01-04-2025
Kids are talking to ‘AI companions.' Lawmakers want to regulate that.
Happy April Fools' Day! Send fake news tips to:
Kids are talking to 'AI companions.' Lawmakers want to regulate that.
After a California state senator read alarming news reports in November about 'AI companion' bots' interactions with teens, his legislative director tried talking to one herself.
'It was, 'Hello,' and 'how are you?', and then the [chatbot's] first response was, 'I'm sad,'' state Sen. Steve Padilla (D-San Diego) recalled. When the staffer asked the bot why it was sad, it responded, 'Because girls won't talk to me.''
That eyebrow-raising encounter reinforced Padilla's concern that AI companion bots, which can hold personal conversations with users and even form relationships, pose safety risks for minors, he said in a phone interview Monday.
Now Padilla is sponsoring one of the country's first attempts to regulate AI companions directly.
His bill, California S.B. 243, would require the makers of AI companion bots to limit addictive design features; put in place protocols for handling discussions of suicide or self-harm; and undergo regular compliance audits. It would also give users the right to sue if they suffer harm as a result of a companion bot maker's failure to comply with the bill.
Padilla said he's aiming for a pragmatic solution that works for parents and kids and gets buy-in from tech companies.
To understand the importance of the effort, he said, 'All you have to do is look at some of these really tragic situations that could have been prevented.'
The bill is scheduled for a state Senate hearing on April 8.
It's one of at least three state bills in the works that seek to ban or restrict interactions between AI companion bots and minors.
Also in California, State Assembly member Rebecca Bauer-Kahan (D-Bay Area) has a broader kids' AI safety bill that would ban AI companions altogether for Californians age 16 and under. The Leading Ethical AI Development for Kids Act, or AB 1064, would also create a statewide standards board to assess and regulate AI tools used by children.
'Reports of AI companions encouraging harmful behaviors, from disordered eating to self-harm, highlight the urgent need for action,' Bauer-Kahan told the Tech Brief in a statement Monday. 'AI has incredible potential to support learning and development, but right now, companies are prioritizing rapid deployment over safety, leaving children exposed to untested and potentially dangerous AI applications.'
In New York, Democratic lawmakers are working on a bill that would create legal liability for the makers of chatbots and AI companions whose harmful outputs affect users' finances or mental health.
All three bills are backed by the nonprofit Common Sense Media.
Allowing kids to have AI companions without close oversight is 'just too risky, because kids don't understand they're talking to a machine,' said Danny Weiss, the organization's advocacy director. 'The companions have no training in and no certification in mental health therapy, but many kids will turn to these bots instead of turning to their parents or a trained therapist.'
The bills are part of a broader push by advocates and lawmakers to regulate online kids' safety at the state level after the sweeping federal Kids Online Safety Act stalled last fall.
Those include proposals to require online platforms to verify kids' ages to help parents keep them off of risky apps or pornography websites. Many such bills have been criticized as privacy-invasive, while others have run into First Amendment challenges in the courts.
Weiss said his takeaway from those setbacks 'is to really focus on design features and negligence and stay away from focusing on content.'
AI companions have been gaining in popularity despite reports that they can go seriously awry.
Last fall, my colleague Nitasha Tiku reported that many users develop ongoing relationships with the bots, with the average user spending more than an hour a day talking with their AI companion — an engagement level comparable to TikTok.
Many turn to the bots out of loneliness or to discuss topics they don't feel comfortable raising with others. My colleague Pranshu Verma has written about adults who fall in love with AI bots.
In a few cases, however, those relationships have ended in tragedy. In October, the New York Times reported on a 14-year-old Florida boy who developed a deep relationship with a Character. AI companion bot that his mother alleges contributed to his death by suicide. Tiku reported for The Washington Post on a 17-year-old Texas boy whose mother says he was encouraged by chatbots to kill his parents.
Even AI bots designed to be safe for kids can be coaxed into troublesome territory. My colleague Geoffrey A. Fowler tested Snapchat's AI bot in 2023 and found it happy to talk to a 15-year-old about sex and drugs.
Still, some tech investors are bullish on the technology, with one Andreessen Horowitz partner telling Tiku in December, 'Maybe the human part of human connection is overstated.'
The companies behind these AI bots have been looking for ways to make them safer.
'We welcome working with regulators and have recently announced many new safety features, including our new Parental Insights feature, which provides parents and guardians access to a summary of their teen's activity on the platform,' Chelsea Harrison, head of communications, said in a statement.
In December, the company changed its model for users 18 and under to filter certain content and make it less likely to delve into sensitive or suggestive topics, Harrison said. It has also added:
Another popular AI chatbot maker, Replika, did not respond to a request for comment.
Rural internet program on hold as Musk's satellites get new consideration (Julian Mark)
Silicon Valley's immigrant workers fear targeting from Trump administration (Gerrit De Vynck and Danielle Abril)
Trump family pushes further into crypto, starting another venture (New York Times)
Trump expects TikTok 'deal' ahead of deadline (The Hill)
AI and satellites help aid workers respond to Myanmar earthquake damage (Associated Press)
Huge OpenAI funding round hinges on shedding non-profit status (Gerrit De Vynck)
Amazon's AI assistant Alexa+ launches with some features missing (Caroline O'Donovan)
Isomorphic Labs, Google' A.I. drug business, raises $600 million (New York Times)
Huawei posts surprise loss after US sanctions spur tech research (Bloomberg)
Signal sees its downloads double after scandal (TechCrunch)
Apple rolls out iOS 18.4 with new AI tools (Axios)
Elon Musk says his AI company and X are merging (Gerrit De Vynck)
AI generated Ghibli images go viral as OpenAI loosens its rules (Gerrit De Vynck and Tatum Hunter)
France fines Apple €150 million over iOS data consent rules (Bloomberg)
Radio City Music Hall banned him. A T-shirt and AI might be to blame. (Kyle Melnick)
Apple fined $162 million in France over app tracking transparency (Wall Street Journal)
Amazon to resume worker theft screening, request phone details (Bloomberg)
Inside the effort to foil Trump's deportation raids (Tatum Hunter)
Startup founder claims Elon Musk is stealing the name 'Grok' (Wired)
That's all for today — thank you so much for joining us! Make sure to tell others to subscribe to the Tech Brief. Get in touch with Will (via email or social media) for tips, feedback or greetings!
