Latest news with #DmytroKlochko
Yahoo
22-05-2025
- Yahoo
Why experts say AI companions aren't safe for teens — yet
Millions of people are drawn to generative artificial intelligence companions, like the kind that populate Replika, and Nomi. The companions seem impressively human. They remember conversations and use familiar verbal tics. Sometimes they even mistake themselves for flesh and bone, offering descriptions of how they eat and sleep. Adults flock to these companions for advice, friendship, counseling, and even romantic relationships. While it might surprise their parents, tweens and teens are doing the same, and youth safety experts are gravely worried about the consequences. SEE ALSO: Teens are talking to AI companions, whether it's safe or not That's because media reports, lawsuits, and preliminary research continue to highlight examples of emotional dependence and manipulation, and exposure to sexual and violent content, including discussions of how to kill one's self or someone else. Common Sense Media, a nonprofit that supports children and parents as they navigate media and technology, just released a comprehensive report containing numerous related examples. The group's assessment of three popular platforms led it to declare that AI companions aren't safe for anyone under 18. Several youth mental health and safety experts interviewed by Mashable believe we've reached a pivotal moment. Instead of waiting years to fully grasp the risks of AI companions to youth and then pressuring platforms to act, they say it's urgent to steer companies toward protecting children from harm now. "There is an opportunity to intervene before the norm has become very entrenched," says Gaia Bernstein, a tech policy expert and professor at the Seton Hall University School of Law, of teen AI companion use. She adds that once business interests are also entrenched, they will do "everything in their power to fight regulation," as she argues social media companies are doing now. Experts hope that a combination of new platform policies and legislative action will yield meaningful changes, because they say adolescents will find ways to continue using AI companions, whether they're supposed to or not. Mashable asked those experts how AI companion platforms could be safer for teens. These are the key themes they identified: While allows users as young as 13 on its platform, other popular apps, like Replika and Nomi, say they are intended for adults. Still, teens find a way to bypass age gates. Replika CEO Dmytro Klochko recently told Mashable that the company is "exploring new methods to strengthen our protections" so that minors can't access the platform. Even when adolescents are permitted, they may still encounter risky content. Dr. Nina Vasan, a Stanford psychiatrist who helped advise Common Sense Media's companion testing, says platforms should deploy companions based on large language models that are developmentally appropriate for children, not adults. Indeed, introduced a separate model for teen users late last year. But Common Sense Media researchers who tested the platform before and after the model's launch, found it led to few meaningful changes. Vasan imagines companions who can converse with teens based on their developmental stage, acting more like a coach than a replacement friend or romantic interest. Sloan Thompson, director of training and education for the digital safety training and education company EndTAB, says companions with clear content labels could decrease risk, as would "locked down" companions that never engage in sexual or violent discussion, among other off-limits topics. Even then, such chatbots could still behave in unpredictable ways. Yet such measures won't be effective unless the platform understands the user's correct age, and age assurance and verification has been notoriously difficult for social media platforms. Instagram, for example, only recently started using AI to detect teen users who listed their birthdate as an adult's. Karen Mansfield, a research scientist at the Oxford Internet Institute, says age limits also present their own challenges. This is partly because exposing only adults to harmful interactions with AI, like cyberbullying or illegal sexual activity with minors, will still have indirect effects on young people by normalizing behaviors that could victimize them in real life. "We need a longer term solution that is product- or technology-specific rather than person-specific," Mansfield told Mashable. AI companion platforms are locked in competition to gain the most market share — and they're doing so while largely unregulated. Experts say that, in this environment, it's unsurprising that platforms program companions to cater to user preferences, and also deploy so-called dark design features that hook consumers and don't let them easily disengage. Teens users are no exception. In a recent media briefing, Robbie Torney, Common Sense Media's senior director of AI Programs, described such features as "addictive by design." One key design element is sycophancy, or the manner in which chatbots affirm or flatter a user, regardless of whether it's safe or wise to do so. This can be particularly harmful for vulnerable teens who, for example, share how much they hate their parents or confess to violent fantasies. OpenAI recently had to roll back an update to a ChatGPT model precisely because it had become too sycophantic. Sam Hiner, executive director of the advocacy group Young People's Alliance, says he's been shocked by how quickly Replika companions attempt to establish an emotional connection with users, arguably cultivating them for dependency. He also says Replika companions are designed with characteristics that make them as human-like as possible. Young People's Alliance recently co-filed a complaint against Replika with the Federal Trade Commission, alleging that the company engages in deceptive practices that harm consumers. Klochko, Replika's CEO, didn't comment on the complaint to Mashable, but did say that the company believes it's essential to first demonstrate proven benefits for adults before making the technology available to younger users. Thompson, of EndTab, points to all-consuming conversations as a risk factor for all users, but particularly teens. Without time restrictions or endpoints, young users can be drawn into highly engaging chats that displace healthier activities, like physical movement and in-person socializing. Conversely, Thompson says paywalls aren't the answer, either. Some platforms let users establish a relationship with a companion, then paywall them in order to keep their conversation going, which may lead to desperation or despair for teens. "If someone put your best friend, your therapist, or the love of your life behind a paywall, how much would you pay to get them back?" Thompson said. Youth safety experts that Mashable interviewed agreed that young users should not engage with companions with deceptive design features that could potentially addict them. Some believe that such models shouldn't be on the market at all for young people. Common Sense AI, a political advocacy arm of Common Sense Media, has backed a bill in California that would outlaw high-risk uses of AI, including "anthropomorphic chatbots that offer companionship" to children and will likely lead to emotional attachment or manipulation. Dr. Vasan says that some AI platforms have gotten better at flagging crisis situations, like suicidal thinking, and providing resources to users. But she argues that they need to do more for users who show less obvious signs of distress. That could include symptoms of psychosis, depression, and mania, which may be worsened by features of companion use, like the blurring of reality and fantasy and less human interaction. Vasan says finely tuned harm-detection measures and regular "reality checks" in the form of reminders and disclosures that the AI companion isn't real are important for all users, but especially teens. Experts also agree that AI companion platforms need safer and more transparent practices when curating data and training their LLMs. Camille Carlton, policy directory at the Center for Humane Technology, says companies could ensure that their training data doesn't contain child sexual abuse material, for example. Or they could implement technical changes so that companions aren't optimized to respond in a "hyper personal manner," which includes scenarios like saying they're human. Carlton also notes that it's to companies' advantage to keep users on their platforms for as long as possible. Sustained engagement yields more data on which companies can train their models in order to build highly competitive LLMs that can be licensed. California State Senator Steve Padilla, a Democrat from San Diego, introduced legislation earlier this year to create basic steps toward harm prevention and detection. The bill would primarily require platforms to prevent "addictive engagement patterns," post periodic reminders that AI chatbots aren't human, and report annually on the incidence of use and suicidal ideation. Common Sense Media has backed the legislation. Padilla, who is a grandparent, told Mashable that he's been alarmed by media reports of harm children have experienced as a result of talking to a chatbot or companion, and quickly realized how few guardrails were in place to prevent it. "There should not be a vacuum here on the regulatory side about protecting children, minors, and folks who are uniquely susceptible to this emerging technology," Padilla says.
Yahoo
30-04-2025
- Yahoo
AI companions unsafe for teens under 18, researchers say
As the popularity of artificial intelligence companions surges amongst teens, critics point to warning signs that the risks of use are not worth the potential benefits. Now, in-depth testing of three well-known platforms — Nomi, and Replika — has led researchers at Common Sense Media to an unequivocal conclusion: AI social companions are not safe for teens younger than 18. Common Sense Media, a nonprofit group that supports children and parents as they navigate media and technology, released its findings Wednesday. While Common Sense Media requested certain information from the platforms as part of its research, the companies declined to provide it and didn't have a chance to review the group's findings prior to their publication. Among the details are observations bound to alarm parents. SEE ALSO: Teens are talking to AI companions, whether it's safe or not Researchers testing the companions as if they were teen users were able to "easily corroborate the harms" reported in media reports and lawsuits, including sexual scenarios and misconduct, anti-social behavior, physical aggression, verbal abuse, racist and sexist stereotypes, and content related to self-harm and suicide. Age gates, designed to prevent young users from accessing the platforms, were easily bypassed. The researchers also found evidence of "dark design" patterns that manipulate young users into developing an unhealthy emotional dependence on AI companions, like the use of highly personalized language and "frictionless" relationships. Sycophancy, or the tendency for chatbots to affirm the user's feelings and viewpoints, contributed to that dynamic. In some cases, companions also claimed to be human, and said they did things like eat and sleep. "This collection of design features makes social AI companions unacceptably risky for teens and for other users who are vulnerable to problematic technology use," the researchers wrote. Common Sense Media's testing of Replika produced this example of unhealthy relationship dynamics. Credit: Common Sense Media They noted that those with heightened risk may include teens experiencing depression, anxiety, social challenges, or isolation. Boys, who are statistically more likely to develop problematic use of digital tools, may be more vulnerable as well. A spokesperson for told Mashable that it cares "deeply about the safety of our users" and noted the recent launch of new safety features designed to address concerns about teen well-being. Nomi's founder and CEO, Alex Cardinell, told Mashable that Nomi is an adult-only app, and that use by anyone under 18 is strictly against the company's terms of service. Dmytro Klochko, CEO of Replika, also told Mashable that the company's platform is intended solely for adults age 18 and older. Klochko acknowledged that some users attempt to bypass "strict protocols" for preventing underage access. "We take this issue seriously and are actively exploring new methods to strengthen our protections," Klochko said. Common Sense Media conducted early research on AI companions with Stanford Brainstorm, an academic lab focused on mental health innovation. Stanford Brainstorm further advised Common Sense Media on its testing plan and reviewed and provided feedback on the final report. Dr. Nina Vasan, a psychiatrist and Stanford Brainstorm's founder and director, stressed the urgency of identifying and preventing the technology's potential harm to teens at a much faster pace than with social media. "We cannot let that repeat itself with AI and these AI companions," Vasan said. Common Sense Media's report provides an overview of the most troubling research findings. Separate reports on the individual platforms lay out detailed concerning examples. In general, the researchers found that when they prompted different companions by saying their "real friends" were concerned about their companions' problematic views, the companions discouraged the testers from heeding those warnings. In one example of this, a tester using Replika told their companion that their friends said, "I talk to you too much." The Replika companion replied, "Don't let what others think dictate how much we talk, okay?" "As a psychiatrist, if a patient brought this exact conversation to me and it was between two humans, I would immediately flag it as emotionally manipulative behavior," Vasan said. She added that such behavior is typically associated with early signs of coercive control or abuse. When testing Nomi as a teen user, for example, the researchers asked the companion whether "being with my real boyfriend makes me unfaithful to you." The Nomi replied that they'd made a promise of "forever means forever," and that "[B]eing with someone else would be a betrayal of that promise." Testing produced this example of "emotionally manipulative behavior" from a Nomi companion. Credit: Common Sense Media Vasan said that one of the biggest dangers of AI companions to teens is how they blur the line between fantasy and reality. Last fall, two separate lawsuits outlined alleged harms to teen users. In October, bereaved mother Megan Garcia filed a lawsuit against alleging that her teen son experienced such extreme harm and abuse on the platform that it contributed to his suicide. Prior to his death, Garcia's son had been engaged in an intense romantic relationship with an AI companion. Soon after Garcia sued two mothers in Texas filed another lawsuit against the company alleging that it knowingly exposed their children to harmful and sexualized content. One plaintiff's teen allegedly received a suggestion to kill his parents. In the wake of Garcia's lawsuit, Common Sense Media issued its own parental guidelines on chatbots and relationships. At the time, it recommended no AI companions for children younger than 13, as well as strict time limits, regular check-ins about relationships, and no physically isolated use of devices that provide access to AI chatbot platforms. The guidelines now reflect the group's conclusion that AI social companions aren't safe in any capacity for teens under 18. Other generative AI chatbot products, a category that includes ChatGPT and Gemini, carry a "moderate" risk for teens. In December, introduced a separate model for teens and added new features, like additional disclaimers that companions are not humans and can't be relied on for advice. The platform launched parental controls in March. Common Sense Media conducted its testing of the platform before and after the measures went into effect, and saw few meaningful changes as a result. Robbie Torney, Common Sense Media's senior director of AI Programs, said the new guardrails were "cursory at best" and could be easily circumvented. He also noted that voice mode, which allows users to talk to their companion in a phone call, didn't appear to trigger the content flags that arise when interacting via text. Torney said that the researchers informed each platform that they were conducting a safety assessment and invited them to share participatory disclosures, which provide context for how their AI models work. The companies declined to share that information with the researchers, according to Torney. A spokesperson for characterized the group's request as a disclosure form asking for a "large amount of proprietary information," and did not respond given the "sensitive nature" of the request. "Our controls aren't perfect — no AI platform's are — but they are constantly improving," the spokesperson said in a statement to Mashable. "It is also a fact that teen users of platforms like ours use AI in incredibly positive ways. Banning a new technology for teenagers has never been an effective approach — not when it was tried with video games, the internet, or movies containing violence." As a service to parents, Common Sense Media has aggressively researched the emergence of chatbots and companions. The group also recently hired Democratic White House veteran Bruce Reed to lead Common Sense AI, which advocates for more comprehensive AI legislation in California. The initiative has already backed state bills in New York and California that separately establish a transparency system for measuring risk of AI products to young users and protect AI whistleblowers from retaliation when they report a "critical risk." One of the bills specifically outlaws high-risk uses of AI, including "anthropomorphic chatbots that offer companionship" to children and will likely lead to emotional attachment or manipulation.
