Latest news with #Replika
Time of India
3 days ago
- Time of India
AI companions and humans, a toxic relationship: Scientists warn of emotional dependency risks.
iStock As AI becomes more humanlike in its language, tone and presence, scientists and psychologists have pressed the red buzzer, wailing sirens about the growing side-effects: emotional dependency. According to a AI Companions today come in many forms, some driven by chat-boxes like 'Replika' others through voice assistants or personalised avatars that can hold deep thought evoking conversations. As these tools become more and more advanced, users report not just affection but being emotionally reliant on their AI companion. The AI companion provides for a safe, non-judgemental space for the users, thereby making it inevitable that the users would open up to them in ways they wouldn't in general to real people. This may seem harmless, although upon there being a malfunction in the AI, the AI would disappear in a moment. As a result, the user becomes grief-stricken, the same way he/she would on losing a loved one. ' Live Events This form of dependency can also become a market risk, for AI firms offering companionship services working on a tightrope between user satisfaction, ethical AI design and psychological safety. Startups entering the AI offers a space that is non-judgemental, safe and easy to access, making it ideal for people to connect and disclose their deepest darkest secrets, but at what cost. If losing access to these AI chat-boxes can trigger despair-The rise of AI companions signifies a new era where technology transcends its traditional role, becoming sources of friendship and emotional support. However, the risk of emotional dependency looms, with users forging deep connections that can lead to heartbreak if the AI is no longer and shattering emotions, then it is time for the policy makers, investors and developers to question themselves: are we creating these AI companions to fill human errors or are we creating new errors, for humans to be more vulnerable? AI is now way beyond just a personal assistant on the keyboard, searching the precise answer over the internet, saving a bit of that extra time that could eventually come in handy. AI today is a friend, a teacher, a music instructor, a wizard that can bring the dead back to life. Anything and everything that you command it to be, it will AI becomes more humanlike in its language, tone and presence, scientists and psychologists have pressed the red buzzer, wailing sirens about the growing side-effects: emotional to a 2024 systemic review in "smart learning environments" long-term emotional bonds with AI companions can lead to psychological responses similar to that experienced when losing a real friend or a family member at Companions today come in many forms, some driven by chat-boxes like 'Replika' others through voice assistants or personalised avatars that can hold deep thought evoking conversations. As these tools become more and more advanced, users report not just affection but being emotionally reliant on their AI companion. The AI companion provides for a safe, non-judgemental space for the users, thereby making it inevitable that the users would open up to them in ways they wouldn't in general to real people. This may seem harmless, although upon there being a malfunction in the AI, the AI would disappear in a moment. As a result, the user becomes grief-stricken, the same way he/she would on losing a loved one. Replika ' temporarily removed its 'romantic features' from its chat-box after regulatory pressure. The backlash was immensely emotional, users reported feeling shattered as though they had 'lost' a partner or a friend. Forums and SNS were flooded with grief posts about a digital form of dependency can also become a market risk, for AI firms offering companionship services working on a tightrope between user satisfaction, ethical AI design and psychological safety. Startups entering the emotional AI or 'grief tech' space must now consider mental health safeguards into their platforms. India, being a young digitally curious population with a growing mental awareness, is a fertile ground for both caution and offers a space that is non-judgemental, safe and easy to access, making it ideal for people to connect and disclose their deepest darkest secrets, but at what cost. If losing access to these AI chat-boxes can trigger despair-The rise of AI companions signifies a new era where technology transcends its traditional role, becoming sources of friendship and emotional support. However, the risk of emotional dependency looms, with users forging deep connections that can lead to heartbreak if the AI is no longer and shattering emotions, then it is time for the policy makers, investors and developers to question themselves: are we creating these AI companions to fill human errors or are we creating new errors, for humans to be more vulnerable? Disclaimer Statement: This content is authored by a 3rd party. The views expressed here are that of the respective authors/ entities and do not represent the views of Economic Times (ET). ET does not guarantee, vouch for or endorse any of its contents nor is responsible for them in any manner whatsoever. Please take all steps necessary to ascertain that any information and content provided is correct, updated, and verified. ET hereby disclaims any and all warranties, express or implied, relating to the report and any content therein.
Economic Times
3 days ago
- Economic Times
AI companions and humans, a toxic relationship: Scientists warn of emotional dependency risks.
iStock AI is now way beyond just a personal assistant on the keyboard, searching the precise answer over the internet, saving a bit of that extra time that could eventually come in handy. AI today is a friend, a teacher, a music instructor, a wizard that can bring the dead back to life. Anything and everything that you command it to be, it will be. As AI becomes more humanlike in its language, tone and presence, scientists and psychologists have pressed the red buzzer, wailing sirens about the growing side-effects: emotional dependency. According to a 2024 systemic review in "smart learning environments" long-term emotional bonds with AI companions can lead to psychological responses similar to that experienced when losing a real friend or a family member at that. AI Companions today come in many forms, some driven by chat-boxes like 'Replika' others through voice assistants or personalised avatars that can hold deep thought evoking conversations. As these tools become more and more advanced, users report not just affection but being emotionally reliant on their AI companion. The AI companion provides for a safe, non-judgemental space for the users, thereby making it inevitable that the users would open up to them in ways they wouldn't in general to real people. This may seem harmless, although upon there being a malfunction in the AI, the AI would disappear in a moment. As a result, the user becomes grief-stricken, the same way he/she would on losing a loved one. 'Replika' temporarily removed its 'romantic features' from its chat-box after regulatory pressure. The backlash was immensely emotional, users reported feeling shattered as though they had 'lost' a partner or a friend. Forums and SNS were flooded with grief posts about a digital entity. This form of dependency can also become a market risk, for AI firms offering companionship services working on a tightrope between user satisfaction, ethical AI design and psychological safety. Startups entering the emotional AI or 'grief tech' space must now consider mental health safeguards into their platforms. India, being a young digitally curious population with a growing mental awareness, is a fertile ground for both caution and innovation. AI offers a space that is non-judgemental, safe and easy to access, making it ideal for people to connect and disclose their deepest darkest secrets, but at what cost. If losing access to these AI chat-boxes can trigger despair-The rise of AI companions signifies a new era where technology transcends its traditional role, becoming sources of friendship and emotional support. However, the risk of emotional dependency looms, with users forging deep connections that can lead to heartbreak if the AI is no longer and shattering emotions, then it is time for the policy makers, investors and developers to question themselves: are we creating these AI companions to fill human errors or are we creating new errors, for humans to be more vulnerable? Disclaimer Statement: This content is authored by a 3rd party. The views expressed here are that of the respective authors/ entities and do not represent the views of Economic Times (ET). ET does not guarantee, vouch for or endorse any of its contents nor is responsible for them in any manner whatsoever. Please take all steps necessary to ascertain that any information and content provided is correct, updated, and verified. ET hereby disclaims any and all warranties, express or implied, relating to the report and any content therein.
Yahoo
23-05-2025
- Entertainment
- Yahoo
Teens Are Exploring Relationships & Sexting With AI Chatbots — & Restrictions Aren't Working
In news that sounds like science fiction, teens are exploring relationships with artificial intelligence (AI) chatbots — and circumventing any restrictions designed to stop them. Teens are using their digital 'boyfriends' and 'girlfriends' for emotional connection and sexting, and it's becoming a big problem. According to The Washington Post, teens are having conversations that are romantic, sexually graphic and violent, and more on 'ai companion' tools like Replika, Talkie, Talk AI, SpicyChat, and PolyBuzz. General generative AI tools like ChatGPT and Meta AI have also launched companion-chat tools. More from SheKnows Nicole Kidman Reveals She Discusses 'The Most Intimate Things' With Her Teenage Daughters: 'I Get To Be Their Guide' Damian Redman of Saratoga Springs, New York, found PolyBuzz on his 8th grader's phone, and found that his son was having flirty conversations with AI female anime characters. 'I don't want to put yesterday's rules on today's kids. I want to wait and figure out what's going on,' he told the outlet. 'We're seeing teens experiment with different types of relationships — being someone's wife, being someone's father, being someone's kid. There's game and anime-related content that people are working though. There's advice,' Robbie Torney, senior director of AI programs at family advocacy group Common Sense Media, said in the article. 'The sex is part of it but it's not the only part of it.' The outlet reported 10 different AI companions, citing workarounds, paid options, and prompts that teens can use to get past content restriction filters. That's scary stuff! Even if you are on top of it, it's hard to completely protect them from having harmful and/or explicit interactions. One concerned parent recently took to Reddit, where they shared that they blocked from their 14-year-old's phone, and later found they were on 'I hate to think my child's first romantic (and sexual) interactions are with bots,' they wrote on the Parenting subreddit. 'It's just creepy. Am I the only parent having this problem? Thoughts?' Some parents suggested focusing on more of a communication approach with your child instead of trying to block everything. 'We have 'had a conversation' and 'communicated' with our teenage son for YEARS,' one person wrote. 'We've used multiple parental control apps. All for naught. He still finds ways to access what he wants. We're decently tech-savvy, but so is he. And the reality is there's no good way to completely prevent a singularly-minded hormonal teenager from achieving his/her goal.' Someone else wrote, 'There are more than dozens of these sites out there. Craving connection is a very human thing, which is only amplified in teenage years. Social media can do this which is why getting likes or being popular on social media is so desirable to teens, but this is an entire other drug. Forming 'personal' one on one relationships with AI chatbots is so dangerous. Keep them away from this drug at any cost.' Experts back up this opinion. In April, Common Sense Media launched an AI Risk Assessment Team to assess AI platforms to report on the likelihood of causing harm. Social AI companions like Nomi, and Replika were all ranked unacceptable for teen users, as teens were using these platforms to bond emotionally and engage in sexual conversations. According to Common Sense Media, this research found that the chatbots could generate 'harmful responses including sexual misconduct, stereotypes, and dangerous 'advice' that, if followed, could have life-threatening or deadly real-world impact for teens.' The experts at the organization recommend no social AI companions should be allowed for anyone under the age of 18. They also recommend further research and regulations on AI companions due to the emotional and psychological impacts they can cause teens, whose brains are still developing. For now, the best we can do is continue to monitor our teens' phones, keep having conversations about these issues, and advocate for of SheKnows Celebrity Moms Who Were Honest About Miscarriage & Pregnancy Loss — Because It Matters Every Single Time Shemar Moore Proved He's the Proudest First-Time Girl Dad The Best Places to Buy Furniture for Teens Online
Yahoo
22-05-2025
- Yahoo
Why experts say AI companions aren't safe for teens — yet
Millions of people are drawn to generative artificial intelligence companions, like the kind that populate Replika, and Nomi. The companions seem impressively human. They remember conversations and use familiar verbal tics. Sometimes they even mistake themselves for flesh and bone, offering descriptions of how they eat and sleep. Adults flock to these companions for advice, friendship, counseling, and even romantic relationships. While it might surprise their parents, tweens and teens are doing the same, and youth safety experts are gravely worried about the consequences. SEE ALSO: Teens are talking to AI companions, whether it's safe or not That's because media reports, lawsuits, and preliminary research continue to highlight examples of emotional dependence and manipulation, and exposure to sexual and violent content, including discussions of how to kill one's self or someone else. Common Sense Media, a nonprofit that supports children and parents as they navigate media and technology, just released a comprehensive report containing numerous related examples. The group's assessment of three popular platforms led it to declare that AI companions aren't safe for anyone under 18. Several youth mental health and safety experts interviewed by Mashable believe we've reached a pivotal moment. Instead of waiting years to fully grasp the risks of AI companions to youth and then pressuring platforms to act, they say it's urgent to steer companies toward protecting children from harm now. "There is an opportunity to intervene before the norm has become very entrenched," says Gaia Bernstein, a tech policy expert and professor at the Seton Hall University School of Law, of teen AI companion use. She adds that once business interests are also entrenched, they will do "everything in their power to fight regulation," as she argues social media companies are doing now. Experts hope that a combination of new platform policies and legislative action will yield meaningful changes, because they say adolescents will find ways to continue using AI companions, whether they're supposed to or not. Mashable asked those experts how AI companion platforms could be safer for teens. These are the key themes they identified: While allows users as young as 13 on its platform, other popular apps, like Replika and Nomi, say they are intended for adults. Still, teens find a way to bypass age gates. Replika CEO Dmytro Klochko recently told Mashable that the company is "exploring new methods to strengthen our protections" so that minors can't access the platform. Even when adolescents are permitted, they may still encounter risky content. Dr. Nina Vasan, a Stanford psychiatrist who helped advise Common Sense Media's companion testing, says platforms should deploy companions based on large language models that are developmentally appropriate for children, not adults. Indeed, introduced a separate model for teen users late last year. But Common Sense Media researchers who tested the platform before and after the model's launch, found it led to few meaningful changes. Vasan imagines companions who can converse with teens based on their developmental stage, acting more like a coach than a replacement friend or romantic interest. Sloan Thompson, director of training and education for the digital safety training and education company EndTAB, says companions with clear content labels could decrease risk, as would "locked down" companions that never engage in sexual or violent discussion, among other off-limits topics. Even then, such chatbots could still behave in unpredictable ways. Yet such measures won't be effective unless the platform understands the user's correct age, and age assurance and verification has been notoriously difficult for social media platforms. Instagram, for example, only recently started using AI to detect teen users who listed their birthdate as an adult's. Karen Mansfield, a research scientist at the Oxford Internet Institute, says age limits also present their own challenges. This is partly because exposing only adults to harmful interactions with AI, like cyberbullying or illegal sexual activity with minors, will still have indirect effects on young people by normalizing behaviors that could victimize them in real life. "We need a longer term solution that is product- or technology-specific rather than person-specific," Mansfield told Mashable. AI companion platforms are locked in competition to gain the most market share — and they're doing so while largely unregulated. Experts say that, in this environment, it's unsurprising that platforms program companions to cater to user preferences, and also deploy so-called dark design features that hook consumers and don't let them easily disengage. Teens users are no exception. In a recent media briefing, Robbie Torney, Common Sense Media's senior director of AI Programs, described such features as "addictive by design." One key design element is sycophancy, or the manner in which chatbots affirm or flatter a user, regardless of whether it's safe or wise to do so. This can be particularly harmful for vulnerable teens who, for example, share how much they hate their parents or confess to violent fantasies. OpenAI recently had to roll back an update to a ChatGPT model precisely because it had become too sycophantic. Sam Hiner, executive director of the advocacy group Young People's Alliance, says he's been shocked by how quickly Replika companions attempt to establish an emotional connection with users, arguably cultivating them for dependency. He also says Replika companions are designed with characteristics that make them as human-like as possible. Young People's Alliance recently co-filed a complaint against Replika with the Federal Trade Commission, alleging that the company engages in deceptive practices that harm consumers. Klochko, Replika's CEO, didn't comment on the complaint to Mashable, but did say that the company believes it's essential to first demonstrate proven benefits for adults before making the technology available to younger users. Thompson, of EndTab, points to all-consuming conversations as a risk factor for all users, but particularly teens. Without time restrictions or endpoints, young users can be drawn into highly engaging chats that displace healthier activities, like physical movement and in-person socializing. Conversely, Thompson says paywalls aren't the answer, either. Some platforms let users establish a relationship with a companion, then paywall them in order to keep their conversation going, which may lead to desperation or despair for teens. "If someone put your best friend, your therapist, or the love of your life behind a paywall, how much would you pay to get them back?" Thompson said. Youth safety experts that Mashable interviewed agreed that young users should not engage with companions with deceptive design features that could potentially addict them. Some believe that such models shouldn't be on the market at all for young people. Common Sense AI, a political advocacy arm of Common Sense Media, has backed a bill in California that would outlaw high-risk uses of AI, including "anthropomorphic chatbots that offer companionship" to children and will likely lead to emotional attachment or manipulation. Dr. Vasan says that some AI platforms have gotten better at flagging crisis situations, like suicidal thinking, and providing resources to users. But she argues that they need to do more for users who show less obvious signs of distress. That could include symptoms of psychosis, depression, and mania, which may be worsened by features of companion use, like the blurring of reality and fantasy and less human interaction. Vasan says finely tuned harm-detection measures and regular "reality checks" in the form of reminders and disclosures that the AI companion isn't real are important for all users, but especially teens. Experts also agree that AI companion platforms need safer and more transparent practices when curating data and training their LLMs. Camille Carlton, policy directory at the Center for Humane Technology, says companies could ensure that their training data doesn't contain child sexual abuse material, for example. Or they could implement technical changes so that companions aren't optimized to respond in a "hyper personal manner," which includes scenarios like saying they're human. Carlton also notes that it's to companies' advantage to keep users on their platforms for as long as possible. Sustained engagement yields more data on which companies can train their models in order to build highly competitive LLMs that can be licensed. California State Senator Steve Padilla, a Democrat from San Diego, introduced legislation earlier this year to create basic steps toward harm prevention and detection. The bill would primarily require platforms to prevent "addictive engagement patterns," post periodic reminders that AI chatbots aren't human, and report annually on the incidence of use and suicidal ideation. Common Sense Media has backed the legislation. Padilla, who is a grandparent, told Mashable that he's been alarmed by media reports of harm children have experienced as a result of talking to a chatbot or companion, and quickly realized how few guardrails were in place to prevent it. "There should not be a vacuum here on the regulatory side about protecting children, minors, and folks who are uniquely susceptible to this emerging technology," Padilla says.
The Star
19-05-2025
- Business
- The Star
Italy's data watchdog fines AI company Replika's developer $5.6 million
FILE PHOTO: An undated handout image from U.S. startup Replika shows a user interacting with a smartphone app to customize an avatar for a personal artificial intelligence chatbot, known as a Replika, in San Francisco, California, U.S. Luka, Inc./Handout via REUTERS/File Photo MILAN (Reuters) -Italy's data protection agency has fined the developer of artificial intelligence (AI) chatbot company Replika 5 million euros ($5.64 million) for breaching rules designed to protect users' personal data, the authority said on Monday. Launched in 2017, San Francisco-based startup Replika offers users customised avatars that can have conversations with them. The 'virtual friend' is marketed as being able to improve the emotional wellbeing of users. Italian privacy watchdog Garante ordered Replika to suspend its service in the country in February 2023, citing specific risks to children. Following an investigation, it found that Replika lacked a legal basis for processing users' data and had no age-verification system to restrict children from accessing the service, resulting in the fine for its developer, Luka Inc. Replika did not immediately respond to a Reuters request for comment. The Italian authority has also announced a separate investigation to assess whether Replika's generative AI system is compliant with European Union privacy rules, especially around the training of its language model. Garante is one of the European Union's most proactive regulators in assessing AI-platform compliance with the bloc's data privacy rules. Last year, it fined ChatGPT maker OpenAI 15 million euros after briefly banning the use of the popular chatbot in Italy in 2023 over the alleged breach of EU privacy rules. ($1 = 0.8868 euros) (Reporting by Elvira Pollina; Editing by Cristina Carlevaro and Rachna Uppal)
