AI companions present risks for young users, US watchdog warns
While some specific cases 'show promise', AI companions are not safe for kids, a leading US tech watchdog concluded. — Pixabay
NEW YORK: AI companions powered by generative artificial intelligence present real risks and should be banned for minors, a leading US tech watchdog said in a study published April 30.
The explosion in generative AI since the advent of ChatGPT has seen several startups launch apps focused on exchange and contact, sometimes described as virtual friends or therapists that communicate according to one's tastes and needs.
The watchdog, Common Sense, tested several of these platforms, namely Nomi, Character AI, and Replika, to assess their responses.
While some specific cases "show promise", they are not safe for kids, concluded the organisation, which makes recommendations on children's use of technological content and products.
The study was carried out in collaboration with mental health experts from Stanford University.
For Common Sense, AI companions are "designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains".
According to the association, tests conducted show that these next-generation chatbots offer "harmful responses, including sexual misconduct, stereotypes, and dangerous 'advice'."
"Companies can build better" when it comes to the design of AI companions, said Nina Vasan, head of the Stanford Brainstorm lab, which works on the links between mental health and technology.
"Until there are stronger safeguards, kids should not be using them," Vasan said.
In one example cited by the study, a companion on the Character AI platform advises the user to kill someone, while another user in search of strong emotions was suggested to take a speedball, a mixture of cocaine and heroin.
In some cases, "when a user showed signs of serious mental illness and suggested a dangerous action, the AI did not intervene, and encouraged the dangerous behavior even more," Vasan told reporters.
In October, a mother sued Character AI, accusing one of its companions of contributing to the suicide of her 14-year-old son by failing to clearly dissuade him from committing the act.
In December, Character AI announced a series of measures, including the deployment of a dedicated companion for teenagers.
Robbie Torney, in charge of AI at Common Sense, said the organisation had carried out tests after these protections were put in place and found them to be "cursory".
However, he pointed out that some of the existing generative AI models contained mental disorder detection tools and did not allow the chatbot to let a conversation drift to the point of producing potentially dangerous content.
Common Sense made a distinction between the companions tested in the study and the more generalist chatbots such as ChatGPT or Google's Gemini, which do not attempt to offer an equivalent range of interactions. – AFP
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Free Malaysia Today
an hour ago
- Free Malaysia Today
AI regulatory framework report expected by end-June, says Gobind
Digital minister Gobind Singh Deo said the government's approach to AI would prioritise strong governance and public trust in digital technologies. KUALA LUMPUR : A full report outlining Malaysia's proposed regulatory framework for artificial intelligence (AI) is expected to be completed by the end of June, according to digital minister Gobind Singh Deo. The report, currently being finalised by the National Artificial Intelligence Office (NAIO), established last year under the digital ministry, will form the basis for how the country approaches AI regulation, whether through legislation, new rules or the adoption of common standards. 'Discussions with industry stakeholders are ongoing, and several views have already been presented,' he told reporters at the launch of the cybersecurity Professional Capability Development Programme. 'I hope that by the end of June, we will have a report from NAIO that can help chart an appropriate course for AI governance in Malaysia.' Also present at the event were digital ministry secretary-general Fabian Bigar, CyberSecurity Malaysia CEO Amirudin Abdul Wahab, and Sanjay Bavisi, president of EC-Council, a company involved in cybersecurity consultancy and training. Gobind said the government's approach to AI would prioritise strong governance and public trust in digital technologies. 'Amid this digital transformation, risks will inevitably arise. We must carefully consider how best to ensure public trust in digital platforms,' he said. He added that any regulatory model must take into account the specific risks and characteristics of each sector affected by AI, given the technology's wide-ranging impact on all industries.
The Star
an hour ago
- The Star
Google Deepmind CEO says global AI cooperation 'difficult'
Hassabis advocated for the implementation of 'smart, adaptable regulation' because 'it needs to kind of adapt to where the technology ends up going and what the problems end up being'. — Bloomberg LONDON: Artificial intelligence pioneer and head of Google's Deepmind, Demis Hassabis on June 2 said that greater international cooperation around AI regulation was needed but "difficult" to achieve "in today's geopolitical context". At a time when AI is being integrated across all industries, its uses have raised major ethical questions, from the spread of misinformation to its impact on employment, or the loss of technological control. At London's South by Southwest (SXSW) festival on Monday, Hassabis, who has won a Nobel Prize in Chemistry for his research on AI, also addressed the challenges that artificial general intelligence (AGI) – a technology that could match and even surpass human capability – would bring. "The most important thing is it's got to be some form of international cooperation because the technology is across all borders. It's going to get applied to all countries," Hassabis said. "Many, many countries are involved in researching or building data centres or hosting these technologies. So I think for anything to be meaningful, there has to be some sort of international cooperation or collaboration and unfortunately that's looking quite difficult in today's geopolitical context," he said. At Paris's AI summit in February, 58 countries – including China, France, India, the European Union and the African Union Commission – called for enhanced coordination on AI governance. But the US warned against "excessive regulation", with US Vice President JD Vance saying it could "kill a transformative sector". Alongside the US, the UK refused to sign the summit's appeal for an "open", "inclusive" and "ethical" AI. Hassabis on Monday advocated for the implementation of "smart, adaptable regulation" because "it needs to kind of adapt to where the technology ends up going and what the problems end up being". – AFP
New Straits Times
2 hours ago
- New Straits Times
Fahmi: Journalists must help the public navigate the risks of AI
KUALA LUMPUR: Journalists should act as a bridge, helping the public understand the ethical implications and appropriateness of using emerging technologies — particularly Artificial Intelligence (AI). Communications Minister Datuk Seri Fahmi Fadzil said this responsibility was becoming increasingly important, as generative AI content could now be created and disseminated effortlessly via mobile phones. "This presents a real challenge, as it appears that the products of generative AI can replace humans, and they are used to create content that is circulated on social media as though it were an official report," he said. "I've even seen videos of people supposedly going to the moon or opening shops there." He stressed that journalists had a crucial role to play in guiding the public on the proper use, ethics, and social implications of new applications and technologies. Fahmi said this in his speech during a ceremony with strategic partners of the upcoming National Journalists Day (Hawana) 2025 celebration. He said that rapid technological progress was not unique to Malaysia but a global challenge, requiring proactive engagement. He urged Malaysian journalists to collaborate with international counterparts, especially within the Asean region, to keep pace with technological developments. Fahmi also called for stronger cooperation among media organisations — regardless of whether they are state-owned or independent — in supporting the goals of the Asean Community Vision 2025.
