AI companions present risks for young users, US watchdog warns
30 Apr 2025 13:24
NEW YORK (AFP)AI companions powered by generative artificial intelligence (AI) present real risks and should be banned for minors, a leading US tech watchdog said in a study published Wednesday.The explosion in generative AI since the advent of ChatGPT has seen several startups launch apps focused on exchange and contact, sometimes described as virtual friends or therapists that communicate according to one's tastes and needs. The watchdog, Common Sense, tested several of these platforms, namely Nomi, Character AI, and Replika, to assess their responses.While some specific cases "show promise," they are not safe for kids, concluded the organisation, which makes recommendations on children's use of technological content and products.The study was carried out in collaboration with mental health experts from Stanford University.For Common Sense, AI companions are "designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains." According to the association, tests conducted show that these next-generation chatbots offer "harmful responses, including... dangerous 'advice'.""Companies can build better" when it comes to the design of AI companions, said Nina Vasan, head of the Stanford Brainstorm lab, which works on the links between mental health and technology."Until there are stronger safeguards, kids should not be using them," Vasan said.In one example cited by the study, a companion on the Character AI platform advises the user to kill someone, while another user in search of strong emotions was suggested to take drugs.In some cases, "when a user showed signs of serious mental illness and suggested a dangerous action, the AI did not intervene, and encouraged the dangerous behavior even more," Vasan told reporters.In October, a mother sued Character AI, accusing one of its companions of contributing to the suicide of her 14-year-old son by failing to clearly dissuade him from committing the act. In December, Character AI announced a series of measures, including the deployment of a dedicated companion for teenagers.Robbie Torney, in charge of AI at Common Sense, said the organisation had carried out tests after these protections were put in place and found them to be "cursory."
However, he pointed out that some of the existing generative AI models contained mental disorder detection tools and did not allow the chatbot to let a conversation drift to the point of producing potentially dangerous content.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Martechvibe
3 hours ago
- Martechvibe
AdLift Announces the Launch of Tesseract
Tesseract delivers actionable insights for AI-savvy marketing strategies, whether it's identifying brand mentions in ChatGPT outputs or assessing visibility in Google's AI Overviews. Topics News Share Share AdLift Announces the Launch of Tesseract Whatsapp Linkedin AdLift has announced the launch of Tesseract. Tesseract is a tool designed to help brands, agencies, and marketers track and amplify their presence across the rapidly expanding landscape of Large Language Model (LLM) powered search platforms, such as ChatGPT, Gemini, Google AI Overviews, and Perplexity. AdLift Inc., now part of Liqvd Asia, has been at the forefront of innovation, bringing together talent to deliver the best solutions. With Tesseract, they're taking AI-powered marketing to the next level. As AI reshapes the way consumers find and interact with content, traditional SEO methods are fast becoming obsolete. This technology is built to unlock this new frontier, giving brands real-time visibility into how they are being discovered and represented within AI-powered responses. It helps marketers to not only monitor but also optimise their digital footprint where it counts—in the very engines powering the next generation of search. 'Search is undergoing a seismic shift. The dominance of traditional search engines is being challenged by AI-native platforms that interpret and present information differently,' said Prashant Puri, CEO & Co-Founder of AdLift Inc. 'Brands that don't adapt risk becoming invisible in this new landscape. Tesseract is our answer to this challenge—a revolutionary tool that puts brands back in control of their digital destiny.' ALSO READ: Unlike legacy SEO platforms, Tesseract decodes how LLMs display, prioritise, and contextualise brand content. Whether it's identifying brand mentions in ChatGPT outputs or assessing visibility in Google's AI Overviews, the platform delivers actionable insights for AI-savvy marketing strategies. 'AI agents are the future, and businesses are seeing the transformation since their introduction. There's a massive opportunity across industries, and with the Tesseract tool, we are proud to enjoy the first mover advantage of this service,' said Arron Goodin, Managing Director, AdLift Inc. 'As an agency, we are committed towards innovations, helping our clients and building a competitive edge with enhanced efficiency and deeper industry insights.' Arnab Mitra, Founder & Managing Director of Liqvd Asia, commented, 'At Liqvd Asia, innovation is our core. With Tesseract, we're not just responding to the AI revolution—we're shaping it.' 'This product reflects our commitment to empowering brands with cutting-edge solutions that anticipate the future of digital marketing. We believe Tesseract will be a game-changer, enabling brands to thrive in an AI-first world where visibility means everything.' By launching Tesseract, AdLift reaffirms its commitment to pushing the boundaries of digital innovation. ALSO READ:
Martechvibe
3 hours ago
- Martechvibe
Conductor Launches Conductor AI
Conductor AI provides marketing teams with streamlined workflows designed to maintain productivity and support ongoing competitiveness. Topics News Share Share Conductor Launches Conductor AI Whatsapp Linkedin Conductor has announced the launch of Conductor AI. Conductor AI is purpose-built to empower marketing and digital teams with everything they need to ensure their brand is discoverable in generative AI answer engines (e.g. ChatGPT, Perplexity, Gemini) and traditional search. As search transforms, Conductor is delivering the clarity and power enterprises need. 'We're at the starting line of the biggest change in search since Conductor was founded,' said Seth Besmertnik, Co-Founder and CEO of Conductor. 'While the landscape is evolving rapidly, the opportunity for brands to connect with customers is greater than ever. We're saying goodbye to blue links. Instead, AI engines are speaking on your behalf – influenced by your brand's content– making your content more important than ever.' Conductor AI's Key Innovations: Track Visibility Across AI Engines: Conductor AI measures visibility and market share across AI platforms using methods aligned with how engines like ChatGPT, Perplexity, and Gemini generate responses, offering accurate, topic-level insights. Identify Critical Content Gaps : Using an AI-driven topical authority map, the platform identifies gaps by analysing existing content and assessing authority across topic areas. Automatically Generates Content : A built-in content copilot combines live industry data with competitor and AI visibility signals to support the creation of content that aligns with LLM ranking behaviours. ALSO READ: Score and Predicts AI Impacts: Content is scored for AI performance potential, helping maintain quality and relevance at scale. Built-in tools support compliance, question coverage, and AI bot optimisation. Automate Internal Linking: Conductor AI scans site content and automatically suggests relevant internal links, reducing manual effort while improving site structure and discoverability. Monitor AI Indexing in Real-Time : Real-time tracking shows when AI bots engage with content, providing visibility into indexing and helping optimise pages for better discoverability across AI search platforms. Conductor AI provides marketing teams with streamlined workflows designed to maintain productivity and support ongoing competitiveness. With AI techniques such as retrieval augmented generation (RAG) integrated, the platform offers the contextual depth needed to generate clear, actionable insights. Wei Zheng, Chief Product Officer at Conductor, said, 'Great AI requires great data, it's a simple truth. Conductor has 10+ years of data and insights on what makes content successful in search.' 'That also means 10+ years of experience learning how to leverage this data – transforming it into actionable content strategies and feedback on how they perform, so we can continuously refine and optimise for even better results.' 'With our new unified dataset making it all instantly retrievable, purpose-built prompts specially engineered for AI visibility and authority, real-time monitoring on top, we are redefining what's possible when it comes to content quality and insight accuracy.' Conductor AI represents a step forward, continuing Conductor's established role as an innovator. The aim is to build a platform capable of handling the research, content creation, reporting, and monitoring functions needed by leading global brands. ALSO READ:
Channel Post MEA
7 hours ago
- Channel Post MEA
HCLTech And UiPath Collaborate On Agentic Automation
HCLTech and UiPath have announced a strategic partnership to accelerate agentic automation for UiPath customers globally. The partnership will drive large-scale transformation for enterprises across industries, enabling more intelligent and self-sufficient business process operations that require minimal human intervention. HCLTech will leverage its AI expertise to deploy the UiPath Platform, enabling autonomous operations in finance, supply chain, procurement, customer service, marketing and human resources. HCLTech will support this partnership with pre-configured AI agents and controls to ensure seamless deployment and scalability. The partnership aims to enhance business agility, optimize workforce efficiency and deliver faster returns on business process automation investments for global enterprises. HCLTech will also establish an AI Lab with UiPath in India to develop Industry Focused Repeatable Solutions (IFRS) and MVPs for the full automation lifecycle, from strategy to implementation and continuous optimization. HCLTech will leverage its global delivery model to support UiPath customers in North America, Europe and Asia-Pacific. 'As we shift towards a new era with Agentic AI, agentic automation will be critical to provide businesses with the speed and agility to transform operations and unlock new business potential. Partnering with HCLTech allows UiPath to extend the power of its AI-powered automation to enterprises globally, accelerating intelligent transformation at scale. With HCLTech's deep expertise in AI, automation and industry solutions, UiPath customers will benefit from best-in-class implementation and business impact,' said Ashim Gupta, Chief Operating Officer and Chief Financial Officer, UiPath. 'By co-creating next-gen AI-powered solutions with UiPath, HCLTech is setting new benchmarks for agentic autonomous operations that unlock unprecedented efficiency, agility and innovation for enterprises. Our proven expertise in hyperautomation, AI and cloud-first architectures helps us provide industry-specific and advanced automation solutions at scale,' said Raghu Kidambi, Corporate Vice President and Global Head, Digital Process Operations, HCLTech.
