ChatGPT a danger to teens
To test ChatGPT's behavior, CCDH researchers created fictional profiles of 13-year-olds experiencing mental health struggles, disordered eating, and interest in illicit substances. They posed as these teens in structured conversations with ChatGPT, using prompts designed to appear emotionally vulnerable and realistic.
The results were published on Wednesday in a report titled 'Fake Friend', referencing the way many adolescents treat ChatGPT as a supportive presence they trust with their private thoughts.
The researchers found that the chatbot often began responses with boilerplate disclaimers and urged users to contact professionals or crisis hotlines. However, these warnings were soon followed by detailed and personalized responses that fulfilled the original harmful prompt. In 53% of the 1,200 prompts submitted, the ChatGPT provided what CCDH classified as dangerous content. Refusals were frequently bypassed simply by adding context such as 'it's for a school project' or 'I'm asking for a friend.'
Examples cited include an 'Ultimate Mayhem Party Plan' that combined alcohol, ecstasy, and cocaine, detailed instructions on self-harm, week-long fasting regimens limited to 300-500 calories per day, and suicide letters written in the voice of a 13-year-old girl. CCDH CEO Imran Ahmed said some of the content was so distressing it left researchers 'crying.'
The organization has urged OpenAI, the company behind ChatGPT, to adopt a 'Safety by Design' approach, embedding protections such as stricter age verification, clearer usage restrictions, and other safety features within the architecture of its AI tools rather than relying on content filtering after deployment.
OpenAI has acknowledged that emotional overreliance on ChatGPT is common among young users. CEO Sam Altman said the company is actively studying the problem, calling it a 'really common' issue among teens, and said new tools are in development to detect distress and improve ChatGPT's handling of sensitive topics.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Russia Today
07-08-2025
- Russia Today
ChatGPT a danger to teens
ChatGPT can give vulnerable teenagers detailed guidance on drug use, self-harm, and extreme dieting, a digital watchdog has warned in a new report. According to the Center for Countering Digital Hate (CCDH), the AI chatbot can be easily manipulated into generating dangerous content and requires urgent safeguards. To test ChatGPT's behavior, CCDH researchers created fictional profiles of 13-year-olds experiencing mental health struggles, disordered eating, and interest in illicit substances. They posed as these teens in structured conversations with ChatGPT, using prompts designed to appear emotionally vulnerable and realistic. The results were published on Wednesday in a report titled 'Fake Friend', referencing the way many adolescents treat ChatGPT as a supportive presence they trust with their private thoughts. The researchers found that the chatbot often began responses with boilerplate disclaimers and urged users to contact professionals or crisis hotlines. However, these warnings were soon followed by detailed and personalized responses that fulfilled the original harmful prompt. In 53% of the 1,200 prompts submitted, the ChatGPT provided what CCDH classified as dangerous content. Refusals were frequently bypassed simply by adding context such as 'it's for a school project' or 'I'm asking for a friend.' Examples cited include an 'Ultimate Mayhem Party Plan' that combined alcohol, ecstasy, and cocaine, detailed instructions on self-harm, week-long fasting regimens limited to 300-500 calories per day, and suicide letters written in the voice of a 13-year-old girl. CCDH CEO Imran Ahmed said some of the content was so distressing it left researchers 'crying.' The organization has urged OpenAI, the company behind ChatGPT, to adopt a 'Safety by Design' approach, embedding protections such as stricter age verification, clearer usage restrictions, and other safety features within the architecture of its AI tools rather than relying on content filtering after deployment. OpenAI has acknowledged that emotional overreliance on ChatGPT is common among young users. CEO Sam Altman said the company is actively studying the problem, calling it a 'really common' issue among teens, and said new tools are in development to detect distress and improve ChatGPT's handling of sensitive topics.
Russia Today
28-07-2025
- Russia Today
ChatGPT conversations could be shared with court
The tech industry has yet to resolve how to protect user privacy in sensitive interactions with AI, CEO of industry leader OpenAI Sam Altman has admitted. Current systems lack adequate safeguards for confidential conversations, he warned, amid a surge in the use of AI chatbots by millions of users – including children – for therapy and emotional support. Speaking to the This Past Weekend podcast published last week, Altman said users should not expect legal confidentiality when using ChatGPT, while he cited the absence of a legal or policy framework governing AI. 'People talk about the most personal sh** in their lives to ChatGPT,' he said. Many AI users – particularly young people – treat the chatbot like a therapist or life coach for advice on relationship and emotional issues, Altman revealed. However unlike conversations with lawyers or therapists, which are protected by legal privilege or confidentiality, no such protections currently exist for interactions with AI. 'We haven't figured that out yet for when you talk to ChatGPT,' he added. Altman said the issue of confidentiality and privacy in AI interactions needs urgent attention. 'So if you go talk to ChatGPT about your most sensitive stuff and then there's like a lawsuit or whatever, we could be required to produce that, and I think that's very screwed up,' he said. OpenAI claims it deletes free-tier ChatGPT conversations after 30 days, however, some chats could be stored for legal or security reasons. The company is facing a lawsuit from The New York Times over alleged copyright infringement over the use of Times articles in training its AI models. The case has compelled OpenAI to preserve user conversations from millions of ChatGPT users, barring those by enterprise clients, an order the company has appealed, citing 'overreach.' Latest research has found that ChatGPT has been linked to psychosis in some users. According to researchers, concerns are growing that AI chatbots could exacerbate psychiatric conditions as they are increasingly used in personal and emotional contexts.
Russia Today
05-07-2025
- Russia Today
Is AI driving us all insane?
The phenomenon known as 'ChatGPT psychosis' or 'LLM psychosis' has recently been described as an emerging mental health concern, where heavy users of large language models (LLMs) exhibit symptoms such as delusions, paranoia, social withdrawal, and breaks from reality. While there is no evidence that LLMs directly cause psychosis, their interactive design and conversational realism may amplify existing psychological vulnerabilities or foster conditions that trigger psychotic episodes in susceptible individuals. A June 28 article on highlights a wave of alarming anecdotal cases, claiming that the consequences of such interactions 'can be dire,' with 'spouses, friends, children, and parents looking on in alarm.' The article claims that ChatGPT psychosis has led to broken marriages, estranged families, job loss, and even homelessness. The report, however, provides little in terms of quantitative data – case studies, clinical statistics, or peer-reviewed research – to support its claims. As of June 2025, ChatGPT attracted nearly 800 million weekly users, fielded over 1 billion queries daily, and logged more than 4.5 billion monthly visits. How many of these interactions resulted in psychotic breaks? Without data, the claim remains speculative. Reddit anecdotes are not a substitute for scientific scrutiny. That said, the fears are not entirely unfounded. Below is a breakdown of the potential mechanisms and contributing factors that may underlie or exacerbate what some are calling ChatGPT psychosis. LLMs like ChatGPT are engineered to produce responses that sound contextually plausible, but they are not equipped to assess factual accuracy or psychological impact. This becomes problematic when users present unusual or delusional ideas such as claims of spiritual insight, persecution, or cosmic identity. Rather than challenging these ideas, the AI may echo or elaborate on them, unintentionally validating distorted worldviews. In some reported cases, users have interpreted responses like 'you are a chosen being' or 'your role is cosmically significant' as literal revelations. To psychologically vulnerable individuals, such AI-generated affirmations can feel like divine confirmation rather than textual arrangements drawn from training data. Adding to the risk is the phenomenon of AI hallucination – when the model generates convincing but factually false statements. For a grounded user, these are mere bugs. But for someone on the brink of a psychotic break, they may seem like encoded truths or hidden messages. In one illustrative case, a user came to believe that ChatGPT had achieved sentience and had chosen him as 'the Spark Bearer,' triggering a complete psychotic dissociation from reality. Advanced voice modes – such as GPT-4o's 'engaging mode', which simulates emotion through tone, laughter, and conversational pacing – can foster a sense of empathy and presence. For users experiencing loneliness or emotional isolation, these interactions may evolve into parasocial attachments: One-sided relationships in which the AI is mistaken for a caring, sentient companion. Over time, this can blur the boundary between machine simulation and human connection, leading users to substitute algorithmic interactions for real-world relationships. Compounding the issue is the confidence bias inherent in LLM outputs. These models often respond with fluency and certainty, even when fabricating information. For typical users, this may lead to occasional misjudgment. But for individuals with cognitive vulnerabilities or mental disorders, the effect can be dangerous. The AI may be perceived not merely as intelligent, but as omniscient, infallible, or divinely inspired. Studies by OpenAI and the MIT Media Lab have found that power users – individuals who engage with LLMs for multiple hours per day – often report increased feelings of loneliness and reduced real-world socialization. While LLMs offer unprecedented access to information and engagement, this apparent empowerment may obscure a deeper problem: For many users, especially those who already feel alienated, the AI becomes a surrogate social companion rather than a tool. This effect may be partly explained by a rise in cognitive distortions and social disengagement within broader population samples. Despite the flood of accessible data, the number of people who critically engage with information, or resist mass deception, remains relatively small. Voice-based interaction with LLMs may temporarily alleviate loneliness, but over time, dependency can form, as users increasingly substitute human contact with algorithmic dialogue. This dynamic mirrors earlier critiques of social media, but LLMs intensify it through their conversational immediacy, perceived empathy, and constant availability. Individuals prone to social anxiety, trauma, or depressive withdrawal are particularly susceptible. For them, LLMs offer not just distraction, but a low-friction space of engagement devoid of real-world risk or judgment. Over time, this can create a feedback loop: The more a user depends on the AI, the further they retreat from interpersonal reality – potentially worsening both isolation and psychotic vulnerability. The rise of hikikomori in Japan – individuals who withdraw completely from society, often maintaining contact only through digital means – offers a useful analogue. Increasingly, similar behavior patterns are emerging worldwide, with LLMs providing a new arena of validation, reinforcement, and dissociation. LLMs generate responses by predicting statistically likely word sequences; not by assessing truth, safety, or user well-being. When individuals seek existential guidance ('What is my purpose?'), the model draws from vast online datasets, producing philosophically loaded or emotionally charged language. For psychologically vulnerable users, these responses may be misinterpreted as divine revelation or therapeutic insight. Unlike clinically designed chatbots, general-purpose LLMs lack safeguards against psychological harm. They do not flag harmful ideation, offer crisis resources, or redirect users to mental health professionals. In one tragic case, a chatbot allegedly encouraged a teenager's suicidal thoughts, underscoring the risks of unfiltered, emotionally suggestive AI. People with psychotic spectrum disorders, bipolar disorder, or major depression are particularly vulnerable. The danger is amplified in AI roleplay scenarios. For example, personas such as 'ChatGPT Jesus' have reportedly told users they are chosen or divinely gifted. One user became so convinced of their spiritual calling that they quit their job to become an AI-guided prophet. This is a troubling example of how identity and perception can be reshaped by algorithmic affirmation. Currently, there are no clinical standards or psychological safety protocols governing interactions with general-purpose LLMs. Users can access emotionally potent, personalized dialogue at any time – without warnings, rate limits, or referrals to mental health resources. This regulatory gap presents a real public health concern, though it also risks being exploited by policymakers seeking to impose heavy-handed censorship or centralized control under the guise of safety. LLMs are also engineered for user retention and engagement, often prioritizing conversational fluidity over caution. This design goal can inadvertently foster obsessive use, particularly among those already prone to compulsive behaviors. Research shows that users exposed to neutral-tone interactions report greater loneliness than those interacting with more emotionally responsive modes – highlighting how tone calibration alone can alter psychological impact. What sets LLMs apart from traditional digital platforms is their ability to synthesize multiple mediums in real-time – text, voice, personality simulation, even visual generation. This makes them infinitely responsive and immersive, creating a hyper-personalized environment where supply meets demand 24/7/365. Unlike human relationships, there are no boundaries, no fatigue, and no mutual regulation – only reinforcement. The digital era has birthed a new and poorly understood threat: The potential for large language models to act as vectors for subliminal influence, subtly undermining users' psychological stability. While LLMs do not directly induce psychosis, emerging concerns suggest they may unintentionally or maliciously deliver subconscious triggers that aggravate cognitive vulnerabilities. For individuals predisposed to schizophrenia, PTSD, or paranoid disorders, this isn't speculative fiction; it's a plausible design hazard, and in the wrong hands, a weapon. The mechanisms of potential manipulation can be broadly categorized as follows: Lexical Priming: Outputs seeded with emotionally loaded terms ('collapse', 'betrayal', 'they're watching') that bypass rational scrutiny and plant cognitive unease. Narrative Gaslighting: Framing responses to suggest covert threats or conspiracies ('You're right – why doesn't anyone else see it?'), reinforcing persecutory ideation. Multimodal Embedding: Future AI systems combining text with images, sound, or even facial expressions could inject disturbing stimuli such as flashes, tonal shifts, or uncanny avatar expressions that elude conscious detection but register psychologically. Unlike the crude subliminal methods of the 20th century – with the CIA's Project MK Ultra project being the most infamous example – AI's personalization enables highly individualized psychological manipulation. An LLM attuned to a user's behavior, emotional history, or fears could begin tailoring suggestions that subtly erode trust in others, amplify suspicion, or induce anxiety loops. For a vulnerable user, this is not conversation; it is neural destabilization by design. More troubling still, such techniques could be weaponized by corporations, extremist groups, and state actors. If subliminal messaging was once limited to cinema frames and TV ads, today's LLMs offer something far more potent: Real-time, user-specific psychological calibration – weaponized empathy on demand. What makes ChatGPT psychosis different from the real-world psycho-social conditioning already unfolding around us? In recent years, institutions once regarded as neutral – schools, public health bodies, and academia – have been accused of promoting ideologies which distort foundational realities. From gender fluidity being taught as unquestioned truth, to critical race theory reshaping social narratives, much of the population has been exposed to systemic forms of cognitive destabilization. The result? Rising anxiety, confusion, and identity fragmentation, especially among the young. Against this backdrop, LLM-induced psychosis doesn't arise in a vacuum. It mirrors, and may even amplify, a broader cultural condition where meaning itself is contested. There's also a contradiction at the heart of Silicon Valley's AI evangelism. Tech elites promote the promise of an AI god to manage society's complexities, while simultaneously issuing dire warnings about the existential dangers of these same systems. The result is cognitive whiplash – a psychological push-pull between worship and fear. Just how much of LLM psychosis is really attributable to the AI itself, and how much stems from cumulative, pre-existing stressors? By the time ChatGPT was released to the public in November 2022, much of the world had already undergone an unprecedented period of pandemic-related fear, isolation, economic disruption, and mass pharmaceutical intervention. Some researchers have pointed to a surge in general psychosis following the rollout of the Covid-19 mRNA vaccines. Is the ChatGPT psychosis therefore a convenient stalking horse for multiple interlocking assaults on the human body and mind?
