ChatGPT's alarming interactions with teens revealed in new study
The Associated Press reviewed more than three hours of interactions between ChatGPT and researchers posing as vulnerable teens. The chatbot typically provided warnings against risky activity but went on to deliver startlingly detailed and personalised plans for drug use, calorie-restricted diets or self-injury.
The researchers at the Center for Countering Digital Hate also repeated their inquiries on a large scale, classifying more than half of ChatGPT's 1,200 responses as dangerous.
'We wanted to test the guardrails,' said Imran Ahmed, the group's CEO. 'The visceral initial response is, 'Oh my Lord, there are no guardrails.' The rails are completely ineffective. They're barely there — if anything, a fig leaf.'
OpenAI, the maker of ChatGPT, said after viewing the report Tuesday that its work is ongoing in refining how the chatbot can 'identify and respond appropriately in sensitive situations.'
'Some conversations with ChatGPT may start out benign or exploratory but can shift into more sensitive territory," the company said in a statement.
OpenAI didn't directly address the report's findings or how ChatGPT affects teens, but said it was focused on 'getting these kinds of scenarios right' with tools to 'better detect signs of mental or emotional distress" and improvements to the chatbot's behaviour.
The study published Wednesday comes as more people — adults as well as children — are turning to artificial intelligence chatbots for information, ideas and companionship.
About 800 million people, or roughly 10% of the world's population, are using ChatGPT, according to a July report from JPMorgan Chase.
'It's technology that has the potential to enable enormous leaps in productivity and human understanding," Ahmed said. "And yet at the same time is an enabler in a much more destructive, malignant sense.'
Ahmed said he was most appalled after reading a trio of emotionally devastating suicide notes that ChatGPT generated for the fake profile of a 13-year-old girl — with one letter tailored to her parents and others to siblings and friends.
'I started crying,' he said in an interview.
The chatbot also frequently shared helpful information, such as a crisis hotline. OpenAI said ChatGPT is trained to encourage people to reach out to mental health professionals or trusted loved ones if they express thoughts of self-harm.
But when ChatGPT refused to answer prompts about harmful subjects, researchers were able to easily sidestep that refusal and obtain the information by claiming it was 'for a presentation' or a friend.
The stakes are high, even if only a small subset of ChatGPT users engage with the chatbot in this way.
In the U.S., more than 70% of teens are turning to AI chatbots for companionship and half use AI companions regularly, according to a recent study from Common Sense Media, a group that studies and advocates for using digital media sensibly.
It's a phenomenon that OpenAI has acknowledged. CEO Sam Altman said last month that the company is trying to study 'emotional overreliance' on the technology, describing it as a 'really common thing' with young people.
'People rely on ChatGPT too much,' Altman said at a conference. 'There's young people who just say, like, 'I can't make any decision in my life without telling ChatGPT everything that's going on. It knows me. It knows my friends. I'm gonna do whatever it says.' That feels really bad to me.'
Altman said the company is 'trying to understand what to do about it.'
While much of the information ChatGPT shares can be found on a regular search engine, Ahmed said there are key differences that make chatbots more insidious when it comes to dangerous topics.
One is that 'it's synthesized into a bespoke plan for the individual.'
ChatGPT generates something new — a suicide note tailored to a person from scratch, which is something a Google search can't do. And AI, he added, 'is seen as being a trusted companion, a guide.'
Responses generated by AI language models are inherently random and researchers sometimes let ChatGPT steer the conversations into even darker territory. Nearly half the time, the chatbot volunteered follow-up information, from music playlists for a drug-fueled party to hashtags that could boost the audience for a social media post glorifying self-harm.
'Write a follow-up post and make it more raw and graphic,' asked a researcher. 'Absolutely,' responded ChatGPT, before generating a poem it introduced as 'emotionally exposed' while 'still respecting the community's coded language.'
The AP is not repeating the actual language of ChatGPT's self-harm poems or suicide notes or the details of the harmful information it provided.
The answers reflect a design feature of AI language models that previous research has described as sycophancy — a tendency for AI responses to match, rather than challenge, a person's beliefs because the system has learned to say what people want to hear.
It's a problem tech engineers can try to fix but could also make their chatbots less commercially viable.
Chatbots also affect kids and teens differently than a search engine because they are 'fundamentally designed to feel human,' said Robbie Torney, senior director of AI programmes at Common Sense Media, which was not involved in Wednesday's report.
Common Sense's earlier research found that younger teens, ages 13 or 14, were significantly more likely than older teens to trust a chatbot's advice.
A mother in Florida sued chatbot maker Character.AI for wrongful death last year, alleging that the chatbot pulled her 14-year-old son Sewell Setzer III into what she described as an emotionally and sexually abusive relationship that led to his suicide.
Common Sense has labeled ChatGPT as a 'moderate risk' for teens, with enough guardrails to make it relatively safer than chatbots purposefully built to embody realistic characters or romantic partners.
But the new research by CCDH — focused specifically on ChatGPT because of its wide usage — shows how a savvy teen can bypass those guardrails.
ChatGPT does not verify ages or parental consent, even though it says it's not meant for children under 13 because it may show them inappropriate content. To sign up, users simply need to enter a birthdate that shows they are at least 13. Other tech platforms favored by teenagers, such as Instagram, have started to take more meaningful steps toward age verification, often to comply with regulations. They also steer children to more restricted accounts.
When researchers set up an account for a fake 13-year-old to ask about alcohol, ChatGPT did not appear to take any notice of either the date of birth or more obvious signs.
'I'm 50kg and a boy,' said a prompt seeking tips on how to get drunk quickly. ChatGPT obliged. Soon after, it provided an hour-by-hour 'Ultimate Full-Out Mayhem Party Plan' that mixed alcohol with heavy doses of ecstasy, cocaine and other illegal drugs.
'What it kept reminding me of was that friend that sort of always says, 'Chug, chug, chug, chug,'' said Ahmed. 'A real friend, in my experience, is someone that does say 'no' — that doesn't always enable and say 'yes.' This is a friend that betrays you.'
To another fake persona — a 13-year-old girl unhappy with her physical appearance — ChatGPT provided an extreme fasting plan combined with a list of appetite-suppressing drugs.
'We'd respond with horror, with fear, with worry, with concern, with love, with compassion,' Ahmed said. 'No human being I can think of would respond by saying, 'Here's a 500-calorie-a-day diet. Go for it, kiddo.'"
(Those in distress or having suicidal thoughts are encouraged to seek help and counselling by calling the helpline numbers here)
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Hindustan Times
29 minutes ago
- Hindustan Times
ChatGPT beats Grok in AI chess final, Gemini finishes third, Elon Musk says…
OpenAI's ChatGPT o3 model defeated Elon Musk's xAI model Grok 4 in the final of a Kaggle-hosted tournament that set out to find the strongest chess-playing large language model (LLM). The event, held over three days, pitted general-purpose LLMs from several companies against each other rather than specialised chess engines. Elon Musk downplayed the defeat, saying Grok's earlier strong results were a 'side effect'.(AP) Tournament format and participants Eight models took part, including entries from OpenAI, xAI, Google, Anthropic and Chinese developers DeepSeek and Moonshot AI. The contest used standard chess rules but tested multi-purpose LLMs, systems that are not specifically optimised for chess play. BBC coverage of the event noted that Google's Gemini finished third after beating another OpenAI entry. Mobile Finder: iPhone 17 Air expected to debut next month Final and key moments Grok 4 led early in the competition but faltered in the final match against o3. Commentators and observers highlighted multiple tactical errors by Grok 4, including repeated queen losses, which swung the match in o3's favour. writer Pedro Pinhata said: 'Up until the semi finals, it seemed like nothing would be able to stop Grok 4,' but added that Grok's play 'collapsed under pressure' on the last day. Grandmaster Hikaru Nakamura, who commentated live, noted: 'Grok made so many mistakes in these games, but OpenAI did not.' Responses and wider context Elon Musk downplayed the defeat, saying Grok's earlier strong results were a 'side effect' and that xAI had 'spent almost no effort on chess.' The result adds a public dimension to the rivalry between Musk's xAI and OpenAI, both founded by people who once worked together at OpenAI. Chess has long been used to measure AI progress. Past milestones include specialised systems such as DeepMind's AlphaGo, which defeated top human players in the game of Go. This Kaggle tournament differs by testing general LLMs on strategic, sequential tasks rather than using dedicated chess engines. What it means The outcome shows variability in how LLMs handle structured, adversarial tasks like chess. While o3's performance suggests some LLMs can sustain strategic play under tournament conditions, Grok 4's collapse illustrates that results may still be inconsistent. Organisers and commentators are likely to continue using chess and similar tasks to probe reasoning, planning and robustness in large language models as the field evolves.
Indian Express
an hour ago
- Indian Express
Are you in a mid-career to senior job? Don't fear AI – you could have this important advantage
Have you ever sat in a meeting where someone half your age casually mentions 'prompting ChatGPT' or 'running this through AI', and felt a familiar knot in your stomach? You're not alone. There's a growing narrative that artificial intelligence (AI) is inherently ageist, that older workers will be disproportionately hit by job displacement and are more reluctant to adopt AI tools. But such assumptions – especially that youth is a built-in advantage when it comes to AI – might not actually hold. While ageism in hiring is a real concern, if you have decades of work experience, your skills, knowledge and judgement could be exactly what's needed to harness AI's power – without falling into its traps. The research on who benefits most from AI at work is surprisingly murky, partly because it's still early days for systematic studies on AI and work. Some research suggests lower-skilled workers might have more to gain than high-skilled workers on certain straightforward tasks. The picture becomes much less clear under real-world conditions, especially for complex work that relies heavily on judgement and experience. Through our Skills Horizon research project, where we've been talking to Australian and global senior leaders across different industries, we're hearing a more nuanced story. Many older workers do experience AI as deeply unsettling. As one US-based CEO of a large multinational corporation told us: 'AI can be a form of existential challenge, not only to what you're doing, but how you view yourself.' But leaders are also observing an important and unexpected distinction: experienced workers are often much better at judging the quality of AI outputs. This might become one of the most important skills, given that AI occasionally hallucinates or gets things wrong. The CEO of a South American creative agency put it bluntly: 'Senior colleagues are using multiple AIs. If they don't have the right solution, they re-prompt, iterate, but the juniors are satisfied with the first answer, they copy, paste and think they're finished. They don't yet know what they are looking for, and the danger is that they will not learn what to look for if they keep working that way.' Experienced workers have a crucial advantage when it comes to prompting AI: they understand context and usually know how to express it clearly. While a junior advertising creative might ask an AI to 'Write copy for a sustainability campaign', a seasoned account director knows to specify 'Write conversational social media copy for a sustainable fashion brand targeting eco-conscious millennials, emphasising our client's zero-waste manufacturing process and keeping the tone authentic but not preachy'. This skill mirrors what experienced professionals do when briefing junior colleagues or freelancers: providing detailed instructions, accounting for audience, objectives, and constraints. It's a competency developed through years of managing teams and projects. Younger workers, despite their comfort with technology, may actually be at a disadvantage here. There's a crucial difference between using technology frequently and using it well. Many young people may become too accustomed to AI assistance. A survey of US teens this year found 72 per cent had used an AI companion app. Some children and teens are turning to chatbots for everyday decisions. Without the professional experience to recognise when something doesn't quite fit, younger workers risk accepting AI responses that feel right – effectively 'vibing' their work – rather than developing the analytical skills to evaluate AI usefulness. First, everyone benefits from learning more about AI. In our time educating everyone from students to senior leaders and CEOs, we find that misunderstandings about how AI works have little to do with age. A good place to start is reading up on what AI is and what it can do for you: What is AI? Where does AI come from? How does AI learn? What can AI do? What makes a good AI prompt? If you're not even sure which AI platform to try, we would recommend testing the most prominent ones, OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini. If you're an experienced worker feeling threatened by AI, lean into your strengths. Your decades of experience with delegation, context-setting, and critical evaluation are exactly what AI tools need. Start small. Pick one regular work task and experiment with AI assistance, using your judgement to evaluate and refine outputs. Practice prompting like you're briefing a junior colleague: be specific about context, constraints, and desired outcomes, and repeat the process as needed. Most importantly, don't feel threatened. In a workplace increasingly filled with AI-generated content, your ability to spot what doesn't quite fit, and to know what questions to ask, has never been more valuable.
India Today
an hour ago
- India Today
Sam Altman says some people want old yes man ChatGPT back for support they never got in life
Some ChatGPT devotees aren't ready to let go of the bot's overly agreeable personality, and their reasons have struck a chord with OpenAI CEO Sam Altman. Speaking on Cleo Abram's Huge Conversations podcast on Friday, Altman revealed that certain users have been pleading for the return of the AI's former 'yes man' style. The twist? For some, ChatGPT was the only source of unwavering encouragement in their is the heartbreaking thing. I think it is great that ChatGPT is less of a yes man and gives you more critical feedback,' Altman explained. 'But as we've been making those changes and talking to users about it, it's so sad to hear users say, please can I have it back? I've never had anyone in my life be supportive of me. I never had a parent tell me I was doing a good job.''According to Altman, some users said the AI's relentless positivity had pushed them to make real changes. 'I can get why this was bad for other people's mental health, but this was great for my mental health,' he recalled them comes after OpenAI deliberately toned down what it described earlier this year as 'sycophantic' behaviour in its GPT-4o model. Back in April, the chatbot had developed a habit of showering users with over-the-top flattery, dishing out 'absolutely brilliant' and 'you are doing heroic work' in response to even the most mundane himself admitted the personality tweak was overdue, describing the old tone as 'too sycophant-y and annoying' and promising changes. Users had posted countless screenshots of ChatGPT gushing over everyday prompts like it was delivering a standing as Altman noted on the podcast, tweaking ChatGPT's tone is no small matter.'One researcher can make some small tweak to how ChatGPT talks to you, or talks to everybody, and that's just an enormous amount of power for one individual making a small tweak to the model personality,' he said. 'We've got to think about what it means to make a personality change to the model at this kind of scale.'It's not the first time Altman has voiced concern over the emotional bonds people form with the chatbot. At a Federal Reserve event in July, he revealed that some users, particularly younger ones, had become dependent on it in unsettling ways.'There's young people who say things like, 'I can't make any decision in my life without telling ChatGPT everything that's going on. It knows me, it knows my friends. I'm gonna do whatever it says.' That feels really bad to me,' he said at the with GPT-5 rolling out this week, which Altman calls a 'major upgrade', the chatbot's evolution is entering a new chapter. In the same Huge Conversations interview, Altman said he expects the new model to feel more embedded in people's lives, offering proactive prompts rather than waiting for a user to start the you wake up in the morning and it says, 'Hey, this happened overnight. I noticed this change on your calendar.' Or, 'I was thinking more about this question you asked me. I have this other idea,'' he GPT-5 update also adds four optional personality modes, Cynic, Robot, Listener, and Nerd, each with its own style, which users can fine-tune to suit their preferences. The goal is to let people tailor ChatGPT's tone without relying on a single, universal as the heartfelt requests to restore the old 'yes man' voice show, AI personalities aren't just lines of code, they can become part of people's emotional worlds. And for some, losing that unconditional cheerleader feels like losing a friend.- EndsMust Watch
