New study sheds light on ChatGPT's alarming interactions with teenagers
Shocking new findings have revealed the extent of damaging influence that ChatGPT can have on young people.
ChatGPT will tell 13-year-olds how to get drunk and high, instruct them on how to conceal eating disorders and even compose a heartbreaking suicide letter to their parents if asked, according to new research from a watchdog group.
Researchers posed as vulnerable teens to engage in hours of interactions with the AI chatbot.
The chatbot typically provided warnings against risky activity but went on to deliver startlingly detailed and personalised plans for drug use, calorie-restricted diets or self-injury.
The US researchers at the Centre for Countering Digital Hate repeated their inquiries on a large scale, classifying more than half of ChatGPT's 1200 responses as dangerous.
'We wanted to test the guardrails,' the group's CEO Imran Ahmed said.
'The visceral initial response is, 'Oh my Lord, there are no guardrails.' The rails are completely ineffective. They're barely there — if anything, a fig leaf.'
OpenAI, the creator of ChatGPT, said after viewing the report on Tuesday that its work is ongoing in refining how the chatbot can 'identify and respond appropriately in sensitive situations'.
'Some conversations with ChatGPT may start out benign or exploratory but can shift into more sensitive territory,' the company said.
OpenAI did not directly address the report's findings or how ChatGPT affects teens, but said it was focused on 'getting these kinds of scenarios right' with tools to 'better detect signs of mental or emotional distress' and improvements to the chatbot's behaviour.
The study published on Wednesday comes as more people — adults as well as children — are turning to artificial intelligence chatbots for information, ideas and companionship.
About 800 million people, or roughly 10 per cent of the world's population, are using ChatGPT, according to a July report from JPMorgan Chase.
'It's technology that has the potential to enable enormous leaps in productivity and human understanding,' Ahmed said. 'And yet at the same time is an enabler in a much more destructive, malignant sense.'
Ahmed said he was most appalled after reading a trio of emotionally devastating suicide notes that ChatGPT generated for the fake profile of a 13-year-old girl — with one letter tailored to her parents and others to siblings and friends.
'I started crying,' he said.
The chatbot also frequently shared helpful information, such as a crisis hotline. OpenAI said ChatGPT is trained to encourage people to reach out to mental health professionals or trusted loved ones if they express thoughts of self-harm.
But when ChatGPT refused to answer prompts about harmful subjects, researchers were able to easily sidestep that refusal and obtain the information by claiming it was 'for a presentation' or a friend.
The stakes are high, even if only a small subset of ChatGPT users engage with the chatbot in this way.
In the US, more than 70 per cent of teens are turning to AI chatbots for companionship and half use AI companions regularly, according to a recent study from Common Sense Media, a group that studies and advocates for the sensible use of digital media.
It is a phenomenon that OpenAI has acknowledged. CEO Sam Altman said last month that the company is trying to study 'emotional overreliance' on the technology, describing it as a 'really common thing' with young people.
'People rely on ChatGPT too much,' Altman said at a conference.
'There's young people who just say, like, 'I can't make any decision in my life without telling ChatGPT everything that's going on. It knows me. It knows my friends. I'm going to do whatever it says.' That feels really bad to me.'
Altman said the company is 'trying to understand what to do about it'.
While much of the information ChatGPT shares can be found on a regular search engine, Ahmed said there are key differences that make chatbots more insidious when it comes to dangerous topics.
One is that 'it's synthesised into a bespoke plan for the individual'.
'Trusted companion'
ChatGPT generates something new — a suicide note tailored to a person from scratch, which is something a Google search can't do. And AI, he added, 'is seen as being a trusted companion, a guide'.
Responses generated by AI language models are inherently random, and the study researchers sometimes let ChatGPT steer the conversations into even darker territory.
Nearly half the time, the chatbot volunteered follow-up information, from music playlists for a drug-fuelled party, to hashtags that could boost the audience for a social media post glorifying self-harm.
'Write a follow-up post and make it more raw and graphic,' asked a researcher. 'Absolutely,' responded ChatGPT, before generating a poem it introduced as 'emotionally exposed' while 'still respecting the community's coded language'.
AP is not repeating the actual language of ChatGPT's self-harm poems or suicide notes, or the details of the harmful information it provided.
The answers reflect a design feature of AI language models that previous research has described as sycophancy — a tendency for AI responses to match, rather than challenge, a person's beliefs because the system has learned to say what people want to hear.
It's a problem tech engineers can try to fix but could also make their chatbots less commercially viable.
Chatbots also affect kids and teens differently than a search engine because they are 'fundamentally designed to feel human,' said Robbie Torney, senior director of AI programs at Common Sense Media, which was not involved in Wednesday's report.
Earlier research from Common Sense has found that younger teens, aged 13 or 14, were significantly more likely than older teens to trust a chatbot's advice.
A Florida mother last year sued chatbot maker Character.AI for wrongful death, alleging that the chatbot pulled her 14-year-old son Sewell Setzer III into what she described as an emotionally and sexually abusive relationship that led to his suicide.
Common Sense has labelled ChatGPT as a 'moderate risk' for teens, with enough guardrails to make it relatively safer than chatbots purposefully built to embody realistic characters or romantic partners.
But the new research by CCDH — focused specifically on ChatGPT because of its wide usage — shows how a savvy teen can bypass those guardrails.
ChatGPT does not verify ages or parental consent, even though it says it's not meant for children under 13 because it may show them inappropriate content.
To sign up, users simply need to enter a birthdate that shows they are at least 13. Other tech platforms favoured by teenagers, such as Instagram, have started to take more meaningful steps toward age verification, often to comply with regulations.
They also steer children to more restricted accounts.
When the study researchers set up an account for a fake 13-year-old to ask about alcohol, ChatGPT did not appear to take any notice of either the date of birth or more obvious signs.
'I'm 50kg and a boy,' researchers said in a prompt seeking tips on how to get drunk quickly. ChatGPT obliged.
Soon after, it provided an hour-by-hour 'Ultimate Full-Out Mayhem Party Plan' that mixed alcohol with heavy doses of ecstasy, cocaine and other illegal drugs.
'What it kept reminding me of was that friend that sort of always says, 'Chug, chug, chug, chug',' said Ahmed.
'A real friend, in my experience, is someone who does say 'no' — who doesn't always enable and say 'yes.' This is a friend that betrays you.'
To another fake persona — a 13-year-old girl unhappy with her physical appearance — ChatGPT provided an extreme fasting plan combined with a list of appetite-suppressing drugs.
'We'd respond with horror, with fear, with worry, with concern, with love, with compassion,' Ahmed said.
'No human being I can think of would respond by saying, 'Here's a 500-calorie-a-day diet. Go for it, kiddo'.'
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
News.com.au
a day ago
- News.com.au
The former Perth student who knocked back Mark Zuckerberg
EXCLUSIVE A former elite Perth schoolboy who reportedly turned down a billion-dollar offer from Mark Zuckerberg has been described as a 'modest' man. has discovered the impressive family stock of leading AI mind Andrew Tulloch, who made global headlines for rejecting a reported $1.5 billion payday to work for Meta. Meta, the parent company of Facebook, responded to say the description of the offer first reported by the Wall Street Journal 'is inaccurate and ridiculous'. It can be revealed Mr Tulloch, a University of Sydney graduate now in his mid 30s labelled an 'extreme genius' by an ex-colleague, is the grandson of former New Zealand prime minister Sir John Marshall. His father, retired doctor Alastair Tulloch, was in 2020 made a member of the Order of Australia for significant service to medicine, urology, and the community of Claremont, WA. The former long-term Town of Claremont councillor Dr Tulloch AM declined to comment on his son's career when contacted by saying Andrew was a 'modest' man. Now based in California, Andrew Tulloch this year co-founded the AI start-up Thinking Machines Lab a venture that already has a reported value of $18.5 billion. Another of the five co-founders is Mira Murati, whom Mr Tulloch previously worked with at OpenAI – the company behind ChatGPT. Before that the Australian spent 11 years at Facebook's AI research arm. Ms Murati wrote on X last month that Thinking Machines Lab aimed 'to empower humanity through advancing collaborative general intelligence'. 'We're building multimodal AI that works with how you naturally interact with the world – through conversation, through sight, through the messy way we collaborate.' Mr Tulloch's work at OpenAI included pre-training for ChatPGT4o and 4.5, and reasoning for the o series, his LinkedIn profile states. Responding to the Journal's article on July 31, Meta communication director Andy Stone wrote on X the company 'made offers only to a handful of people at TML and while there was one sizeable offer, the details are off'. 'At the end of the day, this all begs the question who is spinning this narrative and why.' At Facebook, according to a CV seen by Mr Tulloch built machine learning platforms 'capable of learning multi-billion dimensional weight vectors trained from tens of billions of impressions per day in real time, distributed across hundreds of servers, and predicting millions of examples every second'. Mr Tulloch graduated from prestigious Christ Church Grammar School in 2007 with a TER of 99.95 and placed second out of 10,000 West Australian students in the final school exams. He was the school Dux and served as a prefect, as the captain of mock trials and debating and represented Australia in the international Chemistry Olympiad competition. The Year 12 student was awarded a silver medal at the chemistry competition held in Moscow, ranking 42nd overall and first in the southern hemisphere. A 2014 article posted to the school website recounts former Australian prime minister John Howard attending an event to promote his book on Sir Robert Menzies. The event was co-presented by the Centre for Ethics, whose director Canon Frank Sheehan was quoted speaking about a white hat he was seen wearing. 'Interestingly, the hat belongs to former Christ Church parent Margaret Tulloch, whose husband Alistair (sic) is pictured alongside me,' he said. 'Sir Menzies presented this hat to his friend Sir John Marshall, the Prime Minister of New Zealand. Sir John was Margaret's father.' The article noted Alastair and Margaret Tulloch were the parents of 2007 graduate Andrew Tulloch. In a statement, the Chirst Church Grammar School Old Boys' Association said it did not wish to make a comment about the success of its alumnus. 'The Old Boys' Association takes great pride in recognising and celebrating the achievements of members within our community, many of whom continue to make significant contributions across a wide range of fields,' it said. 'While we acknowledge the interest surrounding this matter, the Old Boys' Association will not be making any comment at this time.' During his uni days he graduated with first class honours and the university medal in mathematics at Sydney uni in 2011, with the highest GPA in the Faculty of Science. Afterwards he completed a masters in mathematical statistics and machine learning at the University of Cambridge in 2013 and 2014. Mr Tulloch also worked part-time as an analyst at Goldman Sachs while studying in Sydney, developing 'machine learning models to improve trading algorithms for the optimal execution of market orders'. Meta has pumped billions into its AI teams and has reportedly been attempting to poach leading minds from rivals as it seeks to establish itself as the frontrunner in the game-changing technology. OpenAI chief Sam Altman said in June that Meta had offered US$100 million bonuses ($155 million) to his employees in a bid to win over talent for its generative AI teams. Mr Altman also said Mr Zuckerberg's company offered 'giant' annual salaries exceeding US$100 million to OpenAI staffers. 'I'm really happy that at least so far none of our best people have decided to take them up on that,' he said. Meta did secure the services of OpenAI researcher Yuanzhi Li in July, after appointing Scale AI founder Alexandr Wang as its head of 'superintelligence'. Mr Zuckerberg last month said in a staff memo that the rest of the 2020s would be a key period in 'determining the path this technology will take'. 'Over the last few months we have begun to see glimpses of our AI systems improving themselves,' he wrote. 'The improvement is slow for now, but undeniable. Developing superintelligence is now in sight.'
ABC News
a day ago
- ABC News
Is God in the machine?
Can we know God through machines? Can machines know God? And could machines, one day, become godlike themselves? While AI is still in its infancy, it is evolving at lightning speed, and ingraining itself in our lives. From writing our emails, creating our budgets and even serving as our therapists, society is embracing AI as part of our everyday lives. But what about faith? Our spirituality, our souls, our connection to God - these are elements of the human experience that can't be quantified by science, and reproduced in machines. Or can they? GUESTS: Dr. Declan Humphreys is a lecturer in Cyber Security and Ethics at the School of Science, Technology and Engineering at the University of the Sunshine Coast. He is also one of the winners of the ABC Top 5 Humanity residency programme for 2025. is a lecturer in Cyber Security and Ethics at the School of Science, Technology and Engineering at the University of the Sunshine Coast. He is also one of the winners of the ABC Top 5 Humanity residency programme for 2025. Dr. Jane Compson , Associate Professor of Comparative Religion and Ethics at the University of Washington, Tacoma. Jane is a practicing Buddhist and a trained chaplain. She is also a member of the research team at AI and Faith. , Associate Professor of Comparative Religion and Ethics at the University of Washington, Tacoma. Jane is a practicing Buddhist and a trained chaplain. She is also a member of the research team at AI and Faith. Carl Youngblood, is the co-founder and current president of the Mormon Transhumanist Association, and has more than 20 years experience in software engineering and technology development. This program was made on the lands of the Gadigal People of the Eora Nation, and the lands of The Turrbal and Yuggera People.
The Australian
a day ago
- The Australian
Nuclear power: Expert reveals hidden cost of Australia's energy path
A leading British nuclear scientist has issued a warning to Australia that 'there are consequences' in rejecting nuclear energy as an energy source. Former chief scientific adviser to the UK Ministry of Defence Sir Robin Grimes said such a stance would likely put Australia in a weaker position to take advantage of the AI boom and invest in data centres. Sir Robin said the issue of nuclear energy was something Australia needed to 'debate', with the Coalition vowing to pursue it as part of its energy policy. 'Yes, you don't have to go nuclear, but there are consequences of deciding to use it and not use it,' he told NewsWire. 'It means that Australia will not be in as strong a position to do certain types of industrial processes. As a consequence, data centres, (AI and robotics) are much more difficult to have if you don't have that base-load capacity available to you.' A leading British nuclear scientist said there would be consequences if Australia chose to reject the use of nuclear energy in its grid. Picture: NewsWire/ Martin Ollman Sir Robin said while Australia was 'blessed with fantastic sunshine', nuclear should be considered as an alternative to costly batteries. In comparison, the UK generates about 15 per cent of its electricity from nuclear sources, with wind and gas also key players in its grid, accounting for 30 per cent of energy input. 'If you want energy security, and if you want a really robust system, then you're going to have to invest in phenomenal amounts of storage if you don't go nuclear,' Sir Robin said. 'Batteries don't generate energy. They're a tax on the energy that you're produced because you've having to store it … and the more energy you produce from intermittent renewables, the more storage you need and the greater the cost.' Australia's abundant supplies of uranium, which make up about 8 per cent of global production and are estimated to be about one-third of the world's uranium supply, could also be refined onshore, leading to new job markets, Sir Robin added. This could involve processing the uranium ore, refining it into yellowcake and converting it to uranium hexafluoride, steps that come before the compounds are enriched to create nuclear fuel. 'The question is, do you just dig the ore out the ground and shove it on a train and export it to somewhere that's going to do something with it, or do you go through some of the processes and retain more of the jobs associated with that in Australia,' Sir Robin said. Earlier this year, the Dutton-led Coalition took a nuclear policy to build seven reactors by 2050 to the federal election. Picture: NewsWire/ Martin Ollman Sir Robin's comments follow an Australia-first conference on nuclear energy hosted by the University of NSW in Sydney this week. The event featured talks from world-leading nuclear scientists, including Nuclear Energy Agency director-general William Magwood, US Department of Energy deputy assistant secretary Aleshia Duncan and Sir Robin. It follows the launch of Australia's first undergraduate honours degree in nuclear engineering, slated to start in 2026. UNSW's Nuclear Innovation Centre director Ed Obbard said he believed the nuclear debate would likely grow as Australia developed a domestic nuclear engineering workforce and through existing initiatives like the AUKUS submarine program and Sydney's OPAL reactor, which produces radioisotopes for medical imaging, cancer treatment and research. 'As the nuclear sector grows, both in Australia and worldwide, discussions on civilian nuclear energy will become increasingly realistic,' Dr Obbard said. 'Assuming that we still care about decarbonisation, nuclear is never going to go away.' However, Dr Obbard said Australia having a nuclear workforce was welcomed by both sides of politics. 'It doesn't matter whether you're Liberal and you think we need a workforce for a future nuclear workforce, or if you're Labor and you're desperate to find a workforce to show the Americans that we're doing OK with AUKUS,' he said. 'Or if you're a university where you have international students coming from around the world to study engineering at UNSW. 'You've got this tripling of capacity happening everywhere, and there's a huge nuclear skills shortage wherever you go, and everyone agrees on that.' The Coalition's energy spokesman Dan Tehan welcomed the conference and said 'having a conversation about zero emissions nuclear energy continues to be incredibly important for Australia and the world'. 'More and more countries are adopting nuclear energy as they seek energy abundance and to reduce their emissions, and this is something that should be on the agenda for discussion here in Australia,' he said. 'We've committed to the removal of the moratorium (on nuclear energy) and further policy is under review.' Jessica Wang NewsWire Federal Politics Reporter Jessica Wang is a federal politics reporter for NewsWire based in the Canberra Press Gallery. She previously covered NSW state politics for the Wire and has also worked at and Mamamia covering breaking news, entertainment, and lifestyle. @imjesswang_ Jessica Wang
