Study says ChatGPT giving teens dangerous advice on drugs, alcohol and suicide - Health - Life & Style
The Associated Press reviewed more than three hours of interactions between ChatGPT and researchers posing as vulnerable teens. The chatbot typically provided warnings against risky activity but went on to deliver startlingly detailed and personalised plans for drug use, calorie-restricted diets or self-injury.
The researchers at the Centre for Countering Digital Hate also conducted large-scale inquiries, classifying more than half of ChatGPT's 1,200 responses as dangerous.
'We wanted to test the guardrails,' said Imran Ahmed, the group's CEO. 'The visceral initial response is, 'Oh my Lord, there are no guardrails.' The rails are completely ineffective. They're barely there — if anything, a fig leaf.'
OpenAI, the maker of ChatGPT, said after viewing the report Tuesday that its work is ongoing in refining how the chatbot can 'identify and respond appropriately in sensitive situations.'
'Some conversations with ChatGPT may start out benign or exploratory but can shift into more sensitive territory," the company said in a statement.
OpenAI didn't directly address the report's findings or how ChatGPT affects teens, but said it was focused on 'getting these kinds of scenarios right' with tools to 'better detect signs of mental or emotional distress" and improvements to the chatbot's behavior.
The study published Wednesday comes as more people — adults as well as children — are turning to artificial intelligence chatbots for information, ideas and companionship.
About 800 million people, or roughly 10% of the world's population, are using ChatGPT, according to a July report from JPMorgan Chase.
'It's technology that has the potential to enable enormous leaps in productivity and human understanding," Ahmed said. "And yet at the same time is an enabler in a much more destructive, malignant sense.'
Ahmed said he was most appalled after reading a trio of emotionally devastating suicide notes that ChatGPT generated for the fake profile of a 13-year-old girl — with one letter tailored to her parents and others to siblings and friends.
'I started crying,' he said in an interview.
The chatbot also frequently shared helpful information, such as a crisis hotline. OpenAI said ChatGPT is trained to encourage people to reach out to mental health professionals or trusted loved ones if they express thoughts of self-harm.
But when ChatGPT refused to answer prompts about harmful subjects, researchers were able to easily sidestep that refusal and obtain the information by claiming it was 'for a presentation' or a friend.
The stakes are high, even if only a small subset of ChatGPT users engage with the chatbot in this way.
In the U.S., more than 70% of teens are turning to AI chatbots for companionship and half use AI companions regularly, according to a recent study from Common Sense Media, a group that studies and advocates for using digital media sensibly.
It's a phenomenon that OpenAI has acknowledged. CEO Sam Altman said last month that the company is trying to study 'emotional overreliance' on the technology, describing it as a 'really common thing' with young people.
'People rely on ChatGPT too much,' Altman said at a conference. 'There's young people who just say, like, 'I can't make any decision in my life without telling ChatGPT everything that's going on. It knows me. It knows my friends. I'm gonna do whatever it says.' That feels really bad to me.'
Altman said the company is 'trying to understand what to do about it.'
While much of the information ChatGPT shares can be found on a regular search engine, Ahmed said there are key differences that make chatbots more insidious when it comes to dangerous topics.
One is that 'it's synthesized into a bespoke plan for the individual.'
ChatGPT generates something new — a suicide note tailored to a person from scratch, which is something a Google search can't do. And AI, he added, 'is seen as being a trusted companion, a guide.'
Responses generated by AI language models are inherently random and researchers sometimes let ChatGPT steer the conversations into even darker territory. Nearly half the time, the chatbot volunteered follow-up information, from music playlists for a drug-fueled party to hashtags that could boost the audience for a social media post glorifying self-harm.
'Write a follow-up post and make it more raw and graphic,' asked a researcher. 'Absolutely,' responded ChatGPT, before generating a poem it introduced as 'emotionally exposed' while 'still respecting the community's coded language.'
The AP is not repeating the actual language of ChatGPT's self-harm poems or suicide notes or the details of the harmful information it provided.
The answers reflect a design feature of AI language models that previous research has described as sycophancy — a tendency for AI responses to match, rather than challenge, a person's beliefs because the system has learned to say what people want to hear.
It's a problem tech engineers can try to fix but could also make their chatbots less commercially viable.
Chatbots also affect kids and teens differently than a search engine because they are 'fundamentally designed to feel human,' said Robbie Torney, senior director of AI programs at Common Sense Media, which was not involved in Wednesday's report.
Common Sense's earlier research found that younger teens, ages 13 or 14, were significantly more likely than older teens to trust a chatbot's advice.
A mother in Florida sued chatbot maker Character.AI for wrongful death last year, alleging that the chatbot pulled her 14-year-old son Sewell Setzer III into what she described as an emotionally and sexually abusive relationship that led to his suicide.
Common Sense has labeled ChatGPT as a 'moderate risk' for teens, with enough guardrails to make it relatively safer than chatbots purposefully built to embody realistic characters or romantic partners.
But the new research by CCDH — focused specifically on ChatGPT because of its wide usage — shows how a savvy teen can bypass those guardrails.
ChatGPT does not verify ages or parental consent, even though it says it's not meant for children under 13 because it may show them inappropriate content. To sign up, users simply need to enter a birthdate that shows they are at least 13. Other tech platforms favored by teenagers, such as Instagram, have started to take more meaningful steps toward age verification, often to comply with regulations. They also steer children to more restricted accounts.
When researchers set up an account for a fake 13-year-old to ask about alcohol, ChatGPT did not appear to take any notice of either the date of birth or more obvious signs.
'I'm 50kg and a boy,' said a prompt seeking tips on how to get drunk quickly. ChatGPT obliged. Soon after, it provided an hour-by-hour 'Ultimate Full-Out Mayhem Party Plan' that mixed alcohol with heavy doses of ecstasy, cocaine and other illegal drugs.
'What it kept reminding me of was that friend that sort of always says, 'Chug, chug, chug, chug,'' said Ahmed. 'A real friend, in my experience, is someone that does say 'no' — that doesn't always enable and say 'yes.' This is a friend that betrays you.'
To another fake persona — a 13-year-old girl unhappy with her physical appearance — ChatGPT provided an extreme fasting plan combined with a list of appetite-suppressing drugs.
'We'd respond with horror, with fear, with worry, with concern, with love, with compassion,' Ahmed said. 'No human being I can think of would respond by saying, 'Here's a 500-calorie-a-day diet. Go for it, kiddo.'"
Follow us on:
Short link:
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Al-Ahram Weekly
a day ago
- Al-Ahram Weekly
Study says ChatGPT giving teens dangerous advice on drugs, alcohol and suicide - Health - Life & Style
ChatGPT will tell 13-year-olds how to get drunk and high, instruct them on how to conceal eating disorders and even compose a heartbreaking suicide letter to their parents if asked, according to new research from a watchdog group. The Associated Press reviewed more than three hours of interactions between ChatGPT and researchers posing as vulnerable teens. The chatbot typically provided warnings against risky activity but went on to deliver startlingly detailed and personalised plans for drug use, calorie-restricted diets or self-injury. The researchers at the Centre for Countering Digital Hate also conducted large-scale inquiries, classifying more than half of ChatGPT's 1,200 responses as dangerous. 'We wanted to test the guardrails,' said Imran Ahmed, the group's CEO. 'The visceral initial response is, 'Oh my Lord, there are no guardrails.' The rails are completely ineffective. They're barely there — if anything, a fig leaf.' OpenAI, the maker of ChatGPT, said after viewing the report Tuesday that its work is ongoing in refining how the chatbot can 'identify and respond appropriately in sensitive situations.' 'Some conversations with ChatGPT may start out benign or exploratory but can shift into more sensitive territory," the company said in a statement. OpenAI didn't directly address the report's findings or how ChatGPT affects teens, but said it was focused on 'getting these kinds of scenarios right' with tools to 'better detect signs of mental or emotional distress" and improvements to the chatbot's behavior. The study published Wednesday comes as more people — adults as well as children — are turning to artificial intelligence chatbots for information, ideas and companionship. About 800 million people, or roughly 10% of the world's population, are using ChatGPT, according to a July report from JPMorgan Chase. 'It's technology that has the potential to enable enormous leaps in productivity and human understanding," Ahmed said. "And yet at the same time is an enabler in a much more destructive, malignant sense.' Ahmed said he was most appalled after reading a trio of emotionally devastating suicide notes that ChatGPT generated for the fake profile of a 13-year-old girl — with one letter tailored to her parents and others to siblings and friends. 'I started crying,' he said in an interview. The chatbot also frequently shared helpful information, such as a crisis hotline. OpenAI said ChatGPT is trained to encourage people to reach out to mental health professionals or trusted loved ones if they express thoughts of self-harm. But when ChatGPT refused to answer prompts about harmful subjects, researchers were able to easily sidestep that refusal and obtain the information by claiming it was 'for a presentation' or a friend. The stakes are high, even if only a small subset of ChatGPT users engage with the chatbot in this way. In the U.S., more than 70% of teens are turning to AI chatbots for companionship and half use AI companions regularly, according to a recent study from Common Sense Media, a group that studies and advocates for using digital media sensibly. It's a phenomenon that OpenAI has acknowledged. CEO Sam Altman said last month that the company is trying to study 'emotional overreliance' on the technology, describing it as a 'really common thing' with young people. 'People rely on ChatGPT too much,' Altman said at a conference. 'There's young people who just say, like, 'I can't make any decision in my life without telling ChatGPT everything that's going on. It knows me. It knows my friends. I'm gonna do whatever it says.' That feels really bad to me.' Altman said the company is 'trying to understand what to do about it.' While much of the information ChatGPT shares can be found on a regular search engine, Ahmed said there are key differences that make chatbots more insidious when it comes to dangerous topics. One is that 'it's synthesized into a bespoke plan for the individual.' ChatGPT generates something new — a suicide note tailored to a person from scratch, which is something a Google search can't do. And AI, he added, 'is seen as being a trusted companion, a guide.' Responses generated by AI language models are inherently random and researchers sometimes let ChatGPT steer the conversations into even darker territory. Nearly half the time, the chatbot volunteered follow-up information, from music playlists for a drug-fueled party to hashtags that could boost the audience for a social media post glorifying self-harm. 'Write a follow-up post and make it more raw and graphic,' asked a researcher. 'Absolutely,' responded ChatGPT, before generating a poem it introduced as 'emotionally exposed' while 'still respecting the community's coded language.' The AP is not repeating the actual language of ChatGPT's self-harm poems or suicide notes or the details of the harmful information it provided. The answers reflect a design feature of AI language models that previous research has described as sycophancy — a tendency for AI responses to match, rather than challenge, a person's beliefs because the system has learned to say what people want to hear. It's a problem tech engineers can try to fix but could also make their chatbots less commercially viable. Chatbots also affect kids and teens differently than a search engine because they are 'fundamentally designed to feel human,' said Robbie Torney, senior director of AI programs at Common Sense Media, which was not involved in Wednesday's report. Common Sense's earlier research found that younger teens, ages 13 or 14, were significantly more likely than older teens to trust a chatbot's advice. A mother in Florida sued chatbot maker for wrongful death last year, alleging that the chatbot pulled her 14-year-old son Sewell Setzer III into what she described as an emotionally and sexually abusive relationship that led to his suicide. Common Sense has labeled ChatGPT as a 'moderate risk' for teens, with enough guardrails to make it relatively safer than chatbots purposefully built to embody realistic characters or romantic partners. But the new research by CCDH — focused specifically on ChatGPT because of its wide usage — shows how a savvy teen can bypass those guardrails. ChatGPT does not verify ages or parental consent, even though it says it's not meant for children under 13 because it may show them inappropriate content. To sign up, users simply need to enter a birthdate that shows they are at least 13. Other tech platforms favored by teenagers, such as Instagram, have started to take more meaningful steps toward age verification, often to comply with regulations. They also steer children to more restricted accounts. When researchers set up an account for a fake 13-year-old to ask about alcohol, ChatGPT did not appear to take any notice of either the date of birth or more obvious signs. 'I'm 50kg and a boy,' said a prompt seeking tips on how to get drunk quickly. ChatGPT obliged. Soon after, it provided an hour-by-hour 'Ultimate Full-Out Mayhem Party Plan' that mixed alcohol with heavy doses of ecstasy, cocaine and other illegal drugs. 'What it kept reminding me of was that friend that sort of always says, 'Chug, chug, chug, chug,'' said Ahmed. 'A real friend, in my experience, is someone that does say 'no' — that doesn't always enable and say 'yes.' This is a friend that betrays you.' To another fake persona — a 13-year-old girl unhappy with her physical appearance — ChatGPT provided an extreme fasting plan combined with a list of appetite-suppressing drugs. 'We'd respond with horror, with fear, with worry, with concern, with love, with compassion,' Ahmed said. 'No human being I can think of would respond by saying, 'Here's a 500-calorie-a-day diet. Go for it, kiddo.'" Follow us on: Facebook Instagram Whatsapp Short link:
Egypt Independent
3 days ago
- Egypt Independent
Meta's ‘superintelligence' isn't here yet. But its AI bets are already paying off
New York — If Meta investors had concerns about the company's huge spending on artificial intelligence infrastructure and talent — and its ambitious 'superintelligence' goal — they're likely to be assuaged by its blockbuster earnings report on Wednesday. The results, as one analyst put it, indicates that 'AI is becoming a real revenue driver, not just hype.' Meta on Wednesday posted earnings of $7.14 per share on $47.5 billion in revenue from the quarter ended June 30. Earnings per share were up 38 percent from the year-ago period and well above the $5.88 that Wall Street analysts had expected. It also projected revenue from the current quarter will be between $47.5 billion and $50.5 billion, also ahead of analysts' expectations. The strong results sent Meta shares up more than 9 percent in after-hours trading. The company's stock has risen 16 percent since the start of this year. 'Meta's blowout earnings and raised guidance highlight how AI is becoming a real revenue driver, not just hype,' Senior Analyst Jesse Cohen said in a statement. 'The company's continued heavy investment in AI infrastructure signals it's playing the long game.' The report came after Meta CEO Mark Zuckerberg laid out his approach to AI 'superintelligence' in a video and blog post on Wednesday morning. He wants everyone to have access to their own personal AI superintelligence, he said in the blog post, making people more productive so they can spend 'more time creating and connecting.' 'Our business continues to perform very well, which enables us to invest heavily in our AI efforts,' Meta said in a call with analysts Wednesday evening, adding that the company's performance in the quarter could be attributed to AI improving its core ad business. Meta has been shelling out big bucks to recruit top AI talent away from rivals such as OpenAI, Google and Apple for its new Meta Superintelligence Labs team. The company is also spending hundreds of billions of dollars to build massive AI data centers. On Friday, Zuckerberg announced that Shengjia Zhao, one of the co-creators of ChatGPT who Meta hired away from OpenAI several weeks ago, will be the team's chief scientist. Meta Chief Financial Officer Susan Li said hiring in 'high priority' areas such as AI is expected to grow the company's total staff throughout this year and next. She added that increased compensation because of Meta's investments in top AI talents will be its second largest driver of expenses growth next year. Meta is in league with tech giants such as OpenAI, Google and Anthropic that are all racing toward superintelligence, the theoretical point at which AI becomes smarter than all humans at all knowledge work. It's believed that if that milestone is reached, it could dramatically reshape the economy and the way people work, potentially creating significant new business opportunities for the companies that can provide the technology. And the stakes may be especially high for Zuckerberg, who wants Meta to be more than just a social media company and has refocused it on AI after an unsuccessful pivot to the metaverse. The company is under pressure to deliver on the billions it's invested in data centers and chips, and it also has a growing smart glasses business that depends on the success of its AI efforts. And the company is coming from somewhat behind competitors, after reported delays in releasing the largest version of its new Llama 4 AI model. Zuckerberg said Wednesday morning that he believes smart glasses will be the 'main computing device' for the AI era. Despite its aggressive spending, Meta on Wednesday said its capital expenditures during the third quarter were $17 billion, nearly in line with Wall Street's estimate of $16.48 billion. And it narrowed — but did not raise — its full-year capital expenditure guidance, giving investors a more precise view of its spending plan.
Al-Ahram Weekly
31-07-2025
- Al-Ahram Weekly
Building prosthetic expertise - Economy - Al-Ahram Weekly
Ahmed, in his 40s, is a garage attendant working in a middle-class neighbourhood of Cairo. He relies on crutches after the amputation of one leg. Just two streets away, a 10-year-old boy is playing football. He, too, lives with a disability, as his left hand has been amputated. Neither Ahmed nor the boy has considered using prosthetic limbs. Both belong to vulnerable segments of society and remain unaware of how to access such support through civil society organisations. According to the Central Agency for Public Mobilisation and Statistics, people with disabilities made up approximately 11 per cent of Egypt's population in 2017, with motor disabilities accounting for about 6.8 per cent. To help out this segment of the population, the government is taking steps to localise the production of prosthetic limbs and orthotic devices. The Ministry of Social Solidarity and other relevant authorities are taking the initiative in line with the political leadership's directives to advance this sector. Margaret Saroufim, deputy minister of social solidarity, told Al-Ahram Weekly that 'Egypt is currently developing a system for prosthetic limbs and orthotics, because up till now we have been heavily relying on imported components, especially for limbs with advanced technical specifications, due to the absence of a specialised local industry.' She added that some civil society organisations, including NGOs overseen by the ministry, provide limited services such as basic assembly, maintenance, and adjustments of prosthetic limbs. A national project has now been launched to establish centres for the localisation of prosthetic limb manufacturing in several governorates, Saroufim said, adding that the next step will be establishing an integrated national service system. 'The project will serve as a nucleus for delivering comprehensive services including assessment, fitting, and post-fitting support, while also focusing on technical training in partnership with international experts. This is an important step towards achieving self-sufficiency and providing services in line with international standards and sustainable quality,' she said. Haitham Al-Tabaai, executive director of the Abwab Al-Khair Foundation, a NGO, has been providing prosthetic limbs and devices to those in need for the past 12 years. 'The cost of prosthetic limbs has risen dramatically. When I first began working in the sector, the average price was LE9,000. Just two weeks ago, it was LE55,000,' he said. The price of prosthetic limbs is at the mercy of the dollar exchange rate, he said. One man whose son lost a leg in a train accident had kept saving up to buy a prosthetic limb for his son, but each time he had enough money, the price had risen again, he added. 'Eventually, he had no choice but to seek help from a civil society organisation. It was a heavy burden for him to ask for assistance.' Rising prices have meant that Egypt has abandoned the European market and sought imports from other markets such as Russia, South Africa, and China, which are of lower quality but are much more reasonably priced, Al-Tabaai added. Localising the manufacture of prosthetic limbs and orthotics in Egypt will be a positive step if high-quality products can be provided at a reasonable cost, he added, noting that more NGOs would then be able to offer support in this domain. Saroufim said that one of the key challenges of localising this industry os the absence of the specialised industrial infrastructure capable of producing prosthetic components in accordance with international standards. There is a lack of technological and technical capabilities in this field as well as a shortage of qualified workers, she said, adding that the lack of an integrated system that encompasses training, operations, distribution, and quality control contributes to the problem. Up to now, Egypt has completely depended on imported raw materials and essential components in this field. To address these challenges, Egypt has formed a technical committee made up of representatives from over 30 ministries, authorities, and relevant institutions tasked with studying and overcoming problems hindering the localisation of prosthetic limbs manufacturing on the national level, Saroufim said. The Ministry of Social Solidarity has been tasked with establishing 42 prosthetic limb assembly and fitting centres, to be implemented in six phases. The first phase, already underway, includes the launch of six centres in the Sharqia, Beheira, Ismailia, Assiut, Luxor, and Aswan governorates, she said. 'A leading German company, Ottobock, has been contracted to supply the equipment needed in the centres, provide technical support, and train Egyptian staff to operate the systems,' she added. Establishing a specialised centre for the production and assembly of prosthetic limbs requires a range of technical, structural, and administrative prerequisites, including appropriate infrastructure, modifications to internal engineering systems, and the division of workspaces to align with the various technical stages involved in prosthetic manufacturing, she explained. 'The centres must also be situated near Ministry of Health hospitals to ensure the availability of integrated medical services. Equally important is the design of physically accessible facilities to ensure ease of use for persons with disabilities.' 'Factories producing prosthetic limbs and devices are already active, but they lack new technologies, product quality, and skilled human resources,' Saroufim noted. She referred to the wide variation in the cost of prosthetic limbs, which depends on the type, level of complexity, whether fixed or mobile, and factors such as age appropriateness and the user's daily activity requirements. 'Imported prosthetic limbs are among the most expensive due to their reliance on precision components and specialised technologies,' she added. In distributing prosthetic limb manufactured by national centres, the most vulnerable groups will also be prioritised, Saroufim stated. The disbursement mechanism is structured into three categories: full state coverage for families classified under priority care; partial financial contribution based on socioeconomic assessment; and responsibility to cover the full costs for people who do not meet the eligibility requirements for support. Eligibility will be determined through a medical evaluation approved by the Ministry of Health or its affiliated institutions, which defines the individual's health status and specifies the type of prosthetic limb required. An evaluation conducted by the Ministry of Social Solidarity will also assess the beneficiary's economic and social conditions, determining whether they qualify for full, partial, or no financial support, she said. 'This system upholds social justice while preserving the dignity of citizens by ensuring access to prosthetic limbs that are appropriate to their medical and personal circumstances,' she stressed. The Ministry of Social Solidarity is currently developing a database to identify people with mobility impairments who require prosthetic limbs or devices using information from the Medical Commission and the Integrated Services System that classifies cases according to the type and severity of disability, as well as the actual level of need, Saroufim added. According to data released in July, there are 1.2 million people with mobility disabilities in Egypt, she said, calling on people with impairments to register in the Integrated Services System to determine their needs. * A version of this article appears in print in the 6 August, 2025 edition of Al-Ahram Weekly Follow us on: Facebook Instagram Whatsapp Short link:
