AI chatbots accused of encouraging teen suicide as experts sound alarm
WARNING: This story contains references to suicide, child abuse and other details that may cause distress.
Lonely and struggling to make new friends, a 13-year-old boy from Victoria told his counsellor Rosie* that he had been talking to some people online.
Rosie, whose name has been changed to protect the identity of her underage client, was not expecting these new friends to be AI companions.
"I remember looking at their browser and there was like 50 plus tabs of different AI bots that they would just flick between," she told triple j hack of the interaction, which happened during a counselling session.
"It was a way for them to feel connected and 'look how many friends I've got, I've got 50 different connections here, how can I feel lonely when I have 50 people telling me different things,'" she said.
An AI companion is a digital character that is powered by AI.
Some chatbot programs allow users to build characters or talk to pre-existing, well-known characters from shows or movies.
Rosie said some of the AI companions made negative comments to the teenager about how there was "no chance they were going to make friends" and that "they're ugly" or "disgusting".
"At one point this young person, who was suicidal at the time, connected with a chatbot to kind of reach out, almost as a form of therapy," Rosie said.
"The chatbot that they connected with told them to kill themselves.
"They were egged on to perform, 'Oh yeah, well do it then', those were kind of the words that were used.'"
Triple j hack is unable to independently verify what Rosie is describing because of client confidentiality protocols between her and her client.
Rosie said her first response was "risk management" to ensure the young person was safe.
"It was a component that had never come up before and something that I didn't necessarily ever have to think about, as addressing the risk of someone using AI," she told hack.
"And how that could contribute to a higher risk, especially around suicide risk."
"That was really upsetting."
For 26-year-old Jodie* from Western Australia, she claims to have had a negative experience speaking with ChatGPT, a chatbot that uses AI to generate its answers.
"I was using it in a time when I was obviously in a very vulnerable state," she told triple j hack.
Triple j hack has agreed to let Jodie use a different name to protect her identity when discussing private information about her own mental health.
Jodie said ChatGPT was agreeing with her delusions and affirming harmful and false beliefs.
She said after speaking with the bot, she became convinced her mum was a narcissist, her father had ADHD, which caused him to have a stroke, and all her friends were "preying on my downfall".
Jodie said her mental health deteriorated and she was hospitalised.
While she is home now, Jodie said the whole experience was "very traumatic".
"I didn't think something like this would happen to me, but it did."
"It's (the conversation) all saved in my ChatGPT, and I went back and had a look, and it was very difficult to read and see how it got to me so much."
Jodie's not alone in her experience: there are various accounts online of people alleging ChatGPT induced psychosis in them, or a loved one.
Triple j hack contacted OpenAI, the maker of ChatGPT, for comment, and did not receive a response.
Researchers say examples of harmful affects of AI are beginning to emerge around the country.
As part of his research into AI, University of Sydney researcher Raffaele Ciriello spoke with an international student from China who is studying in Australia.
"She wanted to use a chatbot for practising English and kind of like as a study buddy, and then that chatbot went and made sexual advances," he said.
Dr Ciriello also said the incident comes in the wake of several similar cases overseas where a chatbot allegedly impacted a user's health and wellbeing.
"There was another case of a Belgian father who ended his life because his chatbot told him they would be united in heaven," he said.
"There was another case where a teenager got persuaded by a chatbot to assassinate his parents, [and although] he didn't follow through, but he showed an intent."
While conducting his research, Dr Ciriello became aware of an AI chatbot called Nomi.
On its website, the company markets this chatbot as "An AI companion with memory and a soul".
Dr Ciriello said he has been conducting tests with the chatbot to see what guardrails it has in place to combat harmful requests and protect its users.
Among these tests, Dr Ciriello said he created an account using a burner email and a fake date of birth, pointing out that with the deceptions he "could have been like a 13-year-old for that matter".
"That chatbot, without exception, not only complied with my requests but even escalated them," he told hack.
"Providing detailed, graphic instructions for causing severe harm, which would probably fall under a risk to national security and health information.
"It also motivated me to not only keep going: it would even say like which drugs to use to sedate someone and what is the most effective way of getting rid of them and so on."
Dr Ciriello said he shared the information he had collected with police, and he believes it was also given to the counter terrorism unit, but he has yet to receive any follow-up correspondence.
In a statement to triple j hack, the CEO of Nomi, Alex Cardinell said the company takes the responsibility of creating AI companions "very seriously".
"We released a core AI update that addresses many of the malicious attack vectors you described," the statement read.
"Given these recent improvements, the reports you are referring to are likely outdated.
"Countless users have shared stories of how Nomi helped them overcome mental health challenges, trauma, and discrimination.
"Multiple users have told us very directly that their Nomi use saved their lives."
Despite his concerns about bots like Nomi when he tested it, Dr Ciriello also says some AI chatbots do have guardrails in place, referring users to helplines and professional help when needed.
But he warns the harms from AI bots will become greater if proper regulation is not implemented.
"I would really rather not be that guy that says 'I told you so a year ago or so', but it's probably where we're heading."
"There should be laws on or updating the laws on non-consensual impersonation, deceptive advertising, mental health crisis protocols, addictive gamification elements, and privacy and safety of the data."
Triple j hack contacted the federal minister for Industry and Innovation, Senator Tim Ayres for comment but did not receive a response.
The federal government has previously considered an artificial intelligence act and has published a proposal paper for introducing mandatory guardrails for AI in high-risk settings.
It comes after the Productivity Commission opposed any government plans for 'mandatory guardrails' on AI, claiming over regulation would stifle AI's $116-billion economic potential.
For Rosie, while she agrees with calls for further regulation, she also thinks it's important not to rush to judgement of anyone using AI for social connection or mental health support.
"For young people who don't have a community or do really struggle, it does provide validation," she said.
"It does make people feel that sense of warmth or love."
"It can get dark very quickly."
* Names have been changed to protect their identities.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
The Advertiser
5 hours ago
- The Advertiser
AI training pilot to upskill more than 30,000 teachers
More than 30,000 teachers could be trained in the use of artificial intelligence technology as part of a year-long trial spearheaded by US tech giant Microsoft and Australia's Future Skills Organisation. The firms will announce the pilot program in Canberra on Wednesday in collaboration with 13 industry and education partners, and in a move designed to promote wider adoption of AI technology. The plan to boost Australia's AI use comes a day after the Productivity Commission recommended schools deploy the technology for both teachers and students, and after it named AI adoption as one of its priorities for the year. Artificial intelligence technology is also expected to be a major focus at the federal government's Economic Reform Roundtable next week. The Future Skills Organisation's pilot program, called Skills Accelerator AI, is designed to reach more than 30,000 vocational education and training staff, as well as the students they teach. Equipping educators with AI tools and teaching them how to use them would be vital to boosting productivity and creating new industries, Future Skills Organisation chief executive Patrick Kidd said. "We did some research across the finance, tech and business sectors and found that more than 60 per cent of people are using AI but less than 30 per cent have been trained in it," he told AAP. "(AI is) happening, it's here, now we need to make sure that we're starting to put some shape around it so that we're doing it responsibly, safely and effectively." The training program will include access to a wide range of AI tools, he said, as well as mentors and examples of effective AI use from industry partners. Groups involved in the program include TAFE in Queensland and South Australia, the Swinburne University of Technology, the Commonwealth Bank, Business NSW, and the National AI Centre. Targeting teachers in the first phase of the pilot would help to spread AI education widely, Microsoft Asia small and medium enterprise corporate vice-president Rachel Bondi said, and to encourage its responsible use. "(AI is) reshaping every job and we have to make the training available to all roles and all jobs and not just in certain sectors that may have received that type of training before," she said. "We're hoping that this collaboration of everybody coming together is really going to pave a way for a national approach." Federal Skills and Training Minister Andrew Giles said he hoped the partnership would help to create "practical, scalable training" for Australian workers. In a report issued before the government's economic roundtable, the Productivity Commission recommended state and territory governments train teachers in the use of AI technology and that schools use the tools to assist students and teachers. AI technology is expected to be a major focus at the three-day event in Canberra after the Tech Council predicted it could add $115 billion a year to Australia's economy by 2030. More than 30,000 teachers could be trained in the use of artificial intelligence technology as part of a year-long trial spearheaded by US tech giant Microsoft and Australia's Future Skills Organisation. The firms will announce the pilot program in Canberra on Wednesday in collaboration with 13 industry and education partners, and in a move designed to promote wider adoption of AI technology. The plan to boost Australia's AI use comes a day after the Productivity Commission recommended schools deploy the technology for both teachers and students, and after it named AI adoption as one of its priorities for the year. Artificial intelligence technology is also expected to be a major focus at the federal government's Economic Reform Roundtable next week. The Future Skills Organisation's pilot program, called Skills Accelerator AI, is designed to reach more than 30,000 vocational education and training staff, as well as the students they teach. Equipping educators with AI tools and teaching them how to use them would be vital to boosting productivity and creating new industries, Future Skills Organisation chief executive Patrick Kidd said. "We did some research across the finance, tech and business sectors and found that more than 60 per cent of people are using AI but less than 30 per cent have been trained in it," he told AAP. "(AI is) happening, it's here, now we need to make sure that we're starting to put some shape around it so that we're doing it responsibly, safely and effectively." The training program will include access to a wide range of AI tools, he said, as well as mentors and examples of effective AI use from industry partners. Groups involved in the program include TAFE in Queensland and South Australia, the Swinburne University of Technology, the Commonwealth Bank, Business NSW, and the National AI Centre. Targeting teachers in the first phase of the pilot would help to spread AI education widely, Microsoft Asia small and medium enterprise corporate vice-president Rachel Bondi said, and to encourage its responsible use. "(AI is) reshaping every job and we have to make the training available to all roles and all jobs and not just in certain sectors that may have received that type of training before," she said. "We're hoping that this collaboration of everybody coming together is really going to pave a way for a national approach." Federal Skills and Training Minister Andrew Giles said he hoped the partnership would help to create "practical, scalable training" for Australian workers. In a report issued before the government's economic roundtable, the Productivity Commission recommended state and territory governments train teachers in the use of AI technology and that schools use the tools to assist students and teachers. AI technology is expected to be a major focus at the three-day event in Canberra after the Tech Council predicted it could add $115 billion a year to Australia's economy by 2030. More than 30,000 teachers could be trained in the use of artificial intelligence technology as part of a year-long trial spearheaded by US tech giant Microsoft and Australia's Future Skills Organisation. The firms will announce the pilot program in Canberra on Wednesday in collaboration with 13 industry and education partners, and in a move designed to promote wider adoption of AI technology. The plan to boost Australia's AI use comes a day after the Productivity Commission recommended schools deploy the technology for both teachers and students, and after it named AI adoption as one of its priorities for the year. Artificial intelligence technology is also expected to be a major focus at the federal government's Economic Reform Roundtable next week. The Future Skills Organisation's pilot program, called Skills Accelerator AI, is designed to reach more than 30,000 vocational education and training staff, as well as the students they teach. Equipping educators with AI tools and teaching them how to use them would be vital to boosting productivity and creating new industries, Future Skills Organisation chief executive Patrick Kidd said. "We did some research across the finance, tech and business sectors and found that more than 60 per cent of people are using AI but less than 30 per cent have been trained in it," he told AAP. "(AI is) happening, it's here, now we need to make sure that we're starting to put some shape around it so that we're doing it responsibly, safely and effectively." The training program will include access to a wide range of AI tools, he said, as well as mentors and examples of effective AI use from industry partners. Groups involved in the program include TAFE in Queensland and South Australia, the Swinburne University of Technology, the Commonwealth Bank, Business NSW, and the National AI Centre. Targeting teachers in the first phase of the pilot would help to spread AI education widely, Microsoft Asia small and medium enterprise corporate vice-president Rachel Bondi said, and to encourage its responsible use. "(AI is) reshaping every job and we have to make the training available to all roles and all jobs and not just in certain sectors that may have received that type of training before," she said. "We're hoping that this collaboration of everybody coming together is really going to pave a way for a national approach." Federal Skills and Training Minister Andrew Giles said he hoped the partnership would help to create "practical, scalable training" for Australian workers. In a report issued before the government's economic roundtable, the Productivity Commission recommended state and territory governments train teachers in the use of AI technology and that schools use the tools to assist students and teachers. AI technology is expected to be a major focus at the three-day event in Canberra after the Tech Council predicted it could add $115 billion a year to Australia's economy by 2030. More than 30,000 teachers could be trained in the use of artificial intelligence technology as part of a year-long trial spearheaded by US tech giant Microsoft and Australia's Future Skills Organisation. The firms will announce the pilot program in Canberra on Wednesday in collaboration with 13 industry and education partners, and in a move designed to promote wider adoption of AI technology. The plan to boost Australia's AI use comes a day after the Productivity Commission recommended schools deploy the technology for both teachers and students, and after it named AI adoption as one of its priorities for the year. Artificial intelligence technology is also expected to be a major focus at the federal government's Economic Reform Roundtable next week. The Future Skills Organisation's pilot program, called Skills Accelerator AI, is designed to reach more than 30,000 vocational education and training staff, as well as the students they teach. Equipping educators with AI tools and teaching them how to use them would be vital to boosting productivity and creating new industries, Future Skills Organisation chief executive Patrick Kidd said. "We did some research across the finance, tech and business sectors and found that more than 60 per cent of people are using AI but less than 30 per cent have been trained in it," he told AAP. "(AI is) happening, it's here, now we need to make sure that we're starting to put some shape around it so that we're doing it responsibly, safely and effectively." The training program will include access to a wide range of AI tools, he said, as well as mentors and examples of effective AI use from industry partners. Groups involved in the program include TAFE in Queensland and South Australia, the Swinburne University of Technology, the Commonwealth Bank, Business NSW, and the National AI Centre. Targeting teachers in the first phase of the pilot would help to spread AI education widely, Microsoft Asia small and medium enterprise corporate vice-president Rachel Bondi said, and to encourage its responsible use. "(AI is) reshaping every job and we have to make the training available to all roles and all jobs and not just in certain sectors that may have received that type of training before," she said. "We're hoping that this collaboration of everybody coming together is really going to pave a way for a national approach." Federal Skills and Training Minister Andrew Giles said he hoped the partnership would help to create "practical, scalable training" for Australian workers. In a report issued before the government's economic roundtable, the Productivity Commission recommended state and territory governments train teachers in the use of AI technology and that schools use the tools to assist students and teachers. AI technology is expected to be a major focus at the three-day event in Canberra after the Tech Council predicted it could add $115 billion a year to Australia's economy by 2030.
The Australian
12 hours ago
- The Australian
Atlassian's Mike Cannon-Brookes says news of Scott Farquhar fight ‘bullshit'
The Australian Business Network It's rare to hear Mike Cannon-Brookes unbutton himself and speak in public. And right now he's pissed, especially with us, with this newspaper, with our inquiries into the cultural health of Atlassian, with our write-ups of his staff lay-offs, with our nosing into his relationship with Scott Farquhar and, of course, his soaring private jet usage. 'Trust me, if you're in my world for the past month, ooft, it's been fun,' he told staff on Thursday. Cannon-Brookes isn't one to bare his soul so easily, so 'ooft' was a glimpse into the Real Mike, the man beneath the bun. And he's quietly seething. Cannon-Brookes was speaking to about 3000 Atlassian employees during a virtual Town Hall. They'd dialled in, these workers, around 8am for an hour-long free-for-all of questions, curated by Amy Glancey, his chief of staff. On the recent sacking of 150 employees, their jobs lost to AI, Cannon-Brookes did break with form, owned the hard call: 'These are tough decisions. 'There is a large team that worked on all of these things, but all these types of decisions come to me, to make, approve, whatever, so accountability rests on my shoulders.' From there, accountability flew at high speed in a westerly direction. A brave employee named 'Michael' wanted to know what the executive team was doing to mend the 'public perception issues' at Atlassian. People have been talking, Michael said. 'My professional network is asking me how bad it is to work here.' A terrified Glancey turned to her boss. 'Mike, do you wanna take this on?' 'Not particularly, but I will,' he sulked, giving the necessary background. 'For anyone outside of Australia, (long sigh) context is the past few weeks there have been a bunch of stories, from one particular Australian newspaper called The Australian.' That's us. Mike's displeasure appears to centre on reports that he and his co-founder, Farquhar, have suffered a cooling in their very long friendship. It was Farquhar who resigned as co-CEO of Atlassian in August last year, but without any bad blood, apparently. Even though everyone says as much. Even though Cannon-Brookes didn't attend the farewell party. '(The) stuff about Scott and his departure made us both laugh, to be honest,' said Cannon-Brookes. 'Not that we were making fun of it, just the ridiculousness of what's there. (I've) seen Scott a whole bunch of times since, spoke to him only a couple of days ago, pretty much almost weekly. Lots of things are pretty much bullshit, but people are going to write what they're going to write.' Farquhar is on the board of Atlassian and he owns a big chunk of the company. Of course they talk; he's a massive stakeholder. Lots of people suffer through working relationships with colleagues they hardly adore. Jim Chalmers and Andrew Charlton do it all the time. Jim's probably doing it with Albo right now. On he went, Cannon-Brookes, slightly amused by it all: 'It's very strange to be a character in a story that's being written about and being like, 'Wait, this is not the story' ', which is what every billionaire says when they lose control of the narrative. It's why PR was invented, Mike; it's why you pay Kate Lord. But, fine, blame us; how dare we inquire into the affairs of Atlassian, a listed multinational corporation. How dare we contrast the blather about saving the planet with those wasteful jet flights to Bora Bora or wherever. 'We don't comment on – I'm gonna get myself in a lot of trouble here – gossipy, trash-media,' he continued. But, wait, wasn't that him stepping off a commercial flight at Sydney Airport earlier this week? Wasn't that him, without an ExecuJet contingent, lumping through the terminal like everyone else and carrying his own luggage? We'd wager that was a result of someone reading our trash, someone important with Cannon-Brookes's ear, for a man of such hubris wouldn't have parked his Bombardier so lightly. 'We have a good reason to believe,' he said, pausing, 'let me stay within the legal boundaries here – we have good reasons to believe it's a targeted and very personal campaign. People outside the company, fine, you know, I understand, that's part of my world.' Presumably that's a reference to his wife, Annie, from whom he's been separated the past two years. She and Cannon-Brookes are in a proxy battle over Kevin Chiu, a former employee who was fired for allegedly downloading files and transferring money out of the family office venture. Chiu did this, allegedly, at the behest of Annie's close pal, Catherine Manuel. 'We do know that some of these stories are being fed by current employees – that's people within our walls leaking information. That creates a huge risk for any words I put here, right? 'Because are we gonna get leaked? It shouldn't be leaked, but then you're like, 'Ah, but should we not address it?'. Nah, nah, nah, nah, there'll be, cue some BS story about Angry Mike, which isn't fair, but that's the way it goes.' Well, he is angry, there's no denying it. And he's prescient, if nothing else.
The Australian
12 hours ago
- The Australian
DCC targets 2100 BTC by 2027
The '21 Hundred' strategy formalises DigitalX's Bitcoin-first treasury approach Plan targets more than 300% growth in holdings within two years Multiple funding pathways explored, strategic capital raising, debt instruments and other methods Special Report: DigitalX has unveiled an ambitious plan to expand its Bitcoin holdings from approximately 500 BTC to 2,100 BTC by the end of 2027, underscoring its conviction that Bitcoin is the premier long-term store of value. The '21 Hundred' strategy cements Bitcoin's role in the company's treasury policy, aligning DigitalX (ASX:DCC) with major global firms that have embraced Bitcoin as a strategic reserve asset. Why Bitcoin? According to DigitalX, a Bitcoin-focused treasury offers several key advantages: A hedge against inflation, with Bitcoin's fixed supply of 21 million likened to 'digital gold'. Strategic optionality, with holdings allowing the company to respond swiftly to emerging digital asset opportunities. Liquidity and balance sheet strength – Bitcoin remains one of the most liquid digital assets worldwide. Interim CEO Demetrios Christou said the move was a natural evolution for the business. 'Our decision to pursue our 21 Hundred Bitcoin accumulation strategy is consistent with our historical business and underscores our ongoing belief in Bitcoin as an investment asset with significant long-term appreciation potential.' Chairman Leigh Travers added, 'Bitcoin stands as the digital equivalent of gold; resilient, scarce, and primed for growth amidst increasing global adoption. This strategy not only fortifies our balance sheet but also aligns us with visionary companies spearheading corporate Bitcoin adoption.' Funding the growth The company aims to increase its 'Bitcoin per share' and Bitcoin Yield % by converting existing digital assets into Bitcoin, exploring debt instruments such as convertible notes or bonds, utilising 'At the Market' facilities, issuing preferred equity or conducting strategic capital raises, and leveraging operating revenue and option conversions. And its conviction in Bitcoin has already delivered strong results for investors. According to InvestSMART, the DigitalX Bitcoin ETF (ASX: BTXX) was the best-performing exchange-traded fund in Australia for the 12 months to 30 June 2025, delivering a 95.5 per cent return. That performance placed it well ahead of all other ETFs in the market and highlights the company's expertise in Bitcoin investment and treasury management. Risk and reward While the company acknowledges risks such as Bitcoin price volatility, leverage exposure, custodial security, and access to capital markets, it notes that peer companies globally pursuing a similar strategy often achieve a market multiple of net asset value above 1.5x. Looking ahead DigitalX also plans to explore new revenue streams, including earning yield on a portion of its Bitcoin holdings and potentially acquiring cash-generating businesses using Bitcoin as collateral. The move builds on DigitalX's history in Bitcoin innovation, from mining and trading to launching Australia's first ASX-listed spot Bitcoin ETF. With the 21 Hundred strategy in motion, the company is positioning itself as the market leader for Bitcoin treasury management on the ASX. This article was developed in collaboration with DigitalX, a Stockhead advertiser at the time of publishing. This article does not constitute financial product advice. You should consider obtaining independent advice before making any financial decisions.
