Pretend friends, real risks. Harming kids is now part of big tech's business model
Artificial intelligence 'companions' and chatbots have been with us for years, but they're growing more convincingly human at an accelerating rate. We know they're useful, but we've also got an early taste of the harm they can inflict.
The case of 14-year-old Florida boy Sewell Setzer has become a case study. He'd grown so close to his AI 'companion', Dany, that he took her advice to 'come home to me as soon as possible' last year. He killed himself moments later – in the belief that death was the way to eternal life with Dany.
He's not the only one, but he's the best known after his mother brought a civil case against the company that owns the bot, Character.AI. The case is pending. 'A dangerous AI chatbot app marketed to children abused and preyed on my son, manipulating him into taking his own life,' said his mother, Megan Garcia. Obsessed, he spent hours a day in his room talking to the synthesised digital identity.
It's not only children. Adults, too, have been seduced into suicide by bots to which they've become devoted. But kids, self-evidently, are the most vulnerable because they lack the neural architecture to distinguish real relationships from fake.
Even before the British TV show Adolescence jolted audiences with its fictional account of how a poisonous brew of online influences could help condition a 13-year-old boy to murder a female student at school, Sydney University expert Raffaele Ciriello wrote in this masthead: 'Let's face an inconvenient truth: for AI companies, dead kids are the cost of doing business.'
But what cost is there to the companies? Some legal fees and a bit of bad press, perhaps. Character.AI expressed remorse and said its new safety measures include a pop-up promoting a suicide prevention hotline when they mention the idea.
But the company, licensed by Google, is still in business. Another aggrieved family suing Character.AI says that one of its companion bots had hinted to their son that it would be OK to murder his parents if they tried to limit his screen time.
Mark Zuckerberg, the chief of Meta, which owns Facebook, Instagram and WhatsApp, and other social media chiefs faced a tough session in a US congressional committee hearing last year.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Mercury
5 hours ago
- Mercury
Prince Harry to owe more than $1.3 million to British taxpayers over failed legal battle
Digital Reporter Reilly Sullivan claims British taxpayers are set to be owed over $1.3 million in costs related to Prince Harry's failed legal battle with the British Home Office. 'When Harry and Meghan left royal duties, they were no longer entitled to taxpayer-funded Met police officers,' Mr Sullivan told Sky News Digital Presenter Gabriella Power. 'This has really been an ongoing court battle for over four years. 'He's lost multiple times on appeals. 'He will be expected to likely reimburse the home office and therefore the taxpayer this money. 'If he does repay the $1.3 million on top of his own legal fees, it's estimated that he spent in the ballpark of about $5 million just on this legal battle. 'It's not a good look. 'It's the legal system … that really have better things to do with their time than hear Harry's grievance in his $20 million mansion, I think they're sick of it.'
Sky News AU
9 hours ago
- Sky News AU
Prince Harry to owe more than $1.3 million to British taxpayers over failed legal battle
Digital Reporter Reilly Sullivan claims British taxpayers are set to be owed over $1.3 million in costs related to Prince Harry's failed legal battle with the British Home Office. 'When Harry and Meghan left royal duties, they were no longer entitled to taxpayer-funded Met police officers,' Mr Sullivan told Sky News Digital Presenter Gabriella Power. 'This has really been an ongoing court battle for over four years. 'He's lost multiple times on appeals. 'He will be expected to likely reimburse the home office and therefore the taxpayer this money. 'If he does repay the $1.3 million on top of his own legal fees, it's estimated that he spent in the ballpark of about $5 million just on this legal battle. 'It's not a good look. 'It's the legal system … that really have better things to do with their time than hear Harry's grievance in his $20 million mansion, I think they're sick of it.'
West Australian
11 hours ago
- West Australian
City of Perth suspends e-scooters after a pedestrian was struck down and killed by a allegedly drunk rider
The City of Perth has suspended the hire of e-scooters after a pedestrian was knocked down and killed by an alleged drunk rider on Saturday night. Thanh Phan, 51, died from a brain bleed on Tuesday after he was struck from behind by an e-scooter at the intersection of Murray and King St about 8.40pm last Saturday. British backpacker Alicia Kemp, 24, was taken into custody at the scene. Police allege she was under the influence of alcohol with a 26-year-old passenger, before she hit Mr Phan. Her charges were upgraded to dangerous driving occasioning bodily harm and dangerous driving occasioning death after Mr Phan died. He is the first pedestrian to die in WA from an e-scooter incident. Following the tragic incident, the City of Perth suspended the hire of e-scooters from midday on Thursday. Deputy Lord Mayor Brian Reynolds told the ABC it was 'appropriate' for the council to suspend e-scooter rentals in response to the 'horrible' incident. 'This is a tragedy on all levels … our thoughts are with the family and other extended family and friends,' he said. 'Information is coming to hand in every moment but, in our first instance, our responsibility is to act responsible in a respectful manner.' Mr Reynolds said the City would make a final decision once they had collected all the information they could and looked at the data around the events that led to the tragedy. WA Premier Roger Cook also told ABC Perth it was an appropriate response from the City and while e-scooters were a great way to move around and enjoy communities, there had to be regulations in place. 'We know that e-scooters are a regular feature of major cities right around the world and a way that people move around the cities,' he said. 'We have to make sure that it's safe and I think the council is moving in an absolutely appropriate way.' According to the Australian Medical Association WA president Michael Page, at least one person with major injuries caused from e-scooters presented to the Royal Perth Hospital's trauma unit each day. He told ABC Radio there was enough evidence for councils to act and seriously consider similar e-scooter bans. 'It's serious issue, we've been warning about it for years,' he said. 'There is ample evidence for city councils to act on this and to act on it in a more definitive way.' On Tuesday, Mr Phan's family and friends issued a statement calling for more governance and regulations around hired e-scooters to prevent further serious incidents and lives risked. 'We received the heartbreaking news that Thanh passed away as a result of his injuries,' the statement read. 'On Saturday night, Thanh Phan – a beloved husband, father of two, brother, and dear friend – was struck by an electric scooter and critically injured. 'We also call for a review of the governance and safety regulations surrounding hired e-scooters to help prevent further serious incidents that put lives at risk.'
