Top scientist wants to prevent AI from going rogue
Canadian computer science professor Yoshua Bengio is considered one of the godfathers of the artificial intelligence revolution and on Tuesday announced the launch of LawZero, a non-profit organization intended to mitigate the technology's inherent risks.
The winner of the Turing Award, also known as the Nobel Prize for computer science, has been warning for several years of the risks of AI, whether through its malicious use or the software itself going awry.
Those risks are increasing with the development of so-called AI agents, a use of the technology that tasks computers with making decisions that were once made by human workers.
The goal of these agents is to build virtual employees that can do practically any job a human can, at a fraction of the cost.
"Currently, AI is developed to maximize profit," Bengio said, adding it was being deployed even as it persists to show flaws.
Moreover, for Bengio, giving AI human-like agency will easily be used for malicious purposes such as disinformation, bioweapons, and cyberattacks.
"If we lose control of rogue super-intelligent AIs, they could greatly harm humanity," he said.
One of the first objectives at LawZero will be to develop Scientist AI, a form of specially trained AI that can be used as a guardrail to ensure other AIs are behaving properly, the company said.
The organization already has over 15 researchers and has received funding from Schmidt Sciences, a charity set up by former Google boss Eric Schmidt and his wife Wendy.
The project comes as powerful large language models (or LLMs) from OpenAI, Google and Anthropic are deployed across all sectors of the digital economy, while still showing significant problems.
These include AI models that show a capability to deceive and fabricate false information even as they increase productivity.
In a recent example, AI company Anthropic said that during safety testing, its latest AI model tried to blackmail an engineer to avoid being replaced by another system.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Euronews
10 hours ago
- Euronews
ABBA's Björn Ulvaeus discusses writing musical with AI and ABBA future
ABBA's Björn Ulvaeus was at the inaugural edition of London's SXSW festival yesterday and revealed he is writing a new musical using AI. He referred to artificial intelligence as 'such a great tool' and discussed his project during a talk at SXSW London. 'It is like having another songwriter in the room with a huge reference frame,' he said. 'It is really an extension of your mind. You have access to things that you didn't think of before.' Ulvaeus discussed the technology's limitations, saying that it is 'very bad at lyrics' and that he believed AI's most useful application was to help artists overcome writer's block. 'You can prompt a lyric you have written about something, and you're stuck maybe, and you want this song to be in a certain style,' he explained. 'You can ask it, how would you extend? Where would you go from here? It usually comes out with garbage, but sometimes there is something in it that gives you another idea.' Une publication partagée par CISAC (@cisacnews) Ulvaeus previously warned of the 'existential challenge' AI represents to the music industry. He is the president of the International Confederation of Societies of Authors and Composers (CISAC), a non-profit organisation that represents songwriters and composers around the world, collecting and paying royalties to its members whose music has been used in broadcasts and on streaming services. The organisation has produced reports on AI use in music. Most recently, one of their studies suggested that music creators could lose nearly a quarter of their income to AI by 2028. Regarding this report, Ulvaeus stated that governments have the power to step in and give a helping hand to creatives. 'For creators of all kinds, from songwriters to film directors, screenwriters to film composers, AI has the power to unlock new and exciting opportunities — but we have to accept that, if badly regulated, generative AI also has the power to cause great damage to human creators, to their careers and livelihoods.' 'Which of these two scenarios will be the outcome?' Ulvaeus continued. 'This will be determined in large part by the choices made by policy makers, in legislative reviews that are going on across the world right now. It's critical that we get these regulations right, protect creators' rights and help develop an AI environment that safeguards human creativity and culture.' During the SXSW discussion in London, Ulvaeus also noted that he was 'three quarters' of the way through writing the follow-up to the Swedish legends' hologram-based ABBA Voyage concert series. ABBA has just celebrated the third anniversary of their acclaimed virtual concert experience 'Voyage' by introducing new songs to the setlist. ABBA Voyage first kicked off in May 2022, and was due to wrap in November 2024, but has since been extended to January 2026 due to overwhelming demand. Elswehere, SXSW London has faced intense criticism after former UK prime ministers Tony Blair and David Cameron were among the unannounced speakers. Screenshots were leaked of the un-shared programme that included Blair talking on a panel called Government and AI, which also featured Technology Secretary and Labour Friends of Israel member Peter Kyle. Blair spoke at the conference's opening day, saying that Britain needs to fully embrace artificial intelligence in public services and that we 'could have AI tutors' along with 'AI nurses, AI doctors'. The panel appearance, which was not announced to the public or artists, prompted many artists to cancel their planned performances at the festival. Sam Akpro, Rat Party, Magnus Westwell, Saliah and LVRA were amongst the artists who pulled out, with the latter accusing the festival of 'artwashing', saying that 'whilst the music team were pulling together a diverse, 'cool' lineup, the conference team were booking speakers from multiple organisations deeply complicit in the current genocide of Palestinian people.' 'I implore artists to engage, rather than ignore, those things that affect us and strive to protect the most marginalised voices in the world,' LVRA added. 'I urge us as a community to think bigger, and better, than the scraps offered to us today. Morten Harket, frontman of celebrated Norwegian synth-pop band A-Ha, has revealed that he has Parkinson's disease. The news was shared by the band in a statement on their website which read: 'This isn't the sort of news anyone wants to deliver to the world, but here it is – Morten has Parkinson's disease.' The pop icon, aged 65, shared further details of the diagnosis in the post, and explained why he has sharing the news after previously keeping details on his health 'strictly private'. 'I've got no problem accepting the diagnosis. With time I've taken to heart my 94-year-old father's attitude to the way the organism gradually surrenders: 'I use whatever works',' he wrote. 'Part of me wanted to reveal it. Like I said, acknowledging the diagnosis wasn't a problem for me; it's my need for peace and quiet to work that has been stopping me. I'm trying the best I can to prevent my entire system from going into decline.' Harket said he underwent neurological procedures to have electrodes implanted inside his brain last year and that this had reduced the symptoms. He continued: 'It's a difficult balancing act between taking the medication and managing its side effects. There's so much to weigh up when you're emulating the masterful way the body handles every complex movement, or social matters and invitations, or day-to-day life in general.' Regarding whether Harket can still perform and sing, he wrote: 'I don't really know. I don't feel like singing, and for me that's a sign. I'm broadminded in terms of what I think works; I don't expect to be able to achieve full technical control. The question is whether I can express myself with my voice. As things stand now, that's out of the question. But I don't know whether I'll be able to manage it at some point in the future.' Parkinson's is the second most common neurodegenerative disorder in the world, behind Alzheimer's. It causes deterioration in the brain's nervous system, leading to tremors and other symptoms that can become progressively worse over time. Common symptoms include involuntary shaking, slower-than-usual movement, and stiffness in the muscles. The disease can be treated with surgery and medication, but there is no cure. It is not known what exactly leads to people developing the condition. Other famous faces who have had Parkinson's diagnoses include Back To The Future actor Michael J. Fox, heavy metal legend Ozzy Osbourne and Scottish comedian Billy Connolly.
France 24
11 hours ago
- France 24
Sci-fi writer Charles Stross' dark take on Silicon Valley 'religion'
Beyond an exhilarating story, Stross' 2005 book "Accelerando" was a thought experiment with ideas like transhumanism, technological "singularity" and rationalism -- concepts that had been circulating in Silicon Valley from the late 1980s -- and which many believe still animate powerful figures like Elon Musk and Peter Thiel. "I was basically trying to bottle up all the future shock I was living with... on the edge of a nervous breakdown from dealing with an exponential growth curve" as an overworked programmer at a dotcom-era startup, Stross told AFP. Originally published as a series of short stories, "Accelerando" went on to win a Locus Award, one of science fiction writing's major honours. The novel follows three generations living through a "singularity" -- a theorised moment when technological progress accelerates to a pace beyond which almost anything becomes possible. Among Stross' inspirations was "Extropians", a pre-social-media mailing list popular among techies that hosted discussions among "some interesting and very odd people... very much into self-improving AI, the singularity, cryonics, space colonisation... they had a strong libertarian bent," he remembers. "Extropians" would also inspire figures like Ray Kurzweil, futurist and Google "AI visionary", who Stross believes "strip-mined" the conversations there for his books predicting the singularity. Chapters of "Accelerando" track anarchic inventor Manfred, who struggles with relatable 21st-century problems like battles over digital copyright and remembering who and where he is without his smart glasses. Another follows his daughter Amber, who uploads her mind into a computer to set off for another star system in the memory banks of a tiny starship. The book also features Amber's son Sirhan, who lives in a solar system largely transformed into computing hardware to support ever-more uploaded minds and AIs. Silicon Valley religion Such out-there scenarios are central to what AI researcher Timnit Gebru and intellectual historian Emile Torres have dubbed "TESCREAL" -- short for "Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism". In a 2024 paper, they described this "bundle" as one of the "ideologies driving the race to attempt to build Artificial General Intelligence" smarter than humans -- and traced its roots back to "the Anglo-American eugenics tradition of the 20th Century". "TESCREAL is what you get when a bunch of relatively bright, technologically-interested former Christians... reinvent religion," Stross said. "Christianity is a template for syncretistic religions" -- belief systems "which pick and match (ideas) from all over the place and glom them together," he added. "TESCREAL is doing exactly the same thing with a bunch of technology-related memes." Some statements and projects of today's tech titans echo this complex of beliefs, which foresees humans evolving beyond their present form, achieving immortality -- perhaps by merging with AI -- and multiplying throughout the universe. Elon Musk, for example, has spoken about making humans a "multiplanetary" species, was one of the original backers of OpenAI's stated mission to develop "artificial intelligence (that) benefits all of humanity" and founded Neuralink, a brain implant startup that aims to one day "expand how we experience the world". And OpenAI boss Sam Altman mused in a 2017 blog post about when humans would "merge" with machines, a process he believed "has already started" and "is probably going to happen sooner than most people think". 'Escapist fiction, big ideas' Stross said that with the likes of Musk close to power in the Trump administration and the threat of climate change hanging over the world, he is "fleeing screaming from writing about the near future". With "reality around us going to hell in a handbasket," he sees his aversion to the present mirrored in readers' appetite for "cosy escapist stuff". "I'm an entertainer... although I've always tried to do entertainment by combining regular escapist fiction with some big ideas," Stross said. Two decades later, his writing is circling back to TESCREAL as he imagines a future where its promises go unrealised. "What if there is not a singularity but everybody believes in it?" he mused. "What if we get half-baked versions of the tech?" His current projects include a story in which humanity's far-future descendants "have religions... based on TESCREAL, and there are holy wars over who will be allowed to set the rules in the AI upload heaven that nobody's actually built yet."
France 24
19 hours ago
- France 24
Reddit sues AI giant Anthropic over content use
The lawsuit in a California state court represents the latest front in the growing battle between content providers and AI companies over the use of data to train increasingly sophisticated language models that power the generative AI revolution. Anthropic, valued at $61.5 billion and heavily backed by Amazon, was founded in 2021 by former executives from OpenAI, the creator of ChatGPT. The company, known for its Claude chatbot and AI models, positions itself as focused on AI safety and responsible development. "This case is about the two faces of Anthropic: the public face that attempts to ingratiate itself into the consumer's consciousness with claims of righteousness and respect for boundaries and the law, and the private face that ignores any rules that interfere with its attempts to further line its pockets," the suit said. According to the complaint, Anthropic has been training its models on Reddit content since at least December 2021, with CEO Dario Amodei co-authoring research papers that specifically identified high-quality content for data training. The lawsuit alleges that despite Anthropic's public claims that it had blocked its bots from accessing Reddit, the company's automated systems continued to harvest Reddit's servers more than 100,000 times in subsequent months. Reddit is seeking monetary damages and a court injunction to force Anthropic to comply with its user agreement terms. The company has requested a jury trial. In an email to AFP, Anthropic said "We disagree with Reddit's claims and will defend ourselves vigorously." Reddit has entered into licensing agreements with other AI giants including Google and OpenAI, which allow those companies to use Reddit content under terms that protect user privacy and provide compensation to the platform. Those deals have helped lift Reddit's share price since it went public in 2024. Reddit shares closed up more than six percent on Wednesday following news of the lawsuit. Musicians, book authors, visual artists and news publications have sued the various AI companies that used their data without permission or payment. AI companies generally defend their practices by claiming fair use, arguing that training AI on large datasets fundamentally changes the original content and is necessary for innovation. Though most of these lawsuits are still in early stages, their outcomes could have a profound effect on the shape of the AI industry.
