Computer scientist to develop 'honest' AI that will spot rogue systems and flag 'harmful behaviour'
Yoshua Bengio, a renowned computer scientist described as one of the 'godfathers' of AI, will be president of LawZero, an organisation committed to the safe design of the cutting-edge technology that has sparked a $1tn (€877bn) arms race.
Starting with funding of about $30m (€26.3m) and more than a dozen researchers, Bengio is developing a system called Scientist AI that will act as a guardrail against AI agents — which carry out tasks without human intervention — showing deceptive or self-preserving behaviour, such as trying to avoid being turned off.
Describing the current suite of AI agents as 'actors' seeking to imitate humans and please users, he said the Scientist AI system would be more like a 'psychologist' that can understand and predict bad behaviour.
'We want to build AIs that will be honest and not deceptive,' Bengio said.
'It is theoretically possible to imagine machines that have no self, no goal for themselves, that are just pure knowledge machines — like a scientist who knows a lot of stuff.'
However, unlike current generative AI tools, Bengio's system will not give definitive answers and will instead give probabilities for whether an answer is correct.
'It has a sense of humility that it isn't sure about the answer,' he said.
Deployed alongside an AI agent, Bengio's model would flag potentially harmful behaviour by an autonomous system — having gauged the probability of its actions causing harm.
Scientist AI will 'predict the probability that an agent's actions will lead to harm' and, if that probability is above a certain threshold, that agent's proposed action will then be blocked.
LawZero's initial backers include AI safety body the Future of Life Institute, Jaan Tallinn, a founding engineer of Skype, and Schmidt Sciences, a research body founded by former Google chief executive Eric Schmidt.
Bengio said the first step for LawZero would be demonstrating the methodology behind the concept works — and then persuading companies or governments to support larger, more powerful versions. Open-source AI models, which are freely available to deploy and adapt, would be the starting point for training LawZero's systems, Bengio added.
'The point is to demonstrate the methodology so that then we can convince either donors or governments or AI labs to put the resources that are needed to train this at the same scale as the current frontier AIs. It is really important that the guardrail AI be at least as smart as the AI agent that it is trying to monitor and control,' he said.
Bengio, a professor at the University of Montreal, earned the 'godfather' moniker after sharing the 2018 Turing award — seen as the equivalent of a Nobel prize for computing — with Geoffrey Hinton, himself a subsequent Nobel winner, and Yann LeCun, the chief AI scientist at Mark Zuckerberg's Meta.
A leading voice on AI safety, he chaired the recent International AI Safety report, which warned autonomous agents could cause 'severe' disruption if they become 'capable of completing longer sequences of tasks without human supervision'.
The Guardian
Read More
The real cost of slightly funnier AI is the health of a poor black community
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Irish Times
a day ago
- Irish Times
Dementia and AI: ‘I've never seen something quite like this'
When Frank Poulsen first started talking to artificial intelligence companion Sunny earlier this year, she was 'kind of formal,' he said. But over time, their conversations have become more familiar and engaging. 'It's really uncanny,' said the 72-year-old. 'It really does feel like you're having a conversation with someone who's interested.' One advantage of Sunny is that she doesn't mind when Poulsen, who was diagnosed with mild cognitive impairment in 2019, repeats himself. 'He can tell a story and forget he told it and tell it again,' said Poulsen's wife and caregiver, Cheryl Poulsen. 'Sunny has no judgment.' Sunny, which was developed by AI start-up NewDays, is part of a rapidly expanding body of AI products designed to help people with dementia . READ MORE 'I've never seen something quite like this in terms of a technology moving as quickly as it is, with as much money poured behind it,' said Luke Stoeckel, a programme director and project scientist at the National Institute on Aging in Maryland, in the United States, who works on AI applications for dementia care. Stoeckel mentioned AI-based dementia diagnostics as another hot area. Experts say AI companions like Sunny have the potential to help keep the brain active and alleviate loneliness in dementia patients, and to reduce some of the burden on caregivers. But there can be risks when it comes to deploying this technology in such a vulnerable population. Babak Parviz, the co-founder and CEO of NewDays, said he was inspired to develop Sunny in part by his own father's diagnosis with Alzheimer's disease. He knew there was no cure, but he was encouraged by research showing that cognitive training exercises and guided conversations with a clinician can, in some cases, improve dementia symptoms. NewDays' program is intended to be used by people with mild cognitive impairment or mild dementia. Sometimes, Sunny prompts the person to reminisce about old memories. Other times, it leads them through more explicit cognitive training exercises, such as asking them to remember and use specific words in a sentence. When Frank Poulsen opens up his computer and converses with Sunny, it sounds like 'he was just having a chat, like he was talking to a friend,' Cheryl Poulsen said. NewDays' website claims that its program can 'delay decline and improve quality of life' and that it is 'backed by a large body of evidence from more than 500 clinical trials and papers.' That includes the I-CONECT study, which found that, over the course of six months, 'cognitively stimulating semistructured conversations with trained interviewers' conducted virtually improved cognition in older adults with mild cognitive impairment. [ 'Dementia is just one part of who they are. My husband's underlying personality hasn't really changed' Opens in new window ] But in those studies, the conversations and brain training exercises were led by humans, said Julian De Freitas, director of the Ethical Intelligence Lab at Harvard Business School. Talking to an AI companion could produce different results, he added. NewDays has not run any studies yet about whether using its program confers the same benefits. Dr Parviz said Sunny's conversations are similar to a human's and that his team is 'pretty confident' it will see results comparable to what the existing clinical trials have demonstrated. But he did not provide data to back that up. There is more compelling evidence that an AI companion can help people feel less lonely – a serious concern for older adults, whether or not they have cognitive impairment. Dr De Freitas's own research has demonstrated that, at least in healthy adults, talking to an empathetic chatbot is roughly as good at reducing loneliness as having a conversation with a human stranger. Anecdotally, Dr Joe Verghese, the chair of the neurology department at Stony Brook University Renaissance School of Medicine, said the caregivers of some of his dementia patients reported that talking to an AI companion lifted their loved ones' moods. 'I don't think these actually improve cognition just by those interactions,' Dr Verghese said. 'But certainly improving quality of life and improving mood in patients with cognitive impairment is an important outcome in itself.' [ I am a university lecturer witnessing how AI is shrinking our ability to think Opens in new window ] AI companions could also be a boon for caregivers and families because they can serve as an extra set of eyes and ears, particularly for people living in specialist facilities for people with memory-related conditions such as dementia. We have to remember that this is a very vulnerable population One AI start-up, CloudMind, has been pilot testing its software, called BrightPath. The companion, via an app on a smartphone or tablet, chats with dementia patients about their favourite memories and hobbies – in one instance, a patient and the companion even sang together. In addition to facilitating conversation, BrightPath produces a daily summary of what the patient talked about and an assessment of the patient's mood. For the family, that can provide 'a peek of their loved one's life, even if they're apart,' said Monica Tsai, CloudMind's chief executive. The AI companion might also help nursing staff identify problems faster. If it hears certain key prompts, such as 'I'm thirsty' or 'I fell,' BrightPath is programmed to instruct the patient to ring their call button. While experts say there is real potential for AI to benefit adults with cognitive impairment, they also have some serious concerns. One is privacy. 'The more human-like the chatbot is, the more you're disclosing various information about yourself,' Dr De Freitas said. 'If this data is housed at a company, it could monetise that. It could also accidentally leak that information if it's hacked.' There is also a risk of further isolation if families or care-home staff use the technology as a replacement for human interaction, rather than as an add-on. 'That would be tragic,' Dr Verghese said. [ Exercise and socialising can help people at risk of dementia, large-scale study finds Opens in new window ] In addition, AI chatbots are known to make things up and take conversations down dark and disturbing paths, which could be especially confusing and distressing for a person with dementia. Dr Parviz and Dr Tsai both acknowledged such hallucinations as a potential issue and said they had programmed 'guard rails' into their AI software to prevent them from happening. But neither specified how that might work. 'We have to remember that this is a very vulnerable population,' Dr De Freitas said, adding: 'We need to think about having additional protections.' - This article originally appeared in the New York Times
Irish Times
2 days ago
- Irish Times
The Irish Times view on Ireland's AI future: the clock is ticking
Given the scale of the claims being made for artificial intelligence, it is striking how slow the Oireachtas has been to give the subject sustained attention. AI is routinely described as having the potential to transform society, disrupt the global political order and even alter what it means to be human. Yet it was only this year that the Joint Committee on Artificial Intelligence began its work, holding its first public session in June. This week its chair, Fianna Fáil TD Malcolm Byrne, set out his thinking on what must happen next. Byrne believes Ireland could position itself as 'the AI island' but warns that the opportunity will be lost without swift and decisive action. He says he would be disappointed if both a new AI office and an AI observatory are not operational by next year. The office will be tasked with implementing the EU's AI Act, while the observatory will assess the technology's effects, from employment disruption to identifying future skills needs. In Byrne's view, those who embrace AI will displace those who do not, whether they are doctors, architects or lawyers. He welcomes the Government's plan for an AI summit during Ireland's EU presidency in 2026 and argues for clear ethical frameworks in education, where students are already using AI tools. He also points to the technology's current uses in Ireland, from automating recycling processes to analysing tax data. Such calls for urgency are sensible and overdue. But preparing the State for the changes ahead will require far more than offices, observatories and summits. The debate is complicated by the sweeping and often speculative claims surrounding AI, from the elimination of entire job categories to science-fiction visions of superintelligent machines destroying the human race. This discourse is unfolding against the backdrop of a global investment surge, with leading AI companies commanding extraordinary valuations and data centres proliferating at breakneck speed. History suggests such frenzies rarely end without turbulence. READ MORE Byrne's proposals are shaped by the EU's AI Act, which will impose a detailed regulatory roadmap over the coming years. That approach contrasts sharply with the let it rip stance favoured by Donald Trump's administration in the US, and with the UK's less prescriptive, more innovation-oriented strategy. Which philosophy will prove more effective is an open question but the divergence will have real consequences for competitiveness. If Ireland is serious about becoming 'the AI island', it must reckon with the reality that others are moving faster and with considerable resources. Ambition is necessary, but so too is a clear-eyed appraisal of the scale of the challenge and the pace of change. Without that, the island will be an observer, not a leader, in the age of AI.
Irish Times
2 days ago
- Irish Times
Ireland could be ‘AI Island' but must act promptly to set up artificial intelligence office and observatory
Ireland could position itself as 'the AI island' but risks getting left behind unless decisions are taken promptly on important related infrastructure, the chairman of the Oireachtas committee examining the technology has said. Fianna Fáil TD Malcolm Byrne said he would be 'disappointed' if an AI office and an AI observatory are not 'up and running' next year, because if we're not showing that level of ambition now, this technology is going to bypass us'. The Government is to set up the artificial intelligence office to oversee the implementation of the European Union's AI Act, which sets rules governing the development, marketing and use of AI in the bloc. The observatory would gather data, carry out research and make recommendations to prepare the State for changes coming because of the technology. 'We know there's going to be huge disruption in employment, so we need to know what sectors is it impacting, where are jobs of the future going to be. And then what upskilling and reskilling do we need to do to make sure everybody can benefit,' said Mr Byrne. READ MORE 'It won't be a case of robots getting rid of all workers. What will happen is workers who use AI will replace those who don't and that will apply in virtually every sector of society.' The Wicklow-Wexford TD welcomed plans by Minister of State for Artificial Intelligence Niamh Smyth to host an AI summit next year during Ireland's presidency of the European Union. In response to a parliamentary question, she said AI and the role it can play 'in our societies and economies will be a central theme' of the presidency, which begins next July. Mr Byrne said Ireland 'wants to be at the forefront of the debate, but also on the policy issues around ensuring that AI is used safely, in the interests of citizens'. Crash: Brian Cowen and the unravelling of Ireland: Part one Listen | 52:35 'We've also got to start those debates around how we deal with AI within our education and training systems, but also much bigger public debates,' he added. Citing education, he said Irish students are using AI already, 'and it's important there's an ethical framework set up about where and when they should use it'. He noted that Open AI has done a deal with the Estonian government and every 16- and 17-year-old has access to an educational version of its ChatGPT AI chatbot. He said it means the material being used 'is the trusted data that is covered as part of the curriculum'. Mr Byrne cited examples of where AI is already having an impact in the State. Greyhound Household Recycling, one of the biggest recycling companies in the country, uses AI to sort and separate paper, plastic, cardboard, metal and other items with about 98 per cent accuracy. Revenue is among the most advanced of any State agency or Government department in the use of AI platforms, employing them to analyse the vast amounts of data they have on tax collection.
