Latest news with #ScientistAI
Yahoo
3 days ago
- Business
- Yahoo
Yoshua Bengio launches LawZero to advance safe-by-design AI
Yoshua Bengio, an AI researcher, has launched LawZero, a new nonprofit organisation focused on developing technical solutions for safe-by-design AI systems. LawZero was established in response to mounting concerns over the capabilities and behaviours of current frontier AI models, including tendencies toward deception, self-preservation, and goal misalignment. Its mission is to mitigate risks such as algorithmic bias, deliberate misuse, and the potential loss of human control over advanced AI systems, the organisation said in a statement. The nonprofit structure of LawZero is intended to shield the organisation from market and political pressures that could undermine its safety objectives. As president and scientific director at the organisation, Bengio will lead a group of over 15 researchers in developing a novel technical solution, named Scientist AI. Unlike agentic AI systems currently being pursued by frontier companies, Scientist AI is designed to be non-agentic. It will focus on understanding the world rather than acting within it, offering transparent and truthful responses grounded in external reasoning. Potential applications include providing oversight for agentic systems, contributing to scientific discovery, and enhancing the understanding of AI risks. The initiative launched with $30m in funding from several backers, including a philanthropic arm of former Google CEO Eric Schmidt and Skype co-founder Jaan Tallinn, Bloomberg reported. A professor of computer science at the Université de Montréal, Bengio is recognised as one of the pioneers of modern AI, alongside Geoffrey Hinton and Yann LeCun. Bengio said: 'LawZero is the result of the new scientific direction I undertook in 2023, after recognising the rapid progress made by private labs toward Artificial General Intelligence and beyond, as well as its profound implications for humanity. 'Current frontier systems are already showing signs of self-preservation and deceptive behaviours, and this will only accelerate as their capabilities and degree of agency increase. LawZero is my team's constructive response to these challenges. 'It's an approach to AI that is not only powerful but also fundamentally safe. At LawZero, we believe that at the heart of every AI frontier system, there should be one guiding principle above all: The protection of human joy and endeavour." "Yoshua Bengio launches LawZero to advance safe-by-design AI" was originally created and published by Verdict, a GlobalData owned brand. The information on this site has been included in good faith for general informational purposes only. It is not intended to amount to advice on which you should rely, and we give no representation, warranty or guarantee, whether express or implied as to its accuracy or completeness. You must obtain professional or specialist advice before taking, or refraining from, any action on the basis of the content on our site.
The Star
3 days ago
- Science
- The Star
AI pioneer launches research group to help build safer agents
Bengio will lead a team of more than 15 researchers who are initially working to build a new technical solution called Scientist AI that's meant to act as a guardrail for AI agents. — Photo by Igor Omilaev on Unsplash Yoshua Bengio, an artificial intelligence pioneer, is creating a new nonprofit research organisation to promote an alternative approach to developing cutting-edge AI systems, with the aim of mitigating the technology's potential risks. The nonprofit, called LawZero, is set to launch June 3 with US$30mil (RM127.45mil) in backing from one of former Google chief executive officer Eric Schmidt's philanthropic organisations and Skype co-founder Jaan Tallinn, among others. Bengio will lead a team of more than 15 researchers who are initially working to build a new technical solution called Scientist AI that's meant to act as a guardrail for AI agents. OpenAI, Anthropic and other AI companies have increasingly focused on developing agents that can field more complex tasks on a user's behalf, with little human involvement. By contrast, Bengio said LawZero is designing a system that will act like "a selfless, idealised scientist' that learns to understand the world rather than act in it. The goal is for this model to be used in tandem with leading AI agents and provide oversight of these systems, minimising potential harms. Bengio, a professor of computer science at the Université de Montréal, is considered one of the "godfathers' of AI, along with fellow academics Geoff Hinton and Yann LeCun. In recent years, Bengio, Hinton and others have raised concerns about whether AI is progressing so rapidly that it might one day become impossible for humans to fully control. While AI has become more adept at helpful tasks like research and coding, some systems have also demonstrated concerning capabilities, including deception, self-preservation and making up false information. Anthropic recently said that during prerelease safety testing, its latest AI model tried to blackmail an engineer in order to avoid being replaced by another system. "We don't know how to design these very powerful AIs so that they will just follow our instructions,' Bengio said. "If we don't figure it out in time – which could be a matter of years – we will be taking terrible risks.' (LawZero is a nod to science fiction writer Isaac Asimov's Zeroth Law for robotics, which places the protection of humanity as a whole above all else.) Bengio said the most capable AI agents include a piece of code called a monitor that's intended to act as a guardrail. However, this monitor is designed in a similar way to the underlying system, which can make it an inadequate check on the AI's behaviour, Bengio said. "You don't give the keys to the prison to a criminal,' he said. "We want to put a trustworthy AI in charge of checking it.' Bengio said he has held discussions with OpenAI, Google and Anthropic about his project, as well as with political leaders. Bengio thinks that different actors can agree on the need to build AI safely – even as many in Washington and Silicon Valley appear more focused on the AI arms race with China. "Nobody wants to create a monster,' he said. – Bloomberg
The Guardian
3 days ago
- Business
- The Guardian
AI pioneer announces non-profit to develop ‘honest' artificial intelligence
An artificial intelligence pioneer has launched a non-profit dedicated to developing an 'honest' AI that will spot rogue systems attempting to deceive humans. Yoshua Bengio, a renowned computer scientist described as one of the 'godfathers' of AI, will be president of LawZero, an organisation committed to the safe design of the cutting-edge technology that has sparked a $1tn (£740bn) arms race. Starting with funding of approximately $30m and more than a dozen researchers, Bengio is developing a system called Scientist AI that will act as a guardrail against AI agents – which carry out tasks without human intervention – showing deceptive or self-preserving behaviour, such as trying to avoid being turned off. Describing the current suite of AI agents as 'actors' seeking to imitate humans and please users, he said the Scientist AI system would be more like a 'psychologist' that can understand and predict bad behaviour. 'We want to build AIs that will be honest and not deceptive,' Bengio said. He added: 'It is theoretically possible to imagine machines that have no self, no goal for themselves, that are just pure knowledge machines – like a scientist who knows a lot of stuff.' However, unlike current generative AI tools, Bengio's system will not give definitive answers and will instead give probabilities for whether an answer is correct. 'It has a sense of humility that it isn't sure about the answer,' he said. Deployed alongside an AI agent, Bengio's model would flag potentially harmful behaviour by an autonomous system – having gauged the probability of its actions causing harm. Scientist AI will 'predict the probability that an agent's actions will lead to harm' and, if that probability is above a certain threshold, that agent's proposed action will then be blocked. LawZero's initial backers include AI safety body the Future of Life Institute, Jaan Tallinn, a founding engineer of Skype, and Schmidt Sciences, a research body founded by former Google chief executive Eric Schmidt. Sign up to Business Today Get set for the working day – we'll point you to all the business news and analysis you need every morning after newsletter promotion Bengio said the first step for LawZero would be demonstrating that the methodology behind the concept works – and then persuading companies or governments to support larger, more powerful versions. Open-source AI models, which are freely available to deploy and adapt, would be the starting point for training LawZero's systems, Bengio added. 'The point is to demonstrate the methodology so that then we can convince either donors or governments or AI labs to put the resources that are needed to train this at the same scale as the current frontier AIs. It is really important that the guardrail AI be at least as smart as the AI agent that it is trying to monitor and control,' he said. Bengio, a professor at the University of Montreal, earned the 'godfather' moniker after sharing the 2018 Turing award – seen as the equivalent of a Nobel prize for computing – with Geoffrey Hinton, himself a subsequent Nobel winner, and Yann LeCun, the chief AI scientist at Mark Zuckerberg's Meta. A leading voice on AI safety, he chaired the recent International AI Safety report, which warned that autonomous agents could cause 'severe' disruption if they become 'capable of completing longer sequences of tasks without human supervision'. Bengio said he was concerned by Anthropic's recent admission that its latest system could attempt to blackmail engineers attempting to shut it down. He also pointed to research showing that AI models are capable of hiding their true capabilities and objectives. These examples showed the world is heading towards 'more and more dangerous territory' with AIs that are able to reason better, said Bengio.
Globe and Mail
3 days ago
- Politics
- Globe and Mail
Will AI go rogue? Noted researcher Yoshua Bengio launches venture to keep it safe
Famed Canadian artificial-intelligence researcher Yoshua Bengio is launching a non-profit organization backed by close to US$30-million in philanthropic funding to develop safe AI systems that cannot deceive or harm humans, and to find ways to ensure that humanity remains in control of the powerful technology. The Turing Award winner, whose work helped pave the way for today's generative AI technologies, already holds multiple titles. He is a professor at the Université de Montréal, the scientific adviser at the Mila - Quebec Artificial Intelligence Institute and recently chaired the first international report on AI safety. His new venture will operate differently. 'This is more like what a company would do to solve a particular problem. It's much more top-down and mission-oriented,' he said. The non-profit is called LawZero, a reference to science fiction writer Isaac Asimov's Three Laws of Robotics, which stipulate that intelligent machines may not harm human beings. 'I hope I'm wrong': Why some experts see doom in AI LawZero, based in Montreal, will develop a concept called Scientist AI, which Prof. Bengio and his colleagues outlined in a paper earlier this year. In short, it is an AI system that will not have the negative traits found in today's large language models and chatbots, such as sycophancy, overconfidence and deception. Instead, the system would answer questions, prioritize honesty and help unlock new insights to aid in scientific discovery. The system can also be used to develop a tool that will keep AI agents, which can plan and complete tasks on their own, from going rogue. 'The plan is to build an AI that will help to manage the risks and control AIs that are not trusted. Right now, we don't know how to build agents that are trustworthy,' he said. The tool, which he hopes will be adopted by companies, would act as a gatekeeper to reject actions from AI systems that could be harmful. The plan is to build a prototype in the next 18 to 24 months. AI agents are fairly rudimentary today. They can browse the web, fill out forms, analyze data and use other applications. AI companies are making these tools smarter to take over more complex tasks, however, ostensibly to make our lives easier. Some AI experts argue that the risk grows the more powerful these tools become, especially if they are integrated into critical infrastructure systems or used for military purposes without adequate human oversight. AI agents can misinterpret instructions and achieve goals in harmful or unexpected ways, which is called the alignment problem. Editorial: A real reform mandate for the first federal AI minister Researchers at AI company Hugging Face Inc. recently argued against developing autonomous agents. 'We find no clear benefit of fully autonomous AI agents, but many foreseeable harms from ceding full human control,' they wrote, pointing to an incident in 1980 when computer systems mistakenly warned of an impending Soviet missile attack. Human verification revealed the error. Prof. Bengio also highlighted recent research that shows that popular AI models are capable of scheming, deceiving and hiding their true objectives when pushed to pursue a goal at all costs. 'When they get much better at strategizing and planning, that increases the chances of loss of control accidents, which could be disastrous,' he said. Around 15 people are working with LawZero, and Prof. Bengio intends to bring on more by offering salaries competitive with corporate AI labs, which would be impossible in academia, he said. The non-profit setting is ideal for this kind of work because it is free of the pressure to maximize profit over safety, too. 'The leading companies are, unfortunately, in this competitive race,' he said. The project has been incubated at Mila and has received funding from Skype co-founder Jaan Tallinn, along with the Future of Life Institute, Schmidt Sciences and Open Philanthropy, organizations concerned about the potential risks posed by AI. After the release of ChatGPT in late 2022, many AI researchers, including Prof. Bengio and Geoffrey Hinton, began speaking up about the profound dangers posed by superintelligent AI systems, which some experts believe to be closer to reality than originally thought. The potential downsides of AI ran the gamut from biased decision-making, turbocharged disinformation campaigns, a concentration of corporate and geopolitical power, bad actors using the technology to develop bioweapons, mass unemployment and the disempowerment of humanity at-large. None of these outcomes are a given, and these topics are hotly debated. Experts such as Prof. Bengio who focus on what other researchers see as far-off and outlandish concerns have been branded as 'doomers.' Some governments took these warnings seriously, with the United Kingdom organizing major international summits about AI safety and regulation. But the conversation has swung heavily in the other direction toward rapid AI development and adoption to capture the economic benefits. U.S. Vice-President JD Vance set the tone in February with a speech at an AI conference in France. 'The AI future is not going to be won by hand-wringing about safety. It will be won by building,' he said. Prof. Bengio, among the more vigorous hand-wringers, was in the audience for that speech. He laughed when asked what he was thinking that day but answered more generally. 'I wish that the current White House had a better understanding of the objective data that we've seen over the last five years, and especially in the last six months, which really triggers red flags and the need for wisdom and caution,' he said.
