AI pioneer Bengio launches $30M nonprofit to rethink safety

Axios

2 days ago

Machine learning pioneer Yoshua Bengio is launching a new nonprofit lab backed by roughly $30 million in funding to make AI systems act less like humans.
Why it matters: While the move bucks a trend toward AI that acts independently, Bengio and others argue the current approach risks creating systems that may pursue their own self-preservation at the expense of humanity.
"We've been getting inspiration from humans as the template for building intelligent machines, but that's crazy, right?" Bengio said in an interview.
"If we continue on this path, that means we're going to be creating entities — like us — that don't want to die, and that may be smarter than us and that we're not sure if they're going to behave according to our norms and our instructions," he said.
Driving the news: Bengio, a Montreal-based researcher who has long warned about the risks of a technology he helped develop, has raised about $30 million for the nonprofit, dubbed LawZero.
LawZero currently has about 15 staffers, Bengio said, "but the goal is to hire many more."
Bengio is among those who have called for tougher regulation of AI development and even the breakup of big tech companies.
Earlier this year he gave a TED Talk urging greater caution and collaboration.
The big picture: There's a growing sense of worry among critics — and even AI practitioners — that safety is taking a back seat as companies and countries race to be first with AI that can best humans in a wide variety of tasks, so-called artificial general intelligence (AGI).
Bengio said there is also a high risk in concentrating control of advanced AI in a handful of companies.
"You don't want AGI or superintelligence to be in the hands of one person or one company only deciding what to do, or even one government," Bengio said. "So you need very strong checks and balances."
Between the lines: Bengio says a large part of the problem is how current systems are trained. During initial training, the systems are taught to mimic humans and then they're honed by seeing which responses people find most appealing.
"Both of these give rise to uncontrolled agency," Bengio said.
Some early glimmers of this are already appearing, such as Anthropic's latest model which, in a test scenario, sought to blackmail its engineers to avoid shutdown.
By contrast, Bengio says he wants to create AI systems that have intellectual distance from humans and act as more of a detached scientist than a personal companion or human agent.
"The training principle is completely different, but it can exploit a lot of the recent advances that have happened in machine learning," he said.
Yes, but: Bengio told Axios that the $30 million should be enough to fund the basic research effort for about 18 months.