AI security risks are in the spotlight—but hackers say models are still alarmingly easy to attack
One of the biggest AI vibe shifts of 2025 so far is the sudden, massive pivot from AI 'safety' to AI 'security.'
Since the release of ChatGPT in November 2022, AI safety advocates, who typically focus on broad, long-term and often theoretical risks, have held the spotlight. There have been daily headlines about concerns that humans could lose control of AI systems that seek to harm humanity, or that rogue nations could use AI to develop genetically modified pandemics that then cause human extinction. There was the May 2023 open letter that called on all AI labs to 'immediately pause for at least 6 months the training of AI systems more powerful than GPT-4'—signed by 30,000, including Elon Musk. The Biden Administration spun out the AI Safety Institute as part of the small NIST agency (the National Institute of Standards and Technology), while the U.K. launched its own AI Safety Institute and held the first of three high-profile AI Safety Summits.
Oh, how times have changed: The head of the U.S. AI Safety Institute, Elizabeth Kelly, has departed, a move seen by many as a sign that the Trump administration was shifting course on AI policy. The third AI Safety Summit held in Paris earlier this month was renamed the AI Action Summit. There, the French government announced a national institute to 'assess and secure AI,' while U.S. Vice President JD Vance focused squarely on AI and national security, saying 'we will safeguard American AI and chip technologies from theft and misuse.'
Focusing on keeping AI models secure from those seeking to break in may seem more immediate and actionable than tackling the potential for all-powerful AI that could conceivably go off the rails. However, the world's best ethical hackers, or those who test systems in order to find and fix weaknesses before malicious hackers can exploit them, say AI security—like traditional cybersecurity—is far from easy.
AI security risks are no joke: A user could trick an LLM into generating detailed instructions for conducting cyberattacks or harmful activities. An AI model could be manipulated to reveal sensitive or private data in its training set. Meanwhile, self-driving cars could be subtly modified; deepfake videos could spread misinformation; and chatbots could impersonate real people as part of scams.
More than two years since OpenAI's ChatGPT burst onto the scene, hackers from the Def Con security conference, the largest annual gathering for ethical hackers, have warned that it is still far too easy to break into AI systems and tools. In a recent report called the Hackers' Almanack published in partnership with the University of Chicago, they said that AI vulnerabilities would continue to pose serious risks without a fundamental overhaul of current security practices.
At the moment, most companies focus on 'red teaming' their AI models. Red teaming means stress-testing an AI model by simulating attacks, probing for vulnerabilities, and identifying weaknesses. The goal is to uncover security issues like the potential for jailbreaks, misinformation and hallucinations, privacy leaks, and 'prompt injection'—that is, when malicious users trick the model into disobeying its own rules.
But in the Hackers' Almanack, Sven Cattell, founder of Def Con's AI Village and AI security startup nbdh.ai, said red teaming is 'B.S.' The problem, he wrote, is that the processes created to monitor the flaws and vulnerabilities of AI models are themselves flawed. With a technology as powerful as LLMs there will always be 'unknown unknowns' that stress-testing and evaluations miss, Cattell said.
Even the largest companies can't imagine and protect against every possible use and restriction that could ever be projected onto generative AI, he explained. 'For a small team at Microsoft, Stanford, NIST or the EU, there will always be a use or edge case that they didn't think of,' he wrote.
The only way for AI security to succeed is for security organizations to cooperate and collaborate, he emphasized, including creating versions of time-tested cybersecurity programs that let companies and developers disclose, share, and fix AI 'bugs,' or vulnerabilities. As Fortune reported after the Def Con conference last August, there is currently no way to report vulnerabilities related to the unexpected behavior of an AI model, and no public database of LLM vulnerabilities, as there has been for other types of software for decades.
'If we want to have a model that we can confidently say 'does not output toxic content' or 'helps with programming tasks in Javascript, but also does not help produce malicious payloads for bad actors' we need to work together,' Cattell wrote.
And with that, here's more AI news.
Sharon Goldmansharon.goldman@fortune.com@sharongoldman
This story was originally featured on Fortune.com
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
CNBC
21 minutes ago
- CNBC
Google's Pixel 10 launch wasn't about the phones but the strategic AI play
While Google made a big splash with its Pixel 10 series of smartphones, it was the software features that were strategically important for the tech giant's bid to compete with players like OpenAI and Perplexity in consumer AI. As it introduced its latest devices on Wednesday, Alphabet-owned Google showed off a slew of artificial intelligence features that are powered by the firm's Gemini AI models. "Magic Cue," for example, can scour various apps for information and deliver it to users when required. "Camera Coach" can give users tips on how to adjust framing and other aspects of a picture for the perfect shot. Live translation for phone calls is also available. All of this gives a glimpse into the so-called "agentic AI" future that tech giants are hoping to reach, where super-smart AI assistants can carry out complex tasks. It is a pivotal time for Google to come up with answers, as fears mount that users and revenue from its core search product could be eroded as more people turn to rivals like Perplexity and OpenAI's ChatGPT. Before Google lies a unique opportunity — the company develops Android, the operating system that is installed across more than three billion devices globally, many of which are smartphones. "The company is leapfrogging rivals like OpenAI and DeepSeek by leveraging its access to billions of Android users, enabling a more effective distribution, integration, and a wider range of use cases for Gemini at scale," Neil Shah, partner at Counterpoint Research, told CNBC. Ben Wood, chief analyst at CCS Insight, said the smartphone is the "most pervasive consumer device on the planet" and that Google now has an "opportunity to get people hooked on Gemini." Google doesn't need to sell a high volume of Pixel phones to find AI success with consumers. In fact, Pixel had just a 0.3% share of the global smartphone market in the first half of the year, compared to 23% for Samsung and 11.8% for Apple, according to the International Data Corporation. But Google's aim with its smartphones is to show off the best that Android has to offer in terms of software and AI. At that point, Android licensers, which include the likes of Samsung and Xiaomi, may adopt some of those features on their new handsets. This cycle would in turn spread Google's Gemini and AI tools to more users. "This massive user base creates a "flywheel effect" of adoption, usage, and feedback, further solidifying Gemini's position as a master agent on the most widely used device on the planet—the smartphone," Shah said. The timing is also advantageous because of struggles at rival Apple. The Cupertino giant's lack of AI strategy has concerned investors, with the iPhone showing very few features compared to Google's offerings. "Google has their tails up because Apple has dropped the ball. When Apple gets AI right it will be a fantastic experience. But right now, Google and all Android licensees have a window of opportunity," Wood said. Yet while there is now a land grab for users between major AI players, questions still linger over how Google will eventually monetize its AI services.
Yahoo
an hour ago
- Yahoo
Mark Zuckerberg freezes AI hiring amid bubble fears
Mark Zuckerberg has blocked recruitment of artificial intelligence staff at Meta, slamming the brakes on a multibillion-dollar hiring spree amid fears of an AI bubble. The tech giant has frozen hiring across its 'superintelligence labs', with only rare exceptions that must be approved by AI chief Alexandr Wang. It marks a sharp reversal for the company, which recently embarked on an unprecedented hiring spree that saw it offer pay packets reportedly worth up to $1bn (£740m) to senior researchers at rival businesses. Meta's pause in recruitment comes after technology shares have tumbled this week, fuelled by concerns that heavy investments in AI are not paying off. Companies such as Nvidia, Arm and Palantir have all fallen. The freeze went into effect last week, before the market sell-off in recent days. Stock market volatility was largely prompted by a report from the Massachusetts Institute of Technology, which claimed that 95pc of companies were getting 'zero return' on their AI investments. A Meta spokesman sought to downplay the freeze, saying: 'All that's happening here is some basic organisational planning: creating a solid structure for our new superintelligence efforts after bringing people on board and undertaking yearly budgeting and planning exercises.' It comes after the company has been offering top researchers at rival companies, including OpenAI and Google, enormous pay deals to join Meta Superintelligence Labs as Mr Zuckerberg seeks to dominate the field. It has also promised unprecedented investments in AI data centres. The company's billionaire chief executive has become personally involved in developing cutting-edge AI after the disappointing release of its latest systems, personally messaging top researchers at Silicon Valley AI companies. However, the division has been disrupted by repeated strategy overhauls, which led to the delayed release of its latest 'Behemoth' AI model. Talent hunt Mr Zuckerberg has said he wants to develop a 'personal superintelligence' that acts as a permanent superhuman assistant and lives in smart glasses. 'We believe in putting this power in people's hands to direct it towards what they value in their own lives,' he wrote last month. 'This is distinct from others in the industry who believe superintelligence should be directed centrally towards automating all valuable work, and then humanity will live on a dole of its output.' Mr Zuckerberg recently told investors that he wanted 'small, talent-dense teams' to be driving its AI work, rather than large groups of researchers. Despite this, the company has said that the cost of paying staff will significantly increase in the coming years. Analysts at Morgan Stanley warned this week that the pay surge may 'dilute shareholder value without any clear innovation gains'. Concerns about AI progress have been amplified by the modest response to GPT-5, the much-anticipated new version of ChatGPT. Sam Altman, OpenAI's chief executive, has compared hype around AI to the dotcom bubble at the turn of the century. Broaden your horizons with award-winning British journalism. Try The Telegraph free for 1 month with unlimited access to our award-winning website, exclusive app, money-saving offers and more.
Engadget
an hour ago
- Engadget
Alaska Air will offer Starlink in-flight internet starting next year
In-flight internet is crappy, but more and more airlines think that Starlink is the solution. The latest company to sign with the SpaceX affiliate is Alaska Air Group, which announced that it will start offering Starlink Wi-Fi next year and expand the service to its entire fleet by 2027. "With Starlink already live on [Alaska Air Group subsidiary] Hawaiian Airlines, we're proud that we'll offer... gate-to-gate connectivity on nearly every aircraft across both airlines," CEO Ben Minicucci said in a statement. The company noted in a separate announcement that it will offer the perk for free to members of its new loyalty program called Atmos Rewards. T-Mobile, a partner with Alaska, will also offer a "seamless, ad-free Wi-Fi log-on" to the in-flight Starlink service, with more details to be announced later this year. Alaska Air touted the benefits of "ultra-fast speeds... up to 7x faster than the geostationary satellite-based Wi-Fi systems that most airlines use today." Other airlines may jump on board soon, too. British Airways is also on the verge of announcing a Starlink deal, Bloomberg reported, and SpaceX has also reportedly been in conversation with Dubai's Emirates. Both of those are flagship carriers in their respective nations, so winning the business would be a large coup for Starlink against legacy operators like Viasat and Echostar. Switching to Starlink isn't necessarily cheap, though. It reportedly costs around $300,000 to equip a 737 and around half a million to install the system on a 787 Dreamliner. On top of that, airlines pay around $120 monthly per seat, plus another $120 for live TV, according to Bloomberg 's sources. (None of the airlines in negotiations have confirmed any details.) Despite those costs, carriers see reliable in-flight internet as a potential game-changer, as it would allow customers to work, communicate and stream videos or live TV. If the latter can be done reliably, it might even allow airlines to get rid of heavy and expensive on-demand entertainment systems. The main downside for potential customers is SpaceX's owner, Elon Musk. Some may view his fractured relationship with US president Donald Trump as a negative, while end-users may be turned off by his political affiliations — something that has seemingly affected sales of his Tesla EVs of late.
