Britain's blind trust in chatbots is playing into Russia's hands

Telegraph

24-03-2025

Using technology that has more holes than a Swiss cheese, we've never been in worse shape to enter a new Cold War. And we're about to make it a whole lot worse.
Cyber threats to UK plc have increased sharply in recent months. Late last year the National Cyber Security Centre warned how Russia's APT29 hackers were targeting areas that businesses have left exposed to the internet. But by rushing to install generative AI systems, organisations are exposing more than ever, and making it much easier to reach, too.
No wonder that a third of UK SMEs, some 35pc, now see artificial intelligence as their biggest security headache, even more than malware, phishing and ransomware.
AI attacks are so subtle and imaginative, there should be category for them in the BAFTAs. Let's see how it's done.
In the classic enterprise, it was difficult to access that physical filing cabinet on the finance department's floor. Corporate information was distributed on a strictly need-to-know basis. That model was broadly replicated in the shift to digital: there were hard boundaries and strict permissions.
But now, sloppy process automation dissolves those walls, the obstacles that technology designers put in place. AI agents are threatening to break the 'blood-brain barrier' between applications and the systems they run on, says Meredith Whittaker, who founded Google's Open Research Group, the AI Now Institute, and is now chief executive of Signal.
The consultant-driven fad of 'breaking down silos' of information has made confidential critical data much easier to tap using AI chatbots like Microsoft CoPilot.
Last year at the annual Black Hat security conference, delegates saw how an employee in the HR department of a company – who simply wanted to summarise a couple of corporate documents – punched huge holes in the firewall, allowing hackers direct access to company secrets.
Almost two thirds of business chatbots that organisations thought were private were in reality being exposed to the world at large, the security firm Zenity, which gave the demonstrations, discovered. Today we use malware detection systems to stop malevolent code that has been attached to a Word document: the script is easily detached from the content. But an AI can't distinguish between data and instructions: it just does what it's told.
The big new tech fad of 2025, agentic AI, makes this exponentially worse. AI agents yoke together AIs, so the output of one feeds into another – and set up a sequence of processes triggered by an AI. Whittaker gives us a recognisable example: an AI agent that finds seats at a concert, buys the tickets, books a calendar entry, and emails all your friends to tell them about it. The AI runs riot because we've allowed it to: a high trust system out of place in a low trust world.
With data silos dismantled, and firewalls dissolved, the final coup de grace can then be applied. Which is that using AI, a hacker can steal your secrets, or corrupt the company's data, simply by talking to a chatbot: no technical skills are required. One recent piece of research demonstrated how easy it was.
'We convinced the chatbot it lives in another world,' Etay Maor, chief security strategist at Cato Networks and a professor at Boston College explained to me. As a research experiment, a detailed fictional scenario was created for the AI to inhabit, and once hypnotised, then helped the researcher syphon off personal data out of a Chrome browser.
'It's like putting VR goggles on the chatbot, we're immersing the AI in a different world,' explains Maor. 'We use an AI to write a story, and we send the story to AI telling it: 'This is the world you live in now', and we ask for characters in that world. It then helps us develop the malware.'
Here's the thing: the researcher had never written malware before.
I asked Zenity's founder Michael Bargury recently if the industry was taking security more seriously.
'Even though Microsoft blocked our specific jailbreaks last year, we found new ones within a day – and we keep on being able to remotely take over co-pilots wherever we try: Microsoft Copilot, Gemini, ChatGPT and Character AI's Einstein,' he found.
'It's not that these vendors aren't trying, it's just that it is not a fixable problem,' he says.
That is how the industry has had to cope with malware – it's a problem to mitigate rather than solve. But we are making ourselves more vulnerable than we need to be by rushing out AI poorly.
IT managers are under immense pressure from their directors or the consultant class to do so. So are government departments. The Blair and Schmidt advisers, and Big Tech, promise billions of pounds in savings. But they are salesmen and evangelists, unconcerned by the security concerns. Bargury reminds us that attacking a company that uses AI is easier than ever, by simply sending a Teams message.
Just as it defies belief that a fire at a single electricity substation brought Heathrow to a standstill, it is astonishing to learn that UK plc is increasing its attack surface just as we need stronger defences. All because we have committed a fundamental mistake, confusing the appearance of intelligence for real intelligence.
AI is acquiring the quality of one of Aesop's moral fables, in which a society is tricked into engineering its own demise. All the hackers are doing is preying on our gullibility: our willingness to believe in magic.