Policymakers who think AI can help rescue flagging UK economy should take heed

The Guardian

15-06-2025

From helping consultants diagnose cancer, to aiding teachers in drawing up lesson plans – and flooding social media with derivative slop – generative artificial intelligence is being adopted across the economy at breakneck speed.
Yet a growing number of voices are starting to question how much of an asset the technology can be to the UK's sluggish economy. Not least because there is no escaping a persistent flaw: large language models (LLMs) remain prone to casually making things up.
It's a phenomenon known as 'hallucination'. In a recent blogpost, the barrister Tahir Khan cited three cases in which lawyers had used large language models to formulate legal filings or arguments – only to find they slipped in fictitious supreme court cases, and made up regulations, or nonexistent laws.
'Hallucinated legal texts often appear stylistically legitimate, formatted with citations, statutes, and judicial opinions, creating an illusion of credibility that can mislead even experienced legal professionals,' he warned.
In a recent episode of his podcast, the broadcaster Adam Buxton read out excerpts from a book he had bought online, purporting to be a compilation of quotes and anecdotes about his own life, many of which were superficially plausible – but completely fictitious.
The tech-sceptic journalist Ed Zitron argued in a recent blogpost that the tendency of ChatGPT (and every other chatbot) to 'assert something to be true, when it isn't', meant it was, 'a non-starter for most business customers, where (obviously) what you write has to be true'.
Academics at the University of Glasgow have said that because the models are not set up to solve problems, or to reason, but to predict the most plausible-sounding sentence based on the reams of data they have hoovered up, a better word for their factual hiccups is not 'hallucinations' but 'bullshit'.
In a paper from last year that glories in the title 'ChatGPT is bullshit', Michael Townsen Hicks and his colleagues say: 'Large language models simply aim to replicate human speech or writing. This means that their primary goal, insofar as they have one, is to produce human-like text. They do so by estimating the likelihood that a particular word will appear next, given the text that has come before.'
In other words, the 'hallucinations' are not glitches likely to be ironed out – but integral to the models. A recent paper in New Scientist suggested they are getting more frequent.
Even the cutting-edge forms of AI known as 'large reasoning models' suffer 'accuracy collapse' when faced with complex problems, according to a much-shared paper from Apple last week.
None of this is to subtract from the usefulness of LLMs for many analytical tasks – and neither are LLMs the full extent of generative AI; but it does make it risky to lean on chatbots as authorities – as those lawyers found.
If LLMs really are more bullshitters than reasoning machines, that has several profound implications.
First, it raises questions about the extent to which AI should really be replacing – rather than augmenting or assisting – human employees, who take ultimate responsibility for what they produce.
Last year's joint winner of the Nobel prize for economics Daron Acemoglu says that given its issues with accuracy, generative AI as currently conceived will only replace a narrowly defined set of roles, in the foreseeable future. 'It's going to impact a bunch of office jobs that are about data summary, visual matching, pattern recognition, etc. And those are essentially about 5% of the economy,' he said in October.
He calls for more research effort to be directed towards building AI tools that workers can use rather than bots aimed at replacing them altogether.
Sign up to Business Today
Get set for the working day – we'll point you to all the business news and analysis you need every morning
after newsletter promotion
If he is right, AI is unlikely to come to the rescue of countries – in particular the UK – whose productivity has never recovered from the global financial crisis and some of whose policymakers are ardently hoping the AI fairy will help workers do more with less.
Second, the patchier the benefits of AI, the lower the costs society should be ready to accept, and the more we should be trying to ensure they are borne, and where possible mitigated, by the originators of the models.
These include massive energy costs but also the obvious downsides for politics and democracy of flooding the public realm with invented content. As Sandra Wachter of the Oxford Internet Institute put this recently: 'Everybody's just throwing their empty cans into the forest. So it's going to be much harder to have a nice walk out there because it's just being polluted, and because those systems can pollute so much quicker than humans could.'
Third, governments should rightly be open to adopting new technologies, including AI – but with a clear understanding of what they can and can't do, alongside a healthy scepticism of some of their proponents' wilder (and riskier) claims.
To ministers' credit, last week's spending review talked as much about 'digitisation' as about AI as a way of improving the delivery of public services.
Ministers are well aware that long before swathes of civil servants are in line to be replaced by chatbots, the UK's put-upon citizenry would like to be able to hear from their doctor in some other format than a letter.
ChatGPT and its rivals have awesome power: they can synthesise vast amounts of information and present it in whatever style and format you choose, and they're great for unearthing the accumulated wisdom of the web.
But as anyone who has met a charming bullshitter in their life will tell you (and who hasn't?), it is a mistake to think they will solve all your problems – and wise to keep your wits about you.