26-03-2025
Too Good To Be Human? AI's Surprising Bias Against Quality Writing
Robot hands and fingers point to laptop button advisor chatbot robotic artificial intelligence ... More concept
The first Turing Test may have been conducted at the ball in My Fair Lady. Professor Higgins has wagered with his friend, Pickering, that he can transform a flower girl into a lady through the science of language. He knows that people judge others by their manner of speech , and he'll use his skills as a professor of elocution to pull off the ruse.
The final test comes when a rival professor conducts his own appraisal of Eliza on the dance floor. His verdict: 'She is a fraud!' His logic is captured in the song, 'You Did It!'
Artificial Intelligence has faced a similar 'fool the inspector' challenge since Alan Turing first posed his famous test in a 1950 paper titled,'Computing Machinery and Intelligence.' Turing's very practical test proposes that a computer is intelligent if a person cannot distinguish between the computer and another person during an online chat.
Many experts believe we've passed Turing's test with generative AI models. The latest version of Claude (Claude Sonnet 3.7) was just released, and it writes remarkably well. I provided Claude with an outline for an article, including the key points to stress, along with an interview text, and it wrote a clear, interesting, coherent article. It was (almost) indistinguishable from something that I might have written.
I decided to try a reverse Turing test. My question was whether other AIs thought a given article was written by a person or by an AI. Gemini was certain the article I gave it was written by an AI. ChatGPT thought it plausible that the article was written by either a human or a machine (or a combination of both). Claude credited the human.
Interested, I put six of my Forbes columns through the test by asking, 'Was this written by an AI?' The articles are, of course, written by a human (me). In five out of six cases, Gemini thought they were AI-written. The model was transparent about its logic and about the 'tells' it uses to identify AI-written text. Several of these fit the category of what might be called good writing: structured argumentation; use of data and statistics; referencing sources; focus on practical solutions; and a concluding call to action. Ironically, these are the aspirations of many an essay writer! In some cases, unfortunately, Gemini also found that the writing 'lacks a distinct personality or voice…which is often characteristic of AI-generated text.' Oh, well.
Gemini's summary for the article How to Jump Start Learning At Work was:
It reminded me of the song from My Fair Lady: 'This writing is too good, it said. That clearly indicates that it is AI…'
ChatGPT seemed confused. It considered three authorship possibilities for each article: Purely AI-Generated; Human + AI Collaboration; and Purely Human-Written. In most cases, it favored a human collaborating with an AI, but it hedged by finding that all three options were plausible in five out of six cases.
Claude identified half of the articles as 'indeterminate' and half as human-generated (phew!). It did this based on the presence of personal voice, individual experience cited in the article, and the nuance of the argument (perceived by the first so-called third-generation LLM). Its summary, for the article cited above, was:
A few observations.
1. AI generally assumes thatwell-written articles are written by an AI. In other words, AI has a low regard for human-written text!
2. AI-written text is, indeed, getting very good. We should use it where we can to make writing better – but without delegating the thinking. AI will increase both the efficiency and clarity of business communications.
3. There will be an art to the collaboration between AIs and people as they work together to create good writing. The partnership is likely to involve iteration and the use of several tools. The best way of learning to do this will be by doing.
The ability of AI to write well creates another challenge. Content that sounds good but which is entirely derivative will become very easy to create (and to promote using AI). It will be easy to become even more overwhelmed by marginally useful information.
For centuries, we lived in a curated media world, where content was scarce and editors were in control. That world was disrupted in less than a generation by user-generated content like blogs, podcasts, and YouTube, which began to overwhelm our ability to process them. AI will move us into another era, one in which the volume of this user-generated content has increased so dramatically that it will inevitably alienate readers.
What will be the consequence? I think that people's media preferences will revert from open, public content to curated, paid content. The model originally spawned from scarcity is likely to be recovered as a consequence of abundance (or a scarcity of attention).
