The Biggest Signs That AI Wrote a Paper, According to a Professor

Gizmodo

6 hours ago

Mark Massaro has taught English Composition at Florida Southwestern State College for years, but his job became significantly more difficult in 2023. Not long after AI apps like ChatGPT became freely available, higher education throughout the U.S. was hit with a tsunami of automated cheating. Students have been using AI to write essays—and the trend appears to be worsening as time passes. Massaro said that, out of 25 students in his classes, it isn't unusual for as many as five to turn in papers that appear to be written by AI.
When ChatGPT first launched, and it became apparent that students were using it to cheat, Massaro says he used to run papers through multiple AI scanners a day. If a majority of them labeled the prose as generated by AI, he believed them. Now, however, the school doesn't allow them to upload students' work to such applications because it could be considered a breach of their privacy. It also doesn't help that the accuracy of AI detectors is unreliable.
Instead, Massaro says he must depend on his own wits to assess whether a paper was illegitimately conjured or not. To do this, he's put together a checklist of tell-tale signs that a paper is AI. Whenever a paper is handed in, he consults that list. He shared some of those red flags with Gizmodo, creating a clearer picture of the hellish conditions that AI has imposed on America's educators.
One hint that a paper may be generated is a plethora of em dashes. For whatever reason, apps like ChatGPT like to include plenty of these little. Now, Massaro has a test that he does. If he suspects a student of having generated their essay with AI, he will call them up to his desk and ask them to produce an em-dash on the computer. Frequently, the student doesn't even know how to do it, he said. 'I'll be like, your paper is full of em-dashes, just show me how you did it,' he said. If the student struggles, it's obvious what has happened.
Massaro says, contrary to the style stipulations of the traditional college essay, it's common for text that was copied and pasted from a chatbot to have no paragraph indentations. The output from ChatGPT and its ilk notably doesn't have any need for indents. If a student's paper is defined by big blocky chunks of text, there's a good chance that a robot wrote it.
If a student hands in a perfectly worded paper that doesn't have much to say, this could be another tell-tale sign of algorithmic generation, Massaro said. While there are plenty of students who may be good writers but may not have a lot of original thoughts, auto-generated writing has several tells that become obvious over time. Massaro describes the AI style as featuring 'uniform sentence and paragraph length,' in which every 'paragraph is about the same size, with a rhythmic, mechanical feel.' The writing may also have a 'hyper-formal tone,' he said, which is often characterized by an 'overly polished academic voice not typical of student work.'
Another red flag is a total absence of any participation in the in-class drafting process, Massaro said. As part of his composition classes, Massaro hosts peer reviews and conferences. A student may also claim that they met with the school's Writing Center to put the essay together, but, in reality, did not do this either. When students turn in these essays, they may also include 'unusual vocabulary for the student' that is characterized by 'sudden jumps in sophistication not seen in previous work or class discussions.'
Massaro said he occasionally hands out personal reflection papers. A student who actually writes the paper based on, say, their experiences working at the local 7-Eleven, will include all sorts of weird personal details that an AI would never think to invent, Massaro said. On the other hand, if the paper is auto-generated, it may wax on in a very abstract way about the importance of 'friendship' to humanity, he chuckled.
Shockingly, some students have been so sloppy with their cheating that they've actually left prompt inputs from their interactions with a chatbot in the final essay that they turned in, Massaro says.
It's well known that AI has a habit of 'hallucinating' information. In a student's paper, these AI-spawned references will manifest themselves in many different forms, including plainly incorrect or invented information and fake citations. This can create hours of additional work for Massaro, as he's forced to hunt down whether an obscure fact in an essay is real or not. This sometimes includes poring over real academic journals and books that were cited in the student's papers, but that may not feature the citations included in the student's bibliography (because they're fake).
Confronting a student about cheating can be awkward and difficult, Massaro said, although it is usually easy to tell what has happened right off the bat, he added. Frequently, a guilty student will admit that they used AI, or will simply not respond to Massaro's email (many of his students are remote). Innocent students often protest, and Massaro said he tends to believe them.
The biggest disappointment about the rise of AI-writing is that it is robbing students of their ability to cultivate their own intellectual and creative identity, he said. 'This is the time when they're supposed to be finding their voice,' he said. Instead, AI is doing the talking (and the thinking) for them.