If you think AI writing is easy to spot, a new study suggests you may be overestimating your ability to tell the difference. Researchers from Villanova University have found that readers struggled to tell AI-generated stories from human-written ones and frequently rated the AI versions higher.
The study, led by Dr. Deena Skolnick Weisberg and published in the journal Judgment and Decision Making (via The Guardian), asked more than 1,600 participants to rate one of six short stories, three written by a human and the rest generated by ChatGPT, on quality and engagement. Participants were told who wrote their story, though that label wasn’t always accurate.
AI stories scored higher, but the “human” label still won points
Readers who got an AI-generated story rated it as more absorbing and higher quality than readers who got a human-written one. At the same time, stories labeled as human-written scored better than those labeled AI, regardless of who actually wrote them.
Weisberg says the results show AI has become far more capable of producing readable, engaging fiction than most people assume. But that doesn’t mean human authors have become irrelevant, she added.
Most readers couldn’t spot the difference
In a follow-up test, participants were given one human-written and one AI-written story on the same theme and asked to identify which was which. They largely failed, guessing correctly only about as often as random chance would predict. The one group that performed noticeably better was people who described themselves as more familiar with AI systems, suggesting that spotting AI writing may be a learned skill rather than an instinct.
The Commonwealth Short Story Prize ran into this exact problem earlier this year, when readers online flagged three of its five winning entries as AI-generated, and an AI-detection tool backed them up. The prize’s own publisher ran a separate check, and it came back inconclusive. If professional judges and readers with a reputation on the line can’t settle the question, expecting casual readers to spot the difference is asking too much.