In August 2023, Vanderbilt University turned off Turnitin's AI detector and explained why in public. Over roughly 75,000 submissions, the tool had flagged about 750 papers as AI written. Vanderbilt could not tell which of those flags were correct, and neither could Turnitin, so they disabled it indefinitely.
Seven hundred and fifty students. That is what a one percent error rate looks like at one university in one year.
The number that should have ended the conversation
The most damaging finding came from Stanford. Liang, Yuksekgonul, Mao, Wu and Zou ran seven GPT detectors against 91 TOEFL essays written by non-native English speakers, all of them human written. The detectors flagged an average of 61.3% of them as AI generated. More than half the detectors flagged over 90% of the essays at least once. The same detectors classified essays by US eighth graders almost perfectly.
The mechanism is not mysterious. These tools mostly measure how statistically predictable text is. Someone writing carefully in their second language uses a narrower range of vocabulary and more conventional sentence structures, because that is what writing in a second language does. The signal the detector reads as "machine" is largely the signal of someone being careful in a language they learned later.
The consequences do not fall evenly. They fall on international students, which in most research universities means a substantial fraction of the graduate population.
What the vendors themselves say
This is where a fair reading matters, because the vendors are not silent and their responses are not worthless.
Turnitin reports around 98% accuracy with a false positive rate under 1% on documents where more than 20% of the text is AI generated. They have also published their own research arguing that for documents meeting their 300 word minimum, the difference in false positive rate between native and non-native English writers is small and not statistically significant.
Two things about that. It is vendor published research on the vendor's own product, which does not make it wrong but does mean it has not cleared the bar we would demand of any other empirical claim in this conversation. And Turnitin themselves acknowledge that below the word count threshold, the gap between those two groups of writers grows and exceeds their 1% target. Short assignments are not an edge case in undergraduate teaching. They are most of it.
Then there is the detector that got withdrawn by the company with the most to gain from it working. OpenAI shut down its own AI text classifier on 20 July 2023, citing low accuracy. Their published evaluation caught 26% of AI written text while incorrectly flagging 9% of human writing.
| Reported result | Source | |
|---|---|---|
| Seven detectors on TOEFL essays | 61.3% of human text flagged as AI | Stanford, Patterns, 2023 |
| OpenAI's own classifier | 26% detection, 9% false positives | OpenAI, withdrawn July 2023 |
| Turnitin at Vanderbilt | ~750 flags in ~75,000 papers | Vanderbilt, detector disabled 2023 |
| Turnitin, vendor reported | <1% false positives above 20% AI content | Turnitin |
The part that has no technical fix
Suppose the detectors improved. Suppose a vendor genuinely reached 99.5% accuracy across all writer populations.
You still cannot appeal the result.
A detector outputs a probability. It cannot show its reasoning, cannot point to which sentence triggered the score, and cannot be cross examined. The student sits in a meeting and says "I wrote it myself" against a number, and the number came from a system that will not explain itself. The burden of proof has quietly inverted: the accused is asked to demonstrate innocence, and no method exists for doing so.
Some students have started recording their screens while writing. Some submit version histories from Google Docs. Both are attempts to answer a question the tools were never built to answer, using evidence nobody agreed in advance to accept.
That is the actual failure. Not that detection is inaccurate, though it is. The failure is that a probabilistic accusation admits no defence, and we deployed it across education anyway.
Recording instead of detecting
The alternative is not a better classifier. It is not needing one.
If the record of how a document was written is captured while it is being written, there is nothing to infer afterwards. The editor already knows which text was typed, which was pasted, which came from an accepted AI suggestion, and who did each of those things. That information exists at the moment of writing and costs nothing to keep. It is only unavailable later because most tools throw it away.
TeXposit keeps it. Every project accumulates a signed authorship record, exportable as a provenance report showing which contributors wrote which parts and how much was AI assisted, signed with HMAC-SHA256 so that any modification invalidates the signature. Anyone can verify a report at app.texposit.com/provenance using the report ID or the PDF itself.
The difference is the direction of the argument. A detector produces a probability that you must argue against. A provenance report is a record you can produce, and the person evaluating it does not have to take your word for anything.
What this does not solve
Provenance is easy to oversell, so here are the limits plainly.
It cannot help a student accused today of writing something last term in a tool that kept no record. Provenance is captured during writing or not at all, and it cannot be reconstructed afterwards.
It is not a detector and cannot be used as one. It says nothing about a document written elsewhere. If an institution wanted to use it to catch people, it would not work, and we would consider that a misuse of it.
It covers text written in the editor. Paste in a paragraph drafted somewhere else and the record shows a paste, not its origin. That narrows the gap without closing it.
And it does not measure quality or intellectual contribution. A supervisor who shaped an entire argument may have typed almost none of it. Provenance describes text composition, which is a genuinely different thing from authorship in the sense the byline means. We have written about those limits at more length.
Where this leaves institutions
The honest position is that AI detectors do not currently support the decisions being made with them. Vanderbilt reached that conclusion in 2023 and published it. Several universities followed. The evidence since has not moved in the detectors' favour.
An integrity policy has to rest on something. The choice is between a classifier that disproportionately flags international students and cannot explain itself, and a record that was captured when the information still existed. Only one of those gives an honest author a way to be believed.
Every TeXposit project generates a provenance record automatically, on the free tier. You can see an example report, verify a document, or read the provenance guide.