Fakespot

What Fakespot measured

Against 110 samples spanning news, book prose, poetry and academic abstracts — machine text from GPT-4, ChatGPT, Llama, Mistral, Cohere and GPT-3 — Fakespot reached an AUC of 0.999. It scored machine text 0.995 on average and modern human text 0.112. AUC is the probability that it ranks a random AI passage above a random human one, so 1.0 is perfect separation and 0.5 is a coin flip.

A second corpus tests the failure that matters most. It is 26 passages published between 1532 and 1915 — Machiavelli, Austen, Melville, Kafka — where any high score is an error by construction, because the writing predates language models by a century or more. Fakespot scored those at 0.645. That is 0.533 above its score for modern human writing, the kind of gap that makes older or more formal prose look machine-written. Its ensemble weight is reduced accordingly.

How it decides

Fakespot contributes one row of the 23-row grid, alongside neural classifiers, GPT-2-based statistical tests and linguistic heuristics. No single engine sets the verdict. Weights come from measurement: each detector's share is proportional to how well it separated known-human from known-AI text, reduced if it showed bias against a particular kind of writing.

That matters because detectors fail in correlated ways. The neural classifiers were trained on overlapping data, so when they are wrong they tend to be wrong together, and averaging lets one confident engine drag the result. Agreement between independent engine families is the signal worth trusting.

How the 23 engines reach a verdict

SlopTotal does not ask one model for an opinion. It runs 23 independent detectors and shows you all 23 rows, because the interesting information is usually in the disagreement. Nine are neural classifiers fine-tuned to separate human from machine text, among them DeBERTa-v3-large, three RoBERTa variants and a 4.4M-parameter BERT-tiny that answers in milliseconds. Seven are statistical tests that need no detector training at all: they measure how predictable your text looks to GPT-2 Medium, using perplexity, token rank distribution, log-rank and cross-entropy ratios between two different language models. The remaining seven read the prose itself, counting contractions, em-dash habits, sentence-length variance, hedging and stock openings.

The final score is not an average. Averaging lets a single confident engine drag the result, and detectors fail in correlated ways: the neural classifiers were largely trained on similar data, so when they are wrong they tend to be wrong together. Scoring instead anchors on the classifiers that measured both accurate and unbiased in our own evaluation, then blends that against the full weighted set, and treats agreement between independent engine families as the real signal of confidence.

Every engine weight comes from measurement rather than intuition. Each detector's contribution is proportional to how well it actually separated known-human from known-AI text in testing, then reduced if it showed bias against any particular kind of writing. Detectors trained on the same benchmark we test against are damped, because a model graded on its own training distribution always flatters itself.

Tested against two corpora, not one

Accuracy claims mean little without saying what was tested. SlopTotal is evaluated against machine text from six different model families — GPT-4, ChatGPT, Llama, Mistral, Cohere and GPT-3 — across four kinds of writing: news reporting, book prose, poetry and academic abstracts. Measuring one domain is how detectors end up with impressive numbers that collapse in the wild; we found one of our own engines scoring near-perfectly on abstracts while being almost exactly backwards on everything else.

The second corpus is the one most detectors skip. It is prose published between 1532 and 1915 — Machiavelli, Austen, Melville, Kafka — where a high score cannot be anything but an error, because the text predates language models by a century or more. Any detector that quietly punishes older or more formal writing gets caught by that set immediately, and several of ours did. The results are published in the repository, including the failures.

Where detection stops working

Short text is unreliable and no detector honestly says otherwise. Under about 80 words the score swings hard on word choice alone; results settle at roughly 200 words and above. If you paste a tweet and get a confident answer, distrust the answer rather than the tweet.

Lightly edited AI is the hard case. Text that a person has rewritten sentence by sentence carries fewer machine fingerprints with every pass, and there is no threshold at which editing stops mattering. Source code is a genuine gap: these engines are trained on natural language, and in our own testing they neither falsely accuse human code nor reliably catch machine-written code, so we do not claim they can.

A score is evidence, not a verdict. Use the 23 rows to decide where to look and what to ask, then ask. No detector output should by itself decide a grade, a hire, or a publication — and any tool that encourages you to treat it that way is selling certainty it does not have.