AI detector benchmark

git clone https://github.com/pablocaeg/sloptotal.git
cd sloptotal/tests/eval/slopbench
SLOPTOTAL_URL=http://127.0.0.1:8000 python score.py results.jsonl
python analyze.py results.jsonl

What the engines look for

Classifiers trained on text from any LLM, statistical tests that measure how predictable each word is, and checks for the structure and phrasing AI models overuse.

Short texts give weak signals. Use at least 80 words, and 200 or more where you can.

How to read the result

The score runs from 0 to 100. From 55, fewer than 2 in 100 human texts score that high; from 80, fewer than 1 in 200.

Look at the engine table too. When most engines agree, the score is strong evidence. When they split, the text is often AI-assisted or heavily edited, and it deserves a closer human read.

Context

The numbers and the harness that produced them are both public. On SlopBench the calibrated ensemble reaches AUC 0.942, calls 67% of AI texts "Likely AI" and 1.7% of human texts. Per-engine figures are published too, including the uncomfortable ones: classifiers that score near-perfectly on the RAID benchmark drop to AUC 0.75 on current models, and the GPT-2 perplexity tests to about 0.64.

How the 23 engines reach a score

SlopTotal does not ask one model for an opinion. It runs 23 independent detectors and shows all 23 scores, because the useful information is often in the disagreement. Nine are neural classifiers trained to separate human from machine text, among them DeBERTa-v3-large, RoBERTa-large and a 4.4M-parameter BERT-tiny. Seven are statistical tests that need no detector training: they measure how predictable the text is to GPT-2 models, with perplexity, token ranks, log-rank and the cross-entropy between two models. The other seven read the prose itself: sentence-length variance, vocabulary, structure, hedging and the stock phrases AI models overuse.

The overall score is not an average. A logistic model fitted on SlopBench weighs each engine by what it adds once the others are known, so engines that fail in the same way do not count twice. Very short texts are pulled toward the middle, because a few words cannot carry much evidence.

The verdict bands are set on human writing: 45 is where the top 5% of human texts begin, 55 the top 2% and 80 the top 0.5%. So a score of 55 or more means fewer than 2 in 100 human texts score that high. Other languages have their own fit, and each language is marked supported, experimental or not yet reliable.

Measured on 3,252 texts

An accuracy figure means little without saying what it was measured on. SlopTotal is measured on SlopBench, an open benchmark of 1,626 human texts, every one written before ChatGPT existed, and 1,626 AI texts on the same topics and at the same lengths from 14 current models (GPT-6, DeepSeek, Qwen, Llama, Mistral, Gemma, GLM among them). It spans news, encyclopedias, science abstracts, Q&A answers, Reddit posts, reviews, student essays, non-native English and fiction, because a detector tested on one kind of writing ends up with numbers that collapse in the wild.

Two slices are there to catch unfairness: 87 passages of literature published between 1532 and 1915, where any high score is an error by construction, and essays by non-native English writers, the people detectors most often accuse wrongly. Every headline number is measured on kinds of writing the model was not tuned on, and the results are published in the repository, failures included.

Where detection stops working

Short text is unreliable, in every detector. Under about 80 words the score swings on word choice alone, and it settles from about 200 words. If a tweet gets a confident answer, distrust the answer.

Lightly edited AI text is the hard case. Each pass of human rewriting removes machine fingerprints, and there is no point at which editing stops mattering. Source code is out of scope: the engines are trained on natural language and do not reliably detect AI-written code, so SlopTotal does not claim to.

A score is evidence, not a verdict. Use the 23 scores to decide where to look and what to ask, then ask. No detector output should decide a grade, a hire or a publication on its own.