Cheap hallucination checks catch most errors, and half of your good answers too

ProtForge research · 29 October 2026 · code and data below

Most teams with a RAG product reach for a cheap automatic check before they reach for a human review: does the answer use words the sources don't contain, does a small entailment model agree with it, do the numbers match. We measured how far those checks get on real model output, using 900 answers that people had already labelled sentence by sentence.

Setup

RAGTruth (Niu et al., ACL 2024, MIT licence) contains about 18,000 answers from six LLMs to retrieval-grounded prompts in three tasks: question answering, summarisation and data-to-text writing, with human-annotated hallucination spans. We call an answer hallucinated if it has at least one annotated span.

We tested three detectors that run on a CPU with no API calls:

• Lexical: share of content words and numbers in the answer that do not occur in the sources.
• Numeric: share of numbers and dates in the answer, after normalising formats (1,000 / 1000 / 1k), that do not occur in the sources.
• NLI: each answer sentence is checked by cross-encoder/nli-deberta-v3-small against its three closest source sentences and their concatenation; the answer scores as its least-supported sentence.

Plus a logistic-regression ensemble of the three. Thresholds and ensemble weights were fit on 300 training answers and frozen; results are on 900 test answers stratified by task and model (150 per model, 300 per task). The lexical and numeric detectors were also run on the full 2,700-answer test split, with the same result (lexical AUROC 0.785).

Results

AUROC: probability that a random hallucinated answer scores higher than a random clean one. 0.5 is chance (dashed line), 1.0 is perfect.

n = 900 test answers, 35% hallucinated. Precision, recall and F1 at the threshold tuned on the training sample. The numeric detector's best threshold flags everything, so its recall of 1.00 means nothing.

By task

AUROC within each task (300 answers each). Pooled AUROC is higher than any single task's because the detectors partly pick up the task itself, and tasks differ a lot in how often models hallucinate.

By generating model

150 answers per model. The confidence intervals here are wide; read the pattern, not the second decimal.

What each detector misses

What this means if you ship a RAG product

A cheap detector is a useful triage signal, not a quality gate. If you block or rewrite every flagged answer, you will degrade about half of your correct ones. If you only log the flags, they are a reasonable way to decide which answers a person should look at first.

The detectors also disagree in ways that depend on your task and your model, and the ranking flips between them. Neither the public leaderboard nor this page tells you which check works on your traffic. The only reliable way to know is to label a few hundred of your own answers and measure, which is a week of work, not a quarter.

Two things we did not test, deliberately: LLM-as-judge detectors (more accurate, but they cost per call and have their own failure modes; the code includes the interface) and fine-tuned detectors trained on RAGTruth itself, which would flatter the numbers on this benchmark.

Code and reproduction

groundcheck-0.1.0.zip (MIT): the detectors, a CLI that scores one answer against its sources sentence by sentence, the benchmark, per-answer test scores and all metrics.

pip install -e .
groundcheck score --source source.txt --response answer.txt --detector nli
groundcheck bench   # downloads RAGTruth, reproduces this page

The full run took 2.5 hours on a laptop CPU. Limitations: English only; RAGTruth's domains and 2023-era models; answer-level labels (we ignore span positions); a 900-answer test sample for the NLI rows.

Citation

Niu et al. (2024). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. ACL 2024. arXiv:2401.00396.