Who checks the checker? We measured it, and published what we found.
Answer Check reads the sources an AI answer cites and shows which numbers and quotes are really in them. Before collecting a single answer we wrote down how we would test it and what would count as passing, and sealed that plan so it could not be changed afterwards. People then judged each figure against its source without seeing what Answer Check had said. Here is everything, including the measure it failed.
The results
When Answer Check says a figure is found in the source, how often is that wrong?
4 of 119 items people could judge. 95% interval 1.3% to 8.3%.
When it says a figure is not in the source, how often is the figure really there?
74 of 89. 95% interval 74.0% to 89.5%.
When we change a correct figure on purpose, how often does it notice?
98 of 100 altered figures caught.
Checked again on the same stored sources, does it give the identical result?
120 of 120 answers gave the same receipt fingerprint.
What this means if you use it
- A ✓ found in source is reliable. When Answer Check said a figure was in its source, people agreed about 97% of the time. This is the measure that matters most, because the harm the tool exists to prevent is telling you something is backed when it isn’t. Read it with the interval beside it: with 119 items, the true rate could be as high as 8.3%. The plan judges on the measured figure.
- A ✕ not in source is too often wrong. In 74 of 89 cases the figure was in the source, written differently: in words, in a table, or in another phrasing Answer Check doesn’t match. Until that changes, read a ✕ as “couldn’t confirm”, and look at the nearest passages it shows you before concluding the AI made something up.
- It notices a changed figure. We altered 100 correct figures (scaled up or down, or one digit changed) and checked them against the same sources. It flagged 98.
- It is repeatable. The check is code, not a model. Run again on the same stored sources, every one of the 120 answers gave an identical result.
How it was done
- The answers. 40 fixed questions on health, medicines, money, science and technology, each asked once of Claude, ChatGPT and Perplexity with web search on: 120 answers, 0 failed. Each was checked as it arrived, and the text of every source it read was stored, so the check can be repeated on exactly what was compared.
- The sample. Items were drawn at random, using a Bitcoin block that did not exist until after the answers were sealed, so nobody could choose them: 120 that Answer Check marked found and all 105 it marked not found.
- The judges were people, not a model. For each item a labeller saw the claim, the figure and the stored source, and answered supported, not supported or can’t tell. They did not see Answer Check’s verdict or which assistant wrote the answer. Lawrence Fine labelled all 225. Cory Hill labelled 56 of them without seeing Lawrence’s labels. Cory works with Graticus commercially, so he is not an outside reviewer; the labels from both are published below so anyone can re-judge them.
- Agreement. Before any discussion the two agreed on 50 of 56 (Cohen’s kappa 0.66). They settled the other 6 together, looking at the source; both original labels are kept beside the agreed one.
- What wasn’t judged. 83 of 391 cited sources could not be read (blocked, paywalled or gone), so their figures were “couldn’t check” and not labelled. 185 cited claims had no number or quote to check. Labellers marked 17 items can’t tell, each with a reason, and those are left out of the shares above. Labellers were asked to note any figure Answer Check failed to pick up; they noted none.
A side finding about the assistants
This measures the assistants, not Answer Check. Weighted back to all the figures they cited, the estimated share people judged not supported by the cited source:
| Assistant | Answers | Citing a checkable figure | Figures cited | Judged by people | Not supported | Estimate |
|---|---|---|---|---|---|---|
| Claude | 40 | 40 | 556 | 156 | 6 | 3% |
| ChatGPT | 40 | 22 | 63 | 37 | 9 | 19% |
| Perplexity | 40 | 22 | 51 | 15 | 4 | 20% |
Treat this with care. Each assistant answered the same 40 questions, but every Claude answer cited specific figures from its sources while only about half of the ChatGPT and Perplexity answers did, so their estimates rest on few judged items: Perplexity’s on 15. The estimate weights the judged items back to all the figures each assistant cited. One run of 40 questions, one model version each, in October 2026.
Check it yourself
Each sealed file has a proof: a qualified timestamp from Sectigo and a Bitcoin anchor. Drop a .canonical.json file and its .proof.json into verify.graticus.com to confirm it has not changed since it was sealed. The check runs on your own computer.
- plan.canonical.json · proofThe plan, sealed 22 October before any answer existed
- corpus.canonical.json · proofThe 120 answers, every item and the checker's verdicts, sealed before labelling
- results.canonical.json · proofThese results, sealed 28 October
- draw.jsonWhich items were drawn, from which Bitcoin block, and the 100 altered figures
- labels-first.jsonEvery label from the first labeller
- labels-second.jsonEvery label from the second labeller
- adjudication.jsonThe agreed label for each disagreement
The stored source texts are not republished here because they belong to their publishers; each is identified by its fingerprint in the corpus file. Results fingerprint (SHA-256): 0dbb1c2faa3f604b1b84379c6fbf8f0788381b40dbb5cccf21d9a773bdcd6a89
What happens next
Anything we change or analyse after seeing these results is published separately and marked as such, so it cannot be confused with what the sealed plan measured. The first change will be how a ✕ is worded, so it no longer suggests a figure is absent when it may only be phrased differently. A second round will measure that change the same way.