{"corpus":{"assistants":["Claude, with web search","ChatGPT, with search","Perplexity"],"collection":"Each question is asked once of each assistant through the vendor's API with web search on, exactly as written, in a fresh request, within 7 days after this plan's seal. The reply is turned into the text a person would paste, links included (src/eval/vendors.ts); the raw reply is stored beside it so the conversion can be checked. A failed request is retried twice, then recorded as failed: counted, not dropped.","freezing":"Each answer is run through Answer Check at the pinned commit as soon as it is collected. The text of every source read is stored with its SHA-256, so the check can be re-run on exactly what was compared. The answers, the stored sources and the checker's verdicts are sealed together as the corpus before any labelling.","questions":"The 40 questions in questions below, fixed by this plan.","unit":"An item: one number or quotation a claim states, with the source the claim cites, as Answer Check extracts it."},"decisions":["Collection through the vendors' APIs with web search, for repeatability (decided 22 Oct 2026). Perplexity through its Agent API, preset \"medium\".","The second labeller is chosen by Lawrence before labelling starts; their name is reported with the results.","The results will be published on graticus.com and check.graticus.com whatever they show (decided 22 Oct 2026)."],"exclusions":[{"applied_before_seeing_output":true,"reason":"None at the answer level. An answer with no citations, a refusal or a failed collection is counted and reported, not dropped."},{"applied_before_seeing_output":true,"reason":"Items whose sources could not be read (verdict \"couldn't check\") are not labelled: there is nothing to compare. They are counted (M5)."}],"labelling":{"adjudication":"Disagreements on the overlap are discussed by the two labellers and the agreed label recorded beside both originals. Agreement before discussion is reported (Cohen's kappa). No model labels or adjudicates: the point is to check an automated checker against people.","blind_to_key":true,"blind_to_system":true,"how":"For each item the labeller sees the claim, the number or quote it states, and the stored source text with a search box. Not shown: Answer Check's verdict, the passage it found, or which assistant wrote the answer. The labeller answers: SUPPORTED (the source states this figure or these words, for the thing the claim is about); NOT SUPPORTED (it doesn't, or states it only about something else); CAN'T TELL (with a reason). Rounding the source's own figure counts as supported only if the claim says 'about' or the rounding is to the claim's stated precision.","labellers":"Two people. Lawrence Fine labels every drawn item; a second labeller, chosen by Lawrence and named in the results, labels a quarter of them, chosen by the same seed.","second_labeller":"named before labelling starts; blind to Lawrence's labels"},"measures":{"M1":"Primary. False reassurance: of labelled items in the found stratum (CAN'T TELL excluded), the share labelled NOT SUPPORTED. 95% Wilson interval reported.","M2":"Secondary, with a mark. False alarm: of labelled items in the not-found stratum (CAN'T TELL excluded), the share labelled SUPPORTED. 95% Wilson interval reported.","M3":"Secondary, with a mark. Planted figures caught: of the 100 planted items, the share Answer Check marks not found in source.","M4":"Secondary, with a mark. Reproducibility: re-running the pinned checker on the stored sources gives the same receipt fingerprint for every answer.","M5":"Descriptive. Across the whole corpus: answers with no citations; sources that couldn't be read, by reason; claims with nothing to check; CAN'T TELL labels and their reasons.","M6":"Descriptive. Figures in drawn claims that Answer Check did not extract at all (labellers note any number or quote it missed).","M7":"Descriptive, by assistant. Share of labelled items NOT SUPPORTED, weighted back to the corpus by stratum size. A side finding about the assistants, not a measure of Answer Check."},"models":{"chatgpt":"gpt-6.1-sol","claude":"claude-opus-5-5","note":"Claude and ChatGPT are named models. Perplexity's API takes a preset, its own search settings over a model it chooses (in October 2026 the \"medium\" preset runs an OpenAI model); the model that actually answered is recorded with every Perplexity answer. Claude is given one instruction beyond the question, \"Search the web to answer, and cite your sources.\", because through the API it otherwise answers from memory with nothing to cite and this model refuses a forced search; ChatGPT and Perplexity search by default and get the question alone. Checked on a trial question outside the 40 before sealing.","perplexity":"medium"},"passMarks":{"basis":"Set before any answer exists, with no prior measurement of this checker. M1 is held tightest because false reassurance is the harm the tool exists to prevent; its 5% matches the brief checker's lookup mark (round 3: met). M2 is looser (15%) because the checker reports only exact figures and quotes, so a source that states the figure differently ('four and a half', a table) will be flagged, and the page always shows the nearest passages for the reader to judge; the brief checker's round-2 rate of flags that were not real errors (20 of 29) shows false alarms are the checker's known weakness. M3's 95% because a changed figure is the case the tool is built for. M4's 100% because the check is code, not a model. A share is judged on its point estimate; the interval is reported beside it. Fewer than 60 labelled items in a stratum and that measure is 'not measured', not met or failed.","falseAlarmMax":0.15,"falseReassuranceMax":0.05,"minimumLabelledPerStratum":60,"plantedCaughtMin":0.95,"reproducibleMin":1},"planted":{"detail":"100 items drawn by the same seed from those marked found in source. In each, the stated number is replaced by a value derived from the seed: multiplied by one of 0.5, 0.8, 1.25, 1.5 or 2, or with one digit changed, rounded as the original was. A replacement that appears anywhere in the source's stored text is redrawn. The altered claim is checked against the same stored source. Ground truth is known by construction: the changed figure is not in the source.","size":100},"primary":{"direction":"lower is better","metric":"M1: of items Answer Check marked found in source, the share people label NOT SUPPORTED","why_primary":"This is the harmful error: telling someone a figure is backed by its source when it isn't."},"questions":[{"area":"health","id":"Q01","text":"How much caffeine is safe for an adult to drink in a day?"},{"area":"health","id":"Q02","text":"How much protein does an adult need per day?"},{"area":"health","id":"Q03","text":"How many steps a day lower the risk of dying early?"},{"area":"health","id":"Q04","text":"How much water should an adult drink each day?"},{"area":"health","id":"Q05","text":"How much sleep do teenagers need?"},{"area":"health","id":"Q06","text":"How much added sugar should adults limit themselves to per day?"},{"area":"health","id":"Q07","text":"What share of adults in the US have high blood pressure?"},{"area":"health","id":"Q08","text":"How effective is the flu vaccine in a typical season?"},{"area":"medicines","id":"Q09","text":"What is the maximum daily dose of acetaminophen for adults?"},{"area":"medicines","id":"Q10","text":"How much weight do people lose on semaglutide in clinical trials?"},{"area":"medicines","id":"Q11","text":"How common are muscle side effects from statins?"},{"area":"medicines","id":"Q12","text":"What is the half-life of ibuprofen?"},{"area":"medicines","id":"Q13","text":"How effective is the HPV vaccine at preventing cervical cancer?"},{"area":"medicines","id":"Q14","text":"What share of Americans take at least one prescription drug?"},{"area":"medicines","id":"Q15","text":"How long do antidepressants take to start working?"},{"area":"medicines","id":"Q16","text":"What is the success rate of IVF for women under 35?"},{"area":"money","id":"Q17","text":"What is the average annual return of the S&P 500 over the long run?"},{"area":"money","id":"Q18","text":"How much does the typical American have saved for retirement?"},{"area":"money","id":"Q19","text":"What is the US federal minimum wage, and when did it last change?"},{"area":"money","id":"Q20","text":"How much does a year at a four-year public university cost in the US?"},{"area":"money","id":"Q21","text":"What share of new restaurants close in their first year?"},{"area":"money","id":"Q22","text":"How much credit card debt does the average US household carry?"},{"area":"money","id":"Q23","text":"What was the US inflation rate in 2023?"},{"area":"money","id":"Q24","text":"How much does an average US household spend on groceries a month?"},{"area":"science","id":"Q25","text":"How much has the global average temperature risen since pre-industrial times?"},{"area":"science","id":"Q26","text":"How fast is the sea level rising each year?"},{"area":"science","id":"Q27","text":"How much CO2 does a typical passenger car emit in a year?"},{"area":"science","id":"Q28","text":"What share of the world's electricity came from renewables last year?"},{"area":"science","id":"Q29","text":"How many species are estimated to live on Earth?"},{"area":"science","id":"Q30","text":"How far is the Moon from the Earth?"},{"area":"science","id":"Q31","text":"How much plastic enters the ocean each year?"},{"area":"science","id":"Q32","text":"How efficient are typical home solar panels?"},{"area":"technology","id":"Q33","text":"How many people use the internet worldwide?"},{"area":"technology","id":"Q34","text":"How long do electric car batteries last?"},{"area":"technology","id":"Q35","text":"How much energy does a ChatGPT query use compared with a Google search?"},{"area":"technology","id":"Q36","text":"How many data breaches were reported in the US last year?"},{"area":"technology","id":"Q37","text":"How much screen time do children aged 8 to 12 have per day?"},{"area":"technology","id":"Q38","text":"What share of jobs could be automated by AI?"},{"area":"technology","id":"Q39","text":"How many people die in road crashes worldwide each year?"},{"area":"technology","id":"Q40","text":"How many people speak English worldwide?"}],"registered_by":"Lawrence Fine, Graticus","reporting":"The results are sealed the same way as this plan and the corpus. Nothing in this plan may change after sealing; a changed plan is a new plan with its own seal, and both are reported. Analysis done after seeing results is published separately and marked as such.","rule":{"detail":"From the sealed corpus, 120 items are drawn from those Answer Check marked found in source, and 120 from those it marked not found, or all of a stratum if it has fewer. Each item is ranked by SHA-256(seed || item id) and the lowest ranks are taken. Items are drawn by verdict because the two error rates that matter are conditional on the verdict: when it says found, and when it says not found.","method":"stratified random draw by verdict","seed":"SHA-256(hash of the first Bitcoin block whose median time past is after the corpus seal's timestamp || SHA-256 of the corpus canonical file)","seed_issued_by":"the Bitcoin network, after the corpus is fixed; scripts/eval.ts draw <block hash> recomputes the draw"},"system":{"commit":"002fc350675180fbb4d05156fa7e2df8c289fed5","files":["src/parse.ts","src/fetchSource.ts","src/check.ts","src/figures.ts","scripts/eval.ts"],"name":"Graticus Answer Check (check.graticus.com), the answer checker only. \"Before you ask\" is out of scope: it judges nothing.","repository":"lgfine-dot/answer-check"},"title":"Answer Check: when it says a figure is or isn't in its source, how often is it right? A pre-registered evaluation, round 1","verdicts":"Each measure with a mark is met, not met, or not measured. All are reported, met or not, with the item ids, labels and fingerprints they were computed from."}