An agent that checks every number against the evidence, measured with planted errors

The problem

A technical report, a README or a case study is full of figures: percentages, timings, versions, licences, dates and "we did X" sentences. Between the figure and the original data there are hand copies, roundings and values left over from an earlier version. Checking by hand takes hours, and if you ask an LLM "is this true?" it tends to say yes. We wanted an agent that cannot accept anything it cannot point a finger at, and to know with numbers when the complex part is worth it.

What we did: three designs and a rule

• Retrieval (round 1): a claim extractor (rules + LLM), a deterministic evidence retriever (search, JSON paths, CSV rows, arithmetic that recomputes sums and ratios), an LLM judge with its output forced into a schema and a mandatory citation, and a deterministic guard that downgrades any OK without a real citation to UNVERIFIED.
• LLM only: the same extractor and model, but the evidence folder pasted into the prompt, with no tools and no guard.
• Hybrid v2 (the one we publish): if the relevant evidence files fit the context budget (a parameter) the judge reads them whole; if not, it uses retrieval. Both paths end in the same guard: no OK without a cited span that contains the numbers, no MISMATCH without a cited span and a found value.
• The model is set by environment variable (any OpenAI-compatible endpoint). Temperature 0, fixed seed, every call logged.

How we measured it

A labelled benchmark, built and hash-sealed before the second round ran. Each report exists as a clean version and two versions with planted errors of eight types: a changed number, a unit or rounding slip, a wrong sum or ratio, numbers on the wrong system, a wrong licence, an outdated date, a new claim with no evidence and a "we did X" with no evidence. Ground truth comes from code: every clean claim carries a check against the evidence files, and every planted error must fail it.

• Small reports (test): 213 labelled claims in 12 reports, 138 planted errors, new ones, different from round 1.
• Large evidence (test): 181 claims in 6 reports, 87 planted errors, over 1.6 MB of real logs of ours that do not fit in context. The reports are templated text over those logs: they test deep lookup, not prose.
• Prompts and configuration were tuned on the development sets only; the test sets were run once, except report 10 of the small test set (v2), redone from scratch after the model server crashed.

Numbers: small reports

In brackets, the 95 % interval (bootstrap over claims). "Flagged" counts a MISMATCH or an UNVERIFIED on a planted error as a catch; "passed as OK" is the planted errors the system accepted. Here pasting the evidence wins or ties: the hybrid flags 94 % and LLM only 93 %, but LLM only gives the exact verdict on 83 % of the 138 planted errors against 63 % and leaves 5 % of correct claims UNVERIFIED against 14 %. Only one of the two repositories has evidence that fits whole; on the other, LLM only sees 41 % of it and still does better. There is no reason to complicate things while the evidence fits.

Numbers: large evidence

Here LLM only sees 4 % of the evidence (it is truncated) and stops being usable: precision of MISMATCH 29 %, 12 % of correct claims called MISMATCH and 67 % left UNVERIFIED. The hybrid keeps 62 % precision (a small base: 8 MISMATCH verdicts), 1 % false alarms and 21 % UNVERIFIED. But it lets 10 % of planted errors pass as OK (interval 5–17 %) against 2 % for LLM only, and its exact verdict is right on only 25 % of the errors: it tends to say "not supported" instead of "this is wrong".

Cost is similar across the three LLM systems: $0.06–0.07 per report on small reports and $0.02–0.03 on large evidence: an estimate, seconds × RunPod's listed RTX 4090 price ($0.34/h Community, $0.89/h Secure, https://www.runpod.io/pricing, page dated 2026-10-08, checked again on 2026-10-17). Local model on a shared GPU; prices change.

A sample output

First rows of a real hybrid run on a report with planted errors, each verdict with the line that proves it.

What fails and what we did not measure

• Planted errors that slip through as OK: 2 on small reports and 9 on large evidence. Real examples, with the model's reason:

• 0.0013 USD (comfyui-photo-preset, line 10): Evidence E1 lists max usd_per_image as 0.001336, which rounds to 0.0013.
• 0.09 % (comfyui-photo-preset, line 117): Classic value matches; preset value is 0.0 in E4.
• 4.066 (ltest, line 35): Evidence E1, E2, E3, E4 all show latency_s=4.066.
• 3.673 (ltest, line 34): Evidence E1 explicitly states latency_s=3.673 for tag judge_llm_only.
• 5.738 (ltest, line 33): Evidence E1 shows latency_s is 5.738 for tag judge.

• False alarms on small reports: 8 correct claims called MISMATCH, for example:

• 0.85 (comfyui-photo-preset, line 26): Evidence states median 0.861, report claims 0.85.
• $0.89/h (canine-vi-diacritics, line 48): report says $0.89/h; evidence says price_date 2026-10-16 (results.json)

• Batching 8 claims per call did not speed anything up: metrics were identical in development and time per report was 563 s vs 561 s on sdev and 345 s vs 345 s on ldev. Every development report took the retrieval path, so the whole-evidence batched path ran for the first time on the test; 17 of 1941 small-set claims ended UNVERIFIED because a batch did not return the schema.
• Not measured: other models (one, local), other domains or languages, PDF or images, evidence above 1.6 MB, evidence beyond these two small repositories and one large set, run-to-run variation (one run per system, except the redone report). We labelled a sample of the checkable claims, not all of them. The agent only knows the evidence folder you give it: it cannot tell whether your data is right, only whether the report reflects it.

What a client gets

A verifier made for your own reports: the extractor tuned to your formats, the retriever wired to your results folders, logs or spreadsheets, the "whole evidence or retrieval" rule calibrated to your data volume, the model you prefer (local or paid, configurable endpoint), an evaluation on a labelled benchmark made from your documents and an honest list of what fails.

Agent systems: from €1,500, excl. VAT. It is a starting price: after the first conversation, the proposal fixes scope and price. Details on the Technical teams page, or write to hello@grawwww.xyz. We answer in under 48 hours.

Credits and licences

• Our code, README and results: MIT licence. The benchmark uses copies of the READMEs, reports and result files of our own repositories (MIT) and of our logs; the reports with planted errors are test data, not statements about those projects.
• Third-party text quoted in that material: Vietnamese Wikipedia sentences (CC BY-SA 4.0 / 3.0 and GFDL, authors in each article's history) and the MIT notice of Comfy Org's templates.
• Model: a community fine-tune of Qwen3.8-27B (Alibaba Cloud) by Hugging Face user Jiunsong, run locally as a Q4_K_M quantisation with llama.cpp (MIT). Both the base model and this fine-tune are published under Apache-2.0. Its author reduced the model's refusal behaviour; we use it only to read short evidence snippets and return verdicts, and the benchmark numbers apply to this exact file only. No weights are redistributed and we are not affiliated with Alibaba Cloud. Exact revision, checksum and sources are in the repository's licences.md.