We gave the same claims to two models. They contradict each other on 1 in 17.
of claims got flatly opposite verdicts: one model said the sources support it, the other said they do not. Same claim, same sources, same prompt.
Window claims drawn from 2026-05-03 → 2026-08-09, run 2026-08-13
Source systems operated by weranai — method in Methodology
Our first report measured how often an AI writer states things its sources do not support, and closed with a limitation we could not resolve: the fact checker doing the measuring is itself a model, with an error rate we had never quantified. This report is that measurement.
The method is blunt on purpose. We took 399 claims the pipeline had already judged, together with the exact source material the writer was given, and put each one to two models under identical conditions: same neutral prompt, same sources, same three-way output contract. The only variable is the model.
They agree most of the time
87.2% agreement. On its own that reads reassuring, and it is the number most people would publish and stop at.
The 5.8% underneath it is the finding. Those are claims where one model said the sources support the statement and the other said they do not. Not a shade of confidence apart: opposite verdicts, on the same sentence, against the same documents. Roughly one in seventeen.
The agreement is concentrated in the easy cases
Splitting by what the pipeline had originally decided shows where the consensus actually lives.
| Claims the pipeline had… | n | Agreement | Contradiction |
|---|---|---|---|
| Pipeline had passed it | 199 | 89.9% | 4.5% |
| Pipeline had flagged it | 200 | 84.5% | 7.0% |
On claims that had passed cleanly, the models contradict each other 4.5% of the time. On claims that had been flagged as problematic, 7.0% — more than half again as often.
That is the wrong way round for anyone relying on this. Automated checking is worth having precisely for the hard cases; the easy ones were never the risk. Agreement is highest where it matters least.
The two models are not equally strict
| Model | Unsupported | Supported | Unclear |
|---|---|---|---|
| Claude Sonnet 4.5 | 150 (37.6%) | 225 | 24 |
| GPT-4o | 168 (42.1%) | 225 | 6 |
GPT-4o rules 42.1% of claims unsupported. Claude, 37.6%. Same claims, same sources, four and a half points apart.
The practical consequence is the uncomfortable one. A fabrication rate is not a property of the system being measured. It is a property of the pair. Swap the checker and your published error rate moves, with nothing about the writer having changed. Our first report put the figure near 30%. Run the same articles past a different checker and that number would be different, and neither run would be wrong.
The two also disagree about how much doubt to admit. Claude returned “unclear” 24 times, GPT-4o 6. A checker that rarely hedges looks more decisive in a dashboard and tells you less.
What they are actually disagreeing about
Reading the contradictions, almost none look like a mistake by either model. They are disagreements about how much inference a source is allowed to carry.
Claim: “semiconductor names led the decline”
Sources: a headline saying chip stocks slid
UNSUPPORTED the sources do not say semiconductors led
SUPPORTED chip stocks slid, which implies semiconductors led the decline
Claim: “deuterium — heavy hydrogen with an extra neutron”
Sources: deuterium described as “heavy hydrogen”
UNSUPPORTED the sources never mention the extra neutron
SUPPORTED heavy hydrogen implies an extra neutron
Claim: “Now it's cutting back to approximately 70,000”
Sources: Meta has 78,000 employees and is laying off roughly 10%
UNSUPPORTED the figure 70,000 does not appear in the sources
SUPPORTED 78,000 minus roughly 10% supports approximately 70,000
One model treats a source as covering what follows from it. The other requires the statement to be present. Both positions are defensible, and the choice between them is editorial, not technical.
Which means “supported by the sources” has no single definition, and every automated checker silently supplies its own. If you deploy one without deciding where that line sits, the model decides for you and never tells you which rule it applied.
What this does and does not show
- Agreement is not accuracy. Both models can be wrong together, and nothing here catches that. Measuring accuracy needs claims with human-established ground truth, which we do not have at this scale.
- Two models, one prompt, one run at temperature zero. Different prompts would produce different rates; the wording of a checking instruction is itself a variable we have not explored.
- The claims come from our pipeline’s output, which skews toward the kinds of sentences it writes. A different writer would give a different mix.
- We deliberately did not use the pipeline’s historical verdicts as a baseline. Those were produced under other conditions — whole article, capped length, up to five claims at once — so comparing against them would measure the conditions, not the models.
- 399 of 400 sampled claims returned usable verdicts from both models. One failed to parse. None were lost to rate limiting.
The full run, including every claim and both stated reasons, is in data/experiments/checker-agreement-2026-08-13.json. Method and definitions are in Methodology. The rate this report qualifies is in the first report.