We fact-checked 5,500 claims from AI-written news. 30% weren't supported by the sources.
factual claims had no support in the sources the system was given. Not wild hallucination — plausible context filled in from model memory.
Window 2026-05-03 → 2026-08-04
Source systems operated by weranai — method in Methodology
We run an autonomous editorial pipeline: a set of agents that research, write, and publish news articles without a human in the loop. Every article passes through an automated fact-checker before publication, which reads each factual claim and tries to trace it back to the source material the writing agent was given.
Over three months that check ran on 1,140 articles and examined 5,500 individual claims. This is what it found.
Claim-level results
The unit that matters is the claim, not the article. An article can read perfectly and still contain a figure nobody can trace.
| Verdict | Claims | Share | Meaning |
|---|---|---|---|
| VERIFIED | 3,621 | 65.8% | Supported by the source packet |
| FLAGGED | 1,635 | 29.7% | No support found in the sources |
| RECENT | 146 | 2.7% | Too recent to verify at write time |
| CORRECTED | 98 | 1.8% | Rewritten to match the source |
Nearly one in three factual claims had no backing in the material the system had in front of it. Reading the flagged examples, almost none are absurd. They are plausible: a specific exchange rate, a year for a past event, a job title. The model fills gaps with things that sound right, and the gaps are invisible unless you check claim by claim.
Article-level results
A single unsupported claim is enough to hold an article back, so the article-level pass rate is much harsher than the claim-level one.
| Status | Articles | Share | Meaning |
|---|---|---|---|
| NEEDS_REVIEW | 833 | 73.1% | At least one unsupported claim |
| PASS | 269 | 23.6% | Every claim traced to a source |
| MINOR_ISSUES | 28 | 2.5% | Small corrections applied |
| PENDING | 10 | 0.9% | Check did not complete |
Under 23.6% of articles came through with every claim traced. Put differently: left unattended, roughly three out of four AI-written news articles would have published at least one figure nobody could source.
Fabrication is not evenly distributed
The clearest finding is that the failure rate depends on the subject. The same models, the same prompts, the same pipeline — very different results by topic.
| Topic | Clean | Total | Pass rate | |
|---|---|---|---|---|
| Technology | 98 | 320 | 30.6% | |
| Space | 27 | 99 | 27.3% | |
| AI | 52 | 205 | 25.4% | |
| Markets/Investing | 52 | 251 | 20.7% | |
| World politics | 40 | 265 | 15.1% |
World politics fails roughly twice as often as technology. Our reading: topics with dense primary documentation — product specs, filings, release notes — give the writer something concrete to anchor to. Geopolitical coverage rewards synthesis and context, which is exactly where a model reaches for memory instead of sources.
If that holds generally, it implies the risk of using generative systems for reporting is not uniform. It is highest precisely where verification is hardest and the cost of being wrong is greatest.
What this does and does not show
- The checker only ran on news articles. Procedural content written against vendor documentation was never part of this sample, and we would expect it to behave differently.
- “Unsupported” means absent from the source packet given to the writer. Some flagged claims may well be true in the world — they simply were not traceable at write time, which is the standard that matters before publishing.
- The fact-checker is itself a model. It has its own error rate, which we have not yet measured. That measurement is on the list.
Full method, definitions, and the exact status vocabulary are in Methodology.