The study's central finding is that five frontier LLMs reached unanimous agreement on only 33% of 1,000 already-adjudicated fact-check claims. Lenz argues this is materially worse than the 70-80% inter-rater reliability seen among professional human fact-checkers, demonstrating the models can't reliably recognize even settled questions.
By submitting the study with the disagreement percentage front-and-center in the title, kostaj frames the finding as significant evidence that LLMs fail at recognition tasks, not just novel epistemic work. The 309-point score and 208-comment thread reflect community resonance with the reliability concern.
The editorial argues the dissents aren't randomly distributed but cluster precisely on politically charged topics, contested scientific consensus, and geopolitics. This pattern suggests the models are reflecting the contested nature of their training corpora rather than failing at random — they disagree most where rigor is most needed.
Lenz.io published a study running 1,000 real-world fact-check claims — the kind already adjudicated by professional fact-checkers at Snopes, PolitiFact, and AFP — through five frontier LLMs: OpenAI's GPT, Anthropic's Claude, Google's Gemini, xAI's Grok, and DeepSeek. Each model was given the same claim, the same prompt, and asked for a verdict on a standard true/false/mixed scale.
The five models reached unanimous agreement on only 33% of claims. On the remaining 67%, at least one model dissented — and on a non-trivial slice, the split was 3–2 or even 2–2–1 across the verdict categories. This isn't the familiar "LLMs hallucinate sometimes" finding. This is five systems, trained on overlapping corpora of the public internet, looking at the same sentence and producing materially different answers about whether it's true.
The HN thread (309 points) zeroed in on the awkward implication: the dataset is *real fact-check claims*, meaning humans with subpoena power, primary sources, and editorial processes had already produced an answer. The models weren't being asked to do novel epistemic work. They were being asked to recognize a settled question — and two-thirds of the time, at least one of them got a different answer than its peers.
The instinctive read is "models are unreliable, news at 11." That undersells what the disagreement pattern actually shows. The dissents aren't randomly distributed across claims — they cluster on politically charged topics, contested scientific consensus, and anything touching national identity or recent geopolitics. In other words: the models disagree most exactly where you'd hope a fact-checker would be most rigorous.
Compare this to inter-rater reliability work in human fact-checking, where professional checkers hit roughly 70–80% agreement on contested claims and >90% on clear-cut ones. Five frontier LLMs sitting at 33% unanimity is well below the human floor, and the gap isn't closing with scale — GPT-4-class models don't agree with each other meaningfully more than the prior generation did. Throwing more parameters at the problem hasn't produced convergence on truth; it's produced more confident divergence.
The community reaction split along predictable lines. One camp argued this is evidence the models are working as designed: alignment is supposed to inject editorial judgment, and different alignment teams make different calls. The other camp pointed out that this is precisely the problem — if your fact-checker's answer depends on which lab fine-tuned it, you don't have a fact-checker, you have a focus group with an API. A senior commenter put it bluntly: "This is RLHF leaking into epistemics. The models aren't disagreeing about reality; they're disagreeing about which reality their reviewers preferred."
There's also a quieter technical story underneath. The disagreements correlate with how each model handles uncertainty: Claude tends toward "mixed" verdicts more often, Gemini toward refusals, Grok toward confident contrarianism, GPT toward institutional consensus, DeepSeek toward whatever its (less-documented) post-training pushes it toward. These aren't bugs to be patched. They're the deliberate output of five different teams making five different bets about what a helpful assistant should say when the truth is contested.
If you're using a single LLM as a verification layer — RAG groundedness checks, content moderation, claim extraction for analytics, automated fact-tagging — you've quietly outsourced editorial judgment to whichever vendor's API you're calling. The lesson isn't "don't use LLMs for this"; it's "a single LLM is not a source of truth, it's one vote in a panel that you've decided not to convene."
Three concrete moves:
Run an ensemble for high-stakes verdicts. If you're flagging misinformation, validating user claims, or auto-moderating contested content, query two or three models and surface the disagreement to a human (or to your logs). The 33% unanimity number means roughly two-thirds of your decisions are model-choice-dependent — which is exactly the rate at which you should be suspicious of any single answer. Cost is the obvious objection, but for verdict-level work the per-call cost is negligible against the cost of being systematically wrong on a third of cases.
Log the verdict and the model. If your pipeline has a `is_factual: true` field somewhere in the database, add a `verdict_source` column. When you swap from Claude 4.x to Claude 5.x or migrate from GPT to Gemini for cost reasons, you'll want to know whether your moderation outcomes shifted because the world changed or because the model did. Most teams don't track this and discover the drift only when a journalist asks.
Stop calling it ground truth in your docs. Internal language matters. If your team refers to LLM output as "the fact-check," engineers downstream will treat it as authoritative. Call it "the model's verdict" or "the GPT label" — the friction of typing the model name keeps the epistemics honest.
The Lenz study is going to get cited a lot in the next round of EU AI Act and FTC enforcement debates, and rightly so — it's the cleanest evidence yet that "AI fact-checking" is a category error at the current state of the art. For practitioners, the takeaway is narrower and more useful: treat LLM verdicts the way you treat any single sensor reading — as input to a decision, never the decision itself. Expect the next 12 months to bring vendor pushback ("our model has improved factuality by X%"), new benchmarks that conveniently show convergence on whatever the vendor optimized for, and at least one high-profile lawsuit where a platform's moderation decisions get traced back to a single model's idiosyncrasies. Build like that's already happened.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.