One Fake Source Broke Every Agent

The pitch for AI agents that browse the web goes like this: give them a search engine, let them cross-reference sources, and they'll separate fact from fiction better than any human could. A new benchmark just showed that pitch is fantasy.

The Synthetic Web

Researchers Shrey Shah and Levent Ozgur built something clever for their 2026 preprint: a procedurally generated mini-internet with thousands of hyperlinked articles, each tagged with ground-truth credibility labels. News sites, blogs, research pages, conspiracy outlets — all interconnected, all timestamped, all cross-cited. The contamination filtering is tight: any question a model can answer without tools gets thrown out, isolating the agent's ability to actually reason from retrieved evidence.

Then they ran six frontier models through it. In standard conditions — no adversarial manipulation — GPT-5 hit 65.1% accuracy. Not stellar, but functional. Humans scored 98%.

Then they injected a single misinformation article into search rankings.

One Article, Total Collapse

GPT-5's accuracy cratered to 18.2%. That's a 47-point drop from one fake source. o3 fell from 48.4% to 16.7%. o1 went from 39% to 8.4%. GPT-4o collapsed to 3.8%. The smaller reasoning models — o4-mini and o1-mini — were already at or near zero and stayed there.

Think of it like a jury trial where eleven witnesses tell the truth and one lies. A human juror weighs testimony, notices contradictions, asks follow-up questions. These agents? They're the juror who hears the loudest voice and stops listening. Humans dropped only 5 percentage points under the same adversarial conditions, landing at 93%.

The models didn't just get answers wrong — they got confident about wrong answers. GPT-5's Expected Calibration Error more than doubled, jumping from 0.298 to 0.641. The agents became more sure of themselves precisely when they should have become less sure.

They Didn't Even Try Harder

Here's what the headlines miss: the agents barely changed their search behavior when confronted with conflicting information. GPT-5 made 6.45 tool calls in normal conditions and 6.61 in adversarial ones. That's essentially flat. Only 62% of GPT-5's adversarial queries involved five or more tool calls. For o1, that number was 13%. For GPT-4o, 7%.

A competent researcher who encounters a suspicious claim digs deeper. These models encountered a suspicious claim and shrugged. They had unlimited access to truthful sources — thousands of them — and couldn't be bothered to look. The problem isn't information access. It's that these models have no instinct for when something smells wrong.

What This Actually Means

This result should worry anyone building agentic systems for high-stakes domains. Search engine optimization already manipulates what humans see. An adversary who understands how AI agents rank and consume sources doesn't need to flood the internet with misinformation — they just need to place one well-crafted article where the agent will find it first.

The accountability question gets worse when your agent can't even tell it's been fooled. Current safety documentation doesn't test for adversarial source injection. Current benchmarks don't model it. And current models show "catastrophic failures" — that's the paper's language, not mine — when exposed to it.

Shah and Ozgur's benchmark is a diagnostic tool, not a solution. But it reveals a gap between how we talk about AI agents and how they actually behave when the information environment turns hostile. We've been testing these systems in clean rooms. The real web isn't a clean room.