Benchmark Watch
Evaluation notes, benchmark interpretation, leaderboard skepticism, and measurement failures.
Guides and explainers
Detailed guides and practical technical analysis.
No guides are published for this topic yet.
Latest analysis
Recent research, benchmark reviews and technical updates.
Practical tools
Templates for budgets and project planning
BoredTools offers practical spreadsheets for budgets, freelance work and small projects.
Agent Reliability Scores Are Getting Worse, Not Better
SWE-Bench scores tick up every quarter, but production failure rates aren't dropping. A METR study found half of test-passing PRs wouldn't be merged. The more capable we make agents, the less reliably they behave.
RAG for Legal: Building Document Retrieval That Survives Court
More than 300 documented instances of AI-generated fake citations have appeared in court filings since mid-2023. The question isn't whether to use AI for legal research — it's how to build retrieval systems that hold up under adversarial scrutiny.
Agent Benchmarks Won't Sit Still
Static agent benchmarks assume frozen environments. ProEvolve evolved one environment into 200 with 3,000 task sandboxes. Every frontier model failed in structurally different ways when familiar tools disappeared.
The UK Is Letting AI Diagnose Your Dog
ManyPets routes every insurance claim through an AI agent. 55% need zero human involvement. In the same year, the RCVS dropped the physical exam requirement for prescribing. Each piece works. Nobody's testing the integration.
Agentic RAG: How AI Agents Are Rewriting Retrieval
The old retrieve-once-generate-once pipeline is dead, and agents killed it. Four architectural patterns are reshaping how production systems handle knowledge retrieval.
How to Evaluate AI Models Without Trusting Benchmarks
Benchmarks are contaminated, gamed, and misleading. Here's how to build evaluation systems that predict real-world model performance.
Knowledge Graphs Just Made RAG Worth the Complexity
Retrieval-augmented generation was supposed to solve the hallucination problem. It didn't. Most RAG systems still return the wrong chunk, miss the...
Config Files Are Now Your Security Surface
Agentic coding assistants went from autocomplete to autonomous operators in under two years. Now they're editing production code, filing pull requests,...
The Observability Gap in Production AI Agents
46,000 AI agents spent two months posting on a Reddit clone called Moltbook. They generated 3 million comments. Not a single human was involved. When...
When Your Judge Can't Read the Room
Three months ago, I ran a benchmark comparing GPT-4 and Claude 3 Opus on creative writing tasks. GPT-4 won by a comfortable margin according to my...