Benchmark Watch
Evaluation notes, benchmark interpretation, leaderboard skepticism, and measurement failures.
Guides and explainers
Detailed guides and practical technical analysis.
No guides are published for this topic yet.
Latest analysis
Recent research, benchmark reviews and technical updates.
Practical tools
Templates for budgets and project planning
BoredTools offers practical spreadsheets for budgets, freelance work and small projects.
Terminal Agents Need Progress Curves, Not Victory Screens
Long-Horizon-Terminal-Bench landed on arXiv in July 2026 with an awkward result for terminal-agent buyers: even the strongest tested model still failed...
Million-Token Context Still Fails the Workload Test
Anthropic reported on February 5, 2026 that Claude Opus 4.6 scored 76% on the 8-needle 1M-token MRCR v2 test while Claude Sonnet 4.5 scored 18.5% on the...
Agent Evals Need Harnesses, Not More Scoreboards
AgentCompass first landed on arXiv on 15 July 2026 with a practical complaint: agent evaluation is fragmented, tightly coupled, and hard to reproduce...
Coding Agent Benchmarks Hit the Generalization Wall
Scale's SWE-Bench Pro public leaderboard reports that top models scoring above 70% on SWE-Bench Verified fall to 23.3% for OpenAI GPT-5 and 23.1% for...
Agent Test-Time Scaling Needs Reuse, Not More Rollouts
General AgentBench reports that running agents for more interaction steps or more sampled trajectories did not reliably improve ten leading agents,...
Multi-Agent Finance Workflows Need Cost Curves, Not More Agents
A March 2026 benchmark on financial-document processing makes the uncomfortable point: the most accurate multi-agent architecture was not the obvious...
Self-Improving Agents Have an Evaluator Problem
Anthropic's June 2026 update on recursive self-improvement is not a distant sci-fi warning. The company says its engineers now ship 8x as much code per...
Coding Agents Need Trajectory Reviews, Not Pass Bits
Most coding-agent benchmarks still compress a whole run into one bit: did the task pass? AgentLens argues that users experience the whole trajectory...
Tool-Use Agents Need Failure Labels, Not Pass Rates
Tool-use agents can fail in ways a final accuracy score hides, because the same wrong answer can come from skipped tools, ignored outputs, fabricated...
Agent Leaderboards Can Be Cheaper Without Being Safer
A March 2026 paper on efficient agent benchmarking found that mid-difficulty task subsets can remove large parts of an agent benchmark while preserving...