Benchmark Watch
Evaluation notes, benchmark interpretation, leaderboard skepticism, and measurement failures.
Guides and explainers
Detailed guides and practical technical analysis.
No guides are published for this topic yet.
Latest analysis
Recent research, benchmark reviews and technical updates.
Practical tools
Templates for budgets and project planning
BoredTools offers practical spreadsheets for budgets, freelance work and small projects.
Multimodal Memory Tests Expose the Personal-Agent Gap
Product teams are turning memory into the selling point for personal agents. The hard question is no longer whether they can remember a preference; it is...
Power Grid Agents Need Constraint Tests, Not Chat Scores
A June 2026 power-systems benchmark argues that language-model agents can solve grid-engineering tasks, but the useful signal is narrower: the agent must...
TerminalWorld Makes Agent Benchmarks Harder to Fake
TerminalWorld turns public terminal recordings into validated agent tasks. The signal is not a higher leaderboard score. It is a harder benchmark supply chain.
Tool Agents Need State Diffs, Not API Call Scores
Tool-use agents are moving from choosing the right API to changing live product state. That makes a clean function-call score too small a test because...
Multilingual Agents Need Workflow Tests, Not Translation Scores
PolyWorkBench landed on arXiv on 7 July 2026 with a useful correction to enterprise-agent hype: a global workflow is not a translated English task...
Agent Benchmarks Need Runtime Receipts, Not Model Labels
RuBench's revised 19 July 2026 release contains a small but important warning for coding-agent buyers: one audited product configuration silently...
Data Agents Need Exploration Budgets, Not SQL Magic
Data Agent Benchmark landed on arXiv on 21 March 2026 with a result that should make enterprise analytics teams pause: the best tested frontier model...
Assistant Agents Need Reminder Tests, Not Recall Scores
Most agent-memory benchmarks ask whether a model can recover old information. PM-Bench asks a harsher question: can an agent remember to do the right...
The 12-to-72 Problem: Computer-Use Agents Hit Human Scores but Miss the Point
Computer-use agents jumped from 12% to 72% on OSWorld in 18 months. The scores look like progress. The latency and efficiency numbers tell a different story.
Why Multi-Agent Papers Don't Replicate in Production
A paper from Tran and Kiela tested 28 multi-agent configurations across four architectures: Sequential, Parallel, Debate, and Ensemble. Every single one...