Benchmark Watch
Evaluation notes, benchmark interpretation, leaderboard skepticism, and measurement failures.
Guides and explainers
Detailed guides and practical technical analysis.
No guides are published for this topic yet.
Latest analysis
Recent research, benchmark reviews and technical updates.
Practical tools
Templates for budgets and project planning
BoredTools offers practical spreadsheets for budgets, freelance work and small projects.
ComplexMCP Finds Tool-State Gaps
ComplexMCP is a useful benchmark because it tests the part of tool use that simple function-calling tasks avoid: one action changes the state that the...
Android Bench Tests Long Coding Work
Android Bench turns mobile coding evaluation from small patch repair into long-horizon Android work. Google says the updated benchmark adds multi-day...
KNOWS Tests Browser Work After Search
The KNOWS paper turns web-agent evaluation from finding information into producing a usable document, spreadsheet or slide deck...
OSWorld-Pro Exposes Click Failures
OSWorld-Pro turns desktop automation into a step-by-step audit. The paper extends OSWorld-style computer-use evaluation with process checks, so a model...
PM-Bench Tests Delayed Agent Memory
PM-Bench turns prospective memory into an agent evaluation problem: can a system remember to act on a future cue while ordinary work continues? The...
Can Security Agents Work Under Human Custody?
Hack The Box's 2026 Global Cyber Skills Benchmark is a useful adoption signal because it measures where security practitioners chose to use AI agents...
Bazaar Shows Pricing Agents Miss Profit
[Bazaar](https://arxiv.org/abs/2608.00102) is a benchmark submitted on 30 July 2026 for testing whether a language-model merchant can learn customer...
SWE-Bench ProMax Tests Refactoring Depth
SWE-Bench ProMax moves coding-agent evaluation from single-issue repair towards large, behaviour-preserving refactors. The paper was submitted on 10...
Tool Agents Need Taint Boundaries, Not Bigger Scopes
APPA landed on arXiv on 27 July 2026 with a blunt result for tool-agent security: permission prompts are too late once poisoned or confidential data has...
Chip-Design Agents Need Token ROI, Not Demo Flows
FluxBench landed on arXiv on 20 July 2026 with a useful warning for chip-design teams: two agents can start from the same foundation model and still...