Benchmark Watch
Evaluation notes, benchmark interpretation, leaderboard skepticism, and measurement failures.
Deep Dives and Frameworks
Implementation playbooks, operator patterns, and durable analysis.
No deep-dive content is currently available for this path.
Signals, Maps, and Watch Lists
Production-oriented analysis, benchmarks, and market/system intelligence.
External tools
Execution tooling is separate
Swarm Signal keeps the analysis layer. Use BoredTools for reusable production templates and trackers.
Tool-Use Agents Need Failure Labels, Not Pass Rates
Tool-use agents can fail in ways a final accuracy score hides, because the same wrong answer can come from skipped tools, ignored outputs, fabricated...
Computer-Use Agents Fail Long Workflows, Not Mouse Clicks
Computer-use agents are clearing more short benchmark tasks, but the new failure line is workflow length. A June 2026 benchmark called OSWorld 2.0 tests...
Agent Leaderboards Can Be Cheaper Without Being Safer
A March 2026 paper on efficient agent benchmarking found that mid-difficulty task subsets can remove large parts of an agent benchmark while preserving...
Multimodal Memory Tests Expose the Personal-Agent Gap
Product teams are turning memory into the selling point for personal agents. The hard question is no longer whether they can remember a preference; it is...
Power Grid Agents Need Constraint Tests, Not Chat Scores
A June 2026 power-systems benchmark argues that language-model agents can solve grid-engineering tasks, but the useful signal is narrower: the agent must...
TerminalWorld Makes Agent Benchmarks Harder to Fake
TerminalWorld turns public terminal recordings into validated agent tasks. The signal is not a higher leaderboard score. It is a harder benchmark supply chain.
Million-Token Context Still Fails the Workload Test
Anthropic reported on February 5, 2026 that Claude Opus 4.6 scored 76% on the 8-needle 1M-token MRCR v2 test while Claude Sonnet 4.5 scored 18.5% on the...
Coding Agent Benchmarks Hit the Generalization Wall
Scale's SWE-Bench Pro public leaderboard reports that top models scoring above 70% on SWE-Bench Verified fall to 23.3% for OpenAI GPT-5 and 23.1% for...
Self-Improving Agents Have an Evaluator Problem
Anthropic's June 2026 update on recursive self-improvement is not a distant sci-fi warning. The company says its engineers now ship 8x as much code per...
The 12-to-72 Problem: Computer-Use Agents Hit Human Scores but Miss the Point
Computer-use agents jumped from 12% to 72% on OSWorld in 18 months. The scores look like progress. The latency and efficiency numbers tell a different story.