LISTEN TO THIS ARTICLE

Agent Test-Time Scaling Needs Reuse, Not More Rollouts

General AgentBench reports that running agents for more interaction steps or more sampled trajectories did not reliably improve ten leading agents, because sequential scaling hit a context ceiling and parallel scaling hit a verification gap General AgentBench.

Evidence base: General AgentBench submitted on 22 February 2026, an April 2026 agentic-coding test-time compute study, a May 2026 Pareto analysis of multi-agent reasoning, the 15 July 2026 CIPHER data-science agent paper, WorkBench Revisited, and practitioner guidance from Anthropic and OpenAI General AgentBench.

Key takeaways

  • Main change: agent scaling is moving from "run more attempts" to "select and reuse better state".
  • Practical implication: stored rollout summaries may matter more than raw trace volume.
  • Caveat or risk: extra compute helps only when the system can verify or rank candidates.
  • Recommendation: budget for selection, summaries, and replay tests before buying more parallel workers.

The April agentic-coding paper gives the cleanest positive result.

What This Benchmark Actually Tests

General AgentBench tests whether agents can work across search, coding, reasoning, and tool-use tasks inside one framework, rather than optimising for a single narrow benchmark General AgentBench. That matters because test-time scaling is useful only if extra attempts can be checked against the real task.

The old test-time compute story was clean. Sample more, think longer, vote harder. That works best when the output is short and cheap to check. Agents are messier. A run contains tool calls, failed branches, hidden state, external side effects, and a handoff that a user has to trust.

For models and frontiers, the uncomfortable result is that neither sequential nor parallel scaling produced effective gains in General AgentBench General AgentBench. That fits Swarm Signal's earlier warning in Coding Agents Need Trajectory Reviews, Not Pass Bits. A pile of unranked trajectories is delayed judgement.

What The New Papers Add

The April agentic-coding paper gives the cleanest positive result. It represents each rollout as a compact summary of hypotheses, progress, and failure modes, then uses those summaries for selection and reuse Scaling Test-Time Compute for Agentic Coding. In reported examples, Claude-4.5-Opus rose from 70.9% to 77.6% on SWE-Bench Verified and from 46.9% to 59.1% on Terminal-Bench v2.0 Scaling Test-Time Compute for Agentic Coding.

CIPHER reaches a similar conclusion from data science agents. Its authors separate candidate initial-state generation from selection, execute selected routes in parallel, and aggregate the results CIPHER. The important move is deciding which states deserve execution before spending the expensive part of the budget.

The counterweight is the May Pareto study. Across 34 configurations and more than 100 evaluations on MMLU-Pro and BBH, inference scaling improved accuracy by up to 7.1 percentage points at the highest evaluated budget, which was 20 times the chain-of-thought compute budget Multi-Agent Reasoning Improves Compute Efficiency. The question is whether the marginal gain beats the marginal bill.

Cheap rollouts tempt teams to hide weak selection behind more attempts.

Cost Is Now A Benchmark Result

WorkBench Revisited makes the cost axis explicit by reporting completion, harmful side effects, and estimated dollar cost per task across 24 models released between 2023 and 2026 WorkBench Revisited. It also says equivalent task-completion cost has fallen roughly a hundredfold since 2024 WorkBench Revisited.

That cost drop cuts both ways. Cheap rollouts tempt teams to hide weak selection behind more attempts. Anthropic's agent guidance argues for simple composed workflows where needed, not default agent loops Building effective agents. OpenAI's agent guidance treats tool behaviour, guardrails, and evaluation as design work rather than magic produced by extra sampling OpenAI agent safety.

Inference from the papers above: the useful agent stack will look less like blind retries and more like controlled search. Generate candidate plans. Summarise failed and promising rollouts. Rank them with task-specific evidence.

Operator takeaway

If you are scaling agent runs, measure cost per successful task, side-effect rate, and selection accuracy before adding more workers WorkBench Revisited.

One practical action: store compact rollout summaries beside full traces, then test whether future runs improve when they can reuse those summaries.

One thing to avoid: treating parallel retries as reliability. Without a verifier, parallelism just creates more confident uncertainty.

Source trail

Research:

Industry and practitioner context:

Related Swarm Signal analysis: