LISTEN TO THIS ARTICLE

Terminal Agents Need Progress Curves, Not Victory Screens

Long-Horizon-Terminal-Bench landed on arXiv in July 2026 with an awkward result for terminal-agent buyers: even the strongest tested model still failed most long terminal workflows Long-Horizon-Terminal-Bench.

Evidence base as of 26 July 2026: Long-Horizon-Terminal-Bench, Terminal-Bench 2.0, Terminal-Bench 2.1 public task context, TAB for terminal-agent instruction alignment, TUA-Bench for general-purpose terminal-use agents, and recent Swarm Signal coverage on harnesses, runtime receipts, and trajectory review Long-Horizon-Terminal-Bench.

Key takeaways

  • Main change: long terminal workflows need dense progress grading, not only final pass/fail.
  • Practical implication: benchmark reports should preserve partial progress, retry count, token spend, wall-clock time, and failure point.
  • Caveat or risk: progress curves can reward busy failure if the subtasks are badly designed.
  • Recommendation: treat terminal-agent claims as workflow measurements, not model victory screens.

It derives 89 terminal tasks from Terminal-Bench 2.1 and tests whether agents can use relevant environmental cues while ignoring plausible distractors TAB.

What This Benchmark Actually Tests

Long-Horizon-Terminal-Bench contains 46 terminal tasks across nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing Long-Horizon-Terminal-Bench. It is still close to agent design rather than abstract model ranking because each task runs through a terminal-style environment and grades intermediate subtasks.

The numbers are the point. The paper reports an average of 9.9 million tokens per task, roughly 231 episodes, and 85.3 minutes of execution time per run across 15 frontier models Long-Horizon-Terminal-Bench. That is a different measurement object from a short coding patch, a browser click path, or a static reasoning prompt.

Terminal-Bench 2.0 already pushed agents into realistic command-line workflows with 89 tasks and comprehensive tests for verification Terminal-Bench 2.0. As checked on 26 July 2026, the public Terminal-Bench site presents Terminal-Bench 2.1, Terminal-Bench 2.0, Terminal-Bench 1.0, science work in development, and long-running single-task challenges Terminal-Bench.

Inference: terminal-agent evaluation is moving from "did it finish?" toward "where did the run stall, how much work did it consume, and was the intermediate state real?"

Final Scores Hide The Useful Failure

Swarm Signal's trajectory-review piece argued that pass bits hide bad intermediate behaviour. The harness piece argued that benchmark, harness, environment, and scorer must be versioned separately. Yesterday's runtime-receipts piece added the product-runtime layer.

Long-Horizon-Terminal-Bench makes the same case with time and cost pressure Long-Horizon-Terminal-Bench. If an agent spends millions of tokens and more than an hour before failing, the final zero is too coarse. For a production evaluation, operators need to know whether it failed at dependency setup, state recovery, instruction selection, verification, or the last integration step.

TAB adds a second warning. It derives 89 terminal tasks from Terminal-Bench 2.1 and tests whether agents can use relevant environmental cues while ignoring plausible distractors TAB. The authors report a gap between task completion and task alignment, and note that prompt-injection defences can suppress useful cues as well as distractors TAB.

That matters because a terminal is full of instructions: README files, stack traces, test output, comments, shell history, and generated artefacts. A model can look capable while following everything. It can also look secure while ignoring the clue needed to finish.

Put those results beside Long-Horizon-Terminal-Bench and the operating requirement is clear.

What Transfers To Production

The production lesson is not that Long-Horizon-Terminal-Bench predicts every engineering job. It is that terminal agents need a progress ledger.

TUA-Bench broadens the frame beyond coding. It introduces 120 terminal-use tasks across five task families, including routine digital activities, live-web information seeking, and scientific or engineering workflows TUA-Bench. Its abstract reports 65.8% overall performance for the strongest frontier agent, with gaps across both tracks TUA-Bench.

Put those results beside Long-Horizon-Terminal-Bench and the operating requirement is clear. A useful terminal-agent run should record the task spec, environment image, allowed tools, network policy, cue provenance, command history, file diffs, evaluator version, token spend, wall-clock time, partial-reward curve, and final verifier output.

Without that, the buyer gets theatre. With it, the team can see whether the agent is improving at planning, recovery, verification, or just spending more tokens before failing.

The Counterargument

Dense grading can be gamed. If the intermediate subtasks are weak, an agent can collect partial credit while making a mess of the actual job. That is my inference from the benchmark design tradeoff: the partial-reward target can become the thing a system optimises.

That objection is fair. It is also an argument for better progress curves, not for final-score nostalgia. The partial rewards should be tied to durable environment state, hidden verification, and fake-proof artefacts, which is the direction Long-Horizon-Terminal-Bench takes with reference solutions or simulation engines and fine-grained graded subtasks Long-Horizon-Terminal-Bench. A victory screen is easier to game.

Operator takeaway

If you are evaluating terminal agents, stop asking only whether the final verifier passed.

One practical action: store progress checkpoints, command logs, file-state hashes, token cost, runtime, and failure labels beside each benchmark row, following the paper's focus on intermediate rewards, execution time, and error patterns Long-Horizon-Terminal-Bench.

One thing to measure: the first irreversible point where the run leaves the intended path.

One thing to avoid: treating a terminal-agent leaderboard as a model comparison when the result is really a workflow, harness, and runtime measurement.

Source trail

Research:

Industry and documentation:

Related Swarm Signal analysis: