LISTEN TO THIS ARTICLE

Coding Agents Need Trajectory Reviews, Not Pass Bits

Most coding-agent benchmarks still compress a whole run into one bit: did the task pass? AgentLens argues that users experience the whole trajectory instead: instruction following, tool use, verification, recovery, and communication AgentLens.

Evidence base: AgentLens, Beyond the Leaderboard's synthesis of 27 agent benchmark and audit papers across 19 benchmarks, OpenAI agent-evaluation guidance, OpenAI Agents SDK tracing documentation, and LangChain's production-monitoring guidance AgentLens.

Key takeaways

  • Main change: coding-agent evaluation is moving from binary pass/fail checks toward reviewed execution trajectories.
  • Practical implication: teams need to score how an agent worked, not just whether one verifier passed.
  • Caveat or risk: trajectory review can become another subjective judge unless it is tied to concrete trace fields.
  • Recommendation: keep pass/fail verification, then add trajectory labels for tool use, recovery, and user-facing behaviour.

For coding agents, a green verifier is necessary but incomplete.

The signal

AgentLens is useful because it attacks a real blind spot in the models and frontier systems stack. A coding agent can pass a final test after wasting context, ignoring instructions, producing risky edits, or recovering only because the harness caught it AgentLens.

The paper pairs formal verification with LLM-written trajectory reviews and side-by-side comparisons, so each run gets an explanation of why the score looks the way it does AgentLens.

The timing matters. The first AgentLens version landed on 7 July 2026, with a revised version on 14 July 2026 AgentLens. Inference from that timing and design: coding-agent evaluation is borrowing from production QA, not just programming-contest scoring.

What This Benchmark Actually Tests

The important unit is the trajectory. AgentLens asks whether the agent followed instructions, used tools sensibly, verified its work, recovered from mistakes, and communicated acceptably while completing an interactive coding task AgentLens.

That is different from Swarm Signal's earlier coding-agent benchmark generalisation coverage. Generalisation asks whether benchmark wins transfer. Trajectory review asks what behaviour produced the win.

Why Pass Bits Hide Too Much

Beyond the Leaderboard makes the broader case. It synthesises 27 benchmark, taxonomy, and audit papers from 2023 to 2026, spanning 19 distinct benchmarks, and groups agent limitations into six clusters: tool invocation errors, planning failures, long-horizon degradation, multi-agent coordination failures, safety and security failures, and measurement-validity problems Beyond the Leaderboard.

That synthesis is a warning against single-number comfort. It reports that strong sub-task performance does not reliably turn into end-to-end success, and that failures can compound as task length grows Beyond the Leaderboard.

For coding agents, a green verifier is necessary but incomplete. The benchmark should preserve inspected files, commands run, failed tests noticed, state invented, and handoff quality.

The fair objection is that trajectory review can smuggle taste into evaluation.

Production relevance

This is already how production agent tooling is drifting. OpenAI's agent-evaluation guidance says trace grading captures an end-to-end record of model calls, tool calls, guardrails, and handoffs, then lets graders score traces against structured criteria OpenAI agent evals.

The OpenAI Agents SDK documentation says tracing records LLM generations, tool calls, handoffs, guardrails, and custom events during a run OpenAI Agents SDK tracing. LangChain's production-monitoring guidance makes the same operational point: agent monitoring has to inspect conversations, tool paths, quality, cost, latency, and failure patterns LangChain.

Inference: the serious benchmark is becoming a replayable trace plus a judgement layer. That judgement can be formal, human, LLM-assisted, or mixed. What matters is that it sees the steps the user saw.

The counterargument

The fair objection is that trajectory review can smuggle taste into evaluation. One reviewer may value terse communication; another may value verbose explanation.

That is real. The answer is not to remove trajectory review. It is to make the rubric explicit. Score instruction adherence, tool discipline, verification, recovery, and handoff separately. Keep the binary task result beside those labels so nobody mistakes politeness for correctness.

Operator takeaway

If you operate coding agents, keep the pass bit, then add a trajectory review before comparing models.

One practical action: sample failed and passed runs, then label instruction following, tool use, verification, recovery, and handoff quality.

One thing to measure: percentage of green runs that still required unsafe retries, ignored instructions, or unclear user handoff.

One thing to avoid: promoting a coding agent because it passes more tasks while making worse intermediate decisions.

Source trail

Research:

Industry and documentation:

Related Swarm Signal analysis: