LISTEN TO THIS ARTICLE

The paper When Better Turns Do Not Make Better Agents tests a quiet failure in agent evaluation: a model can improve at predicting the next local action while still failing the full customer-support workflow when it has to run autonomously Laskar et al.. That matters for teams buying or building support agents, coding agents and tool-using systems, because many evaluation reports still mix local step quality with end-to-end task completion.

Evidence base: the primary workflow-evaluation paper, BenchAgent's protocol-aligned comparison of agent workflows, a recent synthesis of tool-use and planning failures, and related Swarm Signal coverage of benchmark drift and evaluation cost Laskar et al..

Key takeaways

  • Local step scores are not enough: the paper reports that supervised fine-tuning improved text-turn and next-turn success under gold-history evaluation, but the fine-tuned models still failed under holistic workflow evaluation Laskar et al..
  • The strict trajectory-completion result stayed very low in the tested customer-support workflows, so a better turn predictor did not become a reliable autonomous operator Laskar et al..
  • Evaluation reports should split text quality, local action correctness, tool execution and final workflow completion instead of compressing them into one agent score Laskar et al..

The strongest practical lesson is that local competence can be real and still insufficient.

What next-turn scoring hides

Gold-history next-turn evaluation gives the model the correct interaction history, then asks it to choose or produce the next move. That is useful when a team wants to know whether the model recognises a good response, a plausible tool call or a locally correct handoff. It is weaker when the production system has to live with its own earlier mistakes.

The new paper makes that distinction explicit. It studies pre-SFT and supervised fine-tuned language models on multi-turn customer-support workflows, then compares local gold-history results with autonomous execution Laskar et al.. The local protocol says whether the model can continue from a clean reference state. The autonomous protocol asks whether the model can preserve state, recover from its own errors and complete the workflow without the benchmark quietly correcting the path Laskar et al..

That is a different kind of evidence from a static leaderboard. Swarm Signal has already covered Messier's attempt to normalise benchmark drift and mid-range task selection for cheaper agent evaluation. This paper adds another constraint: the metric itself has to match the operating unit.

Workflow success is a separate outcome

The strongest practical lesson is that local competence can be real and still insufficient. The authors report that supervised fine-tuning consistently improved text-turn success and increased overall next-turn success for every model under gold-history evaluation Laskar et al.. In the holistic workflow setting, however, the same fine-tuned systems did not complete strict trajectories reliably Laskar et al..

For operators, this argues against a familiar shortcut: testing a support agent by sampling isolated messages, then assuming the deployed agent will manage the whole case. A customer-support workflow has branching state, tool calls, policy constraints, previous commitments and escalation decisions. If the benchmark repairs the state before every step, it is no longer testing whether the agent can keep the case coherent.

The synthesis Beyond the Leaderboard reaches the same conclusion from a wider literature review. It groups failures across tool use, planning, long-horizon reasoning, multi-agent coordination, safety and measurement validity, and warns that strong performance on individual sub-tasks does not reliably transfer to end-to-end success Albayaydh et al.. The local result is not fake; it is answering a narrower question.

BenchAgent points in this direction for multi-agent and workflow comparisons.

What to ask before trusting an agent score

The buying question should move from "what is the benchmark score?" to "what execution state was the model responsible for?" A useful evaluation report should answer a small set of questions:

  • Did the model run from a clean reference history or from its own prior outputs?
  • Were tool calls executed, mocked or judged as text?
  • Was success scored at each turn, at final answer time or across the full trajectory?
  • Were recoveries, escalations and invalid tool arguments counted separately?

BenchAgent points in this direction for multi-agent and workflow comparisons. It normalises benchmark loading, tool access, answer contracts, usage accounting and trajectory logging so workflow changes can be compared without mixing them with protocol differences Fu et al.. Its results also caution against treating extra agents as automatic lift: under a substrate-internal comparison, most tested multi-agent workflows trailed the matched single-agent anchor Fu et al..

Together, the papers suggest a plain rule. If the product risk is the whole workflow, the evaluation unit has to be the whole workflow. Local turn metrics can diagnose where a system fails, but they should not be the release gate for a system that can issue refunds, file tickets, edit records or close support cases.

How to adapt a release test

Teams do not need to discard next-turn tests. They need to stop using them alone. A better release pack for a tool-using support agent should include:

  • A local next-turn set for response quality and obvious policy mistakes.
  • An autonomous trajectory set where the agent carries its own state.
  • A tool-execution report that separates valid calls, bad arguments, failed calls and recovery attempts.
  • A final workflow-completion score with strict and partial outcomes.
  • A regression set built from real failure modes, not only clean happy paths.

This also changes how fine-tuning claims should be read. Fine-tuning that improves local action prediction may still be valuable if it reduces a known class of bad turns. It should not be sold as workflow reliability unless the autonomous trajectories improve as well. The same caution applies to orchestration changes, memory layers and multi-agent wrappers: the evidence has to show better task completion, not just cleaner intermediate moves.

For teams with limited evaluation budgets, start with realistic full trajectories rather than a large pile of isolated turns. Use local metrics to explain the failures after the trajectory run. That order keeps the benchmark aligned with the operational question: can the agent finish the work, and can the operator see why it failed when it does not?

Source trail

Research and technical sources:

Related Swarm Signal analysis: