LISTEN TO THIS ARTICLE

Agent Evals Need Harnesses, Not More Scoreboards

AgentCompass first landed on arXiv on 15 July 2026 with a practical complaint: agent evaluation is fragmented, tightly coupled, and hard to reproduce AgentCompass.

Evidence base: the AgentCompass paper and repository, OpenAI's agent-evaluation guidance, LangChain's AgentEvals package, IBM Research's survey of agent benchmark methods, and recent Swarm Signal coverage of benchmark cost, trajectory review, and tool-use failure labels AgentCompass.

Key takeaways

  • Main change: agent evaluation is moving from isolated scoreboards toward reusable harness infrastructure.
  • Practical implication: teams need to version benchmark, harness, environment, and trajectory data separately.
  • Caveat or risk: a shared harness can standardise the wrong behaviour if production constraints are missing.
  • Recommendation: compare agents only after the evaluation runtime, tools, retries, and trace schema are fixed.

It is the split between what is being tested and how the test is executed AgentCompass.

The Signal

AgentCompass describes an open-source evaluation infrastructure for LLM-based agents, organised around Benchmark, Harness, and Environment components AgentCompass. Its July 2026 version reports native support for more than 20 benchmarks across tool use, web and research, scientific reasoning, agentic coding, and productivity AgentCompass.

That matters for agent design because many agent results still mix model ability with harness choices. Change the tool loop, retry policy, sandbox, or scorer, and the same model can look meaningfully different.

Swarm Signal has already covered why agent leaderboards can be cheaper without being safer and why coding agents need trajectory reviews. AgentCompass adds the infrastructure angle: if the harness is not reusable, the number is hard to trust.

Scores Depend On The Runtime

The strongest part of AgentCompass is not the leaderboard table. It is the split between what is being tested and how the test is executed AgentCompass. The benchmark defines the task and scorer. The harness defines the agent procedure. The environment supplies the execution boundary.

That sounds like plumbing. It is actually the measurement system.

In the paper's experiments, model performance varies with the employed harness, and the authors cite examples where scores move away from official baselines under the unified protocol AgentCompass. Inference: teams should stop treating a benchmark name as a complete evaluation spec.

The useful question is not "what did the model score on SWE-bench?" It is "which harness, which workspace, which scorer mode, which retry rules, and which trace fields produced that score?"

The same trace that supports evaluation also supports incident review.

Production Evals Need Replay

OpenAI's agent-evaluation guidance frames the production version of this problem around traces, graders, datasets, and evaluation runs OpenAI agent evals. LangChain's AgentEvals project focuses on evaluators for agent trajectories, not just final answers LangChain AgentEvals.

IBM Research made the broader research case from a survey of 120 agent evaluation methods, arguing that agent tests need better realism, interaction measurement, and clearer links to useful behaviour IBM Research.

The pattern is converging. A serious agent eval preserves the run, not just the result. It records tool calls, environment feedback, retries, cost, latency, failure labels, and the scorer. Without that replay layer, a production team cannot tell whether a model improved or the harness got kinder.

This is especially relevant to tool-use agents and agent observability. The same trace that supports evaluation also supports incident review.

The Counterargument

There is a fair objection: unified harnesses can create a new monoculture. If everyone optimises for the same reusable runtime, agents may learn the harness instead of the job.

That risk is real. The answer is not to return to bespoke scripts. It is to make the harness explicit and rotate production-like environments into the suite. A standard runtime should make assumptions visible, not freeze them.

Operator takeaway

If you run agent evaluations, version the harness with the same seriousness as the benchmark.

One practical action: record benchmark spec, harness version, environment image, tool permissions, retry policy, scorer mode, and trace schema for every comparison.

One thing to avoid: buying or promoting an agent from a scoreboard that does not disclose the execution wrapper.

Source trail

Research:

Industry and documentation:

Related Swarm Signal analysis: