LISTEN TO THIS ARTICLE

The Beyond Agent Harnesses paper names a production failure that ordinary workspace inspection misses: an agent can see the same files and memory while the real authority to publish, spend, approve or mutate has changed elsewhere Li et al.. That matters because many release checks still ask whether the planner had enough context, not whether the final action was still authorised at the execution boundary.

Evidence base: the primary cross-substrate authority paper, a September 2026 review of tool-agent authorisation architectures, the Agent Action Decision Protocol Internet-Draft, MasDrift's authorisation-preservation benchmark, and related Swarm Signal coverage of delegated authorisation and covert tool channels Li et al..

Key takeaways

  • Main result: in the paper's first controlled experiment, authority-blind evidence achieved 0/32 final semantic successes, while raw receipts and a typed authority relation each reached 32/32 Li et al..
  • Planning context is not enough: in a second experiment with 96 planning calls, workspace-visible evidence still produced 12/16 unsafe publication decisions Li et al..
  • Practical control: a deterministic execution guard blocked all six unsafe first-action intents in the paper's replay experiment while permitting all 12 valid authorised publish intents Li et al..
  • Caveat: the paper uses controlled mini-benchmarks, so it supports a release-test pattern rather than a universal failure rate Li et al..

More summaries, richer traces or clearer handoff notes help only if they include the live authority fact.

What This Benchmark Actually Tests

The paper studies cases where workspace evidence and model-visible memory are insufficient because the decisive authority fact is stored in a runtime, registry or approval service Li et al.. Its experiments compare planner-observation fixes with execution-time checks, using Git lineage, recorded execution attempts, deterministic oracles and two model routes Li et al..

The authority gap is outside the workspace

The paper's core claim is simple: decision-relevant authorisation can live outside both the agent's model-visible memory and its current files Li et al.. A repository, ticket, document or draft can be identical while the safe next action changes because an approval expired, a previous attempt already ran, a budget was consumed, or a human revoked the route.

That is the cross-substrate authority gap. The substrate that carries the plan is not the same substrate that carries permission. If the agent only reads the workspace, it can make a locally coherent decision that is globally unsafe.

Swarm Signal has covered nearby failures in MasDrift's delegation tests and tool-agent covert channels. The new paper moves the question from "did delegation preserve the user's boundary?" to "where is the boundary stored at the moment the action becomes real?"

More evidence does not fix a missing fact

The first experiment is useful because it separates packaging from substance. The authors test authority-blind candidate evidence, raw receipts and a typed relation across a 128-cell controlled evidence ablation Li et al.. Authority-blind evidence fails completely on final semantic success, while both receipt-based conditions succeed Li et al..

That finding should make teams wary of context-heavy fixes. More summaries, richer traces or clearer handoff notes help only if they include the live authority fact. If the important state sits in a scheduler, approval system, payment ledger, incident queue or prior execution log, the planner cannot infer it reliably from files alone.

The September 2026 review Authorization Architectures for Tool-Using AI Agents frames the same issue at a system level. It argues that consequential actions should be traceable to a human principal, bounded by actual delegation and contestable after the fact, then identifies runtime enforcement and aggregation bounds as unresolved gaps Surapani et al.. The cross-substrate paper gives that gap a concrete test.

A model can choose an apparently sensible action from stale or incomplete state.

Planner approval is weaker than execution approval

The second experiment shows why the check has to sit late. With workspace-visible evidence, planners still made 12 unsafe publication decisions out of 16 tested cases Li et al.. Even when the authority relation was made available, first actions were unreliable: 15/32 were correct, while 11/32 were invalid or absent Li et al..

For operators, the lesson is not that planning is useless. It is that planning cannot be the last authorisation layer for consequential tools. A model can choose an apparently sensible action from stale or incomplete state. The mutation boundary can still ask a narrower question: is this exact action, with these exact arguments, authorised now?

The Agent Action Decision Protocol draft takes a similar direction by separating per-action authorisation from an agent's identity and standing capabilities, including mutable state such as budgets, reservations, approval lifecycle and kill switches AADP draft. It is an individual draft rather than an adopted standard, but the design pressure is clear: authority belongs with the proposed action, not only with the agent account.

Runtime guards make stale plans harmless

The paper's third experiment is the cleanest operational result. The authors replay the same 32 model-generated first-action intents with no additional model calls, then add a deterministic execution guard Li et al.. The guard prevents every unsafe intent from becoming an effect and permits every valid authorised publish intent in that replay Li et al..

That is the useful design distinction. A planner may still propose the wrong action. The system becomes safer when an execution guard can reject it without asking the planner to reason again. This also reduces duplicate-action risk: if a previous attempt already committed the action, the authority check can return "already used", "expired" or "requires fresh approval" before the tool runs AADP draft.

MasDrift points to the same boundary from another angle. It found that re-anchoring pending calls to the original user request reduced unauthorised actions across evaluated model configurations, with a 1.6-point pooled completion cost, while carrying attenuated policy through the delegation chain could cost up to 36.3 points of completion MasDrift. Source-anchored checks beat hoping that intent survives every handoff.

What to change in release tests

Teams evaluating tool-using systems should add at least one cross-substrate test beside ordinary task completion:

  • Give the agent a workspace snapshot where two runs look identical from files and memory.
  • Put the decisive authority fact in another system, such as an approval record, prior execution receipt, budget counter or scheduler row.
  • Ask the agent to plan the next action, but require the tool wrapper to decide again at execution time.
  • Score planner correctness separately from mutation prevention Li et al..
  • Treat duplicate prevention, expired approvals and revoked routes as first-class outcomes.

This is not just a security test. It is a reliability test for any agent that can publish, charge, email, merge, schedule, buy, delete or approve, because the authorisation decision point is where consequential tool invocation becomes enforceable and accountable Surapani et al.. The safe interface should be able to say: the plan may be reasonable, but this particular action is no longer authorised.

Source trail

Research, technical and standards-track sources:

Related Swarm Signal analysis: