LISTEN TO THIS ARTICLE

Evidence base: ToolMenuBench, OSGuard, WorkBench Revisited, Agents' Last Exam, OpenAI function-calling documentation, Anthropic tool-use documentation, and the Model Context Protocol tools specification. Full source links are in the Source trail.

Agent teams often treat the tool menu as plumbing. The model gets a catalogue of available actions, then the real evaluation starts: did it pick the right tool, pass the right arguments, and finish the task?

That framing is too late. The tool menu itself changes the task.

Key takeaways

  • Tool exposure is now an evaluation variable, not a fixed implementation detail.
  • Smaller visible tool sets can improve reliability when the filter preserves the causal path needed for the task.
  • Safety checks that judge isolated actions can miss end-to-end failures caused by over-broad tool access.
  • The operator lesson is simple: test which tools are visible, not only whether the model can call them.

What This Benchmark Actually Tests

The useful signal in ToolMenuBench is its target. The benchmark does not ask only whether an agent can call a tool correctly; it asks how the visible menu shapes task success, wrong-tool calls, premature actions, token usage, and risky-tool exposure ToolMenuBench.

That is a better match for production than a clean function-calling test. Real agents rarely see one obvious tool. They see a crowded interface: read tools, write tools, destructive tools, admin tools, overlapping tools, and stale tools left around because someone thought the model might need them later.

A tool menu is not a list of capabilities. It is a runtime policy surface.

The headline number is large, but it should be read carefully. In the authors' controlled evaluation, causal minimal tool filtering improved task success from 32.1% under all-tools exposure to 85.7%, while reducing average token usage by roughly 98% ToolMenuBench. That is a benchmark result, not a universal production law, because the paper reports a controlled evaluation rather than live deployment data ToolMenuBench. It does show that the default "show the model every tool" pattern deserves more suspicion.

They do not solve the upstream question of which tools should be present.

Why Tool Menus Fail

Tool menus fail because they combine three risks that teams usually evaluate separately; that is an inference from ToolMenuBench's separate reporting of visible-tool count, wrong-tool calls, premature actions, token usage, and risky-tool exposure ToolMenuBench.

First, they increase choice pressure. More visible tools means more room for near-miss selection: the tool that sounds right, has the right noun in its description, but changes the wrong object. The existing agent tool-use patterns guide covers selection, parameter construction, execution, result interpretation, and chaining as separate stages. ToolMenuBench puts pressure on the stage before all of them: which actions should be visible at all?

Second, they increase risk exposure. If an agent can see a destructive write tool during a read-only task, the system has already widened the blast radius. The model may never call that tool. In the ToolMenuBench paper submitted on 13 June 2026, risky-tool exposure is treated as a measured downstream metric, not a cosmetic property ToolMenuBench.

Third, they inflate context. Tool descriptions, schemas, caveats, and examples consume tokens before the agent has started doing useful work. OpenAI's function-calling documentation recommends strict schema mode for reliable function-call shape, while Anthropic's tool-use documentation frames tool execution as an application-controlled loop OpenAI function calling, Anthropic tool use. Those controls help with call shape and execution boundaries. They do not solve the upstream question of which tools should be present.

What Does And Does Not Transfer To Production

What does not transfer to production is the exact 32.1% to 85.7% result, because that number belongs to ToolMenuBench's controlled setup ToolMenuBench.

What does transfer is the test design. If your agent has more tools than it needs for a task, evaluate the menu as part of the system. Run the same workflow with all tools visible, a simple keyword filter, a state-aware filter, and a strict allowlist. Then compare success, wrong-tool calls, risky-tool exposure, token use, and premature actions, which are the categories ToolMenuBench makes visible ToolMenuBench.

OSGuard makes the safety issue clearer. It argues that computer-use agents can complete realistic desktop and web tasks while reaching the nominal goal through unsafe shortcuts OSGuard. Its benchmark separates action-level guardrail decisions from risk-augmented end-to-end execution, because the authors designed OSGuard to diagnose both isolated unsafe-action recognition and full-task safety OSGuard.

That maps directly onto tool menus. A guardrail can block one proposed dangerous call. A better menu prevents the dangerous call from being in scope for that task. Those are not the same control. This is the same reason production agent evals need to inspect the path, not only the final answer.

WorkBench Revisited points in the same direction from a different angle. The June 2026 update reports major progress on workplace-agent task completion, from 43% for GPT-4 in March 2024 to 89% for the best agent in the authors' June 2026 revisit, while unintended harmful actions fell from 26% to 2.5% in that benchmark setting WorkBench Revisited. That is genuine progress. It also leaves an uncomfortable remainder: rare harmful actions still matter when the action is irreversible.

The design question is not "can the model usually avoid the wrong tool?" It is "why was the wrong tool visible for this job?"

A tool that is valid after approval may be invalid before approval.

What Operators Should Test

Start with three tool-menu fixtures.

First, run the same tasks with all tools visible, a lexical filter, a state-aware filter, and a task-specific allowlist. Measure task success, wrong-tool calls, token use, and whether any destructive tool was exposed during a non-destructive task.

Second, test distractors. Add tools with similar names and overlapping descriptions. If the agent calls the wrong one, do not only tune the prompt. Treat the menu as a failing interface.

Third, test temporal scope. A tool that is valid after approval may be invalid before approval. A tool that is safe after a search result may be unsafe before the agent has identified the target object. The Model Context Protocol tools specification says tools expose named capabilities with descriptions and input schemas MCP tools specification. That is necessary, but it is not enough. The menu also needs state.

Agents' Last Exam gives the broader reason to care. Its authors argue that agent benchmarks need sustained measurement on economically valuable workflows, not only isolated task slices; the benchmark covers more than 1,000 tasks across 55 subfields and 13 industry clusters, with the hardest tier below 1% average full pass rate across mainstream harness and backbone configurations in the paper's reported results Agents' Last Exam. If agents are still far from dependable on hard workflows, then avoidable interface risk is not a detail; that is my inference from the paper's reported hardest-tier result Agents' Last Exam.

The Counterargument

Over-filtering can also break agents. If the menu hides a tool the task genuinely needs, the model may fail, ask for help, or improvise around the missing capability. Tool filtering is not a virtue by itself.

That is why causal filtering matters more than small menus. The goal is not fewer tools. The goal is the smallest visible set that preserves the path from user intent to safe completion.

In production, this probably means dynamic menus: read-only by default, write tools unlocked by state, destructive tools gated by approval, admin tools isolated by role, and tool descriptions treated as eval targets rather than copywriting.

Operator takeaway

Stop treating the tool list as neutral context.

For builders, the immediate move is practical: add a tool-menu axis to your agent evals. Run the same task with different visible tool sets. Track wrong-tool calls, premature actions, risky-tool exposure, token use, and completion, matching the metric categories ToolMenuBench reports ToolMenuBench. My recommendation from that evidence is to keep the menu that improves the task without widening the blast radius ToolMenuBench.

One thing to avoid: giving the model every tool because the prompt says "only use what you need."

That is not an interface. It is hope with JSON attached.

Source trail

Research papers

Implementation documentation

Related Swarm Signal analysis