Signal Benchmark Watch

Multimodal Agents Score 40% Where Humans Score 72%

Every frontier lab now ships models that see, hear, and read. The assumption is that more modalities mean more capable agents. The benchmarks tell a...

By Tyler · May 23, 2026 · 6 min read

Evidence trail: source links, evidence base, and editorial method appear below. Editorial standards.

Key finding

Every frontier lab now ships models that see, hear, and read. The assumption is that more modalities mean more capable agents. The benchmarks tell a...

Why it matters

Use this section to judge execution impact before implementation.

Evidence base

Claims are grounded in cited papers, benchmarks, and implementation observations where available.

Operator takeaway

Pair this with an execution review of your current monitoring, rollback, and eval loops.

Where this breaks

Assumptions become fragile when upstream systems or data distributions shift.

Use this if

You are standardising AI operations with explicit reliability constraints.

Avoid this if

The failure tolerance is low and you need defensive controls first.

{"version":"0.3.1","atoms":[],"cards":[["html",{"html":"<div style="background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%); border-radius: 12px; padding: 20px; margin: 20px 0; text-align: center;"><p style="color: #e94560; font-weight: bold; margin: 0 0 12px 0; font-size: 14px; letter-spacing: 2px;">LISTEN TO THIS ARTICLE

<audio controls="" preload="none" style="width: 100%; max-width: 500px;" src="https://swarmsignal.net/audio/multimodal-agents-score-40-where-humans-score-72.mp3\">Your browser does not support the audio element.

"}],["hr",{}],["image",{"src":"https://swarmsignal.net/content/images/2026/06/quote_multimodal-agents-score-40-where-humans-score-72_01.webp","alt":"Every frontier lab now ships models that see, hear, and read."}],["image",{"src":"https://swarmsignal.net/content/images/2026/06/quote_multimodal-agents-score-40-where-humans-score-72_02.webp","alt":"The assumption is that more modalities mean more capable agents."}]],"markups":[["a",["href","https://arxiv.org/html/2601.01743v1"]],["a",["href","https://o-mega.ai/articles/the-2025-2026-guide-to-ai-computer-use-benchmarks-and-top-ai-agents"]],["a",["href","https://arxiv.org/html/2601.12560v1"]],["a",["href","https://link.springer.com/article/10.1007/s44267-025-00093-y"]],["a",["href","https://www.bentoml.com/blog/multimodal-ai-a-guide-to-open-source-vision-language-models"]],["a",["href","https://www.microsoft.com/en-us/dynamics-365/blog/it-professional/2026/02/04/multimodal-agent-score/"]],["a",["href","https://arxiv.org/abs/2509.10769"]],["a",["href","https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026"]],["a",["href","https://machinelearningmastery.com/7-agentic-ai-trends-to-watch-in-2026/"]],["a",["href","https://www.rootsanalysis.com/key-insights/top-ai-agent-companies-in-2026.html"]],["a",["href","https://www.technologyreview.com/2026/01/05/1130662/whats-next-for-ai-in-2026/"]],["strong"],["a",["href","https://swarmsignal.net/when-models-see-and-speak/"]],["a",["href","https://swarmsignal.net/robots-with-reasoning/"]],["a",["href","https://swarmsignal.net/scaling-laws-explained-for-practitioners-what-actually-matte/"]],["a",["href","https://swarmsignal.net/agent-reliability-scores-getting-worse/"]],["a",["href","https://swarmsignal.net/mcp-server-architecture-guide/"]],["a",["href","https://swarmsignal.net/best-ai-red-teaming-tools-2026/"]],["a",["href","https://swarmsignal.net/agent-tool-use-patterns-guide/"]],["a",["href","https://swarmsignal.net/#/portal/signup"]],["a",["href","https://payhip.com/b/oq1HI?utm_source=swarmsignal&utm_medium=article_footer&utm_campaign=ss15"]],["a",["href","https://swarmsignal.net/ai-safety-evals-guardrails/"]],["a",["href","https://swarmsignal.net/models-frontiers/"]]],"sections":[[10,0],[1,"p",[[0,[],0,"Every frontier lab now ships models that see, hear, and read. The assumption is that more modalities mean more capable agents. The benchmarks tell a different story."]]],[1,"p",[[0,[],0,"On "],[0,[0],1,"OSWorld"],[0,[],0,", the most realistic test of agents operating real computer environments, the best multimodal agents complete roughly 40% of tasks. Humans score 72% on the same tasks. That gap isn't closing as fast as the marketing suggests."]]],[10,2],[10,3],[1,"h2",[[0,[],0,"Where the Eyes Fail"]]],[1,"p",[[0,[],0,"The bottleneck isn't understanding. It's grounding -- the act of mapping visual perception to precise action."]]],[1,"p",[[0,[],0,"GPT-4V connected to a GUI "],[0,[1],1,"clicks the wrong button"],[0,[],0," because it misidentifies an icon. It misses pop-up dialogs. It confuses a loading spinner for a static element. These are not reasoning failures. They're perception-to-action failures, and they compound across multi-step tasks. A "],[0,[2],1,"January 2026 survey of agentic architectures"],[0,[],0," found that nondeterminism in long-horizon tasks makes "],[0,[21],1,"evaluation"],[0,[],0," itself unreliable, because agents fail differently each time they run."]]],[1,"p",[[0,[],0,"Text-only agents that parse DOM and HTML directly sidestep the problem entirely. They trade generality for reliability. For structured web tasks, "],[0,[3],1,"a text agent reading the page source often outperforms"],[0,[],0," a multimodal agent looking at the screenshot."]]],[1,"h2",[[0,[],0,"The Open-Weight Surprise"]]],[1,"p",[[0,[],0,"One development worth watching: Qwen3-VL-235B, an open-weight model from Alibaba, now "],[0,[4],1,"matches or surpasses GPT-5 and Gemini-2.5-Pro"],[0,[],0," on multimodal benchmarks. The smaller Qwen2.5-Omni-7B is already deployed in BMW vehicles for in-car assistance. Open-weight multimodal models reaching parity with proprietary ones happened faster than most forecasts predicted."]]],[1,"p",[[0,[],0,"This matters because multimodal agent deployment has been gated partly by cost. Running vision models at "],[0,[22],1,"inference"],[0,[],0," is expensive. Open-weight alternatives that teams can host and optimize internally change the economics."]]],[1,"h2",[[0,[],0,"The Evaluation Problem Nobody Solved"]]],[1,"p",[[0,[],0,"Microsoft published a formal "],[0,[5],1,"Multimodal Agent Score"],[0,[],0," standard in February 2026. The fact that a standardized evaluation metric needed to be invented this late tells you how immature the field remains. AgentArch, an "],[0,[6],1,"enterprise-focused benchmark"],[0,[],0,", found that few existing evaluation suites target real enterprise workflows at all. Most benchmarks test toy tasks."]]],[1,"p",[[0,[],0,"The "],[0,[7],1,"International AI Safety Report"],[0,[],0," flagged a related concern: hallucination rates of 20-30% in language models persist and compound in multimodal settings where inputs are noisy or ambiguous."]]],[1,"p",[[0,[],0,"Meanwhile, the production numbers stay sobering. Gartner projects "],[0,[8],1,"40% of enterprise apps will embed AI agents by end of 2026"],[0,[],0,", up from less than 5% in 2025. But fewer than one in four organizations that experiment with agents have "],[0,[9],1,"successfully scaled them to production"],[0,[],0,". Adding vision doesn't simplify that transition."]]],[1,"h2",[[0,[],0,"What This Changes"]]],[1,"p",[[0,[],0,"The multimodal agent story for 2026 isn't "agents can now see." They could see in 2025. The story is that seeing and acting remain fundamentally different capabilities, and the gap between them is wider than most product roadmaps acknowledge."]]],[1,"p",[[0,[],0,"For practitioners, the calculus is straightforward. If your agent operates in a structured digital environment with accessible DOM or APIs, a text-only agent will be more reliable and cheaper. Multimodal perception is worth the cost when the input is genuinely unstructured: scanned documents, physical environments, interfaces without programmatic access."]]],[1,"p",[[0,[],0,"The "],[0,[10],1,"Agentic AI Foundation"],[0,[],0,", formed under the Linux Foundation in late 2025, is standardizing agent infrastructure through MCP and related protocols. But infrastructure for how agents communicate doesn't solve the harder question of whether they can reliably act on what they see."]]],[1,"p",[[0,[],0,"Forty percent task completion is progress. It's also an honest number in a field that doesn't produce enough of them."]]],[10,1],[1,"p",[[0,[11],1,"Related reading:"]]],[3,"ul",[[[0,[12],1,"When Models See and Speak: The Multimodal Agent Arrives"]],[[0,[13],1,"Robots With Reasoning: When Language Models Meet the Physical World"]],[[0,[14],1,"Scaling Laws Explained for Practitioners"]],[[0,[15],1,"Agent Reliability Scores Are Getting Worse, Not Better"]]]],[1,"h3",[[0,[],0,"Keep reading"]]],[3,"ul",[[[0,[16],1,"MCP Server Architecture Guide"]],[[0,[17],1,"Best AI Red-Teaming Tools (2026)"]],[[0,[18],1,"Agent Tool-Use Patterns Guide"]]]],[1,"p",[[0,[19],1,"Join the Swarm Signal newsletter"]]],[1,"p",[[0,[20],1,"Get the Freelance Command Center on Payhip"]]]]}

AgentArch, an " , 0, 6 ,1,"enterprise-focused benchmark" , 0, ,0,", found that few existing evaluation suites target real enterprise workflows at all.

External tools

Execution tooling is separate

Swarm Signal keeps the analysis layer. Use BoredTools for templates, checklists, and execution tools.

Open BoredTools Open Budget Tracker

Swarm Signal

Up Next

Queue is empty. Click "+ Queue" on any article to add it.