LISTEN TO THIS ARTICLE

Multilingual Agents Need Workflow Tests, Not Translation Scores

PolyWorkBench landed on arXiv on 7 July 2026 with a useful correction to enterprise-agent hype: a global workflow is not a translated English task PolyWorkBench.

Evidence base: PolyWorkBench, MAPS, GAIA-v2-LILT, TelcoAgent-Bench, WorkBench, and current localisation-industry coverage of multilingual agent evaluation PolyWorkBench.

Key takeaways

  • Main change: multilingual agent evaluation is moving from translated prompts to full workplace trajectories.
  • Practical implication: teams need tests where instructions, source files, tool outputs, and final artefacts can use different languages.
  • Caveat or risk: some measured language gaps are benchmark-design errors, not only model failures.
  • Recommendation: log language role, tool action, evaluator, and output consistency for every production workflow.

The more useful result is that the same underlying model can move by 8 to 21 Pass@1 points when the harness changes PolyWorkBench.

What This Benchmark Actually Tests

As described in its July 2026 paper, PolyWorkBench contains 67 long-horizon workplace tasks across commerce, knowledge work, legal analysis, localisation, and manufacturing PolyWorkBench. It covers ten languages and deliberately mixes the language of the instruction, source material, and expected output PolyWorkBench.

That matters for real-world AI deployment because enterprise work rarely stays inside one clean language lane. A procurement agent may read Japanese invoices, compare German contract clauses, use an English ERP tool, and return a Spanish summary. A benchmark that only translates the prompt misses the workflow risk that PolyWorkBench is designed to expose PolyWorkBench.

The paper's strongest design choice is the workflow shape. Tasks require documents, spreadsheets, logs, contracts, or multilingual resources, then grade structured outputs through structural checks, executable verification, and semantic assessment PolyWorkBench.

Inference: the test target is not language fluency. It is whether language switching breaks planning, tool use, and evidence assembly.

The Signal

The headline number is not a disaster score; it is a sign that some scaffolded frontier agents can complete many tasks while still leaving fragile deployment assumptions visible PolyWorkBench. The best reported entry, Claude Opus 4.8 with ClaudeCode, reaches 0.921 Pass@1, while only three entries exceed 0.79 and most fall below 0.77 PolyWorkBench. The more useful result is that the same underlying model can move by 8 to 21 Pass@1 points when the harness changes PolyWorkBench.

That connects directly to agent eval harnesses and runtime receipts. If the harness changes the result that much, a multilingual-agent claim needs the model, scaffold, language mix, tools, verifier, and trace schema in the receipt.

The benchmark also shows domain-specific fracture PolyWorkBench. As reported in the July 2026 results, strong models can remain around 0.85 to 0.90 on knowledge, legal, and manufacturing tasks while falling to 0.50 to 0.65 on commerce tasks PolyWorkBench. One global average hides which part of the business process is breaking.

Translation Is Not The Whole Test

MAPS makes the broader safety case. It extends four agentic benchmarks into eleven languages, producing 805 unique tasks and 9,660 language-specific instances MAPS. Its authors report degradation in both performance and security when moving from English to other languages, with severity tied to translated input share MAPS.

GAIA-v2-LILT adds the warning that benchmark construction itself can create false gaps GAIA-v2-LILT. Its audited multilingual GAIA workflow improved agent success rates by up to 32.7% over minimally translated versions, with the closest audited setting within 3.1% of English performance GAIA-v2-LILT.

That does not weaken the multilingual-agent problem. It sharpens it. If a test uses poor translation, it can punish the wrong thing. If it uses only fluent translation, it can still miss process failure. The production test has to preserve functional alignment, cultural context, tool constraints, and final-output requirements.

TelcoAgent-Bench shows the same pattern in a narrower operational domain. It tests multilingual telecom troubleshooting in English and Arabic and reports that models can understand problems reasonably while struggling to follow ordered troubleshooting steps and remain stable across scenario variations TelcoAgent-Bench.

It asks whether the same workflow still works when the language role changes at each step.

What Transfers To Production

Older workplace-agent benchmarks already showed why final answers are too thin WorkBench. WorkBench used five databases, 26 tools, and 690 workplace tasks, with the best tested ReAct agent completing 43% of tasks and the weakest completing 3% WorkBench.

PolyWorkBench adds the missing global-enterprise dimension. It asks whether the same workflow still works when the language role changes at each step. As reported in the July 2026 PolyWorkBench paper, that mirrors its domain mix across commerce, knowledge work, legal analysis, localisation, and manufacturing PolyWorkBench.

What transfers to production is the failure shape: instrument multilingual handoffs, tool calls, and final artefacts as one trajectory. What does not transfer cleanly is the exact leaderboard score, because a company's language mix, tools, terminology, and review policy will differ from the benchmark design PolyWorkBench.

For production teams, the minimum eval row should include instruction language, source-language set, expected-output language, tool language assumptions, harness version, verifier type, failed step, and whether the final artefact preserved meaning across languages.

That is different from asking a model to translate a document. It is asking whether an agent can keep the business process intact while language becomes part of the environment.

The Counterargument

There is a fair objection: most teams can route multilingual work through specialist translation tools before the agent acts. In some workflows, that is the right architecture.

The risk is pretending that preprocessing removes the evaluation burden MAPS. MAPS and GAIA-v2-LILT both show that translation quality and language role can change agent behaviour, not just wording GAIA-v2-LILT. If the agent reads translated inputs but calls tools whose labels, schemas, or evidence are still language-specific, the system can still drift. If the final artefact must be legally, commercially, or culturally aligned, the translation boundary becomes part of the workflow, not a warm-up step GAIA-v2-LILT.

Operator takeaway

If you deploy agents across regions, test the workflow in the languages where work actually happens.

One practical action: add a multilingual trajectory lane to agent evaluation. Start with one workflow where instructions, source artefacts, and final output use different languages, then store tool calls, verifier outputs, language transitions, and final artefact checks.

One thing to avoid: approving a global agent from English-only workplace scores plus a separate translation benchmark, because PolyWorkBench and MAPS both test multilingual effects inside agent trajectories PolyWorkBench.

Source trail

Research:

Industry and commentary:

Related Swarm Signal analysis: