LISTEN TO THIS ARTICLE

SWE-Bench Mobile tests coding tools against industry mobile-development work rather than public Python bug fixes. Its uncomfortable result is that the strongest tested configuration solved only 12% of tasks, and the same model could vary by up to 6x depending on the coding agent wrapped around it SWE-Bench Mobile.

Evidence base as of 14 August 2026: the SWE-Bench Mobile paper, the official benchmark site, the original SWE-bench paper, SWE-bench Verified, and recent Swarm Signal coverage of coding-agent trajectory review and terminal-agent progress measurement SWE-Bench Mobile.

Key takeaways

  • Main result: tested coding agents still struggled with production-shaped mobile tasks SWE-Bench Mobile.
  • Practical implication: buying a coding agent by model leaderboard alone is unsafe when the surrounding tool can move the same model by up to 6x SWE-Bench Mobile.
  • Caveat: the benchmark is hosted and task details are private, so it is strongest as a contamination-resistant buyer test rather than a fully inspectable open research corpus SWE-Bench Mobile.
  • Release decision: require mobile, design and product-requirement tasks in any evaluation for teams shipping iOS or visually specified product work.

The final pass bit matters, but the route to that bit tells you whether the result is repeatable.

What This Benchmark Actually Tests

SWE-bench made software repair measurable by asking systems to patch real GitHub issues from 12 popular Python repositories, across 2,294 problems SWE-bench. SWE-bench Verified then narrowed that style of evaluation to 500 human-validated tasks, reducing ambiguous or unsolvable cases SWE-bench Verified.

SWE-Bench Mobile changes the job shape. It reports 50 authentic mobile-development tasks, 449 human-verified test cases, 22 agent-model configurations, four coding agents and nine models SWE-Bench Mobile. The benchmark is built from a production iOS codebase, mixes Swift and Objective-C, and gives the system product requirement documents plus Figma designs rather than only issue text SWE-Bench Mobile. The official site frames the task as evaluating coding agents on real-world mobile development work from an industry-level iOS codebase SWE-Bench Mobile.

That matters because mobile work often fails outside the patch hunk. A system has to interpret design intent, locate the right view or controller, respect platform constraints, update assets or gestures, and avoid breaking surrounding behaviour. Text-only issue repair does not reliably measure that whole loop.

Model ranking is the wrong first question

The paper evaluates 22 agent-model configurations across Cursor, Codex, Claude Code and OpenCode, then reports that agent design matters as much as model capability SWE-Bench Mobile. The same underlying model can perform very differently when the tool loop, prompting, file navigation and test execution change.

For buyers, the lesson is blunt: do not ask which model won before asking which coding environment produced the run. A model exposed through a weak workspace loop may miss the target files, underuse the design artefact, over-edit adjacent code or fail to run the decisive tests. A less fashionable model inside a better agent shell may deliver a more usable patch.

This is the same measurement problem Swarm Signal has covered in coding-agent trajectory reviews and terminal-agent progress curves. The final pass bit matters, but the route to that bit tells you whether the result is repeatable.

Product specs are not GitHub issues

SWE-Bench Mobile’s tasks come from real product requirement documents and include Figma designs, acceptance criteria and a large mobile codebase SWE-Bench Mobile About. That is a different cognitive load from reading an issue description and editing a known failing function.

Production inference: teams should not copy the benchmark blindly. They should copy its input realism. If the real ticket starts with a design, the evaluation should include a design. If the real codebase has platform-specific APIs and older Objective-C modules beside Swift code, the evaluation should include that mess SWE-Bench Mobile.

Otherwise the evaluation becomes a comfort test: it may prove that a system can solve a simplified version of the work after the hard parts have been converted into text, not that it can handle the actual product loop.

Use the public result to identify failure shapes: design interpretation, platform APIs, multi-file changes, generated tests, tool-loop variance and prompt sensitivity.

Hosted tests reduce contamination but reduce auditability

The benchmark authors release SWE-Bench Mobile as a hosted benchmark challenge to limit data contamination and preserve fair evaluation SWE-Bench Mobile. The official site says task details are private and invites research collaboration rather than publishing the full task set SWE-Bench Mobile.

That trade-off is defensible, but it changes how teams should read the result. A hosted benchmark can make memorisation harder because the task details are not public, while also making independent inspection harder SWE-Bench Mobile About. The right response is not to dismiss the score, but to avoid treating it as a universal ranking without a local replication layer.

Use the public result to identify failure shapes: design interpretation, platform APIs, multi-file changes, generated tests, tool-loop variance and prompt sensitivity. Then build a private equivalent from your own mobile backlog.

What transfers to production

SWE-Bench Mobile transfers to production as a procurement pattern, not as a direct forecast for every mobile team. The benchmark's useful shape is the combination of real product requirements, design artefacts, platform code, generated patches and hidden hosted evaluation SWE-Bench Mobile.

What transfers: coding-agent evaluations should include the artefacts and constraints the team actually uses. What does not transfer automatically: the exact 12% top success rate, because local codebase structure, test quality, tool access and review rules can move the result SWE-Bench Mobile.

The evaluation to run before procurement

For mobile coding-agent procurement, use a small decision gate:

  • five historical tickets with PRDs, designs and acceptance criteria;
  • one ticket requiring gesture or layout behaviour;
  • one ticket touching networking or persistence;
  • one ticket involving assets or media;
  • a fixed time budget and a fixed allowed-tool policy;
  • trajectory logs showing files read, commands run, tests attempted and failed assumptions.

Score four things separately: functional pass, visual fit, code review burden and operator time. Keep the model, agent shell, prompt, repository state, test set and reviewer notes together. If those records split apart, the comparison becomes a model story when the real variable may be the agent loop SWE-Bench Mobile.

SWE-Bench Mobile does not prove that coding agents are ready for unsupervised mobile feature work. It shows that the wrapper around the model is now part of the product being evaluated. Buying the model without testing the wrapper is no longer a serious evaluation.

Source trail

Research and official sources:

Related Swarm Signal analysis: