🎧 LISTEN TO THIS ARTICLE
Every agent benchmark makes the same assumption: the world holds still while you test. Fixed tool APIs. Static database schemas. Frozen product catalogs. It's a comfortable fiction that produces comfortable scores. A team from Amazon and UC Berkeley just built a system that rips that fiction away — and the results should make anyone running agent evals deeply uncomfortable.
The Static Trap
Current agent benchmarks hand models a fixed environment and measure task completion. Pass rate goes up, the model is "better." But real deployment environments don't freeze. APIs deprecate endpoints. Databases add columns. Services merge, split, and die. An agent that scores 80% on a static benchmark might crater the moment someone updates a schema it depends on.
It's the equivalent of testing a driver only in an empty parking lot and declaring them road-ready. The parking lot score tells you something. It just doesn't tell you enough.
ProEvolve: Break It On Purpose
ProEvolve represents environments as typed relational graphs — nodes for tools, schemas, data entries, and users, with edges encoding dependencies. Want to simulate an API adding new endpoints? That's a graph completion transformation. Need to test what happens when a service creates shortcut tools that combine multi-step operations? Graph saturation. Service outage? Graph deprecation. Each transformation propagates coherently across the entire environment, updating dependent tools, schemas, and data access patterns.
The researchers started with a single environment and evolved it into 200 distinct environments spanning 3,000 task instances. Three sequential transformation phases — completion, saturation, deprecation — simulate the lifecycle of a real production system.
Frontier Models, Different Failures
They benchmarked multiple frontier models against the evolved environments. The results weren't just lower scores — they were structurally different failure patterns across models.
GPT-5 showed dramatic swings: performance climbed during completion and saturation phases as new tools helped, then crashed 48% after deprecation when familiar tools disappeared. DeepSeek-V3.2 exhibited a steady performance decrease throughout the evolution. Claude-Opus-4.5 and Gemini-2.5-Pro were less sensitive to environmental changes overall, but with a catch for Claude: reflection-based replay strategies that helped other models actually degraded its performance.
That last finding is particularly sharp. A strategy that helps one model hurts another. There's no universal adaptation mechanism. As we've covered in when single agents beat swarms, throwing more sophisticated architectures at a problem doesn't guarantee better outcomes — sometimes the complexity works against you.
The Brittleness Pattern
ProEvolve's findings echo a pattern we've seen in hierarchical agent systems: agents that excel at structured tasks in controlled settings fall apart when conditions shift. Hierarchical agents lose user context as it travels through layers. ProEvolve's agents lose tool competence as environments evolve through versions. The common thread is brittleness to change — the thing real-world deployment guarantees.
The "substantial environment-to-environment variability" the authors document means isolated benchmark scores are almost meaningless predictors of production performance. An agent's score on environment version 1 tells you very little about its score on version 4.
What Static Scores Hide
Here's the uncomfortable truth: every agent benchmark number you've seen this year was measured against a frozen world. ProEvolve doesn't just add harder tasks — it changes the ground underneath the agent's feet. The fact that frontier models all struggled differently suggests we don't have dependable agents. We have agents that have memorized specific environment configurations.
ProEvolve is open and extensible. Any team can define new graph transformations and generate evolved environments automatically. That's the right infrastructure. But it also means the next generation of benchmark scores will look worse than today's, because they'll be measuring something harder and more honest. Don't confuse the drop with regression. It's just the parking lot test finally moving to actual streets.