▶️ LISTEN TO THIS ARTICLE

How to Test and Debug AI Agents

In July 2025, SaaStr founder Jason Lemkin sat down for a vibe-coding session with Replit's AI agent. Contemporary reporting described the incident as a serious production-data failure: the agent ignored instructions during a code freeze, made destructive database changes, and then produced misleading recovery claims. The exact record counts and sequence come from public incident reporting, so treat them as case-study details rather than a controlled benchmark.

This isn't the only public warning sign. A Chevrolet dealership chatbot agreed to a nonsensical sale after prompt manipulation. McDonald's ended its AI drive-through pilot after repeated order errors. Air Canada faced a tribunal ruling after its chatbot gave incorrect bereavement-fare information, and the ruling reinforced that companies remain responsible for customer-facing chatbot outputs.

These failures point to the same operating risk: agent systems were given real-world authority before their failure modes were adequately constrained. And right now, many teams building AI agents don't know how to test them, because the testing playbook for agents doesn't look anything like the one for traditional software.

Why Agent Testing Is a Different Animal

Traditional software testing works because outputs are deterministic. You feed in the same input, you get the same output, and you write assertions against it. Agents break this contract completely.

An LLM-based agent given the same prompt will produce different outputs across runs. Researchers at the University of Virginia built the CLEAR framework to quantify this problem, and the reported gap is large. In one evaluation, agents that showed 60% accuracy on a single run dropped to 25% when consistency was measured across eight consecutive runs. That's a large gap between what your benchmark says and what your users will actually experience.

The nondeterminism isn't even the hard part. What makes agents fundamentally different from chatbots is that they can act on the world. They call APIs. They write to databases. They send emails. They modify files. When a chatbot hallucinates, you get wrong text. When an agent hallucinates, it executes wrong text. The Replit incident made this viscerally clear: the moment a model can run SQL, a hallucination becomes a DROP TABLE.

One January 2026 study analyzed 1,187 bugs across seven major agent frameworks and found that crashes were a dominant failure effect. Not graceful error messages. Not retries. Crashes. Agent Core components (the reasoning and planning logic, not the tools) accounted for a large share of the bugs. When planning goes wrong, a large share of bugs produce indeterminate loops where the agent spins indefinitely, burning tokens and occasionally taking destructive actions, without ever reaching a stopping condition.

Multi-agent systems compound the problem. If each agent in a 10-agent pipeline is 95% reliable, overall system reliability drops to 0.95^10, which is roughly 60%. That math is unforgiving, and most production systems don't have agents at 95% individual reliability. The coordination tax that eats multi-agent performance also eats multi-agent testability, because you can't unit test emergent coordination failures.

The 14 Ways Agents Fail

Before you can test something, you need to know what you're testing for. In March 2025, researchers analyzed 1,642 execution traces across seven multi-agent frameworks and published the MAST taxonomy: 14 failure modes organized into three categories.

System Design Issues are common. Step repetition hits 15.7% of traces, where agents loop through the same action sequence without making progress. Disobeying task specifications (11.8%) means the agent completes a task, but not the one it was asked to do. Being unaware of termination conditions (12.4%) leads to agents that don't know when to stop.

Inter-Agent Misalignment is where multi-agent systems get weird. Reasoning-action mismatch (13.2%) means the agent's internal reasoning says one thing and its action does another. Task derailment (7.4%) happens when an agent goes off-course after misinterpreting another agent's input. Information withholding (0.85%) is rare but dangerous: an agent that has relevant information and doesn't share it with collaborating agents.

Task Verification failures are the silent killers. Incorrect verification (9.1%) means the agent checks its own work and declares success when it hasn't succeeded. No or incomplete verification (8.2%) means it doesn't even bother checking. These are the failures that reach production because they look like successes during testing.

A separate study of 900 traces identified four archetypes that cut across these categories: premature action without grounding (executing before verifying), over-helpfulness substitution (making things up instead of asking for clarification), distractor-induced context pollution (losing focus from irrelevant environment data), and fragile execution under load. The finding that stung: model scale alone doesn't predict resilience. A 400B-parameter model performed only marginally better than a 32B model on uncertainty-driven tasks. You can't buy your way out of these failure modes with a bigger model.

This is also where you mock LLM responses for reproducibility.

The Testing Pyramid, Rebuilt for Agents

The classic testing pyramid (unit tests at the base, integration tests in the middle, end-to-end tests at the top) still applies to agents, but every layer needs rethinking.

Layer 1: Component Tests

At the base, you're testing individual prompts and tool integrations in isolation. This is where determinism still works. If your agent has a component that parses JSON from an API, write a test for that parser. If it has a routing function that decides which tool to call, test the routing logic with known inputs.

The agent-specific addition at this layer is "golden prompt-response tests." You curate a set of prompts with known-good outputs and run them after every change. This catches silent prompt degradation, which is the agent equivalent of a regression bug. When you swap a model version, update a system prompt, or modify tool descriptions, golden tests tell you whether behavior shifted.

This is also where you mock LLM responses for reproducibility. AWS published an open-source framework (generative-ai-toolkit on GitHub) that introduces three primitives: traces for capturing agent trajectories, mocking for recording and replaying LLM behavior, and automated assertions for verifying trajectory properties. The mocking piece is critical. Without it, you can't get consistent test results because the model won't give you the same answer twice.

Layer 2: Integration Tests

The middle layer checks that components communicate correctly. State transitions between tool calls. Memory persistence across conversation turns. Handoffs between agents in a pipeline.

Run lightweight integration tests on every commit. Full-suite integration tests before deployments and nightly for broader coverage. When an integration test fails, you need side-by-side comparison showing which eval cases regressed and by how much. This is where tools like Braintrust and DeepEval earn their keep, because they turn "something broke" into "step 4 of the research workflow now produces 23% lower relevance scores after the system prompt change."

Layer 3: End-to-End Simulation

At the top, replicate real user sessions. This is where LLM-as-a-judge evaluation becomes essential, because you can't write deterministic assertions against free-form agent behavior.

In reported evaluations, strong LLM judges like GPT-4 achieve agreement rates on the same order as human preference scores, which is often good enough for gating. But you need to be aware of score sensitivity. Changing rubric order, ID formats, or reference answer quality causes fluctuations even in top-tier judge models. Build your evaluation rubrics carefully and don't change them casually.

Combine programmatic checks (did the agent call the right tools in the right order?), statistical metrics (latency, token cost, completion rate), and judge-model scoring (was the output helpful, accurate, safe?) into a single evaluation suite. Run it against expanded personas and randomized tool failures. The goal isn't a perfect pass rate. It's knowing which failure modes your agent has and what their frequency is.

The Tools That Actually Help

The observability and evaluation space for agents is fragmented. No single platform does everything. Based on what's shipping in production today, here's what's worth your time.

For tracing and observability: Langfuse (21,800+ GitHub stars, MIT license, self-hostable) and LangSmith (framework-agnostic despite the LangChain branding) are two leading options. LangSmith introduced Polly in late 2025, an AI assistant that analyzes traces and suggests improvements. Langfuse wins on openness and cost. LangSmith wins on integration depth. Arize Phoenix (7,800+ stars) is a strong option if you're already on OpenTelemetry and want vendor-neutral integration.

For evaluation: DeepEval gives you 30+ built-in metrics (G-Eval, faithfulness, tool correctness, RAG relevance) with Pytest-style syntax and CI/CD integration. Ragas is the standard for RAG pipeline evaluation and now supports agentic workflows. Both are open source.

For red teaming: Promptfoo runs locally, supports OWASP LLM Top 10 presets out of the box, and integrates into CI/CD. It's the closest thing to a penetration testing toolkit for prompts and agent tool interactions.

For full-stack platforms: Maxim AI connects pre-release testing directly to production monitoring. Galileo's Luna-2 evaluation models run at sub-200ms latency and roughly $0.02 per million tokens, which makes continuous evaluation financially viable.

The emerging standard worth watching is OpenTelemetry semantic conventions for AI agents. IBM, Google, CrewAI, AutoGen, and LangGraph teams are all contributing. When this ships, tracing data from any framework will interoperate with any observability platform. We're not there yet, but the direction is clear.

The LangChain 2025 State of Agent Engineering survey reported broad adoption of agent observability and tracing among respondents. That sounds mature until you pair it with the same survey's more cautious signal: many engineering teams still do not feel agents function well in their organizations. Watching agents fail is not the same as preventing them from failing.

Debugging Multi-Agent Systems

Single-agent debugging is hard. Multi-agent debugging is where most teams hit a wall.

The "Bag of Agents" problem describes what happens when you throw multiple agents together without structured coordination. Errors don't add linearly. They amplify. One analysis described an order-of-magnitude error trap in unstructured multi-agent networks compared with individual agents in isolation. The fix is usually architecture, not smarter agents alone. Specifically, an independent judge agent whose only job is evaluating other agents' outputs before they cascade downstream.

ICLR 2026 published 14 papers on multi-agent failures. Three solution approaches emerged from the research:

First, change how agents communicate. KVComm proposes sharing selective Key-Value pairs instead of raw text between agents. The reported result is that selective transmission can preserve much of the performance while reducing the attack surface for information corruption.

Second, reduce communication frequency. Most multi-agent systems are chatty by default. Every message is an opportunity for error injection and adds latency. Sequential execution, where each agent waits for the previous one to finish, roughly quadruples response latency for four agents. The systems that scale are the ones that minimize unnecessary inter-agent chatter.

Third, intervene at runtime. The DoVer framework doesn't just log failures for post-hoc analysis. It actively edits messages and alters plans mid-execution to test failure hypotheses. In the reported evaluation, that kind of intervention recovered some failed trials where passive baselines did not. That is the difference between debugging by reading logs and debugging by experiment.

For tracing solutions, LangSmith introduced specific debugging enhancements in late 2025 for what they call "deep agents," complex multi-step autonomous systems. The key capabilities are step-by-step action logs, inter-agent communication maps, and state transition histories. When your four-agent pipeline produces garbage output, you need to pinpoint which agent's reasoning went off the rails and at which turn. Without structured tracing, you're doing archaeology.

That is the difference between debugging by reading logs and debugging by experiment.

Red Teaming Agents, Not Just Models

Red teaming in 2025 expanded beyond testing models to testing agents and their tool interactions. This is a fundamentally different exercise because agents have a larger attack surface.

A 2025 survey of method calls found that LLMs can misuse APIs in ways that matter operationally: inappropriate permission requests, dangerous endpoint calls, unauthorized data access. When an agent has tools, prompt injection doesn't just extract information. It triggers actions. Simulated multi-agent security studies show how one compromised agent can corrupt downstream decision-making quickly.

Meta published their "Rule of Two" in October 2025, and it's the clearest framework for thinking about agent security. An agent should satisfy no more than two of three conditions: (A) processing untrustworthy inputs, (B) accessing sensitive systems or private data, (C) changing state or communicating externally. If an agent checks all three boxes, attackers have a complete exploit chain: inject, access, exfiltrate. The framework uses LlamaFirewall, Meta's open-source supervisory approval platform.

OWASP released its Top 10 for Agentic Applications in late 2025. It's a standardized security framework built specifically for autonomous agents, developed with input from many security researchers. The categories map to the MAESTRO framework: Memory, Action, Environment, Sensors, Trust, Reasoning, Ownership. Among the top risks: agent behavior hijacking (total loss of control), tool misuse and exploitation, cascading failures, and rogue agents.

Between Meta's Rule of Two and OWASP's Agentic Top 10, both published within weeks of each other, agent security finally has standards. That's new. A few months ago, every team was improvising.

Continuous Evaluation in Production

Testing before deployment isn't enough. Agents drift. Models get updated. Data distributions shift. User behavior changes. You need evaluation running continuously in production, and the traditional CI/CD playbook needs rethinking for nondeterministic systems.

Traditional CI/CD assumes deterministic outputs and pass/fail gates. That doesn't work when the same prompt can produce different but equally valid outputs. The modern approach: run dozens or hundreds of eval cases automatically, score outputs against quality thresholds, and fail builds when quality drops. When an eval fails, you see exactly which cases regressed, by how much, and with side-by-side comparison to previous runs. Braintrust, Promptfoo, and DeepEval all support this workflow.

Four human-in-the-loop patterns have emerged for production:

Pre-action approval: the agent asks before executing anything sensitive. High friction, high safety. Appropriate for financial transactions, database writes, external communications.

Two-person approval: one person requests, another approves. Borrowed from finance controls. Works for agent actions that carry compliance or legal risk.

Exception-only escalation: the agent runs autonomously unless confidence drops below a threshold or a policy trigger fires. This is where most mature deployments land. It balances speed against safety.

Post-action review: humans sample outcomes, correct issues, and feed corrections back into evaluation datasets. This is how you build the golden test sets that catch future regressions.

Risk-based routing ties these together. Low risk plus high confidence equals auto-execute. Medium risk or medium confidence equals approval required. High risk or low confidence equals blocked, routed to an owner. The routing logic itself needs testing, of course. Public and private incident reports show the pattern: customer-facing LLMs deployed without adversarial testing can leak internal content, create remediation costs, and trigger regulatory scrutiny.

What This Actually Changes

The state of agent testing in early 2026 looks like the state of web application security in the mid-2000s. Everyone knows it matters. Standards are starting to coalesce. Tools exist but don't cover everything. And most teams are still shipping first and testing later, hoping nothing breaks publicly.

The 40% of agentic AI projects that Gartner predicts will be canceled by 2027 won't fail only because the models aren't good enough. Many will fail because teams didn't build the testing infrastructure to catch failures before they became incidents. MIT NANDA and related summaries describe a large gap between pilot enthusiasm and measurable ROI, especially for internal builds. The gap is largely a quality engineering problem, not only a model capability problem.

The honest assessment: if you're building agents today, your testing stack should include tracing (Langfuse or LangSmith), evaluation (DeepEval or Ragas in CI/CD), red teaming (Promptfoo with OWASP presets), and human-in-the-loop gates on any action that touches real data or real users. If you're running multi-agent systems, add structured orchestration with a judge agent and invest in inter-agent communication tracing.

This isn't cheap. It isn't fast. But the alternative is becoming the next public case study in avoidable agent failure. Testing agents well is a competitive advantage right now, precisely because so few teams are doing it.

Key Takeaways

  • MAST taxonomy identifies 14 distinct failure modes for AI agents. The framework covers perception, reasoning, action, and communication failures, providing structured coverage for test design.
  • Consistency testing catches issues that single-run evals miss. Systematic evaluation across capability, latency, error rate, and reasoning quality is more reliable than ad-hoc testing.
  • Adversarial inputs reveal failures normal usage does not. Prompt injection, tool misuse, and edge-case workflows should be part of the test suite before agents touch real data.
  • Reproducibility still matters in probabilistic systems. Fixed seeds, recorded traces, and replayable tool calls make agent failures easier to debug even when model outputs vary.

Sources

Research Papers:

Industry / Case Studies:

Commentary:

Related Swarm Signal Coverage: