On this page
Small, runnable experiments for engineers building AI systems. Reproduce a failure, inspect the result, then adapt the test to your own application.
Choose a problem
Your agent generates a correct answer but chooses a wrong one
Replay answer selection compares selection policies against saved candidate traces. Separate the chance of generating a correct answer from the ability to recognise it. Inspect the failures hidden by an overall accuracy number and see what changes when a selector can abstain. The accounting extension separates overlapping call time, failed attempts and verifier costs when replaying a deadline policy.
Take away: a replay harness, labelled candidate fixtures and per-task diagnostics you can use when changing a verifier or sampling budget.
Your retriever finds a convincing document that it should not use
Test retrieval freshness and permissions exercises expired documents, superseded revisions and access restrictions. Compare ranking alone with an explicit eligibility check, including cases where the right result is no result.
Take away: a small regression suite for the documents allowed into an answer, with fixtures you can replace with sanitised examples from your own corpus.
Your tool times out after completing the action
Make tool retries safe reproduces an ambiguous timeout using a local database. Compare a repeated write with a stable idempotency key, and test what happens when the same key arrives with a different request.
Take away: an executable failure case and a concrete contract to check before an agent retries a tool with side effects.
Your outbox worker crashes after the remote action succeeds
Recover an outbox after a crash stops real worker processes at the boundary between two durable stores. Restart the dispatcher and inspect why downstream deduplication is still needed.
Take away: a crash harness, an idempotent downstream contract and a duplicate-producing negative control.
Your filtered vector search returns fewer useful neighbours
Test pgvector filtering and recall compares an exact eligible result set with HNSW and iterative scans in a real, pinned PostgreSQL instance. Inspect the query plans and vary filter selectivity.
Take away: a reproducible recall experiment with synthetic vectors, SQL plans and measured timings.
Run the examples
The introductory examples use Python's standard library. The pgvector lab also needs Docker to run a pinned PostgreSQL image. None of the labs calls an LLM or requires an API key. Each lab includes the source, the input data, the expected output and instructions for running its checks. Start with the unchanged fixture; only then substitute your own inputs.
These are deliberately small, synthetic cases. The displayed results are produced by the supplied programs, not a survey of production systems or a comparison of commercial models. They show how a failure can occur and how to test for it. They do not estimate how frequently it occurs in your application.
Turn a lab into an application test
- Choose a real failure. Start with a sanitised trace, document pair or tool operation your team can explain.
- Write down the expected outcome. Include when to abstain or ask a person, not just what a successful answer looks like.
- Keep tuning and evaluation separate. Freeze some cases before adjusting thresholds or policies. Inspect individual failures as well as totals.
- Test the real boundary. A local replay cannot prove a hosted retriever enforces permissions or a remote tool honours an idempotency key. Add an integration test for that specific contract.
- Record what changed. Keep the fixture version, policy, output and known limitations with the release so another engineer can reproduce the decision.
For a broader release checklist, use testing an agent before production. For architectural background, start with single versus multi-agent systems or test-time compute.
How we build these labs
Swarm Signal develops each example around a specific engineering question, checks the relevant primary documentation and runs the supplied code. The lab distinguishes observed fixture output from external research findings. Source links and the exact code version are provided so you can inspect both.
This series follows the practical evaluation principle of defining tasks, graders and observable outcomes rather than relying on a convincing transcript. Anthropic's guide to agent evaluations provides useful background. The fixtures and code here are Swarm Signal's own teaching examples.
Found a failing case? Tell us what you ran, including the lab version, command and sanitised input. Do not include credentials, customer data or private documents.
Sources
- Anthropic: Demystifying evals for AI agents. Background on tasks, graders, trajectories and outcomes; it is not evidence for the synthetic results in this series.
Updated 8 September 2026 · Swarm Signal