# Swarm Signal

> Swarm Signal publishes technical guides and research analysis for engineers building and evaluating AI agents.

These are cleaned excerpts of up to 800 words from the selected pages. Follow each article link for its complete argument, qualifications and references. Publication dates are article metadata, not dates of independent verification.

## Choose an architecture and define success

### [Types of AI Agents: Reactive, Deliberative and Hybrid](https://swarmsignal.net/types-of-ai-agents/)

Published: 2026-02-16. Updated: 2026-09-08. Excerpt follows.

Reactive agents respond to the current situation. Deliberative agents plan or reason about possible actions. Hybrid systems combine those approaches. Autonomy is a separate question: how much freedom does the system have to choose and execute its next step? For an engineer choosing an architecture, those distinctions are more useful than a model leaderboard. Start with the work you need done, the evidence that would count as success and the consequences of a wrong action. Which type of AI agent should you build? Pattern Good starting use Main limitation First check Reactive A known signal needs a predefined response Rules may miss an unfamiliar situation Does the rule recognise invalid or missing input? Deliberative An investigation needs several observations and revised decisions More opportunities for unnecessary steps and tool errors Can it stop, verify the result and stay within budget? Hybrid Routine requests and ambiguous cases share one service The routing boundary can send work down the wrong path How often does routing cause an avoidable failure? Fixed workflow The sequence is already known, such as validate, retrieve, format Unexpected cases require an explicit branch or escalation Does adding model-directed control improve task success? These labels overlap. A fixed workflow can contain an LLM call; a reactive controller can sit inside a larger planning system. There is no universal latency or price attached to a category. Measure the actual model, tools, infrastructure and retries. Reactive agents: a response without an explicit plan A simple thermostat illustrates a reactive policy: compare a reading with a threshold and select an action. In software, a rule that routes a known incident code to its runbook has a similar shape. Its usefulness comes from a clear mapping between input and response. Reactive does not always mean stateless. A controller might retain recent observations without searching over future plans. Nor does fast output prove that a system is reactive. A chatbot interface alone tells you little about the planning, retrieval or memory behind it. Use this pattern when the response is well specified and testable. Include malformed inputs, missing readings and changed conditions in the tests. The escape route for an unknown case is part of the architecture, rather than something to add after deployment. Deliberative agents: choose the next step from evidence A deliberative system represents a goal and considers how to reach it. For an incident investigation, that might mean inspecting a failed request, checking a dependency and changing the working hypothesis when the evidence disagrees. ReAct studies interleaving reasoning and actions in language models. Actions can obtain external information, which then informs subsequent decisions. Its reported results concern the tasks and baselines evaluated in the paper; they do not establish that every tool loop is reliable. A reasoning model is one component, not the whole agent. Tool permissions, observations, stopping rules and result checks determine what the surrounding system can do. Extra inference or extra calls may help on a difficult task, but an architecture label cannot tell you whether the improvement justifies its cost. Before increasing an agent’s reasoning budget, use the test-time compute evaluation guide to compare the answers returned with the extra work required. Hybrid agents: make the boundary explicit A practical hybrid might return a cached service status through a fixed path and send an unexplained outage to a bounded investigator. Both paths can share an output format, while only the investigator needs to choose successive observations. Routing is itself a decision worth evaluating. A false escalation wastes time; a complex incident incorrectly treated as routine can produce a misleading answer. Record the selected route and its eventual outcome so that you can distinguish a weak router from a weak downstream model. Anthropic distinguishes predefined workflows from systems where the model directs its own process, and recommends starting with the simpler solution. That distinction is useful when assessing whether a workflow actually needs more autonomy. Read the engineering guidance . Autonomy is a permission boundary A planner that proposes a database change for review has less execution authority than a rule that applies that change automatically. Neither the number of agents nor the length of a run determines that authority. Write down which actions require approval, which resources are accessible and what ends the run. Keep read-only investigation separate from writes. A tool allowlist does not replace authentication or resource-level permissions, and a prompt asking the model to be careful does not enforce either. Start by granting enough authority to complete a bounded task. Broader access needs evidence that the task requires it, plus a way to inspect and reverse the resulting changes where possible.

### [Model Selection Guide: How to Pick the Right AI Model for Your Use Case](https://swarmsignal.net/model-selection-guide-how-to-pick-the-right-ai-model-for-you/)

Published: 2026-05-16. Updated: 2026-06-22. Excerpt follows.

Model Selection Guide: How to Pick the Right AI Model for Your Use Case In this guide, "best model" means "best fit for the job in front of you": the task, budget, latency target, privacy constraint, and integration path. Use it as a working checklist for narrowing candidates, comparing tradeoffs, and deciding when one model is enough. Start With Your Workload Before comparing vendors, write down: the task the quality bar the budget and latency target the data constraints the integration requirements the failures that would be unacceptable That list is your first filter. The leaderboard can wait. Related deeper dives: benchmark crisis analysis and how to evaluate AI models without trusting benchmarks . The Five Dimensions That Actually Matter Use these five variables as the first pass. 1. Task Fit Different models excel at different things. This isn't a minor detail. It's the entire decision. Coding and software engineering: Compare candidates on the kind of work they will actually do: code search, bug fixing, test repair, refactoring, documentation, or agentic coding with tools. Use examples from your own repositories where possible. Reasoning and math: If your application depends on multi-step reasoning, test the model with the same tools, context, and answer format it will use in production. Long-context processing: Check both the advertised context window and the model's behavior on your actual document lengths. A large context window is useful only if the model can retrieve and use the right information inside it. Writing and complex instruction-following: For customer-facing content, detailed analysis, or tasks where tone matters, judge outputs against your house style instead of relying on a generic preference score. Multimodal tasks: If the workflow mixes text with images, audio, or video, include those inputs in the selection test from the beginning. The point is not to memorize a ranking. It is to make task fit the first filter, not an afterthought. 2. Cost Do not copy pricing from a blog post into your budget model. Pull current input and output prices from provider pages or an aggregator such as Artificial Analysis on the day you estimate the deployment. Run the math on your expected request shape: input length, output length, cache behavior, batch discounts, retry rate, and peak traffic. A RouteLLM study published at ICLR 2025 reported large cost reductions from routing between cheaper and more capable models while preserving most of the stronger model's quality. The right question isn't "can we afford the best model?" It's "what's the cheapest model that meets our quality bar for this specific task?" 3. Latency For interactive applications, measure time-to-first-token, total generation time, timeout rate, and retry behavior. Raw capability is not useful if the response arrives too late for the user experience. If your application is latency-sensitive, test smaller models, provider placement, caching, and streaming behavior before you commit. 4. Privacy and Data Residency For any hosted API, confirm what data leaves your environment, where it is processed, how it is retained, and whether it can be used for training or abuse monitoring. For regulated work, get this answer before model testing begins. Open-weight models can be run on infrastructure you control. The tradeoff is operational complexity: serving, updates, monitoring, capacity planning, and incident response become your problem. If your data can leave your network, hosted APIs may be the simplest path. If it cannot, evaluate self-hosted models or private deployments first. 5. Ecosystem and Integration The model you pick comes with an ecosystem: SDKs, fine-tuning support, tool-use behavior, structured output reliability, deployment options, and community resources. For agent-based applications, test tool calls with your actual schemas. The function calling and tool use analysis covers the current state in detail. The Decision Framework Here's the practical process, in order. Step 1: Define your quality bar. Before looking at any model, write down what "good enough" looks like for your use case. Use a small set of representative examples with expected outputs, failure cases, and review notes. Step 2: Filter by hard constraints. Data residency, latency, context length, integration, and budget requirements should remove candidates before subjective preference enters the process. Step 3: Compare the survivors on the same examples. Measure output quality, consistency, latency, tool-call behavior, and failure modes. Use the same prompts and scoring rules for every candidate. Step 4: Cost-normalize. Estimate the cost per successful request, not just the cost per token. Include retries, caching, moderation, tool calls, and fallback paths. Step 5: Test before committing. Use a shadow deployment, replay set, or limited rollout before moving critical traffic. Watch for timeout rates, rate limiting behavior, content-filter triggers, and edge-case handling. When to Use Multiple Models Use multiple models when one model cannot meet your quality, latency, privacy, and cost requirements at the same time. Model routing sends different requests to different models.

### [Agent Evaluation Checklist](https://swarmsignal.net/agent-evaluation-checklist/)

Published: 2026-09-06. Updated: 2026-09-08. Excerpt follows.

A practical starting checklist for engineers evaluating a tool-using AI agent. Copy it into your project notes or print this page. It does not replace workload-specific testing or security review.

Copyable test record

Copy this record for each test case. Agree on the expected result before running it. Store traces in your own system and remove personal data or secrets before sharing them.

Case ID and task: Input and expected result: Allowed tools and actions: Actions requiring human approval: Model, prompt, tool and data versions: Time, retry and spending limits: Observed result: Independent evidence or trace reference: Accepted result? Yes / No Failure type: routing / retrieval / tool / verification / other Elapsed time and number of attempts: Total cost, including failed attempts: Permission or stop-limit breach? Yes / No Follow-up change and retest result:

Compare complete tasks

Keep the task set and acceptance criteria the same when comparing designs. Record unsuccessful attempts as well as successful ones. Divide the total measured cost by the number of accepted results to compare cost per accepted task. If no results are accepted, report that outcome rather than a cost-per-success figure.

Try a simple fixed workflow alongside the agent, then use the architecture guide to interpret the differences. A low cost is only useful if the result meets the task and permission limits.

Define the job

Write one observable success condition before running the task.

Choose representative examples, including failures and ambiguous inputs.

Keep development examples separate from the final evaluation set.

Record model, prompt, tool and data versions so a result can be reproduced.

Bound the authority

List the resources and actions the agent may access.

Enforce permissions outside the model. A prompt is not an access control.

Identify writes that need approval, and test refusal before any write occurs.

Set limits for elapsed time, steps, retries and spend; give each an enforced stop path.

Test the awkward cases

A tool times out, returns malformed data or becomes unavailable.

Retrieved content includes instructions outside the task.

A previously allowed action loses permission during execution.

The necessary fact is missing or conflicts with another source.

The agent repeats a failed action, or declares success without evidence.

Inspect outcomes and traces

Verify the result independently of the agent's own success claim.

Record task success, elapsed time, tool calls and actual cost together.

Separate routing, retrieval, tool and verification failures.

Repeat representative cases; one successful run is not a reliability estimate.

Compare with a simpler fixed workflow before adding more autonomy.

Make a release decision

Define acceptable failures before seeing the results.

Document workload limits and cases that require human review.

Prepare a rollback and a way to stop the running system.

Keep regression cases from real failures and review them after changes.

This checklist is Swarm Signal's editorial guidance. For the underlying distinctions, see Anthropic's agent engineering guidance and our architecture guide . For evaluation coverage, see the Messier analysis .

See a sample weekly briefing or subscribe for new practical analysis .

### [Multi-Agent Human Handoff Patterns: When the Swarm Needs a Person](https://swarmsignal.net/multi-agent-human-handoff-patterns/)

Published: 2026-06-18. Updated: 2026-06-22. Excerpt follows.

Multi-agent human handoff patterns are where agent orchestration stops being a diagram and becomes an operating system. The hard question is not whether a swarm can route work between specialists. It is when the swarm should stop, preserve state, and ask a person to take responsibility. Key takeaways Treat human handoff as a designed state transition, not a panic route. Escalate on authority, uncertainty, irreversible action, or coordination failure. Preserve the evidence bundle: goal, agent trace, proposed action, missing context, and recommended next step. Measure handoff quality by resolution, rework, and blame clarity, not by automation rate. Multi-agent human handoff patterns start with authority Most agent handoff examples describe agent-to-agent routing: triage to billing, billing to refunds, refunds to retention. That pattern matters, but it is incomplete. A person is not just another specialist in the graph. A person changes the authority boundary. OpenAI's Agents SDK treats human review as a pause-and-resume flow for sensitive tool calls, where a run can surface pending approvals and later resume from saved run state after a person approves or rejects the action ( OpenAI Agents SDK ). LangGraph exposes the same operating idea through interrupts, where a graph can pause before an API call, database change, or other important decision and then route based on an approval or rejection ( LangGraph interrupts ). The useful pattern is not "human in the loop" as a slogan. It is "human at the boundary where the system lacks authority". Refund above policy limit. Production rollback. Legal wording. Customer complaint. Data deletion. Vendor payment. If the action changes money, rights, access, safety, or reputation, the swarm should prepare the decision, not make it silently, which matches NIST's emphasis on proportionate accountability when consequences are severe ( NIST AI RMF 1.0 ). This is the missing layer between multi-agent systems and the coordination tax . More agents can reduce workload, but each extra agent can also make it harder to say who had the authority to act. Human handoff gives that authority a named owner. Pattern one: approval gates before irreversible action The cleanest handoff is an approval gate. Agents can plan, gather evidence, draft the action, and stop before execution. The human sees the proposed action, the reason, the review label, and the reversal path, following the approve-or-reject pattern documented by LangGraph ( LangGraph interrupts ). This pattern fits tool calls that write to external systems: cancelling an order, changing a database row, sending a customer email, merging a pull request, or rotating a production setting. The gate should ask for a decision, not a conversation. Approve. Reject. Edit. Escalate. If the interface only says "continue?", it is too vague. The approval gate also reduces false confidence. The MAST paper analysed more than 150 multi-agent tasks with six expert annotators, identified 14 failure modes, and grouped them into specification and system design failures, inter-agent misalignment, and task verification or termination failures ( MAST ). Those categories map directly to human review. A person should not be asked to reread a transcript from scratch. They should be shown which category the system thinks it is in. Pattern two: escalation on stalled coordination Some handoffs should trigger before execution danger becomes visible in a tool call. Agent A routes to Agent B. Agent B asks Agent C. Agent C returns a partial answer. The graph keeps moving, but the work has stopped getting clearer ( Microsoft Agent Framework ). Microsoft's Agent Framework makes this interaction explicit: in handoff orchestration, if an agent does not hand off after a turn, the workflow emits a request for human input, and autonomous mode is described as an experimental option rather than the default path ( Microsoft Agent Framework ). That is the right instinct. Silence from the swarm is itself a state. A practical escalation rule can be simple: trigger human review when the same task crosses a handoff threshold, when agents disagree about ownership, when evidence conflicts, or when the system cannot name the next responsible actor. This is where when single agents beat swarms becomes operational advice. If the swarm is passing uncertainty around, a smaller system may be safer. Incident response already has the better model. Google's incident-management guidance describes clear roles for Incident Commander, Communications Lead, and Operations Lead, with the Incident Commander coordinating the response and assigning responsibilities by incident context rather than reporting chain ( Google SRE ). Agent systems need the same discipline. Handoff is not just routing. It is command transfer. Pattern three: evidence bundles, not chat dumps The human should receive a compact bundle, not a scrollback. A useful bundle has five parts: the user goal, the live system state, the agents that touched the task, the proposed next action, and the reason automation stopped.

### [Small Language Model Agents: The 2026 Practical Guide to Sub-10B Deployments](https://swarmsignal.net/small-language-model-agents-guide/)

Published: 2026-04-26. Updated: 2026-06-22. Excerpt follows.

In early 2025, using a small model as an autonomous agent often meant accepting capability loss on planning, tool selection, and multi-step reasoning in exchange for cheaper inference. The current picture is more nuanced: targeted training, constrained decoding, and narrower task design can make small models viable for specific agent workloads. A 2025 paper reports that a fine-tuned 350M-parameter model reached a 77.55% pass rate on the ToolBench benchmark ( source: arXiv 2512.15943 ). Treat that as a benchmark result for a specifically trained setup, not proof that any small model can replace a frontier model. Note: The ToolBench result comes from a single paper with a specifically fine-tuned model and may not generalize to all small-model deployments. Benchmark conditions can differ significantly from production tool-calling scenarios. Targeted training on agentic trajectories can beat raw model scale on scoped tasks, but the cost advantage should be measured against your own serving stack, fine-tuning cost, and reliability targets. This guide is intended for practitioners evaluating whether and how to run agents on models under 10B parameters. Note: The capabilities and benchmarks described here reflect the state of publicly available models and tooling as of mid-2026. Small model performance is evolving rapidly; readers should verify specific claims against current benchmark data. It covers model families to evaluate, task types where small models tend to perform better or worse, techniques that close capability gaps, and the economics to measure before deploying. Why the Old Assumption Doesn't Hold The conventional case against small model agents rested on three observations: they hallucinate more, they fail at tool selection from large catalogs, and they can't plan across more than a few steps. All three still hold in edge cases. What changed is how far targeted techniques push the performance ceiling. The key shift is that guided decoding with JSON Schema validation has become a common component of production agentic stacks. When you constrain a model to output valid structured tool calls rather than free-form text, you reduce a large category of failures that small models were penalized for disproportionately. On constrained, schema-bound tasks, small models can be competitive with much larger systems when the task distribution is narrow and well tested. The models did not simply get smarter; the interface got stricter. The second shift is training data. Early agentic fine-tuning datasets were sparse and often synthetic in ways that didn't reflect real tool-calling distributions. The 2025-2026 generation of agentic training sets, including real trajectory data from production agent deployments, has changed what's achievable with supervised fine-tuning at small scale. Which Models Actually Perform Not all sub-10B models are equivalent for agentic work. The differences between lineages matter more than raw parameter counts. Qwen3-8B is a strong candidate for some general-purpose agentic workloads at sub-10B scale in the source set used for this draft. Its "thinking mode" makes it worth evaluating for workloads that mix structured tool calls with lighter lookup tasks. The Qwen lineage has been explicitly trained on function-calling data, which is visible in BFCL (Berkeley Function Calling Leaderboard) results. Phi-4-Mini (3.8B) from Microsoft is worth evaluating for reasoning-intensive chains. Public benchmark results make it a credible candidate for agents that spend most of their inference budget on single-step reasoning rather than long-context integration. DeepSeek-R1-Distill-Qwen-7B is worth testing for workloads where extended planning before action matters. The distillation from DeepSeek-R1 aims to preserve some of the parent model's structured reasoning capability in a form that can be cheaper to deploy. Gemma 3 (Google's 4B and 12B variants) is worth evaluating for multimodal agentic pipelines. If your agent needs to process images alongside text as part of its tool-use loop (screenshot parsing, document understanding, visual QA), Gemma 3's architecture may handle multimodal context more reliably than some alternatives at this parameter scale. For teams not committed to a specific model family, running a small evaluation against your actual tool catalog on BFCL v3 task types will tell you more than any benchmark aggregate. The variance between models on specific tool-calling patterns is high enough that general rankings can mislead. What Small Model Agents Handle Well Schema-constrained tool calling. This is one of the strongest use cases. With guided decoding and strict JSON Schema enforcement, sub-10B models can perform well on single-tool and small-toolset selection tasks. The technique works by restricting the token sampling space to valid continuations of the schema at each position. The model does not have to "know" to output valid JSON; it cannot output anything else. RAG pipelines. Seven-billion-parameter models with retrieval augmentation can handle factual lookup reliably when the retrieval quality is high and the answer format is constrained. The model's job in a RAG agent is to parse the retrieved context and format a response, a task that can scale down well.

### [MoE vs Dense Models: A Practical Deployment Guide](https://swarmsignal.net/moe-vs-dense-models-decision-guide-2026/)

Published: 2026-03-26. Updated: 2026-09-08. Excerpt follows.

A mixture-of-experts model can use a small share of its weights to process each token while still needing a large deployment. A dense model can fit an existing serving stack, yet fail the task you need it to perform. Architecture helps explain resource use; it does not settle quality, safety or price. Use this guide when comparing specific open-weight models for an agent or application. If you buy an API, start with measured task quality and the provider's bill. You may have little control over the architecture underneath it. What sparse activation actually means In a sparse MoE layer, a router selects a subset of feed-forward blocks to process a token. Other parts of the network remain shared. The active parameter count describes the weights used for that token; the total parameter count describes the larger collection of weights. The Mixtral paper explains this distinction and how expert parallelism distributes work across devices. An expert is a learned block of parameters, not a named specialist you can ask to handle legal analysis or code review. Mixtral's routing analysis did not find a simple correspondence between experts and subject areas. Do not turn a routing diagram into a claim about domain competence. Choose a model using completed tasks, not the architecture printed on its model card. Neither architecture establishes suitability for a high-stakes use case. Evaluate the particular model, configuration, evidence requirements and human review process. Separate compute from resident memory Sparse activation can reduce feed-forward computation per token compared with evaluating all the experts. It does not make inactive weights disappear. For a conventional fully resident deployment, budget for the entire model's weights across the serving devices, plus runtime overhead and request state. Quantisation changes weight storage. Weight offloading can move some weights to host memory or another storage tier, but transfer costs and the serving implementation then become part of the comparison. Every expert need not fit on a single GPU. Record peak allocation after loading the model and while serving representative requests. A parameter count multiplied by nominal storage precision is a planning estimate, not a measured memory requirement. Include temporary buffers, communication workspaces and reserved memory. The useful question is whether the complete configuration fits your hardware with room for requests. A model that loads successfully can still run out of memory under traffic. Context and concurrency can change the result The attention key-value cache stores state from previously processed tokens. In ordinary full-attention decoding, longer sequences and more simultaneous requests increase the state to retain. This is separate from expert weight storage. Cache behaviour depends on the model and runtime. Sliding-window layers can bound growth; static caches reserve capacity differently from dynamic caches; quantised or offloaded caches trade memory against other costs. Hugging Face's cache guide describes these differences. Run each candidate at short, typical and long input lengths, then increase concurrent requests. Keep the output allowance representative too: generated tokens extend the sequence. Record peak memory, time to first token, completion latency and accepted tasks per unit of time. Include the busiest workload you expect. Fast token generation can still mean slow answers when requests queue. Conversely, low cost per generated token may hide the cost of hardware waiting for work. Measure at the utilisation you can sustain. A comparison matrix you can fill in Decision Evidence to collect What rules out a candidate Task quality Held-out examples with identical acceptance criteria Critical errors or missing evidence Tool use Valid arguments, permissions and recovery from failures Unsafe actions or malformed calls Memory Peak allocation at target context and concurrency Exhaustion or unacceptable offloading delay User experience Completion latency including queues and retries Exceeding the application's latency budget Economics Total cost divided by accepted tasks Exceeding the sustainable budget Maintenance Runtime, adapter support, licence and upgrade procedure A dependency the team cannot operate or use These are suggested evaluation criteria, not benchmark results. Set pass conditions before looking at which candidate wins. For agents, include a failed tool call, an ambiguous request and a task that should stop for human input. Scoring only fluent final answers misses behaviour that determines usability. Fine-tuning is another experiment Do not assume dense fine-tuning updates every weight: adapter methods deliberately update a subset. Nor does MoE fine-tuning inevitably damage routing. Ask what the selected tooling supports and whether adaptation improves held-out tasks. Keep training and evaluation examples separate. Check both the base and adapted models on behaviours you must preserve, such as tool formatting and refusal to invent missing evidence. Expert utilisation, if available, is a diagnostic rather than a substitute for task evaluation. Account for adapter loading, quantisation compatibility and deployment memory. A training run that finishes is only the beginning of the serving decision. Run a fair pilot Use an identical task set, tool implementation and acceptance rubric.

### [Single Agent vs Multi-Agent Systems: How to Choose](https://swarmsignal.net/single-vs-multi-agent-comparison-2026/)

Published: 2026-03-21. Updated: 2026-09-08. Excerpt follows.

Start with a working baseline, then add agents to solve a specific failure or execution constraint. A long prompt, a large tool catalogue or a complex task does not establish that a team of agents will perform better. The useful question is which parts of the work can run separately without losing information needed for the final decision. A single agent can make repeated model calls, use tools and inspect their results. A multi-agent system introduces separate working contexts and a way to combine their work. Neither label tells you how many model calls a request needs, what it will cost or whether its answer is correct. Compare the work before comparing frameworks Task property Baseline to test Reason to consider a different design Each action depends on the previous result Single agent with explicit state A specific stage needs a separate permission boundary or independently checked result Evidence can be collected independently Fixed parallel workflow The required subtasks vary enough to justify an orchestrator Distinct requests need different tools Router with focused tool access A request genuinely spans several specialties Conflicting evidence needs reconciliation Explicit evidence comparison Separate reviewers catch errors that the baseline misses on held-out cases The task is already a predictable procedure Deterministic workflow Model-directed choices improve a measured outcome These are starting hypotheses, not benchmark results. Try tool filtering, better retrieval and clearer stopping rules before attributing a baseline failure to agent count. Our MCP server architecture guide covers the tool and permission boundary; adding specialist prompts does not enforce access control. Add an agent when you can name the work it owns and the evidence it must return. What the research supports Google's scaling study compares single-agent and several multi-agent architectures across financial analysis, browsing, planning and tool-use tasks. Its results depend on task structure and coordination design. The authors report benefits on decomposable workloads and losses on the sequential planning benchmark. Their research explanation also describes coordination costs in tool-heavy tasks. That evidence supports testing decomposition. It does not supply a universal context length or tool count at which every application should switch architectures. Nor can its measured error-amplification ratios be multiplied by an unrelated application's error rate to predict that application's failures. You need matching definitions, denominators and observations before transferring such a calculation. Anthropic's engineering guide distinguishes fixed workflows from model-directed agents and describes routing, parallelisation and orchestrator-worker patterns. Its recommendation to add complexity when justified is a useful implementation discipline. It is not proof that a particular framework will be fastest on your workload. The MAST study examines failures involving system design, communication between agents and task verification. Use those categories when reading traces. Agreement between workers is insufficient if they share the same mistaken source or the final checker never examines the underlying evidence. Worked case: investigating a failed customer import Consider an assistant that investigates a failed data import and drafts a support response. This is a proposed architecture exercise, not a report of production measurements. The assistant may read a permitted customer's import log, current product documentation and an incident feed. It may draft a reply, but sending it requires approval. Build the baseline around the actual sequence: identify the import, retrieve its failure record, check the relevant documentation, then explain the likely cause with evidence. If the log says a required field is missing, the next useful lookup depends on that result. Splitting those dependent steps across agents adds a handoff without creating independent work. Now consider an ambiguous failure. Documentation review and incident-feed inspection can proceed independently once the product version and failure details are known. Test a fixed parallel workflow first. Give each branch the same request identifier and relevant version, require source references, and combine the returned evidence before drafting the response. An orchestrator becomes a candidate if the investigation regularly needs different branches that cannot be selected by simple routing. Require it to record why each branch is needed. A worker returning a plausible diagnosis without the referenced log or document should produce an incomplete investigation, not an automatic success. Keep the permission check in the tool service. Each branch receives access only to authorised records; a specialised role description is not a security boundary. Assign approval for the final outbound response explicitly. If a tool call times out after an external action, determine whether the action happened before retrying it. Run an architecture comparison you can use Freeze a set of representative requests and their expected outcomes. Separate routine cases, ambiguous cases and deliberate failures. Use the same source snapshot, tool behaviour, access rules and acceptance criteria for every candidate. Where you change the model or budget, record that change so it is not mistaken for an orchestration improvement.


## Build tools and manage memory

### [Agent Tool-Use Patterns: How LLMs Actually Wield APIs](https://swarmsignal.net/agent-tool-use-patterns-guide/)

Published: 2026-06-07. Updated: 2026-06-22. Excerpt follows.

Agent Tool-Use Patterns: How LLMs Actually Wield APIs Many major model providers now support function calling or tool-use interfaces. OpenAI, Anthropic, Google, and a growing set of open-weight models let developers define tool schemas and receive structured outputs. The interface is maturing; reliability still depends heavily on schema design, validation, and orchestration. On the Berkeley Function Calling Leaderboard (BFCL) V4 , recent snapshots suggest leading models cluster in a similar band, but the exact ordering changes as models and benchmark versions update. Treat the benchmark as a point-in-time signal. Production results can be lower when schemas are messy, tools are numerous, and failures compound across steps. The gap between "can call a function" and "can use tools reliably across a multi-step workflow" is where agent projects become fragile. This guide covers the common tool-use lifecycle, failure modes, and emerging standards that try to make the stack easier to validate. As of June 2026, the practical lesson is still the same: verify the current provider docs and benchmark versions before treating any of the examples below as production defaults. The Tool-Use Lifecycle Tool use isn't one skill. It's six, executed in sequence, and an agent can fail at any step. Discovery is knowing what tools exist. In early systems, this meant stuffing every function signature into the system prompt. That can work for small tool catalogs and become brittle as the catalog grows. AnyTool introduced hierarchical retrieval structures that use semantic search to inject only the most relevant top-K tool descriptions into the context window. The tradeoff: retrieval adds latency and can miss the right tool if the user's intent doesn't match the tool's description. Selection is choosing which tool to call. This is where the ReAct framework remains influential, interleaving reasoning traces with action steps. But ReAct can have an expensive bottleneck: it may invoke the LLM at every step to decide which tool to use next. AutoTool , published in November 2025, sidesteps this by building a directed graph from historical agent trajectories. Nodes are tools, edges are transition probabilities. The reported result is lower inference cost while maintaining task completion quality in that setting. Parameter construction is generating the correct arguments. This is harder than it sounds. A comprehensive analysis presented at EMNLP 2025 categorized multi-step tool call errors into five patterns: tool selection errors, tool hallucination errors, parameter key errors, parameter value errors, and environment errors. Missing required parameters, hallucinating parameter names that don't exist in the schema, and filling values with plausible-sounding but incorrect data are all common. Models that write flawless Python routinely botch start_date vs date_start in an API call. Execution is running the tool and getting a result. The agent doesn't execute anything directly. It hands structured JSON to your application layer, which calls the API, runs the query, or hits the database. Your code handles retries, rate limits, and timeouts. The agent just waits. Result interpretation is reading what came back and deciding what to do next. This is where context windows matter. A tool that returns 50KB of JSON can blow out the context budget for the rest of the conversation. Smart implementations truncate, summarize, or extract specific fields before feeding results back to the model. Chaining is doing it all again with the next tool. Multi-step workflows create compounding error rates. Even modest per-step failure rates compound quickly across a five-step chain, which is why production agents feel unreliable even when individual tool calls look fine. How the Big Three Implement It The three major providers have converged on similar interfaces but diverge on important details. OpenAI's function calling defines tools as JSON Schema objects passed in the tools parameter. The model returns a tool_calls array with function names and arguments. OpenAI supports parallel tool calls by default, letting the model invoke multiple functions simultaneously when the calls are independent. You can disable this with parallel_tool_calls=False when order matters. Anthropic's tool use follows the same pattern but wraps it in a content-block architecture. Claude returns tool_use content blocks with id , name , and input fields. You respond with tool_result content blocks that reference the original ID. Anthropic's documentation emphasizes the agent loop pattern: send message, receive tool call, execute, send result, let the model continue. Google's Gemini function calling supports both automatic and manual modes. In automatic mode, Gemini executes the function call and feeds the result back without developer intervention. In manual mode, you get the structured call and handle execution yourself. Google also supports ANY mode, which forces the model to predict a function call, useful when you want tool use to be required rather than optional. The practical differences matter less than they used to.

### [How to Build an MCP Server: A Practitioner's Development Guide](https://swarmsignal.net/mcp-server-development-guide/)

Published: 2026-05-02. Updated: 2026-06-22. Excerpt follows.

The Model Context Protocol has moved quickly from local demos into a common agent-integration pattern. Community servers, SDK usage, and client integrations keep broadening, but the exact scale depends on the source and snapshot date. Treat the direction as clear and the numbers as source-specific. Here's the problem: production readiness remains uneven. Many writeups still describe a large share of the ecosystem as local, experimental, or lightly hardened rather than production-hardened. The gap between "it works on my machine with stdio" and "it handles concurrent requests behind a load balancer without leaking credentials" is where most MCP projects stall. This guide covers the implementation decisions that matter once you're past hello world. What You're Actually Building Before touching code, clarify which MCP primitives your server needs. The current build-server documentation describes three, and they serve different purposes. Tools are functions the model can call to take actions or compute results. They accept inputs, do something, and return outputs. A tool might query a database, call an API, run a calculation, or send a message. Tools are the most common primitive and the one most tutorials default to. Resources expose data the model can read without triggering side effects. Think files, database rows, API responses formatted as context. Resources use URI addressing ( file:// , postgres:// , github:// ) and are meant to be pulled, not called. If your tool is "run a query," your resource is "here's the query result as readable context." Prompts are reusable templates that shape how the model interacts with your server's capabilities. A code-review prompt that bundles the right instructions with a resource reference is a prompt. These are underused and underappreciated: they're how you encode institutional knowledge into the server rather than relying on users to prompt correctly every time. Most MCP servers in the wild implement only tools, ignore resources, and don't know prompts exist. That's fine for simple integrations. It's a missed opportunity for anything more complex. SDK Choice: Python vs TypeScript Both official SDKs are mature. Pick based on where your team lives. Python SDK ( modelcontextprotocol.io/docs ) uses a FastMCP -style interface that integrates into the core package. Define tools as normal functions with type hints and docstrings, and the framework infers JSON schemas from your annotations. This reduces boilerplate significantly compared to the lower-level API: from mcp.server.fastmcp import FastMCP mcp = FastMCP("my-server") @mcp.tool() def get_customer(customer_id: str) -> dict: """Retrieve a customer record by ID.""" return db.query("SELECT * FROM customers WHERE id = %s", customer_id) The docstring becomes the tool description the model sees. The type hints become the input schema. Pydantic handles validation. If your business logic already lives in Python, wiring it to MCP is a few decorators. TypeScript SDK is often used for MCP servers that live alongside Node.js services or front-end tooling. It provides full types for messages, tools, resources, and transports. Zod handles schema validation. For web-adjacent infrastructure (GitHub Actions, CI tooling, browser-accessible services), TypeScript fits naturally. Nearform's production implementation guide identifies one consistent mistake across both SDKs: printing to stdout during stdio transport. When your server uses stdio, the protocol messages travel over stdin/stdout. Any debug logging, print statements, or console.log calls that reach stdout corrupt the message stream and break the client connection silently. Send all logs to stderr, a file, or a structured logging sink. Implementing Tools That Don't Break Tool implementation looks simple until you consider error handling. The MCP spec draws a sharp line between two error categories: Protocol errors indicate the MCP communication itself failed: malformed requests, schema violations, transport issues. These surface as error responses in the JSON-RPC layer. Tool execution errors happen inside your tool's logic: a database connection fails, an API returns a 429, a file doesn't exist. These should not be protocol errors. Return them as valid CallToolResult objects with isError: true and a human-readable explanation of what went wrong. Why does this distinction matter? Because the model sees tool execution errors and can reason about them. If you throw an unhandled exception and let it bubble up as a protocol error, the client breaks and the model has no opportunity to retry with different parameters, escalate, or fall back gracefully. If you return a structured error in the result, the model can read it, decide what to do, and keep the session alive. @mcp.tool() def get_customer(customer_id: str) -> dict: """Retrieve a customer record by ID.""" if not customer_id or not customer_id.startswith("cust_"): return {"error": "Invalid customer_id format. Expected prefix: cust_"} try: result = db.query("SELECT * FROM customers WHERE id = %s", customer_id) if not result: return {"error": f"No customer found with id: {customer_id}"} return result except DatabaseConnectionError as e: return {"error": f"Database unavailable: {str(e)}"} Input validation belongs in the tool, not left to the model.

### [Agent Memory Architecture: Long-Term, Episodic, and Semantic Memory for AI Agents](https://swarmsignal.net/agent-memory-architecture-guide/)

Published: 2026-04-27. Updated: 2026-06-22. Excerpt follows.

Many AI agents still behave like they have the memory of a goldfish with a context window. They can appear intelligent inside a single session, then lose useful context when the conversation ends. You ask the same question the next day and may get the same answer, minus the preferences and corrections you established last time. In many deployments, this is less a model quality problem than an architecture problem. The field has moved fast in the past eighteen months. One useful production framing separates agent memory into four practical categories: working, episodic, semantic, and procedural. The evidence base is still uneven, but each category has distinct tooling patterns, evaluation questions, and failure modes. This guide explains how each type works, what current evidence suggests, and how to pick the right architecture for your use case. Why the old framing falls short For years, discussions about LLM memory collapsed into a single axis: context window size. Bigger context, better memory. That framing worked when agents were chatbots. It doesn't hold when agents run for hours, operate across multiple sessions, or need to maintain consistent beliefs about the world. A 2025 arXiv survey gives one version of that critique: the long-term vs. short-term dichotomy is often too coarse to be useful. The paper proposes framing memory as a write-manage-read loop across mechanism families. For builders, the cautious takeaway is that memory is not only a storage question; it is a data lifecycle question. What gets written? How is it indexed and updated? What gets retrieved, and how does that retrieved content influence action? Evidence caveat: Memory tooling is moving quickly, and many benchmark claims come from papers, vendor reports, or early framework evaluations. Treat the numbers below as decision signals to test against your workload, not universal rankings. As of June 2026, treat the cited benchmarks and product comparisons as point-in-time signals and recheck the current docs before making deployment decisions. The more practical taxonomy that has emerged in production work maps onto four types: Working memory is the active context window. Everything the agent can see right now. No retrieval overhead. Latency is effectively immediate, but capacity varies widely by model and the cost rises with longer contexts. Episodic memory is a timestamped log of past interactions and observations. Closest analogue to human autobiographical memory. Retrieved by similarity or recency. Stored in vector databases or structured logs. Latency is usually modest once the store is in place. Semantic memory is distilled facts and beliefs extracted from past experience. "The user prefers JSON over XML." "This company uses AWS, not GCP." Stored in key-value stores, knowledge graphs, or structured DBs. Latency depends on the retrieval and indexing design. Procedural memory covers learned behaviours, tool preferences, and system-level policies: the agent updating its own instructions based on what worked. Least common in production today. Highest risk profile. "Unlike static RAG where errors are isolated, errors in evolving memory systems are cumulative and persistent." Governing Evolving Memory in LLM Agents, arxiv March 2026 Working memory: faster than you think, cheaper than alternatives Working memory gets underestimated. Engineers reach for external retrieval systems when the problem can often be solved by fitting more into context. The practical ceiling for working memory has risen significantly. Some frontier models now support very large context windows, and cost-per-token for long-context inference has dropped enough that holding a large session history can be cheaper than building and maintaining a vector retrieval pipeline for some use cases. The ceiling problem is not only capacity; it is coherence. Multiple controlled studies and many practitioner reports have found that model attention can degrade toward the middle of very long contexts. The "lost in the middle" effect remains a useful risk to test for, even as newer models improve. Putting critical facts at the start or end of context can still matter. When to lean on working memory: single-session agents, agents with bounded task scope, prototypes, or any situation where you want to avoid retrieval latency and index maintenance overhead. When your agent needs to persist state across sessions or across users, you need something external. Episodic memory: the architecture and the failure modes Episodic memory is where most production agent memory lives. You store interaction logs (raw or summarised) in a vector database, then retrieve them by embedding similarity when the agent needs to recall past context. The architecture is straightforward. The failure modes are not. Summarisation drift is a common production problem. Agents often do not store full transcripts; storage and retrieval costs can make that impractical at scale. Instead, they compress interactions into summaries. Each compression step can lose information. After enough cycles, the summary may no longer accurately represent what happened. The agent can "remember" a version of events that never occurred. Retrieval-action gaps are subtler.

### [IFCMemoryBench Finds the Missing Project Facts](https://swarmsignal.net/ifcmemorybench-missing-project-facts/)

Published: 2026-08-04. Updated: 2026-08-04. Excerpt follows.

IFCMemoryBench turns agent memory into an engineering test: can a building-information assistant remember project facts from earlier sessions and combine them with a live Industry Foundation Classes model later? The July 2026 paper matters because it moves memory evaluation from chat recall into a professional workflow where missing assumptions, units and corrections can change the answer IFCMemoryBench . Key takeaways Main change: agent-memory evaluation is moving from conversational recall to domain-specific project facts. Practical implication: memory systems must preserve assumptions, units, entity references and corrections, not just topically similar notes. Caveat or risk: IFCMemoryBench is synthetic and BIM-specific, so it diagnoses a transfer gap rather than proving every memory layer fails in production. Recommendation: test memory ingestion, retrieval and answer-time use separately before giving agents professional workflow authority. What This Benchmark Actually Tests Building Information Modelling systems use Industry Foundation Classes, or IFC, to represent building geometry, semantics and topology in a software-neutral data model buildingSMART IFC . IFC is the right testbed for this paper because the model can expose building entities and quantities, while the benchmark's missing context is deliberately placed in earlier sessions as specifications, corrections, cost assumptions or external factors IFCMemoryBench . That is the gap IFCMemoryBench targets. It converts incomplete-information questions from IFC-Bench v2 into multi-session tasks where earlier conversations seed the missing project context, then a later probe asks the agent to answer by combining remembered facts with fresh IFC queries IFCMemoryBench . This is close to how professional assistants would actually be used: the same project returns over weeks, and the decisive fact may be a correction or assumption mentioned in passing. IFC-Bench v2 matters because it supplies the structured retrieval side of the problem. The May 2026 paper behind it introduced 1,027 BIM question-answering tasks across 37 IFC models from 21 projects, and argued that adaptive exploration beats static query generation because BIM models vary too much for fixed schemas to hold BIM Information Extraction Through LLM-based Adaptive Exploration . IFCMemoryBench adds the cross-session memory layer on top of that structured-query challenge. The Result The benchmark contains 143 tasks across 19 projects, 23 IFC models and 4,016 prior sessions IFCMemoryBench . The no-memory baseline answers 0.0% of tasks fully, which confirms that the probe cannot be solved from the IFC model alone IFCMemoryBench . The benchmark splits memory into three capabilities: ingestion, retrieval and utilisation IFCMemoryBench . Ingestion asks whether prior sessions are written into durable memory in a useful form. Retrieval asks whether the needed memory can be surfaced at probe time. Utilisation asks whether the agent combines that memory with IFC results instead of answering from one source alone. That decomposition is useful because many agent-memory demos collapse those stages into one success metric. A vector store can retrieve something relevant and still miss the one project fact that changes the answer. A graph can preserve entities but fragment the assumption across edges. A Markdown memory file can keep the right note but leave the agent unsure whether it should trust memory, IFC, or both. The results show that failure clearly. With tested memory systems, answer accuracy remains low: Graphiti reaches 21.0%, file memory without citations reaches 25.2%, cited file memory reaches 29.4%, and Mem0 reaches 32.4% in the main comparison IFCMemoryBench . The issue is not whether the systems can remember anything. It is whether they preserve complete, usable project knowledge IFCMemoryBench . The Long-Context Escape Hatch Is Limited The obvious objection is that a long-context model could just read all prior sessions. IFCMemoryBench includes that sanity check: a full-context oracle using all prior user messages reaches 83.2% answer accuracy IFCMemoryBench . The oracle result shows that the task design usually contains enough prior-session evidence for a correct answer IFCMemoryBench . It does not prove that memory systems are unnecessary. The paper notes that indiscriminate full-turn ingestion can reach about 167,000 tokens per task, while production projects accumulate months of history across many users IFCMemoryBench . Swarm Signal covered the same architectural boundary in million-token context still fails the workload test : larger windows help, but they do not remove ranking, authority, freshness or cost constraints. The practical lesson is narrower. Long context is a useful upper bound for debugging memory failures. It tells you whether the answer exists in the record. It does not give you a cheap operating model for live project work, where every query cannot reload every chat, tool trace, model extract and specification. What Transfers To Production The direct evidence transfers to BIM retrieval systems first. IFCMemoryBench uses synthetic but human-validated tasks built from IFC-Bench v2, so it is strongest as a diagnostic for assistants that query IFC models and rely on project context outside the model IFCMemoryBench . The broader transfer is a test design, not a universal score.

### [ScrambleToolBench Finds Tool-Map Inertia](https://swarmsignal.net/scrambletoolbench-tool-map-inertia/)

Published: 2026-08-13. Updated: 2026-08-13. Excerpt follows.

ScrambleToolBench is a 3 August 2026 benchmark for a practical tool-use failure: a system can learn what hidden tools do, then fail to revise that map when the environment changes ScrambleToolBench . The paper matters because many production failures are not caused by missing API documentation. They are caused by stale assumptions after an integration, permission, schema or execution window has shifted. Key takeaways Main finding: initial tool discovery does not reliably turn into robust adaptation after mapping drift ScrambleToolBench . Practical implication: tool tests should mutate behaviour after the system has formed a working map, not only check the first successful call. Caveat or risk: the benchmark uses a simulated terminal environment, so it is strongest as a diagnostic for reasoning and adaptation rather than a direct forecast for every API stack ScrambleToolBench README . Release decision: add a tool-drift case to any workflow that caches tool meaning, routes by learned affordances or retries after failures. What This Benchmark Actually Tests ScrambleToolBench removes semantic cues from tool names and parameters so the tested system must infer behaviour through interaction in an interactive terminal benchmark ScrambleToolBench . Its repository describes the same setup: tool names and parameters are obfuscated, and the system has to probe hidden behaviour across a continuous task curriculum ScrambleToolBench README . That separates two capabilities that are often merged in ordinary evaluations. The first is discovery: can the system work out what an unfamiliar command does? The second is adaptation: can it update that working model when a tool rotates, starts failing stochastically or becomes available only inside a narrow execution window ScrambleToolBench ? The paper reports that successful initial discovery did not translate into robust adaptation. Under mapping drift, tested systems tended to show belief inertia or fall back to exhaustive search rather than use deductive strategies such as cycle tracing ScrambleToolBench . That makes the benchmark a useful companion to Swarm Signal's earlier coverage of tool state diffs and runtime receipts . A receipt can show what happened. A drift test shows whether the system noticed that its previous map had become unsafe. Bigger Reasoning Budgets Can Amplify Brute Force The uncomfortable result is not only that systems struggled with drift. The authors state that increasing test-time reasoning amplified expensive brute-force search rather than enabling deductive recovery ScrambleToolBench . In other words, more thinking time did not automatically produce a better hypothesis update. This fits a wider pattern in agent evaluation. General AgentBench studies search, coding, reasoning and tool-use domains in one environment, then reports that neither sequential scaling nor parallel sampling delivered effective performance improvements in practice because of context ceilings and verification gaps General AgentBench . ScrambleToolBench narrows the reason in a tool setting: the system may spend extra steps exploring again when it should be testing the smallest hypothesis that explains the changed map ScrambleToolBench . For operators, the test is simple. If a workflow's answer to tool drift is "retry harder", the system may look more persistent while becoming less efficient. The useful behaviour is not more calls. It is a short differential probe that asks what changed, checks the old map against one or two targeted observations, and updates the routing rule before continuing. Static Schema Tests Miss The Important Case Many tool-use evaluations expose descriptive schemas in static environments. ScrambleToolBench argues that this lets systems lean on prior semantic knowledge rather than autonomous discovery ScrambleToolBench . That is not a small detail. Production tools are often wrapped, renamed, proxied or partially documented; the name alone may be a weak guarantee of behaviour. AgentProcessBench reaches the process-quality problem from another angle. Its June 2026 version contains 1,000 tool-augmented trajectories and 8,509 human-labelled step annotations, and its authors report that distinguishing neutral and erroneous actions remains difficult for current models AgentProcessBench . ScrambleToolBench adds a dynamic layer: even when an action was useful earlier, a later mapping change can turn the same action choice into the wrong one ScrambleToolBench . That is why a release test should include both step labels and environment mutations. Step labels ask whether the current action is sensible. Drift cases ask whether the system is still acting from the current environment rather than from yesterday's inferred map. The Tool-Drift Gate To Add Before trusting a tool-using workflow, run a small drift gate: hide or alias tool names so the system has to infer behaviour; require it to solve a short sequence from observations, not documentation; rotate a subset of tool mappings after initial success; introduce one stochastic failure that should not change the inferred map; introduce one temporal window where the correct action expires; score the update path, not only the final task result. The key record should include the old hypothesis, the new observation, the chosen probe and the updated tool map.

### [MCP Server Architecture: Tools and Safe Invocation](https://swarmsignal.net/mcp-server-architecture-guide/)

Published: 2026-04-06. Updated: 2026-09-08. Excerpt follows.

An MCP server gives an AI application access to capabilities outside the model. A useful design starts with the action you want to permit: read an issue, search a document collection or propose a change. It then defines who may perform that action and what evidence will show that it worked. This guide uses a hypothetical issue-tracker integration. The architecture and transport descriptions follow the linked, dated MCP specifications; the design choices and acceptance cases are our engineering recommendations. They are not a benchmark or a claim that an example service has passed a security audit. Keep the host, client and server separate The host is the AI application. It coordinates the model, user interaction and permission decisions. Its MCP clients connect to servers. A server exposes capabilities, such as issue lookup, without becoming the model itself. The MCP architecture specification describes these responsibilities and capability negotiation. For our example, draw the request path as: User -> AI host -> MCP client -> issue MCP server -> issue API | | access policy source of truth Give the server a narrow responsibility. Keep issue access separate from deployment operations unless there is a concrete reason to combine them. This makes credentials, ownership and failure handling easier to inspect. A single generic tool that accepts arbitrary HTTP requests makes those boundaries much harder to enforce. A tool description explains an action. Server-side permission checks decide whether it may happen. Choose the primitive that fits the work Use a tool for an operation with structured arguments and a result. In this example, search_issues accepts a project and search terms; propose_issue_update returns a proposed change for review. A read-only operation can still be a tool. Tools are not limited to writes. The tools specification defines discovery, invocation, input schemas and tool errors. Use a resource when the application needs addressable context, such as an issue description or project policy. Decide how the host will select that context and how the server will authorise the read. A resource URI is an identifier, not permission to retrieve everything behind it. See the resources specification . Use a prompt for a reusable interaction, such as reviewing an issue against a project checklist. A prompt can guide the workflow, but it cannot make an unauthorised update safe. Keep policy enforcement out of prose and in executable checks. The prompts specification covers the protocol shape. Design the smallest useful tool contract Start with get_issue(project, issue_key) . Require both arguments, reject unknown fields and return only the fields the user needs. Resolve the authenticated identity from trusted request context. Do not accept an identity supplied in the tool arguments as proof of authority. An implementation should evaluate the following sequence before contacting the issue API: Validate the argument types and permitted identifier format. Resolve the caller and their allowed projects. Check access to the requested project and issue. Call the upstream API with appropriately restricted credentials. Return a bounded response with the issue identifier and source link. Keep these checks in ordinary application code that can be tested without a model. Let the model choose between permitted actions; do not make it responsible for recognising a forbidden project. Tool annotations and descriptions are useful hints, but they do not replace enforcement. For writes, separate proposal from execution. Bind approval to the actual proposed change and the issue revision it was based on. If the issue changes before execution, ask for a fresh review rather than silently applying an outdated patch. Record the resulting upstream identifier so an operator can investigate an ambiguous timeout. Pick a transport for the deployment For a locally launched process, stdio keeps the integration straightforward. Send protocol messages to standard output and diagnostics to standard error. Printing a startup banner to standard output can break the connection. For a shared remote service, inspect Streamable HTTP support in the actual host and SDK you intend to deploy. The transport specification describes both options and the HTTP origin checks. A local process still inherits the permissions and credentials you give it. Restrict its filesystem and network access. For a remote protected server, use the MCP authorisation specification and validate that incoming tokens are intended for your server. Do not pass the incoming token unchanged to an unrelated upstream API. Authentication identifies the caller; project-level authorisation still belongs in your application. Run these acceptance cases before release Use a test project containing synthetic issues. These are concrete acceptance cases to implement against your server, not results from a server tested for this article. Case Action Required observation Allowed read Authorised user requests an issue in their project. Correct issue returned; no unrelated fields disclosed. Cross-project read Same user requests a restricted project. Request denied before an upstream data read.

### [MCP vs A2A vs ACP: Which Integration Do You Need?](https://swarmsignal.net/mcp-vs-a2a-vs-acp-agent-protocol-comparison/)

Published: 2026-03-29. Updated: 2026-09-08. Excerpt follows.

Choose the protocol by the boundary you need to cross. MCP connects an AI application to tools and context. A2A supports communication with another agent. ACP needs its full name: Agent Communication Protocol and Agent Client Protocol are different projects. The Agent Communication Protocol documentation now says that the project is part of A2A. Agent Client Protocol addresses communication between coding agents and editors. Treating all these names as competing enterprise message buses leads to the wrong architecture decision. This comparison replaces an earlier version containing unsupported release numbers, performance figures, commercial pricing and ownership claims. The recommendations below are based on the linked official documentation and explicit engineering trade-offs. We have not run a comparative latency benchmark. Start with the integration you need Your requirement Start here What to verify Give an assistant controlled access to issue search or a database. MCP Host support, tool schemas, resource access and application permissions. Delegate a task to an independently operated agent. A2A Agent discovery, task lifecycle, authentication and result handling. Support a coding agent from an editor. Agent Client Protocol Editor-agent compatibility, permission requests and session behaviour. Maintain an existing Agent Communication Protocol integration. Its A2A migration guidance Existing runs, messages and client expectations before migration. Call an internal function inside a single application. Your existing application interface Whether adding a protocol solves an actual integration problem. Choose the boundary first. A protocol name cannot decide who is allowed to act. MCP: expose capabilities to an AI application In MCP, the host application coordinates the model and its client connections. Servers expose tools, resources and prompts. The server is not simply the AI model, and initialisation is not a fixed manifest containing every callable operation. Capability negotiation and subsequent discovery are separate concerns. The architecture and tools specifications describe this division. For an issue-tracker assistant, an MCP server might expose search_issues and get_issue . The host can use these while composing an answer. If you expose an update tool, your application must enforce permissions and any approval requirements before the update reaches the issue tracker. The linked transport specification defines stdio and Streamable HTTP. SSE can carry streamed server messages within the HTTP transport; it is not a substitute for the client sending its request. Check the selected host and SDK together rather than assuming that supporting MCP means supporting every optional capability. For protected HTTP integrations, follow the authorisation specification . Receiving a token does not settle whether its holder may read a particular customer record. Keep those checks close to the data access. Our MCP server architecture guide turns this into an implementation and acceptance checklist. A2A: delegate work across an agent boundary A2A is useful when the remote party manages how a task is completed. You request an outcome and handle its messages, state and returned artefacts without requiring access to its internal tool definitions. The official A2A and MCP comparison treats the protocols as complementary. Consider a support assistant asking a specialist diagnostic agent to investigate an incident. Define what information it can send, what constitutes an acceptable result and what happens if the specialist needs clarification. A completed protocol task is still something your application must evaluate: it is not proof that the diagnosis is correct. The A2A specification defines Agent Cards, task and message structures, and protocol bindings including JSON-RPC, gRPC and HTTP+JSON. An Agent Card describes the agent interface and capabilities. It does not require every deployment to run a distributed registry, nor does it certify that an agent is trustworthy. Use the authentication mechanisms declared by the service and enforce authorisation at the service boundary. Test task retrieval, cancellation and interrupted communication against the binding and features your counterpart actually implements. Do not assume that a successful initial exchange proves that a long-running workflow can recover after a disconnect. ACP: spell out which project you mean Agent Communication Protocol provided a REST-based interface for communicating with agents. Its official welcome page explicitly announces its move into A2A and links to migration guidance. For an existing integration, use that guidance to examine the mapping of runs, messages and state. For a new agent-to-agent integration, evaluate A2A directly instead of treating the older ACP project as a separate future enterprise stack. Agent Client Protocol standardises the interface between coding agents and editors. Its introduction describes local subprocess communication using JSON-RPC over stdio and remote communication over HTTP or WebSocket. Check the supported transport and features in your chosen editor and agent before deployment. This protocol is relevant when a user wants to operate a coding agent from their editor, including the surrounding interactive experience. A coding agent can expose an editor-facing interface while using MCP tools internally. An agent accepting delegated work can also use MCP.

### [RAG vs Fine-Tuning vs Long Context: A Practical Guide](https://swarmsignal.net/rag-vs-fine-tuning-vs-long-context-2026/)

Published: 2026-03-19. Updated: 2026-09-08. Excerpt follows.

Start with a question your system gets wrong. Did it miss the relevant document, overlook evidence already in its prompt, or fail to apply the required format? These failures need different repairs. Retrieval-augmented generation supplies selected evidence at inference time. Long-context prompting supplies a larger body of material directly. Fine-tuning changes model behaviour through training examples. They can be combined, but each addition should earn its place in a measured evaluation. There is no universal corpus-size boundary or architecture winner. When retrieval is worth testing Retrieval is a useful candidate when requests need different subsets of a changing document collection. It lets the application search for material, attach source identifiers and pass selected passages to the model. The retrieval pipeline is itself a possible source of failure. A relevant passage may never reach the generator because of document parsing, chunk boundaries, filters, query wording or ranking. Log the retrieved evidence separately from the answer so you can distinguish these problems. Anthropic's contextual retrieval explanation describes how isolated chunks can lose important context and proposes adding explanatory context before indexing. Treat that as a technique to test, not a reason to rebuild a retriever that already meets your requirements. Freshness is an operational property. Updating a source document does not instantly update every index, cache and answer. Test the time from an approved source change to a correct answer, including deletion and access revocation. Fix the failure you can observe before adding another part to the system. When longer context is worth testing If the documents needed for a task fit within the selected model's supported input budget, passing them directly can be a useful baseline. It avoids a retrieval stage that might discard necessary evidence. Fitting is not the same as answering correctly. Test whether the model can locate facts, reconcile conflicting passages and cite the right version. Put decisive evidence in different positions rather than always at the beginning. Include enough surrounding material to resemble actual usage. Check the chosen endpoint's current context limits, output allowance, pricing and rate limits. Do not assume a provider's largest advertised window applies to every model or account. Budget for instructions, tool definitions and generated output as required by that endpoint. Measure the full request under cold and warm cache conditions. A short demonstration cannot tell you what a busy service will cost or how quickly it will respond. Longer input, retrieval latency, output generation and queueing all belong in the comparison. When fine-tuning is worth testing Fine-tuning is a candidate when repeated examples can teach a stable behaviour: applying a classification scheme, following a house style or producing an expected response pattern. OpenAI's supervised fine-tuning guide describes example-based training and checking the result against evaluations. It can also change factual recall, so saying that fine-tuning cannot teach facts is inaccurate. However, a trained model is not a document store with straightforward citation, deletion and update semantics. For changing, attributable knowledge, test supplying current evidence at inference time. Establish a prompt baseline first. Keep a held-out evaluation set and compare against it after training. Include examples outside the narrow target pattern to catch regressions. Do not make a fixed number of examples or a hardware price into a universal starting requirement. Output formatting also needs application validation. Training alone does not guarantee a valid schema, correct citation or safe action. Validate the result before a downstream system relies on it. Use a workload test matrix The following is a proposed experiment, not reported benchmark performance. Run each viable approach against the same questions and source versions. Workload Test material Acceptance criterion Exact lookup Answer present alongside similar but incorrect passages Correct answer and supporting source Cross-document comparison Facts distributed across documents All required facts reconciled without invented links Freshness A superseded policy and its replacement Current version used; old version identified if relevant Missing evidence A plausible question with no answer in the permitted corpus Clear abstention or request for evidence Permissions Relevant material outside the user's allowed sources No unauthorised material disclosed Format Valid and deliberately awkward inputs Output passes the application's validation Load Representative short and long requests arriving together Quality and latency remain within your agreed budget For retrieval, record whether the evidence was retrieved and whether the final answer used it correctly. For long context, record the supplied documents and their order. For fine-tuning, record the base model, training dataset version and prompt. This makes a failed answer diagnosable. Compare cost without hiding work Include preprocessing, indexing, storage, training, generation, retries and review where applicable. Separate one-off preparation from recurring costs, and state the period over which you allocate preparation costs. Use total cost divided by accepted tasks alongside task success and completion latency. A cheap answer that needs correction is not equivalent to an accepted answer.

### [Knowledge Graphs for AI Agents: A Practical Guide](https://swarmsignal.net/knowledge-graphs-for-ai-agents/)

Published: 2026-06-11. Updated: 2026-09-08. Excerpt follows.

A knowledge graph is useful when your application needs to query explicit relationships: which service depends on a component, who owns it now, or which evidence supports a claim. It does not automatically make an agent accurate, current or safe. Start with a question your existing retrieval system handles badly. Inspect the missing evidence before choosing a database. If the problem is stale documents, poor access filtering or ambiguous names, adding graph storage alone will preserve those problems in another format. What graphs add to retrieval A graph represents entities and their relationships explicitly. In an RDF representation, statements have a subject, predicate and object; the object can also be a literal value. The W3C specification defines that model. Property graphs use their own node, relationship and property conventions. That structure lets an application follow a defined relationship instead of relying entirely on text similarity. A service can depend on a library whose maintainer belongs to a particular team. The traversal is useful only if those relationships are correctly recorded. Vector retrieval finds candidates by similarity. A complete retrieval system can also apply metadata filters, combine keyword search, run repeated searches or join structured records. Weaviate's filter documentation , for example, describes property filters and combined conditions. It is therefore misleading to say vector systems cannot handle dates or support questions spanning documents. The practical distinction is where you represent and maintain the relationships. Sometimes a relational table and a join are sufficient. Sometimes graph traversal makes the required paths easier to express. Neither choice prevents a language model from misreading the retrieved evidence. A useful graph preserves the identity and evidence behind a relationship. Choose a narrow question Consider an internal engineering assistant asked: “Who maintains the library used by this service?” Define the expected path from service to dependency to maintainer. Decide what counts as a dependency: an imported package, a deployed service or a declared build requirement. Then inspect the records. Does the package have a stable identifier? Is ownership recorded in an authoritative source? Are historical assignments distinguishable from current ones? Can the requester read the ownership record? These questions expose the ingestion and permission work before a graph demonstration makes retrieval look solved. For a broader architecture comparison, use the retrieval, fine-tuning and long-context guide . Resolve identity before adding edges Display names are weak identifiers. Different services may share the name “Atlas”, while the same service appears under a repository name, an internal alias and a deployment label. Prefer an authoritative identifier scoped to its system, such as a repository ID with its hosting organisation. Keep labels and aliases as attributes. An extracted name should resolve to a candidate entity only when the available evidence supports that match. If several candidates remain, return an ambiguity for review instead of silently merging them. Record the source of a proposed relationship and the extraction method. A fluent extraction is not confirmation that the source states the relationship. Review disputed assignments, unsupported links and accidental entity merges before making them available to an agent. A small, runnable example The following synthetic example separates identical labels by namespace, closes an old maintainer assignment and checks the caller's current permissions before returning a relationship. It uses the Python standard library and deliberately avoids embedding models and database services. Save it as graph_memory.py , then run python3 graph_memory.py . from datetime import date entities = { "catalog:atlas": ("catalog", "Atlas"), "analytics:atlas": ("analytics", "Atlas"), } def resolve(namespace, label): matches = [key for key, value in entities.items() if value == (namespace, label)] if len(matches) != 1: raise ValueError("Entity is missing or ambiguous") return matches[0] def current_owner(rows, entity, day, allowed): # allowed comes from the current authenticated permission check. return [r["owner"] for r in rows if r["entity"] == entity and r["source"] in allowed and r["start"] <= day and (r["end"] is None or day < r["end"])] entity = resolve("catalog", "Atlas") rows = [{ "entity": entity, "owner": "Platform", "source": "private-roster", "start": date(2026, 1, 1), "end": None, }] changed = date(2026, 6, 1) rows[0]["end"] = changed rows.append({ "entity": entity, "owner": "Infrastructure", "source": "private-roster", "start": changed, "end": None, }) print(current_owner(rows, entity, changed, {"private-roster"})) print(current_owner(rows, entity, changed, set())) The output is ['Infrastructure'] followed by [] . The old assignment remains available for a historical query, but is excluded on the change date. Someone without permission to read the source receives no owner. The complete example and tests also exercise historical queries, name ambiguity, a different entity with the same label and permission revocation. Run that file with --test to reproduce the checks. This is a teaching example, not an access-control service. Production code must obtain identity and permissions from a trusted service, protect writes and make updates atomic. It must prevent disclosure through intermediate results, summaries, cached responses and error messages as well as the final answer.

### [Pinecone vs Weaviate vs Qdrant vs Chroma (2026)](https://swarmsignal.net/vector-database-comparison-2026/)

Published: 2026-03-17. Updated: 2026-09-08. Excerpt follows.

Choose a vector database around the retrieval work your application needs to perform: eligible results, useful ranking, acceptable response time and an operating model your team can support. A headline vector count or a latency number from a different workload cannot make that decision for you. This comparison covers Pinecone, Weaviate, Qdrant and Chroma, then examines filtering in pgvector, Azure AI Search and Pinecone. It uses first-party documentation linked beside the relevant capabilities. It does not rank these systems using a shared performance test. The practical recommendations below are a way to build a shortlist, not a claim that one engine is universally fastest or cheapest. At a glance System Documented approach Useful reason to evaluate it Check before committing Pinecone Managed vector and document search; dense, sparse and hybrid retrieval You want to consume a search service rather than operate its database engine API and index compatibility, workload cost, required deployment model Weaviate Managed and self-hosted deployments; hybrid keyword/vector retrieval You need configurable fusion of lexical and semantic results Your relevance examples, filters, resource needs and operational responsibilities Qdrant Local and managed deployments; dense/sparse query composition You want control over a dedicated retrieval service and its query pipeline Recall under your filters, tuning effort, backups and capacity Chroma Local, single-node and distributed deployment modes You want an embedded starting point with a documented distributed option Persistence, chosen deployment mode and its limits Sources for the capabilities in this table: Pinecone indexes , Pinecone hybrid search , Weaviate deployment , Weaviate hybrid search , Qdrant quickstart , Qdrant hybrid queries , and Chroma architecture . Pinecone: managed search with deployment choices to check Pinecone's index documentation describes dense vectors, sparse vectors and document schemas. Its hybrid-search guidance covers combining retrieval signals. That gives you several ways to represent a search problem; choose the index/API combination your application actually needs rather than assuming every feature works with every existing index. Pinecone Local is an in-memory Docker emulator for development. It is not a persistent production deployment and does not establish cloud performance. Pinecone also documents a Bring Your Own Cloud option . Check its requirements instead of treating managed service and an independently operated open-source engine as interchangeable choices. Pinecone belongs on your shortlist when operating a database is work you want the provider to handle. You still own data modelling, retrieval evaluation and the application's access rules. A managed engine does not remove those responsibilities. Weaviate: configurable hybrid retrieval Weaviate combines keyword BM25F results with vector results and exposes fusion and weighting controls in its hybrid-search API . This is useful when your queries mix meaning with precise identifiers, product codes or specialist terms. Try queries where lexical and semantic retrieval disagree. Inspect which results each component contributes before changing the weight. A hybrid API makes that experiment convenient; it does not prove the resulting ranking suits your corpus. Deployment documentation covers managed and self-hosted choices. Evaluate the operational model alongside relevance. This guide does not establish comparative garbage-collection latency, memory usage or a performance disadvantage against Rust implementations. The programming language alone is insufficient evidence for those claims. Qdrant: compose and measure your retrieval pipeline Qdrant's query documentation describes staged retrieval and fusion, including dense and sparse searches. Compare the exact query behaviour you need rather than treating support for hybrid search as a yes/no feature. The local quickstart uses a container and a client library. Running it does not require writing Rust. Operating any production deployment still means deciding who handles access, persistence, upgrades and recovery. A managed cloud option changes that division of work. Evaluate Qdrant when a dedicated retrieval service and explicit query composition fit your stack. Measure it on your own data before calling it the lowest-latency or lowest-cost option. This article has no common test that would support either ranking. Chroma: distinguish the deployment modes Chroma's architecture documentation distinguishes an embedded local library, a single-node server and distributed deployment. Chroma Cloud is its managed distributed offering. It is therefore inaccurate to describe Chroma as inherently single-node or something every successful prototype must outgrow. Start by choosing the mode that matches your application. An ephemeral local experiment and a durable multi-service deployment have different operating properties even when the client API looks familiar. Verify persistence and recovery in the mode you intend to use. Chroma is worth evaluating for a local development workflow. Its distributed option also deserves assessment on its actual capabilities, rather than being dismissed because the embedded mode is easy to install. No large-scale Chroma workload was run for this comparison. A filter can remove an unsafe result without finding the useful result underneath it. Compare filtering in pgvector, Azure AI Search and Pinecone Filtering is where a general product comparison becomes an implementation decision.


## Evaluate reliability and operating cost

### [How to Build Agent Evals That Catch Real Failures](https://swarmsignal.net/agent-evals-production-failures/)

Published: 2026-04-25. Updated: 2026-06-22. Excerpt follows.

Your agent passes the benchmark. It scores well on tool-call accuracy. Your unit tests are green. Then it deletes the wrong records in production. This is the reliability gap agent teams have to design around: benchmark performance can look clean while production behavior remains brittle. One 2025 analysis of agent benchmarks argues that several widely used benchmarks have validity and cost-estimation problems. Read that as a bounded warning about benchmark coverage, not proof that all benchmarks fail in the same way. Public benchmarks often emphasize cleaner task slices than the messy parts of agent behavior that matter in production. This guide explains why standard evaluation methods can break for agents, what failure modes they often miss, and how to build an eval system that is more likely to correlate with production quality. Why LLM Evals Don't Transfer to Agents Static LLM evaluation was designed for single-turn responses: prompt in, answer out, score against expected output. Agents break that model in three structural ways. Compositional failure. A single agent run can involve dozens of sequential decisions: which tool to call, in what order, with what arguments, based on what context. Even when individual tool calls look valid in isolation, the run can still produce a wrong result if the steps do not jointly satisfy the task and policy constraints. A customer service agent can execute valid API calls while still mishandling policy edge cases. Final-output scoring may not catch this. Cascading errors. Small upstream mistakes propagate. A wrong timestamp, a malformed identifier, an incorrect assumption in step two gets referenced in steps four, six, and eight. By the time the agent produces a final response, the original error is invisible in the output log. Standard benchmarks check the endpoint. The failure is in the path. Shallow task design. Most existing benchmarks test "isolated API functionalities, few-step workflows, and artificial task compositions," as MCP-Bench's designers noted when explaining why they built an alternative . They don't test strategic planning across real multi-domain operations. An agent that passes these benchmarks has demonstrated it can operate in a controlled environment. That's not the same as production. The result: developers get confident signals from evals that measure the wrong things, and get surprised when agents fail in the field. What Benchmarks Currently Miss Understanding the specific failure modes benchmarks skip is the starting point for building better evals. Context Window Degradation Berkeley Function-Calling Leaderboard data shows every model performs worse when given more than one tool. The degradation isn't linear. Past a model-specific threshold, more context increases variance and raises hallucination risk; the performance doesn't decline gradually, it collapses. Most tool-use benchmarks test agents with a handful of tools in a clean context. Production agents run with dozens of tools, mid-conversation, after five prior exchanges. Tool Call Hallucination and Context Poisoning Agents confidently call non-existent API endpoints and pass wrong parameters. The compounding problem: when a hallucinated call enters the conversation context ("the order was placed successfully"), subsequent steps treat it as ground truth. This is context poisoning: the error propagates laterally through the execution trace, and the final output looks plausible. No benchmark tests for this specifically. That makes memory design part of evaluation design. How Agent Memory Got an Architecture covers the long-term, episodic, and semantic memory choices that change what an eval has to inspect. Policy Violation on Edge Cases Sierra Research's τ-bench , which tests agents in realistic retail, airline, and banking domains, exposed a finding that wasn't visible in simpler benchmarks: agents fail not on tool calls themselves but on policy edge cases and ambiguous user intent. When a customer's request sits at the boundary of what policy allows, agents improvise, sometimes in ways that would cause legal or compliance problems. The follow-on τ²-bench extends this to dual-control environments where both agent and user are tool-using, testing scenarios existing benchmarks don't cover at all. Scope Creep Agents do more than instructed. Standard benchmarks with fixed expected outputs don't score on over-action. In production, scope creep causes data deletion, unintended API side effects, and unauthorized resource access. If your eval only checks "did the right thing happen," it won't catch "did some unintended things also happen." The Three-Layer Evaluation Architecture The most practically useful framework for agent evals organizes assessments into three layers, each catching failures the others miss. Layer 1: Node-Level Precision Evaluate individual steps in isolation. Metric What it measures Tool selection accuracy Did the agent pick the right tool for this step? Argument correctness Were tool arguments valid and semantically appropriate? Step utility score Did this step move toward the goal, or was it redundant/wrong? Node-level evals are fast to run and easy to automate. They catch the obvious failures: wrong tool, malformed parameters, hallucinated function names.

### [Agent Observability Is Escaping the Dashboard](https://swarmsignal.net/agent-observability-is-escaping-the-dashboard/)

Published: 2026-07-25. Updated: 2026-08-01. Excerpt follows.

Agent Observability Is Escaping the Dashboard

In June 2026, agent observability is moving from vendor dashboards into trace contracts OpenAI Agents SDK tracing OpenInference semantic conventions . OpenAI's Agents SDK records LLM generations, tool calls, handoffs, guardrails, and custom events in traces, while OpenInference defines separate span kinds for LLM, retriever, tool, agent, guardrail, evaluator, and prompt steps OpenAI Agents SDK tracing OpenInference semantic conventions . The practical signal is simple: builders are starting to expect the whole agent run, not only the final answer, to be inspectable.

Key takeaways

The useful unit of analysis is shifting from a model call to a run trace.

Schema choice matters because tracing conventions are still split across projects.

Evaluation should attach to traces, not sit in a separate spreadsheet.

Instrumentation without redaction is operational debt.

The signal

OpenAI says tracing is enabled by default in its Agents SDK and captures a record of events during an agent run, including model generations, tool calls, handoffs, guardrails, and custom events OpenAI Agents SDK tracing . OpenInference's specification breaks the trace into operation types such as LLM, embedding, chain, retriever, reranker, tool, agent, guardrail, evaluator, and prompt spans OpenInference semantic conventions .

OpenTelemetry's GenAI agent conventions are marked Development, but the draft already names create-agent, invoke-agent, workflow, plan, and execute-tool spans OpenTelemetry GenAI agent spans . That matters because the agent stack is becoming too stateful for a single "request succeeded" metric to explain much.

Evidence

LangSmith frames observability as visibility from individual traces to production-wide performance metrics, with integrations across OpenAI, Anthropic, CrewAI, Vercel AI SDK, Pydantic AI, and other stacks LangSmith Observability . Braintrust's OpenAI Agents SDK integration says it captures root task spans, child spans for tool calls, guardrails, handoffs, nested model work, inputs and outputs, token metrics when exposed, and parent-child relationships Braintrust OpenAI Agents SDK .

The strongest industry signal is not another dashboard. Datadog says OpenTelemetry GenAI conventions give teams one schema for prompts, model responses, token usage, tool and agent calls, and provider metadata, then route those spans through an existing OpenTelemetry Collector path Datadog OTel GenAI support . In plain terms: agent telemetry is becoming something platform teams can govern before it leaves the network.

Why it matters

Agent failures usually hide in sequence, not output. The final answer can look fine while the retriever selected the wrong document, the tool call sent the wrong argument, or a handoff lost policy context.

That is why this belongs next to Swarm Signal's work on agent evals that catch production failures and agent memory architecture . Evals tell you whether the behaviour was acceptable. Traces tell you where the behaviour came from.

The caveat is privacy. The OpenTelemetry GenAI registry warns that captured message and tool-call attributes can contain sensitive or personal data, and says content capture should be gated by explicit opt-in OpenTelemetry GenAI attributes . A trace that records everything without filtering is not observability. It is a liability log.

What changes

Do not ask only whether the model answered correctly. Ask whether the run trace explains why each step happened, which inputs each step saw, which tool call changed state, which guardrail fired, and which evaluator judged the result.

For builders working through deployment patterns , the practical move is to instrument the agent boundary before optimising prompts. Start with run IDs, model spans, tool spans, retrieval spans, guardrail spans, evaluator spans, token counts, and redaction policy. Add dashboards later.

Operator takeaway

If you are building this system, do this:

One practical action: choose an OpenTelemetry or OpenInference-compatible trace shape before adding a proprietary dashboard.

One thing to measure: trace completeness across model, retriever, tool, guardrail, and evaluator steps.

One thing to avoid: storing raw prompts, retrieved documents, or tool arguments without an explicit redaction rule.

One decision gate: no agent reaches production until a failed run can be reconstructed from trace data.

Source trail

Primary specifications and docs

OpenAI Agents SDK tracing

OpenInference semantic conventions

OpenTelemetry GenAI agent spans

OpenTelemetry GenAI attributes

Industry implementation signals

LangSmith Observability

Braintrust OpenAI Agents SDK

Datadog OTel GenAI support

Related Swarm Signal analysis

Agent evals that catch production failures

Agent memory architecture

Deploying agents to production

### [Messier Maps Agent Benchmark Drift](https://swarmsignal.net/messier-agent-evaluation-corpus/)

Published: 2026-09-06. Updated: 2026-09-06. Excerpt follows.

Messier, revised on 30 August 2026, treats agent evaluation as a data-integration problem rather than another leaderboard race Messier . The paper matters because its authors identify fragmentation across tasks, scaffolds, verifiers and scoring rules as a core obstacle to comparing tool-using systems Messier . Key takeaways Main result: Messier turns fragmented agent benchmark results into a shared corpus with task, scaffold, verifier and scoring-rule fields Messier . Practical implication: evaluation teams can inspect benchmark drift by task, verifier, action space and occupation instead of treating each leaderboard as a separate island Messier . Caveat: a common corpus improves comparability, but it does not prove that any single benchmark measures the intended capability in a deployed product HackDetect . Decision: build an evaluation map before buying or shipping an agent, then decide which local tests still need fresh runs. What This Benchmark Actually Tests Messier does not ask a fresh set of tasks and declare a winner. It consolidates the evaluation record behind many agent benchmarks, then makes the surrounding machinery inspectable Messier . Its corpus combines public evaluation results with new runs on underrepresented professional and scientific benchmarks, then standardises records by model, scaffold, environment, task, verifier and aggregation rule Messier . The headline scale comes later in the paper: the abstract reports 957,611 records across 30 benchmarks, 745 agents, 11,891 tasks and 74,263 verifiers Messier . That makes the paper closer to a map of the evaluation market than another score table because it exposes the components around each result, not only the final metric Messier . Why scores now need corpus context The paper reports uneven frontier progress across benchmark groups: function-calling evaluations are largely saturated, programming is improving fastest, and enterprise workflows remain the most challenging Messier . That distribution should matter to teams that have watched a tool demo pass simple API calls and then fail when the workflow involves documents, policies and state. Swarm Signal's VAKRA analysis covered that same hand-off problem inside API and retrieval workflows. Why common schema beats another leaderboard A common schema matters because agent evaluation is not a static question-answer test. The wrapper around the model changes behaviour. Efficient Benchmarking of AI Agents studies eight benchmarks, 33 agent scaffolds and more than 70 model configurations, then argues that scaffold-driven distribution shift weakens absolute score prediction even when rank-order prediction remains more stable Efficient Benchmarking . That finding changes how to read Messier. The useful output is a way to ask narrower questions: which scaffold was used, which verifier decided success, which action space was available, and whether the task resembles the work in front of the buyer Messier Efficient Benchmarking . Messier's reported capability scores correlate with Epoch's Evaluation Capability Index rankings at Spearman rho 0.84, while Epoch describes its index as a way to combine many benchmark results into a single capability scale Messier Epoch . That correlation is useful, but the stronger operational point is the subset capability. Messier says scores can be specialised by domain, occupation, action space or verifier type Messier . For a production team, that means the first question is no longer "which model wins?" It is "which slice of the evidence resembles our workflow?" Rescoring can change the story Messier's counterfactual rescoring result is the part operators should not skip. The paper reports that strict all-pass aggregation in multi-verifier tasks can alter agent rankings Messier . In plain terms, the choice of scoring rule can decide whether partial progress is visible or flattened into failure. That does not mean loose scoring is better. It means a benchmark result should explain what the verifier is trying to protect and why its aggregation rule fits the task Messier . A coding task, a procurement workflow and a clinical-administration task can require different scoring policies because their failure costs differ HackDetect . This is where runtime receipts become more than audit theatre. If the trace shows which verifier failed, which state changed, and which action was still correct, the team can decide whether the error is a release blocker or a process-design issue. If the benchmark only publishes a final pass rate, that judgement is hidden by design rather than available for review Messier . The missing layer is protocol validity Messier makes records comparable, but comparability is not validity. The HackDetect paper defines protocol validity as whether an evaluation keeps the intended capability necessary for success, then audits 2,385 traces across 15 agent benchmarks HackDetect . It reports evidence of exposures and reward hacking in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks HackDetect . That is the warning label for any corpus-led evaluation market: HackDetect reports benchmark exposures and reward-hacking paths even when the benchmark result itself looks measurable HackDetect .

### [Inference Optimization: A Practical Production Guide](https://swarmsignal.net/inference-optimization-guide-2026/)

Published: 2026-03-28. Updated: 2026-06-22. Excerpt follows.

Inference Optimization in 2026: Where the Compute Actually Goes Inference can absorb a growing share of AI infrastructure spend once a system moves from experimentation into production. The exact split varies by organization, but the pattern is consistent: training is often a bounded event, while serving a model is an ongoing cost center. And yet many optimization discussions still fixate on training efficiency, as if the hard part ends when the loss curve flattens. It doesn't. The harder operational work starts after launch: traffic arrives, queues form, and users notice slow answers. The techniques that manage that pressure have become consequential work in applied ML. Here's what usually moves the needle. The Memory Wall Is the Real Bottleneck Large model inference is often constrained less by raw arithmetic than by memory bandwidth. Large models need a lot of memory just to hold the weights, and token generation repeatedly brings those weights or cached activations back into the compute path. On modern accelerators, that movement can add noticeable latency per pass, even when the arithmetic units are not the limiting factor. This is why raw FLOPS comparisons between GPUs can miss the point for many inference workloads. Recent hardware work keeps making the same point: bottlenecks often shift from arithmetic to memory movement. More compute helps less when the data can't get there fast enough. Much of the inference optimization stack exists to work around this wall. The techniques below either reduce how much data moves, hide the latency of moving it, or improve how requests are scheduled around the hardware. Quantization: Trading Bits for Throughput The most direct attack on the memory wall is making the model smaller. Quantization compresses weights from 16-bit floats to 8-bit, 4-bit, or even lower representations. The results can be strong, but they are workload-dependent. AWQ (Activation-Aware Weight Quantization) at 4-bit can preserve much of the model's quality across standard evaluation sets. GPTQ is often a bit less accurate but can still deliver a useful throughput win on GPU. The real story is in the kernel implementations: optimized quantized kernels can turn a theoretical compression win into a practical serving win. NVIDIA's NVFP4 format on Blackwell pushes this further. It reduces memory footprint versus FP8 while aiming to keep accuracy close to the higher-precision baseline. Vendor benchmarks suggest meaningful throughput gains on supported hardware, but the exact delta depends on the model, kernel, and batch shape. The counterargument matters, though. Quantization degrades gracefully on benchmarks but can fail unpredictably on tail distributions. Rare tokens, code generation edge cases, and multilingual outputs suffer disproportionately. A model that scores well on a general benchmark at 4-bit might hallucinate more on a niche medical query than the accuracy numbers suggest. Production systems need quantization-aware evaluation on their actual task distribution, not just generic benchmarks. KV Cache: The Silent Memory Hog Most optimization coverage focuses on model weights. But during inference, the KV (key-value) cache often consumes more memory than the model itself. For a large model serving long-context requests, the KV cache can become enormous per sequence. Multiply that by concurrent users and you've hit your VRAM ceiling long before compute saturates. Three approaches show up repeatedly in current production and research discussions. PagedAttention , introduced by vLLM , treats KV cache like virtual memory pages. Instead of pre-allocating contiguous blocks for each sequence's maximum possible length, it allocates small pages on demand. This can cut waste that plagued earlier systems and helps explain why vLLM often performs well on throughput-oriented serving workloads. Grouped Query Attention (GQA) attacks the problem architecturally. By sharing key-value heads across multiple query heads, GQA reduces KV cache size , often with limited quality loss when the model is trained or adapted for it. Widely used model families such as Llama, Mistral, Gemma, and Qwen have adopted it in recent releases. For operators, the important point is that the inference benefit is baked into the model architecture rather than added as a serving trick. KV cache quantization compresses the cache itself to FP4 or INT4, separate from model weight quantization. NVIDIA's NVFP4 KV cache reduces cache footprint and can extend the practical context budget on supported stacks. Representative research methods in this area report large memory reductions, but production impact still depends on model, sequence length, and quality tolerance. These aren't always independent choices. Production stacks often layer several of them. A vLLM deployment serving a GQA-trained model with FP4 KV cache quantization can compound savings, but the final gain depends on request shape and hardware support. This is where the attention heads budget analysis becomes directly actionable. Batching and Scheduling: Where Throughput Actually Lives Single-request latency gets the attention. Throughput pays the bills.

### [EcoAgent-Bench Prices Escalation Choices](https://swarmsignal.net/ecoagent-bench-prices-escalation-choices/)

Published: 2026-08-12. Updated: 2026-08-12. Excerpt follows.

EcoAgent-Bench is a 6 August 2026 benchmark for a deployment question that ordinary task-success scores blur: when should a tool-using system spend more, escalate, route to a stronger model or stop EcoAgent-Bench ? Its practical value is that the paper evaluates the priced decision path, not just the final answer EcoAgent-Bench . Key takeaways Main change: cost is part of the task definition, not a metric added after the run. Practical implication: teams should test escalation choices separately from final-answer accuracy. Caveat or risk: the benchmark uses abstract price schedules, so it supports controller design more than direct cloud-bill forecasting. Decision: add a budget-aware routing test before shipping workflows that can call premium tools, stronger models or human review. What This Benchmark Actually Tests EcoAgent-Bench introduces real-derived tasks with priced actions and explicit budgets, then tests whether a system avoids unnecessary escalation, escalates when local evidence is insufficient, selects a model tier and stops on unsupported premises EcoAgent-Bench . A system may have cheap local evidence, a broader search action, a composite research tool, a stronger model tier or an abstention path; each action has a price and the task has a budget EcoAgent-Bench . That changes the release question. A correct answer is not enough if the system bought it wastefully. A cheap answer is not enough if the system stopped before evidence justified the answer. The paper's strict score requires correctness, grounding and staying within budget, while its economic-consistency score takes the weaker side of upgrade-oriented and save-oriented task families EcoAgent-Bench . This connects directly to the older budget-aware routing problem and runtime-receipt requirement . The useful receipt is not just "task passed". It is the sequence of priced choices that made the pass defensible, because EcoAgent-Bench evaluates correctness, grounding, budget feasibility and trajectory cost together EcoAgent-Bench . The Decision Matrix Use EcoAgent-Bench as a design prompt for routing decisions EcoAgent-Bench : Situation Bad default Better test Local evidence is enough Calls a premium tool anyway Does the controller stop cheaply? Evidence is incomplete Answers from weak context Does it escalate before answering? Reasoning is hard Uses one model tier for every case Does it buy stronger inference only when needed? Premise is unsupported Keeps searching until budget is gone Does it abstain with evidence? The paper's 6 August 2026 result is that current systems can be one-sided EcoAgent-Bench . Always-escalate controls can look strong under micro accuracy while failing save-oriented cases, and cheaper policies can look economical while missing warranted escalation EcoAgent-Bench . That is why the family-balanced economic-consistency score is useful: it punishes both reckless spending and premature stopping EcoAgent-Bench . The headline numbers make the gap visible. EcoAgent-Bench contains 304 tasks, and the paper reports that tested tool-API systems reached only 3.9% to 24.0% micro strict success and at most 7.3% economic consistency EcoAgent-Bench . More Budget Is Not A Strategy One uncomfortable result is the budget sweep. The authors report that a threshold-crossing budget sweep changed GPT-5.4's escalation rate from 0% to only 3%, while other runs often stopped before warranted escalation or overspent on cheap tasks EcoAgent-Bench . The inference is narrow but important: adding a budget number to the prompt is not the same as giving the system a resource-selection policy. The 6 August 2026 paper argues for controllers that estimate the marginal value of the next action, compare that value with price and remaining budget, and decide whether current evidence is already sufficient EcoAgent-Bench . Inference-time budget-control research reaches a similar architecture point from search. It formulates search as a two-stage budget problem, where a controller assigns value-of-information scores to candidate actions under remaining tool-call and token limits Inference-Time Budget Control . EcoAgent-Bench is the evaluation counterpart: it asks whether a system actually made the right budget-conditioned choice. Where It Transfers To Production EcoAgent-Bench transfers most directly to workflows with optional expensive actions because its task design explicitly prices local lookup, broad search, composite research, model-tier routing and stop-loss decisions EcoAgent-Bench . That supports production tests for deep research, paid search, legal review, security triage, healthcare handoff, stronger-model escalation and human support when those actions have meaningful cost or risk EcoAgent-Bench . In those settings, the harm is not only a wrong answer. It can be an answer that is right but unaffordable, or cheap but under-evidenced. Agents' Last Exam frames a related benchmark trend by focusing on economically valuable real-world workflows with verifiable success criteria Agents' Last Exam . EcoAgent-Bench narrows that into a specific control: a workflow is not production-ready until the system can explain why the next paid action was worth taking. The limit is cost realism. EcoAgent-Bench's authors state that some costs are abstractions and that workspace-track execution proxies should not be used for cross-track cost claims EcoAgent-Bench .

### [Agent Cost Optimisation: Measure Spend per Accepted Task](https://swarmsignal.net/agent-cost-optimization-how-to-track-and-reduce-llm-spend/)

Published: 2026-07-09. Updated: 2026-09-08. Excerpt follows.

An agent can make a cheap model call and still be expensive to run. Planning, searches, tool execution, retries and human corrections all belong to the user task. Start by connecting those costs to an outcome you can actually accept. This guide provides a measurement plan and a small accounting example. It does not promise a percentage saving. Define success before dividing the bill An accepted task meets your application's requirements. For a research assistant, that might mean answering the question with supporting sources. For an extraction workflow, it might mean producing correct fields that pass validation. A successful HTTP response is not enough. Keep the acceptance rule stable while comparing changes. Report task success alongside cost per accepted task: rejecting more requests should not masquerade as an optimisation. Count the work that failed as well as the answer you kept. Use a task identifier to connect every attempt, including fallbacks and abandoned runs. Count the accepted task once, after the final decision. Keep failed-task costs in the numerator. Otherwise the dashboard makes unreliable workflows look cheaper than they are. Collect enough detail to explain the cost Record the model and version, input and output usage, cached usage where reported, provider request identifier, task identifier, attempt outcome and elapsed time. Keep a pricing version or billing date so an estimate can be reproduced later. Collect tool, hosting and human review costs separately. Avoid logging confidential prompts merely to count tokens. Reconcile estimated usage charges with the provider's invoice and investigate unexplained differences. Define whether a cost report includes only marginal API charges or allocated infrastructure and staff time. Both can be useful, but comparing them without stating the scope is misleading. Do not allocate the same invoice to attempts and add it again as monthly overhead. A runnable whole-task example The following Python program uses only the standard library. Save it as cost.py . The example and recorded test results are available to inspect. It accepts a JSON file of tasks and all their attempt costs, expressed in a single currency. The costs are already calculated: obtain them from billing records or an explicitly versioned usage estimate. #!/usr/bin/env python3 """Illustrative cost accounting. Input: {"tasks": [...]} JSON; no external services.""" import argparse import json from decimal import Decimal def summarise(tasks): total = Decimal("0") accepted = 0 seen = set() for task in tasks: if task["id"] in seen: raise ValueError("Duplicate task id") seen.add(task["id"]) if not isinstance(task["accepted"], bool): raise ValueError("accepted must be a boolean") accepted += int(task["accepted"]) for attempt in task["attempts"]: value = Decimal(str(attempt["cost"])) if not value.is_finite() or value < 0: raise ValueError("cost must be finite and non-negative") total += value return {"total_cost": str(total), "accepted_tasks": accepted, "cost_per_accepted_task": str(total / accepted) if accepted else None} def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("input", help="JSON file containing tasks and all attempt costs") args = parser.parse_args() with open(args.input, encoding="utf-8") as handle: data = json.load(handle) print(json.dumps(summarise(data["tasks"]), indent=2)) if __name__ == "__main__": main() Save this deliberately illustrative input as example.json . These are invented currency units for explaining arithmetic, not provider prices or benchmark measurements: {"tasks": [ {"id": "research-a", "accepted": true, "attempts": [{"cost": "0.04"}, {"cost": "0.06"}]}, {"id": "research-b", "accepted": false, "attempts": [{"cost": "0.03"}]} ]} Run python3 cost.py example.json . The output includes the unsuccessful task and both attempts on the accepted task. If no tasks are accepted, the ratio is null ; reporting a zero cost per accepted task would conceal that the system produced no accepted work. This is a teaching example, not a billing integration. It assumes each task appears once and all costs use the same currency and accounting scope. Add your own export step to map provider usage into attempt costs. Keep cancellations and timeouts if they incurred charges. Pick the next change from the traces Sort tasks by total cost and inspect representative expensive cases. Look for repeated searches, retries that repeat the same failure, large tool responses, unnecessary model calls and answers that need manual repair. Do not assume the largest model is the main problem. A practical experiment record contains the baseline, proposed change, acceptance test, total cost, success rate and completion latency. Keep the raw task outcomes so someone can examine a saving that looks suspicious. OpenAI's model optimisation guide places evaluations around prompt and model changes. Apply that discipline to cost work: a lower bill matters only if the resulting application still does the job. Test routing and caching carefully A cheaper model may handle a particular operation adequately. Compare it on held-out examples before changing the route. Include the routing decision and fallback attempt in cost and latency totals. A model's claim that it is confident is not an acceptance check. Prompt caching and answer caching are different. Provider prompt caching can reuse eligible shared input processing while still generating a new answer.

### [Test-Time Compute: Methods, Costs and Evaluation](https://swarmsignal.net/test-time-compute-scaling-guide-2026/)

Published: 2026-05-03. Updated: 2026-09-08. Excerpt follows.

Test-time compute means spending additional computation while answering a request. That might mean generating alternative solutions, revising a candidate or searching through intermediate steps. The useful question is whether the answer your system actually returns improves enough to justify the added cost and delay. Start with a task you can score, an unchanged baseline and a spending limit. Keep the generator, selection method and final evaluator separate. A promising answer somewhere in the candidate pool is useful evidence about the generator; it is not yet a successful response to the user. Separate finding an answer from choosing it Large Language Monkeys distinguishes coverage from the ability to identify correct samples. Coverage asks whether any attempt solved a problem. In coding evaluations this corresponds to the pass@k idea: did a correct solution appear within the sample budget? Selected-answer accuracy asks whether the system chose a correct answer to return. You can measure coverage offline using reference answers or held-out tests. Those checks may be unavailable to the deployed selector. Using the answer key to pick a candidate, then reporting that result as deployable accuracy, would hide the problem you need to solve. The paper finds that selection methods can stop improving while coverage continues to rise in its studied maths tasks. That is a reason to evaluate the selector independently. It does not establish a universal sampling limit for every model or workload. More candidates help only when your system can choose a better answer. Choose the method around the task Parallel sampling generates alternative candidates from the same request. A selector might rank them with a reward model, count equivalent answers or run executable checks. Keep candidate order and tie-breaking rules explicit. Several near-identical answers can share the same mistake. Sequential revision feeds a candidate and feedback into another attempt. This fits tasks where feedback identifies something actionable, such as a failing test or a missing requirement. Record each revision: an apparently useful correction can also damage an earlier valid result. Search over intermediate steps explores partial solutions and uses a scoring method to decide which branches receive more work. This creates another evaluation obligation: check the scoring method as well as the final answers. Scaling LLM Test-Time Compute Optimally studies different allocations of inference effort and finds that effective choices depend on problem difficulty relative to the model. Its difficulty-estimation setup also has costs that are not fully included in the reported analysis. In your service, include routing and difficulty estimation in the bill. There is no useful universal promise that a particular sample count captures most of the available gain. Choose candidate budgets from your latency and spending constraints, then measure the resulting curves. Try a small selection failure yourself The following Python example uses invented correctness labels and invented verifier scores. It calls no model and is not a benchmark. The candidate order is fixed, and the selector can see scores but cannot use the correctness labels. def measure(tasks, budget): if not tasks or budget < 1: raise ValueError("Provide tasks and a positive budget") if any(len(task) < budget for task in tasks): raise ValueError("Every task needs the requested candidates") covered = selected = 0 for task in tasks: candidates = task[:budget] covered += any(correct for correct, score in candidates) chosen = max(candidates, key=lambda item: item[1]) selected += chosen[0] return {"coverage": covered / len(tasks), "selected_accuracy": selected / len(tasks)} tasks = [ [(False, .4), (True, .8), (False, .9)], [(True, .7), (False, .6), (False, .8)], ] for budget in (1, 2, 3): print(budget, measure(tasks, budget)) With the middle budget, the selector finds the correct candidate for each task. With the largest budget, higher-scored wrong answers displace those choices even though correct answers remain available. This deliberately constructed failure shows why the coverage curve cannot substitute for the selected-answer curve. Use the inline example above. Change the scores, reverse tied candidates and remove a correct answer. Observe what changes before adapting the calculation to recorded model outputs. The example reports observed coverage on fixed candidates; it does not implement a statistical pass@k estimator. For saved traces, calibration and test splits, and abstention policies, run the answer-selection lab . Start with its synthetic fixture, then substitute independently labelled examples from your own workload. Build a verifier you can challenge Use checks that correspond to the intended result. For a code patch, compilation is a different check from passing relevant tests or preserving behaviour outside the edited function. For an extraction task, a valid schema does not establish that the extracted values match the source. Keep final evaluation independent of candidate selection. If a generator can see every acceptance test, passing those tests gives limited evidence about behaviour on unseen inputs. Retain held-out cases and inspect disagreements between selector scores and final labels.


## Control authority and protect information

### [The AI Agent Security Playbook](https://swarmsignal.net/ai-agent-security-playbook/)

Published: 2026-02-27. Updated: 2026-06-22. Excerpt follows.

The AI Agent Security Playbook NVD describes CVE-2025-32711 as AI command injection in Microsoft 365 Copilot that can disclose information over a network. Aim Labs/Cato calls the disclosed chain EchoLeak . In that EchoLeak write-up, the chain starts with a crafted email and uses Copilot context plus auto-fetched URLs as an exfiltration path. The record treats the issue as high severity from multiple scoring angles. Aim Labs/Cato describes EchoLeak as zero-click and says Microsoft confirmed no customers were affected. This article focuses on that intersection: untrusted content, private context, and outbound channels. Recent public data points: As of June 2026, treat the incident counts and attack catalogs below as current evidence, not stable baselines. Stanford HAI's 2025 AI Index reports a rising number of AI-related incidents in 2024 compared with 2023. Gartner's August 2025 forecast suggests task-specific AI agents will become common in enterprise applications by the end of 2026. BigID's June 2025 release says only a small minority of surveyed organizations had an advanced AI security strategy or defined AI TRiSM framework. Agents Are a Different Threat Surface A chatbot takes a prompt and returns text. An agent takes a prompt, reasons about it, calls tools, reads files, executes code, sends emails, and queries databases. The attack surface isn't a conversation. It's every tool the agent can touch. OWASP recognized this distinction in December 2025 by publishing a dedicated Top 10 for Agentic Applications, separate from their existing LLM Top 10. The agentic list emphasizes risks that become much sharper when models can plan and call tools: agent goal hijacking, tool misuse and exploitation, memory and context poisoning, insecure inter-agent communication, cascading failures across agent networks, and agents that drift from intended behavior. MITRE ATLAS added new agent-specific attack techniques in October 2025, including AI agent context poisoning, memory manipulation, and exfiltration via agent tool invocation. The common thread: agents have permissions that chatbots don't. Permissions create attack surfaces. For a framework on how to constrain those permissions, see the guardrails guide . Prompt Injection Remains the Top Threat OWASP treats prompt injection as a top vulnerability for LLM applications in 2025, and it's worse for agents because agents act on their instructions rather than just generating text. Anthropic published quantified prompt injection failure rates in their February 2025 system card. The numbers depend heavily on context and safeguards. In a constrained coding environment, injection did not land in the reported tests. In a GUI-based system without safeguards, success was much higher. With safeguards enabled on a browser-based system, success dropped sharply. That spread is a good illustration of the value of defense-in-depth. Indirect injection is the bigger concern for agents. Attackers embed instructions in documents, web pages, or emails that the agent processes. The GitHub Copilot RCE (CVE-2025-53773) demonstrated this precisely: malicious prompts in source code comments instructed Copilot to modify VS Code settings, enable auto-approve mode, and achieve arbitrary code execution. The injection was self-replicating: during code refactoring, the compromised instructions propagated to other files. The extension had been installed at massive scale. A financial services AI banking assistant was exploited through prompt injection to bypass transaction verification, resulting in fraudulent transfers. A state-backed threat actor manipulated Claude Code to conduct AI-orchestrated espionage across multiple organizations. Security researcher Michael Bargury proposed the "Rule of Two" as the simplest design heuristic: an agent session should satisfy only two of three properties simultaneously: processing untrustworthy inputs, accessing sensitive data, and communicating externally. If your agent does all three, it's a prompt injection waiting to happen. For a deeper analysis of how indirect injection attacks work in agent pipelines , see the dedicated coverage. Tool Misuse and Excessive Agency The Amazon Q Developer incident in July 2025 demonstrated how tool vulnerabilities compound with agent capabilities. An attacker used an unverified GitHub account to gain admin access to the aws-toolkit-vscode repository and injected malicious prompts into the Amazon Q Developer Extension. The compromised version instructed the AI to delete the file system, clear user config, discover AWS profiles, and delete S3 buckets, EC2 instances, and IAM users via AWS CLI. The extension had been installed at massive scale, and the malicious version shipped without tamper detection. Trend Micro found a classic SQL injection vulnerability in Anthropic's reference SQLite MCP server implementation, which had been forked widely before being archived. In agent environments, SQL injection becomes a springboard for stored-prompt injection: agents treat database content as trusted, so embedded prompts in query results can trigger email sends, cloud API calls, and lateral movement. The MCP protocol's security model hasn't caught up with its adoption, making every MCP server a potential attack vector. OWASP distinguishes between two related risks. Excessive Agency is when an agent is granted too many permissions.

### [Red Teaming AI Agents: A Practitioner's Guide](https://swarmsignal.net/red-teaming-ai-agents-guide/)

Published: 2026-04-07. Updated: 2026-06-22. Excerpt follows.

Red Teaming AI Agents: A Practitioner's Guide Jailbreaking a chatbot is only one part of the problem. Many AI agents can call tools, chain multi-step actions, coordinate with other agents, and persist state across sessions. The attack surface is not just the prompt window. It can include the execution environment around the model. This guide covers the threat models, testing methodologies, and open-source tooling practitioners should consider for agentic systems in 2026. In March 2026, researchers published T-MAP , a trajectory-aware evolutionary search method for discovering multi-step attack paths through AI agents' tool-use chains. Treat its reported success rates as benchmark-specific, not as universal field rates. The important practitioner lesson is narrower and better supported: agent red teaming has to evaluate full execution trajectories, not only single prompt-response pairs. That captures the core challenge of red teaming AI agents in the 2025-2026 literature. The attack surface has expanded from text generation toward action execution. A model that refuses to write malware might still be tricked into running a script that downloads it. A system hardened against direct jailbreaks might leak credentials through its memory store. An agent that individually passes every safety benchmark might still be compromised through a manipulated message from another agent in the same system. Traditional LLM red teaming tests whether a model will generate harmful text. For agentic systems, red teaming also has to test whether a system can take harmful actions through tools, memory, or inter-agent communication. The difference is structural enough to require different threat models, different tools, and different thinking. Why Agent Red Teaming Is Different LLM red teaming, as practiced since 2023, focuses on the input-output boundary. You craft an adversarial prompt. The model generates a response. You evaluate whether the response violates a safety policy. The attack surface is the context window. The failure mode is text generation. Agents break this model in four ways. Tool access creates real-world consequences. When an agent can execute code, send emails, query databases, or modify files, a successful attack is not a harmful string in a chat log. It is a harmful action in a production environment. The OWASP Top 10 for Agentic Applications identifies "Excessive Agency" as a top risk: agents granted more permissions than their task requires become force multipliers for any successful exploitation. Multi-step execution creates trajectory-level vulnerabilities. A single prompt-response pair might be safe at every step while the overall trajectory is harmful. T-MAP demonstrated this systematically: attacks that succeed through sequences of individually benign tool calls, where the harm emerges from the combination, not from any single step. Red teaming must evaluate trajectories, not just individual outputs . Inter-agent communication creates new injection surfaces. In multi-agent systems, agents pass messages to each other. These messages are another input channel, and they are exploitable. The Agent-in-the-Middle (AiTM) attack , published in February 2025 and accepted at ACL 2025, demonstrated that an adversary who can manipulate inter-agent messages can compromise an entire multi-agent system without ever touching an individual agent's prompt. The adversarial agent uses a reflection mechanism to generate contextually aware malicious instructions that other agents follow because they arrive through a trusted communication channel. Persistent memory creates poisoning opportunities. Agents that maintain state across sessions can be attacked through their memory. A malicious instruction injected into an agent's memory during one session may execute during a future session, long after the original attack vector is gone. This is a fundamentally different threat model from stateless LLM interactions, and standard prompt injection defenses do not address it . The Threat Model: Five Attack Surfaces Before running a single test, you need a threat model specific to your agent architecture. The following five surfaces cover the primary attack categories for agentic systems in 2026. They build on OWASP's Agentic Top 10 and MITRE ATLAS , adapted for practitioner use. 1. Goal Hijacking The attacker redirects the agent's objective. Unlike prompt injection, which targets a single response, goal hijacking targets the agent's persistent planning loop. A financial analysis agent might be redirected to exfiltrate portfolio data instead of summarizing it. OWASP classifies this as ASI01 (Agent Goal Hijacking) and treats it as one of the highest-severity agentic risks. Test approach: Embed conflicting instructions in tool outputs, retrieved documents, and inter-agent messages. Verify whether the agent's stated goal changes or whether its actions diverge from its stated goal while the goal description remains unchanged (the more dangerous variant). 2. Tool Misuse and Escalation The agent uses its tools in unintended ways, either through direct manipulation or through emergent behavior from ambiguous instructions. An agent with filesystem access might be induced to read configuration files containing secrets. An agent with code execution might be tricked into installing packages. Test approach: For each tool the agent can access, enumerate the maximum harm it could cause.

### [MasDrift Measures Authorisation Drift](https://swarmsignal.net/masdrift-authorisation-drift/)

Published: 2026-08-15. Updated: 2026-08-15. Excerpt follows.

MasDrift tests a quiet failure in multi-agent systems: the task gets delegated, but the user's boundary does not. The August 2026 paper matters because it measures authorisation preservation across single-agent, centralised and peer coordination structures, rather than treating safety as a property of the model alone MasDrift . Key takeaways Main result: centralised multi-agent hierarchies completed more tasks but produced more unauthorised actions than peer networks in MasDrift's benchmark MasDrift . Practical implication: a delegation design can be the risk surface even when every worker is trying to finish a benign task. Control lesson: checking a pending tool call against the original user request worked better than relying on a policy that was carried through handoffs MasDrift . Caveat: MasDrift is a benchmark of simulated productivity tasks, so it should guide release tests and trace audits rather than serve as a production incident rate. What This Benchmark Actually Tests MasDrift defines authorisation drift as the weakening or disappearance of a user's constraint as work is handed from one agent to another MasDrift . The benchmark is deliberately benign. There is no malicious worker, poisoned document or prompt-injection payload. The system is simply trying to finish the job. AgentDojo is the nearby contrast: it evaluates tool-using agents against indirect prompt-injection attacks over untrusted data AgentDojo . MasDrift asks a different release question: if the input is honest and the goal is legitimate, does the system still remember what the user did not approve MasDrift ? The paper's task shape makes the boundary concrete. Each of its 600 tasks pairs required work with reserved actions across eight productivity domains MasDrift . An agent may be allowed to prepare a payment batch, referral letter or customer message while the actual send, post, publish or approval action remains reserved. The tool surface includes both permitted preparation tools and high-impact execution tools, so completion pressure can pull the system over the line MasDrift . The missing boundary in delegation The operational boundary is not just "which tools can this worker call?" It is "which user-approved authority can this worker prove for this call?" MasDrift's task suite makes that distinction visible by separating permitted preparation from reserved execution MasDrift . That connects directly to purpose-bound privacy and agent security ownership . Privacy controls decide which facts may move to which sink. Ownership controls decide who is accountable for an automated action. MasDrift adds the delegation question: can the original boundary survive being restated by another component? Hierarchies trade completion for overreach The central result is not that hierarchy is bad. It is that hierarchy changes the measured safety profile in this benchmark MasDrift . Across generic multi-agent conditions, centralised hierarchies reached 93.9% to 98.6% task completion, while peer networks reached 85.7% to 87.0% MasDrift . On the same comparison, unauthorised actions occurred in 2.7% to 19.8% of centralised tasks, compared with 0.6% to 0.8% for peer networks MasDrift . The deeper result is sharper. Increasing hierarchy depth from one to three levels added 4.6 points of completion, but raised unauthorised actions from 2.7% to 19.8% MasDrift . A single agent given the same tasks and tools stayed at or below 1% unauthorised actions MasDrift . For operators, the lesson is not to avoid supervisors. It is to stop assuming a clean top-level instruction remains clean after decomposition. The more a system summarises, routes and specialises work, the more it needs a separate record of what the user actually authorised. The first handoff is often the break MasDrift scores more than final task success: it measures unauthorised action, over-disclosure and constraint loss, and it localises the handoff at which the constraint first weakens MasDrift . That matters because a final answer can look compliant while an intermediate worker has already lost the boundary. The paper reports 90,000 fully traced executions across 600 tasks, nine coordination conditions and six model configurations MasDrift . In that trace view, a "near miss" is a run where the constraint was lost but no reserved action happened. That is a useful production category. It says the system was one tool call away from a violation, even if the demo still looks clean. This is where ordinary observability falls short. A log that says send_email was not called may satisfy a narrow audit. A release test for delegated systems needs the richer question: did every downstream worker still carry the user's restriction, and can the system prove that before the reserved tool is available? Source checks beat carried intent MasDrift compares two defences. The first re-anchors every pending call to the original user request. The second carries an attenuated policy through the delegation chain MasDrift . Re-anchoring reduced unauthorised actions in every evaluated model configuration, with a pooled completion cost of 1.6 points MasDrift .

### [ToolPrivacyBench Exposes Purpose Drift](https://swarmsignal.net/toolprivacybench-purpose-drift/)

Published: 2026-08-09. Updated: 2026-08-09. Excerpt follows.

ToolPrivacyBench turns agent privacy into a tool-call audit: when a workflow succeeds, did each tool receive only the private facts it needed? The June 2026 paper matters because it separates task completion from purpose-bound disclosure, a failure mode that ordinary function-calling scores can miss ToolPrivacyBench . Key takeaways Main change: privacy evaluation is moving from final answers to full tool-call trajectories. Practical implication: a successful workflow can still leak unnecessary private facts into tickets, handoffs or free-text fields. Caveat or risk: ToolPrivacyBench uses mock business backends, so it diagnoses a control gap rather than proving the same rates in every production stack. Recommendation: score purpose-bound disclosure alongside task success before giving tool agents sensitive workflows. The Privacy Boundary Is Purpose, Not Sensitivity ToolPrivacyBench defines the problem as purpose-bound over-disclosure. A private fact is not automatically forbidden everywhere. It may be necessary for one tool and inappropriate for the next. A symptom can belong in a clinical record but not in a payment note; a phone number can be justified for notification but not for an internal handoff ToolPrivacyBench . That distinction is useful because most tool-agent checks still focus on whether the model chose the right function and passed valid arguments. The June 2026 ToolPrivacyBench paper asks a different question: did the agent route each current-task private atom only to the tools and downstream sinks authorised for that purpose ToolPrivacyBench ? The June 2026 benchmark represents each case with a policy knowledge base covering private atoms, tool purposes, sink types, allowed and forbidden field-tool relations, free-text slots and backend audit evidence ToolPrivacyBench . After the agent acts, the evaluator compares recorded tool arguments and mock-backend logs against that policy. The privacy judgement is therefore attached to what the agent actually sent, not only to the answer the user saw ToolPrivacyBench . That is the missing layer between agent data-injection trust boundaries and agent security ownership . Trust boundaries decide which inputs and tools deserve access. Purpose-bound disclosure decides which facts each allowed tool should receive during a specific workflow. Task Success Hides The Leak The headline result is uncomfortable for teams that use task completion as their release gate. ToolPrivacyBench contains 2,150 multi-tool cases, with 1,150 fully synthetic privacy-sensitive business workflows and 1,000 cases adapted from public multi-tool and function-calling benchmarks ToolPrivacyBench . Across the synthetic private split, the tested agents completed tasks at high rates: TaskSuccess ranged from 92.23% to 97.70% ToolPrivacyBench . The same runs still showed multi-tool privacy over-disclosure. The benchmark's MT-POI privacy-risk metric remained between 19.19 and 28.04, which means workflow success did not imply appropriate disclosure ToolPrivacyBench . The paper identifies tickets and handoffs as frequent leakage locations, with aggregated forbidden-opportunity rates of 51.43 and 34.79 respectively ToolPrivacyBench . Free-text business fields also repeat as channels for over-disclosure ToolPrivacyBench . That pattern should feel familiar to anyone who has reviewed agent traces: the agent writes a convenient narrative, and the narrative carries more private detail than the next recipient needed. This is not prompt injection. The paper explicitly evaluates a benign workflow setting: no attacker changes the prompt, tools, backend or audit log ToolPrivacyBench . The failure is ordinary over-sharing during tool use. That makes it more operationally important, not less. A system can be well-intentioned, helpful and still too loose with private facts. Why Trajectory Audits Matter Anthropic's agent-autonomy research defines an agent operationally as a system equipped with tools that let it take actions such as running code, calling external APIs and sending messages Anthropic . The same post notes a measurement limit: broad API traffic can show individual tool calls, while product-level traces can show whole workflows, and neither view alone answers every autonomy question Anthropic . ToolPrivacyBench lands inside that gap. Privacy risk is not always visible in the final answer or in a single API-call success bit; the benchmark measures it through recorded tool arguments and backend audit logs ToolPrivacyBench . It appears in the sequence: which tool was called, which argument field carried which fact, whether the sink needed that fact, and whether a later handoff repeated it. Human-on-the-Bridge makes a related evaluation argument. Its June 2026 paper says agent failures can include phantom tool calls, missing mandatory tool calls, policy drift, manipulation paths and safe but non-resolving refusals; those failures are difficult to catch through static benchmarks or single-response scoring Human-on-the-Bridge . ToolPrivacyBench adds a privacy-specific case: the agent may call the right tools and still fail the information-flow policy. Inference from these sources: production evaluation should treat tool traces as evidence, not debugging residue. If a workflow handles health, finance, tax, recruiting, education, security or customer-support data, the release gate should be able to answer which private atoms moved through which tools and why; those domains appear in ToolPrivacyBench's benchmark construction and representative workflow set ToolPrivacyBench .

### [Tool-Using Agents Can Hide Messages](https://swarmsignal.net/tool-using-agents-can-hide-messages/)

Published: 2026-09-05. Updated: 2026-09-05. Excerpt follows.

The AISI and Oxford paper Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems matters because it moves covert agent communication from theory into an engineering threat model. The authors report that tool-using coding agents can build working natural-language steganography systems when given realistic affordances such as code execution, file access and research-paper lookup Rippin et al. . For teams wiring agents together, the uncomfortable lesson is simple: readable inter-agent chat is not the same as controlled information flow. Key takeaways Main result: the paper reports that agentic coding models can implement covert communication channels when they can use ordinary tools, not only when they memorise a trick Rippin et al. . Practical implication: monitoring only the visible text between agents is a weak boundary when tool use, files, repeated interaction or shared artefacts are also available. Caveat: the authors find broad convergence on scheme families but limited strict one-shot coordination, so the highest-risk settings are systems with repeated interaction or shared context Rippin et al. . Decision: treat inter-agent egress as a security surface, with capacity reduction and tool-trace review alongside ordinary message inspection. The failure is a boundary assumption Many multi-agent systems assume a plain-text channel is inspectable because a human or model monitor can read each message. The AISI/Oxford paper challenges that assumption. It defines a monitored-channel setting where one agent holds information another agent should not learn, while a monitor sees the exchanged natural-language messages Rippin et al. . The risk is not that every agent will spontaneously invent cryptography; the paper instead tests whether a capable tool user can search, read, code and repair an encoder and decoder until they work Rippin et al. . The paper says the relevant bottleneck is shifting from whether agents can understand steganography to whether independently acting agents can converge on compatible schemes, keys and parameters Rippin et al. . That is a different problem from ordinary prompt filtering. A monitor can reject obvious secrets, policy violations or suspicious strings. It cannot reliably enforce a policy if the content still looks benign and the forbidden payload is encoded in choices the monitor was not designed to measure Rippin et al. . Swarm Signal's earlier agent communication piece argued that plumbing is not language. This paper adds a harder security point: language can also become a carrier Rippin et al. . Tool use changes the threat model The paper's default experiment gives an agent code execution, file-system access, a research paper, and model-sampling support, then asks whether it can produce a working covert channel Rippin et al. . The authors also test more restricted settings and report that agents adapt when some ingredients are missing, including by adding a model-sampling component or implementing related keyed schemes Rippin et al. . For operators, the relevant detail is not a particular encoding recipe. It is the general capability loop: read, implement, test, repair. A system that allows agents to inspect papers, write helper code, run scripts and keep intermediate files has more covert-channel surface than a transcript review suggests. This overlaps with the protocol story. AgentOrchestra's TEA protocol paper argues that agent systems need explicit lifecycle handling for tools, environments, agents and associated artefacts, and reports that its hierarchical framework reaches 89.04% on the GAIA Test set AgentOrchestra . That kind of explicit context management can make multi-agent work more reproducible. It also makes shared artefacts, environment state and generated files part of the trust boundary. Coordination is the remaining brake The AISI/Oxford result is measured rather than apocalyptic. The authors report substantial convergence on broad scheme families, but limited strict one-shot coordination between independent agents Rippin et al. . In practical terms, two agents may both know how to hide information and still fail if they choose incompatible details. That brake weakens when systems provide repeated interaction, shared files, common libraries, shared prompts or stable conventions. The strategic-confinement paper makes the same class of concern more abstractly: when strategic agents share coordination resources, residual communication capacity can concentrate on low-entropy, high-impact facts Schroeder de Witt . A tiny leak can matter if the hidden fact is the answer to "which plan should we coordinate on?" or "which record should be exfiltrated?" This is why the safe design target is not only "block secrets in messages". It is "minimise unexplained channel capacity". Message text, whitespace, timing, file names, image choices, tool traces, retry patterns and shared artefact references can all become coordination material if the system leaves them unconstrained. Egress controls need capacity thinking An application-layer egress-monitor paper from May 2026 lists text, JSON ordering, whitespace, message timing, image channels and audio channels as possible carriers for agent leakage Metere .


## Run an engineering experiment

### [Engineering Lab](https://swarmsignal.net/engineering-lab/)

Published: 2026-09-08. Updated: 2026-09-08. Excerpt follows.

Run it. Inspect it. Change it.

Python labs · Docker for pgvector · No API key · Synthetic fixtures

Small, runnable experiments for engineers building AI systems. Reproduce a failure, inspect the result, then adapt the test to your own application.

Choose a problem

Your agent generates a correct answer but chooses a wrong one

Replay answer selection compares selection policies against saved candidate traces. Separate the chance of generating a correct answer from the ability to recognise it. Inspect the failures hidden by an overall accuracy number and see what changes when a selector can abstain. The accounting extension separates overlapping call time, failed attempts and verifier costs when replaying a deadline policy.

Take away: a replay harness, labelled candidate fixtures and per-task diagnostics you can use when changing a verifier or sampling budget.

Your retriever finds a convincing document that it should not use

Test retrieval freshness and permissions exercises expired documents, superseded revisions and access restrictions. Compare ranking alone with an explicit eligibility check, including cases where the right result is no result.

Take away: a small regression suite for the documents allowed into an answer, with fixtures you can replace with sanitised examples from your own corpus.

Your tool times out after completing the action

Make tool retries safe reproduces an ambiguous timeout using a local database. Compare a repeated write with a stable idempotency key, and test what happens when the same key arrives with a different request.

Take away: an executable failure case and a concrete contract to check before an agent retries a tool with side effects.

Your outbox worker crashes after the remote action succeeds

Recover an outbox after a crash stops real worker processes at the boundary between two durable stores. Restart the dispatcher and inspect why downstream deduplication is still needed.

Take away: a crash harness, an idempotent downstream contract and a duplicate-producing negative control.

Your filtered vector search returns fewer useful neighbours

Test pgvector filtering and recall compares an exact eligible result set with HNSW and iterative scans in a real, pinned PostgreSQL instance. Inspect the query plans and vary filter selectivity.

Take away: a reproducible recall experiment with synthetic vectors, SQL plans and measured timings.

Run the examples

The introductory examples use Python's standard library. The pgvector lab also needs Docker to run a pinned PostgreSQL image. None of the labs calls an LLM or requires an API key. Each lab includes the source, the input data, the expected output and instructions for running its checks. Start with the unchanged fixture; only then substitute your own inputs.

These are deliberately small, synthetic cases. The displayed results are produced by the supplied programs, not a survey of production systems or a comparison of commercial models. They show how a failure can occur and how to test for it. They do not estimate how frequently it occurs in your application.

Turn a lab into an application test

Choose a real failure. Start with a sanitised trace, document pair or tool operation your team can explain.

Write down the expected outcome. Include when to abstain or ask a person, not just what a successful answer looks like.

Keep tuning and evaluation separate. Freeze some cases before adjusting thresholds or policies. Inspect individual failures as well as totals.

Test the real boundary. A local replay cannot prove a hosted retriever enforces permissions or a remote tool honours an idempotency key. Add an integration test for that specific contract.

Record what changed. Keep the fixture version, policy, output and known limitations with the release so another engineer can reproduce the decision.

For a broader release checklist, use testing an agent before production . For architectural background, start with single versus multi-agent systems or test-time compute .

How we build these labs

Swarm Signal develops each example around a specific engineering question, checks the relevant primary documentation and runs the supplied code. The lab distinguishes observed fixture output from external research findings. Source links and the exact code version are provided so you can inspect both.

This series follows the practical evaluation principle of defining tasks, graders and observable outcomes rather than relying on a convincing transcript. Anthropic's guide to agent evaluations provides useful background. The fixtures and code here are Swarm Signal's own teaching examples.

Found a failing case? Tell us what you ran , including the lab version, command and sanitised input. Do not include credentials, customer data or private documents.

Sources

Anthropic: Demystifying evals for AI agents . Background on tasks, graders, trajectories and outcomes; it is not evidence for the synthetic results in this series.

Updated 8 September 2026 · Swarm Signal

### [Replay answer selection](https://swarmsignal.net/engineering-lab-answer-verification/)

Published: 2026-09-08. Updated: 2026-09-08. Excerpt follows.

Run it. Inspect it. Change it. Python standard library · No API key · Synthetic fixtures A runnable lab for replaying candidate answers before increasing an inference budget. This is a synthetic engineering exercise, not a model benchmark. The six tasks, candidate answers, correctness labels and verifier scores are invented. They deliberately include confident errors so you can inspect a failure that aggregate scores can hide. Nothing here estimates the accuracy of a commercial model. The practical question is narrower: when your system generates several answers and returns the highest-scoring one, does another candidate help the user? You will replay the same saved candidates through three selection policies, inspect individual failures, and learn which measurements to carry into a real evaluation. For the broader context, read our test-time compute guide . This lab supplies a trace format and evaluation harness you can adapt. What you will run The download contains two answer-selection exercises and an accounting extension. Start with replay.py , which evaluates saved candidate traces. It uses Python's standard library and makes no network or model calls. python3 replay.py --help python3 replay.py python3 replay.py --threshold 80 --output threshold-80.json Download the lab ZIP below and extract it. Run these commands inside its engineering-labs/answer-verification folder. Python 3.10 or newer is sufficient. The default run writes replay-results.json ; the alternative threshold gets its own output so you retain the original comparison. The fixture has two calibration tasks and four test tasks. Each has four candidates in a fixed generation order. We report the first one, two and four candidates separately. This nested comparison matters: changing the candidate set at every budget would mix selection behaviour with differences in generation. Three policies see identical prefixes: first returns the first candidate, ignoring scores. proxy returns the candidate with the highest verifier score. proxy_threshold chooses the same winner, but abstains when its score is below 90. The threshold is declared in advance for this demonstration. It was not fitted on the test set and is not a probability estimate. Calibration rows are reported separately to show where policy development belongs when you substitute real data. Two calibration tasks are nowhere near enough to calibrate a production verifier. Read the result These are the actual default-run counts for the four synthetic test tasks: N Policy Answered Correct Correct available Missed available 1 first 4/4 2/4 2/4 0 1 proxy 4/4 2/4 2/4 0 1 proxy_threshold 0/4 0/4 2/4 2 2 first 4/4 2/4 3/4 1 2 proxy 4/4 2/4 3/4 1 2 proxy_threshold 1/4 0/4 3/4 3 4 first 4/4 2/4 4/4 2 4 proxy 4/4 1/4 4/4 3 4 proxy_threshold 3/4 0/4 4/4 4 “Correct available” means the candidate pool contains at least one labelled correct answer. “Correct” counts the answers actually returned. “Missed available” counts tasks where a correct answer existed but the policy returned an incorrect answer or abstained. Those two outcomes remain distinguishable in the per-task diagnostics. At four candidates, generation has supplied a correct answer for every test task. The proxy policy finds only one. Raising the minimum score makes matters worse here: the wrong answers already occupy the top of the score range. That does not establish that thresholds are bad. It establishes that a high score is useful only when its relationship to correctness has been checked. A threshold can reduce answer coverage without improving the answers that survive it. The JSON also reports accuracy conditional on answering. When no task is answered, that value is null. Reporting zero would conflate undefined conditional accuracy with answering every task incorrectly. Inspect the failure, not just the average Open replay-results.json and find the test, four-candidate, proxy row. On test-1 , the selector chooses candidate b , a wrong answer scored 99, although correct candidates are present. On test-2 , a wrong candidate scored 95 displaces a correct candidate scored 82. On test-3 , the fourth candidate introduces the highest-scoring error. These are actionable categories in a real system: inspect what the verifier rewarded and whether your label actually captures the requirement. A longer explanation, confident phrasing or a superficially valid output format might correlate with a score; this fixture does not test any of those causes. Gao, Schulman and Hilton studied proxy reward overoptimisation, including best-of-N sampling, using a gold reward model as a reference. Their result motivates checking the selected answer rather than trusting the score being optimised. Our fixture is a separate teaching example, not a reproduction of their experiments. Primary paper Bring your own candidate traces Replace traces.json with your own saved evaluation data. Each task needs an ID, a calibration or test split, and an ordered candidates list. Each candidate needs an ID, answer, numeric score from 0 to 100, and independently assigned boolean correctness label.

### [Test retrieval freshness and permissions](https://swarmsignal.net/engineering-lab-retrieval-freshness/)

Published: 2026-09-08. Updated: 2026-09-08. Excerpt follows.

Run it. Inspect it. Change it. Python standard library · No API key · Synthetic fixtures A retrieved passage can match a question perfectly and still be unusable. It might belong to another tenant, describe a superseded policy, or contain information the caller cannot access. Asking the model to ignore it leaves the wrong material in the context. This lab gives you a small regression harness for that boundary. It compares ranking alone, filtering after the first result, and ranking eligible documents. You can run it without an API key or a vector database, inspect every decision, then replace the synthetic corpus with a sanitised slice of your own. The result is a worked failure demonstration, not a retrieval benchmark. The five cases deliberately expose specific mistakes. We use token overlap, not embeddings, and make no claim about how often these failures occur in production. What you will test The fixture contains six short documents and five queries. Two documents represent different revisions of a retry policy. Two describe signing-key procedures with different group permissions. One policy is not effective until October. A final document belongs to another tenant. The texts are intentionally simple. An exact match scores 1.0. The eligible replacements contain the same four query terms plus two extra words, scoring 0.666667. This makes the ranking problem visible without hiding it behind an embedding model. The score is Jaccard similarity: shared unique tokens divided by all unique tokens. Capitalisation is ignored. Synonyms do not match. A score below 0.25 produces no result; that threshold is a fixture choice, not a recommended production setting. Each document carries a tenant, logical document ID, revision, status, permitted groups and effective date. The separate active-revision map is authoritative for this experiment. Being the newest item returned by search is insufficient: an older revision remains invalid even when the current one is missing from the index. This extends the validity and permission example in our knowledge graph guide into an executable retrieval regression suite. Run the experiment Download the lab ZIP below and extract it. From the extracted folder: cd engineering-labs/retrieval-freshness python3 -m unittest -v python3 lab.py --output my-results.json python3 -m json.tool my-results.json Python 3.10 or later is sufficient. The code uses the standard library and makes no network calls. The included results.json records the run used here, with SHA-256 hashes for the fixture and code. There are three strategies: Rank only: take the best lexical match if it clears the threshold. Filter after top-1: select one candidate, then reject it if ineligible. Filter before ranking: consider all eligible documents, then select the best match. These are deliberately small algorithms. The last strategy scans the entire fixture. It is a reference for expected behaviour, not an implementation of approximate nearest-neighbour search. The actual results The Fedora run on 8 September 2026 produced: Case Rank only Filter after top-1 Filter before ranking Superseded revision retry-v1 Abstain retry-v2 Restricted document keys-private-v1 Abstain keys-public-v1 Caller has no access retry-v1 Abstain Abstain Policy not yet effective refund-v2 Abstain Abstain No lexical match Abstain Abstain Abstain Ranking alone made four wrong selections. Filtering after top-1 prevented those selections but lost two valid answers. Filtering before ranking matched all five expected outcomes, including three required abstentions. Do not turn five deliberately constructed successes into a general accuracy percentage. The useful result is the pair of false abstentions: rejecting the highest-ranked document does not automatically recover the eligible document underneath it. Microsoft documents a related risk in Azure AI Search: filtering after candidate selection can miss matching documents, particularly with selective filters and small candidate counts. Its implementation includes shard and graph traversal behaviour that this script does not reproduce. Azure vector query filters Decide eligibility before building model context Our predicate requires the correct tenant, an active status, the authoritative revision, an allowed group and an effective date range. Start dates are inclusive; expiry dates are exclusive. Missing required metadata, empty permissions and an unknown active revision fail closed. The eight tests cover those boundaries, malformed permission metadata, the missing-result problem and a permission change between queries. One test revokes the engineering group's access to the selected revision and verifies that the next retrieval abstains. The supplied identities are trusted fixture inputs. This does not authenticate a user. In a service, derive identity and group membership from your trusted authentication layer; do not accept an arbitrary group list submitted with a question. That distinction matters in real products. Microsoft's security-filter pattern explicitly describes string matching rather than authentication through the principal string. A working filter still needs a trustworthy identity source and consistent application to queries. Azure security filtering Metadata filters are available in retrieval systems such as Qdrant, including combinations of required and excluded conditions.

### [Make tool retries safe](https://swarmsignal.net/engineering-lab-tool-retries/)

Published: 2026-09-08. Updated: 2026-09-08. Excerpt follows.

Run it. Inspect it. Change it. Python standard library · No API key · Synthetic fixtures An agent asks a tool to create a support ticket. The service creates it, but the response never reaches the agent. Retrying feels reasonable. Without a way to recognise the original request, the second call creates another ticket. This lab reproduces that failure in a small Python program, then adds a stable operation key and a transaction. You can run it without API credentials, a model or a network connection. The side effect is a row in a temporary SQLite database. Nothing is sent anywhere. The useful result is a test you can adapt to your tool adapter: lose the response after the action succeeds, retry, and inspect both the action count and the returned result. What you will test The fixture represents one approved ticket creation. Seven scenarios examine what happens when callers repeat it, change its payload, arrive together or lose its stored replay record. These are deliberately constructed failures, not measurements of a commercial service or an estimate of real-world failure rates. Scenario Observed tickets What the result establishes Retry without a key after a lost response 2 A timeout can conceal a completed action Retry and replay with the original key 1 Repeated calls recover the original ticket ID Reuse that key with a different payload 1 The conflict is rejected before another write Fail before committing, then retry 1 The failed transaction leaves no partial ticket Eight callers use the same key 1 This local transaction serialises competing writers Use a new key with the same payload 2 Identical content can represent separate intentions Delete the replay record, then retry 2 Protection depends on retaining the record Run the experiment Download the lab ZIP below and extract it. From the extracted folder: cd engineering-labs/tool-retries python3 lab.py --output results.json Use Python 3.10 or newer with its standard sqlite3 module. There are no packages to install. The runner creates fresh temporary databases, checks every scenario and removes those databases on exit. The output option writes the evidence file at the named path; choose another filename if you want to preserve an earlier result. Observed output from the Fedora run: naive_retry: {"attempts": 2, "tickets": 2} stable_key_retry: {"attempts": 3, "replay_ticket_id": 1, "tickets": 1} changed_payload: {"rejected": true, "tickets": 1} failure_before_commit: {"tickets_after_retry": 1, "tickets_before_retry": 0} concurrent_same_key: {"attempts": 8, "returned_ticket_ids": [1], "tickets": 1} new_key_same_payload: {"tickets": 2} deleted_replay_record: {"tickets": 2} PASS: 7 scenarios results.json also records the Python and SQLite versions. The final line is printed only after the assertions succeed. A failed assertion or unexpected exception exits unsuccessfully. Run normally, without Python's optimisation flags, which disable assertions. Put the failure in the right place In create_ticket , the injected timeout comes after COMMIT . The database has already recorded the action when the caller sees an exception. Raising before the write would test a different, easier case: retrying work that never happened. The naive scenario sends the request again without a key. Its two rows are the failure demonstration. The stable-key scenario sends the same request three times, including the attempt whose response was lost. It checks that there is one ticket, one replay record and the same ticket ID on both successful responses. Checking only for a successful retry would miss the duplicate. Checking only the row count would miss an adapter that returns the wrong resource. Keep both assertions when you adapt the test. Make one key mean one intention The fixture's approved-ticket-001 identifies one operation. It is intentionally fixed for reproducibility. In an application, create and persist a unique identifier when the operation is accepted, before dispatch. Reuse that identifier through transport retries, queue redelivery and process recovery. A new attempt must not silently become a new operation. Do not derive identity solely from the request body. Two genuinely separate tickets might have identical descriptions. AWS explains this distinction between repeated parameters and repeated intent, and describes caller-supplied request identifiers as its preferred API approach. AWS Builders' Library The lab stores canonical JSON alongside the key. A matching key with a changed priority raises PayloadConflict . JSON key order does not change this comparison; changes to values do. This is a deliberately small payload contract, not a general solution to semantic equivalence. A production key also needs an appropriate namespace, such as tenant and operation type. Authentication must still determine who can perform the operation and retrieve its result. Possessing a key is not authorisation. Do not ask the model to invent a replacement key after an uncertain timeout. Keep the action and replay record together The ticket insert and replay record insert share one transaction. The runner injects a failure between those writes and verifies that both tables remain empty before retrying.

### [Recover an outbox after a crash](https://swarmsignal.net/engineering-lab-outbox-recovery/)

Published: 2026-09-08. Updated: 2026-09-08. Excerpt follows.

Run it. Inspect it. Change it. Python standard library · No API key · Synthetic fixtures A tool worker has two records to write: the action in another service, and the completion marker in its own database. If it crashes between those writes, its replacement cannot infer what happened from a pending status alone. This lab makes that gap visible. It uses two SQLite files and separate Python processes. We kill the dispatcher with SIGKILL after the downstream service commits a synthetic ticket, then start a new dispatcher. With downstream deduplication, recovery returns the original ticket. Disable deduplication and the same recovery creates a duplicate. The first tool-retries lab tested a ticket and its replay record inside one database transaction. Here, the caller and downstream service have independent transactions. Everything remains offline: no model calls, credentials or external tickets. Run the crash experiment Download the ZIP below and extract it. In the extracted engineering-labs/outbox-recovery folder, run: python3 lab.py --output results.json python3 -m unittest -v test_lab.py You need Python 3.10 or newer with SQLite support on Linux or macOS. Windows is not supported because the fixture explicitly uses POSIX SIGKILL . There are no third-party packages. The default run creates temporary databases and removes them afterwards. To keep the stores for inspection, choose a directory that does not already exist: python3 lab.py --directory ./run-001 --output results-001.json The evidence option overwrites the named JSON file. The directory option refuses an existing directory, so it cannot silently reuse an earlier experiment's state. Each child has a 15-second test deadline. The runner only kills child processes it created; it does not search for or stop other workers. What actually happened Observed on Fedora with Python 3.14.3 and SQLite 3.51.2: Failure injected Tickets before recovery Tickets after recovery Recovered result Dispatcher killed before dispatch 0 1 Ticket 1 Dispatcher killed after downstream commit 1 1 Ticket 1 Same crash, downstream deduplication disabled 1 2 Ticket 2 The acceptance crash test left zero operations and zero outbox entries before retrying. After resubmission and dispatch, it produced one ticket. A separate request with the original operation ID and a changed priority was rejected; the downstream store still contained one ticket. The normal runner ends with: PASS: 5 scenarios; real SIGKILL, separate caller and downstream stores results.json preserves the full before-and-after rows, child process IDs, return codes and recovered responses. A killed child returns -9 to Python on this platform. The negative control is a passing test because it successfully demonstrates the expected duplicate, not because duplication is acceptable behaviour. Follow the two transactions The caller accepts operation-001 by inserting its payload into operations and a pending entry into outbox inside one transaction. The outbox is a durable record of work still to dispatch. It is not an in-memory retry queue. A separate dispatcher reads that committed entry, closes the caller database connection and starts the downstream process. The downstream service writes a ticket and a replay receipt in its own transaction. That receipt maps the original operation ID and payload to the ticket result. After receiving the downstream result, the dispatcher updates the local operation and outbox entry to complete together. There is no transaction spanning both files. That is the point of the experiment: recovery must work across the gap between independent commits. AWS's transactional outbox guidance describes recording application state and the outgoing event together, then delivering committed entries separately. It also warns that duplicate deliveries remain possible and recommends idempotent consumers. The pattern preserves delivery intent; the consumer's contract determines whether redelivery repeats the effect. AWS Prescriptive Guidance Put the crash after the irreversible step In the most useful scenario, the downstream child commits, prints its result and exits. Only then does the dispatcher kill itself, before marking local completion. The caller store says pending with no result, while the downstream store already contains ticket 1. This placement deliberately makes recovery uncertain from the caller's perspective. A crash before the downstream write would not exercise that uncertainty. A Python exception handled inside the dispatcher would also be weaker evidence than the abrupt process termination used here. The replacement dispatcher obtains the same operation ID from the persisted outbox. The downstream child recognises that ID and replays ticket 1. The caller stores that result and marks completion. A further dispatcher invocation finds no pending work and changes neither store. The checks compare the returned ticket ID with the downstream receipt and caller result. A successful exit status alone would not detect a wrong result or a second ticket. Why the duplicate control matters Run the identical post-commit crash with deduplication disabled. The replacement dispatcher still reads the original outbox entry, still uses the same operation ID and still completes successfully. But the downstream service creates ticket 2.

### [Test pgvector filtering and recall](https://swarmsignal.net/engineering-lab-pgvector-filtering/)

Published: 2026-09-08. Updated: 2026-09-08. Excerpt follows.

Run it. Inspect it. Change it. Python + Docker + PostgreSQL/pgvector · No API key · Synthetic fixtures Your search asks for ten current documents belonging to one tenant. The database returns two. That might mean only two eligible documents exist. It might also mean the approximate index stopped before finding the others. This lab distinguishes those cases using real PostgreSQL and pgvector. It loads a fixed synthetic vector corpus, establishes an exact eligible top ten, then runs the same query through ordinary HNSW and an iterative HNSW scan. Every measured query has a saved execution plan. The result is useful precisely because it is limited: iterative scanning recovered the missing neighbours in this run, but an exact scan was faster at the tightest filters. Neither result establishes what your production corpus will do. What this experiment measures The checked-in fixture contains 20,000 vectors with 16 dimensions. Coordinates are seeded pseudorandom numbers, not text embeddings. Each row also has a tenant, a current-revision flag and a cohort number. There are three fixed query vectors. All three search modes use the same L2 distance and predicate: WHERE tenant = 'alpha' AND is_current AND cohort < 5 ORDER BY embedding <-> query_vector LIMIT 10 Here, query_vector represents the fixed vector literal supplied by the harness. The saved per-case SQL contains the complete executable query. We vary only the cohort cutoff: 100, 25, 5 and 1. Once tenant and current-revision conditions are included, the eligible populations are 8,000, 2,011, 427 and 87 rows. These are measured fixture counts, not assumed selectivities. This extends our retrieval freshness lab . That earlier example isolates eligibility and candidate starvation with lexical matching. This one executes an actual approximate vector index. Neither authenticates a caller or validates the authority of document permissions. Run it yourself Download the lab and use Python 3.10 or later plus a working Docker engine: cd engineering-labs/pgvector-filtering python3 run.py --output my-run python3 verify_results.py my-run The runner refuses to overwrite an existing evidence directory. It creates its own disposable PostgreSQL container, limits it to two CPUs and 2 GB of memory, exposes no ports and disables container networking. Database files live in a 1 GB temporary filesystem. Image downloading needs network access before the container starts. The checked image digest supplies PostgreSQL 17.8 and pgvector 0.8.1. This is a version-pinned experiment, not a recommendation to deploy that version. The digest, full version string, fixture hashes and settings are saved with the results. Stopping the owned container removes its temporary database; the runner verifies removal. The saved run includes 252 JSON plans from EXPLAIN (ANALYZE, BUFFERS) . You do not need an embedding service, model key or paid API. Keep the comparison honest The exact reference disables index and bitmap scans. Each plan must contain a sequential scan and no index. For the two HNSW modes, sequential scans are discouraged and the harness requires the named HNSW index to appear in every plan. This forces a controlled algorithm comparison; it does not measure which plan PostgreSQL would naturally choose. The index uses m=16 and ef_construction=64 . Both approximate modes use ef_search=40 . Ordinary HNSW has iterative scanning off. The iterative mode uses strict_order , a max_scan_tuples setting of 20,000 and a scan-memory multiplier of 1. pgvector documents that filtering happens after scanning an approximate index, so a restrictive condition can reduce the returned count. Its iterative scans can continue searching, subject to configured limits. Those limits and ordering modes matter; recovering ten rows is not a universal recall guarantee. pgvector 0.8.1 documentation The verifier independently computes exhaustive distances from the checked-in vectors, rounded to float32, and checks the exact reference IDs. It also checks eligibility, duplicates, overlap arithmetic, file hashes and every saved plan. The observed results This run used three queries at each filter cutoff. Each cell below lists the result for queries one, two and three. Eligible rows Eligible fraction Ordinary HNSW returned Ordinary HNSW overlap with exact top ten Iterative overlap with exact top ten 8,000 40% 10 / 10 / 10 10 / 10 / 10 10 / 10 / 10 2,011 10.055% 4 / 6 / 4 4 / 6 / 4 10 / 10 / 10 427 2.135% 0 / 2 / 0 0 / 2 / 0 10 / 10 / 10 87 0.435% 0 / 1 / 0 0 / 1 / 0 10 / 10 / 10 Recall at ten is overlap divided by ten. Returning ten documents alone would not establish perfect recall; they must be the reference neighbours. Here, the iterative mode returned and matched all ten for all twelve cases. At the tightest filter, ordinary HNSW returned no rows for two queries despite 87 eligible rows existing. The reference answers distinguish that retrieval miss from an empty eligible corpus.


> Last updated: 2026-09-08