Make tool retries safe

# Tool retries lab

An executable demonstration of an ambiguous timeout after a committed local action. It does not contact APIs or use credentials.

## Run

Requires Python 3.10+ with SQLite support. No dependencies.

```sh
cd engineering-labs/tool-retries
python3 lab.py --output results.json
python3 lab.py --help
```

`--output` overwrites the named JSON file. Each run uses fresh temporary databases that are removed on exit. Do not use `python -O`: scenario checks are assertions. Seven scenarios verify duplicate creation without a key, replay with a stable key, payload conflict rejection, pre-commit rollback, eight concurrent callers, separate operation keys and replay-record deletion. `results.json` is the observed Fedora run, not a forecast or external-service benchmark.

Files: `article.md` (publication draft), `lab.py` (implementation and checks), `fixture.json` (synthetic approved operation), `results.json` (observed execution), `SOURCES.md` (primary-source provenance).

The database transaction protects local rows only. It cannot atomically cover an external ticket API, email send or other remote action. Keys have no authentication power. The fixture is reproducible, not a production key-generation scheme. Production integration requires a provider-specific contract, tenant scoping, durable caller identity, retention policy, operational retry limits and recovery testing.

# Primary sources and evidence

Reviewed 8 September 2026. All prose and executable fixtures are original to this lab. No API calls, payments, messages or real customer operations were performed.

- Malcolm Featonby, AWS Builders' Library, [Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/). Opened and reviewed the sections on caller request identifiers, differing intent, parameter comparison and late requests. Supports the distinction between duplicate parameters and duplicate intent; does not establish this lab's performance.
- Stripe, [Idempotent requests](https://docs.stripe.com/api/idempotent_requests). Opened current official reference. Supports replay of saved status/body including 500 responses, parameter mismatch rejection, pruning after at least 24 hours and validation/concurrency exceptions. Lab deliberately does not emulate all Stripe behaviour or access Stripe.
- SQLite, [Transaction](https://www.sqlite.org/lang_transaction.html). Opened current official transaction documentation, last-updated date shown as 2026-02-18. Supports the single-writer model, BEGIN IMMEDIATE and possible SQLITE_BUSY behaviour. Python connection timeout controls waiting in this demonstration.

## Original evidence

`python3 lab.py --output results.json` executed successfully on Fedora. The JSON records runtime versions and all seven observed scenario results. Assertions check persistent ticket and replay-record counts and returned identifiers, not just absence of exceptions. The experiment injects Python exceptions; it does not emulate process kill, physical power loss, network partitions or remote API consistency.

No source text is reproduced verbatim in the article. Statements about external services are linked beside the relevant paragraph; the local results come from the checked-in executable rather than documentation inference.

# Make tool retries safe

An agent asks a tool to create a support ticket. The service creates it, but the response never reaches the agent. Retrying feels reasonable. Without a way to recognise the original request, the second call creates another ticket.

This lab reproduces that failure in a small Python program, then adds a stable operation key and a transaction. You can run it without API credentials, a model or a network connection. The side effect is a row in a temporary SQLite database. Nothing is sent anywhere.

The useful result is a test you can adapt to your tool adapter: lose the response **after** the action succeeds, retry, and inspect both the action count and the returned result.

## What you will test

The fixture represents one approved ticket creation. Seven scenarios examine what happens when callers repeat it, change its payload, arrive together or lose its stored replay record. These are deliberately constructed failures, not measurements of a commercial service or an estimate of real-world failure rates.

| Scenario | Observed tickets | What the result establishes |
| --- | ---: | --- |
| Retry without a key after a lost response | 2 | A timeout can conceal a completed action |
| Retry and replay with the original key | 1 | Repeated calls recover the original ticket ID |
| Reuse that key with a different payload | 1 | The conflict is rejected before another write |
| Fail before committing, then retry | 1 | The failed transaction leaves no partial ticket |
| Eight callers use the same key | 1 | This local transaction serialises competing writers |
| Use a new key with the same payload | 2 | Identical content can represent separate intentions |
| Delete the replay record, then retry | 2 | Protection depends on retaining the record |

## Run the experiment

Download the lab ZIP below and extract it. From the extracted folder:

```sh
cd engineering-labs/tool-retries
python3 lab.py --output results.json
```

Use Python 3.10 or newer with its standard `sqlite3` module. There are no packages to install. The runner creates fresh temporary databases, checks every scenario and removes those databases on exit. The output option writes the evidence file at the named path; choose another filename if you want to preserve an earlier result.

Observed output from the Fedora run:

```text
naive_retry: {"attempts": 2, "tickets": 2}
stable_key_retry: {"attempts": 3, "replay_ticket_id": 1, "tickets": 1}
changed_payload: {"rejected": true, "tickets": 1}
failure_before_commit: {"tickets_after_retry": 1, "tickets_before_retry": 0}
concurrent_same_key: {"attempts": 8, "returned_ticket_ids": [1], "tickets": 1}
new_key_same_payload: {"tickets": 2}
deleted_replay_record: {"tickets": 2}
PASS: 7 scenarios
```

`results.json` also records the Python and SQLite versions. The final line is printed only after the assertions succeed. A failed assertion or unexpected exception exits unsuccessfully. Run normally, without Python's optimisation flags, which disable assertions.

## Put the failure in the right place

In `create_ticket`, the injected timeout comes after `COMMIT`. The database has already recorded the action when the caller sees an exception. Raising before the write would test a different, easier case: retrying work that never happened.

The naive scenario sends the request again without a key. Its two rows are the failure demonstration. The stable-key scenario sends the same request three times, including the attempt whose response was lost. It checks that there is one ticket, one replay record and the same ticket ID on both successful responses.

Checking only for a successful retry would miss the duplicate. Checking only the row count would miss an adapter that returns the wrong resource. Keep both assertions when you adapt the test.

## Make one key mean one intention

The fixture's `approved-ticket-001` identifies one operation. It is intentionally fixed for reproducibility. In an application, create and persist a unique identifier when the operation is accepted, before dispatch. Reuse that identifier through transport retries, queue redelivery and process recovery. A new attempt must not silently become a new operation.

Do not derive identity solely from the request body. Two genuinely separate tickets might have identical descriptions. AWS explains this distinction between repeated parameters and repeated intent, and describes caller-supplied request identifiers as its preferred API approach. [AWS Builders' Library](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)

The lab stores canonical JSON alongside the key. A matching key with a changed priority raises `PayloadConflict`. JSON key order does not change this comparison; changes to values do. This is a deliberately small payload contract, not a general solution to semantic equivalence.

A production key also needs an appropriate namespace, such as tenant and operation type. Authentication must still determine who can perform the operation and retrieve its result. Possessing a key is not authorisation. Do not ask the model to invent a replacement key after an uncertain timeout.

## Keep the action and replay record together

The ticket insert and replay record insert share one transaction. The runner injects a failure between those writes and verifies that both tables remain empty before retrying.

`BEGIN IMMEDIATE` obtains SQLite's write transaction before checking for a previous request. SQLite permits only one simultaneous writer, so a competing caller waits or receives a busy error. This demo uses a ten-second connection timeout; its eight threads all reach a barrier before calling the tool. [SQLite transaction documentation](https://www.sqlite.org/lang_transaction.html)

This is a correctness example, not a throughput benchmark. Holding a database transaction while performing a slow external action is not an equivalent solution. If the ticket actually lives in another service, the local commit cannot make that remote write atomic with your replay record.

## Carry the contract across the tool boundary

For a remote tool, inspect its actual idempotency contract. Does it accept a caller key? Which operations support it? How long are keys retained? What happens when parameters change, or two requests arrive together?

Stripe provides a concrete example: its documented implementation replays the first saved status and body, including a `500`; rejects changed parameters for an existing key; and can prune keys after at least 24 hours. Validation failures and some concurrent conflicts do not save a result. Those are Stripe-specific behaviours, not promises made by this SQLite lab. [Stripe API reference](https://docs.stripe.com/api/idempotent_requests)

If a service cannot deduplicate requests or reliably locate a prior result, an ambiguous timeout may require reconciliation or review. An in-memory dictionary in your agent cannot establish that the remote action happened once.

## Use this as an adapter acceptance test

Replace the simulated ticket operation with a sandbox adapter and preserve the seven assertions. Add your service's documented retention and retry behaviours. Test recovery with the same persisted operation ID after restarting the caller. Keep normal retry budgets and deadlines separate from duplicate prevention.

The last scenario deliberately deletes the replay record. The same key then creates a second ticket. That boundary matters: deduplication lasts only as long as the system can recognise the operation. This lab proves a narrow local property under its tested failures. It does not provide universal exactly-once execution, test power loss or establish the behaviour of your external tool.


For the surrounding tool design, read [agent tool-use patterns](https://swarmsignal.net/agent-tool-use-patterns-guide/). Use the [agent evaluation checklist](https://swarmsignal.net/agent-evaluation-checklist/) to place this failure test alongside your other release checks.

## Sources

- [AWS Builders’ Library: Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
- [Stripe API reference: Idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [SQLite: Transaction](https://www.sqlite.org/lang_transaction.html)

{
  "operation_key": "approved-ticket-001",
  "payload": {
    "project": "demo-agent",
    "title": "Investigate a synthetic queue backlog",
    "priority": "normal"
  }
}

#!/usr/bin/env python3
"""A local SQLite retry experiment. No network calls or external side effects."""
import argparse
from concurrent.futures import ThreadPoolExecutor
from contextlib import closing
import json
from pathlib import Path
import platform
import sqlite3
import tempfile
import threading


class PayloadConflict(ValueError):
    pass


def initialise(path):
    with closing(sqlite3.connect(path)) as db:
        db.executescript('''
            CREATE TABLE tickets (id INTEGER PRIMARY KEY, payload TEXT NOT NULL);
            CREATE TABLE requests (
                key TEXT PRIMARY KEY, payload TEXT NOT NULL, response TEXT NOT NULL
            );
        ''')


def create_ticket(path, payload, key=None, fault=None):
    """Commit local ticket and replay record atomically; inject faults explicitly."""
    encoded = json.dumps(payload, sort_keys=True, separators=(',', ':'), allow_nan=False)
    with closing(sqlite3.connect(path, timeout=10, isolation_level=None)) as db:
        try:
            db.execute('BEGIN IMMEDIATE')
            previous = db.execute(
                'SELECT payload, response FROM requests WHERE key = ?', (key,)
            ).fetchone() if key is not None else None
            if previous:
                if previous[0] != encoded:
                    raise PayloadConflict('same key, different payload')
                response = json.loads(previous[1])
            else:
                cursor = db.execute('INSERT INTO tickets(payload) VALUES (?)', (encoded,))
                response = {'ticket_id': cursor.lastrowid}
                if fault == 'before_commit':
                    raise RuntimeError('injected failure before commit')
                if key is not None:
                    db.execute('INSERT INTO requests VALUES (?, ?, ?)',
                               (key, encoded, json.dumps(response, sort_keys=True)))
            db.execute('COMMIT')
        except Exception:
            if db.in_transaction:
                db.execute('ROLLBACK')
            raise
    if fault == 'after_commit':
        raise TimeoutError('server committed; response lost')
    return response


def count(path, table='tickets'):
    if table not in ('tickets', 'requests'):
        raise ValueError('unknown table')
    with closing(sqlite3.connect(path)) as db:
        return db.execute('SELECT COUNT(*) FROM ' + table).fetchone()[0]


def expect_error(kind, function):
    try:
        function()
    except kind:
        return
    raise AssertionError('expected ' + kind.__name__)


def run():
    fixture = json.loads(Path(__file__).with_name('fixture.json').read_text())
    payload, key = fixture['payload'], fixture['operation_key']
    scenarios = {}
    with tempfile.TemporaryDirectory(prefix='tool-retries-') as folder:
        def database(name):
            path = Path(folder) / (name + '.sqlite')
            initialise(path)
            return path

        naive = database('naive')
        expect_error(TimeoutError, lambda: create_ticket(naive, payload, fault='after_commit'))
        create_ticket(naive, payload)
        assert count(naive) == 2
        scenarios['naive_retry'] = {'attempts': 2, 'tickets': count(naive)}

        stable = database('stable')
        expect_error(TimeoutError, lambda: create_ticket(stable, payload, key, 'after_commit'))
        first = create_ticket(stable, payload, key)
        replay = create_ticket(stable, payload, key)
        assert first == replay == {'ticket_id': 1}
        assert count(stable) == count(stable, 'requests') == 1
        scenarios['stable_key_retry'] = {'attempts': 3, 'tickets': count(stable), 'replay_ticket_id': replay['ticket_id']}

        expect_error(PayloadConflict, lambda: create_ticket(stable, {**payload, 'priority': 'urgent'}, key))
        assert count(stable) == 1
        scenarios['changed_payload'] = {'rejected': True, 'tickets': count(stable)}

        rollback = database('rollback')
        expect_error(RuntimeError, lambda: create_ticket(rollback, payload, key, 'before_commit'))
        assert count(rollback) == count(rollback, 'requests') == 0
        create_ticket(rollback, payload, key)
        assert count(rollback) == count(rollback, 'requests') == 1
        scenarios['failure_before_commit'] = {'tickets_before_retry': 0, 'tickets_after_retry': count(rollback)}

        concurrent = database('concurrent')
        barrier = threading.Barrier(8, timeout=10)
        def simultaneous(_):
            barrier.wait()
            return create_ticket(concurrent, payload, key)
        with ThreadPoolExecutor(max_workers=8) as pool:
            responses = list(pool.map(simultaneous, range(8)))
        ids = sorted({response['ticket_id'] for response in responses})
        assert ids == [1] and count(concurrent) == count(concurrent, 'requests') == 1
        scenarios['concurrent_same_key'] = {'attempts': 8, 'tickets': count(concurrent), 'returned_ticket_ids': ids}

        new_key = database('new_key')
        create_ticket(new_key, payload, key)
        create_ticket(new_key, payload, key + '-new-intent')
        assert count(new_key) == 2
        scenarios['new_key_same_payload'] = {'tickets': count(new_key)}

        expired = database('expired')
        create_ticket(expired, payload, key)
        with closing(sqlite3.connect(expired)) as db:
            db.execute('DELETE FROM requests WHERE key = ?', (key,))
            db.commit()
        create_ticket(expired, payload, key)
        assert count(expired) == 2
        scenarios['deleted_replay_record'] = {'tickets': count(expired)}
    return {'experiment': 'local-sqlite-tool-retries-v1', 'python': platform.python_version(),
            'sqlite': sqlite3.sqlite_version, 'passed': True, 'scenarios': scenarios}


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument('--output', type=Path, help='Write an evidence JSON file (overwrites that file).')
    args = parser.parse_args()
    results = run()
    if args.output:
        args.output.write_text(json.dumps(results, indent=2) + '\n')
    for name, values in results['scenarios'].items():
        print(name + ': ' + json.dumps(values, sort_keys=True))
    print('PASS: 7 scenarios')


if __name__ == '__main__':
    main()

{
  "experiment": "local-sqlite-tool-retries-v1",
  "python": "3.14.3",
  "sqlite": "3.51.2",
  "passed": true,
  "scenarios": {
    "naive_retry": {
      "attempts": 2,
      "tickets": 2
    },
    "stable_key_retry": {
      "attempts": 3,
      "tickets": 1,
      "replay_ticket_id": 1
    },
    "changed_payload": {
      "rejected": true,
      "tickets": 1
    },
    "failure_before_commit": {
      "tickets_before_retry": 0,
      "tickets_after_retry": 1
    },
    "concurrent_same_key": {
      "attempts": 8,
      "tickets": 1,
      "returned_ticket_ids": [
        1
      ]
    },
    "new_key_same_payload": {
      "tickets": 2
    },
    "deleted_replay_record": {
      "tickets": 2
    }
  }
}