Replay answer selection

# Answer verification lab

Start with the saved-trace replay. All six tasks and scores are invented teaching data, not model measurements. Python 3.10+; standard library only; no API calls or credentials.

```sh
python3 replay.py --help
python3 replay.py
python3 replay.py --threshold 80 --output threshold-80.json
```

`traces.json` has two calibration and four test tasks. `replay-results.json` records three policies at budgets 1, 2 and 4, including answer coverage, correctness and per-task selected IDs. `article.md` explains how to import your own traces and keep evaluation labels independent of selection. The threshold is illustrative, not calibrated.

The script checks label independence, stable rank ties, abstention, metric bounds and nested pool availability. Bad thresholds exit 2. Candidate IDs must be unique within each task; scores must be numeric 0..100 and correctness labels boolean. The default sweep requires four candidates per task. Never replace independent correctness labels with the verifier's own assessment.

An optional exact calculation explains the failure mechanism:

```sh
python3 run.py --help
python3 run.py
```

`fixture.json` describes three weighted answer archetypes; `results.json` is generated with exact rational arithmetic. The programme checks its formula against exhaustive enumeration for N=1..5, probability bounds and monotonic pool availability for N=1..32, and an oracle-ranking control. A zero candidate weight exits 2 with a positive-weight error. Distinct proxy scores are required by this optional fixture.

`SOURCES.md` records primary-source provenance. No result is a production forecast. For real policy choices add measured cost, latency, enough held-out tasks and uncertainty estimates, as described in the article.

# Source provenance

Reviewed 8 September 2026. Primary publication abstracts opened directly with the web research tool. No third-party summaries used as evidence.

- Gao, Schulman and Hilton, *Scaling Laws for Reward Model Overoptimization*, arXiv:2210.10760v1, 19 October 2022. https://arxiv.org/abs/2210.10760v1
  - Supports: optimisation of a proxy reward model can reduce performance under the study's gold reward model; study includes best-of-N sampling and reinforcement learning.
  - Boundary: the gold reward model is a stand-in for human preferences. This lab neither reproduces the paper nor copies its fitted scaling laws.
- Lightman et al., *Let's Verify Step by Step*, arXiv:2305.20050v1, 31 May 2023. https://arxiv.org/abs/2305.20050v1
  - Supports: distinction between final-outcome and intermediate-step supervision; reported process-supervision advantage in the paper's MATH experiments.
  - Boundary: does not establish a universal advantage on arbitrary engineering tasks.

All fixture answers, frequencies and proxy scores are original synthetic teaching data created for this lab. Result percentages are computed exactly from that fixture using rational arithmetic, not taken from a paper, sampled from a model, or estimated from users.

# Cost and deadline accounting extension

This accompanies the existing Swarm Signal answer-verification lab. It is included in the lab ZIP under the accounting subfolder and does not replace the original candidate replay.

**Measured:** actual local Python sleep-job start/end spans, recorded with a monotonic clock on Fedora. **Invented:** usage quantities, unit prices, verifier scores, labels and candidate work delays. No language model or paid service is called. These timings are not inference latency and these amounts are not bills.

## Run

Python 3.10+, standard library only. From this directory:

```sh
python3 record_demo.py --help
python3 account.py --help
python3 record_demo.py --output-dir my-run
python3 account.py --directory my-run
python3 -m unittest -v test_account.py
```

The supplied `sample/` directory preserves one actual execution. To reproduce its accounting without changing its timings, run `python3 account.py`. Fresh recordings have different wall times and can change which candidates meet the fixed deadline on a busy machine.

## Files

- `fixture.json`: all synthetic inputs, frozen policies and price provenance.
- `record_demo.py`: executes four overlapping workers per task; successful candidates receive a second local verification stage, and one deliberately fails.
- `account.py`: validates spans, counts usage from every call, selects only candidates whose verification has finished, then applies independent labels.
- `sample/traces.json`: actual recorded spans plus explicitly synthetic usage and scores.
- `sample/labels.json`: evaluation labels, passed separately from selection.
- `sample/policies.json`: predeclared all-completed and 50ms policies. Neither is fitted on these tasks.
- `sample/results.json`: accounting and per-task decisions; hashes bind it to the three input files.
- `article-section.md`: concise extension for the existing page.
- `test_account.py`: fresh-runtime, accounting, label-independence and malformed-trace checks.

## Accounting contract

Every observed generation and verification call contributes input/output usage, including failed calls. Missing usage is an error, not zero cost. The example estimates costs using a declared USD price per million units. It intentionally does not ingest invoices, retries with different rates, cache pricing, minimum fees or reported provider bill totals. Extend that schema explicitly before using it for bill reconciliation.

Task `elapsed_ms` is observed from task launch until the executor has completed. Summed call duration counts each worker's occupied time and can exceed wall time because spans overlap. Each start/end is relative to the same task origin.

Deadline replay changes answer eligibility only. It retains all recorded call costs for every policy. `cancellation_savings_usd: null` means unknown; it does not imply cancellation saves nothing. A saved trace alone cannot prove which billed work a different scheduler would avoid.

## Import your own traces

Follow `sample/traces.json`: unique task/span IDs, test split, nonnegative finite relative milliseconds, generation/verification role, success/failed status, usage on every call, and a score on successful verification. One generation and at most one verification span per candidate are supported. The verifier must start after generation finishes. Add a separate candidate ID for an additional attempt in this minimal format.

Use `provider_usage` for authentic usage provenance and `estimated_unit_prices` with a price date when using a price estimate. Independent correctness labels belong in `labels.json`, never in the selector. Freeze policy files using separate calibration data before evaluating held-out tasks. Label flips cannot alter selected IDs; the tests enforce that property.

All-completed answers are selected from the full recorded pool. A deadline answers from verified candidates finished by its cut-off, or abstains. This is a counterfactual availability calculation, not a measured deadline service. Do not claim cost or latency savings without running and measuring the actual policy.

# Primary implementation sources

Reviewed 8 September 2026:

- Python time documentation: https://docs.python.org/3/library/time.html#time.perf_counter_ns . Performance-counter differences include elapsed sleep; nanosecond values avoid floating-point counter precision loss before differences are taken.
- Python concurrent.futures documentation: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor . Used for explicitly bounded local worker execution.

The example contains no researched provider price or billing claim. Fixture prices, usage, scores and correctness are invented. Local execution supplies only recorded timings. No paper result is represented as evidence for this fixture.

#!/usr/bin/env python3
"""Account for all recorded calls, then replay frozen selection deadlines."""
import argparse
from decimal import Decimal, InvalidOperation
import hashlib
import json
import math
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def require(condition, message):
    if not condition:
        raise ValueError(message)

def number(value):
    return type(value) in (int, float) and math.isfinite(value)

def validate(data, labels, policies):
    require(data.get("currency") == "USD", "This example requires USD; convert explicitly before import")
    require(data.get("price_provenance") in ("synthetic_fixture", "estimated_unit_prices"), "Declare price provenance")
    require(isinstance(data.get("price_date"), str) and bool(data["price_date"]), "Price date required")
    for name in ("input_per_million", "output_per_million"):
        value = Decimal(data["prices"][name])
        require(value.is_finite() and value >= 0, "Prices must be finite and nonnegative")
    tasks = data["tasks"]
    require(bool(tasks), "Tasks cannot be empty")
    require(len({t["id"] for t in tasks}) == len(tasks), "Duplicate task IDs")
    require(bool(policies) and len({p["name"] for p in policies}) == len(policies), "Policies need unique names")
    for p in policies:
        require(p["deadline_ms"] is None or (number(p["deadline_ms"]) and p["deadline_ms"] >= 0), "Invalid deadline")
    for task in tasks:
        require(task["split"] == "test", "This replay accepts frozen held-out test tasks only")
        require(number(task["elapsed_ms"]) and task["elapsed_ms"] >= 0, "Invalid elapsed time")
        spans = task["spans"]
        require(bool(spans) and len({s["id"] for s in spans}) == len(spans), "Duplicate or empty spans")
        generation = {}
        verification = {}
        for s in spans:
            require(s["role"] in ("generation", "verification") and s["status"] in ("success", "failed"), "Invalid span type")
            require(number(s["start_ms"]) and number(s["end_ms"]) and 0 <= s["start_ms"] <= s["end_ms"] <= task["elapsed_ms"], "Invalid span boundaries")
            require(s.get("usage_provenance") in ("synthetic_fixture", "provider_usage"), "Every call needs usage provenance, including failures")
            for unit in ("input_units", "output_units"):
                require(type(s.get(unit)) is int and s[unit] >= 0, "Every call needs nonnegative integer usage, including failures")
            bucket = generation if s["role"] == "generation" else verification
            require(s["candidate_id"] not in bucket, "Only one generation and verification span per candidate")
            bucket[s["candidate_id"]] = s
            if s["role"] == "verification" and s["status"] == "success":
                require(number(s["score"]), "Verifier score must be finite")
        for cid, verifier in verification.items():
            require(cid in generation and generation[cid]["status"] == "success", "Verifier needs successful generation")
            require(verifier["start_ms"] >= generation[cid]["end_ms"], "Verifier starts before generation completes")
        require(task["id"] in labels, "Missing task labels")
        for cid in generation:
            require(type(labels[task["id"]].get(cid)) is bool, "Missing boolean correctness label")

def cost(span, prices):
    return (Decimal(span["input_units"]) * Decimal(prices["input_per_million"]) +
            Decimal(span["output_units"]) * Decimal(prices["output_per_million"])) / Decimal(1_000_000)

def select(task, deadline):
    # Evaluation labels are intentionally absent from this function.
    ready = [s for s in task["spans"] if s["role"] == "verification" and s["status"] == "success"
             and (deadline is None or s["end_ms"] <= deadline)]
    # Tie-break by candidate ID, independently of labels and trace serialisation order.
    return min(ready, key=lambda s: (-s["score"], s["candidate_id"]))["candidate_id"] if ready else None

def analyse(data, labels, policies):
    validate(data, labels, policies)
    prices = data["prices"]
    tasks = []
    for task in data["tasks"]:
        total = sum((cost(s, prices) for s in task["spans"]), Decimal(0))
        verifier = sum((cost(s, prices) for s in task["spans"] if s["role"] == "verification"), Decimal(0))
        failed = sum((cost(s, prices) for s in task["spans"] if s["status"] == "failed"), Decimal(0))
        tasks.append({"task_id": task["id"], "recorded_wall_ms": round(task["elapsed_ms"], 4),
                      "summed_call_ms": round(sum(s["end_ms"]-s["start_ms"] for s in task["spans"]), 4),
                      "estimated_cost_usd": str(total), "verification_cost_usd": str(verifier),
                      "failed_call_cost_usd": str(failed), "call_count": len(task["spans"])})
    total_cost = sum((Decimal(t["estimated_cost_usd"]) for t in tasks), Decimal(0))
    results = []
    for policy in policies:
        decisions = []
        for task in data["tasks"]:
            selected = select(task, policy["deadline_ms"])
            decisions.append({"task_id": task["id"], "selected_id": selected,
                              "correct": labels[task["id"]][selected] if selected is not None else False})
        answered = sum(d["selected_id"] is not None for d in decisions)
        correct = sum(d["correct"] for d in decisions)
        results.append({"policy": policy["name"], "counterfactual_deadline_ms": policy["deadline_ms"],
                        "attempted": len(tasks), "answered": answered, "correct": correct,
                        "correct_per_attempt": correct / len(tasks),
                        "accuracy_when_answered": correct / answered if answered else None,
                        "accounted_cost_usd": str(total_cost),
                        "estimated_cost_per_correct_usd": str(total_cost / correct) if correct else None,
                        "cancellation_savings_usd": None, "decisions": decisions})
    return {"kind": data["kind"], "price_provenance": data["price_provenance"],
            "cost_scope": "all observed calls retained for every policy; cancellation savings unknown",
            "latency_scope": "recorded wall time is actual local work; deadlines are counterfactual cut-offs, not measured latency",
            "tasks": tasks, "policies": results}

def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--directory", type=Path, default=ROOT / "sample")
    parser.add_argument("--output", type=Path)
    args = parser.parse_args()
    try:
        inputs = {name: (args.directory / (name + ".json")).read_bytes() for name in ("traces", "labels", "policies")}
        result = analyse(*(json.loads(inputs[name]) for name in ("traces", "labels", "policies")))
    except (ValueError, KeyError, TypeError, InvalidOperation) as error:
        parser.error(str(error))
    result["input_sha256"] = {name: hashlib.sha256(value).hexdigest() for name, value in inputs.items()}
    output = args.output or args.directory / "results.json"
    output.write_text(json.dumps(result, indent=2) + "\n")
    for task in result["tasks"]:
        print(task["task_id"], "wall_ms", task["recorded_wall_ms"], "sum_call_ms", task["summed_call_ms"],
              "estimated_USD", task["estimated_cost_usd"], "failed_USD", task["failed_call_cost_usd"],
              "verifier_USD", task["verification_cost_usd"])
    for policy in result["policies"]:
        print(policy["policy"], "answered", policy["answered"], "correct", policy["correct"],
              "accounted_USD", policy["accounted_cost_usd"], "cancellation savings unknown")

if __name__ == "__main__":
    main()

## Add cost and deadline accounting

Generating four candidates does not tell you what an answer costs. The accounting extension adds failed calls, verification overhead and overlapping spans to the existing replay. It also shows why a deadline is not evidence of a billing saving.

**This is a local execution exercise.** We measured Python workers that sleep for short intervals on Fedora. Usage quantities, prices, scores and correctness labels are invented. There are no model calls, provider bills or measured inference latencies. The useful result is the accounting method you can inspect and adapt.

Download the lab ZIP below, open `engineering-labs/answer-verification/accounting`, and run:

```sh
python3 record_demo.py --output-dir my-run
python3 account.py --directory my-run
python3 -m unittest -v test_account.py
```

Python 3.10 or newer is enough. The supplied sample preserves the original execution. Running `python3 account.py` replays that sample without recording new timings.

Each task launches four overlapping candidate workers. Three complete and then run a verification stage; one deliberately fails. Every call has usage, including the failure. The synthetic rates are $0.50 per million input units and $2.00 per million output units, dated 8 September 2026. These are teaching values, not a provider quotation.

The saved execution produced:

| Task | Observed local wall time | Summed call duration | Estimated total | Failed call | Verification |
| --- | --- | --- | --- | --- | --- |
| heldout-1 | 70.6542ms | 150.4078ms | $0.00043 | $0.00004 | $0.00012 |
| heldout-2 | 70.5080ms | 150.3999ms | $0.00043 | $0.00004 | $0.00012 |

Failed-call and verification amounts are parts of the total, not additional charges. Summing call durations would overstate elapsed time because workers overlap. These exact timings belong to the saved run; another machine or a busy scheduler will produce different values.

Two frozen policies see the same traces. All-completed selects the highest-scoring verified candidate after all work finishes. The 50ms policy considers only candidates whose verification finished by that cut-off. Its selection function cannot access correctness labels; those live in a separate file and are applied afterwards.

In this synthetic run, all-completed returned two correct answers. The deadline policy returned two wrong answers. Both accounting rows retain the full $0.00086 of observed work. That is deliberate: replaying an earlier cut-off does not establish that ongoing requests would have been cancelled, or that cancellation would have reduced billed usage.

The output therefore marks cancellation savings as unknown. It also keeps the deadline separate from observed wall time. The programme has not measured a service returning within 50ms; it has inspected which answers were available then.

To adapt this for real evaluation, import sanitised spans with a common task clock, versioned policy settings and usage provenance. Include unsuccessful requests and verifier calls. Missing usage should trigger investigation rather than silently becoming zero. The supplied calculator supports a simple unit-price estimate; extend it explicitly for recorded bills, caching, retries or provider-specific fees.

Develop policies on separate calibration data, then freeze them before evaluating held-out tasks. Report correct outcomes per attempted task, abstentions and estimated cost per correct outcome together. When there are no correct outcomes, cost per correct is undefined, so the output uses null.

The regression suite records fresh local spans, checks the arithmetic, flips labels to prove selection is unchanged, and rejects malformed or incomplete traces. The exercise gives you a way to test the accounting before applying it to real requests. It makes no claim that the illustrated deadline, prices or outcome rates suit a production workload.

Implementation references: [Python performance counter](https://docs.python.org/3/library/time.html#time.perf_counter_ns) and [thread pool executor](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor).

{
  "description": "Invented usage, prices, scores and answer labels; workload is local sleep, not inference.",
  "price_date": "2026-09-08",
  "currency": "USD",
  "prices": {
    "input_per_million": "0.50",
    "output_per_million": "2.00"
  },
  "policies": [
    {
      "name": "all_completed",
      "deadline_ms": null
    },
    {
      "name": "deadline_50ms",
      "deadline_ms": 50
    }
  ],
  "tasks": [
    {
      "id": "heldout-1",
      "split": "test"
    },
    {
      "id": "heldout-2",
      "split": "test"
    }
  ],
  "candidates": [
    {
      "id": "a",
      "sleep_ms": 10,
      "status": "success",
      "score": 60,
      "correct": true,
      "input_units": 100,
      "output_units": 20
    },
    {
      "id": "b",
      "sleep_ms": 30,
      "status": "success",
      "score": 90,
      "correct": false,
      "input_units": 100,
      "output_units": 20
    },
    {
      "id": "c",
      "sleep_ms": 60,
      "status": "success",
      "score": 95,
      "correct": true,
      "input_units": 100,
      "output_units": 20
    },
    {
      "id": "d",
      "sleep_ms": 20,
      "status": "failed",
      "score": null,
      "correct": false,
      "input_units": 80,
      "output_units": 0
    }
  ],
  "verification": {
    "sleep_ms": 10,
    "input_units": 60,
    "output_units": 5
  }
}

#!/usr/bin/env python3
"""Record actual local sleep spans; all usage, scores and prices are synthetic."""
import argparse
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import platform
import time

ROOT = Path(__file__).resolve().parent

def record(fixture):
    traces = []
    labels = {}
    for task in fixture["tasks"]:
        started = time.perf_counter_ns()
        def now():
            return (time.perf_counter_ns() - started) / 1_000_000
        def candidate_work(c):
            spans = []
            start = now()
            time.sleep(c["sleep_ms"] / 1000)
            end = now()
            spans.append({"id": c["id"] + "-generate", "candidate_id": c["id"], "role": "generation",
                          "status": c["status"], "start_ms": start, "end_ms": end,
                          "input_units": c["input_units"], "output_units": c["output_units"],
                          "usage_provenance": "synthetic_fixture"})
            if c["status"] == "success":
                start = now()
                time.sleep(fixture["verification"]["sleep_ms"] / 1000)
                end = now()
                spans.append({"id": c["id"] + "-verify", "candidate_id": c["id"], "role": "verification",
                              "status": "success", "start_ms": start, "end_ms": end,
                              "input_units": fixture["verification"]["input_units"],
                              "output_units": fixture["verification"]["output_units"],
                              "usage_provenance": "synthetic_fixture", "score": c["score"]})
            return spans
        with ThreadPoolExecutor(max_workers=4) as pool:
            groups = list(pool.map(candidate_work, fixture["candidates"]))
        elapsed = now()
        traces.append({"id": task["id"], "split": task["split"], "elapsed_ms": elapsed,
                       "spans": sorted([span for group in groups for span in group], key=lambda s: s["id"])})
        labels[task["id"]] = {c["id"]: c["correct"] for c in fixture["candidates"]}
    return traces, labels

def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--output-dir", type=Path, default=ROOT / "sample")
    args = parser.parse_args()
    fixture_bytes = (ROOT / "fixture.json").read_bytes()
    fixture = json.loads(fixture_bytes)
    traces, labels = record(fixture)
    args.output_dir.mkdir(parents=True, exist_ok=True)
    (args.output_dir / "traces.json").write_text(json.dumps({
        "kind": "actual local sleep timings; synthetic scores, usage and prices",
        "recorded_at": datetime.now(timezone.utc).isoformat(), "python_version": platform.python_version(),
        "fixture_sha256": hashlib.sha256(fixture_bytes).hexdigest(),
        "price_provenance": "synthetic_fixture", "price_date": fixture["price_date"],
        "currency": fixture["currency"], "prices": fixture["prices"], "tasks": traces}, indent=2) + "\n")
    (args.output_dir / "labels.json").write_text(json.dumps(labels, indent=2) + "\n")
    (args.output_dir / "policies.json").write_text(json.dumps(fixture["policies"], indent=2) + "\n")
    print("Recorded", len(traces), "tasks with real local spans; synthetic usage only.")

if __name__ == "__main__":
    main()

{
  "heldout-1": {
    "a": true,
    "b": false,
    "c": true,
    "d": false
  },
  "heldout-2": {
    "a": true,
    "b": false,
    "c": true,
    "d": false
  }
}

[
  {
    "name": "all_completed",
    "deadline_ms": null
  },
  {
    "name": "deadline_50ms",
    "deadline_ms": 50
  }
]

{
  "kind": "actual local sleep timings; synthetic scores, usage and prices",
  "price_provenance": "synthetic_fixture",
  "cost_scope": "all observed calls retained for every policy; cancellation savings unknown",
  "latency_scope": "recorded wall time is actual local work; deadlines are counterfactual cut-offs, not measured latency",
  "tasks": [
    {
      "task_id": "heldout-1",
      "recorded_wall_ms": 70.6542,
      "summed_call_ms": 150.4078,
      "estimated_cost_usd": "0.00043",
      "verification_cost_usd": "0.00012",
      "failed_call_cost_usd": "0.00004",
      "call_count": 7
    },
    {
      "task_id": "heldout-2",
      "recorded_wall_ms": 70.508,
      "summed_call_ms": 150.3999,
      "estimated_cost_usd": "0.00043",
      "verification_cost_usd": "0.00012",
      "failed_call_cost_usd": "0.00004",
      "call_count": 7
    }
  ],
  "policies": [
    {
      "policy": "all_completed",
      "counterfactual_deadline_ms": null,
      "attempted": 2,
      "answered": 2,
      "correct": 2,
      "correct_per_attempt": 1.0,
      "accuracy_when_answered": 1.0,
      "accounted_cost_usd": "0.00086",
      "estimated_cost_per_correct_usd": "0.00043",
      "cancellation_savings_usd": null,
      "decisions": [
        {
          "task_id": "heldout-1",
          "selected_id": "c",
          "correct": true
        },
        {
          "task_id": "heldout-2",
          "selected_id": "c",
          "correct": true
        }
      ]
    },
    {
      "policy": "deadline_50ms",
      "counterfactual_deadline_ms": 50,
      "attempted": 2,
      "answered": 2,
      "correct": 0,
      "correct_per_attempt": 0.0,
      "accuracy_when_answered": 0.0,
      "accounted_cost_usd": "0.00086",
      "estimated_cost_per_correct_usd": null,
      "cancellation_savings_usd": null,
      "decisions": [
        {
          "task_id": "heldout-1",
          "selected_id": "b",
          "correct": false
        },
        {
          "task_id": "heldout-2",
          "selected_id": "b",
          "correct": false
        }
      ]
    }
  ],
  "input_sha256": {
    "traces": "8c928a7dd548feac754acb85b16ace2129058293377c5bd87392e8d5b7dc0c51",
    "labels": "037075fe81adc8a501ecefb026ae7d5950465e68ac93476690e39515263ca71d",
    "policies": "6ce45cbfb48641d80483c5af5b953aa1bbc0cce73b9c80c49938ba0463902db5"
  }
}

{
  "kind": "actual local sleep timings; synthetic scores, usage and prices",
  "recorded_at": "2026-09-08T19:11:18.978070+00:00",
  "python_version": "3.14.3",
  "fixture_sha256": "fe09987f596491306979e742cc9425de080a972782d821293204bf2bb7ee8619",
  "price_provenance": "synthetic_fixture",
  "price_date": "2026-09-08",
  "currency": "USD",
  "prices": {
    "input_per_million": "0.50",
    "output_per_million": "2.00"
  },
  "tasks": [
    {
      "id": "heldout-1",
      "split": "test",
      "elapsed_ms": 70.654226,
      "spans": [
        {
          "id": "a-generate",
          "candidate_id": "a",
          "role": "generation",
          "status": "success",
          "start_ms": 0.165461,
          "end_ms": 10.224749,
          "input_units": 100,
          "output_units": 20,
          "usage_provenance": "synthetic_fixture"
        },
        {
          "id": "a-verify",
          "candidate_id": "a",
          "role": "verification",
          "status": "success",
          "start_ms": 10.228109,
          "end_ms": 20.286437,
          "input_units": 60,
          "output_units": 5,
          "usage_provenance": "synthetic_fixture",
          "score": 60
        },
        {
          "id": "b-generate",
          "candidate_id": "b",
          "role": "generation",
          "status": "success",
          "start_ms": 0.274242,
          "end_ms": 30.333594,
          "input_units": 100,
          "output_units": 20,
          "usage_provenance": "synthetic_fixture"
        },
        {
          "id": "b-verify",
          "candidate_id": "b",
          "role": "verification",
          "status": "success",
          "start_ms": 30.337604,
          "end_ms": 40.404342,
          "input_units": 60,
          "output_units": 5,
          "usage_provenance": "synthetic_fixture",
          "score": 90
        },
        {
          "id": "c-generate",
          "candidate_id": "c",
          "role": "generation",
          "status": "success",
          "start_ms": 0.383403,
          "end_ms": 60.434517,
          "input_units": 100,
          "output_units": 20,
          "usage_provenance": "synthetic_fixture"
        },
        {
          "id": "c-verify",
          "candidate_id": "c",
          "role": "verification",
          "status": "success",
          "start_ms": 60.439697,
          "end_ms": 70.496054,
          "input_units": 60,
          "output_units": 5,
          "usage_provenance": "synthetic_fixture",
          "score": 95
        },
        {
          "id": "d-generate",
          "candidate_id": "d",
          "role": "generation",
          "status": "failed",
          "start_ms": 0.470514,
          "end_ms": 20.527129,
          "input_units": 80,
          "output_units": 0,
          "usage_provenance": "synthetic_fixture"
        }
      ]
    },
    {
      "id": "heldout-2",
      "split": "test",
      "elapsed_ms": 70.508024,
      "spans": [
        {
          "id": "a-generate",
          "candidate_id": "a",
          "role": "generation",
          "status": "success",
          "start_ms": 0.12034,
          "end_ms": 10.176838,
          "input_units": 100,
          "output_units": 20,
          "usage_provenance": "synthetic_fixture"
        },
        {
          "id": "a-verify",
          "candidate_id": "a",
          "role": "verification",
          "status": "success",
          "start_ms": 10.179708,
          "end_ms": 20.237446,
          "input_units": 60,
          "output_units": 5,
          "usage_provenance": "synthetic_fixture",
          "score": 60
        },
        {
          "id": "b-generate",
          "candidate_id": "b",
          "role": "generation",
          "status": "success",
          "start_ms": 0.190841,
          "end_ms": 30.247833,
          "input_units": 100,
          "output_units": 20,
          "usage_provenance": "synthetic_fixture"
        },
        {
          "id": "b-verify",
          "candidate_id": "b",
          "role": "verification",
          "status": "success",
          "start_ms": 30.251173,
          "end_ms": 40.306691,
          "input_units": 60,
          "output_units": 5,
          "usage_provenance": "synthetic_fixture",
          "score": 90
        },
        {
          "id": "c-generate",
          "candidate_id": "c",
          "role": "generation",
          "status": "success",
          "start_ms": 0.254481,
          "end_ms": 60.313425,
          "input_units": 100,
          "output_units": 20,
          "usage_provenance": "synthetic_fixture"
        },
        {
          "id": "c-verify",
          "candidate_id": "c",
          "role": "verification",
          "status": "success",
          "start_ms": 60.316985,
          "end_ms": 70.374423,
          "input_units": 60,
          "output_units": 5,
          "usage_provenance": "synthetic_fixture",
          "score": 95
        },
        {
          "id": "d-generate",
          "candidate_id": "d",
          "role": "generation",
          "status": "failed",
          "start_ms": 0.315982,
          "end_ms": 20.372757,
          "input_units": 80,
          "output_units": 0,
          "usage_provenance": "synthetic_fixture"
        }
      ]
    }
  ]
}

#!/usr/bin/env python3
"""Small regression suite including a fresh local recording and malformed imports."""
import copy
from decimal import Decimal
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
import account

ROOT = Path(__file__).resolve().parent

class AccountingTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.temp = tempfile.TemporaryDirectory()
        cls.directory = Path(cls.temp.name)
        subprocess.run([sys.executable, str(ROOT / "record_demo.py"), "--output-dir", str(cls.directory)], check=True, capture_output=True)
        cls.data = json.loads((cls.directory / "traces.json").read_text())
        cls.labels = json.loads((cls.directory / "labels.json").read_text())
        cls.policies = json.loads((cls.directory / "policies.json").read_text())

    @classmethod
    def tearDownClass(cls):
        cls.temp.cleanup()

    def test_actual_spans_and_all_costs(self):
        result = account.analyse(self.data, self.labels, self.policies)
        for task in result["tasks"]:
            self.assertEqual(task["call_count"], 7)
            self.assertGreater(task["recorded_wall_ms"], 0)
            self.assertGreater(task["summed_call_ms"], task["recorded_wall_ms"])
            self.assertEqual(Decimal(task["estimated_cost_usd"]), Decimal("0.00043"))
            self.assertEqual(Decimal(task["verification_cost_usd"]), Decimal("0.00012"))
            self.assertEqual(Decimal(task["failed_call_cost_usd"]), Decimal("0.00004"))
        self.assertEqual({p["accounted_cost_usd"] for p in result["policies"]}, {"0.00086"})
        self.assertTrue(all(p["cancellation_savings_usd"] is None for p in result["policies"]))

    def test_labels_cannot_change_selection(self):
        flipped = {task: {cid: not correct for cid, correct in labels.items()} for task, labels in self.labels.items()}
        a = account.analyse(self.data, self.labels, self.policies)
        b = account.analyse(self.data, flipped, self.policies)
        self.assertEqual([[d["selected_id"] for d in p["decisions"]] for p in a["policies"]],
                         [[d["selected_id"] for d in p["decisions"]] for p in b["policies"]])

    def test_deadline_requires_finished_verification(self):
        task = self.data["tasks"][0]
        first_end = min(s["end_ms"] for s in task["spans"] if s["role"] == "verification")
        self.assertIsNone(account.select(task, first_end - 0.001))
        self.assertIsNotNone(account.select(task, first_end))

    def test_missing_failed_call_usage_rejected(self):
        data = copy.deepcopy(self.data)
        failed = next(s for s in data["tasks"][0]["spans"] if s["status"] == "failed")
        del failed["input_units"]
        with self.assertRaisesRegex(ValueError, "including failures"):
            account.analyse(data, self.labels, self.policies)

    def test_nonfinite_time_and_duplicate_spans_rejected(self):
        data = copy.deepcopy(self.data)
        data["tasks"][0]["spans"][0]["end_ms"] = float("nan")
        with self.assertRaisesRegex(ValueError, "boundaries"):
            account.analyse(data, self.labels, self.policies)
        data = copy.deepcopy(self.data)
        data["tasks"][0]["spans"].append(data["tasks"][0]["spans"][0])
        with self.assertRaisesRegex(ValueError, "Duplicate"):
            account.analyse(data, self.labels, self.policies)

    def test_verifier_cannot_finish_before_its_generation(self):
        data = copy.deepcopy(self.data)
        verifier = next(s for s in data["tasks"][0]["spans"] if s["role"] == "verification")
        verifier["start_ms"] = 0
        with self.assertRaisesRegex(ValueError, "before generation"):
            account.analyse(data, self.labels, self.policies)

    def test_bad_cli_trace_exits_two(self):
        data = copy.deepcopy(self.data)
        data["tasks"][0]["elapsed_ms"] = -1
        with tempfile.TemporaryDirectory() as temp:
            path = Path(temp)
            for name, value in (("traces", data), ("labels", self.labels), ("policies", self.policies)):
                (path / (name + ".json")).write_text(json.dumps(value))
            run = subprocess.run([sys.executable, str(ROOT / "account.py"), "--directory", str(path)], capture_output=True, text=True)
            self.assertEqual(run.returncode, 2)
            self.assertIn("Invalid elapsed time", run.stderr)

if __name__ == "__main__":
    unittest.main()

{
  "scope": "Fedora local execution only; synthetic unit usage and prices; no inference or billing benchmark",
  "checks": [
    {
      "command": [
        "python3",
        "-m",
        "unittest",
        "-v",
        "test_account.py"
      ],
      "exit_code": 0,
      "stdout": "",
      "stderr": "test_actual_spans_and_all_costs (test_account.AccountingTests.test_actual_spans_and_all_costs) ... ok\ntest_bad_cli_trace_exits_two (test_account.AccountingTests.test_bad_cli_trace_exits_two) ... ok\ntest_deadline_requires_finished_verification (test_account.AccountingTests.test_deadline_requires_finished_verification) ... ok\ntest_labels_cannot_change_selection (test_account.AccountingTests.test_labels_cannot_change_selection) ... ok\ntest_missing_failed_call_usage_rejected (test_account.AccountingTests.test_missing_failed_call_usage_rejected) ... ok\ntest_nonfinite_time_and_duplicate_spans_rejected (test_account.AccountingTests.test_nonfinite_time_and_duplicate_spans_rejected) ... ok\ntest_verifier_cannot_finish_before_its_generation (test_account.AccountingTests.test_verifier_cannot_finish_before_its_generation) ... ok\n\n----------------------------------------------------------------------\nRan 7 tests in 0.230s\n\nOK\n"
    },
    {
      "command": [
        "python3",
        "account.py"
      ],
      "exit_code": 0,
      "stdout": "heldout-1 wall_ms 70.6542 sum_call_ms 150.4078 estimated_USD 0.00043 failed_USD 0.00004 verifier_USD 0.00012\nheldout-2 wall_ms 70.508 sum_call_ms 150.3999 estimated_USD 0.00043 failed_USD 0.00004 verifier_USD 0.00012\nall_completed answered 2 correct 2 accounted_USD 0.00086 cancellation savings unknown\ndeadline_50ms answered 2 correct 0 accounted_USD 0.00086 cancellation savings unknown\n",
      "stderr": ""
    },
    {
      "command": [
        "python3",
        "account.py",
        "--help"
      ],
      "exit_code": 0,
      "stdout": "usage: account.py [-h] [--directory DIRECTORY] [--output OUTPUT]\n\nAccount for all recorded calls, then replay frozen selection deadlines.\n\noptions:\n  -h, --help            show this help message and exit\n  --directory DIRECTORY\n  --output OUTPUT\n",
      "stderr": ""
    },
    {
      "command": [
        "python3",
        "record_demo.py",
        "--help"
      ],
      "exit_code": 0,
      "stdout": "usage: record_demo.py [-h] [--output-dir OUTPUT_DIR]\n\nRecord actual local sleep spans; all usage, scores and prices are synthetic.\n\noptions:\n  -h, --help            show this help message and exit\n  --output-dir OUTPUT_DIR\n",
      "stderr": ""
    }
  ],
  "sha256": {
    "test_account.py": "7202208a629d36ad2d10470069e563fc12565bfd323124ac476ec431a61aabb2",
    "README.md": "3965f2c9c428077dbe73b39833ff17ebb06f14753dd4885e42c248d4eaa40f2a",
    "SOURCES.md": "3c3cf07640675a47e1f7a53fcf1332bd795298fcdbd4c16aa55759a57b4359b3",
    "fixture.json": "fe09987f596491306979e742cc9425de080a972782d821293204bf2bb7ee8619",
    "account.py": "bbbd8f3801f2292e919c3da6aa6a5f2f690aaea7405213dd3e20155c014ca6e8",
    "record_demo.py": "ef126d44f28cea6b942a8f7f58e53cc572fd2b31273611448f78b3665693b2e5",
    "article-section.md": "d7a23ca5c4e3d9c8c3a053f5c402c51a1a53f628c534ea8d477bf61a929fa044",
    "sample/traces.json": "8c928a7dd548feac754acb85b16ace2129058293377c5bd87392e8d5b7dc0c51",
    "sample/results.json": "c2c6bf1ca085430f9ffcd5452e36a3fa9e1134343533cf89a1593972b09f7ab1",
    "sample/labels.json": "037075fe81adc8a501ecefb026ae7d5950465e68ac93476690e39515263ca71d",
    "sample/policies.json": "6ce45cbfb48641d80483c5af5b953aa1bbc0cce73b9c80c49938ba0463902db5"
  }
}

# Replay answer selection

**A runnable lab for replaying candidate answers before increasing an inference budget.**

This is a synthetic engineering exercise, not a model benchmark. The six tasks, candidate answers, correctness labels and verifier scores are invented. They deliberately include confident errors so you can inspect a failure that aggregate scores can hide. Nothing here estimates the accuracy of a commercial model.

The practical question is narrower: when your system generates several answers and returns the highest-scoring one, does another candidate help the user? You will replay the same saved candidates through three selection policies, inspect individual failures, and learn which measurements to carry into a real evaluation.

For the broader context, read our [test-time compute guide](/test-time-compute-scaling-guide-2026/). This lab supplies a trace format and evaluation harness you can adapt.

## What you will run

The download contains two answer-selection exercises and an accounting extension. Start with `replay.py`, which evaluates saved candidate traces. It uses Python's standard library and makes no network or model calls.

```sh
python3 replay.py --help
python3 replay.py
python3 replay.py --threshold 80 --output threshold-80.json
```

Download the lab ZIP below and extract it. Run these commands inside its engineering-labs/answer-verification folder. Python 3.10 or newer is sufficient. The default run writes `replay-results.json`; the alternative threshold gets its own output so you retain the original comparison.

The fixture has two calibration tasks and four test tasks. Each has four candidates in a fixed generation order. We report the first one, two and four candidates separately. This nested comparison matters: changing the candidate set at every budget would mix selection behaviour with differences in generation.

Three policies see identical prefixes:

- **first** returns the first candidate, ignoring scores.
- **proxy** returns the candidate with the highest verifier score.
- **proxy_threshold** chooses the same winner, but abstains when its score is below 90.

The threshold is declared in advance for this demonstration. It was not fitted on the test set and is not a probability estimate. Calibration rows are reported separately to show where policy development belongs when you substitute real data. Two calibration tasks are nowhere near enough to calibrate a production verifier.

## Read the result

These are the actual default-run counts for the four synthetic test tasks:

| N | Policy | Answered | Correct | Correct available | Missed available |
| --- | --- | --- | --- | --- | --- |
| 1 | first | 4/4 | 2/4 | 2/4 | 0 |
| 1 | proxy | 4/4 | 2/4 | 2/4 | 0 |
| 1 | proxy_threshold | 0/4 | 0/4 | 2/4 | 2 |
| 2 | first | 4/4 | 2/4 | 3/4 | 1 |
| 2 | proxy | 4/4 | 2/4 | 3/4 | 1 |
| 2 | proxy_threshold | 1/4 | 0/4 | 3/4 | 3 |
| 4 | first | 4/4 | 2/4 | 4/4 | 2 |
| 4 | proxy | 4/4 | 1/4 | 4/4 | 3 |
| 4 | proxy_threshold | 3/4 | 0/4 | 4/4 | 4 |

“Correct available” means the candidate pool contains at least one labelled correct answer. “Correct” counts the answers actually returned. “Missed available” counts tasks where a correct answer existed but the policy returned an incorrect answer or abstained. Those two outcomes remain distinguishable in the per-task diagnostics.

At four candidates, generation has supplied a correct answer for every test task. The proxy policy finds only one. Raising the minimum score makes matters worse here: the wrong answers already occupy the top of the score range.

That does not establish that thresholds are bad. It establishes that a high score is useful only when its relationship to correctness has been checked. A threshold can reduce answer coverage without improving the answers that survive it.

The JSON also reports accuracy conditional on answering. When no task is answered, that value is null. Reporting zero would conflate undefined conditional accuracy with answering every task incorrectly.

## Inspect the failure, not just the average

Open `replay-results.json` and find the test, four-candidate, proxy row. On `test-1`, the selector chooses candidate `b`, a wrong answer scored 99, although correct candidates are present. On `test-2`, a wrong candidate scored 95 displaces a correct candidate scored 82. On `test-3`, the fourth candidate introduces the highest-scoring error.

These are actionable categories in a real system: inspect what the verifier rewarded and whether your label actually captures the requirement. A longer explanation, confident phrasing or a superficially valid output format might correlate with a score; this fixture does not test any of those causes.

Gao, Schulman and Hilton studied proxy reward overoptimisation, including best-of-N sampling, using a gold reward model as a reference. Their result motivates checking the selected answer rather than trusting the score being optimised. Our fixture is a separate teaching example, not a reproduction of their experiments. [Primary paper](https://arxiv.org/abs/2210.10760v1)

## Bring your own candidate traces

Replace `traces.json` with your own saved evaluation data. Each task needs an ID, a calibration or test split, and an ordered candidates list. Each candidate needs an ID, answer, numeric score from 0 to 100, and independently assigned boolean correctness label.

Keep task IDs unique and provide at least four candidates per task for the default budget sweep. Scores may tie; the earlier candidate wins. If your verifier uses another scale, document a consistent transformation rather than interpreting 90 as a universal threshold.

Crucially, the selector never reads correctness labels. The programme checks this by flipping every label and requiring the selected IDs to remain unchanged. Labels are for scoring the decision afterwards.

Use executable checks where the task permits them: a numerical reference answer, a schema plus business constraints, or held-out code tests. Otherwise define a review rubric and independently adjudicate disagreements. Do not use the same verifier's preferred answer as ground truth.

Lightman and colleagues distinguish feedback on final outcomes from feedback on intermediate steps. Their MATH experiments support investigating supervision design, but do not prove step-level checking will solve your task. [Primary paper](https://arxiv.org/abs/2305.20050v1)

## Make the budget decision

Develop thresholds and scoring changes on calibration tasks, then freeze them before evaluating held-out test tasks. Log model version, sampling settings, prompt version, candidate order and verifier version alongside the traces. Keep tasks together when splitting, so paraphrases or shared source documents do not leak across the boundary.

Add actual generation and verification costs, latency and timeout outcomes before deciding that a policy is cheaper. The original replay counts candidates; the accounting extension below adds recorded local spans and explicit price estimates. Neither measures model-provider billing. For larger datasets, report uncertainty over tasks and compare policies on the same task pools. Repeated candidates within one task are not independent evaluation cases.

The optional `python3 run.py` exercise calculates exact probabilities for three weighted answer archetypes. It checks its formula against exhaustive enumeration and an oracle-ranking control. It explains the mechanism, while the replay harness provides the more useful route into your own traces.

The decision rule is practical: if correct answers become available but selection repeatedly misses them, inspect verification before paying for a larger candidate pool. If correct answers remain absent, selection alone cannot recover them. This lab helps separate those two failures without pretending that six invented tasks settle your production design.

For cost accounting around this evaluation, see our [guide to tracking and reducing LLM spend](/agent-cost-optimization-how-to-track-and-reduce-llm-spend/).

## Add cost and deadline accounting

Generating four candidates does not tell you what an answer costs. The accounting extension adds failed calls, verification overhead and overlapping spans to the existing replay. It also shows why a deadline is not evidence of a billing saving.

**This is a local execution exercise.** We measured Python workers that sleep for short intervals on Fedora. Usage quantities, prices, scores and correctness labels are invented. There are no model calls, provider bills or measured inference latencies. The useful result is the accounting method you can inspect and adapt.

Download the lab ZIP below, open `engineering-labs/answer-verification/accounting`, and run:

```sh
python3 record_demo.py --output-dir my-run
python3 account.py --directory my-run
python3 -m unittest -v test_account.py
```

Python 3.10 or newer is enough. The supplied sample preserves the original execution. Running `python3 account.py` replays that sample without recording new timings.

Each task launches four overlapping candidate workers. Three complete and then run a verification stage; one deliberately fails. Every call has usage, including the failure. The synthetic rates are $0.50 per million input units and $2.00 per million output units, dated 8 September 2026. These are teaching values, not a provider quotation.

The saved execution produced:

| Task | Observed local wall time | Summed call duration | Estimated total | Failed call | Verification |
| --- | --- | --- | --- | --- | --- |
| heldout-1 | 70.6542ms | 150.4078ms | $0.00043 | $0.00004 | $0.00012 |
| heldout-2 | 70.5080ms | 150.3999ms | $0.00043 | $0.00004 | $0.00012 |

Failed-call and verification amounts are parts of the total, not additional charges. Summing call durations would overstate elapsed time because workers overlap. These exact timings belong to the saved run; another machine or a busy scheduler will produce different values.

Two frozen policies see the same traces. All-completed selects the highest-scoring verified candidate after all work finishes. The 50ms policy considers only candidates whose verification finished by that cut-off. Its selection function cannot access correctness labels; those live in a separate file and are applied afterwards.

In this synthetic run, all-completed returned two correct answers. The deadline policy returned two wrong answers. Both accounting rows retain the full $0.00086 of observed work. That is deliberate: replaying an earlier cut-off does not establish that ongoing requests would have been cancelled, or that cancellation would have reduced billed usage.

The output therefore marks cancellation savings as unknown. It also keeps the deadline separate from observed wall time. The programme has not measured a service returning within 50ms; it has inspected which answers were available then.

To adapt this for real evaluation, import sanitised spans with a common task clock, versioned policy settings and usage provenance. Include unsuccessful requests and verifier calls. Missing usage should trigger investigation rather than silently becoming zero. The supplied calculator supports a simple unit-price estimate; extend it explicitly for recorded bills, caching, retries or provider-specific fees.

Develop policies on separate calibration data, then freeze them before evaluating held-out tasks. Report correct outcomes per attempted task, abstentions and estimated cost per correct outcome together. When there are no correct outcomes, cost per correct is undefined, so the output uses null.

The regression suite records fresh local spans, checks the arithmetic, flips labels to prove selection is unchanged, and rejects malformed or incomplete traces. The exercise gives you a way to test the accounting before applying it to real requests. It makes no claim that the illustrated deadline, prices or outcome rates suit a production workload.

Implementation references: [Python performance counter](https://docs.python.org/3/library/time.html#time.perf_counter_ns) and [thread pool executor](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor).

## Sources

- Gao, Schulman and Hilton, [Scaling Laws for Reward Model Overoptimization](https://arxiv.org/abs/2210.10760v1), 2022.
- Lightman and colleagues, [Let’s Verify Step by Step](https://arxiv.org/abs/2305.20050v1), 2023.

{
  "description": "Synthetic answer archetypes for the prompt: What is 17 * 19? Weights and proxy scores are deliberately invented, not model measurements.",
  "expected_answer": 323,
  "candidates": [
    {
      "id": "correct",
      "answer": 323,
      "text": "17 * (20 - 1) = 340 - 17 = 323.",
      "weight": 6,
      "proxy_score": 80
    },
    {
      "id": "ordinary_error",
      "answer": 313,
      "text": "17 * 19 = 313.",
      "weight": 3,
      "proxy_score": 20
    },
    {
      "id": "confident_error",
      "answer": 333,
      "text": "Verified carefully: 17 * 19 is definitely 333.",
      "weight": 1,
      "proxy_score": 99
    }
  ]
}

{
  "kind": "synthetic teaching fixture; scores and answers invented",
  "threshold": 90,
  "trace_sha256": "ec933285fbbad8721669e43b0bbf245224cfb8246a12607c4e3517392a5df06b",
  "validation": "passed: label independence, ties, abstention, bounds, nested availability",
  "rows": [
    {
      "split": "calibration",
      "n": 1,
      "policy": "first",
      "tasks": 2,
      "answered": 2,
      "correct": 1,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 1,
      "missed_available_correct_tasks": 0,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "cal-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "calibration",
      "n": 1,
      "policy": "proxy",
      "tasks": 2,
      "answered": 2,
      "correct": 1,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 1,
      "missed_available_correct_tasks": 0,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "cal-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "calibration",
      "n": 1,
      "policy": "proxy_threshold",
      "tasks": 2,
      "answered": 0,
      "correct": 0,
      "answer_rate_pct": 0.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": null,
      "available_correct_tasks": 1,
      "missed_available_correct_tasks": 1,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "cal-2",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "calibration",
      "n": 2,
      "policy": "first",
      "tasks": 2,
      "answered": 2,
      "correct": 1,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 1,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "cal-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    },
    {
      "split": "calibration",
      "n": 2,
      "policy": "proxy",
      "tasks": 2,
      "answered": 2,
      "correct": 1,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 1,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "cal-2",
          "selected_id": "b",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "calibration",
      "n": 2,
      "policy": "proxy_threshold",
      "tasks": 2,
      "answered": 1,
      "correct": 0,
      "answer_rate_pct": 50.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": 0.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 2,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "cal-2",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    },
    {
      "split": "calibration",
      "n": 4,
      "policy": "first",
      "tasks": 2,
      "answered": 2,
      "correct": 1,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 1,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "cal-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    },
    {
      "split": "calibration",
      "n": 4,
      "policy": "proxy",
      "tasks": 2,
      "answered": 2,
      "correct": 0,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": 0.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 2,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "cal-2",
          "selected_id": "d",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    },
    {
      "split": "calibration",
      "n": 4,
      "policy": "proxy_threshold",
      "tasks": 2,
      "answered": 2,
      "correct": 0,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": 0.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 2,
      "diagnostics": [
        {
          "task_id": "cal-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "cal-2",
          "selected_id": "d",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    },
    {
      "split": "test",
      "n": 1,
      "policy": "first",
      "tasks": 4,
      "answered": 4,
      "correct": 2,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 0,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        },
        {
          "task_id": "test-3",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-4",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 1,
      "policy": "proxy",
      "tasks": 4,
      "answered": 4,
      "correct": 2,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 0,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        },
        {
          "task_id": "test-3",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-4",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 1,
      "policy": "proxy_threshold",
      "tasks": 4,
      "answered": 0,
      "correct": 0,
      "answer_rate_pct": 0.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": null,
      "available_correct_tasks": 2,
      "missed_available_correct_tasks": 2,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-2",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        },
        {
          "task_id": "test-3",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-4",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 2,
      "policy": "first",
      "tasks": 4,
      "answered": 4,
      "correct": 2,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 3,
      "missed_available_correct_tasks": 1,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-3",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-4",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 2,
      "policy": "proxy",
      "tasks": 4,
      "answered": 4,
      "correct": 2,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 3,
      "missed_available_correct_tasks": 1,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-2",
          "selected_id": "b",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-3",
          "selected_id": "b",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-4",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 2,
      "policy": "proxy_threshold",
      "tasks": 4,
      "answered": 1,
      "correct": 0,
      "answer_rate_pct": 25.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": 0.0,
      "available_correct_tasks": 3,
      "missed_available_correct_tasks": 3,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-2",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-3",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-4",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": false,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 4,
      "policy": "first",
      "tasks": 4,
      "answered": 4,
      "correct": 2,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 50.0,
      "accuracy_when_answered_pct": 50.0,
      "available_correct_tasks": 4,
      "missed_available_correct_tasks": 2,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-2",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-3",
          "selected_id": "a",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        },
        {
          "task_id": "test-4",
          "selected_id": "a",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    },
    {
      "split": "test",
      "n": 4,
      "policy": "proxy",
      "tasks": 4,
      "answered": 4,
      "correct": 1,
      "answer_rate_pct": 100.0,
      "correct_per_task_pct": 25.0,
      "accuracy_when_answered_pct": 25.0,
      "available_correct_tasks": 4,
      "missed_available_correct_tasks": 3,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-2",
          "selected_id": "c",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-3",
          "selected_id": "d",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-4",
          "selected_id": "d",
          "selected_correct": true,
          "correct_available": true,
          "missed_available_correct": false
        }
      ]
    },
    {
      "split": "test",
      "n": 4,
      "policy": "proxy_threshold",
      "tasks": 4,
      "answered": 3,
      "correct": 0,
      "answer_rate_pct": 75.0,
      "correct_per_task_pct": 0.0,
      "accuracy_when_answered_pct": 0.0,
      "available_correct_tasks": 4,
      "missed_available_correct_tasks": 4,
      "diagnostics": [
        {
          "task_id": "test-1",
          "selected_id": "b",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-2",
          "selected_id": "c",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-3",
          "selected_id": "d",
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        },
        {
          "task_id": "test-4",
          "selected_id": null,
          "selected_correct": false,
          "correct_available": true,
          "missed_available_correct": true
        }
      ]
    }
  ]
}

#!/usr/bin/env python3
"""Replay saved candidate traces. Scores drive selection; labels are evaluation only."""
import argparse
import hashlib
import json
from pathlib import Path

ROOT = Path(__file__).resolve().parent
POLICIES = ("first", "proxy", "proxy_threshold")

def load(path):
    data = json.loads(path.read_text())
    tasks = data["tasks"]
    if not tasks or len({t["id"] for t in tasks}) != len(tasks):
        raise ValueError("Tasks must be nonempty with unique IDs")
    for task in tasks:
        if task["split"] not in ("calibration", "test"):
            raise ValueError("Each task needs a calibration or test split")
        candidates = task["candidates"]
        if not candidates or len({c["id"] for c in candidates}) != len(candidates):
            raise ValueError("Candidates must be nonempty with unique IDs per task")
        for c in candidates:
            if type(c["correct"]) is not bool or type(c["score"]) not in (int, float) or not 0 <= c["score"] <= 100:
                raise ValueError("Candidate correct must be boolean and score must be 0..100")
    return data

def select(candidates, policy, threshold):
    # Deliberately do not access the correctness label here.
    if policy == "first":
        return candidates[0]
    winner = max(candidates, key=lambda c: c["score"])
    if policy == "proxy_threshold" and winner["score"] < threshold:
        return None
    return winner

def replay(tasks, n, policy, threshold):
    diagnostics = []
    for task in tasks:
        if len(task["candidates"]) < n:
            raise ValueError(f"Task {task['id']} has fewer than {n} candidates")
        pool = task["candidates"][:n]
        winner = select(pool, policy, threshold)
        available = any(c["correct"] for c in pool)
        correct = bool(winner and winner["correct"])
        diagnostics.append({"task_id": task["id"], "selected_id": winner["id"] if winner else None,
                            "selected_correct": correct, "correct_available": available,
                            "missed_available_correct": available and not correct})
    count = len(tasks)
    answered = sum(d["selected_id"] is not None for d in diagnostics)
    correct = sum(d["selected_correct"] for d in diagnostics)
    return {"tasks": count, "answered": answered, "correct": correct,
            "answer_rate_pct": round(answered / count * 100, 2),
            "correct_per_task_pct": round(correct / count * 100, 2),
            "accuracy_when_answered_pct": round(correct / answered * 100, 2) if answered else None,
            "available_correct_tasks": sum(d["correct_available"] for d in diagnostics),
            "missed_available_correct_tasks": sum(d["missed_available_correct"] for d in diagnostics),
            "diagnostics": diagnostics}

def validate(tasks, threshold):
    # Rank ties preserve candidate order, not ground-truth labels.
    tie = [{"id": "wrong", "score": 90, "correct": False}, {"id": "right", "score": 90, "correct": True}]
    assert select(tie, "proxy", threshold)["id"] == "wrong"
    assert select([{"id": "low", "score": 0}], "proxy_threshold", 1) is None
    # Selection must remain invariant when evaluation labels are inverted.
    for task in tasks:
        pool = task["candidates"]
        inverted = [dict(c, correct=not c["correct"]) for c in pool]
        for policy in POLICIES:
            a, b = select(pool, policy, threshold), select(inverted, policy, threshold)
            assert (a["id"] if a else None) == (b["id"] if b else None)
        previous_available = False
        for n in range(1, len(pool) + 1):
            available = any(c["correct"] for c in pool[:n])
            assert not previous_available or available
            previous_available = available

def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--traces", type=Path, default=ROOT / "traces.json")
    parser.add_argument("--output", type=Path, default=ROOT / "replay-results.json")
    parser.add_argument("--threshold", type=float, default=90, help="Predeclared score cut-off, not a calibrated probability")
    args = parser.parse_args()
    if not 0 <= args.threshold <= 100:
        parser.error("threshold must be 0..100")
    try:
        data = load(args.traces)
        validate(data["tasks"], args.threshold)
        rows = []
        for split in ("calibration", "test"):
            tasks = [t for t in data["tasks"] if t["split"] == split]
            if not tasks:
                raise ValueError(f"Missing {split} tasks")
            for n in (1, 2, 4):
                for policy in POLICIES:
                    row = {"split": split, "n": n, "policy": policy, **replay(tasks, n, policy, args.threshold)}
                    assert row["correct"] <= row["answered"] <= row["tasks"]
                    assert row["correct"] <= row["available_correct_tasks"]
                    rows.append(row)
    except (ValueError, KeyError) as error:
        parser.error(str(error))
    output = {"kind": data["kind"], "threshold": args.threshold,
              "trace_sha256": hashlib.sha256(args.traces.read_bytes()).hexdigest(),
              "validation": "passed: label independence, ties, abstention, bounds, nested availability",
              "rows": rows}
    args.output.write_text(json.dumps(output, indent=2) + "\n")
    print("| N | Policy | Answered | Correct | Correct available | Missed available |")
    print("| --- | --- | --- | --- | --- | --- |")
    for r in rows:
        if r["split"] == "test":
            print(f"| {r['n']} | {r['policy']} | {r['answered']}/{r['tasks']} | {r['correct']}/{r['tasks']} | {r['available_correct_tasks']}/{r['tasks']} | {r['missed_available_correct_tasks']} |")
    print("Validation passed; wrote", args.output.name)

if __name__ == "__main__":
    main()

{
  "kind": "exact synthetic calculation, not an LLM benchmark",
  "fixture_sha256": "8ce54d390e686c65649b0e252f3acf60930b5534e89a6f0207e08f0f22466bfe",
  "assumptions": [
    "independent draws with replacement",
    "fixed invented scores",
    "exact-answer ground truth"
  ],
  "validation": {
    "exhaustive_enumeration_n": [
      1,
      2,
      3,
      4,
      5
    ],
    "probability_and_monotonicity_n": [
      1,
      32
    ],
    "oracle_control": "passed"
  },
  "rows": [
    {
      "candidates": 1,
      "correct_available_pct": 60.0,
      "selected_correct_pct": 60.0,
      "mean_selected_proxy_score": 63.9,
      "selected_correct_exact": "3/5",
      "correct_available_exact": "3/5"
    },
    {
      "candidates": 2,
      "correct_available_pct": 84.0,
      "selected_correct_pct": 72.0,
      "mean_selected_proxy_score": 78.21,
      "selected_correct_exact": "18/25",
      "correct_available_exact": "21/25"
    },
    {
      "candidates": 4,
      "correct_available_pct": 97.44,
      "selected_correct_pct": 64.8,
      "mean_selected_proxy_score": 86.0481,
      "selected_correct_exact": "81/125",
      "correct_available_exact": "609/625"
    },
    {
      "candidates": 8,
      "correct_available_pct": 99.9345,
      "selected_correct_pct": 43.0402,
      "mean_selected_proxy_score": 90.8172,
      "selected_correct_exact": "269001/625000",
      "correct_available_exact": "390369/390625"
    },
    {
      "candidates": 16,
      "correct_available_pct": 100.0,
      "selected_correct_pct": 18.5302,
      "mean_selected_proxy_score": 95.4793,
      "selected_correct_exact": "5790687955641/31250000000000",
      "correct_available_exact": "152587825089/152587890625"
    }
  ]
}

#!/usr/bin/env python3
"""Exact synthetic best-of-N probabilities; no model calls or random sampling."""
import argparse
from fractions import Fraction
import hashlib
import itertools
import json
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def load_fixture(path):
    data = json.loads(path.read_text())
    candidates = data["candidates"]
    if not candidates or any(type(c["weight"]) is not int or c["weight"] <= 0 for c in candidates):
        raise ValueError("Candidate weights must be positive integers")
    if len({c["id"] for c in candidates}) != len(candidates):
        raise ValueError("Candidate IDs must be unique")
    if len({c["proxy_score"] for c in candidates}) != len(candidates):
        raise ValueError("This fixture requires distinct proxy scores")
    total = sum(c["weight"] for c in candidates)
    return data, [(c, Fraction(c["weight"], total)) for c in candidates]

def calculate(data, weighted, n):
    if n < 1:
        raise ValueError("N must be positive")
    cumulative = Fraction(0)
    selected_correct = Fraction(0)
    expected_proxy = Fraction(0)
    selected_mass = Fraction(0)
    for candidate, probability in sorted(weighted, key=lambda item: item[0]["proxy_score"]):
        win_probability = (cumulative + probability) ** n - cumulative ** n
        cumulative += probability
        selected_mass += win_probability
        expected_proxy += win_probability * candidate["proxy_score"]
        if candidate["answer"] == data["expected_answer"]:
            selected_correct += win_probability
    assert selected_mass == 1
    correct_mass = sum((p for c, p in weighted if c["answer"] == data["expected_answer"]), Fraction(0))
    available = 1 - (1 - correct_mass) ** n
    return selected_correct, available, expected_proxy

def validate(data, weighted):
    # Independent exhaustive enumeration checks the closed-form winner calculation.
    for n in range(1, 6):
        selected = available = score = Fraction(0)
        for sequence in itertools.product(weighted, repeat=n):
            probability = Fraction(1)
            for _, p in sequence:
                probability *= p
            winner = max((c for c, _ in sequence), key=lambda c: c["proxy_score"])
            selected += probability * (winner["answer"] == data["expected_answer"])
            available += probability * any(c["answer"] == data["expected_answer"] for c, _ in sequence)
            score += probability * winner["proxy_score"]
        assert (selected, available, score) == calculate(data, weighted, n)
    previous_available = Fraction(0)
    previous_proxy = Fraction(0)
    for n in range(1, 33):
        selected, available, score = calculate(data, weighted, n)
        assert 0 <= selected <= available <= 1
        assert available >= previous_available and score >= previous_proxy
        previous_available, previous_proxy = available, score
    # An oracle ranking must recover every pool containing a correct answer.
    oracle = [(dict(c, proxy_score=(1000 if c["answer"] == data["expected_answer"] else c["proxy_score"])), p) for c, p in weighted]
    for n in (1, 2, 4, 8, 16):
        selected, available, _ = calculate(data, oracle, n)
        assert selected == available

def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--fixture", type=Path, default=ROOT / "fixture.json")
    parser.add_argument("--output", type=Path, default=ROOT / "results.json")
    args = parser.parse_args()
    try:
        data, weighted = load_fixture(args.fixture)
        validate(data, weighted)
    except (ValueError, KeyError) as error:
        parser.error(str(error))
    rows = []
    print("| Candidates | Correct answer available | Proxy-selected accuracy | Mean selected proxy score |")
    print("| --- | --- | --- | --- |")
    for n in (1, 2, 4, 8, 16):
        selected, available, score = calculate(data, weighted, n)
        row = {"candidates": n, "correct_available_pct": round(float(available * 100), 4),
               "selected_correct_pct": round(float(selected * 100), 4),
               "mean_selected_proxy_score": round(float(score), 4),
               "selected_correct_exact": str(selected), "correct_available_exact": str(available)}
        rows.append(row)
        print(f"| {n} | {row['correct_available_pct']:.2f}% | {row['selected_correct_pct']:.2f}% | {row['mean_selected_proxy_score']:.2f} |")
    result = {"kind": "exact synthetic calculation, not an LLM benchmark",
              "fixture_sha256": hashlib.sha256(args.fixture.read_bytes()).hexdigest(),
              "assumptions": ["independent draws with replacement", "fixed invented scores", "exact-answer ground truth"],
              "validation": {"exhaustive_enumeration_n": [1, 2, 3, 4, 5], "probability_and_monotonicity_n": [1, 32], "oracle_control": "passed"},
              "rows": rows}
    args.output.write_text(json.dumps(result, indent=2) + "\n")
    print("Validation passed; wrote", args.output.name)

if __name__ == "__main__":
    main()

{
  "kind": "synthetic teaching fixture; scores and answers invented",
  "tasks": [
    {
      "id": "cal-1",
      "split": "calibration",
      "candidates": [
        {
          "id": "a",
          "answer": "323",
          "correct": true,
          "score": 80
        },
        {
          "id": "b",
          "answer": "333",
          "correct": false,
          "score": 99
        },
        {
          "id": "c",
          "answer": "313",
          "correct": false,
          "score": 30
        },
        {
          "id": "d",
          "answer": "323",
          "correct": true,
          "score": 85
        }
      ],
      "prompt": "Calculate 17 * 19",
      "expected_answer": "323"
    },
    {
      "id": "cal-2",
      "split": "calibration",
      "candidates": [
        {
          "id": "a",
          "answer": "42",
          "correct": false,
          "score": 40
        },
        {
          "id": "b",
          "answer": "44",
          "correct": true,
          "score": 82
        },
        {
          "id": "c",
          "answer": "44",
          "correct": true,
          "score": 87
        },
        {
          "id": "d",
          "answer": "46",
          "correct": false,
          "score": 98
        }
      ],
      "prompt": "Calculate 11 * 4",
      "expected_answer": "44"
    },
    {
      "id": "test-1",
      "split": "test",
      "candidates": [
        {
          "id": "a",
          "answer": "12",
          "correct": true,
          "score": 80
        },
        {
          "id": "b",
          "answer": "14",
          "correct": false,
          "score": 99
        },
        {
          "id": "c",
          "answer": "12",
          "correct": true,
          "score": 85
        },
        {
          "id": "d",
          "answer": "10",
          "correct": false,
          "score": 60
        }
      ],
      "prompt": "Calculate 3 * 4",
      "expected_answer": "12"
    },
    {
      "id": "test-2",
      "split": "test",
      "candidates": [
        {
          "id": "a",
          "answer": "16",
          "correct": false,
          "score": 40
        },
        {
          "id": "b",
          "answer": "18",
          "correct": true,
          "score": 82
        },
        {
          "id": "c",
          "answer": "20",
          "correct": false,
          "score": 95
        },
        {
          "id": "d",
          "answer": "18",
          "correct": true,
          "score": 83
        }
      ],
      "prompt": "Calculate 3 * 6",
      "expected_answer": "18"
    },
    {
      "id": "test-3",
      "split": "test",
      "candidates": [
        {
          "id": "a",
          "answer": "35",
          "correct": true,
          "score": 70
        },
        {
          "id": "b",
          "answer": "35",
          "correct": true,
          "score": 75
        },
        {
          "id": "c",
          "answer": "35",
          "correct": true,
          "score": 85
        },
        {
          "id": "d",
          "answer": "37",
          "correct": false,
          "score": 98
        }
      ],
      "prompt": "Calculate 5 * 7",
      "expected_answer": "35"
    },
    {
      "id": "test-4",
      "split": "test",
      "candidates": [
        {
          "id": "a",
          "answer": "48",
          "correct": false,
          "score": 20
        },
        {
          "id": "b",
          "answer": "50",
          "correct": false,
          "score": 30
        },
        {
          "id": "c",
          "answer": "49",
          "correct": true,
          "score": 60
        },
        {
          "id": "d",
          "answer": "49",
          "correct": true,
          "score": 70
        }
      ],
      "prompt": "Calculate 7 * 7",
      "expected_answer": "49"
    }
  ]
}

{
  "date": "2026-09-08",
  "host": "Fedora",
  "python": "stdlib",
  "checks": [
    "replay.py default: exit 0 and replay-results.json written",
    "replay.py --help: exit 0",
    "replay.py --threshold 80 with temporary output: exit 0",
    "replay.py --threshold 101: exit 2",
    "run.py default: exit 0 and results.json written",
    "run.py --help: exit 0",
    "run.py zero-weight temporary fixture: exit 2"
  ],
  "scope": "Synthetic local fixtures only; no model benchmark or production change"
}