Test pgvector filtering and recall

# pgvector filtered ANN lab

Real PostgreSQL/pgvector experiment with 20,000 checked-in synthetic 16D vectors, three query vectors and four filter cutoffs. No embeddings, external model calls or vendor comparison.

## Run

Python 3.10+ standard library and a Docker engine with internet access to pull the pinned image. Linux x86_64 was tested. Allow 2 GB container memory plus Docker overhead; use a remote machine if the laptop is constrained.

```bash
cd pgvector-filtering
python3 run.py --output my-run
python3 verify_results.py my-run
```

The output directory must not already exist. The runner creates only a randomly named owned container, no published ports, no container network, two CPUs, 2 GB RAM, 1 GB tmpfs data, 128 MB shared memory. Local Docker socket commands load/query it. Trust authentication is restricted to this disposable offline container; it is not a deployment template. It stops the container and verifies automatic removal. Evidence files remain.

Pinned image:
`pgvector/pgvector@sha256:3e8b3adfd27b5707128f60956f62a793c3c9326ea8cfaf0eab7adccb5d700b21`

Observed PostgreSQL 17.8, pgvector 0.8.1, x86_64. Tag originally resolved from `pgvector/pgvector:0.8.1-pg17`; execution uses the digest.

## Files

- `vectors.csv`: 20,000 vectors plus synthetic eligibility metadata; committed fixture is authoritative.
- `queries.json`: three fixed vectors.
- `generate_fixture.py`: seeded generator; rerunning intentionally overwrites only these two fixture files.
- `run.py`: builds the database and saves IDs, overlap/recall/counts, timings and every measured plan.
- `verify_results.py`: independently checks exhaustive float32 distance reference IDs, eligibility, hashes, metrics and plans.
- `verified-run/`: actual complete final run and verification receipt.
- `article.md`, `SOURCES.md`: explanation and research boundaries.

```bash
python3 verify_results.py verified-run
python3 run.py --help
```

## Experimental contract

Identical SQL predicate for all modes: tenant alpha, current flag true and cohort below cutoff. Cutoffs 100/25/5/1 yield actual eligible counts 8000/2011/427/87.

Exact disables index and bitmap scans; all plans must show sequential scan and no index. HNSW disables sequential-scan preference and bitmap scans; all plans must show `items_embedding_hnsw`. This is deliberately forced, not the natural planner choice. L2, m=16, ef_construction=64; ef_search=40 both approximate modes; iterative off versus strict_order with max_scan_tuples=20000, scan_mem_multiplier=1. Single-threaded index build.

One warmup and seven measured repetitions per case/mode by default. Mode order shuffled every repetition. Report EXPLAIN ANALYZE execution time, not end-to-end latency; instrumentation included, planning/connection/Docker excluded. No cold-cache or concurrent-load claims. All 252 measured JSON plans include BUFFERS. No assertion assumes which ANN mode must win: only measured eligibility, index use and reference/metric validity are required.

## Interpretation and limits

The final run recovered all exact top10 IDs with iterative scanning, but at tight filters the exact scan was faster. Ordinary HNSW returned fewer than ten under selective filters. Three queries and artificial 16D coordinates cannot establish production behaviour, semantic relevance or vendor performance. Index rebuilds can change approximate outcomes despite identical fixture bytes. Container resources are capped, not isolated CPU cores.

The fixture does not verify permissions, stale metadata, authentication, cache invalidation or document lifecycle. For that conceptual boundary see the separate retrieval-freshness lab. Preserve trusted identity and metadata controls when adapting this harness; do not copy a synthetic tenant string as an authorisation mechanism.

# Sources and evidence

Research checked 8 September 2026.

- [pgvector README pinned to v0.8.1](https://github.com/pgvector/pgvector/blob/v0.8.1/README.md#filtering): approximate-index filtering, candidate search settings, iterative scan modes and limits. The experiment uses this older pinned version for reproducibility, not an assertion that it is current.
- [PostgreSQL 17 EXPLAIN](https://www.postgresql.org/docs/17/using-explain.html): interpreting actual plans, execution measurement and buffer information.
- [PostgreSQL 17 planner method configuration](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-ENABLE): planner switches are experimental controls in this harness, not permanent tuning recommendations.

All numeric results are locally measured on Fedora in the disposable container identified in `verified-run/results.json`. The data are generated synthetic vectors and synthetic metadata, not text embeddings or user data. No latency/recall numbers are copied from vendor material. `verify_results.py` checks every plan and independently checks the reference neighbours. Seven repeated execution measurements per query/mode share one final index build; an exploratory run under `results/` is excluded from the download because its asynchronous container-removal check was incomplete. The final run corrected that proof and confirms removal.

# Measure pgvector recall when filters leave few matches

Your search asks for ten current documents belonging to one tenant. The database returns two. That might mean only two eligible documents exist. It might also mean the approximate index stopped before finding the others.

This lab distinguishes those cases using real PostgreSQL and pgvector. It loads a fixed synthetic vector corpus, establishes an exact eligible top ten, then runs the same query through ordinary HNSW and an iterative HNSW scan. Every measured query has a saved execution plan.

The result is useful precisely because it is limited: iterative scanning recovered the missing neighbours in this run, but an exact scan was faster at the tightest filters. Neither result establishes what your production corpus will do.

## What this experiment measures

The checked-in fixture contains 20,000 vectors with 16 dimensions. Coordinates are seeded pseudorandom numbers, not text embeddings. Each row also has a tenant, a current-revision flag and a cohort number. There are three fixed query vectors.

All three search modes use the same L2 distance and predicate:

```sql
WHERE tenant = 'alpha'
  AND is_current
  AND cohort < 5
ORDER BY embedding <-> query_vector
LIMIT 10
```

Here, `query_vector` represents the fixed vector literal supplied by the harness. The saved per-case SQL contains the complete executable query.

We vary only the cohort cutoff: 100, 25, 5 and 1. Once tenant and current-revision conditions are included, the eligible populations are 8,000, 2,011, 427 and 87 rows. These are measured fixture counts, not assumed selectivities.

This extends our [retrieval freshness lab](/engineering-lab-retrieval-freshness/). That earlier example isolates eligibility and candidate starvation with lexical matching. This one executes an actual approximate vector index. Neither authenticates a caller or validates the authority of document permissions.

## Run it yourself

Download the lab and use Python 3.10 or later plus a working Docker engine:

```bash
cd engineering-labs/pgvector-filtering
python3 run.py --output my-run
python3 verify_results.py my-run
```

The runner refuses to overwrite an existing evidence directory. It creates its own disposable PostgreSQL container, limits it to two CPUs and 2 GB of memory, exposes no ports and disables container networking. Database files live in a 1 GB temporary filesystem. Image downloading needs network access before the container starts.

The checked image digest supplies PostgreSQL 17.8 and pgvector 0.8.1. This is a version-pinned experiment, not a recommendation to deploy that version. The digest, full version string, fixture hashes and settings are saved with the results. Stopping the owned container removes its temporary database; the runner verifies removal.

The saved run includes 252 JSON plans from `EXPLAIN (ANALYZE, BUFFERS)`. You do not need an embedding service, model key or paid API.

## Keep the comparison honest

The exact reference disables index and bitmap scans. Each plan must contain a sequential scan and no index. For the two HNSW modes, sequential scans are discouraged and the harness requires the named HNSW index to appear in every plan. This forces a controlled algorithm comparison; it does not measure which plan PostgreSQL would naturally choose.

The index uses `m=16` and `ef_construction=64`. Both approximate modes use `ef_search=40`. Ordinary HNSW has iterative scanning off. The iterative mode uses `strict_order`, a `max_scan_tuples` setting of 20,000 and a scan-memory multiplier of 1.

pgvector documents that filtering happens after scanning an approximate index, so a restrictive condition can reduce the returned count. Its iterative scans can continue searching, subject to configured limits. Those limits and ordering modes matter; recovering ten rows is not a universal recall guarantee. [pgvector 0.8.1 documentation](https://github.com/pgvector/pgvector/blob/v0.8.1/README.md#iterative-index-scans)

The verifier independently computes exhaustive distances from the checked-in vectors, rounded to float32, and checks the exact reference IDs. It also checks eligibility, duplicates, overlap arithmetic, file hashes and every saved plan.

## The observed results

This run used three queries at each filter cutoff. Each cell below lists the result for queries one, two and three.

| Eligible rows | Eligible fraction | Ordinary HNSW returned | Ordinary HNSW overlap with exact top ten | Iterative overlap with exact top ten |
|---|---|---|---|---|
| 8,000 | 40% | 10 / 10 / 10 | 10 / 10 / 10 | 10 / 10 / 10 |
| 2,011 | 10.055% | 4 / 6 / 4 | 4 / 6 / 4 | 10 / 10 / 10 |
| 427 | 2.135% | 0 / 2 / 0 | 0 / 2 / 0 | 10 / 10 / 10 |
| 87 | 0.435% | 0 / 1 / 0 | 0 / 1 / 0 | 10 / 10 / 10 |

Recall at ten is overlap divided by ten. Returning ten documents alone would not establish perfect recall; they must be the reference neighbours. Here, the iterative mode returned and matched all ten for all twelve cases.

At the tightest filter, ordinary HNSW returned no rows for two queries despite 87 eligible rows existing. The reference answers distinguish that retrieval miss from an empty eligible corpus.

## The latency trade-off

For each case and mode, the harness performs a warmup and seven measured repetitions. Mode order is shuffled each round with a fixed seed. The following values are medians of the 21 execution times per filter and mode.

| Eligible rows | Exact scan | Ordinary HNSW | Iterative HNSW |
|---|---|---|---|
| 8,000 | 1.724 ms | 0.810 ms | 0.809 ms |
| 2,011 | 1.131 ms | 0.803 ms | 1.442 ms |
| 427 | 0.912 ms | 0.808 ms | 3.415 ms |
| 87 | 0.854 ms | 0.804 ms | 8.673 ms |

These are PostgreSQL execution times with EXPLAIN instrumentation, excluding planning, connections and Docker overhead. They are not application response times. PostgreSQL explains the distinction between planner estimates and actual execution measurements in its [EXPLAIN guidance](https://www.postgresql.org/docs/17/using-explain.html).

The small corpus is warm in memory. The nearly constant ordinary-HNSW timing comes with missing results, so treating it as the winner on latency would ignore the task. Conversely, iterative scanning does more work to recover neighbours. At 87 eligible rows, it was slower than the exhaustive reference in this run.

## Turn this into a useful acceptance test

Replace the synthetic vectors with a sanitised corpus and preserve exact reference results for a manageable query sample. Keep the production predicate identical across modes. Record returned count and reference overlap separately.

Then vary one setting at a time: candidate search size, iterative scan limits or index construction. Include filters correlated with vector neighbourhoods; this fixture's metadata is not designed to reproduce your document distribution. Inspect natural planner choices in a separate run before adopting forced settings.

Repeat after rebuilding the index. Fixed inputs do not promise an identical HNSW graph or timings on another machine. This run covers one final index build, no concurrent load, no ingestion churn, no semantic relevance labels and no production permission system.

For wider storage choices, read our [vector database comparison](/vector-database-comparison-2026/). Use this lab to establish your workload's recall boundary before making a vendor decision.

## Sources

- [pgvector 0.8.1: filtering and iterative scans](https://github.com/pgvector/pgvector/blob/v0.8.1/README.md#filtering)
- [PostgreSQL 17: using EXPLAIN](https://www.postgresql.org/docs/17/using-explain.html)

Documentation checked and experiment run on 8 September 2026. All reported numbers come from the downloadable `verified-run` evidence.

{
  "invalid_repeats_exit": 2,
  "message": "--repeats must be at least 3",
  "docker_started": false
}

"""Regenerate the checked-in synthetic corpus; no embeddings or private text."""
import csv
import json
from pathlib import Path
import random

HERE = Path(__file__).resolve().parent
rng = random.Random(20260908)
with (HERE / "vectors.csv").open("w", newline="") as handle:
    writer = csv.writer(handle)
    writer.writerow(["id", "tenant", "is_current", "cohort", "embedding"])
    for identifier in range(1, 20001):
        vector = [round(rng.uniform(-1, 1), 6) for _ in range(16)]
        writer.writerow([identifier, "alpha" if identifier % 2 else "beta",
                         identifier % 5 != 0, rng.randrange(100),
                         json.dumps(vector, separators=(",", ":"))])
queries = [{"id": f"q{i}", "vector": [round(rng.uniform(-1, 1), 6) for _ in range(16)]}
           for i in range(1, 4)]
(HERE / "queries.json").write_text(json.dumps(queries, indent=2) + "\n")

[
  {
    "id": "q1",
    "vector": [
      -0.250991,
      0.218829,
      0.970669,
      -0.540408,
      -0.8071,
      -0.780703,
      0.640392,
      -0.774388,
      -0.589287,
      0.490063,
      -0.444166,
      0.708039,
      0.146432,
      0.725603,
      0.439718,
      0.953679
    ]
  },
  {
    "id": "q2",
    "vector": [
      0.297705,
      -0.727826,
      0.753227,
      0.352118,
      0.921456,
      -0.936784,
      -0.691736,
      0.795657,
      0.1138,
      -0.803514,
      0.431931,
      -0.849935,
      0.525868,
      -0.341543,
      -0.606619,
      -0.097937
    ]
  },
  {
    "id": "q3",
    "vector": [
      -0.434177,
      -0.631676,
      0.644363,
      0.081957,
      -0.018091,
      0.394836,
      -0.023347,
      -0.264493,
      -0.659755,
      0.159411,
      -0.324511,
      0.083379,
      -0.99169,
      0.482471,
      0.613097,
      -0.385371
    ]
  }
]

#!/usr/bin/env python3
"""Run a disposable, pinned PostgreSQL/pgvector filtered-search experiment."""
import argparse
import csv
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import platform
import random
import statistics
import subprocess
import time
import uuid

HERE = Path(__file__).resolve().parent
IMAGE = "pgvector/pgvector@sha256:3e8b3adfd27b5707128f60956f62a793c3c9326ea8cfaf0eab7adccb5d700b21"
MODES = ("exact", "hnsw", "iterative")
BASE = "SET jit=off; SET max_parallel_workers_per_gather=0; SET work_mem='32MB'; "
SETTINGS = {
    "exact": BASE + "SET enable_indexscan=off; SET enable_bitmapscan=off; SET enable_seqscan=on;",
    "hnsw": BASE + "SET enable_seqscan=off; SET enable_bitmapscan=off; SET hnsw.ef_search=40; SET hnsw.iterative_scan=off;",
    "iterative": BASE + "SET enable_seqscan=off; SET enable_bitmapscan=off; SET hnsw.ef_search=40; SET hnsw.iterative_scan=strict_order; SET hnsw.max_scan_tuples=20000; SET hnsw.scan_mem_multiplier=1;"
}


def command(args, data=None, timeout=120):
    result = subprocess.run(args, input=data, text=True, capture_output=True, timeout=timeout)
    if result.returncode:
        raise RuntimeError(f"Command failed: {args[0]}: {result.stderr.strip()}")
    return result.stdout.strip()


def nodes(plan):
    yield plan
    for child in plan.get("Plans", []):
        yield from nodes(child)


def validate_plan(mode, plan):
    entries = list(nodes(plan["Plan"]))
    if mode == "exact":
        assert any(node["Node Type"] == "Seq Scan" for node in entries), entries
        assert not any("Index Name" in node for node in entries), entries
    else:
        assert any(node.get("Index Name") == "items_embedding_hnsw" for node in entries), entries


def query_sql(vector, threshold):
    value = json.dumps(vector, separators=(",", ":"))
    return (f"SELECT id, embedding <-> '{value}'::vector AS distance FROM items "
            f"WHERE tenant='alpha' AND is_current AND cohort < {threshold} "
            f"ORDER BY embedding <-> '{value}'::vector LIMIT 10")


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--output", type=Path, default=HERE / "results")
    parser.add_argument("--repeats", type=int, default=7)
    args = parser.parse_args()
    if args.repeats < 3:
        parser.error("--repeats must be at least 3")
    if args.output.exists():
        parser.error("output directory already exists; choose a new directory")
    command(["docker", "info", "--format", "{{.ServerVersion}}"])
    command(["docker", "pull", IMAGE], timeout=300)
    args.output.mkdir(parents=True)
    name = "ss-pgvector-lab-" + uuid.uuid4().hex[:12]
    receipt = {"started_utc": datetime.now(timezone.utc).isoformat(), "image": IMAGE,
               "container_name": name, "python": platform.python_version(),
               "resources": {"cpus": 2, "memory": "2g", "network": "none", "postgres_data_tmpfs": "1g"},
               "fixture_sha256": hashlib.sha256((HERE / "vectors.csv").read_bytes()).hexdigest(),
               "queries_sha256": hashlib.sha256((HERE / "queries.json").read_bytes()).hexdigest(),
               "run_code_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
               "timing_policy": "One warmup per mode/case, then seven repetitions by default. Mode order shuffled each round with seed 901. Report PostgreSQL EXPLAIN ANALYZE Execution Time, excluding planning, connection and Docker overhead; instrumentation included. Same single index build. Warm shared buffers/OS cache; no cold-cache or concurrency claims.",
               "settings": SETTINGS, "container_removed": False}
    started = False
    try:
        command(["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
                 "--cpus", "2", "--memory", "2g", "--shm-size", "128m",
                 "--tmpfs", "/var/lib/postgresql/data:rw,size=1g",
                 "-e", "POSTGRES_HOST_AUTH_METHOD=trust", IMAGE,
                 "-c", "shared_buffers=128MB", "-c", "max_parallel_maintenance_workers=0"])
        started = True
        for _ in range(60):
            ready = subprocess.run(["docker", "exec", name, "pg_isready", "-U", "postgres"],
                                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
            if ready.returncode == 0:
                break
            time.sleep(0.5)
        else:
            raise RuntimeError("PostgreSQL did not become ready")

        def sql(statement):
            return command(["docker", "exec", "-i", name, "psql", "-XqAt", "-v",
                            "ON_ERROR_STOP=1", "-U", "postgres"], statement)

        sql("CREATE EXTENSION vector; CREATE TABLE items (id integer PRIMARY KEY, tenant text NOT NULL, is_current boolean NOT NULL, cohort integer NOT NULL, embedding vector(16) NOT NULL);")
        sql("COPY items FROM STDIN WITH (FORMAT csv, HEADER true);\n" + (HERE / "vectors.csv").read_text() + "\\.\n")
        sql("SET maintenance_work_mem='256MB'; CREATE INDEX items_embedding_hnsw ON items USING hnsw (embedding vector_l2_ops) WITH (m=16, ef_construction=64);")
        sql("VACUUM ANALYZE items;")
        receipt["postgres"] = sql("SELECT version();")
        receipt["pgvector"] = sql("SELECT extversion FROM pg_extension WHERE extname='vector';")
        receipt["docker_image_id"] = command(["docker", "inspect", name, "--format", "{{.Image}}"])
        receipt["row_count"] = int(sql("SELECT count(*) FROM items;"))
        receipt["index_definition"] = sql("SELECT indexdef FROM pg_indexes WHERE indexname='items_embedding_hnsw';")
        queries = json.loads((HERE / "queries.json").read_text())
        with (HERE / "vectors.csv").open() as handle:
            fixture = {int(row["id"]): row for row in csv.DictReader(handle)}
        rows = []
        order_rng = random.Random(901)
        for threshold in (100, 25, 5, 1):
            eligible = {identifier for identifier, row in fixture.items()
                        if row["tenant"] == "alpha" and row["is_current"] == "True" and int(row["cohort"]) < threshold}
            assert len(eligible) >= 10
            for query in queries:
                statement = query_sql(query["vector"], threshold)
                case = f"cohort-{threshold}-{query['id']}"
                case_dir = args.output / case
                case_dir.mkdir()
                (case_dir / "query.sql").write_text(statement + ";\n")
                timings = {mode: [] for mode in MODES}
                first_plans, selections = {}, {}
                for mode in MODES:
                    selection = json.loads(sql(SETTINGS[mode] +
                        " SELECT coalesce(json_agg(r), '[]'::json) FROM (" + statement + ") r;"))
                    ids = [entry["id"] for entry in selection]
                    assert len(ids) == len(set(ids)) and set(ids).issubset(eligible)
                    selections[mode] = selection
                    warmup = json.loads(sql(SETTINGS[mode] +
                        " EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) " + statement + ";"))[0]
                    validate_plan(mode, warmup)
                for repetition in range(args.repeats):
                    mode_order = list(MODES)
                    order_rng.shuffle(mode_order)
                    for mode in mode_order:
                        plan = json.loads(sql(SETTINGS[mode] +
                            " EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) " + statement + ";"))[0]
                        validate_plan(mode, plan)
                        timings[mode].append(plan["Execution Time"])
                        (case_dir / f"{mode}-plan-{repetition + 1}.json").write_text(json.dumps(plan, indent=2) + "\n")
                        first_plans.setdefault(mode, plan)
                reference = {entry["id"] for entry in selections["exact"]}
                assert len(reference) == 10
                for mode in MODES:
                    ids = [entry["id"] for entry in selections[mode]]
                    overlap = len(set(ids) & reference)
                    rows.append({"case": case, "cohort_limit": threshold,
                                 "eligible_count": len(eligible), "eligible_fraction": len(eligible) / len(fixture),
                                 "mode": mode, "returned_count": len(ids), "overlap_with_exact": overlap,
                                 "recall_at_10": overlap / 10, "ids": ids,
                                 "exact_ids": sorted(reference), "selected_distances": selections[mode],
                                 "execution_ms": timings[mode], "median_execution_ms": statistics.median(timings[mode]),
                                 "min_execution_ms": min(timings[mode]), "max_execution_ms": max(timings[mode]),
                                 "plan_verified": True})
                print(f"{case}: " + ", ".join(f"{m}={len(selections[m])} rows" for m in MODES), flush=True)
        receipt["repeats"] = args.repeats
        receipt["cases"] = 12
        receipt["rows"] = rows
        receipt["status"] = "passed"
    finally:
        if started:
            command(["docker", "stop", name])
            for _ in range(50):
                inspected = subprocess.run(["docker", "inspect", name], capture_output=True)
                if inspected.returncode != 0:
                    receipt["container_removed"] = True
                    break
                time.sleep(0.1)
            if not receipt["container_removed"]:
                raise RuntimeError("Owned container did not finish removal")
        receipt["finished_utc"] = datetime.now(timezone.utc).isoformat()
        (args.output / "results.json").write_text(json.dumps(receipt, indent=2) + "\n")
    print(f"Evidence: {args.output / 'results.json'}")


if __name__ == "__main__":
    main()

"""Verify saved evidence against fixture and an independent exhaustive distance reference."""
import argparse
import csv
import hashlib
import json
from pathlib import Path
import struct
from run import HERE, validate_plan


def f32(value):
    return struct.unpack("f", struct.pack("f", value))[0]


def verify(output):
    result = json.loads((output / "results.json").read_text())
    assert result["status"] == "passed" and result["container_removed"]
    for name, key in [("vectors.csv", "fixture_sha256"), ("queries.json", "queries_sha256"),
                      ("run.py", "run_code_sha256")]:
        assert hashlib.sha256((HERE / name).read_bytes()).hexdigest() == result[key]
    with (HERE / "vectors.csv").open() as handle:
        fixture = list(csv.DictReader(handle))
    queries = {query["id"]: [f32(value) for value in query["vector"]]
               for query in json.loads((HERE / "queries.json").read_text())}
    corpus = {int(row["id"]): [f32(value) for value in json.loads(row["embedding"])] for row in fixture}
    plans_checked = 0
    assert len(result["rows"]) == 36
    for row in result["rows"]:
        query = queries[row["case"].split("-")[-1]]
        eligible = {int(item["id"]) for item in fixture
                    if item["tenant"] == "alpha" and item["is_current"] == "True"
                    and int(item["cohort"]) < row["cohort_limit"]}
        ranked = sorted(eligible, key=lambda identifier: sum((left-right)**2
                        for left, right in zip(query, corpus[identifier])))
        assert set(ranked[:10]) == set(row["exact_ids"]), row["case"]
        assert set(row["ids"]).issubset(eligible)
        assert len(row["ids"]) == len(set(row["ids"])) == row["returned_count"]
        overlap = len(set(row["ids"]) & set(row["exact_ids"]))
        assert overlap == row["overlap_with_exact"]
        assert overlap / 10 == row["recall_at_10"]
        assert len(eligible) == row["eligible_count"]
        assert len(row["execution_ms"]) == result["repeats"]
        for repetition in range(1, result["repeats"] + 1):
            plan = json.loads((output / row["case"] / f"{row['mode']}-plan-{repetition}.json").read_text())
            validate_plan(row["mode"], plan)
            assert plan["Execution Time"] == row["execution_ms"][repetition - 1]
            assert "Shared Hit Blocks" in plan["Plan"]
            plans_checked += 1
    return {"verified": True, "result_rows": 36, "plans_checked": plans_checked,
            "independent_reference_cases": 12, "fixture_and_code_hashes_match": True,
            "container_removed": True}


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    print(json.dumps(verify(args.output), indent=2))