brain-of-reese/benchmarks/phase188_live_gate

Phase 188 task 04 — the live recall / parity gate (index OFF vs ON)

Measured on live production (brain.reeseapps.com's database, reached at localhost:5432 through the dev-tunnel the app uses), on the live embedding endpoint, against a restored 1,000-document live-replica KB. Nothing here ran against the synthetic 5,000-row fixture — that corpus belongs to task 03 and it is not a proxy for the shapes real embeddings take.

Reproduce with .agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py (its --help has the four commands); the battery itself is uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt.

Prerequisites (measured, not assumed)

Check	Value
SELECT extversion FROM pg_extension WHERE extname = 'vector'	0.8.0 (>= 0.5.0 required; hnsw available)
alembic current before	head (0028), index absent
Replica corpus	1,000 documents, 8,866 chunks, 8,866 embeddings, heap 13 MB (from a 95 MB plain-SQL dump)
maintenance_work_mem	64 MB (the container's default — postgres:17 never got the phase's 512 MB, see below)
/dev/shm in brain-of-reese_db_1	63 MB
Planner costs	seq_page_cost = 1, random_page_cost = 4 (default, SSD-class)
Disk free on the Postgres data volume	2.2 TB free of 3.7 TB (41 % used)
Production relation, from the 2026-10-15 incident record (not measured in this task)	414,220 chunks, heap 3,000 MB
Index after alembic upgrade head	ix_chunks_embedding_hnsw valid, 34 MB on the replica
alembic_version inside the dump	0011 — so alembic upgrade head replays 0012…0029 into the restored corpus and 0029's CREATE INDEX CONCURRENTLY builds against real embeddings. Restoring into a schema already at head would build nothing and measure nothing.

The battery, index OFF vs index ON (eval_retrieval)

battery_index_off.txt was captured with the index dropped (uv run alembic downgrade -1), battery_index_on.txt with it present (uv run alembic upgrade head). Both runs embed the seven battery questions through the live endpoint.

The two captures are byte-identical — every best_cosine, every fts_hits, every one of the five numbered hits per question, and every DEFLECT/HIGH verdict. The only difference is the plan the server chose, which battery output does not print (the deterministic probe below does).

Why the battery alone cannot gate this. At ef_search = 100 its OFF and ON captures disagreed by up to ~3e-4 on printed cosines — above this phase's 1e-4 tolerance — and the battery cannot say where that came from. The probe can: the endpoint is deterministic (endpoint_self_jitter, one question embedded twice, measured 0.000e+00), and the tier reading shows the mechanism — at that depth the two top-100s agree on only 0.9814 of rows on average (0.94 worst), a substituted tail candidate shifts the RRF ranks, and since a document contributes several chunks a printed hit can keep its document while changing its chunk, moving the printed cosine while the top-5 list and top_score hold. At the shipped 400 the two captures are byte-identical. Reading all 100 rows with the vectors pinned is what separates the two effects. So the gate ran a second time with the question vectors pinned (embed once, then measure both sides from the cache): the 1e-4 tolerance was kept, and the input was made identical instead of the assertion weakened (LOCKED A3).

The deterministic gate (pinned vectors, same code path)

deterministic_gate_compare.txt — A = index dropped (the pre-phase behaviour: Limit → Sort → Hash Join → Seq Scan on chunks), B = index present at the shipped ef_search, question vectors identical on both sides:

Check	Result
A8 verdicts (all 7)	identical — every question HIGH (answer), the out-of-KB Mongolia probe included (best_cosine=0.6070, under the 0.62 threshold, but A8 deflects only when the cosine is under it and there is no full-text support — it has 5 FTS hits, unchanged by this index)
Top-5 document sets and order (all 7)	identical
Suggested documents (all 7)	identical
top_score / best_cosine delta	0.000e+00 (gate ≤ 1e-4)
Cosine-tier score delta, ranks 1–5 (the fused window)	0.000e+00 (gate ≤ 1e-4)
Cosine-tier top-100 candidate-set overlap	1.0000 on all 7 (floor 0.99)
Rank swaps that are not exact cosine ties	0 (every swap is a tie broken by access path)
Tier latency (median over questions)	38.3 ms → 15.3 ms
Plan, index present	Limit → Nested Loop → Index Scan on ix_chunks_embedding_hnsw

deterministic_gate_recall.txt — the same tier forced onto the index with enable_indexscan = on at each depth, recall@100 against the exact scan, on real embeddings:

ef_search	mean recall@100	min recall@100	max per-chunk cosine delta	max top_score delta	tier latency
100 (the A7 candidate count — task 02's default)	0.9814	0.94	2.195e-03	0.000e+00	13.8 ms
200	1.0000	1.0000	0.0	0.000e+00	14.1 ms
400 (shipped)	1.0000	1.0000	0.0	0.000e+00	14.6 ms

Two HNSW graph builds were measured (the index is dropped and rebuilt between the OFF and ON runs). The 100-row candidate set at ef_search = 100 came out 0.9814 mean / 0.94 min on one build and 0.9843 / 0.94 on the other; 200 came out 1.0000 / 1.0000 and 0.9971 / 0.99; 400 came out 1.0000 / 1.0000 both times.

Decision: BOR_RETRIEVAL_EF_SEARCH ships at 400

• At the task-02 default (ef_search = 100 == LIMIT) the user-visible numbers did not move at all — verdicts, top-5 sets, suggested documents and top_score were identical, to 0.000e+00. So the task's parity gate passes at 100 and the deflection calibration was never in question.
• What moved was the 100-row candidate set the cosine tier hands the RRF fusion: mean 0.9814, min 0.94 — below this phase's own acceptance floor (mean ≥ 0.99, min ≥ 0.95), and matching the counter-case task 03 recorded for high-effective-rank data. One substitution at rank ~89 of 100 was enough to move 22 tail ranks.
• LOCKED A3 names exactly one remedy: raise ef_search (towards 400). The thresholds were not touched. 200 measured borderline (1.0000 on one graph, 0.9971 on another); 400 measured exact on both, and on real embeddings it costs 0.5 ms (14.1 → 14.6 ms) against the depth-100 search and 23 ms against the sequential scan it replaces.
• The default in app/config.py is therefore 400, still overridable with BOR_RETRIEVAL_EF_SEARCH, and phase 188's clamp log line still fires if an operator sets it below the candidate count.

Production-order probe (production_order_*.log)

The replica's 8,866 chunks are 13 MB of heap, which makes it the wrong place to ask "does production use the index, and what does it cost?": the relation is an order of magnitude smaller than the one that produced the incident, and pgvector costs an HNSW scan partly on ef_search, so a small-relation plan does not transfer. A throwaway database (bor_scale188b, created and dropped during this task) answered it at the production row count, using the replica's own embeddings scaled ×47 with each copy multiplied element-wise by a random near-one jitter vector, so all 416,702 vectors are distinct (415,950 unique) and the heap is 557 MB:

Measurement	Value
Build through the real migration (uv run alembic upgrade head, maintenance_work_mem = 64 MB)	161 s, peak RSS of the alembic process 81 MB
Index size	1,625 MB (vs a 557 MB heap — HNSW at m = 16, 768-dim float4 costs ~3.9 KB/row: 414k rows ⇒ plan for ~1.6 GB)
Plan at ef_search 100 / 200 / 400	HNSW index scan at every depth (index path cost 3,333 / 5,621 / 9,806 vs a seq-scan-and-sort alternative an order of magnitude above)
Tier latency	~550 ms sequential → 12.0–14.9 ms with the index (≈ 40×), the ~1 ms SET LOCAL included
Top-100 overlap vs the exact scan	0.23–0.64, not a valid recall measurement — see the caveat below

Caveat, recorded because it is a limitation of this gate. A corpus built by jittering 8,866 real embeddings into 47 near-copies each is a pathological graph for a top-100 recall metric: "the exact top-100" is then 100 knife-edge slots inside a handful of near-duplicate clusters, which is not the shape of a real KB. The numbers are reported so the latency, plan and build-cost readings at production row count are on the record, and the recall verdict is taken from the live replica, where every vector is a real, distinct chunk. After the production build, the owner should re-run the battery on the real 414k-chunk KB exactly as this task did on the replica; tests/integration/test_chunks_hnsw_recall.py now fails (rather than passing vacuously) if a search at the shipped depth is not actually reading the index.

Task 05 re-verification — the battery at the shipped default, from scratch

task05_battery_index_off.txt / task05_battery_index_on.txt are a clean-room re-run of the same gate, performed by task 05 to confirm the verdict still holds at the shipped BOR_RETRIEVAL_EF_SEARCH = 400 before the phase shipped. Nothing was reused from task 04's state: the dump was restored into a throwaway database (bor_p188gate, created and dropped inside the run, so the dev database was never touched), which lands at alembic_version = 0011 and was taken to 0028 (index absent — the OFF arm), then to head through the real migration (0029 built ix_chunks_embedding_hnsw against the 8,866 real embeddings: valid, 34 MB, ~1 s wall clock, alembic peak RSS 81 MB — the ON arm). Both arms ran uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt with live embeddings against live Postgres.

The two captures are byte-identical (Python string equality, not a diff summary): all seven best_cosine values, all seven fts_hits counts, all 35 numbered hits with their scores, and all seven gate verdicts — five HIGH, the Mongolia probe HIGH on its 5 FTS hits with best_cosine = 0.6070 under the 0.62 threshold, and the two Besta questions behaving as out-of-KB asks on a replica that has no IKEA document (grep -c -i ikea in the dump = 0, verified before drawing any conclusion from them).

Arm	Schema state	Index	Battery capture
OFF	0028	absent	task05_battery_index_off.txt
ON	0029 (head)	valid, 34 MB	task05_battery_index_on.txt
Equality	—	—	identical, byte for byte

Finding: maintenance_work_mem = 512MB cannot be used in this container

ERROR:  could not resize shared memory segment "/PostgreSQL.3366409952" to 533761504 bytes: No space left on device

brain-of-reese_db_1 has a 63 MB /dev/shm, so any maintenance_work_mem above ~63 MB makes a parallel HNSW build die, and the failed build leaves an INVALID index behind. This is a deviation from D5, which suggested SET maintenance_work_mem = '512MB': on this host the value is not applicable, and the container's 64 MB default builds the index successfully. The failure was observed live at production order, and it exercised the self-heal for real: migration 0029 detected indisvalid = false, logged dropping INVALID … rebuilding, dropped it and rebuilt it valid. The task's recorded command is therefore "run alembic upgrade head as-is"; raising maintenance_work_mem is only worth doing after /dev/shm is raised too (podman run --shm-size=1g), and a raise that fails is recoverable.