ducoterra
f20d40e4fe
**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
186 lines
12 KiB
Markdown
186 lines
12 KiB
Markdown
# Phase 188 task 04 — the live recall / parity gate (index OFF vs ON)
|
||
|
||
Measured on **live production** (`brain.reeseapps.com`'s database, reached at
|
||
`localhost:5432` through the dev-tunnel the app uses), on the **live embedding
|
||
endpoint**, against a restored **1,000-document live-replica KB**. Nothing here
|
||
ran against the synthetic 5,000-row fixture — that corpus belongs to task 03 and
|
||
it is not a proxy for the shapes real embeddings take.
|
||
|
||
Reproduce with `.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py`
|
||
(its `--help` has the four commands); the battery itself is
|
||
`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`.
|
||
|
||
## Prerequisites (measured, not assumed)
|
||
|
||
| Check | Value |
|
||
| --- | --- |
|
||
| `SELECT extversion FROM pg_extension WHERE extname = 'vector'` | **0.8.0** (>= 0.5.0 required; `hnsw` available) |
|
||
| `alembic current` before | head (0028), index absent |
|
||
| Replica corpus | 1,000 documents, **8,866 chunks**, 8,866 embeddings, heap **13 MB** (from a 95 MB plain-SQL dump) |
|
||
| `maintenance_work_mem` | **64 MB** (the container's default — `postgres:17` never got the phase's 512 MB, see below) |
|
||
| `/dev/shm` in `brain-of-reese_db_1` | **63 MB** |
|
||
| Planner costs | `seq_page_cost = 1`, `random_page_cost = 4` (default, SSD-class) |
|
||
| Disk free on the Postgres data volume | **2.2 TB free of 3.7 TB** (41 % used) |
|
||
| Production relation, from the 2026-10-15 incident record (not measured in this task) | 414,220 chunks, heap **3,000 MB** |
|
||
| Index after `alembic upgrade head` | `ix_chunks_embedding_hnsw` **valid**, **34 MB** on the replica |
|
||
| `alembic_version` inside the dump | **`0011`** — so `alembic upgrade head` replays **0012…0029** into the restored corpus and 0029's `CREATE INDEX CONCURRENTLY` builds against **real** embeddings. Restoring into a schema already at head would build nothing and measure nothing. |
|
||
|
||
## The battery, index OFF vs index ON (`eval_retrieval`)
|
||
|
||
`battery_index_off.txt` was captured with the index dropped
|
||
(`uv run alembic downgrade -1`), `battery_index_on.txt` with it present
|
||
(`uv run alembic upgrade head`). Both runs embed the seven battery questions
|
||
through the live endpoint.
|
||
|
||
The two captures are **byte-identical** — every `best_cosine`, every `fts_hits`,
|
||
every one of the five numbered hits per question, and every `DEFLECT`/`HIGH`
|
||
verdict. The only difference is the plan the server chose, which battery output
|
||
does not print (the deterministic probe below does).
|
||
|
||
**Why the battery alone cannot gate this.** At `ef_search = 100` its OFF and ON
|
||
captures disagreed by up to ~3e-4 on printed cosines — above this phase's 1e-4
|
||
tolerance — and the battery cannot say where that came from. The probe can: the
|
||
endpoint is deterministic (`endpoint_self_jitter`, one question embedded twice,
|
||
measured **0.000e+00**), and the tier reading shows the mechanism — at that depth the
|
||
two top-100s agree on only 0.9814 of rows on average (0.94 worst), a substituted tail
|
||
candidate shifts the RRF ranks, and since a document contributes several chunks a
|
||
printed hit can keep its document while changing its chunk, moving the printed
|
||
cosine while the top-5 list and `top_score` hold. At the shipped 400 the two
|
||
captures are byte-identical. Reading all 100 rows with the vectors pinned is what
|
||
separates the two effects. So the gate ran
|
||
a second time with the question vectors **pinned** (`embed` once, then
|
||
`measure` both sides from the cache): the 1e-4 tolerance was kept, and the input
|
||
was made identical instead of the assertion weakened (LOCKED A3).
|
||
|
||
## The deterministic gate (pinned vectors, same code path)
|
||
|
||
`deterministic_gate_compare.txt` — A = index dropped (the pre-phase behaviour:
|
||
`Limit → Sort → Hash Join → Seq Scan on chunks`), B = index present at the
|
||
shipped `ef_search`, question vectors identical on both sides:
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| A8 verdicts (all 7) | **identical** — every question `HIGH (answer)`, the out-of-KB Mongolia probe included (`best_cosine=0.6070`, under the 0.62 threshold, but A8 deflects only when the cosine is under it **and** there is no full-text support — it has 5 FTS hits, unchanged by this index) |
|
||
| Top-5 document sets **and order** (all 7) | **identical** |
|
||
| Suggested documents (all 7) | **identical** |
|
||
| `top_score` / `best_cosine` delta | **0.000e+00** (gate ≤ 1e-4) |
|
||
| Cosine-tier score delta, ranks 1–5 (the fused window) | **0.000e+00** (gate ≤ 1e-4) |
|
||
| Cosine-tier top-100 **candidate-set overlap** | **1.0000 on all 7** (floor 0.99) |
|
||
| Rank swaps that are not exact cosine ties | **0** (every swap is a tie broken by access path) |
|
||
| Tier latency (median over questions) | **38.3 ms → 15.3 ms** |
|
||
| Plan, index present | `Limit → Nested Loop → Index Scan on ix_chunks_embedding_hnsw` |
|
||
|
||
`deterministic_gate_recall.txt` — the same tier forced onto the index with
|
||
`enable_indexscan = on` at each depth, recall@100 against the exact scan, on
|
||
**real embeddings**:
|
||
|
||
| `ef_search` | mean recall@100 | min recall@100 | max per-chunk cosine delta | max `top_score` delta | tier latency |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| 100 (the A7 candidate count — task 02's default) | 0.9814 | 0.94 | 2.195e-03 | **0.000e+00** | 13.8 ms |
|
||
| 200 | 1.0000 | 1.0000 | 0.0 | 0.000e+00 | 14.1 ms |
|
||
| 400 (**shipped**) | 1.0000 | 1.0000 | 0.0 | 0.000e+00 | 14.6 ms |
|
||
|
||
Two HNSW graph builds were measured (the index is dropped and rebuilt between the
|
||
OFF and ON runs). The 100-row candidate set at `ef_search = 100` came out
|
||
0.9814 mean / 0.94 min on one build and 0.9843 / 0.94 on the other; `200` came
|
||
out 1.0000 / 1.0000 and 0.9971 / 0.99; `400` came out 1.0000 / 1.0000 both times.
|
||
|
||
## Decision: `BOR_RETRIEVAL_EF_SEARCH` ships at 400
|
||
|
||
* At the task-02 default (`ef_search = 100 == LIMIT`) the **user-visible** numbers
|
||
did not move at all — verdicts, top-5 sets, suggested documents and
|
||
`top_score` were identical, to `0.000e+00`. So the task's parity gate passes at
|
||
100 and the deflection calibration was never in question.
|
||
* What moved was the **100-row candidate set** the cosine tier hands the RRF
|
||
fusion: mean 0.9814, min 0.94 — below this phase's own acceptance floor
|
||
(mean ≥ 0.99, min ≥ 0.95), and matching the counter-case task 03 recorded for
|
||
high-effective-rank data. One substitution at rank ~89 of 100 was enough to
|
||
move 22 tail ranks.
|
||
* LOCKED A3 names exactly one remedy: **raise `ef_search`** (towards 400). The
|
||
thresholds were not touched. `200` measured borderline (1.0000 on one graph,
|
||
0.9971 on another); **400 measured exact on both**, and on real embeddings it
|
||
costs **0.5 ms** (14.1 → 14.6 ms) against the depth-100 search and **23 ms**
|
||
against the sequential scan it replaces.
|
||
* The default in `app/config.py` is therefore **400**, still overridable with
|
||
`BOR_RETRIEVAL_EF_SEARCH`, and phase 188's clamp log line still fires if an
|
||
operator sets it below the candidate count.
|
||
|
||
## Production-order probe (`production_order_*.log`)
|
||
|
||
The replica's 8,866 chunks are 13 MB of heap, which makes it the wrong place to
|
||
ask "does production use the index, and what does it cost?": the relation is an
|
||
order of magnitude smaller than the one that produced the incident, and pgvector
|
||
costs an HNSW scan partly on `ef_search`, so a small-relation plan does not
|
||
transfer. A throwaway database (`bor_scale188b`, created and dropped during
|
||
this task) answered it at the production row count, using the replica's own
|
||
embeddings scaled ×47 with each copy multiplied element-wise by a random
|
||
near-one jitter vector, so all **416,702** vectors are distinct (415,950 unique)
|
||
and the heap is 557 MB:
|
||
|
||
| Measurement | Value |
|
||
| --- | --- |
|
||
| Build through the real migration (`uv run alembic upgrade head`, `maintenance_work_mem = 64 MB`) | **161 s**, peak RSS of the alembic process 81 MB |
|
||
| Index size | **1,625 MB** (vs a 557 MB heap — HNSW at `m = 16`, 768-dim float4 costs ~3.9 KB/row: 414k rows ⇒ plan for ~1.6 GB) |
|
||
| Plan at `ef_search` 100 / 200 / 400 | **HNSW index scan at every depth** (index path cost 3,333 / 5,621 / 9,806 vs a seq-scan-and-sort alternative an order of magnitude above) |
|
||
| Tier latency | **~550 ms sequential → 12.0–14.9 ms** with the index (≈ **40×**), the ~1 ms `SET LOCAL` included |
|
||
| Top-100 overlap vs the exact scan | 0.23–0.64, **not a valid recall measurement** — see the caveat below |
|
||
|
||
**Caveat, recorded because it is a limitation of this gate.** A corpus built by
|
||
jittering 8,866 real embeddings into 47 near-copies each is a pathological graph
|
||
for a top-100 recall metric: "the exact top-100" is then 100 knife-edge slots
|
||
inside a handful of near-duplicate clusters, which is not the shape of a real KB.
|
||
The numbers are reported so the latency, plan and build-cost readings at
|
||
production row count are on the record, and the **recall** verdict is taken from
|
||
the live replica, where every vector is a real, distinct chunk. After the
|
||
production build, the owner should re-run the battery on the real 414k-chunk KB
|
||
exactly as this task did on the replica; `tests/integration/test_chunks_hnsw_recall.py`
|
||
now fails (rather than passing vacuously) if a search at the shipped depth is not
|
||
actually reading the index.
|
||
|
||
## Task 05 re-verification — the battery at the shipped default, from scratch
|
||
|
||
`task05_battery_index_off.txt` / `task05_battery_index_on.txt` are a clean-room
|
||
re-run of the same gate, performed by task 05 to confirm the verdict still holds
|
||
at the shipped `BOR_RETRIEVAL_EF_SEARCH = 400` before the phase shipped. Nothing
|
||
was reused from task 04's state: the dump was restored into a **throwaway
|
||
database** (`bor_p188gate`, created and dropped inside the run, so the dev
|
||
database was never touched), which lands at `alembic_version = 0011` and was
|
||
taken to **`0028`** (index absent — the OFF arm), then to **`head`** through the
|
||
real migration (0029 built `ix_chunks_embedding_hnsw` against the 8,866 real
|
||
embeddings: **valid**, **34 MB**, ~1 s wall clock, `alembic` peak RSS **81 MB**
|
||
— the ON arm). Both arms ran
|
||
`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`
|
||
with live embeddings against live Postgres.
|
||
|
||
**The two captures are byte-identical** (Python string equality, not a diff
|
||
summary): all seven `best_cosine` values, all seven `fts_hits` counts, all 35
|
||
numbered hits with their scores, and all seven gate verdicts — five `HIGH`, the
|
||
Mongolia probe `HIGH` on its 5 FTS hits with `best_cosine = 0.6070` under the
|
||
0.62 threshold, and the two Besta questions behaving as out-of-KB asks on a
|
||
replica that has no IKEA document (`grep -c -i ikea` in the dump = 0, verified
|
||
before drawing any conclusion from them).
|
||
|
||
| Arm | Schema state | Index | Battery capture |
|
||
| --- | --- | --- | --- |
|
||
| OFF | `0028` | absent | `task05_battery_index_off.txt` |
|
||
| ON | `0029` (head) | valid, 34 MB | `task05_battery_index_on.txt` |
|
||
| Equality | — | — | **identical, byte for byte** |
|
||
|
||
## Finding: `maintenance_work_mem = 512MB` cannot be used in this container
|
||
|
||
```
|
||
ERROR: could not resize shared memory segment "/PostgreSQL.3366409952" to 533761504 bytes: No space left on device
|
||
```
|
||
|
||
`brain-of-reese_db_1` has a **63 MB `/dev/shm`**, so any `maintenance_work_mem`
|
||
above ~63 MB makes a parallel HNSW build die, and the failed build leaves an
|
||
**INVALID** index behind. This is a deviation from D5, which suggested
|
||
`SET maintenance_work_mem = '512MB'`: on this host the value is not applicable,
|
||
and the container's **64 MB default builds the index successfully**. The failure
|
||
was observed live at production order, and it exercised the self-heal for real:
|
||
migration 0029 detected `indisvalid = false`, logged
|
||
`dropping INVALID … rebuilding`, dropped it and rebuilt it valid. The task's
|
||
recorded command is therefore "run `alembic upgrade head` as-is"; raising
|
||
`maintenance_work_mem` is only worth doing after `/dev/shm` is raised too
|
||
(`podman run --shm-size=1g`), and a raise that fails is recoverable.
|