phase: 188_chunks_hnsw_index

**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.

AGENTS.md

@@ -140,7 +140,31 @@
     clause) therefore satisfies `peak >= current`, never a peak below the
     run's own footprint, and an unknowable peak is never fabricated from
     the current reading); UI-side,
-    honor the "never stale" feedback contract (PLAN §7.4).
+    honor the "never stale" feedback contract (PLAN §7.4). Phase 188 adds
+    the retrieval tier's ONE operator-visible line: when
+    `BOR_RETRIEVAL_EF_SEARCH` sits below the A7 candidate count
+    `_vector_candidates` logs `retrieval: ef_search=%d raised to %d (A7
+    candidate count)` at WARNING and runs the search at the candidate
+    count anyway — the clamp is never silent, and the healthy
+    (ef_search >= candidate count) turn logs nothing. The value itself
+    reaches Postgres as `SET LOCAL hnsw.ef_search` inside the turn's own
+    transaction (LOCKED A4: no global GUC, nothing survives the request or
+    a pool recycle). Task 04's live gate (the 1,000-document live-replica KB,
+    the live embedding endpoint, question vectors PINNED so the access path is the
+    one variable — `benchmarks/phase188_live_gate/`) found no A8, top-5 or
+    `top_score` difference at any
+    depth and shipped `retrieval_ef_search = 400`: at the candidate count the
+    graph recovered the exact top-100 for 0.9814 of rows on average (0.94
+    worst), at 400 for all of them, and NO threshold
+    (`relevance_threshold`, `lexical_support_floor`, `name_hit_bonus`,
+    `hybrid_top_k`, `hybrid_vector_candidates`) moved to get there (LOCKED A3).
+    Two things the operator needs from the same run: building the index costs
+    ~161 s per 417k chunks and ~1.6 GB (3.9 KB/row at `m = 16`, 768-dim), and
+    `maintenance_work_mem = '512MB'` is NOT applicable in this container —
+    `/dev/shm` is 63 MB, so a raise above that dies with `could not resize
+    shared memory segment … No space left on device` and leaves an INVALID
+    index for the migration's self-heal to drop and rebuild (observed live);
+    the container's 64 MB default builds it fine.
 11. **Never block the event loop:** anything blocking (a `subprocess`,
     synchronous file/network I/O, a heavy walk) reached from an `async`
     handler or a spawned background coroutine goes through
@@ -246,6 +270,44 @@
     stored text, a changed file re-indexes and the fast path still fires
     after).
 
+16. **The cosine tier is indexed — keep it indexed, and keep `0029` the ONE
+    definition of that index:** `ix_chunks_embedding_hnsw` (HNSW over
+    `chunks.embedding`, `vector_cosine_ops`, `m = 16`,
+    `ef_construction = 64`) exists because every turn's
+    `ORDER BY embedding <=> :vec LIMIT 100` full-scanned ~3 GB on the
+    interactive path (phase 188). Build and drop it ONLY through migration
+    `0029` (`CREATE/DROP INDEX CONCURRENTLY` inside
+    `op.get_context().autocommit_block()`, which pre-drops an INVALID remnant
+    and raises if the build does not validate) — never an inline
+    `CREATE INDEX` in app code, a test, or a runbook step; never a second
+    vector index, a `halfvec` cast, or a rewritten `ORDER BY` (the `<=>`
+    operator is what the graph serves, LOCKED A3/A6). A test that needs the
+    index OFF moves the SCHEMA (`command.downgrade(cfg, "0028")` /
+    `command.upgrade(cfg, "0029")`) after rolling back its own session — an
+    open snapshot parks a `CONCURRENTLY` build indefinitely — and asserts the
+    catalog state it claims (`pg_index.indisvalid`, `pg_get_indexdef`).
+    `hnsw.ef_search` is set with `SET LOCAL` inside the turn's own transaction
+    and the `int()`-coerced setting is the only source (no global `SET`, no
+    pool-wide state, nothing survives a recycle); the shipped default is
+    **400** because that is the depth the live gate measured exact-top-100
+    parity at, so "tuning" it back to the candidate count is a recall change,
+    and closing a recall gap by moving an honesty threshold is forbidden
+    (LOCKED A3). The `ef_search` a turn gets is
+    `max(retrieval_ef_search, hybrid_vector_candidates)` with a WARNING when
+    clamped. Pinned by `tests/integration/test_chunks_hnsw_index.py` (DDL
+    round trip + INVALID self-heal + drift guard),
+    `tests/integration/test_chunks_hnsw_recall.py` (the tier's captured
+    statement names the index, `SET LOCAL` reaches the connection, recall at
+    the shipped depth with seq scans refused), and
+    `tests/e2e/test_vector_index_recall.py` (the owner-visible arm: the SAME
+    cited document with the index dropped and restored, the index VALID after a
+    real importer run, the near-miss still deflecting in both arms, and the
+    clamp read off the app's own captured stdout at `=150` / `=50`). That
+    suite's sentinels are calibrated against
+    `tests/fixtures/hnswkb/` — an 11-note tracked KB whose BYTES own the
+    quoted cosine/lexical margins, so editing a note there moves the numbers in
+    the module docstring and the pins together.
+
 ## Quick reference
 ```bash
 podman compose up -d db                      # start Postgres 17 + pgvector