phase: 188_chunks_hnsw_index

**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.

app/config.py

@@ -349,6 +349,44 @@ class Settings(BaseSettings):
     # live 2774-chunk KB for "How did I install gitlab?" (gitlab.md:1 at
     # vrank 100 / lrank 3 → fused 0.0221 vs the template's 0.0164).
     hybrid_vector_candidates: int = 100
+    #: HNSW search depth for the cosine tier (phase 188, D3/LOCKED A4):
+    #: ``SET LOCAL hnsw.ef_search`` immediately before
+    #: ``app.rag.retriever._vector_candidates``'s SELECT — per transaction,
+    #: so no request leaks state into another and no global GUC (nor
+    #: connection-pool) setting is touched. A value BELOW the A7 candidate
+    #: count cannot be honoured — it would return FEWER candidates than A7
+    #: promises (a silent recall regression) — so the retriever CLAMPS it up
+    #: to the candidate count with one WARNING line instead. ``0`` is refused
+    #: here for the same reason: the clamp would silently rewrite it, which
+    #: is exactly what an operator reading ``0`` as "off" would not expect.
+    #: Negative values fail startup loudly (the ``show_text_max_*`` validator
+    #: pattern, phase 182).
+    #:
+    #: Why 400 and not the candidate count (task 04's live battery, the
+    #: 1,000-document live replica, 8,866 real embeddings, the 7-question
+    #: battery with the vectors PINNED so only the index differs):
+    #:
+    #: * ``ef_search = 100`` (``== LIMIT``, the floor the clamp enforces)
+    #:   recovers mean **0.9814** / min **0.94** of the exact top-100 — under
+    #:   this phase's own 0.99 recall floor (LOCKED A3: the remedy for a
+    #:   recall miss is a higher ``ef_search``, never a relaxed assertion).
+    #:   Production holds 47× the chunks, where a fixed ``ef_search`` is
+    #:   weaker still, not stronger.
+    #: * ``ef_search = 400`` recovers **1.0000** of the exact top-100 with a
+    #:   max per-candidate ``top_score`` delta of exactly ``0.0`` — the
+    #:   honesty gate (A8) reads that number, and no verdict, top-5 set or
+    #:   suggested list moved at ANY depth measured.
+    #: * it is also the depth at which the PLANNER actually uses the index on
+    #:   a small relation: at ``ef_search = 100`` Postgres costs the HNSW scan
+    #:   above a sequential one and full-scans anyway (measured 51.5 ms vs the
+    #:   forced-index 2.6 ms); at 400 the plan names
+    #:   ``ix_chunks_embedding_hnsw`` at 5.6 ms — still ~9× under the
+    #:   sequential scan this phase exists to remove.
+    #:
+    #: The A7 clamp floor stays at ``hybrid_vector_candidates`` (100): with a
+    #: 400 default a stock install never warns. Postgres caps ``ef_search`` at
+    #: 1000.
+    retrieval_ef_search: int = 400
     hybrid_lexical_candidates: int = 30
     rrf_k: int = 60
     #: Recency boost on the RRF-fused retrieval score (phase 106, D6): the
@@ -797,6 +835,21 @@ class Settings(BaseSettings):
             )
         return v
 
+    @field_validator("retrieval_ef_search")
+    @classmethod
+    def _retrieval_ef_search_positive(cls, v: int) -> int:
+        """``0``/negative is a typo, not a switch (phase 188, D3). There is
+        no ``ef_search=0`` to turn the index off — the retriever clamps any
+        value below the A7 candidate count UP to it, so a ``0`` that read as
+        "off" would silently become 100 instead of failing. Fail loud at
+        startup, naming the field (the ``show_text_max_chars`` pattern)."""
+        if v <= 0:
+            raise ValueError(
+                "retrieval_ef_search must be > 0 (HNSW ef_search; the "
+                "retriever clamps values below the candidate count up to it)"
+            )
+        return v
+
     @field_validator("embed_question_max_chars")
     @classmethod
     def _embed_question_max_chars_positive(cls, v: int) -> int:

app/core/settings_catalog.py

@@ -205,6 +205,9 @@ CATALOG: dict[str, SettingMeta] = {
         "Retrieval & honesty",
         "Hybrid vector candidates",
     ),
+    "retrieval_ef_search": SettingMeta(
+        "retrieval_ef_search", "Retrieval & honesty", "Vector ef_search"
+    ),
     "hybrid_lexical_candidates": SettingMeta(
         "hybrid_lexical_candidates",
         "Retrieval & honesty",

app/rag/retriever.py

@@ -91,6 +91,7 @@ Deterministic tie-break for equal fused scores:
 """
 from __future__ import annotations
 
+import logging
 import math
 import re
 import unicodedata
@@ -107,6 +108,12 @@ from app.config import get_settings
 from app.models import Chunk, Document
 from app.schemas import SourceRef
 
+#: Phase 188 (D3): the ONE logger of this module — its only line so far is
+#: the ``ef_search`` clamp WARNING (the retrieval path was silent by
+#: design; a knob that silently degraded recall is exactly what needs a
+#: line in the server log).
+logger = logging.getLogger("app.rag.retriever")
+
 #: Shared overflow marker (phase 15; imported by ``app.rag.prompts``)
 #: — used by the steering (<tuning>) section, the phase-118 NULL-summary
 #: suggestion preview fallback (A5), and the capped agent ``read``
@@ -436,7 +443,31 @@ def _vector_candidates(
     (two-phase import in progress) are skipped. Each candidate carries
     its ``Chunk.is_summary`` flag (phase 30) so a summary hit stays
     identifiable after fusion.
+
+    Phase 188 (D3, LOCKED A4): the HNSW tier's search depth is set
+    EXPLICITLY with ``SET LOCAL hnsw.ef_search`` immediately before the
+    SELECT — never a global ``SET``, so the value lives and dies with the
+    request's transaction (nothing leaks into another request, nothing
+    survives a pool recycle, no connection-pool-wide state). The
+    ``ef_search`` a turn actually gets is
+    ``max(settings.retrieval_ef_search, limit)``: an ``ef_search`` BELOW
+    the A7 candidate count would return FEWER candidates than the cosine
+    tier promises (a silent recall regression the honesty gate can turn
+    into a deflect), so it is CLAMPED up to *limit* with one WARNING line
+    instead of being honoured. The value is ``int()``-coerced before it
+    reaches SQL — no operator-supplied string is ever interpolated
+    (:func:`app.config.Settings.retrieval_ef_search` refuses ``0`` and
+    negatives at startup). The setting is read through
+    :func:`app.config.get_settings` exactly like every other retrieval
+    knob in this module.
     """
+    ef_search = get_settings().retrieval_ef_search
+    if ef_search < limit:
+        logger.warning(
+            "retrieval: ef_search=%d raised to %d (A7 candidate count)", ef_search, limit
+        )
+        ef_search = limit
+    db.execute(text(f"SET LOCAL hnsw.ef_search = {int(ef_search)}"))
     distance = Chunk.embedding.cosine_distance(question_embedding)
     rows = db.execute(
         select(Chunk, distance.label("distance"), Document)