phase: 188_chunks_hnsw_index

**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.

tests/unit/test_config.py

@@ -553,6 +553,39 @@ def test_name_hit_bonus_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> Non
         _settings()
 
 
+def test_retrieval_ef_search_default_and_env_override(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Phase 188 (D3, raised by task 04's live battery): the HNSW
+    ``ef_search`` for the cosine tier defaults to 400 — FOUR times the A7
+    candidate count, because the battery measured ``ef_search = 100``
+    (``== LIMIT``) recovering only mean 0.9814 / min 0.94 of the exact
+    top-100 on the 1,000-document live replica's real embeddings, under this
+    phase's own 0.99 floor — and 400 recovered 1.0000 with a max ``top_score``
+    delta of exactly 0. It stays env-tunable (``BOR_RETRIEVAL_EF_SEARCH``) in
+    either direction, and the retriever applies it per transaction with
+    ``SET LOCAL``."""
+    monkeypatch.delenv("BOR_RETRIEVAL_EF_SEARCH", raising=False)
+    assert _settings().retrieval_ef_search == 400
+    monkeypatch.setenv("BOR_RETRIEVAL_EF_SEARCH", "100")
+    assert _settings().retrieval_ef_search == 100
+
+
+def test_retrieval_ef_search_refuses_zero_and_negative(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """There is no ``ef_search=0`` to switch the index off: the retriever
+    clamps anything below the A7 candidate count UP to it, so a ``0`` an
+    operator might read as "off" would silently become 100. It fails at
+    startup naming the field instead (phase 188, D3; the
+    ``show_text_max_chars`` fail-loud family — with ``0`` refused the way
+    ``recency_half_life_days`` refuses it)."""
+    for bad in ("0", "-1"):
+        monkeypatch.setenv("BOR_RETRIEVAL_EF_SEARCH", bad)
+        with pytest.raises(ValidationError, match="retrieval_ef_search"):
+            _settings()
+
+
 def test_recency_half_life_rejects_non_positive(
     monkeypatch: pytest.MonkeyPatch,
 ) -> None:

tests/unit/test_retriever.py

@@ -697,6 +697,155 @@ def test_vector_candidates_default_is_summary_false_for_legacy_chunks() -> None:
     assert out[0].is_summary is False
 
 
+# ---------------------------------------------------------------------------
+# Phase 188 (task 02): the HNSW ``ef_search`` knob — SET LOCAL, clamped
+# ---------------------------------------------------------------------------
+
+
+def _one_vector_row_session() -> _FakeSession:
+    """A stub session whose one rowset is a single ordinary chunk."""
+    doc = _doc("ef.md", "EF_BODY")
+    chunk = Chunk(
+        id=uuid.uuid4(),
+        document_id=doc.id,
+        position=0,
+        content="ef content",
+        is_summary=False,
+    )
+    return _FakeSession([(chunk, 0.25, doc)])
+
+
+def _sql(session: _FakeSession) -> list[str]:
+    """The recorded statements as SQL text, in call order."""
+    return [str(stmt) for stmt, _params in session.statements]
+
+
+def _ef_statement(session: _FakeSession) -> str | None:
+    for sql in _sql(session):
+        if sql.startswith("SET LOCAL hnsw.ef_search"):
+            return sql
+    return None
+
+
+def test_vector_candidates_sets_ef_search_first(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Phase 188 (D3/LOCKED A4): the tier opens with
+    ``SET LOCAL hnsw.ef_search = <value>`` BEFORE the ``<=>`` SELECT (a
+    ``SET`` that arrived after the SELECT would tune the next turn
+    instead of this one), and it is LOCAL — never a session/global ``SET``.
+    The default (400 — the depth task 04's live battery reached exact
+    top-100 parity at) is what goes out when the operator set nothing."""
+    settings = Settings(_env_file=None)  # pyright: ignore[reportCallIssue]
+    assert settings.retrieval_ef_search == 400  # the production default
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    out = _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert len(out) == 1, "the candidates still come back"
+    statements = _sql(session)
+    assert statements[0] == "SET LOCAL hnsw.ef_search = 400"
+    assert len(statements) == 2, "exactly one SET plus the candidate SELECT"
+    assert "<=>" in statements[1]
+
+
+def test_vector_candidates_clamps_ef_search_below_the_candidate_count(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """``ef_search`` BELOW the candidate count cannot be honoured — it
+    returns FEWER candidates than the A7 tier promises — so it is CLAMPED
+    up to the count with one WARNING line (the phase-188 D3 rule), and the
+    SELECT still asks for the full ``LIMIT``."""
+    settings = Settings(_env_file=None, retrieval_ef_search=10)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    with caplog.at_level("WARNING", logger="app.rag.retriever"):
+        _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 100"
+    messages = [r.getMessage() for r in caplog.records]
+    assert "retrieval: ef_search=10 raised to 100 (A7 candidate count)" in messages
+    select_sql = _sql(session)[1]
+    # the LIMIT stays the tier's own (parameterised) count — the clamp
+    # raises the SEARCH depth, it never shrinks the promise
+    assert "LIMIT :param_1" in select_sql
+
+
+def test_vector_candidates_ef_search_equal_to_the_limit_does_not_warn(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """The clamp is STRICTLY below: ``ef_search == limit`` (the default
+    100/100 shape) is the healthy state and logs nothing."""
+    settings = Settings(_env_file=None, retrieval_ef_search=100)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    with caplog.at_level("WARNING", logger="app.rag.retriever"):
+        _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 100"
+    assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
+
+
+def test_vector_candidates_passes_ef_search_above_the_limit(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """Above the candidate count the operator's number goes through
+    untouched (the recall dial the phase-188 battery may raise it to) and
+    nothing warns."""
+    settings = Settings(_env_file=None, retrieval_ef_search=200)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    with caplog.at_level("WARNING", logger="app.rag.retriever"):
+        _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 200"
+    assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
+
+
+def test_vector_candidates_ef_search_statement_is_int_only(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The interpolated fragment is ``int()``-coerced: the SET line carries
+    digits only — no quote, no semicolon, nothing a string could smuggle
+    into the statement (the value is a ``Settings`` int, but the coercion
+    is the belt beside the suspenders)."""
+    settings = Settings(_env_file=None, retrieval_ef_search=137)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    line = _ef_statement(session)
+    assert line is not None
+    value = line.split("=", 1)[1].strip()
+    assert value == "137" and value.isdigit()
+
+
+def test_ef_search_below_one_refuses_startup_the_clamp_has_no_off_state() -> None:
+    """The clamp means ``ef_search=0`` could never mean "off" — it would be
+    silently rewritten to the candidate count — so ``0`` and negatives are
+    refused at settings construction instead (phase 188, D3; the
+    ``show_text_max_*`` family's fail-loud pattern)."""
+    from pydantic import ValidationError
+
+    for bad in (0, -1):
+        with pytest.raises(ValidationError, match="retrieval_ef_search"):
+            Settings(_env_file=None, retrieval_ef_search=bad)  # pyright: ignore[reportCallIssue]
+
+
+def test_ef_search_default_clears_the_a7_candidate_count() -> None:
+    """The two numbers the clamp rule reads: the shipped default sits ABOVE
+    ``hybrid_vector_candidates`` (so a default install never warns), and it
+    is not merely equal to it.
+
+    Equality was the task-02 default and the task-04 live battery retired it:
+    ``ef_search == LIMIT`` is the weakest depth an HNSW graph search can be
+    handed, and on the 1,000-document live replica's real embeddings it
+    recovered mean 0.9814 / min 0.94 of the exact top-100 — under this
+    phase's 0.99 floor. 400 recovered 1.0000 with every A8 verdict, top-5 set
+    and ``top_score`` unchanged, and it costs the tier nothing measurable
+    (12-14 ms against 12-13 ms at 100, on 416k chunks, versus the ~550 ms
+    sequential scan this index replaces).
+    """
+    settings = Settings(_env_file=None)  # pyright: ignore[reportCallIssue]
+    assert settings.hybrid_vector_candidates == 100  # the A7 clamp floor
+    assert settings.retrieval_ef_search == 400 > settings.hybrid_vector_candidates
+
+
 def _lexical_row(is_summary: bool, doc_path: str) -> object:
     """One row of ``_LEXICAL_SQL`` (attribute access, as SQLAlchemy returns)."""
     doc = _doc(doc_path, "DOC_BODY")

tests/unit/test_settings_catalog.py

@@ -73,6 +73,36 @@ def test_non_overridable_is_exactly_locked_a3() -> None:
     assert set(NON_OVERRIDABLE) == {"database_url", "embedding_dim"}
 
 
+def test_ef_search_row_is_a_plain_live_overridable_row() -> None:
+    """Phase 188 (task 02, D3): the HNSW ``ef_search`` row sits in
+    ``Retrieval & honesty`` (beside ``hybrid_vector_candidates`` — the two
+    numbers the clamp rule reads together) with the ``Vector ef_search``
+    label and is plain — not ``secret`` (an int), not ``restart_required``
+    (``_vector_candidates`` reads ``get_settings()`` per turn, so a saved
+    override applies on the next question), overridable (the overlay is
+    generic over ``int`` fields). ``0``/negative fails startup, not the
+    panel.
+
+    The default this row ships with is 400, not the A7 candidate count it
+    launched at: task 04's live gate measured the graph recovering the exact
+    top-100 for 0.9814 of rows on average (0.94 at worst) at 100 against 1.0000
+    at 400, on the restored live KB with the vectors pinned
+    (``benchmarks/phase188_live_gate/``). It stays live-overridable in either
+    direction, which is what makes that a tuning decision and not a redeploy.
+    """
+    meta = CATALOG["retrieval_ef_search"]
+    assert meta.group == "Retrieval & honesty"
+    assert meta.label == "Vector ef_search"
+    assert meta.secret is False
+    assert meta.restart_required is False
+    assert meta.overridable is True
+    assert "retrieval_ef_search" not in SECRETS
+    assert "retrieval_ef_search" not in RESTART_REQUIRED
+    assert "retrieval_ef_search" not in NON_OVERRIDABLE
+    assert Settings.model_fields["retrieval_ef_search"].annotation is int
+    assert Settings.model_fields["retrieval_ef_search"].default == 400
+
+
 def test_restart_required_is_the_boot_wired_set() -> None:
     # LOCKED A4: logging, the gzip middleware (compress_min_bytes), the
     # static mount (static_dir), create_engine (db_pool_*), and