phase: 188_chunks_hnsw_index

**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
2026-10-17 03:18:23 -04:00
parent c054b47df5
commit f20d40e4fe
81 changed files with 5076 additions and 2 deletions
@@ -697,6 +697,155 @@ def test_vector_candidates_default_is_summary_false_for_legacy_chunks() -> None:
    assert out[0].is_summary is False


+# ---------------------------------------------------------------------------
+# Phase 188 (task 02): the HNSW ``ef_search`` knob — SET LOCAL, clamped
+# ---------------------------------------------------------------------------
+
+
+def _one_vector_row_session() -> _FakeSession:
+    """A stub session whose one rowset is a single ordinary chunk."""
+    doc = _doc("ef.md", "EF_BODY")
+    chunk = Chunk(
+        id=uuid.uuid4(),
+        document_id=doc.id,
+        position=0,
+        content="ef content",
+        is_summary=False,
+    )
+    return _FakeSession([(chunk, 0.25, doc)])
+
+
+def _sql(session: _FakeSession) -> list[str]:
+    """The recorded statements as SQL text, in call order."""
+    return [str(stmt) for stmt, _params in session.statements]
+
+
+def _ef_statement(session: _FakeSession) -> str | None:
+    for sql in _sql(session):
+        if sql.startswith("SET LOCAL hnsw.ef_search"):
+            return sql
+    return None
+
+
+def test_vector_candidates_sets_ef_search_first(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Phase 188 (D3/LOCKED A4): the tier opens with
+    ``SET LOCAL hnsw.ef_search = <value>`` BEFORE the ``<=>`` SELECT (a
+    ``SET`` that arrived after the SELECT would tune the next turn
+    instead of this one), and it is LOCAL — never a session/global ``SET``.
+    The default (400 — the depth task 04's live battery reached exact
+    top-100 parity at) is what goes out when the operator set nothing."""
+    settings = Settings(_env_file=None)  # pyright: ignore[reportCallIssue]
+    assert settings.retrieval_ef_search == 400  # the production default
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    out = _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert len(out) == 1, "the candidates still come back"
+    statements = _sql(session)
+    assert statements[0] == "SET LOCAL hnsw.ef_search = 400"
+    assert len(statements) == 2, "exactly one SET plus the candidate SELECT"
+    assert "<=>" in statements[1]
+
+
+def test_vector_candidates_clamps_ef_search_below_the_candidate_count(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """``ef_search`` BELOW the candidate count cannot be honoured — it
+    returns FEWER candidates than the A7 tier promises — so it is CLAMPED
+    up to the count with one WARNING line (the phase-188 D3 rule), and the
+    SELECT still asks for the full ``LIMIT``."""
+    settings = Settings(_env_file=None, retrieval_ef_search=10)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    with caplog.at_level("WARNING", logger="app.rag.retriever"):
+        _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 100"
+    messages = [r.getMessage() for r in caplog.records]
+    assert "retrieval: ef_search=10 raised to 100 (A7 candidate count)" in messages
+    select_sql = _sql(session)[1]
+    # the LIMIT stays the tier's own (parameterised) count — the clamp
+    # raises the SEARCH depth, it never shrinks the promise
+    assert "LIMIT :param_1" in select_sql
+
+
+def test_vector_candidates_ef_search_equal_to_the_limit_does_not_warn(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """The clamp is STRICTLY below: ``ef_search == limit`` (the default
+    100/100 shape) is the healthy state and logs nothing."""
+    settings = Settings(_env_file=None, retrieval_ef_search=100)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    with caplog.at_level("WARNING", logger="app.rag.retriever"):
+        _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 100"
+    assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
+
+
+def test_vector_candidates_passes_ef_search_above_the_limit(
+    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
+) -> None:
+    """Above the candidate count the operator's number goes through
+    untouched (the recall dial the phase-188 battery may raise it to) and
+    nothing warns."""
+    settings = Settings(_env_file=None, retrieval_ef_search=200)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    with caplog.at_level("WARNING", logger="app.rag.retriever"):
+        _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 200"
+    assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
+
+
+def test_vector_candidates_ef_search_statement_is_int_only(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The interpolated fragment is ``int()``-coerced: the SET line carries
+    digits only — no quote, no semicolon, nothing a string could smuggle
+    into the statement (the value is a ``Settings`` int, but the coercion
+    is the belt beside the suspenders)."""
+    settings = Settings(_env_file=None, retrieval_ef_search=137)  # pyright: ignore[reportCallIssue]
+    monkeypatch.setattr(retriever, "get_settings", lambda: settings)
+    session = _one_vector_row_session()
+    _vector_candidates(session, [0.0] * 768, limit=100)  # pyright: ignore[reportArgumentType]
+    line = _ef_statement(session)
+    assert line is not None
+    value = line.split("=", 1)[1].strip()
+    assert value == "137" and value.isdigit()
+
+
+def test_ef_search_below_one_refuses_startup_the_clamp_has_no_off_state() -> None:
+    """The clamp means ``ef_search=0`` could never mean "off" — it would be
+    silently rewritten to the candidate count — so ``0`` and negatives are
+    refused at settings construction instead (phase 188, D3; the
+    ``show_text_max_*`` family's fail-loud pattern)."""
+    from pydantic import ValidationError
+
+    for bad in (0, -1):
+        with pytest.raises(ValidationError, match="retrieval_ef_search"):
+            Settings(_env_file=None, retrieval_ef_search=bad)  # pyright: ignore[reportCallIssue]
+
+
+def test_ef_search_default_clears_the_a7_candidate_count() -> None:
+    """The two numbers the clamp rule reads: the shipped default sits ABOVE
+    ``hybrid_vector_candidates`` (so a default install never warns), and it
+    is not merely equal to it.
+
+    Equality was the task-02 default and the task-04 live battery retired it:
+    ``ef_search == LIMIT`` is the weakest depth an HNSW graph search can be
+    handed, and on the 1,000-document live replica's real embeddings it
+    recovered mean 0.9814 / min 0.94 of the exact top-100 — under this
+    phase's 0.99 floor. 400 recovered 1.0000 with every A8 verdict, top-5 set
+    and ``top_score`` unchanged, and it costs the tier nothing measurable
+    (12-14 ms against 12-13 ms at 100, on 416k chunks, versus the ~550 ms
+    sequential scan this index replaces).
+    """
+    settings = Settings(_env_file=None)  # pyright: ignore[reportCallIssue]
+    assert settings.hybrid_vector_candidates == 100  # the A7 clamp floor
+    assert settings.retrieval_ef_search == 400 > settings.hybrid_vector_candidates
+
+
 def _lexical_row(is_summary: bool, doc_path: str) -> object:
    """One row of ``_LEXICAL_SQL`` (attribute access, as SQLAlchemy returns)."""
    doc = _doc(doc_path, "DOC_BODY")