phase: 188_chunks_hnsw_index
**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
This commit is contained in:
@@ -109,6 +109,15 @@ os.environ["BOR_ANSWER_TEXT"] = str(_Settings.model_fields["answer_text"].defaul
|
||||
os.environ["BOR_SHOW_TEXT_MAX_CHARS"] = str(
|
||||
_Settings.model_fields["show_text_max_chars"].default
|
||||
)
|
||||
|
||||
# Phase 188: the same leak class for the HNSW ``ef_search`` knob — an
|
||||
# operator's shell may legitimately carry ``BOR_RETRIEVAL_EF_SEARCH``
|
||||
# raised after the recall battery, and the default / clamp / statement-text
|
||||
# pins (unit + integration) must see the code default (100 = the A7
|
||||
# candidate count, so a default run never clamps).
|
||||
os.environ["BOR_RETRIEVAL_EF_SEARCH"] = str(
|
||||
_Settings.model_fields["retrieval_ef_search"].default
|
||||
)
|
||||
os.environ["BOR_SHOW_TEXT_MAX_BLOCKS"] = str(
|
||||
_Settings.model_fields["show_text_max_blocks"].default
|
||||
)
|
||||
|
||||
@@ -156,6 +156,14 @@ def app_server(mock_llm: int) -> Iterator[str]:
|
||||
env["BOR_SHOW_TEXT_MAX_BLOCKS"] = str(
|
||||
_Settings.model_fields["show_text_max_blocks"].default
|
||||
)
|
||||
# Phase 188: the same leak guard for the HNSW ``ef_search`` knob — an
|
||||
# operator's shell may legitimately carry ``BOR_RETRIEVAL_EF_SEARCH``
|
||||
# raised after the recall battery, and the story suites' retrieval pins
|
||||
# must see the CODE default (100 = the A7 candidate count, so a default
|
||||
# run never clamps).
|
||||
env["BOR_RETRIEVAL_EF_SEARCH"] = str(
|
||||
_Settings.model_fields["retrieval_ef_search"].default
|
||||
)
|
||||
env.setdefault("BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese")
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
|
||||
@@ -0,0 +1,889 @@
|
||||
"""Phase 188 (task 05) E2E (Playwright, mock-only): the ``chunks.embedding``
|
||||
HNSW index reaches the owner's chat, changes nothing the owner sees, and the
|
||||
one retrieval knob the phase adds is honoured on a real turn.
|
||||
|
||||
Source: owner incident 2026-10-15. ``pg_indexes`` for ``chunks`` listed the
|
||||
PK, ``ix_chunks_document_id`` and ``ix_chunks_tsv`` and NOTHING on
|
||||
``embedding``, so every turn's A7 cosine tier (``ORDER BY embedding <=> :vec
|
||||
LIMIT 100``) full-scanned a ~3 GB, 414,220-row relation on the interactive
|
||||
path. Migration ``0029`` adds ``ix_chunks_embedding_hnsw`` (HNSW,
|
||||
``vector_cosine_ops``, ``m = 16``, ``ef_construction = 64``) and task 02 adds
|
||||
``BOR_RETRIEVAL_EF_SEARCH`` — shipped at **400** on task 04's measured
|
||||
decision — applied as ``SET LOCAL hnsw.ef_search`` inside the turn's own
|
||||
transaction.
|
||||
|
||||
**What this module owns, and what it does not.** The plan-level proof (the
|
||||
tier's captured statement really names the index, and a turn's ``SET LOCAL``
|
||||
reaches the connection) lives in
|
||||
``tests/integration/test_chunks_hnsw_recall.py`` (tasks 03/04) and the DDL
|
||||
round trip in ``tests/integration/test_chunks_hnsw_index.py`` (task 01).
|
||||
THIS file is the last mile: one question, in a real browser, over a real
|
||||
import, with the index DROPPED and RESTORED around it — an A/B the owner
|
||||
could run on their own box. The corpus is 11 small documents, so Postgres'
|
||||
cost model may well serve BOTH arms from a sequential scan; that is stated,
|
||||
not hidden. An index at this size is not a speed claim, it is a
|
||||
**non-regression** claim, and the only non-regression that matters is the
|
||||
answer. So what is compared between the arms is what reached the model: the
|
||||
durable retrieval record (``query_log.sources`` / ``top_score`` /
|
||||
``chunk_hits`` / ``fts_hits`` — LOCKED A3's witness), the ``done`` frame's
|
||||
tiers, and the rendered "Nearby docs, in case" row. The DDL state is read
|
||||
back from the catalog INSIDE each arm, so "OFF" can never quietly mean "ON".
|
||||
|
||||
Both sentinels are calibrated against the deterministic mock embedder (the
|
||||
``tests/e2e/test_retrieval_quality.py`` convention, not invented) and the
|
||||
margins are quoted in the units the honesty gate reads:
|
||||
|
||||
* ANSWERED — ``"How do I raise the tilecache eviction watermark so the hot
|
||||
zoom levels stay pinned?"`` → best cosine **0.6187**, lexical (FTS) hits
|
||||
**10**, leading document ``hnswkb/infra/tilecache.md``. ``0.6187 >= 0.30``
|
||||
is A8's HIGH quadrant by a margin of 0.32, so neither arm can flip on a
|
||||
rounding difference.
|
||||
* DEFLECTED — ``"Which bow should I draw for a baroque viola sonata?"`` (no
|
||||
note in this KB can answer it) → best cosine **0.1706**, lexical hits **0**.
|
||||
A8 deflects only on weak cosine AND zero lexical hits, and both halves hold
|
||||
with room: 0.13 BELOW the 0.30 threshold, and a lexical side that is empty
|
||||
rather than merely low.
|
||||
|
||||
Run in isolation (DB up: ``podman compose up -d db``, schema at head so
|
||||
``0029`` has run — the module fails loudly, never vacuously, if it has not):
|
||||
|
||||
uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the cosine and lexical
|
||||
numbers above are properties of the deterministic mock, and the parity pins
|
||||
compare them exactly between the two arms.
|
||||
|
||||
Isolation (the house shape): ONE module app on its own port (the conftest
|
||||
session app owns 8123 in a combined run), the mock LLM on the conftest port,
|
||||
a dedicated ``BOR_SOURCES_DIR`` / ``BOR_PDF_PAGE_DIR`` so nothing writes into
|
||||
the operator's, the shared E2E Postgres truncated BEFORE **and** AFTER every
|
||||
test, and the KB re-imported through the REAL importer for every test — so
|
||||
the HNSW insert path runs in every arm, including the ones that then drop the
|
||||
index. The admin enters through the real form login.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule — the rendered DOM is the
|
||||
observable, the durable row is the witness, and the two are read as one
|
||||
comparison):
|
||||
|
||||
1. ``test_answered_question_cites_the_same_document_indexed_or_not`` — the answered
|
||||
sentinel, twice: with ``ix_chunks_embedding_hnsw`` present, then after the
|
||||
real ``0029`` downgrade drops it (restored in ``finally``, and again by the
|
||||
module guard). Both arms ground (no ``.is-deflected``, the mock marker
|
||||
renders), both cite the SAME document — ``hnswkb/infra/tilecache.md`` first
|
||||
in ``query_log.sources`` and first in the rendered row — and agree on
|
||||
``top_score`` to within ``1e-9``, on ``chunk_hits`` and ``fts_hits``, and
|
||||
on the byte-identical answer. The DDL really moved; the answer did not.
|
||||
2. ``test_index_exists_and_is_valid_after_import`` — the import that just ran
|
||||
left the index present, VALID in ``pg_index``, HNSW over
|
||||
``vector_cosine_ops`` per ``pg_get_indexdef``, and carrying pages. An
|
||||
INVALID index is silent (the failure ``0029`` exists to catch), so the
|
||||
post-import state is pinned instead of assumed.
|
||||
3. ``test_near_miss_question_still_deflects`` — the out-of-KB sentinel in
|
||||
BOTH DDL arms: the amber deflection, the "Maybe try" chips, the ``done``
|
||||
frame citing nothing, and the row's two gate inputs read in the units A8
|
||||
defines. An index must never manufacture a high ``top_score``.
|
||||
4. ``test_ef_search_setting_is_honoured`` — two extra apps on their own ports
|
||||
with stdout captured (the surface the operator greps): ``BOR_RETRIEVAL_EF_SEARCH=150``
|
||||
answers the sentinel normally and logs NO clamp line; ``=50`` answers it
|
||||
just as well — same leading doc, same ``top_score`` — and logs the
|
||||
operator-visible ``retrieval: ef_search=50 raised to 100 (A7 candidate
|
||||
count)`` WARNING. The clamp keeps its promise in the browser, not only in
|
||||
a unit test.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config as AlembicConfig
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import func, select, text
|
||||
|
||||
from alembic import command
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import QueryLog
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
MOCK_PORT,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The tracked fixture KB — 11 markdown notes over 6 folders. Tracked rather
|
||||
#: than generated: the cosine/lexical calibration in the module docstring
|
||||
#: belongs to THESE bytes, so a re-run ranks the same way on any day.
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "hnswkb"
|
||||
|
||||
#: The ``source`` the importer records for a walked root (``Path(path).name``).
|
||||
SOURCE_NAME = "hnswkb"
|
||||
N_DOCS = 11
|
||||
|
||||
#: Dedicated module ports (the house env-overridable pattern): the conftest
|
||||
#: session app owns 8123, phase 187's modules own 8203/8204.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_VECIDX", "8205"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
EF_OK_PORT = int(os.environ.get("E2E_APP_PORT_VECIDX_EF150", "8206"))
|
||||
EF_OK_URL = f"http://127.0.0.1:{EF_OK_PORT}"
|
||||
EF_CLAMP_PORT = int(os.environ.get("E2E_APP_PORT_VECIDX_EF50", "8207"))
|
||||
EF_CLAMP_URL = f"http://127.0.0.1:{EF_CLAMP_PORT}"
|
||||
|
||||
#: Migration 0029 and the revision it sits on — the pair the drop/restore
|
||||
#: walks, moved in the migration's OWN directions. This file contains no
|
||||
#: inline ``CREATE INDEX``/``DROP INDEX``: 0029 is the single definition of
|
||||
#: the index these pins name.
|
||||
REV_HNSW = "0029"
|
||||
REV_BEFORE = "0028"
|
||||
|
||||
#: The index 0029 builds.
|
||||
INDEX_NAME = "ix_chunks_embedding_hnsw"
|
||||
|
||||
#: The answered sentinel (measured, module docstring): cosine 0.6187, 10
|
||||
#: lexical hits, ``infra/tilecache.md`` first in the fused record.
|
||||
ANSWER_QUESTION = (
|
||||
"How do I raise the tilecache eviction watermark so the hot zoom levels "
|
||||
"stay pinned?"
|
||||
)
|
||||
#: The document the owner asked about — the one both arms must cite.
|
||||
CITED_DOC = f"{SOURCE_NAME}/infra/tilecache.md"
|
||||
|
||||
#: The near-miss sentinel (measured): cosine 0.1706, 0 lexical hits.
|
||||
DEFLECT_QUESTION = "Which bow should I draw for a baroque viola sonata?"
|
||||
|
||||
#: The mock's grounded answer marker (``tests/e2e/mock_llm.py``).
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
#: The honest-deflection phrase the mock answers with (the house constant —
|
||||
#: ``test_honest_deflection.py`` et al., phase 52's copy as the deterministic
|
||||
#: mock speaks it).
|
||||
DEFLECT_PHRASE = r"haven't done anything like that"
|
||||
|
||||
#: The app's honesty gate — the values the module app boots with, named once
|
||||
#: here so the pins quote the gate they are measured against.
|
||||
GATE_THRESHOLD = 0.30
|
||||
GATE_LEXICAL_FLOOR = 0.15
|
||||
|
||||
#: The A7 cosine tier's candidate count (``hybrid_vector_candidates``) — the
|
||||
#: floor the ``ef_search`` clamp raises to. Read from the code default instead
|
||||
#: of restated, so a change to the knob moves the pin with it.
|
||||
A7_CANDIDATES = int(_Settings.model_fields["hybrid_vector_candidates"].default)
|
||||
|
||||
#: The suggested tier's size (``suggested_docs``) — the split between what the
|
||||
#: prompt is seeded with and what renders as "Nearby docs, in case".
|
||||
SUGGESTED_DOCS = int(_Settings.model_fields["suggested_docs"].default)
|
||||
|
||||
#: The clamp line ``app/rag/retriever.py`` logs (AGENTS rule 10: a knob that
|
||||
#: would degrade recall is never silent), matched on the app's captured
|
||||
#: stdout: ``… WARNING app.rag.retriever :: retrieval: ef_search=50 raised
|
||||
#: to 100 (A7 candidate count)``.
|
||||
CLAMP_LINE = re.compile(
|
||||
r"WARNING\s+app\.rag\.retriever\s+::\s+retrieval: ef_search=(\d+) "
|
||||
r"raised to (\d+) \(A7 candidate count\)"
|
||||
)
|
||||
|
||||
#: Numeric slack between the arms' ``top_score``. Both arms run the same exact
|
||||
#: cosine tier over this corpus, so the true slack is zero; the epsilon only
|
||||
#: absorbs the float's travel through the row.
|
||||
SCORE_SLACK = 1e-9
|
||||
|
||||
#: Turn budget: a mock turn lands in well under a second; 30 s is the house
|
||||
#: head-room with no client-side hard timeout.
|
||||
SETTLE_TIMEOUT_MS = 30_000
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The module app + the two ef_search apps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(scratch: Path, ef_search: str | None = None) -> dict[str, str]:
|
||||
"""App env: the mock LLM, the mock-calibrated honesty gate (the conftest
|
||||
convention — both floors stay ``<=`` the threshold or the app refuses to
|
||||
boot), no git-source fallback, dedicated directories so nothing writes
|
||||
into the operator's.
|
||||
|
||||
``ef_search=None`` pins the knob to the CODE default (the phase 188 leak
|
||||
guard: an operator's ``.env`` value never reaches a test); any other value
|
||||
is what the app boots with.
|
||||
"""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = f"http://127.0.0.1:{MOCK_PORT}/v1"
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = str(GATE_THRESHOLD)
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = str(GATE_LEXICAL_FLOOR)
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = str(GATE_LEXICAL_FLOOR)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages")
|
||||
env["BOR_RETRIEVAL_EF_SEARCH"] = (
|
||||
ef_search
|
||||
if ef_search is not None
|
||||
else str(_Settings.model_fields["retrieval_ef_search"].default)
|
||||
)
|
||||
return env
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _running_app(
|
||||
url: str, port: int, env: dict[str, str], log_path: Path | None = None
|
||||
) -> Iterator[tuple[str, Path | None]]:
|
||||
"""Boot the app on *port*, wait for ``/api/health``, tear it down.
|
||||
|
||||
With *log_path* the app's stdout+stderr go to that file — the same surface
|
||||
the operator greps (``grep ' :: ' server.log``), so the clamp pin reads
|
||||
what a human would read rather than a test-private logger.
|
||||
"""
|
||||
args = [
|
||||
sys.executable,
|
||||
"-m",
|
||||
"uvicorn",
|
||||
"app.main:app",
|
||||
"--host",
|
||||
"127.0.0.1",
|
||||
"--port",
|
||||
str(port),
|
||||
"--log-level",
|
||||
"warning",
|
||||
]
|
||||
if log_path is None:
|
||||
proc = subprocess.Popen(args, cwd=REPO, env=env) # noqa: S603
|
||||
try:
|
||||
_wait_http(f"{url}/api/health")
|
||||
yield url, None
|
||||
finally:
|
||||
proc.terminate()
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
proc.wait(timeout=10)
|
||||
return
|
||||
with open(log_path, "wb") as log:
|
||||
proc = subprocess.Popen( # noqa: S603
|
||||
args, cwd=REPO, env=env, stdout=log, stderr=subprocess.STDOUT
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{url}/api/health")
|
||||
yield url, log_path
|
||||
finally:
|
||||
proc.terminate()
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
proc.wait(timeout=10)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The app under tests 1-3: the SHIPPED default ``retrieval_ef_search``
|
||||
(400 — task 04's measured decision) over the fixture KB."""
|
||||
if USE_REAL_LLM:
|
||||
pytest.skip(
|
||||
"mock-only suite — the cosine/lexical calibration and the two-arm "
|
||||
"parity pins belong to the deterministic mock "
|
||||
"(E2E_REAL_LLM unsupported)"
|
||||
)
|
||||
assert mock_llm
|
||||
scratch = tmp_path_factory.mktemp("bor_vector_index_app")
|
||||
with _running_app(APP_URL, APP_PORT, _app_env(scratch)) as (url, _log):
|
||||
yield url
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The index state, read from the catalog (never inferred)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _index_state(db: Any) -> dict[str, Any] | None:
|
||||
"""The ``pg_index`` row for :data:`INDEX_NAME`, or ``None`` when absent.
|
||||
|
||||
``None`` is the OFF arm. ``valid=False`` is a state nothing here may treat
|
||||
as ON: an INVALID index exists, is maintained on every insert, and serves
|
||||
no queries — silently, which is the failure ``0029`` was written to catch.
|
||||
"""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT i.indisvalid, pg_table_size(c.oid) AS size_bytes, "
|
||||
"pg_get_indexdef(c.oid) AS indexdef "
|
||||
"FROM pg_index i "
|
||||
"JOIN pg_class c ON c.oid = i.indexrelid "
|
||||
"JOIN pg_namespace n ON n.oid = c.relnamespace "
|
||||
"WHERE c.relname = :name AND n.nspname = 'public'"
|
||||
),
|
||||
{"name": INDEX_NAME},
|
||||
).first()
|
||||
if row is None:
|
||||
return None
|
||||
return {
|
||||
"valid": bool(row.indisvalid),
|
||||
"size_bytes": int(row.size_bytes),
|
||||
"indexdef": str(row.indexdef),
|
||||
}
|
||||
|
||||
|
||||
def _alembic_config() -> AlembicConfig:
|
||||
cfg = AlembicConfig(str(REPO / "alembic.ini"))
|
||||
cfg.set_main_option("script_location", str(REPO / "alembic"))
|
||||
cfg.attributes["configure_logger"] = False
|
||||
return cfg
|
||||
|
||||
|
||||
def _alembic_version() -> str:
|
||||
with SessionLocal() as db:
|
||||
version = db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
db.rollback()
|
||||
return str(version)
|
||||
|
||||
|
||||
def _set_index_present(present: bool) -> None:
|
||||
"""Move the schema to the index OFF / ON state through 0029's own
|
||||
``downgrade`` / ``upgrade``, and PROVE the move landed.
|
||||
|
||||
Both directions run in the migration's ``autocommit_block`` (a
|
||||
``CONCURRENTLY`` DDL cannot run inside a transaction). This process's own
|
||||
session is rolled back FIRST — an open snapshot is what parks a
|
||||
``CONCURRENTLY`` build behind the test session forever, the failure task 01
|
||||
documented.
|
||||
"""
|
||||
with SessionLocal() as probe:
|
||||
probe.rollback()
|
||||
cfg = _alembic_config()
|
||||
if present:
|
||||
command.upgrade(cfg, REV_HNSW)
|
||||
else:
|
||||
command.downgrade(cfg, REV_BEFORE)
|
||||
with SessionLocal() as db:
|
||||
state = _index_state(db)
|
||||
version = db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
db.rollback()
|
||||
if present:
|
||||
assert state is not None and state["valid"], (
|
||||
f"restoring {REV_HNSW} left no VALID {INDEX_NAME}: {state}"
|
||||
)
|
||||
assert str(version) == REV_HNSW, f"alembic_version did not return: {version}"
|
||||
else:
|
||||
assert state is None, f"{INDEX_NAME} survived the {REV_BEFORE} downgrade: {state}"
|
||||
assert str(version) == REV_BEFORE, f"alembic_version did not move: {version}"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def _index_at_head() -> Iterator[None]:
|
||||
"""Module guard: the schema STARTS and ENDS at 0029 with a valid index.
|
||||
|
||||
A suite that silently ran with the index missing would pass tests 1 and 3
|
||||
for the wrong reason, so entering is an assertion rather than a hope and
|
||||
the teardown repairs whatever a failed test left behind.
|
||||
"""
|
||||
version = _alembic_version()
|
||||
if version != REV_HNSW:
|
||||
pytest.fail(
|
||||
f"phase 188's task-05 suite needs the schema at {REV_HNSW} "
|
||||
f"(found {version!r}) — run `uv run alembic upgrade head`"
|
||||
)
|
||||
with SessionLocal() as db:
|
||||
state = _index_state(db)
|
||||
db.rollback()
|
||||
if state is None or not state["valid"]:
|
||||
_set_index_present(True)
|
||||
yield
|
||||
_set_index_present(True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The KB, imported through the REAL importer with the index in place
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _mock_llm_client() -> LLMClient:
|
||||
"""An embedder aimed at the mock (no ``.env``: the test process must not
|
||||
inherit the operator's live endpoint)."""
|
||||
kwargs: dict[str, Any] = {
|
||||
"_env_file": None,
|
||||
"llm_base_url": f"http://127.0.0.1:{MOCK_PORT}/v1",
|
||||
}
|
||||
settings = _Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return LLMClient(settings)
|
||||
|
||||
|
||||
async def _import_kb() -> ImportSummary:
|
||||
return await import_sources([FIXTURES], _mock_llm_client())
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread (Playwright owns the test loop)."""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
thread = Thread(target=runner)
|
||||
thread.start()
|
||||
thread.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry + query log (the phase-184 table list)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def kb(mock_llm: int) -> Iterator[ImportSummary]:
|
||||
"""A fresh KB of the tracked fixture tree, imported with the HNSW index
|
||||
PRESENT, so every arm — including the ones that then drop it — put its
|
||||
embeddings into the graph.
|
||||
|
||||
One fixture owns the whole boundary so the order cannot drift: truncate →
|
||||
assert the index is live → import → test → truncate. The index assertion
|
||||
runs AFTER the truncate because a TRUNCATE of an HNSW-indexed table is
|
||||
exactly the insert path this suite is about.
|
||||
"""
|
||||
_truncate_all()
|
||||
with SessionLocal() as db:
|
||||
state = _index_state(db)
|
||||
db.rollback()
|
||||
assert state is not None and state["valid"], (
|
||||
f"{INDEX_NAME} must be present and valid BEFORE the import — the "
|
||||
f"point of this suite is inserts into a live graph: {state}"
|
||||
)
|
||||
|
||||
summary = _run_in_thread(_import_kb())
|
||||
assert summary.errors == 0, f"the fixture KB failed to import: {summary}"
|
||||
assert summary.added == N_DOCS, f"expected {N_DOCS} fixture notes: {summary}"
|
||||
assert summary.summaries == N_DOCS, f"every note needs a summary: {summary}"
|
||||
yield summary
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers: the wire, the DOM, the durable row
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream (a response
|
||||
#: clone read in the background) — the wire-level witness, independent of the
|
||||
#: rendering (the ``test_agent_document_tools.py`` hook).
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable response: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _frames(page: Page, since: int) -> list[dict[str, Any]]:
|
||||
"""This turn's SSE frames, once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 10.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw[since:] if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame ({len(parsed)} frames) — "
|
||||
"was the hook installed before the turn?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _done(frames: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""The turn's ONE ``done`` frame — the shape task 03 pins."""
|
||||
dones = [f for f in frames if f.get("type") == "done"]
|
||||
assert len(dones) == 1, f"expected exactly one done frame, got {dones}"
|
||||
return dones[0]
|
||||
|
||||
|
||||
def _new_chat(page: Page) -> None:
|
||||
"""Clear the conversation so the NEXT turn is a first turn (no history
|
||||
from the previous arm rides the prompt). The control the owner uses."""
|
||||
page.click("#new-chat-btn")
|
||||
expect(page.locator(".msg")).to_have_count(0)
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> int:
|
||||
"""Send *question* as the turn's ONLY message (phase 162's rule: a trailing
|
||||
phrase changes what retrieval sees). Returns the frame count so far."""
|
||||
seen = int(page.evaluate("() => (window.__sseFrames || []).length"))
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
return seen
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The settle sync: the client re-labels Send on the ``done`` frame, and
|
||||
the server writes the ``query_log`` row just before yielding it."""
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=SETTLE_TIMEOUT_MS)
|
||||
|
||||
|
||||
def _row(question: str) -> dict[str, Any]:
|
||||
"""The durable record for *question* as a plain snapshot — exactly one row
|
||||
(the caller clears the log between arms that ask the same question).
|
||||
|
||||
Read as values, not as an ORM instance: the row outlives its session here
|
||||
(the arm compares it against another arm's), and a detached instance
|
||||
raises the moment an attribute is touched.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
QueryLog.top_score,
|
||||
QueryLog.fts_hits,
|
||||
QueryLog.chunk_hits,
|
||||
QueryLog.deflected,
|
||||
QueryLog.sources,
|
||||
).where(QueryLog.question == question)
|
||||
).all()
|
||||
assert len(rows) == 1, f"expected one query_log row for {question!r}, got {len(rows)}"
|
||||
row = rows[0]
|
||||
return {
|
||||
"top_score": float(row.top_score),
|
||||
"fts_hits": row.fts_hits,
|
||||
"chunk_hits": int(row.chunk_hits),
|
||||
"deflected": bool(row.deflected),
|
||||
"sources": str(row.sources),
|
||||
}
|
||||
|
||||
|
||||
def _recorded(row: dict[str, Any]) -> list[str]:
|
||||
"""``query_log.sources`` as the ordered ``source/path`` list the server
|
||||
wrote: the retrieval tier that reached the model (LOCKED A3)."""
|
||||
return [part.strip() for part in row["sources"].split(",") if part.strip()]
|
||||
|
||||
|
||||
def _arm(page: Page, question: str, *, expect_index: bool) -> dict[str, Any]:
|
||||
"""Ask *question* once and collect everything that must not move between
|
||||
the arms — the DOM, the wire, the durable row, and the DDL state the arm
|
||||
ACTUALLY ran under."""
|
||||
with SessionLocal() as db:
|
||||
state = _index_state(db)
|
||||
db.rollback()
|
||||
if expect_index:
|
||||
assert state is not None and state["valid"], (
|
||||
f"the ON arm ran WITHOUT a valid {INDEX_NAME}: {state}"
|
||||
)
|
||||
else:
|
||||
assert state is None, f"the OFF arm still had {INDEX_NAME}: {state}"
|
||||
|
||||
since = _submit(page, question)
|
||||
_wait_settled(page)
|
||||
done = _done(_frames(page, since))
|
||||
|
||||
brain = page.locator(".msg.brain").last
|
||||
row = _row(question)
|
||||
recorded = _recorded(row)
|
||||
return {
|
||||
"answer": brain.locator(".bubble").first.inner_text(),
|
||||
"nearby": brain.locator(".related-docs .related-doc").all_inner_texts(),
|
||||
"chips": brain.locator(".source-chip").count(),
|
||||
"deflected_bubbles": page.locator(".msg.brain.is-deflected").count(),
|
||||
"done": done,
|
||||
"index_state": state,
|
||||
"row_deflected": row["deflected"],
|
||||
"top_score": float(row["top_score"]),
|
||||
"fts_hits": row["fts_hits"],
|
||||
"chunk_hits": row["chunk_hits"],
|
||||
"recorded": recorded,
|
||||
"suggested": recorded[:SUGGESTED_DOCS],
|
||||
"nearby_recorded": recorded[SUGGESTED_DOCS:],
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1 — the answered sentinel: the SAME document, index ON and index OFF
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_answered_question_cites_the_same_document_indexed_or_not(
|
||||
page: Page, app_url: str
|
||||
) -> None:
|
||||
"""The A7/A8 contract in the UI: an index is an ACCESS PATH, so the same
|
||||
question must reach the same document whether or not the graph exists."""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
login(page, app_url, next="/") # phase 79: the chat is login-gated
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
# --- ON: the shipped state (0029 applied, index valid) ------------------
|
||||
on = _arm(page, ANSWER_QUESTION, expect_index=True)
|
||||
assert on["row_deflected"] is False
|
||||
assert on["deflected_bubbles"] == 0
|
||||
assert MOCK_ANSWER_MARKER in on["answer"]
|
||||
# The measured margin, not just the direction.
|
||||
assert on["top_score"] >= GATE_THRESHOLD
|
||||
assert on["recorded"][0] == CITED_DOC, on["recorded"]
|
||||
assert on["chunk_hits"] > 0
|
||||
# Phase 119 (LOCKED A1): a zero-read grounded turn chips nothing — the
|
||||
# index changes nothing about that either.
|
||||
assert on["chips"] == 0
|
||||
# The tier that reached the model, and the tier the page rendered.
|
||||
assert CITED_DOC in on["suggested"]
|
||||
assert on["nearby"], "the fixture KB must overflow into the rendered row"
|
||||
assert on["nearby"] == on["nearby_recorded"]
|
||||
assert on["done"]["sources"] == []
|
||||
assert [f"{s['source']}/{s['path']}" for s in on["done"]["related"]] == (
|
||||
on["nearby_recorded"]
|
||||
)
|
||||
|
||||
# --- OFF: the real 0029 downgrade, restored in `finally` ----------------
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
_new_chat(page)
|
||||
_set_index_present(False)
|
||||
try:
|
||||
off = _arm(page, ANSWER_QUESTION, expect_index=False)
|
||||
finally:
|
||||
_set_index_present(True)
|
||||
|
||||
# Same verdict, same document, same numbers: the DDL moved, the answer
|
||||
# did not.
|
||||
assert off["row_deflected"] is False
|
||||
assert off["deflected_bubbles"] == 0
|
||||
assert MOCK_ANSWER_MARKER in off["answer"]
|
||||
assert off["recorded"] == on["recorded"], (
|
||||
f"retrieval order moved with the index:\n ON={on['recorded']}\n"
|
||||
f" OFF={off['recorded']}"
|
||||
)
|
||||
assert off["recorded"][0] == CITED_DOC
|
||||
assert off["suggested"] == on["suggested"]
|
||||
assert off["nearby"] == on["nearby"]
|
||||
assert abs(off["top_score"] - on["top_score"]) <= SCORE_SLACK, (
|
||||
f"top_score moved with the index: {on['top_score']} vs "
|
||||
f"{off['top_score']}"
|
||||
)
|
||||
assert off["chunk_hits"] == on["chunk_hits"]
|
||||
assert off["fts_hits"] == on["fts_hits"]
|
||||
assert off["answer"] == on["answer"]
|
||||
for key in ("deflected", "sources", "related", "suggestions"):
|
||||
assert off["done"].get(key) == on["done"].get(key), (
|
||||
f"the done frame's {key} moved with the index: "
|
||||
f"{on['done'].get(key)!r} vs {off['done'].get(key)!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2 — the index exists and is VALID after a real import
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_index_exists_and_is_valid_after_import(kb: ImportSummary) -> None:
|
||||
"""The import that just ran left a USABLE graph, not a silent INVALID one.
|
||||
|
||||
``0029`` post-validates its own build because a ``CONCURRENTLY`` build that
|
||||
dies leaves the index INVALID — costing insert time on every sync and
|
||||
serving nothing, with no error anywhere. The same check belongs after a
|
||||
real import: chunks go into the graph on every sync, and an index that
|
||||
drifted INVALID is invisible to every other surface the app has.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
state = _index_state(db)
|
||||
counts = db.execute(
|
||||
text("SELECT count(*) AS n, count(embedding) AS with_vec FROM chunks")
|
||||
).one()
|
||||
stored = db.scalar(select(func.count()).select_from(QueryLog))
|
||||
db.rollback()
|
||||
assert stored == 0, "this pin reads the catalog only — no turn has run"
|
||||
|
||||
assert state is not None, f"{INDEX_NAME} is MISSING after the import"
|
||||
assert state["valid"] is True, (
|
||||
f"{INDEX_NAME} exists but is INVALID after the import — every sync now "
|
||||
f"pays for a graph that answers nothing: {state['indexdef']}"
|
||||
)
|
||||
definition = state["indexdef"]
|
||||
assert INDEX_NAME in definition, definition
|
||||
assert "hnsw" in definition, definition
|
||||
assert "vector_cosine_ops" in definition, definition
|
||||
assert "embedding" in definition, definition
|
||||
|
||||
# Not an empty shell over an empty table: the import put real embeddings
|
||||
# in, and the graph carries pages for them.
|
||||
assert int(counts.n) > 0, "the fixture import produced no chunks"
|
||||
assert int(counts.with_vec) == int(counts.n), (
|
||||
f"chunks without an embedding would be invisible to the tier: {counts}"
|
||||
)
|
||||
# Content chunks + one summary chunk per document (phase 30/118) — the
|
||||
# importer's own two counts, so a row that never reached the graph shows
|
||||
# up here instead of as an invisible hole.
|
||||
assert int(counts.n) == kb.chunks + kb.summaries, (
|
||||
f"the graph holds {counts.n} rows, the import reported "
|
||||
f"{kb.chunks} content + {kb.summaries} summary chunks"
|
||||
)
|
||||
assert state["size_bytes"] > 0, f"{INDEX_NAME} carries no pages: {state}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3 — the near-miss question still deflects (both arms)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_near_miss_question_still_deflects(page: Page, app_url: str) -> None:
|
||||
"""A8 in the browser, index ON and OFF: weak cosine AND zero lexical hits
|
||||
still deflects, and an index never manufactures a high ``top_score``."""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
login(page, app_url, next="/")
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
on = _arm(page, DEFLECT_QUESTION, expect_index=True)
|
||||
assert on["row_deflected"] is True
|
||||
assert on["deflected_bubbles"] == 1
|
||||
# The gate's two inputs, in the units A8 defines them.
|
||||
assert 0.0 < on["top_score"] < GATE_THRESHOLD
|
||||
assert on["fts_hits"] == 0, "deflection is only reached with zero hits"
|
||||
# The owner-visible shape of the deflection (phase 52/118).
|
||||
expect(page.locator(".msg.brain.is-deflected .bubble")).to_contain_text(
|
||||
DEFLECT_PHRASE, timeout=SETTLE_TIMEOUT_MS
|
||||
)
|
||||
chips = page.locator(".msg.brain.is-deflected .maybe-try .suggestion-chip")
|
||||
expect(chips.first).to_be_visible(timeout=SETTLE_TIMEOUT_MS)
|
||||
assert chips.count() >= 2, "a deflection must offer 2-3 alternatives"
|
||||
assert all(text.strip() for text in chips.all_inner_texts())
|
||||
assert on["done"]["deflected"] is True
|
||||
assert on["done"]["sources"] == [], "a deflected answer cites nothing"
|
||||
|
||||
# OFF arm: the same verdict, for the same measured reason.
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
_new_chat(page)
|
||||
_set_index_present(False)
|
||||
try:
|
||||
off = _arm(page, DEFLECT_QUESTION, expect_index=False)
|
||||
finally:
|
||||
_set_index_present(True)
|
||||
|
||||
assert off["row_deflected"] is True
|
||||
assert off["deflected_bubbles"] == 1
|
||||
assert 0.0 < off["top_score"] < GATE_THRESHOLD
|
||||
assert off["fts_hits"] == 0
|
||||
assert abs(off["top_score"] - on["top_score"]) <= SCORE_SLACK
|
||||
assert off["recorded"] == on["recorded"]
|
||||
assert MOCK_ANSWER_MARKER not in off["answer"]
|
||||
assert off["done"]["deflected"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4 — the ef_search knob on a real turn, and the clamp it cannot bypass
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _log_lines(log_path: Path, pattern: re.Pattern[str]) -> list[str]:
|
||||
"""Every clamp line the app logged — read off the operator's surface."""
|
||||
return [
|
||||
line
|
||||
for line in log_path.read_text(encoding="utf-8", errors="replace").splitlines()
|
||||
if pattern.search(line)
|
||||
]
|
||||
|
||||
|
||||
def test_ef_search_setting_is_honoured(
|
||||
page: Page, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> None:
|
||||
"""``BOR_RETRIEVAL_EF_SEARCH`` reaches the turn, and a value BELOW the A7
|
||||
candidate count is neither silent nor harmful."""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
|
||||
# --- 150 (>= the 100-candidate tier): a normal turn, no clamp line -----
|
||||
ok_dir = tmp_path_factory.mktemp("bor_ef_ok")
|
||||
with _running_app(
|
||||
EF_OK_URL, EF_OK_PORT, _app_env(ok_dir, "150"), ok_dir / "ef150.log"
|
||||
) as (ok_url, ok_log):
|
||||
assert ok_log is not None
|
||||
login(page, ok_url, next="/")
|
||||
page.evaluate(SSE_HOOK)
|
||||
on_ok = _arm(page, ANSWER_QUESTION, expect_index=True)
|
||||
assert on_ok["row_deflected"] is False
|
||||
assert MOCK_ANSWER_MARKER in on_ok["answer"]
|
||||
assert on_ok["recorded"][0] == CITED_DOC
|
||||
assert _log_lines(ok_log, CLAMP_LINE) == [], (
|
||||
"ef_search=150 sits ABOVE the A7 candidate count and must not "
|
||||
f"clamp: {_log_lines(ok_log, CLAMP_LINE)}"
|
||||
)
|
||||
|
||||
# --- 50 (< the tier): the clamp fires and the turn is UNCHANGED --------
|
||||
clamp_dir = tmp_path_factory.mktemp("bor_ef_clamp")
|
||||
with _running_app(
|
||||
EF_CLAMP_URL, EF_CLAMP_PORT, _app_env(clamp_dir, "50"), clamp_dir / "ef50.log"
|
||||
) as (clamp_url, clamp_log):
|
||||
assert clamp_log is not None
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
login(page, clamp_url, next="/")
|
||||
page.evaluate(SSE_HOOK)
|
||||
clamped = _arm(page, ANSWER_QUESTION, expect_index=True)
|
||||
assert clamped["row_deflected"] is False
|
||||
assert MOCK_ANSWER_MARKER in clamped["answer"]
|
||||
assert clamped["recorded"][0] == CITED_DOC
|
||||
assert clamped["recorded"] == on_ok["recorded"]
|
||||
assert abs(clamped["top_score"] - on_ok["top_score"]) <= SCORE_SLACK
|
||||
|
||||
lines = _log_lines(clamp_log, CLAMP_LINE)
|
||||
assert lines, (
|
||||
"ef_search=50 is BELOW the A7 candidate count and MUST log the "
|
||||
"clamp (AGENTS rule 10 — a knob that would degrade recall is never "
|
||||
"silent). Tail of the captured log:\n"
|
||||
f"{clamp_log.read_text(encoding='utf-8', errors='replace')[-3000:]}"
|
||||
)
|
||||
for line in lines:
|
||||
match = CLAMP_LINE.search(line)
|
||||
assert match is not None, line
|
||||
assert int(match.group(1)) == 50, line
|
||||
assert int(match.group(2)) == A7_CANDIDATES, line
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Shop vacuum
|
||||
|
||||
Dust collection layout for the garage bench and the tools it serves.
|
||||
|
||||
- The cyclone bin empties at the driveway door, twice a week.
|
||||
- Replace the final filter each season or the airflow drops noticeably.
|
||||
- The hose is the flexible 4 inch line, never the rigid conduit.
|
||||
- The saw runs on its own circuit so the motor start does not brown out
|
||||
the bench outlets.
|
||||
|
||||
Airflow is measured at the hood with a cheap gauge; anything under the
|
||||
marker means the cyclone needs emptying before anything else is checked.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Coffee grinder notes
|
||||
|
||||
Burr alignment and dose calibration for the counter grinder. Nothing here
|
||||
applies to the commercial machine at the shop.
|
||||
|
||||
- Grind finer when the shot runs short and sour.
|
||||
- Clean the chute weekly; static holds grounds in the bend.
|
||||
- Recalibrate the dose after any burr change with a blind basket test.
|
||||
- Replace the basket screen when the base of the puck turns muddy.
|
||||
|
||||
The dial clicks are not linear near the fine end, so two clicks of
|
||||
adjustment there behave like five clicks in the middle of the range.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Garden beds
|
||||
|
||||
Crop rotation for the four raised beds behind the shed, following the
|
||||
legume, brassica, root, fruit order so the beds rest every fourth year.
|
||||
|
||||
- Bed one: legumes after the last frost date clears the soil.
|
||||
- Bed two: heavy feeders get a top dressing of compost in spring.
|
||||
- Bed three: roots only, no manure or the carrots fork.
|
||||
- Bed four: fruiting crops staked and watered at the base.
|
||||
|
||||
The irrigation line is one drip tape per bed on a fifteen minute timer,
|
||||
and the mulch goes back on after every harvest rather than once a season.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Kubernetes cluster
|
||||
|
||||
The cluster is three nodes with a control plane on `k8s-master-1` and two
|
||||
workers. It runs the shelf services that want scheduling rather than a
|
||||
static container.
|
||||
|
||||
- The ingress controller listens on 8443 and forwards to cluster services.
|
||||
- etcd snapshots run nightly to the backup host and are kept for a month.
|
||||
- Worker nodes join with a short-lived token minted for one hour.
|
||||
- Storage is a local-path provisioner pinned to the nvme mount.
|
||||
- Pod security labels are set per namespace, not cluster wide.
|
||||
|
||||
Upgrades are one control plane replica at a time, then the workers with a
|
||||
cordon and drain. The ingress is the last thing to move because the
|
||||
services behind it are checked by the shelf watchdog.
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Proxmox nodes
|
||||
|
||||
Three Proxmox hosts form the shelf cluster: node one is the hypervisor and
|
||||
gateway, node two runs the always-on containers, node three is the GPU box.
|
||||
|
||||
- Storage is an LVM-thin pool on the nvme, over-commit watched weekly.
|
||||
- Live migration uses a dedicated bridge, not the management iface.
|
||||
- The watchdog restarts a container that misses three health checks.
|
||||
- Corosync traffic rides the management bridge with multicast.
|
||||
- Backups of the VM configs are taken before every host upgrade.
|
||||
|
||||
Node three is pinned for the inference workloads, so a migration off it is
|
||||
manual and only happens for a scheduled host reboot.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
# Restic backups
|
||||
|
||||
Restic backs up the media volume to the offsite bucket every night at 01:20
|
||||
with a lock timeout of five minutes.
|
||||
|
||||
- The repository password lives in the vault, never in the env file.
|
||||
- Prune keeps fourteen daily and eight weekly snapshots.
|
||||
- Verification runs one read pass every Sunday, sampled rather than full.
|
||||
- A failed lock means the previous run is still going; check before
|
||||
forcing anything.
|
||||
- Restores are tested quarterly against a scratch directory.
|
||||
|
||||
The media volume changes slowly, so a nightly run is usually small and the
|
||||
sampled verification is enough to catch a corrupt pack file early.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
# Ntfy relay
|
||||
|
||||
The ntfy relay runs on `notify-host-1` and fans alert payloads out to the
|
||||
phones and the desktop notifier.
|
||||
|
||||
- Topic namespaces are per host, so a host can only publish into its own.
|
||||
- Retention is twelve hours; anything older is dropped on the sweep.
|
||||
- The access log lives at `/var/log/ntfy/access.log` and rotates weekly.
|
||||
- SMTP fallback is disabled on purpose: a failed push is a signal, not
|
||||
something to paper over with an email.
|
||||
- The relay is fronted by the proxy on 443 with mutual TLS for publishers.
|
||||
|
||||
Publishing is a plain POST with the topic in the path. The relay stamps a
|
||||
monotonic sequence number so a subscriber can detect a gap after a
|
||||
reconnect, and the gap detection is what the shelf watchdog polls.
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
# Tilecache
|
||||
|
||||
Tilecache serves the vector tiles for the map boxes from `map-host-2`,
|
||||
listening on port 8077 behind the reverse proxy. Everything below is the
|
||||
shape of that cache on the shelf, not a general tile server guide.
|
||||
|
||||
## Cache budget and eviction
|
||||
|
||||
The cache budget is 40 GiB on the nvme. The eviction watermark sits at 88
|
||||
percent of that budget: once the cache crosses the watermark, tilecache
|
||||
evicts the coldest tiles until it drops back under it.
|
||||
|
||||
Raise the eviction watermark so the hot zoom levels stay pinned through a
|
||||
bulk re-render:
|
||||
|
||||
1. `tilecache config set eviction-watermark 92`
|
||||
2. `tilecache config set pin-zoom-floor 12`
|
||||
3. `systemctl restart tilecache`
|
||||
|
||||
Zoom levels 12 to 15 are the hot set for the atlas layers, and the pin
|
||||
floor keeps them resident while the watermark holds. Zoom 16 and above is
|
||||
rendered on demand and is always evictable.
|
||||
|
||||
## Origin fetches
|
||||
|
||||
Origin fetches go to `origin.maps.internal`, and a slow origin reply is
|
||||
treated as a miss after 900 ms. A stale origin reply keeps the previous
|
||||
tile pinned rather than evicting the hot zoom set, so a watermark raise is
|
||||
safe during a bulk re-render.
|
||||
|
||||
## Watermark maths
|
||||
|
||||
At a 40 GiB budget the 88 percent watermark means eviction starts at 35.2
|
||||
GiB. The 92 percent setting moves that to 36.8 GiB, which buys roughly one
|
||||
extra render pass of zoom 14 before anything cold is dropped.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
# Wireguard hub
|
||||
|
||||
The Wireguard hub on `edge-host-1` is the only way into the shelf from a
|
||||
laptop. Each device gets one peer entry and one allowed-IPs line.
|
||||
|
||||
- Listen port 51820/udp, one peer per device.
|
||||
- Allowed IPs are per-device /32s; the hub does not advertise the LAN.
|
||||
- Persistent keepalive is 25 seconds for the roaming peers.
|
||||
- Key rotation is a manual ceremony: generate, swap the peer file, reload.
|
||||
- The hub hands out addresses from 10.66.0.0/24, one /32 per peer.
|
||||
|
||||
A peer that roams between the mobile network and wifi must keep the same
|
||||
endpoint or the hub drops the stale mapping after the keepalive window.
|
||||
The routing table is re-read on a reload, so a swapped peer file needs no
|
||||
reboot of the host.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Sourdough timetable
|
||||
|
||||
Bulk fermentation schedule for the weekend bakes at the kitchen bench.
|
||||
|
||||
- Stretch and fold every forty minutes for the first two hours.
|
||||
- Bulk ends when the dough has grown by half and jiggles, not by clock.
|
||||
- Cold retard overnight in the fridge at four degrees.
|
||||
- Bake from cold with steam for the first twenty minutes.
|
||||
|
||||
The starter is fed one to five in the morning so it peaks mid afternoon,
|
||||
which is when the mix actually happens.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Packing list
|
||||
|
||||
Carry-on checklist for a week of trips with one bag and a laptop.
|
||||
|
||||
- One shirt per two days, always folded not rolled.
|
||||
- Spare battery in the personal item, never the checked bag.
|
||||
- Toiletries under the hundred millilitre rule in one clear pouch.
|
||||
- A single pair of shoes that survives both rain and a dinner.
|
||||
|
||||
The laptop bag holds the cables, the drive and the paper documents, and it
|
||||
is the thing that gets opened at every security checkpoint, so it stays
|
||||
boring and easy to repack.
|
||||
@@ -0,0 +1,354 @@
|
||||
"""Integration: migration 0029 (chunks.embedding HNSW index) — phase 188, task 01.
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the house pattern of
|
||||
``test_migration_0028.py`` (fixture upgrades to head at setup and
|
||||
teardown so the dev DB is never left below head; tests target the
|
||||
0028 → 0029 step explicitly so later migrations cannot break them):
|
||||
|
||||
* upgrade 0028 → 0029 → ``ix_chunks_embedding_hnsw`` exists,
|
||||
``pg_index.indisvalid`` is TRUE, and ``pg_get_indexdef`` names
|
||||
``USING hnsw`` + ``vector_cosine_ops`` + ``m=16`` + ``ef_construction=64``
|
||||
(the D1 shape the recall pins of task 03 depend on);
|
||||
* downgrade to 0028 → the index is GONE (A13 — reversible, ``CONCURRENTLY``),
|
||||
the ``chunks`` table and its PK survive;
|
||||
* upgrade back to head → the index is back and valid (round-trip);
|
||||
* the D2 self-heal: with an INVALID index of that name deliberately
|
||||
present, ``upgrade()`` drops it and rebuilds it VALID — no
|
||||
``relation already exists`` error and no leftover INVALID index.
|
||||
The INVALID state is forced with the documented superuser catalog
|
||||
trick ``UPDATE pg_index SET indisvalid = false WHERE
|
||||
indexrelid = CAST(:name AS regclass)`` — the same failure a killed
|
||||
``CREATE INDEX CONCURRENTLY`` leaves behind (pgvector cannot fail a
|
||||
build on data, and the dev/test DB role is a superuser; a role
|
||||
without catalog UPDATE rights gets a clean skip).
|
||||
|
||||
Pure-helper matrix tests (no DB): ``should_drop_invalid`` and
|
||||
``is_valid`` over ``None`` / ``True`` / ``False``.
|
||||
|
||||
**Why every ``command.*`` call here is preceded by ``db.rollback()``
|
||||
when the session was used.** psycopg3 keeps its transaction snapshot
|
||||
alive while the session sits idle-in-transaction (measured: ``pg_stat
|
||||
_activity.backend_xmin`` stays set where psql's clears), and ``CREATE
|
||||
INDEX CONCURRENTLY`` phase 3 waits for every older-snapshot
|
||||
transaction to finish — an unreleased session parks the build FOREVER
|
||||
(observed on the dev box). The test session must therefore hold no
|
||||
open transaction while the concurrent DDL runs; the rollback is what
|
||||
makes the file deterministic rather than flaky.
|
||||
**Task 02 adds the ``ef_search`` half:** the knob the index needs is
|
||||
applied with ``SET LOCAL`` inside the turn's own transaction, so the file
|
||||
pins the SCOPING against the real server — after a ``SET LOCAL`` the same
|
||||
transaction reports the (clamped) value and a NEW transaction reports the
|
||||
server's own default again (LOCKED A4: no global GUC, nothing surviving
|
||||
the request or a pool recycle).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import logging
|
||||
import types
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import ProgrammingError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.config import Settings
|
||||
from app.db import db_available
|
||||
from app.rag import retriever
|
||||
from app.rag.retriever import _vector_candidates
|
||||
|
||||
MIGRATION_PATH = Path("alembic/versions/0029_chunks_embedding_hnsw.py")
|
||||
INDEX_NAME = "ix_chunks_embedding_hnsw"
|
||||
|
||||
|
||||
def _load_migration() -> types.ModuleType:
|
||||
"""Import the 0029 module by file path (``alembic/versions`` is not a
|
||||
package — the house way to reach a migration's own helpers)."""
|
||||
spec = importlib.util.spec_from_file_location("m0029_chunks_embedding_hnsw", MIGRATION_PATH)
|
||||
assert spec is not None and spec.loader is not None, f"missing {MIGRATION_PATH}"
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run — 0029's own
|
||||
INVALID self-heal makes that repair safe when the interrupted
|
||||
migration was 0029 itself); teardown upgrades to head no matter
|
||||
what happened, so the dev DB is never left below head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL (the 0018 house pattern): a still-running snapshot would
|
||||
# park the CONCURRENTLY build, and teardown is exactly where a
|
||||
# failed test leaves one open.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _index_present(db: Session) -> bool:
|
||||
return (
|
||||
db.execute(
|
||||
text("SELECT 1 FROM pg_indexes WHERE indexname = :n"), {"n": INDEX_NAME}
|
||||
).scalar()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _index_valid(db: Session) -> bool | None:
|
||||
"""``pg_index.indisvalid`` for the index, ``None`` when it is absent."""
|
||||
return db.execute(
|
||||
text(
|
||||
"SELECT i.indisvalid FROM pg_index i"
|
||||
" JOIN pg_class c ON c.oid = i.indexrelid"
|
||||
" WHERE c.relname = :n"
|
||||
),
|
||||
{"n": INDEX_NAME},
|
||||
).scalar()
|
||||
|
||||
|
||||
def _index_def(db: Session) -> str | None:
|
||||
return db.execute(
|
||||
text("SELECT pg_get_indexdef(c.oid) FROM pg_class c WHERE c.relname = :n"),
|
||||
{"n": INDEX_NAME},
|
||||
).scalar()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure helpers (no DB, no I/O)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_should_drop_invalid_matrix() -> None:
|
||||
"""``should_drop_invalid``: absent → False, valid → False,
|
||||
INVALID → True (the only state the pre-drop fires for)."""
|
||||
mod = _load_migration()
|
||||
assert mod.should_drop_invalid(None) is False, "absent index needs no drop"
|
||||
assert mod.should_drop_invalid(True) is False, "a valid index must be left alone"
|
||||
assert mod.should_drop_invalid(False) is True, "an INVALID remnant must be dropped"
|
||||
|
||||
|
||||
def test_is_valid_matrix() -> None:
|
||||
"""``is_valid``: only a present, validated index counts as valid."""
|
||||
mod = _load_migration()
|
||||
assert mod.is_valid(None) is False, "absent is not valid"
|
||||
assert mod.is_valid(True) is True, "validated is valid"
|
||||
assert mod.is_valid(False) is False, "INVALID is not valid"
|
||||
|
||||
|
||||
def test_migration_module_contract() -> None:
|
||||
"""The module's identity: revision 0029 on top of 0028, and the one
|
||||
index name the whole phase pins on."""
|
||||
mod = _load_migration()
|
||||
assert mod.revision == "0029"
|
||||
assert mod.down_revision == "0028"
|
||||
assert mod.INDEX_NAME == INDEX_NAME
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Migration round-trip against the real Alembic engine
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_upgrade_creates_a_valid_hnsw_index(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0028 → 0029: the index exists, VALIDATED, with the D1
|
||||
shape — HNSW, ``vector_cosine_ops`` (the operator class A7's ``<=>``
|
||||
needs), ``m=16``, ``ef_construction=64`` — and the ``chunks`` table
|
||||
itself survives the additive step untouched."""
|
||||
command.downgrade(alembic, "0028") # start from the pre-0029 state
|
||||
assert _version(db) == "0028"
|
||||
assert not _index_present(db), f"{INDEX_NAME} must be absent at 0028"
|
||||
|
||||
db.rollback() # no open snapshot while the CONCURRENTLY build runs
|
||||
command.upgrade(alembic, "0029")
|
||||
assert _version(db) == "0029", "alembic_version must be at 0029"
|
||||
assert _index_present(db), "the index must exist at 0029"
|
||||
assert _index_valid(db) is True, "a fresh build must be VALID (D2 post-check)"
|
||||
|
||||
raw = _index_def(db)
|
||||
assert raw is not None
|
||||
definition = raw.replace("'", "") # pg renders WITH options quoted
|
||||
assert "USING hnsw" in definition, f"not an HNSW index: {definition}"
|
||||
assert "vector_cosine_ops" in definition, (
|
||||
f"cosine operator class required for <=>: {definition}"
|
||||
)
|
||||
assert "m=16" in definition, f"m=16 expected: {definition}"
|
||||
assert "ef_construction=64" in definition, f"ef_construction=64 expected: {definition}"
|
||||
|
||||
|
||||
def test_downgrade_removes_it(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade 0029 → 0028: the index is gone (A13 — reversible), and
|
||||
the ``chunks`` relation with its PK survives (the drop touches no
|
||||
data)."""
|
||||
command.upgrade(alembic, "0029")
|
||||
assert _index_present(db)
|
||||
|
||||
db.rollback() # no open snapshot while the CONCURRENTLY drop runs
|
||||
command.downgrade(alembic, "0028")
|
||||
assert _version(db) == "0028"
|
||||
assert not _index_present(db), f"{INDEX_NAME} must be dropped by the downgrade"
|
||||
assert _index_valid(db) is None, "no pg_index row may survive the downgrade"
|
||||
|
||||
pk = db.execute(
|
||||
text("SELECT 1 FROM pg_indexes WHERE indexname = 'chunks_pkey'")
|
||||
).scalar()
|
||||
assert pk is not None, "chunks_pkey must survive the downgrade"
|
||||
|
||||
|
||||
def test_upgrade_head_again(db: Session, alembic: Config) -> None:
|
||||
"""A13 round-trip: downgrade to 0028, upgrade back to HEAD — the
|
||||
index returns, valid, with the same definition."""
|
||||
command.downgrade(alembic, "0028")
|
||||
command.upgrade(alembic, "head")
|
||||
assert _index_present(db), "the index must be back after re-upgrade"
|
||||
assert _index_valid(db) is True, "the rebuilt index must be VALID"
|
||||
raw = _index_def(db)
|
||||
assert raw is not None and "USING hnsw" in raw
|
||||
|
||||
|
||||
def test_invalid_index_is_dropped_and_rebuilt(db: Session, alembic: Config) -> None:
|
||||
"""D2 self-heal: an INVALID index of this name (what a killed
|
||||
``CREATE INDEX CONCURRENTLY`` leaves behind) is dropped and rebuilt
|
||||
VALID on the next ``upgrade()``.
|
||||
|
||||
The INVALID state is forced with the documented superuser catalog
|
||||
update ``UPDATE pg_index SET indisvalid = false WHERE indexrelid =
|
||||
CAST(:name AS regclass)`` — pgvector cannot fail an HNSW build on
|
||||
data, and timing a real interruption is nondeterministic; the dev/
|
||||
test DB role is a superuser, and a role without catalog UPDATE
|
||||
rights skips cleanly instead of leaving the DB inconsistent (the
|
||||
stamp that follows the forced state, never before it)."""
|
||||
command.upgrade(alembic, "head")
|
||||
assert _index_valid(db) is True, "precondition: a valid index to corrupt"
|
||||
|
||||
try:
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE pg_index SET indisvalid = false"
|
||||
" WHERE indexrelid = CAST(:n AS regclass)"
|
||||
),
|
||||
{"n": INDEX_NAME},
|
||||
)
|
||||
db.commit()
|
||||
except ProgrammingError as exc: # non-superuser role
|
||||
db.rollback()
|
||||
pytest.skip(f"forcing INVALID needs catalog UPDATE rights: {exc}")
|
||||
|
||||
# The stamp moves the version table WITHOUT touching the index, so
|
||||
# the next upgrade() re-runs 0029 against the corrupted state.
|
||||
command.stamp(alembic, "0028")
|
||||
assert _version(db) == "0028"
|
||||
assert _index_valid(db) is False, "the forced INVALID state must be visible"
|
||||
|
||||
db.rollback() # no open snapshot of ours while the CONCURRENTLY run rebuilds
|
||||
command.upgrade(alembic, "0029")
|
||||
assert _index_present(db), "no 'relation already exists' — it was pre-dropped"
|
||||
assert _index_valid(db) is True, "the rebuilt index must be VALID (no leftover INVALID)"
|
||||
raw = _index_def(db)
|
||||
assert raw is not None and "USING hnsw" in raw, "the rebuild is the HNSW index itself"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Task 02: SET LOCAL hnsw.ef_search — applied, clamped, transaction-scoped
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _show_ef_search(db: Session) -> str:
|
||||
return str(db.execute(text("SHOW hnsw.ef_search")).scalar())
|
||||
|
||||
|
||||
def _server_default_ef_search(db: Session) -> str:
|
||||
"""The server's own reset value for the GUC (the value a NEW
|
||||
transaction must fall back to). Read from ``pg_settings`` rather than
|
||||
hardcoded: pgvector renders its default, and the pin is about the
|
||||
FALLBACK, not the number."""
|
||||
value = db.execute(
|
||||
text("SELECT reset_val FROM pg_settings WHERE name = 'hnsw.ef_search'")
|
||||
).scalar()
|
||||
assert value is not None, "pgvector must register hnsw.ef_search in this session"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _stub_settings(**kwargs: object) -> Settings:
|
||||
return Settings(_env_file=None, **kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def test_set_local_ef_search_reaches_the_server_in_the_same_transaction(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The D3 rule against a real server: the value the tier sends is the
|
||||
value Postgres runs the search with. The setting here (10) sits BELOW
|
||||
the limit (250), so the server must report the CLAMPED value — the
|
||||
clamp is not a Python-side fiction."""
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=10)
|
||||
)
|
||||
db.rollback() # a clean transaction boundary for the SET LOCAL
|
||||
_vector_candidates(db, [0.0] * 768, limit=250) # pyright: ignore[reportArgumentType]
|
||||
assert _show_ef_search(db) == "250", "the clamped value must be live in this tx"
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_set_local_ef_search_does_not_survive_the_transaction(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""LOCKED A4 pinned on the wire: once the turn's transaction ends, the
|
||||
server is back on ITS default — no global GUC mutation, no state a
|
||||
pooled connection carries into the next request."""
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=317)
|
||||
)
|
||||
db.rollback()
|
||||
_vector_candidates(db, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
assert _show_ef_search(db) == "317"
|
||||
|
||||
db.rollback() # the end of the turn's transaction
|
||||
after = _show_ef_search(db)
|
||||
default = _server_default_ef_search(db)
|
||||
assert after == default, f"a new tx must see the server default, got {after}"
|
||||
assert after != "317"
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_ef_search_clamp_logs_one_warning_line(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The clamp leaves exactly one operator-visible line in the server log
|
||||
(AGENTS rule 10), and the pass-through case leaves none."""
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=5)
|
||||
)
|
||||
db.rollback()
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.retriever"):
|
||||
_vector_candidates(db, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
db.rollback()
|
||||
lines = [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()]
|
||||
assert lines == ["retrieval: ef_search=5 raised to 100 (A7 candidate count)"]
|
||||
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=100)
|
||||
)
|
||||
caplog.clear()
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.retriever"):
|
||||
_vector_candidates(db, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
db.rollback()
|
||||
assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
|
||||
@@ -0,0 +1,612 @@
|
||||
"""Integration: the cosine tier USES ``ix_chunks_embedding_hnsw`` and the
|
||||
top-100 did not move (phase 188, task 03 — the deterministic half of D4).
|
||||
|
||||
**The production numbers this pin stands in for** (read 2026-10-15, read-only):
|
||||
414,220 chunks — all with a non-null ``embedding`` — a ``chunks`` relation of
|
||||
≈ **3,000 MB** (database 3,147 MB), ``shared_buffers=128MB``, ``work_mem=4MB``,
|
||||
a 3,904 MB host with **no swap**, pgvector **0.8.0**, and ``pg_indexes`` for
|
||||
``chunks`` listing only ``chunks_pkey``, ``ix_chunks_document_id`` and
|
||||
``ix_chunks_tsv`` (GIN): **no index on ``embedding``**. Every chat turn's A7
|
||||
cosine tier (``ORDER BY embedding <=> :vec LIMIT 100``) was therefore a
|
||||
sequential scan of that 3 GB relation.
|
||||
|
||||
Two CI pins, both deterministic and LLM-free:
|
||||
|
||||
1. **The plan pin** — ``EXPLAIN (FORMAT JSON)`` of the tier's own SELECT (the
|
||||
statement object captured from
|
||||
:func:`app.rag.retriever._vector_candidates`, compiled with literal binds so
|
||||
the planner sees a real constant) names ``ix_chunks_embedding_hnsw`` and
|
||||
contains no ``Seq Scan on chunks``. Its vacuity guard runs the SAME
|
||||
statement with index scans refused and MUST see the seq scan, so the pin
|
||||
cannot pass on a plan the index never influenced.
|
||||
2. **The recall pin** — the index-backed top-100 against the EXACT cosine
|
||||
top-100 of the fixture corpus: ``|overlap| / 100 >= 0.99`` for EVERY query
|
||||
at the SHIPPED ``ef_search`` (400 since task 04's live battery; 100, the
|
||||
A7 candidate count, was the task-02 default and the battery retired it),
|
||||
each returned row's ``1 − distance`` equal to the exact cosine within
|
||||
**1e-6**, and the mean one step DEEPER never below the mean at the shipped
|
||||
value. Every recall run also re-reads the plan it just ran, because an
|
||||
``ef_search`` the planner declines turns the "index" pin into a second
|
||||
measurement of the sequential scan — a vacuous pass, and the one failure
|
||||
mode this file exists to catch.
|
||||
|
||||
**Measured on this corpus** (5,000 chunks, ``TOP_K = 100``, 30 fixed-seed
|
||||
queries, ``m = 16`` / ``ef_construction = 64`` from migration 0029):
|
||||
``ef_search=100`` → mean top-100 overlap **0.9993**, per-query minimum **0.99**
|
||||
(two of the thirty return 99/100), zero queries below the floor;
|
||||
``ef_search=200`` and the shipped **400** → mean **1.0000**. Worst
|
||||
``|cosine − exact cosine|`` over every
|
||||
returned row: **5.7e-07**, and 1e-6 is the tightest tolerance that can hold —
|
||||
``_vector_candidates`` publishes ``round(1 − distance, 6)``, which alone can move
|
||||
a value by 5e-7, on top of the float4 rounding of the distance itself (~6e-8 at
|
||||
this magnitude). The exact reference is computed over the vectors as the
|
||||
database returned them, so the comparison measures the index. Setup stays inside the runtime guard:
|
||||
generate ≈ 0.5 s, seed ≈ 2 s, ``CREATE INDEX CONCURRENTLY`` ≈ 1.4 s, read-back +
|
||||
exact top-100 ≈ 3.5 s — ≈ 8 s against the ~60 s target.
|
||||
|
||||
**What 5,000 rows proves and what it does not.** It is enough for a PLAN pin
|
||||
(the planner needs a populated relation to prefer an ordered index scan over a
|
||||
seq scan + top-N sort) and enough for a deterministic index-vs-exact
|
||||
regression; it is not a proxy for 414k rows, because recall is a function of the
|
||||
corpus shape and of graph density at 83× the size. The prod-scale gate is task
|
||||
04 — the live retrieval battery, index off vs on, over the 1,000-document
|
||||
replica KB, where every ``top_score``, top-5 set and A8 verdict must match.
|
||||
|
||||
**Corpus shape, and why it is not uniform noise.** One
|
||||
``random.Random(20261015)`` stream in a fixed order: a 16-direction principal
|
||||
subspace (one random unit direction per disjoint 48-dim block of the 768 —
|
||||
disjoint blocks are orthonormal by construction, so no Gram–Schmidt and no
|
||||
numerical drift), 40 unit centroids inside it, 5,000 chunks of
|
||||
``normalise(centroid + 2.0 × a random subspace direction)`` (125 per cluster),
|
||||
then 30 queries drawn uniformly from the same subspace. Low EFFECTIVE rank (16
|
||||
of the 768 stored dimensions carry the structure; every dimension is still
|
||||
non-zero) is deliberate: it is how trained text embeddings behave (their
|
||||
spectrum concentrates far below the nominal dimension), and it is the shape
|
||||
under which ``ef_search == LIMIT`` can be expected to return the exact top-100.
|
||||
|
||||
The counter-case is recorded here because measuring it is a finding of this
|
||||
task: the same 5,000 rows with the cluster bias thinned to a whisper (centroid
|
||||
weight 0.1 over per-dimension Gaussian noise across all 768 dimensions —
|
||||
effectively iid, the hardest shape a graph index can be handed, and the one
|
||||
this file's sweep ran at centroid weights 0.1, 0.3 and 0.5 alike) measures mean
|
||||
recall@100 ≈ **0.63** at ``ef_search=100``, ≈ **0.85** at 200 and **1.00** from
|
||||
400 up, whatever ``m`` / ``ef_construction`` migration 0029 ships. ``ef_search``
|
||||
at the candidate count is therefore NOT a free win on high-effective-rank data;
|
||||
if the live battery moves a ``top_score`` or a top-5 set, LOCKED A3's answer is
|
||||
to raise ``BOR_RETRIEVAL_EF_SEARCH`` (towards 400), never to relax an assertion
|
||||
here.
|
||||
|
||||
The index is built by migration **0029** and never by an inline ``CREATE
|
||||
INDEX`` — the migration is the single definition of the index these pins name
|
||||
(``0029 → 0028`` → seed unindexed → ``upgrade 0029``). As in
|
||||
``test_chunks_hnsw_index.py``, the seeding session holds NO open transaction
|
||||
while the ``CONCURRENTLY`` DDL runs (an idle-in-transaction snapshot parks the
|
||||
build), and teardown leaves the dev DB at head, index valid, fixture rows gone.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import operator
|
||||
import random
|
||||
import struct
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import Chunk, Document
|
||||
from app.rag import retriever
|
||||
from app.rag.retriever import _vector_candidates
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The corpus (D4's deterministic half) — one fixed stream, no wall clock
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
DIM = 768 # A6: the embedding dimension is fixed at table creation
|
||||
CENTROIDS = 40
|
||||
SUBSPACE = 16 # effective rank of the corpus (see the module docstring)
|
||||
BLOCK = DIM // SUBSPACE # 48 dims per basis direction, disjoint by construction
|
||||
NOISE = 2.0 # |noise| / |centroid| — cos(chunk, own centroid) ~= 0.45
|
||||
CORPUS_CHUNKS = 5000
|
||||
CHUNKS_PER_DOCUMENT = 10
|
||||
QUERIES = 30
|
||||
SEED = 20261015 # the date of the incident that started phase 188
|
||||
FIXTURE_SOURCE = "phase-188-hnsw-recall"
|
||||
|
||||
TOP_K = 100 # A7's cosine candidate count — the LIMIT the tier promises
|
||||
EF_PIN = 400 # the shipped default (Settings.retrieval_ef_search)
|
||||
EF_RAISED = 600 # the monotonicity probe
|
||||
RECALL_FLOOR = 0.99 # LOCKED A3 — a raised ef_search is the only remedy
|
||||
COSINE_TOLERANCE = 1e-6 # "exact to 6 decimals"
|
||||
|
||||
#: The pins need a small relation: the exact reference is a Python scan over
|
||||
#: every row. A DB holding more is a restored replica rather than the dev
|
||||
#: corpus — that scale is task 04's live battery, and 30 × N dot products here
|
||||
#: would only make the suite crawl.
|
||||
MAX_RELATION_ROWS = 20_000
|
||||
|
||||
INDEX_NAME = "ix_chunks_embedding_hnsw"
|
||||
RELATION = "chunks"
|
||||
|
||||
|
||||
def _f32(value: float) -> float:
|
||||
"""The ``float4`` the database holds for a query component.
|
||||
|
||||
A question vector goes to Postgres as ``str(float)`` text and comes back as
|
||||
``float4`` (``strtof`` rounds to nearest, the same value ``struct.pack``
|
||||
produces), so the exact reference rounds the QUERY the way the server does.
|
||||
The corpus needs no modelling — its components are read back from the
|
||||
database — which is what makes the comparison measure the INDEX rather than
|
||||
a precision artefact of the fixture.
|
||||
"""
|
||||
return struct.unpack("f", struct.pack("f", value))[0]
|
||||
|
||||
|
||||
def _norm(vec: list[float]) -> float:
|
||||
return math.sqrt(sum(v * v for v in vec))
|
||||
|
||||
|
||||
def _unit(vec: list[float]) -> list[float]:
|
||||
n = _norm(vec)
|
||||
return [v / n for v in vec] if n else vec
|
||||
|
||||
|
||||
def _generate() -> tuple[list[list[float]], list[list[float]]]:
|
||||
"""Fixed-seed ``(corpus, queries)``, all L2-normalised.
|
||||
|
||||
Normalising makes cosine the dot product, which is what keeps the exact
|
||||
reference cheap enough to run in CI. One stream in a fixed order (basis →
|
||||
centroids → chunks → queries) is what turns the recall floor into a
|
||||
regression pin instead of a measurement: byte-identical input, therefore a
|
||||
byte-identical graph, therefore a byte-identical overlap.
|
||||
"""
|
||||
rng = random.Random(SEED)
|
||||
basis = [_unit([rng.gauss(0.0, 1.0) for _ in range(BLOCK)]) for _ in range(SUBSPACE)]
|
||||
|
||||
def subspace_vector() -> list[float]:
|
||||
"""A random unit vector inside the 16-direction subspace."""
|
||||
coefs = [rng.gauss(0.0, 1.0) for _ in range(SUBSPACE)]
|
||||
n = _norm(coefs) or 1.0
|
||||
out = [0.0] * DIM
|
||||
for k, direction in enumerate(basis):
|
||||
weight = coefs[k] / n
|
||||
start = k * BLOCK
|
||||
for j in range(BLOCK):
|
||||
out[start + j] = weight * direction[j]
|
||||
return out
|
||||
|
||||
centroids = [subspace_vector() for _ in range(CENTROIDS)]
|
||||
corpus: list[list[float]] = []
|
||||
for i in range(CORPUS_CHUNKS):
|
||||
centroid = centroids[i % CENTROIDS]
|
||||
noise = subspace_vector()
|
||||
corpus.append(_unit([c + NOISE * n for c, n in zip(centroid, noise, strict=True)]))
|
||||
queries = [subspace_vector() for _ in range(QUERIES)]
|
||||
return corpus, queries
|
||||
|
||||
|
||||
def _seed(session: Session, corpus: list[list[float]]) -> None:
|
||||
"""Insert the corpus — 500 fixture documents × 10 chunks, one commit.
|
||||
|
||||
Ordinary content chunks (``is_summary`` False, embedding set): the
|
||||
NULL-embedding skip is pinned in ``tests/unit/test_retriever.py``, and a
|
||||
NULL row would sit in neither the exact set nor the index.
|
||||
"""
|
||||
documents = [
|
||||
Document(
|
||||
id=uuid.uuid4(),
|
||||
source=FIXTURE_SOURCE,
|
||||
path=f"{FIXTURE_SOURCE}/{d}.md",
|
||||
full_path=f"/tmp/{FIXTURE_SOURCE}/{d}.md",
|
||||
title=f"hnsw recall fixture {d}",
|
||||
content="fixture content",
|
||||
content_hash="0" * 64,
|
||||
)
|
||||
for d in range(CORPUS_CHUNKS // CHUNKS_PER_DOCUMENT)
|
||||
]
|
||||
session.add_all(documents)
|
||||
session.flush()
|
||||
session.add_all(
|
||||
[
|
||||
Chunk(
|
||||
id=uuid.uuid4(),
|
||||
document_id=documents[i // CHUNKS_PER_DOCUMENT].id,
|
||||
position=i % CHUNKS_PER_DOCUMENT,
|
||||
content=f"fixture chunk {i}",
|
||||
embedding=vector,
|
||||
is_summary=False,
|
||||
)
|
||||
for i, vector in enumerate(corpus)
|
||||
]
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exact reference (pure Python, float4-modelled inputs)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _exact_top(stored: dict[Any, list[float]], query: list[float]) -> list[tuple[float, Any]]:
|
||||
"""The EXACT cosine top-:data:`TOP_K` of *stored* for *query*.
|
||||
|
||||
Mirrors pgvector's ``vector_cosine_distance``: three float8 accumulators
|
||||
(``dot``, ``norm_a``, ``norm_b``) over the components left to right, then
|
||||
``dot / sqrt(norm_a * norm_b)``. The norms are recomputed from the stored
|
||||
components rather than assumed to be 1, because the stored component is the
|
||||
float4 of a normalised double.
|
||||
|
||||
Pure Python (no numpy in this project — A3 keeps the dependency list
|
||||
closed): ``sum(map(mul, …))`` is ~25 µs per 768-dim dot product, so 30
|
||||
queries over 5,000 rows cost ~3 s. The one residual inaccuracy is the
|
||||
read-back itself: pgvector prints each float4 in shortest-round-trip form,
|
||||
so a component comes back within ~1 ulp of the float4 (measured max
|
||||
7.4e-09 absolute over the fixture corpus) — three orders below the 1e-6
|
||||
this file pins.
|
||||
"""
|
||||
q = [_f32(v) for v in query]
|
||||
q_norm = math.sqrt(sum(map(operator.mul, q, q)))
|
||||
scored = [
|
||||
(
|
||||
sum(map(operator.mul, vec, q)) / (math.sqrt(sum(map(operator.mul, vec, vec))) * q_norm),
|
||||
chunk_id,
|
||||
)
|
||||
for chunk_id, vec in stored.items()
|
||||
]
|
||||
scored.sort(key=lambda item: -item[0])
|
||||
return scored[:TOP_K]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Capturing the tier's own statement (the plan pin must explain THAT SQL)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _StatementRecorder:
|
||||
"""Records what :func:`_vector_candidates` sends, answering with no rows."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.statements: list[Any] = []
|
||||
|
||||
def execute(self, statement: Any, params: Any = None) -> _StatementRecorder:
|
||||
self.statements.append(statement)
|
||||
return self
|
||||
|
||||
def all(self) -> list[Any]:
|
||||
return []
|
||||
|
||||
|
||||
def _vector_statement_sql() -> str:
|
||||
"""The tier's cosine SELECT as one literal SQL string, ready for EXPLAIN.
|
||||
|
||||
A hand-written copy would drift the moment the retriever changes and would
|
||||
then pin SQL no request ever runs, so the statement OBJECT is captured.
|
||||
``literal_binds`` gives the planner a real vector constant
|
||||
(``pgvector.sqlalchemy.VECTOR`` ships a literal processor, and the resulting
|
||||
unknown string literal coerces to ``vector`` through the ``<=>`` operator's
|
||||
own signature). The ``SET LOCAL`` line is deliberately not part of it:
|
||||
``EXPLAIN`` takes one statement, and that line tunes the search depth, not
|
||||
the shape of the plan.
|
||||
"""
|
||||
recorder = _StatementRecorder()
|
||||
_vector_candidates(recorder, [0.0] * DIM, TOP_K) # pyright: ignore[reportArgumentType]
|
||||
set_local, select_statement = recorder.statements
|
||||
assert str(set_local) == f"SET LOCAL hnsw.ef_search = {EF_PIN}", (
|
||||
f"the tier's ef_search line changed ({set_local}) — the capture must be the SELECT"
|
||||
)
|
||||
sql = str(
|
||||
select_statement.compile(
|
||||
dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}
|
||||
)
|
||||
)
|
||||
assert "<=>" in sql, f"the cosine operator is gone from the tier: {sql[:200]}"
|
||||
return sql
|
||||
|
||||
|
||||
def _plan(session: Session, sql: str) -> list[dict[str, Any]]:
|
||||
"""Every node of ``EXPLAIN (FORMAT JSON)`` for *sql*, flattened."""
|
||||
document = session.execute(text(f"EXPLAIN (FORMAT JSON) {sql}")).scalar()
|
||||
assert isinstance(document, list) and document, f"no plan for {sql[:120]}: {document!r}"
|
||||
nodes: list[dict[str, Any]] = []
|
||||
|
||||
def walk(node: dict[str, Any]) -> None:
|
||||
nodes.append(node)
|
||||
for child in node.get("Plans", []):
|
||||
walk(child)
|
||||
|
||||
walk(document[0]["Plan"])
|
||||
return nodes
|
||||
|
||||
|
||||
def _describe(nodes: list[dict[str, Any]]) -> str:
|
||||
return ", ".join(
|
||||
f"{node.get('Node Type')}[{node.get('Index Name') or node.get('Relation Name') or '?'}]"
|
||||
for node in nodes
|
||||
)
|
||||
|
||||
|
||||
def _uses_index(nodes: list[dict[str, Any]], index_name: str) -> bool:
|
||||
return any(
|
||||
"Index Scan" in str(node.get("Node Type", "")) and node.get("Index Name") == index_name
|
||||
for node in nodes
|
||||
)
|
||||
|
||||
|
||||
def _seq_scans(nodes: list[dict[str, Any]], relation: str) -> bool:
|
||||
return any(
|
||||
node.get("Node Type") == "Seq Scan" and node.get("Relation Name") == relation
|
||||
for node in nodes
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture: seed unindexed, build through 0029, read the corpus back once
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def hnsw_corpus() -> Iterator[dict[str, Any]]:
|
||||
"""Seed the corpus, build the index with migration 0029, precompute exact.
|
||||
|
||||
``0029 → 0028`` → seed → ``0029``: seeding with the index present would pay
|
||||
a graph insert per row, and building after the load is what the operator
|
||||
does on the production box anyway. Teardown deletes ONLY this module's rows
|
||||
(``source = FIXTURE_SOURCE``; the chunk rows go with them by the DB-level
|
||||
cascade — AGENTS rule 12, no chunk hydration) and leaves the dev DB at head
|
||||
with the index valid, exactly as it was found.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
session = SessionLocal()
|
||||
config = Config() # no alembic.ini — env.py reads the URL from app config
|
||||
config.set_main_option("script_location", "alembic")
|
||||
try:
|
||||
preexisting = session.execute(text(f"SELECT count(*) FROM {RELATION}")).scalar() or 0
|
||||
if preexisting > MAX_RELATION_ROWS:
|
||||
pytest.skip(
|
||||
f"{RELATION} holds {preexisting} rows: the exact reference is a Python "
|
||||
"scan, so these pins need the small dev corpus. Production scale "
|
||||
"is task 04's live battery."
|
||||
)
|
||||
corpus, queries = _generate()
|
||||
session.rollback() # no open snapshot while the CONCURRENTLY DDL runs
|
||||
command.downgrade(config, "0028")
|
||||
session.rollback()
|
||||
_seed(session, corpus)
|
||||
session.rollback()
|
||||
command.upgrade(config, "0029") # the ONE definition of the index
|
||||
session.rollback()
|
||||
# Planner statistics for the plan pin: a freshly loaded relation has
|
||||
# none, and a row estimate of 0 makes "seq scan + top-N sort" look free.
|
||||
session.execute(text(f"ANALYZE {RELATION}"))
|
||||
session.commit()
|
||||
rows = session.execute(select(Chunk.id, Chunk.embedding)).all()
|
||||
stored: dict[Any, list[float]] = {row[0]: row[1] or [] for row in rows}
|
||||
assert len(stored) >= preexisting + CORPUS_CHUNKS, (
|
||||
f"expected {CORPUS_CHUNKS} fixture rows on top of {preexisting}, "
|
||||
f"the relation holds {len(stored)}"
|
||||
)
|
||||
# The exact sets are the expensive half (30 × N dot products); the two
|
||||
# recall pins differ only in the ef_search they ask the index for.
|
||||
yield {
|
||||
"queries": queries,
|
||||
"exact": [_exact_top(stored, query) for query in queries],
|
||||
"chunk_ids": set(stored),
|
||||
"plan_sql": _vector_statement_sql(),
|
||||
}
|
||||
finally:
|
||||
session.rollback()
|
||||
session.execute(
|
||||
text("DELETE FROM documents WHERE source = :src"), {"src": FIXTURE_SOURCE}
|
||||
)
|
||||
session.commit()
|
||||
session.rollback()
|
||||
command.upgrade(config, "head") # the dev DB never stays below head
|
||||
session.close()
|
||||
|
||||
|
||||
def _stub_settings(**kwargs: object) -> Settings:
|
||||
"""A :class:`Settings` off the env file (the ``test_chunks_hnsw_index.py``
|
||||
helper's shape — an operator's shell must not tune these pins)."""
|
||||
return Settings(_env_file=None, **kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _force_index_scan(session: Session) -> None:
|
||||
"""Put the planner on the index, because a recall pin has to measure the
|
||||
index and not the planner's preference.
|
||||
|
||||
Postgres costs an HNSW scan partly on ``ef_search``, so on a relation this
|
||||
small it can legitimately prefer a seq scan + top-N sort — and a recall run
|
||||
that did that would score the EXACT scan twice and report a perfect 1.0000
|
||||
without visiting a single graph node. That is not hypothetical: the live
|
||||
gate (``benchmarks/phase188_live_gate/``) watched exactly this happen on the
|
||||
8,866-chunk replica, where the planner used the index at ``ef_search = 40``
|
||||
and refused it at 100.
|
||||
|
||||
``enable_seqscan = off`` is Postgres' own "take any other plan if one
|
||||
exists" lever. It is set at SESSION scope because ``_vector_candidates``
|
||||
opens its own transaction, so a ``SET LOCAL`` issued here would die with the
|
||||
caller's transaction instead of reaching the tier. Every caller resets it in
|
||||
a ``finally`` — a leaked ``off`` would slow every later test on this pooled
|
||||
connection — and the assertion beside it proves the lever took effect.
|
||||
"""
|
||||
session.rollback()
|
||||
session.execute(text("SET enable_seqscan = off"))
|
||||
|
||||
|
||||
def _release_index_scan(session: Session) -> None:
|
||||
session.rollback()
|
||||
session.execute(text("RESET enable_seqscan"))
|
||||
session.rollback()
|
||||
|
||||
|
||||
def _set_ef_search(monkeypatch: pytest.MonkeyPatch, value: int) -> None:
|
||||
"""Drive the tier at *value* the way an operator does — the setting."""
|
||||
settings = _stub_settings(retrieval_ef_search=value)
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pin 1 — the planner uses the index (with its vacuity guard)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plan_uses_the_hnsw_index(db: Session, hnsw_corpus: dict[str, Any]) -> None:
|
||||
"""The 2026-10-15 incident WAS a plan, so the pin is a plan: the tier's own
|
||||
``ORDER BY embedding <=> <vector> LIMIT 100`` runs through
|
||||
``ix_chunks_embedding_hnsw`` and never reads ``chunks`` sequentially.
|
||||
|
||||
Explaining the captured statement is also what proves D1's "no query
|
||||
rewrite": the ``<=>`` the retriever already emits is exactly the operator
|
||||
``vector_cosine_ops`` serves.
|
||||
"""
|
||||
nodes = _plan(db, hnsw_corpus["plan_sql"])
|
||||
assert _uses_index(nodes, INDEX_NAME), (
|
||||
f"{INDEX_NAME} is not in the plan — the incident is back: [{_describe(nodes)}]"
|
||||
)
|
||||
assert not _seq_scans(nodes, RELATION), (
|
||||
f"a sequential scan of {RELATION} survived the index: [{_describe(nodes)}]"
|
||||
)
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_plan_without_the_index_seq_scans_the_relation(
|
||||
db: Session, hnsw_corpus: dict[str, Any]
|
||||
) -> None:
|
||||
"""Vacuity guard: the same statement, same session, same statistics, with
|
||||
index scans refused, MUST fall back to ``Seq Scan on chunks``.
|
||||
|
||||
``SET LOCAL enable_indexscan = off`` rather than dropping the index: a
|
||||
``DROP INDEX CONCURRENTLY`` + rebuild would cost a second graph build and
|
||||
leave the corpus unindexed if the rebuild failed, while this GUC is
|
||||
transaction-scoped (it dies with the rollback) and removes exactly the
|
||||
access path under test. Without this test the plan pin would pass vacuously
|
||||
on any plan that merely mentions some index.
|
||||
"""
|
||||
sql = hnsw_corpus["plan_sql"]
|
||||
db.rollback()
|
||||
db.execute(text("SET LOCAL enable_indexscan = off"))
|
||||
db.execute(text("SET LOCAL enable_indexonlyscan = off"))
|
||||
nodes = _plan(db, sql)
|
||||
assert _seq_scans(nodes, RELATION), (
|
||||
f"the vacuity guard itself did not seq-scan {RELATION}: [{_describe(nodes)}]"
|
||||
)
|
||||
assert not _uses_index(nodes, INDEX_NAME), (
|
||||
f"{INDEX_NAME} was used with index scans disabled: [{_describe(nodes)}]"
|
||||
)
|
||||
db.rollback()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pin 2 — the approximate top-100 IS the exact top-100
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_top100_recall_at_the_shipped_ef_search(
|
||||
db: Session,
|
||||
hnsw_corpus: dict[str, Any],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""At the shipped ``ef_search`` the index returns the exact top-100 for
|
||||
every query, exact to 1e-6 on the score the honesty gate reads.
|
||||
|
||||
Two different failures surface here: a ``top_score`` that moved — A8's
|
||||
deflect/answer verdict flips on that one number, so it is pinned per row,
|
||||
not per query — and an overlap below the floor, i.e. a candidate that never
|
||||
reaches RRF. The failure message carries every query's overlap so a future
|
||||
``ef_search`` raise has its evidence in the report instead of in a re-run.
|
||||
"""
|
||||
queries: list[list[float]] = hnsw_corpus["queries"]
|
||||
exact: list[list[tuple[float, Any]]] = hnsw_corpus["exact"]
|
||||
chunk_ids: set[Any] = hnsw_corpus["chunk_ids"]
|
||||
_set_ef_search(monkeypatch, EF_PIN)
|
||||
|
||||
_force_index_scan(db)
|
||||
try:
|
||||
depth_nodes = _plan(db, hnsw_corpus["plan_sql"])
|
||||
assert _uses_index(depth_nodes, INDEX_NAME), (
|
||||
f"this recall run at ef_search={EF_PIN} did not use {INDEX_NAME} even "
|
||||
"with sequential scans refused — the pin would be measuring a scan "
|
||||
f"and calling it parity: [{_describe(depth_nodes)}]"
|
||||
)
|
||||
|
||||
overlaps: list[float] = []
|
||||
worst_delta = 0.0
|
||||
for index, (query, exact_rows) in enumerate(zip(queries, exact, strict=True)):
|
||||
exact_ids = {chunk_id for _, chunk_id in exact_rows}
|
||||
exact_scores = {chunk_id: cosine for cosine, chunk_id in exact_rows}
|
||||
candidates = _vector_candidates(db, query, TOP_K)
|
||||
db.rollback() # end the turn's transaction, SET LOCAL included
|
||||
assert len(candidates) == TOP_K, (
|
||||
f"query {index}: the tier returned {len(candidates)} candidates, "
|
||||
f"not the {TOP_K} the A7 cosine tier promises"
|
||||
)
|
||||
got_ids = {candidate.chunk_id for candidate in candidates}
|
||||
assert got_ids <= chunk_ids, (
|
||||
f"query {index}: a candidate is outside the corpus the exact "
|
||||
"reference was computed over"
|
||||
)
|
||||
overlaps.append(len(exact_ids & got_ids) / TOP_K)
|
||||
for candidate in candidates:
|
||||
exact_cosine = exact_scores.get(candidate.chunk_id)
|
||||
if exact_cosine is not None:
|
||||
worst_delta = max(worst_delta, abs(candidate.cosine - exact_cosine))
|
||||
finally:
|
||||
_release_index_scan(db)
|
||||
|
||||
mean = sum(overlaps) / len(overlaps)
|
||||
below = [(i, overlap) for i, overlap in enumerate(overlaps) if overlap < RECALL_FLOOR]
|
||||
assert not below, (
|
||||
f"top-{TOP_K} recall below {RECALL_FLOOR} at ef_search={EF_PIN} for "
|
||||
f"{len(below)}/{len(overlaps)} queries: {below}; per-query overlap="
|
||||
f"{[round(overlap, 4) for overlap in overlaps]} mean={mean:.4f} — the "
|
||||
"remedy is a higher BOR_RETRIEVAL_EF_SEARCH, never this floor (LOCKED A3)"
|
||||
)
|
||||
assert worst_delta < COSINE_TOLERANCE, (
|
||||
f"1 - distance drifted from the exact cosine by {worst_delta:.3e} "
|
||||
f"(tolerance {COSINE_TOLERANCE:.0e}) — top_score would not be exact"
|
||||
)
|
||||
|
||||
|
||||
def test_recall_never_below_the_pinned_floor_when_ef_search_raised(
|
||||
db: Session,
|
||||
hnsw_corpus: dict[str, Any],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Monotonicity sanity: a deeper search cannot recover FEWER of the exact
|
||||
top-100 than a shallower one.
|
||||
|
||||
This is the property the operator leans on when the live battery (task 04)
|
||||
says "raise it" — ``ef_search`` is a recall dial, not a lottery. A
|
||||
regression here means the value never reaches the server (the ``SET LOCAL``
|
||||
of task 02), which is exactly what the shipped 400 would then silently not
|
||||
do. ``EF_RAISED`` sits ABOVE the shipped value, so the probe still measures
|
||||
a deeper search after task 04 moved the default.
|
||||
"""
|
||||
exact: list[list[tuple[float, Any]]] = hnsw_corpus["exact"]
|
||||
queries: list[list[float]] = hnsw_corpus["queries"]
|
||||
|
||||
def mean_overlap(ef_search: int) -> float:
|
||||
_set_ef_search(monkeypatch, ef_search)
|
||||
total = 0.0
|
||||
for query, exact_rows in zip(queries, exact, strict=True):
|
||||
exact_ids = {chunk_id for _, chunk_id in exact_rows}
|
||||
candidates = _vector_candidates(db, query, TOP_K)
|
||||
db.rollback()
|
||||
total += len(exact_ids & {candidate.chunk_id for candidate in candidates}) / TOP_K
|
||||
return total / len(exact)
|
||||
|
||||
pinned = mean_overlap(EF_PIN)
|
||||
raised = mean_overlap(EF_RAISED)
|
||||
assert raised >= pinned, (
|
||||
f"ef_search={EF_RAISED} recovered less of the exact top-{TOP_K} "
|
||||
f"({raised:.4f}) than ef_search={EF_PIN} ({pinned:.4f})"
|
||||
)
|
||||
@@ -46,6 +46,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -48,6 +48,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -50,6 +50,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -57,6 +57,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -60,6 +60,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -52,6 +52,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -52,6 +52,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -58,6 +58,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -68,6 +68,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -48,6 +48,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -58,6 +58,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -53,6 +53,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -59,6 +59,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -553,6 +553,39 @@ def test_name_hit_bonus_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> Non
|
||||
_settings()
|
||||
|
||||
|
||||
def test_retrieval_ef_search_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 188 (D3, raised by task 04's live battery): the HNSW
|
||||
``ef_search`` for the cosine tier defaults to 400 — FOUR times the A7
|
||||
candidate count, because the battery measured ``ef_search = 100``
|
||||
(``== LIMIT``) recovering only mean 0.9814 / min 0.94 of the exact
|
||||
top-100 on the 1,000-document live replica's real embeddings, under this
|
||||
phase's own 0.99 floor — and 400 recovered 1.0000 with a max ``top_score``
|
||||
delta of exactly 0. It stays env-tunable (``BOR_RETRIEVAL_EF_SEARCH``) in
|
||||
either direction, and the retriever applies it per transaction with
|
||||
``SET LOCAL``."""
|
||||
monkeypatch.delenv("BOR_RETRIEVAL_EF_SEARCH", raising=False)
|
||||
assert _settings().retrieval_ef_search == 400
|
||||
monkeypatch.setenv("BOR_RETRIEVAL_EF_SEARCH", "100")
|
||||
assert _settings().retrieval_ef_search == 100
|
||||
|
||||
|
||||
def test_retrieval_ef_search_refuses_zero_and_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""There is no ``ef_search=0`` to switch the index off: the retriever
|
||||
clamps anything below the A7 candidate count UP to it, so a ``0`` an
|
||||
operator might read as "off" would silently become 100. It fails at
|
||||
startup naming the field instead (phase 188, D3; the
|
||||
``show_text_max_chars`` fail-loud family — with ``0`` refused the way
|
||||
``recency_half_life_days`` refuses it)."""
|
||||
for bad in ("0", "-1"):
|
||||
monkeypatch.setenv("BOR_RETRIEVAL_EF_SEARCH", bad)
|
||||
with pytest.raises(ValidationError, match="retrieval_ef_search"):
|
||||
_settings()
|
||||
|
||||
|
||||
def test_recency_half_life_rejects_non_positive(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
|
||||
@@ -697,6 +697,155 @@ def test_vector_candidates_default_is_summary_false_for_legacy_chunks() -> None:
|
||||
assert out[0].is_summary is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 188 (task 02): the HNSW ``ef_search`` knob — SET LOCAL, clamped
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _one_vector_row_session() -> _FakeSession:
|
||||
"""A stub session whose one rowset is a single ordinary chunk."""
|
||||
doc = _doc("ef.md", "EF_BODY")
|
||||
chunk = Chunk(
|
||||
id=uuid.uuid4(),
|
||||
document_id=doc.id,
|
||||
position=0,
|
||||
content="ef content",
|
||||
is_summary=False,
|
||||
)
|
||||
return _FakeSession([(chunk, 0.25, doc)])
|
||||
|
||||
|
||||
def _sql(session: _FakeSession) -> list[str]:
|
||||
"""The recorded statements as SQL text, in call order."""
|
||||
return [str(stmt) for stmt, _params in session.statements]
|
||||
|
||||
|
||||
def _ef_statement(session: _FakeSession) -> str | None:
|
||||
for sql in _sql(session):
|
||||
if sql.startswith("SET LOCAL hnsw.ef_search"):
|
||||
return sql
|
||||
return None
|
||||
|
||||
|
||||
def test_vector_candidates_sets_ef_search_first(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 188 (D3/LOCKED A4): the tier opens with
|
||||
``SET LOCAL hnsw.ef_search = <value>`` BEFORE the ``<=>`` SELECT (a
|
||||
``SET`` that arrived after the SELECT would tune the next turn
|
||||
instead of this one), and it is LOCAL — never a session/global ``SET``.
|
||||
The default (400 — the depth task 04's live battery reached exact
|
||||
top-100 parity at) is what goes out when the operator set nothing."""
|
||||
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
assert settings.retrieval_ef_search == 400 # the production default
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
session = _one_vector_row_session()
|
||||
out = _vector_candidates(session, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
assert len(out) == 1, "the candidates still come back"
|
||||
statements = _sql(session)
|
||||
assert statements[0] == "SET LOCAL hnsw.ef_search = 400"
|
||||
assert len(statements) == 2, "exactly one SET plus the candidate SELECT"
|
||||
assert "<=>" in statements[1]
|
||||
|
||||
|
||||
def test_vector_candidates_clamps_ef_search_below_the_candidate_count(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""``ef_search`` BELOW the candidate count cannot be honoured — it
|
||||
returns FEWER candidates than the A7 tier promises — so it is CLAMPED
|
||||
up to the count with one WARNING line (the phase-188 D3 rule), and the
|
||||
SELECT still asks for the full ``LIMIT``."""
|
||||
settings = Settings(_env_file=None, retrieval_ef_search=10) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
session = _one_vector_row_session()
|
||||
with caplog.at_level("WARNING", logger="app.rag.retriever"):
|
||||
_vector_candidates(session, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 100"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert "retrieval: ef_search=10 raised to 100 (A7 candidate count)" in messages
|
||||
select_sql = _sql(session)[1]
|
||||
# the LIMIT stays the tier's own (parameterised) count — the clamp
|
||||
# raises the SEARCH depth, it never shrinks the promise
|
||||
assert "LIMIT :param_1" in select_sql
|
||||
|
||||
|
||||
def test_vector_candidates_ef_search_equal_to_the_limit_does_not_warn(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The clamp is STRICTLY below: ``ef_search == limit`` (the default
|
||||
100/100 shape) is the healthy state and logs nothing."""
|
||||
settings = Settings(_env_file=None, retrieval_ef_search=100) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
session = _one_vector_row_session()
|
||||
with caplog.at_level("WARNING", logger="app.rag.retriever"):
|
||||
_vector_candidates(session, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 100"
|
||||
assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
|
||||
|
||||
|
||||
def test_vector_candidates_passes_ef_search_above_the_limit(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Above the candidate count the operator's number goes through
|
||||
untouched (the recall dial the phase-188 battery may raise it to) and
|
||||
nothing warns."""
|
||||
settings = Settings(_env_file=None, retrieval_ef_search=200) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
session = _one_vector_row_session()
|
||||
with caplog.at_level("WARNING", logger="app.rag.retriever"):
|
||||
_vector_candidates(session, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
assert _ef_statement(session) == "SET LOCAL hnsw.ef_search = 200"
|
||||
assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
|
||||
|
||||
|
||||
def test_vector_candidates_ef_search_statement_is_int_only(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The interpolated fragment is ``int()``-coerced: the SET line carries
|
||||
digits only — no quote, no semicolon, nothing a string could smuggle
|
||||
into the statement (the value is a ``Settings`` int, but the coercion
|
||||
is the belt beside the suspenders)."""
|
||||
settings = Settings(_env_file=None, retrieval_ef_search=137) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
session = _one_vector_row_session()
|
||||
_vector_candidates(session, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
line = _ef_statement(session)
|
||||
assert line is not None
|
||||
value = line.split("=", 1)[1].strip()
|
||||
assert value == "137" and value.isdigit()
|
||||
|
||||
|
||||
def test_ef_search_below_one_refuses_startup_the_clamp_has_no_off_state() -> None:
|
||||
"""The clamp means ``ef_search=0`` could never mean "off" — it would be
|
||||
silently rewritten to the candidate count — so ``0`` and negatives are
|
||||
refused at settings construction instead (phase 188, D3; the
|
||||
``show_text_max_*`` family's fail-loud pattern)."""
|
||||
from pydantic import ValidationError
|
||||
|
||||
for bad in (0, -1):
|
||||
with pytest.raises(ValidationError, match="retrieval_ef_search"):
|
||||
Settings(_env_file=None, retrieval_ef_search=bad) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def test_ef_search_default_clears_the_a7_candidate_count() -> None:
|
||||
"""The two numbers the clamp rule reads: the shipped default sits ABOVE
|
||||
``hybrid_vector_candidates`` (so a default install never warns), and it
|
||||
is not merely equal to it.
|
||||
|
||||
Equality was the task-02 default and the task-04 live battery retired it:
|
||||
``ef_search == LIMIT`` is the weakest depth an HNSW graph search can be
|
||||
handed, and on the 1,000-document live replica's real embeddings it
|
||||
recovered mean 0.9814 / min 0.94 of the exact top-100 — under this
|
||||
phase's 0.99 floor. 400 recovered 1.0000 with every A8 verdict, top-5 set
|
||||
and ``top_score`` unchanged, and it costs the tier nothing measurable
|
||||
(12-14 ms against 12-13 ms at 100, on 416k chunks, versus the ~550 ms
|
||||
sequential scan this index replaces).
|
||||
"""
|
||||
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
assert settings.hybrid_vector_candidates == 100 # the A7 clamp floor
|
||||
assert settings.retrieval_ef_search == 400 > settings.hybrid_vector_candidates
|
||||
|
||||
|
||||
def _lexical_row(is_summary: bool, doc_path: str) -> object:
|
||||
"""One row of ``_LEXICAL_SQL`` (attribute access, as SQLAlchemy returns)."""
|
||||
doc = _doc(doc_path, "DOC_BODY")
|
||||
|
||||
@@ -73,6 +73,36 @@ def test_non_overridable_is_exactly_locked_a3() -> None:
|
||||
assert set(NON_OVERRIDABLE) == {"database_url", "embedding_dim"}
|
||||
|
||||
|
||||
def test_ef_search_row_is_a_plain_live_overridable_row() -> None:
|
||||
"""Phase 188 (task 02, D3): the HNSW ``ef_search`` row sits in
|
||||
``Retrieval & honesty`` (beside ``hybrid_vector_candidates`` — the two
|
||||
numbers the clamp rule reads together) with the ``Vector ef_search``
|
||||
label and is plain — not ``secret`` (an int), not ``restart_required``
|
||||
(``_vector_candidates`` reads ``get_settings()`` per turn, so a saved
|
||||
override applies on the next question), overridable (the overlay is
|
||||
generic over ``int`` fields). ``0``/negative fails startup, not the
|
||||
panel.
|
||||
|
||||
The default this row ships with is 400, not the A7 candidate count it
|
||||
launched at: task 04's live gate measured the graph recovering the exact
|
||||
top-100 for 0.9814 of rows on average (0.94 at worst) at 100 against 1.0000
|
||||
at 400, on the restored live KB with the vectors pinned
|
||||
(``benchmarks/phase188_live_gate/``). It stays live-overridable in either
|
||||
direction, which is what makes that a tuning decision and not a redeploy.
|
||||
"""
|
||||
meta = CATALOG["retrieval_ef_search"]
|
||||
assert meta.group == "Retrieval & honesty"
|
||||
assert meta.label == "Vector ef_search"
|
||||
assert meta.secret is False
|
||||
assert meta.restart_required is False
|
||||
assert meta.overridable is True
|
||||
assert "retrieval_ef_search" not in SECRETS
|
||||
assert "retrieval_ef_search" not in RESTART_REQUIRED
|
||||
assert "retrieval_ef_search" not in NON_OVERRIDABLE
|
||||
assert Settings.model_fields["retrieval_ef_search"].annotation is int
|
||||
assert Settings.model_fields["retrieval_ef_search"].default == 400
|
||||
|
||||
|
||||
def test_restart_required_is_the_boot_wired_set() -> None:
|
||||
# LOCKED A4: logging, the gzip middleware (compress_min_bytes), the
|
||||
# static mount (static_dir), create_engine (db_pool_*), and
|
||||
|
||||
Reference in New Issue
Block a user