phase: 188_chunks_hnsw_index
**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
This commit is contained in:
@@ -0,0 +1,354 @@
|
||||
"""Integration: migration 0029 (chunks.embedding HNSW index) — phase 188, task 01.
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the house pattern of
|
||||
``test_migration_0028.py`` (fixture upgrades to head at setup and
|
||||
teardown so the dev DB is never left below head; tests target the
|
||||
0028 → 0029 step explicitly so later migrations cannot break them):
|
||||
|
||||
* upgrade 0028 → 0029 → ``ix_chunks_embedding_hnsw`` exists,
|
||||
``pg_index.indisvalid`` is TRUE, and ``pg_get_indexdef`` names
|
||||
``USING hnsw`` + ``vector_cosine_ops`` + ``m=16`` + ``ef_construction=64``
|
||||
(the D1 shape the recall pins of task 03 depend on);
|
||||
* downgrade to 0028 → the index is GONE (A13 — reversible, ``CONCURRENTLY``),
|
||||
the ``chunks`` table and its PK survive;
|
||||
* upgrade back to head → the index is back and valid (round-trip);
|
||||
* the D2 self-heal: with an INVALID index of that name deliberately
|
||||
present, ``upgrade()`` drops it and rebuilds it VALID — no
|
||||
``relation already exists`` error and no leftover INVALID index.
|
||||
The INVALID state is forced with the documented superuser catalog
|
||||
trick ``UPDATE pg_index SET indisvalid = false WHERE
|
||||
indexrelid = CAST(:name AS regclass)`` — the same failure a killed
|
||||
``CREATE INDEX CONCURRENTLY`` leaves behind (pgvector cannot fail a
|
||||
build on data, and the dev/test DB role is a superuser; a role
|
||||
without catalog UPDATE rights gets a clean skip).
|
||||
|
||||
Pure-helper matrix tests (no DB): ``should_drop_invalid`` and
|
||||
``is_valid`` over ``None`` / ``True`` / ``False``.
|
||||
|
||||
**Why every ``command.*`` call here is preceded by ``db.rollback()``
|
||||
when the session was used.** psycopg3 keeps its transaction snapshot
|
||||
alive while the session sits idle-in-transaction (measured: ``pg_stat
|
||||
_activity.backend_xmin`` stays set where psql's clears), and ``CREATE
|
||||
INDEX CONCURRENTLY`` phase 3 waits for every older-snapshot
|
||||
transaction to finish — an unreleased session parks the build FOREVER
|
||||
(observed on the dev box). The test session must therefore hold no
|
||||
open transaction while the concurrent DDL runs; the rollback is what
|
||||
makes the file deterministic rather than flaky.
|
||||
**Task 02 adds the ``ef_search`` half:** the knob the index needs is
|
||||
applied with ``SET LOCAL`` inside the turn's own transaction, so the file
|
||||
pins the SCOPING against the real server — after a ``SET LOCAL`` the same
|
||||
transaction reports the (clamped) value and a NEW transaction reports the
|
||||
server's own default again (LOCKED A4: no global GUC, nothing surviving
|
||||
the request or a pool recycle).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import logging
|
||||
import types
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import ProgrammingError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.config import Settings
|
||||
from app.db import db_available
|
||||
from app.rag import retriever
|
||||
from app.rag.retriever import _vector_candidates
|
||||
|
||||
MIGRATION_PATH = Path("alembic/versions/0029_chunks_embedding_hnsw.py")
|
||||
INDEX_NAME = "ix_chunks_embedding_hnsw"
|
||||
|
||||
|
||||
def _load_migration() -> types.ModuleType:
|
||||
"""Import the 0029 module by file path (``alembic/versions`` is not a
|
||||
package — the house way to reach a migration's own helpers)."""
|
||||
spec = importlib.util.spec_from_file_location("m0029_chunks_embedding_hnsw", MIGRATION_PATH)
|
||||
assert spec is not None and spec.loader is not None, f"missing {MIGRATION_PATH}"
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run — 0029's own
|
||||
INVALID self-heal makes that repair safe when the interrupted
|
||||
migration was 0029 itself); teardown upgrades to head no matter
|
||||
what happened, so the dev DB is never left below head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL (the 0018 house pattern): a still-running snapshot would
|
||||
# park the CONCURRENTLY build, and teardown is exactly where a
|
||||
# failed test leaves one open.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _index_present(db: Session) -> bool:
|
||||
return (
|
||||
db.execute(
|
||||
text("SELECT 1 FROM pg_indexes WHERE indexname = :n"), {"n": INDEX_NAME}
|
||||
).scalar()
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _index_valid(db: Session) -> bool | None:
|
||||
"""``pg_index.indisvalid`` for the index, ``None`` when it is absent."""
|
||||
return db.execute(
|
||||
text(
|
||||
"SELECT i.indisvalid FROM pg_index i"
|
||||
" JOIN pg_class c ON c.oid = i.indexrelid"
|
||||
" WHERE c.relname = :n"
|
||||
),
|
||||
{"n": INDEX_NAME},
|
||||
).scalar()
|
||||
|
||||
|
||||
def _index_def(db: Session) -> str | None:
|
||||
return db.execute(
|
||||
text("SELECT pg_get_indexdef(c.oid) FROM pg_class c WHERE c.relname = :n"),
|
||||
{"n": INDEX_NAME},
|
||||
).scalar()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure helpers (no DB, no I/O)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_should_drop_invalid_matrix() -> None:
|
||||
"""``should_drop_invalid``: absent → False, valid → False,
|
||||
INVALID → True (the only state the pre-drop fires for)."""
|
||||
mod = _load_migration()
|
||||
assert mod.should_drop_invalid(None) is False, "absent index needs no drop"
|
||||
assert mod.should_drop_invalid(True) is False, "a valid index must be left alone"
|
||||
assert mod.should_drop_invalid(False) is True, "an INVALID remnant must be dropped"
|
||||
|
||||
|
||||
def test_is_valid_matrix() -> None:
|
||||
"""``is_valid``: only a present, validated index counts as valid."""
|
||||
mod = _load_migration()
|
||||
assert mod.is_valid(None) is False, "absent is not valid"
|
||||
assert mod.is_valid(True) is True, "validated is valid"
|
||||
assert mod.is_valid(False) is False, "INVALID is not valid"
|
||||
|
||||
|
||||
def test_migration_module_contract() -> None:
|
||||
"""The module's identity: revision 0029 on top of 0028, and the one
|
||||
index name the whole phase pins on."""
|
||||
mod = _load_migration()
|
||||
assert mod.revision == "0029"
|
||||
assert mod.down_revision == "0028"
|
||||
assert mod.INDEX_NAME == INDEX_NAME
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Migration round-trip against the real Alembic engine
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_upgrade_creates_a_valid_hnsw_index(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0028 → 0029: the index exists, VALIDATED, with the D1
|
||||
shape — HNSW, ``vector_cosine_ops`` (the operator class A7's ``<=>``
|
||||
needs), ``m=16``, ``ef_construction=64`` — and the ``chunks`` table
|
||||
itself survives the additive step untouched."""
|
||||
command.downgrade(alembic, "0028") # start from the pre-0029 state
|
||||
assert _version(db) == "0028"
|
||||
assert not _index_present(db), f"{INDEX_NAME} must be absent at 0028"
|
||||
|
||||
db.rollback() # no open snapshot while the CONCURRENTLY build runs
|
||||
command.upgrade(alembic, "0029")
|
||||
assert _version(db) == "0029", "alembic_version must be at 0029"
|
||||
assert _index_present(db), "the index must exist at 0029"
|
||||
assert _index_valid(db) is True, "a fresh build must be VALID (D2 post-check)"
|
||||
|
||||
raw = _index_def(db)
|
||||
assert raw is not None
|
||||
definition = raw.replace("'", "") # pg renders WITH options quoted
|
||||
assert "USING hnsw" in definition, f"not an HNSW index: {definition}"
|
||||
assert "vector_cosine_ops" in definition, (
|
||||
f"cosine operator class required for <=>: {definition}"
|
||||
)
|
||||
assert "m=16" in definition, f"m=16 expected: {definition}"
|
||||
assert "ef_construction=64" in definition, f"ef_construction=64 expected: {definition}"
|
||||
|
||||
|
||||
def test_downgrade_removes_it(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade 0029 → 0028: the index is gone (A13 — reversible), and
|
||||
the ``chunks`` relation with its PK survives (the drop touches no
|
||||
data)."""
|
||||
command.upgrade(alembic, "0029")
|
||||
assert _index_present(db)
|
||||
|
||||
db.rollback() # no open snapshot while the CONCURRENTLY drop runs
|
||||
command.downgrade(alembic, "0028")
|
||||
assert _version(db) == "0028"
|
||||
assert not _index_present(db), f"{INDEX_NAME} must be dropped by the downgrade"
|
||||
assert _index_valid(db) is None, "no pg_index row may survive the downgrade"
|
||||
|
||||
pk = db.execute(
|
||||
text("SELECT 1 FROM pg_indexes WHERE indexname = 'chunks_pkey'")
|
||||
).scalar()
|
||||
assert pk is not None, "chunks_pkey must survive the downgrade"
|
||||
|
||||
|
||||
def test_upgrade_head_again(db: Session, alembic: Config) -> None:
|
||||
"""A13 round-trip: downgrade to 0028, upgrade back to HEAD — the
|
||||
index returns, valid, with the same definition."""
|
||||
command.downgrade(alembic, "0028")
|
||||
command.upgrade(alembic, "head")
|
||||
assert _index_present(db), "the index must be back after re-upgrade"
|
||||
assert _index_valid(db) is True, "the rebuilt index must be VALID"
|
||||
raw = _index_def(db)
|
||||
assert raw is not None and "USING hnsw" in raw
|
||||
|
||||
|
||||
def test_invalid_index_is_dropped_and_rebuilt(db: Session, alembic: Config) -> None:
|
||||
"""D2 self-heal: an INVALID index of this name (what a killed
|
||||
``CREATE INDEX CONCURRENTLY`` leaves behind) is dropped and rebuilt
|
||||
VALID on the next ``upgrade()``.
|
||||
|
||||
The INVALID state is forced with the documented superuser catalog
|
||||
update ``UPDATE pg_index SET indisvalid = false WHERE indexrelid =
|
||||
CAST(:name AS regclass)`` — pgvector cannot fail an HNSW build on
|
||||
data, and timing a real interruption is nondeterministic; the dev/
|
||||
test DB role is a superuser, and a role without catalog UPDATE
|
||||
rights skips cleanly instead of leaving the DB inconsistent (the
|
||||
stamp that follows the forced state, never before it)."""
|
||||
command.upgrade(alembic, "head")
|
||||
assert _index_valid(db) is True, "precondition: a valid index to corrupt"
|
||||
|
||||
try:
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE pg_index SET indisvalid = false"
|
||||
" WHERE indexrelid = CAST(:n AS regclass)"
|
||||
),
|
||||
{"n": INDEX_NAME},
|
||||
)
|
||||
db.commit()
|
||||
except ProgrammingError as exc: # non-superuser role
|
||||
db.rollback()
|
||||
pytest.skip(f"forcing INVALID needs catalog UPDATE rights: {exc}")
|
||||
|
||||
# The stamp moves the version table WITHOUT touching the index, so
|
||||
# the next upgrade() re-runs 0029 against the corrupted state.
|
||||
command.stamp(alembic, "0028")
|
||||
assert _version(db) == "0028"
|
||||
assert _index_valid(db) is False, "the forced INVALID state must be visible"
|
||||
|
||||
db.rollback() # no open snapshot of ours while the CONCURRENTLY run rebuilds
|
||||
command.upgrade(alembic, "0029")
|
||||
assert _index_present(db), "no 'relation already exists' — it was pre-dropped"
|
||||
assert _index_valid(db) is True, "the rebuilt index must be VALID (no leftover INVALID)"
|
||||
raw = _index_def(db)
|
||||
assert raw is not None and "USING hnsw" in raw, "the rebuild is the HNSW index itself"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Task 02: SET LOCAL hnsw.ef_search — applied, clamped, transaction-scoped
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _show_ef_search(db: Session) -> str:
|
||||
return str(db.execute(text("SHOW hnsw.ef_search")).scalar())
|
||||
|
||||
|
||||
def _server_default_ef_search(db: Session) -> str:
|
||||
"""The server's own reset value for the GUC (the value a NEW
|
||||
transaction must fall back to). Read from ``pg_settings`` rather than
|
||||
hardcoded: pgvector renders its default, and the pin is about the
|
||||
FALLBACK, not the number."""
|
||||
value = db.execute(
|
||||
text("SELECT reset_val FROM pg_settings WHERE name = 'hnsw.ef_search'")
|
||||
).scalar()
|
||||
assert value is not None, "pgvector must register hnsw.ef_search in this session"
|
||||
return str(value)
|
||||
|
||||
|
||||
def _stub_settings(**kwargs: object) -> Settings:
|
||||
return Settings(_env_file=None, **kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def test_set_local_ef_search_reaches_the_server_in_the_same_transaction(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The D3 rule against a real server: the value the tier sends is the
|
||||
value Postgres runs the search with. The setting here (10) sits BELOW
|
||||
the limit (250), so the server must report the CLAMPED value — the
|
||||
clamp is not a Python-side fiction."""
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=10)
|
||||
)
|
||||
db.rollback() # a clean transaction boundary for the SET LOCAL
|
||||
_vector_candidates(db, [0.0] * 768, limit=250) # pyright: ignore[reportArgumentType]
|
||||
assert _show_ef_search(db) == "250", "the clamped value must be live in this tx"
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_set_local_ef_search_does_not_survive_the_transaction(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""LOCKED A4 pinned on the wire: once the turn's transaction ends, the
|
||||
server is back on ITS default — no global GUC mutation, no state a
|
||||
pooled connection carries into the next request."""
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=317)
|
||||
)
|
||||
db.rollback()
|
||||
_vector_candidates(db, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
assert _show_ef_search(db) == "317"
|
||||
|
||||
db.rollback() # the end of the turn's transaction
|
||||
after = _show_ef_search(db)
|
||||
default = _server_default_ef_search(db)
|
||||
assert after == default, f"a new tx must see the server default, got {after}"
|
||||
assert after != "317"
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_ef_search_clamp_logs_one_warning_line(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The clamp leaves exactly one operator-visible line in the server log
|
||||
(AGENTS rule 10), and the pass-through case leaves none."""
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=5)
|
||||
)
|
||||
db.rollback()
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.retriever"):
|
||||
_vector_candidates(db, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
db.rollback()
|
||||
lines = [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()]
|
||||
assert lines == ["retrieval: ef_search=5 raised to 100 (A7 candidate count)"]
|
||||
|
||||
monkeypatch.setattr(
|
||||
retriever, "get_settings", lambda: _stub_settings(retrieval_ef_search=100)
|
||||
)
|
||||
caplog.clear()
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.retriever"):
|
||||
_vector_candidates(db, [0.0] * 768, limit=100) # pyright: ignore[reportArgumentType]
|
||||
db.rollback()
|
||||
assert [r.getMessage() for r in caplog.records if "ef_search" in r.getMessage()] == []
|
||||
@@ -0,0 +1,612 @@
|
||||
"""Integration: the cosine tier USES ``ix_chunks_embedding_hnsw`` and the
|
||||
top-100 did not move (phase 188, task 03 — the deterministic half of D4).
|
||||
|
||||
**The production numbers this pin stands in for** (read 2026-10-15, read-only):
|
||||
414,220 chunks — all with a non-null ``embedding`` — a ``chunks`` relation of
|
||||
≈ **3,000 MB** (database 3,147 MB), ``shared_buffers=128MB``, ``work_mem=4MB``,
|
||||
a 3,904 MB host with **no swap**, pgvector **0.8.0**, and ``pg_indexes`` for
|
||||
``chunks`` listing only ``chunks_pkey``, ``ix_chunks_document_id`` and
|
||||
``ix_chunks_tsv`` (GIN): **no index on ``embedding``**. Every chat turn's A7
|
||||
cosine tier (``ORDER BY embedding <=> :vec LIMIT 100``) was therefore a
|
||||
sequential scan of that 3 GB relation.
|
||||
|
||||
Two CI pins, both deterministic and LLM-free:
|
||||
|
||||
1. **The plan pin** — ``EXPLAIN (FORMAT JSON)`` of the tier's own SELECT (the
|
||||
statement object captured from
|
||||
:func:`app.rag.retriever._vector_candidates`, compiled with literal binds so
|
||||
the planner sees a real constant) names ``ix_chunks_embedding_hnsw`` and
|
||||
contains no ``Seq Scan on chunks``. Its vacuity guard runs the SAME
|
||||
statement with index scans refused and MUST see the seq scan, so the pin
|
||||
cannot pass on a plan the index never influenced.
|
||||
2. **The recall pin** — the index-backed top-100 against the EXACT cosine
|
||||
top-100 of the fixture corpus: ``|overlap| / 100 >= 0.99`` for EVERY query
|
||||
at the SHIPPED ``ef_search`` (400 since task 04's live battery; 100, the
|
||||
A7 candidate count, was the task-02 default and the battery retired it),
|
||||
each returned row's ``1 − distance`` equal to the exact cosine within
|
||||
**1e-6**, and the mean one step DEEPER never below the mean at the shipped
|
||||
value. Every recall run also re-reads the plan it just ran, because an
|
||||
``ef_search`` the planner declines turns the "index" pin into a second
|
||||
measurement of the sequential scan — a vacuous pass, and the one failure
|
||||
mode this file exists to catch.
|
||||
|
||||
**Measured on this corpus** (5,000 chunks, ``TOP_K = 100``, 30 fixed-seed
|
||||
queries, ``m = 16`` / ``ef_construction = 64`` from migration 0029):
|
||||
``ef_search=100`` → mean top-100 overlap **0.9993**, per-query minimum **0.99**
|
||||
(two of the thirty return 99/100), zero queries below the floor;
|
||||
``ef_search=200`` and the shipped **400** → mean **1.0000**. Worst
|
||||
``|cosine − exact cosine|`` over every
|
||||
returned row: **5.7e-07**, and 1e-6 is the tightest tolerance that can hold —
|
||||
``_vector_candidates`` publishes ``round(1 − distance, 6)``, which alone can move
|
||||
a value by 5e-7, on top of the float4 rounding of the distance itself (~6e-8 at
|
||||
this magnitude). The exact reference is computed over the vectors as the
|
||||
database returned them, so the comparison measures the index. Setup stays inside the runtime guard:
|
||||
generate ≈ 0.5 s, seed ≈ 2 s, ``CREATE INDEX CONCURRENTLY`` ≈ 1.4 s, read-back +
|
||||
exact top-100 ≈ 3.5 s — ≈ 8 s against the ~60 s target.
|
||||
|
||||
**What 5,000 rows proves and what it does not.** It is enough for a PLAN pin
|
||||
(the planner needs a populated relation to prefer an ordered index scan over a
|
||||
seq scan + top-N sort) and enough for a deterministic index-vs-exact
|
||||
regression; it is not a proxy for 414k rows, because recall is a function of the
|
||||
corpus shape and of graph density at 83× the size. The prod-scale gate is task
|
||||
04 — the live retrieval battery, index off vs on, over the 1,000-document
|
||||
replica KB, where every ``top_score``, top-5 set and A8 verdict must match.
|
||||
|
||||
**Corpus shape, and why it is not uniform noise.** One
|
||||
``random.Random(20261015)`` stream in a fixed order: a 16-direction principal
|
||||
subspace (one random unit direction per disjoint 48-dim block of the 768 —
|
||||
disjoint blocks are orthonormal by construction, so no Gram–Schmidt and no
|
||||
numerical drift), 40 unit centroids inside it, 5,000 chunks of
|
||||
``normalise(centroid + 2.0 × a random subspace direction)`` (125 per cluster),
|
||||
then 30 queries drawn uniformly from the same subspace. Low EFFECTIVE rank (16
|
||||
of the 768 stored dimensions carry the structure; every dimension is still
|
||||
non-zero) is deliberate: it is how trained text embeddings behave (their
|
||||
spectrum concentrates far below the nominal dimension), and it is the shape
|
||||
under which ``ef_search == LIMIT`` can be expected to return the exact top-100.
|
||||
|
||||
The counter-case is recorded here because measuring it is a finding of this
|
||||
task: the same 5,000 rows with the cluster bias thinned to a whisper (centroid
|
||||
weight 0.1 over per-dimension Gaussian noise across all 768 dimensions —
|
||||
effectively iid, the hardest shape a graph index can be handed, and the one
|
||||
this file's sweep ran at centroid weights 0.1, 0.3 and 0.5 alike) measures mean
|
||||
recall@100 ≈ **0.63** at ``ef_search=100``, ≈ **0.85** at 200 and **1.00** from
|
||||
400 up, whatever ``m`` / ``ef_construction`` migration 0029 ships. ``ef_search``
|
||||
at the candidate count is therefore NOT a free win on high-effective-rank data;
|
||||
if the live battery moves a ``top_score`` or a top-5 set, LOCKED A3's answer is
|
||||
to raise ``BOR_RETRIEVAL_EF_SEARCH`` (towards 400), never to relax an assertion
|
||||
here.
|
||||
|
||||
The index is built by migration **0029** and never by an inline ``CREATE
|
||||
INDEX`` — the migration is the single definition of the index these pins name
|
||||
(``0029 → 0028`` → seed unindexed → ``upgrade 0029``). As in
|
||||
``test_chunks_hnsw_index.py``, the seeding session holds NO open transaction
|
||||
while the ``CONCURRENTLY`` DDL runs (an idle-in-transaction snapshot parks the
|
||||
build), and teardown leaves the dev DB at head, index valid, fixture rows gone.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import operator
|
||||
import random
|
||||
import struct
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import Chunk, Document
|
||||
from app.rag import retriever
|
||||
from app.rag.retriever import _vector_candidates
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The corpus (D4's deterministic half) — one fixed stream, no wall clock
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
DIM = 768 # A6: the embedding dimension is fixed at table creation
|
||||
CENTROIDS = 40
|
||||
SUBSPACE = 16 # effective rank of the corpus (see the module docstring)
|
||||
BLOCK = DIM // SUBSPACE # 48 dims per basis direction, disjoint by construction
|
||||
NOISE = 2.0 # |noise| / |centroid| — cos(chunk, own centroid) ~= 0.45
|
||||
CORPUS_CHUNKS = 5000
|
||||
CHUNKS_PER_DOCUMENT = 10
|
||||
QUERIES = 30
|
||||
SEED = 20261015 # the date of the incident that started phase 188
|
||||
FIXTURE_SOURCE = "phase-188-hnsw-recall"
|
||||
|
||||
TOP_K = 100 # A7's cosine candidate count — the LIMIT the tier promises
|
||||
EF_PIN = 400 # the shipped default (Settings.retrieval_ef_search)
|
||||
EF_RAISED = 600 # the monotonicity probe
|
||||
RECALL_FLOOR = 0.99 # LOCKED A3 — a raised ef_search is the only remedy
|
||||
COSINE_TOLERANCE = 1e-6 # "exact to 6 decimals"
|
||||
|
||||
#: The pins need a small relation: the exact reference is a Python scan over
|
||||
#: every row. A DB holding more is a restored replica rather than the dev
|
||||
#: corpus — that scale is task 04's live battery, and 30 × N dot products here
|
||||
#: would only make the suite crawl.
|
||||
MAX_RELATION_ROWS = 20_000
|
||||
|
||||
INDEX_NAME = "ix_chunks_embedding_hnsw"
|
||||
RELATION = "chunks"
|
||||
|
||||
|
||||
def _f32(value: float) -> float:
|
||||
"""The ``float4`` the database holds for a query component.
|
||||
|
||||
A question vector goes to Postgres as ``str(float)`` text and comes back as
|
||||
``float4`` (``strtof`` rounds to nearest, the same value ``struct.pack``
|
||||
produces), so the exact reference rounds the QUERY the way the server does.
|
||||
The corpus needs no modelling — its components are read back from the
|
||||
database — which is what makes the comparison measure the INDEX rather than
|
||||
a precision artefact of the fixture.
|
||||
"""
|
||||
return struct.unpack("f", struct.pack("f", value))[0]
|
||||
|
||||
|
||||
def _norm(vec: list[float]) -> float:
|
||||
return math.sqrt(sum(v * v for v in vec))
|
||||
|
||||
|
||||
def _unit(vec: list[float]) -> list[float]:
|
||||
n = _norm(vec)
|
||||
return [v / n for v in vec] if n else vec
|
||||
|
||||
|
||||
def _generate() -> tuple[list[list[float]], list[list[float]]]:
|
||||
"""Fixed-seed ``(corpus, queries)``, all L2-normalised.
|
||||
|
||||
Normalising makes cosine the dot product, which is what keeps the exact
|
||||
reference cheap enough to run in CI. One stream in a fixed order (basis →
|
||||
centroids → chunks → queries) is what turns the recall floor into a
|
||||
regression pin instead of a measurement: byte-identical input, therefore a
|
||||
byte-identical graph, therefore a byte-identical overlap.
|
||||
"""
|
||||
rng = random.Random(SEED)
|
||||
basis = [_unit([rng.gauss(0.0, 1.0) for _ in range(BLOCK)]) for _ in range(SUBSPACE)]
|
||||
|
||||
def subspace_vector() -> list[float]:
|
||||
"""A random unit vector inside the 16-direction subspace."""
|
||||
coefs = [rng.gauss(0.0, 1.0) for _ in range(SUBSPACE)]
|
||||
n = _norm(coefs) or 1.0
|
||||
out = [0.0] * DIM
|
||||
for k, direction in enumerate(basis):
|
||||
weight = coefs[k] / n
|
||||
start = k * BLOCK
|
||||
for j in range(BLOCK):
|
||||
out[start + j] = weight * direction[j]
|
||||
return out
|
||||
|
||||
centroids = [subspace_vector() for _ in range(CENTROIDS)]
|
||||
corpus: list[list[float]] = []
|
||||
for i in range(CORPUS_CHUNKS):
|
||||
centroid = centroids[i % CENTROIDS]
|
||||
noise = subspace_vector()
|
||||
corpus.append(_unit([c + NOISE * n for c, n in zip(centroid, noise, strict=True)]))
|
||||
queries = [subspace_vector() for _ in range(QUERIES)]
|
||||
return corpus, queries
|
||||
|
||||
|
||||
def _seed(session: Session, corpus: list[list[float]]) -> None:
|
||||
"""Insert the corpus — 500 fixture documents × 10 chunks, one commit.
|
||||
|
||||
Ordinary content chunks (``is_summary`` False, embedding set): the
|
||||
NULL-embedding skip is pinned in ``tests/unit/test_retriever.py``, and a
|
||||
NULL row would sit in neither the exact set nor the index.
|
||||
"""
|
||||
documents = [
|
||||
Document(
|
||||
id=uuid.uuid4(),
|
||||
source=FIXTURE_SOURCE,
|
||||
path=f"{FIXTURE_SOURCE}/{d}.md",
|
||||
full_path=f"/tmp/{FIXTURE_SOURCE}/{d}.md",
|
||||
title=f"hnsw recall fixture {d}",
|
||||
content="fixture content",
|
||||
content_hash="0" * 64,
|
||||
)
|
||||
for d in range(CORPUS_CHUNKS // CHUNKS_PER_DOCUMENT)
|
||||
]
|
||||
session.add_all(documents)
|
||||
session.flush()
|
||||
session.add_all(
|
||||
[
|
||||
Chunk(
|
||||
id=uuid.uuid4(),
|
||||
document_id=documents[i // CHUNKS_PER_DOCUMENT].id,
|
||||
position=i % CHUNKS_PER_DOCUMENT,
|
||||
content=f"fixture chunk {i}",
|
||||
embedding=vector,
|
||||
is_summary=False,
|
||||
)
|
||||
for i, vector in enumerate(corpus)
|
||||
]
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exact reference (pure Python, float4-modelled inputs)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _exact_top(stored: dict[Any, list[float]], query: list[float]) -> list[tuple[float, Any]]:
|
||||
"""The EXACT cosine top-:data:`TOP_K` of *stored* for *query*.
|
||||
|
||||
Mirrors pgvector's ``vector_cosine_distance``: three float8 accumulators
|
||||
(``dot``, ``norm_a``, ``norm_b``) over the components left to right, then
|
||||
``dot / sqrt(norm_a * norm_b)``. The norms are recomputed from the stored
|
||||
components rather than assumed to be 1, because the stored component is the
|
||||
float4 of a normalised double.
|
||||
|
||||
Pure Python (no numpy in this project — A3 keeps the dependency list
|
||||
closed): ``sum(map(mul, …))`` is ~25 µs per 768-dim dot product, so 30
|
||||
queries over 5,000 rows cost ~3 s. The one residual inaccuracy is the
|
||||
read-back itself: pgvector prints each float4 in shortest-round-trip form,
|
||||
so a component comes back within ~1 ulp of the float4 (measured max
|
||||
7.4e-09 absolute over the fixture corpus) — three orders below the 1e-6
|
||||
this file pins.
|
||||
"""
|
||||
q = [_f32(v) for v in query]
|
||||
q_norm = math.sqrt(sum(map(operator.mul, q, q)))
|
||||
scored = [
|
||||
(
|
||||
sum(map(operator.mul, vec, q)) / (math.sqrt(sum(map(operator.mul, vec, vec))) * q_norm),
|
||||
chunk_id,
|
||||
)
|
||||
for chunk_id, vec in stored.items()
|
||||
]
|
||||
scored.sort(key=lambda item: -item[0])
|
||||
return scored[:TOP_K]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Capturing the tier's own statement (the plan pin must explain THAT SQL)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _StatementRecorder:
|
||||
"""Records what :func:`_vector_candidates` sends, answering with no rows."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.statements: list[Any] = []
|
||||
|
||||
def execute(self, statement: Any, params: Any = None) -> _StatementRecorder:
|
||||
self.statements.append(statement)
|
||||
return self
|
||||
|
||||
def all(self) -> list[Any]:
|
||||
return []
|
||||
|
||||
|
||||
def _vector_statement_sql() -> str:
|
||||
"""The tier's cosine SELECT as one literal SQL string, ready for EXPLAIN.
|
||||
|
||||
A hand-written copy would drift the moment the retriever changes and would
|
||||
then pin SQL no request ever runs, so the statement OBJECT is captured.
|
||||
``literal_binds`` gives the planner a real vector constant
|
||||
(``pgvector.sqlalchemy.VECTOR`` ships a literal processor, and the resulting
|
||||
unknown string literal coerces to ``vector`` through the ``<=>`` operator's
|
||||
own signature). The ``SET LOCAL`` line is deliberately not part of it:
|
||||
``EXPLAIN`` takes one statement, and that line tunes the search depth, not
|
||||
the shape of the plan.
|
||||
"""
|
||||
recorder = _StatementRecorder()
|
||||
_vector_candidates(recorder, [0.0] * DIM, TOP_K) # pyright: ignore[reportArgumentType]
|
||||
set_local, select_statement = recorder.statements
|
||||
assert str(set_local) == f"SET LOCAL hnsw.ef_search = {EF_PIN}", (
|
||||
f"the tier's ef_search line changed ({set_local}) — the capture must be the SELECT"
|
||||
)
|
||||
sql = str(
|
||||
select_statement.compile(
|
||||
dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}
|
||||
)
|
||||
)
|
||||
assert "<=>" in sql, f"the cosine operator is gone from the tier: {sql[:200]}"
|
||||
return sql
|
||||
|
||||
|
||||
def _plan(session: Session, sql: str) -> list[dict[str, Any]]:
|
||||
"""Every node of ``EXPLAIN (FORMAT JSON)`` for *sql*, flattened."""
|
||||
document = session.execute(text(f"EXPLAIN (FORMAT JSON) {sql}")).scalar()
|
||||
assert isinstance(document, list) and document, f"no plan for {sql[:120]}: {document!r}"
|
||||
nodes: list[dict[str, Any]] = []
|
||||
|
||||
def walk(node: dict[str, Any]) -> None:
|
||||
nodes.append(node)
|
||||
for child in node.get("Plans", []):
|
||||
walk(child)
|
||||
|
||||
walk(document[0]["Plan"])
|
||||
return nodes
|
||||
|
||||
|
||||
def _describe(nodes: list[dict[str, Any]]) -> str:
|
||||
return ", ".join(
|
||||
f"{node.get('Node Type')}[{node.get('Index Name') or node.get('Relation Name') or '?'}]"
|
||||
for node in nodes
|
||||
)
|
||||
|
||||
|
||||
def _uses_index(nodes: list[dict[str, Any]], index_name: str) -> bool:
|
||||
return any(
|
||||
"Index Scan" in str(node.get("Node Type", "")) and node.get("Index Name") == index_name
|
||||
for node in nodes
|
||||
)
|
||||
|
||||
|
||||
def _seq_scans(nodes: list[dict[str, Any]], relation: str) -> bool:
|
||||
return any(
|
||||
node.get("Node Type") == "Seq Scan" and node.get("Relation Name") == relation
|
||||
for node in nodes
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture: seed unindexed, build through 0029, read the corpus back once
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def hnsw_corpus() -> Iterator[dict[str, Any]]:
|
||||
"""Seed the corpus, build the index with migration 0029, precompute exact.
|
||||
|
||||
``0029 → 0028`` → seed → ``0029``: seeding with the index present would pay
|
||||
a graph insert per row, and building after the load is what the operator
|
||||
does on the production box anyway. Teardown deletes ONLY this module's rows
|
||||
(``source = FIXTURE_SOURCE``; the chunk rows go with them by the DB-level
|
||||
cascade — AGENTS rule 12, no chunk hydration) and leaves the dev DB at head
|
||||
with the index valid, exactly as it was found.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
session = SessionLocal()
|
||||
config = Config() # no alembic.ini — env.py reads the URL from app config
|
||||
config.set_main_option("script_location", "alembic")
|
||||
try:
|
||||
preexisting = session.execute(text(f"SELECT count(*) FROM {RELATION}")).scalar() or 0
|
||||
if preexisting > MAX_RELATION_ROWS:
|
||||
pytest.skip(
|
||||
f"{RELATION} holds {preexisting} rows: the exact reference is a Python "
|
||||
"scan, so these pins need the small dev corpus. Production scale "
|
||||
"is task 04's live battery."
|
||||
)
|
||||
corpus, queries = _generate()
|
||||
session.rollback() # no open snapshot while the CONCURRENTLY DDL runs
|
||||
command.downgrade(config, "0028")
|
||||
session.rollback()
|
||||
_seed(session, corpus)
|
||||
session.rollback()
|
||||
command.upgrade(config, "0029") # the ONE definition of the index
|
||||
session.rollback()
|
||||
# Planner statistics for the plan pin: a freshly loaded relation has
|
||||
# none, and a row estimate of 0 makes "seq scan + top-N sort" look free.
|
||||
session.execute(text(f"ANALYZE {RELATION}"))
|
||||
session.commit()
|
||||
rows = session.execute(select(Chunk.id, Chunk.embedding)).all()
|
||||
stored: dict[Any, list[float]] = {row[0]: row[1] or [] for row in rows}
|
||||
assert len(stored) >= preexisting + CORPUS_CHUNKS, (
|
||||
f"expected {CORPUS_CHUNKS} fixture rows on top of {preexisting}, "
|
||||
f"the relation holds {len(stored)}"
|
||||
)
|
||||
# The exact sets are the expensive half (30 × N dot products); the two
|
||||
# recall pins differ only in the ef_search they ask the index for.
|
||||
yield {
|
||||
"queries": queries,
|
||||
"exact": [_exact_top(stored, query) for query in queries],
|
||||
"chunk_ids": set(stored),
|
||||
"plan_sql": _vector_statement_sql(),
|
||||
}
|
||||
finally:
|
||||
session.rollback()
|
||||
session.execute(
|
||||
text("DELETE FROM documents WHERE source = :src"), {"src": FIXTURE_SOURCE}
|
||||
)
|
||||
session.commit()
|
||||
session.rollback()
|
||||
command.upgrade(config, "head") # the dev DB never stays below head
|
||||
session.close()
|
||||
|
||||
|
||||
def _stub_settings(**kwargs: object) -> Settings:
|
||||
"""A :class:`Settings` off the env file (the ``test_chunks_hnsw_index.py``
|
||||
helper's shape — an operator's shell must not tune these pins)."""
|
||||
return Settings(_env_file=None, **kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _force_index_scan(session: Session) -> None:
|
||||
"""Put the planner on the index, because a recall pin has to measure the
|
||||
index and not the planner's preference.
|
||||
|
||||
Postgres costs an HNSW scan partly on ``ef_search``, so on a relation this
|
||||
small it can legitimately prefer a seq scan + top-N sort — and a recall run
|
||||
that did that would score the EXACT scan twice and report a perfect 1.0000
|
||||
without visiting a single graph node. That is not hypothetical: the live
|
||||
gate (``benchmarks/phase188_live_gate/``) watched exactly this happen on the
|
||||
8,866-chunk replica, where the planner used the index at ``ef_search = 40``
|
||||
and refused it at 100.
|
||||
|
||||
``enable_seqscan = off`` is Postgres' own "take any other plan if one
|
||||
exists" lever. It is set at SESSION scope because ``_vector_candidates``
|
||||
opens its own transaction, so a ``SET LOCAL`` issued here would die with the
|
||||
caller's transaction instead of reaching the tier. Every caller resets it in
|
||||
a ``finally`` — a leaked ``off`` would slow every later test on this pooled
|
||||
connection — and the assertion beside it proves the lever took effect.
|
||||
"""
|
||||
session.rollback()
|
||||
session.execute(text("SET enable_seqscan = off"))
|
||||
|
||||
|
||||
def _release_index_scan(session: Session) -> None:
|
||||
session.rollback()
|
||||
session.execute(text("RESET enable_seqscan"))
|
||||
session.rollback()
|
||||
|
||||
|
||||
def _set_ef_search(monkeypatch: pytest.MonkeyPatch, value: int) -> None:
|
||||
"""Drive the tier at *value* the way an operator does — the setting."""
|
||||
settings = _stub_settings(retrieval_ef_search=value)
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pin 1 — the planner uses the index (with its vacuity guard)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plan_uses_the_hnsw_index(db: Session, hnsw_corpus: dict[str, Any]) -> None:
|
||||
"""The 2026-10-15 incident WAS a plan, so the pin is a plan: the tier's own
|
||||
``ORDER BY embedding <=> <vector> LIMIT 100`` runs through
|
||||
``ix_chunks_embedding_hnsw`` and never reads ``chunks`` sequentially.
|
||||
|
||||
Explaining the captured statement is also what proves D1's "no query
|
||||
rewrite": the ``<=>`` the retriever already emits is exactly the operator
|
||||
``vector_cosine_ops`` serves.
|
||||
"""
|
||||
nodes = _plan(db, hnsw_corpus["plan_sql"])
|
||||
assert _uses_index(nodes, INDEX_NAME), (
|
||||
f"{INDEX_NAME} is not in the plan — the incident is back: [{_describe(nodes)}]"
|
||||
)
|
||||
assert not _seq_scans(nodes, RELATION), (
|
||||
f"a sequential scan of {RELATION} survived the index: [{_describe(nodes)}]"
|
||||
)
|
||||
db.rollback()
|
||||
|
||||
|
||||
def test_plan_without_the_index_seq_scans_the_relation(
|
||||
db: Session, hnsw_corpus: dict[str, Any]
|
||||
) -> None:
|
||||
"""Vacuity guard: the same statement, same session, same statistics, with
|
||||
index scans refused, MUST fall back to ``Seq Scan on chunks``.
|
||||
|
||||
``SET LOCAL enable_indexscan = off`` rather than dropping the index: a
|
||||
``DROP INDEX CONCURRENTLY`` + rebuild would cost a second graph build and
|
||||
leave the corpus unindexed if the rebuild failed, while this GUC is
|
||||
transaction-scoped (it dies with the rollback) and removes exactly the
|
||||
access path under test. Without this test the plan pin would pass vacuously
|
||||
on any plan that merely mentions some index.
|
||||
"""
|
||||
sql = hnsw_corpus["plan_sql"]
|
||||
db.rollback()
|
||||
db.execute(text("SET LOCAL enable_indexscan = off"))
|
||||
db.execute(text("SET LOCAL enable_indexonlyscan = off"))
|
||||
nodes = _plan(db, sql)
|
||||
assert _seq_scans(nodes, RELATION), (
|
||||
f"the vacuity guard itself did not seq-scan {RELATION}: [{_describe(nodes)}]"
|
||||
)
|
||||
assert not _uses_index(nodes, INDEX_NAME), (
|
||||
f"{INDEX_NAME} was used with index scans disabled: [{_describe(nodes)}]"
|
||||
)
|
||||
db.rollback()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pin 2 — the approximate top-100 IS the exact top-100
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_top100_recall_at_the_shipped_ef_search(
|
||||
db: Session,
|
||||
hnsw_corpus: dict[str, Any],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""At the shipped ``ef_search`` the index returns the exact top-100 for
|
||||
every query, exact to 1e-6 on the score the honesty gate reads.
|
||||
|
||||
Two different failures surface here: a ``top_score`` that moved — A8's
|
||||
deflect/answer verdict flips on that one number, so it is pinned per row,
|
||||
not per query — and an overlap below the floor, i.e. a candidate that never
|
||||
reaches RRF. The failure message carries every query's overlap so a future
|
||||
``ef_search`` raise has its evidence in the report instead of in a re-run.
|
||||
"""
|
||||
queries: list[list[float]] = hnsw_corpus["queries"]
|
||||
exact: list[list[tuple[float, Any]]] = hnsw_corpus["exact"]
|
||||
chunk_ids: set[Any] = hnsw_corpus["chunk_ids"]
|
||||
_set_ef_search(monkeypatch, EF_PIN)
|
||||
|
||||
_force_index_scan(db)
|
||||
try:
|
||||
depth_nodes = _plan(db, hnsw_corpus["plan_sql"])
|
||||
assert _uses_index(depth_nodes, INDEX_NAME), (
|
||||
f"this recall run at ef_search={EF_PIN} did not use {INDEX_NAME} even "
|
||||
"with sequential scans refused — the pin would be measuring a scan "
|
||||
f"and calling it parity: [{_describe(depth_nodes)}]"
|
||||
)
|
||||
|
||||
overlaps: list[float] = []
|
||||
worst_delta = 0.0
|
||||
for index, (query, exact_rows) in enumerate(zip(queries, exact, strict=True)):
|
||||
exact_ids = {chunk_id for _, chunk_id in exact_rows}
|
||||
exact_scores = {chunk_id: cosine for cosine, chunk_id in exact_rows}
|
||||
candidates = _vector_candidates(db, query, TOP_K)
|
||||
db.rollback() # end the turn's transaction, SET LOCAL included
|
||||
assert len(candidates) == TOP_K, (
|
||||
f"query {index}: the tier returned {len(candidates)} candidates, "
|
||||
f"not the {TOP_K} the A7 cosine tier promises"
|
||||
)
|
||||
got_ids = {candidate.chunk_id for candidate in candidates}
|
||||
assert got_ids <= chunk_ids, (
|
||||
f"query {index}: a candidate is outside the corpus the exact "
|
||||
"reference was computed over"
|
||||
)
|
||||
overlaps.append(len(exact_ids & got_ids) / TOP_K)
|
||||
for candidate in candidates:
|
||||
exact_cosine = exact_scores.get(candidate.chunk_id)
|
||||
if exact_cosine is not None:
|
||||
worst_delta = max(worst_delta, abs(candidate.cosine - exact_cosine))
|
||||
finally:
|
||||
_release_index_scan(db)
|
||||
|
||||
mean = sum(overlaps) / len(overlaps)
|
||||
below = [(i, overlap) for i, overlap in enumerate(overlaps) if overlap < RECALL_FLOOR]
|
||||
assert not below, (
|
||||
f"top-{TOP_K} recall below {RECALL_FLOOR} at ef_search={EF_PIN} for "
|
||||
f"{len(below)}/{len(overlaps)} queries: {below}; per-query overlap="
|
||||
f"{[round(overlap, 4) for overlap in overlaps]} mean={mean:.4f} — the "
|
||||
"remedy is a higher BOR_RETRIEVAL_EF_SEARCH, never this floor (LOCKED A3)"
|
||||
)
|
||||
assert worst_delta < COSINE_TOLERANCE, (
|
||||
f"1 - distance drifted from the exact cosine by {worst_delta:.3e} "
|
||||
f"(tolerance {COSINE_TOLERANCE:.0e}) — top_score would not be exact"
|
||||
)
|
||||
|
||||
|
||||
def test_recall_never_below_the_pinned_floor_when_ef_search_raised(
|
||||
db: Session,
|
||||
hnsw_corpus: dict[str, Any],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Monotonicity sanity: a deeper search cannot recover FEWER of the exact
|
||||
top-100 than a shallower one.
|
||||
|
||||
This is the property the operator leans on when the live battery (task 04)
|
||||
says "raise it" — ``ef_search`` is a recall dial, not a lottery. A
|
||||
regression here means the value never reaches the server (the ``SET LOCAL``
|
||||
of task 02), which is exactly what the shipped 400 would then silently not
|
||||
do. ``EF_RAISED`` sits ABOVE the shipped value, so the probe still measures
|
||||
a deeper search after task 04 moved the default.
|
||||
"""
|
||||
exact: list[list[tuple[float, Any]]] = hnsw_corpus["exact"]
|
||||
queries: list[list[float]] = hnsw_corpus["queries"]
|
||||
|
||||
def mean_overlap(ef_search: int) -> float:
|
||||
_set_ef_search(monkeypatch, ef_search)
|
||||
total = 0.0
|
||||
for query, exact_rows in zip(queries, exact, strict=True):
|
||||
exact_ids = {chunk_id for _, chunk_id in exact_rows}
|
||||
candidates = _vector_candidates(db, query, TOP_K)
|
||||
db.rollback()
|
||||
total += len(exact_ids & {candidate.chunk_id for candidate in candidates}) / TOP_K
|
||||
return total / len(exact)
|
||||
|
||||
pinned = mean_overlap(EF_PIN)
|
||||
raised = mean_overlap(EF_RAISED)
|
||||
assert raised >= pinned, (
|
||||
f"ef_search={EF_RAISED} recovered less of the exact top-{TOP_K} "
|
||||
f"({raised:.4f}) than ef_search={EF_PIN} ({pinned:.4f})"
|
||||
)
|
||||
@@ -46,6 +46,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -48,6 +48,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -50,6 +50,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -57,6 +57,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -60,6 +60,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -52,6 +52,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -52,6 +52,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -58,6 +58,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -68,6 +68,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -48,6 +48,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -58,6 +58,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -53,6 +53,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
@@ -59,6 +59,12 @@ def alembic(db: Session) -> Iterator[Config]:
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
|
||||
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
|
||||
# older-snapshot transaction — an unreleased session would park
|
||||
# the build forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user