brain-of-reese/.agents/phases/complete/188_chunks_hnsw_index/00_phase.md

Phase 188 — The vector search stops full-scanning 414,220 embeddings

Source: found while investigating the 2026-10-15 memory incident — pg_indexes for chunks lists chunks_pkey, ix_chunks_document_id, ix_chunks_tsv (GIN) and no index on embedding, so every chat turn runs ORDER BY embedding <=> :vec LIMIT 100 (A7's cosine tier) as a sequential scan of a 3 GB relation: per-turn CPU on the interactive path, plus 3 GB of page-cache churn on a 3,904 MB box that the sync also needs. Story: n/a (incident-adjacent, owner directive 2026-10-15) Depends on: none functionally; LAST of the six incident phases — it is the only one that touches the schema and the retrieval quality contract, so it lands after the sync's memory work is in.

Context: .agents/PLAN.md §2 — A3 (PostgreSQL 17 + pgvector, cosine <=>, one system), A6 (dim 768, fixed at table creation, a mismatch fails loudly), A7 (hybrid: cosine top-100 ∪ FTS top-30 → RRF k=60 → full text of the top-N=2 documents), A8 (the honesty gate: deflect only when best cosine < BOR_RELEVANCE_THRESHOLD 0.62 AND (zero FTS hits OR best cosine < BOR_LEXICAL_SUPPORT_FLOOR 0.35)), A13 (every migration ships a TESTED downgrade), A16 (per-phase unit + integration + >90% + one dedicated E2E file), A21 (LLM calls are free; the interactive path's latency is what matters); app/models.py — Chunk.embedding: Mapped[list[float] | None] = mapped_column(Vector(EMBEDDING_DIM)) (L241); app/rag/retriever.py — _vector_candidates(db, question_embedding, limit) (L431) building distance = Chunk.embedding.cosine_distance(question_embedding) (L440) .where(Chunk.embedding.is_not(None)) (L444) .order_by(distance).limit(limit) (L446), the fusion caller at L690–711 (vector = _vector_candidates(db, question_embedding, v_n)), _lexical_candidates (L632); app/config.py (the BOR_ Settings pattern, relevance_threshold / lexical_support_floor / suggested_docs as the retrieval-knob neighbours); app/core/settings_catalog.py — GROUPS (L72, the "Retrieval & honesty" group) + the set(CATALOG) == set(Settings.model_fields) completeness pin; alembic/versions/ — head is 0028 (uv run alembic heads), so this phase adds 0029; db/Containerfile — ARG PGVECTOR_VERSION=v0.8.0 (HNSW + halfvec available; verified 0.8.0 on both the dev and the production instance); scripts/eval_retrieval.py (+ tests/fixtures/retrieval_battery.txt, the phase-119/162 regression battery — uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt, AIPI_KEY required); benchmarks/model_benchmarks.csv (the recorded-measurement home); tests/e2e/test_retrieval_quality.py, tests/e2e/test_search_recall_small_sources.py, tests/e2e/test_chat_rag.py, tests/unit/test_retriever.py, tests/unit/test_retriever_recency.py.

Production numbers (read 2026-10-15, read-only)

414,220 chunks, all 414,220 with a non-null embedding, chunks relation ≈ 3,000 MB, database 3,147 MB, shared_buffers=128MB, work_mem=4MB, host 3,904 MB with no swap. pgvector 0.8.0.

Objective

Add an HNSW cosine index on chunks.embedding through a reversible migration built CONCURRENTLY (A3's one-system rule respected — no new service, no new dependency), give the retrieval tier an explicit hnsw.ef_search knob that can never be set below the A7 top-100 candidate count, and prove with a deterministic recall pin plus the live retrieval battery that the approximate index does not move the top hits, the top_score, or the honesty-gate verdicts.

Design (shared by all tasks — the executor reads this, not the chat)

• D1 — the index is on the existing vector(768) column, vector_cosine_ops (task 01). CREATE INDEX CONCURRENTLY … USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64) — the pgvector defaults' neighbourhood, which keeps the graph buildable on a 4 GB box. halfvec(768) (half the index size) was considered and REJECTED for this phase (A2): it needs embedding::halfvec(768) in _vector_candidates AND a quantisation-recall study, i.e. two quality risks in one phase. The index is smaller-with-halfvec; if the prod build does not fit, that is a follow-up phase, not a silent compromise here.
• D2 — CONCURRENTLY needs autocommit, and an interrupted build leaves an INVALID index (task 01). Migration alembic/versions/0029_chunks_embedding_hnsw.py runs its DDL inside with op.get_context().autocommit_block(): and, before creating, drops any existing INVALID index of that name (select indisvalid from pg_index join pg_class … → DROP INDEX CONCURRENTLY IF EXISTS ix_chunks_embedding_hnsw when indisvalid is false), then re-checks indisvalid afterwards and raises if the build did not validate — because a CONCURRENTLY build that dies (a 3,904 MB box is exactly where it dies) leaves an index that costs insert time and buys nothing, silently. Upgrade and downgrade both run in the autocommit block; the downgrade is DROP INDEX CONCURRENTLY IF EXISTS … and is TESTED (A13) by alembic downgrade -1 / upgrade head in the task's verification.
• D3 — ef_search is explicit and clamped (task 02). retrieval_ef_search: int = 100 (BOR_RETRIEVAL_EF_SEARCH, "Retrieval & honesty" catalogue group): _vector_candidates issues SET LOCAL hnsw.ef_search = :ef (per-transaction — no leakage between requests, no global GUC change; A12 keeps state where it belongs) immediately before the SELECT, and a value below the A7 candidate count is CLAMPED UP to it with one WARNING line (retrieval: ef_search=%d raised to %d (A7 candidate count)) rather than silently degrading recall. Negative values refuse startup the way the phase-182 show_text_max_* knobs do.
• D4 — recall is pinned twice: deterministic in CI, live against the battery (tasks 03 + 04).
  • CI (deterministic, no LLM): seed ~5,000 chunks with fixed-seed 768-dim vectors, compute the EXACT cosine top-100 in Python (numpy-free is fine — a pure-Python dot product over the seeded corpus, or reuse the existing helper if tests/ already has one), then assert the index-backed query's top-100 set overlaps the exact set by ≥ 0.99 at ef_search=100 and that 1 - distance (top_score) matches the exact value to 6 decimals for the returned rows. Plus the plan pin: EXPLAIN (FORMAT JSON) for the real _vector_candidates statement names ix_chunks_embedding_hnsw (and does NOT show Seq Scan on chunks).
  • Live (A8's honesty contract): the retrieval battery, index OFF vs ON, on the 1,000-document live-replica KB (the restore-test-db skill) with AIPI_KEY set — every question's top_score, top-5 document set and deflection verdict must be identical (a difference means RAISING ef_search, never accepting it). The before/after numbers are recorded in benchmarks/ and the phase report; the build's wall time and the box's free RAM before/after are recorded too (they are the deploy-runbook numbers).
• D5 — the build is an operator step with a runbook (task 01 + README). The migration is correct, but building HNSW over 414k × 768 on the production 3,904 MB box needs a documented pre-flight: free-RAM check, maintenance_work_mem set explicitly for the session (e.g. 512 MB, never left at the 4 MB work_mem default), the build run against the postgres container directly (podman exec … psql) if the app container is memory-limited (the owner's separate task), autovacuum left running afterwards, and the INVALID-index recovery path. README.md gains that runbook section; the phase's own CI build runs against the small test corpus, not 414k rows.
• NOT touched: the embedding dim (A6), the <=> operator and the hybrid fusion (A7), the thresholds themselves (A8 — unchanged unless the battery forces an ef_search raise), the tsv GIN index, chunks/documents columns, the importer/sync (phases 183–186), the frontend, any new dependency or service (A3/A12).

Dependencies

— (none; deliberately the LAST of the six incident phases — schema + retrieval-quality risk after the sync's memory work)

Tasks

1. 01_hnsw_migration.md — migration 0029 (autocommit block, CONCURRENTLY, the INVALID-index pre-drop + post-validate), the tested downgrade, the migration tests.
2. 02_ef_search_setting.md — retrieval_ef_search on Settings + the catalogue group + .env.example, SET LOCAL in _vector_candidates, the clamp-and-warn rule, the negative-refuses-startup pin.
3. 03_plan_and_recall_pins.md — the EXPLAIN plan pin (index used, no seq scan) and the deterministic ≥0.99 overlap / exact-top_score recall pin on a fixed-seed corpus.
4. 04_recall_battery_gate.md — the live battery index-off vs index-on on the 1,000-doc replica KB; the recorded numbers in benchmarks/ + the phase report; the ef_search raise loop if anything moved.
5. 05_e2e_index_chat_and_runbook.md — tests/e2e/test_vector_index_recall.py (the phase's dedicated Playwright suite), the README.md build runbook, the full gates, the docs pass, the atomic commit.

Testing & Quality

• Unit: the retrieval_ef_search default/clamp/negative rules; the migration module's helpers (the INVALID-index decision function is a pure function over a query result, so it unit-tests without a DB).
• Integration: tests/integration/test_chunks_hnsw_index.py — upgrade/downgrade round-trip on the test DB, pg_index.indisvalid asserted, the INVALID-index recovery path exercised by creating an INVALID index deliberately (… WHERE false won't produce INVALID; use the documented UPDATE pg_index SET indisvalid=false superuser trick or a deliberately interrupted build — pick the one the test DB allows and document it), the EXPLAIN plan pin, the deterministic recall pin. tests/unit/test_retriever.py + test_retriever_recency.py + tests/integration/test_chat_db_sessions.py green with SET LOCAL in play.
• Live gate: the retrieval battery off/on, identical verdicts, recorded.
• E2E (AGENTS rule 4, task 05): tests/e2e/test_vector_index_recall.py, run in isolation.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).
• Lint/types: uv run ruff check . && uv run pyright.

Documentation & Consistency

• Final pass reviews README.md and AGENTS.md in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
• README.md: the 0029 deploy section (the D5 build runbook: pre-flight free RAM, maintenance_work_mem, CONCURRENTLY semantics, the INVALID-index recovery, expected build time from task 04's measurement) and the BOR_RETRIEVAL_EF_SEARCH knob (default 100, the clamp, "raise it if recall matters"). AGENTS.md: the quick-reference gains nothing new unless a command changes — the retrieval battery line already exists; keep it accurate.

Completion Criteria

• alembic/versions/0029_… creates ix_chunks_embedding_hnsw CONCURRENTLY inside an autocommit block, pre-drops an INVALID index of that name, post-validates indisvalid, and its downgrade is tested (upgrade head → downgrade -1 → upgrade head green on the test DB, A13).
• retrieval_ef_search (default 100) is applied with SET LOCAL in _vector_candidates, clamped up with a WARNING below the A7 candidate count, negative refuses startup, and it appears in CATALOG ("Retrieval & honesty") + .env.example + README.md.
• The EXPLAIN pin shows the index used and no seq scan; the deterministic recall pin shows ≥0.99 top-100 overlap and exact top_scores.
• The live battery ran index-off and index-on over the 1,000-doc replica with IDENTICAL top scores, top-5 sets and deflection verdicts, and the numbers (plus build time + free-RAM readings) are recorded in benchmarks/ and the phase report.
• uv run pytest green, app/ coverage >90%, ruff + pyright clean.
• uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov green in isolation.
• README.md and AGENTS.md accurate and consistent with the whole project (the runbook included).
• One atomic --no-gpg-sign Conventional Commit (e.g. perf(retrieval): index chunks.embedding with HNSW and pin recall so cosine stops full-scanning 414k vectors); the phase dir moves to .agents/phases/complete/.

Locked decisions (owner-confirmed 2026-10-15)

• A1 — HNSW on vector(768) with vector_cosine_ops, built CONCURRENTLY. No query rewrite, no new dependency, <=> uses it directly (A3).
• A2 — halfvec quantisation is NOT part of this phase (it needs a statement cast + its own recall study); revisit only if the production build does not fit in RAM.
• A3 — recall is a gate, not a hope. The honesty gate (A8) reads top_score; an approximate index that changes a top hit can flip a deflect. The deterministic ≥0.99 pin AND the live battery must pass, and the remedy is a higher ef_search, never a relaxed assertion.
• A4 — SET LOCAL, per transaction. No global GUC mutation, no connection-pool-wide state (A12: no aux services, restart clears state).
• A5 — the production build is an operator action with a runbook, executed after the owner's container-memory work lands; the phase's CI never builds a 414k-row graph.