Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 183_sync_passive_delete_cascade
Build and Push Containers / build-and-push-app (push) Successful in 2m7s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
All gates green. Final state verified.

**Implemented / verified**
- Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after.
- Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings.
- **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`).

**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%)
- `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors

**Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands).

**Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit).

**Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify.

**Next pending phase**: `184_sync_bounded_session`.
This commit is contained in:
ducoterra
2026-10-15 16:00:28 -04:00
parent b34b8ca672
commit 7b5ea7c2ee
52 changed files with 3044 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/188_chunks_hnsw_index/00_phase.md
+62
View File
@@ -0,0 +1,62 @@
# Phase 188 — The vector search stops full-scanning 414,220 embeddings
**Source:** found while investigating the 2026-10-15 memory incident — `pg_indexes` for `chunks` lists `chunks_pkey`, `ix_chunks_document_id`, `ix_chunks_tsv` (GIN) and **no index on `embedding`**, so every chat turn runs `ORDER BY embedding <=> :vec LIMIT 100` (A7's cosine tier) as a sequential scan of a 3 GB relation: per-turn CPU on the interactive path, plus 3 GB of page-cache churn on a 3,904 MB box that the sync also needs.
**Story:** n/a (incident-adjacent, owner directive 2026-10-15)
**Depends on:** none functionally; LAST of the six incident phases — it is the only one that touches the schema and the retrieval quality contract, so it lands after the sync's memory work is in.
**Context:** `.agents/PLAN.md` §2 — **A3** (PostgreSQL 17 + pgvector, cosine `<=>`, one system), **A6** (dim **768**, fixed at table creation, a mismatch fails loudly), **A7** (hybrid: **cosine top-100** ∪ FTS top-30 → RRF k=60 → full text of the top-N=2 documents), **A8** (the honesty gate: deflect only when best cosine < `BOR_RELEVANCE_THRESHOLD` 0.62 AND (zero FTS hits OR best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` 0.35)), **A13** (every migration ships a TESTED downgrade), **A16** (per-phase unit + integration + >90% + one dedicated E2E file), **A21** (LLM calls are free; the interactive path's latency is what matters); `app/models.py` — `Chunk.embedding: Mapped[list[float] | None] = mapped_column(Vector(EMBEDDING_DIM))` (**L241**); `app/rag/retriever.py` — **`_vector_candidates(db, question_embedding, limit)` (L431)** building `distance = Chunk.embedding.cosine_distance(question_embedding)` (**L440**) `.where(Chunk.embedding.is_not(None))` (**L444**) `.order_by(distance).limit(limit)` (**L446**), the fusion caller at **L690–711** (`vector = _vector_candidates(db, question_embedding, v_n)`), `_lexical_candidates` (L632); `app/config.py` (the `BOR_` `Settings` pattern, `relevance_threshold` / `lexical_support_floor` / `suggested_docs` as the retrieval-knob neighbours); `app/core/settings_catalog.py` — `GROUPS` (**L72**, the `"Retrieval & honesty"` group) + the `set(CATALOG) == set(Settings.model_fields)` completeness pin; `alembic/versions/` — head is **`0028`** (`uv run alembic heads`), so this phase adds **`0029`**; `db/Containerfile` — **`ARG PGVECTOR_VERSION=v0.8.0`** (HNSW + `halfvec` available; verified 0.8.0 on both the dev and the production instance); `scripts/eval_retrieval.py` (+ `tests/fixtures/retrieval_battery.txt`, the phase-119/162 regression battery — `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`, `AIPI_KEY` required); `benchmarks/model_benchmarks.csv` (the recorded-measurement home); `tests/e2e/test_retrieval_quality.py`, `tests/e2e/test_search_recall_small_sources.py`, `tests/e2e/test_chat_rag.py`, `tests/unit/test_retriever.py`, `tests/unit/test_retriever_recency.py`.
## Production numbers (read 2026-10-15, read-only)
414,220 chunks, **all 414,220 with a non-null `embedding`**, `chunks` relation ≈ **3,000 MB**, database 3,147 MB, `shared_buffers=128MB`, `work_mem=4MB`, host 3,904 MB with **no swap**. pgvector **0.8.0**.
## Objective
Add an HNSW cosine index on `chunks.embedding` through a reversible migration built `CONCURRENTLY` (A3's one-system rule respected — no new service, no new dependency), give the retrieval tier an explicit `hnsw.ef_search` knob that can never be set below the A7 top-100 candidate count, and prove with a deterministic recall pin plus the live retrieval battery that the approximate index does not move the top hits, the `top_score`, or the honesty-gate verdicts.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the index is on the existing `vector(768)` column, `vector_cosine_ops` (task 01).** `CREATE INDEX CONCURRENTLY … USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)` — the pgvector defaults' neighbourhood, which keeps the graph buildable on a 4 GB box. `halfvec(768)` (half the index size) was considered and REJECTED for this phase (A2): it needs `embedding::halfvec(768)` in `_vector_candidates` AND a quantisation-recall study, i.e. two quality risks in one phase. The index is smaller-with-halfvec; if the prod build does not fit, that is a follow-up phase, not a silent compromise here.
- **D2 — `CONCURRENTLY` needs autocommit, and an interrupted build leaves an INVALID index (task 01).** Migration `alembic/versions/0029_chunks_embedding_hnsw.py` runs its DDL inside `with op.get_context().autocommit_block():` and, before creating, drops any existing INVALID index of that name (`select indisvalid from pg_index join pg_class …` → `DROP INDEX CONCURRENTLY IF EXISTS ix_chunks_embedding_hnsw` when `indisvalid` is false), then re-checks `indisvalid` afterwards and raises if the build did not validate — because a `CONCURRENTLY` build that dies (a 3,904 MB box is exactly where it dies) leaves an index that costs insert time and buys nothing, silently. Upgrade and downgrade both run in the autocommit block; the downgrade is `DROP INDEX CONCURRENTLY IF EXISTS …` and is TESTED (A13) by `alembic downgrade -1` / `upgrade head` in the task's verification.
- **D3 — `ef_search` is explicit and clamped (task 02).** `retrieval_ef_search: int = 100` (`BOR_RETRIEVAL_EF_SEARCH`, `"Retrieval & honesty"` catalogue group): `_vector_candidates` issues `SET LOCAL hnsw.ef_search = :ef` (per-transaction — no leakage between requests, no global GUC change; A12 keeps state where it belongs) immediately before the SELECT, and a value below the A7 candidate count is CLAMPED UP to it with one WARNING line (`retrieval: ef_search=%d raised to %d (A7 candidate count)`) rather than silently degrading recall. Negative values refuse startup the way the phase-182 `show_text_max_*` knobs do.
- **D4 — recall is pinned twice: deterministic in CI, live against the battery (tasks 03 + 04).**
- **CI (deterministic, no LLM):** seed ~5,000 chunks with fixed-seed 768-dim vectors, compute the EXACT cosine top-100 in Python (`numpy`-free is fine — a pure-Python dot product over the seeded corpus, or reuse the existing helper if `tests/` already has one), then assert the index-backed query's top-100 set overlaps the exact set by **≥ 0.99** at `ef_search=100` and that `1 - distance` (`top_score`) matches the exact value to 6 decimals for the returned rows. Plus the plan pin: `EXPLAIN (FORMAT JSON)` for the real `_vector_candidates` statement names `ix_chunks_embedding_hnsw` (and does NOT show `Seq Scan on chunks`).
- **Live (A8's honesty contract):** the retrieval battery, index OFF vs ON, on the 1,000-document live-replica KB (the `restore-test-db` skill) with `AIPI_KEY` set — every question's `top_score`, top-5 document set and deflection verdict must be identical (a difference means RAISING `ef_search`, never accepting it). The before/after numbers are recorded in `benchmarks/` and the phase report; the build's wall time and the box's free RAM before/after are recorded too (they are the deploy-runbook numbers).
- **D5 — the build is an operator step with a runbook (task 01 + README).** The migration is correct, but building HNSW over 414k × 768 on the production 3,904 MB box needs a documented pre-flight: free-RAM check, `maintenance_work_mem` set explicitly for the session (e.g. 512 MB, never left at the 4 MB `work_mem` default), the build run against the postgres container directly (`podman exec … psql`) if the app container is memory-limited (the owner's separate task), `autovacuum` left running afterwards, and the INVALID-index recovery path. `README.md` gains that runbook section; the phase's own CI build runs against the small test corpus, not 414k rows.
- **NOT touched:** the embedding dim (A6), the `<=>` operator and the hybrid fusion (A7), the thresholds themselves (A8 — unchanged unless the battery forces an `ef_search` raise), the `tsv` GIN index, `chunks`/`documents` columns, the importer/sync (phases 183–186), the frontend, any new dependency or service (A3/A12).
## Dependencies
— (none; deliberately the LAST of the six incident phases — schema + retrieval-quality risk after the sync's memory work)
## Tasks
1. `01_hnsw_migration.md` — migration `0029` (autocommit block, `CONCURRENTLY`, the INVALID-index pre-drop + post-validate), the tested downgrade, the migration tests.
2. `02_ef_search_setting.md` — `retrieval_ef_search` on `Settings` + the catalogue group + `.env.example`, `SET LOCAL` in `_vector_candidates`, the clamp-and-warn rule, the negative-refuses-startup pin.
3. `03_plan_and_recall_pins.md` — the `EXPLAIN` plan pin (index used, no seq scan) and the deterministic ≥0.99 overlap / exact-`top_score` recall pin on a fixed-seed corpus.
4. `04_recall_battery_gate.md` — the live battery index-off vs index-on on the 1,000-doc replica KB; the recorded numbers in `benchmarks/` + the phase report; the `ef_search` raise loop if anything moved.
5. `05_e2e_index_chat_and_runbook.md` — `tests/e2e/test_vector_index_recall.py` (the phase's dedicated Playwright suite), the `README.md` build runbook, the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the `retrieval_ef_search` default/clamp/negative rules; the migration module's helpers (the INVALID-index decision function is a pure function over a query result, so it unit-tests without a DB).
- **Integration:** `tests/integration/test_chunks_hnsw_index.py` — upgrade/downgrade round-trip on the test DB, `pg_index.indisvalid` asserted, the INVALID-index recovery path exercised by creating an INVALID index deliberately (`… WHERE false` won't produce INVALID; use the documented `UPDATE pg_index SET indisvalid=false` superuser trick or a deliberately interrupted build — pick the one the test DB allows and document it), the `EXPLAIN` plan pin, the deterministic recall pin. `tests/unit/test_retriever.py` + `test_retriever_recency.py` + `tests/integration/test_chat_db_sessions.py` green with `SET LOCAL` in play.
- **Live gate:** the retrieval battery off/on, identical verdicts, recorded.
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_vector_index_recall.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the `0029` deploy section (the D5 build runbook: pre-flight free RAM, `maintenance_work_mem`, `CONCURRENTLY` semantics, the INVALID-index recovery, expected build time from task 04's measurement) and the `BOR_RETRIEVAL_EF_SEARCH` knob (default 100, the clamp, "raise it if recall matters"). `AGENTS.md`: the quick-reference gains nothing new unless a command changes — the retrieval battery line already exists; keep it accurate.
## Completion Criteria
- [ ] `alembic/versions/0029_…` creates `ix_chunks_embedding_hnsw` `CONCURRENTLY` inside an autocommit block, pre-drops an INVALID index of that name, post-validates `indisvalid`, and its downgrade is tested (`upgrade head` → `downgrade -1` → `upgrade head` green on the test DB, A13).
- [ ] `retrieval_ef_search` (default 100) is applied with `SET LOCAL` in `_vector_candidates`, clamped up with a WARNING below the A7 candidate count, negative refuses startup, and it appears in `CATALOG` (`"Retrieval & honesty"`) + `.env.example` + `README.md`.
- [ ] The `EXPLAIN` pin shows the index used and no seq scan; the deterministic recall pin shows ≥0.99 top-100 overlap and exact `top_score`s.
- [ ] The live battery ran index-off and index-on over the 1,000-doc replica with IDENTICAL top scores, top-5 sets and deflection verdicts, and the numbers (plus build time + free-RAM readings) are recorded in `benchmarks/` and the phase report.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the runbook included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(retrieval): index chunks.embedding with HNSW and pin recall so cosine stops full-scanning 414k vectors`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — HNSW on `vector(768)` with `vector_cosine_ops`, built `CONCURRENTLY`.** No query rewrite, no new dependency, `<=>` uses it directly (A3).
- **A2 — `halfvec` quantisation is NOT part of this phase** (it needs a statement cast + its own recall study); revisit only if the production build does not fit in RAM.
- **A3 — recall is a gate, not a hope.** The honesty gate (A8) reads `top_score`; an approximate index that changes a top hit can flip a deflect. The deterministic ≥0.99 pin AND the live battery must pass, and the remedy is a higher `ef_search`, never a relaxed assertion.
- **A4 — `SET LOCAL`, per transaction.** No global GUC mutation, no connection-pool-wide state (A12: no aux services, restart clears state).
- **A5 — the production build is an operator action with a runbook**, executed after the owner's container-memory work lands; the phase's CI never builds a 414k-row graph.