Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 188_chunks_hnsw_index
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Details
Build and Push Containers / build-and-push-db (push) Successful in 12s
Details

Browse Source 
**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
This commit is contained in:
ducoterra
2026-10-17 03:18:23 -04:00
parent c054b47df5
commit f20d40e4fe
81 changed files with 5076 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__00_phase.a1.err
View File
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__00_phase.a1.md
+10
View File
@@ -0,0 +1,10 @@
**Phase 188 — final verification pass: all criteria green.**
- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__00_phase.a1.validate
+145
View File
@@ -0,0 +1,145 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 231 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 325 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/memory.py 32 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 865 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 259 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6946 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__01_hnsw_migration.a1.err
View File
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__01_hnsw_migration.a1.md
+16
View File
@@ -0,0 +1,16 @@
**Task 01 complete — all green.**
**Implemented**
- `alembic/versions/0029_chunks_embedding_hnsw.py` — `ix_chunks_embedding_hnsw` (`USING hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64)`) built `CONCURRENTLY` in an autocommit block; pure helpers `should_drop_invalid`/`is_valid`; INVALID pre-drop self-heal + post-validate raise; `DROP INDEX CONCURRENTLY IF EXISTS` downgrade (A13).
- `tests/integration/test_chunks_hnsw_index.py` — create/valid/definition pin, downgrade, head re-upgrade round-trip, forced-INVALID self-heal (documented superuser `UPDATE pg_index SET indisvalid=false` trick, clean skip without catalog rights), pure helper matrices, module contract.
- **Defect found & fixed suite-wide:** psycopg3 keeps its snapshot alive while idle-in-transaction (psql does not); a session left open parks `CREATE INDEX CONCURRENTLY` phase 3 **forever** (reproduced 5×, lock graph + `pg_terminate_backend` unblock prove it). Patched the 13 pre-0028 migration-test fixtures (0004–0017, 0028) with the 0018 house `db.rollback()`-before-repair-DDL pattern and made every alembic call in the new file transaction-free.
- `README.md` — deploy note under Production Deployment (CONCURRENTLY semantics, `maintenance_work_mem`/RAM pre-flight, INVALID self-heal, the post-build/pre-stamp `already exists` recovery). AGENTS.md: no doc impact (no command/env change).
**Verification (exact results)**
- Dev DB: `alembic upgrade head → downgrade -1 → upgrade head` green; index def renders `USING hnsw … WITH (m='16', ef_construction='64')`, `indisvalid=t`; head=0029.
- `uv run pytest tests/integration/test_chunks_hnsw_index.py` → **7 passed in 0.43s** (deterministic).
- `uv run pytest` → **4577 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **4577 passed, TOTAL 99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- `uv run pytest tests/e2e/test_smoke.py --no-cov` → 3 passed (index live, no retrieval regression).
**Decisions/deviations**: kept the task-prescribed unconditional `CREATE` (post-build crash → loud `already exists` + documented manual drop; halfvec stays out per A2). Next pending task: `188_chunks_hnsw_index/02_ef_search_setting.md`.
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__01_hnsw_migration.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__01_hnsw_migration.a1.validate
+145
View File
@@ -0,0 +1,145 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 231 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/memory.py 32 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 865 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6932 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__02_ef_search_setting.a1.err
View File
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__02_ef_search_setting.a1.md
+18
View File
@@ -0,0 +1,18 @@
**Task 02 complete — all green.**
**Implemented**
- `app/config.py` — `retrieval_ef_search: int = 100` (`BOR_RETRIEVAL_EF_SEARCH`) with the fail-loud validator: `0` **and** negatives refuse startup (the clamp has no "off" state, phase-182 `show_text_max_*` family).
- `app/rag/retriever.py` — `_vector_candidates` now opens with `SET LOCAL hnsw.ef_search = {int(ef)}` immediately before the `<=>` SELECT (module gained its ONE logger); below the A7 candidate count it CLAMPS up and logs `retrieval: ef_search=%d raised to %d (A7 candidate count)`. Setting read via `get_settings()` (the module's existing convention — no new plumbing).
- `app/core/settings_catalog.py` — `SettingMeta("retrieval_ef_search", "Retrieval & honesty", "Vector ef_search")` beside `hybrid_vector_candidates`; `.env.example` + README knob documented.
- Pins: 8 unit (`tests/unit/test_retriever.py` — SET-before-SELECT, clamp+warn, equal/pass-through silent, default 100, int-only statement, ≤0 refuses, default == candidate count), 2 `test_config.py`, 1 `test_settings_catalog.py` row/type pin, 3 integration in `test_chunks_hnsw_index.py` (clamped value live in the same tx; **new tx falls back to the server default** — LOCKED A4; exactly one clamp WARNING line).
- Leak-guard pins added for the new knob in `tests/conftest.py` and `tests/e2e/conftest.py` (house phase-61/106/182 pattern).
**Verification (exact)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4590 passed, TOTAL 99%** (>90%); `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors.
- E2E each in isolation: smoke 3, chat_rag 3, retrieval_quality 4, admin_settings_panel 12, admin_settings_layout 5 — all passed. Dev DB verified at `0029` with `ix_chunks_embedding_hnsw` valid.
**Docs**: `README.md` (How-retrieval step 1 + env table row), `AGENTS.md` (rule 10 gains the clamp-warning + `SET LOCAL` contract), `.env.example`.
**Decisions/deviations**: clamp compares against the call's own `LIMIT` (per D3); no `ef_search` override param added (task 03 can monkeypatch `get_settings`); measured server default = 40, max 1000 (pgvector 0.8.0), so the integration pin reads `pg_settings.reset_val` instead of hardcoding. **Pre-existing, not introduced here**: running several `tests/e2e/` modules in ONE pytest session fails (every module spawns uvicorn on the same `APP_PORT`) — reproduced byte-identically (7 failed/8 passed) on a clean `git archive HEAD` tree; each file passes in isolation per AGENTS rule 9, so left untouched.
**Next pending task**: `188_chunks_hnsw_index/03_plan_and_recall_pins.md`.
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__02_ef_search_setting.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__02_ef_search_setting.a1.validate
+145
View File
@@ -0,0 +1,145 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
...................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 231 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 325 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/memory.py 32 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 865 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 259 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6946 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__03_plan_and_recall_pins.a1.err
View File
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__03_plan_and_recall_pins.a1.md
+19
View File
@@ -0,0 +1,19 @@
**Implemented** (task 03 — `.agents/phases/todo/188_chunks_hnsw_index/03_plan_and_recall_pins.md`)
- New `tests/integration/test_chunks_hnsw_recall.py` (567 lines): fixed-seed 5,000-chunk / 768-dim corpus (40 centroids + noise inside a 16-direction principal subspace, L2-normalised), index built **only** via migration 0029 (`0029→0028→seed→0029`), teardown deletes just `source='phase-188-hnsw-recall'` and restores head (dev DB verified: 0 chunks, 0029, `indisvalid=t`).
- `test_plan_uses_the_hnsw_index` — `EXPLAIN (FORMAT JSON)` of the tier's **captured** SELECT (statement object from `_vector_candidates`, `literal_binds`): names `ix_chunks_embedding_hnsw`, no `Seq Scan on chunks`.
- `test_plan_without_the_index_seq_scans_the_relation` — vacuity guard via `SET LOCAL enable_indexscan/indexonlyscan = off` (documented choice over DROP/re-`CREATE CONCURRENTLY`).
- `test_top100_recall_at_ef_search_100` — pure-Python exact cosine top-100 (float4-modelled) vs the index: per-query floor 0.99, per-row `1-distance` within 1e-6, evidence-carrying failure message. Measured: mean **0.9993**, min 0.99 (2/30 at 99/100), worst cosine delta **5.7e-07** (bounded by the tier's `round(…,6)`).
- `test_recall_never_below_the_pinned_floor_when_ef_search_raised` — monotonicity: ef=100 → 0.9993, ef=200 → **1.0000**.
- Module docstring records the prod numbers (414,220 chunks / ~3,000 MB / no index / pgvector 0.8.0 / 3,904 MB no swap), why 5,000 rows is a plan pin not a prod proxy (task 04 is that gate), the measured overlaps, ef values, and ~8 s setup vs the 60 s guard.
**Verification (exact)**
- `uv run pytest tests/integration/test_chunks_hnsw_recall.py -v` → **4 passed** (29 s); with `test_chunks_hnsw_index.py` + `tests/unit/test_retriever.py` → 105 checks, 0 failures.
- `uv run pytest --cov=app --cov-report=term-missing` → **4594 passed, TOTAL 99%** (>90%). `uv run ruff check .` → All checks passed. `uv run pyright` → **0 errors**.
**Docs**: no doc impact — task 03 is CI-only; `BOR_RETRIEVAL_EF_SEARCH` already in README/.env.example from task 02; the build runbook stays task 05's. No `app/` change.
**Decisions / flags**
- Corpus design: the task's "40 clusters + noise" is actively hostile to HNSW when clusters are tight (measured recall collapsed to 0.00–0.65 — disconnected layer-0 graph); the pin therefore uses a noise-dominated, low-effective-rank (16-of-768) clustered corpus, which is also how trained embeddings behave.
- **Flag for task 04 / owner**: on effectively-iid 768-dim data the same 5,000-row / `m=16` / `ef_construction=64` graph measures recall@100 ≈ **0.63 at ef_search=100**, 0.85 at 200, 1.00 from 400. Per LOCKED D3/A3 I did **not** change the shipped default (100); if the live battery moves a `top_score`/top-5 set, the remedy is raising `BOR_RETRIEVAL_EF_SEARCH` toward 400.
**Next pending task**: `188_chunks_hnsw_index/04_recall_battery_gate.md`.
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__03_plan_and_recall_pins.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__03_plan_and_recall_pins.a1.validate
+145
View File
@@ -0,0 +1,145 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 231 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 325 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/memory.py 32 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 865 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 259 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6946 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__04_recall_battery_gate.a1.err
View File
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__04_recall_battery_gate.a1.md
+16
View File
@@ -0,0 +1,16 @@
Task 04 complete. Final state of the work:
**Implementation** — Gate run on the live endpoint + restored 1,000-doc live replica (8,866 real embeddings, dump at `alembic_version=0011`, so migration 0029 built the graph against real vectors).
- Battery `eval_retrieval` OFF vs ON: **byte-identical** (all `best_cosine`, all 35 hits, all 7 A8 verdicts).
- Pinned-vector gate (probe embeds once; endpoint self-jitter measured 0.000e+00): verdicts, top-5 sets/order, suggested docs and `top_score` **identical, delta 0.000e+00**; top-100 candidate-set overlap **1.0000 on all 7**; every rank swap an exact tie; tier 38.3 → 15.3 ms; `Limit → Nested Loop → Index Scan on ix_chunks_embedding_hnsw` vs `Limit → Sort → Seq Scan`.
- Recall@100 sweep: `ef=100` 0.9814 mean / 0.94 min (under the phase's 0.99 floor, per-chunk delta 2.195e-03) → `ef=400` **1.0000/1.0000**. **Applied LOCKED A3's only remedy: `retrieval_ef_search` default 100 → 400** (`app/config.py`, `.env.example`, `README.md`, `AGENTS.md` rule 10, and the 4 pins that encoded 100). **No honesty threshold touched.**
- Production order (throwaway 416,702-chunk / 557 MB DB, created and dropped): build **161 s**, index **1,625 MB**, planner takes HNSW at ef 100/200/400, tier **~550 ms → 12–15 ms**.
- **Deviation flagged:** D5's `maintenance_work_mem='512MB'` is unusable here — `/dev/shm` is 63 MB; reproduced `could not resize shared memory segment … No space left on device` at production order, which exercised 0029's INVALID-index self-heal for real. Also fixed a vacuity hole in task 03's recall pin (pgvector costs HNSW partly on `ef_search`, so the pin now refuses seq scans and asserts the index is in the plan).
**Tests / gates** — `uv run pytest --cov=app`: **4594 passed**, `app/` **99%** (>90%). `ruff check .`: All checks passed. `pyright`: 0 errors. `tests/e2e/test_smoke.py`: exit 0.
**Docs** — new `benchmarks/phase188_live_gate/` (both battery captures, compare + recall outputs, pinned vectors, production-order build/latency logs, the 512 MB failure log, README) + pointer in `benchmarks/README.md`; report at `.agents/reports/188_chunks_hnsw_index/task04_recall_battery_gate.md` (+ the probe).
**Deviations** — (1) the ×47 production-order corpus is valid for build/plan/latency but *not* recall (47 near-copies per real vector); recall verdict comes from the real-embedding replica, stated in both docs. (2) Phase task files untouched; no git index/history mutations — 34 paths left in the working tree. Dev DB returned to its start state (empty, 0029, index valid); scratch DBs dropped.
**Next** — task 05 (`prod_index_build_runbook`): carry the 161 s / 1.6 GB figures, the shm constraint, and a post-build battery re-run on the real 414k KB.
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__04_recall_battery_gate.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__04_recall_battery_gate.a1.validate
+145
View File
@@ -0,0 +1,145 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 231 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 325 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/memory.py 32 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 865 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 259 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6946 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__05_e2e_index_chat_and_runbook.a1.err
View File
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__05_e2e_index_chat_and_runbook.a1.md
+12
View File
@@ -0,0 +1,12 @@
All gates green. Final state:
**Implemented (task 05)**
- `tests/e2e/test_vector_index_recall.py` (889 lines, mock-only, own app on 8205 + two ef-apps on 8206/8207 with stdout captured) — the four required tests: `test_answered_question_cites_the_same_document_indexed_or_not` (arms split by the real `0029` downgrade/upgrade, catalog state re-read inside each arm), `test_index_exists_and_is_valid_after_import`, `test_near_miss_question_still_deflects` (both arms), `test_ef_search_setting_is_honoured` (`=150` answers with no clamp line; `=50` answers identically and logs `retrieval: ef_search=50 raised to 100 (A7 candidate count)`).
- `tests/fixtures/hnswkb/` — tracked 11-note KB imported through the real importer with the index live; sentinels measured, not guessed (answered: cosine **0.6187**, fts 10, `hnswkb/infra/tilecache.md` first; near-miss: **0.1706**, fts 0).
- README: new **`Deploying migration 0029 (HNSW)`** runbook (pre-flight incl. `/dev/shm`, CONCURRENTLY semantics, 161 s / 1.6 GB / 3.9 KB-per-row, INVALID recovery, the `maintenance_work_mem > /dev/shm` trap, the 4 GB-box ordering pointer); **fixed a pre-existing docs bug** — the env table still said `BOR_RETRIEVAL_EF_SEARCH = 100` after task 04 shipped 400. AGENTS.md: new rule 16 (index owned only by `0029`, `SET LOCAL` only, never re-tune 400 or move a threshold, how a test may take the index off). `benchmarks/phase188_live_gate/`: task-05 parity section + captures.
**Gates** — `test_vector_index_recall.py` 4 passed in isolation; `test_retrieval_quality` 4, `test_search_recall_small_sources` 7, `test_chat_rag` 3, `test_smoke` 3, all green in isolation; `uv run pytest --cov=app` **4594 passed, app/ coverage 99%**; `ruff check .` clean; `pyright` **0 errors**. Retrieval battery re-verified clean-room on the live replica (throwaway DB `bor_p188gate`, restored at `0011` → `0028` OFF → `head` ON, index valid/34 MB): the two captures are **byte-identical**; DB dropped, dev DB left as found (head `0029`, index valid, KB empty).
**Deviations** — (1) no commit and no phase-dir move: git-mutating commands were disallowed for this run, so `00_phase.md`/task files and the commit remain untouched; (2) phase 119 LOCKED A1 gives a zero-read grounded turn **zero chips**, so test 1 pins "the same cited document" through the durable `query_log` record + the rendered "Nearby docs, in case" row + the `done` tiers (and pins `chips == 0` in both arms), stated in the module docstring; (3) one transient full-suite failure I caused by running the new E2E module concurrently with it (shared Postgres) — both pass serially, re-run clean.
**Next pending** — task 05's step 5 (the atomic `--no-gpg-sign` commit + move `188_chunks_hnsw_index/` to `complete/`), then `.agents/phases/todo/189_chat_save_long_turns/`.
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__05_e2e_index_chat_and_runbook.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/188_chunks_hnsw_index/188_chunks_hnsw_index__05_e2e_index_chat_and_runbook.a1.validate
+145
View File
@@ -0,0 +1,145 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 231 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 325 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/memory.py 32 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 865 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 259 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6946 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py
+629
View File
@@ -0,0 +1,629 @@
"""Phase 188 task 04 — the DETERMINISTIC index off/on gate probe (live KB, live endpoint).
Why this probe exists next to `scripts/eval_retrieval.py`
---------------------------------------------------------
`eval_retrieval` cannot gate this task on its own, for two reasons. It prints
five hits per question, while the difference between the exact and the
approximate search lives in the TAIL of the cosine tier's 100-row candidate set
(at `ef_search = 100` on the live replica the exact and approximate top-100
agreed on 0.9814 of the rows on average, 0.94 at worst, with a per-chunk cosine
moved by up to 2.195e-03 while `top_score` stayed at 0.000e+00). Five printed rows
cannot show a 100-row candidate set, and they conflate the cosine tier with the RRF
fusion: an early off/on battery pair at `ef_search = 100` moved printed cosines by
up to ~3e-4 (above the 1e-4 tolerance) because a substituted tail candidate shifts
the fusion ranks and a document's several chunks let a hit keep its document while
changing its chunk.
This probe embeds the battery ONCE, caches the vectors, and measures both sides from
the cache, so the access path is the only variable and all 100 rows are visible
separately from the fused five. It also reports the endpoint's own float jitter
(`endpoint_self_jitter`, one question embedded twice) rather than assuming it either
way — measured 0.000e+00 here, i.e. the endpoint is deterministic and every delta
below is the index's doing. LOCKED A3: the tolerance is never widened; the input is
pinned.
What it measures per question (all through the real code path,
`app.rag.retriever._vector_candidates` / `retrieve`):
* `exact` — the cosine tier with `enable_indexscan = off` (the exact
top-100 a seq scan produces, i.e. the pre-phase behaviour)
* `app` — the tier exactly as a request runs it (SET LOCAL ef_search
from Settings, planner free to use the index)
* `ef100` / `ef200` / `ef400` — the tier pinned at each ef_search
* latency of each of the above (median of `--reps`, own transaction)
* the full `retrieve()` pipeline: best_cosine, fts_hits, the A8 verdict,
the top-5 documents (the same best-per-doc order eval_retrieval prints)
* the EXPLAIN node types of the tier statement (anti-vacuity: `exact`
MUST seq-scan, `app` MUST name ix_chunks_embedding_hnsw when it exists)
Gate for `compare` (LOCKED A3 — nothing here is ever relaxed to pass):
* identical A8 verdicts,
* identical top-5 document lists and identical suggested documents,
* identical `top_score` / `best_cosine` within 1e-4,
* identical scores within 1e-4 for every rank inside the fused window
(`hybrid_top_k = 5`) — the ranks that reach RRF and the deflection gate,
* the cosine tier's top-100 candidate SET overlap at or above 0.99.
Tail ranks are gated as a SET overlap rather than a score delta: an HNSW search
may hand back a different tail chunk at a near-identical cosine, which changes
which tail chunk the fusion could have raised without moving anything the turn
shows the model. That is exactly the difference this gate is allowed to see, and
the reason `eval_retrieval` alone cannot answer the task.
Usage (repo root, live KB restored, the app's LLM key reachable —
`BOR_LLM_API_KEY` in `.env`, or `$AIPI_KEY` as the fallback app/config.py reads)
-----------------------------------------------------------------
PYTHONPATH=. uv run python \
.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py embed \
--out /tmp/gate_vectors.json
# index OFF (alembic downgrade -1)
PYTHONPATH=. uv run python .../task04_recall_ab_probe.py measure \
--vectors /tmp/gate_vectors.json --out /tmp/gate_off.json --reps 5
# index ON (alembic upgrade head)
PYTHONPATH=. uv run python .../task04_recall_ab_probe.py measure \
--vectors /tmp/gate_vectors.json --out /tmp/gate_on.json --reps 5
PYTHONPATH=. uv run python .../task04_recall_ab_probe.py compare \
--a /tmp/gate_off.json --b /tmp/gate_on.json
# recall@100 of the approximate index vs the exact top-100
PYTHONPATH=. uv run python .../task04_recall_ab_probe.py recall \
--path /tmp/gate_on.json
"""
from __future__ import annotations
import argparse
import asyncio
import json
import statistics
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
from dotenv import load_dotenv
from sqlalchemy import text
from sqlalchemy.dialects import postgresql
from sqlalchemy.orm import Session
load_dotenv()
BATTERY = Path("tests/fixtures/retrieval_battery.txt")
TOP_K = 100 # A7's cosine candidate count (the clamp floor)
EF_GRID = (100, 200, 400)
INDEX_NAME = "ix_chunks_embedding_hnsw"
CANDIDATE_SET_FLOOR = 0.99 # the phase's own recall floor (LOCKED A3)
TURN_TOP_K = 5 # app/config.py hybrid_top_k — what the turn actually fuses
def questions() -> list[str]:
lines = BATTERY.read_text(encoding="utf-8").splitlines()
return [ln.strip() for ln in lines if ln.strip() and not ln.lstrip().startswith("#")]
def cmd_embed(args: argparse.Namespace) -> int:
from app.config import get_settings
from app.db import db_available
from app.rag.llm import LLMClient
if not db_available():
print("probe: Postgres is down", file=sys.stderr)
return 1
qs = questions()
vectors, jitter = asyncio.run(_embed_all(LLMClient(get_settings()), qs))
out = {"questions": qs, "vectors": vectors, "endpoint_self_jitter": jitter}
Path(args.out).write_text(json.dumps(out))
print(f"probe: embedded {len(qs)} questions -> {args.out} (endpoint self-jitter {jitter:.3e})")
return 0
async def _embed_all(llm: Any, qs: list[str]) -> tuple[list[list[float]], float]:
"""One loop for every embedding the probe needs (the SDK client is
loop-bound: a second asyncio.run with the same client raises
``Event loop is closed``). Also embeds the first question twice more to
measure the endpoint's OWN run-to-run float jitter — the number that
explains why the plain off/on battery diff moves at all."""
vectors = await llm.embed(qs)
first = (await llm.embed([qs[0]]))[0]
again = (await llm.embed([qs[0]]))[0]
jitter = max(abs(a - b) for a, b in zip(first, again, strict=True))
await llm._client.close() # pyright: ignore[reportPrivateUsage]
return vectors, jitter
def _tier_statement(vector: list[float]) -> Any:
"""The tier's own SELECT (statement object captured from the code)."""
from app.rag.retriever import _vector_candidates
class Rec:
def __init__(self) -> None:
self.statements: list[Any] = []
def execute(self, statement: Any, params: Any = None) -> Rec:
self.statements.append(statement)
return self
def all(self) -> list[Any]:
return []
rec = Rec()
_vector_candidates(rec, vector, TOP_K) # pyright: ignore[reportArgumentType]
return rec.statements[1].compile(
dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}
)
def _plan_nodes(session: Session, sql: Any) -> list[str]:
raw = session.execute(text(f"EXPLAIN (FORMAT JSON) {sql}")).scalar()
doc = json.loads(raw) if isinstance(raw, str) else raw
def walk(plan: dict[str, Any]) -> Iterator[dict[str, Any]]:
yield plan
for child in plan.get("Plans") or []:
yield from walk(child)
nodes = []
for p in walk(doc[0]["Plan"]):
name = p.get("Index Name") or p.get("Relationship Name") or ""
nodes.append(f"{p['Node Type']}" + (f" on {name}" if name else ""))
return nodes
def _pipeline(
session: Session, question: str, vector: list[float], force_index: bool
) -> dict[str, Any]:
"""The whole turn's retrieval, in ONE transaction, optionally with the
planner forced onto the HNSW index (``enable_seqscan = off``).
Why the force: at ``ef_search = 100`` pgvector costs the approximate scan
ABOVE a sequential scan of the 8,866-chunk / 13 MB replica, so the planner
picks the seq scan and an "index ON" measurement would silently measure the
exact scan twice (a vacuous gate). Forcing the index measures the path
production (414,220 chunks, ~3,000 MB) actually takes. The force is
session-local (``SET LOCAL``) and rolls back with the transaction.
"""
from app.config import get_settings
from app.rag.retriever import retrieve, select_suggested
settings = get_settings()
session.rollback()
session.execute(text("begin"))
if force_index:
session.execute(text("SET LOCAL enable_seqscan = off"))
# the plan a turn actually gets: the tier's own SET LOCAL, in the SAME
# transaction it runs in (EXPLAIN with the server's ef_search=40 would
# report a plan no request ever uses)
session.execute(text(f"SET LOCAL hnsw.ef_search = {int(settings.retrieval_ef_search)}"))
chunks = retrieve(session, question, vector)
best_cosine = max((c.cosine for c in chunks), default=0.0)
fts_hits = sum(1 for c in chunks if c.fts_hit)
best_by_doc: dict[str, Any] = {}
for c in chunks:
best_by_doc.setdefault(f"{c.document.source}/{c.document.path}", c)
out = {
"best_cosine": best_cosine,
"fts_hits": fts_hits,
"verdict": (
"LOW (deflect)"
if best_cosine < settings.relevance_threshold and fts_hits == 0
else "HIGH (answer)"
),
"top5": list(best_by_doc)[:5],
"suggested": [f"{d.source}/{d.path}" for d in select_suggested(chunks)],
"plan": _plan_nodes(session, _tier_statement(vector)),
}
session.rollback()
return out
def _timed_tier(
session: Session,
vector: list[float],
setup: str | None,
reps: int,
ef_override: int | None = None,
) -> dict[str, Any]:
"""Median/min/max milliseconds of the cosine tier, one transaction per rep.
``ef_override`` goes through the REAL knob (``BOR_RETRIEVAL_EF_SEARCH`` +
the ``get_settings`` cache), never a hand-written ``SET LOCAL``: the tier
issues its own ``SET LOCAL hnsw.ef_search`` from Settings, so a probe-side
``SET`` would simply be overwritten by the line under test.
"""
import os
from app.config import get_settings
from app.rag.retriever import _vector_candidates
previous_env = os.environ.get("BOR_RETRIEVAL_EF_SEARCH")
if ef_override is not None:
os.environ["BOR_RETRIEVAL_EF_SEARCH"] = str(ef_override)
get_settings.cache_clear()
timings: list[float] = []
rows: list[dict[str, Any]] = []
try:
for i in range(reps):
session.rollback() # end any implicit transaction -> SET LOCAL is fresh
if setup:
session.execute(text("begin"))
session.execute(text(setup))
t0 = time.perf_counter()
got = _vector_candidates(session, vector, TOP_K)
timings.append((time.perf_counter() - t0) * 1000.0)
if i == 0:
rows = [
{
"rank": rank,
"chunk_id": str(c.chunk_id),
"cosine": c.cosine,
"doc": f"{c.document.source}/{c.document.path}",
}
for rank, c in enumerate(got, start=1)
]
session.rollback()
finally:
if ef_override is not None:
if previous_env is None:
os.environ.pop("BOR_RETRIEVAL_EF_SEARCH", None)
else:
os.environ["BOR_RETRIEVAL_EF_SEARCH"] = previous_env
get_settings.cache_clear()
return {
"ms_median": round(statistics.median(timings), 3),
"ms_min": round(min(timings), 3),
"ms_max": round(max(timings), 3),
"rows": rows,
}
def cmd_measure(args: argparse.Namespace) -> int:
from app.config import get_settings
from app.db import SessionLocal, db_available
if not db_available():
print("probe: Postgres is down", file=sys.stderr)
return 1
payload = json.loads(Path(args.vectors).read_text())
qs: list[str] = payload["questions"]
vectors: list[list[float]] = payload["vectors"]
settings = get_settings()
with SessionLocal() as db:
index_present = db.execute(
text("select count(*) from pg_indexes where indexname = :n"), {"n": INDEX_NAME}
).scalar()
ef_setting = settings.retrieval_ef_search
per_question: list[dict[str, Any]] = []
for q, vec in zip(qs, vectors, strict=True):
record: dict[str, Any] = {"question": q}
record["exact"] = _timed_tier(
db,
vec,
"SET LOCAL enable_indexscan = off",
args.reps,
)
record["app"] = _timed_tier(db, vec, None, args.reps)
# the APPROXIMATE path, forced (the planner declines it at this
# corpus size — see _pipeline's docstring), at each ef_search
for ef in EF_GRID:
record[f"forced{ef}"] = _timed_tier(
db,
vec,
"SET LOCAL enable_seqscan = off",
args.reps,
ef_override=ef,
)
# anti-vacuity plans, same session, same vector
sql = _tier_statement(vec)
db.execute(text("begin"))
db.execute(text("SET LOCAL enable_indexscan = off"))
record["plan_exact"] = _plan_nodes(db, sql)
db.rollback()
db.execute(text("begin"))
record["plan_app"] = _plan_nodes(db, sql)
db.rollback()
# the pipeline a turn actually runs, with the A8 gate: once as the
# planner chooses, once forced onto the approximate index
record["pipeline"] = _pipeline(db, q, vec, force_index=False)
record["pipeline_forced"] = _pipeline(db, q, vec, force_index=True)
per_question.append(record)
print(
f" {q!r}: verdict={record['pipeline']['verdict']} "
f"exact={record['exact']['ms_median']}ms app={record['app']['ms_median']}ms "
f"plan_app={record['plan_app'][1] if len(record['plan_app']) > 1 else '?'}"
)
Path(args.out).write_text(
json.dumps(
{
"index_present": int(index_present or 0),
"ef_search_setting": ef_setting,
"reps": args.reps,
"vector_candidates": settings.hybrid_vector_candidates,
"questions": qs,
"records": per_question,
}
)
)
print(
f"probe: measured {len(per_question)} questions -> {args.out} "
f"(index_present={index_present})"
)
return 0
def _docs(rec: dict[str, Any]) -> list[str]:
return [r["doc"] for r in rec["rows"]]
def cmd_compare(args: argparse.Namespace) -> int:
a = json.loads(Path(args.a).read_text())
b = json.loads(Path(args.b).read_text())
print(f"A={args.a} index_present={a['index_present']} ef={a['ef_search_setting']}")
print(f"B={args.b} index_present={b['index_present']} ef={b['ef_search_setting']}")
failures: list[str] = []
max_score_delta = 0.0
max_tier_delta = 0.0 # every rank of the tier, reported
max_window_delta = 0.0 # ranks <= hybrid_top_k, gated
for ra, rb in zip(a["records"], b["records"], strict=True):
assert ra["question"] == rb["question"]
q = ra["question"]
if ra["pipeline"]["verdict"] != rb["pipeline"]["verdict"]:
failures.append(
f"{q!r}: verdict {ra['pipeline']['verdict']} != "
f"{rb['pipeline']['verdict']}"
)
if ra["pipeline"]["top5"] != rb["pipeline"]["top5"]:
failures.append(
f"{q!r}: top5 changed\n A={ra['pipeline']['top5']}\n"
f" B={rb['pipeline']['top5']}"
)
if set(ra["pipeline"]["top5"]) != set(rb["pipeline"]["top5"]):
failures.append(f"{q!r}: top5 SET changed")
if ra["pipeline"]["suggested"] != rb["pipeline"]["suggested"]:
failures.append(f"{q!r}: suggested list changed")
delta = abs(ra["pipeline"]["best_cosine"] - rb["pipeline"]["best_cosine"])
max_score_delta = max(max_score_delta, delta)
if delta > 1e-4:
failures.append(
f"{q!r}: top_score delta {delta:.9f} > 1e-4 "
f"(A={ra['pipeline']['best_cosine']:.9f} B={rb['pipeline']['best_cosine']:.9f})"
)
# The candidate SET (the A7 tier's promise to the fusion) and the
# SCORES. Order is NOT the gate: `ORDER BY distance LIMIT 100` breaks
# ties by access path, so an index scan and a top-N sort legitimately
# return the same rows in a different order whenever two chunks share a
# cosine. What must not move is which chunks are in the list, their
# scores, and anything the turn shows the model.
da, dbb = ra["app"], rb["app"]
ids_a = {r["chunk_id"] for r in da["rows"]}
ids_b = {r["chunk_id"] for r in dbb["rows"]}
overlap = len(ids_a & ids_b) / max(1, len(ids_a))
if overlap < CANDIDATE_SET_FLOOR:
failures.append(
f"{q!r}: the cosine tier's top-{TOP_K} candidate set overlap is "
f"{overlap:.4f}, below the {CANDIDATE_SET_FLOOR} floor "
f"(A-only={sorted(ids_a - ids_b)[:3]}…, B-only={sorted(ids_b - ids_a)[:3]}…)"
)
for row_a, row_b in zip(da["rows"], dbb["rows"], strict=False):
d = abs(row_a["cosine"] - row_b["cosine"])
max_tier_delta = max(max_tier_delta, d)
# The 1e-4 tolerance binds where the turn actually reads: the
# fused window is TOP_K=5 (app/config.py), and ranks 1-5 are the
# ones whose scores reach RRF and the deflection gate. Ranks below
# that only ever change WHICH tail chunk is in the list, and that
# is measured by the overlap floor above, not by a score delta.
if row_a["rank"] <= TURN_TOP_K:
max_window_delta = max(max_window_delta, d)
if row_a["rank"] <= TURN_TOP_K and d > 1e-4:
failures.append(
f"{q!r}: rank-{row_a['rank']} cosine delta {d:.9f} > 1e-4 "
f"(A={row_a['cosine']:.9f} B={row_b['cosine']:.9f})"
)
tie_swaps = sum(
1
for a, b in zip(da["rows"], dbb["rows"], strict=False)
if a["chunk_id"] != b["chunk_id"]
)
exact_ties = all(
abs(a["cosine"] - b["cosine"]) == 0.0
for a, b in zip(da["rows"], dbb["rows"], strict=False)
if a["chunk_id"] != b["chunk_id"]
)
same_top5 = "SAME" if ra["pipeline"]["top5"] == rb["pipeline"]["top5"] else "DIFFERENT"
same_sugg = (
"SAME" if ra["pipeline"]["suggested"] == rb["pipeline"]["suggested"] else "DIFFERENT"
)
print(
f" {q!r}: verdict {ra['pipeline']['verdict']} / "
f"{rb['pipeline']['verdict']} "
f"| top5 {same_top5} | suggested {same_sugg} "
f"| candidate-set overlap {overlap:.4f} "
f"| best_cosine delta {delta:.3e} "
f"| rank swaps {tie_swaps}"
f"{' (all exact ties)' if exact_ties else ' <-- NOT ties'}"
)
print(f"\nmax top_score (best_cosine) delta = {max_score_delta:.3e} (gate <= 1e-4)")
print(
f"max in-window (rank <= {TURN_TOP_K}) tier cosine delta = "
f"{max_window_delta:.3e} (gate <= 1e-4)"
)
print(
f"max tier cosine delta over all {TOP_K} ranks = {max_tier_delta:.3e} (reported only)"
)
print(
f"tail note: ranks below the fused window are gated as the candidate-SET "
f"overlap (>= {CANDIDATE_SET_FLOOR}), not as a score delta: a graph search may "
"hand back a different tail chunk at a near-identical score without moving "
"anything the turn reads."
)
for label, side in (("A", a), ("B", b)):
r0 = side["records"][0]
print(f"{label} plan(exact)={r0['plan_exact'][:2]} plan(app)={r0['plan_app'][:2]}")
print(
f"tier latency ms (median over questions): A exact={_med(a, 'exact')} "
f"app={_med(a, 'app')} | "
f"B exact={_med(b, 'exact')} app={_med(b, 'app')}"
)
if failures:
print(f"\nGATE FAIL ({len(failures)} differences):")
for f in failures:
print(f" - {f}")
return 1
print("\nGATE PASS: identical verdicts, identical top-5 sets and order, deltas <= 1e-4")
return 0
def _med(side: dict[str, Any], key: str) -> float:
return round(statistics.median([r[key]["ms_median"] for r in side["records"]]), 3)
def cmd_recall(args: argparse.Namespace) -> int:
"""THE APPROXIMATE-PATH GATE, run on the index-ON side.
Three things per question, all on the SAME pinned vector:
* recall@100 of the forced HNSW scan against the exact top-100
* the per-chunk cosine delta at equal rank, and the best-cosine
(``top_score``) delta
* the A8 verdict, the top-5 documents and the suggested list of the
forced-index pipeline against the planner's own pipeline
A vacuity guard is part of the gate: the forced plan MUST name
``ix_chunks_embedding_hnsw`` and the exact plan MUST be a Sort over a seq
scan, otherwise the two sides measured the same thing twice.
"""
side = json.loads(Path(args.path).read_text())
if not side["index_present"]:
print("recall: the index is not present — run this on the index-ON side", file=sys.stderr)
return 2
failures: list[str] = []
out: dict[str, Any] = {}
for ef in EF_GRID:
key = f"forced{ef}"
overlaps: list[float] = []
deltas: list[float] = []
best_deltas: list[float] = []
for r in side["records"]:
exact = [x["chunk_id"] for x in r["exact"]["rows"]]
got = [x["chunk_id"] for x in r[key]["rows"]]
overlaps.append(len(set(exact) & set(got)) / max(1, len(exact)))
deltas.append(
max(
(
abs(a["cosine"] - b["cosine"])
for a, b in zip(r["exact"]["rows"], r[key]["rows"], strict=False)
),
default=0.0,
)
)
best_deltas.append(
abs(
max(x["cosine"] for x in r["exact"]["rows"])
- max(x["cosine"] for x in r[key]["rows"])
)
)
out[key] = {
"mean_overlap": round(statistics.mean(overlaps), 6),
"min_overlap": round(min(overlaps), 6),
"max_cosine_delta": round(max(deltas), 9),
"max_top_score_delta": round(max(best_deltas), 9),
"ms_median": round(
statistics.median([r[key]["ms_median"] for r in side["records"]]), 3
),
}
print(
f" {key}: mean recall@100={out[key]['mean_overlap']:.4f} "
f"min={out[key]['min_overlap']:.4f} "
f"max per-chunk cosine delta={out[key]['max_cosine_delta']:.3e} "
f"max top_score delta={out[key]['max_top_score_delta']:.3e} "
f"latency={out[key]['ms_median']}ms"
)
if ef == side["ef_search_setting"] and out[key]["min_overlap"] < 1.0:
print(
f" note: at the SHIPPED ef_search={ef} the approximate top-100 differs from the "
f"exact one in "
f"{round((1 - out[key]['min_overlap']) * 100, 2)}% of the worst question"
)
print("\n pipeline parity (A8 verdict / top-5 / suggested), planner vs forced index:")
for r in side["records"]:
a, b = r["pipeline"], r["pipeline_forced"]
same_v = a["verdict"] == b["verdict"]
same_5 = a["top5"] == b["top5"]
same_s = a["suggested"] == b["suggested"]
d = abs(a["best_cosine"] - b["best_cosine"])
print(
f" {r['question']!r}: verdict {'SAME' if same_v else 'DIFFERENT'} "
f"({a['verdict']} / {b['verdict']}) | top5 {'SAME' if same_5 else 'DIFFERENT'} "
f"| suggested {'SAME' if same_s else 'DIFFERENT'} | top_score delta {d:.3e}"
)
if not same_v:
failures.append(f"{r['question']!r}: verdict moves with the index")
if not same_5:
failures.append(
f"{r['question']!r}: top-5 changes with the index\n"
f" planner ={a['top5']}\n index ={b['top5']}"
)
if d > 1e-4:
failures.append(
f"{r['question']!r}: top_score delta {d:.3e} > 1e-4 on the forced index"
)
print("\n plans (vacuity guards):")
r0 = side["records"][0]
print(f" exact: {r0['plan_exact'][:3]}")
print(f" forced index (pipeline): {r0['pipeline_forced']['plan'][:3]}")
if not any("ix_chunks_embedding_hnsw" in n for n in r0["pipeline_forced"]["plan"]):
failures.append(
"the forced pipeline did NOT use ix_chunks_embedding_hnsw — "
"the gate measured nothing"
)
if not any("Seq Scan" in n for n in r0["plan_exact"]):
failures.append("the exact side was not a seq scan — the gate measured the same plan twice")
print(
f"\n latency ms over questions: exact(seq)={_med(side, 'exact')} "
f"planner={_med(side, 'app')} "
+ " ".join(f"forced{ef}={_med(side, f'forced{ef}')}" for ef in EF_GRID)
)
if failures:
print(f"\nGATE FAIL ({len(failures)}):")
for f in failures:
print(f" - {f}")
return 1
print("\nGATE PASS: the approximate path returns the exact verdicts, top-5s and top_scores")
Path(args.path).with_suffix(".recall.json").write_text(json.dumps(out, indent=2))
return 0
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="task04_recall_ab_probe")
sub = p.add_subparsers(dest="cmd", required=True)
e = sub.add_parser("embed")
e.add_argument("--out", required=True)
m = sub.add_parser("measure")
m.add_argument("--vectors", required=True)
m.add_argument("--out", required=True)
m.add_argument("--reps", type=int, default=5)
c = sub.add_parser("compare")
c.add_argument("--a", required=True)
c.add_argument("--b", required=True)
r = sub.add_parser("recall")
r.add_argument("--path", required=True)
r = sub.add_parser("gate")
r.add_argument("--path", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
if args.cmd == "embed":
return cmd_embed(args)
if args.cmd == "measure":
return cmd_measure(args)
if args.cmd == "compare":
return cmd_compare(args)
if args.cmd == "gate":
return cmd_recall(args)
return cmd_recall(args)
if __name__ == "__main__":
sys.exit(main())
.agents/reports/188_chunks_hnsw_index/task04_recall_battery_gate.md
+227
View File
@@ -0,0 +1,227 @@
# Phase 188 · Task 04 — Recall + deflection gate on the live retrieval battery
**Status: complete.** The index changes nothing the turn shows the model, at every
`ef_search` measured. One knob moved as the task's remedy: `retrieval_ef_search`
ships at **400** instead of the candidate count. No honesty threshold moved.
## Where the measurement ran (prerequisites, checked)
Live `brain-of-reese_db_1` (**PostgreSQL 17.11**, Debian image, host port 5432) and the
**live embedding endpoint** — model `embed`, 768 dims, the key from
`BOR_LLM_API_KEY` in the local `.env` (never printed, never committed). The battery
and the probe both go through `app.rag.embeddings`, so the question vectors are the
same ones a real turn would compute.
| Prerequisite | Measured |
| --- | --- |
| `SELECT extversion FROM pg_extension WHERE extname='vector'` | **0.8.0** (≥ 0.5.0 required) — the running server matches this checkout's pin |
| `alembic current` before the task | head `0028`, `ix_chunks_embedding_hnsw` absent |
| `maintenance_work_mem` | 64 MB (the container default; nobody ever applied D5's 512 MB) |
| `/dev/shm` inside `brain-of-reese_db_1` | **63 MB** |
| Planner | `seq_page_cost=1`, `random_page_cost=4` (default) |
| Disk on the Postgres data volume | **2.2 TB free of 3.7 TB** (41 % used) |
| Production relation, from the 2026-10-15 incident record (not measured here) | 414,220 chunks, heap **3,000 MB** |
| Restored gate corpus | **1,000** documents, **8,866** chunks, 8,866 embeddings, heap **13 MB** (95 MB plain-SQL dump) |
The replica came from `data/bor_live_kb_replica.dump.sql` (plain-SQL `pg_dump`,
schema + data + its 10 own indexes, `CREATE EXTENSION IF NOT EXISTS vector`) via
the `restore-test-db` procedure, into a freshly dropped `public` schema, with a
disposable pre-restore backup taken first.
**The one prerequisite that decides what this task can measure:** the dump carries
`alembic_version = 0011`, so the restored corpus lands on revision 0011 and
`alembic upgrade head` replays **0012…0029 into it** — which is exactly why the
gate is trustworthy on the shape of the real migration: 0029's
`CREATE INDEX CONCURRENTLY` ran against 8,866 **real** embeddings (seconds, valid
on the first attempt) rather than against an empty table. Restoring into a schema
already at head and skipping the upgrade would have built nothing and measured
nothing. The dev database was returned to the exact state it was in when the task
started (empty, at head `0029`, index valid) before the suite ran.
## Step 1–2 — the battery, index OFF vs ON
`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`
with the index dropped (`alembic downgrade -1`) and with it present
(`alembic upgrade head`). Captures:
[`battery_index_off.txt`](../../../benchmarks/phase188_live_gate/battery_index_off.txt),
[`battery_index_on.txt`](../../../benchmarks/phase188_live_gate/battery_index_on.txt).
**The two captures are byte-identical** (7,193 bytes each, `diff` empty): every
`best_cosine`, every `fts_hits`, all 35 numbered hits, all 7 A8 verdicts. Battery
wall time 1.35 s with the index, 1.49 s without.
**Why the battery alone cannot gate this, recorded because it nearly mis-measured
it.** Two reasons, both measured rather than assumed:
* It prints **five** hits per question, while the difference between an exact and an
approximate search lives in the **tail** of the tier's 100-row candidate set. At
`ef_search = 100` the exact and approximate top-100 agreed on 0.9814 of rows on
average (0.94 worst) with a per-chunk cosine moved by up to 2.195e-03, while
`top_score` — the rank-1 number the honesty gate reads — stayed at 0.000e+00. The
battery prints a perfect match while the tier has changed its hand.
* At the task-02 default its OFF and ON captures disagreed by up to **~3e-4** on
printed cosines — above the 1e-4 tolerance — and the cause needed isolating before
anything could be concluded. The probe's own measurement of the endpoint
(`endpoint_self_jitter`, one question embedded twice) is **0.000e+00**: the endpoint
is deterministic, so that 3e-4 was **the index**. Which is what the pinned tier
then explained — at `ef_search = 100` a substituted tail candidate changes the RRF
ranks, and because a document contributes several chunks, a printed hit can keep
its document and change its chunk, moving the printed cosine while the top-5 list
and `top_score` stay identical. At the shipped 400 the two captures are
byte-identical: the recall result (1.0000) showing up in the report a human reads.
Pinning is what makes that separation possible — with the vectors fixed, the tier can
be read on its own across all 100 rows instead of 5, and a delta can only mean the
access path.
So the gate ran with the input pinned — `embed` once, cache, `measure` both sides
from the cache ([`question_vectors.json`](../../../benchmarks/phase188_live_gate/question_vectors.json))
— which is also what LOCKED A3 demands: never widen the tolerance, remove the noise
instead.
## Steps 3–4 — the deterministic gate (pinned vectors, same code path)
[`deterministic_gate_compare.txt`](../../../benchmarks/phase188_live_gate/deterministic_gate_compare.txt)
— A = index dropped (the pre-phase `Limit → Sort → Hash Join → Seq Scan on chunks`),
B = index present at the shipped `ef_search`, identical question vectors:
| Gate | Result |
| --- | --- |
| A8 verdicts, 7/7 | **identical** |
| Top-5 document sets **and order**, 7/7 | **identical** |
| Suggested documents, 7/7 | **identical** |
| `top_score` (`best_cosine`) delta | **0.000e+00** (≤ 1e-4) |
| Tier score delta, ranks 1–5 (`hybrid_top_k`, the fused window) | **0.000e+00** (≤ 1e-4) |
| Cosine-tier top-100 **candidate-set overlap** | **1.0000 on all 7** (floor 0.99) |
| Rank swaps that are not exact cosine ties | **0** |
| Tier latency, median over questions | **38.3 ms → 15.3 ms** |
Ranks below the fused window are gated as a candidate **set** overlap rather than a
score delta, and that distinction earned its place: an HNSW search can hand back a
different tail chunk at a near-identical cosine, which moves which tail chunk the
fusion could have raised and nothing else. Every swap observed in the shipped
configuration was an exact cosine tie broken by the access path.
[`deterministic_gate_recall.txt`](../../../benchmarks/phase188_live_gate/deterministic_gate_recall.txt)
— the tier forced onto the index at each depth, recall@100 against the exact scan,
**real embeddings**, two independent graph builds (the index is dropped and rebuilt
between the OFF and ON runs):
| `ef_search` | mean recall@100 | min | max per-chunk cosine delta | max `top_score` delta | tier latency |
| --- | --- | --- | --- | --- | --- |
| 100 — task 02's default, `== LIMIT` | 0.9814 / 0.9843 | 0.94 | 2.195e-03 | **0.000e+00** | 13.8 ms |
| 200 | 1.0000 / 0.9971 | 1.00 / 0.99 | 0.0 | 0.0 | 14.1 ms |
| 400 — **shipped** | **1.0000 / 1.0000** | **1.00** | **0.0** | **0.0** | 14.6 ms |
## Step 5 — the remedy, and what it did not touch
Nothing user-visible moved at `ef_search = 100`, so the task's parity gate passed at
the task-02 default. What moved was the 100-row candidate set the cosine tier hands
the RRF fusion — 0.9814 mean / 0.94 minimum — below this phase's own acceptance
floor and matching the counter-case task 03 recorded for high-effective-rank data
(one substitution at rank ~89 of 100 reorders 22 tail ranks). **LOCKED A3 names one
remedy: raise `ef_search` (towards 400).** Applied:
* `app/config.py` — `retrieval_ef_search: int = 400`, comment carrying the numbers.
* `.env.example`, `README.md` (retrieval section + env table), `AGENTS.md` (rule 10)
— the default, why it is 400, and that the thresholds stayed put.
* `tests/unit/test_config.py`, `tests/unit/test_retriever.py`,
`tests/unit/test_settings_catalog.py`, `tests/integration/test_chunks_hnsw_recall.py`
(`EF_PIN = 400`, `EF_RAISED = 600`) — every pin that encoded 100.
* **Untouched (LOCKED A3):** `relevance_threshold` 0.62, `lexical_support_floor` 0.60,
`name_hit_bonus` 0.005, `hybrid_top_k` 5, `hybrid_vector_candidates` 100, and every
recall floor and tolerance in this phase's tests.
## Production-order probe — and the two things it found
The replica is 13 MB of heap, so the planner legitimately declines a 34 MB index
below ~17 MB at `random_page_cost = 4`; the replica cannot answer "does production
use it, and what does it cost". A throwaway database (`bor_scale188b`, created and
dropped in the task) reproduced the production row count from the replica's own
embeddings ×47, each copy multiplied element-wise by a random near-one jitter vector
so all **416,702** vectors are distinct and the heap is 557 MB
([`production_order_build.log`](../../../benchmarks/phase188_live_gate/production_order_build.log),
[`production_order_latency.txt`](../../../benchmarks/phase188_live_gate/production_order_latency.txt)):
| Measurement | Value |
| --- | --- |
| Build through `uv run alembic upgrade head`, `maintenance_work_mem = 64 MB` | **161 s**, alembic process peak RSS 81 MB |
| Index size | **1,625 MB** — ~3.9 KB/row at `m=16`, 768-dim float4: plan for **~1.6 GB** on the real 414k |
| Plan at `ef_search` 100 / 200 / 400 | **HNSW index scan at every depth** (cost 3,333 / 5,621 / 9,806) |
| Tier latency | **~550 ms seq → 12.0–14.9 ms** with the index (~40×), the ~1 ms `SET LOCAL` included |
**Finding 1 — `SET maintenance_work_mem = '512MB'` (D5) is unusable in this
container, and the failure was reproduced at production order**
([`build_maintenance_work_mem_512MB_failure.log`](../../../benchmarks/phase188_live_gate/build_maintenance_work_mem_512MB_failure.log)):
`could not resize shared memory segment "/PostgreSQL.3366409952" to 533761504 bytes:
No space left on device` — `/dev/shm` is **63 MB**, and the dying build leaves an
**INVALID** index. That made the self-heal run for real: migration 0029 found
`indisvalid = false`, logged `dropping INVALID … rebuilding`, dropped it, and
rebuilt it valid (`pg_stat_progress_create_index` watched the second build at
860,544 of 872,448 version-0 tuples). So the D5 pre-check stays, but the recorded
procedure is "run `alembic upgrade head` with the container default"; raising
`maintenance_work_mem` only makes sense together with `--shm-size`. The container
default costs time, not correctness.
**Finding 2 — a recall pin must not depend on the planner cooperating.** pgvector
costs an HNSW scan partly on `ef_search`: at production order the index path cost
**3,333 / 5,621 / 9,806** at `ef_search` 100 / 200 / 400 respectively, so a deeper
value makes the index a less attractive plan and, on a relation small enough for a
top-N sort to win, it can lose. Task 03's statement pin EXPLAINs the tier's SELECT
without the tier's own `SET LOCAL hnsw.ef_search`, so what it proves is the plan the
*server* default produces — a recall run at a depth the planner declines would score
the exact scan twice and report 1.0000. `test_top100_recall_at_the_shipped_ef_search`
now refuses sequential scans for the measurement (session-scoped, reset in a
`finally`) and asserts `ix_chunks_embedding_hnsw` is in the plan before scoring, so
it fails loudly instead of passing vacuously. On the live replica the planner did take
the index at the shipped depth (`Limit → Nested Loop → Index Scan on
ix_chunks_embedding_hnsw`, with the OFF side correctly
`Limit → Sort → Seq Scan`), and at production order it took it at all three depths.
**Limitation, stated plainly:** the ×47 jittered corpus is *not* a valid recall
substrate — "the exact top-100" of 47 near-copies of 8,866 vectors is 100 knife-edge
slots inside a few clusters, which is not a real KB. Its overlap numbers (0.23–0.64)
are excluded from the recall verdict, which comes from the live replica where every
vector is a real chunk. After the production build the owner should re-run the
battery on the real 414k KB exactly as here; that is task 05's runbook step.
## Verification
| Gate | Result |
| --- | --- |
| `uv run pytest -q --cov=app --cov-report=term-missing` | **all passed**, `app/` coverage **99%** (> 90%) |
| `uv run ruff check .` | **All checks passed!** |
| `uv run pyright` | **0 errors, 0 warnings** |
| `uv run pytest tests/e2e/test_smoke.py -v --no-cov` | **exit 0** (app boots indexed, corpus assertion skips on the empty dev KB) |
| `uv run alembic downgrade -1 && uv run alembic upgrade head` | round trip verified twice on the live replica; index valid after each |
## Files
* `app/config.py` — `retrieval_ef_search` default `100 → 400` with the evidence.
* `.env.example`, `README.md`, `AGENTS.md` — the new default and why.
* `tests/unit/test_config.py`, `tests/unit/test_retriever.py`,
`tests/unit/test_settings_catalog.py`,
`tests/integration/test_chunks_hnsw_recall.py` (+ its module docstring's measured
numbers) — pins moved to the shipped depth, vacuity guard added.
* `benchmarks/phase188_live_gate/` — the captures, the gate outputs, the
production-order logs, the failure log, the pinned vectors, and the README that
ties them together; `benchmarks/README.md` gained a "Retrieval gate" pointer.
* `.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py` — the probe
(`embed` / `measure` / `compare` / `recall` / `gate`).
## Reproduce
```bash
# index OFF
uv run alembic downgrade -1
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
uv run alembic upgrade head
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
# deterministic, one embedding per question
P=.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py
PYTHONPATH=. uv run python $P embed --out /tmp/v.json
PYTHONPATH=. uv run python $P measure --vectors /tmp/v.json --out /tmp/off.json # index dropped
PYTHONPATH=. uv run python $P measure --vectors /tmp/v.json --out /tmp/on.json # index present
PYTHONPATH=. uv run python $P compare --a /tmp/off.json --b /tmp/on.json
PYTHONPATH=. uv run python $P gate --path /tmp/on.json
```