phase: 188_chunks_hnsw_index
**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
This commit is contained in:
@@ -963,7 +963,20 @@ curl -s -X DELETE localhost:8000/api/steering/<note-id> # remove
|
||||
Every question is embedded and also lexically tokenized (OR-joined, English
|
||||
stemming) and searched **twice** against Postgres:
|
||||
|
||||
1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`)
|
||||
1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`) over
|
||||
the `ix_chunks_embedding_hnsw` HNSW index (phase 188), searched at
|
||||
`BOR_RETRIEVAL_EF_SEARCH` (default **`400`**, set per transaction with
|
||||
`SET LOCAL` — never a global setting). 400 is not a guess: the live battery
|
||||
in [`benchmarks/phase188_live_gate/`](benchmarks/phase188_live_gate/README.md)
|
||||
measured the graph recovering the exact top-100 for only 0.9814 of rows on
|
||||
average (0.94 worst) at `100` — the candidate count — against 1.0000 at 400,
|
||||
with every A8 verdict, top-5 set and `top_score` unchanged at both depths
|
||||
(the honesty thresholds were never touched). A value below the candidate
|
||||
count is **clamped up** to it with one WARNING line in the server log
|
||||
(`retrieval: ef_search=… raised to …`): a smaller `ef_search` returns
|
||||
fewer candidates than the tier promises, which is a silent recall
|
||||
regression the honesty gate can turn into a deflect. Raise it (max 1000)
|
||||
if recall matters more than the tier's latency.
|
||||
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
|
||||
top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`), led by
|
||||
**name hits** — documents whose *path* matches a question token (`gitea`
|
||||
@@ -1736,6 +1749,7 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
|
||||
| Env | Default | Meaning |
|
||||
|-----|---------|---------|
|
||||
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | Cosine list width for the RRF fusion |
|
||||
| `BOR_RETRIEVAL_EF_SEARCH` | `400` | HNSW `ef_search` for the cosine tier (phase 188) — `SET LOCAL` per transaction; clamped UP to `BOR_HYBRID_VECTOR_CANDIDATES` with a WARNING when lower (that clamp is the recall floor, so `0`/negative fails startup); raise it for more recall, Postgres caps it at 1000. 400 is the measured depth at which the graph recovers the exact top-100 (see [How retrieval works](#how-retrieval-works-hybrid)) |
|
||||
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
||||
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
||||
| `BOR_RECENCY_BOOST` | `0.0007` | Max additive fused-score bonus a zero-age doc gets (phase 106); `0` = off |
|
||||
@@ -1982,6 +1996,14 @@ start while either is empty.
|
||||
|
||||
The entrypoint runs `alembic upgrade head` automatically on start.
|
||||
|
||||
> **Deploy note (migration `0029`, phase 188):** this step builds
|
||||
> `ix_chunks_embedding_hnsw` — an HNSW index over `chunks.embedding` — and
|
||||
> it is the one migration in this repo that takes **minutes** on a full
|
||||
> corpus and needs free RAM to do it. It will not lock the table, it can
|
||||
> be interrupted, and it self-heals — but read
|
||||
> [Deploying migration 0029 (HNSW)](#deploying-migration-0029-hnsw) before
|
||||
> running it on the production box.
|
||||
|
||||
Or run the whole stack from compose (app + db):
|
||||
```bash
|
||||
podman compose --profile prod up -d --build
|
||||
@@ -1994,6 +2016,127 @@ Production hardening: app runs as non-root (uid 10001), slim image,
|
||||
healthcheck on `/api/health`, debugpy off unless `DEBUGPY=1`, all assets
|
||||
served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
|
||||
### Deploying migration 0029 (HNSW)
|
||||
|
||||
Migration `0029` builds `ix_chunks_embedding_hnsw` — an HNSW graph over the
|
||||
existing `vector(768)` column (`m = 16`, `ef_construction = 64`), so the
|
||||
cosine tier of every chat turn stops sequential-scanning
|
||||
[`benchmarks/phase188_live_gate/`](benchmarks/phase188_live_gate/README.md).
|
||||
Everything changes about the **plan**; nothing changes about the **answer**
|
||||
(the measured parity is in the benchmark link above and in
|
||||
`tests/integration/test_chunks_hnsw_recall.py`). It is still the migration
|
||||
to run deliberately rather than as a side effect of a deploy.
|
||||
|
||||
**Pre-flight — all four, on the machine that runs it.**
|
||||
|
||||
```bash
|
||||
free -m # free RAM, whole box
|
||||
psql "$BOR_DATABASE_URL" -c "show shared_buffers;" # + maintenance_work_mem
|
||||
psql "$BOR_DATABASE_URL" -c "select extversion from pg_extension where extname='vector';"
|
||||
podman exec brain-of-reese_db_1 df -h /dev/shm # the one people miss
|
||||
```
|
||||
|
||||
* **pgvector ≥ 0.5.0** (`0.8.0` measured). HNSW does not exist before 0.5 —
|
||||
the migration fails with `access method "hnsw" does not exist`.
|
||||
* **Disk:** budget **~3.9 KB per chunk row** — the real ~414k chunks need
|
||||
roughly **1.6 GB** for the index on top of the ~3 GB heap. A build that
|
||||
runs out of disk is the one failure the self-heal cannot talk its way out
|
||||
of.
|
||||
* **RAM:** the build itself is not the hungry part — a 416,702-row build
|
||||
peaked at **81 MB** of RSS in the `alembic` process at the container's
|
||||
default `maintenance_work_mem` (64 MB), and 1.6 GB of index is page cache
|
||||
afterwards, which Linux drops under pressure. What the box needs is room
|
||||
for the build to finish while the app keeps serving.
|
||||
* **`/dev/shm`:** see the `maintenance_work_mem` note below — inside this
|
||||
project's compose container it is **63 MB**, and that is what caps the
|
||||
knob, not the box's RAM.
|
||||
|
||||
**Build it.**
|
||||
|
||||
```bash
|
||||
uv run alembic upgrade head
|
||||
```
|
||||
|
||||
`CREATE INDEX CONCURRENTLY` takes `ShareUpdateExclusiveLock`: reads and
|
||||
writes continue, the app keeps answering through the whole build, and the
|
||||
build takes **~161 s for 416,702 vectors** at `maintenance_work_mem = 64 MB`
|
||||
(measured, `production_order_build.log`) — budget minutes, not seconds, and
|
||||
more on a 4 GB-class box.
|
||||
|
||||
*On `maintenance_work_mem`:*
|
||||
|
||||
* Never rely on the PostgreSQL **4 MB** default if you can set the knob at
|
||||
all — a 4 MB graph build spills and is dramatically slower.
|
||||
* **But do not raise it above `/dev/shm` in this container.** D5's original
|
||||
suggestion (`SET maintenance_work_mem = '512MB'`) was run at production
|
||||
order and failed:
|
||||
`could not resize shared memory segment "/PostgreSQL.3366409952" to 533761504 bytes: No space left on device`
|
||||
— the container's `/dev/shm` is **63 MB**, and the dying build leaves an
|
||||
**INVALID** index behind (`build_maintenance_work_mem_512MB_failure.log`).
|
||||
The migration's self-heal then did its job (dropped the INVALID graph,
|
||||
rebuilt it valid), so nothing was lost, but the recorded procedure for
|
||||
THIS container is: **`alembic upgrade head` with the container default
|
||||
(64 MB)**, which costs time and not correctness. Raising
|
||||
`maintenance_work_mem` only makes sense together with a larger
|
||||
`--shm-size` / `shm_size:` on the db container.
|
||||
|
||||
**`CONCURRENTLY` semantics, in the owner's terms.** No read or write is
|
||||
blocked. The build is long, and if it dies — OOM killer, cancel, network
|
||||
drop, container restart — Postgres does **not** roll it back: it leaves the
|
||||
index present and marked `indisvalid = false`. That state is silent (it
|
||||
costs insert time on every sync and serves no query), which is why `0029`
|
||||
post-validates its own build and **raises** instead of reporting success.
|
||||
|
||||
**Recovery.**
|
||||
|
||||
```bash
|
||||
# "did it actually build?" — one row, `t` is the answer
|
||||
psql "$BOR_DATABASE_URL" -c "select i.indisvalid from pg_index i
|
||||
join pg_class c on c.oid = i.indexrelid where c.relname = 'ix_chunks_embedding_hnsw';"
|
||||
|
||||
# the recovery for BOTH cases is the same command:
|
||||
uv run alembic upgrade head
|
||||
```
|
||||
|
||||
* `indisvalid = f` after a killed build → `upgrade head` drops the INVALID
|
||||
remnant and rebuilds (the migration's own self-heal).
|
||||
* `relation "ix_chunks_embedding_hnsw" already exists` → the build committed
|
||||
but the version stamp did not land. `0029` handles the INVALID case;
|
||||
a VALID leftover of the same name is dropped by hand and re-run:
|
||||
`DROP INDEX CONCURRENTLY ix_chunks_embedding_hnsw` then
|
||||
`alembic upgrade head`.
|
||||
* Watching a long build from a second terminal:
|
||||
`select * from pg_stat_progress_create_index;`
|
||||
* Rolling back is cheap and touches no data:
|
||||
`uv run alembic downgrade -1` is `DROP INDEX CONCURRENTLY IF EXISTS` — an
|
||||
index is a derived structure, and the schema underneath is 0028's exactly.
|
||||
|
||||
**After it lands.** Turn on the index in the app's own terms and confirm the
|
||||
owner-visible parity, in this order:
|
||||
|
||||
```bash
|
||||
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
|
||||
uv run python -m scripts.api_payload_audit # the app must be running
|
||||
```
|
||||
|
||||
The battery's verdicts should read exactly as they did before the build
|
||||
(the A7/A8 contract — `top_score` included). The tier's own speed is the
|
||||
part that changed: **~550 ms → ~12–15 ms** per cosine search at production
|
||||
order, and the plan flips from `Limit → Sort → Seq Scan on chunks` to
|
||||
`Limit → Nested Loop → Index Scan on ix_chunks_embedding_hnsw` — verifiable
|
||||
with `EXPLAIN` on the tier's statement, or by reading
|
||||
`tests/integration/test_chunks_hnsw_recall.py`'s pin of the same thing.
|
||||
|
||||
**Order of operations on a 4 GB-class box.** Land the phase 183–187 memory
|
||||
work first (`passive_deletes` cascade, the session-release cadence, the
|
||||
prune's six-column read, the deferred-`content` fast path, the reported
|
||||
`rss_mb`/`rss_peak_mb` pair — see
|
||||
[Operational notes: the sync's memory](#operational-notes-the-syncs-memory-phase-187)).
|
||||
The box that OOM-killed the 2026-10-15 sync is the box this build lands on;
|
||||
1.6 GB of new index plus a graph build on top of a sync that still holds its
|
||||
walked documents is how a build gets killed and leaves an INVALID graph for
|
||||
somebody else to notice.
|
||||
|
||||
**Reverse-proxy body size:** a legal chat/save POST is now large — one
|
||||
conversation can carry a full-length answer (see
|
||||
[Conversation payload limits](#conversation-payload-limits)) — so an
|
||||
|
||||
Reference in New Issue
Block a user