# Phase 169 — Embedding generations: stamp every vector, measure comparability, re-embed in place

**Source:** owner request 2026-10-09 — "I'm about to upgrade the embedding model from embedding gemma to embedding gemma 2. The dimensions are the same, but the embeddings may not be comparable. What do I need to do on brain of reese's side?"
**Story:** n/a (owner-directed infrastructure work)
**Context:** `app/models.py:232-249` (`Chunk`, `embedding: Vector(EMBEDDING_DIM)` at L241 — the ONLY vector column in the schema), `app/config.py:102` (`llm_embed_model`), `:142` (`embedding_dim=768`), `:151` (`relevance_threshold=0.62`), `:160` (`lexical_support_floor=0.35`), `:179` (`source_usefulness_floor=0.35`), `:225` (`embed_question_max_chars`), `:548-578` (the two floor ⇄ threshold validators), `app/rag/llm.py:314-322` (`EmbeddingDimensionError` — the ONLY guard, dim-based), `:213-221` (`_TOKENS_PER_CHAR` / `_REQUEST_TOKEN_BUDGET`, written against the current endpoint's ~1024-token cap), `:346` (`embed()`), `:400` (`embed_one()`), `app/rag/importer.py:1008-1023` (content-chunk write), `:1587-1589` (the `is_summary` `position = -1` chunk), `:51-57` (the phase-118 "content-unchanged work is NOT a KB change" convention), `app/rag/retriever.py:433-455` (`_vector_list`, the brute-force cosine scan), `app/api/docs.py:488-491` (the manual-summary write), `app/api/chat.py:740` (the question embed), `app/api/sync.py:844` (`sync: done`), `app/core/auth.py:83` (`require_admin`), `app/main.py:146-172` (router includes), `app/core/settings_overlay.py` + `app/core/settings_catalog.py` (phase 168: the DB-over-env overlay — a stored override BEATS `.env`), `alembic/versions/0028_app_settings.py` (head — next free: **0029**), `scripts/regen_folder_summaries.py` (the one-shot entrypoint idiom this phase's script mirrors), `scripts/eval_retrieval.py`, `scripts/test_embed_model.py`, `scripts/llm_probe.py`, `tests/fixtures/retrieval_battery.txt` (the 43-line gated battery), `benchmarks/model_benchmarks.csv`, `tests/e2e/mock_llm.py` (`POST /v1/embeddings` → deterministic 768-dim bag-of-words), `tests/e2e/conftest.py:53,64` (the global env pins) and `:105-186` (the `app_server` env contract), `README.md:1408-1410` (the "Embedding dimension mismatch" troubleshooting entry).

## Objective

An embedding-model change that KEEPS the dimension (768 → 768) is today invisible and therefore silently destructive: nothing records which model produced a vector, `EmbeddingDimensionError` cannot fire, and cosine scores from two different geometries get compared on one list. This phase makes a vector's generation a fact in the database (`chunks.embedding_model`), makes a mismatch loud in the admin UI and the sync log, ships the two tools that answer "are the old vectors still usable?" (`--compare`, measured in cosine agreement) and "make them usable" (`--apply`, a resumable in-place re-embed over the text already stored), and writes the operator runbook the live upgrade follows.

## The problem, precisely (the executor reads this, not the chat)

- **No generation is recorded.** `chunks.embedding` is a bare `Vector(768)`; `grep -rn "embedding_model" app/ alembic/` is empty. A row does not know what made it.
- **No guard can fire.** The only loud failure is `EmbeddingDimensionError` (`app/rag/llm.py:314-322`), which compares dimension against `BOR_EMBEDDING_DIM`. Same dims → no error, ever.
- **A re-sync re-embeds NOTHING.** `documents.file_hash` matches → the importer returns through `_unchanged_doc` with "ZERO embed" (`app/rag/importer.py:70-84`, `:1391-1403`). Flipping `BOR_LLM_EMBED_MODEL` and pressing Sync leaves 100 % of the corpus in the old space while every QUESTION embeds in the new one (`app/api/chat.py:740`) — retrieval degrades with no error anywhere.
- **The thresholds are model-calibrated, not absolute.** `relevance_threshold=0.62` / `lexical_support_floor=0.35` / `source_usefulness_floor=0.35` gate honesty (A8) and citation tiering, and `app/config.py:147-150` records that they were hand-re-tuned on 2026-08-21 *for the current model's* cosine distribution ("the default never discriminated"). A new model moves the whole distribution.
- **No ANN index to rebuild** — `pg_indexes` for `chunks` is `chunks_pkey`, `ix_chunks_document_id`, `ix_chunks_tsv` only; `_vector_list` is a brute-force scan. (If a hnsw/ivfflat index is ever added, it must be built AFTER a re-embed.)
- **Scale:** the live KB (and `data/bor_live_kb_replica.dump.sql`) is **1,000 documents / 8,866 chunks**, ~1,000 of them the `is_summary` `position = -1` row. At the benchmarked 93.8 vec/s a full re-embed is minutes — so a quiesced in-place run beats a blue/green column.

## Design (shared by all tasks)

**D1 — One nullable `chunks.embedding_model` text column (task 01, migration `0029`).** The value is the `llm_embed_model` string that was in effect when the vector was written. Nullable and **never backfilled by the migration**: for pre-phase rows the true producing model is genuinely unknown, and writing the current setting into them would be a lie that survives the very migration this phase exists to make detectable. `NULL` therefore means *unknown* and is reported as its own bucket. No index: the column is read by `GROUP BY` audits and by the re-embed's `WHERE` over ≤ 10⁴ rows, where a seq scan beats maintaining an index on a low-cardinality column on every chunk write.

**D2 — One write helper, so a fourth path cannot forget (task 02).** `app/core/embedding_generation.py::stamp_embedding(chunk, vector, model)` sets `embedding` and `embedding_model` together. Every path that writes a vector goes through it: the content chunks (`app/rag/importer.py:1008-1023`), the summary chunk (`:1587-1589`), and the manual-summary edit (`app/api/docs.py:488-491`). A unit pin asserts no `Chunk` vector write in `app/` bypasses it (source-scan style, the house static-pin idiom).

**D3 — Comparability is MEASURED, not assumed (task 04).** `scripts/reembed_embeddings.py --compare` takes N chunks (`--sample`, default 25, `ORDER BY id` — deterministic, no RNG), embeds their **stored** `content` with the endpoint/env it is given, and reports the cosine distribution (mean / median / p10 / min) against the **stored** vectors. Bands (LOCKED A4): mean ≥ **0.995** → `COMPATIBLE` (same geometry — stamp, don't re-embed); mean < **0.90** → `INCOMPATIBLE` (re-embed); between → `DRIFT` (re-embed + flag). The 0.995 bar exists because pgvector stores `float4` while the app writes Python floats: an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never to exactly 1.0. This is the tool that answers the owner's actual question instead of guessing, and it doubles as the post-migration smoke test (it must read ≈ 1.0 afterwards).

**D4 — Three modes, one script (task 04), mirroring `scripts/regen_folder_summaries.py`'s entrypoint idiom** (`get_settings()` + `configure_logging` + `configure_debugging()` + `ensure_admin_configured`, one event loop, one `LLMClient`, `DEBUGPY` honored and never imported by default — AGENTS rule 7):
- `--compare` (default when no mode flag): read-only, prints the verdict, exits 0 on COMPATIBLE and non-zero otherwise so the runbook can branch on it. **It never writes.** Because it is its own process it can be run on prod with `BOR_LLM_EMBED_MODEL=<new> uv run python -m scripts.reembed_embeddings --compare` while the app keeps serving the OLD model — the measurement does not touch the running app.
- `--stamp-only`: set `embedding_model` to the configured model on every chunk that has a vector, touching no vector. Valid **only** on a COMPATIBLE verdict — the runbook's step for "the endpoint changed name but not geometry".
- `--apply`: re-embed in place keyed on `chunk.id`, selecting `WHERE embedding_model IS DISTINCT FROM :configured`, committing per batch, resumable by construction (re-running finishes what is left, so a mid-run outage is never a dead end). Batching is delegated to `LLMClient.embed()` — no second batching implementation. Per-batch progress plus one PLAN §9 summary line: `reembed: model=<stamp> mode=… chunks=… batches=… remaining=… duration_ms=…`.
- Both writers **fail loud**: an `EmbeddingError` on a batch logs, stops, exits non-zero and prints the resume command. A chunk is NEVER skipped into `embedding = NULL` — a NULL embedding silently disappears from `_vector_list` (`app/rag/retriever.py:433-455`), which is the exact silent-degradation class this phase kills.
- Both writers **never** touch `content`, `position`, `tsv`, `documents.*`, or `sources_meta.version` (the phase-118 convention, `app/rag/importer.py:51-57`: content-unchanged work is not a KB change, so saved chats must not go stale because the vectors were refreshed).

**D5 — A mismatch is loud in three places (tasks 03, 05, 06), and never silent:**
- `app/core/embedding_generation.py::generation_report(session, expected_model)` → a pure, unit-tested counts object: `total`, `with_vector`, `null_vectors`, `by_model` (with a `None`/unknown bucket), `mismatched`, `unknown`, `stale`.
- **After every sync** (`app/api/sync.py`, right after the `sync: done` line at `:844`): one INFO line `embeddings model=<configured> chunks=N mismatched=M unknown=K`. A **WARNING** only when `mismatched > 0`; `unknown > 0` alone stays INFO, because every pre-phase KB is 100 % unknown and would scream on every sync forever.
- **Admin endpoint** `GET /api/admin/embeddings` (router-wide `Depends(require_admin)`, the `tokens`/`ui_settings` pattern) returning that report as the JSON the card and the runbook both read. Payload is a handful of numbers — it must stay under the phase-167 byte budget and appears in the `grep ' :: api '` audit.
- **Settings view card** (phase 168's `#view-settings`, task 06): a read-only "Embedding generation" card above the settings table — `role="status"` when clean, `role="alert"` naming the offending stamp, the chunk counts, and the remediation command (`uv run python -m scripts.reembed_embeddings --apply`). Text states the state; never color alone (B5).

**D6 — Retrieval stays unfiltered, deliberately (LOCKED A3).** No `WHERE embedding_model = :model` in `_vector_list`: during a backfill it would silently return nothing and deflect every turn (a worse failure than the one being fixed), it costs a predicate on the hot path of every turn, and the migration window is minutes. The column is for DETECTION; the script is for REPAIR. This rejection is recorded in the code docstring so the next reader does not "fix" it.

**D7 — Threshold re-tuning is operator work, backed by tooling (task 07 docs).** The phase ships the measurement + the runbook; it does NOT change `relevance_threshold` / `lexical_support_floor` / `source_usefulness_floor` defaults, because the right numbers depend on the model the owner actually points at. The runbook makes the order explicit: re-embed FIRST, then re-tune (retuning against a mixed corpus tunes to noise), then verify with `tests/fixtures/retrieval_battery.txt` and the deflection rate. It also names the phase-168 trap: those keys may now be `app_settings` rows, and **a stored override beats `.env`** (`app/core/settings_overlay.py`) — so `select key, value from app_settings where key in ('relevance_threshold','lexical_support_floor','source_usefulness_floor','llm_embed_model');` is step one of any value change.

## Dependencies

- `168_admin_settings_panel` (**todo — must complete first**, this phase runs after it in the pipeline) — the overlay that makes `app_settings` beat `.env` (D7's trap), the catalog-completeness pin, and the `#view-settings` view + `/api/admin/settings` + `tests/e2e/test_admin_settings_panel.py` that this phase's card and E2E extend.
- `97_kb_tree_catalog` + `156_table_ux_polish` (complete) — the admin catalogue/bounded-panel and `.cell-clip` contracts the card must not violate (AGENTS rule 5).
- `167_api_payload_diet` (complete) — the 256 KB response budget + `scripts.api_payload_audit` the new endpoint is audited against.
- `82_security_headers` + `78_static_background` + `54_asset_cache_bust_revalidation` (complete) — the CSP/`default-src 'self'` and locally-served-assets posture the card's JS lives inside (AGENTS rule 6, the No CDN Rule), and the `?v=` busting the E2E cache contract keeps.
- `119_name_signal_read_chips` + `162_search_recall_small_sources` (complete) — own `tests/fixtures/retrieval_battery.txt`, which is this phase's measuring stick; the phase READS it and never edits it (the gated items are the retuning ruler).
- `118_summary_seed_context` (complete) — the "content-unchanged work is not a KB change" convention (`app/rag/importer.py:51-57`) that D4 inherits for `sources_meta`.
- `122_image_documents` + `137_pdf_image_context` (complete) — image docs store the vision description as `chunks.content`, which is what makes K3 (a re-embed never re-calls vision) true.

## Tasks

1. `01_embedding_model_column.md` — `Chunk.embedding_model` + reversible migration `0029_chunks_embedding_model.py` + its up/down test.
2. `02_stamp_write_paths.md` — `stamp_embedding()` + the three vector-write call sites routed through it + the no-bypass source pin.
3. `03_generation_report_and_logs.md` — `generation_report()` + the post-sync INFO/WARNING lines + the greppable log contract.
4. `04_reembed_script.md` — `scripts/reembed_embeddings.py` with `--compare` / `--stamp-only` / `--apply`, the pure cosine helpers, resumability, the §9 summary line, unit + integration coverage.
5. `05_admin_embeddings_endpoint.md` — `GET /api/admin/embeddings` (admin-only), the schema, the auth matrix, the byte-budget pin.
6. `06_settings_status_card.md` — the read-only "Embedding generation" card in `#view-settings` (`role=status`/`role=alert`, text-first).
7. `07_e2e_generation_and_docs.md` — `tests/e2e/test_embedding_generation.py`, the README runbook + AGENTS quick-reference updates, the gates, the atomic commit.

## Testing & Quality

- **Unit:** `tests/unit/test_embedding_generation.py` (`stamp_embedding` sets both fields; `cosine` on orthogonal / identical / zero-norm input; `generation_report` buckets — clean, mismatched, NULL-vector rows, unknown stamps, empty table), `tests/unit/test_reembed_embeddings.py` (the `scripts.reembed_embeddings` import idiom of `tests/unit/test_regen_folder_summaries.py:34`: mode parsing, COMPATIBLE/DRIFT/INCOMPATIBLE band edges at exactly 0.995 and 0.90, the deterministic `ORDER BY id` selection, `--apply` failing loud without writing a NULL, `--stamp-only` not touching `embedding`, no `sources_meta` bump), `tests/unit/test_embedding_write_paths.py` (the no-bypass source pin: no `.embedding =` / `embedding=` vector write in `app/` outside `stamp_embedding`), `tests/unit/test_settings_frontend_view.py` extension or a sibling (the card exists, no CDN, DOM built with `createElement`/`textContent`).
- **Integration:** `tests/integration/test_migration_0029.py` (up AND down, A13), `tests/integration/test_admin_embeddings_api.py` (auth matrix anonymous / token-user / admin; counts correct against a seeded corpus with mixed stamps and one NULL-embedding chunk; the response body carries no API key), `tests/integration/test_sync_embedding_log.py` (the post-sync INFO line and the WARNING only on `mismatched > 0`), and an importer pin that a synced doc's content chunk AND its `is_summary` chunk both carry the configured stamp.
- **Coverage:** **>90%** on `app/` — `uv run pytest --cov=app --cov-report=term-missing`.
- **E2E (AGENTS rule 4):** `uv run pytest tests/e2e/test_embedding_generation.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_admin_settings_panel.py`, `test_admin_auth.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_llm_retry.py`, `test_sync_button.py`, `test_cache_busting.py`.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
- **Log contract (AGENTS rule 10):** the two new greppable lines — `embeddings model=… chunks=… mismatched=… unknown=…` (post-sync) and `reembed: model=… mode=… chunks=… batches=… remaining=… duration_ms=…` — plus `GET /api/admin/embeddings` present in the byte audit and under the bar (`uv run python -m scripts.api_payload_audit`).

## Documentation & Consistency

- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it will stand after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Task 07 rewrites `README.md`'s "Embedding dimension mismatch" troubleshooting entry (`:1408-1410`) — it currently says the fix is "drop + recreate the chunks table", which is right for a DIM change and dangerously incomplete for a same-dim swap — and adds a "Switching the embedding model" runbook section; `AGENTS.md`'s quick reference gains the compare/apply commands.

## Completion Criteria

- [ ] `chunks.embedding_model` exists (migration `0029`, reversible), and EVERY vector write in `app/` stamps it — proven by the no-bypass pin and by an importer integration pin covering both a content chunk and the `position = -1` summary chunk.
- [ ] `uv run python -m scripts.reembed_embeddings --compare` prints a mean/median/p10/min cosine agreement and a `COMPATIBLE` / `DRIFT` / `INCOMPATIBLE` verdict, writes nothing, and exits non-zero unless COMPATIBLE — proven against a mock endpoint on a seeded corpus (identical vectors → COMPATIBLE; different vectors → INCOMPATIBLE).
- [ ] `--apply` re-embeds only the chunks whose stamp differs, commits per batch, is resumable (a second run reports `remaining=0`), never leaves an embedding NULL, and does not bump `sources_meta.version` or touch `documents`/`tsv`.
- [ ] A mixed-generation KB is LOUD: `GET /api/admin/embeddings` reports the counts, `#view-settings` shows a `role="alert"` card naming the stamp and the repair command, and the post-sync log WARNs. An all-`unknown` (pre-phase) KB WARNs nothing and shows a `role="status"` card.
- [ ] Retrieval is UNCHANGED by the new column — a mixed-stamp corpus still returns its hits (the D6 pin: detection, not filtering).
- [ ] `README.md` has the runbook (measure → decide → back up → migrate → re-embed → verify → retune → smoke) including the phase-168 `app_settings`-beats-`.env` warning; `AGENTS.md`'s quick reference lists both script modes.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
- [ ] no behavior change in completed phases (retrieval order, the A8 gate, sync counts, saved-chat staleness all untouched).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.

## Locked decisions (owner-confirmed 2026-10-09 during phase authoring)

- **A1 — one nullable `chunks.embedding_model` text column**, stamped with the `llm_embed_model` in effect at write time; `NULL` means *unknown* and the migration NEVER backfills a guess.
- **A2 — every vector write goes through one helper** (`stamp_embedding`) so no path can desynchronize the pair; a source-scan test enforces it.
- **A3 — detection only.** Retrieval never filters on the stamp — a mixed corpus still ranks. (Rejected: a `WHERE embedding_model = …` predicate in `_vector_list`, which silently deflects every turn during a backfill and taxes the hot path; the reason is in the code docstring.)
- **A4 — comparability is measured in cosine agreement**, bands `COMPATIBLE ≥ 0.995 > DRIFT ≥ 0.90 > INCOMPATIBLE`, with the float4-round-trip rationale for the 0.995 bar; `--stamp-only` is legal only on COMPATIBLE.
- **A5 — per-batch commits + a resumable `WHERE embedding_model IS DISTINCT FROM :configured` selection**, fail-loud on any batch error, never a NULL embedding, and NEVER a `sources_meta.version` bump (a re-embed is not a KB change — phase 118's convention).
- **A6 — NO new `Settings` field.** Every knob is a CLI flag, so `tests/unit/test_settings_catalog.py`'s `set(CATALOG) == set(Settings.model_fields)` pin is untouched and the phase-168 panel stays display-only for this feature.
- **A7 — one story, one phase, one Playwright file** (AGENTS rule 4): the backend is covered by unit + integration, the visible contract by `tests/e2e/test_embedding_generation.py`.
- **A8 — the three A8/A13 tuning defaults are NOT changed by this phase** — they are operator-side, post-re-embed, battery-driven (D7). If the owner's post-migration battery says a default is now wrong, that is a separate, deliberate change.

## Assumptions (the request left these open; the executor does not guess)

- **K1:** the stamp is the model NAME. An operator who re-points the same alias server-side (aipi keeps serving `embed` but now runs gemma-2) is caught by `--compare`, not by the column — the column cannot see a name that did not change, and faking it with a vector fingerprint per row costs an endpoint call per write. This boundary is stated in the column docstring and the README runbook.
- **K2:** the corpus is ≤ ~10⁴ chunks (live and replica: 8,866), so one quiesced in-process pass is minutes; no parallelism, no job queue, no progress table (PLAN A12: in-process state, no Valkey).
- **K3:** every embedded text is already in `chunks.content` — image docs store the VISION DESCRIPTION as their content — so a re-embed never re-calls the vision model and never re-reads a PDF.
- **K4:** `--compare`'s sample is `ORDER BY id LIMIT :n` (deterministic across runs; no RNG, no `RANDOM()`), and N defaults to 25 with `--sample` for more.
- **K5:** the card is read-only and admin-only (no controls, no "fix it now" button — a re-embed is a maintenance-window operation, not a click).
- **K6:** no new fixtures and no dump files are committed (`data/` stays gitignored); tests seed small corpora in the house way.
- **K7:** quiescing = "do not run a sync while `--apply` runs" (single app process). The script does not take a lock; the README runbook states the order.
- **K8:** `embedding_dim` stays 768 and stays non-overridable (phase 168 A3); a real DIM change remains the drop-and-recreate path in the README — this phase is for the same-dim swap that path does not cover.

## Commit

```bash
git add app/ alembic/ scripts/ frontend/ tests/ README.md AGENTS.md .agents/phases/ && \
git commit --no-gpg-sign \
  -m "feat(rag): stamp the embedding generation on every chunk and ship the in-place re-embed tooling" \
  -m "A same-dimension embedding-model swap was invisible: nothing recorded which model made a vector, EmbeddingDimensionError cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the file_hash fast path returns with zero embeds. chunks.embedding_model (migration 0029, nullable, never backfilled) now records it, scripts/reembed_embeddings.py measures comparability in cosine agreement (--compare) and repairs in place (--stamp-only / --apply, per-batch commits, resumable, never a NULL embedding, never a sources_meta bump), and a mixed generation is loud in GET /api/admin/embeddings, the Settings card and the post-sync log. Retrieval is deliberately unfiltered: detection, not repair."
```