docs(phases): add phase 169 (embedding generation stamp + in-place re-embed) and the migrate-embed-model skill

An embedding-model swap that keeps the dimension is invisible on this
app: nothing records which model made a vector, EmbeddingDimensionError
cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the
documents.file_hash fast path returns with zero embeds - so questions
embed in one geometry and chunks score in another with no error anywhere.
The owner's upgrade (embedding gemma -> embedding gemma 2) is exactly
that case.

Phase 169, seven tasks: chunks.embedding_model (reversible migration
0029, nullable, never backfilled - NULL means unknown by design), one
write helper so every vector write stamps its generation, a
generation_report() that turns the column into counts plus a stale
verdict logged after every sync, scripts/reembed_embeddings.py with
--compare (cosine agreement against the stored vectors, so 'are they
compatible?' is measured instead of guessed) / --stamp-only / --apply
(per-batch commits, resumable, never a NULL embedding, never a
sources_meta bump), an admin-only GET /api/admin/embeddings, a read-only
card in the phase-168 Settings view, and the dedicated Playwright file
plus the README runbook.

Owner-locked during authoring: detection only - retrieval never filters
by generation (a WHERE embedding_model predicate would silently deflect
every turn mid-backfill), comparability bands COMPATIBLE >= 0.995 for
the float4 round trip, a re-embed is not a KB change so saved chats stay
fresh, and no new Settings field so the phase-168 catalog pin is
untouched.

.agents/skills/migrate-embed-model is the operator twin: probe and
benchmark the new model, measure comparability as its own process
without disturbing the running app, back up and migrate production
(idle check, pg_dump off-box, alembic, flip then re-embed with the same
env), then re-tune relevance_threshold and the two floors against the
retrieval battery - after the corpus is on one generation, and with the
phase-168 warning that an app_settings row beats .env - and smoke-test
the result.

LLM-Generated: true

.agents/phases/todo/169_embedding_generation_migration/00_phase.md

@@ -0,0 +1,121 @@
+# Phase 169 — Embedding generations: stamp every vector, measure comparability, re-embed in place
+
+**Source:** owner request 2026-10-09 — "I'm about to upgrade the embedding model from embedding gemma to embedding gemma 2. The dimensions are the same, but the embeddings may not be comparable. What do I need to do on brain of reese's side?"
+**Story:** n/a (owner-directed infrastructure work)
+**Context:** `app/models.py:232-249` (`Chunk`, `embedding: Vector(EMBEDDING_DIM)` at L241 — the ONLY vector column in the schema), `app/config.py:102` (`llm_embed_model`), `:142` (`embedding_dim=768`), `:151` (`relevance_threshold=0.62`), `:160` (`lexical_support_floor=0.35`), `:179` (`source_usefulness_floor=0.35`), `:225` (`embed_question_max_chars`), `:548-578` (the two floor ⇄ threshold validators), `app/rag/llm.py:314-322` (`EmbeddingDimensionError` — the ONLY guard, dim-based), `:213-221` (`_TOKENS_PER_CHAR` / `_REQUEST_TOKEN_BUDGET`, written against the current endpoint's ~1024-token cap), `:346` (`embed()`), `:400` (`embed_one()`), `app/rag/importer.py:1008-1023` (content-chunk write), `:1587-1589` (the `is_summary` `position = -1` chunk), `:51-57` (the phase-118 "content-unchanged work is NOT a KB change" convention), `app/rag/retriever.py:433-455` (`_vector_list`, the brute-force cosine scan), `app/api/docs.py:488-491` (the manual-summary write), `app/api/chat.py:740` (the question embed), `app/api/sync.py:844` (`sync: done`), `app/core/auth.py:83` (`require_admin`), `app/main.py:146-172` (router includes), `app/core/settings_overlay.py` + `app/core/settings_catalog.py` (phase 168: the DB-over-env overlay — a stored override BEATS `.env`), `alembic/versions/0028_app_settings.py` (head — next free: **0029**), `scripts/regen_folder_summaries.py` (the one-shot entrypoint idiom this phase's script mirrors), `scripts/eval_retrieval.py`, `scripts/test_embed_model.py`, `scripts/llm_probe.py`, `tests/fixtures/retrieval_battery.txt` (the 43-line gated battery), `benchmarks/model_benchmarks.csv`, `tests/e2e/mock_llm.py` (`POST /v1/embeddings` → deterministic 768-dim bag-of-words), `tests/e2e/conftest.py:53,64` (the global env pins) and `:105-186` (the `app_server` env contract), `README.md:1408-1410` (the "Embedding dimension mismatch" troubleshooting entry).
+
+## Objective
+
+An embedding-model change that KEEPS the dimension (768 → 768) is today invisible and therefore silently destructive: nothing records which model produced a vector, `EmbeddingDimensionError` cannot fire, and cosine scores from two different geometries get compared on one list. This phase makes a vector's generation a fact in the database (`chunks.embedding_model`), makes a mismatch loud in the admin UI and the sync log, ships the two tools that answer "are the old vectors still usable?" (`--compare`, measured in cosine agreement) and "make them usable" (`--apply`, a resumable in-place re-embed over the text already stored), and writes the operator runbook the live upgrade follows.
+
+## The problem, precisely (the executor reads this, not the chat)
+
+- **No generation is recorded.** `chunks.embedding` is a bare `Vector(768)`; `grep -rn "embedding_model" app/ alembic/` is empty. A row does not know what made it.
+- **No guard can fire.** The only loud failure is `EmbeddingDimensionError` (`app/rag/llm.py:314-322`), which compares dimension against `BOR_EMBEDDING_DIM`. Same dims → no error, ever.
+- **A re-sync re-embeds NOTHING.** `documents.file_hash` matches → the importer returns through `_unchanged_doc` with "ZERO embed" (`app/rag/importer.py:70-84`, `:1391-1403`). Flipping `BOR_LLM_EMBED_MODEL` and pressing Sync leaves 100 % of the corpus in the old space while every QUESTION embeds in the new one (`app/api/chat.py:740`) — retrieval degrades with no error anywhere.
+- **The thresholds are model-calibrated, not absolute.** `relevance_threshold=0.62` / `lexical_support_floor=0.35` / `source_usefulness_floor=0.35` gate honesty (A8) and citation tiering, and `app/config.py:147-150` records that they were hand-re-tuned on 2026-08-21 *for the current model's* cosine distribution ("the default never discriminated"). A new model moves the whole distribution.
+- **No ANN index to rebuild** — `pg_indexes` for `chunks` is `chunks_pkey`, `ix_chunks_document_id`, `ix_chunks_tsv` only; `_vector_list` is a brute-force scan. (If a hnsw/ivfflat index is ever added, it must be built AFTER a re-embed.)
+- **Scale:** the live KB (and `data/bor_live_kb_replica.dump.sql`) is **1,000 documents / 8,866 chunks**, ~1,000 of them the `is_summary` `position = -1` row. At the benchmarked 93.8 vec/s a full re-embed is minutes — so a quiesced in-place run beats a blue/green column.
+
+## Design (shared by all tasks)
+
+**D1 — One nullable `chunks.embedding_model` text column (task 01, migration `0029`).** The value is the `llm_embed_model` string that was in effect when the vector was written. Nullable and **never backfilled by the migration**: for pre-phase rows the true producing model is genuinely unknown, and writing the current setting into them would be a lie that survives the very migration this phase exists to make detectable. `NULL` therefore means *unknown* and is reported as its own bucket. No index: the column is read by `GROUP BY` audits and by the re-embed's `WHERE` over ≤ 10⁴ rows, where a seq scan beats maintaining an index on a low-cardinality column on every chunk write.
+
+**D2 — One write helper, so a fourth path cannot forget (task 02).** `app/core/embedding_generation.py::stamp_embedding(chunk, vector, model)` sets `embedding` and `embedding_model` together. Every path that writes a vector goes through it: the content chunks (`app/rag/importer.py:1008-1023`), the summary chunk (`:1587-1589`), and the manual-summary edit (`app/api/docs.py:488-491`). A unit pin asserts no `Chunk` vector write in `app/` bypasses it (source-scan style, the house static-pin idiom).
+
+**D3 — Comparability is MEASURED, not assumed (task 04).** `scripts/reembed_embeddings.py --compare` takes N chunks (`--sample`, default 25, `ORDER BY id` — deterministic, no RNG), embeds their **stored** `content` with the endpoint/env it is given, and reports the cosine distribution (mean / median / p10 / min) against the **stored** vectors. Bands (LOCKED A4): mean ≥ **0.995** → `COMPATIBLE` (same geometry — stamp, don't re-embed); mean < **0.90** → `INCOMPATIBLE` (re-embed); between → `DRIFT` (re-embed + flag). The 0.995 bar exists because pgvector stores `float4` while the app writes Python floats: an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never to exactly 1.0. This is the tool that answers the owner's actual question instead of guessing, and it doubles as the post-migration smoke test (it must read ≈ 1.0 afterwards).
+
+**D4 — Three modes, one script (task 04), mirroring `scripts/regen_folder_summaries.py`'s entrypoint idiom** (`get_settings()` + `configure_logging` + `configure_debugging()` + `ensure_admin_configured`, one event loop, one `LLMClient`, `DEBUGPY` honored and never imported by default — AGENTS rule 7):
+- `--compare` (default when no mode flag): read-only, prints the verdict, exits 0 on COMPATIBLE and non-zero otherwise so the runbook can branch on it. **It never writes.** Because it is its own process it can be run on prod with `BOR_LLM_EMBED_MODEL=<new> uv run python -m scripts.reembed_embeddings --compare` while the app keeps serving the OLD model — the measurement does not touch the running app.
+- `--stamp-only`: set `embedding_model` to the configured model on every chunk that has a vector, touching no vector. Valid **only** on a COMPATIBLE verdict — the runbook's step for "the endpoint changed name but not geometry".
+- `--apply`: re-embed in place keyed on `chunk.id`, selecting `WHERE embedding_model IS DISTINCT FROM :configured`, committing per batch, resumable by construction (re-running finishes what is left, so a mid-run outage is never a dead end). Batching is delegated to `LLMClient.embed()` — no second batching implementation. Per-batch progress plus one PLAN §9 summary line: `reembed: model=<stamp> mode=… chunks=… batches=… remaining=… duration_ms=…`.
+- Both writers **fail loud**: an `EmbeddingError` on a batch logs, stops, exits non-zero and prints the resume command. A chunk is NEVER skipped into `embedding = NULL` — a NULL embedding silently disappears from `_vector_list` (`app/rag/retriever.py:433-455`), which is the exact silent-degradation class this phase kills.
+- Both writers **never** touch `content`, `position`, `tsv`, `documents.*`, or `sources_meta.version` (the phase-118 convention, `app/rag/importer.py:51-57`: content-unchanged work is not a KB change, so saved chats must not go stale because the vectors were refreshed).
+
+**D5 — A mismatch is loud in three places (tasks 03, 05, 06), and never silent:**
+- `app/core/embedding_generation.py::generation_report(session, expected_model)` → a pure, unit-tested counts object: `total`, `with_vector`, `null_vectors`, `by_model` (with a `None`/unknown bucket), `mismatched`, `unknown`, `stale`.
+- **After every sync** (`app/api/sync.py`, right after the `sync: done` line at `:844`): one INFO line `embeddings model=<configured> chunks=N mismatched=M unknown=K`. A **WARNING** only when `mismatched > 0`; `unknown > 0` alone stays INFO, because every pre-phase KB is 100 % unknown and would scream on every sync forever.
+- **Admin endpoint** `GET /api/admin/embeddings` (router-wide `Depends(require_admin)`, the `tokens`/`ui_settings` pattern) returning that report as the JSON the card and the runbook both read. Payload is a handful of numbers — it must stay under the phase-167 byte budget and appears in the `grep ' :: api '` audit.
+- **Settings view card** (phase 168's `#view-settings`, task 06): a read-only "Embedding generation" card above the settings table — `role="status"` when clean, `role="alert"` naming the offending stamp, the chunk counts, and the remediation command (`uv run python -m scripts.reembed_embeddings --apply`). Text states the state; never color alone (B5).
+
+**D6 — Retrieval stays unfiltered, deliberately (LOCKED A3).** No `WHERE embedding_model = :model` in `_vector_list`: during a backfill it would silently return nothing and deflect every turn (a worse failure than the one being fixed), it costs a predicate on the hot path of every turn, and the migration window is minutes. The column is for DETECTION; the script is for REPAIR. This rejection is recorded in the code docstring so the next reader does not "fix" it.
+
+**D7 — Threshold re-tuning is operator work, backed by tooling (task 07 docs).** The phase ships the measurement + the runbook; it does NOT change `relevance_threshold` / `lexical_support_floor` / `source_usefulness_floor` defaults, because the right numbers depend on the model the owner actually points at. The runbook makes the order explicit: re-embed FIRST, then re-tune (retuning against a mixed corpus tunes to noise), then verify with `tests/fixtures/retrieval_battery.txt` and the deflection rate. It also names the phase-168 trap: those keys may now be `app_settings` rows, and **a stored override beats `.env`** (`app/core/settings_overlay.py`) — so `select key, value from app_settings where key in ('relevance_threshold','lexical_support_floor','source_usefulness_floor','llm_embed_model');` is step one of any value change.
+
+## Dependencies
+
+- `168_admin_settings_panel` (**todo — must complete first**, this phase runs after it in the pipeline) — the overlay that makes `app_settings` beat `.env` (D7's trap), the catalog-completeness pin, and the `#view-settings` view + `/api/admin/settings` + `tests/e2e/test_admin_settings_panel.py` that this phase's card and E2E extend.
+- `97_kb_tree_catalog` + `156_table_ux_polish` (complete) — the admin catalogue/bounded-panel and `.cell-clip` contracts the card must not violate (AGENTS rule 5).
+- `167_api_payload_diet` (complete) — the 256 KB response budget + `scripts.api_payload_audit` the new endpoint is audited against.
+- `82_security_headers` + `78_static_background` + `54_asset_cache_bust_revalidation` (complete) — the CSP/`default-src 'self'` and locally-served-assets posture the card's JS lives inside (AGENTS rule 6, the No CDN Rule), and the `?v=` busting the E2E cache contract keeps.
+- `119_name_signal_read_chips` + `162_search_recall_small_sources` (complete) — own `tests/fixtures/retrieval_battery.txt`, which is this phase's measuring stick; the phase READS it and never edits it (the gated items are the retuning ruler).
+- `118_summary_seed_context` (complete) — the "content-unchanged work is not a KB change" convention (`app/rag/importer.py:51-57`) that D4 inherits for `sources_meta`.
+- `122_image_documents` + `137_pdf_image_context` (complete) — image docs store the vision description as `chunks.content`, which is what makes K3 (a re-embed never re-calls vision) true.
+
+## Tasks
+
+1. `01_embedding_model_column.md` — `Chunk.embedding_model` + reversible migration `0029_chunks_embedding_model.py` + its up/down test.
+2. `02_stamp_write_paths.md` — `stamp_embedding()` + the three vector-write call sites routed through it + the no-bypass source pin.
+3. `03_generation_report_and_logs.md` — `generation_report()` + the post-sync INFO/WARNING lines + the greppable log contract.
+4. `04_reembed_script.md` — `scripts/reembed_embeddings.py` with `--compare` / `--stamp-only` / `--apply`, the pure cosine helpers, resumability, the §9 summary line, unit + integration coverage.
+5. `05_admin_embeddings_endpoint.md` — `GET /api/admin/embeddings` (admin-only), the schema, the auth matrix, the byte-budget pin.
+6. `06_settings_status_card.md` — the read-only "Embedding generation" card in `#view-settings` (`role=status`/`role=alert`, text-first).
+7. `07_e2e_generation_and_docs.md` — `tests/e2e/test_embedding_generation.py`, the README runbook + AGENTS quick-reference updates, the gates, the atomic commit.
+
+## Testing & Quality
+
+- **Unit:** `tests/unit/test_embedding_generation.py` (`stamp_embedding` sets both fields; `cosine` on orthogonal / identical / zero-norm input; `generation_report` buckets — clean, mismatched, NULL-vector rows, unknown stamps, empty table), `tests/unit/test_reembed_embeddings.py` (the `scripts.reembed_embeddings` import idiom of `tests/unit/test_regen_folder_summaries.py:34`: mode parsing, COMPATIBLE/DRIFT/INCOMPATIBLE band edges at exactly 0.995 and 0.90, the deterministic `ORDER BY id` selection, `--apply` failing loud without writing a NULL, `--stamp-only` not touching `embedding`, no `sources_meta` bump), `tests/unit/test_embedding_write_paths.py` (the no-bypass source pin: no `.embedding =` / `embedding=` vector write in `app/` outside `stamp_embedding`), `tests/unit/test_settings_frontend_view.py` extension or a sibling (the card exists, no CDN, DOM built with `createElement`/`textContent`).
+- **Integration:** `tests/integration/test_migration_0029.py` (up AND down, A13), `tests/integration/test_admin_embeddings_api.py` (auth matrix anonymous / token-user / admin; counts correct against a seeded corpus with mixed stamps and one NULL-embedding chunk; the response body carries no API key), `tests/integration/test_sync_embedding_log.py` (the post-sync INFO line and the WARNING only on `mismatched > 0`), and an importer pin that a synced doc's content chunk AND its `is_summary` chunk both carry the configured stamp.
+- **Coverage:** **>90%** on `app/` — `uv run pytest --cov=app --cov-report=term-missing`.
+- **E2E (AGENTS rule 4):** `uv run pytest tests/e2e/test_embedding_generation.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_admin_settings_panel.py`, `test_admin_auth.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_llm_retry.py`, `test_sync_button.py`, `test_cache_busting.py`.
+- **Lint/types:** `uv run ruff check . && uv run pyright`.
+- **Log contract (AGENTS rule 10):** the two new greppable lines — `embeddings model=… chunks=… mismatched=… unknown=…` (post-sync) and `reembed: model=… mode=… chunks=… batches=… remaining=… duration_ms=…` — plus `GET /api/admin/embeddings` present in the byte audit and under the bar (`uv run python -m scripts.api_payload_audit`).
+
+## Documentation & Consistency
+
+- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it will stand after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
+- Task 07 rewrites `README.md`'s "Embedding dimension mismatch" troubleshooting entry (`:1408-1410`) — it currently says the fix is "drop + recreate the chunks table", which is right for a DIM change and dangerously incomplete for a same-dim swap — and adds a "Switching the embedding model" runbook section; `AGENTS.md`'s quick reference gains the compare/apply commands.
+
+## Completion Criteria
+
+- [ ] `chunks.embedding_model` exists (migration `0029`, reversible), and EVERY vector write in `app/` stamps it — proven by the no-bypass pin and by an importer integration pin covering both a content chunk and the `position = -1` summary chunk.
+- [ ] `uv run python -m scripts.reembed_embeddings --compare` prints a mean/median/p10/min cosine agreement and a `COMPATIBLE` / `DRIFT` / `INCOMPATIBLE` verdict, writes nothing, and exits non-zero unless COMPATIBLE — proven against a mock endpoint on a seeded corpus (identical vectors → COMPATIBLE; different vectors → INCOMPATIBLE).
+- [ ] `--apply` re-embeds only the chunks whose stamp differs, commits per batch, is resumable (a second run reports `remaining=0`), never leaves an embedding NULL, and does not bump `sources_meta.version` or touch `documents`/`tsv`.
+- [ ] A mixed-generation KB is LOUD: `GET /api/admin/embeddings` reports the counts, `#view-settings` shows a `role="alert"` card naming the stamp and the repair command, and the post-sync log WARNs. An all-`unknown` (pre-phase) KB WARNs nothing and shows a `role="status"` card.
+- [ ] Retrieval is UNCHANGED by the new column — a mixed-stamp corpus still returns its hits (the D6 pin: detection, not filtering).
+- [ ] `README.md` has the runbook (measure → decide → back up → migrate → re-embed → verify → retune → smoke) including the phase-168 `app_settings`-beats-`.env` warning; `AGENTS.md`'s quick reference lists both script modes.
+- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
+- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
+- [ ] no behavior change in completed phases (retrieval order, the A8 gate, sync counts, saved-chat staleness all untouched).
+- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
+
+## Locked decisions (owner-confirmed 2026-10-09 during phase authoring)
+
+- **A1 — one nullable `chunks.embedding_model` text column**, stamped with the `llm_embed_model` in effect at write time; `NULL` means *unknown* and the migration NEVER backfills a guess.
+- **A2 — every vector write goes through one helper** (`stamp_embedding`) so no path can desynchronize the pair; a source-scan test enforces it.
+- **A3 — detection only.** Retrieval never filters on the stamp — a mixed corpus still ranks. (Rejected: a `WHERE embedding_model = …` predicate in `_vector_list`, which silently deflects every turn during a backfill and taxes the hot path; the reason is in the code docstring.)
+- **A4 — comparability is measured in cosine agreement**, bands `COMPATIBLE ≥ 0.995 > DRIFT ≥ 0.90 > INCOMPATIBLE`, with the float4-round-trip rationale for the 0.995 bar; `--stamp-only` is legal only on COMPATIBLE.
+- **A5 — per-batch commits + a resumable `WHERE embedding_model IS DISTINCT FROM :configured` selection**, fail-loud on any batch error, never a NULL embedding, and NEVER a `sources_meta.version` bump (a re-embed is not a KB change — phase 118's convention).
+- **A6 — NO new `Settings` field.** Every knob is a CLI flag, so `tests/unit/test_settings_catalog.py`'s `set(CATALOG) == set(Settings.model_fields)` pin is untouched and the phase-168 panel stays display-only for this feature.
+- **A7 — one story, one phase, one Playwright file** (AGENTS rule 4): the backend is covered by unit + integration, the visible contract by `tests/e2e/test_embedding_generation.py`.
+- **A8 — the three A8/A13 tuning defaults are NOT changed by this phase** — they are operator-side, post-re-embed, battery-driven (D7). If the owner's post-migration battery says a default is now wrong, that is a separate, deliberate change.
+
+## Assumptions (the request left these open; the executor does not guess)
+
+- **K1:** the stamp is the model NAME. An operator who re-points the same alias server-side (aipi keeps serving `embed` but now runs gemma-2) is caught by `--compare`, not by the column — the column cannot see a name that did not change, and faking it with a vector fingerprint per row costs an endpoint call per write. This boundary is stated in the column docstring and the README runbook.
+- **K2:** the corpus is ≤ ~10⁴ chunks (live and replica: 8,866), so one quiesced in-process pass is minutes; no parallelism, no job queue, no progress table (PLAN A12: in-process state, no Valkey).
+- **K3:** every embedded text is already in `chunks.content` — image docs store the VISION DESCRIPTION as their content — so a re-embed never re-calls the vision model and never re-reads a PDF.
+- **K4:** `--compare`'s sample is `ORDER BY id LIMIT :n` (deterministic across runs; no RNG, no `RANDOM()`), and N defaults to 25 with `--sample` for more.
+- **K5:** the card is read-only and admin-only (no controls, no "fix it now" button — a re-embed is a maintenance-window operation, not a click).
+- **K6:** no new fixtures and no dump files are committed (`data/` stays gitignored); tests seed small corpora in the house way.
+- **K7:** quiescing = "do not run a sync while `--apply` runs" (single app process). The script does not take a lock; the README runbook states the order.
+- **K8:** `embedding_dim` stays 768 and stays non-overridable (phase 168 A3); a real DIM change remains the drop-and-recreate path in the README — this phase is for the same-dim swap that path does not cover.
+
+## Commit
+
+```bash
+git add app/ alembic/ scripts/ frontend/ tests/ README.md AGENTS.md .agents/phases/ && \
+git commit --no-gpg-sign \
+  -m "feat(rag): stamp the embedding generation on every chunk and ship the in-place re-embed tooling" \
+  -m "A same-dimension embedding-model swap was invisible: nothing recorded which model made a vector, EmbeddingDimensionError cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the file_hash fast path returns with zero embeds. chunks.embedding_model (migration 0029, nullable, never backfilled) now records it, scripts/reembed_embeddings.py measures comparability in cosine agreement (--compare) and repairs in place (--stamp-only / --apply, per-batch commits, resumable, never a NULL embedding, never a sources_meta bump), and a mixed generation is loud in GET /api/admin/embeddings, the Settings card and the post-sync log. Retrieval is deliberately unfiltered: detection, not repair."
+```

.agents/phases/todo/169_embedding_generation_migration/01_embedding_model_column.md

@@ -0,0 +1,33 @@
+# Task 01 — `chunks.embedding_model`: the nullable generation stamp (phase 169 D1, LOCKED A1)
+
+**Phase:** `169_embedding_generation_migration` · **Source:** owner request 2026-10-09 — an embedding gemma → embedding gemma 2 upgrade keeps 768 dims, so nothing detects the swap
+**Story:** n/a
+
+## Objective
+
+`chunks` gains one nullable `embedding_model` text column recording which embedding model produced each vector, with a tested downgrade — the schema fact every later task (stamping, the report, the script, the card) reads.
+
+## Work
+
+1. `app/models.py` — in `class Chunk` (L232), directly after `embedding: Mapped[list[float] | None] = mapped_column(Vector(EMBEDDING_DIM))` (L241), add:
+   - `embedding_model: Mapped[str | None] = mapped_column(Text, nullable=True)`
+   - a comment in the file's established voice stating: the value is the `llm_embed_model` in effect when the vector was written; **`NULL` means UNKNOWN (a pre-phase-169 row), never "no model"**; the dimension cannot distinguish generations because `chunks.embedding` is fixed at 768 by A6 (`EMBEDDING_DIM`, L106), so this column is the ONLY thing that distinguishes two geometries of the same width; and a name change is what is recorded — a server-side re-point that KEEPS the name is detected by `scripts/reembed_embeddings.py --compare`, not here (K1).
+2. New `alembic/versions/0029_chunks_embedding_model.py` — `revision = "0029"`, `down_revision = "0028"` (verify with `uv run alembic heads` first: `0028_app_settings` is head).
+   - `upgrade()`: `op.add_column("chunks", sa.Column("embedding_model", sa.Text(), nullable=True))`
+   - `downgrade()`: `op.drop_column("chunks", "embedding_model")` (A13 — every migration ships a tested downgrade).
+   - NO index, NO `UPDATE`: an index on a low-cardinality column costs every chunk write for an audit `GROUP BY` over ≤ 10⁴ rows (D1), and a backfill would stamp pre-phase rows with a model name nobody verified (A1 — the migration must never guess).
+   - Module docstring: the 2026-10-09 embedding-gemma-2 question, why 768 → 768 is invisible to `EmbeddingDimensionError` (`app/rag/llm.py:314-322`), why a re-sync fixes nothing (the `file_hash` fast path, `app/rag/importer.py:70-84`), and that `NULL` = unknown by design.
+3. New `tests/integration/test_migration_0029.py` — copy the structure of `tests/integration/test_migration_0028.py` (the up/down harness, the temp-database idiom it uses): upgrade `head` puts `embedding_model` on `chunks` (assert through `sa.inspect(...).get_columns("chunks")`), the column is nullable and text-typed, a chunk row can be inserted with it `NULL` and with a value, and downgrade removes it cleanly with the surviving rows intact.
+4. `README.md` — in the schema section (`grep -n "chunks" README.md` to locate the table listing), add the new column to the `chunks` description with the `NULL` = unknown note; leave the troubleshooting rewrite for task 07.
+
+## Testing & Quality
+- Unit/integration: `tests/integration/test_migration_0029.py` (up, down, nullable both ways, rows survive the downgrade). The rest of the model surface is unchanged, so the existing model/migration suites must stay green untouched.
+- Coverage: **>90%** on this task's new/modified code (the migration module is exercised by the up/down test; `app/models.py` is already covered).
+
+## Completion Criteria
+- [ ] `uv run alembic upgrade head` reports `0029` and `\d chunks` (via `podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c '\d chunks'`) shows `embedding_model | text`.
+- [ ] `uv run alembic downgrade -1` then `upgrade head` both succeed (A13), pinned by `tests/integration/test_migration_0029.py`.
+- [ ] No `UPDATE`, no `CREATE INDEX` in the migration; no existing chunk row gains a non-NULL stamp.
+- [ ] `uv run pytest` green; ruff + pyright clean.
+- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide).
+- [ ] no behavior change in completed work (retrieval, sync counts, the A8 gate untouched).

.agents/phases/todo/169_embedding_generation_migration/02_stamp_write_paths.md

@@ -0,0 +1,32 @@
+# Task 02 — Stamp every vector write: `stamp_embedding()` + the three call sites (phase 169 D2, LOCKED A2)
+
+**Phase:** `169_embedding_generation_migration` · **Story:** n/a
+
+## Objective
+
+Every code path that writes `chunks.embedding` also writes `chunks.embedding_model` from the same settings read, so the pair can never desynchronize — enforced by a source-scan test so a future fourth path fails the build rather than shipping a silent lie.
+
+## Work
+
+1. New `app/core/embedding_generation.py` — the phase's home for the generation logic (pure, no HTTP, no `app.db` import at module level; task 03 extends this same module):
+   - `def stamp_embedding(chunk: Chunk, vector: list[float], model: str | None) -> None` — sets `chunk.embedding = list(vector)` and `chunk.embedding_model = model`. Nothing else: no flush, no commit, no validation beyond what SQLAlchemy does (the caller owns the transaction, exactly like the current code).
+   - `def current_embedding_model(settings: Settings | None = None) -> str` — returns `settings.llm_embed_model` (defaults to `get_settings()`), the ONE place the stamp value is derived, so a later move to a fingerprinted stamp changes one function (K1).
+   - Module docstring: why the pair exists (A6 fixes the column at 768 so the dimension cannot distinguish two geometries), the "write the pair together" rule, and the K1 boundary (a name that did not change is invisible here — `scripts/reembed_embeddings.py --compare` is the tool for that).
+2. `app/rag/importer.py` — the content-chunk write (L1008-1023): inside `for row, vec in zip(doc.chunks, vectors, strict=True):` replace `row.embedding = vec` with `stamp_embedding(row, vec, current_embedding_model(settings))`. Read `settings` the way the surrounding function already does (it takes/uses `settings` — do NOT add a second `get_settings()` call inside the loop).
+3. `app/rag/importer.py` — the summary-chunk write in `_store_summary` (L1587-1589): replace `chunk.embedding = vector` with the helper. This is the `is_summary` `position = -1` row — the one an operator would otherwise discover months later still carries the old geometry.
+4. `app/api/docs.py` — the manual-summary edit (L488-491): replace `summary_chunk.embedding = vector` with the helper (the file already constructs `LLMClient()` there; the stamp still comes from `current_embedding_model()`, NOT from the client).
+5. Greppable invariant comment at each of the three sites (`# phase 169: the vector and its generation are written together (LOCKED A2)`) — short, one line each.
+6. New `tests/unit/test_embedding_generation.py` — `stamp_embedding` sets BOTH attributes; a `Chunk` built through it carries the model; calling it twice overwrites both; `current_embedding_model()` reads `llm_embed_model` from a passed `Settings` and from `get_settings()`.
+7. New `tests/unit/test_embedding_write_paths.py` — the no-bypass pin, in the house static-source-scan style (`grep` the `app/` tree from the test, like the existing frontend/CDN pins do): every assignment matching `\.embedding\s*=` or a `Chunk(...)` construction passing `embedding=` inside `app/` lives in `app/core/embedding_generation.py`. Fail with an assertion message naming the offending file:line and the rule ("write vectors through `stamp_embedding` so the generation is stamped with them"). Also pin the reverse: the three call sites still exist (so the pin cannot be satisfied by deleting the calls).
+
+## Testing & Quality
+- Unit: as above.
+- Integration (extend the existing importer/sync suites — locate them with `grep -rln "is_summary" tests/integration`): a synced markdown doc's content chunks AND its `is_summary` chunk all carry `embedding_model == "embed"` (the E2E/fixture model name); a doc edited through the summary API (`app/api/docs.py`) re-stamps its summary chunk; a doc RE-imported with changed bytes re-stamps every content chunk it re-embeds. Run the touched suite, then the whole suite.
+- Coverage: **>90%** on `app/core/embedding_generation.py` and the modified lines.
+
+## Completion Criteria
+- [ ] `grep -rn "\.embedding\s*=" app/ --include=*.py | grep -v __pycache__` shows assignments ONLY inside `app/core/embedding_generation.py`.
+- [ ] A sync against the mock endpoint leaves `select distinct embedding_model from chunks` = one value, the configured `BOR_LLM_EMBED_MODEL`, across content and summary chunks (integration-pinned).
+- [ ] `uv run pytest tests/unit/test_embedding_write_paths.py -v` passes, and adding a bypassing `row.embedding = [...]` anywhere in `app/` makes it fail (verified once by hand, then reverted).
+- [ ] `uv run pytest` green; coverage on the new module >90%; ruff + pyright clean.
+- [ ] no behavior change in completed work — chunk counts, sync summary numbers, `sources_meta` bumps, retrieval order and the A8 gate are all byte-identical to pre-task.

.agents/phases/todo/169_embedding_generation_migration/03_generation_report_and_logs.md

@@ -0,0 +1,33 @@
+# Task 03 — `generation_report()` + the post-sync loud lines (phase 169 D5, LOCKED A3)
+
+**Phase:** `169_embedding_generation_migration` · **Story:** n/a
+
+## Objective
+
+One pure, unit-tested function turns `chunks.embedding_model` into counts-per-generation plus a `stale` verdict, and the sync run reports it on every success — INFO always, WARNING only when chunks actually carry a different generation than the model the app embeds questions with.
+
+## Work
+
+1. `app/core/embedding_generation.py` (from task 02) — add:
+   - `@dataclass(frozen=True) class GenerationReport` with `expected_model: str`, `embedding_dim: int`, `total_chunks: int`, `with_vector: int`, `null_vector: int`, `by_model: tuple[ModelCount, ...]`, `mismatched: int`, `unknown: int`, `stale: bool`. `ModelCount` is a small frozen dataclass `(model: str | None, chunks: int, embedded: int)` — `model is None` is the UNKNOWN bucket.
+   - `def generation_report(db: Session, expected_model: str | None = None, embedding_dim: int | None = None) -> GenerationReport` — ONE aggregate query (`select(Chunk.embedding_model, func.count(), func.count(Chunk.embedding)).group_by(Chunk.embedding_model)`), then pure Python bucketing: `mismatched` = chunks whose stamp is non-NULL and `!= expected_model`; `unknown` = chunks whose stamp IS NULL; `stale = mismatched > 0`. NULL-vector rows stay inside their model's `chunks` count and show up in `embedded < chunks` — a chunk that lost its vector is a separate defect and must remain visible, not folded into the generation story.
+   - **D6 in writing:** the docstring states that nothing here filters retrieval (`app/rag/retriever.py:433-455` keeps its unfiltered cosine scan) and why — a `WHERE embedding_model = …` predicate would silently return nothing mid-backfill and deflect every turn, which is worse than the mixed ranking it prevents, and taxes the hot path of every turn for a migration window measured in minutes (LOCKED A3, rejected alternative).
+2. `app/api/sync.py` — immediately after `logger.info("sync: done detail=%s", _status.detail)` (L844, the SUCCESS terminal only — the cancelled and failed paths log nothing new):
+   - open a short-lived `SessionLocal()` (the file's own convention, L538/L805), call `generation_report(db, settings.llm_embed_model, settings.embedding_dim)`, close it;
+   - always one INFO line, greppable and cron-safe (PLAN §9): `embeddings model=<expected> chunks=<total> mismatched=<n> unknown=<n> null_vector=<n>`;
+   - escalate to **WARNING** only when `mismatched > 0`, appending the repair hint: `— chunks were embedded by another model, run: uv run python -m scripts.reembed_embeddings --apply`. `unknown > 0` alone NEVER warns: a pre-phase-169 KB is 100 % unknown, and a permanent warning is noise that trains the owner to ignore the one signal that matters (D5).
+   - the whole block is wrapped in `except Exception` (broad, with a `# noqa: BLE001` and a comment, matching this file's background-task style): a reporting hiccup logs `embeddings report unavailable: <type>` at WARNING and **never** changes the sync's terminal state, counts, or `_status.detail`.
+3. New `tests/unit/test_embedding_generation_report.py` — clean corpus (all stamps equal → `stale=False`, `mismatched=0`), a mixed corpus (two named stamps → the counts split, `stale=True`), the UNKNOWN bucket (NULL stamps counted, `unknown=N`, NOT counted as mismatched), NULL-vector rows (`with_vector`/`null_vector` correct, `embedded < chunks`), an empty table (all zeros, `stale=False`), and the `expected_model`/`embedding_dim` defaults resolving from `get_settings()`.
+4. New `tests/integration/test_sync_embedding_log.py` — seed a tiny corpus, run the sync the way the existing sync integration suites do (`grep -rln "sync: done\|/api/sync" tests/integration` to find the harness and copy it), and assert with `caplog`: the INFO line on success with the right numbers; the WARNING + repair hint appears after `UPDATE chunks SET embedding_model='older-model'` on a subset; NO warning when the only oddity is `embedding_model IS NULL`; no new line on a cancelled run; a broken report (monkeypatch `generation_report` to raise) leaves the sync terminal `done` and its counts untouched.
+
+## Testing & Quality
+- Unit + integration as above.
+- Coverage: **>90%** on `app/core/embedding_generation.py` (every bucket branch) and the new sync block (success / warning / unknown-only / raising paths).
+
+## Completion Criteria
+- [ ] `generation_report()` returns correct counts for the five pinned corpus shapes, and `stale` is `True` ONLY for a non-NULL stamp mismatch.
+- [ ] A successful sync logs exactly one `embeddings model=… chunks=… mismatched=… unknown=… null_vector=…` line: `grep`-able on stdout in dev (`uv run uvicorn app.main:app --reload` + a sync) and pinned via `caplog`.
+- [ ] The WARNING fires on a mismatch and does NOT fire on an all-unknown corpus; the repair hint names `scripts.reembed_embeddings --apply`.
+- [ ] A raising `generation_report` cannot change a sync's terminal state or counts.
+- [ ] `uv run pytest` green (the existing sync suites untouched), coverage >90% on the new code, ruff + pyright clean.
+- [ ] no behavior change in completed work (the sync's `detail` payload, `sources_meta` bumping and the saved-chat staleness gate are byte-identical).

.agents/phases/todo/169_embedding_generation_migration/04_reembed_script.md

@@ -0,0 +1,42 @@
+# Task 04 — `scripts/reembed_embeddings.py`: measure comparability, then repair in place (phase 169 D3/D4, LOCKED A4/A5)
+
+**Phase:** `169_embedding_generation_migration` · **Story:** n/a
+
+## Objective
+
+One script answers the owner's real question — *"are the stored vectors still usable against this endpoint?"* — by measuring cosine agreement (`--compare`), and repairs the corpus when they are not (`--apply`), with `--stamp-only` for the "endpoint changed name, not geometry" case. This is the tool a same-dimension embedding-model swap has never had.
+
+## Work
+
+1. `app/core/embedding_generation.py` — the pure math + selection, so coverage and unit tests stay clean and the script stays thin glue:
+   - `cosine(a: Sequence[float], b: Sequence[float]) -> float` — pure stdlib (`math.fsum`), `0.0` for a zero-norm operand (never `NaN`/`ZeroDivisionError`), `ValueError` naming both lengths when `len(a) != len(b)` (a length difference is a bug, not a score).
+   - Band constants, documented where they are declared: `COMPATIBLE_MEAN = 0.995` and `INCOMPATIBLE_MEAN = 0.90`, plus `VERDICT_COMPATIBLE / VERDICT_DRIFT / VERDICT_INCOMPATIBLE`. The 0.995 rationale goes in the docstring **verbatim**: pgvector stores `float4` while the app writes Python floats, so an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never to exactly 1.0 — anything meaningfully below that is a different geometry, and "expect exactly 1.0" would make every healthy corpus read as DRIFT.
+   - `@dataclass(frozen=True) class AgreementStats` — `n`, `mean`, `median`, `p10`, `min`, `worst: tuple[tuple[int, float], ...]` (the ≤ 3 lowest-agreement chunk ids), `verdict: str`; `agreement(samples: Iterable[tuple[int, Sequence[float], Sequence[float]]]) -> AgreementStats` computes it (`(chunk_id, stored, fresh)` triples; empty input → `n=0`, `verdict=VERDICT_INCOMPATIBLE` — "nothing measured" is never "compatible").
+   - Selection helpers (all `ORDER BY Chunk.id`, deterministic, no RNG — K4): `sample_chunks(db, limit)` → `id, content, embedding` rows `WHERE embedding IS NOT NULL`; `pending_chunks(db, model, limit=None)` → `id, content` rows `WHERE embedding IS NULL OR embedding_model IS DISTINCT FROM :model`; `count_pending(db, model) -> int`; `count_missing_vector(db) -> int`.
+   - `stamp_rows(db, model, ids) -> int` — the `--stamp-only` UPDATE, **restricted to rows that actually have a vector** (`embedding IS NOT NULL`) so a vector-less row is never stamped as if it did.
+2. New `scripts/reembed_embeddings.py` — the entrypoint mirrors `scripts/regen_folder_summaries.py` EXACTLY (the house one-shot idiom): `get_settings()` → `configure_logging(settings.log_level)` → `configure_debugging()` (debugpy only under `DEBUGPY=1`, never imported otherwise — AGENTS rule 7) → `ensure_admin_configured(settings)`, a boot failure prints `reembed_embeddings: boot failed: <e>` to stderr and returns 1; one `LLMClient()`; one `asyncio.run(...)`; a short-lived `SessionLocal()` per unit of work (closed in `finally`); one cron-safe PLAN §9 summary line.
+   - **Modes** (`argparse mutually exclusive group`, default `--compare`): `--compare` | `--stamp-only` | `--apply`; plus `--sample N` (default 25, compare only), `--limit N` (apply/stamp-only: cap the rows this run touches — the debug/smoke knob), `--dry-run` (apply/stamp-only: report `count_pending` and write nothing).
+   - **Exit codes as module constants**, documented in the parser epilog: `EXIT_OK = 0`, `EXIT_FAILED = 1` (boot/endpoint/DB failure), `EXIT_INCOMPATIBLE = 3` (the measurement answered "no") — so the runbook branches on "the answer is no" without confusing it with "the tool broke".
+   - **First log line of every run**: `reembed: starting mode=<mode> model=<llm_embed_model> dim=<embedding_dim> base=<sanitized>` with the base URL through `app.core.errors.sanitize_error` (the phase-84 SEC-13 rule: a `user:pass@` in `BOR_LLM_BASE_URL` must never reach a log). This line is also the operator's proof the run used the env the app will run with — the rule that keeps the stamp honest (K1).
+   - `--compare`: `sample_chunks` → `await llm.embed([content …])` → `agreement(...)` per pair → print a human table (per-chunk id + cosine, and the aggregate), then `reembed: compare model=… n=… mean=… median=… p10=… min=… verdict=…` and, for a non-COMPATIBLE verdict, one more line stating the repair command (`--apply`) and the caveat that `--stamp-only` is only legal on COMPATIBLE. **It writes nothing, ever** — no session writes, no commit (pin it).
+   - `--apply`: loop `pending_chunks(db, model, limit)` in id-ordered pages (default page 200 rows), per page `vectors = await llm.embed([content …])`, `zip(rows, vectors, strict=True)`, `stamp_embedding(row, vec, model)` (task 02's helper — the pair is written together), `session.commit()` per page. Progress INFO per page: `reembed: progress done=%d pending_total=%d`. **Fail loud:** any `EmbeddingError` logs `reembed: FAILED after %d rows — resume with: uv run python -m scripts.reembed_embeddings --apply` and returns `EXIT_FAILED`; no row is ever written with `embedding=None`, and no row is silently skipped (a NULL embedding vanishes from `_vector_list`, `app/rag/retriever.py:433-455` — exactly the silent class this phase kills).
+   - `--stamp-only`: `stamp_rows(...)`, loud INFO line naming the count, and a WARNING that repeats what must be true for this to be correct ("only valid on a COMPATIBLE --compare verdict against the SAME endpoint").
+   - **Never touched:** `content`, `position`, `tsv`, `documents.*`, `sources_meta.version` (the phase-118 convention, `app/rag/importer.py:51-57` — a re-embed is not a KB change, so saved chats must not go stale). No `chunk_document` call, no PDF read, no vision call (K3 — the embedded text already lives in `chunks.content`, image docs included).
+   - Final summary line (all modes): `reembed: model=<stamp> mode=<mode> chunks=<processed|sampled> batches=<llm.embed_batches> remaining=<count_pending after> duration_ms=<…>`. `remaining=0` after a clean `--apply` is the resumability proof; a second run prints `pending=0` and does nothing.
+   - No whitespace special-case for `content`: the chunker never yields empty chunks, and an endpoint rejection on a pathological row fails loud on the batch rule above rather than being quietly skipped.
+   - Module docstring: the embedding-gemma → gemma-2 question this exists for, the three modes, the bands + float4 rationale, the resumability contract, the "run it with the app's own env" rule, and the K1 boundary (a same-name server-side re-point is invisible to the stamp and is exactly what `--compare` catches).
+3. New `tests/unit/test_reembed_embeddings.py` — the `from scripts import reembed_embeddings as reembed` idiom of `tests/unit/test_regen_folder_summaries.py:34`: mode mutual-exclusion and defaults, `--sample`/`--limit`/`--dry-run` parsing, the exit-code constants, the band edges (mean exactly 0.995 → COMPATIBLE, 0.9949 → not, exactly 0.90 → INCOMPATIBLE boundary handled per the declared rule, empty sample → INCOMPATIBLE), `cosine` (identical → 1.0, orthogonal → 0.0, zero-norm → 0.0 not NaN, length mismatch → `ValueError` naming both lengths), the deterministic sample/pending ordering, and that the base URL in the starting line is sanitized (an `https://user:pass@host` env yields no `user:pass` in the log).
+4. New `tests/integration/test_reembed_apply.py` — seed a small corpus (a handful of docs + chunks) with `embedding_model='old-model'`; with a stubbed `LLMClient` returning known vectors: `--dry-run` writes nothing (`count_pending` unchanged, no UPDATE issued); `--apply` stamps every row, sets the new vectors, leaves `content`/`position`/`tsv`/`documents` and `sources_meta.version` untouched, and a SECOND `--apply` reports `pending=0` / `remaining=0` (the resumability pin); a partial failure (the stub raises `EmbeddingError` on the second page) leaves page 1 committed, exits `EXIT_FAILED`, logs the resume line, and NO row ends up with `embedding IS NULL`; `--stamp-only` changes only `embedding_model` (vector bytes bit-identical — compare against a snapshot) and skips vector-less rows; `--compare` against a stub that echoes the stored vectors → COMPATIBLE / exit 0, against different vectors → INCOMPATIBLE / exit 3, and the chunks table byte-identical afterwards (the read-only pin).
+
+## Testing & Quality
+- Unit + integration as above; the existing `scripts` suites (`tests/unit/test_regen_folder_summaries.py`, `tests/unit/test_llm_probe.py`, `tests/integration/test_import_docs_*.py`) stay green untouched.
+- Coverage: **>90%** on `scripts/reembed_embeddings.py` and on `app/core/embedding_generation.py` (every mode, every band edge, every failure path).
+
+## Completion Criteria
+- [ ] `uv run python -m scripts.reembed_embeddings --help` shows the three modes, `--sample`, `--limit`, `--dry-run` and the exit-code epilog.
+- [ ] Against the restored 1,000-doc replica locally (`restore-test-db` skill) the three modes do exactly what the docstring says — `--compare` prints a verdict and leaves the dump's fingerprint byte-identical; `--apply` on a `UPDATE chunks SET embedding_model='old-model'` corpus re-embeds all 8,866 chunks and reports `remaining=0`; a second run is a no-op.
+- [ ] No code path can write `embedding = NULL`; no path bumps `sources_meta.version`; `--compare` issues no UPDATE (pinned).
+- [ ] `--stamp-only` on a COMPATIBLE corpus leaves every `embedding` value bit-identical and stamps all vector rows.
+- [ ] `uv run pytest` green, coverage >90% on the script and module, ruff + pyright clean.
+- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the full runbook lands in task 07; the phase's final pass re-checks project-wide).
+- [ ] no behavior change in completed work (the app, the sync, and retrieval are untouched by this task — only the new script + helpers).

.agents/phases/todo/169_embedding_generation_migration/05_admin_embeddings_endpoint.md

@@ -0,0 +1,30 @@
+# Task 05 — `GET /api/admin/embeddings`: the generation report as an admin endpoint (phase 169 D5)
+
+**Phase:** `169_embedding_generation_migration` · **Story:** n/a
+
+## Objective
+
+A read-only, admin-only endpoint exposing task 03's `GenerationReport` as JSON — the single source the Settings card (task 06) and the operator runbook (task 07 + the `migrate-embed-model` skill) both read to answer "is this KB on one embedding generation?".
+
+## Work
+
+1. `app/schemas.py` — two response models beside the other read models (the `HealthResponse` style, no `response_model_exclude` tricks):
+   - `EmbeddingModelCount(BaseModel)`: `model: str | None`, `chunks: int`, `embedded: int` — `model=None` is the UNKNOWN bucket, and `embedded < chunks` names the vector-less rows inside that stamp.
+   - `EmbeddingGeneration(BaseModel)`: `configured_model: str`, `embedding_dim: int`, `total_chunks: int`, `with_vector: int`, `null_vector: int`, `by_model: list[EmbeddingModelCount]`, `mismatched: int`, `unknown: int`, `stale: bool`.
+   - Docstring notes: numbers only, no text from the corpus; and NO tuning values here — `relevance_threshold` / the two floors and their `db`/`env`/`default` source badges are phase 168's `GET /api/admin/settings` job, never duplicated (two sources of truth for a tuning value is how a runbook lies).
+2. New `app/api/admin_embeddings.py` — `router = APIRouter(prefix="/admin/embeddings", tags=["embeddings"], dependencies=[Depends(require_admin)])` (the `tokens.py`/`ui_settings.py` router-wide pattern: anonymous AND token-user both 403), one route `GET ""` → `EmbeddingGeneration`, `get_db` session, `generation_report(db, settings.llm_embed_model, settings.embedding_dim)` (task 03) mapped to the schema. Module docstring: what "stale" means (a chunk whose stamped model ≠ the model the app embeds questions with, i.e. two geometries on one cosine list), that detection is not filtering (LOCKED A3), and the repair pointer to `scripts.reembed_embeddings`.
+3. `app/main.py` — register it with the other admin routers (`app.include_router(admin_embeddings_router, prefix="/api")`, next to the `ui_settings`/`tokens` includes at L146-172).
+4. New `tests/integration/test_admin_embeddings_api.py` — the auth matrix copied from the existing admin-surface suite (`grep -rln "require_admin" tests/integration`): anonymous → 403, token user → 403, admin session → 200; against seeded corpora: clean (one stamp → `stale=false`, `mismatched=0`), mixed (two stamps → counts split, `stale=true`, `mismatched` exact), unknown (NULL stamps counted in `unknown` and NOT in `mismatched`), vector-less rows (`null_vector` and the per-stamp `embedded < chunks`), empty DB (all zeros, `stale=false`); the response body contains no `api_key` / `password` / `secret` substring; the payload is tiny (< 4 KB — the phase-167 byte budget is a 256 KB bar, and a handful of counters must never approach it).
+5. Byte-audit check (AGENTS rule 10): with the app running, `uv run python -m scripts.api_payload_audit` lists `/api/admin/embeddings` under the bar, and `grep ' :: api '` shows the route's line.
+
+## Testing & Quality
+- Integration as above; unit coverage comes free from task 03's report tests (the route is a mapping, so pin it here instead of duplicating the bucket tests).
+- Coverage: **>90%** on `app/api/admin_embeddings.py` and the new schemas.
+
+## Completion Criteria
+- [ ] `GET /api/admin/embeddings` returns 200 for the admin and 403 for anonymous and token users (all pinned).
+- [ ] The JSON matches the four seeded corpus shapes exactly, `stale` is true only for a non-NULL mismatch, and `unknown` never counts toward `mismatched`.
+- [ ] No secret substring in the body; the payload is a few hundred bytes and appears under the bar in `scripts.api_payload_audit`.
+- [ ] `uv run pytest` green, coverage >90% on the new route, ruff + pyright clean.
+- [ ] README.md updated for the new endpoint in its API listing (task 07 finishes the runbook; the final pass re-checks project-wide).
+- [ ] no behavior change in completed work.

.agents/phases/todo/169_embedding_generation_migration/06_settings_status_card.md

@@ -0,0 +1,34 @@
+# Task 06 — the "Embedding generation" card in the Settings view (phase 169 D5, AGENTS rule 5)
+
+**Phase:** `169_embedding_generation_migration` · **Story:** n/a
+
+## Objective
+
+The admin sees, unprompted and in the browser, whether the corpus and the configured embedding model agree — one read-only card at the top of phase 168's `#view-settings` that is quiet-clean on a healthy KB and unmistakably loud when chunks were embedded by another model.
+
+## Work
+
+1. `frontend/index.html` — inside `#view-settings` (the phase-168 view), ABOVE the settings `.table-wrap` card, a new card in the file's existing card idiom:
+   - `<section class="card" id="embedding-status" aria-labelledby="embedding-status-title">` with `<h2 id="embedding-status-title">Embedding generation</h2>`, a `<p id="embedding-status-line" role="status" aria-live="polite">` for the state line, and a `<p id="embedding-status-fix" hidden>` wrapping the repair command in a `<code>` element.
+   - No new view, no nav entry, no router change — this is one card inside a route that already exists (`VIEW` / `VIEW_PATH` / `VIEW_MODULES` in `frontend/assets/router.js` are untouched), which is what keeps the phase's blast radius small.
+2. `frontend/assets/settings.js` (the phase-168 module — extend it, do not fork it) — on view activation, `fetch("/api/admin/embeddings")` once and render through ONE function:
+   - **clean** (`stale=false`, `unknown=0`): the line reads `All N chunks are on "<model>" (dim D).` in `role="status"`; the fix paragraph stays `hidden`.
+   - **unknown only** (`stale=false`, `unknown>0`): `role="status"` and the sentence names it as pre-phase rows, not an error — `N of M chunks predate the stamp (unknown model); K have no vector.` plus the informational `--stamp-only`-or-`--apply` hint in the `<code>` line. This never paints as an error (task 03's INFO-not-WARNING rule, mirrored in the UI).
+   - **stale** (`stale=true`): the SAME line element flips to `role="alert"` and names the offending stamp, the count, the configured model, and the command: `8,612 of 8,866 chunks were embedded by "embed" — the app is set to "embed2". Run: uv run python -m scripts.reembed_embeddings --apply`. Numbers via `toLocaleString()`; the model name via `textContent` (never string-built HTML).
+   - **fetch failure**: `role="alert"`, `Embedding generation status unavailable.` — never a blank card and never a stale previous value left on screen (PLAN §7.4 "never stale").
+   - All DOM built with `createElement` / `textContent` (the house rule — no `innerHTML` with fetched data, no CDN, AGENTS rule 6); a `null`-model bucket label is the literal text `unknown`.
+3. `frontend/assets/styles.css` — the card needs no new layout primitives: reuse `.card`, and add `overflow-wrap: anywhere;` to the command `<code>` so the repair command wraps instead of clipping (an operator must be able to SELECT and copy it — an ellipsis would break the one action the card offers). Contrast: the stale state uses the existing warning/danger text token on its background (≥ 4.5:1, the theme tokens phase 92/93 established), and the state is carried by the WORDS and the `role`, never by color alone (B5).
+4. New `tests/unit/test_settings_embedding_card.py` — the static source pins in the house style (read the two files, assert): the section + `aria-labelledby` + `role="status"` + `aria-live="polite"` exist; the fix `<code>` exists and starts `hidden`; `settings.js` references `/api/admin/embeddings`, sets `role` to `alert` on the stale path, uses `textContent` for the model name, contains no `innerHTML` assignment with fetched data, and no `https://` src/href (the No-CDN pin).
+
+## Testing & Quality
+- Unit: the static pins above.
+- Integration: none needed beyond task 05's endpoint suite (the card is client-side; its behavior is E2E-pinned in task 07). If phase 168's view has an integration render pin, extend it with the card's presence for an admin and its absence for a token user.
+- Coverage: **>90%** on the changed JS surface as far as the static pins reach (the runtime paths are covered by task 07's E2E).
+
+## Completion Criteria
+- [ ] An admin opening `/settings.html` sees the card above the settings table, inside the 72rem container, with the correct state for the seeded data (checked in task 07's E2E).
+- [ ] A stale corpus shows `role="alert"` naming the old stamp, the counts, the configured model, and a fully visible, copyable `scripts.reembed_embeddings --apply` command; a clean corpus shows `role="status"` and no command.
+- [ ] An all-unknown corpus is a `role="status"` card, never an alert.
+- [ ] The state is legible with color removed (words + `role` carry it), focus styles are untouched, and no `innerHTML` / CDN appears in the changed JS.
+- [ ] `uv run pytest` green, ruff + pyright clean, `test_cache_busting.py` and `test_admin_theme_tab.py` still green in isolation (the asset-busting and admin-view contracts survive).
+- [ ] no behavior change in completed work (the settings table, its Save/Reset behavior, and `/api/admin/settings` are untouched).

.agents/phases/todo/169_embedding_generation_migration/07_e2e_generation_and_docs.md

@@ -0,0 +1,50 @@
+# Task 07 — E2E for the generation card, the README runbook, the gates, the commit (phase 169, AGENTS rules 4/8/9)
+
+**Phase:** `169_embedding_generation_migration` · **Story:** n/a
+
+## Objective
+
+Prove in a real browser that a mixed embedding generation is visible and that detection does not filter retrieval, write the operator runbook the live upgrade follows, run every gate, and land the phase as one atomic commit.
+
+## Work
+
+1. New `tests/e2e/test_embedding_generation.py` — the phase's dedicated Playwright file (AGENTS rule 4), built on the shared fixtures (`from e2e.auth_helpers import login, login_with_token`, the `page` / `app_url` / `db_ready` fixtures, `from app.db import SessionLocal`, and the `TRUNCATE chunks, documents, query_log, kb_overview, git_sources` cleanup `tests/e2e/test_sync_button.py:177` uses — TRUNCATE before AND after). Seed the corpus the way that suite seeds it, so the mock embeddings endpoint (`tests/e2e/mock_llm.py`: deterministic 768-dim bag-of-words) stamps chunks with `BOR_LLM_EMBED_MODEL`. Tests:
+   - **clean corpus** — `login(page, app_url, next="/settings.html")`, open the view, `#embedding-status-line` has `role="status"`, its text contains the configured model name and the chunk count, and `#embedding-status-fix` is hidden.
+   - **mixed corpus is loud** — `UPDATE chunks SET embedding_model = 'older-model'` for a known subset, reload, assert `#embedding-status-line` has `role="alert"` and that its text names `older-model`, the mismatched count, the configured model, and the literal `scripts.reembed_embeddings --apply`; the command element is visible and its full text is present in the DOM (the AGENTS rule 5 clip contract — nothing ellipsised away).
+   - **detection, never filtering (LOCKED A3)** — with the SAME mixed corpus, ask a chat question whose answer is pinned by the seeded corpus and assert the turn still grounds on the same source as it did before the stamps were changed (the retrieval list is unchanged by the column — this is the pin that stops a future task from "fixing" the mixed state by hiding chunks).
+   - **all-unknown is calm** — `UPDATE chunks SET embedding_model = NULL`, assert `role="status"` (NOT `alert`) and that the text says `unknown` rather than naming an error.
+   - **walled surface** — `page`-less API requests (`APIRequestContext`): anonymous and `login_with_token` users get 403 from `GET /api/admin/embeddings`, the admin gets 200 (mirror phase 168's `test_anonymous_and_token_user_are_walled`).
+   - **the script repairs what the card reports** — run the real script in-process against the E2E database (`BOR_LLM_BASE_URL` points at the mock endpoint, so `--apply` is deterministic and free): set all stamps to `older-model`, `reembed.main(["--apply"])` returns 0, the DB reports one stamp again, and reloading `/settings.html` returns to the clean `role="status"` card. (If `LLMClient()` construction needs the env the subprocess-style fixture provides, drive it with the same env the app_server fixture uses — do not add a new env contract.)
+   - Module docstring: the embedding-gemma → gemma-2 story, the four states, and why the A3 pin matters.
+2. `README.md` — two edits, both substantive:
+   - **New section "Switching the embedding model (same dimension)"** — the operator runbook, in order: (1) baseline the battery and the cosine percentiles; (2) probe the new model (`scripts.llm_probe`, `scripts.test_embed_model`); (3) measure comparability with `uv run python -m scripts.reembed_embeddings --compare` run as its OWN process with `BOR_LLM_EMBED_MODEL=<new>` — which does not disturb the running app — and read the verdict bands (COMPATIBLE ≥ 0.995 / DRIFT / INCOMPATIBLE, with the float4-round-trip reason for the bar); (4) back up (`pg_dump -Fc`, copied off-box, verified); (5) `uv run alembic upgrade head`, then `--stamp-only` on COMPATIBLE or `--apply` otherwise; (6) verify — `--compare` now ≈ 1.0, `GET /api/admin/embeddings` `stale=false`, the Settings card clean, `remaining=0`; (7) ONLY NOW re-tune `BOR_RELEVANCE_THRESHOLD` / `BOR_LEXICAL_SUPPORT_FLOOR` / `BOR_SOURCE_USEFULNESS_FLOOR` against the battery, remembering the two validators cap the floors at the threshold and that **a value stored via the Settings panel (`app_settings`) beats `.env`** (phase 168), so check `select key, value from app_settings where key in ('llm_embed_model','relevance_threshold','lexical_support_floor','source_usefulness_floor');` before and after any change; (8) smoke: the battery's gated items, an out-of-KB question still deflects, and the next N turns' deflection rate in `query_log` looks like the pre-migration one. State the maintenance-window rule (do not run a sync while `--apply` runs) and the K1 boundary (a server-side alias re-point that keeps the model NAME is invisible to the stamp — `--compare` is the detector).
+   - **Rewrite the troubleshooting entry at `:1408-1410`** — "Embedding dimension mismatch" keeps its drop-and-recreate advice (a real DIM change, A6) and gains the pointer that a SAME-dim swap is the invisible one: run `--compare`, then `--apply`; and that the Settings card / `GET /api/admin/embeddings` / the post-sync `embeddings model=… mismatched=…` line are where a mismatch announces itself.
+3. `AGENTS.md` — the quick-reference block gains the two commands (with a one-line "why" in the surrounding style):
+   ```
+   uv run python -m scripts.reembed_embeddings --compare   # are the stored vectors still usable? (cosine agreement verdict)
+   uv run python -m scripts.reembed_embeddings --apply     # re-embed the chunks whose generation != BOR_LLM_EMBED_MODEL (resumable)
+   ```
+   and rule 10's log list gains the two new greppable lines (`embeddings model=… chunks=… mismatched=… unknown=…`, `reembed: model=… mode=… … remaining=… duration_ms=…`).
+4. Full gates (AGENTS rule 9, all of them, in this order):
+   - `uv run pytest`
+   - `uv run pytest --cov=app --cov-report=term-missing` → `app/` coverage **>90%**
+   - `uv run pytest tests/e2e/test_embedding_generation.py -v --no-cov` → green **in isolation**
+   - regression set, each **in isolation**: `tests/e2e/test_admin_settings_panel.py`, `test_admin_auth.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_llm_retry.py`, `test_sync_button.py`, `test_cache_busting.py`, `test_admin_theme_tab.py`
+   - `uv run ruff check . && uv run pyright`
+   - `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` on the restored replica (locally, via the `restore-test-db` skill) — the battery's gate items keep passing with the new column present (nothing here should move them; a movement means something in this phase changed ranking, which is a regression)
+   - `uv run python -m scripts.api_payload_audit` (app running) — `/api/admin/embeddings` present and under the bar
+5. The Documentation & Consistency final pass — read `README.md` and `AGENTS.md` end to end against the post-phase project, fix every inconsistency found **including pre-existing ones this phase did not cause** (e.g. a stale command, a wrong default, a section count), and record what was fixed in the commit body.
+6. Move `.agents/phases/todo/169_embedding_generation_migration/` → `.agents/phases/complete/` and commit exactly as the overview's **Commit** block specifies (`--no-gpg-sign`, Conventional Commit, plus the `LLM-Generated: true` trailer).
+
+## Testing & Quality
+- This task OWNS the phase's whole-suite gate: 100 % passing, `app/` coverage >90%, the phase E2E plus the eight-file regression set green in isolation.
+- Coverage: **>90%** on `app/` (the phase's new modules included).
+
+## Completion Criteria
+- [ ] `tests/e2e/test_embedding_generation.py` passes in isolation and covers all six behaviors above, including the A3 "the retrieval list is unchanged by the stamp" pin.
+- [ ] The README runbook exists and is self-sufficient (a competent operator who has never seen this chat can execute a production embedding swap from it alone, including the `app_settings`-beats-`.env` warning and the retune-after-re-embed ordering); the `:1408` troubleshooting entry no longer tells anyone to drop the table for a same-dim swap.
+- [ ] `AGENTS.md`'s quick reference and rule 10 name both script modes and both log lines.
+- [ ] Every gate in step 4 is green, with the coverage number recorded in the commit body.
+- [ ] `README.md` and `AGENTS.md` are accurate and consistent with the whole project (verified in the final pass, fixes listed in the commit body).
+- [ ] no behavior change in completed phases: the battery's ranks, the A8 gate verdicts, sync counts, `sources_meta` behavior and saved-chat staleness are unchanged.
+- [ ] One atomic `--no-gpg-sign` Conventional Commit; `.agents/phases/complete/169_embedding_generation_migration/` exists and the todo copy is gone.

.agents/skills/migrate-embed-model/SKILL.md

@@ -0,0 +1,357 @@
+---
+name: migrate-embed-model
+description: Walks a developer through migrating Brain of Reese to a new embedding model — probe and benchmark the new model, MEASURE whether the stored 768-dim vectors are still comparable (cosine-agreement verdict), back up and run the migration on production (re-embed in place, or stamp only), then re-tune the honesty/citation thresholds and smoke-test them against the retrieval battery. Use when the user asks to change, upgrade, switch or roll back the embedding model (e.g. "upgrade the embed model to embedding gemma 2", "switch BOR to a new embedding model", "can I change the embed model without re-importing?", "re-embed the knowledge base"). Same-dimension swaps are the dangerous case — nothing errors, retrieval just degrades silently. Does NOT apply when the output DIMENSION changes (that is the drop-and-recreate path in README.md) and does not test chat/summary models (see test-chat-model / test-summary-model).
+---
+
+# Migrate the Embedding Model
+
+An embedding-model change that keeps the dimension (768 → 768) produces **no
+error anywhere** in Brain of Reese: `chunks.embedding` is fixed at
+`BOR_EMBEDDING_DIM`, `EmbeddingDimensionError` (`app/rag/llm.py`) only fires on
+a dimension mismatch, and a re-sync re-embeds **nothing** because the
+`documents.file_hash` fast path returns with zero embeds. The corpus then
+scores questions with a different geometry and retrieval silently decays.
+
+This skill is the controlled procedure for that change: **measure first, then
+decide, then migrate, then re-tune, then smoke-test.** Four stages, in that
+order — never skip stage 2 (the measurement) and never re-tune before the
+re-embed.
+
+```
+stage 1  probe + benchmark the new model          (is it 768? is it any good?)
+stage 2  measure comparability  --compare         (do I have to re-embed at all?)
+stage 3  prod migration                           (backup → migrate → re-embed → verify)
+stage 4  re-tune the thresholds + smoke test      (battery-gated)
+```
+
+## Prerequisites
+
+- **Phase 169 must be applied** — the tooling lives there. Check before
+  anything else; if either is missing, STOP and tell the user to run the phase
+  (`phased-execution` skill) rather than improvising:
+  ```bash
+  ls scripts/reembed_embeddings.py
+  uv run alembic heads          # must be >= 0029
+  ```
+- The db container is up: `podman compose up -d db`
+- `.env` has `BOR_LLM_BASE_URL` / `BOR_LLM_API_KEY` (or `$AIPI_KEY`) pointing at
+  the endpoint that serves BOTH the old and the new embedding model.
+- **Keep the old model served** on the endpoint for the whole bake window —
+  rollback means re-embedding back to it (stage "Rollback").
+- Production access: read the prod host / DSN / service name from your
+  deployment repo or the prod env file. Never hardcode them into this file or a
+  commit. Below, `prod` means "run the quoted command there".
+
+## Rules (non-negotiable)
+
+- **Never switch the app's model before stage 2 is measured and a backup
+  exists.** `--apply` overwrites vectors; the `pg_dump` is the only undo.
+- **Never `--stamp-only` without a `COMPATIBLE` verdict.** Stamping claims
+  "these vectors belong to this model", and after that claim nothing will ever
+  re-embed them.
+- **Never re-tune the thresholds before the corpus is on ONE generation.**
+  Retuning against a mixed corpus tunes to noise (stage 4 is after stage 3).
+- **Never run `--apply` while a sync is running** (confirm idle first).
+- **The script and the app must use the same model name.** `--apply` stamps
+  `BOR_LLM_EMBED_MODEL` from ITS OWN environment; running it with a different
+  env than the app's makes every stamp a lie. Its first log line prints the
+  model and the sanitized base URL — read that line before walking away.
+- **Check the `app_settings` overlay before changing any value** (phase 168):
+  a value saved in the Settings panel is a DB row and **beats `.env`**. Every
+  stage that changes a value starts with the `select … from app_settings`
+  check in step 4.1.
+- **Do not edit `app/` or `alembic/` from this skill.** If the new model
+  exposes an app defect (e.g. a different input-token ceiling), report it as a
+  phase candidate — the batching constants in `app/rag/llm.py`
+  (`_TOKENS_PER_CHAR`, `_REQUEST_TOKEN_BUDGET`) are code, not a drive-by edit.
+- Retrieval is **never** filtered by generation (phase 169 LOCKED A3): a mixed
+  corpus keeps ranking, it just ranks wrongly. The card, the endpoint and the
+  sync log are the signals — trust them over "it seems to work".
+- ntfy (if used at all) is **outbound only** — milestones, never questions.
+
+## Stage 1 — Probe and benchmark the new model
+
+Point at the new model *for one process at a time*; do not touch the running
+app yet.
+
+```bash
+NEW=embeddinggemma-2          # whatever the endpoint calls it
+BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.llm_probe
+BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.test_embed_model --model "$NEW" --runs 3
+```
+
+- `llm_probe` prints the output dimension against `BOR_EMBEDDING_DIM` (768).
+  **If the dimension differs, STOP** — this skill does not apply. That is the
+  README's "Embedding dimension mismatch" path (`BOR_EMBEDDING_DIM` + drop and
+  recreate the chunks table), which is destructive in a different way.
+- `test_embed_model` runs the three gated checks (dimension / cosine margin ≥
+  0.05 on the fixed semantic-pair corpus / speed) and **appends its rows to
+  `benchmarks/model_benchmarks.csv` automatically**. Record the numbers; a
+  cosine-margin failure means the model is not fit to serve, full stop.
+- Note the **speed** (the old model's reference is ~94 vec/s → the live corpus
+  of 1,000 docs / 8,866 chunks re-embeds in minutes; if the new model is
+  10× slower, plan the window accordingly).
+- **Input ceiling check** — the app batches embeddings against the current
+  endpoint's ~1024-token hard rejection. Probe it once with a long input:
+  ```bash
+  BOR_LLM_EMBED_MODEL=$NEW uv run python - <<'PY'
+  import asyncio, httpx
+  from app.config import get_settings
+  from app.rag.llm import LLMClient
+  async def go():
+      llm = LLMClient()
+      for n in (1200, 4000, 12000):
+          try:
+              await llm.embed_one("word " * n)
+              print(f"ok   ~{n} chars")
+          except Exception as e:
+              print(f"FAIL ~{n} chars: {type(e).__name__}: {str(e)[:120]}")
+  asyncio.run(go())
+  PY
+  ```
+  Fewer accepted chars than the old model → `BOR_EMBED_QUESTION_MAX_CHARS`
+  (env) must come down AND the `_REQUEST_TOKEN_BUDGET` constant
+  (`app/rag/llm.py`) is now wrong: flag it as a phase, don't hand-edit.
+  More accepted → nothing to do; the existing budget is merely conservative.
+
+## Stage 2 — Measure comparability (the answer to "are they compatible?")
+
+Run the measurement as its **own process with an env override**. It reads and
+writes nothing, and it does not disturb the running app — the app keeps serving
+the old model while you measure.
+
+```bash
+BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.reembed_embeddings --compare --sample 25
+```
+
+It embeds each sampled chunk's **stored text** with the new endpoint and
+compares against the **stored vector**, printing `mean / median / p10 / min`
+cosine agreement and a verdict:
+
+| verdict | mean cosine | what it means | what to do |
+|---|---|---|---|
+| `COMPATIBLE` | ≥ 0.995 | same geometry, different name/build | stage 3 with `--stamp-only` — **no re-embed needed** |
+| `DRIFT` | 0.90 – 0.995 | related but not the same space | stage 3 with `--apply` (cheap; note it in the record) |
+| `INCOMPATIBLE` | < 0.90 | different space — this is the normal case for a model upgrade | stage 3 with `--apply` |
+
+- Exit codes: `0` COMPATIBLE, `3` non-compatible verdict (the answer is "no"),
+  `1` the tool/boot/endpoint failed (something broke). Branch on those, never
+  on the pretty text.
+- The bar is **0.995, not 1.0**, because pgvector stores `float4` while the app
+  writes Python floats: an unchanged model reproduces a vector to float4
+  round-trip precision (~0.9999), never exactly 1.0. Expecting exactly 1.0
+  makes a healthy corpus read as `DRIFT`.
+- Also print the **baseline** you are protecting, so stage 4 has a "before":
+  ```bash
+  uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt \
+    | tee /tmp/embed_baseline_$(date +%Y%m%d_%H%M%S).txt
+  podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c "
+    select percentile_cont(0.1) within group (order by top_score) as p10,
+           percentile_cont(0.5) within group (order by top_score) as p50,
+           percentile_cont(0.9) within group (order by top_score) as p90,
+           count(*) filter (where deflected) as deflected, count(*) as turns
+    from query_log where created_at > now() - interval '90 days';"
+  ```
+  (On prod, run the equivalent `psql` against the prod database and keep both
+  outputs out of git — `/tmp` is fine.)
+
+## Stage 3 — Migrate production
+
+Do these back to back in one maintenance window. The corpus is
+1,000 docs / 8,866 chunks ≈ minutes at the benchmarked speed; the mixed window
+is as long as the re-embed, and the Settings card tells you when it is over.
+
+### 3.1 Confirm nothing is mid-flight, then BACK UP
+
+```bash
+curl -s -b "$JAR" https://<prod-host>/api/sync/status     # must be idle (verify the route in app/api/sync.py)
+
+podman exec <prod-db-container> pg_dump -U <user> <db> -Fc \
+  > /tmp/pre_embed_migration_$(date +%Y%m%d_%H%M%S).dump
+ls -lh /tmp/pre_embed_migration_*.dump
+podman exec <prod-db-container> pg_restore --list < /tmp/pre_embed_migration_<timestamp>.dump | head
+```
+
+Copy the dump **off the app box**, and only continue once `pg_restore --list`
+reads it. The dump — not the rollback section below — is the real undo.
+
+### 3.2 Apply the schema
+
+```bash
+uv run alembic upgrade head      # 0029_chunks_embedding_model, reversible
+```
+
+Adding `chunks.embedding_model` is instant and non-destructive; existing rows
+stay `NULL` = **unknown by design** (never backfilled with a guess).
+
+### 3.3 Flip the app, then re-embed (or stamp)
+
+Flip the app's model, then immediately run the repair with the **same** env:
+
+- `.env` + restart (`BOR_LLM_EMBED_MODEL=$NEW`), **or**
+- the admin **Settings** panel → `llm_embed_model` → Save (phase 168 applies it
+  live, no restart — and writes an `app_settings` row, which then beats `.env`
+  for the rest of the app's life; that is fine, just know where the truth is).
+
+```bash
+# the normal case (DRIFT / INCOMPATIBLE):
+uv run python -m scripts.reembed_embeddings --apply
+# only on a COMPATIBLE verdict — rewrites no vectors, just the stamps:
+uv run python -m scripts.reembed_embeddings --stamp-only
+```
+
+Read the first log line (`mode`, `model`, `dim`, `base=`) before walking away,
+and the summary line when it finishes:
+
+```
+reembed: model=<stamp> mode=apply chunks=8866 batches=<n> remaining=0 duration_ms=<ms>
+```
+
+`remaining=0` is the goal. The run is resumable by construction (it selects
+`WHERE embedding IS NULL OR embedding_model IS DISTINCT FROM <model>` and
+commits per batch): if it dies mid-way, fix the cause and re-run — it picks up
+where it stopped. It never writes a NULL embedding and it never bumps
+`sources_meta.version`, so saved chats do not go stale over a re-embed.
+
+### 3.4 Verify the migration before touching any value
+
+```bash
+podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c "
+  select coalesce(embedding_model,'<unknown>') as model, count(*) as chunks,
+         count(embedding) as with_vector
+  from chunks group by 1 order by 2 desc;"
+
+uv run python -m scripts.reembed_embeddings --compare --sample 25   # must now read COMPATIBLE (~1.0)
+
+curl -s -b "$JAR" https://<prod-host>/api/admin/embeddings          # "stale": false, "unknown": 0, "null_vector": 0
+```
+
+Then in the browser: **Settings → Embedding generation** shows the clean
+`status` card, not the `alert`. And the post-sync line agrees — trigger nothing,
+just check the last sync's log:
+
+```bash
+grep ' :: embeddings model=' <app log> | tail -1     # mismatched=0 unknown=0
+```
+
+## Stage 4 — Re-tune the values, then smoke-test them
+
+The three honesty/citation values were hand-tuned against the OLD model's
+cosine distribution (`app/config.py`: `relevance_threshold=0.62`,
+`lexical_support_floor=0.35`, `source_usefulness_floor=0.35`). A new model moves
+the whole distribution, so they are almost certainly wrong after stage 3 — in
+whichever direction the percentiles moved.
+
+### 4.1 Know which layer wins
+
+```sql
+select key, value from app_settings
+where key in ('llm_embed_model','relevance_threshold',
+              'lexical_support_floor','source_usefulness_floor');
+```
+
+A row here **beats `.env`** (phase 168's overlay). Change values in ONE place —
+the panel (live, DB row) or `.env` (restart, env value) — and never let a stale
+DB row silently pin an old number.
+
+### 4.2 Measure, then set
+
+1. Re-run the battery and collect two numbers: the **best cosine of known-good
+   questions** (from `/tmp/embed_baseline_*.txt`'s same questions) and the best
+   cosine of the battery's **out-of-KB control** question.
+2. `relevance_threshold` goes **between the two clusters** (midpoint). Good
+   hits clustering lower than before → lower it; nonsense questions scoring
+   higher → raise it.
+3. `lexical_support_floor` sits at the bottom of the good-hit cluster — it is
+   the bar a lexical hit needs vector corroboration to flip a turn to HIGH.
+   `source_usefulness_floor` (the citation tier) sits at the same bar by
+   default.
+4. The validators enforce `0 ≤ floor ≤ relevance_threshold`
+   (`app/config.py`'s `_lexical_support_floor_bounds` /
+   `_source_usefulness_floor_bounds`) — a floor above the threshold is refused
+   at startup (or a 422 in the panel), which is the intended guard, not a bug.
+5. Apply → re-run the battery → iterate. Two or three passes is normal; keep
+   each pass's numbers.
+
+Symptom → first move:
+
+| symptom after the swap | first move |
+|---|---|
+| real questions deflect, cosines cluster low | lower `relevance_threshold`, then both floors (keep floors ≤ threshold) |
+| out-of-KB questions answered confidently | raise `relevance_threshold` |
+| a single weak token match promotes to HIGH | raise `lexical_support_floor` |
+| citations present but irrelevant | raise `source_usefulness_floor` |
+| `embedding failed (too-large)` on questions | lower `BOR_EMBED_QUESTION_MAX_CHARS`; if import batches also fail, `_REQUEST_TOKEN_BUDGET` needs a phase |
+
+### 4.3 Smoke test
+
+- [ ] `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` — every gated item in that file passes (the gitea / forgejo / gateway name questions, the qwen3.8-27b incident regression, both IKEA `BESTÅ`/`besta` spellings, and the out-of-KB control's verdict). This file is the ruler — **read it, never edit it** to make a run pass.
+- [ ] A real question in the browser answers with citations and a plausible `top_score` (per-turn line: `grep ' :: chat: ' <app log> | tail -5`).
+- [ ] The out-of-KB control question still gets the amber honest deflection — a model that scores everything high has NOT been "tuned", it has been blinded.
+- [ ] `GET /api/admin/embeddings` → `stale: false`, `unknown: 0`, `null_vector: 0`; the Settings card is the clean `status` variant.
+- [ ] `psql`: the new cosine percentiles sit where the retuned threshold expects them, and `select deflected, count(*) from query_log group by 1` over the next day matches the stage-2 baseline's deflection rate within a few points.
+- [ ] `uv run python -m scripts.api_payload_audit` still clean (the migration added one small admin endpoint).
+
+### 4.4 Record it
+
+- `benchmarks/model_benchmarks.csv` already has stage 1's rows (written by
+  `scripts.test_embed_model`).
+- If a **default** in `app/config.py` changed as a result, that is its own
+  deliberate commit (`tune(rag): …`, with the before/after percentiles in the
+  body) — never smuggled into a migration.
+- Commit any `.env.example` / README diff. Note in the body: date, old → new
+  model, the `--compare` verdict + mean cosine before and after, chunks
+  re-embedded, and the final three values.
+
+## Rollback
+
+Pick by what went wrong:
+
+1. **Vectors unchanged (`--stamp-only` path):** flip `llm_embed_model` back
+   (panel or `.env` + restart). Done — the vectors were never touched.
+2. **Vectors re-embedded, app box still healthy:** flip `llm_embed_model` back
+   to the old model (which must still be served) and run
+   `uv run python -m scripts.reembed_embeddings --apply` again — it re-embeds
+   the corpus back into the old geometry because the selection is "stamp differs
+   from the configured model". Minutes, same window as the forward migration.
+3. **Something worse (bad dump, half-run, corrupted corpus):** restore the stage
+   3.1 dump — the exact pre-migration state, vectors and stamps included:
+   ```bash
+   podman exec <prod-db-container> pg_restore -U <user> -d <db> --clean --if-exists --no-owner \
+     < /tmp/pre_embed_migration_<timestamp>.dump
+   ```
+   Then verify the fingerprint (`documents` / `chunks` counts and
+   `select count(*) from chunks where embedding_model is not null`) before
+   sending traffic back.
+
+## Troubleshooting
+
+- **`EmbeddingDimensionError` / dimension ≠ 768** — the dimension changed, so
+  this skill is the wrong tool: README → "Embedding dimension mismatch"
+  (`BOR_EMBEDDING_DIM`, drop + recreate `chunks`, re-import). Do not "make it
+  fit" by resizing the column.
+- **`reembed: FAILED … resume with:`** — the run stopped loud, on purpose. The
+  batches already committed are done; the rest are pending. Find the cause
+  (endpoint outage, oversized input, a pathological chunk) and re-run the
+  printed command. Never hand-edit vectors.
+- **`remaining=` never reaches 0** — some rows never match. Look at them:
+  `select id, length(content), embedding_model from chunks where embedding_model
+   is distinct from '<configured>' order by id limit 20;` Empty `content` or a
+  content the endpoint rejects is the usual answer.
+- **Card stuck on `unknown`** — those are pre-phase-169 rows (stamp `NULL`,
+  which is *unknown*, not "wrong"). The sync logs them at INFO and never warns,
+  and the card keeps the calm `status` variant. They resolve the first time
+  they are legitimately re-embedded, or now via `--stamp-only` (COMPATIBLE
+  only) / `--apply`.
+- **Verdict `DRIFT` with mean 0.97-0.99** — usually a different server build /
+  quantization of a nominally identical model. Re-embed anyway (it is minutes);
+  note it in the record so nobody re-litigates it later.
+- **The card and the app disagree about which model is live** — the overlay:
+  a `app_settings.llm_embed_model` row beats `.env`. `select key, value from
+  app_settings where key='llm_embed_model';`, then fix it in the panel (or
+  `delete from app_settings where key='llm_embed_model';` to fall back to env).
+- **Everything still looks fine but answers got worse** — that is the failure
+  mode this whole procedure exists for, and it hides in the `top_score`
+  column, not in the logs. Compare stage 2's percentile snapshot with
+  `query_log.top_score` over the last day; a collapsed spread (e.g. p10 ≈ p90)
+  means a mixed generation or a threshold that no longer discriminates.