brain-of-reese/.agents/phases/todo/169_embedding_generation_migration/00_phase.md

Phase 169 — Embedding generations: stamp every vector, measure comparability, re-embed in place

Source: owner request 2026-10-09 — "I'm about to upgrade the embedding model from embedding gemma to embedding gemma 2. The dimensions are the same, but the embeddings may not be comparable. What do I need to do on brain of reese's side?" Story: n/a (owner-directed infrastructure work) Context: app/models.py:232-249 (Chunk, embedding: Vector(EMBEDDING_DIM) at L241 — the ONLY vector column in the schema), app/config.py:102 (llm_embed_model), :142 (embedding_dim=768), :151 (relevance_threshold=0.62), :160 (lexical_support_floor=0.35), :179 (source_usefulness_floor=0.35), :225 (embed_question_max_chars), :548-578 (the two floor ⇄ threshold validators), app/rag/llm.py:314-322 (EmbeddingDimensionError — the ONLY guard, dim-based), :213-221 (_TOKENS_PER_CHAR / _REQUEST_TOKEN_BUDGET, written against the current endpoint's ~1024-token cap), :346 (embed()), :400 (embed_one()), app/rag/importer.py:1008-1023 (content-chunk write), :1587-1589 (the is_summary position = -1 chunk), :51-57 (the phase-118 "content-unchanged work is NOT a KB change" convention), app/rag/retriever.py:433-455 (_vector_list, the brute-force cosine scan), app/api/docs.py:488-491 (the manual-summary write), app/api/chat.py:740 (the question embed), app/api/sync.py:844 (sync: done), app/core/auth.py:83 (require_admin), app/main.py:146-172 (router includes), app/core/settings_overlay.py + app/core/settings_catalog.py (phase 168: the DB-over-env overlay — a stored override BEATS .env), alembic/versions/0028_app_settings.py (head — next free: 0029), scripts/regen_folder_summaries.py (the one-shot entrypoint idiom this phase's script mirrors), scripts/eval_retrieval.py, scripts/test_embed_model.py, scripts/llm_probe.py, tests/fixtures/retrieval_battery.txt (the 43-line gated battery), benchmarks/model_benchmarks.csv, tests/e2e/mock_llm.py (POST /v1/embeddings → deterministic 768-dim bag-of-words), tests/e2e/conftest.py:53,64 (the global env pins) and :105-186 (the app_server env contract), README.md:1408-1410 (the "Embedding dimension mismatch" troubleshooting entry).

Objective

An embedding-model change that KEEPS the dimension (768 → 768) is today invisible and therefore silently destructive: nothing records which model produced a vector, EmbeddingDimensionError cannot fire, and cosine scores from two different geometries get compared on one list. This phase makes a vector's generation a fact in the database (chunks.embedding_model), makes a mismatch loud in the admin UI and the sync log, ships the two tools that answer "are the old vectors still usable?" (--compare, measured in cosine agreement) and "make them usable" (--apply, a resumable in-place re-embed over the text already stored), and writes the operator runbook the live upgrade follows.

The problem, precisely (the executor reads this, not the chat)

• No generation is recorded. chunks.embedding is a bare Vector(768); grep -rn "embedding_model" app/ alembic/ is empty. A row does not know what made it.
• No guard can fire. The only loud failure is EmbeddingDimensionError (app/rag/llm.py:314-322), which compares dimension against BOR_EMBEDDING_DIM. Same dims → no error, ever.
• A re-sync re-embeds NOTHING. documents.file_hash matches → the importer returns through _unchanged_doc with "ZERO embed" (app/rag/importer.py:70-84, :1391-1403). Flipping BOR_LLM_EMBED_MODEL and pressing Sync leaves 100 % of the corpus in the old space while every QUESTION embeds in the new one (app/api/chat.py:740) — retrieval degrades with no error anywhere.
• The thresholds are model-calibrated, not absolute. relevance_threshold=0.62 / lexical_support_floor=0.35 / source_usefulness_floor=0.35 gate honesty (A8) and citation tiering, and app/config.py:147-150 records that they were hand-re-tuned on 2026-08-21 for the current model's cosine distribution ("the default never discriminated"). A new model moves the whole distribution.
• No ANN index to rebuild — pg_indexes for chunks is chunks_pkey, ix_chunks_document_id, ix_chunks_tsv only; _vector_list is a brute-force scan. (If a hnsw/ivfflat index is ever added, it must be built AFTER a re-embed.)
• Scale: the live KB (and data/bor_live_kb_replica.dump.sql) is 1,000 documents / 8,866 chunks, ~1,000 of them the is_summary position = -1 row. At the benchmarked 93.8 vec/s a full re-embed is minutes — so a quiesced in-place run beats a blue/green column.

Design (shared by all tasks)

D1 — One nullable chunks.embedding_model text column (task 01, migration 0029). The value is the llm_embed_model string that was in effect when the vector was written. Nullable and never backfilled by the migration: for pre-phase rows the true producing model is genuinely unknown, and writing the current setting into them would be a lie that survives the very migration this phase exists to make detectable. NULL therefore means unknown and is reported as its own bucket. No index: the column is read by GROUP BY audits and by the re-embed's WHERE over ≤ 10⁴ rows, where a seq scan beats maintaining an index on a low-cardinality column on every chunk write.

D2 — One write helper, so a fourth path cannot forget (task 02). app/core/embedding_generation.py::stamp_embedding(chunk, vector, model) sets embedding and embedding_model together. Every path that writes a vector goes through it: the content chunks (app/rag/importer.py:1008-1023), the summary chunk (:1587-1589), and the manual-summary edit (app/api/docs.py:488-491). A unit pin asserts no Chunk vector write in app/ bypasses it (source-scan style, the house static-pin idiom).

D3 — Comparability is MEASURED, not assumed (task 04). scripts/reembed_embeddings.py --compare takes N chunks (--sample, default 25, ORDER BY id — deterministic, no RNG), embeds their stored content with the endpoint/env it is given, and reports the cosine distribution (mean / median / p10 / min) against the stored vectors. Bands (LOCKED A4): mean ≥ 0.995 → COMPATIBLE (same geometry — stamp, don't re-embed); mean < 0.90 → INCOMPATIBLE (re-embed); between → DRIFT (re-embed + flag). The 0.995 bar exists because pgvector stores float4 while the app writes Python floats: an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never to exactly 1.0. This is the tool that answers the owner's actual question instead of guessing, and it doubles as the post-migration smoke test (it must read ≈ 1.0 afterwards).

D4 — Three modes, one script (task 04), mirroring scripts/regen_folder_summaries.py's entrypoint idiom (get_settings() + configure_logging + configure_debugging() + ensure_admin_configured, one event loop, one LLMClient, DEBUGPY honored and never imported by default — AGENTS rule 7):

• --compare (default when no mode flag): read-only, prints the verdict, exits 0 on COMPATIBLE and non-zero otherwise so the runbook can branch on it. It never writes. Because it is its own process it can be run on prod with BOR_LLM_EMBED_MODEL=<new> uv run python -m scripts.reembed_embeddings --compare while the app keeps serving the OLD model — the measurement does not touch the running app.
• --stamp-only: set embedding_model to the configured model on every chunk that has a vector, touching no vector. Valid only on a COMPATIBLE verdict — the runbook's step for "the endpoint changed name but not geometry".
• --apply: re-embed in place keyed on chunk.id, selecting WHERE embedding_model IS DISTINCT FROM :configured, committing per batch, resumable by construction (re-running finishes what is left, so a mid-run outage is never a dead end). Batching is delegated to LLMClient.embed() — no second batching implementation. Per-batch progress plus one PLAN §9 summary line: reembed: model=<stamp> mode=… chunks=… batches=… remaining=… duration_ms=….
• Both writers fail loud: an EmbeddingError on a batch logs, stops, exits non-zero and prints the resume command. A chunk is NEVER skipped into embedding = NULL — a NULL embedding silently disappears from _vector_list (app/rag/retriever.py:433-455), which is the exact silent-degradation class this phase kills.
• Both writers never touch content, position, tsv, documents.*, or sources_meta.version (the phase-118 convention, app/rag/importer.py:51-57: content-unchanged work is not a KB change, so saved chats must not go stale because the vectors were refreshed).

D5 — A mismatch is loud in three places (tasks 03, 05, 06), and never silent:

• app/core/embedding_generation.py::generation_report(session, expected_model) → a pure, unit-tested counts object: total, with_vector, null_vectors, by_model (with a None/unknown bucket), mismatched, unknown, stale.
• After every sync (app/api/sync.py, right after the sync: done line at :844): one INFO line embeddings model=<configured> chunks=N mismatched=M unknown=K. A WARNING only when mismatched > 0; unknown > 0 alone stays INFO, because every pre-phase KB is 100 % unknown and would scream on every sync forever.
• Admin endpoint GET /api/admin/embeddings (router-wide Depends(require_admin), the tokens/ui_settings pattern) returning that report as the JSON the card and the runbook both read. Payload is a handful of numbers — it must stay under the phase-167 byte budget and appears in the grep ' :: api ' audit.
• Settings view card (phase 168's #view-settings, task 06): a read-only "Embedding generation" card above the settings table — role="status" when clean, role="alert" naming the offending stamp, the chunk counts, and the remediation command (uv run python -m scripts.reembed_embeddings --apply). Text states the state; never color alone (B5).

D6 — Retrieval stays unfiltered, deliberately (LOCKED A3). No WHERE embedding_model = :model in _vector_list: during a backfill it would silently return nothing and deflect every turn (a worse failure than the one being fixed), it costs a predicate on the hot path of every turn, and the migration window is minutes. The column is for DETECTION; the script is for REPAIR. This rejection is recorded in the code docstring so the next reader does not "fix" it.

D7 — Threshold re-tuning is operator work, backed by tooling (task 07 docs). The phase ships the measurement + the runbook; it does NOT change relevance_threshold / lexical_support_floor / source_usefulness_floor defaults, because the right numbers depend on the model the owner actually points at. The runbook makes the order explicit: re-embed FIRST, then re-tune (retuning against a mixed corpus tunes to noise), then verify with tests/fixtures/retrieval_battery.txt and the deflection rate. It also names the phase-168 trap: those keys may now be app_settings rows, and a stored override beats .env (app/core/settings_overlay.py) — so select key, value from app_settings where key in ('relevance_threshold','lexical_support_floor','source_usefulness_floor','llm_embed_model'); is step one of any value change.

Dependencies

• 168_admin_settings_panel (todo — must complete first, this phase runs after it in the pipeline) — the overlay that makes app_settings beat .env (D7's trap), the catalog-completeness pin, and the #view-settings view + /api/admin/settings + tests/e2e/test_admin_settings_panel.py that this phase's card and E2E extend.
• 97_kb_tree_catalog + 156_table_ux_polish (complete) — the admin catalogue/bounded-panel and .cell-clip contracts the card must not violate (AGENTS rule 5).
• 167_api_payload_diet (complete) — the 256 KB response budget + scripts.api_payload_audit the new endpoint is audited against.
• 82_security_headers + 78_static_background + 54_asset_cache_bust_revalidation (complete) — the CSP/default-src 'self' and locally-served-assets posture the card's JS lives inside (AGENTS rule 6, the No CDN Rule), and the ?v= busting the E2E cache contract keeps.
• 119_name_signal_read_chips + 162_search_recall_small_sources (complete) — own tests/fixtures/retrieval_battery.txt, which is this phase's measuring stick; the phase READS it and never edits it (the gated items are the retuning ruler).
• 118_summary_seed_context (complete) — the "content-unchanged work is not a KB change" convention (app/rag/importer.py:51-57) that D4 inherits for sources_meta.
• 122_image_documents + 137_pdf_image_context (complete) — image docs store the vision description as chunks.content, which is what makes K3 (a re-embed never re-calls vision) true.

Tasks

1. 01_embedding_model_column.md — Chunk.embedding_model + reversible migration 0029_chunks_embedding_model.py + its up/down test.
2. 02_stamp_write_paths.md — stamp_embedding() + the three vector-write call sites routed through it + the no-bypass source pin.
3. 03_generation_report_and_logs.md — generation_report() + the post-sync INFO/WARNING lines + the greppable log contract.
4. 04_reembed_script.md — scripts/reembed_embeddings.py with --compare / --stamp-only / --apply, the pure cosine helpers, resumability, the §9 summary line, unit + integration coverage.
5. 05_admin_embeddings_endpoint.md — GET /api/admin/embeddings (admin-only), the schema, the auth matrix, the byte-budget pin.
6. 06_settings_status_card.md — the read-only "Embedding generation" card in #view-settings (role=status/role=alert, text-first).
7. 07_e2e_generation_and_docs.md — tests/e2e/test_embedding_generation.py, the README runbook + AGENTS quick-reference updates, the gates, the atomic commit.

Testing & Quality

• Unit: tests/unit/test_embedding_generation.py (stamp_embedding sets both fields; cosine on orthogonal / identical / zero-norm input; generation_report buckets — clean, mismatched, NULL-vector rows, unknown stamps, empty table), tests/unit/test_reembed_embeddings.py (the scripts.reembed_embeddings import idiom of tests/unit/test_regen_folder_summaries.py:34: mode parsing, COMPATIBLE/DRIFT/INCOMPATIBLE band edges at exactly 0.995 and 0.90, the deterministic ORDER BY id selection, --apply failing loud without writing a NULL, --stamp-only not touching embedding, no sources_meta bump), tests/unit/test_embedding_write_paths.py (the no-bypass source pin: no .embedding = / embedding= vector write in app/ outside stamp_embedding), tests/unit/test_settings_frontend_view.py extension or a sibling (the card exists, no CDN, DOM built with createElement/textContent).
• Integration: tests/integration/test_migration_0029.py (up AND down, A13), tests/integration/test_admin_embeddings_api.py (auth matrix anonymous / token-user / admin; counts correct against a seeded corpus with mixed stamps and one NULL-embedding chunk; the response body carries no API key), tests/integration/test_sync_embedding_log.py (the post-sync INFO line and the WARNING only on mismatched > 0), and an importer pin that a synced doc's content chunk AND its is_summary chunk both carry the configured stamp.
• Coverage: >90% on app/ — uv run pytest --cov=app --cov-report=term-missing.
• E2E (AGENTS rule 4): uv run pytest tests/e2e/test_embedding_generation.py -v --no-cov green in isolation; regression set green in isolation: test_admin_settings_panel.py, test_admin_auth.py, test_configurable_brand.py, test_global_tuning.py, test_llm_retry.py, test_sync_button.py, test_cache_busting.py.
• Lint/types: uv run ruff check . && uv run pyright.
• Log contract (AGENTS rule 10): the two new greppable lines — embeddings model=… chunks=… mismatched=… unknown=… (post-sync) and reembed: model=… mode=… chunks=… batches=… remaining=… duration_ms=… — plus GET /api/admin/embeddings present in the byte audit and under the bar (uv run python -m scripts.api_payload_audit).

Documentation & Consistency

• The final pass reviews README.md and AGENTS.md in full against the project as it will stand after this phase — fixing every inconsistency found, including pre-existing ones not caused by this phase.
• Task 07 rewrites README.md's "Embedding dimension mismatch" troubleshooting entry (:1408-1410) — it currently says the fix is "drop + recreate the chunks table", which is right for a DIM change and dangerously incomplete for a same-dim swap — and adds a "Switching the embedding model" runbook section; AGENTS.md's quick reference gains the compare/apply commands.

Completion Criteria

• chunks.embedding_model exists (migration 0029, reversible), and EVERY vector write in app/ stamps it — proven by the no-bypass pin and by an importer integration pin covering both a content chunk and the position = -1 summary chunk.
• uv run python -m scripts.reembed_embeddings --compare prints a mean/median/p10/min cosine agreement and a COMPATIBLE / DRIFT / INCOMPATIBLE verdict, writes nothing, and exits non-zero unless COMPATIBLE — proven against a mock endpoint on a seeded corpus (identical vectors → COMPATIBLE; different vectors → INCOMPATIBLE).
• --apply re-embeds only the chunks whose stamp differs, commits per batch, is resumable (a second run reports remaining=0), never leaves an embedding NULL, and does not bump sources_meta.version or touch documents/tsv.
• A mixed-generation KB is LOUD: GET /api/admin/embeddings reports the counts, #view-settings shows a role="alert" card naming the stamp and the repair command, and the post-sync log WARNs. An all-unknown (pre-phase) KB WARNs nothing and shows a role="status" card.
• Retrieval is UNCHANGED by the new column — a mixed-stamp corpus still returns its hits (the D6 pin: detection, not filtering).
• README.md has the runbook (measure → decide → back up → migrate → re-embed → verify → retune → smoke) including the phase-168 app_settings-beats-.env warning; AGENTS.md's quick reference lists both script modes.
• uv run pytest green, app/ coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
• README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
• no behavior change in completed phases (retrieval order, the A8 gate, sync counts, saved-chat staleness all untouched).
• One atomic --no-gpg-sign Conventional Commit; the phase dir moves to .agents/phases/complete/.

Locked decisions (owner-confirmed 2026-10-09 during phase authoring)

• A1 — one nullable chunks.embedding_model text column, stamped with the llm_embed_model in effect at write time; NULL means unknown and the migration NEVER backfills a guess.
• A2 — every vector write goes through one helper (stamp_embedding) so no path can desynchronize the pair; a source-scan test enforces it.
• A3 — detection only. Retrieval never filters on the stamp — a mixed corpus still ranks. (Rejected: a WHERE embedding_model = … predicate in _vector_list, which silently deflects every turn during a backfill and taxes the hot path; the reason is in the code docstring.)
• A4 — comparability is measured in cosine agreement, bands COMPATIBLE ≥ 0.995 > DRIFT ≥ 0.90 > INCOMPATIBLE, with the float4-round-trip rationale for the 0.995 bar; --stamp-only is legal only on COMPATIBLE.
• A5 — per-batch commits + a resumable WHERE embedding_model IS DISTINCT FROM :configured selection, fail-loud on any batch error, never a NULL embedding, and NEVER a sources_meta.version bump (a re-embed is not a KB change — phase 118's convention).
• A6 — NO new Settings field. Every knob is a CLI flag, so tests/unit/test_settings_catalog.py's set(CATALOG) == set(Settings.model_fields) pin is untouched and the phase-168 panel stays display-only for this feature.
• A7 — one story, one phase, one Playwright file (AGENTS rule 4): the backend is covered by unit + integration, the visible contract by tests/e2e/test_embedding_generation.py.
• A8 — the three A8/A13 tuning defaults are NOT changed by this phase — they are operator-side, post-re-embed, battery-driven (D7). If the owner's post-migration battery says a default is now wrong, that is a separate, deliberate change.

Assumptions (the request left these open; the executor does not guess)

• K1: the stamp is the model NAME. An operator who re-points the same alias server-side (aipi keeps serving embed but now runs gemma-2) is caught by --compare, not by the column — the column cannot see a name that did not change, and faking it with a vector fingerprint per row costs an endpoint call per write. This boundary is stated in the column docstring and the README runbook.
• K2: the corpus is ≤ ~10⁴ chunks (live and replica: 8,866), so one quiesced in-process pass is minutes; no parallelism, no job queue, no progress table (PLAN A12: in-process state, no Valkey).
• K3: every embedded text is already in chunks.content — image docs store the VISION DESCRIPTION as their content — so a re-embed never re-calls the vision model and never re-reads a PDF.
• K4: --compare's sample is ORDER BY id LIMIT :n (deterministic across runs; no RNG, no RANDOM()), and N defaults to 25 with --sample for more.
• K5: the card is read-only and admin-only (no controls, no "fix it now" button — a re-embed is a maintenance-window operation, not a click).
• K6: no new fixtures and no dump files are committed (data/ stays gitignored); tests seed small corpora in the house way.
• K7: quiescing = "do not run a sync while --apply runs" (single app process). The script does not take a lock; the README runbook states the order.
• K8: embedding_dim stays 768 and stays non-overridable (phase 168 A3); a real DIM change remains the drop-and-recreate path in the README — this phase is for the same-dim swap that path does not cover.

Commit

git add app/ alembic/ scripts/ frontend/ tests/ README.md AGENTS.md .agents/phases/ && \
git commit --no-gpg-sign \
  -m "feat(rag): stamp the embedding generation on every chunk and ship the in-place re-embed tooling" \
  -m "A same-dimension embedding-model swap was invisible: nothing recorded which model made a vector, EmbeddingDimensionError cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the file_hash fast path returns with zero embeds. chunks.embedding_model (migration 0029, nullable, never backfilled) now records it, scripts/reembed_embeddings.py measures comparability in cosine agreement (--compare) and repairs in place (--stamp-only / --apply, per-batch commits, resumable, never a NULL embedding, never a sources_meta bump), and a mixed generation is loud in GET /api/admin/embeddings, the Settings card and the post-sync log. Retrieval is deliberately unfiltered: detection, not repair."