diff --git a/.agents/phases/todo/169_embedding_generation_migration/00_phase.md b/.agents/phases/todo/169_embedding_generation_migration/00_phase.md new file mode 100644 index 0000000..9cd309b --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/00_phase.md @@ -0,0 +1,121 @@ +# Phase 169 — Embedding generations: stamp every vector, measure comparability, re-embed in place + +**Source:** owner request 2026-10-09 — "I'm about to upgrade the embedding model from embedding gemma to embedding gemma 2. The dimensions are the same, but the embeddings may not be comparable. What do I need to do on brain of reese's side?" +**Story:** n/a (owner-directed infrastructure work) +**Context:** `app/models.py:232-249` (`Chunk`, `embedding: Vector(EMBEDDING_DIM)` at L241 — the ONLY vector column in the schema), `app/config.py:102` (`llm_embed_model`), `:142` (`embedding_dim=768`), `:151` (`relevance_threshold=0.62`), `:160` (`lexical_support_floor=0.35`), `:179` (`source_usefulness_floor=0.35`), `:225` (`embed_question_max_chars`), `:548-578` (the two floor ⇄ threshold validators), `app/rag/llm.py:314-322` (`EmbeddingDimensionError` — the ONLY guard, dim-based), `:213-221` (`_TOKENS_PER_CHAR` / `_REQUEST_TOKEN_BUDGET`, written against the current endpoint's ~1024-token cap), `:346` (`embed()`), `:400` (`embed_one()`), `app/rag/importer.py:1008-1023` (content-chunk write), `:1587-1589` (the `is_summary` `position = -1` chunk), `:51-57` (the phase-118 "content-unchanged work is NOT a KB change" convention), `app/rag/retriever.py:433-455` (`_vector_list`, the brute-force cosine scan), `app/api/docs.py:488-491` (the manual-summary write), `app/api/chat.py:740` (the question embed), `app/api/sync.py:844` (`sync: done`), `app/core/auth.py:83` (`require_admin`), `app/main.py:146-172` (router includes), `app/core/settings_overlay.py` + `app/core/settings_catalog.py` (phase 168: the DB-over-env overlay — a stored override BEATS `.env`), `alembic/versions/0028_app_settings.py` (head — next free: **0029**), `scripts/regen_folder_summaries.py` (the one-shot entrypoint idiom this phase's script mirrors), `scripts/eval_retrieval.py`, `scripts/test_embed_model.py`, `scripts/llm_probe.py`, `tests/fixtures/retrieval_battery.txt` (the 43-line gated battery), `benchmarks/model_benchmarks.csv`, `tests/e2e/mock_llm.py` (`POST /v1/embeddings` → deterministic 768-dim bag-of-words), `tests/e2e/conftest.py:53,64` (the global env pins) and `:105-186` (the `app_server` env contract), `README.md:1408-1410` (the "Embedding dimension mismatch" troubleshooting entry). + +## Objective + +An embedding-model change that KEEPS the dimension (768 → 768) is today invisible and therefore silently destructive: nothing records which model produced a vector, `EmbeddingDimensionError` cannot fire, and cosine scores from two different geometries get compared on one list. This phase makes a vector's generation a fact in the database (`chunks.embedding_model`), makes a mismatch loud in the admin UI and the sync log, ships the two tools that answer "are the old vectors still usable?" (`--compare`, measured in cosine agreement) and "make them usable" (`--apply`, a resumable in-place re-embed over the text already stored), and writes the operator runbook the live upgrade follows. + +## The problem, precisely (the executor reads this, not the chat) + +- **No generation is recorded.** `chunks.embedding` is a bare `Vector(768)`; `grep -rn "embedding_model" app/ alembic/` is empty. A row does not know what made it. +- **No guard can fire.** The only loud failure is `EmbeddingDimensionError` (`app/rag/llm.py:314-322`), which compares dimension against `BOR_EMBEDDING_DIM`. Same dims → no error, ever. +- **A re-sync re-embeds NOTHING.** `documents.file_hash` matches → the importer returns through `_unchanged_doc` with "ZERO embed" (`app/rag/importer.py:70-84`, `:1391-1403`). Flipping `BOR_LLM_EMBED_MODEL` and pressing Sync leaves 100 % of the corpus in the old space while every QUESTION embeds in the new one (`app/api/chat.py:740`) — retrieval degrades with no error anywhere. +- **The thresholds are model-calibrated, not absolute.** `relevance_threshold=0.62` / `lexical_support_floor=0.35` / `source_usefulness_floor=0.35` gate honesty (A8) and citation tiering, and `app/config.py:147-150` records that they were hand-re-tuned on 2026-08-21 *for the current model's* cosine distribution ("the default never discriminated"). A new model moves the whole distribution. +- **No ANN index to rebuild** — `pg_indexes` for `chunks` is `chunks_pkey`, `ix_chunks_document_id`, `ix_chunks_tsv` only; `_vector_list` is a brute-force scan. (If a hnsw/ivfflat index is ever added, it must be built AFTER a re-embed.) +- **Scale:** the live KB (and `data/bor_live_kb_replica.dump.sql`) is **1,000 documents / 8,866 chunks**, ~1,000 of them the `is_summary` `position = -1` row. At the benchmarked 93.8 vec/s a full re-embed is minutes — so a quiesced in-place run beats a blue/green column. + +## Design (shared by all tasks) + +**D1 — One nullable `chunks.embedding_model` text column (task 01, migration `0029`).** The value is the `llm_embed_model` string that was in effect when the vector was written. Nullable and **never backfilled by the migration**: for pre-phase rows the true producing model is genuinely unknown, and writing the current setting into them would be a lie that survives the very migration this phase exists to make detectable. `NULL` therefore means *unknown* and is reported as its own bucket. No index: the column is read by `GROUP BY` audits and by the re-embed's `WHERE` over ≤ 10⁴ rows, where a seq scan beats maintaining an index on a low-cardinality column on every chunk write. + +**D2 — One write helper, so a fourth path cannot forget (task 02).** `app/core/embedding_generation.py::stamp_embedding(chunk, vector, model)` sets `embedding` and `embedding_model` together. Every path that writes a vector goes through it: the content chunks (`app/rag/importer.py:1008-1023`), the summary chunk (`:1587-1589`), and the manual-summary edit (`app/api/docs.py:488-491`). A unit pin asserts no `Chunk` vector write in `app/` bypasses it (source-scan style, the house static-pin idiom). + +**D3 — Comparability is MEASURED, not assumed (task 04).** `scripts/reembed_embeddings.py --compare` takes N chunks (`--sample`, default 25, `ORDER BY id` — deterministic, no RNG), embeds their **stored** `content` with the endpoint/env it is given, and reports the cosine distribution (mean / median / p10 / min) against the **stored** vectors. Bands (LOCKED A4): mean ≥ **0.995** → `COMPATIBLE` (same geometry — stamp, don't re-embed); mean < **0.90** → `INCOMPATIBLE` (re-embed); between → `DRIFT` (re-embed + flag). The 0.995 bar exists because pgvector stores `float4` while the app writes Python floats: an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never to exactly 1.0. This is the tool that answers the owner's actual question instead of guessing, and it doubles as the post-migration smoke test (it must read ≈ 1.0 afterwards). + +**D4 — Three modes, one script (task 04), mirroring `scripts/regen_folder_summaries.py`'s entrypoint idiom** (`get_settings()` + `configure_logging` + `configure_debugging()` + `ensure_admin_configured`, one event loop, one `LLMClient`, `DEBUGPY` honored and never imported by default — AGENTS rule 7): +- `--compare` (default when no mode flag): read-only, prints the verdict, exits 0 on COMPATIBLE and non-zero otherwise so the runbook can branch on it. **It never writes.** Because it is its own process it can be run on prod with `BOR_LLM_EMBED_MODEL= uv run python -m scripts.reembed_embeddings --compare` while the app keeps serving the OLD model — the measurement does not touch the running app. +- `--stamp-only`: set `embedding_model` to the configured model on every chunk that has a vector, touching no vector. Valid **only** on a COMPATIBLE verdict — the runbook's step for "the endpoint changed name but not geometry". +- `--apply`: re-embed in place keyed on `chunk.id`, selecting `WHERE embedding_model IS DISTINCT FROM :configured`, committing per batch, resumable by construction (re-running finishes what is left, so a mid-run outage is never a dead end). Batching is delegated to `LLMClient.embed()` — no second batching implementation. Per-batch progress plus one PLAN §9 summary line: `reembed: model= mode=… chunks=… batches=… remaining=… duration_ms=…`. +- Both writers **fail loud**: an `EmbeddingError` on a batch logs, stops, exits non-zero and prints the resume command. A chunk is NEVER skipped into `embedding = NULL` — a NULL embedding silently disappears from `_vector_list` (`app/rag/retriever.py:433-455`), which is the exact silent-degradation class this phase kills. +- Both writers **never** touch `content`, `position`, `tsv`, `documents.*`, or `sources_meta.version` (the phase-118 convention, `app/rag/importer.py:51-57`: content-unchanged work is not a KB change, so saved chats must not go stale because the vectors were refreshed). + +**D5 — A mismatch is loud in three places (tasks 03, 05, 06), and never silent:** +- `app/core/embedding_generation.py::generation_report(session, expected_model)` → a pure, unit-tested counts object: `total`, `with_vector`, `null_vectors`, `by_model` (with a `None`/unknown bucket), `mismatched`, `unknown`, `stale`. +- **After every sync** (`app/api/sync.py`, right after the `sync: done` line at `:844`): one INFO line `embeddings model= chunks=N mismatched=M unknown=K`. A **WARNING** only when `mismatched > 0`; `unknown > 0` alone stays INFO, because every pre-phase KB is 100 % unknown and would scream on every sync forever. +- **Admin endpoint** `GET /api/admin/embeddings` (router-wide `Depends(require_admin)`, the `tokens`/`ui_settings` pattern) returning that report as the JSON the card and the runbook both read. Payload is a handful of numbers — it must stay under the phase-167 byte budget and appears in the `grep ' :: api '` audit. +- **Settings view card** (phase 168's `#view-settings`, task 06): a read-only "Embedding generation" card above the settings table — `role="status"` when clean, `role="alert"` naming the offending stamp, the chunk counts, and the remediation command (`uv run python -m scripts.reembed_embeddings --apply`). Text states the state; never color alone (B5). + +**D6 — Retrieval stays unfiltered, deliberately (LOCKED A3).** No `WHERE embedding_model = :model` in `_vector_list`: during a backfill it would silently return nothing and deflect every turn (a worse failure than the one being fixed), it costs a predicate on the hot path of every turn, and the migration window is minutes. The column is for DETECTION; the script is for REPAIR. This rejection is recorded in the code docstring so the next reader does not "fix" it. + +**D7 — Threshold re-tuning is operator work, backed by tooling (task 07 docs).** The phase ships the measurement + the runbook; it does NOT change `relevance_threshold` / `lexical_support_floor` / `source_usefulness_floor` defaults, because the right numbers depend on the model the owner actually points at. The runbook makes the order explicit: re-embed FIRST, then re-tune (retuning against a mixed corpus tunes to noise), then verify with `tests/fixtures/retrieval_battery.txt` and the deflection rate. It also names the phase-168 trap: those keys may now be `app_settings` rows, and **a stored override beats `.env`** (`app/core/settings_overlay.py`) — so `select key, value from app_settings where key in ('relevance_threshold','lexical_support_floor','source_usefulness_floor','llm_embed_model');` is step one of any value change. + +## Dependencies + +- `168_admin_settings_panel` (**todo — must complete first**, this phase runs after it in the pipeline) — the overlay that makes `app_settings` beat `.env` (D7's trap), the catalog-completeness pin, and the `#view-settings` view + `/api/admin/settings` + `tests/e2e/test_admin_settings_panel.py` that this phase's card and E2E extend. +- `97_kb_tree_catalog` + `156_table_ux_polish` (complete) — the admin catalogue/bounded-panel and `.cell-clip` contracts the card must not violate (AGENTS rule 5). +- `167_api_payload_diet` (complete) — the 256 KB response budget + `scripts.api_payload_audit` the new endpoint is audited against. +- `82_security_headers` + `78_static_background` + `54_asset_cache_bust_revalidation` (complete) — the CSP/`default-src 'self'` and locally-served-assets posture the card's JS lives inside (AGENTS rule 6, the No CDN Rule), and the `?v=` busting the E2E cache contract keeps. +- `119_name_signal_read_chips` + `162_search_recall_small_sources` (complete) — own `tests/fixtures/retrieval_battery.txt`, which is this phase's measuring stick; the phase READS it and never edits it (the gated items are the retuning ruler). +- `118_summary_seed_context` (complete) — the "content-unchanged work is not a KB change" convention (`app/rag/importer.py:51-57`) that D4 inherits for `sources_meta`. +- `122_image_documents` + `137_pdf_image_context` (complete) — image docs store the vision description as `chunks.content`, which is what makes K3 (a re-embed never re-calls vision) true. + +## Tasks + +1. `01_embedding_model_column.md` — `Chunk.embedding_model` + reversible migration `0029_chunks_embedding_model.py` + its up/down test. +2. `02_stamp_write_paths.md` — `stamp_embedding()` + the three vector-write call sites routed through it + the no-bypass source pin. +3. `03_generation_report_and_logs.md` — `generation_report()` + the post-sync INFO/WARNING lines + the greppable log contract. +4. `04_reembed_script.md` — `scripts/reembed_embeddings.py` with `--compare` / `--stamp-only` / `--apply`, the pure cosine helpers, resumability, the §9 summary line, unit + integration coverage. +5. `05_admin_embeddings_endpoint.md` — `GET /api/admin/embeddings` (admin-only), the schema, the auth matrix, the byte-budget pin. +6. `06_settings_status_card.md` — the read-only "Embedding generation" card in `#view-settings` (`role=status`/`role=alert`, text-first). +7. `07_e2e_generation_and_docs.md` — `tests/e2e/test_embedding_generation.py`, the README runbook + AGENTS quick-reference updates, the gates, the atomic commit. + +## Testing & Quality + +- **Unit:** `tests/unit/test_embedding_generation.py` (`stamp_embedding` sets both fields; `cosine` on orthogonal / identical / zero-norm input; `generation_report` buckets — clean, mismatched, NULL-vector rows, unknown stamps, empty table), `tests/unit/test_reembed_embeddings.py` (the `scripts.reembed_embeddings` import idiom of `tests/unit/test_regen_folder_summaries.py:34`: mode parsing, COMPATIBLE/DRIFT/INCOMPATIBLE band edges at exactly 0.995 and 0.90, the deterministic `ORDER BY id` selection, `--apply` failing loud without writing a NULL, `--stamp-only` not touching `embedding`, no `sources_meta` bump), `tests/unit/test_embedding_write_paths.py` (the no-bypass source pin: no `.embedding =` / `embedding=` vector write in `app/` outside `stamp_embedding`), `tests/unit/test_settings_frontend_view.py` extension or a sibling (the card exists, no CDN, DOM built with `createElement`/`textContent`). +- **Integration:** `tests/integration/test_migration_0029.py` (up AND down, A13), `tests/integration/test_admin_embeddings_api.py` (auth matrix anonymous / token-user / admin; counts correct against a seeded corpus with mixed stamps and one NULL-embedding chunk; the response body carries no API key), `tests/integration/test_sync_embedding_log.py` (the post-sync INFO line and the WARNING only on `mismatched > 0`), and an importer pin that a synced doc's content chunk AND its `is_summary` chunk both carry the configured stamp. +- **Coverage:** **>90%** on `app/` — `uv run pytest --cov=app --cov-report=term-missing`. +- **E2E (AGENTS rule 4):** `uv run pytest tests/e2e/test_embedding_generation.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_admin_settings_panel.py`, `test_admin_auth.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_llm_retry.py`, `test_sync_button.py`, `test_cache_busting.py`. +- **Lint/types:** `uv run ruff check . && uv run pyright`. +- **Log contract (AGENTS rule 10):** the two new greppable lines — `embeddings model=… chunks=… mismatched=… unknown=…` (post-sync) and `reembed: model=… mode=… chunks=… batches=… remaining=… duration_ms=…` — plus `GET /api/admin/embeddings` present in the byte audit and under the bar (`uv run python -m scripts.api_payload_audit`). + +## Documentation & Consistency + +- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it will stand after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**. +- Task 07 rewrites `README.md`'s "Embedding dimension mismatch" troubleshooting entry (`:1408-1410`) — it currently says the fix is "drop + recreate the chunks table", which is right for a DIM change and dangerously incomplete for a same-dim swap — and adds a "Switching the embedding model" runbook section; `AGENTS.md`'s quick reference gains the compare/apply commands. + +## Completion Criteria + +- [ ] `chunks.embedding_model` exists (migration `0029`, reversible), and EVERY vector write in `app/` stamps it — proven by the no-bypass pin and by an importer integration pin covering both a content chunk and the `position = -1` summary chunk. +- [ ] `uv run python -m scripts.reembed_embeddings --compare` prints a mean/median/p10/min cosine agreement and a `COMPATIBLE` / `DRIFT` / `INCOMPATIBLE` verdict, writes nothing, and exits non-zero unless COMPATIBLE — proven against a mock endpoint on a seeded corpus (identical vectors → COMPATIBLE; different vectors → INCOMPATIBLE). +- [ ] `--apply` re-embeds only the chunks whose stamp differs, commits per batch, is resumable (a second run reports `remaining=0`), never leaves an embedding NULL, and does not bump `sources_meta.version` or touch `documents`/`tsv`. +- [ ] A mixed-generation KB is LOUD: `GET /api/admin/embeddings` reports the counts, `#view-settings` shows a `role="alert"` card naming the stamp and the repair command, and the post-sync log WARNs. An all-`unknown` (pre-phase) KB WARNs nothing and shows a `role="status"` card. +- [ ] Retrieval is UNCHANGED by the new column — a mixed-stamp corpus still returns its hits (the D6 pin: detection, not filtering). +- [ ] `README.md` has the runbook (measure → decide → back up → migrate → re-embed → verify → retune → smoke) including the phase-168 `app_settings`-beats-`.env` warning; `AGENTS.md`'s quick reference lists both script modes. +- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation. +- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass). +- [ ] no behavior change in completed phases (retrieval order, the A8 gate, sync counts, saved-chat staleness all untouched). +- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`. + +## Locked decisions (owner-confirmed 2026-10-09 during phase authoring) + +- **A1 — one nullable `chunks.embedding_model` text column**, stamped with the `llm_embed_model` in effect at write time; `NULL` means *unknown* and the migration NEVER backfills a guess. +- **A2 — every vector write goes through one helper** (`stamp_embedding`) so no path can desynchronize the pair; a source-scan test enforces it. +- **A3 — detection only.** Retrieval never filters on the stamp — a mixed corpus still ranks. (Rejected: a `WHERE embedding_model = …` predicate in `_vector_list`, which silently deflects every turn during a backfill and taxes the hot path; the reason is in the code docstring.) +- **A4 — comparability is measured in cosine agreement**, bands `COMPATIBLE ≥ 0.995 > DRIFT ≥ 0.90 > INCOMPATIBLE`, with the float4-round-trip rationale for the 0.995 bar; `--stamp-only` is legal only on COMPATIBLE. +- **A5 — per-batch commits + a resumable `WHERE embedding_model IS DISTINCT FROM :configured` selection**, fail-loud on any batch error, never a NULL embedding, and NEVER a `sources_meta.version` bump (a re-embed is not a KB change — phase 118's convention). +- **A6 — NO new `Settings` field.** Every knob is a CLI flag, so `tests/unit/test_settings_catalog.py`'s `set(CATALOG) == set(Settings.model_fields)` pin is untouched and the phase-168 panel stays display-only for this feature. +- **A7 — one story, one phase, one Playwright file** (AGENTS rule 4): the backend is covered by unit + integration, the visible contract by `tests/e2e/test_embedding_generation.py`. +- **A8 — the three A8/A13 tuning defaults are NOT changed by this phase** — they are operator-side, post-re-embed, battery-driven (D7). If the owner's post-migration battery says a default is now wrong, that is a separate, deliberate change. + +## Assumptions (the request left these open; the executor does not guess) + +- **K1:** the stamp is the model NAME. An operator who re-points the same alias server-side (aipi keeps serving `embed` but now runs gemma-2) is caught by `--compare`, not by the column — the column cannot see a name that did not change, and faking it with a vector fingerprint per row costs an endpoint call per write. This boundary is stated in the column docstring and the README runbook. +- **K2:** the corpus is ≤ ~10⁴ chunks (live and replica: 8,866), so one quiesced in-process pass is minutes; no parallelism, no job queue, no progress table (PLAN A12: in-process state, no Valkey). +- **K3:** every embedded text is already in `chunks.content` — image docs store the VISION DESCRIPTION as their content — so a re-embed never re-calls the vision model and never re-reads a PDF. +- **K4:** `--compare`'s sample is `ORDER BY id LIMIT :n` (deterministic across runs; no RNG, no `RANDOM()`), and N defaults to 25 with `--sample` for more. +- **K5:** the card is read-only and admin-only (no controls, no "fix it now" button — a re-embed is a maintenance-window operation, not a click). +- **K6:** no new fixtures and no dump files are committed (`data/` stays gitignored); tests seed small corpora in the house way. +- **K7:** quiescing = "do not run a sync while `--apply` runs" (single app process). The script does not take a lock; the README runbook states the order. +- **K8:** `embedding_dim` stays 768 and stays non-overridable (phase 168 A3); a real DIM change remains the drop-and-recreate path in the README — this phase is for the same-dim swap that path does not cover. + +## Commit + +```bash +git add app/ alembic/ scripts/ frontend/ tests/ README.md AGENTS.md .agents/phases/ && \ +git commit --no-gpg-sign \ + -m "feat(rag): stamp the embedding generation on every chunk and ship the in-place re-embed tooling" \ + -m "A same-dimension embedding-model swap was invisible: nothing recorded which model made a vector, EmbeddingDimensionError cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the file_hash fast path returns with zero embeds. chunks.embedding_model (migration 0029, nullable, never backfilled) now records it, scripts/reembed_embeddings.py measures comparability in cosine agreement (--compare) and repairs in place (--stamp-only / --apply, per-batch commits, resumable, never a NULL embedding, never a sources_meta bump), and a mixed generation is loud in GET /api/admin/embeddings, the Settings card and the post-sync log. Retrieval is deliberately unfiltered: detection, not repair." +``` diff --git a/.agents/phases/todo/169_embedding_generation_migration/01_embedding_model_column.md b/.agents/phases/todo/169_embedding_generation_migration/01_embedding_model_column.md new file mode 100644 index 0000000..c262770 --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/01_embedding_model_column.md @@ -0,0 +1,33 @@ +# Task 01 — `chunks.embedding_model`: the nullable generation stamp (phase 169 D1, LOCKED A1) + +**Phase:** `169_embedding_generation_migration` · **Source:** owner request 2026-10-09 — an embedding gemma → embedding gemma 2 upgrade keeps 768 dims, so nothing detects the swap +**Story:** n/a + +## Objective + +`chunks` gains one nullable `embedding_model` text column recording which embedding model produced each vector, with a tested downgrade — the schema fact every later task (stamping, the report, the script, the card) reads. + +## Work + +1. `app/models.py` — in `class Chunk` (L232), directly after `embedding: Mapped[list[float] | None] = mapped_column(Vector(EMBEDDING_DIM))` (L241), add: + - `embedding_model: Mapped[str | None] = mapped_column(Text, nullable=True)` + - a comment in the file's established voice stating: the value is the `llm_embed_model` in effect when the vector was written; **`NULL` means UNKNOWN (a pre-phase-169 row), never "no model"**; the dimension cannot distinguish generations because `chunks.embedding` is fixed at 768 by A6 (`EMBEDDING_DIM`, L106), so this column is the ONLY thing that distinguishes two geometries of the same width; and a name change is what is recorded — a server-side re-point that KEEPS the name is detected by `scripts/reembed_embeddings.py --compare`, not here (K1). +2. New `alembic/versions/0029_chunks_embedding_model.py` — `revision = "0029"`, `down_revision = "0028"` (verify with `uv run alembic heads` first: `0028_app_settings` is head). + - `upgrade()`: `op.add_column("chunks", sa.Column("embedding_model", sa.Text(), nullable=True))` + - `downgrade()`: `op.drop_column("chunks", "embedding_model")` (A13 — every migration ships a tested downgrade). + - NO index, NO `UPDATE`: an index on a low-cardinality column costs every chunk write for an audit `GROUP BY` over ≤ 10⁴ rows (D1), and a backfill would stamp pre-phase rows with a model name nobody verified (A1 — the migration must never guess). + - Module docstring: the 2026-10-09 embedding-gemma-2 question, why 768 → 768 is invisible to `EmbeddingDimensionError` (`app/rag/llm.py:314-322`), why a re-sync fixes nothing (the `file_hash` fast path, `app/rag/importer.py:70-84`), and that `NULL` = unknown by design. +3. New `tests/integration/test_migration_0029.py` — copy the structure of `tests/integration/test_migration_0028.py` (the up/down harness, the temp-database idiom it uses): upgrade `head` puts `embedding_model` on `chunks` (assert through `sa.inspect(...).get_columns("chunks")`), the column is nullable and text-typed, a chunk row can be inserted with it `NULL` and with a value, and downgrade removes it cleanly with the surviving rows intact. +4. `README.md` — in the schema section (`grep -n "chunks" README.md` to locate the table listing), add the new column to the `chunks` description with the `NULL` = unknown note; leave the troubleshooting rewrite for task 07. + +## Testing & Quality +- Unit/integration: `tests/integration/test_migration_0029.py` (up, down, nullable both ways, rows survive the downgrade). The rest of the model surface is unchanged, so the existing model/migration suites must stay green untouched. +- Coverage: **>90%** on this task's new/modified code (the migration module is exercised by the up/down test; `app/models.py` is already covered). + +## Completion Criteria +- [ ] `uv run alembic upgrade head` reports `0029` and `\d chunks` (via `podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c '\d chunks'`) shows `embedding_model | text`. +- [ ] `uv run alembic downgrade -1` then `upgrade head` both succeed (A13), pinned by `tests/integration/test_migration_0029.py`. +- [ ] No `UPDATE`, no `CREATE INDEX` in the migration; no existing chunk row gains a non-NULL stamp. +- [ ] `uv run pytest` green; ruff + pyright clean. +- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide). +- [ ] no behavior change in completed work (retrieval, sync counts, the A8 gate untouched). diff --git a/.agents/phases/todo/169_embedding_generation_migration/02_stamp_write_paths.md b/.agents/phases/todo/169_embedding_generation_migration/02_stamp_write_paths.md new file mode 100644 index 0000000..9cb66ce --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/02_stamp_write_paths.md @@ -0,0 +1,32 @@ +# Task 02 — Stamp every vector write: `stamp_embedding()` + the three call sites (phase 169 D2, LOCKED A2) + +**Phase:** `169_embedding_generation_migration` · **Story:** n/a + +## Objective + +Every code path that writes `chunks.embedding` also writes `chunks.embedding_model` from the same settings read, so the pair can never desynchronize — enforced by a source-scan test so a future fourth path fails the build rather than shipping a silent lie. + +## Work + +1. New `app/core/embedding_generation.py` — the phase's home for the generation logic (pure, no HTTP, no `app.db` import at module level; task 03 extends this same module): + - `def stamp_embedding(chunk: Chunk, vector: list[float], model: str | None) -> None` — sets `chunk.embedding = list(vector)` and `chunk.embedding_model = model`. Nothing else: no flush, no commit, no validation beyond what SQLAlchemy does (the caller owns the transaction, exactly like the current code). + - `def current_embedding_model(settings: Settings | None = None) -> str` — returns `settings.llm_embed_model` (defaults to `get_settings()`), the ONE place the stamp value is derived, so a later move to a fingerprinted stamp changes one function (K1). + - Module docstring: why the pair exists (A6 fixes the column at 768 so the dimension cannot distinguish two geometries), the "write the pair together" rule, and the K1 boundary (a name that did not change is invisible here — `scripts/reembed_embeddings.py --compare` is the tool for that). +2. `app/rag/importer.py` — the content-chunk write (L1008-1023): inside `for row, vec in zip(doc.chunks, vectors, strict=True):` replace `row.embedding = vec` with `stamp_embedding(row, vec, current_embedding_model(settings))`. Read `settings` the way the surrounding function already does (it takes/uses `settings` — do NOT add a second `get_settings()` call inside the loop). +3. `app/rag/importer.py` — the summary-chunk write in `_store_summary` (L1587-1589): replace `chunk.embedding = vector` with the helper. This is the `is_summary` `position = -1` row — the one an operator would otherwise discover months later still carries the old geometry. +4. `app/api/docs.py` — the manual-summary edit (L488-491): replace `summary_chunk.embedding = vector` with the helper (the file already constructs `LLMClient()` there; the stamp still comes from `current_embedding_model()`, NOT from the client). +5. Greppable invariant comment at each of the three sites (`# phase 169: the vector and its generation are written together (LOCKED A2)`) — short, one line each. +6. New `tests/unit/test_embedding_generation.py` — `stamp_embedding` sets BOTH attributes; a `Chunk` built through it carries the model; calling it twice overwrites both; `current_embedding_model()` reads `llm_embed_model` from a passed `Settings` and from `get_settings()`. +7. New `tests/unit/test_embedding_write_paths.py` — the no-bypass pin, in the house static-source-scan style (`grep` the `app/` tree from the test, like the existing frontend/CDN pins do): every assignment matching `\.embedding\s*=` or a `Chunk(...)` construction passing `embedding=` inside `app/` lives in `app/core/embedding_generation.py`. Fail with an assertion message naming the offending file:line and the rule ("write vectors through `stamp_embedding` so the generation is stamped with them"). Also pin the reverse: the three call sites still exist (so the pin cannot be satisfied by deleting the calls). + +## Testing & Quality +- Unit: as above. +- Integration (extend the existing importer/sync suites — locate them with `grep -rln "is_summary" tests/integration`): a synced markdown doc's content chunks AND its `is_summary` chunk all carry `embedding_model == "embed"` (the E2E/fixture model name); a doc edited through the summary API (`app/api/docs.py`) re-stamps its summary chunk; a doc RE-imported with changed bytes re-stamps every content chunk it re-embeds. Run the touched suite, then the whole suite. +- Coverage: **>90%** on `app/core/embedding_generation.py` and the modified lines. + +## Completion Criteria +- [ ] `grep -rn "\.embedding\s*=" app/ --include=*.py | grep -v __pycache__` shows assignments ONLY inside `app/core/embedding_generation.py`. +- [ ] A sync against the mock endpoint leaves `select distinct embedding_model from chunks` = one value, the configured `BOR_LLM_EMBED_MODEL`, across content and summary chunks (integration-pinned). +- [ ] `uv run pytest tests/unit/test_embedding_write_paths.py -v` passes, and adding a bypassing `row.embedding = [...]` anywhere in `app/` makes it fail (verified once by hand, then reverted). +- [ ] `uv run pytest` green; coverage on the new module >90%; ruff + pyright clean. +- [ ] no behavior change in completed work — chunk counts, sync summary numbers, `sources_meta` bumps, retrieval order and the A8 gate are all byte-identical to pre-task. diff --git a/.agents/phases/todo/169_embedding_generation_migration/03_generation_report_and_logs.md b/.agents/phases/todo/169_embedding_generation_migration/03_generation_report_and_logs.md new file mode 100644 index 0000000..a5ce7fd --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/03_generation_report_and_logs.md @@ -0,0 +1,33 @@ +# Task 03 — `generation_report()` + the post-sync loud lines (phase 169 D5, LOCKED A3) + +**Phase:** `169_embedding_generation_migration` · **Story:** n/a + +## Objective + +One pure, unit-tested function turns `chunks.embedding_model` into counts-per-generation plus a `stale` verdict, and the sync run reports it on every success — INFO always, WARNING only when chunks actually carry a different generation than the model the app embeds questions with. + +## Work + +1. `app/core/embedding_generation.py` (from task 02) — add: + - `@dataclass(frozen=True) class GenerationReport` with `expected_model: str`, `embedding_dim: int`, `total_chunks: int`, `with_vector: int`, `null_vector: int`, `by_model: tuple[ModelCount, ...]`, `mismatched: int`, `unknown: int`, `stale: bool`. `ModelCount` is a small frozen dataclass `(model: str | None, chunks: int, embedded: int)` — `model is None` is the UNKNOWN bucket. + - `def generation_report(db: Session, expected_model: str | None = None, embedding_dim: int | None = None) -> GenerationReport` — ONE aggregate query (`select(Chunk.embedding_model, func.count(), func.count(Chunk.embedding)).group_by(Chunk.embedding_model)`), then pure Python bucketing: `mismatched` = chunks whose stamp is non-NULL and `!= expected_model`; `unknown` = chunks whose stamp IS NULL; `stale = mismatched > 0`. NULL-vector rows stay inside their model's `chunks` count and show up in `embedded < chunks` — a chunk that lost its vector is a separate defect and must remain visible, not folded into the generation story. + - **D6 in writing:** the docstring states that nothing here filters retrieval (`app/rag/retriever.py:433-455` keeps its unfiltered cosine scan) and why — a `WHERE embedding_model = …` predicate would silently return nothing mid-backfill and deflect every turn, which is worse than the mixed ranking it prevents, and taxes the hot path of every turn for a migration window measured in minutes (LOCKED A3, rejected alternative). +2. `app/api/sync.py` — immediately after `logger.info("sync: done detail=%s", _status.detail)` (L844, the SUCCESS terminal only — the cancelled and failed paths log nothing new): + - open a short-lived `SessionLocal()` (the file's own convention, L538/L805), call `generation_report(db, settings.llm_embed_model, settings.embedding_dim)`, close it; + - always one INFO line, greppable and cron-safe (PLAN §9): `embeddings model= chunks= mismatched= unknown= null_vector=`; + - escalate to **WARNING** only when `mismatched > 0`, appending the repair hint: `— chunks were embedded by another model, run: uv run python -m scripts.reembed_embeddings --apply`. `unknown > 0` alone NEVER warns: a pre-phase-169 KB is 100 % unknown, and a permanent warning is noise that trains the owner to ignore the one signal that matters (D5). + - the whole block is wrapped in `except Exception` (broad, with a `# noqa: BLE001` and a comment, matching this file's background-task style): a reporting hiccup logs `embeddings report unavailable: ` at WARNING and **never** changes the sync's terminal state, counts, or `_status.detail`. +3. New `tests/unit/test_embedding_generation_report.py` — clean corpus (all stamps equal → `stale=False`, `mismatched=0`), a mixed corpus (two named stamps → the counts split, `stale=True`), the UNKNOWN bucket (NULL stamps counted, `unknown=N`, NOT counted as mismatched), NULL-vector rows (`with_vector`/`null_vector` correct, `embedded < chunks`), an empty table (all zeros, `stale=False`), and the `expected_model`/`embedding_dim` defaults resolving from `get_settings()`. +4. New `tests/integration/test_sync_embedding_log.py` — seed a tiny corpus, run the sync the way the existing sync integration suites do (`grep -rln "sync: done\|/api/sync" tests/integration` to find the harness and copy it), and assert with `caplog`: the INFO line on success with the right numbers; the WARNING + repair hint appears after `UPDATE chunks SET embedding_model='older-model'` on a subset; NO warning when the only oddity is `embedding_model IS NULL`; no new line on a cancelled run; a broken report (monkeypatch `generation_report` to raise) leaves the sync terminal `done` and its counts untouched. + +## Testing & Quality +- Unit + integration as above. +- Coverage: **>90%** on `app/core/embedding_generation.py` (every bucket branch) and the new sync block (success / warning / unknown-only / raising paths). + +## Completion Criteria +- [ ] `generation_report()` returns correct counts for the five pinned corpus shapes, and `stale` is `True` ONLY for a non-NULL stamp mismatch. +- [ ] A successful sync logs exactly one `embeddings model=… chunks=… mismatched=… unknown=… null_vector=…` line: `grep`-able on stdout in dev (`uv run uvicorn app.main:app --reload` + a sync) and pinned via `caplog`. +- [ ] The WARNING fires on a mismatch and does NOT fire on an all-unknown corpus; the repair hint names `scripts.reembed_embeddings --apply`. +- [ ] A raising `generation_report` cannot change a sync's terminal state or counts. +- [ ] `uv run pytest` green (the existing sync suites untouched), coverage >90% on the new code, ruff + pyright clean. +- [ ] no behavior change in completed work (the sync's `detail` payload, `sources_meta` bumping and the saved-chat staleness gate are byte-identical). diff --git a/.agents/phases/todo/169_embedding_generation_migration/04_reembed_script.md b/.agents/phases/todo/169_embedding_generation_migration/04_reembed_script.md new file mode 100644 index 0000000..a66c240 --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/04_reembed_script.md @@ -0,0 +1,42 @@ +# Task 04 — `scripts/reembed_embeddings.py`: measure comparability, then repair in place (phase 169 D3/D4, LOCKED A4/A5) + +**Phase:** `169_embedding_generation_migration` · **Story:** n/a + +## Objective + +One script answers the owner's real question — *"are the stored vectors still usable against this endpoint?"* — by measuring cosine agreement (`--compare`), and repairs the corpus when they are not (`--apply`), with `--stamp-only` for the "endpoint changed name, not geometry" case. This is the tool a same-dimension embedding-model swap has never had. + +## Work + +1. `app/core/embedding_generation.py` — the pure math + selection, so coverage and unit tests stay clean and the script stays thin glue: + - `cosine(a: Sequence[float], b: Sequence[float]) -> float` — pure stdlib (`math.fsum`), `0.0` for a zero-norm operand (never `NaN`/`ZeroDivisionError`), `ValueError` naming both lengths when `len(a) != len(b)` (a length difference is a bug, not a score). + - Band constants, documented where they are declared: `COMPATIBLE_MEAN = 0.995` and `INCOMPATIBLE_MEAN = 0.90`, plus `VERDICT_COMPATIBLE / VERDICT_DRIFT / VERDICT_INCOMPATIBLE`. The 0.995 rationale goes in the docstring **verbatim**: pgvector stores `float4` while the app writes Python floats, so an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never to exactly 1.0 — anything meaningfully below that is a different geometry, and "expect exactly 1.0" would make every healthy corpus read as DRIFT. + - `@dataclass(frozen=True) class AgreementStats` — `n`, `mean`, `median`, `p10`, `min`, `worst: tuple[tuple[int, float], ...]` (the ≤ 3 lowest-agreement chunk ids), `verdict: str`; `agreement(samples: Iterable[tuple[int, Sequence[float], Sequence[float]]]) -> AgreementStats` computes it (`(chunk_id, stored, fresh)` triples; empty input → `n=0`, `verdict=VERDICT_INCOMPATIBLE` — "nothing measured" is never "compatible"). + - Selection helpers (all `ORDER BY Chunk.id`, deterministic, no RNG — K4): `sample_chunks(db, limit)` → `id, content, embedding` rows `WHERE embedding IS NOT NULL`; `pending_chunks(db, model, limit=None)` → `id, content` rows `WHERE embedding IS NULL OR embedding_model IS DISTINCT FROM :model`; `count_pending(db, model) -> int`; `count_missing_vector(db) -> int`. + - `stamp_rows(db, model, ids) -> int` — the `--stamp-only` UPDATE, **restricted to rows that actually have a vector** (`embedding IS NOT NULL`) so a vector-less row is never stamped as if it did. +2. New `scripts/reembed_embeddings.py` — the entrypoint mirrors `scripts/regen_folder_summaries.py` EXACTLY (the house one-shot idiom): `get_settings()` → `configure_logging(settings.log_level)` → `configure_debugging()` (debugpy only under `DEBUGPY=1`, never imported otherwise — AGENTS rule 7) → `ensure_admin_configured(settings)`, a boot failure prints `reembed_embeddings: boot failed: ` to stderr and returns 1; one `LLMClient()`; one `asyncio.run(...)`; a short-lived `SessionLocal()` per unit of work (closed in `finally`); one cron-safe PLAN §9 summary line. + - **Modes** (`argparse mutually exclusive group`, default `--compare`): `--compare` | `--stamp-only` | `--apply`; plus `--sample N` (default 25, compare only), `--limit N` (apply/stamp-only: cap the rows this run touches — the debug/smoke knob), `--dry-run` (apply/stamp-only: report `count_pending` and write nothing). + - **Exit codes as module constants**, documented in the parser epilog: `EXIT_OK = 0`, `EXIT_FAILED = 1` (boot/endpoint/DB failure), `EXIT_INCOMPATIBLE = 3` (the measurement answered "no") — so the runbook branches on "the answer is no" without confusing it with "the tool broke". + - **First log line of every run**: `reembed: starting mode= model= dim= base=` with the base URL through `app.core.errors.sanitize_error` (the phase-84 SEC-13 rule: a `user:pass@` in `BOR_LLM_BASE_URL` must never reach a log). This line is also the operator's proof the run used the env the app will run with — the rule that keeps the stamp honest (K1). + - `--compare`: `sample_chunks` → `await llm.embed([content …])` → `agreement(...)` per pair → print a human table (per-chunk id + cosine, and the aggregate), then `reembed: compare model=… n=… mean=… median=… p10=… min=… verdict=…` and, for a non-COMPATIBLE verdict, one more line stating the repair command (`--apply`) and the caveat that `--stamp-only` is only legal on COMPATIBLE. **It writes nothing, ever** — no session writes, no commit (pin it). + - `--apply`: loop `pending_chunks(db, model, limit)` in id-ordered pages (default page 200 rows), per page `vectors = await llm.embed([content …])`, `zip(rows, vectors, strict=True)`, `stamp_embedding(row, vec, model)` (task 02's helper — the pair is written together), `session.commit()` per page. Progress INFO per page: `reembed: progress done=%d pending_total=%d`. **Fail loud:** any `EmbeddingError` logs `reembed: FAILED after %d rows — resume with: uv run python -m scripts.reembed_embeddings --apply` and returns `EXIT_FAILED`; no row is ever written with `embedding=None`, and no row is silently skipped (a NULL embedding vanishes from `_vector_list`, `app/rag/retriever.py:433-455` — exactly the silent class this phase kills). + - `--stamp-only`: `stamp_rows(...)`, loud INFO line naming the count, and a WARNING that repeats what must be true for this to be correct ("only valid on a COMPATIBLE --compare verdict against the SAME endpoint"). + - **Never touched:** `content`, `position`, `tsv`, `documents.*`, `sources_meta.version` (the phase-118 convention, `app/rag/importer.py:51-57` — a re-embed is not a KB change, so saved chats must not go stale). No `chunk_document` call, no PDF read, no vision call (K3 — the embedded text already lives in `chunks.content`, image docs included). + - Final summary line (all modes): `reembed: model= mode= chunks= batches= remaining= duration_ms=<…>`. `remaining=0` after a clean `--apply` is the resumability proof; a second run prints `pending=0` and does nothing. + - No whitespace special-case for `content`: the chunker never yields empty chunks, and an endpoint rejection on a pathological row fails loud on the batch rule above rather than being quietly skipped. + - Module docstring: the embedding-gemma → gemma-2 question this exists for, the three modes, the bands + float4 rationale, the resumability contract, the "run it with the app's own env" rule, and the K1 boundary (a same-name server-side re-point is invisible to the stamp and is exactly what `--compare` catches). +3. New `tests/unit/test_reembed_embeddings.py` — the `from scripts import reembed_embeddings as reembed` idiom of `tests/unit/test_regen_folder_summaries.py:34`: mode mutual-exclusion and defaults, `--sample`/`--limit`/`--dry-run` parsing, the exit-code constants, the band edges (mean exactly 0.995 → COMPATIBLE, 0.9949 → not, exactly 0.90 → INCOMPATIBLE boundary handled per the declared rule, empty sample → INCOMPATIBLE), `cosine` (identical → 1.0, orthogonal → 0.0, zero-norm → 0.0 not NaN, length mismatch → `ValueError` naming both lengths), the deterministic sample/pending ordering, and that the base URL in the starting line is sanitized (an `https://user:pass@host` env yields no `user:pass` in the log). +4. New `tests/integration/test_reembed_apply.py` — seed a small corpus (a handful of docs + chunks) with `embedding_model='old-model'`; with a stubbed `LLMClient` returning known vectors: `--dry-run` writes nothing (`count_pending` unchanged, no UPDATE issued); `--apply` stamps every row, sets the new vectors, leaves `content`/`position`/`tsv`/`documents` and `sources_meta.version` untouched, and a SECOND `--apply` reports `pending=0` / `remaining=0` (the resumability pin); a partial failure (the stub raises `EmbeddingError` on the second page) leaves page 1 committed, exits `EXIT_FAILED`, logs the resume line, and NO row ends up with `embedding IS NULL`; `--stamp-only` changes only `embedding_model` (vector bytes bit-identical — compare against a snapshot) and skips vector-less rows; `--compare` against a stub that echoes the stored vectors → COMPATIBLE / exit 0, against different vectors → INCOMPATIBLE / exit 3, and the chunks table byte-identical afterwards (the read-only pin). + +## Testing & Quality +- Unit + integration as above; the existing `scripts` suites (`tests/unit/test_regen_folder_summaries.py`, `tests/unit/test_llm_probe.py`, `tests/integration/test_import_docs_*.py`) stay green untouched. +- Coverage: **>90%** on `scripts/reembed_embeddings.py` and on `app/core/embedding_generation.py` (every mode, every band edge, every failure path). + +## Completion Criteria +- [ ] `uv run python -m scripts.reembed_embeddings --help` shows the three modes, `--sample`, `--limit`, `--dry-run` and the exit-code epilog. +- [ ] Against the restored 1,000-doc replica locally (`restore-test-db` skill) the three modes do exactly what the docstring says — `--compare` prints a verdict and leaves the dump's fingerprint byte-identical; `--apply` on a `UPDATE chunks SET embedding_model='old-model'` corpus re-embeds all 8,866 chunks and reports `remaining=0`; a second run is a no-op. +- [ ] No code path can write `embedding = NULL`; no path bumps `sources_meta.version`; `--compare` issues no UPDATE (pinned). +- [ ] `--stamp-only` on a COMPATIBLE corpus leaves every `embedding` value bit-identical and stamps all vector rows. +- [ ] `uv run pytest` green, coverage >90% on the script and module, ruff + pyright clean. +- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the full runbook lands in task 07; the phase's final pass re-checks project-wide). +- [ ] no behavior change in completed work (the app, the sync, and retrieval are untouched by this task — only the new script + helpers). diff --git a/.agents/phases/todo/169_embedding_generation_migration/05_admin_embeddings_endpoint.md b/.agents/phases/todo/169_embedding_generation_migration/05_admin_embeddings_endpoint.md new file mode 100644 index 0000000..b5e1478 --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/05_admin_embeddings_endpoint.md @@ -0,0 +1,30 @@ +# Task 05 — `GET /api/admin/embeddings`: the generation report as an admin endpoint (phase 169 D5) + +**Phase:** `169_embedding_generation_migration` · **Story:** n/a + +## Objective + +A read-only, admin-only endpoint exposing task 03's `GenerationReport` as JSON — the single source the Settings card (task 06) and the operator runbook (task 07 + the `migrate-embed-model` skill) both read to answer "is this KB on one embedding generation?". + +## Work + +1. `app/schemas.py` — two response models beside the other read models (the `HealthResponse` style, no `response_model_exclude` tricks): + - `EmbeddingModelCount(BaseModel)`: `model: str | None`, `chunks: int`, `embedded: int` — `model=None` is the UNKNOWN bucket, and `embedded < chunks` names the vector-less rows inside that stamp. + - `EmbeddingGeneration(BaseModel)`: `configured_model: str`, `embedding_dim: int`, `total_chunks: int`, `with_vector: int`, `null_vector: int`, `by_model: list[EmbeddingModelCount]`, `mismatched: int`, `unknown: int`, `stale: bool`. + - Docstring notes: numbers only, no text from the corpus; and NO tuning values here — `relevance_threshold` / the two floors and their `db`/`env`/`default` source badges are phase 168's `GET /api/admin/settings` job, never duplicated (two sources of truth for a tuning value is how a runbook lies). +2. New `app/api/admin_embeddings.py` — `router = APIRouter(prefix="/admin/embeddings", tags=["embeddings"], dependencies=[Depends(require_admin)])` (the `tokens.py`/`ui_settings.py` router-wide pattern: anonymous AND token-user both 403), one route `GET ""` → `EmbeddingGeneration`, `get_db` session, `generation_report(db, settings.llm_embed_model, settings.embedding_dim)` (task 03) mapped to the schema. Module docstring: what "stale" means (a chunk whose stamped model ≠ the model the app embeds questions with, i.e. two geometries on one cosine list), that detection is not filtering (LOCKED A3), and the repair pointer to `scripts.reembed_embeddings`. +3. `app/main.py` — register it with the other admin routers (`app.include_router(admin_embeddings_router, prefix="/api")`, next to the `ui_settings`/`tokens` includes at L146-172). +4. New `tests/integration/test_admin_embeddings_api.py` — the auth matrix copied from the existing admin-surface suite (`grep -rln "require_admin" tests/integration`): anonymous → 403, token user → 403, admin session → 200; against seeded corpora: clean (one stamp → `stale=false`, `mismatched=0`), mixed (two stamps → counts split, `stale=true`, `mismatched` exact), unknown (NULL stamps counted in `unknown` and NOT in `mismatched`), vector-less rows (`null_vector` and the per-stamp `embedded < chunks`), empty DB (all zeros, `stale=false`); the response body contains no `api_key` / `password` / `secret` substring; the payload is tiny (< 4 KB — the phase-167 byte budget is a 256 KB bar, and a handful of counters must never approach it). +5. Byte-audit check (AGENTS rule 10): with the app running, `uv run python -m scripts.api_payload_audit` lists `/api/admin/embeddings` under the bar, and `grep ' :: api '` shows the route's line. + +## Testing & Quality +- Integration as above; unit coverage comes free from task 03's report tests (the route is a mapping, so pin it here instead of duplicating the bucket tests). +- Coverage: **>90%** on `app/api/admin_embeddings.py` and the new schemas. + +## Completion Criteria +- [ ] `GET /api/admin/embeddings` returns 200 for the admin and 403 for anonymous and token users (all pinned). +- [ ] The JSON matches the four seeded corpus shapes exactly, `stale` is true only for a non-NULL mismatch, and `unknown` never counts toward `mismatched`. +- [ ] No secret substring in the body; the payload is a few hundred bytes and appears under the bar in `scripts.api_payload_audit`. +- [ ] `uv run pytest` green, coverage >90% on the new route, ruff + pyright clean. +- [ ] README.md updated for the new endpoint in its API listing (task 07 finishes the runbook; the final pass re-checks project-wide). +- [ ] no behavior change in completed work. diff --git a/.agents/phases/todo/169_embedding_generation_migration/06_settings_status_card.md b/.agents/phases/todo/169_embedding_generation_migration/06_settings_status_card.md new file mode 100644 index 0000000..fc65f4d --- /dev/null +++ b/.agents/phases/todo/169_embedding_generation_migration/06_settings_status_card.md @@ -0,0 +1,34 @@ +# Task 06 — the "Embedding generation" card in the Settings view (phase 169 D5, AGENTS rule 5) + +**Phase:** `169_embedding_generation_migration` · **Story:** n/a + +## Objective + +The admin sees, unprompted and in the browser, whether the corpus and the configured embedding model agree — one read-only card at the top of phase 168's `#view-settings` that is quiet-clean on a healthy KB and unmistakably loud when chunks were embedded by another model. + +## Work + +1. `frontend/index.html` — inside `#view-settings` (the phase-168 view), ABOVE the settings `.table-wrap` card, a new card in the file's existing card idiom: + - `
` with `

Embedding generation

`, a `

` for the state line, and a `