brain-of-reese/.agents/skills/migrate-embed-model/SKILL.md

---
name: migrate-embed-model
description: Walks a developer through migrating Brain of Reese to a new embedding model — probe and benchmark the new model, MEASURE whether the stored 768-dim vectors are still comparable (cosine-agreement verdict), back up and run the migration on production (re-embed in place, or stamp only), then re-tune the honesty/citation thresholds and smoke-test them against the retrieval battery. Use when the user asks to change, upgrade, switch or roll back the embedding model (e.g. "upgrade the embed model to embedding gemma 2", "switch BOR to a new embedding model", "can I change the embed model without re-importing?", "re-embed the knowledge base"). Same-dimension swaps are the dangerous case — nothing errors, retrieval just degrades silently. Does NOT apply when the output DIMENSION changes (that is the drop-and-recreate path in README.md) and does not test chat/summary models (see test-chat-model / test-summary-model).
---

# Migrate the Embedding Model

An embedding-model change that keeps the dimension (768 → 768) produces **no
error anywhere** in Brain of Reese: `chunks.embedding` is fixed at
`BOR_EMBEDDING_DIM`, `EmbeddingDimensionError` (`app/rag/llm.py`) only fires on
a dimension mismatch, and a re-sync re-embeds **nothing** because the
`documents.file_hash` fast path returns with zero embeds. The corpus then
scores questions with a different geometry and retrieval silently decays.

This skill is the controlled procedure for that change: **measure first, then
decide, then migrate, then re-tune, then smoke-test.** Four stages, in that
order — never skip stage 2 (the measurement) and never re-tune before the
re-embed.

```
stage 1  probe + benchmark the new model          (is it 768? is it any good?)
stage 2  measure comparability  --compare         (do I have to re-embed at all?)
stage 3  prod migration                           (backup → migrate → re-embed → verify)
stage 4  re-tune the thresholds + smoke test      (battery-gated)
```

## Prerequisites

- **Phase 169 must be applied** — the tooling lives there. Check before
  anything else; if either is missing, STOP and tell the user to run the phase
  (`phased-execution` skill) rather than improvising:
  ```bash
  ls scripts/reembed_embeddings.py
  uv run alembic heads          # must be >= 0029
  ```
- The db container is up: `podman compose up -d db`
- `.env` has `BOR_LLM_BASE_URL` / `BOR_LLM_API_KEY` (or `$AIPI_KEY`) pointing at
  the endpoint that serves BOTH the old and the new embedding model.
- **Keep the old model served** on the endpoint for the whole bake window —
  rollback means re-embedding back to it (stage "Rollback").
- Production access: read the prod host / DSN / service name from your
  deployment repo or the prod env file. Never hardcode them into this file or a
  commit. Below, `prod` means "run the quoted command there".

## Rules (non-negotiable)

- **Never switch the app's model before stage 2 is measured and a backup
  exists.** `--apply` overwrites vectors; the `pg_dump` is the only undo.
- **Never `--stamp-only` without a `COMPATIBLE` verdict.** Stamping claims
  "these vectors belong to this model", and after that claim nothing will ever
  re-embed them.
- **Never re-tune the thresholds before the corpus is on ONE generation.**
  Retuning against a mixed corpus tunes to noise (stage 4 is after stage 3).
- **Never run `--apply` while a sync is running** (confirm idle first).
- **The script and the app must use the same model name.** `--apply` stamps
  `BOR_LLM_EMBED_MODEL` from ITS OWN environment; running it with a different
  env than the app's makes every stamp a lie. Its first log line prints the
  model and the sanitized base URL — read that line before walking away.
- **Check the `app_settings` overlay before changing any value** (phase 168):
  a value saved in the Settings panel is a DB row and **beats `.env`**. Every
  stage that changes a value starts with the `select … from app_settings`
  check in step 4.1.
- **Do not edit `app/` or `alembic/` from this skill.** If the new model
  exposes an app defect (e.g. a different input-token ceiling), report it as a
  phase candidate — the batching constants in `app/rag/llm.py`
  (`_TOKENS_PER_CHAR`, `_REQUEST_TOKEN_BUDGET`) are code, not a drive-by edit.
- Retrieval is **never** filtered by generation (phase 169 LOCKED A3): a mixed
  corpus keeps ranking, it just ranks wrongly. The card, the endpoint and the
  sync log are the signals — trust them over "it seems to work".
- ntfy (if used at all) is **outbound only** — milestones, never questions.

## Stage 1 — Probe and benchmark the new model

Point at the new model *for one process at a time*; do not touch the running
app yet.

```bash
NEW=embeddinggemma-2          # whatever the endpoint calls it
BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.llm_probe
BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.test_embed_model --model "$NEW" --runs 3
```

- `llm_probe` prints the output dimension against `BOR_EMBEDDING_DIM` (768).
  **If the dimension differs, STOP** — this skill does not apply. That is the
  README's "Embedding dimension mismatch" path (`BOR_EMBEDDING_DIM` + drop and
  recreate the chunks table), which is destructive in a different way.
- `test_embed_model` runs the three gated checks (dimension / cosine margin ≥
  0.05 on the fixed semantic-pair corpus / speed) and **appends its rows to
  `benchmarks/model_benchmarks.csv` automatically**. Record the numbers; a
  cosine-margin failure means the model is not fit to serve, full stop.
- Note the **speed** (the old model's reference is ~94 vec/s → the live corpus
  of 1,000 docs / 8,866 chunks re-embeds in minutes; if the new model is
  10× slower, plan the window accordingly).
- **Input ceiling check** — the app batches embeddings against the current
  endpoint's ~1024-token hard rejection. Probe it once with a long input:
  ```bash
  BOR_LLM_EMBED_MODEL=$NEW uv run python - <<'PY'
  import asyncio, httpx
  from app.config import get_settings
  from app.rag.llm import LLMClient
  async def go():
      llm = LLMClient()
      for n in (1200, 4000, 12000):
          try:
              await llm.embed_one("word " * n)
              print(f"ok   ~{n} chars")
          except Exception as e:
              print(f"FAIL ~{n} chars: {type(e).__name__}: {str(e)[:120]}")
  asyncio.run(go())
  PY
  ```
  Fewer accepted chars than the old model → `BOR_EMBED_QUESTION_MAX_CHARS`
  (env) must come down AND the `_REQUEST_TOKEN_BUDGET` constant
  (`app/rag/llm.py`) is now wrong: flag it as a phase, don't hand-edit.
  More accepted → nothing to do; the existing budget is merely conservative.

## Stage 2 — Measure comparability (the answer to "are they compatible?")

Run the measurement as its **own process with an env override**. It reads and
writes nothing, and it does not disturb the running app — the app keeps serving
the old model while you measure.

```bash
BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.reembed_embeddings --compare --sample 25
```

It embeds each sampled chunk's **stored text** with the new endpoint and
compares against the **stored vector**, printing `mean / median / p10 / min`
cosine agreement and a verdict:

| verdict | mean cosine | what it means | what to do |
|---|---|---|---|
| `COMPATIBLE` | ≥ 0.995 | same geometry, different name/build | stage 3 with `--stamp-only` — **no re-embed needed** |
| `DRIFT` | 0.90 – 0.995 | related but not the same space | stage 3 with `--apply` (cheap; note it in the record) |
| `INCOMPATIBLE` | < 0.90 | different space — this is the normal case for a model upgrade | stage 3 with `--apply` |

- Exit codes: `0` COMPATIBLE, `3` non-compatible verdict (the answer is "no"),
  `1` the tool/boot/endpoint failed (something broke). Branch on those, never
  on the pretty text.
- The bar is **0.995, not 1.0**, because pgvector stores `float4` while the app
  writes Python floats: an unchanged model reproduces a vector to float4
  round-trip precision (~0.9999), never exactly 1.0. Expecting exactly 1.0
  makes a healthy corpus read as `DRIFT`.
- Also print the **baseline** you are protecting, so stage 4 has a "before":
  ```bash
  uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt \
    | tee /tmp/embed_baseline_$(date +%Y%m%d_%H%M%S).txt
  podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c "
    select percentile_cont(0.1) within group (order by top_score) as p10,
           percentile_cont(0.5) within group (order by top_score) as p50,
           percentile_cont(0.9) within group (order by top_score) as p90,
           count(*) filter (where deflected) as deflected, count(*) as turns
    from query_log where created_at > now() - interval '90 days';"
  ```
  (On prod, run the equivalent `psql` against the prod database and keep both
  outputs out of git — `/tmp` is fine.)

## Stage 3 — Migrate production

Do these back to back in one maintenance window. The corpus is
1,000 docs / 8,866 chunks ≈ minutes at the benchmarked speed; the mixed window
is as long as the re-embed, and the Settings card tells you when it is over.

### 3.1 Confirm nothing is mid-flight, then BACK UP

```bash
curl -s -b "$JAR" https://<prod-host>/api/sync/status     # must be idle (verify the route in app/api/sync.py)

podman exec <prod-db-container> pg_dump -U <user> <db> -Fc \
  > /tmp/pre_embed_migration_$(date +%Y%m%d_%H%M%S).dump
ls -lh /tmp/pre_embed_migration_*.dump
podman exec <prod-db-container> pg_restore --list < /tmp/pre_embed_migration_<timestamp>.dump | head
```

Copy the dump **off the app box**, and only continue once `pg_restore --list`
reads it. The dump — not the rollback section below — is the real undo.

### 3.2 Apply the schema

```bash
uv run alembic upgrade head      # 0029_chunks_embedding_model, reversible
```

Adding `chunks.embedding_model` is instant and non-destructive; existing rows
stay `NULL` = **unknown by design** (never backfilled with a guess).

### 3.3 Flip the app, then re-embed (or stamp)

Flip the app's model, then immediately run the repair with the **same** env:

- `.env` + restart (`BOR_LLM_EMBED_MODEL=$NEW`), **or**
- the admin **Settings** panel → `llm_embed_model` → Save (phase 168 applies it
  live, no restart — and writes an `app_settings` row, which then beats `.env`
  for the rest of the app's life; that is fine, just know where the truth is).

```bash
# the normal case (DRIFT / INCOMPATIBLE):
uv run python -m scripts.reembed_embeddings --apply
# only on a COMPATIBLE verdict — rewrites no vectors, just the stamps:
uv run python -m scripts.reembed_embeddings --stamp-only
```

Read the first log line (`mode`, `model`, `dim`, `base=`) before walking away,
and the summary line when it finishes:

```
reembed: model=<stamp> mode=apply chunks=8866 batches=<n> remaining=0 duration_ms=<ms>
```

`remaining=0` is the goal. The run is resumable by construction (it selects
`WHERE embedding IS NULL OR embedding_model IS DISTINCT FROM <model>` and
commits per batch): if it dies mid-way, fix the cause and re-run — it picks up
where it stopped. It never writes a NULL embedding and it never bumps
`sources_meta.version`, so saved chats do not go stale over a re-embed.

### 3.4 Verify the migration before touching any value

```bash
podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c "
  select coalesce(embedding_model,'<unknown>') as model, count(*) as chunks,
         count(embedding) as with_vector
  from chunks group by 1 order by 2 desc;"

uv run python -m scripts.reembed_embeddings --compare --sample 25   # must now read COMPATIBLE (~1.0)

curl -s -b "$JAR" https://<prod-host>/api/admin/embeddings          # "stale": false, "unknown": 0, "null_vector": 0
```

Then in the browser: **Settings → Embedding generation** shows the clean
`status` card, not the `alert`. And the post-sync line agrees — trigger nothing,
just check the last sync's log:

```bash
grep ' :: embeddings model=' <app log> | tail -1     # mismatched=0 unknown=0
```

## Stage 4 — Re-tune the values, then smoke-test them

The three honesty/citation values were hand-tuned against the OLD model's
cosine distribution (`app/config.py`: `relevance_threshold=0.62`,
`lexical_support_floor=0.35`, `source_usefulness_floor=0.35`). A new model moves
the whole distribution, so they are almost certainly wrong after stage 3 — in
whichever direction the percentiles moved.

### 4.1 Know which layer wins

```sql
select key, value from app_settings
where key in ('llm_embed_model','relevance_threshold',
              'lexical_support_floor','source_usefulness_floor');
```

A row here **beats `.env`** (phase 168's overlay). Change values in ONE place —
the panel (live, DB row) or `.env` (restart, env value) — and never let a stale
DB row silently pin an old number.

### 4.2 Measure, then set

1. Re-run the battery and collect two numbers: the **best cosine of known-good
   questions** (from `/tmp/embed_baseline_*.txt`'s same questions) and the best
   cosine of the battery's **out-of-KB control** question.
2. `relevance_threshold` goes **between the two clusters** (midpoint). Good
   hits clustering lower than before → lower it; nonsense questions scoring
   higher → raise it.
3. `lexical_support_floor` sits at the bottom of the good-hit cluster — it is
   the bar a lexical hit needs vector corroboration to flip a turn to HIGH.
   `source_usefulness_floor` (the citation tier) sits at the same bar by
   default.
4. The validators enforce `0 ≤ floor ≤ relevance_threshold`
   (`app/config.py`'s `_lexical_support_floor_bounds` /
   `_source_usefulness_floor_bounds`) — a floor above the threshold is refused
   at startup (or a 422 in the panel), which is the intended guard, not a bug.
5. Apply → re-run the battery → iterate. Two or three passes is normal; keep
   each pass's numbers.

Symptom → first move:

| symptom after the swap | first move |
|---|---|
| real questions deflect, cosines cluster low | lower `relevance_threshold`, then both floors (keep floors ≤ threshold) |
| out-of-KB questions answered confidently | raise `relevance_threshold` |
| a single weak token match promotes to HIGH | raise `lexical_support_floor` |
| citations present but irrelevant | raise `source_usefulness_floor` |
| `embedding failed (too-large)` on questions | lower `BOR_EMBED_QUESTION_MAX_CHARS`; if import batches also fail, `_REQUEST_TOKEN_BUDGET` needs a phase |

### 4.3 Smoke test

- [ ] `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` — every gated item in that file passes (the gitea / forgejo / gateway name questions, the qwen3.8-27b incident regression, both IKEA `BESTÅ`/`besta` spellings, and the out-of-KB control's verdict). This file is the ruler — **read it, never edit it** to make a run pass.
- [ ] A real question in the browser answers with citations and a plausible `top_score` (per-turn line: `grep ' :: chat: ' <app log> | tail -5`).
- [ ] The out-of-KB control question still gets the amber honest deflection — a model that scores everything high has NOT been "tuned", it has been blinded.
- [ ] `GET /api/admin/embeddings` → `stale: false`, `unknown: 0`, `null_vector: 0`; the Settings card is the clean `status` variant.
- [ ] `psql`: the new cosine percentiles sit where the retuned threshold expects them, and `select deflected, count(*) from query_log group by 1` over the next day matches the stage-2 baseline's deflection rate within a few points.
- [ ] `uv run python -m scripts.api_payload_audit` still clean (the migration added one small admin endpoint).

### 4.4 Record it

- `benchmarks/model_benchmarks.csv` already has stage 1's rows (written by
  `scripts.test_embed_model`).
- If a **default** in `app/config.py` changed as a result, that is its own
  deliberate commit (`tune(rag): …`, with the before/after percentiles in the
  body) — never smuggled into a migration.
- Commit any `.env.example` / README diff. Note in the body: date, old → new
  model, the `--compare` verdict + mean cosine before and after, chunks
  re-embedded, and the final three values.

## Rollback

Pick by what went wrong:

1. **Vectors unchanged (`--stamp-only` path):** flip `llm_embed_model` back
   (panel or `.env` + restart). Done — the vectors were never touched.
2. **Vectors re-embedded, app box still healthy:** flip `llm_embed_model` back
   to the old model (which must still be served) and run
   `uv run python -m scripts.reembed_embeddings --apply` again — it re-embeds
   the corpus back into the old geometry because the selection is "stamp differs
   from the configured model". Minutes, same window as the forward migration.
3. **Something worse (bad dump, half-run, corrupted corpus):** restore the stage
   3.1 dump — the exact pre-migration state, vectors and stamps included:
   ```bash
   podman exec <prod-db-container> pg_restore -U <user> -d <db> --clean --if-exists --no-owner \
     < /tmp/pre_embed_migration_<timestamp>.dump
   ```
   Then verify the fingerprint (`documents` / `chunks` counts and
   `select count(*) from chunks where embedding_model is not null`) before
   sending traffic back.

## Troubleshooting

- **`EmbeddingDimensionError` / dimension ≠ 768** — the dimension changed, so
  this skill is the wrong tool: README → "Embedding dimension mismatch"
  (`BOR_EMBEDDING_DIM`, drop + recreate `chunks`, re-import). Do not "make it
  fit" by resizing the column.
- **`reembed: FAILED … resume with:`** — the run stopped loud, on purpose. The
  batches already committed are done; the rest are pending. Find the cause
  (endpoint outage, oversized input, a pathological chunk) and re-run the
  printed command. Never hand-edit vectors.
- **`remaining=` never reaches 0** — some rows never match. Look at them:
  `select id, length(content), embedding_model from chunks where embedding_model
   is distinct from '<configured>' order by id limit 20;` Empty `content` or a
  content the endpoint rejects is the usual answer.
- **Card stuck on `unknown`** — those are pre-phase-169 rows (stamp `NULL`,
  which is *unknown*, not "wrong"). The sync logs them at INFO and never warns,
  and the card keeps the calm `status` variant. They resolve the first time
  they are legitimately re-embedded, or now via `--stamp-only` (COMPATIBLE
  only) / `--apply`.
- **Verdict `DRIFT` with mean 0.97-0.99** — usually a different server build /
  quantization of a nominally identical model. Re-embed anyway (it is minutes);
  note it in the record so nobody re-litigates it later.
- **The card and the app disagree about which model is live** — the overlay:
  a `app_settings.llm_embed_model` row beats `.env`. `select key, value from
  app_settings where key='llm_embed_model';`, then fix it in the panel (or
  `delete from app_settings where key='llm_embed_model';` to fall back to env).
- **Everything still looks fine but answers got worse** — that is the failure
  mode this whole procedure exists for, and it hides in the `top_score`
  column, not in the logs. Compare stage 2's percentile snapshot with
  `query_log.top_score` over the last day; a collapsed spread (e.g. p10 ≈ p90)
  means a mixed generation or a threshold that no longer discriminates.