brain-of-reese/.agents/skills/migrate-embed-model/SKILL.md

name, description

name

description

migrate-embed-model

Walks a developer through migrating Brain of Reese to a new embedding model — probe and benchmark the new model, MEASURE whether the stored 768-dim vectors are still comparable (cosine-agreement verdict), back up and run the migration on production (re-embed in place, or stamp only), then re-tune the honesty/citation thresholds and smoke-test them against the retrieval battery. Use when the user asks to change, upgrade, switch or roll back the embedding model (e.g. "upgrade the embed model to embedding gemma 2", "switch BOR to a new embedding model", "can I change the embed model without re-importing?", "re-embed the knowledge base"). Same-dimension swaps are the dangerous case — nothing errors, retrieval just degrades silently. Does NOT apply when the output DIMENSION changes (that is the drop-and-recreate path in README.md) and does not test chat/summary models (see test-chat-model / test-summary-model).

Migrate the Embedding Model

An embedding-model change that keeps the dimension (768 → 768) produces no error anywhere in Brain of Reese: chunks.embedding is fixed at BOR_EMBEDDING_DIM, EmbeddingDimensionError (app/rag/llm.py) only fires on a dimension mismatch, and a re-sync re-embeds nothing because the documents.file_hash fast path returns with zero embeds. The corpus then scores questions with a different geometry and retrieval silently decays.

This skill is the controlled procedure for that change: measure first, then decide, then migrate, then re-tune, then smoke-test. Four stages, in that order — never skip stage 2 (the measurement) and never re-tune before the re-embed.

stage 1  probe + benchmark the new model          (is it 768? is it any good?)
stage 2  measure comparability  --compare         (do I have to re-embed at all?)
stage 3  prod migration                           (backup → migrate → re-embed → verify)
stage 4  re-tune the thresholds + smoke test      (battery-gated)

Prerequisites

• Phase 169 must be applied — the tooling lives there. Check before anything else; if either is missing, STOP and tell the user to run the phase (phased-execution skill) rather than improvising:

  ls scripts/reembed_embeddings.py
  uv run alembic heads          # must be >= 0029
  

• The db container is up: podman compose up -d db
• .env has BOR_LLM_BASE_URL / BOR_LLM_API_KEY (or $AIPI_KEY) pointing at the endpoint that serves BOTH the old and the new embedding model.
• Keep the old model served on the endpoint for the whole bake window — rollback means re-embedding back to it (stage "Rollback").
• Production access: read the prod host / DSN / service name from your deployment repo or the prod env file. Never hardcode them into this file or a commit. Below, prod means "run the quoted command there".

Rules (non-negotiable)

• Never switch the app's model before stage 2 is measured and a backup exists. --apply overwrites vectors; the pg_dump is the only undo.
• Never --stamp-only without a COMPATIBLE verdict. Stamping claims "these vectors belong to this model", and after that claim nothing will ever re-embed them.
• Never re-tune the thresholds before the corpus is on ONE generation. Retuning against a mixed corpus tunes to noise (stage 4 is after stage 3).
• Never run --apply while a sync is running (confirm idle first).
• The script and the app must use the same model name. --apply stamps BOR_LLM_EMBED_MODEL from ITS OWN environment; running it with a different env than the app's makes every stamp a lie. Its first log line prints the model and the sanitized base URL — read that line before walking away.
• Check the app_settings overlay before changing any value (phase 168): a value saved in the Settings panel is a DB row and beats .env. Every stage that changes a value starts with the select … from app_settings check in step 4.1.
• Do not edit app/ or alembic/ from this skill. If the new model exposes an app defect (e.g. a different input-token ceiling), report it as a phase candidate — the batching constants in app/rag/llm.py (_TOKENS_PER_CHAR, _REQUEST_TOKEN_BUDGET) are code, not a drive-by edit.
• Retrieval is never filtered by generation (phase 169 LOCKED A3): a mixed corpus keeps ranking, it just ranks wrongly. The card, the endpoint and the sync log are the signals — trust them over "it seems to work".
• ntfy (if used at all) is outbound only — milestones, never questions.

Stage 1 — Probe and benchmark the new model

Point at the new model for one process at a time; do not touch the running app yet.

NEW=embeddinggemma-2          # whatever the endpoint calls it
BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.llm_probe
BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.test_embed_model --model "$NEW" --runs 3

• llm_probe prints the output dimension against BOR_EMBEDDING_DIM (768). If the dimension differs, STOP — this skill does not apply. That is the README's "Embedding dimension mismatch" path (BOR_EMBEDDING_DIM + drop and recreate the chunks table), which is destructive in a different way.
• test_embed_model runs the three gated checks (dimension / cosine margin ≥ 0.05 on the fixed semantic-pair corpus / speed) and appends its rows to benchmarks/model_benchmarks.csv automatically. Record the numbers; a cosine-margin failure means the model is not fit to serve, full stop.
• Note the speed (the old model's reference is ~94 vec/s → the live corpus of 1,000 docs / 8,866 chunks re-embeds in minutes; if the new model is 10× slower, plan the window accordingly).
• Input ceiling check — the app batches embeddings against the current endpoint's ~1024-token hard rejection. Probe it once with a long input:

  BOR_LLM_EMBED_MODEL=$NEW uv run python - <<'PY'
  import asyncio, httpx
  from app.config import get_settings
  from app.rag.llm import LLMClient
  async def go():
      llm = LLMClient()
      for n in (1200, 4000, 12000):
          try:
              await llm.embed_one("word " * n)
              print(f"ok   ~{n} chars")
          except Exception as e:
              print(f"FAIL ~{n} chars: {type(e).__name__}: {str(e)[:120]}")
  asyncio.run(go())
  PY
  

  Fewer accepted chars than the old model → BOR_EMBED_QUESTION_MAX_CHARS (env) must come down AND the _REQUEST_TOKEN_BUDGET constant (app/rag/llm.py) is now wrong: flag it as a phase, don't hand-edit. More accepted → nothing to do; the existing budget is merely conservative.

Stage 2 — Measure comparability (the answer to "are they compatible?")

Run the measurement as its own process with an env override. It reads and writes nothing, and it does not disturb the running app — the app keeps serving the old model while you measure.

BOR_LLM_EMBED_MODEL=$NEW uv run python -m scripts.reembed_embeddings --compare --sample 25

It embeds each sampled chunk's stored text with the new endpoint and compares against the stored vector, printing mean / median / p10 / min cosine agreement and a verdict:

verdict	mean cosine	what it means	what to do
COMPATIBLE	≥ 0.995	same geometry, different name/build	stage 3 with --stamp-only — no re-embed needed
DRIFT	0.90 – 0.995	related but not the same space	stage 3 with --apply (cheap; note it in the record)
INCOMPATIBLE	< 0.90	different space — this is the normal case for a model upgrade	stage 3 with --apply

• Exit codes: 0 COMPATIBLE, 3 non-compatible verdict (the answer is "no"), 1 the tool/boot/endpoint failed (something broke). Branch on those, never on the pretty text.
• The bar is 0.995, not 1.0, because pgvector stores float4 while the app writes Python floats: an unchanged model reproduces a vector to float4 round-trip precision (~0.9999), never exactly 1.0. Expecting exactly 1.0 makes a healthy corpus read as DRIFT.
• Also print the baseline you are protecting, so stage 4 has a "before":

  uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt \
    | tee /tmp/embed_baseline_$(date +%Y%m%d_%H%M%S).txt
  podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c "
    select percentile_cont(0.1) within group (order by top_score) as p10,
           percentile_cont(0.5) within group (order by top_score) as p50,
           percentile_cont(0.9) within group (order by top_score) as p90,
           count(*) filter (where deflected) as deflected, count(*) as turns
    from query_log where created_at > now() - interval '90 days';"
  

  (On prod, run the equivalent psql against the prod database and keep both outputs out of git — /tmp is fine.)

Stage 3 — Migrate production

Do these back to back in one maintenance window. The corpus is 1,000 docs / 8,866 chunks ≈ minutes at the benchmarked speed; the mixed window is as long as the re-embed, and the Settings card tells you when it is over.

3.1 Confirm nothing is mid-flight, then BACK UP

curl -s -b "$JAR" https://<prod-host>/api/sync/status     # must be idle (verify the route in app/api/sync.py)

podman exec <prod-db-container> pg_dump -U <user> <db> -Fc \
  > /tmp/pre_embed_migration_$(date +%Y%m%d_%H%M%S).dump
ls -lh /tmp/pre_embed_migration_*.dump
podman exec <prod-db-container> pg_restore --list < /tmp/pre_embed_migration_<timestamp>.dump | head

Copy the dump off the app box, and only continue once pg_restore --list reads it. The dump — not the rollback section below — is the real undo.

3.2 Apply the schema

uv run alembic upgrade head      # 0029_chunks_embedding_model, reversible

Adding chunks.embedding_model is instant and non-destructive; existing rows stay NULL = unknown by design (never backfilled with a guess).

3.3 Flip the app, then re-embed (or stamp)

Flip the app's model, then immediately run the repair with the same env:

• .env + restart (BOR_LLM_EMBED_MODEL=$NEW), or
• the admin Settings panel → llm_embed_model → Save (phase 168 applies it live, no restart — and writes an app_settings row, which then beats .env for the rest of the app's life; that is fine, just know where the truth is).

# the normal case (DRIFT / INCOMPATIBLE):
uv run python -m scripts.reembed_embeddings --apply
# only on a COMPATIBLE verdict — rewrites no vectors, just the stamps:
uv run python -m scripts.reembed_embeddings --stamp-only

Read the first log line (mode, model, dim, base=) before walking away, and the summary line when it finishes:

reembed: model=<stamp> mode=apply chunks=8866 batches=<n> remaining=0 duration_ms=<ms>

remaining=0 is the goal. The run is resumable by construction (it selects WHERE embedding IS NULL OR embedding_model IS DISTINCT FROM <model> and commits per batch): if it dies mid-way, fix the cause and re-run — it picks up where it stopped. It never writes a NULL embedding and it never bumps sources_meta.version, so saved chats do not go stale over a re-embed.

3.4 Verify the migration before touching any value

podman exec brain-of-reese_db_1 psql -U reese -d brain_of_reese -c "
  select coalesce(embedding_model,'<unknown>') as model, count(*) as chunks,
         count(embedding) as with_vector
  from chunks group by 1 order by 2 desc;"

uv run python -m scripts.reembed_embeddings --compare --sample 25   # must now read COMPATIBLE (~1.0)

curl -s -b "$JAR" https://<prod-host>/api/admin/embeddings          # "stale": false, "unknown": 0, "null_vector": 0

Then in the browser: Settings → Embedding generation shows the clean status card, not the alert. And the post-sync line agrees — trigger nothing, just check the last sync's log:

grep ' :: embeddings model=' <app log> | tail -1     # mismatched=0 unknown=0

Stage 4 — Re-tune the values, then smoke-test them

The three honesty/citation values were hand-tuned against the OLD model's cosine distribution (app/config.py: relevance_threshold=0.62, lexical_support_floor=0.35, source_usefulness_floor=0.35). A new model moves the whole distribution, so they are almost certainly wrong after stage 3 — in whichever direction the percentiles moved.

4.1 Know which layer wins

select key, value from app_settings
where key in ('llm_embed_model','relevance_threshold',
              'lexical_support_floor','source_usefulness_floor');

A row here beats .env (phase 168's overlay). Change values in ONE place — the panel (live, DB row) or .env (restart, env value) — and never let a stale DB row silently pin an old number.

4.2 Measure, then set

1. Re-run the battery and collect two numbers: the best cosine of known-good questions (from /tmp/embed_baseline_*.txt's same questions) and the best cosine of the battery's out-of-KB control question.
2. relevance_threshold goes between the two clusters (midpoint). Good hits clustering lower than before → lower it; nonsense questions scoring higher → raise it.
3. lexical_support_floor sits at the bottom of the good-hit cluster — it is the bar a lexical hit needs vector corroboration to flip a turn to HIGH. source_usefulness_floor (the citation tier) sits at the same bar by default.
4. The validators enforce 0 ≤ floor ≤ relevance_threshold (app/config.py's _lexical_support_floor_bounds / _source_usefulness_floor_bounds) — a floor above the threshold is refused at startup (or a 422 in the panel), which is the intended guard, not a bug.
5. Apply → re-run the battery → iterate. Two or three passes is normal; keep each pass's numbers.

Symptom → first move:

symptom after the swap	first move
real questions deflect, cosines cluster low	lower relevance_threshold, then both floors (keep floors ≤ threshold)
out-of-KB questions answered confidently	raise relevance_threshold
a single weak token match promotes to HIGH	raise lexical_support_floor
citations present but irrelevant	raise source_usefulness_floor
embedding failed (too-large) on questions	lower BOR_EMBED_QUESTION_MAX_CHARS; if import batches also fail, _REQUEST_TOKEN_BUDGET needs a phase

4.3 Smoke test

• uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt — every gated item in that file passes (the gitea / forgejo / gateway name questions, the qwen3.8-27b incident regression, both IKEA BESTÅ/besta spellings, and the out-of-KB control's verdict). This file is the ruler — read it, never edit it to make a run pass.
• A real question in the browser answers with citations and a plausible top_score (per-turn line: grep ' :: chat: ' <app log> | tail -5).
• The out-of-KB control question still gets the amber honest deflection — a model that scores everything high has NOT been "tuned", it has been blinded.
• GET /api/admin/embeddings → stale: false, unknown: 0, null_vector: 0; the Settings card is the clean status variant.
• psql: the new cosine percentiles sit where the retuned threshold expects them, and select deflected, count(*) from query_log group by 1 over the next day matches the stage-2 baseline's deflection rate within a few points.
• uv run python -m scripts.api_payload_audit still clean (the migration added one small admin endpoint).

4.4 Record it

• benchmarks/model_benchmarks.csv already has stage 1's rows (written by scripts.test_embed_model).
• If a default in app/config.py changed as a result, that is its own deliberate commit (tune(rag): …, with the before/after percentiles in the body) — never smuggled into a migration.
• Commit any .env.example / README diff. Note in the body: date, old → new model, the --compare verdict + mean cosine before and after, chunks re-embedded, and the final three values.

Rollback

Pick by what went wrong:

1. Vectors unchanged (--stamp-only path): flip llm_embed_model back (panel or .env + restart). Done — the vectors were never touched.
2. Vectors re-embedded, app box still healthy: flip llm_embed_model back to the old model (which must still be served) and run uv run python -m scripts.reembed_embeddings --apply again — it re-embeds the corpus back into the old geometry because the selection is "stamp differs from the configured model". Minutes, same window as the forward migration.
3. Something worse (bad dump, half-run, corrupted corpus): restore the stage 3.1 dump — the exact pre-migration state, vectors and stamps included:

   podman exec <prod-db-container> pg_restore -U <user> -d <db> --clean --if-exists --no-owner \
     < /tmp/pre_embed_migration_<timestamp>.dump
   

   Then verify the fingerprint (documents / chunks counts and select count(*) from chunks where embedding_model is not null) before sending traffic back.

Troubleshooting

• EmbeddingDimensionError / dimension ≠ 768 — the dimension changed, so this skill is the wrong tool: README → "Embedding dimension mismatch" (BOR_EMBEDDING_DIM, drop + recreate chunks, re-import). Do not "make it fit" by resizing the column.
• reembed: FAILED … resume with: — the run stopped loud, on purpose. The batches already committed are done; the rest are pending. Find the cause (endpoint outage, oversized input, a pathological chunk) and re-run the printed command. Never hand-edit vectors.
• remaining= never reaches 0 — some rows never match. Look at them: select id, length(content), embedding_model from chunks where embedding_model is distinct from '<configured>' order by id limit 20; Empty content or a content the endpoint rejects is the usual answer.
• Card stuck on unknown — those are pre-phase-169 rows (stamp NULL, which is unknown, not "wrong"). The sync logs them at INFO and never warns, and the card keeps the calm status variant. They resolve the first time they are legitimately re-embedded, or now via --stamp-only (COMPATIBLE only) / --apply.
• Verdict DRIFT with mean 0.97-0.99 — usually a different server build / quantization of a nominally identical model. Re-embed anyway (it is minutes); note it in the record so nobody re-litigates it later.
• The card and the app disagree about which model is live — the overlay: a app_settings.llm_embed_model row beats .env. select key, value from app_settings where key='llm_embed_model';, then fix it in the panel (or delete from app_settings where key='llm_embed_model'; to fall back to env).
• Everything still looks fine but answers got worse — that is the failure mode this whole procedure exists for, and it hides in the top_score column, not in the logs. Compare stage 2's percentile snapshot with query_log.top_score over the last day; a collapsed spread (e.g. p10 ≈ p90) means a mixed generation or a threshold that no longer discriminates.