An embedding-model swap that keeps the dimension is invisible on this
app: nothing records which model made a vector, EmbeddingDimensionError
cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the
documents.file_hash fast path returns with zero embeds - so questions
embed in one geometry and chunks score in another with no error anywhere.
The owner's upgrade (embedding gemma -> embedding gemma 2) is exactly
that case.
Phase 169, seven tasks: chunks.embedding_model (reversible migration
0029, nullable, never backfilled - NULL means unknown by design), one
write helper so every vector write stamps its generation, a
generation_report() that turns the column into counts plus a stale
verdict logged after every sync, scripts/reembed_embeddings.py with
--compare (cosine agreement against the stored vectors, so 'are they
compatible?' is measured instead of guessed) / --stamp-only / --apply
(per-batch commits, resumable, never a NULL embedding, never a
sources_meta bump), an admin-only GET /api/admin/embeddings, a read-only
card in the phase-168 Settings view, and the dedicated Playwright file
plus the README runbook.
Owner-locked during authoring: detection only - retrieval never filters
by generation (a WHERE embedding_model predicate would silently deflect
every turn mid-backfill), comparability bands COMPATIBLE >= 0.995 for
the float4 round trip, a re-embed is not a KB change so saved chats stay
fresh, and no new Settings field so the phase-168 catalog pin is
untouched.
.agents/skills/migrate-embed-model is the operator twin: probe and
benchmark the new model, measure comparability as its own process
without disturbing the running app, back up and migrate production
(idle check, pg_dump off-box, alembic, flip then re-embed with the same
env), then re-tune relevance_threshold and the two floors against the
retrieval battery - after the corpus is on one generation, and with the
phase-168 warning that an app_settings row beats .env - and smoke-test
the result.
LLM-Generated: true
Single consolidated commit for four completed, validated phases (77, 78,
79, 80). The pipeline run left all work uncommitted because the harness
commits only with PHASE_COMMIT=1 while child executors are forbidden from
committing; the phases themselves all passed validation and moved to
.agents/phases/complete/.
Phase 77 — navbar view refresh
- router.js dispatches bor:view-refresh on re-show / active re-click /
popstate (gated on wasMounted; first show and boot exempt)
- History / RAG / Sources / Tuning re-fetch on refresh (admin branch);
Chat deliberately excluded (stream survival)
- History "Refresh" button (admin-only, in-flight disable + status line)
- New story suite tests/e2e/test_navbar_refresh.py (7 tests)
Phase 78 — static background
- Removed the animated glow layers; static 44px grid over the flat --bg
canvas; default and reduced-motion renders byte-identical
- Updated background/theme E2E suites; removed bg-glow test pins
Phase 79 — API tokens
- api_tokens model + migration 0012; hash-only token service
- Admin tokens API + Tokens admin view; POST /api/token-auth;
live-revoking require_user on chat / suggestions / document content
- Frontend token gate with localStorage cache; anonymous E2E suites
migrated to token login
- New story suite tests/e2e/test_api_tokens.py (9 tests)
Phase 80 — history suggestion chips
- last_questions() endpoint with SEED fallback; startNewChat() refetch
- Seed-semantics docs (config.py, .env.example, README)
- Integration state matrix + E2E suite rewritten to the 4 chip states
Also included: phase-76 report artifacts and the repo restore-test-db
skill (previously untracked), scripts/* ruff fixes from phase 77.
Final gate state (phase 80 final pass, covers everything above):
- uv run pytest --cov=app → 1637 passed, 0 failed, app/ coverage 99%
- uv run ruff check . && uv run pyright → clean, 0 errors
- Per-phase story E2E suites green in isolation
New files:
- scripts/model_benchmark.py — shared CSV recorder for all model tests
- scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination)
- scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed)
- .agents/skills/test-summary-model/SKILL.md — skill for testing summary models
- .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models
- benchmarks/README.md — schema documentation
Updated:
- .agents/skills/test-chat-model/SKILL.md — now also records to CSV
All three scripts write to benchmarks/model_benchmarks.csv with one row
per run per check. The CSV accumulates results across runs for comparison.
Codifies the 2026-09-05 turbo comparison workflow as a project skill under
.agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate
(twice, for variance) + the locked derived gate with per-turn wall timing,
interpret the two metrics against the reference model rates (re-read habit:
lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real
regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and
commit the doc. Rules baked in: never touch the battery/thresholds/fixtures,
never edit app code, never commit .env.
ducoterra