The aipi `embed` endpoint was re-pointed to a new backing model and was
re-benchmarked over 3 runs. All three gates pass: output stays 768-dim
(matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the
similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf
vectors, and throughput is 35.6 vec/s on the 50-vector probe.
Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin
tightens and throughput drops ~2.6x, both consistent with a model swap
behind the same alias. Vectors are unit-normalized and deterministic.
Note: same dimension on a different model means any KB indexed under the
old backing model holds stale vectors and needs a re-embed; the local dev
DB is empty, so the phase 119/162 retrieval battery was not runnable here.
LLM-Generated: true
New files:
- scripts/model_benchmark.py — shared CSV recorder for all model tests
- scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination)
- scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed)
- .agents/skills/test-summary-model/SKILL.md — skill for testing summary models
- .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models
- benchmarks/README.md — schema documentation
Updated:
- .agents/skills/test-chat-model/SKILL.md — now also records to CSV
All three scripts write to benchmarks/model_benchmarks.csv with one row
per run per check. The CSV accumulates results across runs for comparison.
ducoterra