ducoterra
8fac4e64d4
The aipi `embed` endpoint was re-pointed to a new backing model and was re-benchmarked over 3 runs. All three gates pass: output stays 768-dim (matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf vectors, and throughput is 35.6 vec/s on the 50-vector probe. Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin tightens and throughput drops ~2.6x, both consistent with a model swap behind the same alias. Vectors are unit-normalized and deterministic. Note: same dimension on a different model means any KB indexed under the old backing model holds stale vectors and needs a re-embed; the local dev DB is empty, so the phase 119/162 retrieval battery was not runnable here. LLM-Generated: true
Model Benchmark Results
CSV recordings from all model tests (chat, summary, embedding).
Files
model_benchmarks.csv— all benchmark results, one row per run per check
Schema
| Column | Description |
|---|---|
date |
Test date (YYYY-MM-DD) |
script |
chat |
model |
Model name (e.g. lite, turbo, embed) |
mode |
fixture / derived / quality / dimension / cosine / speed |
gate_status |
PASS / FAIL |
turns |
Number of questions/turns |
answered |
Turns that produced an answer |
caps |
Turns that hit the round cap |
tool_turns |
Turns emitting ≥1 tool call (chat only) |
emitted / executed |
Tool-call counts (chat only) |
contract |
Numerator for accuracy (chat: well-formed calls; summary: quality 0–100; embed: 1=pass) |
wall_s |
Total wall seconds |
contract_denom / executed_denom / tool_turns_denom |
Denominators for percentages |
extra1_col / extra1_val |
Script-specific metric 1 |
extra2_col / extra2_val |
Script-specific metric 2 |
Scripts
scripts/agent_realmodel_check.py— chat model tool-calling batteryscripts/test_summary_model.py— summary model quality benchmarkscripts/test_embed_model.py— embedding model dimension/cosine/speed benchmarkscripts/model_benchmark.py— shared CSV recorder (imported by all scripts)