From 8fac4e64d4385f656b8c4411159c25ce9e29801c Mon Sep 17 00:00:00 2001 From: ducoterra Date: Fri, 9 Oct 2026 12:23:55 -0400 Subject: [PATCH] docs(agent): record the re-pointed embed embedding benchmark The aipi `embed` endpoint was re-pointed to a new backing model and was re-benchmarked over 3 runs. All three gates pass: output stays 768-dim (matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf vectors, and throughput is 35.6 vec/s on the 50-vector probe. Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin tightens and throughput drops ~2.6x, both consistent with a model swap behind the same alias. Vectors are unit-normalized and deterministic. Note: same dimension on a different model means any KB indexed under the old backing model holds stale vectors and needs a re-embed; the local dev DB is empty, so the phase 119/162 retrieval battery was not runnable here. LLM-Generated: true --- benchmarks/model_benchmarks.csv | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/benchmarks/model_benchmarks.csv b/benchmarks/model_benchmarks.csv index 3bdee19..442473c 100644 --- a/benchmarks/model_benchmarks.csv +++ b/benchmarks/model_benchmarks.csv @@ -11,3 +11,12 @@ date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,execut 2026-09-06,chat,turbo,fixture,PASS,10,10,0,7,9,9,9,113.7,9,9,10,,,, 2026-09-06,chat,turbo,fixture,PASS,10,10,0,7,7,7,7,112.8,7,7,10,,,, 2026-09-06,chat,turbo,derived,FAIL,10,10,0,5,5,5,5,97.2,5,5,10,,,, +2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.2,1,0,1,actual_dim,768,expected_dim,768 +2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43 +2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.6 vec/s,bad_vectors,0 +2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768 +2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43 +2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.5 vec/s,bad_vectors,0 +2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768 +2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43 +2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.6 vec/s,bad_vectors,0