Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agent): record the re-pointed embed embedding benchmark

Browse Source 
The aipi `embed` endpoint was re-pointed to a new backing model and was
re-benchmarked over 3 runs. All three gates pass: output stays 768-dim
(matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the
similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf
vectors, and throughput is 35.6 vec/s on the 50-vector probe.

Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin
tightens and throughput drops ~2.6x, both consistent with a model swap
behind the same alias. Vectors are unit-normalized and deterministic.
Note: same dimension on a different model means any KB indexed under the
old backing model holds stale vectors and needs a re-embed; the local dev
DB is empty, so the phase 119/162 retrieval battery was not runnable here.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-10-09 12:23:55 -04:00
parent a12fdcdf0a
commit 8fac4e64d4
1 changed files with 9 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/model_benchmarks.csv
+9
View File
@@ -11,3 +11,12 @@ date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,execut
2026-09-06,chat,turbo,fixture,PASS,10,10,0,7,9,9,9,113.7,9,9,10,,,,
2026-09-06,chat,turbo,fixture,PASS,10,10,0,7,7,7,7,112.8,7,7,10,,,,
2026-09-06,chat,turbo,derived,FAIL,10,10,0,5,5,5,5,97.2,5,5,10,,,,
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.2,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.6 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.5 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.6 vec/s,bad_vectors,0
1 date script model mode gate_status turns answered caps tool_turns emitted executed contract wall_s contract_denom executed_denom tool_turns_denom extra1_col extra1_val extra2_col extra2_val
11 2026-09-06 chat turbo fixture PASS 10 10 0 7 9 9 9 113.7 9 9 10
12 2026-09-06 chat turbo fixture PASS 10 10 0 7 7 7 7 112.8 7 7 10
13 2026-09-06 chat turbo derived FAIL 10 10 0 5 5 5 5 97.2 5 5 10
14 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.2 1 0 1 actual_dim 768 expected_dim 768
15 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.6 1 0 1 margin 0.217 quality 43
16 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 1.4 1 0 1 rate 35.6 vec/s bad_vectors 0
17 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
18 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.6 1 0 1 margin 0.217 quality 43
19 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 1.4 1 0 1 rate 35.5 vec/s bad_vectors 0
20 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
21 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.6 1 0 1 margin 0.217 quality 43
22 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 1.4 1 0 1 rate 35.6 vec/s bad_vectors 0