Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agent): re-run the embed embedding benchmark (speed variance)

Browse Source 
Second 3-run pass against the re-pointed embed endpoint the same day.
Dimension stays 768/768 and the cosine margin reproduces exactly at 0.217
across all passes, confirming both the model's determinism and that the
margin drop from the 0.283 baseline is a real effect of the model swap.
Throughput measured 59.1/61.4/57.4 vec/s against 35.5/35.6 earlier, a 1.7x
swing on an unchanged model, which shows the speed check measures one batched
request's end-to-end wall time (TLS setup, network, server queueing) rather
than stable throughput - note it gates on NaN/Inf only, never on rate.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-10-09 12:38:27 -04:00
parent 05f7d0645e
commit a88a5a3713
1 changed files with 9 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/model_benchmarks.csv
+9
View File
@@ -20,3 +20,12 @@ date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,execut
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.6 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.1,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.3,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.8,1,0,1,rate,59.1 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.4,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.8,1,0,1,rate,61.4 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.3,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.9,1,0,1,rate,57.4 vec/s,bad_vectors,0
1 date script model mode gate_status turns answered caps tool_turns emitted executed contract wall_s contract_denom executed_denom tool_turns_denom extra1_col extra1_val extra2_col extra2_val
20 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
21 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.6 1 0 1 margin 0.217 quality 43
22 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 1.4 1 0 1 rate 35.6 vec/s bad_vectors 0
23 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.1 1 0 1 actual_dim 768 expected_dim 768
24 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.3 1 0 1 margin 0.217 quality 43
25 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.8 1 0 1 rate 59.1 vec/s bad_vectors 0
26 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
27 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.4 1 0 1 margin 0.217 quality 43
28 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.8 1 0 1 rate 61.4 vec/s bad_vectors 0
29 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
30 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.3 1 0 1 margin 0.217 quality 43
31 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.9 1 0 1 rate 57.4 vec/s bad_vectors 0