Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agent): record the reverted embed embedding benchmark

Browse Source 
The embed alias was reverted to the previous backing model and re-benchmarked
over 3 runs. All gates pass and the numbers reproduce the original baseline
exactly: 768 dims, cosine margin 0.283 (quality 56) versus the 0.217 measured
on the swapped-in model, with throughput back to 109.7-123.3 vec/s.

The margin returning to the identical 0.283 confirms the locked semantic-pair
corpus is a reliable fingerprint for this endpoint, and a raw-vector probe
shows the two models are genuinely different spaces despite both being
unit-normalized 768-dim (max component diff 0.035 on the same input). Any
content embedded while the swapped-in model was live is therefore in an
incompatible space and needs re-embedding.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-10-09 12:41:38 -04:00
parent a88a5a3713
commit 542e568d25
1 changed files with 9 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/model_benchmarks.csv
+9
View File
@@ -29,3 +29,12 @@ date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,execut
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.3,1,0,1,margin,0.217,quality,43
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.9,1,0,1,rate,57.4 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.2,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.3,1,0,1,margin,0.283,quality,56
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.5,1,0,1,rate,109.7 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.1,1,0,1,margin,0.283,quality,56
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.4,1,0,1,rate,123.3 vec/s,bad_vectors,0
2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768
2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.2,1,0,1,margin,0.283,quality,56
2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.4,1,0,1,rate,119.0 vec/s,bad_vectors,0
1 date script model mode gate_status turns answered caps tool_turns emitted executed contract wall_s contract_denom executed_denom tool_turns_denom extra1_col extra1_val extra2_col extra2_val
29 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
30 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.3 1 0 1 margin 0.217 quality 43
31 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.9 1 0 1 rate 57.4 vec/s bad_vectors 0
32 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.2 1 0 1 actual_dim 768 expected_dim 768
33 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.3 1 0 1 margin 0.283 quality 56
34 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.5 1 0 1 rate 109.7 vec/s bad_vectors 0
35 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
36 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.1 1 0 1 margin 0.283 quality 56
37 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.4 1 0 1 rate 123.3 vec/s bad_vectors 0
38 2026-10-09 embed embed dimension PASS 1 1 0 0 0 0 1 0.0 1 0 1 actual_dim 768 expected_dim 768
39 2026-10-09 embed embed cosine PASS 1 1 0 0 0 0 1 0.2 1 0 1 margin 0.283 quality 56
40 2026-10-09 embed embed speed PASS 1 1 0 0 0 0 1 0.4 1 0 1 rate 119.0 vec/s bad_vectors 0