From a88a5a37134a2eb124ea37083857f324466bf525 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Fri, 9 Oct 2026 12:38:27 -0400 Subject: [PATCH] docs(agent): re-run the embed embedding benchmark (speed variance) Second 3-run pass against the re-pointed embed endpoint the same day. Dimension stays 768/768 and the cosine margin reproduces exactly at 0.217 across all passes, confirming both the model's determinism and that the margin drop from the 0.283 baseline is a real effect of the model swap. Throughput measured 59.1/61.4/57.4 vec/s against 35.5/35.6 earlier, a 1.7x swing on an unchanged model, which shows the speed check measures one batched request's end-to-end wall time (TLS setup, network, server queueing) rather than stable throughput - note it gates on NaN/Inf only, never on rate. LLM-Generated: true --- benchmarks/model_benchmarks.csv | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/benchmarks/model_benchmarks.csv b/benchmarks/model_benchmarks.csv index 442473c..ee81021 100644 --- a/benchmarks/model_benchmarks.csv +++ b/benchmarks/model_benchmarks.csv @@ -20,3 +20,12 @@ date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,execut 2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768 2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.6,1,0,1,margin,0.217,quality,43 2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,1.4,1,0,1,rate,35.6 vec/s,bad_vectors,0 +2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.1,1,0,1,actual_dim,768,expected_dim,768 +2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.3,1,0,1,margin,0.217,quality,43 +2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.8,1,0,1,rate,59.1 vec/s,bad_vectors,0 +2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768 +2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.4,1,0,1,margin,0.217,quality,43 +2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.8,1,0,1,rate,61.4 vec/s,bad_vectors,0 +2026-10-09,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.0,1,0,1,actual_dim,768,expected_dim,768 +2026-10-09,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.3,1,0,1,margin,0.217,quality,43 +2026-10-09,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.9,1,0,1,rate,57.4 vec/s,bad_vectors,0