brain-of-reese/benchmarks/model_benchmarks.csv at 6880bd08660560ab94b6f8fef10b7233b088b3ec

Files

T

ducoterra 542e568d25 docs(agent): record the reverted embed embedding benchmark

The embed alias was reverted to the previous backing model and re-benchmarked
over 3 runs. All gates pass and the numbers reproduce the original baseline
exactly: 768 dims, cosine margin 0.283 (quality 56) versus the 0.217 measured
on the swapped-in model, with throughput back to 109.7-123.3 vec/s.

The margin returning to the identical 0.283 confirms the locked semantic-pair
corpus is a reliable fingerprint for this endpoint, and a raw-vector probe
shows the two models are genuinely different spaces despite both being
unit-normalized 768-dim (max component diff 0.035 on the same input). Any
content embedded while the swapped-in model was live is therefore in an
incompatible space and needs re-embedding.

LLM-Generated: true

2026-10-09 12:41:38 -04:00

3.5 KiB

Raw Blame History

1	date	script	model	mode	gate_status	turns	answered	tool_turns	emitted	executed	contract	wall_s	contract_denom	executed_denom	tool_turns_denom	extra1_col	extra1_val	extra2_col	extra2_val
2	2026-09-06	summary	lite	quality	FAIL	8	8	0	0	0	58	26.9	58	0	8	coherence	4.0/5	hallucinations	4/8
3	2026-09-06	summary	lite	quality	FAIL	8	8	0	0	0	53	25.4	53	0	8	coherence	4.0/5	hallucinations	5/8
4	2026-09-06	embed	embed	dimension	PASS	1	1	0	0	0	1	0.1	1	0	1	actual_dim	768	expected_dim	768
5	2026-09-06	embed	embed	cosine	PASS	1	1	0	0	0	1	0.2	1	0	1	margin	0.283	quality	56
6	2026-09-06	embed	embed	speed	PASS	1	1	0	0	0	1	0.5	1	0	1	rate	93.8 vec/s	bad_vectors	0
7	2026-09-06	summary	turbo	quality	FAIL	8	8	0	0	0	63	25.3	63	0	8	coherence	4.0/5	hallucinations	3/8
8	2026-09-06	chat	lite	fixture	PASS	10	10	9	9	9	11	40.4	12	12	10
9	2026-09-06	chat	lite	fixture	PASS	10	10	10	9	9	13	40.5	14	14	10
10	2026-09-06	chat	lite	derived	FAIL	10	10	10	5	5	10	38.3	14	14	10
11	2026-09-06	chat	turbo	fixture	PASS	10	10	7	9	9	9	113.7	9	9	10
12	2026-09-06	chat	turbo	fixture	PASS	10	10	7	7	7	7	112.8	7	7	10
13	2026-09-06	chat	turbo	derived	FAIL	10	10	5	5	5	5	97.2	5	5	10
14	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.2	1	0	1	actual_dim	768	expected_dim	768
15	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.6	1	0	1	margin	0.217	quality	43
16	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	1.4	1	0	1	rate	35.6 vec/s	bad_vectors	0
17	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
18	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.6	1	0	1	margin	0.217	quality	43
19	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	1.4	1	0	1	rate	35.5 vec/s	bad_vectors	0
20	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
21	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.6	1	0	1	margin	0.217	quality	43
22	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	1.4	1	0	1	rate	35.6 vec/s	bad_vectors	0
23	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.1	1	0	1	actual_dim	768	expected_dim	768
24	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.3	1	0	1	margin	0.217	quality	43
25	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.8	1	0	1	rate	59.1 vec/s	bad_vectors	0
26	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
27	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.4	1	0	1	margin	0.217	quality	43
28	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.8	1	0	1	rate	61.4 vec/s	bad_vectors	0
29	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
30	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.3	1	0	1	margin	0.217	quality	43
31	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.9	1	0	1	rate	57.4 vec/s	bad_vectors	0
32	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.2	1	0	1	actual_dim	768	expected_dim	768
33	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.3	1	0	1	margin	0.283	quality	56
34	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.5	1	0	1	rate	109.7 vec/s	bad_vectors	0
35	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
36	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.1	1	0	1	margin	0.283	quality	56
37	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.4	1	0	1	rate	123.3 vec/s	bad_vectors	0
38	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
39	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.2	1	0	1	margin	0.283	quality	56
40	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.4	1	0	1	rate	119.0 vec/s	bad_vectors	0

3.5 KiB Raw Blame History

3.5 KiB

Raw Blame History