brain-of-reese/benchmarks/model_benchmarks.csv at a88a5a37134a2eb124ea37083857f324466bf525

Files

T

ducoterra a88a5a3713 docs(agent): re-run the embed embedding benchmark (speed variance)

Second 3-run pass against the re-pointed embed endpoint the same day.
Dimension stays 768/768 and the cosine margin reproduces exactly at 0.217
across all passes, confirming both the model's determinism and that the
margin drop from the 0.283 baseline is a real effect of the model swap.
Throughput measured 59.1/61.4/57.4 vec/s against 35.5/35.6 earlier, a 1.7x
swing on an unchanged model, which shows the speed check measures one batched
request's end-to-end wall time (TLS setup, network, server queueing) rather
than stable throughput - note it gates on NaN/Inf only, never on rate.

LLM-Generated: true

2026-10-09 12:38:27 -04:00

2.7 KiB

Raw Blame History

1	date	script	model	mode	gate_status	turns	answered	tool_turns	emitted	executed	contract	wall_s	contract_denom	executed_denom	tool_turns_denom	extra1_col	extra1_val	extra2_col	extra2_val
2	2026-09-06	summary	lite	quality	FAIL	8	8	0	0	0	58	26.9	58	0	8	coherence	4.0/5	hallucinations	4/8
3	2026-09-06	summary	lite	quality	FAIL	8	8	0	0	0	53	25.4	53	0	8	coherence	4.0/5	hallucinations	5/8
4	2026-09-06	embed	embed	dimension	PASS	1	1	0	0	0	1	0.1	1	0	1	actual_dim	768	expected_dim	768
5	2026-09-06	embed	embed	cosine	PASS	1	1	0	0	0	1	0.2	1	0	1	margin	0.283	quality	56
6	2026-09-06	embed	embed	speed	PASS	1	1	0	0	0	1	0.5	1	0	1	rate	93.8 vec/s	bad_vectors	0
7	2026-09-06	summary	turbo	quality	FAIL	8	8	0	0	0	63	25.3	63	0	8	coherence	4.0/5	hallucinations	3/8
8	2026-09-06	chat	lite	fixture	PASS	10	10	9	9	9	11	40.4	12	12	10
9	2026-09-06	chat	lite	fixture	PASS	10	10	10	9	9	13	40.5	14	14	10
10	2026-09-06	chat	lite	derived	FAIL	10	10	10	5	5	10	38.3	14	14	10
11	2026-09-06	chat	turbo	fixture	PASS	10	10	7	9	9	9	113.7	9	9	10
12	2026-09-06	chat	turbo	fixture	PASS	10	10	7	7	7	7	112.8	7	7	10
13	2026-09-06	chat	turbo	derived	FAIL	10	10	5	5	5	5	97.2	5	5	10
14	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.2	1	0	1	actual_dim	768	expected_dim	768
15	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.6	1	0	1	margin	0.217	quality	43
16	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	1.4	1	0	1	rate	35.6 vec/s	bad_vectors	0
17	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
18	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.6	1	0	1	margin	0.217	quality	43
19	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	1.4	1	0	1	rate	35.5 vec/s	bad_vectors	0
20	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
21	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.6	1	0	1	margin	0.217	quality	43
22	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	1.4	1	0	1	rate	35.6 vec/s	bad_vectors	0
23	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.1	1	0	1	actual_dim	768	expected_dim	768
24	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.3	1	0	1	margin	0.217	quality	43
25	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.8	1	0	1	rate	59.1 vec/s	bad_vectors	0
26	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
27	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.4	1	0	1	margin	0.217	quality	43
28	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.8	1	0	1	rate	61.4 vec/s	bad_vectors	0
29	2026-10-09	embed	embed	dimension	PASS	1	1	0	0	0	1	0.0	1	0	1	actual_dim	768	expected_dim	768
30	2026-10-09	embed	embed	cosine	PASS	1	1	0	0	0	1	0.3	1	0	1	margin	0.217	quality	43
31	2026-10-09	embed	embed	speed	PASS	1	1	0	0	0	1	0.9	1	0	1	rate	57.4 vec/s	bad_vectors	0

2.7 KiB Raw Blame History

2.7 KiB

Raw Blame History