Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills

Browse Source 
New files:
- scripts/model_benchmark.py — shared CSV recorder for all model tests
- scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination)
- scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed)
- .agents/skills/test-summary-model/SKILL.md — skill for testing summary models
- .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models
- benchmarks/README.md — schema documentation

Updated:
- .agents/skills/test-chat-model/SKILL.md — now also records to CSV

All three scripts write to benchmarks/model_benchmarks.csv with one row
per run per check. The CSV accumulates results across runs for comparison.
This commit is contained in:
ducoterra
2026-09-06 21:58:35 -04:00
parent 70ba8710f3
commit f221b40fce
8 changed files with 1069 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/model_benchmarks.csv
+7
View File
@@ -0,0 +1,7 @@
date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,executed,contract,wall_s,contract_denom,executed_denom,tool_turns_denom,extra1_col,extra1_val,extra2_col,extra2_val
2026-09-06,summary,lite,quality,FAIL,8,8,0,0,0,0,58,26.9,58,0,8,coherence,4.0/5,hallucinations,4/8
2026-09-06,summary,lite,quality,FAIL,8,8,0,0,0,0,53,25.4,53,0,8,coherence,4.0/5,hallucinations,5/8
2026-09-06,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.1,1,0,1,actual_dim,768,expected_dim,768
2026-09-06,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.2,1,0,1,margin,0.283,quality,56
2026-09-06,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.5,1,0,1,rate,93.8 vec/s,bad_vectors,0
2026-09-06,summary,turbo,quality,FAIL,8,8,0,0,0,0,63,25.3,63,0,8,coherence,4.0/5,hallucinations,3/8
1 date script model mode gate_status turns answered caps tool_turns emitted executed contract wall_s contract_denom executed_denom tool_turns_denom extra1_col extra1_val extra2_col extra2_val
2 2026-09-06 summary lite quality FAIL 8 8 0 0 0 0 58 26.9 58 0 8 coherence 4.0/5 hallucinations 4/8
3 2026-09-06 summary lite quality FAIL 8 8 0 0 0 0 53 25.4 53 0 8 coherence 4.0/5 hallucinations 5/8
4 2026-09-06 embed embed dimension PASS 1 1 0 0 0 0 1 0.1 1 0 1 actual_dim 768 expected_dim 768
5 2026-09-06 embed embed cosine PASS 1 1 0 0 0 0 1 0.2 1 0 1 margin 0.283 quality 56
6 2026-09-06 embed embed speed PASS 1 1 0 0 0 0 1 0.5 1 0 1 rate 93.8 vec/s bad_vectors 0
7 2026-09-06 summary turbo quality FAIL 8 8 0 0 0 0 63 25.3 63 0 8 coherence 4.0/5 hallucinations 3/8