ducoterra
f221b40fce
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
8 lines
760 B
CSV
8 lines
760 B
CSV
date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,executed,contract,wall_s,contract_denom,executed_denom,tool_turns_denom,extra1_col,extra1_val,extra2_col,extra2_val
|
|
2026-09-06,summary,lite,quality,FAIL,8,8,0,0,0,0,58,26.9,58,0,8,coherence,4.0/5,hallucinations,4/8
|
|
2026-09-06,summary,lite,quality,FAIL,8,8,0,0,0,0,53,25.4,53,0,8,coherence,4.0/5,hallucinations,5/8
|
|
2026-09-06,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.1,1,0,1,actual_dim,768,expected_dim,768
|
|
2026-09-06,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.2,1,0,1,margin,0.283,quality,56
|
|
2026-09-06,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.5,1,0,1,rate,93.8 vec/s,bad_vectors,0
|
|
2026-09-06,summary,turbo,quality,FAIL,8,8,0,0,0,0,63,25.3,63,0,8,coherence,4.0/5,hallucinations,3/8
|