ducoterra
f221b40fce
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
760 B
760 B
| 1 | date | script | model | mode | gate_status | turns | answered | caps | tool_turns | emitted | executed | contract | wall_s | contract_denom | executed_denom | tool_turns_denom | extra1_col | extra1_val | extra2_col | extra2_val |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 2026-09-06 | summary | lite | quality | FAIL | 8 | 8 | 0 | 0 | 0 | 0 | 58 | 26.9 | 58 | 0 | 8 | coherence | 4.0/5 | hallucinations | 4/8 |
| 3 | 2026-09-06 | summary | lite | quality | FAIL | 8 | 8 | 0 | 0 | 0 | 0 | 53 | 25.4 | 53 | 0 | 8 | coherence | 4.0/5 | hallucinations | 5/8 |
| 4 | 2026-09-06 | embed | embed | dimension | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.1 | 1 | 0 | 1 | actual_dim | 768 | expected_dim | 768 |
| 5 | 2026-09-06 | embed | embed | cosine | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.2 | 1 | 0 | 1 | margin | 0.283 | quality | 56 |
| 6 | 2026-09-06 | embed | embed | speed | PASS | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0.5 | 1 | 0 | 1 | rate | 93.8 vec/s | bad_vectors | 0 |
| 7 | 2026-09-06 | summary | turbo | quality | FAIL | 8 | 8 | 0 | 0 | 0 | 0 | 63 | 25.3 | 63 | 0 | 8 | coherence | 4.0/5 | hallucinations | 3/8 |