feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
This commit is contained in:
@@ -0,0 +1,34 @@
|
||||
# Model Benchmark Results
|
||||
|
||||
CSV recordings from all model tests (chat, summary, embedding).
|
||||
|
||||
## Files
|
||||
|
||||
- `model_benchmarks.csv` — all benchmark results, one row per run per check
|
||||
|
||||
## Schema
|
||||
|
||||
| Column | Description |
|
||||
|---|---|
|
||||
| `date` | Test date (YYYY-MM-DD) |
|
||||
| `script` | `chat` | `summary` | `embed` |
|
||||
| `model` | Model name (e.g. `lite`, `turbo`, `embed`) |
|
||||
| `mode` | `fixture` / `derived` / `quality` / `dimension` / `cosine` / `speed` |
|
||||
| `gate_status` | `PASS` / `FAIL` |
|
||||
| `turns` | Number of questions/turns |
|
||||
| `answered` | Turns that produced an answer |
|
||||
| `caps` | Turns that hit the round cap |
|
||||
| `tool_turns` | Turns emitting ≥1 tool call (chat only) |
|
||||
| `emitted` / `executed` | Tool-call counts (chat only) |
|
||||
| `contract` | Numerator for accuracy (chat: well-formed calls; summary: quality 0–100; embed: 1=pass) |
|
||||
| `wall_s` | Total wall seconds |
|
||||
| `contract_denom` / `executed_denom` / `tool_turns_denom` | Denominators for percentages |
|
||||
| `extra1_col` / `extra1_val` | Script-specific metric 1 |
|
||||
| `extra2_col` / `extra2_val` | Script-specific metric 2 |
|
||||
|
||||
## Scripts
|
||||
|
||||
- `scripts/agent_realmodel_check.py` — chat model tool-calling battery
|
||||
- `scripts/test_summary_model.py` — summary model quality benchmark
|
||||
- `scripts/test_embed_model.py` — embedding model dimension/cosine/speed benchmark
|
||||
- `scripts/model_benchmark.py` — shared CSV recorder (imported by all scripts)
|
||||
@@ -0,0 +1,7 @@
|
||||
date,script,model,mode,gate_status,turns,answered,caps,tool_turns,emitted,executed,contract,wall_s,contract_denom,executed_denom,tool_turns_denom,extra1_col,extra1_val,extra2_col,extra2_val
|
||||
2026-09-06,summary,lite,quality,FAIL,8,8,0,0,0,0,58,26.9,58,0,8,coherence,4.0/5,hallucinations,4/8
|
||||
2026-09-06,summary,lite,quality,FAIL,8,8,0,0,0,0,53,25.4,53,0,8,coherence,4.0/5,hallucinations,5/8
|
||||
2026-09-06,embed,embed,dimension,PASS,1,1,0,0,0,0,1,0.1,1,0,1,actual_dim,768,expected_dim,768
|
||||
2026-09-06,embed,embed,cosine,PASS,1,1,0,0,0,0,1,0.2,1,0,1,margin,0.283,quality,56
|
||||
2026-09-06,embed,embed,speed,PASS,1,1,0,0,0,0,1,0.5,1,0,1,rate,93.8 vec/s,bad_vectors,0
|
||||
2026-09-06,summary,turbo,quality,FAIL,8,8,0,0,0,0,63,25.3,63,0,8,coherence,4.0/5,hallucinations,3/8
|
||||
|
Reference in New Issue
Block a user