feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
This commit is contained in:
@@ -96,18 +96,34 @@ prints the `MISS` lines naming the condition. Interpret:
|
||||
- **caps > 0** — the phase-72 incident signature; a real regression,
|
||||
say so explicitly.
|
||||
|
||||
### 5. Record + commit
|
||||
### 5. Record to CSV + commit
|
||||
|
||||
Append the model's results to the model-comparison subsection of
|
||||
`TOOL_CALLING_TESTING.md` §3, keeping the `gate:` lines **byte-exact
|
||||
verbatim**, plus 2–4 sentences of interpretation against the reference
|
||||
rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s
|
||||
per full loop).
|
||||
**a. CSV record** — append to `benchmarks/model_benchmarks.csv` via
|
||||
`scripts/model_benchmark.bench_write`. For each of the 3 runs, call:
|
||||
|
||||
```python
|
||||
from scripts.model_benchmark import bench_write
|
||||
|
||||
bench_write(
|
||||
script="chat", model="<model>", mode="fixture",
|
||||
gate_status="PASS", turns=10, answered=10, caps=0,
|
||||
tool_turns=<N>, emitted=<E>, executed=<X>,
|
||||
contract=<C>, wall_s=<wall>,
|
||||
contract_denom=<C_denom>, executed_denom=<E_denom>,
|
||||
)
|
||||
```
|
||||
|
||||
For the derived run, use `mode="derived"`.
|
||||
|
||||
**b. Append to TOOL_CALLING_TESTING.md** — keep the `gate:` lines
|
||||
**byte-exact verbatim**, plus 2–4 sentences of interpretation against
|
||||
the reference rates above and the wall-time baseline (lite ~40–55 s,
|
||||
turbo ~97–135 s per full loop).
|
||||
|
||||
Commit — docs only, house style:
|
||||
|
||||
```bash
|
||||
git add TOOL_CALLING_TESTING.md
|
||||
git add TOOL_CALLING_TESTING.md benchmarks/model_benchmarks.csv
|
||||
git commit --no-gpg-sign \
|
||||
-m "docs(agent): record the <model> comparison on the controlled fixture battery" \
|
||||
-m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"
|
||||
|
||||
Reference in New Issue
Block a user