feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
This commit is contained in:
@@ -96,18 +96,34 @@ prints the `MISS` lines naming the condition. Interpret:
|
||||
- **caps > 0** — the phase-72 incident signature; a real regression,
|
||||
say so explicitly.
|
||||
|
||||
### 5. Record + commit
|
||||
### 5. Record to CSV + commit
|
||||
|
||||
Append the model's results to the model-comparison subsection of
|
||||
`TOOL_CALLING_TESTING.md` §3, keeping the `gate:` lines **byte-exact
|
||||
verbatim**, plus 2–4 sentences of interpretation against the reference
|
||||
rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s
|
||||
per full loop).
|
||||
**a. CSV record** — append to `benchmarks/model_benchmarks.csv` via
|
||||
`scripts/model_benchmark.bench_write`. For each of the 3 runs, call:
|
||||
|
||||
```python
|
||||
from scripts.model_benchmark import bench_write
|
||||
|
||||
bench_write(
|
||||
script="chat", model="<model>", mode="fixture",
|
||||
gate_status="PASS", turns=10, answered=10, caps=0,
|
||||
tool_turns=<N>, emitted=<E>, executed=<X>,
|
||||
contract=<C>, wall_s=<wall>,
|
||||
contract_denom=<C_denom>, executed_denom=<E_denom>,
|
||||
)
|
||||
```
|
||||
|
||||
For the derived run, use `mode="derived"`.
|
||||
|
||||
**b. Append to TOOL_CALLING_TESTING.md** — keep the `gate:` lines
|
||||
**byte-exact verbatim**, plus 2–4 sentences of interpretation against
|
||||
the reference rates above and the wall-time baseline (lite ~40–55 s,
|
||||
turbo ~97–135 s per full loop).
|
||||
|
||||
Commit — docs only, house style:
|
||||
|
||||
```bash
|
||||
git add TOOL_CALLING_TESTING.md
|
||||
git add TOOL_CALLING_TESTING.md benchmarks/model_benchmarks.csv
|
||||
git commit --no-gpg-sign \
|
||||
-m "docs(agent): record the <model> comparison on the controlled fixture battery" \
|
||||
-m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
name: test-embed-model
|
||||
description: Tests an embedding model (BOR_LLM_EMBED_MODEL in .env) across three dimensions — dimension consistency, cosine accuracy on semantic pairs, and embedding speed — then records results in benchmarks/model_benchmarks.csv. Use when the user asks to test or benchmark an embedding model, check embedding quality, or compare embedding models (e.g. "test the embed model", "check embedding dimension", "benchmark embed speed").
|
||||
---
|
||||
|
||||
# Test an Embedding Model (dimension + cosine + speed)
|
||||
|
||||
Tests the configured `BOR_LLM_EMBED_MODEL` (default `embed`) against
|
||||
three independent checks:
|
||||
|
||||
1. **Dimension check** — output vector length matches `BOR_EMBEDDING_DIM`
|
||||
2. **Cosine accuracy** — semantically similar text pairs have higher
|
||||
cosine similarity than dissimilar pairs (5 pairs tested)
|
||||
3. **Speed** — vectors produced per second (50 vectors)
|
||||
|
||||
## Rules (non-negotiable)
|
||||
|
||||
- **Do not touch** the semantic pair texts — they are the controlled
|
||||
benchmark corpus. Changing them is a methodology change: flag it.
|
||||
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
|
||||
not the app. If the model exposes an app defect, report it — don't fix it.
|
||||
- `.env` is gitignored and is **not committed** — the model switch stays
|
||||
a live dev setting, and the summary must say which model `.env` is left
|
||||
on (default: the tested model).
|
||||
|
||||
## Procedure
|
||||
|
||||
All commands run from the repo root with `uv run`.
|
||||
|
||||
### 1. Preconditions
|
||||
|
||||
```bash
|
||||
podman compose up -d db
|
||||
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL|EMBEDDING_DIM)" .env
|
||||
```
|
||||
|
||||
### 2. Switch the model (optional)
|
||||
|
||||
Edit only `BOR_LLM_EMBED_MODEL` in `.env` (leave chat and summary alone):
|
||||
|
||||
```
|
||||
BOR_LLM_EMBED_MODEL=<model>
|
||||
```
|
||||
|
||||
### 3. Run the benchmark
|
||||
|
||||
```bash
|
||||
# Single run (all three checks)
|
||||
uv run python -m scripts.test_embed_model
|
||||
|
||||
# Specify a model explicitly
|
||||
uv run python -m scripts.test_embed_model --model embed-v2
|
||||
|
||||
# Multiple runs for variance
|
||||
uv run python -m scripts.test_embed_model --runs 3
|
||||
```
|
||||
|
||||
Wall time: ~0.5–2 s per run (fast — the embedding endpoint is lightweight).
|
||||
|
||||
### 4. Read the verdicts
|
||||
|
||||
Each check prints `✓` (PASS) or `✗` (FAIL):
|
||||
|
||||
- **dimension**: `actual_dim == expected_dim` (from `BOR_EMBEDDING_DIM`, default 768)
|
||||
- **cosine**: similar-pair margin ≥ 0.05 (similar > dissimilar by at least 0.05 cosine)
|
||||
- **speed**: no NaN/Inf vectors in output; rate reported as vec/s
|
||||
|
||||
Gate: PASS if all three checks pass.
|
||||
|
||||
### 5. Record + commit
|
||||
|
||||
Results are automatically appended to `benchmarks/model_benchmarks.csv`
|
||||
(one row per check per run). Commit:
|
||||
|
||||
```bash
|
||||
git add benchmarks/model_benchmarks.csv
|
||||
git commit --no-gpg-sign \
|
||||
-m "docs(agent): record the <model> embedding benchmark" \
|
||||
-m "<one-paragraph body: dimension match, cosine margin, speed, any failures>"
|
||||
```
|
||||
|
||||
### 6. Summary
|
||||
|
||||
Table of dimension / cosine margin / speed per run, the one-line
|
||||
conclusion, and a note that `.env` is now on `<model>`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **dimension mismatch** — the model's output vectors are a different
|
||||
size than expected. This is a configuration error: check the model's
|
||||
docs for its embedding dimension and update `BOR_EMBEDDING_DIM`.
|
||||
- **cosine margin < 0.05** — the embeddings don't separate similar from
|
||||
dissimilar texts well. The model may be a poor embedding model or
|
||||
the semantic pairs are too generic. Report the margin.
|
||||
- **NaN/Inf vectors** — the model's embedding endpoint is broken or
|
||||
the input text is malformed. Check the endpoint directly.
|
||||
- **very slow** — >100 vec/s is typical for a local endpoint; remote
|
||||
endpoints may be slower due to network latency. Report the rate.
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
name: test-summary-model
|
||||
description: Tests a summary model (BOR_LLM_SUMMARY_MODEL in .env) against a fixed set of 8 source texts from the fixture KB — evaluates coherence, coverage, brevity, and hallucination detection — then records results in benchmarks/model_benchmarks.csv. Use when the user asks to test or benchmark a summary model, compare summary models, or evaluate summary quality (e.g. "test lite summary", "compare summary models", "how good is turbo at summarizing").
|
||||
---
|
||||
|
||||
# Test a Summary Model (quality benchmark)
|
||||
|
||||
Tests the configured `BOR_LLM_SUMMARY_MODEL` (default `lite`) against
|
||||
8 source texts extracted from the fixture KB. Each text is a realistic
|
||||
documentation excerpt (~300–500 chars). The model is asked to summarize
|
||||
each in 2–4 sentences.
|
||||
|
||||
## Rules (non-negotiable)
|
||||
|
||||
- **Do not touch** the fixture source texts — they are the controlled
|
||||
benchmark corpus. Changing them is a methodology change: flag it.
|
||||
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
|
||||
not the app. If the model exposes an app defect, report it — don't fix it.
|
||||
- `.env` is gitignored and is **not committed** — the model switch stays
|
||||
a live dev setting, and the summary must say which model `.env` is left
|
||||
on (default: the tested model).
|
||||
|
||||
## Procedure
|
||||
|
||||
All commands run from the repo root with `uv run`.
|
||||
|
||||
### 1. Preconditions
|
||||
|
||||
```bash
|
||||
podman compose up -d db
|
||||
grep -E "BOR_LLM_(BASE_URL|API_KEY|SUMMARY_MODEL)" .env
|
||||
```
|
||||
|
||||
### 2. Switch the model (optional)
|
||||
|
||||
Edit only `BOR_LLM_SUMMARY_MODEL` in `.env` (leave chat and embed alone):
|
||||
|
||||
```
|
||||
BOR_LLM_SUMMARY_MODEL=<model>
|
||||
```
|
||||
|
||||
### 3. Run the benchmark
|
||||
|
||||
```bash
|
||||
# Single run
|
||||
uv run python -m scripts.test_summary_model
|
||||
|
||||
# Specify a model explicitly
|
||||
uv run python -m scripts.test_summary_model --model turbo
|
||||
|
||||
# Multiple runs for variance
|
||||
uv run python -m scripts.test_summary_model --runs 3
|
||||
```
|
||||
|
||||
Wall time: ~3–5 s per text, ~25–40 s total for 8 texts.
|
||||
|
||||
### 4. Read the verdicts
|
||||
|
||||
The `gate:` line: `PASS|FAIL turns=N answered=N quality=Q hallucinations=H/N coherence=C/5 coverage=V/5 brevity=B/5 (wall Ts)`.
|
||||
|
||||
Scoring rubric:
|
||||
- **coherence** (0–5): non-empty, multi-sentence, starts with capital
|
||||
- **coverage** (0–5): captures ≥2 key numbers/facts from source
|
||||
- **brevity** (0–5): summary length / source length ratio 0.10–0.25 = 5, 0.05–0.35 = 4, etc.
|
||||
- **hallucination**: detected when summary contains >3 uncommon tokens not in source
|
||||
|
||||
Quality score = coherence×0.35 + coverage×0.40 + brevity×0.25, scaled 0–100,
|
||||
with a 5-point penalty per hallucination.
|
||||
|
||||
Gate: PASS if quality ≥ 70 and hallucination rate < 25%.
|
||||
|
||||
### 5. Record + commit
|
||||
|
||||
Results are automatically appended to `benchmarks/model_benchmarks.csv`.
|
||||
Also append a summary line to `benchmarks/README.md` if it exists, or
|
||||
create it:
|
||||
|
||||
```bash
|
||||
git add benchmarks/model_benchmarks.csv
|
||||
git commit --no-gpg-sign \
|
||||
-m "docs(agent): record the <model> summary benchmark" \
|
||||
-m "<one-paragraph body: quality score, hallucination count, wall time, comparison to other models>"
|
||||
```
|
||||
|
||||
### 6. Summary
|
||||
|
||||
Table of quality / hallucinations / coherence / coverage / brevity / wall
|
||||
per run, the one-line conclusion, and a note that `.env` is now on
|
||||
`<model>`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **endpoint down / all turns error** — check
|
||||
`curl -s $BOR_LLM_BASE_URL/models` with the key; the gate will exit 1
|
||||
with `answered<8`. Report, don't retry-loop.
|
||||
- **low brevity** — the model is restating rather than condensing.
|
||||
This is model-specific; try a more explicit prompt or a different model.
|
||||
- **hallucinations** — the model is adding details not in the source.
|
||||
This is the most common failure mode; tighten the prompt or switch models.
|
||||
Reference in New Issue
Block a user