ducoterra
f221b40fce
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
151 lines
6.1 KiB
Markdown
151 lines
6.1 KiB
Markdown
---
|
||
name: test-chat-model
|
||
description: Adds a new chat model to the app (BOR_LLM_CHAT_MODEL in .env) and tests its tool-calling functionality with the controlled methodology from TOOL_CALLING_TESTING.md — fixture battery gate, locked derived gate, per-turn and total wall timing — then records the verdicts in TOOL_CALLING_TESTING.md with a single docs commit. Use when the user asks to add or test a new chat model, benchmark a model on tool calling, or compare models (e.g. "test model turbo", "switch the chat model to X and test it").
|
||
---
|
||
|
||
# Test a Chat Model (controlled tool-calling battery)
|
||
|
||
Switch `BOR_LLM_CHAT_MODEL` to the target model and run the controlled
|
||
tool-calling battery from `TOOL_CALLING_TESTING.md` (repo root). The
|
||
methodology is fixed: 8 hand-written fixture docs with unguessable
|
||
specifics, snapshotted to `tests/fixtures/test_kb.dump.sql` and restored
|
||
in ~0.03 s — **no repo imports, no re-embedding, no app code changes**.
|
||
|
||
The user supplies the model name (e.g. `turbo`). If they don't, ask.
|
||
|
||
## Rules (non-negotiable)
|
||
|
||
- **Do not touch** the battery questions, the thresholds, or the fixture
|
||
documents — changing any of those is a *methodology* change: flag it to
|
||
the user first.
|
||
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
|
||
not the app. If the model exposes an app defect, report it — don't fix
|
||
it here.
|
||
- `.env` is gitignored and is **not committed** — the model switch stays
|
||
a live dev setting, and the summary must say which model `.env` is left
|
||
on (default: the tested model — "add a new model" implies keeping it).
|
||
|
||
## Procedure
|
||
|
||
All commands run from the repo root with `uv run`.
|
||
|
||
### 1. Preconditions (fast fail)
|
||
|
||
```bash
|
||
podman compose up -d db
|
||
ls tests/fixtures/test_kb.dump.sql # must exist
|
||
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL)" .env
|
||
```
|
||
|
||
If the dump is missing, rebuild it once (real embeddings, ~2 s):
|
||
|
||
```bash
|
||
uv run python -m scripts.load_test_kb
|
||
```
|
||
|
||
A gate exit code of **2** means a precondition failed (DB down, schema
|
||
not applied, dump missing) — each prints the actionable fix; fix it and
|
||
re-run. Do not interpret an exit-2 run.
|
||
|
||
### 2. Switch the model
|
||
|
||
Edit the single line in `.env` (leave `BOR_LLM_SUMMARY_MODEL` and
|
||
`BOR_LLM_EMBED_MODEL` alone):
|
||
|
||
```
|
||
BOR_LLM_CHAT_MODEL=<model>
|
||
```
|
||
|
||
### 3. Run the battery (3 runs)
|
||
|
||
```bash
|
||
# a. sanity micro-loop — ~12 s for a fast model, scales with the model
|
||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3
|
||
|
||
# b. full fixture gate, twice (variance matters — see the 92/100/100 spread for lite)
|
||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||
|
||
# c. locked derived battery (phase-72 gate — bare-path traps, executed >= 90% bar)
|
||
uv run python -m scripts.agent_realmodel_check --restore
|
||
```
|
||
|
||
Grep `^gate:` / `^turn ` from each run. Total wall: ~2–8 min depending on
|
||
model speed. Per-turn seconds are on the `turn` lines — capture them,
|
||
they are the latency signal. (No `--concurrency` — the endpoint
|
||
serializes; measured, no gain.)
|
||
|
||
### 4. Read the verdicts
|
||
|
||
Each `gate:` line: `PASS|FAIL turns=10 answered=N caps=N tool-turns=N
|
||
calls X/Y executed (E%) contract C/D (K%) DATE (wall Ts)`. A `FAIL`
|
||
prints the `MISS` lines naming the condition. Interpret:
|
||
|
||
- **contract < 100 %** — real incident-class errors (bad scopes, bare
|
||
paths, hallucinated identities). Correlate with the per-run log lines
|
||
`agent tool=… args=… round=…/…` and the refusal classes in
|
||
`app/rag/agent.py` to name the exact misuse.
|
||
- **executed < contract** — the gap is `ALREADY_IN_CONTEXT` re-reads of
|
||
already-seeded documents: a *model-specific habit* (reference rates:
|
||
`lite` ~100 % of seeded-target turns, `turbo` ~12 %). Report the
|
||
re-read rate, not a verdict.
|
||
- **FAIL on only the ≥6/10 tool-turns usage floor** — the model answered
|
||
seeded questions from `<documents>` context instead of making the
|
||
(refusable) read call the trap design expects. That is the *ideal*
|
||
grounded behavior; report it as a test artifact, accuracy unaffected.
|
||
- **caps > 0** — the phase-72 incident signature; a real regression,
|
||
say so explicitly.
|
||
|
||
### 5. Record to CSV + commit
|
||
|
||
**a. CSV record** — append to `benchmarks/model_benchmarks.csv` via
|
||
`scripts/model_benchmark.bench_write`. For each of the 3 runs, call:
|
||
|
||
```python
|
||
from scripts.model_benchmark import bench_write
|
||
|
||
bench_write(
|
||
script="chat", model="<model>", mode="fixture",
|
||
gate_status="PASS", turns=10, answered=10, caps=0,
|
||
tool_turns=<N>, emitted=<E>, executed=<X>,
|
||
contract=<C>, wall_s=<wall>,
|
||
contract_denom=<C_denom>, executed_denom=<E_denom>,
|
||
)
|
||
```
|
||
|
||
For the derived run, use `mode="derived"`.
|
||
|
||
**b. Append to TOOL_CALLING_TESTING.md** — keep the `gate:` lines
|
||
**byte-exact verbatim**, plus 2–4 sentences of interpretation against
|
||
the reference rates above and the wall-time baseline (lite ~40–55 s,
|
||
turbo ~97–135 s per full loop).
|
||
|
||
Commit — docs only, house style:
|
||
|
||
```bash
|
||
git add TOOL_CALLING_TESTING.md benchmarks/model_benchmarks.csv
|
||
git commit --no-gpg-sign \
|
||
-m "docs(agent): record the <model> comparison on the controlled fixture battery" \
|
||
-m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"
|
||
```
|
||
|
||
### 6. Summary
|
||
|
||
Table of contract / executed / caps / tool-turns / wall per run (vs the
|
||
already-recorded models), the one-line conclusion (more or less
|
||
disciplined than the others, faster or slower), and a note that `.env`
|
||
is now on `<model>`.
|
||
|
||
## Troubleshooting
|
||
|
||
- **endpoint down / all turns error** — check
|
||
`curl -s $BOR_LLM_BASE_URL/models` with the key; the gate will exit 1
|
||
with `answered<10`. Report, don't retry-loop.
|
||
- **slow runs** — per-turn seconds on the `turn` lines show it; the LLM
|
||
is ~95 % of the cost, DB work is milliseconds.
|
||
- **a turn deflects (defl=yes)** — the honesty gate found no grounded
|
||
retrieval; that breaks the battery's design contract. Re-run once; if
|
||
it repeats, the fixture KB or the embed model changed — run
|
||
`uv run python -m scripts.load_test_kb` and check its retrieval report
|
||
(all 10 questions must be grounded).
|