feat(skills): add test-chat-model skill — add a chat model and run the controlled tool-calling battery
Codifies the 2026-09-05 turbo comparison workflow as a project skill under .agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate (twice, for variance) + the locked derived gate with per-turn wall timing, interpret the two metrics against the reference model rates (re-read habit: lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and commit the doc. Rules baked in: never touch the battery/thresholds/fixtures, never edit app code, never commit .env.
This commit is contained in:
@@ -0,0 +1,134 @@
|
||||
---
|
||||
name: test-chat-model
|
||||
description: Adds a new chat model to the app (BOR_LLM_CHAT_MODEL in .env) and tests its tool-calling functionality with the controlled methodology from TOOL_CALLING_TESTING.md — fixture battery gate, locked derived gate, per-turn and total wall timing — then records the verdicts in TOOL_CALLING_TESTING.md with a single docs commit. Use when the user asks to add or test a new chat model, benchmark a model on tool calling, or compare models (e.g. "test model turbo", "switch the chat model to X and test it").
|
||||
---
|
||||
|
||||
# Test a Chat Model (controlled tool-calling battery)
|
||||
|
||||
Switch `BOR_LLM_CHAT_MODEL` to the target model and run the controlled
|
||||
tool-calling battery from `TOOL_CALLING_TESTING.md` (repo root). The
|
||||
methodology is fixed: 8 hand-written fixture docs with unguessable
|
||||
specifics, snapshotted to `tests/fixtures/test_kb.dump.sql` and restored
|
||||
in ~0.03 s — **no repo imports, no re-embedding, no app code changes**.
|
||||
|
||||
The user supplies the model name (e.g. `turbo`). If they don't, ask.
|
||||
|
||||
## Rules (non-negotiable)
|
||||
|
||||
- **Do not touch** the battery questions, the thresholds, or the fixture
|
||||
documents — changing any of those is a *methodology* change: flag it to
|
||||
the user first.
|
||||
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
|
||||
not the app. If the model exposes an app defect, report it — don't fix
|
||||
it here.
|
||||
- `.env` is gitignored and is **not committed** — the model switch stays
|
||||
a live dev setting, and the summary must say which model `.env` is left
|
||||
on (default: the tested model — "add a new model" implies keeping it).
|
||||
|
||||
## Procedure
|
||||
|
||||
All commands run from the repo root with `uv run`.
|
||||
|
||||
### 1. Preconditions (fast fail)
|
||||
|
||||
```bash
|
||||
podman compose up -d db
|
||||
ls tests/fixtures/test_kb.dump.sql # must exist
|
||||
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL)" .env
|
||||
```
|
||||
|
||||
If the dump is missing, rebuild it once (real embeddings, ~2 s):
|
||||
|
||||
```bash
|
||||
uv run python -m scripts.load_test_kb
|
||||
```
|
||||
|
||||
A gate exit code of **2** means a precondition failed (DB down, schema
|
||||
not applied, dump missing) — each prints the actionable fix; fix it and
|
||||
re-run. Do not interpret an exit-2 run.
|
||||
|
||||
### 2. Switch the model
|
||||
|
||||
Edit the single line in `.env` (leave `BOR_LLM_SUMMARY_MODEL` and
|
||||
`BOR_LLM_EMBED_MODEL` alone):
|
||||
|
||||
```
|
||||
BOR_LLM_CHAT_MODEL=<model>
|
||||
```
|
||||
|
||||
### 3. Run the battery (3 runs)
|
||||
|
||||
```bash
|
||||
# a. sanity micro-loop — ~12 s for a fast model, scales with the model
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3
|
||||
|
||||
# b. full fixture gate, twice (variance matters — see the 92/100/100 spread for lite)
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||||
|
||||
# c. locked derived battery (phase-72 gate — bare-path traps, executed >= 90% bar)
|
||||
uv run python -m scripts.agent_realmodel_check --restore
|
||||
```
|
||||
|
||||
Grep `^gate:` / `^turn ` from each run. Total wall: ~2–8 min depending on
|
||||
model speed. Per-turn seconds are on the `turn` lines — capture them,
|
||||
they are the latency signal. (No `--concurrency` — the endpoint
|
||||
serializes; measured, no gain.)
|
||||
|
||||
### 4. Read the verdicts
|
||||
|
||||
Each `gate:` line: `PASS|FAIL turns=10 answered=N caps=N tool-turns=N
|
||||
calls X/Y executed (E%) contract C/D (K%) DATE (wall Ts)`. A `FAIL`
|
||||
prints the `MISS` lines naming the condition. Interpret:
|
||||
|
||||
- **contract < 100 %** — real incident-class errors (bad scopes, bare
|
||||
paths, hallucinated identities). Correlate with the per-run log lines
|
||||
`agent tool=… args=… round=…/…` and the refusal classes in
|
||||
`app/rag/agent.py` to name the exact misuse.
|
||||
- **executed < contract** — the gap is `ALREADY_IN_CONTEXT` re-reads of
|
||||
already-seeded documents: a *model-specific habit* (reference rates:
|
||||
`lite` ~100 % of seeded-target turns, `turbo` ~12 %). Report the
|
||||
re-read rate, not a verdict.
|
||||
- **FAIL on only the ≥6/10 tool-turns usage floor** — the model answered
|
||||
seeded questions from `<documents>` context instead of making the
|
||||
(refusable) read call the trap design expects. That is the *ideal*
|
||||
grounded behavior; report it as a test artifact, accuracy unaffected.
|
||||
- **caps > 0** — the phase-72 incident signature; a real regression,
|
||||
say so explicitly.
|
||||
|
||||
### 5. Record + commit
|
||||
|
||||
Append the model's results to the model-comparison subsection of
|
||||
`TOOL_CALLING_TESTING.md` §3, keeping the `gate:` lines **byte-exact
|
||||
verbatim**, plus 2–4 sentences of interpretation against the reference
|
||||
rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s
|
||||
per full loop).
|
||||
|
||||
Commit — docs only, house style:
|
||||
|
||||
```bash
|
||||
git add TOOL_CALLING_TESTING.md
|
||||
git commit --no-gpg-sign \
|
||||
-m "docs(agent): record the <model> comparison on the controlled fixture battery" \
|
||||
-m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"
|
||||
```
|
||||
|
||||
### 6. Summary
|
||||
|
||||
Table of contract / executed / caps / tool-turns / wall per run (vs the
|
||||
already-recorded models), the one-line conclusion (more or less
|
||||
disciplined than the others, faster or slower), and a note that `.env`
|
||||
is now on `<model>`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **endpoint down / all turns error** — check
|
||||
`curl -s $BOR_LLM_BASE_URL/models` with the key; the gate will exit 1
|
||||
with `answered<10`. Report, don't retry-loop.
|
||||
- **slow runs** — per-turn seconds on the `turn` lines show it; the LLM
|
||||
is ~95 % of the cost, DB work is milliseconds.
|
||||
- **a turn deflects (defl=yes)** — the honesty gate found no grounded
|
||||
retrieval; that breaks the battery's design contract. Re-run once; if
|
||||
it repeats, the fixture KB or the embed model changed — run
|
||||
`uv run python -m scripts.load_test_kb` and check its retrieval report
|
||||
(all 10 questions must be grounded).
|
||||
Reference in New Issue
Block a user