ducoterra
bb2803bebd
Codifies the 2026-09-05 turbo comparison workflow as a project skill under .agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate (twice, for variance) + the locked derived gate with per-turn wall timing, interpret the two metrics against the reference model rates (re-read habit: lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and commit the doc. Rules baked in: never touch the battery/thresholds/fixtures, never edit app code, never commit .env.
135 lines
5.6 KiB
Markdown
135 lines
5.6 KiB
Markdown
---
|
||
name: test-chat-model
|
||
description: Adds a new chat model to the app (BOR_LLM_CHAT_MODEL in .env) and tests its tool-calling functionality with the controlled methodology from TOOL_CALLING_TESTING.md — fixture battery gate, locked derived gate, per-turn and total wall timing — then records the verdicts in TOOL_CALLING_TESTING.md with a single docs commit. Use when the user asks to add or test a new chat model, benchmark a model on tool calling, or compare models (e.g. "test model turbo", "switch the chat model to X and test it").
|
||
---
|
||
|
||
# Test a Chat Model (controlled tool-calling battery)
|
||
|
||
Switch `BOR_LLM_CHAT_MODEL` to the target model and run the controlled
|
||
tool-calling battery from `TOOL_CALLING_TESTING.md` (repo root). The
|
||
methodology is fixed: 8 hand-written fixture docs with unguessable
|
||
specifics, snapshotted to `tests/fixtures/test_kb.dump.sql` and restored
|
||
in ~0.03 s — **no repo imports, no re-embedding, no app code changes**.
|
||
|
||
The user supplies the model name (e.g. `turbo`). If they don't, ask.
|
||
|
||
## Rules (non-negotiable)
|
||
|
||
- **Do not touch** the battery questions, the thresholds, or the fixture
|
||
documents — changing any of those is a *methodology* change: flag it to
|
||
the user first.
|
||
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
|
||
not the app. If the model exposes an app defect, report it — don't fix
|
||
it here.
|
||
- `.env` is gitignored and is **not committed** — the model switch stays
|
||
a live dev setting, and the summary must say which model `.env` is left
|
||
on (default: the tested model — "add a new model" implies keeping it).
|
||
|
||
## Procedure
|
||
|
||
All commands run from the repo root with `uv run`.
|
||
|
||
### 1. Preconditions (fast fail)
|
||
|
||
```bash
|
||
podman compose up -d db
|
||
ls tests/fixtures/test_kb.dump.sql # must exist
|
||
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL)" .env
|
||
```
|
||
|
||
If the dump is missing, rebuild it once (real embeddings, ~2 s):
|
||
|
||
```bash
|
||
uv run python -m scripts.load_test_kb
|
||
```
|
||
|
||
A gate exit code of **2** means a precondition failed (DB down, schema
|
||
not applied, dump missing) — each prints the actionable fix; fix it and
|
||
re-run. Do not interpret an exit-2 run.
|
||
|
||
### 2. Switch the model
|
||
|
||
Edit the single line in `.env` (leave `BOR_LLM_SUMMARY_MODEL` and
|
||
`BOR_LLM_EMBED_MODEL` alone):
|
||
|
||
```
|
||
BOR_LLM_CHAT_MODEL=<model>
|
||
```
|
||
|
||
### 3. Run the battery (3 runs)
|
||
|
||
```bash
|
||
# a. sanity micro-loop — ~12 s for a fast model, scales with the model
|
||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3
|
||
|
||
# b. full fixture gate, twice (variance matters — see the 92/100/100 spread for lite)
|
||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||
|
||
# c. locked derived battery (phase-72 gate — bare-path traps, executed >= 90% bar)
|
||
uv run python -m scripts.agent_realmodel_check --restore
|
||
```
|
||
|
||
Grep `^gate:` / `^turn ` from each run. Total wall: ~2–8 min depending on
|
||
model speed. Per-turn seconds are on the `turn` lines — capture them,
|
||
they are the latency signal. (No `--concurrency` — the endpoint
|
||
serializes; measured, no gain.)
|
||
|
||
### 4. Read the verdicts
|
||
|
||
Each `gate:` line: `PASS|FAIL turns=10 answered=N caps=N tool-turns=N
|
||
calls X/Y executed (E%) contract C/D (K%) DATE (wall Ts)`. A `FAIL`
|
||
prints the `MISS` lines naming the condition. Interpret:
|
||
|
||
- **contract < 100 %** — real incident-class errors (bad scopes, bare
|
||
paths, hallucinated identities). Correlate with the per-run log lines
|
||
`agent tool=… args=… round=…/…` and the refusal classes in
|
||
`app/rag/agent.py` to name the exact misuse.
|
||
- **executed < contract** — the gap is `ALREADY_IN_CONTEXT` re-reads of
|
||
already-seeded documents: a *model-specific habit* (reference rates:
|
||
`lite` ~100 % of seeded-target turns, `turbo` ~12 %). Report the
|
||
re-read rate, not a verdict.
|
||
- **FAIL on only the ≥6/10 tool-turns usage floor** — the model answered
|
||
seeded questions from `<documents>` context instead of making the
|
||
(refusable) read call the trap design expects. That is the *ideal*
|
||
grounded behavior; report it as a test artifact, accuracy unaffected.
|
||
- **caps > 0** — the phase-72 incident signature; a real regression,
|
||
say so explicitly.
|
||
|
||
### 5. Record + commit
|
||
|
||
Append the model's results to the model-comparison subsection of
|
||
`TOOL_CALLING_TESTING.md` §3, keeping the `gate:` lines **byte-exact
|
||
verbatim**, plus 2–4 sentences of interpretation against the reference
|
||
rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s
|
||
per full loop).
|
||
|
||
Commit — docs only, house style:
|
||
|
||
```bash
|
||
git add TOOL_CALLING_TESTING.md
|
||
git commit --no-gpg-sign \
|
||
-m "docs(agent): record the <model> comparison on the controlled fixture battery" \
|
||
-m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"
|
||
```
|
||
|
||
### 6. Summary
|
||
|
||
Table of contract / executed / caps / tool-turns / wall per run (vs the
|
||
already-recorded models), the one-line conclusion (more or less
|
||
disciplined than the others, faster or slower), and a note that `.env`
|
||
is now on `<model>`.
|
||
|
||
## Troubleshooting
|
||
|
||
- **endpoint down / all turns error** — check
|
||
`curl -s $BOR_LLM_BASE_URL/models` with the key; the gate will exit 1
|
||
with `answered<10`. Report, don't retry-loop.
|
||
- **slow runs** — per-turn seconds on the `turn` lines show it; the LLM
|
||
is ~95 % of the cost, DB work is milliseconds.
|
||
- **a turn deflects (defl=yes)** — the honesty gate found no grounded
|
||
retrieval; that breaks the battery's design contract. Re-run once; if
|
||
it repeats, the fixture KB or the embed model changed — run
|
||
`uv run python -m scripts.load_test_kb` and check its retrieval report
|
||
(all 10 questions must be grounded).
|