brain-of-reese/.agents/skills/test-chat-model/SKILL.md

name, description

name

description

test-chat-model

Adds a new chat model to the app (BOR_LLM_CHAT_MODEL in .env) and tests its tool-calling functionality with the controlled methodology from TOOL_CALLING_TESTING.md — fixture battery gate, locked derived gate, per-turn and total wall timing — then records the verdicts in TOOL_CALLING_TESTING.md with a single docs commit. Use when the user asks to add or test a new chat model, benchmark a model on tool calling, or compare models (e.g. "test model turbo", "switch the chat model to X and test it").

Test a Chat Model (controlled tool-calling battery)

Switch BOR_LLM_CHAT_MODEL to the target model and run the controlled tool-calling battery from TOOL_CALLING_TESTING.md (repo root). The methodology is fixed: 8 hand-written fixture docs with unguessable specifics, snapshotted to tests/fixtures/test_kb.dump.sql and restored in ~0.03 s — no repo imports, no re-embedding, no app code changes.

The user supplies the model name (e.g. turbo). If they don't, ask.

Rules (non-negotiable)

• Do not touch the battery questions, the thresholds, or the fixture documents — changing any of those is a methodology change: flag it to the user first.
• Do not edit app code (app/, tests/). This skill tests a model, not the app. If the model exposes an app defect, report it — don't fix it here.
• .env is gitignored and is not committed — the model switch stays a live dev setting, and the summary must say which model .env is left on (default: the tested model — "add a new model" implies keeping it).

Procedure

All commands run from the repo root with uv run.

1. Preconditions (fast fail)

podman compose up -d db
ls tests/fixtures/test_kb.dump.sql        # must exist
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL)" .env

If the dump is missing, rebuild it once (real embeddings, ~2 s):

uv run python -m scripts.load_test_kb

A gate exit code of 2 means a precondition failed (DB down, schema not applied, dump missing) — each prints the actionable fix; fix it and re-run. Do not interpret an exit-2 run.

2. Switch the model

Edit the single line in .env (leave BOR_LLM_SUMMARY_MODEL and BOR_LLM_EMBED_MODEL alone):

BOR_LLM_CHAT_MODEL=<model>

3. Run the battery (3 runs)

# a. sanity micro-loop — ~12 s for a fast model, scales with the model
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3

# b. full fixture gate, twice (variance matters — see the 92/100/100 spread for lite)
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
uv run python -m scripts.agent_realmodel_check --restore --mode fixture

# c. locked derived battery (phase-72 gate — bare-path traps, executed >= 90% bar)
uv run python -m scripts.agent_realmodel_check --restore

Grep ^gate: / ^turn from each run. Total wall: ~2–8 min depending on model speed. Per-turn seconds are on the turn lines — capture them, they are the latency signal. (No --concurrency — the endpoint serializes; measured, no gain.)

4. Read the verdicts

Each gate: line: PASS|FAIL turns=10 answered=N caps=N tool-turns=N calls X/Y executed (E%) contract C/D (K%) DATE (wall Ts). A FAIL prints the MISS lines naming the condition. Interpret:

• contract < 100 % — real incident-class errors (bad scopes, bare paths, hallucinated identities). Correlate with the per-run log lines agent tool=… args=… round=…/… and the refusal classes in app/rag/agent.py to name the exact misuse.
• executed < contract — the gap is ALREADY_IN_CONTEXT re-reads of already-seeded documents: a model-specific habit (reference rates: lite ~100 % of seeded-target turns, turbo ~12 %). Report the re-read rate, not a verdict.
• FAIL on only the ≥6/10 tool-turns usage floor — the model answered seeded questions from <documents> context instead of making the (refusable) read call the trap design expects. That is the ideal grounded behavior; report it as a test artifact, accuracy unaffected.
• caps > 0 — the phase-72 incident signature; a real regression, say so explicitly.

5. Record + commit

Append the model's results to the model-comparison subsection of TOOL_CALLING_TESTING.md §3, keeping the gate: lines byte-exact verbatim, plus 2–4 sentences of interpretation against the reference rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s per full loop).

Commit — docs only, house style:

git add TOOL_CALLING_TESTING.md
git commit --no-gpg-sign \
  -m "docs(agent): record the <model> comparison on the controlled fixture battery" \
  -m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"

6. Summary

Table of contract / executed / caps / tool-turns / wall per run (vs the already-recorded models), the one-line conclusion (more or less disciplined than the others, faster or slower), and a note that .env is now on <model>.

Troubleshooting

• endpoint down / all turns error — check curl -s $BOR_LLM_BASE_URL/models with the key; the gate will exit 1 with answered<10. Report, don't retry-loop.
• slow runs — per-turn seconds on the turn lines show it; the LLM is ~95 % of the cost, DB work is milliseconds.
• a turn deflects (defl=yes) — the honesty gate found no grounded retrieval; that breaks the battery's design contract. Re-run once; if it repeats, the fixture KB or the embed model changed — run uv run python -m scripts.load_test_kb and check its retrieval report (all 10 questions must be grounded).