From bb2803bebd44f5718eef9caaf29aec81615cc5ec Mon Sep 17 00:00:00 2001 From: ducoterra Date: Sat, 5 Sep 2026 00:13:21 -0400 Subject: [PATCH] =?UTF-8?q?feat(skills):=20add=20test-chat-model=20skill?= =?UTF-8?q?=20=E2=80=94=20add=20a=20chat=20model=20and=20run=20the=20contr?= =?UTF-8?q?olled=20tool-calling=20battery?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codifies the 2026-09-05 turbo comparison workflow as a project skill under .agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate (twice, for variance) + the locked derived gate with per-turn wall timing, interpret the two metrics against the reference model rates (re-read habit: lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and commit the doc. Rules baked in: never touch the battery/thresholds/fixtures, never edit app code, never commit .env. --- .agents/skills/test-chat-model/SKILL.md | 134 ++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 .agents/skills/test-chat-model/SKILL.md diff --git a/.agents/skills/test-chat-model/SKILL.md b/.agents/skills/test-chat-model/SKILL.md new file mode 100644 index 0000000..05824e8 --- /dev/null +++ b/.agents/skills/test-chat-model/SKILL.md @@ -0,0 +1,134 @@ +--- +name: test-chat-model +description: Adds a new chat model to the app (BOR_LLM_CHAT_MODEL in .env) and tests its tool-calling functionality with the controlled methodology from TOOL_CALLING_TESTING.md — fixture battery gate, locked derived gate, per-turn and total wall timing — then records the verdicts in TOOL_CALLING_TESTING.md with a single docs commit. Use when the user asks to add or test a new chat model, benchmark a model on tool calling, or compare models (e.g. "test model turbo", "switch the chat model to X and test it"). +--- + +# Test a Chat Model (controlled tool-calling battery) + +Switch `BOR_LLM_CHAT_MODEL` to the target model and run the controlled +tool-calling battery from `TOOL_CALLING_TESTING.md` (repo root). The +methodology is fixed: 8 hand-written fixture docs with unguessable +specifics, snapshotted to `tests/fixtures/test_kb.dump.sql` and restored +in ~0.03 s — **no repo imports, no re-embedding, no app code changes**. + +The user supplies the model name (e.g. `turbo`). If they don't, ask. + +## Rules (non-negotiable) + +- **Do not touch** the battery questions, the thresholds, or the fixture + documents — changing any of those is a *methodology* change: flag it to + the user first. +- **Do not edit app code** (`app/`, `tests/`). This skill tests a model, + not the app. If the model exposes an app defect, report it — don't fix + it here. +- `.env` is gitignored and is **not committed** — the model switch stays + a live dev setting, and the summary must say which model `.env` is left + on (default: the tested model — "add a new model" implies keeping it). + +## Procedure + +All commands run from the repo root with `uv run`. + +### 1. Preconditions (fast fail) + +```bash +podman compose up -d db +ls tests/fixtures/test_kb.dump.sql # must exist +grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL)" .env +``` + +If the dump is missing, rebuild it once (real embeddings, ~2 s): + +```bash +uv run python -m scripts.load_test_kb +``` + +A gate exit code of **2** means a precondition failed (DB down, schema +not applied, dump missing) — each prints the actionable fix; fix it and +re-run. Do not interpret an exit-2 run. + +### 2. Switch the model + +Edit the single line in `.env` (leave `BOR_LLM_SUMMARY_MODEL` and +`BOR_LLM_EMBED_MODEL` alone): + +``` +BOR_LLM_CHAT_MODEL= +``` + +### 3. Run the battery (3 runs) + +```bash +# a. sanity micro-loop — ~12 s for a fast model, scales with the model +uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3 + +# b. full fixture gate, twice (variance matters — see the 92/100/100 spread for lite) +uv run python -m scripts.agent_realmodel_check --restore --mode fixture +uv run python -m scripts.agent_realmodel_check --restore --mode fixture + +# c. locked derived battery (phase-72 gate — bare-path traps, executed >= 90% bar) +uv run python -m scripts.agent_realmodel_check --restore +``` + +Grep `^gate:` / `^turn ` from each run. Total wall: ~2–8 min depending on +model speed. Per-turn seconds are on the `turn` lines — capture them, +they are the latency signal. (No `--concurrency` — the endpoint +serializes; measured, no gain.) + +### 4. Read the verdicts + +Each `gate:` line: `PASS|FAIL turns=10 answered=N caps=N tool-turns=N +calls X/Y executed (E%) contract C/D (K%) DATE (wall Ts)`. A `FAIL` +prints the `MISS` lines naming the condition. Interpret: + +- **contract < 100 %** — real incident-class errors (bad scopes, bare + paths, hallucinated identities). Correlate with the per-run log lines + `agent tool=… args=… round=…/…` and the refusal classes in + `app/rag/agent.py` to name the exact misuse. +- **executed < contract** — the gap is `ALREADY_IN_CONTEXT` re-reads of + already-seeded documents: a *model-specific habit* (reference rates: + `lite` ~100 % of seeded-target turns, `turbo` ~12 %). Report the + re-read rate, not a verdict. +- **FAIL on only the ≥6/10 tool-turns usage floor** — the model answered + seeded questions from `` context instead of making the + (refusable) read call the trap design expects. That is the *ideal* + grounded behavior; report it as a test artifact, accuracy unaffected. +- **caps > 0** — the phase-72 incident signature; a real regression, + say so explicitly. + +### 5. Record + commit + +Append the model's results to the model-comparison subsection of +`TOOL_CALLING_TESTING.md` §3, keeping the `gate:` lines **byte-exact +verbatim**, plus 2–4 sentences of interpretation against the reference +rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s +per full loop). + +Commit — docs only, house style: + +```bash +git add TOOL_CALLING_TESTING.md +git commit --no-gpg-sign \ + -m "docs(agent): record the comparison on the controlled fixture battery" \ + -m "" +``` + +### 6. Summary + +Table of contract / executed / caps / tool-turns / wall per run (vs the +already-recorded models), the one-line conclusion (more or less +disciplined than the others, faster or slower), and a note that `.env` +is now on ``. + +## Troubleshooting + +- **endpoint down / all turns error** — check + `curl -s $BOR_LLM_BASE_URL/models` with the key; the gate will exit 1 + with `answered<10`. Report, don't retry-loop. +- **slow runs** — per-turn seconds on the `turn` lines show it; the LLM + is ~95 % of the cost, DB work is milliseconds. +- **a turn deflects (defl=yes)** — the honesty gate found no grounded + retrieval; that breaks the battery's design contract. Re-run once; if + it repeats, the fixture KB or the embed model changed — run + `uv run python -m scripts.load_test_kb` and check its retrieval report + (all 10 questions must be grounded).