---
name: test-chat-model
description: Adds a new chat model to the app (BOR_LLM_CHAT_MODEL in .env) and tests its tool-calling functionality with the controlled methodology from TOOL_CALLING_TESTING.md — fixture battery gate, locked derived gate, per-turn and total wall timing — then records the verdicts in TOOL_CALLING_TESTING.md with a single docs commit. Use when the user asks to add or test a new chat model, benchmark a model on tool calling, or compare models (e.g. "test model turbo", "switch the chat model to X and test it").
---

# Test a Chat Model (controlled tool-calling battery)

Switch `BOR_LLM_CHAT_MODEL` to the target model and run the controlled
tool-calling battery from `TOOL_CALLING_TESTING.md` (repo root). The
methodology is fixed: 8 hand-written fixture docs with unguessable
specifics, snapshotted to `tests/fixtures/test_kb.dump.sql` and restored
in ~0.03 s — **no repo imports, no re-embedding, no app code changes**.

The user supplies the model name (e.g. `turbo`). If they don't, ask.

## Rules (non-negotiable)

- **Do not touch** the battery questions, the thresholds, or the fixture
  documents — changing any of those is a *methodology* change: flag it to
  the user first.
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
  not the app. If the model exposes an app defect, report it — don't fix
  it here.
- `.env` is gitignored and is **not committed** — the model switch stays
  a live dev setting, and the summary must say which model `.env` is left
  on (default: the tested model — "add a new model" implies keeping it).

## Procedure

All commands run from the repo root with `uv run`.

### 1. Preconditions (fast fail)

```bash
podman compose up -d db
ls tests/fixtures/test_kb.dump.sql        # must exist
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL)" .env
```

If the dump is missing, rebuild it once (real embeddings, ~2 s):

```bash
uv run python -m scripts.load_test_kb
```

A gate exit code of **2** means a precondition failed (DB down, schema
not applied, dump missing) — each prints the actionable fix; fix it and
re-run. Do not interpret an exit-2 run.

### 2. Switch the model

Edit the single line in `.env` (leave `BOR_LLM_SUMMARY_MODEL` and
`BOR_LLM_EMBED_MODEL` alone):

```
BOR_LLM_CHAT_MODEL=<model>
```

### 3. Run the battery (3 runs)

```bash
# a. sanity micro-loop — ~12 s for a fast model, scales with the model
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3

# b. full fixture gate, twice (variance matters — see the 92/100/100 spread for lite)
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
uv run python -m scripts.agent_realmodel_check --restore --mode fixture

# c. locked derived battery (phase-72 gate — bare-path traps, executed >= 90% bar)
uv run python -m scripts.agent_realmodel_check --restore
```

Grep `^gate:` / `^turn ` from each run. Total wall: ~2–8 min depending on
model speed. Per-turn seconds are on the `turn` lines — capture them,
they are the latency signal. (No `--concurrency` — the endpoint
serializes; measured, no gain.)

### 4. Read the verdicts

Each `gate:` line: `PASS|FAIL turns=10 answered=N caps=N tool-turns=N
calls X/Y executed (E%) contract C/D (K%) DATE (wall Ts)`. A `FAIL`
prints the `MISS` lines naming the condition. Interpret:

- **contract < 100 %** — real incident-class errors (bad scopes, bare
  paths, hallucinated identities). Correlate with the per-run log lines
  `agent tool=… args=… round=…/…` and the refusal classes in
  `app/rag/agent.py` to name the exact misuse.
- **executed < contract** — the gap is `ALREADY_IN_CONTEXT` re-reads of
  already-seeded documents: a *model-specific habit* (reference rates:
  `lite` ~100 % of seeded-target turns, `turbo` ~12 %). Report the
  re-read rate, not a verdict.
- **FAIL on only the ≥6/10 tool-turns usage floor** — the model answered
  seeded questions from `<documents>` context instead of making the
  (refusable) read call the trap design expects. That is the *ideal*
  grounded behavior; report it as a test artifact, accuracy unaffected.
- **caps > 0** — the phase-72 incident signature; a real regression,
  say so explicitly.

### 5. Record + commit

Append the model's results to the model-comparison subsection of
`TOOL_CALLING_TESTING.md` §3, keeping the `gate:` lines **byte-exact
verbatim**, plus 2–4 sentences of interpretation against the reference
rates above and the wall-time baseline (lite ~43–55 s, turbo ~105–135 s
per full loop).

Commit — docs only, house style:

```bash
git add TOOL_CALLING_TESTING.md
git commit --no-gpg-sign \
  -m "docs(agent): record the <model> comparison on the controlled fixture battery" \
  -m "<one-paragraph body: the numbers, the re-read rate, the wall time, any MISS nuance>"
```

### 6. Summary

Table of contract / executed / caps / tool-turns / wall per run (vs the
already-recorded models), the one-line conclusion (more or less
disciplined than the others, faster or slower), and a note that `.env`
is now on `<model>`.

## Troubleshooting

- **endpoint down / all turns error** — check
  `curl -s $BOR_LLM_BASE_URL/models` with the key; the gate will exit 1
  with `answered<10`. Report, don't retry-loop.
- **slow runs** — per-turn seconds on the `turn` lines show it; the LLM
  is ~95 % of the cost, DB work is milliseconds.
- **a turn deflects (defl=yes)** — the honesty gate found no grounded
  retrieval; that breaks the battery's design contract. Re-run once; if
  it repeats, the fixture KB or the embed model changed — run
  `uv run python -m scripts.load_test_kb` and check its retrieval report
  (all 10 questions must be grounded).