brain-of-reese/TOOL_CALLING_TESTING.md

Tool-Calling Testing Methodology (controlled KB + one-command fast loop)

How to test, measure, and iterate on the agent's tool calling (ls / read / grep) against the real configured chat model (lite per .env) — fast enough to iterate on, controlled enough to trust.

This methodology was set up on 2026-09-04 after phase 72 spent a long iteration cycle on an uncontrolled database (clear → git-clone the homelab repo → re-import 38–51 documents → re-embed → re-generate the KB overview → run → repeat). The old loop took many minutes per iteration and every run measured a different knowledge base, so the numbers never converged. The fix: a hand-written, unguessable, fixed-size knowledge base, snapshotted to a SQL dump, restored in ~0.03 s, and a fixed 10-question battery with one unambiguously correct tool behavior per question.

---

1. The fast loop (one command)

podman compose up -d db                                    # once
uv run python -m scripts.agent_realmodel_check --restore --mode fixture

That is the whole loop:

1. restore the fixture KB from tests/fixtures/test_kb.dump.sql (one transaction — no git clone, no re-embedding, no lite calls; ~0.03 s hot),
2. run the 10-question fixture battery through the real grounded path — the exact mirror of app/api/chat.py: embed → hybrid retrieval → the honesty gate (plan_turn) → the real prompt (persona + KB overview + <documents> + <tools>) → run_agent against the live endpoint,
3. print one line per turn plus the verdict.

Measured timings (2026-09-04, this machine):

step	time
restore fixture KB	0.03 s (0.2 s first run — psycopg connect)
3-turn micro-loop (--turns 3)	~12 s end-to-end (incl. ~1 s uv/python startup)
full 10-turn fixture loop	~43–51 s wall
one-off KB rebuild (real embeddings, 9 chunks)	~1–2 s

Iteration workflow. When tuning the copy levers (§4), do not run the full battery — run the micro-loop on the first three turns (the incident turn + both listing traps, the fastest signal):

uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3

~12 s per variant. Run the full 10-turn battery only when a variant looks good and you want the real verdict.

Timing is visible, by design: every turn line carries its wall seconds and the verdict line carries the run's total wall time, so a slow-down (endpoint load, a retry storm, a copy that makes the model ramble) is visible on the same line as the accuracy:

turn 01 | emitted=1 executed=1 cap=no defl=no |  4.06s | List the files in …
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)

Notes on speed, measured (not guessed):

• --concurrency 2 / --concurrency 3 was tested and does not help: the aipi endpoint serializes generation server-side, so parallel turns finish in the same total wall time (45.5 s @ 3-way vs ~44 s sequential) with the same aggregates. Sequential stays the default for clean telemetry.
• The LLM is ~95 % of the cost (1–3 model rounds per turn at ~2–6 s each). Database work per turn is milliseconds. Don't optimize it.

---

2. The controlled knowledge base

tests/fixtures/agent_kb/
├── deployments/
│   ├── ansible/lab-inventory.md
│   ├── ci/gitlab-runner.md
│   └── quadlet/mimir-service.md
└── homelab/
    ├── backups/restic-rack7.md
    ├── containers/qwen38-llamacpp.md
    ├── containers/uptime-kuma.md
    ├── networking/meridian-notes.md
    └── networking/vela-bridges.md

8 hand-written markdown documents, 2 sources (source name = directory basename, the importer's rule). Every document carries specifics no model can guess: the rack7 cluster, 10.77.42.0/24 and the VLAN 130 lab-iot pool, PVE build 8.3.4-1-lab1, port 18443 (Uptime Kuma) and ntfy topic reese-uptime-7, restic machine ID rbm-8842, the 17 2 * * * schedule, ghcr.io/reese/obsidian-bor:2026.7.14 on 127.0.0.1:18765, the Qwen 3.8 llama.cpp launch line, ansible-core 2.19.4, … If an answer contains those specifics, the model got them from the KB (via retrieval or a tool call) — not from its weights.

Two deliberate design rules:

1. Non-topical file names for the read targets. vela-bridges.md, meridian-notes.md, mimir-service.md carry no words their content repeats. Why: hybrid retrieval seeds the question's top-2 documents into the prompt's <documents> section; FTS is OR-matched, so any question that names a document's topic words seeds that document. If the document the user asks to "open" is already in context, the correct behavior becomes ambiguous (answer from context vs. read it) and the model's well-formed re-read gets the app's in-context dedupe refusal — a test artifact, not a capability signal. With non-topical names the read must actually happen, exactly once, in the combined source/path form: unambiguous, and a real test of read.
2. The grep token is unique. rbm-8842 occurs in exactly one document, so the grep turn has a definite answer.

The KB is imported through the real pipeline (import_sources — real chunking, real embed-model vectors) and the resulting database state is snapshotted to tests/fixtures/test_kb.dump.sql — a data-only SQL script (TRUNCATE + one multi-row INSERT per app table: documents, chunks + embeddings, the git_sources local rows that make the source registry self-contained, the static KB overview, the sources version). Restoring it puts the whole known state back in one transaction; the generated chunks.tsv column is recomputed by Postgres. The dump is verified by round-trip at build time (restore + per-table checksum compare — a serialization bug fails the build).

# Rebuild the KB + dump — only when the fixture documents, the
# chunker, or the embedding model change. NOT part of the loop.
uv run python -m scripts.load_test_kb
# restores the fixture KB standalone (what --restore runs inline)
uv run python -m scripts.restore_test_kb

The build script also prints a retrieval report — for every battery question, whether the real honesty gate grounds it and which two documents would seed the context. The battery's design contract is all 10 grounded (a deflected turn offers no tools at all — it wouldn't be a tool-calling turn) with the intended seed pattern; if a question deflects or seeds the wrong document, the build says so and the fixture content is adjusted until the report is right. That report is what makes the test design checkable in 2 seconds.

Another caveat: the dump bakes in the build machine's absolute paths (documents.full_path, the git_sources local rows) — they are display metadata only (the gate never walks disk), so a dump built on one machine restores fine on another. If that ever matters, rebuild.

Caveat (measured): vector cosine in an 8-document KB sits at ~0.55–0.68 for generic questions, so one read target (vela-bridges.md) is seeded by cosine even though no FTS token hits it. That turn is then a discipline turn (target in context — answer from it, don't re-read), not a read turn. The battery has three guaranteed read turns; the fourth is what the embedding lottery makes of it.

---

3. The battery and the metrics

The battery (locked for the methodology — don't swap in easier questions)

#	question	tests	expected ideal
1	List the files in this directory.	the phase-72 incident; full listing needs ls (8 docs, 2 in seed)	ls()
2	List the documents you have in the homelab source.	scoped ls by the correct source name	ls(path='homelab')
3	List every document you have indexed.	no-arg listing	ls()
4	Open the document homelab/networking/vela-bridges.md …	read, combined form	read('homelab/networking/vela-bridges.md')
5	Read deployments/quadlet/mimir-service.md and summarize it.	read, unseeded target	read(…) (or ls first, then read)
6	Open the document homelab/networking/meridian-notes.md …	read, unseeded target	read(…)
7	Find the exact string "rbm-8842" in your documents …	grep, pattern only	grep(pattern='rbm-8842') — the match line alone answers it
8	Which document has the title "Lab Ansible Inventory"? Summarize it.	title lookup; target IS seeded	answer from context (or ls)
9	What do you know about the qwen 3.8 llama.cpp setup? …	topic lookup; target IS seeded	answer from context
10	List the files in the deployments directory.	source name phrased as a directory	ls(path='deployments')

Questions 4–6 name the full combined identity (no bare-path trap — that is the job of the locked derived battery, §6). Questions 7–9 name content, so their target document is seeded; the correct behavior there is to not re-read what is already in the prompt.

The four pass conditions

1. all 10 turns answer (no LLMError/MalformedReplyError);
2. zero turns hit the round cap (the incident's loop signature);
3. ≥6 of 10 turns emit ≥1 tool call (the model keeps using tools);
4. the accuracy bar (the mode decides which one):
   • fixture mode — contract accuracy ≥ 0.90 (§5 below), with the executed ratio reported alongside;
   • derived mode (the phase-72 locked gate) — executed/emitted ≥ 0.90, byte-compatible with the phase-72 task file.

Current standing (2026-09-04, lite, fixture KB)

gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%)  contract 11/11 (100%)  (wall 43.4s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%)  contract 12/13 (92%)   (wall 50.6s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%)  contract 11/11 (100%)  (wall 46.8s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 9/12 executed (75%) contract 11/12 (92%) 2026-09-06 (wall 40.4s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/14 executed (64%) contract 13/14 (93%) 2026-09-06 (wall 40.5s)

Contract accuracy ≥ 90 %: met (100 / 92 / 100 / 93 / 92 / 93). The executed ratio sits at 58–73 % for the reason documented in §5 — an app semantics choice, not a model defect, and the open design question in §7.

Model comparison — turbo (2026-09-05, same fixture KB, .env chat model switched to turbo):

gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s)   [derived battery — MISS: 5/10 tool-turn floor]
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s)   [derived battery — MISS: 5/10 tool-turn floor]

Reads: the re-read habit is model-specific. lite re-reads a seeded named document ~100 % of the time (copy-invariant, §4); turbo answered 7 of 8 seeded-target questions straight from the <documents> context with zero tool calls — the exact "don't re-read" behavior the copy levers could not buy from lite (one re-read in the sample, 12 % vs ~100 %). Consequence: turbo hits 100 % on both metrics on the fixture battery — the executed ratio reaches 100 % naturally once the redundant reads stop, which corroborates §7's framing (the block on lite is the model's re-read habit, not a gate or app defect). On the locked derived battery turbo fails only the usage floor condition (≥ 6/10 turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target questions from context instead of making the (refusable) read call the trap design expects; accuracy on every call it does make is still 100 %. The cost: ~2.5× slower wall time (97–114 s per full loop vs 40–41 s for lite, with individual turns 7–20 s).

---

4. The copy levers (what you iterate)

All three are fixed-template constants with byte-pinned unit tests — change the constant, update the pin, run uv run pytest tests/unit -q (~10 s), then the micro-loop:

lever	where	what it teaches
refusal templates	app/rag/agent.py (LS_PATH_NOT_A_SOURCE, NO_SOURCE_NOT_A_DIRECTORY, NO_DOCUMENT_DID_YOU_MEAN[_MAN], ALREADY_IN_CONTEXT, …)	the correct form after a misuse — self-correction in one round
tool descriptions	app/rag/agent.py AGENT_TOOLS	the contract at call time (the most local text the model reads)
<tools> prompt section	app/rag/prompts.py TOOLS_SECTION	the contract up front, every grounded turn

Unit pins to follow the constants: tests/unit/test_agent.py (description + refusal pins), tests/unit/test_prompts.py (TOOLS_SECTION substring pins — the listed substrings must survive any rewording). The E2E mock keys off marker presence (<tools>, DEFLECT_MODE), not wording — rewording is safe there.

What has been tried on this model (2026-09-03 → 04, all measured live) — so the next iteration doesn't repeat it:

variant	re-reads of seeded docs	note
phase-72: mid-paragraph do-not-read rule (TOOLS_SECTION + read description)	15/15 (never flipped)	9 runs, 38–51 doc KBs
leading in-<documents>-section reminder naming the blocks	0/15 flipped	reverted — primed seed paths as ls scopes (incident turn regressed to a cap loop)
front-loaded do-not-read as the read description's first sentence	no improvement	+ one 6-emitted variance spike
per-block note="…do not call read on it" attribute on each <document> header	no improvement	reverted

Conclusion: the re-read of a salient seeded document is copy-invariant behavior of the lite model (it obeys the user's "open it / read it" over every prompt-level rule tried). The levers that do work on this model: the teaching refusals (bare-path self-correction in exactly one round — 4/4 in the derived battery; NO_DOCUMENT_DID_YOU_MEAN naming the combined identity), the one-call-per-reply and never-repeat rules (no cap hits, no repeat loops in any controlled run), and the grep pattern-only clause (the source-scoped-grep misuse is gone).

Do not touch while iterating: the battery questions, the thresholds, the fixture documents (that would be moving the goal posts — if the battery needs changing, it is a methodology change, say so), the refusal mechanics (a refusal is still a refusal, counts in nothing, consumes a round — phase-72 locked decision), the tool names/argument shapes (ls(path?) / read(path) / grep(pattern, path?) — phase-70 locked surface).

---

5. The two metrics — read this before arguing about the numbers

The verdict carries both:

• contract accuracy = emitted calls that are well-formed and target a resolvable entity ÷ emitted (classify_call in scripts/agent_realmodel_check.py, mirroring app/rag/agent._execute_tool's resolution rules gate-side). A call is a contract violation when the model aimed wrong: unknown tool, missing argument, a bare document path where the combined source/path belongs, a nonexistent document identity, a source name where a document belongs (ls(path='.'), ls(path='/'), grep(path='homelab') — the entire phase-72 incident class).
• executed/emitted (the phase-72 locked metric) = calls the app actually executed ÷ emitted. Every refusal class counts against it — including ALREADY_IN_CONTEXT, the app's dedupe refusal when the model reads a document whose full text is already in the <documents> context.

Why the fixture gate's accuracy bar is contract accuracy, and why this is honest rather than goalpost-moving:

1. The re-read is a correct tool call — right tool, well-formed arguments, a real document identity — that the app declines for redundancy. The phase-72 incident the owner was frustrated by (garbage scopes, loops, cap hits) is exactly the class contract accuracy measures, and it is gone: 0 contract violations in 2 of 3 fixture runs, 3 in the third (one directory-scoped grep('mimir-service', path='deployments/quadlet') exploration that self-corrected via ls in two rounds).
2. The executed ratio is blocked at 58–73 % by the re-reads alone — and §4 shows five independent copy variants failed to change that behavior even once. Gating the fast loop on a number no lever can move would make it permanently red and useless for iteration.
3. Both numbers are always printed. Nothing is hidden; the executed ratio stays the pass bar for the locked derived gate.

The remaining question — should a redundant-but-correct read count as a failure at all? — is an app-semantics decision, not a copy lever (§7).

---

6. The derived gate (phase 72, locked)

--mode derived (the default) runs the phase-72 locked battery — derived from the live catalog's first two documents, including the two bare-path traps (read('ansible/lab-inventory.md') without the source prefix, etc.) — with the phase-72 locked conditions, including executed/emitted ≥ 0.90. Against the fixture KB (2026-09-04):

gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) (wall 47.7s)
gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/14 executed (36%) contract 10/14 (71%) 2026-09-06 (wall 38.3s)

Reading that result: the teaching works — every bare-path trap self-corrected in exactly one round (the did-you-mean refusal named the combined identity, the model used it next round), zero cap hits, 10/10 answered. The executed bar fails because the corrected read then hits ALREADY_IN_CONTEXT — the trap question names the document's topic words, so the document is seeded, and the correct combined-form read is dedupe-refused. Same wall as §5, now on the locked gate: the ≥90 % executed bar is unreachable under the current refusal semantics regardless of copy. The gate runs as-is, unchanged, and reports it.

---

7. Open design question (for the owner)

The only thing standing between the lite model and a ≥90 % executed ratio is one refusal's semantics: ALREADY_IN_CONTEXT. Options, with trade-offs:

1. Keep as-is (phase-72 locked): a redundant read is a refusal, counts in nothing. The model is taught not to re-read; the cost is that the executed metric can't reach 90 % while the model's copy-invariant re-read habit exists. Contract accuracy (the capability metric) is ~100 %.
2. Count an in-context read as executed (return the document, dedupe the context — the holder.read_docs dedupe already makes a re-read a no-op content-wise). The executed metric would jump to ~100 %; the teaching signal weakens (the model never sees the refusal it is being taught by).
3. Hybrid: execute it, but mark the turn redundant_reads=N in the log line and the verdict, keeping the signal without the wall.

The controlled methodology makes this a 50-second experiment either way: change the one branch in app/rag/agent.py::_execute_tool, update its unit pins, run the full fixture loop.

---

8. Reproducing from scratch

# 0. Prereqs: the usual dev setup (AGENTS.md quick reference)
podman compose up -d db
cp .env.example .env                # once; LLM endpoint + DB URL
uv run alembic upgrade head

# 1. Build the controlled KB + dump (one-off, ~2 s — real embeddings)
uv run python -m scripts.load_test_kb
#    → prints the retrieval report (all 10 must be grounded) and
#      verifies the dump by round-trip.

# 2. The loop
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3   # ~12 s micro-loop
uv run python -m scripts.agent_realmodel_check --restore --mode fixture             # ~45 s full gate
uv run python -m scripts.agent_realmodel_check --restore                            # phase-72 locked gate

# 3. After touching the copy levers
uv run pytest tests/unit -q                                             # pins in sync?
uv run pytest --cov=app --cov-report=term-missing | tail -3             # >90 %
uv run ruff check . && uv run pyright
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov          # E2E in isolation

Exit codes, both gate and restore/build: 0 pass/ok, 1 fail (with the per-condition breakdown — the MISS lines name the lever to iterate), 2 precondition (DB down, dump missing, schema not applied — each with the actionable fix on the same line).

Diagnosing a bad run: every call is logged by run_agent (agent tool=… args=… round=…/…) — correlate the arguments with the refusal templates in app/rag/agent.py to see which teaching line the model hit, and which refusal class (contract violation vs. in-context dedupe) the rejection was.