brain-of-reese/TOOL_CALLING_TESTING.md

Tool-Calling Testing Methodology (controlled KB + one-command fast loop)

How to test, measure, and iterate on the agent's tool calling (ls / read / grep) against the real configured chat model (lite per .env) — fast enough to iterate on, controlled enough to trust.

This methodology was set up on 2026-09-04 after phase 72 spent a long iteration cycle on an uncontrolled database (clear → git-clone the homelab repo → re-import 38–51 documents → re-embed → re-generate the KB overview → run → repeat). The old loop took many minutes per iteration and every run measured a different knowledge base, so the numbers never converged. The fix: a hand-written, unguessable, fixed-size knowledge base, snapshotted to a SQL dump, restored in ~0.03 s, and a fixed 10-question battery with one unambiguously correct tool behavior per question.

---

1. The fast loop (one command)

podman compose up -d db                                    # once
uv run python -m scripts.agent_realmodel_check --restore --mode fixture

That is the whole loop:

1. restore the fixture KB from tests/fixtures/test_kb.dump.sql (one transaction — no git clone, no re-embedding, no lite calls; ~0.03 s hot),
2. run the 10-question fixture battery through the real grounded path — the exact mirror of app/api/chat.py: embed → hybrid retrieval → the honesty gate (plan_turn) → the real prompt (persona + KB overview + <documents> + <tools>) → run_agent against the live endpoint,
3. print one line per turn plus the verdict.

Measured timings (2026-09-04, this machine):

step	time
restore fixture KB	0.03 s (0.2 s first run — psycopg connect)
3-turn micro-loop (--turns 3)	~12 s end-to-end (incl. ~1 s uv/python startup)
full 10-turn fixture loop	~43–51 s wall
one-off KB rebuild (real embeddings, 9 chunks)	~1–2 s

Iteration workflow. When tuning the copy levers (§4), do not run the full battery — run the micro-loop on the first three turns (the incident turn + both listing traps, the fastest signal):

uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3

~12 s per variant. Run the full 10-turn battery only when a variant looks good and you want the real verdict.

Timing is visible, by design: every turn line carries its wall seconds and the verdict line carries the run's total wall time, so a slow-down (endpoint load, a retry storm, a copy that makes the model ramble) is visible on the same line as the accuracy:

turn 01 | emitted=1 executed=1 cap=no defl=no |  4.06s | List the files in …
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)

Notes on speed, measured (not guessed):

• --concurrency 2 / --concurrency 3 was tested and does not help: the aipi endpoint serializes generation server-side, so parallel turns finish in the same total wall time (45.5 s @ 3-way vs ~44 s sequential) with the same aggregates. Sequential stays the default for clean telemetry.
• The LLM is ~95 % of the cost (1–3 model rounds per turn at ~2–6 s each). Database work per turn is milliseconds. Don't optimize it.

---

2. The controlled knowledge base

tests/fixtures/agent_kb/
├── deployments/
│   ├── ansible/lab-inventory.md
│   ├── ci/gitlab-runner.md
│   └── quadlet/mimir-service.md
└── homelab/
    ├── backups/restic-rack7.md
    ├── containers/qwen38-llamacpp.md
    ├── containers/uptime-kuma.md
    ├── networking/meridian-notes.md
    └── networking/vela-bridges.md

8 hand-written markdown documents, 2 sources (source name = directory basename, the importer's rule). Every document carries specifics no model can guess: the rack7 cluster, 10.77.42.0/24 and the VLAN 130 lab-iot pool, PVE build 8.3.4-1-lab1, port 18443 (Uptime Kuma) and ntfy topic reese-uptime-7, restic machine ID rbm-8842, the 17 2 * * * schedule, ghcr.io/reese/obsidian-bor:2026.7.14 on 127.0.0.1:18765, the Qwen 3.8 llama.cpp launch line, ansible-core 2.19.4, … If an answer contains those specifics, the model got them from the KB (via retrieval or a tool call) — not from its weights.

Two deliberate design rules:

1. Non-topical file names for the read targets. vela-bridges.md, meridian-notes.md, mimir-service.md carry no words their content repeats. Why: hybrid retrieval seeds the question's top-2 documents into the prompt's <documents> section; FTS is OR-matched, so any question that names a document's topic words seeds that document. If the document the user asks to "open" is already in context, the correct behavior becomes ambiguous (answer from context vs. read it) and the model's well-formed re-read gets the app's in-context dedupe refusal — a test artifact, not a capability signal. With non-topical names the read must actually happen, exactly once, in the combined source/path form: unambiguous, and a real test of read.
2. The grep token is unique. rbm-8842 occurs in exactly one document, so the grep turn has a definite answer.

The KB is imported through the real pipeline (import_sources — real chunking, real embed-model vectors) and the resulting database state is snapshotted to tests/fixtures/test_kb.dump.sql — a data-only SQL script (TRUNCATE + one multi-row INSERT per app table: documents, chunks + embeddings, the git_sources local rows that make the source registry self-contained, the static KB overview, the sources version, and — since the 2026-09-11 phase-94 refresh — the sync-time folder_summaries rows the drill-down ls shows). The build generates those rows against the live lite endpoint through the real sync-time generator (4 rows for this KB: the two source roots + homelab/containers + homelab/networking — the ≥ 2-document folders; single-document folders carry no row by design). Restoring it puts the whole known state back in one transaction; the generated chunks.tsv column is recomputed by Postgres. The dump is verified by round-trip at build time (restore + per-table checksum compare — a serialization bug fails the build).

# Rebuild the KB + dump — only when the fixture documents, the
# chunker, the embedding model, or the folder-summary prompt change
# (its `lite` output is baked in — phase 94). NOT part of the loop.
uv run python -m scripts.load_test_kb
# restores the fixture KB standalone (what --restore runs inline)
uv run python -m scripts.restore_test_kb

The build script also prints a retrieval report — for every battery question, whether the real honesty gate grounds it and which two documents would seed the context. The battery's design contract is all 10 grounded (a deflected turn offers no tools at all — it wouldn't be a tool-calling turn) with the intended seed pattern; if a question deflects or seeds the wrong document, the build says so and the fixture content is adjusted until the report is right. That report is what makes the test design checkable in 2 seconds.

Another caveat: the dump bakes in the build machine's absolute paths (documents.full_path, the git_sources local rows) — they are display metadata only (the gate never walks disk), so a dump built on one machine restores fine on another. If that ever matters, rebuild.

Caveat (measured): vector cosine in an 8-document KB sits at ~0.55–0.68 for generic questions, so one read target (vela-bridges.md) is seeded by cosine even though no FTS token hits it. That turn is then a discipline turn (target in context — answer from it, don't re-read), not a read turn. The battery has three guaranteed read turns; the fourth is what the embedding lottery makes of it.

---

3. The battery and the metrics

The battery (locked for the methodology — don't swap in easier questions)

#	question	tests	expected ideal
1	List the files in this directory.	the phase-72 incident; the top level lists the sources — a full file listing drills them (phase 94)	ls() → drill ls('deployments') + ls('homelab') (+ leaf folders)
2	List the documents you have in the homelab source.	scoped ls by the correct source name; the result is the source's folders (+ counts/summaries) — file lines come from the leaf drills (phase 94)	ls(path='homelab') (+ leaf drills)
3	List every document you have indexed.	the "list everything" expectation is now the whole tree — no single call lists every file (phase 94)	ls() → drill both sources and every folder
4	Open the document homelab/networking/vela-bridges.md …	read, combined form	read('homelab/networking/vela-bridges.md')
5	Read deployments/quadlet/mimir-service.md and summarize it.	read, unseeded target	read(…) (or ls first, then read)
6	Open the document homelab/networking/meridian-notes.md …	read, unseeded target	read(…)
7	Find the exact string "rbm-8842" in your documents …	grep, pattern only	grep(pattern='rbm-8842') — the match line alone answers it
8	Which document has the title "Lab Ansible Inventory"? Summarize it.	title lookup; target IS seeded	answer from context (or ls)
9	What do you know about the qwen 3.8 llama.cpp setup? …	topic lookup; target IS seeded	answer from context
10	List the files in the deployments directory.	source name phrased as a directory; the result is the source's folders — file lines come from the leaf drills (phase 94)	ls(path='deployments') (+ leaf drills)

Questions 4–6 name the full combined identity (no bare-path trap — that is the job of the locked derived battery, §6). Questions 7–9 name content, so their target document is seeded; the correct behavior there is to not re-read what is already in the prompt.

Phase-94 note: the listing questions' (1, 2, 3, 10) correct behavior moved with the owner-permitted ls surface change (2026-09-10, TODO.md L4 — 00_phase.md of 94_ls_tree_drilldown): a full file listing is now a drill sequence (ls() → source → folder; one level per call, the old whole-KB flood is structurally gone), and a folder line's count + sync-time summary is the tree's picture of that subtree. The questions themselves are unchanged (the methodology's locked battery) — only the expected-behavior definition moved.

The four pass conditions

1. all 10 turns answer (no LLMError/MalformedReplyError);
2. zero turns hit the round cap (the incident's loop signature);
3. ≥6 of 10 turns emit ≥1 tool call (the model keeps using tools);
4. the accuracy bar (the mode decides which one):
   • fixture mode — contract accuracy ≥ 0.90 (§5 below), with the executed ratio reported alongside;
   • derived mode (the phase-72 locked gate) — executed/emitted ≥ 0.90, byte-compatible with the phase-72 task file.

Current standing (2026-09-04, lite, fixture KB)

gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%)  contract 11/11 (100%)  (wall 43.4s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%)  contract 12/13 (92%)   (wall 50.6s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%)  contract 11/11 (100%)  (wall 46.8s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 9/12 executed (75%) contract 11/12 (92%) 2026-09-06 (wall 40.4s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/14 executed (64%) contract 13/14 (93%) 2026-09-06 (wall 40.5s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 13/18 executed (72%) contract 18/18 (100%) 2026-09-11 (wall 43.6s)   [phase-94 ls tree]

Contract accuracy ≥ 90 %: met (100 / 92 / 100 / 93 / 92 / 93 / 100). The executed ratio sits at 58–73 % for the reason documented in §5 — an app semantics choice, not a model defect, and the open design question in §7.

Model comparison — turbo (2026-09-05, same fixture KB, .env chat model switched to turbo):

gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s)   [derived battery — MISS: 5/10 tool-turn floor]
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s)   [derived battery — MISS: 5/10 tool-turn floor]
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 18/19 executed (95%) contract 19/19 (100%) 2026-09-11 (wall 98.7s)   [phase-94 ls tree]

Reads: the re-read habit is model-specific. lite re-reads a seeded named document ~100 % of the time (copy-invariant, §4); turbo answered 7 of 8 seeded-target questions straight from the <documents> context with zero tool calls — the exact "don't re-read" behavior the copy levers could not buy from lite (one re-read in the sample, 12 % vs ~100 %). Consequence: turbo hits 100 % on both metrics on the fixture battery — the executed ratio reaches 100 % naturally once the redundant reads stop, which corroborates §7's framing (the block on lite is the model's re-read habit, not a gate or app defect). On the locked derived battery turbo fails only the usage floor condition (≥ 6/10 turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target questions from context instead of making the (refusable) read call the trap design expects; accuracy on every call it does make is still 100 %. The cost: ~2.5× slower wall time (97–114 s per full loop vs 40–41 s for lite, with individual turns 7–20 s).

---

4. The copy levers (what you iterate)

All three are fixed-template constants with byte-pinned unit tests — change the constant, update the pin, run uv run pytest tests/unit -q (~10 s), then the micro-loop:

lever	where	what it teaches
refusal templates	app/rag/agent.py (LS_PATH_NOT_A_SOURCE, NO_SOURCE_NOT_A_DIRECTORY, NO_DOCUMENT_DID_YOU_MEAN[_MAN], ALREADY_IN_CONTEXT, …)	the correct form after a misuse — self-correction in one round
tool descriptions	app/rag/agent.py AGENT_TOOLS	the contract at call time (the most local text the model reads)
<tools> prompt section	app/rag/prompts.py TOOLS_SECTION	the contract up front, every grounded turn

Unit pins to follow the constants: tests/unit/test_agent.py (description + refusal pins), tests/unit/test_prompts.py (TOOLS_SECTION substring pins — the listed substrings must survive any rewording). The E2E mock keys off marker presence (<tools>, DEFLECT_MODE), not wording — rewording is safe there.

What has been tried on this model (2026-09-03 → 04, all measured live) — so the next iteration doesn't repeat it:

variant	re-reads of seeded docs	note
phase-72: mid-paragraph do-not-read rule (TOOLS_SECTION + read description)	15/15 (never flipped)	9 runs, 38–51 doc KBs
leading in-<documents>-section reminder naming the blocks	0/15 flipped	reverted — primed seed paths as ls scopes (incident turn regressed to a cap loop)
front-loaded do-not-read as the read description's first sentence	no improvement	+ one 6-emitted variance spike
per-block note="…do not call read on it" attribute on each <document> header	no improvement	reverted

Conclusion: the re-read of a salient seeded document is copy-invariant behavior of the lite model (it obeys the user's "open it / read it" over every prompt-level rule tried). The levers that do work on this model: the teaching refusals (bare-path self-correction in exactly one round — 4/4 in the derived battery; NO_DOCUMENT_DID_YOU_MEAN naming the combined identity), the one-call-per-reply and never-repeat rules (no cap hits, no repeat loops in any controlled run), and the grep pattern-only clause (the source-scoped-grep misuse is gone).

Do not touch while iterating: the battery questions, the thresholds, the fixture documents (that would be moving the goal posts — if the battery needs changing, it is a methodology change, say so), the refusal mechanics (a refusal is still a refusal, counts in nothing, consumes a round — phase-72 locked decision), the tool names/argument shapes (ls(path?) / read(path) / grep(pattern, path?) — phase-70 locked surface).

---

5. The two metrics — read this before arguing about the numbers

The verdict carries both:

• contract accuracy = emitted calls that are well-formed and target a resolvable entity ÷ emitted (classify_call in scripts/agent_realmodel_check.py, mirroring app/rag/agent._execute_tool's resolution rules gate-side). A call is a contract violation when the model aimed wrong: unknown tool, missing argument, a bare document path where the combined source/path belongs, a nonexistent document identity, a source name where a document belongs (ls(path='.'), ls(path='/'), grep(path='homelab') — the entire phase-72 incident class). Phase 94 moved the ls contract (owner-permitted surface revision): ls(path) is ALSO contract-correct for an EXISTING source/folder — the drill-down; the violation is now an unknown first segment, a bare folder name (no source prefix), or a folder matching no indexed prefix. The gate-side mirror (classify_call in scripts/agent_realmodel_check.py) follows.
• executed/emitted (the phase-72 locked metric) = calls the app actually executed ÷ emitted. Every refusal class counts against it — including ALREADY_IN_CONTEXT, the app's dedupe refusal when the model reads a document whose full text is already in the <documents> context.

Why the fixture gate's accuracy bar is contract accuracy, and why this is honest rather than goalpost-moving:

1. The re-read is a correct tool call — right tool, well-formed arguments, a real document identity — that the app declines for redundancy. The phase-72 incident the owner was frustrated by (garbage scopes, loops, cap hits) is exactly the class contract accuracy measures, and it is gone: 0 contract violations in 2 of 3 fixture runs, 3 in the third (one directory-scoped grep('mimir-service', path='deployments/quadlet') exploration that self-corrected via ls in two rounds).
2. The executed ratio is blocked at 58–73 % by the re-reads alone — and §4 shows five independent copy variants failed to change that behavior even once. Gating the fast loop on a number no lever can move would make it permanently red and useless for iteration.
3. Both numbers are always printed. Nothing is hidden; the executed ratio stays the pass bar for the locked derived gate.

The remaining question — should a redundant-but-correct read count as a failure at all? — is an app-semantics decision, not a copy lever (§7).

---

6. The derived gate (phase 72, locked)

--mode derived (the default) runs the phase-72 locked battery — derived from the live catalog's first two documents, including the two bare-path traps (read('ansible/lab-inventory.md') without the source prefix, etc.) — with the phase-72 locked conditions, including executed/emitted ≥ 0.90. Against the fixture KB (2026-09-04):

gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) (wall 47.7s)
gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/14 executed (36%) contract 10/14 (71%) 2026-09-06 (wall 38.3s)

Reading that result: the teaching works — every bare-path trap self-corrected in exactly one round (the did-you-mean refusal named the combined identity, the model used it next round), zero cap hits, 10/10 answered. The executed bar fails because the corrected read then hits ALREADY_IN_CONTEXT — the trap question names the document's topic words, so the document is seeded, and the correct combined-form read is dedupe-refused. Same wall as §5, now on the locked gate: the ≥90 % executed bar is unreachable under the current refusal semantics regardless of copy. The gate runs as-is, unchanged, and reports it.

---

7. Open design question (for the owner)

The only thing standing between the lite model and a ≥90 % executed ratio is one refusal's semantics: ALREADY_IN_CONTEXT. Options, with trade-offs:

1. Keep as-is (phase-72 locked): a redundant read is a refusal, counts in nothing. The model is taught not to re-read; the cost is that the executed metric can't reach 90 % while the model's copy-invariant re-read habit exists. Contract accuracy (the capability metric) is ~100 %.
2. Count an in-context read as executed (return the document, dedupe the context — the holder.read_docs dedupe already makes a re-read a no-op content-wise). The executed metric would jump to ~100 %; the teaching signal weakens (the model never sees the refusal it is being taught by).
3. Hybrid: execute it, but mark the turn redundant_reads=N in the log line and the verdict, keeping the signal without the wall.

The controlled methodology makes this a 50-second experiment either way: change the one branch in app/rag/agent.py::_execute_tool, update its unit pins, run the full fixture loop.

---

8. Reproducing from scratch

# 0. Prereqs: the usual dev setup (AGENTS.md quick reference)
podman compose up -d db
cp .env.example .env                # once; LLM endpoint + DB URL
uv run alembic upgrade head

# 1. Build the controlled KB + dump (one-off, ~7 s — real embeddings +
#    4 live-lite folder-summary calls since phase 94)
uv run python -m scripts.load_test_kb
#    → prints the retrieval report (all 10 must be grounded) and
#      verifies the dump by round-trip.

# 2. The loop
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3   # ~12 s micro-loop
uv run python -m scripts.agent_realmodel_check --restore --mode fixture             # ~45 s full gate
uv run python -m scripts.agent_realmodel_check --restore                            # phase-72 locked gate

# 3. After touching the copy levers
uv run pytest tests/unit -q                                             # pins in sync?
uv run pytest --cov=app --cov-report=term-missing | tail -3             # >90 %
uv run ruff check . && uv run pyright
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov          # E2E in isolation

Exit codes, both gate and restore/build: 0 pass/ok, 1 fail (with the per-condition breakdown — the MISS lines name the lever to iterate), 2 precondition (DB down, dump missing, schema not applied — each with the actionable fix on the same line).

Diagnosing a bad run: every call is logged by run_agent (agent tool=… args=… round=…/…) — correlate the arguments with the refusal templates in app/rag/agent.py to see which teaching line the model hit, and which refusal class (contract violation vs. in-context dedupe) the rejection was.

---

9. Phase 94 — the ls drill-down tree (2026-09-11)

Phase 94 rebuilt ls as a drill-down tree (owner-permitted tool-surface revision, TODO.md L4 — the ls result format and path semantics changed; the tool name and the read/grep contract are untouched): ls() lists the synced sources with a per-source sync-time summary, ls(source) lists its folders (each with its summary) + its own files, ls(source/folder) drills one level deeper; file lines are capped at 50 with a grep pointer. This section is the controlled battery's verdict on that architecture — the TODO's gate: "ensure accuracy and performance aren't too badly impacted", read per the phase-94 task file as accuracy at or above the recorded baseline (no contract/verdict regression on the 10-question battery) and wall time within ~20% of the baseline total.

Fixture dump refresh (a precondition of this run). The 2026-09-04 dump predates migration 0017 — it had no folder_summaries rows, so the drill-down's summaries would have been absent from the controlled KB. It was rebuilt on 2026-09-11 with uv run python -m scripts.load_test_kb: the same 8 hand-written documents through the real pipeline (real embeddings), the static KB overview + sources version re-stored, and the sync-time folder summaries generated against the live lite endpoint (4 rows: the two source roots + homelab/containers + homelab/networking — the ≥ 2-document folders). The dump now carries the folder_summaries table (TRUNCATE + INSERT + round-trip checksum verification — the house rule); --restore measured 0.03 s hot.

The classifier mirror followed the surface. classify_call (scripts/agent_realmodel_check.py) is documented as the gate-side mirror of app.rag.agent._execute_tool's resolution rules — task 03 changed the ls resolution (a source/folder path is now valid) but the mirror still flagged every folder drill-down as a contract violation. Fixed in this phase: ls(path) is contract-correct for "", a registered source, or an EXISTING source/folder; without the fix the battery would have scored the correct new behavior as inaccurate. (No battery question changed; the §3 expected-behavior notes moved with the owner-permitted surface.)

The runs. Against the live aipi endpoint, the refreshed fixture KB, the real grounded path — two models: the configured chat model (turbo per .env — the verdict) and lite (the .env model temporarily switched, then restored byte-identical — the house procedure of the 2026-09-05 turbo comparison) to check the phase-70/72 standing block, which was recorded under lite. The micro-loop (--turns 3 — the three listing turns) ran twice on turbo first: 12/12 and 13/13 contract (100 %), zero caps — so the full battery went straight to the verdict, with no copy-lever iteration needed (the first full run on each model was green).

$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture
restore: ok in 0.03s (8 docs, 2 sources)
turn 01 | emitted=1 executed=1 cap=no defl=no |  6.80s | List the files in this directory.
turn 02 | emitted=3 executed=3 cap=no defl=no |  9.77s | List the documents you have in the …
turn 03 | emitted=7 executed=7 cap=no defl=no | 14.83s | List every document you have indexed.
turn 04 | emitted=0 executed=0 cap=no defl=no |  8.89s | Open the document …
turn 05 | emitted=1 executed=1 cap=no defl=no | 12.48s | Read …
turn 06 | emitted=1 executed=1 cap=no defl=no | 10.03s | Open the document …
turn 07 | emitted=1 executed=1 cap=no defl=no |  6.85s | Find the exact string "rbm-8842" in …
turn 08 | emitted=0 executed=0 cap=no defl=no |  7.92s | Which document has the title "Lab …
turn 09 | emitted=0 executed=0 cap=no defl=no | 10.90s | What do you know about the qwen 3.8 …
turn 10 | emitted=5 executed=4 cap=no defl=no | 10.08s | List the files in the deployments …
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 18/19 executed (95%) contract 19/19 (100%) 2026-09-11 (wall 98.7s)

(same command; `.env` chat model temporarily `lite`, restored after)
turn 01 | emitted=1 executed=1 cap=no defl=no |  3.82s | List the files in this directory.
turn 02 | emitted=8 executed=5 cap=no defl=no | 10.93s | List the documents you have in the …
turn 03 | emitted=1 executed=1 cap=no defl=no |  2.25s | List every document you have indexed.
turn 04 | emitted=1 executed=0 cap=no defl=no |  3.84s | Open the document …
turn 05 | emitted=3 executed=3 cap=no defl=no |  4.99s | Read …
turn 06 | emitted=1 executed=1 cap=no defl=no |  5.64s | Open the document …
turn 07 | emitted=1 executed=1 cap=no defl=no |  2.25s | Find the exact string "rbm-8842" in …
turn 08 | emitted=1 executed=0 cap=no defl=no |  5.09s | Which document has the title "Lab …
turn 09 | emitted=0 executed=0 cap=no defl=no |  2.93s | What do you know about the qwen 3.8 …
turn 10 | emitted=1 executed=1 cap=no defl=no |  1.75s | List the files in the deployments …
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 13/18 executed (72%) contract 18/18 (100%) 2026-09-11 (wall 43.6s)

Per-turn reading. Both runs show the designed drill-down: the listing turns hop ls() → source → folder (turbo turn 03: top level

• both sources + five leaf folders in 7 calls; lite turn 02: ls('homelab') + all three homelab folders), the read turns keep the combined source/path form (turbo turn 05 drills ls('deployments') → ls('deployments/quadlet') before the read — the "or ls first, then read" branch), the grep turn is pattern-only, and the seeded discipline turns (04/08/09) answer from the <documents> context. Zero cap hits, zero contract violations, 10/10 answered on both models. The non-executed emitted calls are all contract-correct (the contract is 100 %): turbo's one (turn 10) is a second call in a single reply that the one-tool-per-reply rule discards; lite's five are the same discard class (three on turn 02) plus the two documented copy-invariant ALREADY_IN_CONTEXT re-reads of seeded documents (turns 04/08 — §4: 15/15 across every copy variant tried). lite turn 02 also carried one exploratory grep(pattern='homelab') (pattern-only — contract-correct, executed, then the proper folder drills in the following rounds — the same self-correcting exploration class §5 documents pre-94).

Baseline comparison. Against the last recorded baseline per model (the 2026-09-06 runs for turbo; the phase-70/72 standing block for lite):

model	baseline wall	phase-94 wall	Δ wall	baseline contract	phase-94 contract
turbo (configured — the verdict)	112.8–113.7 s	98.7 s	−12.5 % to −13.2 %	100 % (7/7, 9/9)	100 % (19/19)
lite (phase-70/72 standing)	40.4–40.5 s	43.6 s	+7.7–7.9 %	92–100 % (last: 92 / 93)	100 % (18/18)

Both inside the ~20 % band — turbo under its same-model baseline despite executing MORE calls (18 vs 7–9: the drill-down adds rounds, but per-round latency was lower on this day than on the baseline runs — wall is endpoint-load-sensitive, which is why a band, not a point, is the bar), lite +7.7 % — the modest increase the task file expects from a few extra ls hops. Accuracy: contract at or above the same-model baseline on both models; verdict PASS on both; the executed ratios (95 % / 72 %) sit in the documented bands (§4/§5 — discard class + app-semantics, not tool-calling errors).

Conclusion. The phase-94 gate is met: accuracy at or above baseline (100 % contract, PASS, zero cap hits, zero contract violations — on both models) and wall time within the 20 % band on both models (−12.5/−13.2 % same-model turbo, +7.7 % lite standing). The phase-72 incident class is gone under the tree: the flood is structurally impossible (one level per call, 50-line cap) and the listing turns drill cleanly with the folder summaries doing the steering. No copy-lever change was made (nothing regressed to iterate on).

Observation for the owner (not a gate miss). The top level now carries the two source summaries, and on the lite run the two "list everything" turns (01, 03) answered after a single ls() — from the source-level lines — without drilling to the files; turbo drilled (3 / 7 calls on the same turns). The gate measures contract/verdict, which is met either way; if the owner wants listing answers to reach the file level by default, the lever is the ls description / TOOLS_SECTION copy (teach that a folder's file lines appear only after drilling into it) — flagged here rather than acted on unilaterally.

---

10. Phase 118 — summary-seed gate, 2026-09-15 (run 2026-09-16)

Phase 118 re-worked the grounded-turn seed: the <documents> section no longer carries the full text of the top-2 cited documents — it carries the summaries of the top 5 ranked documents (the "start here if these summaries seem right to you" tier), and a document's full text enters the context only through the capped read tool. The fixture battery had to be re-run against the real configured chat model under the re-cut gate semantics (locked A7, owner-confirmed 2026-09-15) before the phase could complete — the phase-70/72/94 tool-copy gate precedent.

Gate semantics changed (locked A7). The phase gate is now condition 1 (all turns answer) AND condition 2 (zero round-cap hits) AND condition 4 (fixture-mode contract accuracy ≥ 0.90). Condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is now the intended fast path, not tool abandonment. The gate script still gates condition 3 on a full (non---turns) run and exits 1 if it is the only miss; the phase verdict reads the four printed conditions under A7, so a run that misses only condition 3 with 1/2/4 green is a PASS (recorded as such, below). The full telemetry is recorded regardless.

Fixture dump rebuild (a legitimate re-run trigger). The import pipeline's summary policy changed (phase 118, task 01 — every document gets a stored summary + one embedded is_summary chunk, markdown included), so the dump was rebuilt with uv run python -m scripts.load_test_kb (the real pipeline). All eight hand-written fixture documents are markdown and each now carries a doc.summary + one is_summary chunk: 8 documents, 17 chunks (9 content + 8 summary). The build verified summary_errors=0, zero failed folder-summary batches, and an all-GROUNDED retrieval report (the honesty gate is untouched by this phase — grounded/deflected is unchanged). The dump is tracked and committed with the phase.

The rebuild also surfaced and fixed a latent dump bug: the _TABLES column lists in scripts/load_test_kb.py predated three migrations, so the snapshot omitted documents.created_at / documents.created_at_manual (phase 106), git_sources.ignore_paths / git_sources.include_hidden, and folder_summaries.manually_edited (phase 97). Restoring such a dump silently re-defaulted created_at to now(), so the build's own round-trip checksum verification FAILED on documents. The column lists now cover every column of each table, and the round-trip verification passes.

The verdict run. uv run python -m scripts.agent_realmodel_check --restore --mode fixture — the configured chat model (turbo per .env), the refreshed fixture KB, the real grounded path, the full 10-question battery (the verdict is always the full battery — --turns N is the copy-iteration micro-loop only):

$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture
restore: ok in 0.04s (8 docs, 2 sources)
turn 01 | emitted=1 executed=1 cap=no defl=no |  9.49s | List the files in this directory.
turn 02 | emitted=4 executed=4 cap=no defl=no | 15.22s | List the documents you have in the …
turn 03 | emitted=8 executed=8 cap=no defl=no | 24.16s | List every document you have indexed.
turn 04 | emitted=1 executed=1 cap=no defl=no | 12.55s | Open the document …
turn 05 | emitted=1 executed=1 cap=no defl=no | 10.45s | Read …
turn 06 | emitted=1 executed=1 cap=no defl=no |  7.99s | Open the document …
turn 07 | emitted=1 executed=1 cap=no defl=no |  6.63s | Find the exact string "rbm-8842" in …
turn 08 | emitted=0 executed=0 cap=no defl=no |  9.92s | Which document has the title "Lab …
turn 09 | emitted=1 executed=1 cap=no defl=no | 10.57s | What do you know about the qwen 3.8 …
turn 10 | emitted=3 executed=3 cap=no defl=no | 11.00s | List the files in the deployments …
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=9 calls 21/21 executed (100%) contract 21/21 (100%) 2026-09-16 (wall 118.1s)

The four conditions (locked A7) and metrics.

condition	gated under A7	result
1. all turns answer	yes	10/10 answered — GREEN
2. zero round-cap hits	yes	caps=0 — GREEN
3. ≥6/10 turns emit ≥1 tool call	no (reported)	9/10 tool-turns
4. contract accuracy ≥ 0.90	yes	21/21 (100%) — GREEN
executed / emitted (reported)	no	21/21 (100%)

Contract line: contract 21/21 (100%). Wall time: 118.1 s. Model: turbo (the configured chat model).

Per-turn reading. The three designed read turns (04/05/06) each emitted exactly one contract-correct read in the combined source/path form — and each executed (the seeded document is now a summary, so read adds its full text rather than being dedupe-refused: 100 % executed, not the ~58–73 % the full-text seed produced). The discipline turns behave exactly as the new contract wants: turn 08 (title lookup, target seeded as a summary) answered straight from the <documents> context with zero tool calls, and turn 09 (the qwen launch arguments) answered from the seeded summary context. The listing turns drill the tree as under phase 94 (turn 03: top level + both sources + the leaf folders in 8 calls). Zero cap hits, zero contract violations, 10/10 answered.

Baseline comparison. Against the phase-94 turbo baseline (98.7 s wall, 100 % contract, 100 % executed on 19 calls): contract and executed are unchanged at 100 %, caps remain 0, and the wall time is 118.1 s (+19.7 %) — inside the ~20 % band the phase-94 gate used as its slowdown tripwire, and within the normal endpoint-load variance of the 97–135 s turbo range recorded in §3. The gate is met on the conditions that gate (1/2/4 all GREEN) and condition 3 is reported (9/10).

read copy — final shape (no copy iteration was needed). Condition 4 did not degrade and condition 2 was never hit, so no --turns micro-loop iteration was required. The final copy is the task-04 rewrite, byte-pinned in tests/unit/test_prompt_lock.py / test_agent.py: the read description in AGENT_TOOLS (app/rag/agent.py) teaches that <documents> shows the summaries of the top-ranked documents (their full texts are NOT in the prompt yet) and that read adds one of them — or any other document — by its combined source/path, and must not re-read an already-read document (that refusal's copy stays accurate: "the full text is already in your prompt"); the read clause of TOOLS_SECTION (app/rag/prompts.py) mirrors it ("The <documents> section holds SUMMARIES — the full text of a suggested document is not in your prompt until you read it"). classify_call needed no re-baselining: a read of a suggested document is a well-formed call targeting a resolvable document — contract-correct under both the old and the new app semantics (its docstring's ALREADY_IN_CONTEXT note already excludes dedupe re-reads from the metric).

Conclusion. The phase-118 summary-seed gate is PASS: conditions 1, 2, and 4 GREEN (condition 3 reported at 9/10, not gated), contract accuracy 100 %, zero cap hits, the full battery all-answered, wall within the ~20 % band. The summary-seed behavior is proven against the real configured chat model — full text enters the context only through the capped read tool, and a summary-only answer is the intended fast path.