brain-of-reese/TOOL_CALLING_TESTING.md

# Tool-Calling Testing Methodology (controlled KB + one-command fast loop)

How to test, measure, and iterate on the agent's tool calling
(`ls` / `read` / `grep`) against the **real configured chat model**
(`lite` per `.env`) — fast enough to iterate on, controlled enough to
trust.

This methodology was set up on 2026-09-04 after phase 72 spent a long
iteration cycle on an uncontrolled database (clear → git-clone the
homelab repo → re-import 38–51 documents → re-embed → re-generate the
KB overview → run → repeat). The old loop took many minutes per
iteration and every run measured a *different* knowledge base, so the
numbers never converged. The fix: **a hand-written, unguessable,
fixed-size knowledge base, snapshotted to a SQL dump, restored in
~0.03 s**, and a fixed 10-question battery with one unambiguously
correct tool behavior per question.

---

## 1. The fast loop (one command)

```bash
podman compose up -d db                                    # once
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
```

That is the whole loop:

1. restore the fixture KB from `tests/fixtures/test_kb.dump.sql`
   (one transaction — **no git clone, no re-embedding, no `lite`
   calls**; ~0.03 s hot),
2. run the 10-question fixture battery through the **real grounded
   path** — the exact mirror of `app/api/chat.py`: embed → hybrid
   retrieval → the honesty gate (`plan_turn`) → the real prompt
   (persona + KB overview + `<documents>` + `<tools>`) → `run_agent`
   against the live endpoint,
3. print one line per turn plus the verdict.

Measured timings (2026-09-04, this machine):

| step | time |
|---|---|
| restore fixture KB | 0.03 s (0.2 s first run — psycopg connect) |
| 3-turn micro-loop (`--turns 3`) | ~12 s end-to-end (incl. ~1 s uv/python startup) |
| full 10-turn fixture loop | ~43–51 s wall |
| one-off KB rebuild (real embeddings, 9 chunks) | ~1–2 s |

**Iteration workflow.** When tuning the copy levers (§4), do not run
the full battery — run the micro-loop on the first three turns (the
incident turn + both listing traps, the fastest signal):

```bash
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3
```

~12 s per variant. Run the full 10-turn battery only when a variant
looks good and you want the real verdict.

**Timing is visible, by design:** every turn line carries its wall
seconds and the verdict line carries the run's total wall time, so a
slow-down (endpoint load, a retry storm, a copy that makes the model
ramble) is visible on the same line as the accuracy:

```
turn 01 | emitted=1 executed=1 cap=no defl=no |  4.06s | List the files in …
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
```

Notes on speed, measured (not guessed):

- `--concurrency 2` / `--concurrency 3` was tested and **does not
  help**: the aipi endpoint serializes generation server-side, so
  parallel turns finish in the same total wall time (45.5 s @ 3-way
  vs ~44 s sequential) with the same aggregates. Sequential stays the
  default for clean telemetry.
- The LLM is ~95 % of the cost (1–3 model rounds per turn at ~2–6 s
  each). Database work per turn is milliseconds. Don't optimize it.

---

## 2. The controlled knowledge base

```
tests/fixtures/agent_kb/
├── deployments/
│   ├── ansible/lab-inventory.md
│   ├── ci/gitlab-runner.md
│   └── quadlet/mimir-service.md
└── homelab/
    ├── backups/restic-rack7.md
    ├── containers/qwen38-llamacpp.md
    ├── containers/uptime-kuma.md
    ├── networking/meridian-notes.md
    └── networking/vela-bridges.md
```

**8 hand-written markdown documents, 2 sources** (source name =
directory basename, the importer's rule). Every document carries
specifics no model can guess: the `rack7` cluster, `10.77.42.0/24`
and the VLAN 130 lab-iot pool, PVE build `8.3.4-1-lab1`, port `18443`
(Uptime Kuma) and ntfy topic `reese-uptime-7`, restic machine ID
`rbm-8842`, the `17 2 * * *` schedule, `ghcr.io/reese/obsidian-bor:2026.7.14`
on `127.0.0.1:18765`, the Qwen 3.8 llama.cpp launch line, ansible-core
`2.19.4`, … If an answer contains those specifics, the model got them
from the KB (via retrieval or a tool call) — not from its weights.

Two deliberate design rules:

1. **Non-topical file names for the `read` targets.**
   `vela-bridges.md`, `meridian-notes.md`, `mimir-service.md` carry no
   words their content repeats. Why: hybrid retrieval seeds the
   question's top-2 documents into the prompt's `<documents>` section;
   FTS is OR-matched, so any question that names a document's topic
   words seeds that document. If the document the user asks to "open"
   is already in context, the *correct* behavior becomes ambiguous
   (answer from context vs. read it) and the model's well-formed
   re-read gets the app's in-context dedupe refusal — a test artifact,
   not a capability signal. With non-topical names the read must
   actually happen, exactly once, in the combined `source/path` form:
   unambiguous, and a real test of `read`.
2. **The grep token is unique.** `rbm-8842` occurs in exactly one
   document, so the `grep` turn has a definite answer.

The KB is imported through the **real pipeline** (`import_sources` —
real chunking, real `embed`-model vectors) and the resulting database
state is snapshotted to **`tests/fixtures/test_kb.dump.sql`** — a
data-only SQL script (TRUNCATE + one multi-row INSERT per app table:
documents, chunks + embeddings, the `git_sources` local rows that make
the source registry self-contained, the static KB overview, the
sources version, and — since the 2026-09-11 phase-94 refresh — the
sync-time `folder_summaries` rows the drill-down `ls` shows). The
build generates those rows against the live `lite` endpoint through
the real sync-time generator (4 rows for this KB: the two source
roots + `homelab/containers` + `homelab/networking` — the ≥ 2-document
folders; single-document folders carry no row by design). Restoring it
puts the whole known state back in one transaction; the generated
`chunks.tsv` column is recomputed by Postgres. The dump is verified by
round-trip at build time (restore + per-table checksum compare — a
serialization bug fails the build).

```bash
# Rebuild the KB + dump — only when the fixture documents, the
# chunker, the embedding model, or the folder-summary prompt change
# (its `lite` output is baked in — phase 94). NOT part of the loop.
uv run python -m scripts.load_test_kb
# restores the fixture KB standalone (what --restore runs inline)
uv run python -m scripts.restore_test_kb
```

The build script also prints a **retrieval report** — for every
battery question, whether the real honesty gate grounds it and which
two documents would seed the context. The battery's design contract
is *all 10 grounded* (a deflected turn offers no tools at all — it
wouldn't be a tool-calling turn) with the intended seed pattern; if a
question deflects or seeds the wrong document, the build says so and
the fixture content is adjusted until the report is right. That report
is what makes the test design *checkable in 2 seconds*.

Another caveat: the dump bakes in the build machine's absolute paths
(`documents.full_path`, the `git_sources` local rows) — they are
display metadata only (the gate never walks disk), so a dump built on
one machine restores fine on another. If that ever matters, rebuild.

Caveat (measured): vector cosine in an 8-document KB sits at
~0.55–0.68 for generic questions, so one read target
(`vela-bridges.md`) is seeded by cosine even though no FTS token
hits it. That turn is then a *discipline turn* (target in context —
answer from it, don't re-read), not a read turn. The battery has three
guaranteed read turns; the fourth is what the embedding lottery makes
of it.

---

## 3. The battery and the metrics

### The battery (locked for the methodology — don't swap in easier questions)

| # | question | tests | expected ideal |
|---|---|---|---|
| 1 | List the files in this directory. | the phase-72 incident; the top level lists the sources — a full file listing drills them (phase 94) | `ls()` → drill `ls('deployments')` + `ls('homelab')` (+ leaf folders) |
| 2 | List the documents you have in the homelab source. | scoped `ls` by the correct source name; the result is the source's folders (+ counts/summaries) — file lines come from the leaf drills (phase 94) | `ls(path='homelab')` (+ leaf drills) |
| 3 | List every document you have indexed. | the "list everything" expectation is now the whole tree — no single call lists every file (phase 94) | `ls()` → drill both sources and every folder |
| 4 | Open the document homelab/networking/vela-bridges.md … | `read`, combined form | `read('homelab/networking/vela-bridges.md')` |
| 5 | Read deployments/quadlet/mimir-service.md and summarize it. | `read`, unseeded target | `read(…)` (or `ls` first, then `read`) |
| 6 | Open the document homelab/networking/meridian-notes.md … | `read`, unseeded target | `read(…)` |
| 7 | Find the exact string "rbm-8842" in your documents … | `grep`, pattern only | `grep(pattern='rbm-8842')` — the match line alone answers it |
| 8 | Which document has the title "Lab Ansible Inventory"? Summarize it. | title lookup; target IS seeded | answer from context (or `ls`) |
| 9 | What do you know about the qwen 3.8 llama.cpp setup? … | topic lookup; target IS seeded | answer from context |
| 10 | List the files in the deployments directory. | source name phrased as a directory; the result is the source's folders — file lines come from the leaf drills (phase 94) | `ls(path='deployments')` (+ leaf drills) |

Questions 4–6 name the **full combined identity** (no bare-path trap —
that is the job of the locked derived battery, §6). Questions 7–9 name
content, so their target document is seeded; the correct behavior
there is to **not** re-read what is already in the prompt.

Phase-94 note: the listing questions' (1, 2, 3, 10) correct behavior
**moved with the owner-permitted `ls` surface change** (2026-09-10,
`TODO.md` L4 — `00_phase.md` of `94_ls_tree_drilldown`): a full file
listing is now a drill sequence (`ls()` → source → folder; one level
per call, the old whole-KB flood is structurally gone), and a folder
line's count + sync-time summary is the tree's picture of that
subtree. The questions themselves are unchanged (the methodology's
locked battery) — only the expected-behavior definition moved.

### The four pass conditions

1. all 10 turns answer (no `LLMError`/`MalformedReplyError`);
2. zero turns hit the round cap (the incident's loop signature);
3. ≥6 of 10 turns emit ≥1 tool call (the model keeps *using* tools);
4. the accuracy bar (the mode decides which one):
   - `fixture` mode — **contract accuracy ≥ 0.90** (§5 below), with
     the executed ratio reported alongside;
   - `derived` mode (the phase-72 locked gate) — **executed/emitted ≥
     0.90**, byte-compatible with the phase-72 task file.

### Current standing (2026-09-04, `lite`, fixture KB)

```
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%)  contract 11/11 (100%)  (wall 43.4s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%)  contract 12/13 (92%)   (wall 50.6s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%)  contract 11/11 (100%)  (wall 46.8s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 9/12 executed (75%) contract 11/12 (92%) 2026-09-06 (wall 40.4s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/14 executed (64%) contract 13/14 (93%) 2026-09-06 (wall 40.5s)
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 13/18 executed (72%) contract 18/18 (100%) 2026-09-11 (wall 43.6s)   [phase-94 ls tree]
```

Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93 / 92 / 93 / 100). The
executed ratio sits at 58–73 % for the reason documented in §5 — an app
semantics choice, not a model defect, and the open design question in
§7.

**Model comparison — `turbo` (2026-09-05, same fixture KB, `.env`
chat model switched to `turbo`):**

```
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s)   [derived battery — MISS: 5/10 tool-turn floor]
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s)   [derived battery — MISS: 5/10 tool-turn floor]
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 18/19 executed (95%) contract 19/19 (100%) 2026-09-11 (wall 98.7s)   [phase-94 ls tree]
```

Reads: the re-read habit is model-specific. `lite` re-reads a seeded
named document ~100 % of the time (copy-invariant, §4); `turbo`
answered 7 of 8 seeded-target questions straight from the
`<documents>` context with **zero** tool calls — the exact
"don't re-read" behavior the copy levers could not buy from `lite`
(one re-read in the sample, 12 % vs ~100 %). Consequence:
`turbo` hits **100 % on both metrics** on the fixture battery — the
executed ratio reaches 100 % naturally once the redundant reads
stop, which corroborates §7's framing (the block on `lite` is the
model's re-read habit, not a gate or app defect). On the locked
derived battery `turbo` fails only the *usage floor* condition (≥ 6/10
turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target
questions from context instead of making the (refusable) read call the
trap design expects; accuracy on every call it does make is still
100 %. The cost: **~2.5× slower wall time** (97–114 s per full loop
vs 40–41 s for `lite`, with individual turns 7–20 s).

---

## 4. The copy levers (what you iterate)

All three are fixed-template constants with byte-pinned unit tests —
change the constant, update the pin, run `uv run pytest tests/unit -q`
(~10 s), then the micro-loop:

| lever | where | what it teaches |
|---|---|---|
| refusal templates | `app/rag/agent.py` (`LS_PATH_NOT_A_SOURCE`, `NO_SOURCE_NOT_A_DIRECTORY`, `NO_DOCUMENT_DID_YOU_MEAN[_MAN]`, `ALREADY_IN_CONTEXT`, …) | the correct form *after* a misuse — self-correction in one round |
| tool descriptions | `app/rag/agent.py` `AGENT_TOOLS` | the contract *at call time* (the most local text the model reads) |
| `<tools>` prompt section | `app/rag/prompts.py` `TOOLS_SECTION` | the contract *up front*, every grounded turn |

Unit pins to follow the constants: `tests/unit/test_agent.py`
(description + refusal pins), `tests/unit/test_prompts.py`
(`TOOLS_SECTION` substring pins — the listed substrings must survive
any rewording). The E2E mock keys off marker *presence* (`<tools>`,
`DEFLECT_MODE`), not wording — rewording is safe there.

**What has been tried on this model (2026-09-03 → 04, all measured
live) — so the next iteration doesn't repeat it:**

| variant | re-reads of seeded docs | note |
|---|---|---|
| phase-72: mid-paragraph do-not-read rule (TOOLS_SECTION + `read` description) | 15/15 (never flipped) | 9 runs, 38–51 doc KBs |
| leading in-`<documents>`-section reminder naming the blocks | 0/15 flipped | **reverted** — primed seed paths as `ls` scopes (incident turn regressed to a cap loop) |
| front-loaded do-not-read as the `read` description's first sentence | no improvement | + one 6-emitted variance spike |
| per-block `note="…do not call read on it"` attribute on each `<document>` header | no improvement | **reverted** |

**Conclusion: the re-read of a salient seeded document is
copy-invariant behavior of the `lite` model** (it obeys the user's
"open it / read it" over every prompt-level rule tried). The levers
that *do* work on this model: the teaching refusals (bare-path
self-correction in exactly one round — 4/4 in the derived battery;
`NO_DOCUMENT_DID_YOU_MEAN` naming the combined identity), the
one-call-per-reply and never-repeat rules (no cap hits, no repeat
loops in any controlled run), and the grep pattern-only clause (the
source-scoped-grep misuse is gone).

**Do not touch while iterating:** the battery questions, the
thresholds, the fixture documents (that would be moving the goal
posts — if the battery needs changing, it is a methodology change,
say so), the refusal *mechanics* (a refusal is still a refusal,
counts in nothing, consumes a round — phase-72 locked decision), the
tool names/argument shapes (`ls(path?)` / `read(path)` /
`grep(pattern, path?)` — phase-70 locked surface).

---

## 5. The two metrics — read this before arguing about the numbers

The verdict carries both:

- **contract accuracy** = emitted calls that are *well-formed and
  target a resolvable entity* ÷ emitted (`classify_call` in
  `scripts/agent_realmodel_check.py`, mirroring
  `app/rag/agent._execute_tool`'s resolution rules gate-side).
  A call is a **contract violation** when the model aimed wrong:
  unknown tool, missing argument, a bare document path where the
  combined `source/path` belongs, a nonexistent document identity, a
  source name where a document belongs (`ls(path='.')`,
  `ls(path='/')`, `grep(path='homelab')` — the entire phase-72
  incident class). Phase 94 moved the `ls` contract (owner-permitted
  surface revision): `ls(path)` is ALSO contract-correct for an
  EXISTING `source/folder` — the drill-down; the violation is now an
  unknown first segment, a bare folder name (no source prefix), or a
  folder matching no indexed prefix. The gate-side mirror
  (`classify_call` in `scripts/agent_realmodel_check.py`) follows.
- **executed/emitted** (the phase-72 locked metric) = calls the app
  actually executed ÷ emitted. Every refusal class counts against it
  — **including `ALREADY_IN_CONTEXT`**, the app's dedupe refusal when
  the model reads a document whose full text is already in the
  `<documents>` context.

Why the fixture gate's accuracy bar is contract accuracy, and why
this is honest rather than goalpost-moving:

1. The re-read is a *correct* tool call — right tool, well-formed
   arguments, a real document identity — that the app declines for
   redundancy. The phase-72 incident the owner was frustrated by
   (garbage scopes, loops, cap hits) is exactly the class contract
   accuracy measures, and it is **gone**: 0 contract violations in 2 of
   3 fixture runs, 3 in the third (one directory-scoped
   `grep('mimir-service', path='deployments/quadlet')` exploration
   that self-corrected via `ls` in two rounds).
2. The executed ratio is blocked at 58–73 % by the re-reads alone —
   and §4 shows five independent copy variants failed to change that
   behavior even once. Gating the fast loop on a number no lever can
   move would make it permanently red and useless for iteration.
3. Both numbers are always printed. Nothing is hidden; the executed
   ratio stays the pass bar for the locked derived gate.

The remaining question — should a redundant-but-correct read count as
a *failure* at all? — is an app-semantics decision, not a copy lever
(§7).

---

## 6. The derived gate (phase 72, locked)

`--mode derived` (the default) runs the phase-72 locked battery —
derived from the live catalog's first two documents, including the two
**bare-path traps** (`read('ansible/lab-inventory.md')` without the
source prefix, etc.) — with the phase-72 locked conditions, including
executed/emitted ≥ 0.90. Against the fixture KB (2026-09-04):

```
gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) (wall 47.7s)
gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/14 executed (36%) contract 10/14 (71%) 2026-09-06 (wall 38.3s)
```

Reading that result: the teaching works — **every bare-path trap
self-corrected in exactly one round** (the did-you-mean refusal named
the combined identity, the model used it next round), zero cap hits,
10/10 answered. The executed bar fails because the corrected read then
hits `ALREADY_IN_CONTEXT` — the trap question names the document's
topic words, so the document is seeded, and the *correct* combined-form
read is dedupe-refused. Same wall as §5, now on the locked gate:
the ≥90 % executed bar is unreachable under the current refusal
semantics regardless of copy. The gate runs as-is, unchanged, and
reports it.

---

## 7. Open design question (for the owner)

The only thing standing between the `lite` model and a ≥90 %
**executed** ratio is one refusal's semantics: `ALREADY_IN_CONTEXT`.
Options, with trade-offs:

1. **Keep as-is** (phase-72 locked): a redundant read is a refusal,
   counts in nothing. The model is *taught* not to re-read; the cost
   is that the executed metric can't reach 90 % while the model's
   copy-invariant re-read habit exists. Contract accuracy (the
   capability metric) is ~100 %.
2. **Count an in-context read as executed** (return the document,
   dedupe the context — the `holder.read_docs` dedupe already makes a
   re-read a no-op content-wise). The executed metric would jump to
   ~100 %; the teaching signal weakens (the model never sees the
   refusal it is being taught by).
3. **Hybrid**: execute it, but mark the turn `redundant_reads=N` in
   the log line and the verdict, keeping the signal without the wall.

The controlled methodology makes this a 50-second experiment either
way: change the one branch in `app/rag/agent.py::_execute_tool`,
update its unit pins, run the full fixture loop.

---

## 8. Reproducing from scratch

```bash
# 0. Prereqs: the usual dev setup (AGENTS.md quick reference)
podman compose up -d db
cp .env.example .env                # once; LLM endpoint + DB URL
uv run alembic upgrade head

# 1. Build the controlled KB + dump (one-off, ~7 s — real embeddings +
#    4 live-lite folder-summary calls since phase 94)
uv run python -m scripts.load_test_kb
#    → prints the retrieval report (all 10 must be grounded) and
#      verifies the dump by round-trip.

# 2. The loop
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3   # ~12 s micro-loop
uv run python -m scripts.agent_realmodel_check --restore --mode fixture             # ~45 s full gate
uv run python -m scripts.agent_realmodel_check --restore                            # phase-72 locked gate

# 3. After touching the copy levers
uv run pytest tests/unit -q                                             # pins in sync?
uv run pytest --cov=app --cov-report=term-missing | tail -3             # >90 %
uv run ruff check . && uv run pyright
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov          # E2E in isolation
```

Exit codes, both gate and restore/build: **0** pass/ok, **1** fail
(with the per-condition breakdown — the MISS lines name the lever to
iterate), **2** precondition (DB down, dump missing, schema not
applied — each with the actionable fix on the same line).

Diagnosing a bad run: every call is logged by `run_agent`
(`agent tool=… args=… round=…/…`) — correlate the arguments with the
refusal templates in `app/rag/agent.py` to see which teaching line the
model hit, and which refusal class (contract violation vs. in-context
dedupe) the rejection was.

---

## 9. Phase 94 — the `ls` drill-down tree (2026-09-11)

Phase 94 rebuilt `ls` as a drill-down tree (owner-permitted
tool-surface revision, `TODO.md` L4 — the `ls` result format and
`path` semantics changed; the tool name and the `read`/`grep`
contract are untouched): `ls()` lists the synced sources with a
per-source sync-time summary, `ls(source)` lists its folders (each
with its summary) + its own files, `ls(source/folder)` drills one
level deeper; file lines are capped at 50 with a grep pointer. This
section is the controlled battery's verdict on that architecture —
the TODO's gate: "ensure accuracy and performance aren't too badly
impacted", read per the phase-94 task file as **accuracy at or above
the recorded baseline** (no contract/verdict regression on the
10-question battery) and **wall time within ~20% of the baseline
total**.

**Fixture dump refresh (a precondition of this run).** The 2026-09-04
dump predates migration 0017 — it had no `folder_summaries` rows, so
the drill-down's summaries would have been absent from the controlled
KB. It was rebuilt on 2026-09-11 with
`uv run python -m scripts.load_test_kb`: the same 8 hand-written
documents through the real pipeline (real embeddings), the static KB
overview + sources version re-stored, and the sync-time folder
summaries generated against the live `lite` endpoint (4 rows: the two
source roots + `homelab/containers` + `homelab/networking` — the
≥ 2-document folders). The dump now carries the `folder_summaries`
table (TRUNCATE + INSERT + round-trip checksum verification — the
house rule); `--restore` measured **0.03 s** hot.

**The classifier mirror followed the surface.** `classify_call`
(`scripts/agent_realmodel_check.py`) is documented as the gate-side
mirror of `app.rag.agent._execute_tool`'s resolution rules — task 03
changed the `ls` resolution (a `source/folder` path is now valid) but
the mirror still flagged every folder drill-down as a contract
violation. Fixed in this phase: `ls(path)` is contract-correct for
`""`, a registered source, or an EXISTING `source/folder`; without
the fix the battery would have scored the *correct* new behavior as
inaccurate. (No battery question changed; the §3 expected-behavior
notes moved with the owner-permitted surface.)

**The runs.** Against the live aipi endpoint, the refreshed fixture
KB, the real grounded path — two models: the **configured** chat
model (`turbo` per `.env` — the verdict) and `lite` (the `.env` model
temporarily switched, then restored byte-identical — the house
procedure of the 2026-09-05 turbo comparison) to check the
phase-70/72 standing block, which was recorded under `lite`. The
micro-loop (`--turns 3` — the three listing turns) ran twice on
`turbo` first: 12/12 and 13/13 contract (100 %), zero caps — so the
full battery went straight to the verdict, with no copy-lever
iteration needed (the first full run on each model was green).

```
$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture
restore: ok in 0.03s (8 docs, 2 sources)
turn 01 | emitted=1 executed=1 cap=no defl=no |  6.80s | List the files in this directory.
turn 02 | emitted=3 executed=3 cap=no defl=no |  9.77s | List the documents you have in the …
turn 03 | emitted=7 executed=7 cap=no defl=no | 14.83s | List every document you have indexed.
turn 04 | emitted=0 executed=0 cap=no defl=no |  8.89s | Open the document …
turn 05 | emitted=1 executed=1 cap=no defl=no | 12.48s | Read …
turn 06 | emitted=1 executed=1 cap=no defl=no | 10.03s | Open the document …
turn 07 | emitted=1 executed=1 cap=no defl=no |  6.85s | Find the exact string "rbm-8842" in …
turn 08 | emitted=0 executed=0 cap=no defl=no |  7.92s | Which document has the title "Lab …
turn 09 | emitted=0 executed=0 cap=no defl=no | 10.90s | What do you know about the qwen 3.8 …
turn 10 | emitted=5 executed=4 cap=no defl=no | 10.08s | List the files in the deployments …
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 18/19 executed (95%) contract 19/19 (100%) 2026-09-11 (wall 98.7s)
```

```
(same command; `.env` chat model temporarily `lite`, restored after)
turn 01 | emitted=1 executed=1 cap=no defl=no |  3.82s | List the files in this directory.
turn 02 | emitted=8 executed=5 cap=no defl=no | 10.93s | List the documents you have in the …
turn 03 | emitted=1 executed=1 cap=no defl=no |  2.25s | List every document you have indexed.
turn 04 | emitted=1 executed=0 cap=no defl=no |  3.84s | Open the document …
turn 05 | emitted=3 executed=3 cap=no defl=no |  4.99s | Read …
turn 06 | emitted=1 executed=1 cap=no defl=no |  5.64s | Open the document …
turn 07 | emitted=1 executed=1 cap=no defl=no |  2.25s | Find the exact string "rbm-8842" in …
turn 08 | emitted=1 executed=0 cap=no defl=no |  5.09s | Which document has the title "Lab …
turn 09 | emitted=0 executed=0 cap=no defl=no |  2.93s | What do you know about the qwen 3.8 …
turn 10 | emitted=1 executed=1 cap=no defl=no |  1.75s | List the files in the deployments …
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 13/18 executed (72%) contract 18/18 (100%) 2026-09-11 (wall 43.6s)
```

**Per-turn reading.** Both runs show the designed drill-down: the
listing turns hop `ls()` → source → folder (turbo turn 03: top level
+ both sources + five leaf folders in 7 calls; lite turn 02:
`ls('homelab')` + all three homelab folders), the read turns keep the
combined `source/path` form (turbo turn 05 drills
`ls('deployments')` → `ls('deployments/quadlet')` before the read —
the "or `ls` first, then `read`" branch), the grep turn is
pattern-only, and the seeded discipline turns (04/08/09) answer from
the `<documents>` context. Zero cap hits, zero contract violations,
10/10 answered on both models. The non-executed emitted calls are all
contract-correct (the contract is 100 %): turbo's one (turn 10) is a
second call in a single reply that the one-tool-per-reply rule
discards; lite's five are the same discard class (three on turn 02)
plus the two documented copy-invariant `ALREADY_IN_CONTEXT` re-reads
of seeded documents (turns 04/08 — §4: 15/15 across every copy
variant tried). lite turn 02 also carried one exploratory
`grep(pattern='homelab')` (pattern-only — contract-correct, executed,
then the proper folder drills in the following rounds — the same
self-correcting exploration class §5 documents pre-94).

**Baseline comparison.** Against the last recorded baseline per model
(the 2026-09-06 runs for `turbo`; the phase-70/72 standing block for
`lite`):

| model | baseline wall | phase-94 wall | Δ wall | baseline contract | phase-94 contract |
|---|---|---|---|---|---|
| `turbo` (configured — the verdict) | 112.8–113.7 s | 98.7 s | −12.5 % to −13.2 % | 100 % (7/7, 9/9) | 100 % (19/19) |
| `lite` (phase-70/72 standing) | 40.4–40.5 s | 43.6 s | +7.7–7.9 % | 92–100 % (last: 92 / 93) | 100 % (18/18) |

Both inside the ~20 % band — `turbo` *under* its same-model baseline
despite executing MORE calls (18 vs 7–9: the drill-down adds rounds,
but per-round latency was lower on this day than on the baseline runs
— wall is endpoint-load-sensitive, which is why a band, not a point,
is the bar), `lite` +7.7 % — the modest increase the task file
expects from a few extra `ls` hops. Accuracy: contract at or above
the same-model baseline on both models; verdict PASS on both; the
executed ratios (95 % / 72 %) sit in the documented bands (§4/§5 —
discard class + app-semantics, not tool-calling errors).

**Conclusion.** The phase-94 gate is **met**: accuracy at or above
baseline (100 % contract, PASS, zero cap hits, zero contract
violations — on both models) and wall time within the 20 % band on
both models (−12.5/−13.2 % same-model `turbo`, +7.7 % `lite`
standing). The phase-72 incident class is gone under the tree: the
flood is structurally impossible (one level per call, 50-line cap)
and the listing turns drill cleanly with the folder summaries doing
the steering. No copy-lever change was made (nothing regressed to
iterate on).

**Observation for the owner (not a gate miss).** The top level now
carries the two source summaries, and on the `lite` run the two
"list everything" turns (01, 03) answered after a single `ls()` —
from the source-level lines — without drilling to the files; `turbo`
drilled (3 / 7 calls on the same turns). The gate measures
contract/verdict, which is met either way; if the owner wants
listing answers to reach the file level by default, the lever is the
`ls` description / `TOOLS_SECTION` copy (teach that a folder's file
lines appear only after drilling into it) — flagged here rather than
acted on unilaterally.

---

## 10. Phase 118 — summary-seed gate, 2026-09-15 (run 2026-09-16)

Phase 118 re-worked the grounded-turn seed: the `<documents>` section
no longer carries the **full text** of the top-2 cited documents — it
carries the **summaries** of the top 5 ranked documents (the
"start here if these summaries seem right to you" tier), and a
document's full text enters the context only through the capped `read`
tool. The fixture battery had to be re-run against the real configured
chat model under the re-cut gate semantics (locked A7, owner-confirmed
2026-09-15) before the phase could complete — the phase-70/72/94
tool-copy gate precedent.

**Gate semantics changed (locked A7).** The phase gate is now
condition **1** (all turns answer) AND condition **2** (zero round-cap
hits) AND condition **4** (fixture-mode contract accuracy ≥ 0.90).
Condition **3** (≥6/10 turns emit ≥1 tool call) is **REPORTED, not
gated** — a summary-only answer is now the *intended* fast path, not
tool abandonment. The gate script still gates condition 3 on a full
(non-`--turns`) run and exits 1 if it is the only miss; the phase
verdict reads the four printed conditions under A7, so a run that
misses **only** condition 3 with 1/2/4 green is a **PASS** (recorded
as such, below). The full telemetry is recorded regardless.

**Fixture dump rebuild (a legitimate re-run trigger).** The import
pipeline's summary policy changed (phase 118, task 01 — every document
gets a stored summary + one embedded `is_summary` chunk, markdown
included), so the dump was rebuilt with
`uv run python -m scripts.load_test_kb` (the real pipeline). All eight
hand-written fixture documents are markdown and each now carries a
`doc.summary` + one `is_summary` chunk: **8 documents, 17 chunks**
(9 content + 8 summary). The build verified
`summary_errors=0`, zero failed folder-summary batches, and an
all-GROUNDED retrieval report (the honesty gate is untouched by this
phase — grounded/deflected is unchanged). The dump is tracked and
committed with the phase.

The rebuild also surfaced and fixed a latent dump bug: the
`_TABLES` column lists in `scripts/load_test_kb.py` predated three
migrations, so the snapshot omitted `documents.created_at` /
`documents.created_at_manual` (phase 106),
`git_sources.ignore_paths` / `git_sources.include_hidden`, and
`folder_summaries.manually_edited` (phase 97). Restoring such a dump
silently re-defaulted `created_at` to `now()`, so the build's own
round-trip checksum verification FAILED on `documents`. The column
lists now cover every column of each table, and the round-trip
verification passes.

**The verdict run.** `uv run python -m scripts.agent_realmodel_check
--restore --mode fixture` — the configured chat model (`turbo` per
`.env`), the refreshed fixture KB, the real grounded path, the full
10-question battery (the verdict is always the full battery —
`--turns N` is the copy-iteration micro-loop only):

```
$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture
restore: ok in 0.04s (8 docs, 2 sources)
turn 01 | emitted=1 executed=1 cap=no defl=no |  9.49s | List the files in this directory.
turn 02 | emitted=4 executed=4 cap=no defl=no | 15.22s | List the documents you have in the …
turn 03 | emitted=8 executed=8 cap=no defl=no | 24.16s | List every document you have indexed.
turn 04 | emitted=1 executed=1 cap=no defl=no | 12.55s | Open the document …
turn 05 | emitted=1 executed=1 cap=no defl=no | 10.45s | Read …
turn 06 | emitted=1 executed=1 cap=no defl=no |  7.99s | Open the document …
turn 07 | emitted=1 executed=1 cap=no defl=no |  6.63s | Find the exact string "rbm-8842" in …
turn 08 | emitted=0 executed=0 cap=no defl=no |  9.92s | Which document has the title "Lab …
turn 09 | emitted=1 executed=1 cap=no defl=no | 10.57s | What do you know about the qwen 3.8 …
turn 10 | emitted=3 executed=3 cap=no defl=no | 11.00s | List the files in the deployments …
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=9 calls 21/21 executed (100%) contract 21/21 (100%) 2026-09-16 (wall 118.1s)
```

**The four conditions (locked A7) and metrics.**

| condition | gated under A7 | result |
|---|---|---|
| 1. all turns answer | yes | 10/10 answered — **GREEN** |
| 2. zero round-cap hits | yes | caps=0 — **GREEN** |
| 3. ≥6/10 turns emit ≥1 tool call | **no (reported)** | 9/10 tool-turns |
| 4. contract accuracy ≥ 0.90 | yes | 21/21 (100%) — **GREEN** |
| executed / emitted (reported) | no | 21/21 (100%) |

Contract line: **contract 21/21 (100%)**. Wall time: **118.1 s**.
Model: **turbo** (the configured chat model).

**Per-turn reading.** The three designed read turns (04/05/06) each
emitted exactly one contract-correct `read` in the combined
`source/path` form — and each **executed** (the seeded document is now
a *summary*, so `read` adds its full text rather than being
dedupe-refused: 100 % executed, not the ~58–73 % the full-text seed
produced). The discipline turns behave exactly as the new contract
wants: turn 08 (title lookup, target seeded as a summary) answered
straight from the `<documents>` context with **zero** tool calls, and
turn 09 (the qwen launch arguments) answered from the seeded summary
context. The listing turns drill the tree as under phase 94 (turn 03:
top level + both sources + the leaf folders in 8 calls). Zero cap
hits, zero contract violations, 10/10 answered.

**Baseline comparison.** Against the phase-94 `turbo` baseline
(98.7 s wall, 100 % contract, 100 % executed on 19 calls): contract
and executed are unchanged at 100 %, caps remain 0, and the wall time
is 118.1 s (+19.7 %) — inside the ~20 % band the phase-94 gate used
as its slowdown tripwire, and within the normal endpoint-load
variance of the 97–135 s `turbo` range recorded in §3. The gate is
met on the conditions that gate (1/2/4 all GREEN) and condition 3 is
reported (9/10).

**`read` copy — final shape (no copy iteration was needed).**
Condition 4 did not degrade and condition 2 was never hit, so no
`--turns` micro-loop iteration was required. The final copy is the
task-04 rewrite, byte-pinned in `tests/unit/test_prompt_lock.py` /
`test_agent.py`: the `read` description in `AGENT_TOOLS`
(`app/rag/agent.py`) teaches that `<documents>` shows the *summaries*
of the top-ranked documents (their full texts are NOT in the prompt
yet) and that `read` adds one of them — or any other document — by
its combined `source/path`, and must not re-read an already-read
document (that refusal's copy stays accurate: "the full text is
already in your prompt"); the `read` clause of `TOOLS_SECTION`
(`app/rag/prompts.py`) mirrors it ("The `<documents>` section holds
SUMMARIES — the full text of a suggested document is not in your
prompt until you `read` it"). `classify_call` needed **no**
re-baselining: a `read` of a suggested document is a well-formed call
targeting a resolvable document — contract-correct under both the old
and the new app semantics (its docstring's `ALREADY_IN_CONTEXT` note
already excludes dedupe re-reads from the metric).

**Conclusion.** The phase-118 summary-seed gate is **PASS**: conditions
1, 2, and 4 GREEN (condition 3 reported at 9/10, not gated), contract
accuracy 100 %, zero cap hits, the full battery all-answered, wall
within the ~20 % band. The summary-seed behavior is proven against the
real configured chat model — full text enters the context only through
the capped `read` tool, and a summary-only answer is the intended fast
path.