diff --git a/TOOL_CALLING_TESTING.md b/TOOL_CALLING_TESTING.md index 07d5e7a..c30927e 100644 --- a/TOOL_CALLING_TESTING.md +++ b/TOOL_CALLING_TESTING.md @@ -205,11 +205,37 @@ gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (6 gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s) ``` -Contract accuracy ≥ 90 %: **met** (100 / 92 / 100). The executed +Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93). The executed ratio sits at 58–73 % for the reason documented in §5 — an app semantics choice, not a model defect, and the open design question in §7. +**Model comparison — `turbo` (2026-09-05, same fixture KB, `.env` +chat model switched to `turbo`):** + +``` +gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s) +gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s) +gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s) [derived battery — MISS: 5/10 tool-turn floor] +``` + +Reads: the re-read habit is model-specific. `lite` re-reads a seeded +named document ~100 % of the time (copy-invariant, §4); `turbo` +answered 7 of 8 seeded-target questions straight from the +`` context with **zero** tool calls — the exact +"don't re-read" behavior the copy levers could not buy from `lite` +(one re-read in the sample, 12 % vs ~100 %). Consequence: +`turbo` hits **100 % on both metrics** on the fixture battery — the +executed ratio reaches 100 % naturally once the redundant reads +stop, which corroborates §7's framing (the block on `lite` is the +model's re-read habit, not a gate or app defect). On the locked +derived battery `turbo` fails only the *usage floor* condition (≥ 6/10 +turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target +questions from context instead of making the (refusable) read call the +trap design expects; accuracy on every call it does make is still +100 %. The cost: **2–3× slower wall time** (105–135 s per full loop +vs 43–55 s, with individual slow turns up to ~34 s). + --- ## 4. The copy levers (what you iterate)