From 70ba8710f36c4b40cac5b9533887a18b214b4234 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Sun, 6 Sep 2026 21:49:57 -0400 Subject: [PATCH] docs(agent): record the turbo sanity check on the controlled fixture battery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2026-09-06 fixture runs: contract 100 %, executed 100 %, wall ~113 s (2 runs). Derived battery: FAIL only on usage floor (5/10 tool-turns) — answers seeded questions from context, which is ideal grounded behavior. Wall time ~2.8× lite (113 s vs 40 s). Model is clean. --- TOOL_CALLING_TESTING.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/TOOL_CALLING_TESTING.md b/TOOL_CALLING_TESTING.md index 05c9645..f5434d7 100644 --- a/TOOL_CALLING_TESTING.md +++ b/TOOL_CALLING_TESTING.md @@ -219,6 +219,9 @@ chat model switched to `turbo`):** gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s) gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s) gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s) [derived battery — MISS: 5/10 tool-turn floor] +gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s) +gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s) +gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s) [derived battery — MISS: 5/10 tool-turn floor] ``` Reads: the re-read habit is model-specific. `lite` re-reads a seeded @@ -235,8 +238,8 @@ derived battery `turbo` fails only the *usage floor* condition (≥ 6/10 turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target questions from context instead of making the (refusable) read call the trap design expects; accuracy on every call it does make is still -100 %. The cost: **2–3× slower wall time** (105–135 s per full loop -vs 43–55 s, with individual slow turns up to ~34 s). +100 %. The cost: **~2.5× slower wall time** (97–114 s per full loop +vs 40–41 s for `lite`, with individual turns 7–20 s). ---