Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agent): record the turbo sanity check on the controlled fixture battery

Browse Source 
2026-09-06 fixture runs: contract 100 %, executed 100 %, wall ~113 s (2 runs). Derived battery: FAIL only on usage floor (5/10 tool-turns) — answers seeded questions from context, which is ideal grounded behavior. Wall time ~2.8× lite (113 s vs 40 s). Model is clean.
This commit is contained in:
ducoterra
2026-09-06 21:49:57 -04:00
parent bf64c0d7e4
commit 70ba8710f3
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TOOL_CALLING_TESTING.md
+5 -2
View File
@@ -219,6 +219,9 @@ chat model switched to `turbo`):**
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s) gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s) gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s) [derived battery — MISS: 5/10 tool-turn floor] gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s) [derived battery — MISS: 5/10 tool-turn floor]
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s) [derived battery — MISS: 5/10 tool-turn floor]
``` ```
Reads: the re-read habit is model-specific. `lite` re-reads a seeded Reads: the re-read habit is model-specific. `lite` re-reads a seeded
@@ -235,8 +238,8 @@ derived battery `turbo` fails only the *usage floor* condition (≥ 6/10
turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target
questions from context instead of making the (refusable) read call the questions from context instead of making the (refusable) read call the
trap design expects; accuracy on every call it does make is still trap design expects; accuracy on every call it does make is still
100 %. The cost: **2–3× slower wall time** (105–135 s per full loop 100 %. The cost: **~2.5× slower wall time** (97–114 s per full loop
vs 43–55 s, with individual slow turns up to ~34 s). vs 40–41 s for `lite`, with individual turns 7–20 s).
--- ---