Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agent): record the turbo comparison on the controlled fixture battery

Browse Source 
turbo (2026-09-05, same fixture KB): fixture gate PASS 100%/100% on both
metrics, two runs (wall 105-135s vs lite 43-55s); the redundant re-read
of seeded documents that capped lite's executed ratio at 58-73% is
model-specific (turbo re-read rate ~12% vs ~100% in-sample), corroborating
section 7's framing. Locked derived battery: turbo fails only the >=6/10
tool-turn usage floor (it answers seeded read-target questions from
context instead of making the refusable read call) - accuracy on all
emitted calls still 100%/100%.
This commit is contained in:
ducoterra
2026-09-05 00:09:55 -04:00
parent 988ff78526
commit ce896ab8ab
1 changed files with 27 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TOOL_CALLING_TESTING.md
+27 -1
View File
@@ -205,11 +205,37 @@ gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (6
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s)
``` ```
Contract accuracy ≥ 90 %: **met** (100 / 92 / 100). The executed Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93). The executed
ratio sits at 58–73 % for the reason documented in §5 — an app ratio sits at 58–73 % for the reason documented in §5 — an app
semantics choice, not a model defect, and the open design question in semantics choice, not a model defect, and the open design question in
§7. §7.
**Model comparison — `turbo` (2026-09-05, same fixture KB, `.env`
chat model switched to `turbo`):**
```
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-05 (wall 105.1s)
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-05 (wall 135.5s)
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-05 (wall 77.1s) [derived battery — MISS: 5/10 tool-turn floor]
```
Reads: the re-read habit is model-specific. `lite` re-reads a seeded
named document ~100 % of the time (copy-invariant, §4); `turbo`
answered 7 of 8 seeded-target questions straight from the
`<documents>` context with **zero** tool calls — the exact
"don't re-read" behavior the copy levers could not buy from `lite`
(one re-read in the sample, 12 % vs ~100 %). Consequence:
`turbo` hits **100 % on both metrics** on the fixture battery — the
executed ratio reaches 100 % naturally once the redundant reads
stop, which corroborates §7's framing (the block on `lite` is the
model's re-read habit, not a gate or app defect). On the locked
derived battery `turbo` fails only the *usage floor* condition (≥ 6/10
turns with ≥ 1 emitted call: 5/10) — it answers the seeded read-target
questions from context instead of making the (refusable) read call the
trap design expects; accuracy on every call it does make is still
100 %. The cost: **2–3× slower wall time** (105–135 s per full loop
vs 43–55 s, with individual slow turns up to ~34 s).
--- ---
## 4. The copy levers (what you iterate) ## 4. The copy levers (what you iterate)