From bf64c0d7e4077801486717063d5ac4b893927b13 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Sun, 6 Sep 2026 21:42:14 -0400 Subject: [PATCH] docs(agent): record the lite comparison on the controlled fixture battery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2026-09-06 fixture runs: contract 92–93 %, executed 64–75 %, wall ~40.5 s (2 runs). Derived battery: FAIL, 36 % executed (38.3 s). Same pattern — copy-invariant re-read habit blocks the ≥90 % executed bar under current ALREADY_IN_CONTEXT refusal semantics. Model is working correctly; the bottleneck is the app's dedupe refusal, not the model. --- TOOL_CALLING_TESTING.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/TOOL_CALLING_TESTING.md b/TOOL_CALLING_TESTING.md index c30927e..05c9645 100644 --- a/TOOL_CALLING_TESTING.md +++ b/TOOL_CALLING_TESTING.md @@ -203,9 +203,11 @@ there is to **not** re-read what is already in the prompt. gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) (wall 43.4s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) (wall 50.6s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s) +gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 9/12 executed (75%) contract 11/12 (92%) 2026-09-06 (wall 40.4s) +gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/14 executed (64%) contract 13/14 (93%) 2026-09-06 (wall 40.5s) ``` -Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93). The executed +Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93 / 92 / 93). The executed ratio sits at 58–73 % for the reason documented in §5 — an app semantics choice, not a model defect, and the open design question in §7. @@ -340,6 +342,7 @@ executed/emitted ≥ 0.90. Against the fixture KB (2026-09-04): ``` gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) (wall 47.7s) +gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/14 executed (36%) contract 10/14 (71%) 2026-09-06 (wall 38.3s) ``` Reading that result: the teaching works — **every bare-path trap