Experimental/brain-of-reese

Fork 0

Files

T

ducoterra 988ff78526

Build and Push Containers / build-and-push-app (push) Successful in 1m51s

Details

Build and Push Containers / build-and-push-db (push) Successful in 14s

Details

fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling

Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).

2026-09-04 13:11:07 -04:00

2.3 KiB

Raw Blame History

Task 03 — `TOOLS_SECTION` Copy: State the Contract Up Front

Phase: 72_teaching_refusals · Story: .agent/user_stories/agent-document-tools.md

Objective

The HIGH prompt's <tools> section says the same two things the new refusals teach — the ls path is a source name, not a directory or file path, and read/grep need the combined source/path string including the source name — so the model carries the contract before it calls a tool, not only after being refused.

Work

app/rag/prompts.py — TOOLS_SECTION rewritten (the E2E mock keys off the <tools> marker's presence, not this wording, so the change is mock-safe):
- ls clause: its optional path argument is a source name (e.g. 'homelab') — not a directory or file path; omit it to list every document.
- read clause: the combined source/path string, exactly as shown in the ls output — including the source name; a bare document path will not resolve.
- grep clause: the locator copy stays (its path is already described as a combined source/path string); add the same bare-path-will-not-resolve note.
- Keep the section's shape: a single paragraph between <tools> and </tools>, still appended after the mode body in the HIGH prompt only (the LOW/deflection prompt never carries it — phase 71's plain-text line stays put).
tests/unit/test_prompts.py — update the TOOLS_SECTION wording pin(s) where they pin the old wording; the <tools>-marker-present-in-HIGH pin, the marker-absent-from-LOW pin, and the byte-identical-LOW-prompt pin stay green as-is.

Testing & Quality

Unit: tests/unit/test_prompts.py — marker present in the HIGH prompt and absent from the LOW prompt; the LOW prompt byte-identical to today; the new wording pinned for the ls source-name clause and the read combined-identity clause.
Coverage: >90% on this task's modified code (the constant itself — the builders are already covered).

Completion Criteria

uv run pytest tests/unit/test_prompts.py -v --no-cov green
The HIGH prompt still ends with the <tools> section (existing section-order pin green); the LOW/deflection prompt is byte-identical to today
uv run ruff check . && uv run pyright clean

2.3 KiB Raw Blame History

Task 03 — TOOLS_SECTION Copy: State the Contract Up Front