brain-of-reese/.agents/phases/complete/72_teaching_refusals/00_phase.md

Phase 72 — Teaching Refusals: End the Post-Harness Tool-Loop Rambling

Story: .agents/user_stories/agent-document-tools.md (this phase repairs the model-facing contract the phase-70 tools reshaped) Context:

• app/rag/agent.py — AGENT_TOOLS (the phase-70 ls / read / grep OpenAI function definitions), _execute_tool (the refusal strings: "No source named '…' — check the ls output.", "No document at '…' — check the ls output."), all_documents (catalog-order bulk loader — reused by the suggestion lookup).
• app/rag/prompts.py — TOOLS_SECTION (HIGH prompt only; the E2E mock keys off the <tools> marker's presence, not its wording).
• tests/unit/test_agent.py (refusal-string pins; the ScriptedLLM + monkeypatched- accessor pattern), tests/integration/test_agent_tools.py (the same pins against real Postgres, kb/src fixtures).
• tests/e2e/mock_llm.py — the deterministic mock tool flows (TOOLS_TRIGGER single-read, MULTI_READ_TRIGGER, SEARCH_TRIGGER; _CATALOG_LINE_RE catalog-line parse) and the dedicated-suite-per-phase E2E house pattern.
• scripts/llm_probe.py — the house live-endpoint probe pattern (python -m scripts.…, argparse, dotenv, printed verdict line); app/api/chat.py — the grounded path the real-model gate mirrors (retrieve → select_documents → build_high_prompt → run_agent)
• Incident (owner chat, 2026-09-03, post phase 70/71): the question "list the files in this directory" produced a Thinking-display trace of the model calling ls(path='app/rag/importer.py') → "No source named 'app/rag/importer.py' — check the ls output.", then ls(path='.') → the same-style refusal, then re-reasoning the same paragraphs over and over across rounds (each round's reasoning_content appends to the open Thinking block) before finally answering from the seed documents alone. Root cause: the harness-trained prior (ls's path = a directory to list) collides with this app's contract (path = a source-name filter), and the terse refusal does not correct the misunderstanding, so the model burns rounds. The identical trap awaits read/grep: a bare document path missing the source prefix (read('app/rag/importer.py')) → "No document at '…'" with no hint of the combined form.

Objective

Make the affected tool refusals teaching so the harness-prior misuse self-corrects in at most one extra round: a scoped ls whose path looks like a document path (contains /) or names an unknown source gets a fixed-template refusal that states the correct contract; a read / scoped-grep argument that resolves to no combined identity but matches an indexed document's path (exact or suffix) gets a "did you mean 'source/path'?" refusal naming the exact combined identity to use. The AGENT_TOOLS path descriptions and the TOOLS_SECTION prompt copy say the same contract up front. Deterministic only — no model participates in detection or repair; the phase-70 harness shape (ls / read(path) / grep(pattern, path?)) is unchanged verbatim. The phase does not pass on mocks alone: a live acceptance gate runs the fixed question battery through run_agent against the real configured chat model (lite per .env) and must PASS before the commit (owner directive, 2026-09-03 — "test with the real lite model until tool calls work consistently; don't pass until a sufficient number of tool calls succeed").

Dependencies

• 70_harness_aligned_tools (complete) — the tool surface this phase teaches (shape untouched).
• 71_scaffolding_guardrails (complete) — the deterministic-guardrail house style this phase follows.

Tasks

1. 01_ls_teaching_refusal.md — ls: path-like and unknown-source scopes get teaching refusals; the ls path description says "source name, not a file or directory path".
2. 02_read_grep_path_suggestion.md — read / grep: an unresolved argument that matches an indexed document path gets the "did you mean 'source/path'?" suggestion; descriptions updated.
3. 03_prompt_copy.md — TOOLS_SECTION copy: the ls path is a source name, not a directory; read/grep need the combined identity including the source name.
4. 04_mock_e2e.md — mock ls-misuse flow, dedicated E2E suite (green in isolation).
5. 05_real_model_gate.md — the live real-lite acceptance gate (scripts/agent_realmodel_check.py): iterate the copy levers until the gate PASSES, then full gates and the commit.

Testing & Quality

• Unit: tests/unit/test_agent.py — the new ls teaching refusals (scope containing / → the document-path line; scope without / unknown → the extended no-source line; both count in nothing, tools stay offered; valid-scope and no-arg listings byte-identical to today); find_path_candidates (exact path match, suffix match, multiple candidates in catalog order capped at 3, zero candidates, no-/ argument → no DB lookup); read/grep wiring (in-context dedupe precedence, valid combined form unchanged, scoped grep suggestion, A5 grep contract regression).
• Integration: tests/integration/test_agent_tools.py — changed pins updated; new end-to-end suggestion cases through run_agent against real Postgres (bare path under one source; the same path under two sources).
• E2E (mandatory, house rule): NEW dedicated suite tests/e2e/test_tool_path_teaching.py, run in isolation — the mock flow (misuse ls(path='.') → teaching refusal → corrected no-arg ls() → listing answer) through the real UI with the two-round shape pinned on the SSE wire; regression suites green in isolation: test_harness_aligned_tools.py, test_agent_document_tools.py, test_agent_unlimited_tools.py, test_search_tool.py, test_chat_rag.py.
• Real-model acceptance gate (owner-locked, the phase's pass condition): uv run python -m scripts.agent_realmodel_check against the live endpoint with the configured chat model (lite) — the fixed 10-question battery (3 ls turns including the incident's "list the files in this directory" and a source-name trap, 4 read turns including two bare-path traps, 1 grep turn, 2 mixed) driven through the real grounded path. PASS = every turn answers (no LLMError/MalformedReplyError), zero turns hit the round cap, ≥6 of 10 turns emit ≥1 tool call, and ≥90% of all emitted tool calls execute (rejections don't count). Until it passes, task 05 iterates the copy levers this phase owns (refusal templates, AGENT_TOOLS descriptions, TOOLS_SECTION) — the question set and thresholds are fixed by the task file and may not be weakened.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).

Completion Criteria

• A scoped ls whose stripped path contains / gets the document-path teaching refusal; an unknown source name without / gets the extended "source name, not a directory" refusal; neither counts in anything; a valid scope and the no-arg listing are byte-identical to today.
• read / scoped grep with an unresolved argument that matches an indexed document path (exact or suffix) gets the "did you mean …?" refusal (one candidate → one combined identity; two or more → up to 3, catalog order); a non-matching argument gets today's refusal byte-identical; the in-context dedupe refusal still wins.
• The AGENT_TOOLS path descriptions for ls / read / grep state the contract explicitly; the tool names and argument shapes are unchanged (rg '"name":' app/rag/agent.py → exactly ls, read, grep).
• TOOLS_SECTION clarifies the source-name ls path and the source-name-required combined identity; the HIGH prompt still ends with the <tools> section; the LOW/deflection prompt is byte-identical to today.
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run ruff check . && uv run pyright clean.
• uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov green in isolation; the regression suites above green in isolation.
• uv run python -m scripts.agent_realmodel_check exits 0 against the live endpoint (all four pass conditions met with the configured model) — the verdict line recorded in the app/rag/agent.py module docstring and in the commit body.
• One --no-gpg-sign commit (message in the Commit block); the phase directory moved to .agents/phases/complete/.

Locked decisions

• The phase-70 tool surface is unchanged (owner lock, 2026-09-03): ls(path?) / read(path) / grep(pattern, path?) — no renames, no argument additions or removals; this phase changes refusal copy, tool descriptions, and prompt copy only.
• Deterministic only (owner 2026-09-03, phase-71 house style): no model in detection or repair; suggestions are a pure catalog lookup (exact or suffix path match, case-sensitive, catalog order, capped at 3); every refusal is a fixed template constant.
• Teach, don't silently fix: a misused call is still a refusal (counts in nothing, consumes a round); the model sees its own argument echoed plus the correct form. No silent argument normalization — ls(path='.') does NOT become a full listing.
• The zero-candidate refusal is byte-identical to today ("No document at '…' — check the ls output.") — no behavior change where the model is not confused; the ls no-source refusal keeps its prefix (the teaching parenthetical is appended).
• No UI change: the Thinking display (phases 17/21/43) works as designed — the fix ends the loop, it does not hide the scratchpad. No SSE contract change (refusals are tool results in the message history; the tool frames already carry the call's name/argument). No model swap (owner keeps lite), no env change, no schema change.
• Real-model gate is a pass condition, not a smoke test (owner directive, 2026-09-03): the phase is NOT complete — and gets NO commit — until scripts/agent_realmodel_check.py PASSES against the real lite model. The 10 questions, the ≥6-of-10 tool-usage floor, the ≥90% executed-call bar, and the zero-cap rule are fixed by task 05's file; the executor may iterate ONLY the copy levers this phase owns (refusal templates, AGENT_TOOLS descriptions, TOOLS_SECTION — with their unit pins updated to follow the constants). Lowering a threshold, swapping in easier questions, or skipping the gate to "make it pass" is forbidden; a gate still failing after iteration stops the phase with the per-turn numbers reported for the owner (fail-loud house style). The verdict line (house precedent: the phase-37 probe verdict in app/rag/agent.py) is recorded in that module's docstring and in the commit body.

Commit

git add -A .agents/ app/ tests/ scripts/ && git commit --no-gpg-sign -m "fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling" -m "<real-model gate verdict line, e.g. real-model gate (lite): 10/10 answered, caps=0, tool-turns=8, calls 21/23 executed (91%) — 2026-09-03>"

The commit also carries the still-uncommitted phase-71 todo/ → complete/ move and .agents/reports/71_scaffolding_guardrails/ (.agents/ is tracked and committed with the phase — AGENTS.md §8; only .agents/phase-sessions/ and .agents/pipeline.log are gitignored).