# Phase 72 — Teaching Refusals: End the Post-Harness Tool-Loop Rambling **Story:** `.agents/user_stories/agent-document-tools.md` (this phase repairs the model-facing contract the phase-70 tools reshaped) **Context:** - `app/rag/agent.py` — `AGENT_TOOLS` (the phase-70 `ls` / `read` / `grep` OpenAI function definitions), `_execute_tool` (the refusal strings: `"No source named '…' — check the ls output."`, `"No document at '…' — check the ls output."`), `all_documents` (catalog-order bulk loader — reused by the suggestion lookup). - `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only; the E2E mock keys off the `` marker's *presence*, not its wording). - `tests/unit/test_agent.py` (refusal-string pins; the `ScriptedLLM` + monkeypatched- accessor pattern), `tests/integration/test_agent_tools.py` (the same pins against real Postgres, `kb`/`src` fixtures). - `tests/e2e/mock_llm.py` — the deterministic mock tool flows (`TOOLS_TRIGGER` single-read, `MULTI_READ_TRIGGER`, `SEARCH_TRIGGER`; `_CATALOG_LINE_RE` catalog-line parse) and the dedicated-suite-per-phase E2E house pattern. - `scripts/llm_probe.py` — the house live-endpoint probe pattern (`python -m scripts.…`, argparse, dotenv, printed verdict line); `app/api/chat.py` — the grounded path the real-model gate mirrors (`retrieve` → `select_documents` → `build_high_prompt` → `run_agent`) - **Incident (owner chat, 2026-09-03, post phase 70/71):** the question "list the files in this directory" produced a Thinking-display trace of the model calling `ls(path='app/rag/importer.py')` → `"No source named 'app/rag/importer.py' — check the ls output."`, then `ls(path='.')` → the same-style refusal, then re-reasoning the same paragraphs over and over across rounds (each round's `reasoning_content` appends to the open Thinking block) before finally answering from the seed documents alone. Root cause: the harness-trained prior (`ls`'s `path` = a directory to list) collides with this app's contract (`path` = a source-name filter), and the terse refusal does not correct the misunderstanding, so the model burns rounds. The identical trap awaits `read`/`grep`: a bare document path missing the source prefix (`read('app/rag/importer.py')`) → `"No document at '…'"` with no hint of the combined form. ## Objective Make the affected tool refusals **teaching** so the harness-prior misuse self-corrects in at most one extra round: a scoped `ls` whose `path` looks like a document path (contains `/`) or names an unknown source gets a fixed-template refusal that states the correct contract; a `read` / scoped-`grep` argument that resolves to no combined identity but *matches an indexed document's `path`* (exact or suffix) gets a `"did you mean 'source/path'?"` refusal naming the exact combined identity to use. The `AGENT_TOOLS` `path` descriptions and the `TOOLS_SECTION` prompt copy say the same contract up front. Deterministic only — no model participates in detection or repair; the phase-70 harness shape (`ls` / `read(path)` / `grep(pattern, path?)`) is unchanged verbatim. The phase does not pass on mocks alone: a live acceptance gate runs the fixed question battery through `run_agent` against the **real configured chat model** (`lite` per `.env`) and must PASS before the commit (owner directive, 2026-09-03 — "test with the real lite model until tool calls work consistently; don't pass until a sufficient number of tool calls succeed"). ## Dependencies - `70_harness_aligned_tools` (complete) — the tool surface this phase teaches (shape untouched). - `71_scaffolding_guardrails` (complete) — the deterministic-guardrail house style this phase follows. ## Tasks 1. `01_ls_teaching_refusal.md` — `ls`: path-like and unknown-source scopes get teaching refusals; the `ls` `path` description says "source name, not a file or directory path". 2. `02_read_grep_path_suggestion.md` — `read` / `grep`: an unresolved argument that matches an indexed document `path` gets the "did you mean 'source/path'?" suggestion; descriptions updated. 3. `03_prompt_copy.md` — `TOOLS_SECTION` copy: the `ls` `path` is a source name, not a directory; `read`/`grep` need the combined identity *including the source name*. 4. `04_mock_e2e.md` — mock `ls`-misuse flow, dedicated E2E suite (green in isolation). 5. `05_real_model_gate.md` — the live real-lite acceptance gate (`scripts/agent_realmodel_check.py`): iterate the copy levers until the gate PASSES, then full gates and the commit. ## Testing & Quality - Unit: `tests/unit/test_agent.py` — the new `ls` teaching refusals (scope containing `/` → the document-path line; scope without `/` unknown → the extended no-source line; both count in nothing, tools stay offered; valid-scope and no-arg listings byte-identical to today); `find_path_candidates` (exact `path` match, suffix match, multiple candidates in catalog order capped at 3, zero candidates, no-`/` argument → no DB lookup); `read`/`grep` wiring (in-context dedupe precedence, valid combined form unchanged, scoped `grep` suggestion, A5 grep contract regression). - Integration: `tests/integration/test_agent_tools.py` — changed pins updated; new end-to-end suggestion cases through `run_agent` against real Postgres (bare path under one source; the same path under two sources). - E2E (mandatory, house rule): NEW dedicated suite `tests/e2e/test_tool_path_teaching.py`, run in isolation — the mock flow (misuse `ls(path='.')` → teaching refusal → corrected no-arg `ls()` → listing answer) through the real UI with the two-round shape pinned on the SSE wire; regression suites green in isolation: `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`. - **Real-model acceptance gate (owner-locked, the phase's pass condition):** `uv run python -m scripts.agent_realmodel_check` against the live endpoint with the configured chat model (`lite`) — the fixed 10-question battery (3 `ls` turns including the incident's "list the files in this directory" and a source-name trap, 4 `read` turns including two bare-path traps, 1 `grep` turn, 2 mixed) driven through the real grounded path. PASS = every turn answers (no `LLMError`/`MalformedReplyError`), zero turns hit the round cap, ≥6 of 10 turns emit ≥1 tool call, and **≥90% of all emitted tool calls execute** (rejections don't count). Until it passes, task 05 iterates the copy levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions, `TOOLS_SECTION`) — the question set and thresholds are fixed by the task file and may not be weakened. - Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`). ## Completion Criteria - [ ] A scoped `ls` whose stripped `path` contains `/` gets the document-path teaching refusal; an unknown source name without `/` gets the extended "source name, not a directory" refusal; neither counts in anything; a valid scope and the no-arg listing are byte-identical to today. - [ ] `read` / scoped `grep` with an unresolved argument that matches an indexed document `path` (exact or suffix) gets the "did you mean …?" refusal (one candidate → one combined identity; two or more → up to 3, catalog order); a non-matching argument gets today's refusal byte-identical; the in-context dedupe refusal still wins. - [ ] The `AGENT_TOOLS` `path` descriptions for `ls` / `read` / `grep` state the contract explicitly; the tool names and argument shapes are unchanged (`rg '"name":' app/rag/agent.py` → exactly `ls`, `read`, `grep`). - [ ] `TOOLS_SECTION` clarifies the source-name `ls` `path` and the source-name-required combined identity; the HIGH prompt still ends with the `` section; the LOW/deflection prompt is byte-identical to today. - [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean. - [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in isolation; the regression suites above green in isolation. - [ ] `uv run python -m scripts.agent_realmodel_check` exits 0 against the live endpoint (all four pass conditions met with the configured model) — the verdict line recorded in the `app/rag/agent.py` module docstring and in the commit body. - [ ] One `--no-gpg-sign` commit (message in the Commit block); the phase directory moved to `.agents/phases/complete/`. ## Locked decisions - **The phase-70 tool surface is unchanged** (owner lock, 2026-09-03): `ls(path?)` / `read(path)` / `grep(pattern, path?)` — no renames, no argument additions or removals; this phase changes refusal copy, tool descriptions, and prompt copy only. - **Deterministic only** (owner 2026-09-03, phase-71 house style): no model in detection or repair; suggestions are a pure catalog lookup (exact or suffix `path` match, case-sensitive, catalog order, capped at 3); every refusal is a fixed template constant. - **Teach, don't silently fix:** a misused call is still a refusal (counts in nothing, consumes a round); the model sees its own argument echoed plus the correct form. No silent argument normalization — `ls(path='.')` does NOT become a full listing. - **The zero-candidate refusal is byte-identical to today** (`"No document at '…' — check the ls output."`) — no behavior change where the model is not confused; the `ls` no-source refusal keeps its prefix (the teaching parenthetical is appended). - **No UI change:** the Thinking display (phases 17/21/43) works as designed — the fix ends the loop, it does not hide the scratchpad. **No SSE contract change** (refusals are tool results in the message history; the `tool` frames already carry the call's `name`/`argument`). **No model swap** (owner keeps `lite`), **no env change**, **no schema change**. - **Real-model gate is a pass condition, not a smoke test** (owner directive, 2026-09-03): the phase is NOT complete — and gets NO commit — until `scripts/agent_realmodel_check.py` PASSES against the real `lite` model. The 10 questions, the ≥6-of-10 tool-usage floor, the ≥90% executed-call bar, and the zero-cap rule are fixed by task 05's file; the executor may iterate ONLY the copy levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions, `TOOLS_SECTION` — with their unit pins updated to follow the constants). Lowering a threshold, swapping in easier questions, or skipping the gate to "make it pass" is forbidden; a gate still failing after iteration stops the phase with the per-turn numbers reported for the owner (fail-loud house style). The verdict line (house precedent: the phase-37 probe verdict in `app/rag/agent.py`) is recorded in that module's docstring and in the commit body. ## Commit ```bash git add -A .agents/ app/ tests/ scripts/ && git commit --no-gpg-sign -m "fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling" -m "" ``` The commit also carries the still-uncommitted phase-71 `todo/` → `complete/` move and `.agents/reports/71_scaffolding_guardrails/` (`.agents/` is tracked and committed with the phase — AGENTS.md §8; only `.agents/phase-sessions/` and `.agents/pipeline.log` are gitignored).