brain-of-reese/.agents/phases/complete/72_teaching_refusals/04_mock_e2e.md

Task 04 — Mock Flow, Dedicated E2E Suite

Phase: 72_teaching_refusals · Story: .agents/user_stories/agent-document-tools.md

Objective

Prove the self-correction loop deterministically through the real UI: a mock-LLM flow that reproduces the incident's ls(path='.') misuse, receives the teaching refusal, corrects to a no-arg ls(), and answers from the catalog — a dedicated Playwright suite pinning the two-round shape on the SSE wire. (The live real-model acceptance gate is task 05 — this task is the deterministic half of the proof.)

Work

1. tests/e2e/mock_llm.py — one new deterministic flow, checked in the flow table before the plain TOOLS_TRIGGER flow (the trigger phrases are disjoint substrings; the ordering rule follows the phase-71 convention):
   • LS_TEACH_TRIGGER = "list the files in this directory" — and the system prompt carries the <tools> section (grounded turn):
     • request 1 (tools offered, no tool-role result in the messages yet): stream ONLY tool_calls deltas — ls with {"path": "."} (synthetic id call_0), finish_reason: "tool_calls", no content (the incident's misuse, deterministic);
     • request 2 (a tool-role result present that is not a catalog listing — i.e. the teaching refusal): stream a tool_calls delta — ls with no arguments (id call_1);
     • request 3 (a tool-role result whose first line matches the ^\d+ documents: catalog header): a deterministic content answer — These are the indexed documents: <first catalog line> (the source: X | path: Y | title: Z line, parsed with the existing _CATALOG_LINE_RE machinery), finish_reason: "stop".
   • Update the module docstring's flow table with the phase-72 note.
2. tests/e2e/test_tool_path_teaching.py (NEW — the phase's dedicated suite, house pattern, run in isolation; DB up, mock LLM):
   • Import a small fixture document set (house fixture pattern: one source, two documents with known source/path/title) and ask a question containing LS_TEACH_TRIGGER.
   • Self-correction — the turn settles (composer re-enables, done observed); the answer bubble contains the first document's source: and path: fields (the catalog reached the model and landed in the answer); no error banner.
   • Two rounds on the wire (the house SSE-capture pattern): the tool frames arrive in order — first name:"ls" with argument:".", then name:"ls" with argument:null — and there is no third tool frame (the loop ended in one correction, not at the round cap).
   • No regression to the plain flow — a follow-up question containing TOOLS_TRIGGER (the single-read flow) in the same session still settles with the read flow's answer (the new flow did not swallow the existing trigger).

Testing & Quality

• E2E: the dedicated suite proves the loop shape (misuse → teaching refusal → corrected call → answer) through the real UI and the SSE wire; the existing regression E2E suites (mock-driven) stay green — run them as the regression check for this task.
• Coverage: unit/integration coverage of app/ stays >90% (this task adds test-only code; uv run pytest --cov=app --cov-report=term-missing as the check).

Completion Criteria

• uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov green in isolation (DB up: podman compose up -d db, mock LLM)
• Regression E2E suites green in isolation: test_harness_aligned_tools.py, test_agent_document_tools.py, test_agent_unlimited_tools.py, test_search_tool.py, test_chat_rag.py
• uv run pytest green; uv run ruff check . && uv run pyright clean
• No commit in this task (the commit happens in task 05, after the real-model gate passes)