brain-of-reese/.agents/phases/complete/71_scaffolding_guardrails/05_e2e_commit.md

Task 05 — Mock Triggers, Dedicated E2E Suite, Gates, Commit

Phase: 71_scaffolding_guardrails · Story: n/a (owner request from chat, 2026-09-03)

Objective

Prove the guardrail end-to-end through the real UI: a mock-LLM scaffolding flow (recovery case + terminal case), a dedicated Playwright suite pinning that raw tokens never reach the DOM, then the full quality gates and the phase commit.

Work

1. tests/e2e/mock_llm.py — two new deterministic flows (checked in the flow table before the plain TOOLS_TRIGGER flow, after SEARCH_TRIGGER ordering rules as they fit — the triggers are independent of the <tools> marker, so both grounded and deflected turns hit them):
   • SCAFFOLD_TRIGGER = "emit raw tool markup" — request 1 (no correction in the system prompt): stream ONLY delta.content chunks carrying the incident text <|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|> (split across ≥2 chunks to exercise the boundary path), finish_reason: "stop", no structured tool_calls, no reasoning. Request 2 (system prompt contains the stable substring of CORRECTION_INSTRUCTION — import it from app.rag.agents so the mock can never drift from the constant): stream a clean plain answer ("Here is the plain-text answer the recovery produced.")
     • finish_reason: "stop".
   • SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup" — every request (recovery included): the same scaffolding-only stream, forever.
   • Update the module docstring's flow table + the phase-71 note.
2. tests/e2e/test_tool_scaffolding_guardrails.py (NEW — the phase's dedicated suite, house pattern, run in isolation; DB up, mock LLM):
   • Recovery case — ask a question containing SCAFFOLD_TRIGGER: the turn settles (the composer re-enables, done observed); the final answer bubble contains the recovery's clean text; document.body.innerText contains neither tool_call_start nor tool_call_end (nor the raw [read(path=…] fragment); no error banner.
   • Terminal case — ask a question containing SCAFFOLD_ALWAYS_TRIGGER: the existing error status renders with the dedicated copy ("The model returned a malformed reply — please try again."); no raw tokens in the DOM; no answer bubble with the scaffolding; the app stays usable — a follow-up plain question (no trigger) gets a normal streamed answer in the same session.
   • No false positive — a plain question (existing CHAT_TRIGGER-style flow, no tools needed): the answer streams byte-clean, no error state, no recovery request visible (the turn settles on the first request).
   • The SSE wire itself: in the recovery case, no delta frame ever carries a tool_call_start/tool_call_end fragment (the existing SSE-capture house pattern) — the strip happens server-side, not in the UI.
3. Gates + commit:
   • uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run ruff check . && uv run pyright clean.
   • E2E in isolation (DB up): test_tool_scaffolding_guardrails.py, then the regression suites test_harness_aligned_tools.py, test_chat_rag.py, test_agent_document_tools.py.
   • One atomic commit (message below); move .agents/phases/todo/71_scaffolding_guardrails/ → .agents/phases/complete/71_scaffolding_guardrails/.

Testing & Quality

• E2E: the new dedicated suite (recovery / terminal / no-false-positive / wire pins) + the three regression suites (isolation runs).
• Coverage: >90% on app/ (phase-level gate).

Completion Criteria

• rg "tool_call_start|tool_call_end" frontend/ → no matches.
• Raw scaffolding is never visible in the DOM in any case (recovery, terminal, mixed) — pinned by the dedicated suite.
• Exactly one recovery per malformed turn (mock request counts implied by the flows); clean turns never carry the correction line (pinned by the unit suites from task 03).
• uv run pytest green; uv run pytest --cov=app TOTAL >90%; uv run ruff check . && uv run pyright clean.
• uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov green in isolation; regression suites green in isolation.
• One --no-gpg-sign commit; phase dir moved to .agents/phases/complete/.

Commit

git add -A .agents/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"