# Phase 71 — Tool-Scaffolding Guardrails: Deterministic Strip + One Bounded Recovery

**Source:** owner request (chat, 2026-09-03) — same incident as phase 70: the
deflected answer streamed the raw model text
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>` into
the UI (the model's own `<|…|>` chat-template tool syntax, emitted as plain
`delta.content` even though no tools were offered). Owner direction: "We also need
guardrails for situations like this… **deterministic guardrails only** right now,
forget using a model for that" — no model of any kind (no `lite` classifier, no
model-authored repair) in the guardrail path; `lite` stays the chat model for
everything (it's faster).
**Story:** n/a (owner request from chat — tool-scaffolding guardrails, 2026-09-03)
**Context:**
- `app/rag/llm.py` — `chat_stream` yields `StreamPiece("content", delta.content)`
  verbatim (L~`content = delta.content; if content: yield`); thinking pieces
  (`delta.reasoning_content`) pass through raw by design; tool-call materialization
  happens at stream end (after the `async for`); phase-48 teardown closes the
  endpoint stream on every exit (must stay intact). `chat_stream_retried` (phase 67)
  wraps it with the retry-before-first-piece rule.
- `app/rag/agent.py` — `run_agent`'s round loop: a round with no `ToolCallPiece`s
  ends the turn (`if not calls: return`); the round cap forces one final
  `tools=None` request (the "forced final answer" pattern this phase reuses for
  recovery); `AgentHolder` (read_docs / tool_calls) is unchanged.
- `app/api/chat.py` — the piece loop (thinking/tool/retry/delta handling +
  `thinking_chars` counter), the deflected path (`chat_stream_retried(...,
  tools=None)` directly), the `LLMError` → terminal error-frame handler, the
  per-turn log line (ends `…total_ms=N retries=N`).
- `app/rag/prompts.py` — `build_deflect_prompt`'s `DEFLECT_MODE` body (the E2E mock
  keys on the marker's *presence*, not the wording — an appended line is safe).
- `tests/e2e/mock_llm.py` — the trigger-string flow table (task 05 adds the
  scaffolding triggers; phase-70 names apply).
- Tests to extend: `tests/unit/test_llm_client.py`, `test_agent.py`,
  `tests/integration/test_chat_api.py` (log-line + error-frame pins),
  `tests/unit/test_prompts.py`.

## Objective
Raw tool-scaffolding tokens can never reach the user as answer text: a deterministic
streaming filter strips known scaffolding from `delta.content` as it flows, and a
round/turn whose visible content ends up empty (scaffolding was the whole "answer")
gets **one** bounded, deterministic recovery (same turn, `tools=None`, a fixed
harness-owned correction line in the system prompt); if the recovery also comes back
empty, the turn settles with a dedicated structured error frame. No model is used to
detect or repair anything.

## Dependencies
- `70_harness_aligned_tools` (todo) — runs first: the mock/prompt/code state this
  phase builds on (new tool names in the mock flows and prompt).
- `67_llm_retry` (complete) — `chat_stream_retried`, the primitive every request
  (including recoveries) goes through.
- `48_stop_generation` (complete) — the phase-48 stream-teardown contract the
  filter integration must not break.

## Tasks
1. `01_filter.md` — `app/rag/scaffolding.py` (new): the pattern registry +
   `ScaffoldingFilter` streaming state machine, with the boundary test matrix.
2. `02_llm_integration.md` — `app/rag/llm.py`: `chat_stream`/`chat_stream_retried`
   accept the filter; content deltas are filtered, the tail flushed before tool
   materialization; `None` = byte-identical raw path.
3. `03_recovery_policy.md` — `app/rag/agent.py` + `app/api/chat.py`: the empty-round
   recovery (one per turn, `tools=None`, correction constant),
   `MalformedReplyError`, the dedicated error copy, and the per-turn log line's
   `scaffold_stripped=N` field.
4. `04_deflect_prompt.md` — the plain-text line in the `DEFLECT_MODE` body
   (prevention; owner-permitted LOW-prompt change).
5. `05_e2e_commit.md` — the mock scaffolding triggers, the dedicated E2E suite,
   full gates, commit.

## Testing & Quality
- Unit (new `tests/unit/test_scaffolding_filter.py`): the filter matrix — span in
  one chunk; span split across chunks at **every** boundary offset of the start
  token; multiple spans in one chunk; standalone `<|tool_calls|>` / `<|tool_call|>`
  tokens stripped; look-alikes **not** stripped (prose containing the words
  "tool_call" or `tool_call_start` without the `<|…|>` delimiters, an unknown
  `<|some_other_token|>`, a lone `<|tool_call_end|>` without a start); a partial
  start token at stream end → `flush()` emits it as-is (no false-positive strip);
  `stripped_chars` accounting; empty chunks.
- Unit: `tests/unit/test_llm_client.py` — `chat_stream` with a filter (content
  filtered, thinking raw, `None` = raw pass-through pinned byte-identical, flush
  order: flushed tail content precedes tool-call pieces, phase-48 teardown intact).
- Unit: `tests/unit/test_agent.py` — grounded recovery matrix (scaffolding-only
  round → exactly one recovery request: `tools=None` + correction line in the
  system prompt + fresh filter → clean answer ends the turn; scaffolding twice →
  `MalformedReplyError`; scaffolding + real content → clean answer, **no**
  recovery; round cap and kill switch unchanged).
- Integration: `tests/integration/test_chat_api.py` — deflected-path recovery matrix
  (same shapes over the SSE endpoint: clean recovery → `done` frame; terminal → the
  dedicated error frame, no `done`, no `query_log` row — same terminal semantics as
  today's `LLMError`); log line carries `scaffold_stripped=N` (0 when nothing was
  stripped — the field is uniform, the phase-67 `retries=N` pattern).
- E2E (mandatory, house rule): NEW dedicated suite
  `tests/e2e/test_tool_scaffolding_guardrails.py`, run in isolation — recovery case
  (raw tokens never in the DOM, clean answer shown) and terminal case (error state,
  no raw tokens, the app stays usable).
- Coverage: **>90%** on `app/` (validate.sh gate).

## Completion Criteria
- [ ] `rg "tool_call_start" frontend/` → no matches (no scaffolding rendering
      path); the filter lives in `app/rag/scaffolding.py` as a pure module (no I/O,
      no model calls).
- [ ] A content stream of pure scaffolding yields zero `delta` frames; a mixed
      stream yields the clean remainder; thinking frames are never filtered.
- [ ] Exactly one recovery per turn (grounded and deflected paths); the recovery
      request is `tools=None` with the fixed correction line in the system prompt;
      a second empty reply settles with the error frame
      "The model returned a malformed reply — please try again."
- [ ] The per-turn log line ends `…retries=N scaffold_stripped=N`; `scaffold_stripped=0`
      on clean turns (uniform field).
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
      `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
      green in isolation; regression suites green in isolation:
      `test_harness_aligned_tools.py`, `test_chat_rag.py`, `test_agent_document_tools.py`.
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
      `.agents/phases/complete/`.

## Locked decisions
- **Owner (chat, 2026-09-03): deterministic only.** No model — `lite` or any other —
  participates in detection or repair. The guardrail is a fixed pattern registry + a
  fixed retry policy. (A model-based classifier/repair was proposed and explicitly
  rejected for now — if it is ever wanted, it is a later phase with its own
  permission.)
- **`lite` stays the chat model** (owner: "I want to use lite for everything since
  it's way faster") — the guardrail is what protects the UX while `lite` is the
  model; no `.env` change in this phase.
- **The pattern registry is the extension point.** Initial entries: the observed
  span form `<|tool_call_start|>…<|tool_call_end|>` (non-greedy, any text between)
  plus the standalone sibling tokens `<|tool_calls|>` and `<|tool_call|>` from the
  same tokenizer family. Every strip logs a warning with the stripped span
  (truncated to 200 chars) — that log line is how a new format gets captured and
  added (pattern + unit fixture), keeping the registry honest (every entry traces
  to an observed capture or the initial incident).
- **Content only, thinking never filtered.** The Thinking block is the model's raw
  reasoning by design (phase 17) and stays raw (collapsible); the guardrail protects
  the answer, not the scratchpad.
- **Recovery is a fixed policy, not a conversation.** One extra request per turn,
  same messages with the harness-owned constant folded into the system prompt
  (single system message — provider-safe), `tools=None`, a fresh filter, the same
  phase-67 retry budget. At most one recovery; the second empty reply is terminal.
  A round with real visible content plus scaffolding needs no recovery (the clean
  content stands).
- **Terminal semantics follow the existing error pattern.** A terminal malformed
  turn settles with a structured `error` frame (dedicated copy), writes no
  `query_log` row, and the UI shows the existing error state — byte-for-byte the
  same shape as today's `LLMError` terminal path.
- **`MalformedReplyError` subclasses `LLMError`** and is raised only by the recovery
  policy (never from inside a stream, so `chat_stream_retried`'s retry rule never
  sees it); `chat.py` catches it before the generic `LLMError` handler.

## Commit
```bash
git add -A .agents/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
```