feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# Task 03 — Recovery Policy: One Bounded `tools=None` Retry + Terminal Error
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
When a round/turn's visible content ends up empty **because scaffolding was the
|
||||
whole answer**, run exactly one deterministic recovery (same turn, `tools=None`,
|
||||
fixed correction line in the system prompt, fresh filter); a second empty reply
|
||||
settles with a dedicated error frame. Both the grounded agent loop and the
|
||||
deflected path get the policy; the per-turn log line gains `scaffold_stripped=N`.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py`:
|
||||
- `CORRECTION_INSTRUCTION: str` — the harness-owned constant (verbatim, single
|
||||
line): "Your previous reply contained raw tool-call markup, which is not
|
||||
interpreted here. Answer the user's question directly in plain text — no
|
||||
tool syntax." (The E2E mock in task 05 keys on a stable substring of it —
|
||||
pick the exact constant now; the mock copies it.)
|
||||
- `class MalformedReplyError(LLMError)` — raised only by the recovery policy
|
||||
(never from inside a stream, so `chat_stream_retried`'s retry rule never
|
||||
sees it). Module docstring: the phase-71 note (deterministic-only, owner
|
||||
permission 2026-09-03).
|
||||
- `run_agent` — per round: create a fresh `ScaffoldingFilter`, pass it to
|
||||
`chat_stream_retried(..., scaffolding=round_filter)`, and count the
|
||||
round's visible content (sum of the lengths of the yielded
|
||||
`StreamPiece("content", …)` texts — the filtered ones). After the round,
|
||||
when `not calls` (today's "the answer was streamed" exit):
|
||||
- if round content > 0 → return (as today).
|
||||
- if round content == 0 and `round_filter.stripped_chars > 0` → **one
|
||||
recovery**: `messages_recovered = [*messages[:-1], {"role": "system",
|
||||
"content": system_prompt + "\n" + CORRECTION_INSTRUCTION}, messages[-1]]`
|
||||
(the correction folds into the ORIGINAL single system message —
|
||||
provider-safe; the user message stays last) — one
|
||||
`chat_stream_retried` request with `tools=None`, a fresh filter, the same
|
||||
`retries`/`delay` budget; yield its pieces through the normal piece flow.
|
||||
If the recovery's visible content > 0 → return. Otherwise →
|
||||
`logger.warning` + `raise MalformedReplyError(…)`.
|
||||
- if round content == 0 and nothing was stripped → return (today's
|
||||
empty/thinking-only answer behavior — the UI handles it; unchanged).
|
||||
- Log one warning per strip event here: `logger.warning("agent: stripped
|
||||
N chars of tool-scaffolding in round %d: %r", …)` with the stripped span
|
||||
truncated to 200 chars (the capture mechanism for new registry entries —
|
||||
the filter exposes the stripped spans for this; add a
|
||||
`stripped_spans: list[str]` to the filter if needed).
|
||||
- A scaffolding-only round that also **carried tool calls** needs no
|
||||
recovery (the clean content stands / the tool ran) — the policy keys on
|
||||
the `not calls` exit only (pinned).
|
||||
2. `app/api/chat.py` — the **deflected** path (the grounded path is covered by
|
||||
`run_agent`):
|
||||
- Create one `ScaffoldingFilter` for the turn's request, pass it to
|
||||
`chat_stream_retried(..., scaffolding=filter)`; count visible content across
|
||||
the piece loop (a `content_chars` counter next to `thinking_chars`).
|
||||
- After the piece loop (deflected branch only): content == 0 and
|
||||
`filter.stripped_chars > 0` → one recovery request: the same
|
||||
`messages` with the system prompt extended by
|
||||
`CORRECTION_INSTRUCTION` (import from `app.rag.agent`), `tools=None`, a
|
||||
**fresh** filter, the same retry budget; stream its pieces through the
|
||||
SAME piece-handling code (extract the piece loop into a small inner
|
||||
helper/coroutine to avoid duplicating the thinking/tool/retry/delta
|
||||
handling — the extraction must be behavior-preserving for the first pass,
|
||||
pinned by the existing integration suite). If the recovery content > 0 →
|
||||
continue to the normal `done` flow; else → the terminal path below.
|
||||
- Catch `MalformedReplyError` **before** the generic `LLMError` handler:
|
||||
`settled = True`, yield `ChatErrorEvent(detail="The model returned a
|
||||
malformed reply — please try again.")`, return (no `query_log` row, no
|
||||
`done` — the existing terminal-error semantics; the generic
|
||||
"dropped the connection" copy stays for transport failures).
|
||||
- Per-turn log line: append `scaffold_stripped=N` after `retries=N` — the sum
|
||||
across the turn's requests (rounds + any recovery; 0 on clean turns —
|
||||
uniform field, the phase-67 `retries=N` pattern). The recovery does not
|
||||
bump `retries=N` (it is not a phase-67 endpoint-retry).
|
||||
- Module docstring: the phase-71 paragraph (deterministic guardrail +
|
||||
recovery + log field).
|
||||
3. `tests/unit/test_agent.py` — the grounded matrix (scripted fake LLM,
|
||||
monkeypatched DB): scaffolding-only round → exactly **two** model requests,
|
||||
the second `tools=None` with `CORRECTION_INSTRUCTION` in its system prompt →
|
||||
clean answer ends the turn, `done`-side state normal (holder untouched by the
|
||||
recovery); scaffolding twice → `MalformedReplyError` (assert it subclasses
|
||||
`LLMError`); scaffolding + real content → one request only, clean content
|
||||
yielded, no recovery; clean turn → one request, no correction in any system
|
||||
prompt; the round cap + kill-switch tests stay green.
|
||||
4. `tests/integration/test_chat_api.py` — the deflected matrix over
|
||||
`POST /api/chat` (mock LLM): scaffolding-only deflected turn → clean recovery
|
||||
answer + `done` frame; scaffolding twice → the error frame with the dedicated
|
||||
copy, no `done`, no `query_log` row (assert against the table); mixed
|
||||
scaffolding+content → clean answer, `scaffold_stripped>0` in the log line;
|
||||
the log-line pin gains `scaffold_stripped=0` on clean turns and the summed
|
||||
value on stripped turns.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` (grounded matrix).
|
||||
- Integration: `tests/integration/test_chat_api.py` (deflected matrix + log line).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/agent.py`,
|
||||
`app/api/chat.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_chat_api.py -v
|
||||
--no-cov` green.
|
||||
- [ ] Exactly one recovery per turn, on both paths; the recovery request is
|
||||
`tools=None` + correction line; a second empty reply → the dedicated error
|
||||
frame, no `query_log` row.
|
||||
- [ ] Clean turns: no correction in any system prompt, `scaffold_stripped=0`,
|
||||
request counts unchanged (kill-switch/deflection byte-identical pins green).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
Reference in New Issue
Block a user