refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
# Task 05 — Mock Triggers, Dedicated E2E Suite, Gates, Commit
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Prove the guardrail end-to-end through the real UI: a mock-LLM scaffolding flow
|
||||
(recovery case + terminal case), a dedicated Playwright suite pinning that raw
|
||||
tokens never reach the DOM, then the full quality gates and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — two new deterministic flows (checked in the flow
|
||||
table **before** the plain `TOOLS_TRIGGER` flow, after `SEARCH_TRIGGER`
|
||||
ordering rules as they fit — the triggers are independent of the `<tools>`
|
||||
marker, so both grounded and deflected turns hit them):
|
||||
- `SCAFFOLD_TRIGGER = "emit raw tool markup"` — request 1 (no correction in
|
||||
the system prompt): stream ONLY `delta.content` chunks carrying the incident
|
||||
text `<|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|>`
|
||||
(split across ≥2 chunks to exercise the boundary path), `finish_reason:
|
||||
"stop"`, no structured `tool_calls`, no reasoning. Request 2 (system prompt
|
||||
contains the stable substring of `CORRECTION_INSTRUCTION` — import it from
|
||||
`app.rag.agents` so the mock can never drift from the constant): stream a
|
||||
clean plain answer ("Here is the plain-text answer the recovery produced.")
|
||||
+ `finish_reason: "stop"`.
|
||||
- `SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup"` — every request
|
||||
(recovery included): the same scaffolding-only stream, forever.
|
||||
- Update the module docstring's flow table + the phase-71 note.
|
||||
2. `tests/e2e/test_tool_scaffolding_guardrails.py` (NEW — the phase's dedicated
|
||||
suite, house pattern, run in isolation; DB up, mock LLM):
|
||||
- **Recovery case** — ask a question containing `SCAFFOLD_TRIGGER`: the turn
|
||||
settles (the composer re-enables, `done` observed); the final answer bubble
|
||||
contains the recovery's clean text; `document.body.innerText` contains
|
||||
**neither** `tool_call_start` nor `tool_call_end` (nor the raw
|
||||
`[read(path=…]` fragment); no error banner.
|
||||
- **Terminal case** — ask a question containing
|
||||
`SCAFFOLD_ALWAYS_TRIGGER`: the existing error status renders with the
|
||||
dedicated copy ("The model returned a malformed reply — please try
|
||||
again."); no raw tokens in the DOM; no answer bubble with the scaffolding;
|
||||
the app stays usable — a follow-up plain question (no trigger) gets a
|
||||
normal streamed answer in the same session.
|
||||
- **No false positive** — a plain question (existing `CHAT_TRIGGER`-style
|
||||
flow, no tools needed): the answer streams byte-clean, no error state, no
|
||||
recovery request visible (the turn settles on the first request).
|
||||
- The SSE wire itself: in the recovery case, no `delta` frame ever carries a
|
||||
`tool_call_start`/`tool_call_end` fragment (the existing SSE-capture house
|
||||
pattern) — the strip happens server-side, not in the UI.
|
||||
3. Gates + commit:
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_scaffolding_guardrails.py`, then the
|
||||
regression suites `test_harness_aligned_tools.py`, `test_chat_rag.py`,
|
||||
`test_agent_document_tools.py`.
|
||||
- One atomic commit (message below); move
|
||||
`.agents/phases/todo/71_scaffolding_guardrails/` →
|
||||
`.agents/phases/complete/71_scaffolding_guardrails/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new dedicated suite (recovery / terminal / no-false-positive / wire
|
||||
pins) + the three regression suites (isolation runs).
|
||||
- Coverage: **>90%** on `app/` (phase-level gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start|tool_call_end" frontend/` → no matches.
|
||||
- [ ] Raw scaffolding is never visible in the DOM in any case (recovery,
|
||||
terminal, mixed) — pinned by the dedicated suite.
|
||||
- [ ] Exactly one recovery per malformed turn (mock request counts implied by the
|
||||
flows); clean turns never carry the correction line (pinned by the unit
|
||||
suites from task 03).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agents/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
Reference in New Issue
Block a user