refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
# Task 04 — Mock Flow, Dedicated E2E Suite
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agents/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Prove the self-correction loop deterministically through the real UI: a mock-LLM flow
|
||||
that reproduces the incident's `ls(path='.')` misuse, receives the teaching refusal,
|
||||
corrects to a no-arg `ls()`, and answers from the catalog — a dedicated Playwright
|
||||
suite pinning the two-round shape on the SSE wire. (The live real-model acceptance
|
||||
gate is task 05 — this task is the deterministic half of the proof.)
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — one new deterministic flow, checked in the flow table
|
||||
**before** the plain `TOOLS_TRIGGER` flow (the trigger phrases are disjoint
|
||||
substrings; the ordering rule follows the phase-71 convention):
|
||||
- `LS_TEACH_TRIGGER = "list the files in this directory"` — **and** the system
|
||||
prompt carries the `<tools>` section (grounded turn):
|
||||
* request 1 (tools offered, no `tool`-role result in the messages yet): stream
|
||||
ONLY `tool_calls` deltas — `ls` with `{"path": "."}` (synthetic id `call_0`),
|
||||
`finish_reason: "tool_calls"`, no content (the incident's misuse,
|
||||
deterministic);
|
||||
* request 2 (a `tool`-role result present that is **not** a catalog listing —
|
||||
i.e. the teaching refusal): stream a `tool_calls` delta — `ls` with no
|
||||
arguments (id `call_1`);
|
||||
* request 3 (a `tool`-role result whose first line matches the
|
||||
`^\d+ documents:` catalog header): a deterministic content answer —
|
||||
`These are the indexed documents: <first catalog line>` (the
|
||||
`source: X | path: Y | title: Z` line, parsed with the existing
|
||||
`_CATALOG_LINE_RE` machinery), `finish_reason: "stop"`.
|
||||
- Update the module docstring's flow table with the phase-72 note.
|
||||
2. `tests/e2e/test_tool_path_teaching.py` (NEW — the phase's dedicated suite, house
|
||||
pattern, run in isolation; DB up, mock LLM):
|
||||
- Import a small fixture document set (house fixture pattern: one source, two
|
||||
documents with known `source`/`path`/`title`) and ask a question containing
|
||||
`LS_TEACH_TRIGGER`.
|
||||
- **Self-correction** — the turn settles (composer re-enables, `done` observed);
|
||||
the answer bubble contains the first document's `source:` and `path:` fields
|
||||
(the catalog reached the model and landed in the answer); no error banner.
|
||||
- **Two rounds on the wire** (the house SSE-capture pattern): the `tool` frames
|
||||
arrive in order — first `name:"ls"` with `argument:"."`, then `name:"ls"` with
|
||||
`argument:null` — and there is **no** third `tool` frame (the loop ended in one
|
||||
correction, not at the round cap).
|
||||
- **No regression to the plain flow** — a follow-up question containing
|
||||
`TOOLS_TRIGGER` (the single-read flow) in the same session still settles with
|
||||
the read flow's answer (the new flow did not swallow the existing trigger).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the dedicated suite proves the loop shape (misuse → teaching refusal →
|
||||
corrected call → answer) through the real UI and the SSE wire; the existing
|
||||
regression E2E suites (mock-driven) stay green — run them as the regression check
|
||||
for this task.
|
||||
- Coverage: unit/integration coverage of `app/` stays >90% (this task adds test-only
|
||||
code; `uv run pytest --cov=app --cov-report=term-missing` as the check).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation (DB up: `podman compose up -d db`, mock LLM)
|
||||
- [ ] Regression E2E suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`, `test_chat_rag.py`
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No commit in this task (the commit happens in task 05, after the real-model
|
||||
gate passes)
|
||||
Reference in New Issue
Block a user