feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)
This commit is contained in:
@@ -0,0 +1,143 @@
|
||||
# Phase 70 — Harness-Aligned Agent Tools: `ls` / `read(path)` / `grep(pattern, path?)`
|
||||
|
||||
**Source:** owner request (chat, 2026-09-03) — live incident: the question "What are the
|
||||
correct llama.cpp arguments for Qwen 3.8?" was answered with the raw model text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>`
|
||||
(`query_log`: top_score=0.040, fts_hits=0, deflected=true — the turn was DEFLECTED, so
|
||||
no tools were even offered). Diagnosis: the chat model (`lite` per `.env`; owner keeps
|
||||
`lite` for everything — it's faster, owner decision 2026-09-03) reaches for the tool
|
||||
shapes it was trained on — a `read` tool taking a **single** `path` argument — and our
|
||||
`read_document(source, path)` fights that prior (the `_resolve_document` combined-form
|
||||
self-correction and "teach the split" refusals are the scar tissue). Owner direction:
|
||||
"match existing harnesses as much as possible" — the mapping below mirrors the
|
||||
**pi.dev** tool surface (`dist/core/tools/`: `read{path}`, `ls{path?}`,
|
||||
`grep{pattern, path?}`).
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md` (this phase reshapes the tools
|
||||
that story delivered; phase 68's search contract rides along renamed)
|
||||
**Context:**
|
||||
- `app/rag/agent.py` — `AGENT_TOOLS` (OpenAI function definitions:
|
||||
`list_documents` / `read_document(source, path)` / `search_documents(pattern,
|
||||
source?, path?)`), `_execute_tool` (execution + refusals: `ALREADY_IN_CONTEXT`,
|
||||
`UNKNOWN_TOOL`, `MISSING_READ_ARGS`, `MISSING_SEARCH_ARGS`; combined-form
|
||||
self-correction in `_resolve_document`), `AgentHolder`, `run_agent` (round cap
|
||||
`BOR_AGENT_MAX_ROUNDS`, default 10; `0` = no-tools kill switch).
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only, ends the prompt; the E2E
|
||||
mock keys off the `<tools>` marker's *presence*, not the wording).
|
||||
- `app/api/chat.py` — the `ToolCallPiece` → SSE `{"type":"tool","name":…,"argument":…}`
|
||||
derivation (read → `source/path`, search → pattern, list → null), `done.sources`,
|
||||
per-turn log line (`tool_calls=N`).
|
||||
- `tests/e2e/mock_llm.py` — the deterministic mock flows keyed on `TOOLS_TRIGGER`
|
||||
("use your tools"), `MULTI_READ_TRIGGER`, `SEARCH_TRIGGER` + the `<tools>` marker;
|
||||
the flows emit `tool_calls` deltas with the current names/args and parse the
|
||||
`source: X | path: Y | title: Z` catalog lines.
|
||||
- `frontend/assets/app.js` — `.tool-call` lines (one row per `tool` frame, generic
|
||||
render of `name` + `argument`), docstring name references; `frontend/assets/shared.js`
|
||||
L159 special-cases `t.name === "read_document"` (shared-chat view).
|
||||
- `tests/` — unit: `test_agent.py`, `test_prompts.py`, `test_chat_gate.py`,
|
||||
`test_mock_tool_flow.py`, `test_sse_events.py`; integration: `test_agent_tools.py`,
|
||||
`test_chat_api.py`, `test_api.py`, `test_chats_api.py`; E2E:
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`. `README.md` L173–176 documents the current tool names/args.
|
||||
|
||||
## Objective
|
||||
Rename and reshape the three server-side agent tools to the harness-trained surface —
|
||||
**`ls`**, **`read(path)`** (single combined `source/path` argument), **`grep(pattern,
|
||||
path?)`** — so the model's trained priors emit valid calls instead of fighting the
|
||||
schema. Capabilities and all owner-locked semantics (full-document reads, fixed-
|
||||
substring locator searches, round cap, kill switch, SSE contract shape) are unchanged;
|
||||
only the tool surface, its descriptions, and the ripples (prompt copy, SSE names,
|
||||
mock, tests, docs) change.
|
||||
|
||||
## Dependencies
|
||||
- `68_search_tool` (complete) — the search semantics this phase renames (A5 locked
|
||||
match/output contract preserved).
|
||||
- `45_agent_unlimited_tools` (complete) — the round-cap design the loop keeps.
|
||||
- No todo dependencies; runs on the current head. Phase `71_scaffolding_guardrails`
|
||||
(todo) follows this one (its mock changes build on the new tool names).
|
||||
|
||||
## Tasks
|
||||
1. `01_tool_schemas.md` — `app/rag/agent.py`: `AGENT_TOOLS` → `ls` / `read` / `grep`
|
||||
schemas + descriptions; `_execute_tool` argument handling (single combined `path`),
|
||||
updated refusals; unit tests.
|
||||
2. `02_prompt_section.md` — `app/rag/prompts.py`: `TOOLS_SECTION` rewritten for
|
||||
`ls`/`read`/`grep`; prompt unit pins.
|
||||
3. `03_api_sse.md` — `app/api/chat.py`: SSE `tool`-frame `argument` derivation for the
|
||||
new shapes + module/docstring contract copy; integration tests.
|
||||
4. `04_frontend.md` — `app.js` + `shared.js` name references and the
|
||||
`read_document` special case; frontend unit pins; persisted-chat note.
|
||||
5. `05_mock_e2e_commit.md` — `mock_llm.py` flows on the new names/args, the three
|
||||
existing E2E suites updated, the new dedicated E2E suite, README, full gates,
|
||||
commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` — the new schemas (names, required/optional args,
|
||||
descriptions), `read` path resolution (combined form split at the FIRST slash, bare
|
||||
source name refusal, already-in-context, full content), `ls` scoping (no-arg full
|
||||
catalog, `path`=source scope, unknown-source refusal, empty-source "0 documents"),
|
||||
`grep` (required `pattern`, optional `path` single-doc scope, the locked A5 output
|
||||
contract: fixed substring, case-insensitive, 20 matches, 200-char lines, locator
|
||||
only), the round cap + kill switch (`agent_max_rounds=0` → `tools=None`,
|
||||
byte-identical request) unchanged.
|
||||
- Unit: `tests/unit/test_prompts.py` / `test_chat_gate.py` — `TOOLS_SECTION` copy
|
||||
(marker present, HIGH-only), byte-identical LOW prompt (this phase does not touch
|
||||
the deflection path).
|
||||
- Integration: `tests/integration/test_agent_tools.py`, `test_chat_api.py` — SSE
|
||||
`tool` frames with the new `name`/`argument` values end-to-end (mock LLM),
|
||||
`done.sources` / `query_log.sources` / `tool_calls=N` log line unchanged in meaning.
|
||||
- E2E (mandatory, house rule): NEW dedicated suite
|
||||
`tests/e2e/test_harness_aligned_tools.py`, run in isolation — the `ls` → `read`
|
||||
(combined path) → answer flow and the `grep` → `read` flow over the real UI; plus
|
||||
the three existing suites (`test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`) updated to the new names and
|
||||
green in isolation.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `AGENT_TOOLS` defines exactly `ls` (optional `path`), `read` (required `path`),
|
||||
`grep` (required `pattern`, optional `path`); the old names exist nowhere in
|
||||
`app/` (`rg "list_documents|read_document|search_documents" app/` → no matches).
|
||||
- [ ] `read` accepts the combined `source/path` form (the model's natural shape),
|
||||
splits at the first slash, appends the **full** document (A7-revised: never
|
||||
truncated); `grep` keeps the locked A5 match/output contract and is a locator
|
||||
only; `ls` prints the phase-63 `source: X | path: Y | title: Z` lines.
|
||||
- [ ] SSE `tool` frames carry the new `name` values; `argument` is the single string
|
||||
the model passed (`read`'s `path`, `grep`'s `pattern`, `ls`'s `path`) or null.
|
||||
- [ ] `BOR_AGENT_MAX_ROUNDS=0` still disables the tools entirely (request
|
||||
byte-identical to the no-tools path); the deflected path is byte-identical
|
||||
(LOW prompt and `tools=None` untouched by this phase).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov` green in
|
||||
isolation; regression suites green in isolation: `test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`.
|
||||
- [ ] README tool documentation updated to the new surface.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
|
||||
`.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner (chat, 2026-09-03):** the tool surface is remapped to the pi.dev harness
|
||||
shape — `ls` / `read(path)` / `grep(pattern, path?)` — "to match existing harnesses
|
||||
as much as possible"; `lite` stays the chat model for everything (no model swap);
|
||||
the chat model is NOT flipped to `turbo`. This supersedes the phase-37 tool
|
||||
names/args and the phase-68 tool *name* (`search_documents` → `grep`); the phase-68
|
||||
match/output contract (fixed substring, case-insensitive, 20×200, locator-only) is
|
||||
preserved verbatim.
|
||||
- **`read` is `path`-only.** No `offset`/`limit` (pi has them, but the A7-revised
|
||||
contract is "never truncated" — implementing paging would violate it; `path`-only
|
||||
still carries the trained shape, which is the point).
|
||||
- **The combined `source/path` string is the canonical document identity** in every
|
||||
tool argument, refusal, and result header (it already is in search result lines and
|
||||
`done.sources`). The old two-argument split and its self-correction/teaching
|
||||
refusals are deleted — the model's combined form is now *correct*, not a mistake to
|
||||
fix.
|
||||
- **SSE contract shape unchanged** (A15 extension honoured): `{"type":"tool",
|
||||
"name":…,"argument":…}` — only the `name` values and the `argument` derivation
|
||||
change. `delta`/`thinking`/`retry`/`done`/`error` frames are untouched.
|
||||
- **No env changes, no schema change.** `BOR_AGENT_MAX_ROUNDS` keeps its meaning
|
||||
(round cap; `0` = no-tools kill switch). Saved chats persisting old tool names
|
||||
render fine (the UI renders whatever `name`/`argument` arrive — no migration).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)"
|
||||
```
|
||||
Reference in New Issue
Block a user