feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)

.agent/phases/complete/70_harness_aligned_tools/01_tool_schemas.md

@@ -0,0 +1,96 @@
+# Task 01 — Tool Schemas + Execution: `ls` / `read(path)` / `grep(pattern, path?)`
+
+**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Reshape `AGENT_TOOLS` and `_execute_tool` in `app/rag/agent.py` to the pi.dev harness
+surface — `ls` (optional `path`), `read` (required `path`), `grep` (required
+`pattern`, optional `path`) — with the combined `source/path` form as the canonical
+document identity, keeping every locked capability intact.
+
+## Work
+1. `app/rag/agent.py` — `AGENT_TOOLS` (three OpenAI function definitions; keep the
+   module docstring's loop-contract points accurate, update the names/args references):
+   - **`ls`** — parameters: optional `path` (string): "Source name to list one
+     source's documents (e.g. 'homelab'); omit to list every document." Description:
+     "List the indexed documents as `source: X | path: Y | title: Z` lines."
+   - **`read`** — parameters: required `path` (string): "The document to add to your
+     context, as the combined `source/path` string exactly as shown in the `ls`
+     output (e.g. 'homelab/active/container_caddy/caddy.md')." Description: "Add the
+     full content of one indexed document to your context."
+   - **`grep`** — parameters: required `pattern` (string): "The exact text to search
+     for (a plain substring, not a regex)"; optional `path` (string): "Limit the
+     search to one document, as a combined `source/path` string from the `ls` output
+     (omit to search every document)." Description: "Search the indexed documents
+     for an exact string (case-insensitive) and return up to 20 matching lines as
+     `source/path:line: text` — a locator, not a context-adder: read the winner with
+     `read`."
+2. `app/rag/agent.py` — `_execute_tool` (branch on the new names; `AgentHolder`
+   semantics, round counting, and the DB accessors are unchanged):
+   - `ls`: no `path` → the full `list_catalog` listing (`"N documents:\n"` + the
+     phase-63 lines, unchanged format). With `path`: strip it; if it equals no
+     source name → refusal `No source named '{path}' — check the ls output.` (a
+     rejected call, counts in nothing); else the same listing filtered to that
+     source (`0 documents:` is a valid, counted result).
+   - `read`: require a non-blank string `path` (else
+     `read requires a string argument 'path'.`). Split at the FIRST `/` →
+     `(source, path)` (source names can never contain `/` — importer contract).
+     No `/` in the argument → refusal `No document at '{arg}' — check the ls
+     output.` Already-in-context check (seed + holder) via the resolved pair →
+     `ALREADY_IN_CONTEXT`. Unknown pair → the same `No document at '{arg}' — check
+     the ls output.` refusal (echo the argument as passed, so the model sees its
+     own form). Success → `holder.read_docs.append(doc)`, `holder.tool_calls += 1`,
+     result `f"Document {doc.source}/{doc.path}:\n{doc.content}"` (full content —
+     A7-revised, never truncated).
+   - `grep`: require a non-blank string `pattern` (else
+     `grep requires a string argument 'pattern'`). Optional `path`: if present it
+     must resolve to exactly one document (same first-slash split; unknown →
+     `No document at '{arg}' — check the ls output.`); absent → `all_documents`.
+     Everything downstream of target selection is the **phase-68 contract verbatim**:
+     case-insensitive fixed substring via `grep_document`, catalog order, global cap
+     `SEARCH_MAX_MATCHES=20`, per-line `SEARCH_LINE_LIMIT=200`, no-match lines
+     (`NO_MATCHES` / the scoped variant keyed on the resolved `source/path`),
+     counted as a successful call, `holder.read_docs` untouched (locator only).
+   - `_resolve_document`: replace with the single-input resolver the three tools
+     share — `_resolve_path(db, combined: str) -> tuple[Document | None, str, str]`
+     (split at the first slash, exact pair lookup; return the split pair so refusals
+     can echo/teach). Delete the old two-argument self-correction + "teach the split"
+     refusal branches (the combined form is now the correct input).
+   - Update the module-level refusal constants: `MISSING_READ_ARGS = "read requires
+     a string argument 'path'."`, `MISSING_SEARCH_ARGS = "grep requires a string
+     argument 'pattern'."`; `ALREADY_IN_CONTEXT` / `UNKNOWN_TOOL` unchanged.
+   - Update the module docstring: point 2 (the three tools by new name/shape),
+     point 3 (refusals — no more split-teaching; unknown source for `ls`), and the
+     phase-70 note (owner permission 2026-09-03: harness-aligned surface; phase-68
+     A5 match/output contract preserved; the combined form is canonical).
+3. `tests/unit/test_agent.py` — update every existing test to the new names/args and
+   add:
+   - schema pins: exactly `ls`/`read`/`grep`; required/optional arg sets; old names
+     absent from `AGENT_TOOLS`.
+   - `read`: combined form resolves + full content returned; bare source name
+     (`read(path='homelab')`) → the no-document refusal; missing/blank/
+     non-string `path` → the missing-args refusal; already-in-context (seed and
+     previously-read) → `ALREADY_IN_CONTEXT`; re-read counts nothing.
+   - `ls`: no-arg full catalog (format pinned: `source: X | path: Y | title: Z`);
+     scoped known source (incl. a source with 0 docs → `0 documents:` counted);
+     unknown source → refusal, not counted.
+   - `grep`: whole-KB vs `path`-scoped; the A5 pins unchanged (fixed substring,
+     case-insensitive, 20-cap in catalog order, 200-char truncation, no-match lines
+     counted, read_docs untouched); unknown `path` target → refusal.
+   - loop mechanics unchanged: round cap forcing the final no-tools answer,
+     `agent_max_rounds=0` → exactly one request with `tools=None`, rejected calls
+     consume a round but count nothing in `holder.tool_calls`.
+
+## Testing & Quality
+- Unit: `tests/unit/test_agent.py` (above) — the DB accessors stay monkeypatched
+  module-level functions (no database in unit scope).
+- Coverage: **>90%** on this task's modified code (`app/rag/agent.py`).
+
+## Completion Criteria
+- [ ] `rg "list_documents|read_document|search_documents" app/rag/agent.py` → no
+      matches; `AGENT_TOOLS` names are exactly `ls`, `read`, `grep`.
+- [ ] `uv run pytest tests/unit/test_agent.py -v --no-cov` green (old + new pins).
+- [ ] `uv run pytest --cov=app -k agent` green; `uv run ruff check . && uv run
+      pyright` clean.
+- [ ] No behavior change to the deflected path or the kill switch (pinned by the
+      updated suite).

.agent/phases/complete/70_harness_aligned_tools/02_prompt_section.md

@@ -0,0 +1,42 @@
+# Task 02 — `<tools>` Prompt Section for the New Surface
+
+**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Rewrite `TOOLS_SECTION` in `app/rag/prompts.py` so the HIGH prompt teaches the new
+`ls`/`read`/`grep` shapes in harness language — the last thing the model reads on a
+grounded turn — while the E2E mock's `<tools>`-marker keying and the LOW prompt stay
+untouched.
+
+## Work
+1. `app/rag/prompts.py` — replace the `TOOLS_SECTION` constant (keep the name, keep
+   it HIGH-only, keep it appended after the `<documents>` body, keep the `<tools>` /
+   `</tools>` markers the mock keys on):
+   - New copy (harness-aligned, names/args exactly as task 01): instruct that the
+     context may be extended with three tools — `ls` to list the indexed documents
+     (`source: X | path: Y | title: Z` lines), `grep` to locate an exact string
+     (case-insensitive; a locator, not a context-adder), and `read` to pull in one
+     document by its combined `source/path` (full content) — and that the model
+     should answer as soon as it has what it needs (the round cap is the bound; the
+     prompt does not re-state budgets — phase 45).
+   - Keep the module docstring's statement that the LOW/deflection prompt never
+     carries the section (byte-identical LOW path — this task must not change
+     `build_deflect_prompt` or `build_high_prompt`'s section order).
+2. `tests/unit/test_prompts.py` (and `test_chat_gate.py` where the section is pinned)
+   — update the `TOOLS_SECTION` copy pins: markers present; the new tool names
+   present and the old names (`list_documents`, `read_document`,
+   `search_documents`) absent from the HIGH prompt; HIGH/LOW section order and the
+   byte-identical-when-empty steering/overview behavior unchanged; the LOW prompt is
+   byte-identical to the pre-phase text (no `<tools>`, no new copy).
+
+## Testing & Quality
+- Unit: `tests/unit/test_prompts.py` / `test_chat_gate.py` pins above.
+- Coverage: **>90%** on this task's modified code (`app/rag/prompts.py`).
+
+## Completion Criteria
+- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v
+      --no-cov` green.
+- [ ] HIGH prompt still ends with the `<tools>` section (mock keying intact);
+      `rg "list_documents|read_document|search_documents" app/rag/prompts.py` → no
+      matches.
+- [ ] `uv run ruff check . && uv run pyright` clean.

.agent/phases/complete/70_harness_aligned_tools/03_api_sse.md

@@ -0,0 +1,47 @@
+# Task 03 — SSE `tool` Frames + API Contract Copy
+
+**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Point the `ToolCallPiece` → SSE `tool`-frame derivation in `app/api/chat.py` at the
+new tool shapes (new `name` values, `argument` = the single string the model passed)
+and update the module's contract docstrings; `delta`/`thinking`/`retry`/`done`/
+`error` frames and the `done.sources`/`query_log` semantics stay exactly as they are.
+
+## Work
+1. `app/api/chat.py` — the `ToolCallPiece` branch of the piece loop:
+   - `read` → `argument = piece.arguments.get("path")` (the combined `source/path`
+     as the model passed it; non-string → null).
+   - `grep` → `argument = piece.arguments.get("pattern")` (non-string → null, the
+     existing refusal case).
+   - `ls` → `argument = piece.arguments.get("path")` (the scope, if the model gave
+     one; otherwise null).
+   - One-line rule in the code comment: **`argument` is the single string argument
+     the model passed, or null** — symmetric across the three tools.
+   - `ChatToolEvent` schema (`app/schemas.py`) needs no field change (name/argument
+     are already `str`/`str | None`) — update its docstring's name examples only.
+2. `app/api/chat.py` module docstring — the "Agent document tools" paragraph: new
+   tool names/shapes, the `argument` rule above, phase-70 note (owner permission
+   2026-09-03); keep the SSE shape statement (`{"type":"tool","name":…,"argument":…}`)
+   and the `done.sources`/`query_log`/`tool_calls=N` contract wording accurate.
+3. `tests/integration/test_agent_tools.py`, `test_chat_api.py` (+ `test_api.py` /
+   `test_chats_api.py` where tool frames or names are pinned) — update to the new
+   `name` values and the `argument` rule; add pins: a `read` frame carries the
+   combined path as passed; a `grep` frame carries the pattern; an `ls` frame is
+   null-unscoped / the scope when scoped; a rejected call (unknown `read` path)
+   still emits its frame with the model's argument (frame emission is
+   execution-independent — the existing behavior, now pinned).
+
+## Testing & Quality
+- Integration: the updated suites above (mock LLM emitting the new tool names —
+  `tests/e2e/mock_llm.py` is updated in task 05; until then these integration tests
+  build their own scripted pieces, so they do not depend on the mock).
+- Coverage: **>90%** on this task's modified code (`app/api/chat.py`,
+  `app/schemas.py`).
+
+## Completion Criteria
+- [ ] `uv run pytest tests/integration/test_agent_tools.py tests/integration/test_chat_api.py -v
+      --no-cov` green.
+- [ ] `done.sources`, `query_log.sources`, and the per-turn `tool_calls=N` field
+      report exactly what they did before for the same executed calls (pins green).
+- [ ] `uv run ruff check . && uv run pyright` clean.

.agent/phases/complete/70_harness_aligned_tools/04_frontend.md

@@ -0,0 +1,52 @@
+# Task 04 — Frontend Tool-Line Rendering for the New Names
+
+**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Point the tool-line rendering at the new `ls`/`read`/`grep` names while keeping
+**legacy saved chats** (persisted with the old `list_documents` / `read_document` /
+`search_documents` names, phase 14 persistence) rendering exactly as before — no
+migration.
+
+## Work
+1. `frontend/assets/app.js` — `appendToolLine(wrap, name, argument)`:
+   - New branches: `read` + argument → `📄 Reading <code>argument</code>`;
+     `grep` + argument → `🔎 Searching for <code>argument</code>`; `ls` + argument
+     (scoped) → `🔎 Listing documents in <code>argument</code>`; `ls` unscoped →
+     `🔎 Listing documents`.
+   - **Legacy branches stay**: `read_document` (Reading), `search_documents`
+     (Searching for), everything else (Listing documents) — persisted turns from
+     before this phase must render unchanged. (Implementation: a name→kind map
+     covering both generations, or explicit if-chains — executor's choice; the pins
+     below define the contract.)
+   - All arguments through `textContent` only (the existing "path/pattern is data,
+     never markup" discipline).
+   - Update the section docstring (phase 37 reference + a phase-70 note: names
+     remapped to the harness surface; legacy names still render).
+2. `frontend/assets/shared.js` — the shared-chat tool-line helper (currently
+   `read_document` + argument → `📄 Reading <code>…</code>`, else `🔎 Listing
+   documents`): add the new names (`read` → Reading, `grep` → Searching for with
+   `<code>` argument, `ls` → Listing documents, `ls` + argument → Listing documents
+   in `<code>…</code>`), keep the legacy `read_document` branch, keep
+   `textContent`-only population. Update its docstring comment.
+3. `tests/unit/test_frontend_tool_states.py` — update/add pins (the
+   source-string house pattern):
+   - `app.js` carries branches for `read`, `grep`, `ls` (with/without argument) and
+     still carries `read_document` / `search_documents` (legacy render).
+   - `shared.js` carries the `read` / `grep` / `ls` handling and the legacy
+     `read_document` branch.
+   - Every argument population goes through `textContent` (no `innerHTML` on tool
+     lines).
+   - The old-only copy pins (e.g. exact "Listing documents" string for the default
+     case) stay green.
+
+## Testing & Quality
+- Unit: `tests/unit/test_frontend_tool_states.py` pins above.
+- Coverage: no `app/` code in this task — the **>90%** gate stays green as part of
+  the full suite run.
+
+## Completion Criteria
+- [ ] `uv run pytest tests/unit/test_frontend_tool_states.py -v --no-cov` green.
+- [ ] A persisted chat containing old tool names renders Reading/Searching/Listing
+      lines exactly as before (legacy branches pinned).
+- [ ] `uv run ruff check . && uv run pyright` clean.