feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
# Task 01 — Tool Schemas + Execution: `ls` / `read(path)` / `grep(pattern, path?)`
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Reshape `AGENT_TOOLS` and `_execute_tool` in `app/rag/agent.py` to the pi.dev harness
|
||||
surface — `ls` (optional `path`), `read` (required `path`), `grep` (required
|
||||
`pattern`, optional `path`) — with the combined `source/path` form as the canonical
|
||||
document identity, keeping every locked capability intact.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `AGENT_TOOLS` (three OpenAI function definitions; keep the
|
||||
module docstring's loop-contract points accurate, update the names/args references):
|
||||
- **`ls`** — parameters: optional `path` (string): "Source name to list one
|
||||
source's documents (e.g. 'homelab'); omit to list every document." Description:
|
||||
"List the indexed documents as `source: X | path: Y | title: Z` lines."
|
||||
- **`read`** — parameters: required `path` (string): "The document to add to your
|
||||
context, as the combined `source/path` string exactly as shown in the `ls`
|
||||
output (e.g. 'homelab/active/container_caddy/caddy.md')." Description: "Add the
|
||||
full content of one indexed document to your context."
|
||||
- **`grep`** — parameters: required `pattern` (string): "The exact text to search
|
||||
for (a plain substring, not a regex)"; optional `path` (string): "Limit the
|
||||
search to one document, as a combined `source/path` string from the `ls` output
|
||||
(omit to search every document)." Description: "Search the indexed documents
|
||||
for an exact string (case-insensitive) and return up to 20 matching lines as
|
||||
`source/path:line: text` — a locator, not a context-adder: read the winner with
|
||||
`read`."
|
||||
2. `app/rag/agent.py` — `_execute_tool` (branch on the new names; `AgentHolder`
|
||||
semantics, round counting, and the DB accessors are unchanged):
|
||||
- `ls`: no `path` → the full `list_catalog` listing (`"N documents:\n"` + the
|
||||
phase-63 lines, unchanged format). With `path`: strip it; if it equals no
|
||||
source name → refusal `No source named '{path}' — check the ls output.` (a
|
||||
rejected call, counts in nothing); else the same listing filtered to that
|
||||
source (`0 documents:` is a valid, counted result).
|
||||
- `read`: require a non-blank string `path` (else
|
||||
`read requires a string argument 'path'.`). Split at the FIRST `/` →
|
||||
`(source, path)` (source names can never contain `/` — importer contract).
|
||||
No `/` in the argument → refusal `No document at '{arg}' — check the ls
|
||||
output.` Already-in-context check (seed + holder) via the resolved pair →
|
||||
`ALREADY_IN_CONTEXT`. Unknown pair → the same `No document at '{arg}' — check
|
||||
the ls output.` refusal (echo the argument as passed, so the model sees its
|
||||
own form). Success → `holder.read_docs.append(doc)`, `holder.tool_calls += 1`,
|
||||
result `f"Document {doc.source}/{doc.path}:\n{doc.content}"` (full content —
|
||||
A7-revised, never truncated).
|
||||
- `grep`: require a non-blank string `pattern` (else
|
||||
`grep requires a string argument 'pattern'`). Optional `path`: if present it
|
||||
must resolve to exactly one document (same first-slash split; unknown →
|
||||
`No document at '{arg}' — check the ls output.`); absent → `all_documents`.
|
||||
Everything downstream of target selection is the **phase-68 contract verbatim**:
|
||||
case-insensitive fixed substring via `grep_document`, catalog order, global cap
|
||||
`SEARCH_MAX_MATCHES=20`, per-line `SEARCH_LINE_LIMIT=200`, no-match lines
|
||||
(`NO_MATCHES` / the scoped variant keyed on the resolved `source/path`),
|
||||
counted as a successful call, `holder.read_docs` untouched (locator only).
|
||||
- `_resolve_document`: replace with the single-input resolver the three tools
|
||||
share — `_resolve_path(db, combined: str) -> tuple[Document | None, str, str]`
|
||||
(split at the first slash, exact pair lookup; return the split pair so refusals
|
||||
can echo/teach). Delete the old two-argument self-correction + "teach the split"
|
||||
refusal branches (the combined form is now the correct input).
|
||||
- Update the module-level refusal constants: `MISSING_READ_ARGS = "read requires
|
||||
a string argument 'path'."`, `MISSING_SEARCH_ARGS = "grep requires a string
|
||||
argument 'pattern'."`; `ALREADY_IN_CONTEXT` / `UNKNOWN_TOOL` unchanged.
|
||||
- Update the module docstring: point 2 (the three tools by new name/shape),
|
||||
point 3 (refusals — no more split-teaching; unknown source for `ls`), and the
|
||||
phase-70 note (owner permission 2026-09-03: harness-aligned surface; phase-68
|
||||
A5 match/output contract preserved; the combined form is canonical).
|
||||
3. `tests/unit/test_agent.py` — update every existing test to the new names/args and
|
||||
add:
|
||||
- schema pins: exactly `ls`/`read`/`grep`; required/optional arg sets; old names
|
||||
absent from `AGENT_TOOLS`.
|
||||
- `read`: combined form resolves + full content returned; bare source name
|
||||
(`read(path='homelab')`) → the no-document refusal; missing/blank/
|
||||
non-string `path` → the missing-args refusal; already-in-context (seed and
|
||||
previously-read) → `ALREADY_IN_CONTEXT`; re-read counts nothing.
|
||||
- `ls`: no-arg full catalog (format pinned: `source: X | path: Y | title: Z`);
|
||||
scoped known source (incl. a source with 0 docs → `0 documents:` counted);
|
||||
unknown source → refusal, not counted.
|
||||
- `grep`: whole-KB vs `path`-scoped; the A5 pins unchanged (fixed substring,
|
||||
case-insensitive, 20-cap in catalog order, 200-char truncation, no-match lines
|
||||
counted, read_docs untouched); unknown `path` target → refusal.
|
||||
- loop mechanics unchanged: round cap forcing the final no-tools answer,
|
||||
`agent_max_rounds=0` → exactly one request with `tools=None`, rejected calls
|
||||
consume a round but count nothing in `holder.tool_calls`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` (above) — the DB accessors stay monkeypatched
|
||||
module-level functions (no database in unit scope).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "list_documents|read_document|search_documents" app/rag/agent.py` → no
|
||||
matches; `AGENT_TOOLS` names are exactly `ls`, `read`, `grep`.
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py -v --no-cov` green (old + new pins).
|
||||
- [ ] `uv run pytest --cov=app -k agent` green; `uv run ruff check . && uv run
|
||||
pyright` clean.
|
||||
- [ ] No behavior change to the deflected path or the kill switch (pinned by the
|
||||
updated suite).
|
||||
@@ -0,0 +1,42 @@
|
||||
# Task 02 — `<tools>` Prompt Section for the New Surface
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Rewrite `TOOLS_SECTION` in `app/rag/prompts.py` so the HIGH prompt teaches the new
|
||||
`ls`/`read`/`grep` shapes in harness language — the last thing the model reads on a
|
||||
grounded turn — while the E2E mock's `<tools>`-marker keying and the LOW prompt stay
|
||||
untouched.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — replace the `TOOLS_SECTION` constant (keep the name, keep
|
||||
it HIGH-only, keep it appended after the `<documents>` body, keep the `<tools>` /
|
||||
`</tools>` markers the mock keys on):
|
||||
- New copy (harness-aligned, names/args exactly as task 01): instruct that the
|
||||
context may be extended with three tools — `ls` to list the indexed documents
|
||||
(`source: X | path: Y | title: Z` lines), `grep` to locate an exact string
|
||||
(case-insensitive; a locator, not a context-adder), and `read` to pull in one
|
||||
document by its combined `source/path` (full content) — and that the model
|
||||
should answer as soon as it has what it needs (the round cap is the bound; the
|
||||
prompt does not re-state budgets — phase 45).
|
||||
- Keep the module docstring's statement that the LOW/deflection prompt never
|
||||
carries the section (byte-identical LOW path — this task must not change
|
||||
`build_deflect_prompt` or `build_high_prompt`'s section order).
|
||||
2. `tests/unit/test_prompts.py` (and `test_chat_gate.py` where the section is pinned)
|
||||
— update the `TOOLS_SECTION` copy pins: markers present; the new tool names
|
||||
present and the old names (`list_documents`, `read_document`,
|
||||
`search_documents`) absent from the HIGH prompt; HIGH/LOW section order and the
|
||||
byte-identical-when-empty steering/overview behavior unchanged; the LOW prompt is
|
||||
byte-identical to the pre-phase text (no `<tools>`, no new copy).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_prompts.py` / `test_chat_gate.py` pins above.
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/prompts.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v
|
||||
--no-cov` green.
|
||||
- [ ] HIGH prompt still ends with the `<tools>` section (mock keying intact);
|
||||
`rg "list_documents|read_document|search_documents" app/rag/prompts.py` → no
|
||||
matches.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Task 03 — SSE `tool` Frames + API Contract Copy
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Point the `ToolCallPiece` → SSE `tool`-frame derivation in `app/api/chat.py` at the
|
||||
new tool shapes (new `name` values, `argument` = the single string the model passed)
|
||||
and update the module's contract docstrings; `delta`/`thinking`/`retry`/`done`/
|
||||
`error` frames and the `done.sources`/`query_log` semantics stay exactly as they are.
|
||||
|
||||
## Work
|
||||
1. `app/api/chat.py` — the `ToolCallPiece` branch of the piece loop:
|
||||
- `read` → `argument = piece.arguments.get("path")` (the combined `source/path`
|
||||
as the model passed it; non-string → null).
|
||||
- `grep` → `argument = piece.arguments.get("pattern")` (non-string → null, the
|
||||
existing refusal case).
|
||||
- `ls` → `argument = piece.arguments.get("path")` (the scope, if the model gave
|
||||
one; otherwise null).
|
||||
- One-line rule in the code comment: **`argument` is the single string argument
|
||||
the model passed, or null** — symmetric across the three tools.
|
||||
- `ChatToolEvent` schema (`app/schemas.py`) needs no field change (name/argument
|
||||
are already `str`/`str | None`) — update its docstring's name examples only.
|
||||
2. `app/api/chat.py` module docstring — the "Agent document tools" paragraph: new
|
||||
tool names/shapes, the `argument` rule above, phase-70 note (owner permission
|
||||
2026-09-03); keep the SSE shape statement (`{"type":"tool","name":…,"argument":…}`)
|
||||
and the `done.sources`/`query_log`/`tool_calls=N` contract wording accurate.
|
||||
3. `tests/integration/test_agent_tools.py`, `test_chat_api.py` (+ `test_api.py` /
|
||||
`test_chats_api.py` where tool frames or names are pinned) — update to the new
|
||||
`name` values and the `argument` rule; add pins: a `read` frame carries the
|
||||
combined path as passed; a `grep` frame carries the pattern; an `ls` frame is
|
||||
null-unscoped / the scope when scoped; a rejected call (unknown `read` path)
|
||||
still emits its frame with the model's argument (frame emission is
|
||||
execution-independent — the existing behavior, now pinned).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: the updated suites above (mock LLM emitting the new tool names —
|
||||
`tests/e2e/mock_llm.py` is updated in task 05; until then these integration tests
|
||||
build their own scripted pieces, so they do not depend on the mock).
|
||||
- Coverage: **>90%** on this task's modified code (`app/api/chat.py`,
|
||||
`app/schemas.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/integration/test_agent_tools.py tests/integration/test_chat_api.py -v
|
||||
--no-cov` green.
|
||||
- [ ] `done.sources`, `query_log.sources`, and the per-turn `tool_calls=N` field
|
||||
report exactly what they did before for the same executed calls (pins green).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Task 04 — Frontend Tool-Line Rendering for the New Names
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Point the tool-line rendering at the new `ls`/`read`/`grep` names while keeping
|
||||
**legacy saved chats** (persisted with the old `list_documents` / `read_document` /
|
||||
`search_documents` names, phase 14 persistence) rendering exactly as before — no
|
||||
migration.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — `appendToolLine(wrap, name, argument)`:
|
||||
- New branches: `read` + argument → `📄 Reading <code>argument</code>`;
|
||||
`grep` + argument → `🔎 Searching for <code>argument</code>`; `ls` + argument
|
||||
(scoped) → `🔎 Listing documents in <code>argument</code>`; `ls` unscoped →
|
||||
`🔎 Listing documents`.
|
||||
- **Legacy branches stay**: `read_document` (Reading), `search_documents`
|
||||
(Searching for), everything else (Listing documents) — persisted turns from
|
||||
before this phase must render unchanged. (Implementation: a name→kind map
|
||||
covering both generations, or explicit if-chains — executor's choice; the pins
|
||||
below define the contract.)
|
||||
- All arguments through `textContent` only (the existing "path/pattern is data,
|
||||
never markup" discipline).
|
||||
- Update the section docstring (phase 37 reference + a phase-70 note: names
|
||||
remapped to the harness surface; legacy names still render).
|
||||
2. `frontend/assets/shared.js` — the shared-chat tool-line helper (currently
|
||||
`read_document` + argument → `📄 Reading <code>…</code>`, else `🔎 Listing
|
||||
documents`): add the new names (`read` → Reading, `grep` → Searching for with
|
||||
`<code>` argument, `ls` → Listing documents, `ls` + argument → Listing documents
|
||||
in `<code>…</code>`), keep the legacy `read_document` branch, keep
|
||||
`textContent`-only population. Update its docstring comment.
|
||||
3. `tests/unit/test_frontend_tool_states.py` — update/add pins (the
|
||||
source-string house pattern):
|
||||
- `app.js` carries branches for `read`, `grep`, `ls` (with/without argument) and
|
||||
still carries `read_document` / `search_documents` (legacy render).
|
||||
- `shared.js` carries the `read` / `grep` / `ls` handling and the legacy
|
||||
`read_document` branch.
|
||||
- Every argument population goes through `textContent` (no `innerHTML` on tool
|
||||
lines).
|
||||
- The old-only copy pins (e.g. exact "Listing documents" string for the default
|
||||
case) stay green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_frontend_tool_states.py` pins above.
|
||||
- Coverage: no `app/` code in this task — the **>90%** gate stays green as part of
|
||||
the full suite run.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_frontend_tool_states.py -v --no-cov` green.
|
||||
- [ ] A persisted chat containing old tool names renders Reading/Searching/Listing
|
||||
lines exactly as before (legacy branches pinned).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,143 @@
|
||||
# Phase 70 — Harness-Aligned Agent Tools: `ls` / `read(path)` / `grep(pattern, path?)`
|
||||
|
||||
**Source:** owner request (chat, 2026-09-03) — live incident: the question "What are the
|
||||
correct llama.cpp arguments for Qwen 3.8?" was answered with the raw model text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>`
|
||||
(`query_log`: top_score=0.040, fts_hits=0, deflected=true — the turn was DEFLECTED, so
|
||||
no tools were even offered). Diagnosis: the chat model (`lite` per `.env`; owner keeps
|
||||
`lite` for everything — it's faster, owner decision 2026-09-03) reaches for the tool
|
||||
shapes it was trained on — a `read` tool taking a **single** `path` argument — and our
|
||||
`read_document(source, path)` fights that prior (the `_resolve_document` combined-form
|
||||
self-correction and "teach the split" refusals are the scar tissue). Owner direction:
|
||||
"match existing harnesses as much as possible" — the mapping below mirrors the
|
||||
**pi.dev** tool surface (`dist/core/tools/`: `read{path}`, `ls{path?}`,
|
||||
`grep{pattern, path?}`).
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md` (this phase reshapes the tools
|
||||
that story delivered; phase 68's search contract rides along renamed)
|
||||
**Context:**
|
||||
- `app/rag/agent.py` — `AGENT_TOOLS` (OpenAI function definitions:
|
||||
`list_documents` / `read_document(source, path)` / `search_documents(pattern,
|
||||
source?, path?)`), `_execute_tool` (execution + refusals: `ALREADY_IN_CONTEXT`,
|
||||
`UNKNOWN_TOOL`, `MISSING_READ_ARGS`, `MISSING_SEARCH_ARGS`; combined-form
|
||||
self-correction in `_resolve_document`), `AgentHolder`, `run_agent` (round cap
|
||||
`BOR_AGENT_MAX_ROUNDS`, default 10; `0` = no-tools kill switch).
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only, ends the prompt; the E2E
|
||||
mock keys off the `<tools>` marker's *presence*, not the wording).
|
||||
- `app/api/chat.py` — the `ToolCallPiece` → SSE `{"type":"tool","name":…,"argument":…}`
|
||||
derivation (read → `source/path`, search → pattern, list → null), `done.sources`,
|
||||
per-turn log line (`tool_calls=N`).
|
||||
- `tests/e2e/mock_llm.py` — the deterministic mock flows keyed on `TOOLS_TRIGGER`
|
||||
("use your tools"), `MULTI_READ_TRIGGER`, `SEARCH_TRIGGER` + the `<tools>` marker;
|
||||
the flows emit `tool_calls` deltas with the current names/args and parse the
|
||||
`source: X | path: Y | title: Z` catalog lines.
|
||||
- `frontend/assets/app.js` — `.tool-call` lines (one row per `tool` frame, generic
|
||||
render of `name` + `argument`), docstring name references; `frontend/assets/shared.js`
|
||||
L159 special-cases `t.name === "read_document"` (shared-chat view).
|
||||
- `tests/` — unit: `test_agent.py`, `test_prompts.py`, `test_chat_gate.py`,
|
||||
`test_mock_tool_flow.py`, `test_sse_events.py`; integration: `test_agent_tools.py`,
|
||||
`test_chat_api.py`, `test_api.py`, `test_chats_api.py`; E2E:
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`. `README.md` L173–176 documents the current tool names/args.
|
||||
|
||||
## Objective
|
||||
Rename and reshape the three server-side agent tools to the harness-trained surface —
|
||||
**`ls`**, **`read(path)`** (single combined `source/path` argument), **`grep(pattern,
|
||||
path?)`** — so the model's trained priors emit valid calls instead of fighting the
|
||||
schema. Capabilities and all owner-locked semantics (full-document reads, fixed-
|
||||
substring locator searches, round cap, kill switch, SSE contract shape) are unchanged;
|
||||
only the tool surface, its descriptions, and the ripples (prompt copy, SSE names,
|
||||
mock, tests, docs) change.
|
||||
|
||||
## Dependencies
|
||||
- `68_search_tool` (complete) — the search semantics this phase renames (A5 locked
|
||||
match/output contract preserved).
|
||||
- `45_agent_unlimited_tools` (complete) — the round-cap design the loop keeps.
|
||||
- No todo dependencies; runs on the current head. Phase `71_scaffolding_guardrails`
|
||||
(todo) follows this one (its mock changes build on the new tool names).
|
||||
|
||||
## Tasks
|
||||
1. `01_tool_schemas.md` — `app/rag/agent.py`: `AGENT_TOOLS` → `ls` / `read` / `grep`
|
||||
schemas + descriptions; `_execute_tool` argument handling (single combined `path`),
|
||||
updated refusals; unit tests.
|
||||
2. `02_prompt_section.md` — `app/rag/prompts.py`: `TOOLS_SECTION` rewritten for
|
||||
`ls`/`read`/`grep`; prompt unit pins.
|
||||
3. `03_api_sse.md` — `app/api/chat.py`: SSE `tool`-frame `argument` derivation for the
|
||||
new shapes + module/docstring contract copy; integration tests.
|
||||
4. `04_frontend.md` — `app.js` + `shared.js` name references and the
|
||||
`read_document` special case; frontend unit pins; persisted-chat note.
|
||||
5. `05_mock_e2e_commit.md` — `mock_llm.py` flows on the new names/args, the three
|
||||
existing E2E suites updated, the new dedicated E2E suite, README, full gates,
|
||||
commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` — the new schemas (names, required/optional args,
|
||||
descriptions), `read` path resolution (combined form split at the FIRST slash, bare
|
||||
source name refusal, already-in-context, full content), `ls` scoping (no-arg full
|
||||
catalog, `path`=source scope, unknown-source refusal, empty-source "0 documents"),
|
||||
`grep` (required `pattern`, optional `path` single-doc scope, the locked A5 output
|
||||
contract: fixed substring, case-insensitive, 20 matches, 200-char lines, locator
|
||||
only), the round cap + kill switch (`agent_max_rounds=0` → `tools=None`,
|
||||
byte-identical request) unchanged.
|
||||
- Unit: `tests/unit/test_prompts.py` / `test_chat_gate.py` — `TOOLS_SECTION` copy
|
||||
(marker present, HIGH-only), byte-identical LOW prompt (this phase does not touch
|
||||
the deflection path).
|
||||
- Integration: `tests/integration/test_agent_tools.py`, `test_chat_api.py` — SSE
|
||||
`tool` frames with the new `name`/`argument` values end-to-end (mock LLM),
|
||||
`done.sources` / `query_log.sources` / `tool_calls=N` log line unchanged in meaning.
|
||||
- E2E (mandatory, house rule): NEW dedicated suite
|
||||
`tests/e2e/test_harness_aligned_tools.py`, run in isolation — the `ls` → `read`
|
||||
(combined path) → answer flow and the `grep` → `read` flow over the real UI; plus
|
||||
the three existing suites (`test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`) updated to the new names and
|
||||
green in isolation.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `AGENT_TOOLS` defines exactly `ls` (optional `path`), `read` (required `path`),
|
||||
`grep` (required `pattern`, optional `path`); the old names exist nowhere in
|
||||
`app/` (`rg "list_documents|read_document|search_documents" app/` → no matches).
|
||||
- [ ] `read` accepts the combined `source/path` form (the model's natural shape),
|
||||
splits at the first slash, appends the **full** document (A7-revised: never
|
||||
truncated); `grep` keeps the locked A5 match/output contract and is a locator
|
||||
only; `ls` prints the phase-63 `source: X | path: Y | title: Z` lines.
|
||||
- [ ] SSE `tool` frames carry the new `name` values; `argument` is the single string
|
||||
the model passed (`read`'s `path`, `grep`'s `pattern`, `ls`'s `path`) or null.
|
||||
- [ ] `BOR_AGENT_MAX_ROUNDS=0` still disables the tools entirely (request
|
||||
byte-identical to the no-tools path); the deflected path is byte-identical
|
||||
(LOW prompt and `tools=None` untouched by this phase).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov` green in
|
||||
isolation; regression suites green in isolation: `test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`.
|
||||
- [ ] README tool documentation updated to the new surface.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
|
||||
`.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner (chat, 2026-09-03):** the tool surface is remapped to the pi.dev harness
|
||||
shape — `ls` / `read(path)` / `grep(pattern, path?)` — "to match existing harnesses
|
||||
as much as possible"; `lite` stays the chat model for everything (no model swap);
|
||||
the chat model is NOT flipped to `turbo`. This supersedes the phase-37 tool
|
||||
names/args and the phase-68 tool *name* (`search_documents` → `grep`); the phase-68
|
||||
match/output contract (fixed substring, case-insensitive, 20×200, locator-only) is
|
||||
preserved verbatim.
|
||||
- **`read` is `path`-only.** No `offset`/`limit` (pi has them, but the A7-revised
|
||||
contract is "never truncated" — implementing paging would violate it; `path`-only
|
||||
still carries the trained shape, which is the point).
|
||||
- **The combined `source/path` string is the canonical document identity** in every
|
||||
tool argument, refusal, and result header (it already is in search result lines and
|
||||
`done.sources`). The old two-argument split and its self-correction/teaching
|
||||
refusals are deleted — the model's combined form is now *correct*, not a mistake to
|
||||
fix.
|
||||
- **SSE contract shape unchanged** (A15 extension honoured): `{"type":"tool",
|
||||
"name":…,"argument":…}` — only the `name` values and the `argument` derivation
|
||||
change. `delta`/`thinking`/`retry`/`done`/`error` frames are untouched.
|
||||
- **No env changes, no schema change.** `BOR_AGENT_MAX_ROUNDS` keeps its meaning
|
||||
(round cap; `0` = no-tools kill switch). Saved chats persisting old tool names
|
||||
render fine (the UI renders whatever `name`/`argument` arrive — no migration).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)"
|
||||
```
|
||||
@@ -0,0 +1,78 @@
|
||||
# Task 05 — Mock LLM, E2E Suites, README, Gates, Commit
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Bring the E2E world onto the new surface: the mock LLM's deterministic tool flows
|
||||
emit `ls`/`read`/`grep` calls with the new argument shapes, the three existing tool
|
||||
E2E suites assert the new names/frames, and a NEW dedicated suite pins the remap
|
||||
end-to-end. Then the full quality gates and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — update the deterministic flows (trigger strings
|
||||
`TOOLS_TRIGGER` / `MULTI_READ_TRIGGER` / `SEARCH_TRIGGER` and the `<tools>`
|
||||
marker keying stay unchanged):
|
||||
- READ flow: request 1 streams `ls` (no arguments, synthetic id `call_0`,
|
||||
`finish_reason: "tool_calls"`); request 2 (a `tool`-role catalog result in the
|
||||
messages) streams `read` with the **combined** `source/path` parsed from the
|
||||
first `source: X | path: Y` catalog line (the mock joins them — the catalog
|
||||
format is unchanged, so this is the only parse change); request 3 (a
|
||||
`tool`-role read result — content starting with the `Document source/path:`
|
||||
header) streams the plain answer.
|
||||
- MULTI-READ flow: same, `read` on the first then second combined path.
|
||||
- SEARCH flow: request 1 streams `grep` with the pattern argument (the existing
|
||||
SEARCH_TRIGGER pattern value); request 2 (a `tool`-role search result —
|
||||
`source/path:line: text` lines) streams the plain answer. Update the
|
||||
`tool`-role result detection strings where they key on the old headers.
|
||||
- Update the module docstring's flow descriptions (names/args) + the phase-70
|
||||
note.
|
||||
2. Existing E2E suites — update assertions to the new names/frames (behavior
|
||||
otherwise unchanged): `tests/e2e/test_agent_document_tools.py` (tool frame
|
||||
`name` values: `ls` then `read`; the `.tool-call` line copy "📄 Reading …"),
|
||||
`tests/e2e/test_agent_unlimited_tools.py` (multi-read frames),
|
||||
`tests/e2e/test_search_tool.py` (the `grep` frame + pattern argument).
|
||||
3. NEW `tests/e2e/test_harness_aligned_tools.py` (the phase's dedicated suite,
|
||||
house pattern, run in isolation):
|
||||
- Grounded turn with the READ trigger: the UI shows the `ls` line then the
|
||||
`📄 Reading <source/path>` line, the answer streams, and the done-state
|
||||
sources include the read document; no raw tool markup anywhere in the DOM.
|
||||
- Grounded turn with the SEARCH trigger: the `🔎 Searching for <pattern>` line
|
||||
then the answer.
|
||||
- The SSE wire itself (in-browser `fetch` capture or the existing SSE-capture
|
||||
house pattern): `tool` frames carry `name` ∈ {`ls`, `read`, `grep`} and the
|
||||
`argument` rule (read → combined path as passed; grep → pattern; ls → null
|
||||
unscoped).
|
||||
4. `README.md` — the agent-tools section (L173–176): document `ls` / `read(path)` /
|
||||
`grep(pattern, path?)` with the combined-path identity and the A5 locator
|
||||
semantics; the `.env.example` comment near `BOR_AGENT_MAX_ROUNDS` stays accurate
|
||||
(no env changes this phase).
|
||||
5. Gates + commit:
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_harness_aligned_tools.py`, then the regression
|
||||
suites `test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`, `test_chat_rag.py`.
|
||||
- One atomic commit (message below); move
|
||||
`.agent/phases/todo/70_harness_aligned_tools/` →
|
||||
`.agent/phases/complete/70_harness_aligned_tools/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new dedicated suite + the four regression suites (isolation runs).
|
||||
- Coverage: **>90%** on `app/` (phase-level gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "list_documents|read_document|search_documents" app/ frontend/ tests/
|
||||
README.md` → matches only in legacy-render pins/comments explicitly marked
|
||||
legacy (the frontend legacy branches and their unit pins).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov` green in
|
||||
isolation; regression suites green in isolation.
|
||||
- [ ] README documents the new surface; no stale tool-name copy in `README.md` /
|
||||
`.env.example` / `frontend/`.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)"
|
||||
```
|
||||
@@ -0,0 +1,152 @@
|
||||
# Phase 71 — Tool-Scaffolding Guardrails: Deterministic Strip + One Bounded Recovery
|
||||
|
||||
**Source:** owner request (chat, 2026-09-03) — same incident as phase 70: the
|
||||
deflected answer streamed the raw model text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>` into
|
||||
the UI (the model's own `<|…|>` chat-template tool syntax, emitted as plain
|
||||
`delta.content` even though no tools were offered). Owner direction: "We also need
|
||||
guardrails for situations like this… **deterministic guardrails only** right now,
|
||||
forget using a model for that" — no model of any kind (no `lite` classifier, no
|
||||
model-authored repair) in the guardrail path; `lite` stays the chat model for
|
||||
everything (it's faster).
|
||||
**Story:** n/a (owner request from chat — tool-scaffolding guardrails, 2026-09-03)
|
||||
**Context:**
|
||||
- `app/rag/llm.py` — `chat_stream` yields `StreamPiece("content", delta.content)`
|
||||
verbatim (L~`content = delta.content; if content: yield`); thinking pieces
|
||||
(`delta.reasoning_content`) pass through raw by design; tool-call materialization
|
||||
happens at stream end (after the `async for`); phase-48 teardown closes the
|
||||
endpoint stream on every exit (must stay intact). `chat_stream_retried` (phase 67)
|
||||
wraps it with the retry-before-first-piece rule.
|
||||
- `app/rag/agent.py` — `run_agent`'s round loop: a round with no `ToolCallPiece`s
|
||||
ends the turn (`if not calls: return`); the round cap forces one final
|
||||
`tools=None` request (the "forced final answer" pattern this phase reuses for
|
||||
recovery); `AgentHolder` (read_docs / tool_calls) is unchanged.
|
||||
- `app/api/chat.py` — the piece loop (thinking/tool/retry/delta handling +
|
||||
`thinking_chars` counter), the deflected path (`chat_stream_retried(...,
|
||||
tools=None)` directly), the `LLMError` → terminal error-frame handler, the
|
||||
per-turn log line (ends `…total_ms=N retries=N`).
|
||||
- `app/rag/prompts.py` — `build_deflect_prompt`'s `DEFLECT_MODE` body (the E2E mock
|
||||
keys on the marker's *presence*, not the wording — an appended line is safe).
|
||||
- `tests/e2e/mock_llm.py` — the trigger-string flow table (task 05 adds the
|
||||
scaffolding triggers; phase-70 names apply).
|
||||
- Tests to extend: `tests/unit/test_llm_client.py`, `test_agent.py`,
|
||||
`tests/integration/test_chat_api.py` (log-line + error-frame pins),
|
||||
`tests/unit/test_prompts.py`.
|
||||
|
||||
## Objective
|
||||
Raw tool-scaffolding tokens can never reach the user as answer text: a deterministic
|
||||
streaming filter strips known scaffolding from `delta.content` as it flows, and a
|
||||
round/turn whose visible content ends up empty (scaffolding was the whole "answer")
|
||||
gets **one** bounded, deterministic recovery (same turn, `tools=None`, a fixed
|
||||
harness-owned correction line in the system prompt); if the recovery also comes back
|
||||
empty, the turn settles with a dedicated structured error frame. No model is used to
|
||||
detect or repair anything.
|
||||
|
||||
## Dependencies
|
||||
- `70_harness_aligned_tools` (todo) — runs first: the mock/prompt/code state this
|
||||
phase builds on (new tool names in the mock flows and prompt).
|
||||
- `67_llm_retry` (complete) — `chat_stream_retried`, the primitive every request
|
||||
(including recoveries) goes through.
|
||||
- `48_stop_generation` (complete) — the phase-48 stream-teardown contract the
|
||||
filter integration must not break.
|
||||
|
||||
## Tasks
|
||||
1. `01_filter.md` — `app/rag/scaffolding.py` (new): the pattern registry +
|
||||
`ScaffoldingFilter` streaming state machine, with the boundary test matrix.
|
||||
2. `02_llm_integration.md` — `app/rag/llm.py`: `chat_stream`/`chat_stream_retried`
|
||||
accept the filter; content deltas are filtered, the tail flushed before tool
|
||||
materialization; `None` = byte-identical raw path.
|
||||
3. `03_recovery_policy.md` — `app/rag/agent.py` + `app/api/chat.py`: the empty-round
|
||||
recovery (one per turn, `tools=None`, correction constant),
|
||||
`MalformedReplyError`, the dedicated error copy, and the per-turn log line's
|
||||
`scaffold_stripped=N` field.
|
||||
4. `04_deflect_prompt.md` — the plain-text line in the `DEFLECT_MODE` body
|
||||
(prevention; owner-permitted LOW-prompt change).
|
||||
5. `05_e2e_commit.md` — the mock scaffolding triggers, the dedicated E2E suite,
|
||||
full gates, commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (new `tests/unit/test_scaffolding_filter.py`): the filter matrix — span in
|
||||
one chunk; span split across chunks at **every** boundary offset of the start
|
||||
token; multiple spans in one chunk; standalone `<|tool_calls|>` / `<|tool_call|>`
|
||||
tokens stripped; look-alikes **not** stripped (prose containing the words
|
||||
"tool_call" or `tool_call_start` without the `<|…|>` delimiters, an unknown
|
||||
`<|some_other_token|>`, a lone `<|tool_call_end|>` without a start); a partial
|
||||
start token at stream end → `flush()` emits it as-is (no false-positive strip);
|
||||
`stripped_chars` accounting; empty chunks.
|
||||
- Unit: `tests/unit/test_llm_client.py` — `chat_stream` with a filter (content
|
||||
filtered, thinking raw, `None` = raw pass-through pinned byte-identical, flush
|
||||
order: flushed tail content precedes tool-call pieces, phase-48 teardown intact).
|
||||
- Unit: `tests/unit/test_agent.py` — grounded recovery matrix (scaffolding-only
|
||||
round → exactly one recovery request: `tools=None` + correction line in the
|
||||
system prompt + fresh filter → clean answer ends the turn; scaffolding twice →
|
||||
`MalformedReplyError`; scaffolding + real content → clean answer, **no**
|
||||
recovery; round cap and kill switch unchanged).
|
||||
- Integration: `tests/integration/test_chat_api.py` — deflected-path recovery matrix
|
||||
(same shapes over the SSE endpoint: clean recovery → `done` frame; terminal → the
|
||||
dedicated error frame, no `done`, no `query_log` row — same terminal semantics as
|
||||
today's `LLMError`); log line carries `scaffold_stripped=N` (0 when nothing was
|
||||
stripped — the field is uniform, the phase-67 `retries=N` pattern).
|
||||
- E2E (mandatory, house rule): NEW dedicated suite
|
||||
`tests/e2e/test_tool_scaffolding_guardrails.py`, run in isolation — recovery case
|
||||
(raw tokens never in the DOM, clean answer shown) and terminal case (error state,
|
||||
no raw tokens, the app stays usable).
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start" frontend/` → no matches (no scaffolding rendering
|
||||
path); the filter lives in `app/rag/scaffolding.py` as a pure module (no I/O,
|
||||
no model calls).
|
||||
- [ ] A content stream of pure scaffolding yields zero `delta` frames; a mixed
|
||||
stream yields the clean remainder; thinking frames are never filtered.
|
||||
- [ ] Exactly one recovery per turn (grounded and deflected paths); the recovery
|
||||
request is `tools=None` with the fixed correction line in the system prompt;
|
||||
a second empty reply settles with the error frame
|
||||
"The model returned a malformed reply — please try again."
|
||||
- [ ] The per-turn log line ends `…retries=N scaffold_stripped=N`; `scaffold_stripped=0`
|
||||
on clean turns (uniform field).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation:
|
||||
`test_harness_aligned_tools.py`, `test_chat_rag.py`, `test_agent_document_tools.py`.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
|
||||
`.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner (chat, 2026-09-03): deterministic only.** No model — `lite` or any other —
|
||||
participates in detection or repair. The guardrail is a fixed pattern registry + a
|
||||
fixed retry policy. (A model-based classifier/repair was proposed and explicitly
|
||||
rejected for now — if it is ever wanted, it is a later phase with its own
|
||||
permission.)
|
||||
- **`lite` stays the chat model** (owner: "I want to use lite for everything since
|
||||
it's way faster") — the guardrail is what protects the UX while `lite` is the
|
||||
model; no `.env` change in this phase.
|
||||
- **The pattern registry is the extension point.** Initial entries: the observed
|
||||
span form `<|tool_call_start|>…<|tool_call_end|>` (non-greedy, any text between)
|
||||
plus the standalone sibling tokens `<|tool_calls|>` and `<|tool_call|>` from the
|
||||
same tokenizer family. Every strip logs a warning with the stripped span
|
||||
(truncated to 200 chars) — that log line is how a new format gets captured and
|
||||
added (pattern + unit fixture), keeping the registry honest (every entry traces
|
||||
to an observed capture or the initial incident).
|
||||
- **Content only, thinking never filtered.** The Thinking block is the model's raw
|
||||
reasoning by design (phase 17) and stays raw (collapsible); the guardrail protects
|
||||
the answer, not the scratchpad.
|
||||
- **Recovery is a fixed policy, not a conversation.** One extra request per turn,
|
||||
same messages with the harness-owned constant folded into the system prompt
|
||||
(single system message — provider-safe), `tools=None`, a fresh filter, the same
|
||||
phase-67 retry budget. At most one recovery; the second empty reply is terminal.
|
||||
A round with real visible content plus scaffolding needs no recovery (the clean
|
||||
content stands).
|
||||
- **Terminal semantics follow the existing error pattern.** A terminal malformed
|
||||
turn settles with a structured `error` frame (dedicated copy), writes no
|
||||
`query_log` row, and the UI shows the existing error state — byte-for-byte the
|
||||
same shape as today's `LLMError` terminal path.
|
||||
- **`MalformedReplyError` subclasses `LLMError`** and is raised only by the recovery
|
||||
policy (never from inside a stream, so `chat_stream_retried`'s retry rule never
|
||||
sees it); `chat.py` catches it before the generic `LLMError` handler.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# Task 01 — `app/rag/scaffolding.py`: Pattern Registry + Streaming Filter
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
A pure-Python streaming filter that removes known tool-scaffolding from answer text
|
||||
as it flows — no I/O, no model, fully unit-testable — plus the pattern registry that
|
||||
defines "known" (every entry traces to an observed capture).
|
||||
|
||||
## Work
|
||||
1. `app/rag/scaffolding.py` (new module):
|
||||
- `SCAFFOLD_PATTERNS: tuple[re.Pattern, ...]` — module-level, compiled:
|
||||
1. `re.compile(r"<\|tool_call_start\|>[\s\S]*?<\|tool_call_end\|>")` — the
|
||||
observed span (incident 2026-09-03), non-greedy so multiple spans each
|
||||
strip to their own end token.
|
||||
2. `re.compile(r"<\|tool_calls\|>")` — standalone sibling token (same
|
||||
tokenizer family).
|
||||
3. `re.compile(r"<\|tool_call\|>")` — standalone sibling token.
|
||||
Module docstring: the registry is the extension point — a new entry needs an
|
||||
observed capture (the strip warning log, task 03) + a unit fixture here; no
|
||||
speculative entries.
|
||||
- `class ScaffoldingFilter` — streaming state machine over a **content** stream:
|
||||
- `feed(chunk: str) -> str` — appends to an internal pending buffer and
|
||||
returns the clean text safe to emit now. Algorithm: repeatedly (a) take the
|
||||
leftmost complete match among `SCAFFOLD_PATTERNS` in the pending buffer —
|
||||
drop it (count into `stripped_chars`) and continue; then (b) check the
|
||||
buffer tail for a **live prefix**: the longest suffix that is either a
|
||||
proper prefix of any pattern's literal opening (`<|tool_call_start|>`,
|
||||
`<|tool_calls|>`, `<|tool_call|>`) or an **open span** (a start token with
|
||||
no end token yet in the buffer — everything from that start token onward is
|
||||
held, since the span may continue in future chunks). Emit everything before
|
||||
the held tail; keep the held tail as the new pending state. (Bounded: the
|
||||
held tail in NORMAL state is ≤ the longest opening token minus one char;
|
||||
in an open span it is unbounded but is being *stripped*, never emitted.)
|
||||
- `flush() -> str` — end of stream: the pending tail is emitted **as-is**
|
||||
(a partial marker at EOF is content, not scaffolding — a documented
|
||||
choice, pinned).
|
||||
- `stripped_chars: int` — total characters removed (property or attribute;
|
||||
read by the caller after the round/turn, task 03).
|
||||
- One filter instance per model request (callers create fresh instances —
|
||||
the phase-67 retry-after-dead-attempt case is safe: a dead attempt never
|
||||
fed the filter).
|
||||
- No logging, no settings, no imports beyond `re` (pure module; the strip
|
||||
*warning* log is emitted by the integration layer, task 03, so the module
|
||||
stays I/O-free).
|
||||
2. `tests/unit/test_scaffolding_filter.py` (new) — the matrix from the phase
|
||||
overview:
|
||||
- span in one chunk → stripped, `stripped_chars` exact.
|
||||
- span split across chunks: feed the incident text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>`
|
||||
at **every** split offset of the start token (0..len) plus a mid-span split
|
||||
and an end-token split → nothing ever emits until the span completes; total
|
||||
output empty; `stripped_chars` = full span length.
|
||||
- two spans in one chunk (with text between) → between-text emitted, spans
|
||||
stripped.
|
||||
- standalone `<|tool_calls|>` / `<|tool_call|>` stripped (alone and embedded in
|
||||
a line).
|
||||
- look-alikes **not** stripped, emitted verbatim: `tool_call_start` as prose;
|
||||
`<|tool_call_start|` (missing delimiter close) as prose; an unknown
|
||||
`<|some_other_token|>`; a lone `<|tool_call_end|>` without a start.
|
||||
- partial start token at EOF (`feed("<|tool_call_st")` then `flush()`) →
|
||||
flush emits it as-is, `stripped_chars == 0`.
|
||||
- surrounding text preserved exactly: `hello <|…span…> world` → `hello world`
|
||||
(no reflow beyond the removal); empty chunk no-op; `flush()` on a clean
|
||||
stream emits nothing new.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_scaffolding_filter.py` (above) — exhaustive boundary
|
||||
offsets, parametrized.
|
||||
- Coverage: **>90%** on `app/rag/scaffolding.py` (every branch: match, open-span
|
||||
hold, live-prefix hold, flush-emits-as-is).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_scaffolding_filter.py -v --no-cov` green.
|
||||
- [ ] `rg "import (os|sys|logging|app)" app/rag/scaffolding.py` → no matches (pure
|
||||
module).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Task 02 — Filter Integration in `chat_stream` / `chat_stream_retried`
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Route `delta.content` through the caller-supplied `ScaffoldingFilter` in
|
||||
`app/rag/llm.py` — content filtered, thinking raw, tail flushed before tool-call
|
||||
materialization — with `None` keeping today's byte-identical raw path.
|
||||
|
||||
## Work
|
||||
1. `app/rag/llm.py` — `chat_stream(..., scaffolding: "ScaffoldingFilter | None =
|
||||
None)"` (type imported under `TYPE_CHECKING` or as a string annotation to keep
|
||||
the module's import graph clean — the filter type is only needed for typing):
|
||||
- Content path: `content = delta.content` → when a filter is present,
|
||||
`cleaned = scaffolding.feed(content)`; yield `StreamPiece("content",
|
||||
cleaned)` only when `cleaned` is non-empty (an empty clean result yields
|
||||
**nothing** — no empty `delta` frames). Without a filter the existing
|
||||
`if content: yield` stands untouched (byte-identical).
|
||||
- Thinking path: unchanged — `reasoning_content` pieces are never filtered
|
||||
(locked: the scratchpad stays raw).
|
||||
- End of stream: after the `async for` exhausts and **before** the tool-call
|
||||
materialization block (`if calls and not emitted: …` and the
|
||||
finish-reason emission), `tail = scaffolding.flush()` when a filter is
|
||||
present; yield a content piece for the tail when non-empty. Order pinned:
|
||||
flushed-tail content precedes `ToolCallPiece`s (content-before-tools wire
|
||||
convention).
|
||||
- Teardown (phase 48) untouched: the `finally: await stream.close()` behavior is
|
||||
independent of the filter.
|
||||
- Docstring: a short "Scaffolding guardrail (phase 71)" paragraph — the filter
|
||||
is caller-owned (one per request), content-only, `None` = raw path.
|
||||
2. `app/rag/llm.py` — `chat_stream_retried(..., scaffolding=None)` — pass-through
|
||||
to `llm.chat_stream`. The same filter object is used across retry attempts of
|
||||
one logical request: safe by construction (a restarted attempt only happens
|
||||
when no piece was emitted, i.e. the filter was never fed — note this in the
|
||||
docstring).
|
||||
3. `tests/unit/test_llm_client.py` — scripted-chunk tests (the existing
|
||||
fake-client pattern):
|
||||
- with a filter: a chunk carrying a span mid-stream → the `delta` pieces carry
|
||||
only the clean text; a span split across two chunks → no partial emit;
|
||||
`filter.stripped_chars` correct after consumption.
|
||||
- thinking pieces pass through raw even when they contain a span (pinned).
|
||||
- `scaffolding=None` → the yielded pieces are byte-identical to the pre-phase
|
||||
raw path (a span in content is yielded verbatim — the raw contract for
|
||||
callers that opt out).
|
||||
- flush ordering: a trailing partial-then-complete tail yields its content
|
||||
piece **before** the materialized `ToolCallPiece` (a chunk stream that ends
|
||||
with `…clean tail` + tool_calls deltas).
|
||||
- teardown interaction: the phase-48 `aclose()`/abandon tests
|
||||
(`tests/unit/test_llm_stream_teardown.py`) stay green with and without a
|
||||
filter (run, don't rewrite).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_llm_client.py` (+ the untouched teardown suite green).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/llm.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_llm_client.py tests/unit/test_llm_stream_teardown.py
|
||||
-v --no-cov` green.
|
||||
- [ ] `scaffolding=None` callers (including the kill-switch no-tools requests) are
|
||||
byte-identical (pinned).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,104 @@
|
||||
# Task 03 — Recovery Policy: One Bounded `tools=None` Retry + Terminal Error
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
When a round/turn's visible content ends up empty **because scaffolding was the
|
||||
whole answer**, run exactly one deterministic recovery (same turn, `tools=None`,
|
||||
fixed correction line in the system prompt, fresh filter); a second empty reply
|
||||
settles with a dedicated error frame. Both the grounded agent loop and the
|
||||
deflected path get the policy; the per-turn log line gains `scaffold_stripped=N`.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py`:
|
||||
- `CORRECTION_INSTRUCTION: str` — the harness-owned constant (verbatim, single
|
||||
line): "Your previous reply contained raw tool-call markup, which is not
|
||||
interpreted here. Answer the user's question directly in plain text — no
|
||||
tool syntax." (The E2E mock in task 05 keys on a stable substring of it —
|
||||
pick the exact constant now; the mock copies it.)
|
||||
- `class MalformedReplyError(LLMError)` — raised only by the recovery policy
|
||||
(never from inside a stream, so `chat_stream_retried`'s retry rule never
|
||||
sees it). Module docstring: the phase-71 note (deterministic-only, owner
|
||||
permission 2026-09-03).
|
||||
- `run_agent` — per round: create a fresh `ScaffoldingFilter`, pass it to
|
||||
`chat_stream_retried(..., scaffolding=round_filter)`, and count the
|
||||
round's visible content (sum of the lengths of the yielded
|
||||
`StreamPiece("content", …)` texts — the filtered ones). After the round,
|
||||
when `not calls` (today's "the answer was streamed" exit):
|
||||
- if round content > 0 → return (as today).
|
||||
- if round content == 0 and `round_filter.stripped_chars > 0` → **one
|
||||
recovery**: `messages_recovered = [*messages[:-1], {"role": "system",
|
||||
"content": system_prompt + "\n" + CORRECTION_INSTRUCTION}, messages[-1]]`
|
||||
(the correction folds into the ORIGINAL single system message —
|
||||
provider-safe; the user message stays last) — one
|
||||
`chat_stream_retried` request with `tools=None`, a fresh filter, the same
|
||||
`retries`/`delay` budget; yield its pieces through the normal piece flow.
|
||||
If the recovery's visible content > 0 → return. Otherwise →
|
||||
`logger.warning` + `raise MalformedReplyError(…)`.
|
||||
- if round content == 0 and nothing was stripped → return (today's
|
||||
empty/thinking-only answer behavior — the UI handles it; unchanged).
|
||||
- Log one warning per strip event here: `logger.warning("agent: stripped
|
||||
N chars of tool-scaffolding in round %d: %r", …)` with the stripped span
|
||||
truncated to 200 chars (the capture mechanism for new registry entries —
|
||||
the filter exposes the stripped spans for this; add a
|
||||
`stripped_spans: list[str]` to the filter if needed).
|
||||
- A scaffolding-only round that also **carried tool calls** needs no
|
||||
recovery (the clean content stands / the tool ran) — the policy keys on
|
||||
the `not calls` exit only (pinned).
|
||||
2. `app/api/chat.py` — the **deflected** path (the grounded path is covered by
|
||||
`run_agent`):
|
||||
- Create one `ScaffoldingFilter` for the turn's request, pass it to
|
||||
`chat_stream_retried(..., scaffolding=filter)`; count visible content across
|
||||
the piece loop (a `content_chars` counter next to `thinking_chars`).
|
||||
- After the piece loop (deflected branch only): content == 0 and
|
||||
`filter.stripped_chars > 0` → one recovery request: the same
|
||||
`messages` with the system prompt extended by
|
||||
`CORRECTION_INSTRUCTION` (import from `app.rag.agent`), `tools=None`, a
|
||||
**fresh** filter, the same retry budget; stream its pieces through the
|
||||
SAME piece-handling code (extract the piece loop into a small inner
|
||||
helper/coroutine to avoid duplicating the thinking/tool/retry/delta
|
||||
handling — the extraction must be behavior-preserving for the first pass,
|
||||
pinned by the existing integration suite). If the recovery content > 0 →
|
||||
continue to the normal `done` flow; else → the terminal path below.
|
||||
- Catch `MalformedReplyError` **before** the generic `LLMError` handler:
|
||||
`settled = True`, yield `ChatErrorEvent(detail="The model returned a
|
||||
malformed reply — please try again.")`, return (no `query_log` row, no
|
||||
`done` — the existing terminal-error semantics; the generic
|
||||
"dropped the connection" copy stays for transport failures).
|
||||
- Per-turn log line: append `scaffold_stripped=N` after `retries=N` — the sum
|
||||
across the turn's requests (rounds + any recovery; 0 on clean turns —
|
||||
uniform field, the phase-67 `retries=N` pattern). The recovery does not
|
||||
bump `retries=N` (it is not a phase-67 endpoint-retry).
|
||||
- Module docstring: the phase-71 paragraph (deterministic guardrail +
|
||||
recovery + log field).
|
||||
3. `tests/unit/test_agent.py` — the grounded matrix (scripted fake LLM,
|
||||
monkeypatched DB): scaffolding-only round → exactly **two** model requests,
|
||||
the second `tools=None` with `CORRECTION_INSTRUCTION` in its system prompt →
|
||||
clean answer ends the turn, `done`-side state normal (holder untouched by the
|
||||
recovery); scaffolding twice → `MalformedReplyError` (assert it subclasses
|
||||
`LLMError`); scaffolding + real content → one request only, clean content
|
||||
yielded, no recovery; clean turn → one request, no correction in any system
|
||||
prompt; the round cap + kill-switch tests stay green.
|
||||
4. `tests/integration/test_chat_api.py` — the deflected matrix over
|
||||
`POST /api/chat` (mock LLM): scaffolding-only deflected turn → clean recovery
|
||||
answer + `done` frame; scaffolding twice → the error frame with the dedicated
|
||||
copy, no `done`, no `query_log` row (assert against the table); mixed
|
||||
scaffolding+content → clean answer, `scaffold_stripped>0` in the log line;
|
||||
the log-line pin gains `scaffold_stripped=0` on clean turns and the summed
|
||||
value on stripped turns.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` (grounded matrix).
|
||||
- Integration: `tests/integration/test_chat_api.py` (deflected matrix + log line).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/agent.py`,
|
||||
`app/api/chat.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_chat_api.py -v
|
||||
--no-cov` green.
|
||||
- [ ] Exactly one recovery per turn, on both paths; the recovery request is
|
||||
`tools=None` + correction line; a second empty reply → the dedicated error
|
||||
frame, no `query_log` row.
|
||||
- [ ] Clean turns: no correction in any system prompt, `scaffold_stripped=0`,
|
||||
request counts unchanged (kill-switch/deflection byte-identical pins green).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 04 — Deflect Prompt: Plain-Text-Only Line (Prevention)
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Close the door at the prompt: the deflected (LOW) turn offers no tools, so any tool
|
||||
markup there is always wrong — add one plain-text-only instruction line to the
|
||||
`DEFLECT_MODE` body. This is the owner-permitted change to the otherwise-locked LOW
|
||||
prompt; the `DEFLECT_MODE` marker and everything else in the prompt stay put (the
|
||||
E2E mock keys on the marker's presence, not the wording).
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `build_deflect_prompt`: append one line to the
|
||||
`DEFLECT_MODE` body (after "…propose 2-3 alternative questions."):
|
||||
"Reply in plain text only — you have no tools in this mode." The weak-hit
|
||||
title list follows exactly as today; `build_high_prompt`, `PERSONA`,
|
||||
`TOOLS_SECTION`, and the section order are untouched. Update the module
|
||||
docstring: the phase-71 note (owner-permitted 2026-09-03 LOW-prompt line;
|
||||
the marker-keying contract unchanged).
|
||||
2. `tests/unit/test_prompts.py` (and `test_chat_gate.py` where the deflection body
|
||||
is pinned) — pins: the new line present in the LOW prompt; the `DEFLECT_MODE`
|
||||
marker still present (mock keying); the HIGH prompt unchanged (byte-identical
|
||||
pins stay green — the line must not leak into HIGH); the byte-identical-when-
|
||||
empty steering/overview behavior unchanged.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the prompt pins above.
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/prompts.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v
|
||||
--no-cov` green.
|
||||
- [ ] LOW prompt = pre-phase text + exactly the one new line (a diff pin, or a
|
||||
reconstruction pin in the test).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Task 05 — Mock Triggers, Dedicated E2E Suite, Gates, Commit
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Prove the guardrail end-to-end through the real UI: a mock-LLM scaffolding flow
|
||||
(recovery case + terminal case), a dedicated Playwright suite pinning that raw
|
||||
tokens never reach the DOM, then the full quality gates and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — two new deterministic flows (checked in the flow
|
||||
table **before** the plain `TOOLS_TRIGGER` flow, after `SEARCH_TRIGGER`
|
||||
ordering rules as they fit — the triggers are independent of the `<tools>`
|
||||
marker, so both grounded and deflected turns hit them):
|
||||
- `SCAFFOLD_TRIGGER = "emit raw tool markup"` — request 1 (no correction in
|
||||
the system prompt): stream ONLY `delta.content` chunks carrying the incident
|
||||
text `<|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|>`
|
||||
(split across ≥2 chunks to exercise the boundary path), `finish_reason:
|
||||
"stop"`, no structured `tool_calls`, no reasoning. Request 2 (system prompt
|
||||
contains the stable substring of `CORRECTION_INSTRUCTION` — import it from
|
||||
`app.rag.agent` so the mock can never drift from the constant): stream a
|
||||
clean plain answer ("Here is the plain-text answer the recovery produced.")
|
||||
+ `finish_reason: "stop"`.
|
||||
- `SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup"` — every request
|
||||
(recovery included): the same scaffolding-only stream, forever.
|
||||
- Update the module docstring's flow table + the phase-71 note.
|
||||
2. `tests/e2e/test_tool_scaffolding_guardrails.py` (NEW — the phase's dedicated
|
||||
suite, house pattern, run in isolation; DB up, mock LLM):
|
||||
- **Recovery case** — ask a question containing `SCAFFOLD_TRIGGER`: the turn
|
||||
settles (the composer re-enables, `done` observed); the final answer bubble
|
||||
contains the recovery's clean text; `document.body.innerText` contains
|
||||
**neither** `tool_call_start` nor `tool_call_end` (nor the raw
|
||||
`[read(path=…]` fragment); no error banner.
|
||||
- **Terminal case** — ask a question containing
|
||||
`SCAFFOLD_ALWAYS_TRIGGER`: the existing error status renders with the
|
||||
dedicated copy ("The model returned a malformed reply — please try
|
||||
again."); no raw tokens in the DOM; no answer bubble with the scaffolding;
|
||||
the app stays usable — a follow-up plain question (no trigger) gets a
|
||||
normal streamed answer in the same session.
|
||||
- **No false positive** — a plain question (existing `CHAT_TRIGGER`-style
|
||||
flow, no tools needed): the answer streams byte-clean, no error state, no
|
||||
recovery request visible (the turn settles on the first request).
|
||||
- The SSE wire itself: in the recovery case, no `delta` frame ever carries a
|
||||
`tool_call_start`/`tool_call_end` fragment (the existing SSE-capture house
|
||||
pattern) — the strip happens server-side, not in the UI.
|
||||
3. Gates + commit:
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_scaffolding_guardrails.py`, then the
|
||||
regression suites `test_harness_aligned_tools.py`, `test_chat_rag.py`,
|
||||
`test_agent_document_tools.py`.
|
||||
- One atomic commit (message below); move
|
||||
`.agent/phases/todo/71_scaffolding_guardrails/` →
|
||||
`.agent/phases/complete/71_scaffolding_guardrails/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new dedicated suite (recovery / terminal / no-false-positive / wire
|
||||
pins) + the three regression suites (isolation runs).
|
||||
- Coverage: **>90%** on `app/` (phase-level gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start|tool_call_end" frontend/` → no matches.
|
||||
- [ ] Raw scaffolding is never visible in the DOM in any case (recovery,
|
||||
terminal, mixed) — pinned by the dedicated suite.
|
||||
- [ ] Exactly one recovery per malformed turn (mock request counts implied by the
|
||||
flows); clean turns never carry the correction line (pinned by the unit
|
||||
suites from task 03).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
All task-01 gates green. Final report:
|
||||
|
||||
**Task 01 complete — `ls` / `read(path)` / `grep(pattern, path?)` tool surface in `app/rag/agent.py`**
|
||||
|
||||
- `AGENT_TOOLS` → exactly `ls` (optional `path`), `read` (required `path`), `grep` (required `pattern`, optional `path`) with the task's pinned descriptions; old names gone from `agent.py` (verified `rg` → no matches)
|
||||
- `_execute_tool`: `read`/`grep` take the canonical combined `source/path` (first-slash split via new `_resolve_path`; bare source name → no-document refusal, no DB); `ls` scoping by registered source name (`No source named '…'` refusal; `0 documents:` counted); phase-68 A5 grep contract + refusals/round-cap/kill-switch unchanged; split-teaching branches deleted
|
||||
- New module-level accessor `list_source_names` (registry via `effective_sources` + phase-69 `resolve_source_name`) — the catalog can't distinguish "known source, 0 docs" from "unknown source", the registry can
|
||||
- Tests: `tests/unit/test_agent.py` rewritten (57 tests, new schema/ls-scope/read/grep pins); `tests/integration/test_agent_tools.py` updated to the new surface incl. real-DB scoped-`ls` registry tests
|
||||
|
||||
**Gates:** `pytest tests/unit/test_agent.py --no-cov` → 57 passed · `pytest --cov=app -k agent` → 76 passed, `app/rag/agent.py` **100%** · `ruff check .` clean · `pyright` 0 errors · full suite 1379 passed, **2 failed** — both in `tests/integration/test_chat_api.py` (they script the *old* tool names; fixing them requires `app/api/chat.py`'s SSE `argument` re-derivation, which is task 03's explicit work item — left for it, per no-next-task rule)
|
||||
|
||||
**Decisions:** blank/non-string `ls` `path` = omitted (full catalog, not refusal); already-in-context dedupe runs pre-DB (seed docs never hit the DB); no commit (task 05 owns it).
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/70_harness_aligned_tools/02_prompt_section.md`
|
||||
+294
@@ -0,0 +1,294 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 78%]
|
||||
.......................F...F............................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 99%]
|
||||
............. [100%]
|
||||
=================================== FAILURES ===================================
|
||||
__________ test_grounded_turn_streams_tool_frames_and_cites_read_doc ___________
|
||||
|
||||
client = <starlette.testclient.TestClient object at 0x7fc9c0134130>
|
||||
db = <sqlalchemy.orm.session.Session object at 0x7fc9b02627b0>
|
||||
seeded_kb = <test_chat_api.FakeRagLLM object at 0x7fc9b0535190>
|
||||
caplog = <_pytest.logging.LogCaptureFixture object at 0x7fc9b01fcad0>
|
||||
|
||||
def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""(a) Grounded turn with tool calls: the event sequence is
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
Phase 45: the agent loop keeps offering the tools for the whole
|
||||
turn — the round cap (not per-tool budgets) is the bound."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
StreamPiece("thinking", "Let me list what is indexed…"),
|
||||
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
# the answer request still carries the tools (2 rounds < the
|
||||
# default cap of 10); the fake's tool_script is exhausted, so
|
||||
# it falls back to the thinking + answer stream
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
types = [f["type"] for f in frames]
|
||||
assert types[0] == "thinking"
|
||||
assert types[1] == "tool" and types[2] == "tool" # the two executed calls
|
||||
assert "error" not in types
|
||||
assert types[3:-1] == ["delta"] * (len(types) - 4) # deltas, then done last
|
||||
assert frames[-1]["type"] == "done"
|
||||
|
||||
list_frame, read_frame = frames[1], frames[2]
|
||||
assert set(list_frame) == {"type", "name", "argument"}
|
||||
assert list_frame["name"] == "list_documents"
|
||||
assert list_frame["argument"] is None # the tool takes no parameters
|
||||
assert set(read_frame) == {"type", "name", "argument"}
|
||||
assert read_frame["name"] == "read_document"
|
||||
assert read_frame["argument"] == "docs/homelab/backups.md"
|
||||
|
||||
deltas = [f for f in frames if f["type"] == "delta"]
|
||||
assert len(deltas) >= 2 # genuinely streamed
|
||||
assert "".join(d["text"] for d in deltas) == scripted.answer
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# done.sources = the retrieval docs + the read doc, deduped, order kept.
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
> assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
E AssertionError: assert ('docs', 'hom...es/deploy.j2') == ('docs', 'homelab/backups.md')
|
||||
E
|
||||
E At index 1 diff: 'homelab/templates/deploy.j2' != 'homelab/backups.md'
|
||||
E Use -v to get more diff
|
||||
|
||||
tests/integration/test_chat_api.py:597: AssertionError
|
||||
---------------------------- Captured stdout setup -----------------------------
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=deployments/new-service.md chunks=4
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/backups.md chunks=5
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/tables.md chunks=2
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
------------------------------ Captured log setup ------------------------------
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=deployments/new-service.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/backups.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/tables.md chunks=2
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
INFO app.importer:importer.py:103 import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
2026-09-03 09:53:07 INFO app.agent :: agent tool=list_documents args={} round=1/10
|
||||
2026-09-03 09:53:07 INFO app.agent :: agent tool=read_document args={"source": "docs", "path": "homelab/backups.md"} round=2/10
|
||||
2026-09-03 09:53:07 INFO app.chat :: question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=28 tool_calls=0 total_ms=11 retries=0
|
||||
------------------------------ Captured log call -------------------------------
|
||||
INFO app.agent:agent.py:527 agent tool=list_documents args={} round=1/10
|
||||
INFO app.agent:agent.py:527 agent tool=read_document args={"source": "docs", "path": "homelab/backups.md"} round=2/10
|
||||
INFO app.chat:chat.py:506 question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=28 tool_calls=0 total_ms=11 retries=0
|
||||
______________ test_tool_execution_db_failure_yields_error_event _______________
|
||||
|
||||
client = <starlette.testclient.TestClient object at 0x7fc9b0eea0b0>
|
||||
db = <sqlalchemy.orm.session.Session object at 0x7fc9b02d27b0>
|
||||
seeded_kb = <test_chat_api.FakeRagLLM object at 0x7fc9b0535610>
|
||||
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc9c35c3d20>
|
||||
|
||||
def test_tool_execution_db_failure_yields_error_event(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A tool call that hits a dead DB mid-stream gets the same structured
|
||||
``error`` event as the pre-stream retrieval path — never a severed
|
||||
stream (the "never stale" contract, PLAN §7.4)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[[ToolCallPiece(id="call_1", name="list_documents", arguments={})]]
|
||||
)
|
||||
|
||||
def boom(*_a: Any, **_k: Any) -> Any:
|
||||
raise RuntimeError("db exploded mid tool call")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", boom)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
# The ``tool`` frame went out first (the model requested the call);
|
||||
# the failed execution ends the turn with the structured error event.
|
||||
> assert [f["type"] for f in frames] == ["tool", "error"]
|
||||
E AssertionError: assert ['tool', 'del... 'delta', ...] == ['tool', 'error']
|
||||
E
|
||||
E At index 1 diff: 'delta' != 'error'
|
||||
E Left contains 5 more items, first extra item: 'delta'
|
||||
E Use -v to get more diff
|
||||
|
||||
tests/integration/test_chat_api.py:817: AssertionError
|
||||
---------------------------- Captured stdout setup -----------------------------
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=deployments/new-service.md chunks=4
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/backups.md chunks=5
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/tables.md chunks=2
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
------------------------------ Captured log setup ------------------------------
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=deployments/new-service.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/backups.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/tables.md chunks=2
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
INFO app.importer:importer.py:103 import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
2026-09-03 09:53:08 INFO app.agent :: agent tool=list_documents args={} round=1/10
|
||||
2026-09-03 09:53:08 INFO app.chat :: question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=0 tool_calls=0 total_ms=10 retries=0
|
||||
------------------------------ Captured log call -------------------------------
|
||||
INFO app.agent:agent.py:527 agent tool=list_documents args={} round=1/10
|
||||
INFO app.chat:chat.py:506 question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=0 tool_calls=0 total_ms=10 retries=0
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 153 4 97%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2726 16 99%
|
||||
=========================== short test summary info ============================
|
||||
FAILED tests/integration/test_chat_api.py::test_grounded_turn_streams_tool_frames_and_cites_read_doc
|
||||
FAILED tests/integration/test_chat_api.py::test_tool_execution_db_failure_yields_error_event
|
||||
tests FAILED: unit + integration suite did not pass
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation FAILED (see output above)
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
All gates green — full suite included. Final report:
|
||||
|
||||
**Fixed:** the harness gate runs the full unit+integration suite, so I resolved the two `test_chat_api.py` failures that the phase plans queue for task 03 — the minimal coupled fix, no further task-03 work:
|
||||
- `app/api/chat.py`: SSE `tool`-frame `argument` re-derived for the new surface — `read`'s `path` (combined `source/path`), `grep`'s `pattern`, `ls`'s `path`, else/null for non-strings or unknown tools; module docstring + inline comments updated (old names now gone from `chat.py` too)
|
||||
- `tests/integration/test_chat_api.py`: 5 tests moved to the new surface — `ls`+`read(path=…)` frame test (`argument` == the single string the model passed), search-frame test renamed to `grep` (raw pattern / non-string → null), DB-failure test scripted as `ls` (asserts `name == "ls"`, `["tool","error"]`), deflected + kill-switch scripts renamed
|
||||
|
||||
**Results:** `uv run pytest` → **1381 passed, 0 failed** · `--cov=app` TOTAL **99%** (`app/rag/agent.py` 100%, `app/api/chat.py` 100%) · `uv run pytest tests/unit/test_agent.py --no-cov` → 57 passed · `uv run pytest --cov=app -k agent` → 76 passed · `uv run ruff check .` clean · `uv run pyright` 0 errors · `rg` old names in `app/rag/agent.py` → no matches
|
||||
|
||||
**Decisions:** remaining old-name test references (chats_api, sse_events, mock_tool_flow, frontend_tool_states, llm_*) all pass — they pin opaque data or files owned by tasks 02/04/05, left untouched; deflected path + kill switch byte-identical (pinned).
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/70_harness_aligned_tools/02_prompt_section.md`
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 99%]
|
||||
............. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
Task 02 complete — all gates green.
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/prompts.py`: rewrote `TOOLS_SECTION` copy for the harness-aligned `ls` / `read(path)` / `grep(pattern, path?)` shapes (catalog-line format, case-insensitive locator contract, combined `source/path` + full content, answer-when-ready close — no budget re-statement, phase 45); updated the module docstring and section comment (HIGH-only, LOW byte-identical, mock keys on `<tools>` marker presence); `build_high_prompt`/`build_deflect_prompt` and section order untouched
|
||||
- `tests/unit/test_prompts.py`: new phase-70 pin block (markers, shape copy pins, old names + budget copy absent, HIGH still ends with `<tools>`), explicit no-`<tools>`/no-new-copy LOW assertions
|
||||
- `tests/unit/test_chat_gate.py`: endpoint pins updated (new names in HIGH system prompt, old names out; new copy absent from the deflected path)
|
||||
- `tests/unit/test_agent.py`: re-pinned the old `TOOLS_SECTION` copy pins (the in-file NOTE delegated this re-pin to task 02)
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v --no-cov` → 58 passed
|
||||
- `rg "list_documents|read_document|search_documents" app/rag/prompts.py` → no matches
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 1385 passed; `app/rag/prompts.py` 100% (>90% gate), TOTAL 99%
|
||||
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors
|
||||
|
||||
**Decisions**
|
||||
- E2E/mock still on old names — expected mid-phase; task 05 owns the mock + E2E updates (default `testpaths` excludes E2E; mock keys on the `<tools>` marker only, so keying is intact)
|
||||
- No PLAN.md exists in `.agent/` (referenced by AGENTS.md); the phase overview was treated as binding context
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/70_harness_aligned_tools/03_api_sse.md`
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,16 @@
|
||||
All gates green. Task 03 is complete.
|
||||
|
||||
**Task 03 — SSE `tool` frames + API contract copy: done**
|
||||
|
||||
- `app/api/chat.py`: `ToolCallPiece` → SSE derivation now symmetric — `argument` = the single string the model passed (`read`/`ls` → `path`, `grep` → `pattern`), non-string/omitted → null; module docstring carries the phase-70 note, the `{"type":"tool",…}` shape, and the `done.sources`/`query_log`/`tool_calls=N` contract (verified in place; branch was already rewritten by the earlier interrupted run of this task).
|
||||
- `app/schemas.py`: `ChatToolEvent` docstring re-pointed at `ls`/`read`/`grep` + the argument rule (no field change); `ToolCall` docstring updated incl. the locked no-migration note for legacy saved names.
|
||||
- `tests/integration/test_chat_api.py`: new pin `test_tool_frames_carry_the_model_arguments_regardless_of_execution` — scoped `ls` frame carries `"docs"`; a rejected `read` (unknown path) still emits its frame with the model's argument as passed; `done.sources` stays retrieval-only; log line `tool_calls=1` (rejected calls count nothing). Existing pins: `read` frame = combined path as passed, `grep` frame = pattern, unscoped `ls` = null.
|
||||
- `tests/integration/test_chats_api.py`: saved-chat fixture on new names/args; keeps one `read_document` entry pinning that pre-phase-70 chats still validate (locked, no migration).
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/integration/test_agent_tools.py tests/integration/test_chat_api.py -v --no-cov` → 40 passed
|
||||
- `uv run pytest tests/integration/test_chats_api.py --no-cov` → 54 passed; full `uv run pytest` → green (exit 0)
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → `app/api/chat.py` 100%, `app/schemas.py` 100%, TOTAL 99% (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||
|
||||
**Notes**: `tests/integration/test_api.py` old-name mentions are app.js asset pins — left for task 04 (frontend). Legacy docstring names in `app/rag/llm.py` and the `docs.py` route function name fall to the phase-level `rg` gate in task 05. Next pending task: `04_frontend.md`.
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
.................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,11 @@
|
||||
All green. The other modified files are uncommitted work from completed tasks 01–03 (the commit belongs to task 05 per the phase plan).
|
||||
|
||||
**Task 04 report**
|
||||
|
||||
- `frontend/assets/app.js`: `appendToolLine` now keys on `read`/`grep`/`ls` (new copy: "📄 Reading", "🔎 Searching for", "🔎 Listing documents in <scope>", unscoped "🔎 Listing documents") with legacy `read_document`/`search_documents`/`list_documents` branches preserved byte-identical for persisted turns; status-label ternary in the turn handler extended to the new names (incl. scoped `ls`); section + module docstrings updated with the phase-70 note.
|
||||
- `frontend/assets/shared.js`: `addToolLines` local copy got the same new-name branches (read/grep/ls + scoped ls) with the legacy branches kept; all arguments `textContent`-only; docstring updated.
|
||||
- `tests/unit/test_frontend_tool_states.py`: pins updated/added — new+legacy branch gates in app.js, new shared.js pin test (`test_shared_page_tool_lines_cover_new_and_legacy_names`), no-`innerHTML` textContent discipline; old copy pins kept.
|
||||
- Ripple fix: `tests/integration/test_api.py` emoji guard now strips the fourth tool-line literal `"🔎 Listing documents in "`.
|
||||
- Tests: `uv run pytest tests/unit/test_frontend_tool_states.py -v --no-cov` → 10 passed; `uv run pytest --cov=app --cov-report=term-missing` → 1387 passed, TOTAL 99% (>90%); `uv run ruff check . && uv run pyright` → clean; both JS files pass `node --check`.
|
||||
- Decisions: explicit if-chains (not a name map) for the two-generation gates; scoped-`ls` status label added so the live region mirrors the new line copy (PLAN §7.4). E2E suites remain keyed on the old mock names — task 05's scope.
|
||||
- Next pending task: `.agent/phases/todo/70_harness_aligned_tools/05_mock_e2e_commit.md`.
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
Reference in New Issue
Block a user