fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled methodology (owner directive: stop clearing/re-importing the homelab KB per iteration; measure tool-calling accuracy on a controlled fixture KB, target >90%). Real-model gate verdicts (live, configured chat model 'lite', fixture KB): - Controlled fixture battery (the new methodology's pass condition — contract accuracy >= 90%): PASS, 4 consecutive runs: gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s) - Locked derived battery (phase-72 task 05, executed >= 90% bar, run unchanged on the same fixture KB): gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s) The teaching works — every bare-path trap self-corrects in exactly one round, zero cap hits, zero repeat loops, 10/10 answered. The locked executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the corrected re-reads (the trap question seeds its target, so the correct combined-form read is refused for redundancy) — a copy-invariant model behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04) and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md sections 5 and 7), not a copy lever. Copy changes this phase owns (unit pins updated to follow): - app/rag/agent.py: ls teaching refusals (path-like scope -> document-path line; unknown source -> no-source line with the source-name parenthetical), read/grep 'did you mean source/path?' teaching (find_path_candidates: exact or suffix path match, catalog order, cap 3), ALREADY_IN_CONTEXT naming the correct action (answer from the text already in the prompt), read tool description front-loaded with the do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is the only remaining refusal class; contract accuracy 92-100% across runs) - app/rag/prompts.py: TOOLS_SECTION states the document-identity contract up front (ls path = source name; read/grep = combined source/path including the source name; do-not-read for <documents> documents placed next to the read teaching; one-call-per-reply and never-repeat rules) - tests: refusal pins (unit + integration), new dedicated E2E suite tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in isolation), regression suites green in isolation (harness_aligned_tools, agent_document_tools, agent_unlimited_tools, search_tool, chat_rag). Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff + pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/ move and both phases' .agent/reports/ (AGENTS.md 8).
This commit is contained in:
@@ -0,0 +1,152 @@
|
||||
# Phase 71 — Tool-Scaffolding Guardrails: Deterministic Strip + One Bounded Recovery
|
||||
|
||||
**Source:** owner request (chat, 2026-09-03) — same incident as phase 70: the
|
||||
deflected answer streamed the raw model text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>` into
|
||||
the UI (the model's own `<|…|>` chat-template tool syntax, emitted as plain
|
||||
`delta.content` even though no tools were offered). Owner direction: "We also need
|
||||
guardrails for situations like this… **deterministic guardrails only** right now,
|
||||
forget using a model for that" — no model of any kind (no `lite` classifier, no
|
||||
model-authored repair) in the guardrail path; `lite` stays the chat model for
|
||||
everything (it's faster).
|
||||
**Story:** n/a (owner request from chat — tool-scaffolding guardrails, 2026-09-03)
|
||||
**Context:**
|
||||
- `app/rag/llm.py` — `chat_stream` yields `StreamPiece("content", delta.content)`
|
||||
verbatim (L~`content = delta.content; if content: yield`); thinking pieces
|
||||
(`delta.reasoning_content`) pass through raw by design; tool-call materialization
|
||||
happens at stream end (after the `async for`); phase-48 teardown closes the
|
||||
endpoint stream on every exit (must stay intact). `chat_stream_retried` (phase 67)
|
||||
wraps it with the retry-before-first-piece rule.
|
||||
- `app/rag/agent.py` — `run_agent`'s round loop: a round with no `ToolCallPiece`s
|
||||
ends the turn (`if not calls: return`); the round cap forces one final
|
||||
`tools=None` request (the "forced final answer" pattern this phase reuses for
|
||||
recovery); `AgentHolder` (read_docs / tool_calls) is unchanged.
|
||||
- `app/api/chat.py` — the piece loop (thinking/tool/retry/delta handling +
|
||||
`thinking_chars` counter), the deflected path (`chat_stream_retried(...,
|
||||
tools=None)` directly), the `LLMError` → terminal error-frame handler, the
|
||||
per-turn log line (ends `…total_ms=N retries=N`).
|
||||
- `app/rag/prompts.py` — `build_deflect_prompt`'s `DEFLECT_MODE` body (the E2E mock
|
||||
keys on the marker's *presence*, not the wording — an appended line is safe).
|
||||
- `tests/e2e/mock_llm.py` — the trigger-string flow table (task 05 adds the
|
||||
scaffolding triggers; phase-70 names apply).
|
||||
- Tests to extend: `tests/unit/test_llm_client.py`, `test_agent.py`,
|
||||
`tests/integration/test_chat_api.py` (log-line + error-frame pins),
|
||||
`tests/unit/test_prompts.py`.
|
||||
|
||||
## Objective
|
||||
Raw tool-scaffolding tokens can never reach the user as answer text: a deterministic
|
||||
streaming filter strips known scaffolding from `delta.content` as it flows, and a
|
||||
round/turn whose visible content ends up empty (scaffolding was the whole "answer")
|
||||
gets **one** bounded, deterministic recovery (same turn, `tools=None`, a fixed
|
||||
harness-owned correction line in the system prompt); if the recovery also comes back
|
||||
empty, the turn settles with a dedicated structured error frame. No model is used to
|
||||
detect or repair anything.
|
||||
|
||||
## Dependencies
|
||||
- `70_harness_aligned_tools` (todo) — runs first: the mock/prompt/code state this
|
||||
phase builds on (new tool names in the mock flows and prompt).
|
||||
- `67_llm_retry` (complete) — `chat_stream_retried`, the primitive every request
|
||||
(including recoveries) goes through.
|
||||
- `48_stop_generation` (complete) — the phase-48 stream-teardown contract the
|
||||
filter integration must not break.
|
||||
|
||||
## Tasks
|
||||
1. `01_filter.md` — `app/rag/scaffolding.py` (new): the pattern registry +
|
||||
`ScaffoldingFilter` streaming state machine, with the boundary test matrix.
|
||||
2. `02_llm_integration.md` — `app/rag/llm.py`: `chat_stream`/`chat_stream_retried`
|
||||
accept the filter; content deltas are filtered, the tail flushed before tool
|
||||
materialization; `None` = byte-identical raw path.
|
||||
3. `03_recovery_policy.md` — `app/rag/agent.py` + `app/api/chat.py`: the empty-round
|
||||
recovery (one per turn, `tools=None`, correction constant),
|
||||
`MalformedReplyError`, the dedicated error copy, and the per-turn log line's
|
||||
`scaffold_stripped=N` field.
|
||||
4. `04_deflect_prompt.md` — the plain-text line in the `DEFLECT_MODE` body
|
||||
(prevention; owner-permitted LOW-prompt change).
|
||||
5. `05_e2e_commit.md` — the mock scaffolding triggers, the dedicated E2E suite,
|
||||
full gates, commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (new `tests/unit/test_scaffolding_filter.py`): the filter matrix — span in
|
||||
one chunk; span split across chunks at **every** boundary offset of the start
|
||||
token; multiple spans in one chunk; standalone `<|tool_calls|>` / `<|tool_call|>`
|
||||
tokens stripped; look-alikes **not** stripped (prose containing the words
|
||||
"tool_call" or `tool_call_start` without the `<|…|>` delimiters, an unknown
|
||||
`<|some_other_token|>`, a lone `<|tool_call_end|>` without a start); a partial
|
||||
start token at stream end → `flush()` emits it as-is (no false-positive strip);
|
||||
`stripped_chars` accounting; empty chunks.
|
||||
- Unit: `tests/unit/test_llm_client.py` — `chat_stream` with a filter (content
|
||||
filtered, thinking raw, `None` = raw pass-through pinned byte-identical, flush
|
||||
order: flushed tail content precedes tool-call pieces, phase-48 teardown intact).
|
||||
- Unit: `tests/unit/test_agent.py` — grounded recovery matrix (scaffolding-only
|
||||
round → exactly one recovery request: `tools=None` + correction line in the
|
||||
system prompt + fresh filter → clean answer ends the turn; scaffolding twice →
|
||||
`MalformedReplyError`; scaffolding + real content → clean answer, **no**
|
||||
recovery; round cap and kill switch unchanged).
|
||||
- Integration: `tests/integration/test_chat_api.py` — deflected-path recovery matrix
|
||||
(same shapes over the SSE endpoint: clean recovery → `done` frame; terminal → the
|
||||
dedicated error frame, no `done`, no `query_log` row — same terminal semantics as
|
||||
today's `LLMError`); log line carries `scaffold_stripped=N` (0 when nothing was
|
||||
stripped — the field is uniform, the phase-67 `retries=N` pattern).
|
||||
- E2E (mandatory, house rule): NEW dedicated suite
|
||||
`tests/e2e/test_tool_scaffolding_guardrails.py`, run in isolation — recovery case
|
||||
(raw tokens never in the DOM, clean answer shown) and terminal case (error state,
|
||||
no raw tokens, the app stays usable).
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start" frontend/` → no matches (no scaffolding rendering
|
||||
path); the filter lives in `app/rag/scaffolding.py` as a pure module (no I/O,
|
||||
no model calls).
|
||||
- [ ] A content stream of pure scaffolding yields zero `delta` frames; a mixed
|
||||
stream yields the clean remainder; thinking frames are never filtered.
|
||||
- [ ] Exactly one recovery per turn (grounded and deflected paths); the recovery
|
||||
request is `tools=None` with the fixed correction line in the system prompt;
|
||||
a second empty reply settles with the error frame
|
||||
"The model returned a malformed reply — please try again."
|
||||
- [ ] The per-turn log line ends `…retries=N scaffold_stripped=N`; `scaffold_stripped=0`
|
||||
on clean turns (uniform field).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation:
|
||||
`test_harness_aligned_tools.py`, `test_chat_rag.py`, `test_agent_document_tools.py`.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
|
||||
`.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner (chat, 2026-09-03): deterministic only.** No model — `lite` or any other —
|
||||
participates in detection or repair. The guardrail is a fixed pattern registry + a
|
||||
fixed retry policy. (A model-based classifier/repair was proposed and explicitly
|
||||
rejected for now — if it is ever wanted, it is a later phase with its own
|
||||
permission.)
|
||||
- **`lite` stays the chat model** (owner: "I want to use lite for everything since
|
||||
it's way faster") — the guardrail is what protects the UX while `lite` is the
|
||||
model; no `.env` change in this phase.
|
||||
- **The pattern registry is the extension point.** Initial entries: the observed
|
||||
span form `<|tool_call_start|>…<|tool_call_end|>` (non-greedy, any text between)
|
||||
plus the standalone sibling tokens `<|tool_calls|>` and `<|tool_call|>` from the
|
||||
same tokenizer family. Every strip logs a warning with the stripped span
|
||||
(truncated to 200 chars) — that log line is how a new format gets captured and
|
||||
added (pattern + unit fixture), keeping the registry honest (every entry traces
|
||||
to an observed capture or the initial incident).
|
||||
- **Content only, thinking never filtered.** The Thinking block is the model's raw
|
||||
reasoning by design (phase 17) and stays raw (collapsible); the guardrail protects
|
||||
the answer, not the scratchpad.
|
||||
- **Recovery is a fixed policy, not a conversation.** One extra request per turn,
|
||||
same messages with the harness-owned constant folded into the system prompt
|
||||
(single system message — provider-safe), `tools=None`, a fresh filter, the same
|
||||
phase-67 retry budget. At most one recovery; the second empty reply is terminal.
|
||||
A round with real visible content plus scaffolding needs no recovery (the clean
|
||||
content stands).
|
||||
- **Terminal semantics follow the existing error pattern.** A terminal malformed
|
||||
turn settles with a structured `error` frame (dedicated copy), writes no
|
||||
`query_log` row, and the UI shows the existing error state — byte-for-byte the
|
||||
same shape as today's `LLMError` terminal path.
|
||||
- **`MalformedReplyError` subclasses `LLMError`** and is raised only by the recovery
|
||||
policy (never from inside a stream, so `chat_stream_retried`'s retry rule never
|
||||
sees it); `chat.py` catches it before the generic `LLMError` handler.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# Task 05 — Mock Triggers, Dedicated E2E Suite, Gates, Commit
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Prove the guardrail end-to-end through the real UI: a mock-LLM scaffolding flow
|
||||
(recovery case + terminal case), a dedicated Playwright suite pinning that raw
|
||||
tokens never reach the DOM, then the full quality gates and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — two new deterministic flows (checked in the flow
|
||||
table **before** the plain `TOOLS_TRIGGER` flow, after `SEARCH_TRIGGER`
|
||||
ordering rules as they fit — the triggers are independent of the `<tools>`
|
||||
marker, so both grounded and deflected turns hit them):
|
||||
- `SCAFFOLD_TRIGGER = "emit raw tool markup"` — request 1 (no correction in
|
||||
the system prompt): stream ONLY `delta.content` chunks carrying the incident
|
||||
text `<|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|>`
|
||||
(split across ≥2 chunks to exercise the boundary path), `finish_reason:
|
||||
"stop"`, no structured `tool_calls`, no reasoning. Request 2 (system prompt
|
||||
contains the stable substring of `CORRECTION_INSTRUCTION` — import it from
|
||||
`app.rag.agent` so the mock can never drift from the constant): stream a
|
||||
clean plain answer ("Here is the plain-text answer the recovery produced.")
|
||||
+ `finish_reason: "stop"`.
|
||||
- `SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup"` — every request
|
||||
(recovery included): the same scaffolding-only stream, forever.
|
||||
- Update the module docstring's flow table + the phase-71 note.
|
||||
2. `tests/e2e/test_tool_scaffolding_guardrails.py` (NEW — the phase's dedicated
|
||||
suite, house pattern, run in isolation; DB up, mock LLM):
|
||||
- **Recovery case** — ask a question containing `SCAFFOLD_TRIGGER`: the turn
|
||||
settles (the composer re-enables, `done` observed); the final answer bubble
|
||||
contains the recovery's clean text; `document.body.innerText` contains
|
||||
**neither** `tool_call_start` nor `tool_call_end` (nor the raw
|
||||
`[read(path=…]` fragment); no error banner.
|
||||
- **Terminal case** — ask a question containing
|
||||
`SCAFFOLD_ALWAYS_TRIGGER`: the existing error status renders with the
|
||||
dedicated copy ("The model returned a malformed reply — please try
|
||||
again."); no raw tokens in the DOM; no answer bubble with the scaffolding;
|
||||
the app stays usable — a follow-up plain question (no trigger) gets a
|
||||
normal streamed answer in the same session.
|
||||
- **No false positive** — a plain question (existing `CHAT_TRIGGER`-style
|
||||
flow, no tools needed): the answer streams byte-clean, no error state, no
|
||||
recovery request visible (the turn settles on the first request).
|
||||
- The SSE wire itself: in the recovery case, no `delta` frame ever carries a
|
||||
`tool_call_start`/`tool_call_end` fragment (the existing SSE-capture house
|
||||
pattern) — the strip happens server-side, not in the UI.
|
||||
3. Gates + commit:
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_scaffolding_guardrails.py`, then the
|
||||
regression suites `test_harness_aligned_tools.py`, `test_chat_rag.py`,
|
||||
`test_agent_document_tools.py`.
|
||||
- One atomic commit (message below); move
|
||||
`.agent/phases/todo/71_scaffolding_guardrails/` →
|
||||
`.agent/phases/complete/71_scaffolding_guardrails/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new dedicated suite (recovery / terminal / no-false-positive / wire
|
||||
pins) + the three regression suites (isolation runs).
|
||||
- Coverage: **>90%** on `app/` (phase-level gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start|tool_call_end" frontend/` → no matches.
|
||||
- [ ] Raw scaffolding is never visible in the DOM in any case (recovery,
|
||||
terminal, mixed) — pinned by the dedicated suite.
|
||||
- [ ] Exactly one recovery per malformed turn (mock request counts implied by the
|
||||
flows); clean turns never carry the correction line (pinned by the unit
|
||||
suites from task 03).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
@@ -0,0 +1,165 @@
|
||||
# Phase 72 — Teaching Refusals: End the Post-Harness Tool-Loop Rambling
|
||||
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md` (this phase repairs the model-facing
|
||||
contract the phase-70 tools reshaped)
|
||||
**Context:**
|
||||
- `app/rag/agent.py` — `AGENT_TOOLS` (the phase-70 `ls` / `read` / `grep` OpenAI function
|
||||
definitions), `_execute_tool` (the refusal strings:
|
||||
`"No source named '…' — check the ls output."`,
|
||||
`"No document at '…' — check the ls output."`), `all_documents`
|
||||
(catalog-order bulk loader — reused by the suggestion lookup).
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only; the E2E mock keys off the
|
||||
`<tools>` marker's *presence*, not its wording).
|
||||
- `tests/unit/test_agent.py` (refusal-string pins; the `ScriptedLLM` + monkeypatched-
|
||||
accessor pattern), `tests/integration/test_agent_tools.py` (the same pins against real
|
||||
Postgres, `kb`/`src` fixtures).
|
||||
- `tests/e2e/mock_llm.py` — the deterministic mock tool flows (`TOOLS_TRIGGER` single-read,
|
||||
`MULTI_READ_TRIGGER`, `SEARCH_TRIGGER`; `_CATALOG_LINE_RE` catalog-line parse) and the
|
||||
dedicated-suite-per-phase E2E house pattern.
|
||||
- `scripts/llm_probe.py` — the house live-endpoint probe pattern (`python -m scripts.…`,
|
||||
argparse, dotenv, printed verdict line); `app/api/chat.py` — the grounded path the
|
||||
real-model gate mirrors (`retrieve` → `select_documents` → `build_high_prompt` →
|
||||
`run_agent`)
|
||||
- **Incident (owner chat, 2026-09-03, post phase 70/71):** the question "list the files
|
||||
in this directory" produced a Thinking-display trace of the model calling
|
||||
`ls(path='app/rag/importer.py')` → `"No source named 'app/rag/importer.py' — check the
|
||||
ls output."`, then `ls(path='.')` → the same-style refusal, then re-reasoning the same
|
||||
paragraphs over and over across rounds (each round's `reasoning_content` appends to the
|
||||
open Thinking block) before finally answering from the seed documents alone. Root cause:
|
||||
the harness-trained prior (`ls`'s `path` = a directory to list) collides with this app's
|
||||
contract (`path` = a source-name filter), and the terse refusal does not correct the
|
||||
misunderstanding, so the model burns rounds. The identical trap awaits `read`/`grep`:
|
||||
a bare document path missing the source prefix (`read('app/rag/importer.py')`) →
|
||||
`"No document at '…'"` with no hint of the combined form.
|
||||
|
||||
## Objective
|
||||
Make the affected tool refusals **teaching** so the harness-prior misuse self-corrects in
|
||||
at most one extra round: a scoped `ls` whose `path` looks like a document path (contains
|
||||
`/`) or names an unknown source gets a fixed-template refusal that states the correct
|
||||
contract; a `read` / scoped-`grep` argument that resolves to no combined identity but
|
||||
*matches an indexed document's `path`* (exact or suffix) gets a
|
||||
`"did you mean 'source/path'?"` refusal naming the exact combined identity to use. The
|
||||
`AGENT_TOOLS` `path` descriptions and the `TOOLS_SECTION` prompt copy say the same
|
||||
contract up front. Deterministic only — no model participates in detection or repair; the
|
||||
phase-70 harness shape (`ls` / `read(path)` / `grep(pattern, path?)`) is unchanged
|
||||
verbatim. The phase does not pass on mocks alone: a live acceptance gate runs the
|
||||
fixed question battery through `run_agent` against the **real configured chat model**
|
||||
(`lite` per `.env`) and must PASS before the commit (owner directive, 2026-09-03 —
|
||||
"test with the real lite model until tool calls work consistently; don't pass until a
|
||||
sufficient number of tool calls succeed").
|
||||
|
||||
## Dependencies
|
||||
- `70_harness_aligned_tools` (complete) — the tool surface this phase teaches (shape
|
||||
untouched).
|
||||
- `71_scaffolding_guardrails` (complete) — the deterministic-guardrail house style this
|
||||
phase follows.
|
||||
|
||||
## Tasks
|
||||
1. `01_ls_teaching_refusal.md` — `ls`: path-like and unknown-source scopes get teaching
|
||||
refusals; the `ls` `path` description says "source name, not a file or directory path".
|
||||
2. `02_read_grep_path_suggestion.md` — `read` / `grep`: an unresolved argument that
|
||||
matches an indexed document `path` gets the "did you mean 'source/path'?" suggestion;
|
||||
descriptions updated.
|
||||
3. `03_prompt_copy.md` — `TOOLS_SECTION` copy: the `ls` `path` is a source name, not a
|
||||
directory; `read`/`grep` need the combined identity *including the source name*.
|
||||
4. `04_mock_e2e.md` — mock `ls`-misuse flow, dedicated E2E suite (green in isolation).
|
||||
5. `05_real_model_gate.md` — the live real-lite acceptance gate
|
||||
(`scripts/agent_realmodel_check.py`): iterate the copy levers until the gate
|
||||
PASSES, then full gates and the commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` — the new `ls` teaching refusals (scope containing
|
||||
`/` → the document-path line; scope without `/` unknown → the extended no-source line;
|
||||
both count in nothing, tools stay offered; valid-scope and no-arg listings
|
||||
byte-identical to today); `find_path_candidates` (exact `path` match, suffix match,
|
||||
multiple candidates in catalog order capped at 3, zero candidates, no-`/` argument →
|
||||
no DB lookup); `read`/`grep` wiring (in-context dedupe precedence, valid combined form
|
||||
unchanged, scoped `grep` suggestion, A5 grep contract regression).
|
||||
- Integration: `tests/integration/test_agent_tools.py` — changed pins updated; new
|
||||
end-to-end suggestion cases through `run_agent` against real Postgres (bare path under
|
||||
one source; the same path under two sources).
|
||||
- E2E (mandatory, house rule): NEW dedicated suite `tests/e2e/test_tool_path_teaching.py`,
|
||||
run in isolation — the mock flow (misuse `ls(path='.')` → teaching refusal → corrected
|
||||
no-arg `ls()` → listing answer) through the real UI with the two-round shape pinned on
|
||||
the SSE wire; regression suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_search_tool.py`,
|
||||
`test_chat_rag.py`.
|
||||
- **Real-model acceptance gate (owner-locked, the phase's pass condition):**
|
||||
`uv run python -m scripts.agent_realmodel_check` against the live endpoint with the
|
||||
configured chat model (`lite`) — the fixed 10-question battery (3 `ls` turns including
|
||||
the incident's "list the files in this directory" and a source-name trap, 4 `read`
|
||||
turns including two bare-path traps, 1 `grep` turn, 2 mixed) driven through the real
|
||||
grounded path. PASS = every turn answers (no `LLMError`/`MalformedReplyError`), zero
|
||||
turns hit the round cap, ≥6 of 10 turns emit ≥1 tool call, and **≥90% of all emitted
|
||||
tool calls execute** (rejections don't count). Until it passes, task 05 iterates the
|
||||
copy levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions,
|
||||
`TOOLS_SECTION`) — the question set and thresholds are fixed by the task file and may
|
||||
not be weakened.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A scoped `ls` whose stripped `path` contains `/` gets the document-path teaching
|
||||
refusal; an unknown source name without `/` gets the extended "source name, not a
|
||||
directory" refusal; neither counts in anything; a valid scope and the no-arg
|
||||
listing are byte-identical to today.
|
||||
- [ ] `read` / scoped `grep` with an unresolved argument that matches an indexed document
|
||||
`path` (exact or suffix) gets the "did you mean …?" refusal (one candidate → one
|
||||
combined identity; two or more → up to 3, catalog order); a non-matching argument
|
||||
gets today's refusal byte-identical; the in-context dedupe refusal still wins.
|
||||
- [ ] The `AGENT_TOOLS` `path` descriptions for `ls` / `read` / `grep` state the contract
|
||||
explicitly; the tool names and argument shapes are unchanged
|
||||
(`rg '"name":' app/rag/agent.py` → exactly `ls`, `read`, `grep`).
|
||||
- [ ] `TOOLS_SECTION` clarifies the source-name `ls` `path` and the source-name-required
|
||||
combined identity; the HIGH prompt still ends with the `<tools>` section; the
|
||||
LOW/deflection prompt is byte-identical to today.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL
|
||||
**>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation; the regression suites above green in isolation.
|
||||
- [ ] `uv run python -m scripts.agent_realmodel_check` exits 0 against the live
|
||||
endpoint (all four pass conditions met with the configured model) — the verdict
|
||||
line recorded in the `app/rag/agent.py` module docstring and in the commit body.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); the phase directory
|
||||
moved to `.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **The phase-70 tool surface is unchanged** (owner lock, 2026-09-03): `ls(path?)` /
|
||||
`read(path)` / `grep(pattern, path?)` — no renames, no argument additions or removals;
|
||||
this phase changes refusal copy, tool descriptions, and prompt copy only.
|
||||
- **Deterministic only** (owner 2026-09-03, phase-71 house style): no model in detection
|
||||
or repair; suggestions are a pure catalog lookup (exact or suffix `path` match,
|
||||
case-sensitive, catalog order, capped at 3); every refusal is a fixed template
|
||||
constant.
|
||||
- **Teach, don't silently fix:** a misused call is still a refusal (counts in nothing,
|
||||
consumes a round); the model sees its own argument echoed plus the correct form. No
|
||||
silent argument normalization — `ls(path='.')` does NOT become a full listing.
|
||||
- **The zero-candidate refusal is byte-identical to today**
|
||||
(`"No document at '…' — check the ls output."`) — no behavior change where the model is
|
||||
not confused; the `ls` no-source refusal keeps its prefix (the teaching parenthetical
|
||||
is appended).
|
||||
- **No UI change:** the Thinking display (phases 17/21/43) works as designed — the fix
|
||||
ends the loop, it does not hide the scratchpad. **No SSE contract change** (refusals
|
||||
are tool results in the message history; the `tool` frames already carry the call's
|
||||
`name`/`argument`). **No model swap** (owner keeps `lite`), **no env change**,
|
||||
**no schema change**.
|
||||
- **Real-model gate is a pass condition, not a smoke test** (owner directive,
|
||||
2026-09-03): the phase is NOT complete — and gets NO commit — until
|
||||
`scripts/agent_realmodel_check.py` PASSES against the real `lite` model. The 10
|
||||
questions, the ≥6-of-10 tool-usage floor, the ≥90% executed-call bar, and the
|
||||
zero-cap rule are fixed by task 05's file; the executor may iterate ONLY the copy
|
||||
levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions,
|
||||
`TOOLS_SECTION` — with their unit pins updated to follow the constants). Lowering a
|
||||
threshold, swapping in easier questions, or skipping the gate to "make it pass" is
|
||||
forbidden; a gate still failing after iteration stops the phase with the per-turn
|
||||
numbers reported for the owner (fail-loud house style). The verdict line (house
|
||||
precedent: the phase-37 probe verdict in `app/rag/agent.py`) is recorded in that
|
||||
module's docstring and in the commit body.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ scripts/ && git commit --no-gpg-sign -m "fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling" -m "<real-model gate verdict line, e.g. real-model gate (lite): 10/10 answered, caps=0, tool-turns=8, calls 21/23 executed (91%) — 2026-09-03>"
|
||||
```
|
||||
The commit also carries the still-uncommitted phase-71 `todo/` → `complete/` move and
|
||||
`.agent/reports/71_scaffolding_guardrails/` (`.agent/` is tracked and committed with the
|
||||
phase — AGENTS.md §8; only `.agent/phase-sessions/` and `.agent/pipeline.log` are
|
||||
gitignored).
|
||||
@@ -0,0 +1,64 @@
|
||||
# Task 01 — `ls`: Teaching Refusals for Path-Like and Unknown-Source Scopes
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
A scoped `ls` whose `path` argument is a file/directory path (contains `/`) — or an
|
||||
unknown source name — gets a fixed-template refusal that states the correct contract
|
||||
instead of the terse "check the ls output", so the harness-prior misuse
|
||||
(`ls(path='app/rag/importer.py')`, `ls(path='.')` — the incident) self-corrects in one
|
||||
round. The `ls` tool description makes the same point at request time.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — two refusal template constants next to the existing refusal
|
||||
constants, plus the branch change:
|
||||
- `LS_PATH_NOT_A_SOURCE: str` — one `{path}` field, used when the **stripped** scope
|
||||
contains `/` (a source name can never contain `/` — source names are directory
|
||||
basenames, `app.rag.importer`):
|
||||
`"'{path}' looks like a document path, not a source name. The 'path' argument of ls filters by source name (e.g. 'homelab') — omit it to list every document, or read a document by its combined 'source/path' string."`
|
||||
- `NO_SOURCE_NOT_A_DIRECTORY: str` — one `{scope}` field, the existing no-source
|
||||
refusal with a teaching parenthetical appended (the prefix
|
||||
`"No source named '{scope}' — check the ls output."` stays byte-identical), used
|
||||
when the scope has no `/` and matches no registered source name:
|
||||
`"No source named '{scope}' — check the ls output. (The 'path' argument is a source name, not a directory — omit it to list every document.)"`
|
||||
- `_execute_tool` `ls` branch: non-empty scope with `"/" in scope` →
|
||||
`LS_PATH_NOT_A_SOURCE.format(path=scope)`; non-empty scope without `/` not in
|
||||
`list_source_names(db)` → `NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)`; a valid
|
||||
scope and the no-arg listing are unchanged. Both refusals count in nothing (no
|
||||
`holder.tool_calls` bump) and consume a round — exactly like today's refusal.
|
||||
- `AGENT_TOOLS` → `ls` → `function.parameters.properties.path.description`:
|
||||
`"Source name to list one source's documents (e.g. 'homelab') — a source name, not a file or directory path; omit to list every document."`
|
||||
- Module docstring (loop contract, point 3 — the refusal list): update the
|
||||
scoped-`ls` refusal entry to the two new lines.
|
||||
2. `tests/unit/test_agent.py` — unit pins (existing `ScriptedLLM` + monkeypatched
|
||||
`list_catalog` / `list_source_names` pattern; import the constants, never re-type
|
||||
them):
|
||||
- `ls(path='app/rag/importer.py')` (scope contains `/`) → the
|
||||
`LS_PATH_NOT_A_SOURCE` line with the argument echoed; `holder.tool_calls == 0`;
|
||||
tools stay offered on the next request.
|
||||
- `ls(path='.')` (no `/`, unknown) → the `NO_SOURCE_NOT_A_DIRECTORY` line with
|
||||
`'.'` echoed; `holder.tool_calls == 0`.
|
||||
- `ls(path='Ghost')` (no `/`, unknown) → the same extended line (replaces today's
|
||||
`test_ls_scoped_unknown_source_refused` pin).
|
||||
- Regression: `ls()` no-arg full catalog and `ls(path='<registered source>')` scoped
|
||||
listing (including the `0 documents:` registered-empty-source case) remain
|
||||
byte-identical to today.
|
||||
3. `tests/integration/test_agent_tools.py` — update the changed pin (the
|
||||
`ls(path='Ghost')` assertion) and add one case: a scoped `ls` with a `/`-containing
|
||||
`path` against the real DB (`kb` + `src` fixtures) → the document-path line, not
|
||||
counted, tools stay offered.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: as listed in Work 2–3 — every new refusal line pinned
|
||||
byte-for-byte; the count-in-nothing and tools-stay-offered invariants pinned; the
|
||||
unchanged paths regression-pinned.
|
||||
- Coverage: **>90%** on this task's new/modified code (the `ls` branch in
|
||||
`app/rag/agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov`
|
||||
green (DB up: `podman compose up -d db`)
|
||||
- [ ] The old terse string (no-source refusal without the parenthetical) appears nowhere
|
||||
in `app/` or `tests/`
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No behavior change to valid-scope / no-arg `ls` (regression pins green)
|
||||
@@ -0,0 +1,87 @@
|
||||
# Task 02 — `read` / `grep`: "did you mean 'source/path'?" Suggestions for Bare Document Paths
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
When `read` (or a scoped `grep`) receives an argument that resolves to no combined
|
||||
identity but *does* match an indexed document's `path` (exact or as a suffix), the
|
||||
refusal names the exact combined `source/path` identity to use — the harness prior
|
||||
(`read('app/rag/importer.py')`, missing the source prefix) self-corrects in one round.
|
||||
An argument that matches nothing keeps today's refusal byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py`:
|
||||
- `SUGGESTION_LIMIT = 3` — the cap on suggested identities per refusal.
|
||||
- Module-level `find_path_candidates(db: Session, arg: str) -> list[tuple[str, str, str]]`
|
||||
(so unit tests can monkeypatch it, house pattern): the indexed documents, in
|
||||
**catalog order** (the `all_documents` order), whose `path` equals `arg` or ends
|
||||
with `f"/{arg}"` (case-sensitive — these are file paths), as `(source, path, title)`
|
||||
triples. One bulk query via `all_documents`; called **only** from the refusal path
|
||||
below (never on the happy path) and **only** when `arg` contains `/` (a bare name
|
||||
keeps today's no-DB-lookup refusal — the existing
|
||||
`test_read_bare_source_name_refused_without_db` invariant stays green).
|
||||
- Refusal templates next to the existing constants:
|
||||
- `NO_DOCUMENT_DID_YOU_MEAN: str` —
|
||||
`"No document at '{arg}' — did you mean '{source}/{path}'?"`
|
||||
- `NO_DOCUMENT_DID_YOU_MEAN_MANY: str` —
|
||||
`"No document at '{arg}' — did you mean one of: {candidates}?"` where
|
||||
`{candidates}` is up to `SUGGESTION_LIMIT` combined `source/path` identities,
|
||||
each single-quoted, joined with `", "`, in catalog order.
|
||||
- `_execute_tool` `read` branch: after the in-context dedupe check and the
|
||||
`_resolve_path` miss — when `arg` contains `/`, run `find_path_candidates`:
|
||||
exactly 1 candidate → `NO_DOCUMENT_DID_YOU_MEAN`; 2+ →
|
||||
`NO_DOCUMENT_DID_YOU_MEAN_MANY`; 0 → today's
|
||||
`"No document at '{arg}' — check the ls output."` unchanged. `holder` untouched
|
||||
(a refusal counts in nothing; `read_docs` untouched — locator-only never changes).
|
||||
- `_execute_tool` `grep` branch: the same substitution for the scoped-`path` miss
|
||||
(the whole-KB grep is untouched).
|
||||
- `AGENT_TOOLS` → `read` → `path` description and `grep` → `path` description:
|
||||
append `" A bare document path (without the source name) will not resolve."` to
|
||||
each current text.
|
||||
- Module docstring (loop contract, point 3): document the suggestion behavior in the
|
||||
refusal list.
|
||||
2. `tests/unit/test_agent.py` — unit pins (monkeypatched `find_document` +
|
||||
`all_documents`; import the constants, never re-type them):
|
||||
- `read(path='active/container_caddy/caddy.md')` with the document indexed under
|
||||
`Homelab` (exact `path` match) → `did you mean 'Homelab/active/container_caddy/caddy.md'?`;
|
||||
`holder.read_docs` empty, `holder.tool_calls == 0`.
|
||||
- Suffix match: `read(path='caddy.md')` → the same single suggestion.
|
||||
- Two sources sharing the same `path` →
|
||||
`did you mean one of: 'A/x.md', 'B/x.md'?` in catalog order.
|
||||
- Four sources sharing the `path` → exactly 3 suggestions (the cap).
|
||||
- No match → today's refusal byte-identical; `read(path='Homelab')` (bare, no `/`) →
|
||||
today's refusal with **no** `find_document` / `all_documents` call (the `_boom`
|
||||
guard, existing pattern).
|
||||
- Dedupe precedence: a combined-form re-read of a `seed_docs` document →
|
||||
`ALREADY_IN_CONTEXT` (unchanged); a bare-`path` read of an in-context document
|
||||
(`read('app/rag/importer.py')` with `sample/app/rag/importer.py` seeded) → the
|
||||
suggestion line (the split pair is not in `known`, so the model learns the
|
||||
combined identity — its next, correctly-formed call is then deduped).
|
||||
- Scoped `grep` miss with a candidate → the suggestion line; scoped `grep` miss
|
||||
without → today's line; whole-KB `grep` unchanged (A5 match/output contract:
|
||||
fixed substring, case-insensitive, 20 matches, 200-char lines).
|
||||
- Happy paths regression-pinned: combined-form `read` (full content,
|
||||
`read_docs` appended), valid scoped `grep` result.
|
||||
3. `tests/integration/test_agent_tools.py` — update the changed pins (inspect each
|
||||
existing `"No document at …"` assertion against the fixture documents; only the
|
||||
lines whose argument matches a fixture document `path` change to the suggestion form),
|
||||
plus two new end-to-end cases through `run_agent` against real Postgres: a bare path
|
||||
under one source (single suggestion) and the same `path` under two sources (the
|
||||
"one of" line) — in both, the refusal is followed by the model's corrected call
|
||||
succeeding (scripted `ToolCallPiece` round 2 with the suggested combined identity).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: as listed in Work 2–3 — every new template pinned byte-for-byte;
|
||||
the catalog-order + cap invariant pinned; the zero-candidate and no-DB-lookup
|
||||
invariants pinned; the A5 `grep` contract regression-pinned.
|
||||
- Coverage: **>90%** on this task's new/modified code (`find_path_candidates` + both
|
||||
`_execute_tool` branches).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov`
|
||||
green (DB up: `podman compose up -d db`)
|
||||
- [ ] `find_path_candidates` is module-level (monkeypatchable) and issues at most one
|
||||
bulk query
|
||||
- [ ] The zero-candidate refusal and the bare-name (no-DB-lookup) refusal are
|
||||
byte-identical to today
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 03 — `TOOLS_SECTION` Copy: State the Contract Up Front
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The HIGH prompt's `<tools>` section says the same two things the new refusals teach —
|
||||
the `ls` `path` is a *source name*, not a directory or file path, and `read`/`grep`
|
||||
need the combined `source/path` string *including the source name* — so the model
|
||||
carries the contract before it calls a tool, not only after being refused.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `TOOLS_SECTION` rewritten (the E2E mock keys off the
|
||||
`<tools>` marker's *presence*, not this wording, so the change is mock-safe):
|
||||
- `ls` clause: its optional `path` argument is a *source name* (e.g. `'homelab'`)
|
||||
— **not** a directory or file path; omit it to list every document.
|
||||
- `read` clause: the combined `source/path` string, exactly as shown in the `ls`
|
||||
output — *including the source name*; a bare document path will not resolve.
|
||||
- `grep` clause: the locator copy stays (its `path` is already described as a
|
||||
combined `source/path` string); add the same bare-path-will-not-resolve note.
|
||||
- Keep the section's shape: a single paragraph between `<tools>` and `</tools>`,
|
||||
still appended after the mode body in the HIGH prompt only (the LOW/deflection
|
||||
prompt never carries it — phase 71's plain-text line stays put).
|
||||
2. `tests/unit/test_prompts.py` — update the `TOOLS_SECTION` wording pin(s) where they
|
||||
pin the old wording; the `<tools>`-marker-present-in-HIGH pin, the
|
||||
marker-absent-from-LOW pin, and the byte-identical-LOW-prompt pin stay green as-is.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_prompts.py` — marker present in the HIGH prompt and absent
|
||||
from the LOW prompt; the LOW prompt byte-identical to today; the new wording pinned
|
||||
for the `ls` source-name clause and the read combined-identity clause.
|
||||
- Coverage: **>90%** on this task's modified code (the constant itself — the builders
|
||||
are already covered).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py -v --no-cov` green
|
||||
- [ ] The HIGH prompt still ends with the `<tools>` section (existing section-order pin
|
||||
green); the LOW/deflection prompt is byte-identical to today
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,63 @@
|
||||
# Task 04 — Mock Flow, Dedicated E2E Suite
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Prove the self-correction loop deterministically through the real UI: a mock-LLM flow
|
||||
that reproduces the incident's `ls(path='.')` misuse, receives the teaching refusal,
|
||||
corrects to a no-arg `ls()`, and answers from the catalog — a dedicated Playwright
|
||||
suite pinning the two-round shape on the SSE wire. (The live real-model acceptance
|
||||
gate is task 05 — this task is the deterministic half of the proof.)
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — one new deterministic flow, checked in the flow table
|
||||
**before** the plain `TOOLS_TRIGGER` flow (the trigger phrases are disjoint
|
||||
substrings; the ordering rule follows the phase-71 convention):
|
||||
- `LS_TEACH_TRIGGER = "list the files in this directory"` — **and** the system
|
||||
prompt carries the `<tools>` section (grounded turn):
|
||||
* request 1 (tools offered, no `tool`-role result in the messages yet): stream
|
||||
ONLY `tool_calls` deltas — `ls` with `{"path": "."}` (synthetic id `call_0`),
|
||||
`finish_reason: "tool_calls"`, no content (the incident's misuse,
|
||||
deterministic);
|
||||
* request 2 (a `tool`-role result present that is **not** a catalog listing —
|
||||
i.e. the teaching refusal): stream a `tool_calls` delta — `ls` with no
|
||||
arguments (id `call_1`);
|
||||
* request 3 (a `tool`-role result whose first line matches the
|
||||
`^\d+ documents:` catalog header): a deterministic content answer —
|
||||
`These are the indexed documents: <first catalog line>` (the
|
||||
`source: X | path: Y | title: Z` line, parsed with the existing
|
||||
`_CATALOG_LINE_RE` machinery), `finish_reason: "stop"`.
|
||||
- Update the module docstring's flow table with the phase-72 note.
|
||||
2. `tests/e2e/test_tool_path_teaching.py` (NEW — the phase's dedicated suite, house
|
||||
pattern, run in isolation; DB up, mock LLM):
|
||||
- Import a small fixture document set (house fixture pattern: one source, two
|
||||
documents with known `source`/`path`/`title`) and ask a question containing
|
||||
`LS_TEACH_TRIGGER`.
|
||||
- **Self-correction** — the turn settles (composer re-enables, `done` observed);
|
||||
the answer bubble contains the first document's `source:` and `path:` fields
|
||||
(the catalog reached the model and landed in the answer); no error banner.
|
||||
- **Two rounds on the wire** (the house SSE-capture pattern): the `tool` frames
|
||||
arrive in order — first `name:"ls"` with `argument:"."`, then `name:"ls"` with
|
||||
`argument:null` — and there is **no** third `tool` frame (the loop ended in one
|
||||
correction, not at the round cap).
|
||||
- **No regression to the plain flow** — a follow-up question containing
|
||||
`TOOLS_TRIGGER` (the single-read flow) in the same session still settles with
|
||||
the read flow's answer (the new flow did not swallow the existing trigger).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the dedicated suite proves the loop shape (misuse → teaching refusal →
|
||||
corrected call → answer) through the real UI and the SSE wire; the existing
|
||||
regression E2E suites (mock-driven) stay green — run them as the regression check
|
||||
for this task.
|
||||
- Coverage: unit/integration coverage of `app/` stays >90% (this task adds test-only
|
||||
code; `uv run pytest --cov=app --cov-report=term-missing` as the check).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation (DB up: `podman compose up -d db`, mock LLM)
|
||||
- [ ] Regression E2E suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`, `test_chat_rag.py`
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No commit in this task (the commit happens in task 05, after the real-model
|
||||
gate passes)
|
||||
@@ -0,0 +1,114 @@
|
||||
# Task 05 — Real-Model Acceptance Gate (live `lite`), Full Gates, Commit
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The phase's pass condition (owner directive, 2026-09-03: "test with the real lite
|
||||
model until tool calls work consistently — don't pass until a sufficient number of
|
||||
tool calls succeed"): a live script drives the fixed 10-question battery through the
|
||||
**real** grounded path (real endpoint, configured chat model — `lite` per `.env`,
|
||||
real Postgres KB) and the phase commits only when the gate PASSES. Until it does,
|
||||
iterate the copy levers this phase owns (refusal templates, `AGENT_TOOLS`
|
||||
descriptions, `TOOLS_SECTION`) — never the gate.
|
||||
|
||||
## Work
|
||||
1. `scripts/agent_realmodel_check.py` (NEW — house probe pattern, `scripts/llm_probe.py`
|
||||
as the model: `uv run python -m scripts.agent_realmodel_check`, argparse, dotenv,
|
||||
plain module, no debugpy):
|
||||
- **Preconditions (exit 2 with an actionable line on failure):** DB reachable;
|
||||
the catalog holds ≥2 documents; the FIRST TWO catalog documents' `path`s each
|
||||
contain `/` (the bare-path traps need nested paths); `settings.agent_max_rounds
|
||||
> 0` (the gate needs tools enabled).
|
||||
- **Mirror the grounded path of `app/api/chat.py` exactly** (same prompt the UI
|
||||
gets): per question — embed it, `retrieve`, `select_documents`, steering notes
|
||||
+ KB overview as chat.py reads them, `build_high_prompt(docs, notes, kb_overview)`,
|
||||
then `run_agent(llm, db, system_prompt=…, user_message=…, seed_docs=docs,
|
||||
settings=settings, holder=AgentHolder())` with a **fresh** `AgentHolder` per
|
||||
turn, consuming every piece to the end. Never modify the KB.
|
||||
- **Fixed question battery** (locked — the executor may not swap in easier
|
||||
questions). Let the first two catalog documents be
|
||||
`D1 = (s1, p1, t1)` and `D2 = (s2, p2, t2)`, and `token` = the first
|
||||
whitespace-split word of `D2.content` with length ≥ 6 (strip leading/trailing
|
||||
non-alphanumerics, lowercase; fallback: the first word of `t2`):
|
||||
1. `List the files in this directory.` (the incident)
|
||||
2. `List the documents you have in the {s1} source.`
|
||||
3. `List every document you have indexed.`
|
||||
4. `What does the document {p1} contain? Open it and tell me.` (bare-path `read` trap)
|
||||
5. `Read {s1}/{p1} and summarize it.` (combined form — the correct shape)
|
||||
6. `Open the document {p2} and tell me what it covers.` (bare-path `read` trap)
|
||||
7. `Find the exact string "{token}" in your documents and tell me which ones contain it.` (`grep`)
|
||||
8. `Which document has the title "{t2}"? Read it and summarize.`
|
||||
9. `What do you know about {t1}? Open the relevant document and give me specifics.`
|
||||
10. `List the files in the {s2} directory.` (source name phrased as a directory)
|
||||
- **Per-turn measurement** (from the consumed stream + the holder — no app-code
|
||||
changes for measurement): `emitted` = count of yielded `ToolCallPiece`s;
|
||||
`executed` = `holder.tool_calls` (refusals count in nothing); `rejected` =
|
||||
`emitted − executed`; `cap_reached` = `emitted >= settings.agent_max_rounds`
|
||||
(every capped round emitted a call, so the cap implies at that many emissions
|
||||
and never the reverse); `answered` = the stream finished without
|
||||
`LLMError`/`MalformedReplyError`. Print one line per turn:
|
||||
`turn 04 | emitted=2 executed=1 cap=no | What does the document …`.
|
||||
- **Verdict + pass conditions (locked):**
|
||||
1. all 10 turns `answered`;
|
||||
2. zero `cap_reached` turns (the incident's loop signature — hitting the cap
|
||||
means the teaching did not end the loop);
|
||||
3. ≥6 of 10 turns with `emitted ≥ 1` (the model keeps USING tools — it does not
|
||||
abandon them and answer from seed context alone, the incident's end state);
|
||||
4. `executed / emitted ≥ 0.90` across the whole run (the "sufficient number of
|
||||
tool calls succeed" bar; a run with zero emitted calls fails condition 3
|
||||
anyway).
|
||||
Print the single verdict line in a stable format, e.g.
|
||||
`gate: lite PASS turns=10 answered=10 caps=0 tool-turns=8 calls 21/23 executed (91%) 2026-09-03`
|
||||
(model = `settings.llm_chat_model`, date = run date). **Exit 0 on PASS, 1 on
|
||||
FAIL, 2 on precondition failure.**
|
||||
- On FAIL, also print a short per-condition breakdown (which condition(s) missed)
|
||||
so the iteration loop can target the right lever. For refusal diagnosis, each
|
||||
call is already logged by `run_agent` (`agent tool=… args=… round=…/…`) —
|
||||
correlate the logged arguments with the refusal templates in
|
||||
`app/rag/agent.py` to see which teaching line the model hit.
|
||||
2. **Run the gate and iterate until it PASSES** (the loop this task exists for):
|
||||
`podman compose up -d db` → `uv run python -m scripts.agent_realmodel_check`.
|
||||
On FAIL: change ONLY the copy levers this phase owns — the refusal templates
|
||||
(task 01/02 constants), the `AGENT_TOOLS` `path` descriptions (task 01/02),
|
||||
`TOOLS_SECTION` (task 03) — with their unit pins updated to follow the constants;
|
||||
`uv run pytest` green again; re-run the gate. Repeat. **Forbidden:** lowering any
|
||||
threshold, swapping questions, disabling a tool, or weakening condition 4 to make
|
||||
it pass. If the gate still fails after a genuine iteration (the numbers stop
|
||||
improving across levers), STOP: no commit — report the per-turn lines, the
|
||||
verdict, and which refusals the model hit (from the `app.agent` log) in the task
|
||||
report for the owner (fail-loud house style).
|
||||
3. **Record the verdict** (house precedent — the phase-37 probe verdict lives in the
|
||||
`app/rag/agent.py` module docstring): append one line to that docstring —
|
||||
`Real-model gate (phase 72, task 05 — live vs the configured chat model):
|
||||
<the verdict line, verbatim>`.
|
||||
4. **Full gates + commit:**
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_path_teaching.py`, then the regression
|
||||
suites `test_harness_aligned_tools.py`, `test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`.
|
||||
- Move the phase directory: `mv .agent/phases/todo/72_teaching_refusals
|
||||
.agent/phases/complete/`.
|
||||
- Commit — one, `--no-gpg-sign`, the Commit block of `00_phase.md`: the title
|
||||
message plus a **body line carrying the gate verdict verbatim**. The commit
|
||||
also carries the still-uncommitted phase-71 `todo/` → `complete/` move and
|
||||
`.agent/reports/71_scaffolding_guardrails/` (`.agent/` is tracked — AGENTS.md
|
||||
§8; `git add -A .agent/ app/ tests/ scripts/` picks up everything).
|
||||
|
||||
## Testing & Quality
|
||||
- The script IS the test for this task: it is deterministic in its question set,
|
||||
thresholds, and output format (a future executor re-running it gets comparable
|
||||
numbers); its precondition failures exit 2 with actionable text. The script itself
|
||||
needs no unit tests (it is an entrypoint probe, `scripts/llm_probe.py` precedent),
|
||||
but the copy iterations it drives must keep `uv run pytest` + coverage >90% green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run python -m scripts.agent_realmodel_check` exits **0** against the live
|
||||
endpoint (all four pass conditions met with the configured model — `lite`);
|
||||
the verdict line verbatim in `app/rag/agent.py`'s module docstring
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation; the regression E2E suites green in isolation (the Work-4 list)
|
||||
- [ ] One `--no-gpg-sign` commit whose body carries the gate verdict; the phase
|
||||
directory under `.agent/phases/complete/72_teaching_refusals/`
|
||||
Reference in New Issue
Block a user