refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
@@ -0,0 +1,165 @@
|
||||
# Phase 72 — Teaching Refusals: End the Post-Harness Tool-Loop Rambling
|
||||
|
||||
**Story:** `.agents/user_stories/agent-document-tools.md` (this phase repairs the model-facing
|
||||
contract the phase-70 tools reshaped)
|
||||
**Context:**
|
||||
- `app/rag/agent.py` — `AGENT_TOOLS` (the phase-70 `ls` / `read` / `grep` OpenAI function
|
||||
definitions), `_execute_tool` (the refusal strings:
|
||||
`"No source named '…' — check the ls output."`,
|
||||
`"No document at '…' — check the ls output."`), `all_documents`
|
||||
(catalog-order bulk loader — reused by the suggestion lookup).
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only; the E2E mock keys off the
|
||||
`<tools>` marker's *presence*, not its wording).
|
||||
- `tests/unit/test_agent.py` (refusal-string pins; the `ScriptedLLM` + monkeypatched-
|
||||
accessor pattern), `tests/integration/test_agent_tools.py` (the same pins against real
|
||||
Postgres, `kb`/`src` fixtures).
|
||||
- `tests/e2e/mock_llm.py` — the deterministic mock tool flows (`TOOLS_TRIGGER` single-read,
|
||||
`MULTI_READ_TRIGGER`, `SEARCH_TRIGGER`; `_CATALOG_LINE_RE` catalog-line parse) and the
|
||||
dedicated-suite-per-phase E2E house pattern.
|
||||
- `scripts/llm_probe.py` — the house live-endpoint probe pattern (`python -m scripts.…`,
|
||||
argparse, dotenv, printed verdict line); `app/api/chat.py` — the grounded path the
|
||||
real-model gate mirrors (`retrieve` → `select_documents` → `build_high_prompt` →
|
||||
`run_agent`)
|
||||
- **Incident (owner chat, 2026-09-03, post phase 70/71):** the question "list the files
|
||||
in this directory" produced a Thinking-display trace of the model calling
|
||||
`ls(path='app/rag/importer.py')` → `"No source named 'app/rag/importer.py' — check the
|
||||
ls output."`, then `ls(path='.')` → the same-style refusal, then re-reasoning the same
|
||||
paragraphs over and over across rounds (each round's `reasoning_content` appends to the
|
||||
open Thinking block) before finally answering from the seed documents alone. Root cause:
|
||||
the harness-trained prior (`ls`'s `path` = a directory to list) collides with this app's
|
||||
contract (`path` = a source-name filter), and the terse refusal does not correct the
|
||||
misunderstanding, so the model burns rounds. The identical trap awaits `read`/`grep`:
|
||||
a bare document path missing the source prefix (`read('app/rag/importer.py')`) →
|
||||
`"No document at '…'"` with no hint of the combined form.
|
||||
|
||||
## Objective
|
||||
Make the affected tool refusals **teaching** so the harness-prior misuse self-corrects in
|
||||
at most one extra round: a scoped `ls` whose `path` looks like a document path (contains
|
||||
`/`) or names an unknown source gets a fixed-template refusal that states the correct
|
||||
contract; a `read` / scoped-`grep` argument that resolves to no combined identity but
|
||||
*matches an indexed document's `path`* (exact or suffix) gets a
|
||||
`"did you mean 'source/path'?"` refusal naming the exact combined identity to use. The
|
||||
`AGENT_TOOLS` `path` descriptions and the `TOOLS_SECTION` prompt copy say the same
|
||||
contract up front. Deterministic only — no model participates in detection or repair; the
|
||||
phase-70 harness shape (`ls` / `read(path)` / `grep(pattern, path?)`) is unchanged
|
||||
verbatim. The phase does not pass on mocks alone: a live acceptance gate runs the
|
||||
fixed question battery through `run_agent` against the **real configured chat model**
|
||||
(`lite` per `.env`) and must PASS before the commit (owner directive, 2026-09-03 —
|
||||
"test with the real lite model until tool calls work consistently; don't pass until a
|
||||
sufficient number of tool calls succeed").
|
||||
|
||||
## Dependencies
|
||||
- `70_harness_aligned_tools` (complete) — the tool surface this phase teaches (shape
|
||||
untouched).
|
||||
- `71_scaffolding_guardrails` (complete) — the deterministic-guardrail house style this
|
||||
phase follows.
|
||||
|
||||
## Tasks
|
||||
1. `01_ls_teaching_refusal.md` — `ls`: path-like and unknown-source scopes get teaching
|
||||
refusals; the `ls` `path` description says "source name, not a file or directory path".
|
||||
2. `02_read_grep_path_suggestion.md` — `read` / `grep`: an unresolved argument that
|
||||
matches an indexed document `path` gets the "did you mean 'source/path'?" suggestion;
|
||||
descriptions updated.
|
||||
3. `03_prompt_copy.md` — `TOOLS_SECTION` copy: the `ls` `path` is a source name, not a
|
||||
directory; `read`/`grep` need the combined identity *including the source name*.
|
||||
4. `04_mock_e2e.md` — mock `ls`-misuse flow, dedicated E2E suite (green in isolation).
|
||||
5. `05_real_model_gate.md` — the live real-lite acceptance gate
|
||||
(`scripts/agent_realmodel_check.py`): iterate the copy levers until the gate
|
||||
PASSES, then full gates and the commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` — the new `ls` teaching refusals (scope containing
|
||||
`/` → the document-path line; scope without `/` unknown → the extended no-source line;
|
||||
both count in nothing, tools stay offered; valid-scope and no-arg listings
|
||||
byte-identical to today); `find_path_candidates` (exact `path` match, suffix match,
|
||||
multiple candidates in catalog order capped at 3, zero candidates, no-`/` argument →
|
||||
no DB lookup); `read`/`grep` wiring (in-context dedupe precedence, valid combined form
|
||||
unchanged, scoped `grep` suggestion, A5 grep contract regression).
|
||||
- Integration: `tests/integration/test_agent_tools.py` — changed pins updated; new
|
||||
end-to-end suggestion cases through `run_agent` against real Postgres (bare path under
|
||||
one source; the same path under two sources).
|
||||
- E2E (mandatory, house rule): NEW dedicated suite `tests/e2e/test_tool_path_teaching.py`,
|
||||
run in isolation — the mock flow (misuse `ls(path='.')` → teaching refusal → corrected
|
||||
no-arg `ls()` → listing answer) through the real UI with the two-round shape pinned on
|
||||
the SSE wire; regression suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_search_tool.py`,
|
||||
`test_chat_rag.py`.
|
||||
- **Real-model acceptance gate (owner-locked, the phase's pass condition):**
|
||||
`uv run python -m scripts.agent_realmodel_check` against the live endpoint with the
|
||||
configured chat model (`lite`) — the fixed 10-question battery (3 `ls` turns including
|
||||
the incident's "list the files in this directory" and a source-name trap, 4 `read`
|
||||
turns including two bare-path traps, 1 `grep` turn, 2 mixed) driven through the real
|
||||
grounded path. PASS = every turn answers (no `LLMError`/`MalformedReplyError`), zero
|
||||
turns hit the round cap, ≥6 of 10 turns emit ≥1 tool call, and **≥90% of all emitted
|
||||
tool calls execute** (rejections don't count). Until it passes, task 05 iterates the
|
||||
copy levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions,
|
||||
`TOOLS_SECTION`) — the question set and thresholds are fixed by the task file and may
|
||||
not be weakened.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A scoped `ls` whose stripped `path` contains `/` gets the document-path teaching
|
||||
refusal; an unknown source name without `/` gets the extended "source name, not a
|
||||
directory" refusal; neither counts in anything; a valid scope and the no-arg
|
||||
listing are byte-identical to today.
|
||||
- [ ] `read` / scoped `grep` with an unresolved argument that matches an indexed document
|
||||
`path` (exact or suffix) gets the "did you mean …?" refusal (one candidate → one
|
||||
combined identity; two or more → up to 3, catalog order); a non-matching argument
|
||||
gets today's refusal byte-identical; the in-context dedupe refusal still wins.
|
||||
- [ ] The `AGENT_TOOLS` `path` descriptions for `ls` / `read` / `grep` state the contract
|
||||
explicitly; the tool names and argument shapes are unchanged
|
||||
(`rg '"name":' app/rag/agent.py` → exactly `ls`, `read`, `grep`).
|
||||
- [ ] `TOOLS_SECTION` clarifies the source-name `ls` `path` and the source-name-required
|
||||
combined identity; the HIGH prompt still ends with the `<tools>` section; the
|
||||
LOW/deflection prompt is byte-identical to today.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL
|
||||
**>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation; the regression suites above green in isolation.
|
||||
- [ ] `uv run python -m scripts.agent_realmodel_check` exits 0 against the live
|
||||
endpoint (all four pass conditions met with the configured model) — the verdict
|
||||
line recorded in the `app/rag/agent.py` module docstring and in the commit body.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); the phase directory
|
||||
moved to `.agents/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **The phase-70 tool surface is unchanged** (owner lock, 2026-09-03): `ls(path?)` /
|
||||
`read(path)` / `grep(pattern, path?)` — no renames, no argument additions or removals;
|
||||
this phase changes refusal copy, tool descriptions, and prompt copy only.
|
||||
- **Deterministic only** (owner 2026-09-03, phase-71 house style): no model in detection
|
||||
or repair; suggestions are a pure catalog lookup (exact or suffix `path` match,
|
||||
case-sensitive, catalog order, capped at 3); every refusal is a fixed template
|
||||
constant.
|
||||
- **Teach, don't silently fix:** a misused call is still a refusal (counts in nothing,
|
||||
consumes a round); the model sees its own argument echoed plus the correct form. No
|
||||
silent argument normalization — `ls(path='.')` does NOT become a full listing.
|
||||
- **The zero-candidate refusal is byte-identical to today**
|
||||
(`"No document at '…' — check the ls output."`) — no behavior change where the model is
|
||||
not confused; the `ls` no-source refusal keeps its prefix (the teaching parenthetical
|
||||
is appended).
|
||||
- **No UI change:** the Thinking display (phases 17/21/43) works as designed — the fix
|
||||
ends the loop, it does not hide the scratchpad. **No SSE contract change** (refusals
|
||||
are tool results in the message history; the `tool` frames already carry the call's
|
||||
`name`/`argument`). **No model swap** (owner keeps `lite`), **no env change**,
|
||||
**no schema change**.
|
||||
- **Real-model gate is a pass condition, not a smoke test** (owner directive,
|
||||
2026-09-03): the phase is NOT complete — and gets NO commit — until
|
||||
`scripts/agent_realmodel_check.py` PASSES against the real `lite` model. The 10
|
||||
questions, the ≥6-of-10 tool-usage floor, the ≥90% executed-call bar, and the
|
||||
zero-cap rule are fixed by task 05's file; the executor may iterate ONLY the copy
|
||||
levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions,
|
||||
`TOOLS_SECTION` — with their unit pins updated to follow the constants). Lowering a
|
||||
threshold, swapping in easier questions, or skipping the gate to "make it pass" is
|
||||
forbidden; a gate still failing after iteration stops the phase with the per-turn
|
||||
numbers reported for the owner (fail-loud house style). The verdict line (house
|
||||
precedent: the phase-37 probe verdict in `app/rag/agent.py`) is recorded in that
|
||||
module's docstring and in the commit body.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agents/ app/ tests/ scripts/ && git commit --no-gpg-sign -m "fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling" -m "<real-model gate verdict line, e.g. real-model gate (lite): 10/10 answered, caps=0, tool-turns=8, calls 21/23 executed (91%) — 2026-09-03>"
|
||||
```
|
||||
The commit also carries the still-uncommitted phase-71 `todo/` → `complete/` move and
|
||||
`.agents/reports/71_scaffolding_guardrails/` (`.agents/` is tracked and committed with the
|
||||
phase — AGENTS.md §8; only `.agents/phase-sessions/` and `.agents/pipeline.log` are
|
||||
gitignored).
|
||||
@@ -0,0 +1,64 @@
|
||||
# Task 01 — `ls`: Teaching Refusals for Path-Like and Unknown-Source Scopes
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agents/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
A scoped `ls` whose `path` argument is a file/directory path (contains `/`) — or an
|
||||
unknown source name — gets a fixed-template refusal that states the correct contract
|
||||
instead of the terse "check the ls output", so the harness-prior misuse
|
||||
(`ls(path='app/rag/importer.py')`, `ls(path='.')` — the incident) self-corrects in one
|
||||
round. The `ls` tool description makes the same point at request time.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — two refusal template constants next to the existing refusal
|
||||
constants, plus the branch change:
|
||||
- `LS_PATH_NOT_A_SOURCE: str` — one `{path}` field, used when the **stripped** scope
|
||||
contains `/` (a source name can never contain `/` — source names are directory
|
||||
basenames, `app.rag.importer`):
|
||||
`"'{path}' looks like a document path, not a source name. The 'path' argument of ls filters by source name (e.g. 'homelab') — omit it to list every document, or read a document by its combined 'source/path' string."`
|
||||
- `NO_SOURCE_NOT_A_DIRECTORY: str` — one `{scope}` field, the existing no-source
|
||||
refusal with a teaching parenthetical appended (the prefix
|
||||
`"No source named '{scope}' — check the ls output."` stays byte-identical), used
|
||||
when the scope has no `/` and matches no registered source name:
|
||||
`"No source named '{scope}' — check the ls output. (The 'path' argument is a source name, not a directory — omit it to list every document.)"`
|
||||
- `_execute_tool` `ls` branch: non-empty scope with `"/" in scope` →
|
||||
`LS_PATH_NOT_A_SOURCE.format(path=scope)`; non-empty scope without `/` not in
|
||||
`list_source_names(db)` → `NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)`; a valid
|
||||
scope and the no-arg listing are unchanged. Both refusals count in nothing (no
|
||||
`holder.tool_calls` bump) and consume a round — exactly like today's refusal.
|
||||
- `AGENT_TOOLS` → `ls` → `function.parameters.properties.path.description`:
|
||||
`"Source name to list one source's documents (e.g. 'homelab') — a source name, not a file or directory path; omit to list every document."`
|
||||
- Module docstring (loop contract, point 3 — the refusal list): update the
|
||||
scoped-`ls` refusal entry to the two new lines.
|
||||
2. `tests/unit/test_agent.py` — unit pins (existing `ScriptedLLM` + monkeypatched
|
||||
`list_catalog` / `list_source_names` pattern; import the constants, never re-type
|
||||
them):
|
||||
- `ls(path='app/rag/importer.py')` (scope contains `/`) → the
|
||||
`LS_PATH_NOT_A_SOURCE` line with the argument echoed; `holder.tool_calls == 0`;
|
||||
tools stay offered on the next request.
|
||||
- `ls(path='.')` (no `/`, unknown) → the `NO_SOURCE_NOT_A_DIRECTORY` line with
|
||||
`'.'` echoed; `holder.tool_calls == 0`.
|
||||
- `ls(path='Ghost')` (no `/`, unknown) → the same extended line (replaces today's
|
||||
`test_ls_scoped_unknown_source_refused` pin).
|
||||
- Regression: `ls()` no-arg full catalog and `ls(path='<registered source>')` scoped
|
||||
listing (including the `0 documents:` registered-empty-source case) remain
|
||||
byte-identical to today.
|
||||
3. `tests/integration/test_agent_tools.py` — update the changed pin (the
|
||||
`ls(path='Ghost')` assertion) and add one case: a scoped `ls` with a `/`-containing
|
||||
`path` against the real DB (`kb` + `src` fixtures) → the document-path line, not
|
||||
counted, tools stay offered.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: as listed in Work 2–3 — every new refusal line pinned
|
||||
byte-for-byte; the count-in-nothing and tools-stay-offered invariants pinned; the
|
||||
unchanged paths regression-pinned.
|
||||
- Coverage: **>90%** on this task's new/modified code (the `ls` branch in
|
||||
`app/rag/agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov`
|
||||
green (DB up: `podman compose up -d db`)
|
||||
- [ ] The old terse string (no-source refusal without the parenthetical) appears nowhere
|
||||
in `app/` or `tests/`
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No behavior change to valid-scope / no-arg `ls` (regression pins green)
|
||||
@@ -0,0 +1,87 @@
|
||||
# Task 02 — `read` / `grep`: "did you mean 'source/path'?" Suggestions for Bare Document Paths
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agents/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
When `read` (or a scoped `grep`) receives an argument that resolves to no combined
|
||||
identity but *does* match an indexed document's `path` (exact or as a suffix), the
|
||||
refusal names the exact combined `source/path` identity to use — the harness prior
|
||||
(`read('app/rag/importer.py')`, missing the source prefix) self-corrects in one round.
|
||||
An argument that matches nothing keeps today's refusal byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py`:
|
||||
- `SUGGESTION_LIMIT = 3` — the cap on suggested identities per refusal.
|
||||
- Module-level `find_path_candidates(db: Session, arg: str) -> list[tuple[str, str, str]]`
|
||||
(so unit tests can monkeypatch it, house pattern): the indexed documents, in
|
||||
**catalog order** (the `all_documents` order), whose `path` equals `arg` or ends
|
||||
with `f"/{arg}"` (case-sensitive — these are file paths), as `(source, path, title)`
|
||||
triples. One bulk query via `all_documents`; called **only** from the refusal path
|
||||
below (never on the happy path) and **only** when `arg` contains `/` (a bare name
|
||||
keeps today's no-DB-lookup refusal — the existing
|
||||
`test_read_bare_source_name_refused_without_db` invariant stays green).
|
||||
- Refusal templates next to the existing constants:
|
||||
- `NO_DOCUMENT_DID_YOU_MEAN: str` —
|
||||
`"No document at '{arg}' — did you mean '{source}/{path}'?"`
|
||||
- `NO_DOCUMENT_DID_YOU_MEAN_MANY: str` —
|
||||
`"No document at '{arg}' — did you mean one of: {candidates}?"` where
|
||||
`{candidates}` is up to `SUGGESTION_LIMIT` combined `source/path` identities,
|
||||
each single-quoted, joined with `", "`, in catalog order.
|
||||
- `_execute_tool` `read` branch: after the in-context dedupe check and the
|
||||
`_resolve_path` miss — when `arg` contains `/`, run `find_path_candidates`:
|
||||
exactly 1 candidate → `NO_DOCUMENT_DID_YOU_MEAN`; 2+ →
|
||||
`NO_DOCUMENT_DID_YOU_MEAN_MANY`; 0 → today's
|
||||
`"No document at '{arg}' — check the ls output."` unchanged. `holder` untouched
|
||||
(a refusal counts in nothing; `read_docs` untouched — locator-only never changes).
|
||||
- `_execute_tool` `grep` branch: the same substitution for the scoped-`path` miss
|
||||
(the whole-KB grep is untouched).
|
||||
- `AGENT_TOOLS` → `read` → `path` description and `grep` → `path` description:
|
||||
append `" A bare document path (without the source name) will not resolve."` to
|
||||
each current text.
|
||||
- Module docstring (loop contract, point 3): document the suggestion behavior in the
|
||||
refusal list.
|
||||
2. `tests/unit/test_agent.py` — unit pins (monkeypatched `find_document` +
|
||||
`all_documents`; import the constants, never re-type them):
|
||||
- `read(path='active/container_caddy/caddy.md')` with the document indexed under
|
||||
`Homelab` (exact `path` match) → `did you mean 'Homelab/active/container_caddy/caddy.md'?`;
|
||||
`holder.read_docs` empty, `holder.tool_calls == 0`.
|
||||
- Suffix match: `read(path='caddy.md')` → the same single suggestion.
|
||||
- Two sources sharing the same `path` →
|
||||
`did you mean one of: 'A/x.md', 'B/x.md'?` in catalog order.
|
||||
- Four sources sharing the `path` → exactly 3 suggestions (the cap).
|
||||
- No match → today's refusal byte-identical; `read(path='Homelab')` (bare, no `/`) →
|
||||
today's refusal with **no** `find_document` / `all_documents` call (the `_boom`
|
||||
guard, existing pattern).
|
||||
- Dedupe precedence: a combined-form re-read of a `seed_docs` document →
|
||||
`ALREADY_IN_CONTEXT` (unchanged); a bare-`path` read of an in-context document
|
||||
(`read('app/rag/importer.py')` with `sample/app/rag/importer.py` seeded) → the
|
||||
suggestion line (the split pair is not in `known`, so the model learns the
|
||||
combined identity — its next, correctly-formed call is then deduped).
|
||||
- Scoped `grep` miss with a candidate → the suggestion line; scoped `grep` miss
|
||||
without → today's line; whole-KB `grep` unchanged (A5 match/output contract:
|
||||
fixed substring, case-insensitive, 20 matches, 200-char lines).
|
||||
- Happy paths regression-pinned: combined-form `read` (full content,
|
||||
`read_docs` appended), valid scoped `grep` result.
|
||||
3. `tests/integration/test_agent_tools.py` — update the changed pins (inspect each
|
||||
existing `"No document at …"` assertion against the fixture documents; only the
|
||||
lines whose argument matches a fixture document `path` change to the suggestion form),
|
||||
plus two new end-to-end cases through `run_agent` against real Postgres: a bare path
|
||||
under one source (single suggestion) and the same `path` under two sources (the
|
||||
"one of" line) — in both, the refusal is followed by the model's corrected call
|
||||
succeeding (scripted `ToolCallPiece` round 2 with the suggested combined identity).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: as listed in Work 2–3 — every new template pinned byte-for-byte;
|
||||
the catalog-order + cap invariant pinned; the zero-candidate and no-DB-lookup
|
||||
invariants pinned; the A5 `grep` contract regression-pinned.
|
||||
- Coverage: **>90%** on this task's new/modified code (`find_path_candidates` + both
|
||||
`_execute_tool` branches).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov`
|
||||
green (DB up: `podman compose up -d db`)
|
||||
- [ ] `find_path_candidates` is module-level (monkeypatchable) and issues at most one
|
||||
bulk query
|
||||
- [ ] The zero-candidate refusal and the bare-name (no-DB-lookup) refusal are
|
||||
byte-identical to today
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 03 — `TOOLS_SECTION` Copy: State the Contract Up Front
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agents/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The HIGH prompt's `<tools>` section says the same two things the new refusals teach —
|
||||
the `ls` `path` is a *source name*, not a directory or file path, and `read`/`grep`
|
||||
need the combined `source/path` string *including the source name* — so the model
|
||||
carries the contract before it calls a tool, not only after being refused.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `TOOLS_SECTION` rewritten (the E2E mock keys off the
|
||||
`<tools>` marker's *presence*, not this wording, so the change is mock-safe):
|
||||
- `ls` clause: its optional `path` argument is a *source name* (e.g. `'homelab'`)
|
||||
— **not** a directory or file path; omit it to list every document.
|
||||
- `read` clause: the combined `source/path` string, exactly as shown in the `ls`
|
||||
output — *including the source name*; a bare document path will not resolve.
|
||||
- `grep` clause: the locator copy stays (its `path` is already described as a
|
||||
combined `source/path` string); add the same bare-path-will-not-resolve note.
|
||||
- Keep the section's shape: a single paragraph between `<tools>` and `</tools>`,
|
||||
still appended after the mode body in the HIGH prompt only (the LOW/deflection
|
||||
prompt never carries it — phase 71's plain-text line stays put).
|
||||
2. `tests/unit/test_prompts.py` — update the `TOOLS_SECTION` wording pin(s) where they
|
||||
pin the old wording; the `<tools>`-marker-present-in-HIGH pin, the
|
||||
marker-absent-from-LOW pin, and the byte-identical-LOW-prompt pin stay green as-is.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_prompts.py` — marker present in the HIGH prompt and absent
|
||||
from the LOW prompt; the LOW prompt byte-identical to today; the new wording pinned
|
||||
for the `ls` source-name clause and the read combined-identity clause.
|
||||
- Coverage: **>90%** on this task's modified code (the constant itself — the builders
|
||||
are already covered).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py -v --no-cov` green
|
||||
- [ ] The HIGH prompt still ends with the `<tools>` section (existing section-order pin
|
||||
green); the LOW/deflection prompt is byte-identical to today
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,63 @@
|
||||
# Task 04 — Mock Flow, Dedicated E2E Suite
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agents/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Prove the self-correction loop deterministically through the real UI: a mock-LLM flow
|
||||
that reproduces the incident's `ls(path='.')` misuse, receives the teaching refusal,
|
||||
corrects to a no-arg `ls()`, and answers from the catalog — a dedicated Playwright
|
||||
suite pinning the two-round shape on the SSE wire. (The live real-model acceptance
|
||||
gate is task 05 — this task is the deterministic half of the proof.)
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — one new deterministic flow, checked in the flow table
|
||||
**before** the plain `TOOLS_TRIGGER` flow (the trigger phrases are disjoint
|
||||
substrings; the ordering rule follows the phase-71 convention):
|
||||
- `LS_TEACH_TRIGGER = "list the files in this directory"` — **and** the system
|
||||
prompt carries the `<tools>` section (grounded turn):
|
||||
* request 1 (tools offered, no `tool`-role result in the messages yet): stream
|
||||
ONLY `tool_calls` deltas — `ls` with `{"path": "."}` (synthetic id `call_0`),
|
||||
`finish_reason: "tool_calls"`, no content (the incident's misuse,
|
||||
deterministic);
|
||||
* request 2 (a `tool`-role result present that is **not** a catalog listing —
|
||||
i.e. the teaching refusal): stream a `tool_calls` delta — `ls` with no
|
||||
arguments (id `call_1`);
|
||||
* request 3 (a `tool`-role result whose first line matches the
|
||||
`^\d+ documents:` catalog header): a deterministic content answer —
|
||||
`These are the indexed documents: <first catalog line>` (the
|
||||
`source: X | path: Y | title: Z` line, parsed with the existing
|
||||
`_CATALOG_LINE_RE` machinery), `finish_reason: "stop"`.
|
||||
- Update the module docstring's flow table with the phase-72 note.
|
||||
2. `tests/e2e/test_tool_path_teaching.py` (NEW — the phase's dedicated suite, house
|
||||
pattern, run in isolation; DB up, mock LLM):
|
||||
- Import a small fixture document set (house fixture pattern: one source, two
|
||||
documents with known `source`/`path`/`title`) and ask a question containing
|
||||
`LS_TEACH_TRIGGER`.
|
||||
- **Self-correction** — the turn settles (composer re-enables, `done` observed);
|
||||
the answer bubble contains the first document's `source:` and `path:` fields
|
||||
(the catalog reached the model and landed in the answer); no error banner.
|
||||
- **Two rounds on the wire** (the house SSE-capture pattern): the `tool` frames
|
||||
arrive in order — first `name:"ls"` with `argument:"."`, then `name:"ls"` with
|
||||
`argument:null` — and there is **no** third `tool` frame (the loop ended in one
|
||||
correction, not at the round cap).
|
||||
- **No regression to the plain flow** — a follow-up question containing
|
||||
`TOOLS_TRIGGER` (the single-read flow) in the same session still settles with
|
||||
the read flow's answer (the new flow did not swallow the existing trigger).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the dedicated suite proves the loop shape (misuse → teaching refusal →
|
||||
corrected call → answer) through the real UI and the SSE wire; the existing
|
||||
regression E2E suites (mock-driven) stay green — run them as the regression check
|
||||
for this task.
|
||||
- Coverage: unit/integration coverage of `app/` stays >90% (this task adds test-only
|
||||
code; `uv run pytest --cov=app --cov-report=term-missing` as the check).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation (DB up: `podman compose up -d db`, mock LLM)
|
||||
- [ ] Regression E2E suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`, `test_chat_rag.py`
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No commit in this task (the commit happens in task 05, after the real-model
|
||||
gate passes)
|
||||
@@ -0,0 +1,114 @@
|
||||
# Task 05 — Real-Model Acceptance Gate (live `lite`), Full Gates, Commit
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agents/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The phase's pass condition (owner directive, 2026-09-03: "test with the real lite
|
||||
model until tool calls work consistently — don't pass until a sufficient number of
|
||||
tool calls succeed"): a live script drives the fixed 10-question battery through the
|
||||
**real** grounded path (real endpoint, configured chat model — `lite` per `.env`,
|
||||
real Postgres KB) and the phase commits only when the gate PASSES. Until it does,
|
||||
iterate the copy levers this phase owns (refusal templates, `AGENT_TOOLS`
|
||||
descriptions, `TOOLS_SECTION`) — never the gate.
|
||||
|
||||
## Work
|
||||
1. `scripts/agent_realmodel_check.py` (NEW — house probe pattern, `scripts/llm_probe.py`
|
||||
as the model: `uv run python -m scripts.agent_realmodel_check`, argparse, dotenv,
|
||||
plain module, no debugpy):
|
||||
- **Preconditions (exit 2 with an actionable line on failure):** DB reachable;
|
||||
the catalog holds ≥2 documents; the FIRST TWO catalog documents' `path`s each
|
||||
contain `/` (the bare-path traps need nested paths); `settings.agent_max_rounds
|
||||
> 0` (the gate needs tools enabled).
|
||||
- **Mirror the grounded path of `app/api/chat.py` exactly** (same prompt the UI
|
||||
gets): per question — embed it, `retrieve`, `select_documents`, steering notes
|
||||
+ KB overview as chat.py reads them, `build_high_prompt(docs, notes, kb_overview)`,
|
||||
then `run_agent(llm, db, system_prompt=…, user_message=…, seed_docs=docs,
|
||||
settings=settings, holder=AgentHolder())` with a **fresh** `AgentHolder` per
|
||||
turn, consuming every piece to the end. Never modify the KB.
|
||||
- **Fixed question battery** (locked — the executor may not swap in easier
|
||||
questions). Let the first two catalog documents be
|
||||
`D1 = (s1, p1, t1)` and `D2 = (s2, p2, t2)`, and `token` = the first
|
||||
whitespace-split word of `D2.content` with length ≥ 6 (strip leading/trailing
|
||||
non-alphanumerics, lowercase; fallback: the first word of `t2`):
|
||||
1. `List the files in this directory.` (the incident)
|
||||
2. `List the documents you have in the {s1} source.`
|
||||
3. `List every document you have indexed.`
|
||||
4. `What does the document {p1} contain? Open it and tell me.` (bare-path `read` trap)
|
||||
5. `Read {s1}/{p1} and summarize it.` (combined form — the correct shape)
|
||||
6. `Open the document {p2} and tell me what it covers.` (bare-path `read` trap)
|
||||
7. `Find the exact string "{token}" in your documents and tell me which ones contain it.` (`grep`)
|
||||
8. `Which document has the title "{t2}"? Read it and summarize.`
|
||||
9. `What do you know about {t1}? Open the relevant document and give me specifics.`
|
||||
10. `List the files in the {s2} directory.` (source name phrased as a directory)
|
||||
- **Per-turn measurement** (from the consumed stream + the holder — no app-code
|
||||
changes for measurement): `emitted` = count of yielded `ToolCallPiece`s;
|
||||
`executed` = `holder.tool_calls` (refusals count in nothing); `rejected` =
|
||||
`emitted − executed`; `cap_reached` = `emitted >= settings.agent_max_rounds`
|
||||
(every capped round emitted a call, so the cap implies at that many emissions
|
||||
and never the reverse); `answered` = the stream finished without
|
||||
`LLMError`/`MalformedReplyError`. Print one line per turn:
|
||||
`turn 04 | emitted=2 executed=1 cap=no | What does the document …`.
|
||||
- **Verdict + pass conditions (locked):**
|
||||
1. all 10 turns `answered`;
|
||||
2. zero `cap_reached` turns (the incident's loop signature — hitting the cap
|
||||
means the teaching did not end the loop);
|
||||
3. ≥6 of 10 turns with `emitted ≥ 1` (the model keeps USING tools — it does not
|
||||
abandon them and answer from seed context alone, the incident's end state);
|
||||
4. `executed / emitted ≥ 0.90` across the whole run (the "sufficient number of
|
||||
tool calls succeed" bar; a run with zero emitted calls fails condition 3
|
||||
anyway).
|
||||
Print the single verdict line in a stable format, e.g.
|
||||
`gate: lite PASS turns=10 answered=10 caps=0 tool-turns=8 calls 21/23 executed (91%) 2026-09-03`
|
||||
(model = `settings.llm_chat_model`, date = run date). **Exit 0 on PASS, 1 on
|
||||
FAIL, 2 on precondition failure.**
|
||||
- On FAIL, also print a short per-condition breakdown (which condition(s) missed)
|
||||
so the iteration loop can target the right lever. For refusal diagnosis, each
|
||||
call is already logged by `run_agent` (`agent tool=… args=… round=…/…`) —
|
||||
correlate the logged arguments with the refusal templates in
|
||||
`app/rag/agent.py` to see which teaching line the model hit.
|
||||
2. **Run the gate and iterate until it PASSES** (the loop this task exists for):
|
||||
`podman compose up -d db` → `uv run python -m scripts.agent_realmodel_check`.
|
||||
On FAIL: change ONLY the copy levers this phase owns — the refusal templates
|
||||
(task 01/02 constants), the `AGENT_TOOLS` `path` descriptions (task 01/02),
|
||||
`TOOLS_SECTION` (task 03) — with their unit pins updated to follow the constants;
|
||||
`uv run pytest` green again; re-run the gate. Repeat. **Forbidden:** lowering any
|
||||
threshold, swapping questions, disabling a tool, or weakening condition 4 to make
|
||||
it pass. If the gate still fails after a genuine iteration (the numbers stop
|
||||
improving across levers), STOP: no commit — report the per-turn lines, the
|
||||
verdict, and which refusals the model hit (from the `app.agents` log) in the task
|
||||
report for the owner (fail-loud house style).
|
||||
3. **Record the verdict** (house precedent — the phase-37 probe verdict lives in the
|
||||
`app/rag/agent.py` module docstring): append one line to that docstring —
|
||||
`Real-model gate (phase 72, task 05 — live vs the configured chat model):
|
||||
<the verdict line, verbatim>`.
|
||||
4. **Full gates + commit:**
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_path_teaching.py`, then the regression
|
||||
suites `test_harness_aligned_tools.py`, `test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`.
|
||||
- Move the phase directory: `mv .agents/phases/todo/72_teaching_refusals
|
||||
.agents/phases/complete/`.
|
||||
- Commit — one, `--no-gpg-sign`, the Commit block of `00_phase.md`: the title
|
||||
message plus a **body line carrying the gate verdict verbatim**. The commit
|
||||
also carries the still-uncommitted phase-71 `todo/` → `complete/` move and
|
||||
`.agents/reports/71_scaffolding_guardrails/` (`.agents/` is tracked — AGENTS.md
|
||||
§8; `git add -A .agents/ app/ tests/ scripts/` picks up everything).
|
||||
|
||||
## Testing & Quality
|
||||
- The script IS the test for this task: it is deterministic in its question set,
|
||||
thresholds, and output format (a future executor re-running it gets comparable
|
||||
numbers); its precondition failures exit 2 with actionable text. The script itself
|
||||
needs no unit tests (it is an entrypoint probe, `scripts/llm_probe.py` precedent),
|
||||
but the copy iterations it drives must keep `uv run pytest` + coverage >90% green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run python -m scripts.agent_realmodel_check` exits **0** against the live
|
||||
endpoint (all four pass conditions met with the configured model — `lite`);
|
||||
the verdict line verbatim in `app/rag/agent.py`'s module docstring
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation; the regression E2E suites green in isolation (the Work-4 list)
|
||||
- [ ] One `--no-gpg-sign` commit whose body carries the gate verdict; the phase
|
||||
directory under `.agents/phases/complete/72_teaching_refusals/`
|
||||
Reference in New Issue
Block a user