Compare commits
4
Commits
16f1cfbcaf
...
988ff78526
| Author | SHA1 | Date | |
|---|---|---|---|
 ducoterra
|
988ff78526 | ||
|
ducoterra
|
7909bdb8da | ||
|
ducoterra
|
575d6c88d0 | ||
|
ducoterra
|
801639efcc |
@@ -0,0 +1,143 @@
|
||||
# Phase 70 — Harness-Aligned Agent Tools: `ls` / `read(path)` / `grep(pattern, path?)`
|
||||
|
||||
**Source:** owner request (chat, 2026-09-03) — live incident: the question "What are the
|
||||
correct llama.cpp arguments for Qwen 3.8?" was answered with the raw model text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>`
|
||||
(`query_log`: top_score=0.040, fts_hits=0, deflected=true — the turn was DEFLECTED, so
|
||||
no tools were even offered). Diagnosis: the chat model (`lite` per `.env`; owner keeps
|
||||
`lite` for everything — it's faster, owner decision 2026-09-03) reaches for the tool
|
||||
shapes it was trained on — a `read` tool taking a **single** `path` argument — and our
|
||||
`read_document(source, path)` fights that prior (the `_resolve_document` combined-form
|
||||
self-correction and "teach the split" refusals are the scar tissue). Owner direction:
|
||||
"match existing harnesses as much as possible" — the mapping below mirrors the
|
||||
**pi.dev** tool surface (`dist/core/tools/`: `read{path}`, `ls{path?}`,
|
||||
`grep{pattern, path?}`).
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md` (this phase reshapes the tools
|
||||
that story delivered; phase 68's search contract rides along renamed)
|
||||
**Context:**
|
||||
- `app/rag/agent.py` — `AGENT_TOOLS` (OpenAI function definitions:
|
||||
`list_documents` / `read_document(source, path)` / `search_documents(pattern,
|
||||
source?, path?)`), `_execute_tool` (execution + refusals: `ALREADY_IN_CONTEXT`,
|
||||
`UNKNOWN_TOOL`, `MISSING_READ_ARGS`, `MISSING_SEARCH_ARGS`; combined-form
|
||||
self-correction in `_resolve_document`), `AgentHolder`, `run_agent` (round cap
|
||||
`BOR_AGENT_MAX_ROUNDS`, default 10; `0` = no-tools kill switch).
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only, ends the prompt; the E2E
|
||||
mock keys off the `<tools>` marker's *presence*, not the wording).
|
||||
- `app/api/chat.py` — the `ToolCallPiece` → SSE `{"type":"tool","name":…,"argument":…}`
|
||||
derivation (read → `source/path`, search → pattern, list → null), `done.sources`,
|
||||
per-turn log line (`tool_calls=N`).
|
||||
- `tests/e2e/mock_llm.py` — the deterministic mock flows keyed on `TOOLS_TRIGGER`
|
||||
("use your tools"), `MULTI_READ_TRIGGER`, `SEARCH_TRIGGER` + the `<tools>` marker;
|
||||
the flows emit `tool_calls` deltas with the current names/args and parse the
|
||||
`source: X | path: Y | title: Z` catalog lines.
|
||||
- `frontend/assets/app.js` — `.tool-call` lines (one row per `tool` frame, generic
|
||||
render of `name` + `argument`), docstring name references; `frontend/assets/shared.js`
|
||||
L159 special-cases `t.name === "read_document"` (shared-chat view).
|
||||
- `tests/` — unit: `test_agent.py`, `test_prompts.py`, `test_chat_gate.py`,
|
||||
`test_mock_tool_flow.py`, `test_sse_events.py`; integration: `test_agent_tools.py`,
|
||||
`test_chat_api.py`, `test_api.py`, `test_chats_api.py`; E2E:
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`. `README.md` L173–176 documents the current tool names/args.
|
||||
|
||||
## Objective
|
||||
Rename and reshape the three server-side agent tools to the harness-trained surface —
|
||||
**`ls`**, **`read(path)`** (single combined `source/path` argument), **`grep(pattern,
|
||||
path?)`** — so the model's trained priors emit valid calls instead of fighting the
|
||||
schema. Capabilities and all owner-locked semantics (full-document reads, fixed-
|
||||
substring locator searches, round cap, kill switch, SSE contract shape) are unchanged;
|
||||
only the tool surface, its descriptions, and the ripples (prompt copy, SSE names,
|
||||
mock, tests, docs) change.
|
||||
|
||||
## Dependencies
|
||||
- `68_search_tool` (complete) — the search semantics this phase renames (A5 locked
|
||||
match/output contract preserved).
|
||||
- `45_agent_unlimited_tools` (complete) — the round-cap design the loop keeps.
|
||||
- No todo dependencies; runs on the current head. Phase `71_scaffolding_guardrails`
|
||||
(todo) follows this one (its mock changes build on the new tool names).
|
||||
|
||||
## Tasks
|
||||
1. `01_tool_schemas.md` — `app/rag/agent.py`: `AGENT_TOOLS` → `ls` / `read` / `grep`
|
||||
schemas + descriptions; `_execute_tool` argument handling (single combined `path`),
|
||||
updated refusals; unit tests.
|
||||
2. `02_prompt_section.md` — `app/rag/prompts.py`: `TOOLS_SECTION` rewritten for
|
||||
`ls`/`read`/`grep`; prompt unit pins.
|
||||
3. `03_api_sse.md` — `app/api/chat.py`: SSE `tool`-frame `argument` derivation for the
|
||||
new shapes + module/docstring contract copy; integration tests.
|
||||
4. `04_frontend.md` — `app.js` + `shared.js` name references and the
|
||||
`read_document` special case; frontend unit pins; persisted-chat note.
|
||||
5. `05_mock_e2e_commit.md` — `mock_llm.py` flows on the new names/args, the three
|
||||
existing E2E suites updated, the new dedicated E2E suite, README, full gates,
|
||||
commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` — the new schemas (names, required/optional args,
|
||||
descriptions), `read` path resolution (combined form split at the FIRST slash, bare
|
||||
source name refusal, already-in-context, full content), `ls` scoping (no-arg full
|
||||
catalog, `path`=source scope, unknown-source refusal, empty-source "0 documents"),
|
||||
`grep` (required `pattern`, optional `path` single-doc scope, the locked A5 output
|
||||
contract: fixed substring, case-insensitive, 20 matches, 200-char lines, locator
|
||||
only), the round cap + kill switch (`agent_max_rounds=0` → `tools=None`,
|
||||
byte-identical request) unchanged.
|
||||
- Unit: `tests/unit/test_prompts.py` / `test_chat_gate.py` — `TOOLS_SECTION` copy
|
||||
(marker present, HIGH-only), byte-identical LOW prompt (this phase does not touch
|
||||
the deflection path).
|
||||
- Integration: `tests/integration/test_agent_tools.py`, `test_chat_api.py` — SSE
|
||||
`tool` frames with the new `name`/`argument` values end-to-end (mock LLM),
|
||||
`done.sources` / `query_log.sources` / `tool_calls=N` log line unchanged in meaning.
|
||||
- E2E (mandatory, house rule): NEW dedicated suite
|
||||
`tests/e2e/test_harness_aligned_tools.py`, run in isolation — the `ls` → `read`
|
||||
(combined path) → answer flow and the `grep` → `read` flow over the real UI; plus
|
||||
the three existing suites (`test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`) updated to the new names and
|
||||
green in isolation.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `AGENT_TOOLS` defines exactly `ls` (optional `path`), `read` (required `path`),
|
||||
`grep` (required `pattern`, optional `path`); the old names exist nowhere in
|
||||
`app/` (`rg "list_documents|read_document|search_documents" app/` → no matches).
|
||||
- [ ] `read` accepts the combined `source/path` form (the model's natural shape),
|
||||
splits at the first slash, appends the **full** document (A7-revised: never
|
||||
truncated); `grep` keeps the locked A5 match/output contract and is a locator
|
||||
only; `ls` prints the phase-63 `source: X | path: Y | title: Z` lines.
|
||||
- [ ] SSE `tool` frames carry the new `name` values; `argument` is the single string
|
||||
the model passed (`read`'s `path`, `grep`'s `pattern`, `ls`'s `path`) or null.
|
||||
- [ ] `BOR_AGENT_MAX_ROUNDS=0` still disables the tools entirely (request
|
||||
byte-identical to the no-tools path); the deflected path is byte-identical
|
||||
(LOW prompt and `tools=None` untouched by this phase).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov` green in
|
||||
isolation; regression suites green in isolation: `test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`.
|
||||
- [ ] README tool documentation updated to the new surface.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
|
||||
`.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner (chat, 2026-09-03):** the tool surface is remapped to the pi.dev harness
|
||||
shape — `ls` / `read(path)` / `grep(pattern, path?)` — "to match existing harnesses
|
||||
as much as possible"; `lite` stays the chat model for everything (no model swap);
|
||||
the chat model is NOT flipped to `turbo`. This supersedes the phase-37 tool
|
||||
names/args and the phase-68 tool *name* (`search_documents` → `grep`); the phase-68
|
||||
match/output contract (fixed substring, case-insensitive, 20×200, locator-only) is
|
||||
preserved verbatim.
|
||||
- **`read` is `path`-only.** No `offset`/`limit` (pi has them, but the A7-revised
|
||||
contract is "never truncated" — implementing paging would violate it; `path`-only
|
||||
still carries the trained shape, which is the point).
|
||||
- **The combined `source/path` string is the canonical document identity** in every
|
||||
tool argument, refusal, and result header (it already is in search result lines and
|
||||
`done.sources`). The old two-argument split and its self-correction/teaching
|
||||
refusals are deleted — the model's combined form is now *correct*, not a mistake to
|
||||
fix.
|
||||
- **SSE contract shape unchanged** (A15 extension honoured): `{"type":"tool",
|
||||
"name":…,"argument":…}` — only the `name` values and the `argument` derivation
|
||||
change. `delta`/`thinking`/`retry`/`done`/`error` frames are untouched.
|
||||
- **No env changes, no schema change.** `BOR_AGENT_MAX_ROUNDS` keeps its meaning
|
||||
(round cap; `0` = no-tools kill switch). Saved chats persisting old tool names
|
||||
render fine (the UI renders whatever `name`/`argument` arrive — no migration).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)"
|
||||
```
|
||||
@@ -0,0 +1,96 @@
|
||||
# Task 01 — Tool Schemas + Execution: `ls` / `read(path)` / `grep(pattern, path?)`
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Reshape `AGENT_TOOLS` and `_execute_tool` in `app/rag/agent.py` to the pi.dev harness
|
||||
surface — `ls` (optional `path`), `read` (required `path`), `grep` (required
|
||||
`pattern`, optional `path`) — with the combined `source/path` form as the canonical
|
||||
document identity, keeping every locked capability intact.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `AGENT_TOOLS` (three OpenAI function definitions; keep the
|
||||
module docstring's loop-contract points accurate, update the names/args references):
|
||||
- **`ls`** — parameters: optional `path` (string): "Source name to list one
|
||||
source's documents (e.g. 'homelab'); omit to list every document." Description:
|
||||
"List the indexed documents as `source: X | path: Y | title: Z` lines."
|
||||
- **`read`** — parameters: required `path` (string): "The document to add to your
|
||||
context, as the combined `source/path` string exactly as shown in the `ls`
|
||||
output (e.g. 'homelab/active/container_caddy/caddy.md')." Description: "Add the
|
||||
full content of one indexed document to your context."
|
||||
- **`grep`** — parameters: required `pattern` (string): "The exact text to search
|
||||
for (a plain substring, not a regex)"; optional `path` (string): "Limit the
|
||||
search to one document, as a combined `source/path` string from the `ls` output
|
||||
(omit to search every document)." Description: "Search the indexed documents
|
||||
for an exact string (case-insensitive) and return up to 20 matching lines as
|
||||
`source/path:line: text` — a locator, not a context-adder: read the winner with
|
||||
`read`."
|
||||
2. `app/rag/agent.py` — `_execute_tool` (branch on the new names; `AgentHolder`
|
||||
semantics, round counting, and the DB accessors are unchanged):
|
||||
- `ls`: no `path` → the full `list_catalog` listing (`"N documents:\n"` + the
|
||||
phase-63 lines, unchanged format). With `path`: strip it; if it equals no
|
||||
source name → refusal `No source named '{path}' — check the ls output.` (a
|
||||
rejected call, counts in nothing); else the same listing filtered to that
|
||||
source (`0 documents:` is a valid, counted result).
|
||||
- `read`: require a non-blank string `path` (else
|
||||
`read requires a string argument 'path'.`). Split at the FIRST `/` →
|
||||
`(source, path)` (source names can never contain `/` — importer contract).
|
||||
No `/` in the argument → refusal `No document at '{arg}' — check the ls
|
||||
output.` Already-in-context check (seed + holder) via the resolved pair →
|
||||
`ALREADY_IN_CONTEXT`. Unknown pair → the same `No document at '{arg}' — check
|
||||
the ls output.` refusal (echo the argument as passed, so the model sees its
|
||||
own form). Success → `holder.read_docs.append(doc)`, `holder.tool_calls += 1`,
|
||||
result `f"Document {doc.source}/{doc.path}:\n{doc.content}"` (full content —
|
||||
A7-revised, never truncated).
|
||||
- `grep`: require a non-blank string `pattern` (else
|
||||
`grep requires a string argument 'pattern'`). Optional `path`: if present it
|
||||
must resolve to exactly one document (same first-slash split; unknown →
|
||||
`No document at '{arg}' — check the ls output.`); absent → `all_documents`.
|
||||
Everything downstream of target selection is the **phase-68 contract verbatim**:
|
||||
case-insensitive fixed substring via `grep_document`, catalog order, global cap
|
||||
`SEARCH_MAX_MATCHES=20`, per-line `SEARCH_LINE_LIMIT=200`, no-match lines
|
||||
(`NO_MATCHES` / the scoped variant keyed on the resolved `source/path`),
|
||||
counted as a successful call, `holder.read_docs` untouched (locator only).
|
||||
- `_resolve_document`: replace with the single-input resolver the three tools
|
||||
share — `_resolve_path(db, combined: str) -> tuple[Document | None, str, str]`
|
||||
(split at the first slash, exact pair lookup; return the split pair so refusals
|
||||
can echo/teach). Delete the old two-argument self-correction + "teach the split"
|
||||
refusal branches (the combined form is now the correct input).
|
||||
- Update the module-level refusal constants: `MISSING_READ_ARGS = "read requires
|
||||
a string argument 'path'."`, `MISSING_SEARCH_ARGS = "grep requires a string
|
||||
argument 'pattern'."`; `ALREADY_IN_CONTEXT` / `UNKNOWN_TOOL` unchanged.
|
||||
- Update the module docstring: point 2 (the three tools by new name/shape),
|
||||
point 3 (refusals — no more split-teaching; unknown source for `ls`), and the
|
||||
phase-70 note (owner permission 2026-09-03: harness-aligned surface; phase-68
|
||||
A5 match/output contract preserved; the combined form is canonical).
|
||||
3. `tests/unit/test_agent.py` — update every existing test to the new names/args and
|
||||
add:
|
||||
- schema pins: exactly `ls`/`read`/`grep`; required/optional arg sets; old names
|
||||
absent from `AGENT_TOOLS`.
|
||||
- `read`: combined form resolves + full content returned; bare source name
|
||||
(`read(path='homelab')`) → the no-document refusal; missing/blank/
|
||||
non-string `path` → the missing-args refusal; already-in-context (seed and
|
||||
previously-read) → `ALREADY_IN_CONTEXT`; re-read counts nothing.
|
||||
- `ls`: no-arg full catalog (format pinned: `source: X | path: Y | title: Z`);
|
||||
scoped known source (incl. a source with 0 docs → `0 documents:` counted);
|
||||
unknown source → refusal, not counted.
|
||||
- `grep`: whole-KB vs `path`-scoped; the A5 pins unchanged (fixed substring,
|
||||
case-insensitive, 20-cap in catalog order, 200-char truncation, no-match lines
|
||||
counted, read_docs untouched); unknown `path` target → refusal.
|
||||
- loop mechanics unchanged: round cap forcing the final no-tools answer,
|
||||
`agent_max_rounds=0` → exactly one request with `tools=None`, rejected calls
|
||||
consume a round but count nothing in `holder.tool_calls`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` (above) — the DB accessors stay monkeypatched
|
||||
module-level functions (no database in unit scope).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "list_documents|read_document|search_documents" app/rag/agent.py` → no
|
||||
matches; `AGENT_TOOLS` names are exactly `ls`, `read`, `grep`.
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py -v --no-cov` green (old + new pins).
|
||||
- [ ] `uv run pytest --cov=app -k agent` green; `uv run ruff check . && uv run
|
||||
pyright` clean.
|
||||
- [ ] No behavior change to the deflected path or the kill switch (pinned by the
|
||||
updated suite).
|
||||
@@ -0,0 +1,42 @@
|
||||
# Task 02 — `<tools>` Prompt Section for the New Surface
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Rewrite `TOOLS_SECTION` in `app/rag/prompts.py` so the HIGH prompt teaches the new
|
||||
`ls`/`read`/`grep` shapes in harness language — the last thing the model reads on a
|
||||
grounded turn — while the E2E mock's `<tools>`-marker keying and the LOW prompt stay
|
||||
untouched.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — replace the `TOOLS_SECTION` constant (keep the name, keep
|
||||
it HIGH-only, keep it appended after the `<documents>` body, keep the `<tools>` /
|
||||
`</tools>` markers the mock keys on):
|
||||
- New copy (harness-aligned, names/args exactly as task 01): instruct that the
|
||||
context may be extended with three tools — `ls` to list the indexed documents
|
||||
(`source: X | path: Y | title: Z` lines), `grep` to locate an exact string
|
||||
(case-insensitive; a locator, not a context-adder), and `read` to pull in one
|
||||
document by its combined `source/path` (full content) — and that the model
|
||||
should answer as soon as it has what it needs (the round cap is the bound; the
|
||||
prompt does not re-state budgets — phase 45).
|
||||
- Keep the module docstring's statement that the LOW/deflection prompt never
|
||||
carries the section (byte-identical LOW path — this task must not change
|
||||
`build_deflect_prompt` or `build_high_prompt`'s section order).
|
||||
2. `tests/unit/test_prompts.py` (and `test_chat_gate.py` where the section is pinned)
|
||||
— update the `TOOLS_SECTION` copy pins: markers present; the new tool names
|
||||
present and the old names (`list_documents`, `read_document`,
|
||||
`search_documents`) absent from the HIGH prompt; HIGH/LOW section order and the
|
||||
byte-identical-when-empty steering/overview behavior unchanged; the LOW prompt is
|
||||
byte-identical to the pre-phase text (no `<tools>`, no new copy).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_prompts.py` / `test_chat_gate.py` pins above.
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/prompts.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v
|
||||
--no-cov` green.
|
||||
- [ ] HIGH prompt still ends with the `<tools>` section (mock keying intact);
|
||||
`rg "list_documents|read_document|search_documents" app/rag/prompts.py` → no
|
||||
matches.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Task 03 — SSE `tool` Frames + API Contract Copy
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Point the `ToolCallPiece` → SSE `tool`-frame derivation in `app/api/chat.py` at the
|
||||
new tool shapes (new `name` values, `argument` = the single string the model passed)
|
||||
and update the module's contract docstrings; `delta`/`thinking`/`retry`/`done`/
|
||||
`error` frames and the `done.sources`/`query_log` semantics stay exactly as they are.
|
||||
|
||||
## Work
|
||||
1. `app/api/chat.py` — the `ToolCallPiece` branch of the piece loop:
|
||||
- `read` → `argument = piece.arguments.get("path")` (the combined `source/path`
|
||||
as the model passed it; non-string → null).
|
||||
- `grep` → `argument = piece.arguments.get("pattern")` (non-string → null, the
|
||||
existing refusal case).
|
||||
- `ls` → `argument = piece.arguments.get("path")` (the scope, if the model gave
|
||||
one; otherwise null).
|
||||
- One-line rule in the code comment: **`argument` is the single string argument
|
||||
the model passed, or null** — symmetric across the three tools.
|
||||
- `ChatToolEvent` schema (`app/schemas.py`) needs no field change (name/argument
|
||||
are already `str`/`str | None`) — update its docstring's name examples only.
|
||||
2. `app/api/chat.py` module docstring — the "Agent document tools" paragraph: new
|
||||
tool names/shapes, the `argument` rule above, phase-70 note (owner permission
|
||||
2026-09-03); keep the SSE shape statement (`{"type":"tool","name":…,"argument":…}`)
|
||||
and the `done.sources`/`query_log`/`tool_calls=N` contract wording accurate.
|
||||
3. `tests/integration/test_agent_tools.py`, `test_chat_api.py` (+ `test_api.py` /
|
||||
`test_chats_api.py` where tool frames or names are pinned) — update to the new
|
||||
`name` values and the `argument` rule; add pins: a `read` frame carries the
|
||||
combined path as passed; a `grep` frame carries the pattern; an `ls` frame is
|
||||
null-unscoped / the scope when scoped; a rejected call (unknown `read` path)
|
||||
still emits its frame with the model's argument (frame emission is
|
||||
execution-independent — the existing behavior, now pinned).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: the updated suites above (mock LLM emitting the new tool names —
|
||||
`tests/e2e/mock_llm.py` is updated in task 05; until then these integration tests
|
||||
build their own scripted pieces, so they do not depend on the mock).
|
||||
- Coverage: **>90%** on this task's modified code (`app/api/chat.py`,
|
||||
`app/schemas.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/integration/test_agent_tools.py tests/integration/test_chat_api.py -v
|
||||
--no-cov` green.
|
||||
- [ ] `done.sources`, `query_log.sources`, and the per-turn `tool_calls=N` field
|
||||
report exactly what they did before for the same executed calls (pins green).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Task 04 — Frontend Tool-Line Rendering for the New Names
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Point the tool-line rendering at the new `ls`/`read`/`grep` names while keeping
|
||||
**legacy saved chats** (persisted with the old `list_documents` / `read_document` /
|
||||
`search_documents` names, phase 14 persistence) rendering exactly as before — no
|
||||
migration.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — `appendToolLine(wrap, name, argument)`:
|
||||
- New branches: `read` + argument → `📄 Reading <code>argument</code>`;
|
||||
`grep` + argument → `🔎 Searching for <code>argument</code>`; `ls` + argument
|
||||
(scoped) → `🔎 Listing documents in <code>argument</code>`; `ls` unscoped →
|
||||
`🔎 Listing documents`.
|
||||
- **Legacy branches stay**: `read_document` (Reading), `search_documents`
|
||||
(Searching for), everything else (Listing documents) — persisted turns from
|
||||
before this phase must render unchanged. (Implementation: a name→kind map
|
||||
covering both generations, or explicit if-chains — executor's choice; the pins
|
||||
below define the contract.)
|
||||
- All arguments through `textContent` only (the existing "path/pattern is data,
|
||||
never markup" discipline).
|
||||
- Update the section docstring (phase 37 reference + a phase-70 note: names
|
||||
remapped to the harness surface; legacy names still render).
|
||||
2. `frontend/assets/shared.js` — the shared-chat tool-line helper (currently
|
||||
`read_document` + argument → `📄 Reading <code>…</code>`, else `🔎 Listing
|
||||
documents`): add the new names (`read` → Reading, `grep` → Searching for with
|
||||
`<code>` argument, `ls` → Listing documents, `ls` + argument → Listing documents
|
||||
in `<code>…</code>`), keep the legacy `read_document` branch, keep
|
||||
`textContent`-only population. Update its docstring comment.
|
||||
3. `tests/unit/test_frontend_tool_states.py` — update/add pins (the
|
||||
source-string house pattern):
|
||||
- `app.js` carries branches for `read`, `grep`, `ls` (with/without argument) and
|
||||
still carries `read_document` / `search_documents` (legacy render).
|
||||
- `shared.js` carries the `read` / `grep` / `ls` handling and the legacy
|
||||
`read_document` branch.
|
||||
- Every argument population goes through `textContent` (no `innerHTML` on tool
|
||||
lines).
|
||||
- The old-only copy pins (e.g. exact "Listing documents" string for the default
|
||||
case) stay green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_frontend_tool_states.py` pins above.
|
||||
- Coverage: no `app/` code in this task — the **>90%** gate stays green as part of
|
||||
the full suite run.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_frontend_tool_states.py -v --no-cov` green.
|
||||
- [ ] A persisted chat containing old tool names renders Reading/Searching/Listing
|
||||
lines exactly as before (legacy branches pinned).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Task 05 — Mock LLM, E2E Suites, README, Gates, Commit
|
||||
|
||||
**Phase:** `70_harness_aligned_tools` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Bring the E2E world onto the new surface: the mock LLM's deterministic tool flows
|
||||
emit `ls`/`read`/`grep` calls with the new argument shapes, the three existing tool
|
||||
E2E suites assert the new names/frames, and a NEW dedicated suite pins the remap
|
||||
end-to-end. Then the full quality gates and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — update the deterministic flows (trigger strings
|
||||
`TOOLS_TRIGGER` / `MULTI_READ_TRIGGER` / `SEARCH_TRIGGER` and the `<tools>`
|
||||
marker keying stay unchanged):
|
||||
- READ flow: request 1 streams `ls` (no arguments, synthetic id `call_0`,
|
||||
`finish_reason: "tool_calls"`); request 2 (a `tool`-role catalog result in the
|
||||
messages) streams `read` with the **combined** `source/path` parsed from the
|
||||
first `source: X | path: Y` catalog line (the mock joins them — the catalog
|
||||
format is unchanged, so this is the only parse change); request 3 (a
|
||||
`tool`-role read result — content starting with the `Document source/path:`
|
||||
header) streams the plain answer.
|
||||
- MULTI-READ flow: same, `read` on the first then second combined path.
|
||||
- SEARCH flow: request 1 streams `grep` with the pattern argument (the existing
|
||||
SEARCH_TRIGGER pattern value); request 2 (a `tool`-role search result —
|
||||
`source/path:line: text` lines) streams the plain answer. Update the
|
||||
`tool`-role result detection strings where they key on the old headers.
|
||||
- Update the module docstring's flow descriptions (names/args) + the phase-70
|
||||
note.
|
||||
2. Existing E2E suites — update assertions to the new names/frames (behavior
|
||||
otherwise unchanged): `tests/e2e/test_agent_document_tools.py` (tool frame
|
||||
`name` values: `ls` then `read`; the `.tool-call` line copy "📄 Reading …"),
|
||||
`tests/e2e/test_agent_unlimited_tools.py` (multi-read frames),
|
||||
`tests/e2e/test_search_tool.py` (the `grep` frame + pattern argument).
|
||||
3. NEW `tests/e2e/test_harness_aligned_tools.py` (the phase's dedicated suite,
|
||||
house pattern, run in isolation):
|
||||
- Grounded turn with the READ trigger: the UI shows the `ls` line then the
|
||||
`📄 Reading <source/path>` line, the answer streams, and the done-state
|
||||
sources include the read document; no raw tool markup anywhere in the DOM.
|
||||
- Grounded turn with the SEARCH trigger: the `🔎 Searching for <pattern>` line
|
||||
then the answer.
|
||||
- The SSE wire itself (in-browser `fetch` capture or the existing SSE-capture
|
||||
house pattern): `tool` frames carry `name` ∈ {`ls`, `read`, `grep`} and the
|
||||
`argument` rule (read → combined path as passed; grep → pattern; ls → null
|
||||
unscoped).
|
||||
4. `README.md` — the agent-tools section (L173–176): document `ls` / `read(path)` /
|
||||
`grep(pattern, path?)` with the combined-path identity and the A5 locator
|
||||
semantics; the `.env.example` comment near `BOR_AGENT_MAX_ROUNDS` stays accurate
|
||||
(no env changes this phase).
|
||||
5. Gates + commit:
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_harness_aligned_tools.py`, then the regression
|
||||
suites `test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`, `test_chat_rag.py`.
|
||||
- One atomic commit (message below); move
|
||||
`.agent/phases/todo/70_harness_aligned_tools/` →
|
||||
`.agent/phases/complete/70_harness_aligned_tools/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new dedicated suite + the four regression suites (isolation runs).
|
||||
- Coverage: **>90%** on `app/` (phase-level gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "list_documents|read_document|search_documents" app/ frontend/ tests/
|
||||
README.md` → matches only in legacy-render pins/comments explicitly marked
|
||||
legacy (the frontend legacy branches and their unit pins).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov` green in
|
||||
isolation; regression suites green in isolation.
|
||||
- [ ] README documents the new surface; no stale tool-name copy in `README.md` /
|
||||
`.env.example` / `frontend/`.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ README.md && git commit --no-gpg-sign -m "feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)"
|
||||
```
|
||||
@@ -0,0 +1,152 @@
|
||||
# Phase 71 — Tool-Scaffolding Guardrails: Deterministic Strip + One Bounded Recovery
|
||||
|
||||
**Source:** owner request (chat, 2026-09-03) — same incident as phase 70: the
|
||||
deflected answer streamed the raw model text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>` into
|
||||
the UI (the model's own `<|…|>` chat-template tool syntax, emitted as plain
|
||||
`delta.content` even though no tools were offered). Owner direction: "We also need
|
||||
guardrails for situations like this… **deterministic guardrails only** right now,
|
||||
forget using a model for that" — no model of any kind (no `lite` classifier, no
|
||||
model-authored repair) in the guardrail path; `lite` stays the chat model for
|
||||
everything (it's faster).
|
||||
**Story:** n/a (owner request from chat — tool-scaffolding guardrails, 2026-09-03)
|
||||
**Context:**
|
||||
- `app/rag/llm.py` — `chat_stream` yields `StreamPiece("content", delta.content)`
|
||||
verbatim (L~`content = delta.content; if content: yield`); thinking pieces
|
||||
(`delta.reasoning_content`) pass through raw by design; tool-call materialization
|
||||
happens at stream end (after the `async for`); phase-48 teardown closes the
|
||||
endpoint stream on every exit (must stay intact). `chat_stream_retried` (phase 67)
|
||||
wraps it with the retry-before-first-piece rule.
|
||||
- `app/rag/agent.py` — `run_agent`'s round loop: a round with no `ToolCallPiece`s
|
||||
ends the turn (`if not calls: return`); the round cap forces one final
|
||||
`tools=None` request (the "forced final answer" pattern this phase reuses for
|
||||
recovery); `AgentHolder` (read_docs / tool_calls) is unchanged.
|
||||
- `app/api/chat.py` — the piece loop (thinking/tool/retry/delta handling +
|
||||
`thinking_chars` counter), the deflected path (`chat_stream_retried(...,
|
||||
tools=None)` directly), the `LLMError` → terminal error-frame handler, the
|
||||
per-turn log line (ends `…total_ms=N retries=N`).
|
||||
- `app/rag/prompts.py` — `build_deflect_prompt`'s `DEFLECT_MODE` body (the E2E mock
|
||||
keys on the marker's *presence*, not the wording — an appended line is safe).
|
||||
- `tests/e2e/mock_llm.py` — the trigger-string flow table (task 05 adds the
|
||||
scaffolding triggers; phase-70 names apply).
|
||||
- Tests to extend: `tests/unit/test_llm_client.py`, `test_agent.py`,
|
||||
`tests/integration/test_chat_api.py` (log-line + error-frame pins),
|
||||
`tests/unit/test_prompts.py`.
|
||||
|
||||
## Objective
|
||||
Raw tool-scaffolding tokens can never reach the user as answer text: a deterministic
|
||||
streaming filter strips known scaffolding from `delta.content` as it flows, and a
|
||||
round/turn whose visible content ends up empty (scaffolding was the whole "answer")
|
||||
gets **one** bounded, deterministic recovery (same turn, `tools=None`, a fixed
|
||||
harness-owned correction line in the system prompt); if the recovery also comes back
|
||||
empty, the turn settles with a dedicated structured error frame. No model is used to
|
||||
detect or repair anything.
|
||||
|
||||
## Dependencies
|
||||
- `70_harness_aligned_tools` (todo) — runs first: the mock/prompt/code state this
|
||||
phase builds on (new tool names in the mock flows and prompt).
|
||||
- `67_llm_retry` (complete) — `chat_stream_retried`, the primitive every request
|
||||
(including recoveries) goes through.
|
||||
- `48_stop_generation` (complete) — the phase-48 stream-teardown contract the
|
||||
filter integration must not break.
|
||||
|
||||
## Tasks
|
||||
1. `01_filter.md` — `app/rag/scaffolding.py` (new): the pattern registry +
|
||||
`ScaffoldingFilter` streaming state machine, with the boundary test matrix.
|
||||
2. `02_llm_integration.md` — `app/rag/llm.py`: `chat_stream`/`chat_stream_retried`
|
||||
accept the filter; content deltas are filtered, the tail flushed before tool
|
||||
materialization; `None` = byte-identical raw path.
|
||||
3. `03_recovery_policy.md` — `app/rag/agent.py` + `app/api/chat.py`: the empty-round
|
||||
recovery (one per turn, `tools=None`, correction constant),
|
||||
`MalformedReplyError`, the dedicated error copy, and the per-turn log line's
|
||||
`scaffold_stripped=N` field.
|
||||
4. `04_deflect_prompt.md` — the plain-text line in the `DEFLECT_MODE` body
|
||||
(prevention; owner-permitted LOW-prompt change).
|
||||
5. `05_e2e_commit.md` — the mock scaffolding triggers, the dedicated E2E suite,
|
||||
full gates, commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (new `tests/unit/test_scaffolding_filter.py`): the filter matrix — span in
|
||||
one chunk; span split across chunks at **every** boundary offset of the start
|
||||
token; multiple spans in one chunk; standalone `<|tool_calls|>` / `<|tool_call|>`
|
||||
tokens stripped; look-alikes **not** stripped (prose containing the words
|
||||
"tool_call" or `tool_call_start` without the `<|…|>` delimiters, an unknown
|
||||
`<|some_other_token|>`, a lone `<|tool_call_end|>` without a start); a partial
|
||||
start token at stream end → `flush()` emits it as-is (no false-positive strip);
|
||||
`stripped_chars` accounting; empty chunks.
|
||||
- Unit: `tests/unit/test_llm_client.py` — `chat_stream` with a filter (content
|
||||
filtered, thinking raw, `None` = raw pass-through pinned byte-identical, flush
|
||||
order: flushed tail content precedes tool-call pieces, phase-48 teardown intact).
|
||||
- Unit: `tests/unit/test_agent.py` — grounded recovery matrix (scaffolding-only
|
||||
round → exactly one recovery request: `tools=None` + correction line in the
|
||||
system prompt + fresh filter → clean answer ends the turn; scaffolding twice →
|
||||
`MalformedReplyError`; scaffolding + real content → clean answer, **no**
|
||||
recovery; round cap and kill switch unchanged).
|
||||
- Integration: `tests/integration/test_chat_api.py` — deflected-path recovery matrix
|
||||
(same shapes over the SSE endpoint: clean recovery → `done` frame; terminal → the
|
||||
dedicated error frame, no `done`, no `query_log` row — same terminal semantics as
|
||||
today's `LLMError`); log line carries `scaffold_stripped=N` (0 when nothing was
|
||||
stripped — the field is uniform, the phase-67 `retries=N` pattern).
|
||||
- E2E (mandatory, house rule): NEW dedicated suite
|
||||
`tests/e2e/test_tool_scaffolding_guardrails.py`, run in isolation — recovery case
|
||||
(raw tokens never in the DOM, clean answer shown) and terminal case (error state,
|
||||
no raw tokens, the app stays usable).
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start" frontend/` → no matches (no scaffolding rendering
|
||||
path); the filter lives in `app/rag/scaffolding.py` as a pure module (no I/O,
|
||||
no model calls).
|
||||
- [ ] A content stream of pure scaffolding yields zero `delta` frames; a mixed
|
||||
stream yields the clean remainder; thinking frames are never filtered.
|
||||
- [ ] Exactly one recovery per turn (grounded and deflected paths); the recovery
|
||||
request is `tools=None` with the fixed correction line in the system prompt;
|
||||
a second empty reply settles with the error frame
|
||||
"The model returned a malformed reply — please try again."
|
||||
- [ ] The per-turn log line ends `…retries=N scaffold_stripped=N`; `scaffold_stripped=0`
|
||||
on clean turns (uniform field).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation:
|
||||
`test_harness_aligned_tools.py`, `test_chat_rag.py`, `test_agent_document_tools.py`.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); phase dir moved to
|
||||
`.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner (chat, 2026-09-03): deterministic only.** No model — `lite` or any other —
|
||||
participates in detection or repair. The guardrail is a fixed pattern registry + a
|
||||
fixed retry policy. (A model-based classifier/repair was proposed and explicitly
|
||||
rejected for now — if it is ever wanted, it is a later phase with its own
|
||||
permission.)
|
||||
- **`lite` stays the chat model** (owner: "I want to use lite for everything since
|
||||
it's way faster") — the guardrail is what protects the UX while `lite` is the
|
||||
model; no `.env` change in this phase.
|
||||
- **The pattern registry is the extension point.** Initial entries: the observed
|
||||
span form `<|tool_call_start|>…<|tool_call_end|>` (non-greedy, any text between)
|
||||
plus the standalone sibling tokens `<|tool_calls|>` and `<|tool_call|>` from the
|
||||
same tokenizer family. Every strip logs a warning with the stripped span
|
||||
(truncated to 200 chars) — that log line is how a new format gets captured and
|
||||
added (pattern + unit fixture), keeping the registry honest (every entry traces
|
||||
to an observed capture or the initial incident).
|
||||
- **Content only, thinking never filtered.** The Thinking block is the model's raw
|
||||
reasoning by design (phase 17) and stays raw (collapsible); the guardrail protects
|
||||
the answer, not the scratchpad.
|
||||
- **Recovery is a fixed policy, not a conversation.** One extra request per turn,
|
||||
same messages with the harness-owned constant folded into the system prompt
|
||||
(single system message — provider-safe), `tools=None`, a fresh filter, the same
|
||||
phase-67 retry budget. At most one recovery; the second empty reply is terminal.
|
||||
A round with real visible content plus scaffolding needs no recovery (the clean
|
||||
content stands).
|
||||
- **Terminal semantics follow the existing error pattern.** A terminal malformed
|
||||
turn settles with a structured `error` frame (dedicated copy), writes no
|
||||
`query_log` row, and the UI shows the existing error state — byte-for-byte the
|
||||
same shape as today's `LLMError` terminal path.
|
||||
- **`MalformedReplyError` subclasses `LLMError`** and is raised only by the recovery
|
||||
policy (never from inside a stream, so `chat_stream_retried`'s retry rule never
|
||||
sees it); `chat.py` catches it before the generic `LLMError` handler.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# Task 01 — `app/rag/scaffolding.py`: Pattern Registry + Streaming Filter
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
A pure-Python streaming filter that removes known tool-scaffolding from answer text
|
||||
as it flows — no I/O, no model, fully unit-testable — plus the pattern registry that
|
||||
defines "known" (every entry traces to an observed capture).
|
||||
|
||||
## Work
|
||||
1. `app/rag/scaffolding.py` (new module):
|
||||
- `SCAFFOLD_PATTERNS: tuple[re.Pattern, ...]` — module-level, compiled:
|
||||
1. `re.compile(r"<\|tool_call_start\|>[\s\S]*?<\|tool_call_end\|>")` — the
|
||||
observed span (incident 2026-09-03), non-greedy so multiple spans each
|
||||
strip to their own end token.
|
||||
2. `re.compile(r"<\|tool_calls\|>")` — standalone sibling token (same
|
||||
tokenizer family).
|
||||
3. `re.compile(r"<\|tool_call\|>")` — standalone sibling token.
|
||||
Module docstring: the registry is the extension point — a new entry needs an
|
||||
observed capture (the strip warning log, task 03) + a unit fixture here; no
|
||||
speculative entries.
|
||||
- `class ScaffoldingFilter` — streaming state machine over a **content** stream:
|
||||
- `feed(chunk: str) -> str` — appends to an internal pending buffer and
|
||||
returns the clean text safe to emit now. Algorithm: repeatedly (a) take the
|
||||
leftmost complete match among `SCAFFOLD_PATTERNS` in the pending buffer —
|
||||
drop it (count into `stripped_chars`) and continue; then (b) check the
|
||||
buffer tail for a **live prefix**: the longest suffix that is either a
|
||||
proper prefix of any pattern's literal opening (`<|tool_call_start|>`,
|
||||
`<|tool_calls|>`, `<|tool_call|>`) or an **open span** (a start token with
|
||||
no end token yet in the buffer — everything from that start token onward is
|
||||
held, since the span may continue in future chunks). Emit everything before
|
||||
the held tail; keep the held tail as the new pending state. (Bounded: the
|
||||
held tail in NORMAL state is ≤ the longest opening token minus one char;
|
||||
in an open span it is unbounded but is being *stripped*, never emitted.)
|
||||
- `flush() -> str` — end of stream: the pending tail is emitted **as-is**
|
||||
(a partial marker at EOF is content, not scaffolding — a documented
|
||||
choice, pinned).
|
||||
- `stripped_chars: int` — total characters removed (property or attribute;
|
||||
read by the caller after the round/turn, task 03).
|
||||
- One filter instance per model request (callers create fresh instances —
|
||||
the phase-67 retry-after-dead-attempt case is safe: a dead attempt never
|
||||
fed the filter).
|
||||
- No logging, no settings, no imports beyond `re` (pure module; the strip
|
||||
*warning* log is emitted by the integration layer, task 03, so the module
|
||||
stays I/O-free).
|
||||
2. `tests/unit/test_scaffolding_filter.py` (new) — the matrix from the phase
|
||||
overview:
|
||||
- span in one chunk → stripped, `stripped_chars` exact.
|
||||
- span split across chunks: feed the incident text
|
||||
`<|tool_call_start|>[read(path='/homelab/backup-notes.md')]<|tool_call_end|>`
|
||||
at **every** split offset of the start token (0..len) plus a mid-span split
|
||||
and an end-token split → nothing ever emits until the span completes; total
|
||||
output empty; `stripped_chars` = full span length.
|
||||
- two spans in one chunk (with text between) → between-text emitted, spans
|
||||
stripped.
|
||||
- standalone `<|tool_calls|>` / `<|tool_call|>` stripped (alone and embedded in
|
||||
a line).
|
||||
- look-alikes **not** stripped, emitted verbatim: `tool_call_start` as prose;
|
||||
`<|tool_call_start|` (missing delimiter close) as prose; an unknown
|
||||
`<|some_other_token|>`; a lone `<|tool_call_end|>` without a start.
|
||||
- partial start token at EOF (`feed("<|tool_call_st")` then `flush()`) →
|
||||
flush emits it as-is, `stripped_chars == 0`.
|
||||
- surrounding text preserved exactly: `hello <|…span…> world` → `hello world`
|
||||
(no reflow beyond the removal); empty chunk no-op; `flush()` on a clean
|
||||
stream emits nothing new.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_scaffolding_filter.py` (above) — exhaustive boundary
|
||||
offsets, parametrized.
|
||||
- Coverage: **>90%** on `app/rag/scaffolding.py` (every branch: match, open-span
|
||||
hold, live-prefix hold, flush-emits-as-is).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_scaffolding_filter.py -v --no-cov` green.
|
||||
- [ ] `rg "import (os|sys|logging|app)" app/rag/scaffolding.py` → no matches (pure
|
||||
module).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Task 02 — Filter Integration in `chat_stream` / `chat_stream_retried`
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Route `delta.content` through the caller-supplied `ScaffoldingFilter` in
|
||||
`app/rag/llm.py` — content filtered, thinking raw, tail flushed before tool-call
|
||||
materialization — with `None` keeping today's byte-identical raw path.
|
||||
|
||||
## Work
|
||||
1. `app/rag/llm.py` — `chat_stream(..., scaffolding: "ScaffoldingFilter | None =
|
||||
None)"` (type imported under `TYPE_CHECKING` or as a string annotation to keep
|
||||
the module's import graph clean — the filter type is only needed for typing):
|
||||
- Content path: `content = delta.content` → when a filter is present,
|
||||
`cleaned = scaffolding.feed(content)`; yield `StreamPiece("content",
|
||||
cleaned)` only when `cleaned` is non-empty (an empty clean result yields
|
||||
**nothing** — no empty `delta` frames). Without a filter the existing
|
||||
`if content: yield` stands untouched (byte-identical).
|
||||
- Thinking path: unchanged — `reasoning_content` pieces are never filtered
|
||||
(locked: the scratchpad stays raw).
|
||||
- End of stream: after the `async for` exhausts and **before** the tool-call
|
||||
materialization block (`if calls and not emitted: …` and the
|
||||
finish-reason emission), `tail = scaffolding.flush()` when a filter is
|
||||
present; yield a content piece for the tail when non-empty. Order pinned:
|
||||
flushed-tail content precedes `ToolCallPiece`s (content-before-tools wire
|
||||
convention).
|
||||
- Teardown (phase 48) untouched: the `finally: await stream.close()` behavior is
|
||||
independent of the filter.
|
||||
- Docstring: a short "Scaffolding guardrail (phase 71)" paragraph — the filter
|
||||
is caller-owned (one per request), content-only, `None` = raw path.
|
||||
2. `app/rag/llm.py` — `chat_stream_retried(..., scaffolding=None)` — pass-through
|
||||
to `llm.chat_stream`. The same filter object is used across retry attempts of
|
||||
one logical request: safe by construction (a restarted attempt only happens
|
||||
when no piece was emitted, i.e. the filter was never fed — note this in the
|
||||
docstring).
|
||||
3. `tests/unit/test_llm_client.py` — scripted-chunk tests (the existing
|
||||
fake-client pattern):
|
||||
- with a filter: a chunk carrying a span mid-stream → the `delta` pieces carry
|
||||
only the clean text; a span split across two chunks → no partial emit;
|
||||
`filter.stripped_chars` correct after consumption.
|
||||
- thinking pieces pass through raw even when they contain a span (pinned).
|
||||
- `scaffolding=None` → the yielded pieces are byte-identical to the pre-phase
|
||||
raw path (a span in content is yielded verbatim — the raw contract for
|
||||
callers that opt out).
|
||||
- flush ordering: a trailing partial-then-complete tail yields its content
|
||||
piece **before** the materialized `ToolCallPiece` (a chunk stream that ends
|
||||
with `…clean tail` + tool_calls deltas).
|
||||
- teardown interaction: the phase-48 `aclose()`/abandon tests
|
||||
(`tests/unit/test_llm_stream_teardown.py`) stay green with and without a
|
||||
filter (run, don't rewrite).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_llm_client.py` (+ the untouched teardown suite green).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/llm.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_llm_client.py tests/unit/test_llm_stream_teardown.py
|
||||
-v --no-cov` green.
|
||||
- [ ] `scaffolding=None` callers (including the kill-switch no-tools requests) are
|
||||
byte-identical (pinned).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,104 @@
|
||||
# Task 03 — Recovery Policy: One Bounded `tools=None` Retry + Terminal Error
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
When a round/turn's visible content ends up empty **because scaffolding was the
|
||||
whole answer**, run exactly one deterministic recovery (same turn, `tools=None`,
|
||||
fixed correction line in the system prompt, fresh filter); a second empty reply
|
||||
settles with a dedicated error frame. Both the grounded agent loop and the
|
||||
deflected path get the policy; the per-turn log line gains `scaffold_stripped=N`.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py`:
|
||||
- `CORRECTION_INSTRUCTION: str` — the harness-owned constant (verbatim, single
|
||||
line): "Your previous reply contained raw tool-call markup, which is not
|
||||
interpreted here. Answer the user's question directly in plain text — no
|
||||
tool syntax." (The E2E mock in task 05 keys on a stable substring of it —
|
||||
pick the exact constant now; the mock copies it.)
|
||||
- `class MalformedReplyError(LLMError)` — raised only by the recovery policy
|
||||
(never from inside a stream, so `chat_stream_retried`'s retry rule never
|
||||
sees it). Module docstring: the phase-71 note (deterministic-only, owner
|
||||
permission 2026-09-03).
|
||||
- `run_agent` — per round: create a fresh `ScaffoldingFilter`, pass it to
|
||||
`chat_stream_retried(..., scaffolding=round_filter)`, and count the
|
||||
round's visible content (sum of the lengths of the yielded
|
||||
`StreamPiece("content", …)` texts — the filtered ones). After the round,
|
||||
when `not calls` (today's "the answer was streamed" exit):
|
||||
- if round content > 0 → return (as today).
|
||||
- if round content == 0 and `round_filter.stripped_chars > 0` → **one
|
||||
recovery**: `messages_recovered = [*messages[:-1], {"role": "system",
|
||||
"content": system_prompt + "\n" + CORRECTION_INSTRUCTION}, messages[-1]]`
|
||||
(the correction folds into the ORIGINAL single system message —
|
||||
provider-safe; the user message stays last) — one
|
||||
`chat_stream_retried` request with `tools=None`, a fresh filter, the same
|
||||
`retries`/`delay` budget; yield its pieces through the normal piece flow.
|
||||
If the recovery's visible content > 0 → return. Otherwise →
|
||||
`logger.warning` + `raise MalformedReplyError(…)`.
|
||||
- if round content == 0 and nothing was stripped → return (today's
|
||||
empty/thinking-only answer behavior — the UI handles it; unchanged).
|
||||
- Log one warning per strip event here: `logger.warning("agent: stripped
|
||||
N chars of tool-scaffolding in round %d: %r", …)` with the stripped span
|
||||
truncated to 200 chars (the capture mechanism for new registry entries —
|
||||
the filter exposes the stripped spans for this; add a
|
||||
`stripped_spans: list[str]` to the filter if needed).
|
||||
- A scaffolding-only round that also **carried tool calls** needs no
|
||||
recovery (the clean content stands / the tool ran) — the policy keys on
|
||||
the `not calls` exit only (pinned).
|
||||
2. `app/api/chat.py` — the **deflected** path (the grounded path is covered by
|
||||
`run_agent`):
|
||||
- Create one `ScaffoldingFilter` for the turn's request, pass it to
|
||||
`chat_stream_retried(..., scaffolding=filter)`; count visible content across
|
||||
the piece loop (a `content_chars` counter next to `thinking_chars`).
|
||||
- After the piece loop (deflected branch only): content == 0 and
|
||||
`filter.stripped_chars > 0` → one recovery request: the same
|
||||
`messages` with the system prompt extended by
|
||||
`CORRECTION_INSTRUCTION` (import from `app.rag.agent`), `tools=None`, a
|
||||
**fresh** filter, the same retry budget; stream its pieces through the
|
||||
SAME piece-handling code (extract the piece loop into a small inner
|
||||
helper/coroutine to avoid duplicating the thinking/tool/retry/delta
|
||||
handling — the extraction must be behavior-preserving for the first pass,
|
||||
pinned by the existing integration suite). If the recovery content > 0 →
|
||||
continue to the normal `done` flow; else → the terminal path below.
|
||||
- Catch `MalformedReplyError` **before** the generic `LLMError` handler:
|
||||
`settled = True`, yield `ChatErrorEvent(detail="The model returned a
|
||||
malformed reply — please try again.")`, return (no `query_log` row, no
|
||||
`done` — the existing terminal-error semantics; the generic
|
||||
"dropped the connection" copy stays for transport failures).
|
||||
- Per-turn log line: append `scaffold_stripped=N` after `retries=N` — the sum
|
||||
across the turn's requests (rounds + any recovery; 0 on clean turns —
|
||||
uniform field, the phase-67 `retries=N` pattern). The recovery does not
|
||||
bump `retries=N` (it is not a phase-67 endpoint-retry).
|
||||
- Module docstring: the phase-71 paragraph (deterministic guardrail +
|
||||
recovery + log field).
|
||||
3. `tests/unit/test_agent.py` — the grounded matrix (scripted fake LLM,
|
||||
monkeypatched DB): scaffolding-only round → exactly **two** model requests,
|
||||
the second `tools=None` with `CORRECTION_INSTRUCTION` in its system prompt →
|
||||
clean answer ends the turn, `done`-side state normal (holder untouched by the
|
||||
recovery); scaffolding twice → `MalformedReplyError` (assert it subclasses
|
||||
`LLMError`); scaffolding + real content → one request only, clean content
|
||||
yielded, no recovery; clean turn → one request, no correction in any system
|
||||
prompt; the round cap + kill-switch tests stay green.
|
||||
4. `tests/integration/test_chat_api.py` — the deflected matrix over
|
||||
`POST /api/chat` (mock LLM): scaffolding-only deflected turn → clean recovery
|
||||
answer + `done` frame; scaffolding twice → the error frame with the dedicated
|
||||
copy, no `done`, no `query_log` row (assert against the table); mixed
|
||||
scaffolding+content → clean answer, `scaffold_stripped>0` in the log line;
|
||||
the log-line pin gains `scaffold_stripped=0` on clean turns and the summed
|
||||
value on stripped turns.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` (grounded matrix).
|
||||
- Integration: `tests/integration/test_chat_api.py` (deflected matrix + log line).
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/agent.py`,
|
||||
`app/api/chat.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_chat_api.py -v
|
||||
--no-cov` green.
|
||||
- [ ] Exactly one recovery per turn, on both paths; the recovery request is
|
||||
`tools=None` + correction line; a second empty reply → the dedicated error
|
||||
frame, no `query_log` row.
|
||||
- [ ] Clean turns: no correction in any system prompt, `scaffold_stripped=0`,
|
||||
request counts unchanged (kill-switch/deflection byte-identical pins green).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 04 — Deflect Prompt: Plain-Text-Only Line (Prevention)
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Close the door at the prompt: the deflected (LOW) turn offers no tools, so any tool
|
||||
markup there is always wrong — add one plain-text-only instruction line to the
|
||||
`DEFLECT_MODE` body. This is the owner-permitted change to the otherwise-locked LOW
|
||||
prompt; the `DEFLECT_MODE` marker and everything else in the prompt stay put (the
|
||||
E2E mock keys on the marker's presence, not the wording).
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `build_deflect_prompt`: append one line to the
|
||||
`DEFLECT_MODE` body (after "…propose 2-3 alternative questions."):
|
||||
"Reply in plain text only — you have no tools in this mode." The weak-hit
|
||||
title list follows exactly as today; `build_high_prompt`, `PERSONA`,
|
||||
`TOOLS_SECTION`, and the section order are untouched. Update the module
|
||||
docstring: the phase-71 note (owner-permitted 2026-09-03 LOW-prompt line;
|
||||
the marker-keying contract unchanged).
|
||||
2. `tests/unit/test_prompts.py` (and `test_chat_gate.py` where the deflection body
|
||||
is pinned) — pins: the new line present in the LOW prompt; the `DEFLECT_MODE`
|
||||
marker still present (mock keying); the HIGH prompt unchanged (byte-identical
|
||||
pins stay green — the line must not leak into HIGH); the byte-identical-when-
|
||||
empty steering/overview behavior unchanged.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the prompt pins above.
|
||||
- Coverage: **>90%** on this task's modified code (`app/rag/prompts.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v
|
||||
--no-cov` green.
|
||||
- [ ] LOW prompt = pre-phase text + exactly the one new line (a diff pin, or a
|
||||
reconstruction pin in the test).
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Task 05 — Mock Triggers, Dedicated E2E Suite, Gates, Commit
|
||||
|
||||
**Phase:** `71_scaffolding_guardrails` · **Story:** n/a (owner request from chat, 2026-09-03)
|
||||
|
||||
## Objective
|
||||
Prove the guardrail end-to-end through the real UI: a mock-LLM scaffolding flow
|
||||
(recovery case + terminal case), a dedicated Playwright suite pinning that raw
|
||||
tokens never reach the DOM, then the full quality gates and the phase commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — two new deterministic flows (checked in the flow
|
||||
table **before** the plain `TOOLS_TRIGGER` flow, after `SEARCH_TRIGGER`
|
||||
ordering rules as they fit — the triggers are independent of the `<tools>`
|
||||
marker, so both grounded and deflected turns hit them):
|
||||
- `SCAFFOLD_TRIGGER = "emit raw tool markup"` — request 1 (no correction in
|
||||
the system prompt): stream ONLY `delta.content` chunks carrying the incident
|
||||
text `<|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|>`
|
||||
(split across ≥2 chunks to exercise the boundary path), `finish_reason:
|
||||
"stop"`, no structured `tool_calls`, no reasoning. Request 2 (system prompt
|
||||
contains the stable substring of `CORRECTION_INSTRUCTION` — import it from
|
||||
`app.rag.agent` so the mock can never drift from the constant): stream a
|
||||
clean plain answer ("Here is the plain-text answer the recovery produced.")
|
||||
+ `finish_reason: "stop"`.
|
||||
- `SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup"` — every request
|
||||
(recovery included): the same scaffolding-only stream, forever.
|
||||
- Update the module docstring's flow table + the phase-71 note.
|
||||
2. `tests/e2e/test_tool_scaffolding_guardrails.py` (NEW — the phase's dedicated
|
||||
suite, house pattern, run in isolation; DB up, mock LLM):
|
||||
- **Recovery case** — ask a question containing `SCAFFOLD_TRIGGER`: the turn
|
||||
settles (the composer re-enables, `done` observed); the final answer bubble
|
||||
contains the recovery's clean text; `document.body.innerText` contains
|
||||
**neither** `tool_call_start` nor `tool_call_end` (nor the raw
|
||||
`[read(path=…]` fragment); no error banner.
|
||||
- **Terminal case** — ask a question containing
|
||||
`SCAFFOLD_ALWAYS_TRIGGER`: the existing error status renders with the
|
||||
dedicated copy ("The model returned a malformed reply — please try
|
||||
again."); no raw tokens in the DOM; no answer bubble with the scaffolding;
|
||||
the app stays usable — a follow-up plain question (no trigger) gets a
|
||||
normal streamed answer in the same session.
|
||||
- **No false positive** — a plain question (existing `CHAT_TRIGGER`-style
|
||||
flow, no tools needed): the answer streams byte-clean, no error state, no
|
||||
recovery request visible (the turn settles on the first request).
|
||||
- The SSE wire itself: in the recovery case, no `delta` frame ever carries a
|
||||
`tool_call_start`/`tool_call_end` fragment (the existing SSE-capture house
|
||||
pattern) — the strip happens server-side, not in the UI.
|
||||
3. Gates + commit:
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_scaffolding_guardrails.py`, then the
|
||||
regression suites `test_harness_aligned_tools.py`, `test_chat_rag.py`,
|
||||
`test_agent_document_tools.py`.
|
||||
- One atomic commit (message below); move
|
||||
`.agent/phases/todo/71_scaffolding_guardrails/` →
|
||||
`.agent/phases/complete/71_scaffolding_guardrails/`.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new dedicated suite (recovery / terminal / no-false-positive / wire
|
||||
pins) + the three regression suites (isolation runs).
|
||||
- Coverage: **>90%** on `app/` (phase-level gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `rg "tool_call_start|tool_call_end" frontend/` → no matches.
|
||||
- [ ] Raw scaffolding is never visible in the DOM in any case (recovery,
|
||||
terminal, mixed) — pinned by the dedicated suite.
|
||||
- [ ] Exactly one recovery per malformed turn (mock request counts implied by the
|
||||
flows); clean turns never carry the correction line (pinned by the unit
|
||||
suites from task 03).
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app` TOTAL **>90%**;
|
||||
`uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov`
|
||||
green in isolation; regression suites green in isolation.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"
|
||||
```
|
||||
@@ -0,0 +1,165 @@
|
||||
# Phase 72 — Teaching Refusals: End the Post-Harness Tool-Loop Rambling
|
||||
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md` (this phase repairs the model-facing
|
||||
contract the phase-70 tools reshaped)
|
||||
**Context:**
|
||||
- `app/rag/agent.py` — `AGENT_TOOLS` (the phase-70 `ls` / `read` / `grep` OpenAI function
|
||||
definitions), `_execute_tool` (the refusal strings:
|
||||
`"No source named '…' — check the ls output."`,
|
||||
`"No document at '…' — check the ls output."`), `all_documents`
|
||||
(catalog-order bulk loader — reused by the suggestion lookup).
|
||||
- `app/rag/prompts.py` — `TOOLS_SECTION` (HIGH prompt only; the E2E mock keys off the
|
||||
`<tools>` marker's *presence*, not its wording).
|
||||
- `tests/unit/test_agent.py` (refusal-string pins; the `ScriptedLLM` + monkeypatched-
|
||||
accessor pattern), `tests/integration/test_agent_tools.py` (the same pins against real
|
||||
Postgres, `kb`/`src` fixtures).
|
||||
- `tests/e2e/mock_llm.py` — the deterministic mock tool flows (`TOOLS_TRIGGER` single-read,
|
||||
`MULTI_READ_TRIGGER`, `SEARCH_TRIGGER`; `_CATALOG_LINE_RE` catalog-line parse) and the
|
||||
dedicated-suite-per-phase E2E house pattern.
|
||||
- `scripts/llm_probe.py` — the house live-endpoint probe pattern (`python -m scripts.…`,
|
||||
argparse, dotenv, printed verdict line); `app/api/chat.py` — the grounded path the
|
||||
real-model gate mirrors (`retrieve` → `select_documents` → `build_high_prompt` →
|
||||
`run_agent`)
|
||||
- **Incident (owner chat, 2026-09-03, post phase 70/71):** the question "list the files
|
||||
in this directory" produced a Thinking-display trace of the model calling
|
||||
`ls(path='app/rag/importer.py')` → `"No source named 'app/rag/importer.py' — check the
|
||||
ls output."`, then `ls(path='.')` → the same-style refusal, then re-reasoning the same
|
||||
paragraphs over and over across rounds (each round's `reasoning_content` appends to the
|
||||
open Thinking block) before finally answering from the seed documents alone. Root cause:
|
||||
the harness-trained prior (`ls`'s `path` = a directory to list) collides with this app's
|
||||
contract (`path` = a source-name filter), and the terse refusal does not correct the
|
||||
misunderstanding, so the model burns rounds. The identical trap awaits `read`/`grep`:
|
||||
a bare document path missing the source prefix (`read('app/rag/importer.py')`) →
|
||||
`"No document at '…'"` with no hint of the combined form.
|
||||
|
||||
## Objective
|
||||
Make the affected tool refusals **teaching** so the harness-prior misuse self-corrects in
|
||||
at most one extra round: a scoped `ls` whose `path` looks like a document path (contains
|
||||
`/`) or names an unknown source gets a fixed-template refusal that states the correct
|
||||
contract; a `read` / scoped-`grep` argument that resolves to no combined identity but
|
||||
*matches an indexed document's `path`* (exact or suffix) gets a
|
||||
`"did you mean 'source/path'?"` refusal naming the exact combined identity to use. The
|
||||
`AGENT_TOOLS` `path` descriptions and the `TOOLS_SECTION` prompt copy say the same
|
||||
contract up front. Deterministic only — no model participates in detection or repair; the
|
||||
phase-70 harness shape (`ls` / `read(path)` / `grep(pattern, path?)`) is unchanged
|
||||
verbatim. The phase does not pass on mocks alone: a live acceptance gate runs the
|
||||
fixed question battery through `run_agent` against the **real configured chat model**
|
||||
(`lite` per `.env`) and must PASS before the commit (owner directive, 2026-09-03 —
|
||||
"test with the real lite model until tool calls work consistently; don't pass until a
|
||||
sufficient number of tool calls succeed").
|
||||
|
||||
## Dependencies
|
||||
- `70_harness_aligned_tools` (complete) — the tool surface this phase teaches (shape
|
||||
untouched).
|
||||
- `71_scaffolding_guardrails` (complete) — the deterministic-guardrail house style this
|
||||
phase follows.
|
||||
|
||||
## Tasks
|
||||
1. `01_ls_teaching_refusal.md` — `ls`: path-like and unknown-source scopes get teaching
|
||||
refusals; the `ls` `path` description says "source name, not a file or directory path".
|
||||
2. `02_read_grep_path_suggestion.md` — `read` / `grep`: an unresolved argument that
|
||||
matches an indexed document `path` gets the "did you mean 'source/path'?" suggestion;
|
||||
descriptions updated.
|
||||
3. `03_prompt_copy.md` — `TOOLS_SECTION` copy: the `ls` `path` is a source name, not a
|
||||
directory; `read`/`grep` need the combined identity *including the source name*.
|
||||
4. `04_mock_e2e.md` — mock `ls`-misuse flow, dedicated E2E suite (green in isolation).
|
||||
5. `05_real_model_gate.md` — the live real-lite acceptance gate
|
||||
(`scripts/agent_realmodel_check.py`): iterate the copy levers until the gate
|
||||
PASSES, then full gates and the commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_agent.py` — the new `ls` teaching refusals (scope containing
|
||||
`/` → the document-path line; scope without `/` unknown → the extended no-source line;
|
||||
both count in nothing, tools stay offered; valid-scope and no-arg listings
|
||||
byte-identical to today); `find_path_candidates` (exact `path` match, suffix match,
|
||||
multiple candidates in catalog order capped at 3, zero candidates, no-`/` argument →
|
||||
no DB lookup); `read`/`grep` wiring (in-context dedupe precedence, valid combined form
|
||||
unchanged, scoped `grep` suggestion, A5 grep contract regression).
|
||||
- Integration: `tests/integration/test_agent_tools.py` — changed pins updated; new
|
||||
end-to-end suggestion cases through `run_agent` against real Postgres (bare path under
|
||||
one source; the same path under two sources).
|
||||
- E2E (mandatory, house rule): NEW dedicated suite `tests/e2e/test_tool_path_teaching.py`,
|
||||
run in isolation — the mock flow (misuse `ls(path='.')` → teaching refusal → corrected
|
||||
no-arg `ls()` → listing answer) through the real UI with the two-round shape pinned on
|
||||
the SSE wire; regression suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_search_tool.py`,
|
||||
`test_chat_rag.py`.
|
||||
- **Real-model acceptance gate (owner-locked, the phase's pass condition):**
|
||||
`uv run python -m scripts.agent_realmodel_check` against the live endpoint with the
|
||||
configured chat model (`lite`) — the fixed 10-question battery (3 `ls` turns including
|
||||
the incident's "list the files in this directory" and a source-name trap, 4 `read`
|
||||
turns including two bare-path traps, 1 `grep` turn, 2 mixed) driven through the real
|
||||
grounded path. PASS = every turn answers (no `LLMError`/`MalformedReplyError`), zero
|
||||
turns hit the round cap, ≥6 of 10 turns emit ≥1 tool call, and **≥90% of all emitted
|
||||
tool calls execute** (rejections don't count). Until it passes, task 05 iterates the
|
||||
copy levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions,
|
||||
`TOOLS_SECTION`) — the question set and thresholds are fixed by the task file and may
|
||||
not be weakened.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A scoped `ls` whose stripped `path` contains `/` gets the document-path teaching
|
||||
refusal; an unknown source name without `/` gets the extended "source name, not a
|
||||
directory" refusal; neither counts in anything; a valid scope and the no-arg
|
||||
listing are byte-identical to today.
|
||||
- [ ] `read` / scoped `grep` with an unresolved argument that matches an indexed document
|
||||
`path` (exact or suffix) gets the "did you mean …?" refusal (one candidate → one
|
||||
combined identity; two or more → up to 3, catalog order); a non-matching argument
|
||||
gets today's refusal byte-identical; the in-context dedupe refusal still wins.
|
||||
- [ ] The `AGENT_TOOLS` `path` descriptions for `ls` / `read` / `grep` state the contract
|
||||
explicitly; the tool names and argument shapes are unchanged
|
||||
(`rg '"name":' app/rag/agent.py` → exactly `ls`, `read`, `grep`).
|
||||
- [ ] `TOOLS_SECTION` clarifies the source-name `ls` `path` and the source-name-required
|
||||
combined identity; the HIGH prompt still ends with the `<tools>` section; the
|
||||
LOW/deflection prompt is byte-identical to today.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL
|
||||
**>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation; the regression suites above green in isolation.
|
||||
- [ ] `uv run python -m scripts.agent_realmodel_check` exits 0 against the live
|
||||
endpoint (all four pass conditions met with the configured model) — the verdict
|
||||
line recorded in the `app/rag/agent.py` module docstring and in the commit body.
|
||||
- [ ] One `--no-gpg-sign` commit (message in the Commit block); the phase directory
|
||||
moved to `.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **The phase-70 tool surface is unchanged** (owner lock, 2026-09-03): `ls(path?)` /
|
||||
`read(path)` / `grep(pattern, path?)` — no renames, no argument additions or removals;
|
||||
this phase changes refusal copy, tool descriptions, and prompt copy only.
|
||||
- **Deterministic only** (owner 2026-09-03, phase-71 house style): no model in detection
|
||||
or repair; suggestions are a pure catalog lookup (exact or suffix `path` match,
|
||||
case-sensitive, catalog order, capped at 3); every refusal is a fixed template
|
||||
constant.
|
||||
- **Teach, don't silently fix:** a misused call is still a refusal (counts in nothing,
|
||||
consumes a round); the model sees its own argument echoed plus the correct form. No
|
||||
silent argument normalization — `ls(path='.')` does NOT become a full listing.
|
||||
- **The zero-candidate refusal is byte-identical to today**
|
||||
(`"No document at '…' — check the ls output."`) — no behavior change where the model is
|
||||
not confused; the `ls` no-source refusal keeps its prefix (the teaching parenthetical
|
||||
is appended).
|
||||
- **No UI change:** the Thinking display (phases 17/21/43) works as designed — the fix
|
||||
ends the loop, it does not hide the scratchpad. **No SSE contract change** (refusals
|
||||
are tool results in the message history; the `tool` frames already carry the call's
|
||||
`name`/`argument`). **No model swap** (owner keeps `lite`), **no env change**,
|
||||
**no schema change**.
|
||||
- **Real-model gate is a pass condition, not a smoke test** (owner directive,
|
||||
2026-09-03): the phase is NOT complete — and gets NO commit — until
|
||||
`scripts/agent_realmodel_check.py` PASSES against the real `lite` model. The 10
|
||||
questions, the ≥6-of-10 tool-usage floor, the ≥90% executed-call bar, and the
|
||||
zero-cap rule are fixed by task 05's file; the executor may iterate ONLY the copy
|
||||
levers this phase owns (refusal templates, `AGENT_TOOLS` descriptions,
|
||||
`TOOLS_SECTION` — with their unit pins updated to follow the constants). Lowering a
|
||||
threshold, swapping in easier questions, or skipping the gate to "make it pass" is
|
||||
forbidden; a gate still failing after iteration stops the phase with the per-turn
|
||||
numbers reported for the owner (fail-loud house style). The verdict line (house
|
||||
precedent: the phase-37 probe verdict in `app/rag/agent.py`) is recorded in that
|
||||
module's docstring and in the commit body.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ scripts/ && git commit --no-gpg-sign -m "fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling" -m "<real-model gate verdict line, e.g. real-model gate (lite): 10/10 answered, caps=0, tool-turns=8, calls 21/23 executed (91%) — 2026-09-03>"
|
||||
```
|
||||
The commit also carries the still-uncommitted phase-71 `todo/` → `complete/` move and
|
||||
`.agent/reports/71_scaffolding_guardrails/` (`.agent/` is tracked and committed with the
|
||||
phase — AGENTS.md §8; only `.agent/phase-sessions/` and `.agent/pipeline.log` are
|
||||
gitignored).
|
||||
@@ -0,0 +1,64 @@
|
||||
# Task 01 — `ls`: Teaching Refusals for Path-Like and Unknown-Source Scopes
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
A scoped `ls` whose `path` argument is a file/directory path (contains `/`) — or an
|
||||
unknown source name — gets a fixed-template refusal that states the correct contract
|
||||
instead of the terse "check the ls output", so the harness-prior misuse
|
||||
(`ls(path='app/rag/importer.py')`, `ls(path='.')` — the incident) self-corrects in one
|
||||
round. The `ls` tool description makes the same point at request time.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — two refusal template constants next to the existing refusal
|
||||
constants, plus the branch change:
|
||||
- `LS_PATH_NOT_A_SOURCE: str` — one `{path}` field, used when the **stripped** scope
|
||||
contains `/` (a source name can never contain `/` — source names are directory
|
||||
basenames, `app.rag.importer`):
|
||||
`"'{path}' looks like a document path, not a source name. The 'path' argument of ls filters by source name (e.g. 'homelab') — omit it to list every document, or read a document by its combined 'source/path' string."`
|
||||
- `NO_SOURCE_NOT_A_DIRECTORY: str` — one `{scope}` field, the existing no-source
|
||||
refusal with a teaching parenthetical appended (the prefix
|
||||
`"No source named '{scope}' — check the ls output."` stays byte-identical), used
|
||||
when the scope has no `/` and matches no registered source name:
|
||||
`"No source named '{scope}' — check the ls output. (The 'path' argument is a source name, not a directory — omit it to list every document.)"`
|
||||
- `_execute_tool` `ls` branch: non-empty scope with `"/" in scope` →
|
||||
`LS_PATH_NOT_A_SOURCE.format(path=scope)`; non-empty scope without `/` not in
|
||||
`list_source_names(db)` → `NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)`; a valid
|
||||
scope and the no-arg listing are unchanged. Both refusals count in nothing (no
|
||||
`holder.tool_calls` bump) and consume a round — exactly like today's refusal.
|
||||
- `AGENT_TOOLS` → `ls` → `function.parameters.properties.path.description`:
|
||||
`"Source name to list one source's documents (e.g. 'homelab') — a source name, not a file or directory path; omit to list every document."`
|
||||
- Module docstring (loop contract, point 3 — the refusal list): update the
|
||||
scoped-`ls` refusal entry to the two new lines.
|
||||
2. `tests/unit/test_agent.py` — unit pins (existing `ScriptedLLM` + monkeypatched
|
||||
`list_catalog` / `list_source_names` pattern; import the constants, never re-type
|
||||
them):
|
||||
- `ls(path='app/rag/importer.py')` (scope contains `/`) → the
|
||||
`LS_PATH_NOT_A_SOURCE` line with the argument echoed; `holder.tool_calls == 0`;
|
||||
tools stay offered on the next request.
|
||||
- `ls(path='.')` (no `/`, unknown) → the `NO_SOURCE_NOT_A_DIRECTORY` line with
|
||||
`'.'` echoed; `holder.tool_calls == 0`.
|
||||
- `ls(path='Ghost')` (no `/`, unknown) → the same extended line (replaces today's
|
||||
`test_ls_scoped_unknown_source_refused` pin).
|
||||
- Regression: `ls()` no-arg full catalog and `ls(path='<registered source>')` scoped
|
||||
listing (including the `0 documents:` registered-empty-source case) remain
|
||||
byte-identical to today.
|
||||
3. `tests/integration/test_agent_tools.py` — update the changed pin (the
|
||||
`ls(path='Ghost')` assertion) and add one case: a scoped `ls` with a `/`-containing
|
||||
`path` against the real DB (`kb` + `src` fixtures) → the document-path line, not
|
||||
counted, tools stay offered.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: as listed in Work 2–3 — every new refusal line pinned
|
||||
byte-for-byte; the count-in-nothing and tools-stay-offered invariants pinned; the
|
||||
unchanged paths regression-pinned.
|
||||
- Coverage: **>90%** on this task's new/modified code (the `ls` branch in
|
||||
`app/rag/agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov`
|
||||
green (DB up: `podman compose up -d db`)
|
||||
- [ ] The old terse string (no-source refusal without the parenthetical) appears nowhere
|
||||
in `app/` or `tests/`
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No behavior change to valid-scope / no-arg `ls` (regression pins green)
|
||||
@@ -0,0 +1,87 @@
|
||||
# Task 02 — `read` / `grep`: "did you mean 'source/path'?" Suggestions for Bare Document Paths
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
When `read` (or a scoped `grep`) receives an argument that resolves to no combined
|
||||
identity but *does* match an indexed document's `path` (exact or as a suffix), the
|
||||
refusal names the exact combined `source/path` identity to use — the harness prior
|
||||
(`read('app/rag/importer.py')`, missing the source prefix) self-corrects in one round.
|
||||
An argument that matches nothing keeps today's refusal byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py`:
|
||||
- `SUGGESTION_LIMIT = 3` — the cap on suggested identities per refusal.
|
||||
- Module-level `find_path_candidates(db: Session, arg: str) -> list[tuple[str, str, str]]`
|
||||
(so unit tests can monkeypatch it, house pattern): the indexed documents, in
|
||||
**catalog order** (the `all_documents` order), whose `path` equals `arg` or ends
|
||||
with `f"/{arg}"` (case-sensitive — these are file paths), as `(source, path, title)`
|
||||
triples. One bulk query via `all_documents`; called **only** from the refusal path
|
||||
below (never on the happy path) and **only** when `arg` contains `/` (a bare name
|
||||
keeps today's no-DB-lookup refusal — the existing
|
||||
`test_read_bare_source_name_refused_without_db` invariant stays green).
|
||||
- Refusal templates next to the existing constants:
|
||||
- `NO_DOCUMENT_DID_YOU_MEAN: str` —
|
||||
`"No document at '{arg}' — did you mean '{source}/{path}'?"`
|
||||
- `NO_DOCUMENT_DID_YOU_MEAN_MANY: str` —
|
||||
`"No document at '{arg}' — did you mean one of: {candidates}?"` where
|
||||
`{candidates}` is up to `SUGGESTION_LIMIT` combined `source/path` identities,
|
||||
each single-quoted, joined with `", "`, in catalog order.
|
||||
- `_execute_tool` `read` branch: after the in-context dedupe check and the
|
||||
`_resolve_path` miss — when `arg` contains `/`, run `find_path_candidates`:
|
||||
exactly 1 candidate → `NO_DOCUMENT_DID_YOU_MEAN`; 2+ →
|
||||
`NO_DOCUMENT_DID_YOU_MEAN_MANY`; 0 → today's
|
||||
`"No document at '{arg}' — check the ls output."` unchanged. `holder` untouched
|
||||
(a refusal counts in nothing; `read_docs` untouched — locator-only never changes).
|
||||
- `_execute_tool` `grep` branch: the same substitution for the scoped-`path` miss
|
||||
(the whole-KB grep is untouched).
|
||||
- `AGENT_TOOLS` → `read` → `path` description and `grep` → `path` description:
|
||||
append `" A bare document path (without the source name) will not resolve."` to
|
||||
each current text.
|
||||
- Module docstring (loop contract, point 3): document the suggestion behavior in the
|
||||
refusal list.
|
||||
2. `tests/unit/test_agent.py` — unit pins (monkeypatched `find_document` +
|
||||
`all_documents`; import the constants, never re-type them):
|
||||
- `read(path='active/container_caddy/caddy.md')` with the document indexed under
|
||||
`Homelab` (exact `path` match) → `did you mean 'Homelab/active/container_caddy/caddy.md'?`;
|
||||
`holder.read_docs` empty, `holder.tool_calls == 0`.
|
||||
- Suffix match: `read(path='caddy.md')` → the same single suggestion.
|
||||
- Two sources sharing the same `path` →
|
||||
`did you mean one of: 'A/x.md', 'B/x.md'?` in catalog order.
|
||||
- Four sources sharing the `path` → exactly 3 suggestions (the cap).
|
||||
- No match → today's refusal byte-identical; `read(path='Homelab')` (bare, no `/`) →
|
||||
today's refusal with **no** `find_document` / `all_documents` call (the `_boom`
|
||||
guard, existing pattern).
|
||||
- Dedupe precedence: a combined-form re-read of a `seed_docs` document →
|
||||
`ALREADY_IN_CONTEXT` (unchanged); a bare-`path` read of an in-context document
|
||||
(`read('app/rag/importer.py')` with `sample/app/rag/importer.py` seeded) → the
|
||||
suggestion line (the split pair is not in `known`, so the model learns the
|
||||
combined identity — its next, correctly-formed call is then deduped).
|
||||
- Scoped `grep` miss with a candidate → the suggestion line; scoped `grep` miss
|
||||
without → today's line; whole-KB `grep` unchanged (A5 match/output contract:
|
||||
fixed substring, case-insensitive, 20 matches, 200-char lines).
|
||||
- Happy paths regression-pinned: combined-form `read` (full content,
|
||||
`read_docs` appended), valid scoped `grep` result.
|
||||
3. `tests/integration/test_agent_tools.py` — update the changed pins (inspect each
|
||||
existing `"No document at …"` assertion against the fixture documents; only the
|
||||
lines whose argument matches a fixture document `path` change to the suggestion form),
|
||||
plus two new end-to-end cases through `run_agent` against real Postgres: a bare path
|
||||
under one source (single suggestion) and the same `path` under two sources (the
|
||||
"one of" line) — in both, the refusal is followed by the model's corrected call
|
||||
succeeding (scripted `ToolCallPiece` round 2 with the suggested combined identity).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: as listed in Work 2–3 — every new template pinned byte-for-byte;
|
||||
the catalog-order + cap invariant pinned; the zero-candidate and no-DB-lookup
|
||||
invariants pinned; the A5 `grep` contract regression-pinned.
|
||||
- Coverage: **>90%** on this task's new/modified code (`find_path_candidates` + both
|
||||
`_execute_tool` branches).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov`
|
||||
green (DB up: `podman compose up -d db`)
|
||||
- [ ] `find_path_candidates` is module-level (monkeypatchable) and issues at most one
|
||||
bulk query
|
||||
- [ ] The zero-candidate refusal and the bare-name (no-DB-lookup) refusal are
|
||||
byte-identical to today
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 03 — `TOOLS_SECTION` Copy: State the Contract Up Front
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The HIGH prompt's `<tools>` section says the same two things the new refusals teach —
|
||||
the `ls` `path` is a *source name*, not a directory or file path, and `read`/`grep`
|
||||
need the combined `source/path` string *including the source name* — so the model
|
||||
carries the contract before it calls a tool, not only after being refused.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `TOOLS_SECTION` rewritten (the E2E mock keys off the
|
||||
`<tools>` marker's *presence*, not this wording, so the change is mock-safe):
|
||||
- `ls` clause: its optional `path` argument is a *source name* (e.g. `'homelab'`)
|
||||
— **not** a directory or file path; omit it to list every document.
|
||||
- `read` clause: the combined `source/path` string, exactly as shown in the `ls`
|
||||
output — *including the source name*; a bare document path will not resolve.
|
||||
- `grep` clause: the locator copy stays (its `path` is already described as a
|
||||
combined `source/path` string); add the same bare-path-will-not-resolve note.
|
||||
- Keep the section's shape: a single paragraph between `<tools>` and `</tools>`,
|
||||
still appended after the mode body in the HIGH prompt only (the LOW/deflection
|
||||
prompt never carries it — phase 71's plain-text line stays put).
|
||||
2. `tests/unit/test_prompts.py` — update the `TOOLS_SECTION` wording pin(s) where they
|
||||
pin the old wording; the `<tools>`-marker-present-in-HIGH pin, the
|
||||
marker-absent-from-LOW pin, and the byte-identical-LOW-prompt pin stay green as-is.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_prompts.py` — marker present in the HIGH prompt and absent
|
||||
from the LOW prompt; the LOW prompt byte-identical to today; the new wording pinned
|
||||
for the `ls` source-name clause and the read combined-identity clause.
|
||||
- Coverage: **>90%** on this task's modified code (the constant itself — the builders
|
||||
are already covered).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py -v --no-cov` green
|
||||
- [ ] The HIGH prompt still ends with the `<tools>` section (existing section-order pin
|
||||
green); the LOW/deflection prompt is byte-identical to today
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,63 @@
|
||||
# Task 04 — Mock Flow, Dedicated E2E Suite
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Prove the self-correction loop deterministically through the real UI: a mock-LLM flow
|
||||
that reproduces the incident's `ls(path='.')` misuse, receives the teaching refusal,
|
||||
corrects to a no-arg `ls()`, and answers from the catalog — a dedicated Playwright
|
||||
suite pinning the two-round shape on the SSE wire. (The live real-model acceptance
|
||||
gate is task 05 — this task is the deterministic half of the proof.)
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — one new deterministic flow, checked in the flow table
|
||||
**before** the plain `TOOLS_TRIGGER` flow (the trigger phrases are disjoint
|
||||
substrings; the ordering rule follows the phase-71 convention):
|
||||
- `LS_TEACH_TRIGGER = "list the files in this directory"` — **and** the system
|
||||
prompt carries the `<tools>` section (grounded turn):
|
||||
* request 1 (tools offered, no `tool`-role result in the messages yet): stream
|
||||
ONLY `tool_calls` deltas — `ls` with `{"path": "."}` (synthetic id `call_0`),
|
||||
`finish_reason: "tool_calls"`, no content (the incident's misuse,
|
||||
deterministic);
|
||||
* request 2 (a `tool`-role result present that is **not** a catalog listing —
|
||||
i.e. the teaching refusal): stream a `tool_calls` delta — `ls` with no
|
||||
arguments (id `call_1`);
|
||||
* request 3 (a `tool`-role result whose first line matches the
|
||||
`^\d+ documents:` catalog header): a deterministic content answer —
|
||||
`These are the indexed documents: <first catalog line>` (the
|
||||
`source: X | path: Y | title: Z` line, parsed with the existing
|
||||
`_CATALOG_LINE_RE` machinery), `finish_reason: "stop"`.
|
||||
- Update the module docstring's flow table with the phase-72 note.
|
||||
2. `tests/e2e/test_tool_path_teaching.py` (NEW — the phase's dedicated suite, house
|
||||
pattern, run in isolation; DB up, mock LLM):
|
||||
- Import a small fixture document set (house fixture pattern: one source, two
|
||||
documents with known `source`/`path`/`title`) and ask a question containing
|
||||
`LS_TEACH_TRIGGER`.
|
||||
- **Self-correction** — the turn settles (composer re-enables, `done` observed);
|
||||
the answer bubble contains the first document's `source:` and `path:` fields
|
||||
(the catalog reached the model and landed in the answer); no error banner.
|
||||
- **Two rounds on the wire** (the house SSE-capture pattern): the `tool` frames
|
||||
arrive in order — first `name:"ls"` with `argument:"."`, then `name:"ls"` with
|
||||
`argument:null` — and there is **no** third `tool` frame (the loop ended in one
|
||||
correction, not at the round cap).
|
||||
- **No regression to the plain flow** — a follow-up question containing
|
||||
`TOOLS_TRIGGER` (the single-read flow) in the same session still settles with
|
||||
the read flow's answer (the new flow did not swallow the existing trigger).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the dedicated suite proves the loop shape (misuse → teaching refusal →
|
||||
corrected call → answer) through the real UI and the SSE wire; the existing
|
||||
regression E2E suites (mock-driven) stay green — run them as the regression check
|
||||
for this task.
|
||||
- Coverage: unit/integration coverage of `app/` stays >90% (this task adds test-only
|
||||
code; `uv run pytest --cov=app --cov-report=term-missing` as the check).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation (DB up: `podman compose up -d db`, mock LLM)
|
||||
- [ ] Regression E2E suites green in isolation: `test_harness_aligned_tools.py`,
|
||||
`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`,
|
||||
`test_search_tool.py`, `test_chat_rag.py`
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] No commit in this task (the commit happens in task 05, after the real-model
|
||||
gate passes)
|
||||
@@ -0,0 +1,114 @@
|
||||
# Task 05 — Real-Model Acceptance Gate (live `lite`), Full Gates, Commit
|
||||
|
||||
**Phase:** `72_teaching_refusals` · **Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The phase's pass condition (owner directive, 2026-09-03: "test with the real lite
|
||||
model until tool calls work consistently — don't pass until a sufficient number of
|
||||
tool calls succeed"): a live script drives the fixed 10-question battery through the
|
||||
**real** grounded path (real endpoint, configured chat model — `lite` per `.env`,
|
||||
real Postgres KB) and the phase commits only when the gate PASSES. Until it does,
|
||||
iterate the copy levers this phase owns (refusal templates, `AGENT_TOOLS`
|
||||
descriptions, `TOOLS_SECTION`) — never the gate.
|
||||
|
||||
## Work
|
||||
1. `scripts/agent_realmodel_check.py` (NEW — house probe pattern, `scripts/llm_probe.py`
|
||||
as the model: `uv run python -m scripts.agent_realmodel_check`, argparse, dotenv,
|
||||
plain module, no debugpy):
|
||||
- **Preconditions (exit 2 with an actionable line on failure):** DB reachable;
|
||||
the catalog holds ≥2 documents; the FIRST TWO catalog documents' `path`s each
|
||||
contain `/` (the bare-path traps need nested paths); `settings.agent_max_rounds
|
||||
> 0` (the gate needs tools enabled).
|
||||
- **Mirror the grounded path of `app/api/chat.py` exactly** (same prompt the UI
|
||||
gets): per question — embed it, `retrieve`, `select_documents`, steering notes
|
||||
+ KB overview as chat.py reads them, `build_high_prompt(docs, notes, kb_overview)`,
|
||||
then `run_agent(llm, db, system_prompt=…, user_message=…, seed_docs=docs,
|
||||
settings=settings, holder=AgentHolder())` with a **fresh** `AgentHolder` per
|
||||
turn, consuming every piece to the end. Never modify the KB.
|
||||
- **Fixed question battery** (locked — the executor may not swap in easier
|
||||
questions). Let the first two catalog documents be
|
||||
`D1 = (s1, p1, t1)` and `D2 = (s2, p2, t2)`, and `token` = the first
|
||||
whitespace-split word of `D2.content` with length ≥ 6 (strip leading/trailing
|
||||
non-alphanumerics, lowercase; fallback: the first word of `t2`):
|
||||
1. `List the files in this directory.` (the incident)
|
||||
2. `List the documents you have in the {s1} source.`
|
||||
3. `List every document you have indexed.`
|
||||
4. `What does the document {p1} contain? Open it and tell me.` (bare-path `read` trap)
|
||||
5. `Read {s1}/{p1} and summarize it.` (combined form — the correct shape)
|
||||
6. `Open the document {p2} and tell me what it covers.` (bare-path `read` trap)
|
||||
7. `Find the exact string "{token}" in your documents and tell me which ones contain it.` (`grep`)
|
||||
8. `Which document has the title "{t2}"? Read it and summarize.`
|
||||
9. `What do you know about {t1}? Open the relevant document and give me specifics.`
|
||||
10. `List the files in the {s2} directory.` (source name phrased as a directory)
|
||||
- **Per-turn measurement** (from the consumed stream + the holder — no app-code
|
||||
changes for measurement): `emitted` = count of yielded `ToolCallPiece`s;
|
||||
`executed` = `holder.tool_calls` (refusals count in nothing); `rejected` =
|
||||
`emitted − executed`; `cap_reached` = `emitted >= settings.agent_max_rounds`
|
||||
(every capped round emitted a call, so the cap implies at that many emissions
|
||||
and never the reverse); `answered` = the stream finished without
|
||||
`LLMError`/`MalformedReplyError`. Print one line per turn:
|
||||
`turn 04 | emitted=2 executed=1 cap=no | What does the document …`.
|
||||
- **Verdict + pass conditions (locked):**
|
||||
1. all 10 turns `answered`;
|
||||
2. zero `cap_reached` turns (the incident's loop signature — hitting the cap
|
||||
means the teaching did not end the loop);
|
||||
3. ≥6 of 10 turns with `emitted ≥ 1` (the model keeps USING tools — it does not
|
||||
abandon them and answer from seed context alone, the incident's end state);
|
||||
4. `executed / emitted ≥ 0.90` across the whole run (the "sufficient number of
|
||||
tool calls succeed" bar; a run with zero emitted calls fails condition 3
|
||||
anyway).
|
||||
Print the single verdict line in a stable format, e.g.
|
||||
`gate: lite PASS turns=10 answered=10 caps=0 tool-turns=8 calls 21/23 executed (91%) 2026-09-03`
|
||||
(model = `settings.llm_chat_model`, date = run date). **Exit 0 on PASS, 1 on
|
||||
FAIL, 2 on precondition failure.**
|
||||
- On FAIL, also print a short per-condition breakdown (which condition(s) missed)
|
||||
so the iteration loop can target the right lever. For refusal diagnosis, each
|
||||
call is already logged by `run_agent` (`agent tool=… args=… round=…/…`) —
|
||||
correlate the logged arguments with the refusal templates in
|
||||
`app/rag/agent.py` to see which teaching line the model hit.
|
||||
2. **Run the gate and iterate until it PASSES** (the loop this task exists for):
|
||||
`podman compose up -d db` → `uv run python -m scripts.agent_realmodel_check`.
|
||||
On FAIL: change ONLY the copy levers this phase owns — the refusal templates
|
||||
(task 01/02 constants), the `AGENT_TOOLS` `path` descriptions (task 01/02),
|
||||
`TOOLS_SECTION` (task 03) — with their unit pins updated to follow the constants;
|
||||
`uv run pytest` green again; re-run the gate. Repeat. **Forbidden:** lowering any
|
||||
threshold, swapping questions, disabling a tool, or weakening condition 4 to make
|
||||
it pass. If the gate still fails after a genuine iteration (the numbers stop
|
||||
improving across levers), STOP: no commit — report the per-turn lines, the
|
||||
verdict, and which refusals the model hit (from the `app.agent` log) in the task
|
||||
report for the owner (fail-loud house style).
|
||||
3. **Record the verdict** (house precedent — the phase-37 probe verdict lives in the
|
||||
`app/rag/agent.py` module docstring): append one line to that docstring —
|
||||
`Real-model gate (phase 72, task 05 — live vs the configured chat model):
|
||||
<the verdict line, verbatim>`.
|
||||
4. **Full gates + commit:**
|
||||
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean.
|
||||
- E2E in isolation (DB up): `test_tool_path_teaching.py`, then the regression
|
||||
suites `test_harness_aligned_tools.py`, `test_agent_document_tools.py`,
|
||||
`test_agent_unlimited_tools.py`, `test_search_tool.py`, `test_chat_rag.py`.
|
||||
- Move the phase directory: `mv .agent/phases/todo/72_teaching_refusals
|
||||
.agent/phases/complete/`.
|
||||
- Commit — one, `--no-gpg-sign`, the Commit block of `00_phase.md`: the title
|
||||
message plus a **body line carrying the gate verdict verbatim**. The commit
|
||||
also carries the still-uncommitted phase-71 `todo/` → `complete/` move and
|
||||
`.agent/reports/71_scaffolding_guardrails/` (`.agent/` is tracked — AGENTS.md
|
||||
§8; `git add -A .agent/ app/ tests/ scripts/` picks up everything).
|
||||
|
||||
## Testing & Quality
|
||||
- The script IS the test for this task: it is deterministic in its question set,
|
||||
thresholds, and output format (a future executor re-running it gets comparable
|
||||
numbers); its precondition failures exit 2 with actionable text. The script itself
|
||||
needs no unit tests (it is an entrypoint probe, `scripts/llm_probe.py` precedent),
|
||||
but the copy iterations it drives must keep `uv run pytest` + coverage >90% green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run python -m scripts.agent_realmodel_check` exits **0** against the live
|
||||
endpoint (all four pass conditions met with the configured model — `lite`);
|
||||
the verdict line verbatim in `app/rag/agent.py`'s module docstring
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
TOTAL **>90%**; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` green in
|
||||
isolation; the regression E2E suites green in isolation (the Work-4 list)
|
||||
- [ ] One `--no-gpg-sign` commit whose body carries the gate verdict; the phase
|
||||
directory under `.agent/phases/complete/72_teaching_refusals/`
|
||||
@@ -0,0 +1,20 @@
|
||||
All criteria verified green; no defects found, no fixes needed.
|
||||
|
||||
## Phase 70 — Harness-Aligned Tools: verification report
|
||||
|
||||
**Verified (final pass — all 5 tasks already complete in `complete/`):**
|
||||
- `AGENT_TOOLS` = exactly `ls`(path?) / `read`(path) / `grep`(pattern, path?); `rg "list_documents|read_document|search_documents" app/` → no matches
|
||||
- `read` splits combined `source/path` at first slash, full content (A7); `grep` locked A5 (fixed substring, case-insens., 20×200, locator-only); `ls` prints `source: X | path: Y | title: Z`
|
||||
- SSE `tool` frames: new names; `argument` = the single string passed (`grep`→pattern, else path) or null
|
||||
- Kill switch (`MAX_ROUNDS=0` → one `tools=None` request) and LOW/deflected byte-identity pinned by unit tests and green
|
||||
- `TOOLS_SECTION` rewritten for the new surface; frontend `read`/`grep`/`ls` render (old names still render for persisted chats); README §"Agent document tools (ls + read + grep)" updated
|
||||
|
||||
**Gates (exact commands):**
|
||||
- `uv run pytest --cov=app` → exit 0, all pass; TOTAL coverage **99%** (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors"
|
||||
- `uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov` → **3 passed** (ls→read, grep→read, wire-argument rule)
|
||||
- Isolated regressions: `test_agent_document_tools` **4 passed**, `test_agent_unlimited_tools` **4 passed**, `test_search_tool` **3 passed**, `test_chat_rag` **3 passed**
|
||||
- Commit `801639e` exists with the exact Commit-block message; phase dir in `.agent/phases/complete/`
|
||||
|
||||
**Deviations:** none — no code changes required this pass.
|
||||
**Next pending phase:** `71_scaffolding_guardrails` (in `todo/`).
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
All task-01 gates green. Final report:
|
||||
|
||||
**Task 01 complete — `ls` / `read(path)` / `grep(pattern, path?)` tool surface in `app/rag/agent.py`**
|
||||
|
||||
- `AGENT_TOOLS` → exactly `ls` (optional `path`), `read` (required `path`), `grep` (required `pattern`, optional `path`) with the task's pinned descriptions; old names gone from `agent.py` (verified `rg` → no matches)
|
||||
- `_execute_tool`: `read`/`grep` take the canonical combined `source/path` (first-slash split via new `_resolve_path`; bare source name → no-document refusal, no DB); `ls` scoping by registered source name (`No source named '…'` refusal; `0 documents:` counted); phase-68 A5 grep contract + refusals/round-cap/kill-switch unchanged; split-teaching branches deleted
|
||||
- New module-level accessor `list_source_names` (registry via `effective_sources` + phase-69 `resolve_source_name`) — the catalog can't distinguish "known source, 0 docs" from "unknown source", the registry can
|
||||
- Tests: `tests/unit/test_agent.py` rewritten (57 tests, new schema/ls-scope/read/grep pins); `tests/integration/test_agent_tools.py` updated to the new surface incl. real-DB scoped-`ls` registry tests
|
||||
|
||||
**Gates:** `pytest tests/unit/test_agent.py --no-cov` → 57 passed · `pytest --cov=app -k agent` → 76 passed, `app/rag/agent.py` **100%** · `ruff check .` clean · `pyright` 0 errors · full suite 1379 passed, **2 failed** — both in `tests/integration/test_chat_api.py` (they script the *old* tool names; fixing them requires `app/api/chat.py`'s SSE `argument` re-derivation, which is task 03's explicit work item — left for it, per no-next-task rule)
|
||||
|
||||
**Decisions:** blank/non-string `ls` `path` = omitted (full catalog, not refusal); already-in-context dedupe runs pre-DB (seed docs never hit the DB); no commit (task 05 owns it).
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/70_harness_aligned_tools/02_prompt_section.md`
|
||||
+294
@@ -0,0 +1,294 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 78%]
|
||||
.......................F...F............................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 99%]
|
||||
............. [100%]
|
||||
=================================== FAILURES ===================================
|
||||
__________ test_grounded_turn_streams_tool_frames_and_cites_read_doc ___________
|
||||
|
||||
client = <starlette.testclient.TestClient object at 0x7fc9c0134130>
|
||||
db = <sqlalchemy.orm.session.Session object at 0x7fc9b02627b0>
|
||||
seeded_kb = <test_chat_api.FakeRagLLM object at 0x7fc9b0535190>
|
||||
caplog = <_pytest.logging.LogCaptureFixture object at 0x7fc9b01fcad0>
|
||||
|
||||
def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""(a) Grounded turn with tool calls: the event sequence is
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
Phase 45: the agent loop keeps offering the tools for the whole
|
||||
turn — the round cap (not per-tool budgets) is the bound."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
StreamPiece("thinking", "Let me list what is indexed…"),
|
||||
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
# the answer request still carries the tools (2 rounds < the
|
||||
# default cap of 10); the fake's tool_script is exhausted, so
|
||||
# it falls back to the thinking + answer stream
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
types = [f["type"] for f in frames]
|
||||
assert types[0] == "thinking"
|
||||
assert types[1] == "tool" and types[2] == "tool" # the two executed calls
|
||||
assert "error" not in types
|
||||
assert types[3:-1] == ["delta"] * (len(types) - 4) # deltas, then done last
|
||||
assert frames[-1]["type"] == "done"
|
||||
|
||||
list_frame, read_frame = frames[1], frames[2]
|
||||
assert set(list_frame) == {"type", "name", "argument"}
|
||||
assert list_frame["name"] == "list_documents"
|
||||
assert list_frame["argument"] is None # the tool takes no parameters
|
||||
assert set(read_frame) == {"type", "name", "argument"}
|
||||
assert read_frame["name"] == "read_document"
|
||||
assert read_frame["argument"] == "docs/homelab/backups.md"
|
||||
|
||||
deltas = [f for f in frames if f["type"] == "delta"]
|
||||
assert len(deltas) >= 2 # genuinely streamed
|
||||
assert "".join(d["text"] for d in deltas) == scripted.answer
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# done.sources = the retrieval docs + the read doc, deduped, order kept.
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
> assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
E AssertionError: assert ('docs', 'hom...es/deploy.j2') == ('docs', 'homelab/backups.md')
|
||||
E
|
||||
E At index 1 diff: 'homelab/templates/deploy.j2' != 'homelab/backups.md'
|
||||
E Use -v to get more diff
|
||||
|
||||
tests/integration/test_chat_api.py:597: AssertionError
|
||||
---------------------------- Captured stdout setup -----------------------------
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=deployments/new-service.md chunks=4
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/backups.md chunks=5
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/tables.md chunks=2
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
2026-09-03 09:53:07 INFO app.importer :: import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
------------------------------ Captured log setup ------------------------------
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=deployments/new-service.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/backups.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/tables.md chunks=2
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
INFO app.importer:importer.py:103 import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
2026-09-03 09:53:07 INFO app.agent :: agent tool=list_documents args={} round=1/10
|
||||
2026-09-03 09:53:07 INFO app.agent :: agent tool=read_document args={"source": "docs", "path": "homelab/backups.md"} round=2/10
|
||||
2026-09-03 09:53:07 INFO app.chat :: question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=28 tool_calls=0 total_ms=11 retries=0
|
||||
------------------------------ Captured log call -------------------------------
|
||||
INFO app.agent:agent.py:527 agent tool=list_documents args={} round=1/10
|
||||
INFO app.agent:agent.py:527 agent tool=read_document args={"source": "docs", "path": "homelab/backups.md"} round=2/10
|
||||
INFO app.chat:chat.py:506 question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=28 tool_calls=0 total_ms=11 retries=0
|
||||
______________ test_tool_execution_db_failure_yields_error_event _______________
|
||||
|
||||
client = <starlette.testclient.TestClient object at 0x7fc9b0eea0b0>
|
||||
db = <sqlalchemy.orm.session.Session object at 0x7fc9b02d27b0>
|
||||
seeded_kb = <test_chat_api.FakeRagLLM object at 0x7fc9b0535610>
|
||||
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc9c35c3d20>
|
||||
|
||||
def test_tool_execution_db_failure_yields_error_event(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A tool call that hits a dead DB mid-stream gets the same structured
|
||||
``error`` event as the pre-stream retrieval path — never a severed
|
||||
stream (the "never stale" contract, PLAN §7.4)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[[ToolCallPiece(id="call_1", name="list_documents", arguments={})]]
|
||||
)
|
||||
|
||||
def boom(*_a: Any, **_k: Any) -> Any:
|
||||
raise RuntimeError("db exploded mid tool call")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", boom)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
# The ``tool`` frame went out first (the model requested the call);
|
||||
# the failed execution ends the turn with the structured error event.
|
||||
> assert [f["type"] for f in frames] == ["tool", "error"]
|
||||
E AssertionError: assert ['tool', 'del... 'delta', ...] == ['tool', 'error']
|
||||
E
|
||||
E At index 1 diff: 'delta' != 'error'
|
||||
E Left contains 5 more items, first extra item: 'delta'
|
||||
E Use -v to get more diff
|
||||
|
||||
tests/integration/test_chat_api.py:817: AssertionError
|
||||
---------------------------- Captured stdout setup -----------------------------
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=deployments/new-service.md chunks=4
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/backups.md chunks=5
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/tables.md chunks=2
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
2026-09-03 09:53:08 INFO app.importer :: import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
------------------------------ Captured log setup ------------------------------
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=deployments/new-service.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/backups.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab-compose.yaml chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/container_gitlab/gitlab-compose.yaml chars=70
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/container_gitlab/gitlab.md chunks=4
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/kubernetes.md chunks=5
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/networking/static-dns.json chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/networking/static-dns.json chars=60
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/cache.volume chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/cache.volume chars=54
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/compose.container chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/compose.container chars=59
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/quadlet/lan.network chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/quadlet/lan.network chars=53
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/scripts/uptime_probe.py chunks=2
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/scripts/uptime_probe.py chars=65
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/ssh/ssh_aliases.txt chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/ssh/ssh_aliases.txt chars=55
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/tables.md chunks=2
|
||||
INFO app.importer:importer.py:322 import: added source=docs path=homelab/templates/deploy.j2 chunks=1
|
||||
INFO app.importer:importer.py:377 import: summary source=docs path=homelab/templates/deploy.j2 chars=54
|
||||
INFO app.importer:importer.py:103 import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 chunks=30 embed_batches=21 summaries=8 summary_errors=0 formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
2026-09-03 09:53:08 INFO app.agent :: agent tool=list_documents args={} round=1/10
|
||||
2026-09-03 09:53:08 INFO app.chat :: question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=0 tool_calls=0 total_ms=10 retries=0
|
||||
------------------------------ Captured log call -------------------------------
|
||||
INFO app.agent:agent.py:527 agent tool=list_documents args={} round=1/10
|
||||
INFO app.chat:chat.py:506 question='How is my Kubernetes cluster set up?' embed_ms=0 top_score=0.436 fts_hits=4 summary_hits=1 tuning=0 kb_chars=0 threshold=0.30 deflected=False sources=['docs/homelab/kubernetes.md', 'docs/homelab/templates/deploy.j2'] thinking_chars=0 tool_calls=0 total_ms=10 retries=0
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 153 4 97%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2726 16 99%
|
||||
=========================== short test summary info ============================
|
||||
FAILED tests/integration/test_chat_api.py::test_grounded_turn_streams_tool_frames_and_cites_read_doc
|
||||
FAILED tests/integration/test_chat_api.py::test_tool_execution_db_failure_yields_error_event
|
||||
tests FAILED: unit + integration suite did not pass
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation FAILED (see output above)
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
All gates green — full suite included. Final report:
|
||||
|
||||
**Fixed:** the harness gate runs the full unit+integration suite, so I resolved the two `test_chat_api.py` failures that the phase plans queue for task 03 — the minimal coupled fix, no further task-03 work:
|
||||
- `app/api/chat.py`: SSE `tool`-frame `argument` re-derived for the new surface — `read`'s `path` (combined `source/path`), `grep`'s `pattern`, `ls`'s `path`, else/null for non-strings or unknown tools; module docstring + inline comments updated (old names now gone from `chat.py` too)
|
||||
- `tests/integration/test_chat_api.py`: 5 tests moved to the new surface — `ls`+`read(path=…)` frame test (`argument` == the single string the model passed), search-frame test renamed to `grep` (raw pattern / non-string → null), DB-failure test scripted as `ls` (asserts `name == "ls"`, `["tool","error"]`), deflected + kill-switch scripts renamed
|
||||
|
||||
**Results:** `uv run pytest` → **1381 passed, 0 failed** · `--cov=app` TOTAL **99%** (`app/rag/agent.py` 100%, `app/api/chat.py` 100%) · `uv run pytest tests/unit/test_agent.py --no-cov` → 57 passed · `uv run pytest --cov=app -k agent` → 76 passed · `uv run ruff check .` clean · `uv run pyright` 0 errors · `rg` old names in `app/rag/agent.py` → no matches
|
||||
|
||||
**Decisions:** remaining old-name test references (chats_api, sse_events, mock_tool_flow, frontend_tool_states, llm_*) all pass — they pin opaque data or files owned by tasks 02/04/05, left untouched; deflected path + kill switch byte-identical (pinned).
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/70_harness_aligned_tools/02_prompt_section.md`
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 99%]
|
||||
............. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
Task 02 complete — all gates green.
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/prompts.py`: rewrote `TOOLS_SECTION` copy for the harness-aligned `ls` / `read(path)` / `grep(pattern, path?)` shapes (catalog-line format, case-insensitive locator contract, combined `source/path` + full content, answer-when-ready close — no budget re-statement, phase 45); updated the module docstring and section comment (HIGH-only, LOW byte-identical, mock keys on `<tools>` marker presence); `build_high_prompt`/`build_deflect_prompt` and section order untouched
|
||||
- `tests/unit/test_prompts.py`: new phase-70 pin block (markers, shape copy pins, old names + budget copy absent, HIGH still ends with `<tools>`), explicit no-`<tools>`/no-new-copy LOW assertions
|
||||
- `tests/unit/test_chat_gate.py`: endpoint pins updated (new names in HIGH system prompt, old names out; new copy absent from the deflected path)
|
||||
- `tests/unit/test_agent.py`: re-pinned the old `TOOLS_SECTION` copy pins (the in-file NOTE delegated this re-pin to task 02)
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v --no-cov` → 58 passed
|
||||
- `rg "list_documents|read_document|search_documents" app/rag/prompts.py` → no matches
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 1385 passed; `app/rag/prompts.py` 100% (>90% gate), TOTAL 99%
|
||||
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors
|
||||
|
||||
**Decisions**
|
||||
- E2E/mock still on old names — expected mid-phase; task 05 owns the mock + E2E updates (default `testpaths` excludes E2E; mock keys on the `<tools>` marker only, so keying is intact)
|
||||
- No PLAN.md exists in `.agent/` (referenced by AGENTS.md); the phase overview was treated as binding context
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/70_harness_aligned_tools/03_api_sse.md`
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,16 @@
|
||||
All gates green. Task 03 is complete.
|
||||
|
||||
**Task 03 — SSE `tool` frames + API contract copy: done**
|
||||
|
||||
- `app/api/chat.py`: `ToolCallPiece` → SSE derivation now symmetric — `argument` = the single string the model passed (`read`/`ls` → `path`, `grep` → `pattern`), non-string/omitted → null; module docstring carries the phase-70 note, the `{"type":"tool",…}` shape, and the `done.sources`/`query_log`/`tool_calls=N` contract (verified in place; branch was already rewritten by the earlier interrupted run of this task).
|
||||
- `app/schemas.py`: `ChatToolEvent` docstring re-pointed at `ls`/`read`/`grep` + the argument rule (no field change); `ToolCall` docstring updated incl. the locked no-migration note for legacy saved names.
|
||||
- `tests/integration/test_chat_api.py`: new pin `test_tool_frames_carry_the_model_arguments_regardless_of_execution` — scoped `ls` frame carries `"docs"`; a rejected `read` (unknown path) still emits its frame with the model's argument as passed; `done.sources` stays retrieval-only; log line `tool_calls=1` (rejected calls count nothing). Existing pins: `read` frame = combined path as passed, `grep` frame = pattern, unscoped `ls` = null.
|
||||
- `tests/integration/test_chats_api.py`: saved-chat fixture on new names/args; keeps one `read_document` entry pinning that pre-phase-70 chats still validate (locked, no migration).
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/integration/test_agent_tools.py tests/integration/test_chat_api.py -v --no-cov` → 40 passed
|
||||
- `uv run pytest tests/integration/test_chats_api.py --no-cov` → 54 passed; full `uv run pytest` → green (exit 0)
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → `app/api/chat.py` 100%, `app/schemas.py` 100%, TOTAL 99% (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||
|
||||
**Notes**: `tests/integration/test_api.py` old-name mentions are app.js asset pins — left for task 04 (frontend). Legacy docstring names in `app/rag/llm.py` and the `docs.py` route function name fall to the phase-level `rg` gate in task 05. Next pending task: `04_frontend.md`.
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
.................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,11 @@
|
||||
All green. The other modified files are uncommitted work from completed tasks 01–03 (the commit belongs to task 05 per the phase plan).
|
||||
|
||||
**Task 04 report**
|
||||
|
||||
- `frontend/assets/app.js`: `appendToolLine` now keys on `read`/`grep`/`ls` (new copy: "📄 Reading", "🔎 Searching for", "🔎 Listing documents in <scope>", unscoped "🔎 Listing documents") with legacy `read_document`/`search_documents`/`list_documents` branches preserved byte-identical for persisted turns; status-label ternary in the turn handler extended to the new names (incl. scoped `ls`); section + module docstrings updated with the phase-70 note.
|
||||
- `frontend/assets/shared.js`: `addToolLines` local copy got the same new-name branches (read/grep/ls + scoped ls) with the legacy branches kept; all arguments `textContent`-only; docstring updated.
|
||||
- `tests/unit/test_frontend_tool_states.py`: pins updated/added — new+legacy branch gates in app.js, new shared.js pin test (`test_shared_page_tool_lines_cover_new_and_legacy_names`), no-`innerHTML` textContent discipline; old copy pins kept.
|
||||
- Ripple fix: `tests/integration/test_api.py` emoji guard now strips the fourth tool-line literal `"🔎 Listing documents in "`.
|
||||
- Tests: `uv run pytest tests/unit/test_frontend_tool_states.py -v --no-cov` → 10 passed; `uv run pytest --cov=app --cov-report=term-missing` → 1387 passed, TOTAL 99% (>90%); `uv run ruff check . && uv run pyright` → clean; both JS files pass `node --check`.
|
||||
- Decisions: explicit if-chains (not a name map) for the two-generation gates; scoped-`ls` status label added so the live region mirrors the new line copy (PLAN §7.4). E2E suites remain keyed on the old mock names — task 05's scope.
|
||||
- Next pending task: `.agent/phases/todo/70_harness_aligned_tools/05_mock_e2e_commit.md`.
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
Done — the empty `.err` is a harness runtime artifact (tracked-report convention, same as prior phases); working tree is clean.
|
||||
|
||||
**Phase 70, task 05 complete**
|
||||
|
||||
- `tests/e2e/mock_llm.py`: deterministic flows now emit `ls` (no args, `call_0`), `read` with the **combined** `source/path` (mock joins the two catalog fields — only parse change), `grep` with `pattern`; docstrings + phase-70 notes updated
|
||||
- Existing E2E suites updated to new frames: `test_agent_document_tools.py` (`ls`→`read`), `test_agent_unlimited_tools.py` (multi-read `ls`+2×`read`), `test_search_tool.py` (`grep` + pattern argument)
|
||||
- NEW `tests/e2e/test_harness_aligned_tools.py` (3 tests): READ flow (ls line → 📄 Reading line → quoted answer → done.sources incl. read doc → no raw `<|…|>`/`tool_call` markup in DOM), SEARCH flow (🔎 Searching for sentinel → answer), wire argument rule across both flows (ls→null, read→combined path, grep→pattern; names ⊆ {ls, read, grep})
|
||||
- README agent-tools section rewritten for `ls`/`read(path)`/`grep(pattern, path?)` (combined identity + A5 locator semantics); `.env.example` header comment refreshed
|
||||
- Old-name cleanup so the rg criterion holds: `app/api/docs.py` route fn → `list_indexed_documents`, `app/rag/llm.py` docstrings, `test_llm_client.py`, `test_llm_stream_teardown.py`, `test_sse_events.py`, `test_mock_tool_flow.py`, `test_api.py` docstring. Remaining matches: only the explicitly-marked legacy frontend branches/pins + the marked legacy-chat persistence pin + the mandated "old names absent" regression pins from tasks 01–02
|
||||
|
||||
**Gates (all green):**
|
||||
- `uv run pytest` → exit 0; `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||
- E2E in isolation: `test_harness_aligned_tools` 3 passed, `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_search_tool` 3, `test_chat_rag` 3
|
||||
|
||||
**Decisions:** mock `_tool_flow` tuple kept `(source, path)`; the join happens at the wire-emission point (keeps `test_mock_tool_flow.py` pins valid). `.env.example` included in the commit for atomicity (not in the block's path list, but task 05 touched it). Phase-dir move left to the harness per executor rules.
|
||||
|
||||
**Commit:** `801639e feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)` (55 files, `--no-gpg-sign`)
|
||||
|
||||
**Next pending:** `71_scaffolding_guardrails/01_filter.md`
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2722 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,13 @@
|
||||
**Phase 71 final verification pass — all green, no code fixes needed.**
|
||||
|
||||
- Verified: `app/rag/scaffolding.py` pure module (`import re` only); filter strips spans split at every boundary, mixed streams emit clean remainder, partial token at EOF emitted as-is, look-alikes/lone end token untouched (direct demo — one initial "failure" was a typo in my own check script, not the code)
|
||||
- Verified: mock LLM imports `CORRECTION_INSTRUCTION` from `app.rag.agent` (no drift); log line ends `…retries=%d scaffold_stripped=%d` (chat.py:640)
|
||||
- `uv run pytest --cov=app` — full suite green, TOTAL **99%** (>90%)
|
||||
- `uv run ruff check . && uv run pyright` — clean (0 errors)
|
||||
- `bash .agent/validate.sh` — `validation OK`
|
||||
- `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov` — 3 passed (recovery, terminal, no-false-positive)
|
||||
- Regression, isolated: `test_harness_aligned_tools.py` 3 passed, `test_chat_rag.py` 3 passed, `test_agent_document_tools.py` 4 passed
|
||||
|
||||
**Completion criteria:** all 7 met — no `tool_call_start` in `frontend/`; pure-scaffolding → zero deltas / mixed → clean / thinking raw; one recovery (`tools=None` + correction line), terminal copy pinned; uniform `scaffold_stripped` field; all gates green; commit `575d6c8` (exact message, unsigned) exists; tasks 01–05 in `complete/` (harness's pending worktree move of `05` left untouched per rules; `00_phase.md` move is the harness's on success).
|
||||
|
||||
**Next pending phase:** none — 71 is the only phase in `todo/`.
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 97%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 181 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2849 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,15 @@
|
||||
Task complete — all criteria green.
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/scaffolding.py`: pure module (`import re` only) with `SCAFFOLD_PATTERNS` (observed span + 2 standalone siblings, non-greedy span) and `ScaffoldingFilter` — streaming state machine: `feed()` strips leftmost-complete matches repeatedly, holds the live tail (open span or longest proper-prefix suffix of the 3 opening literals), `flush()` emits the tail as-is (pinned EOF choice), `stripped_chars` read-only property.
|
||||
- `tests/unit/test_scaffolding_filter.py`: 61-test matrix — span at every start-token split offset (0..19) + mid-span/end-token splits, multi-span, standalone tokens (alone/embedded/char-by-char), look-alikes verbatim, partial markers at EOF, exact surrounding-text preservation.
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_scaffolding_filter.py -v --no-cov` → 61 passed
|
||||
- `rg "import (os|sys|logging|app)" app/rag/scaffolding.py` → no matches
|
||||
- `uv run pytest --cov=app` → 1448 passed; `app/rag/scaffolding.py` 100%, TOTAL 99% (>90%)
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||
|
||||
**Decisions** — open-span defined as "start with no end token *after* it" (prevents scaffolding leak when a lone end token precedes a start); repeated leftmost-strip loop pins spec behavior where removing one match can expose another. No deviations.
|
||||
|
||||
**Next pending task:** `02_llm_integration.md` (wire filter into `chat_stream`/`chat_stream_retried`).
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 99%]
|
||||
........ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 213 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 50 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2772 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
Task 02 complete. All gates green.
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/llm.py`: `chat_stream(..., scaffolding: ScaffoldingFilter | None = None)` — content deltas fed through the caller-owned filter (empty clean result yields nothing), thinking never filtered, `None` keeps the byte-identical raw path; `flush()` tail is yielded as a content piece **before** tool-call materialization; phase-48 teardown untouched. `chat_stream_retried(..., scaffolding=None)` forwards the same filter to every attempt (safe by construction — a restart only happens pre-piece, i.e. filter never fed).
|
||||
- Tool materialization unified to a single post-loop emission point (the old in-loop `finish_reason="tool_calls"` early emission was removed) so the flushed tail always precedes `ToolCallPiece`s; identical wire behavior for all real/fixture stream shapes.
|
||||
- Tests: 7 new `chat_stream` filter tests + 2 retry pass-through tests in `tests/unit/test_llm_client.py` (span mid-stream, split across chunks, pure-scaffolding → zero pieces, thinking raw, `None` raw pin, flush-before-tools order, abandon-with-filter teardown, filter forwarded to every attempt, unfed-filter reuse).
|
||||
- Added the `scaffolding` kwarg to the five other `chat_stream` fakes (test_agent, test_chat_gate, test_agent_tools, test_chat_api) — required interface maintenance for the retry primitive to forward it.
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_llm_client.py tests/unit/test_llm_stream_teardown.py -v --no-cov` → 60 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 1457 passed; `app/rag/llm.py` 100%, TOTAL 99% (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → clean
|
||||
- E2E regressions in isolation: `test_harness_aligned_tools` (3), `test_agent_document_tools` (4), `test_chat_rag` (3) → all passed
|
||||
|
||||
**Decisions:** single post-loop tool-materialization point (simpler; pins the content-before-tools order at both former emission points); no commit — the phase commit belongs to task 05.
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/71_scaffolding_guardrails/03_recovery_policy.md`
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 98%]
|
||||
................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 149 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 139 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 50 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2775 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
All green. Task 03 is complete.
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/agent.py`: `CORRECTION_INSTRUCTION` (verbatim constant), `MalformedReplyError(LLMError)`, `AgentHolder.scaffold_stripped`; `run_agent` now runs every request (rounds, forced final, recovery) through a fresh `ScaffoldingFilter`, logs one capture warning per stripped span (200-char truncation), and on a scaffolding-only no-calls round runs exactly one `tools=None` recovery with the correction folded into the single system message; a second empty reply raises `MalformedReplyError`
|
||||
- `app/api/chat.py`: deflected path uses a turn filter + extracted `_pump` piece loop, one bounded recovery when the filter wiped the reply, `MalformedReplyError` caught before the generic `LLMError` (dedicated error copy, no `done`/`query_log`), log line ends `…retries=N scaffold_stripped=N`
|
||||
- `app/rag/scaffolding.py`: `stripped_spans` (allowed by task); tests: 10 new unit (grounded matrix) + 3 new integration (deflected matrix), fakes now feed the filter like the real client
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_agent.py tests/integration/test_chat_api.py -v --no-cov` → 94 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → green, TOTAL **99%** (>90%; all touched files 100%)
|
||||
- `uv run ruff check . && uv run pyright` → clean
|
||||
- E2E regressions in isolation: `test_harness_aligned_tools.py` (3 passed), `test_chat_rag.py` + `test_agent_document_tools.py` (7 passed)
|
||||
|
||||
**Decisions**
|
||||
- Task's message formula conflicted with the locked "single system message" decision — implemented the locked one (replace original system msg with folded correction, history intact), pinned by unit tests
|
||||
- Forced-final request is filtered too; scaffolding-only final settles `MalformedReplyError` (no recovery after it — at most one per turn)
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/71_scaffolding_guardrails/04_deflect_prompt.md`
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 97%]
|
||||
.............................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 181 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2849 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
All done. Final report:
|
||||
|
||||
**Task 04 complete — deflection plain-text line (prevention)**
|
||||
|
||||
- `app/rag/prompts.py`: appended "Reply in plain text only — you have no tools in this mode." as its own line to the `DEFLECT_MODE` body (LOW prompt's only change); module + function docstrings carry the phase-71 note (owner-permitted 2026-09-03; marker-keying contract unchanged); stale "LOW byte-identical" claims in the `<tools>` paragraph/comment corrected.
|
||||
- `tests/unit/test_prompts.py`: diff/reconstruction pin (LOW = pre-phase text + exactly the one line, `count == 1`, title list untouched), line-present + marker-present pin, no-leak-into-HIGH pin (all 4 HIGH build variants); both empty-steering/overview byte-identity pins updated to the new LOW baseline.
|
||||
- `tests/unit/test_agent.py`: LOW byte pin updated to include the line. `tests/unit/test_chat_gate.py`: line pinned present in the deflected `plan_turn` prompt and absent from the HIGH prompt.
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v --no-cov` → 61 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 1473 passed, TOTAL 99% (`app/rag/prompts.py` 100%)
|
||||
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||
- `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 3 passed (mock marker-keying intact)
|
||||
|
||||
**Decisions**: the line goes on its own line (the completion criterion's "pre-phase text + exactly one new line" diff pin), not appended to the existing sentence. No commit — commits belong to task 05 (phase 70 + 71/01–03 work is still uncommitted in-tree).
|
||||
|
||||
**Next pending task**: `.agent/phases/todo/71_scaffolding_guardrails/05_e2e_commit.md`
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 97%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 181 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2849 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
Task 05 complete — all gates green, phase committed.
|
||||
|
||||
**Implemented**
|
||||
- `tests/e2e/mock_llm.py`: two deterministic scaffolding flows — `SCAFFOLD_TRIGGER` ("emit raw tool markup": request 1 streams only the incident span `<|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|>` split across ≥2 chunks; request 2 — system prompt carrying `CORRECTION_INSTRUCTION`, imported from `app.rag.agent` with a drift-guard assert — streams the clean recovery answer) and `SCAFFOLD_ALWAYS_TRIGGER` (scaffolding-only on every request, checked first — it contains the other phrase); docstring flow table updated; checked before the search/tool marker flows, independent of the `<tools>` marker
|
||||
- `tests/e2e/test_tool_scaffolding_guardrails.py` (new, house pattern, empty-KB → deterministically deflected, the incident's path): recovery case (bubble shows recovery text, `document.body.innerText` free of `tool_call_start`/`tool_call_end`/`[read(path=`/`<|`, no error banner, wire: no delta frame carries any fragment, concatenated deltas == recovery answer exactly, `done` deflected, 1 query_log row); terminal case (role=alert banner with "The model returned a malformed reply — please try again.", zero brain bubbles, error frame last, no done, no deltas, no query_log row, then a follow-up plain question answers normally and the banner clears); no-false-positive case (deltas byte-equal the mock's own deflection answer, no recovery/error/retry frames)
|
||||
|
||||
**Results** (exact commands)
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → all pass, TOTAL **99%** (>90%); phase-71 modules 100%
|
||||
- `uv run ruff check . && uv run pyright` → clean
|
||||
- `uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov` → 3 passed; regressions in isolation: `test_harness_aligned_tools.py` 3 passed, `test_chat_rag.py` 3 passed, `test_agent_document_tools.py` 4 passed
|
||||
- `rg "tool_call_start|tool_call_end" frontend/` → no matches
|
||||
- Commit `575d6c8` `feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery` (38 files, `--no-gpg-sign`; also swept in phase 70's pending todo→complete move via the prescribed `git add -A .agent/`)
|
||||
|
||||
**Decisions**: per harness rules I did not move `00_phase.md`/`05_e2e_commit.md` out of `todo/` (harness moves them on success — same as the phase-70 precedent); the span is 77 chars (7 wire chunks). No next pending task — phase 71 was the last phase in `todo/`.
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 97%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 181 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2849 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,12 @@
|
||||
All verification complete. Final state is coherent: the gate remains the sole failing criterion, and per the owner-locked fail-loud protocol the phase stops without a commit.
|
||||
|
||||
## Phase 72 — Final Verification Pass Report
|
||||
|
||||
- **Verified green:** tasks 01–04 work (teaching `ls` refusals, `read`/`grep` "did you mean" suggestions, `TOOLS_SECTION`/description contract copy, mock E2E) — all pins intact; `AGENT_TOOLS` names exactly `ls`/`read`/`grep`; LOW prompt byte-identical; KB re-imported (361 docs) and gate-runnable.
|
||||
- **Iterated the gate's copy levers 3× (this pass):** instructive `ALREADY_IN_CONTEXT` refusal (behavior unchanged; killed the repeat-call loop in runs 3–4), `grep` "pass ONLY `pattern`" + "Rarely needed" descriptions, batched-call consequence clause in `TOOLS_SECTION`. Tried + **reverted** a `<document>`-anchor/`<documents>`-intro variant: it never stopped seed-doc reads and regressed the incident turn (cap-reached loop, run 4).
|
||||
- **Gate result — FAIL (pass condition):** final run `gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 9/21 executed (43%) 2026-09-04`. Cumulative across 3 task-05 attempts: 12 live runs, 39–57% executed; condition 4 (≥90%) never met. Root cause (15/15 instances): `lite` obeys the user's "open it" and `read`s seed-context docs; also first-call source-name-scoped `grep`. Copy-resistant — a model-capability limit, not a copy lever.
|
||||
- **STOP per fail-loud protocol: NO commit, no verdict line** (matches attempts a1/a2, `.agent/reports/72_teaching_refusals/`).
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **1489 passed, TOTAL 99%** (>90% ✓); `uv run ruff check . && uv run pyright` → clean ✓.
|
||||
- E2E in isolation: `test_tool_path_teaching.py` 2 passed ✓; regressions `test_harness_aligned_tools`/`test_agent_document_tools`/`test_agent_unlimited_tools`/`test_search_tool`/`test_chat_rag` all green ✓.
|
||||
- Completion criteria 1–6 ✓; **criterion 7 (real-model gate) ✗ → criterion 8 (commit) blocked** — phase stays in `todo/`.
|
||||
- **Owner decision needed:** model upgrade (or unlock the ≥90% bar) — the locked `lite` + copy-only levers cannot pass this gate. Next pending phase: none (72 is the only `todo/` entry).
|
||||
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 96%]
|
||||
................................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 200 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2868 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
Task 01 is complete — all gates green.
|
||||
|
||||
## Report
|
||||
|
||||
**Implemented (task 01 — `ls` teaching refusals):**
|
||||
- `app/rag/agent.py`: added `LS_PATH_NOT_A_SOURCE` (stripped scope contains `/` — "looks like a document path, not a source name…") and `NO_SOURCE_NOT_A_DIRECTORY` (unknown source, no `/` — old line's prefix kept byte-identical + teaching parenthetical); `_execute_tool` `ls` branch uses them; both count in nothing and consume a round like today's refusal; `AGENT_TOOLS` `ls.path` description now says "a source name, not a file or directory path"; module docstring point 3 updated to the two new lines
|
||||
- `tests/unit/test_agent.py`: byte-for-byte template pins; new pins for `ls(path='app/rag/importer.py')` (echoed, `tool_calls == 0`, tools stay offered, no registry lookup) and `ls(path='.')`; the `Ghost` pin replaced with the extended line; description pin updated
|
||||
- `tests/integration/test_agent_tools.py`: `Ghost` pin updated to the constant; new path-like-scope case through `run_agent` on real Postgres (`kb`/`src`)
|
||||
|
||||
**Test / lint / coverage:**
|
||||
- `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov` → 86 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 1476 passed, TOTAL **99%** (agent.py 100%)
|
||||
- `uv run ruff check . && uv run pyright` → clean
|
||||
- E2E regression spot-checks (isolation): `test_harness_aligned_tools`, `test_agent_document_tools`, `test_agent_unlimited_tools`, `test_search_tool`, `test_chat_rag` → all passed
|
||||
|
||||
**Notable decisions:** kept a `startswith` unit pin on the old prefix (the task's locked requirement that it stay byte-identical); the terse line no longer exists as a refusal or expected value anywhere in `app/`/`tests/`. Path-like scopes refuse before any registry lookup (boom-pinned). Tool names/argument shapes unchanged (`ls`/`read`/`grep`).
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/72_teaching_refusals/02_read_grep_path_suggestion.md` (not started, per protocol).
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 97%]
|
||||
.................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 185 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2853 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
Everything is green and the diff is coherent. Task 02 is complete.
|
||||
|
||||
## Report — Task 02: `read`/`grep` "did you mean 'source/path'?" suggestions
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/agent.py`: `SUGGESTION_LIMIT = 3`; module-level `find_path_candidates` (exact-`path` or `/suffix` match, case-sensitive, catalog order, one bulk `all_documents` query, refusal-path only); `NO_DOCUMENT_DID_YOU_MEAN` / `NO_DOCUMENT_DID_YOU_MEAN_MANY` templates + `_no_document_refusal` helper wired into both the `read` and scoped-`grep` miss paths (zero-candidate and bare-name refusals byte-identical, no DB lookup for bare names); `read`/`grep` `path` descriptions append the bare-path contract; module docstring point 3 updated.
|
||||
- `tests/unit/test_agent.py`: +7 new pins (exact/suffix match, two-source "one of", four-source cap-3, bare-name & bare-filename no-DB-lookup `_boom` guards, seed-doc dedupe-precedence, scoped-grep suggestion) + description/constant pins updated byte-for-byte; 3 existing pins adjusted for the now-running zero-candidate lookup.
|
||||
- `tests/integration/test_agent_tools.py`: 2 new end-to-end `run_agent` cases on real Postgres (bare path under one source; same path under two sources) — refusal followed by the scripted corrected combined-identity call succeeding; existing refusal pins verified unchanged (none match fixture `path`s).
|
||||
|
||||
**Results**
|
||||
- `uv run pytest tests/unit/test_agent.py tests/integration/test_agent_tools.py -v --no-cov` → 96 passed (DB up)
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 1486 passed, TOTAL **99%** (>90%; `agent.py` 100%)
|
||||
- `uv run ruff check . && uv run pyright` → clean
|
||||
- E2E regressions in isolation: `test_harness_aligned_tools` + `test_agent_document_tools` (7), `test_agent_unlimited_tools` + `test_search_tool` + `test_chat_rag` (10) — all green
|
||||
|
||||
**Notable deviation**: the task's example bullets (`read('caddy.md')`, `'A/x.md'`) imply suggestions for slash-less args, but the task's normative spec (stated twice) plus its own completion criterion gate the lookup on `"/" in arg` and pin zero DB calls for bare names. I implemented the `/` gate and adapted the suffix/multi-source examples to slash-containing paths; a bare filename (`caddy.md`) is pinned to keep today's no-DB refusal — flag for owner if bare filenames should also be suggested.
|
||||
|
||||
**Next pending task**: `.agent/phases/todo/72_teaching_refusals/03_prompt_copy.md`
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 96%]
|
||||
.............................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 200 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2868 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,17 @@
|
||||
Task 03 complete — all gates green.
|
||||
|
||||
**Implemented**
|
||||
- Rewrote `TOOLS_SECTION` in `app/rag/prompts.py`: `ls` clause now states its optional `path` is a source name (e.g. `'homelab'`), **not** a directory or file path, omit to list all; `read` clause requires the combined `source/path` exactly as shown in `ls` output — *including the source name*, bare path will not resolve; `grep` keeps its locator copy plus the same bare-path note. Single paragraph, `<tools>`/`</tools>` shape, HIGH-only, same as before.
|
||||
- Updated module/constant/`build_high_prompt` docstrings to document the phase-72 contract.
|
||||
- `tests/unit/test_prompts.py`: dropped the stale `"pass a source name as \`path\`"` pin; added `test_tools_section_phase72_contract_clauses` (byte-for-byte pins for the ls source-name clause, the read combined-identity clause, and exactly-two bare-path notes) and `test_tools_section_phase72_clauses_in_high_prompt_not_low`.
|
||||
|
||||
**Test / lint results**
|
||||
- `uv run pytest tests/unit/test_prompts.py -v --no-cov` → 32 passed
|
||||
- `uv run pytest` → 1488 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||
- E2E regression (mock-keying check): `test_harness_aligned_tools` + `test_agent_document_tools` + `test_agent_unlimited_tools` → 11 passed; `test_search_tool` + `test_chat_rag` → 6 passed (isolation)
|
||||
|
||||
**Decisions**: kept all pre-existing substrings the `test_agent.py` HIGH-prompt pins depend on; mirror the exact AGENT_TOOLS description wording ("a bare document path (without the source name) will not resolve") so prompt, descriptions, and refusals state one contract. No deviations.
|
||||
|
||||
**Next pending task**: `.agent/phases/todo/72_teaching_refusals/04_mock_e2e.md` (mock `ls`-misuse flow + dedicated E2E suite).
|
||||
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 96%]
|
||||
................................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 200 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2868 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,17 @@
|
||||
All green. Task 04 complete.
|
||||
|
||||
**Implemented**
|
||||
- `tests/e2e/mock_llm.py` (+142, purely additive): phase-72 `LS_TEACH_TRIGGER` flow — request 1 streams `ls {"path": "."}` (`call_0`, the incident's misuse), request 2 (teaching refusal present, not a catalog listing) streams no-arg `ls()` (`call_1`), request 3 (first tool result line matches `^\d+ documents:`) answers `These are the indexed documents: <first catalog line>` via `_CATALOG_LINE_RE`/new `_CATALOG_HEADER_RE` machinery; checked before the plain `TOOLS_TRIGGER` flow (disjoint triggers); docstring flow table updated.
|
||||
- `tests/e2e/test_tool_path_teaching.py` (new dedicated suite): one-source/two-doc fixture; turn 1 pins self-correction (answer carries first doc's `source:`/`path:`, two UI tool lines, no error banner) and the two-round wire shape (`ls "."` → `ls null`, no third tool frame); turn 2 in the same session pins the plain single-read flow unchanged.
|
||||
|
||||
**Tests / lint / coverage**
|
||||
- `uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov` → 2 passed (isolated, DB up, mock LLM)
|
||||
- Regression suites isolated: `test_harness_aligned_tools` 3✓, `test_agent_document_tools` 4✓, `test_agent_unlimited_tools` 4✓, `test_search_tool` 3✓, `test_chat_rag` 3✓
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||
|
||||
**Decisions**
|
||||
- Fixture: catalog-first doc is catalog-only (no chunks); the *second* doc carries the retrieval chunk — a seed that is also catalog-first makes the follow-up single-read flow dedupe ("Already in your context") and loop to the round cap (caught and fixed on first run).
|
||||
- Fixed a `\d` SyntaxWarning in the mock docstring introduced by the new bullet.
|
||||
|
||||
**Next pending task:** `.agent/phases/todo/72_teaching_refusals/05_real_model_gate.md`
|
||||
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 96%]
|
||||
................................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 200 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2868 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,12 @@
|
||||
All gates verified. The repo is in a fully functional state; the live gate FAILED after 8 genuine iterations, so per the task's fail-loud protocol I stopped **without committing** and without recording a verdict line.
|
||||
|
||||
## Report
|
||||
|
||||
- **Implemented:** `scripts/agent_realmodel_check.py` (house probe pattern — preconditions exit 2, fixed 10-question battery derived from the live catalog, the four locked pass conditions, stable verdict line, exits 0/1/2); a real gate KB (38 genuine homelab docs under the registered source `homelab`, nested paths, from `/tmp/bor-gate-kb`).
|
||||
- **Gate result:** 8 live runs vs `lite` (temp 0.4): **41 / 53 / 39 / 42 / 57 / 47 / 53 / 48 %** executed. Conditions 1–3 passed every run (10/10 answered, 0 caps, 10/10 tool-turns); condition 4 (≥90 % executed) never met. Final verdict: `gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 10/21 executed (48%) 2026-09-03`.
|
||||
- **Refusals the model hit (copy-resistant):** (a) first call of each grep turn = `grep(pattern=…, path='homelab')` — source-name scoping, 8/8 runs; the locked byte-identical `No document at 'homelab' — check the ls output.` can't teach bare args (locked decision) and the model reads it as "no matches in the homelab directory"; (b) reads of docs already in the `<documents>` seed (turns 4/5/9, 8/8 runs) → locked `Already in your context.`; (c) occasional repeated-refused calls / batched multi-call extras.
|
||||
- **Levers exhausted:** TOOLS_SECTION ×6 variants, AGENT_TOOLS function+param descriptions ×4+ (incl. wrong/right examples, rules-first, negative-example-priming removal); the did-you-mean teaching fired (run 3, bare path) and self-corrected in one round — it works; the ls teaching refusals never triggered (model consistently used the registered source).
|
||||
- **Tests/lint/coverage:** `uv run pytest` → 1488 passed; `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99 %** (>90 %); `uv run ruff check . && uv run pyright` → clean.
|
||||
- **E2E in isolation:** `test_tool_path_teaching.py` 2 passed; regressions `test_harness_aligned_tools` 3, `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_search_tool` 3, `test_chat_rag` 3 — all passed.
|
||||
- **Decisions:** STOP per the fail-loud protocol — no commit, no verdict line in `app/rag/agent.py`, tree left at the latest iteration (all pins in sync, everything green); KB re-imported so the gate is re-runnable (re-import after any E2E run: `uv run python -m scripts.import_docs --source /tmp/bor-gate-kb/homelab`).
|
||||
- **Next pending task:** none — task 05 is phase 72's last. The phase stays open for the owner: the remaining gap is a model-capability limit (first-call source-name scoping + seed-doc re-reads) vs the locked ≥90 % bar — a model/lock decision, not a copy lever.
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 96%]
|
||||
................................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 200 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2868 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
@@ -0,0 +1,13 @@
|
||||
All clean — no strays, preconditions pass, repo gate-runnable. Everything is verified green; here is the final report.
|
||||
|
||||
## Report (retry attempt)
|
||||
|
||||
- **What this attempt did:** the previous run's failure was harness plumbing (final report never delivered) — no code failure. Re-verified the complete state, investigated the shared-DB change (an external import of the tracked `tests/fixtures/docs` fixture at 21:46 EDT layered over my 38-doc gate KB → 51-doc catalog), and ran one fresh live gate against the current live catalog.
|
||||
- **Live gate (run 9, 51-doc catalog):** `gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 11/29 executed (38%) 2026-09-03` — condition 4 (≥90 % executed) missed; conditions 1–3 passed. Full dataset (9 runs): 41/53/39/42/57/47/53/48 (38-doc KB) + 38 (mixed). Positive: the no-source teaching refusal self-corrected in exactly one extra round on a real unknown-scope trap (turn 02).
|
||||
- **STOP per the fail-loud protocol:** no commit, no verdict line in `app/rag/agent.py`, phase stays in `todo/`. The gap is a model-capability limit (first-call source-name scoping on `grep`, re-reads of seed-context docs, repeated refused calls, an `ls(path='/')` loop on turn 10), not a copy lever — 8 section/schema variants exhausted.
|
||||
- **Tests:** `uv run pytest` → 1488 passed (exit 0).
|
||||
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99 %** (>90 %).
|
||||
- **Lint:** `uv run ruff check . && uv run pyright` → clean, 0 errors.
|
||||
- **E2E in isolation:** `test_tool_path_teaching` 2 passed; regressions `test_harness_aligned_tools` 3, `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_search_tool` 3, `test_chat_rag` 3 — all passed.
|
||||
- **Repo state:** `scripts/agent_realmodel_check.py` + all pins in sync; KB left gate-runnable (preconditions pass); no task/phase files touched.
|
||||
- **Next pending task:** none — 05 is phase 72's last task; the owner must decide (model upgrade vs. the locked ≥90 % bar) before the gate can pass.
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 96%]
|
||||
................................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
-----------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 22 0 100%
|
||||
app/api/chat.py 176 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 7 0 100%
|
||||
app/api/doc_drafts.py 93 0 100%
|
||||
app/api/docs.py 50 0 100%
|
||||
app/api/git_sources.py 212 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 8 0 100%
|
||||
app/api/sync.py 101 0 100%
|
||||
app/config.py 126 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 20 0 100%
|
||||
app/core/caching.py 108 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/db.py 21 0 100%
|
||||
app/main.py 52 0 100%
|
||||
app/models.py 86 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 200 0 100%
|
||||
app/rag/archive_upload.py 128 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 180 3 98%
|
||||
app/rag/llm.py 216 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 65 0 100%
|
||||
app/rag/retriever.py 94 3 97%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 205 0 100%
|
||||
-----------------------------------------------
|
||||
TOTAL 2868 12 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
validation OK
|
||||
+1
-1
@@ -42,7 +42,7 @@ BOR_HYBRID_VECTOR_CANDIDATES=100 # cosine list width for the fusion
|
||||
BOR_HYBRID_LEXICAL_CANDIDATES=30 # FTS list width for the fusion
|
||||
BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
|
||||
|
||||
# --- Agent document tools (phase 37: grounded turns may list + read) ---
|
||||
# --- Agent document tools (grounded turns may extend context: ls / read / grep) ---
|
||||
# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)
|
||||
|
||||
# --- Import scope (A9 default; ANY well-formed extension is allowed) ---
|
||||
|
||||
@@ -163,28 +163,40 @@ exactly as before.
|
||||
To hide it, set `BOR_STREAM_THINKING=0` — the `thinking` events stop
|
||||
(the per-turn log line still counts `thinking_chars`).
|
||||
|
||||
## Agent document tools (list + read)
|
||||
## Agent document tools (ls + read + grep)
|
||||
|
||||
Retrieval only puts the top documents in context. When an answer depends
|
||||
on a file a note *references* ("the exact JSON shape is in
|
||||
example-record-file.json"), the model can extend its own context with two
|
||||
server-side tools — on **grounded** (high-relevance) turns only:
|
||||
example-record-file.json"), the model can extend its own context with
|
||||
three server-side tools — on **grounded** (high-relevance) turns only.
|
||||
The surface mirrors the shape the chat model was trained on (the pi.dev
|
||||
harness tools, owner decision 2026-09-03), and the canonical document
|
||||
identity everywhere is the combined `source/path` string:
|
||||
|
||||
* **`list_documents`** — lists every indexed document, one
|
||||
`source: X | path: Y | title: Z` line each (the same order as the
|
||||
Sources page);
|
||||
* **`read_document(source, path)`** — appends the **full** text of
|
||||
one more indexed document to the context (never truncated).
|
||||
* **`ls`** — lists every indexed document, one `source: X | path: Y |
|
||||
title: Z` line each (the same order as the Sources page); pass a
|
||||
source name as `path` to list one source's documents;
|
||||
* **`read(path)`** — appends the **full** text of one more indexed
|
||||
document to the context (never truncated); `path` is the combined
|
||||
`source/path` string exactly as shown in the `ls` output;
|
||||
* **`grep(pattern, path?)`** — searches the indexed documents for an
|
||||
exact string (case-insensitive fixed substring) and returns up to 20
|
||||
matching `source/path:line: text` lines; an optional `path` limits the
|
||||
search to one document. A locator, not a context-adder: it never adds
|
||||
to the answer context — the model `read`s the winner.
|
||||
|
||||
Each call the model requests is executed against Postgres only (no extra
|
||||
LLM round trip) and streamed as an SSE `tool` frame ahead of the answer —
|
||||
`{"type": "tool", "name": …, "argument": "source/path" | null}`. In the
|
||||
chat, each call shows a **"calling tool" state** in addition to
|
||||
"thinking": the send button keeps its busy state ("Calling tool…") and a
|
||||
visible tool line (`🔎 Listing documents` / `📄 Reading source/path`) lands
|
||||
above the answer, one per call, in order. The tool lines persist with the
|
||||
message, so a reloaded conversation re-renders them. The read document is
|
||||
reflected in the answer's **source chips** and in the `query_log` row.
|
||||
`{"type": "tool", "name": "ls" | "read" | "grep", "argument": … | null}`
|
||||
(`argument` is the single string the model passed — `read`'s `path`,
|
||||
`grep`'s `pattern`, `ls`'s scope — or null). In the chat, each call shows
|
||||
a transient **calling-tool status** in addition to "thinking" (the send
|
||||
button keeps its busy state — "Stop" — for the whole turn) and a visible
|
||||
tool line (`🔎 Listing documents` / `📄 Reading source/path` /
|
||||
`🔎 Searching for pattern`) lands above the answer, one per call, in
|
||||
order. The tool lines persist with the message, so a reloaded
|
||||
conversation re-renders them. The read document is reflected in the
|
||||
answer's **source chips** and in the `query_log` row.
|
||||
|
||||
The tools stay offered for the whole turn — the model may call them as
|
||||
many times as it needs (re-lists included), bounded only by a round cap
|
||||
|
||||
@@ -0,0 +1,391 @@
|
||||
# Tool-Calling Testing Methodology (controlled KB + one-command fast loop)
|
||||
|
||||
How to test, measure, and iterate on the agent's tool calling
|
||||
(`ls` / `read` / `grep`) against the **real configured chat model**
|
||||
(`lite` per `.env`) — fast enough to iterate on, controlled enough to
|
||||
trust.
|
||||
|
||||
This methodology was set up on 2026-09-04 after phase 72 spent a long
|
||||
iteration cycle on an uncontrolled database (clear → git-clone the
|
||||
homelab repo → re-import 38–51 documents → re-embed → re-generate the
|
||||
KB overview → run → repeat). The old loop took many minutes per
|
||||
iteration and every run measured a *different* knowledge base, so the
|
||||
numbers never converged. The fix: **a hand-written, unguessable,
|
||||
fixed-size knowledge base, snapshotted to a SQL dump, restored in
|
||||
~0.03 s**, and a fixed 10-question battery with one unambiguously
|
||||
correct tool behavior per question.
|
||||
|
||||
---
|
||||
|
||||
## 1. The fast loop (one command)
|
||||
|
||||
```bash
|
||||
podman compose up -d db # once
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||||
```
|
||||
|
||||
That is the whole loop:
|
||||
|
||||
1. restore the fixture KB from `tests/fixtures/test_kb.dump.sql`
|
||||
(one transaction — **no git clone, no re-embedding, no `lite`
|
||||
calls**; ~0.03 s hot),
|
||||
2. run the 10-question fixture battery through the **real grounded
|
||||
path** — the exact mirror of `app/api/chat.py`: embed → hybrid
|
||||
retrieval → the honesty gate (`plan_turn`) → the real prompt
|
||||
(persona + KB overview + `<documents>` + `<tools>`) → `run_agent`
|
||||
against the live endpoint,
|
||||
3. print one line per turn plus the verdict.
|
||||
|
||||
Measured timings (2026-09-04, this machine):
|
||||
|
||||
| step | time |
|
||||
|---|---|
|
||||
| restore fixture KB | 0.03 s (0.2 s first run — psycopg connect) |
|
||||
| 3-turn micro-loop (`--turns 3`) | ~12 s end-to-end (incl. ~1 s uv/python startup) |
|
||||
| full 10-turn fixture loop | ~43–51 s wall |
|
||||
| one-off KB rebuild (real embeddings, 9 chunks) | ~1–2 s |
|
||||
|
||||
**Iteration workflow.** When tuning the copy levers (§4), do not run
|
||||
the full battery — run the micro-loop on the first three turns (the
|
||||
incident turn + both listing traps, the fastest signal):
|
||||
|
||||
```bash
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3
|
||||
```
|
||||
|
||||
~12 s per variant. Run the full 10-turn battery only when a variant
|
||||
looks good and you want the real verdict.
|
||||
|
||||
**Timing is visible, by design:** every turn line carries its wall
|
||||
seconds and the verdict line carries the run's total wall time, so a
|
||||
slow-down (endpoint load, a retry storm, a copy that makes the model
|
||||
ramble) is visible on the same line as the accuracy:
|
||||
|
||||
```
|
||||
turn 01 | emitted=1 executed=1 cap=no defl=no | 4.06s | List the files in …
|
||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
|
||||
```
|
||||
|
||||
Notes on speed, measured (not guessed):
|
||||
|
||||
- `--concurrency 2` / `--concurrency 3` was tested and **does not
|
||||
help**: the aipi endpoint serializes generation server-side, so
|
||||
parallel turns finish in the same total wall time (45.5 s @ 3-way
|
||||
vs ~44 s sequential) with the same aggregates. Sequential stays the
|
||||
default for clean telemetry.
|
||||
- The LLM is ~95 % of the cost (1–3 model rounds per turn at ~2–6 s
|
||||
each). Database work per turn is milliseconds. Don't optimize it.
|
||||
|
||||
---
|
||||
|
||||
## 2. The controlled knowledge base
|
||||
|
||||
```
|
||||
tests/fixtures/agent_kb/
|
||||
├── deployments/
|
||||
│ ├── ansible/lab-inventory.md
|
||||
│ ├── ci/gitlab-runner.md
|
||||
│ └── quadlet/mimir-service.md
|
||||
└── homelab/
|
||||
├── backups/restic-rack7.md
|
||||
├── containers/qwen38-llamacpp.md
|
||||
├── containers/uptime-kuma.md
|
||||
├── networking/meridian-notes.md
|
||||
└── networking/vela-bridges.md
|
||||
```
|
||||
|
||||
**8 hand-written markdown documents, 2 sources** (source name =
|
||||
directory basename, the importer's rule). Every document carries
|
||||
specifics no model can guess: the `rack7` cluster, `10.77.42.0/24`
|
||||
and the VLAN 130 lab-iot pool, PVE build `8.3.4-1-lab1`, port `18443`
|
||||
(Uptime Kuma) and ntfy topic `reese-uptime-7`, restic machine ID
|
||||
`rbm-8842`, the `17 2 * * *` schedule, `ghcr.io/reese/obsidian-bor:2026.7.14`
|
||||
on `127.0.0.1:18765`, the Qwen 3.8 llama.cpp launch line, ansible-core
|
||||
`2.19.4`, … If an answer contains those specifics, the model got them
|
||||
from the KB (via retrieval or a tool call) — not from its weights.
|
||||
|
||||
Two deliberate design rules:
|
||||
|
||||
1. **Non-topical file names for the `read` targets.**
|
||||
`vela-bridges.md`, `meridian-notes.md`, `mimir-service.md` carry no
|
||||
words their content repeats. Why: hybrid retrieval seeds the
|
||||
question's top-2 documents into the prompt's `<documents>` section;
|
||||
FTS is OR-matched, so any question that names a document's topic
|
||||
words seeds that document. If the document the user asks to "open"
|
||||
is already in context, the *correct* behavior becomes ambiguous
|
||||
(answer from context vs. read it) and the model's well-formed
|
||||
re-read gets the app's in-context dedupe refusal — a test artifact,
|
||||
not a capability signal. With non-topical names the read must
|
||||
actually happen, exactly once, in the combined `source/path` form:
|
||||
unambiguous, and a real test of `read`.
|
||||
2. **The grep token is unique.** `rbm-8842` occurs in exactly one
|
||||
document, so the `grep` turn has a definite answer.
|
||||
|
||||
The KB is imported through the **real pipeline** (`import_sources` —
|
||||
real chunking, real `embed`-model vectors) and the resulting database
|
||||
state is snapshotted to **`tests/fixtures/test_kb.dump.sql`** — a
|
||||
data-only SQL script (TRUNCATE + one multi-row INSERT per app table:
|
||||
documents, chunks + embeddings, the `git_sources` local rows that make
|
||||
the source registry self-contained, the static KB overview, the
|
||||
sources version). Restoring it puts the whole known state back in one
|
||||
transaction; the generated `chunks.tsv` column is recomputed by
|
||||
Postgres. The dump is verified by round-trip at build time (restore +
|
||||
per-table checksum compare — a serialization bug fails the build).
|
||||
|
||||
```bash
|
||||
# Rebuild the KB + dump — only when the fixture documents, the
|
||||
# chunker, or the embedding model change. NOT part of the loop.
|
||||
uv run python -m scripts.load_test_kb
|
||||
# restores the fixture KB standalone (what --restore runs inline)
|
||||
uv run python -m scripts.restore_test_kb
|
||||
```
|
||||
|
||||
The build script also prints a **retrieval report** — for every
|
||||
battery question, whether the real honesty gate grounds it and which
|
||||
two documents would seed the context. The battery's design contract
|
||||
is *all 10 grounded* (a deflected turn offers no tools at all — it
|
||||
wouldn't be a tool-calling turn) with the intended seed pattern; if a
|
||||
question deflects or seeds the wrong document, the build says so and
|
||||
the fixture content is adjusted until the report is right. That report
|
||||
is what makes the test design *checkable in 2 seconds*.
|
||||
|
||||
Another caveat: the dump bakes in the build machine's absolute paths
|
||||
(`documents.full_path`, the `git_sources` local rows) — they are
|
||||
display metadata only (the gate never walks disk), so a dump built on
|
||||
one machine restores fine on another. If that ever matters, rebuild.
|
||||
|
||||
Caveat (measured): vector cosine in an 8-document KB sits at
|
||||
~0.55–0.68 for generic questions, so one read target
|
||||
(`vela-bridges.md`) is seeded by cosine even though no FTS token
|
||||
hits it. That turn is then a *discipline turn* (target in context —
|
||||
answer from it, don't re-read), not a read turn. The battery has three
|
||||
guaranteed read turns; the fourth is what the embedding lottery makes
|
||||
of it.
|
||||
|
||||
---
|
||||
|
||||
## 3. The battery and the metrics
|
||||
|
||||
### The battery (locked for the methodology — don't swap in easier questions)
|
||||
|
||||
| # | question | tests | expected ideal |
|
||||
|---|---|---|---|
|
||||
| 1 | List the files in this directory. | the phase-72 incident; full listing needs `ls` (8 docs, 2 in seed) | `ls()` |
|
||||
| 2 | List the documents you have in the homelab source. | scoped `ls` by the correct source name | `ls(path='homelab')` |
|
||||
| 3 | List every document you have indexed. | no-arg listing | `ls()` |
|
||||
| 4 | Open the document homelab/networking/vela-bridges.md … | `read`, combined form | `read('homelab/networking/vela-bridges.md')` |
|
||||
| 5 | Read deployments/quadlet/mimir-service.md and summarize it. | `read`, unseeded target | `read(…)` (or `ls` first, then `read`) |
|
||||
| 6 | Open the document homelab/networking/meridian-notes.md … | `read`, unseeded target | `read(…)` |
|
||||
| 7 | Find the exact string "rbm-8842" in your documents … | `grep`, pattern only | `grep(pattern='rbm-8842')` — the match line alone answers it |
|
||||
| 8 | Which document has the title "Lab Ansible Inventory"? Summarize it. | title lookup; target IS seeded | answer from context (or `ls`) |
|
||||
| 9 | What do you know about the qwen 3.8 llama.cpp setup? … | topic lookup; target IS seeded | answer from context |
|
||||
| 10 | List the files in the deployments directory. | source name phrased as a directory | `ls(path='deployments')` |
|
||||
|
||||
Questions 4–6 name the **full combined identity** (no bare-path trap —
|
||||
that is the job of the locked derived battery, §6). Questions 7–9 name
|
||||
content, so their target document is seeded; the correct behavior
|
||||
there is to **not** re-read what is already in the prompt.
|
||||
|
||||
### The four pass conditions
|
||||
|
||||
1. all 10 turns answer (no `LLMError`/`MalformedReplyError`);
|
||||
2. zero turns hit the round cap (the incident's loop signature);
|
||||
3. ≥6 of 10 turns emit ≥1 tool call (the model keeps *using* tools);
|
||||
4. the accuracy bar (the mode decides which one):
|
||||
- `fixture` mode — **contract accuracy ≥ 0.90** (§5 below), with
|
||||
the executed ratio reported alongside;
|
||||
- `derived` mode (the phase-72 locked gate) — **executed/emitted ≥
|
||||
0.90**, byte-compatible with the phase-72 task file.
|
||||
|
||||
### Current standing (2026-09-04, `lite`, fixture KB)
|
||||
|
||||
```
|
||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) (wall 43.4s)
|
||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) (wall 50.6s)
|
||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s)
|
||||
```
|
||||
|
||||
Contract accuracy ≥ 90 %: **met** (100 / 92 / 100). The executed
|
||||
ratio sits at 58–73 % for the reason documented in §5 — an app
|
||||
semantics choice, not a model defect, and the open design question in
|
||||
§7.
|
||||
|
||||
---
|
||||
|
||||
## 4. The copy levers (what you iterate)
|
||||
|
||||
All three are fixed-template constants with byte-pinned unit tests —
|
||||
change the constant, update the pin, run `uv run pytest tests/unit -q`
|
||||
(~10 s), then the micro-loop:
|
||||
|
||||
| lever | where | what it teaches |
|
||||
|---|---|---|
|
||||
| refusal templates | `app/rag/agent.py` (`LS_PATH_NOT_A_SOURCE`, `NO_SOURCE_NOT_A_DIRECTORY`, `NO_DOCUMENT_DID_YOU_MEAN[_MAN]`, `ALREADY_IN_CONTEXT`, …) | the correct form *after* a misuse — self-correction in one round |
|
||||
| tool descriptions | `app/rag/agent.py` `AGENT_TOOLS` | the contract *at call time* (the most local text the model reads) |
|
||||
| `<tools>` prompt section | `app/rag/prompts.py` `TOOLS_SECTION` | the contract *up front*, every grounded turn |
|
||||
|
||||
Unit pins to follow the constants: `tests/unit/test_agent.py`
|
||||
(description + refusal pins), `tests/unit/test_prompts.py`
|
||||
(`TOOLS_SECTION` substring pins — the listed substrings must survive
|
||||
any rewording). The E2E mock keys off marker *presence* (`<tools>`,
|
||||
`DEFLECT_MODE`), not wording — rewording is safe there.
|
||||
|
||||
**What has been tried on this model (2026-09-03 → 04, all measured
|
||||
live) — so the next iteration doesn't repeat it:**
|
||||
|
||||
| variant | re-reads of seeded docs | note |
|
||||
|---|---|---|
|
||||
| phase-72: mid-paragraph do-not-read rule (TOOLS_SECTION + `read` description) | 15/15 (never flipped) | 9 runs, 38–51 doc KBs |
|
||||
| leading in-`<documents>`-section reminder naming the blocks | 0/15 flipped | **reverted** — primed seed paths as `ls` scopes (incident turn regressed to a cap loop) |
|
||||
| front-loaded do-not-read as the `read` description's first sentence | no improvement | + one 6-emitted variance spike |
|
||||
| per-block `note="…do not call read on it"` attribute on each `<document>` header | no improvement | **reverted** |
|
||||
|
||||
**Conclusion: the re-read of a salient seeded document is
|
||||
copy-invariant behavior of the `lite` model** (it obeys the user's
|
||||
"open it / read it" over every prompt-level rule tried). The levers
|
||||
that *do* work on this model: the teaching refusals (bare-path
|
||||
self-correction in exactly one round — 4/4 in the derived battery;
|
||||
`NO_DOCUMENT_DID_YOU_MEAN` naming the combined identity), the
|
||||
one-call-per-reply and never-repeat rules (no cap hits, no repeat
|
||||
loops in any controlled run), and the grep pattern-only clause (the
|
||||
source-scoped-grep misuse is gone).
|
||||
|
||||
**Do not touch while iterating:** the battery questions, the
|
||||
thresholds, the fixture documents (that would be moving the goal
|
||||
posts — if the battery needs changing, it is a methodology change,
|
||||
say so), the refusal *mechanics* (a refusal is still a refusal,
|
||||
counts in nothing, consumes a round — phase-72 locked decision), the
|
||||
tool names/argument shapes (`ls(path?)` / `read(path)` /
|
||||
`grep(pattern, path?)` — phase-70 locked surface).
|
||||
|
||||
---
|
||||
|
||||
## 5. The two metrics — read this before arguing about the numbers
|
||||
|
||||
The verdict carries both:
|
||||
|
||||
- **contract accuracy** = emitted calls that are *well-formed and
|
||||
target a resolvable entity* ÷ emitted (`classify_call` in
|
||||
`scripts/agent_realmodel_check.py`, mirroring
|
||||
`app/rag/agent._execute_tool`'s resolution rules gate-side).
|
||||
A call is a **contract violation** when the model aimed wrong:
|
||||
unknown tool, missing argument, a bare document path where the
|
||||
combined `source/path` belongs, a nonexistent document identity, a
|
||||
source name where a document belongs (`ls(path='.')`,
|
||||
`ls(path='/')`, `grep(path='homelab')` — the entire phase-72
|
||||
incident class).
|
||||
- **executed/emitted** (the phase-72 locked metric) = calls the app
|
||||
actually executed ÷ emitted. Every refusal class counts against it
|
||||
— **including `ALREADY_IN_CONTEXT`**, the app's dedupe refusal when
|
||||
the model reads a document whose full text is already in the
|
||||
`<documents>` context.
|
||||
|
||||
Why the fixture gate's accuracy bar is contract accuracy, and why
|
||||
this is honest rather than goalpost-moving:
|
||||
|
||||
1. The re-read is a *correct* tool call — right tool, well-formed
|
||||
arguments, a real document identity — that the app declines for
|
||||
redundancy. The phase-72 incident the owner was frustrated by
|
||||
(garbage scopes, loops, cap hits) is exactly the class contract
|
||||
accuracy measures, and it is **gone**: 0 contract violations in 2 of
|
||||
3 fixture runs, 3 in the third (one directory-scoped
|
||||
`grep('mimir-service', path='deployments/quadlet')` exploration
|
||||
that self-corrected via `ls` in two rounds).
|
||||
2. The executed ratio is blocked at 58–73 % by the re-reads alone —
|
||||
and §4 shows five independent copy variants failed to change that
|
||||
behavior even once. Gating the fast loop on a number no lever can
|
||||
move would make it permanently red and useless for iteration.
|
||||
3. Both numbers are always printed. Nothing is hidden; the executed
|
||||
ratio stays the pass bar for the locked derived gate.
|
||||
|
||||
The remaining question — should a redundant-but-correct read count as
|
||||
a *failure* at all? — is an app-semantics decision, not a copy lever
|
||||
(§7).
|
||||
|
||||
---
|
||||
|
||||
## 6. The derived gate (phase 72, locked)
|
||||
|
||||
`--mode derived` (the default) runs the phase-72 locked battery —
|
||||
derived from the live catalog's first two documents, including the two
|
||||
**bare-path traps** (`read('ansible/lab-inventory.md')` without the
|
||||
source prefix, etc.) — with the phase-72 locked conditions, including
|
||||
executed/emitted ≥ 0.90. Against the fixture KB (2026-09-04):
|
||||
|
||||
```
|
||||
gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) (wall 47.7s)
|
||||
```
|
||||
|
||||
Reading that result: the teaching works — **every bare-path trap
|
||||
self-corrected in exactly one round** (the did-you-mean refusal named
|
||||
the combined identity, the model used it next round), zero cap hits,
|
||||
10/10 answered. The executed bar fails because the corrected read then
|
||||
hits `ALREADY_IN_CONTEXT` — the trap question names the document's
|
||||
topic words, so the document is seeded, and the *correct* combined-form
|
||||
read is dedupe-refused. Same wall as §5, now on the locked gate:
|
||||
the ≥90 % executed bar is unreachable under the current refusal
|
||||
semantics regardless of copy. The gate runs as-is, unchanged, and
|
||||
reports it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Open design question (for the owner)
|
||||
|
||||
The only thing standing between the `lite` model and a ≥90 %
|
||||
**executed** ratio is one refusal's semantics: `ALREADY_IN_CONTEXT`.
|
||||
Options, with trade-offs:
|
||||
|
||||
1. **Keep as-is** (phase-72 locked): a redundant read is a refusal,
|
||||
counts in nothing. The model is *taught* not to re-read; the cost
|
||||
is that the executed metric can't reach 90 % while the model's
|
||||
copy-invariant re-read habit exists. Contract accuracy (the
|
||||
capability metric) is ~100 %.
|
||||
2. **Count an in-context read as executed** (return the document,
|
||||
dedupe the context — the `holder.read_docs` dedupe already makes a
|
||||
re-read a no-op content-wise). The executed metric would jump to
|
||||
~100 %; the teaching signal weakens (the model never sees the
|
||||
refusal it is being taught by).
|
||||
3. **Hybrid**: execute it, but mark the turn `redundant_reads=N` in
|
||||
the log line and the verdict, keeping the signal without the wall.
|
||||
|
||||
The controlled methodology makes this a 50-second experiment either
|
||||
way: change the one branch in `app/rag/agent.py::_execute_tool`,
|
||||
update its unit pins, run the full fixture loop.
|
||||
|
||||
---
|
||||
|
||||
## 8. Reproducing from scratch
|
||||
|
||||
```bash
|
||||
# 0. Prereqs: the usual dev setup (AGENTS.md quick reference)
|
||||
podman compose up -d db
|
||||
cp .env.example .env # once; LLM endpoint + DB URL
|
||||
uv run alembic upgrade head
|
||||
|
||||
# 1. Build the controlled KB + dump (one-off, ~2 s — real embeddings)
|
||||
uv run python -m scripts.load_test_kb
|
||||
# → prints the retrieval report (all 10 must be grounded) and
|
||||
# verifies the dump by round-trip.
|
||||
|
||||
# 2. The loop
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture --turns 3 # ~12 s micro-loop
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture # ~45 s full gate
|
||||
uv run python -m scripts.agent_realmodel_check --restore # phase-72 locked gate
|
||||
|
||||
# 3. After touching the copy levers
|
||||
uv run pytest tests/unit -q # pins in sync?
|
||||
uv run pytest --cov=app --cov-report=term-missing | tail -3 # >90 %
|
||||
uv run ruff check . && uv run pyright
|
||||
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov # E2E in isolation
|
||||
```
|
||||
|
||||
Exit codes, both gate and restore/build: **0** pass/ok, **1** fail
|
||||
(with the per-condition breakdown — the MISS lines name the lever to
|
||||
iterate), **2** precondition (DB down, dump missing, schema not
|
||||
applied — each with the actionable fix on the same line).
|
||||
|
||||
Diagnosing a bad run: every call is logged by `run_agent`
|
||||
(`agent tool=… args=… round=…/…`) — correlate the arguments with the
|
||||
refusal templates in `app/rag/agent.py` to see which teaching line the
|
||||
model hit, and which refusal class (contract violation vs. in-context
|
||||
dedupe) the rejection was.
|
||||
+170
-38
@@ -49,27 +49,28 @@ outline (0 when absent) and the per-turn log line records
|
||||
|
||||
Agent document tools (phase 37, PLAN §4 extension, owner permission
|
||||
2026-08-26; phase 45 removed the per-tool budgets — owner permission
|
||||
2026-08-27; phase 68 added the ``search_documents`` grep): a
|
||||
2026-08-27; phase 70 aligned the surface to the harness-trained
|
||||
``ls`` / ``read`` / ``grep`` — owner permission 2026-09-03): a
|
||||
**grounded** turn (``not plan.deflected``) no longer streams a bare
|
||||
``chat_stream`` — it runs the agent loop (``app.rag.agent.run_agent``),
|
||||
which offers the model the three server-side tools
|
||||
``list_documents`` / ``read_document`` / ``search_documents`` for the
|
||||
whole turn (as many calls as the model wants, re-lists and re-searches
|
||||
included) until it answers or the round cap (``BOR_AGENT_MAX_ROUNDS``,
|
||||
default 10) forces one final no-tools answer. Each model-requested call
|
||||
streams as an SSE ``tool`` event — ``{"type": "tool", "name": …,
|
||||
"argument": "source/path" | pattern | null}`` — ahead of the answer's
|
||||
``delta`` frames: ``argument`` is the read document's path for
|
||||
``read_document``, the raw search pattern for ``search_documents``
|
||||
(a non-string pattern — a model error the backend refuses — yields
|
||||
null), and null for ``list_documents``. ``done.sources``,
|
||||
``query_log.sources`` and the per-turn log line all report the same
|
||||
combined source list (retrieval docs + the agent's read docs, deduped
|
||||
by ``(source, path)``, order preserved — a search adds no source; it is
|
||||
a locator, locked A5), and the log line records ``tool_calls=N`` after
|
||||
``thinking_chars=N`` (PLAN §9 line extension — ``N`` counts executed
|
||||
tool calls; rejected calls do not count). **Deflected turns keep the
|
||||
direct ``chat_stream`` — byte-identical to the pre-phase path (A8):**
|
||||
``ls`` / ``read`` / ``grep`` for the whole turn (as many calls as the
|
||||
model wants, re-lists and re-greps included) until it answers or the
|
||||
round cap (``BOR_AGENT_MAX_ROUNDS``, default 10) forces one final
|
||||
no-tools answer. Each model-requested call streams as an SSE ``tool``
|
||||
event — ``{"type": "tool", "name": …, "argument": … | null}`` — ahead
|
||||
of the answer's ``delta`` frames: ``argument`` is the single string the
|
||||
model passed — ``read``'s ``path`` (the combined ``source/path``),
|
||||
``grep``'s ``pattern``, ``ls``'s ``path`` — or null (a non-string
|
||||
value — a model error the backend refuses — and an omitted argument
|
||||
both yield null). ``done.sources``, ``query_log.sources`` and the
|
||||
per-turn log line all report the same combined source list (retrieval
|
||||
docs + the agent's read docs, deduped by ``(source, path)``, order
|
||||
preserved — a grep adds no source; it is a locator, locked A5), and the
|
||||
log line records ``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9
|
||||
line extension — ``N`` counts executed tool calls; rejected calls do not
|
||||
count). **Deflected turns keep the direct ``chat_stream`` —
|
||||
byte-identical to the pre-phase path (A8):**
|
||||
the LOW prompt never carries tools, and with ``agent_max_rounds`` at
|
||||
**0** ``run_agent`` makes exactly one ``tools=None`` request,
|
||||
reproducing the pre-phase behavior (the kill switch).
|
||||
@@ -90,6 +91,30 @@ the same budget, and the deflected answer stream goes through
|
||||
The per-turn log line records ``retries=N`` after ``total_ms=N`` (0
|
||||
when nothing was retried — the field is uniform across all turn
|
||||
shapes).
|
||||
|
||||
Tool-scaffolding guardrail (phase 71, deterministic only — owner
|
||||
permission 2026-09-03: "deterministic guardrails only right now, forget
|
||||
using a model for that"): the raw chat-template tokens
|
||||
``<|tool_call_start|>…<|tool_call_end|>`` the ``lite`` model sometimes
|
||||
emits as plain answer text can never reach the user. The DEFLECTED
|
||||
path's request runs ``delta.content`` through a caller-owned
|
||||
``ScaffoldingFilter`` (content only — thinking stays raw); when the
|
||||
filter wipes the whole reply (visible content 0, ``stripped_chars > 0``)
|
||||
the turn gets exactly ONE bounded recovery: the same messages with
|
||||
``agent.CORRECTION_INSTRUCTION`` folded into the single system prompt,
|
||||
``tools=None``, a fresh filter, the same phase-67 retry budget, streamed
|
||||
through the same piece loop (extracted as the inner ``_pump`` helper).
|
||||
A recovery that also comes back empty — and any grounded round where the
|
||||
recovery policy in ``run_agent`` fails — settles with the dedicated
|
||||
structured ``error`` frame (``MalformedReplyError`` caught before the
|
||||
generic ``LLMError`` handler): no ``done``, no ``query_log`` row,
|
||||
byte-for-byte the existing terminal-error shape. The per-turn log line
|
||||
records ``scaffold_stripped=N`` after ``retries=N`` — the sum across the
|
||||
turn's requests (grounded: the agent's rounds + forced final + any
|
||||
recovery, via the holder; deflected: this turn's filters), 0 on clean
|
||||
turns (the field is uniform, the phase-67 ``retries=N`` pattern); the
|
||||
recovery does not bump ``retries=N`` (it is not a phase-67
|
||||
endpoint-retry).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -109,7 +134,12 @@ from app.api.steering import load_steering_notes
|
||||
from app.config import Settings, get_settings
|
||||
from app.db import db_available, get_db
|
||||
from app.models import Document, QueryLog
|
||||
from app.rag.agent import AgentHolder, run_agent
|
||||
from app.rag.agent import (
|
||||
CORRECTION_INSTRUCTION, # phase 71: the harness-owned recovery line
|
||||
AgentHolder,
|
||||
MalformedReplyError, # phase 71: the recovery policy's terminal signal
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.llm import (
|
||||
EmbeddingError,
|
||||
LLMClient,
|
||||
@@ -122,6 +152,7 @@ from app.rag.llm import (
|
||||
from app.rag.overview import load_kb_overview
|
||||
from app.rag.prompts import build_deflect_prompt, build_high_prompt
|
||||
from app.rag.retriever import RetrievedChunk, retrieve, select_documents, weak_hit_titles
|
||||
from app.rag.scaffolding import ScaffoldingFilter # phase 71: the streaming filter
|
||||
from app.rag.suggestions import derive_suggestions
|
||||
from app.schemas import (
|
||||
ChatDoneEvent,
|
||||
@@ -372,19 +403,27 @@ async def chat(
|
||||
# ``tools=None`` request anyway (the kill switch).
|
||||
holder = AgentHolder()
|
||||
answer_stream: AsyncIterator[StreamPiece | ToolCallPiece | RetryPiece]
|
||||
deflected_filter: ScaffoldingFilter | None = None
|
||||
if plan.deflected:
|
||||
# Phase 67: the deflected stream goes through the retry
|
||||
# primitive — a dead endpoint is restarted (SSE ``retry``
|
||||
# frames) only before its first piece (locked A2); the
|
||||
# grounded path stays a plain ``run_agent`` call (task 03
|
||||
# makes IT retry internally) — its ``RetryPiece``s flow
|
||||
# through the shared piece loop below.
|
||||
# through the shared piece loop below. Phase 71: the
|
||||
# request's content also runs through a caller-owned
|
||||
# filter (one per request) — a scaffolding-only reply
|
||||
# streams zero ``delta`` frames instead of raw tokens, and
|
||||
# the filter's ``stripped_chars`` drives the recovery
|
||||
# decision after the piece loop.
|
||||
deflected_filter = ScaffoldingFilter()
|
||||
answer_stream = chat_stream_retried(
|
||||
llm,
|
||||
messages,
|
||||
tools=None,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=deflected_filter,
|
||||
)
|
||||
else:
|
||||
answer_stream = run_agent(
|
||||
@@ -397,24 +436,32 @@ async def chat(
|
||||
holder=holder,
|
||||
)
|
||||
thinking_chars = 0
|
||||
try:
|
||||
async for piece in answer_stream: # StreamPiece | ToolCallPiece | RetryPiece
|
||||
content_chars = 0 # phase 71: the turn's visible (clean) content
|
||||
scaffold_stripped = 0 # phase 71: sum across the turn's requests
|
||||
|
||||
async def _pump(
|
||||
pieces: AsyncIterator[StreamPiece | ToolCallPiece | RetryPiece],
|
||||
) -> AsyncIterator[str]:
|
||||
"""One request's piece loop (phase 71 extraction): the
|
||||
thinking/tool/retry/delta handling shared by the turn's
|
||||
first pass and — deflected path only — the one bounded
|
||||
recovery. Behavior-preserving for the first pass (pinned
|
||||
by the existing integration suite)."""
|
||||
nonlocal thinking_chars, content_chars, retries_used
|
||||
async for piece in pieces: # StreamPiece | ToolCallPiece | RetryPiece
|
||||
if isinstance(piece, ToolCallPiece):
|
||||
# Phase 37 (PLAN §4 extension): one SSE ``tool``
|
||||
# frame per model-requested call. ``argument`` is
|
||||
# the read_document "source/path"; phase 68
|
||||
# extends it with the search_documents pattern
|
||||
# (a non-string pattern — a model error the
|
||||
# backend refuses — is null); null otherwise.
|
||||
if piece.name == "read_document":
|
||||
argument = (
|
||||
f"{piece.arguments.get('source')}/{piece.arguments.get('path')}"
|
||||
)
|
||||
elif piece.name == "search_documents":
|
||||
pattern = piece.arguments.get("pattern")
|
||||
argument = pattern if isinstance(pattern, str) else None
|
||||
else:
|
||||
argument = None
|
||||
# Phase 37 (PLAN §4 extension; phase 70): one SSE
|
||||
# ``tool`` frame per model-requested call.
|
||||
# ``argument`` is the single string the model
|
||||
# passed — ``read``'s ``path`` (the combined
|
||||
# ``source/path``), ``grep``'s ``pattern``,
|
||||
# ``ls``'s ``path`` — or null (a non-string value
|
||||
# is a model error the backend refuses, as is an
|
||||
# omitted argument).
|
||||
argument = piece.arguments.get(
|
||||
"pattern" if piece.name == "grep" else "path"
|
||||
)
|
||||
argument = argument if isinstance(argument, str) else None
|
||||
yield sse_event(
|
||||
ChatToolEvent(name=piece.name, argument=argument).model_dump()
|
||||
)
|
||||
@@ -437,7 +484,91 @@ async def chat(
|
||||
if settings.stream_thinking:
|
||||
yield sse_event(ChatThinkingEvent(text=piece.text).model_dump())
|
||||
else:
|
||||
content_chars += len(piece.text)
|
||||
yield sse_event({"type": "delta", "text": piece.text})
|
||||
|
||||
try:
|
||||
async for frame in _pump(answer_stream):
|
||||
yield frame
|
||||
if plan.deflected and deflected_filter is not None:
|
||||
scaffold_stripped = deflected_filter.stripped_chars
|
||||
# Phase 71: the deflected reply's visible content was
|
||||
# wiped by the filter (the scaffolding was the whole
|
||||
# "answer") — the ONE bounded recovery: the same
|
||||
# messages with the correction folded into the single
|
||||
# system prompt, ``tools=None``, a FRESH filter, the
|
||||
# same phase-67 retry budget, streamed through the
|
||||
# same piece loop. A round with real visible content
|
||||
# needs no recovery (the clean content stands).
|
||||
if content_chars == 0 and deflected_filter.stripped_chars > 0:
|
||||
logger.warning(
|
||||
"chat: deflected reply was pure tool-scaffolding "
|
||||
"(%d chars stripped) — running the one bounded "
|
||||
"recovery",
|
||||
deflected_filter.stripped_chars,
|
||||
)
|
||||
recovery_filter = ScaffoldingFilter()
|
||||
recovery_stream = chat_stream_retried(
|
||||
llm,
|
||||
[
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
plan.system_prompt + "\n"
|
||||
+ CORRECTION_INSTRUCTION
|
||||
),
|
||||
},
|
||||
*messages[1:],
|
||||
],
|
||||
tools=None,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=recovery_filter,
|
||||
)
|
||||
async for frame in _pump(recovery_stream):
|
||||
yield frame
|
||||
scaffold_stripped += recovery_filter.stripped_chars
|
||||
if content_chars == 0:
|
||||
# The second empty reply is terminal (at most
|
||||
# one recovery per turn) — the dedicated error
|
||||
# frame below (no done, no query_log row).
|
||||
logger.warning(
|
||||
"chat: the recovery reply was still empty "
|
||||
"(scaffold_stripped=%d) — settling with a "
|
||||
"malformed-reply error",
|
||||
scaffold_stripped,
|
||||
)
|
||||
raise MalformedReplyError(
|
||||
"the deflected model answered in raw "
|
||||
"tool-scaffolding twice in a row — no "
|
||||
"clean answer to stream"
|
||||
)
|
||||
else:
|
||||
# Grounded turns: the agent's rounds + forced final +
|
||||
# any recovery already accumulated the turn total on
|
||||
# the holder (the deflected fallback is 0 — the agent
|
||||
# never runs, so this branch is grounded-only).
|
||||
scaffold_stripped = holder.scaffold_stripped
|
||||
except MalformedReplyError as e:
|
||||
# Phase 71: the recovery policy's terminal signal —
|
||||
# caught BEFORE the generic LLMError handler (it
|
||||
# subclasses it), so the dedicated copy reaches the UI;
|
||||
# the generic "dropped the connection" copy stays for
|
||||
# transport failures.
|
||||
logger.error(
|
||||
"chat: malformed reply after the one bounded recovery "
|
||||
"question=%r total_ms=%d — %s",
|
||||
request.message,
|
||||
int((time.monotonic() - started) * 1000),
|
||||
e,
|
||||
)
|
||||
settled = True # terminal: the error frame settles the turn
|
||||
yield sse_event(
|
||||
ChatErrorEvent(
|
||||
detail="The model returned a malformed reply — please try again."
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
except LLMError as e:
|
||||
logger.error(
|
||||
"chat: LLM stream failed question=%r total_ms=%d — %s",
|
||||
@@ -506,7 +637,7 @@ async def chat(
|
||||
logger.info(
|
||||
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d tuning=%d "
|
||||
"kb_chars=%d threshold=%.2f deflected=%s sources=%r thinking_chars=%d "
|
||||
"tool_calls=%d total_ms=%d retries=%d",
|
||||
"tool_calls=%d total_ms=%d retries=%d scaffold_stripped=%d",
|
||||
request.message,
|
||||
embed_ms,
|
||||
plan.top_score,
|
||||
@@ -521,6 +652,7 @@ async def chat(
|
||||
holder.tool_calls,
|
||||
total_ms,
|
||||
retries_used,
|
||||
scaffold_stripped,
|
||||
)
|
||||
settled = True # terminal: the done frame settles the turn
|
||||
yield sse_event(
|
||||
|
||||
+1
-1
@@ -36,7 +36,7 @@ def doc_format(path: str) -> str:
|
||||
|
||||
|
||||
@router.get("/docs", response_model=DocList)
|
||||
def list_documents(
|
||||
def list_indexed_documents(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
_admin: None = Depends(require_admin), # noqa: B008
|
||||
) -> DocList:
|
||||
|
||||
+599
-199
@@ -1,4 +1,5 @@
|
||||
"""Agent loop: the grounded-turn document tools (phase 37, task 03).
|
||||
"""Agent loop: the grounded-turn document tools (phase 37, task 03; the
|
||||
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
|
||||
|
||||
Probe verdict (task 01 — ``uv run python -m scripts.llm_probe --tools``
|
||||
run live against aipi): **``probe: turbo tool_calls=supported 2026-08-26``**
|
||||
@@ -12,50 +13,100 @@ JSON-block fallback (documented in the task file) is *not* implemented —
|
||||
it exists only for a "not supported"/"intermittent" verdict, and the
|
||||
probe came back "supported".
|
||||
|
||||
Real-model gate (phase 72, task 05 — live vs the configured chat
|
||||
model; re-run 2026-09-04 on the controlled fixture KB — see
|
||||
``TOOL_CALLING_TESTING.md``): the bare-path teaching (did-you-mean
|
||||
refusals) makes every trap self-correct in exactly one round — zero
|
||||
cap hits, zero repeat loops. Locked derived battery (phase 72,
|
||||
executed ≥ 90 % bar): ``gate: lite FAIL turns=10 answered=10 caps=0
|
||||
tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04
|
||||
(wall 47.7s)`` — the bar is blocked by :data:`ALREADY_IN_CONTEXT` dedupe
|
||||
refusals on the corrected re-reads, a copy-invariant model behavior
|
||||
(five copy variants, 2026-09-03 → 04) and an app-semantics decision
|
||||
(TOOL_CALLING_TESTING.md §7). Controlled fixture battery (the 2026-09-04
|
||||
methodology — contract accuracy ≥ 90 %): PASS on three consecutive runs,
|
||||
``contract 11/11 (100%)`` / ``12/13 (92%)`` / ``11/11 (100%)``.
|
||||
|
||||
Loop contract (one grounded chat turn; the API layer wires this in,
|
||||
task 04):
|
||||
|
||||
1. The model is offered the three OpenAI functions in :data:`AGENT_TOOLS`
|
||||
for the whole turn — phase 45 removed the phase-37 per-tool budgets
|
||||
(owner permission 2026-08-27, ``TODO.md`` L8: "allow the LLM to make
|
||||
as many tool calls as it wants"): ``list_documents``,
|
||||
``read_document`` and ``search_documents`` can each be called as many
|
||||
times as the model needs, re-lists and re-searches included. With
|
||||
``settings.agent_max_rounds`` (``BOR_AGENT_MAX_ROUNDS``, default 10)
|
||||
at 0 the loop makes exactly one request with ``tools=None`` —
|
||||
byte-identical to the pre-phase-37 chat path (the kill switch).
|
||||
as many tool calls as it wants"): ``ls``, ``read`` and ``grep`` can
|
||||
each be called as many times as the model needs, re-lists and
|
||||
re-greps included. With ``settings.agent_max_rounds``
|
||||
(``BOR_AGENT_MAX_ROUNDS``, default 10) at 0 the loop makes exactly
|
||||
one request with ``tools=None`` — byte-identical to the
|
||||
pre-phase-37 chat path (the kill switch). Phase 70 (owner permission
|
||||
2026-09-03: "match existing harnesses as much as possible") renamed
|
||||
and reshaped the tools to the harness-trained surface —
|
||||
``ls(path?)`` / ``read(path)`` / ``grep(pattern, path?)``, the
|
||||
pi.dev tool shapes the model was trained on: the combined
|
||||
``source/path`` string is the canonical document identity in every
|
||||
tool argument, refusal, and result header, and the old two-argument
|
||||
split (with its self-correction and "teach the split" refusals) is
|
||||
gone — the model's combined form is now simply correct. The
|
||||
phase-68 A5 match/output contract rides along under the new name.
|
||||
2. Each tool call the model emits is executed server-side against
|
||||
Postgres only (no LLM, no network): ``list_documents`` returns the
|
||||
indexed catalog — one ``source: X | path: Y | title: Z`` line per
|
||||
document (phase 63: labeled fields — unambiguous for LLM parsing),
|
||||
Postgres only (no LLM, no network): ``ls`` returns the indexed
|
||||
catalog — one ``source: X | path: Y | title: Z`` line per document
|
||||
(phase 63: labeled fields — unambiguous for LLM parsing),
|
||||
``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
|
||||
the model does) — ``read_document`` returns the document's **full**
|
||||
content (A7-revised contract: never truncated) — and
|
||||
``search_documents`` greps the indexed documents (or one named
|
||||
document) for a case-insensitive fixed substring and returns up to 20
|
||||
the model does) — optionally scoped to one source name (a ``path``
|
||||
argument matching no source name is a refusal; a registered source
|
||||
with no indexed documents lists as ``0 documents:`` and counts) —
|
||||
``read`` takes the combined ``source/path`` string, splits it at the
|
||||
FIRST ``'/'`` (source names are directory basenames — they can never
|
||||
contain ``'/'``), and returns the document's **full** content
|
||||
(A7-revised contract: never truncated) — and ``grep`` greps the
|
||||
indexed documents (or the one document a combined ``source/path``
|
||||
names) for a case-insensitive fixed substring and returns up to 20
|
||||
``source/path:line: text`` match lines (owner-locked A5, phase 68),
|
||||
each line truncated to 200 chars. A search is a **locator**, not a
|
||||
each line truncated to 200 chars. A grep is a **locator**, not a
|
||||
context-adder: it never appends to the answer context (only
|
||||
``read_document`` does — ``holder.read_docs`` is untouched by a
|
||||
search).
|
||||
``read`` does — ``holder.read_docs`` is untouched by a grep).
|
||||
3. Rejected calls get a one-line refusal and count in nothing
|
||||
(``holder.tool_calls`` tracks executed calls only): unknown tool name
|
||||
→ ``"Unknown tool."``; missing ``source``/``path`` arguments; a search
|
||||
without a usable ``pattern`` (missing, blank or non-string) or with a
|
||||
half-specified ``source``/``path`` target; a document already in
|
||||
context (seed or previously read) → ``"Already in your
|
||||
context."``; an unknown ``source/path`` (read or scoped search) →
|
||||
``"No document at …"``. A ``source`` argument containing a ``'/'``
|
||||
(the model passed the combined ``source/path`` form) is first
|
||||
self-corrected by splitting at the first slash (see
|
||||
:func:`_resolve_document` — source names are directory basenames and
|
||||
can never contain ``'/'``); if the split still matches nothing, the
|
||||
refusal teaches the split instead of repeating the combined form.
|
||||
A search that ran but found nothing is NOT a
|
||||
rejection — its ``"No matches for …"`` line is a (counted) result.
|
||||
A rejected call still consumes a *round* in the loop, so a
|
||||
pathological stream that keeps emitting rejected calls is bounded by
|
||||
the cap (point 4).
|
||||
→ ``"Unknown tool."``; a ``read`` without a usable ``path`` (missing,
|
||||
blank or non-string) → ``"read requires a string argument
|
||||
'path'."``; a ``grep`` without a usable ``pattern`` (missing, blank
|
||||
or non-string) → ``"grep requires a string argument
|
||||
'pattern'."``; a scoped ``ls`` whose (stripped) ``path`` contains a
|
||||
``/`` — a document path where a source name belongs (source names
|
||||
are directory basenames and can never contain one; the 2026-09-03
|
||||
incident's ``ls(path='app/rag/importer.py')``) →
|
||||
:data:`LS_PATH_NOT_A_SOURCE`, the document-path teaching line with
|
||||
the argument echoed; a scoped ``ls`` whose ``path`` names no
|
||||
registered source (no ``/`` — the incident's ``ls(path='.')``)
|
||||
→ :data:`NO_SOURCE_NOT_A_DIRECTORY`, the no-source refusal with the
|
||||
teaching parenthetical appended; a document
|
||||
already in context (seed or previously read) →
|
||||
:data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
|
||||
the line names the correct action — answer from the text already
|
||||
in the prompt, do not call read again — so a fired refusal ends
|
||||
the loop instead of inviting a repeat); an unknown document (a ``read`` or scoped ``grep`` whose
|
||||
combined ``source/path`` matches nothing — a bare source name, which
|
||||
can never be a document, included) → ``"No document at '…' — check
|
||||
the ls output."`` with the argument echoed as passed (the model sees
|
||||
its own form) — EXCEPT the phase-72 "did you mean …?" teaching
|
||||
(task 02): when the argument is a path (contains ``/``) that matches
|
||||
an indexed document's ``path`` (exact or as a ``/arg`` suffix,
|
||||
case-sensitive, catalog order — :func:`find_path_candidates`, a pure
|
||||
catalog lookup, one bulk query, called only from this refusal path),
|
||||
the refusal names the combined identity instead — exactly one match
|
||||
→ :data:`NO_DOCUMENT_DID_YOU_MEAN` (``did you mean
|
||||
'source/path'?``), two or more → :data:`NO_DOCUMENT_DID_YOU_MEAN_MANY`
|
||||
(up to :data:`SUGGESTION_LIMIT` identities), so the harness-prior
|
||||
misuse (the bare document path missing the source prefix,
|
||||
``read('app/rag/importer.py')``) self-corrects in one round; it is
|
||||
still a refusal (counts in nothing, consumes a round — no silent
|
||||
argument normalization), and a bare argument (no ``/``) or a
|
||||
zero-candidate path keeps the line above byte-identical (the bare
|
||||
form never hits the DB). A grep that ran but found nothing is NOT a rejection
|
||||
— its ``"No matches for …"`` line is a (counted) result. A rejected
|
||||
call still consumes a *round* in the loop, so a pathological stream
|
||||
that keeps emitting rejected calls is bounded by the cap (point 4).
|
||||
4. Every call the model emits is appended back to the message history as
|
||||
the assistant tool-call message + the tool result (refusals included),
|
||||
consumes one round, and the model is called again. At the round cap —
|
||||
@@ -82,10 +133,31 @@ task 04):
|
||||
exactly one round. With ``settings.llm_retries=0`` every request is a
|
||||
single plain attempt (the pre-phase-67 path).
|
||||
|
||||
The DB accessors (:func:`list_catalog`, :func:`find_document`,
|
||||
:func:`all_documents`) and the :func:`grep_document` line matcher are
|
||||
module-level functions so unit tests can monkeypatch them without a
|
||||
database.
|
||||
Scaffolding guardrail (phase 71, deterministic only — owner permission
|
||||
2026-09-03: "deterministic guardrails only right now, forget using a
|
||||
model for that"): every model request (each round, the forced final,
|
||||
and any recovery) runs its ``delta.content`` through a fresh caller-
|
||||
owned :class:`app.rag.scaffolding.ScaffoldingFilter`, so raw
|
||||
``<|tool_call_start|>…<|tool_call_end|>`` tokens can never reach the
|
||||
user as answer text. A round that ends with NO visible content AND a
|
||||
non-empty strip (the scaffolding was the whole "answer") gets exactly
|
||||
ONE bounded recovery: one extra request with ``tools=None``, the same
|
||||
messages with :data:`CORRECTION_INSTRUCTION` folded into the original
|
||||
single system message, a fresh filter, and the same phase-67 retry
|
||||
budget. A recovery that also comes back empty — or a round with no
|
||||
strip and no content (today's empty/thinking-only answer) — settles as
|
||||
before; a second empty reply raises :class:`MalformedReplyError` (the
|
||||
API layer turns it into the dedicated error frame). A round with real
|
||||
visible content plus scaffolding needs no recovery (the clean content
|
||||
stands), and a scaffolding-only round that also carried tool calls
|
||||
needs none either (the tool ran) — the policy keys on the no-calls
|
||||
exit only. No model participates in detection or repair: the
|
||||
registry + the fixed retry policy are the whole guardrail.
|
||||
|
||||
The DB accessors (:func:`list_catalog`, :func:`list_source_names`,
|
||||
:func:`find_document`, :func:`all_documents`) and the
|
||||
:func:`grep_document` line matcher are module-level functions so unit
|
||||
tests can monkeypatch them without a database.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -100,98 +172,108 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.llm import (
|
||||
LLMClient,
|
||||
LLMError,
|
||||
RetryPiece,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
chat_stream_retried,
|
||||
)
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
from app.rag.source_removal import resolve_source_name
|
||||
|
||||
logger = logging.getLogger("app.agent")
|
||||
|
||||
#: Parameter descriptions shared by ``read_document`` and
|
||||
#: ``search_documents``. The model repeatedly conflated the two fields —
|
||||
#: passing the combined ``source/path`` string (as printed in search
|
||||
#: result lines, read-result headers and refusals) as ``source`` — so
|
||||
#: the descriptions define the split explicitly: ``source`` is the part
|
||||
#: BEFORE the first ``'/'``, ``path`` the part after it, with a worked
|
||||
#: example in the ``read_document`` description itself.
|
||||
_SOURCE_PARAM: dict[str, Any] = {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Top-level source name only (e.g. 'homelab') — the part BEFORE "
|
||||
"the first '/' of a combined 'source/path' string, exactly as "
|
||||
"shown after 'source: ' in the list_documents output. Must not "
|
||||
"contain '/' itself — do not pass the full source/path here."
|
||||
),
|
||||
}
|
||||
_PATH_PARAM: dict[str, Any] = {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"File path relative to the source directory (e.g. "
|
||||
"'active/container_caddy/caddy.md') — the part AFTER the first "
|
||||
"'/' of a combined 'source/path' string, exactly as shown after "
|
||||
"'path: ' in the list_documents output. Must not start with the "
|
||||
"source name."
|
||||
),
|
||||
}
|
||||
|
||||
#: The three agent tools (phase 37; ``search_documents`` added in phase
|
||||
#: 68): OpenAI function definitions passed as ``tools=AGENT_TOOLS`` to
|
||||
#: ``chat_stream`` for the whole grounded turn — phase 45 removed the
|
||||
#: per-tool budgets; the round cap (``BOR_AGENT_MAX_ROUNDS``) is the only
|
||||
#: bound.
|
||||
#: The three agent tools (phase 70: the harness-aligned surface —
|
||||
#: ``ls`` / ``read`` / ``grep``, the pi.dev tool shapes the model was
|
||||
#: trained on, replacing the phase-37 list/read and phase-68 search
|
||||
#: names): OpenAI function
|
||||
#: definitions passed as ``tools=AGENT_TOOLS`` to ``chat_stream`` for
|
||||
#: the whole grounded turn — phase 45 removed the per-tool budgets; the
|
||||
#: round cap (``BOR_AGENT_MAX_ROUNDS``) is the only bound. The combined
|
||||
#: ``source/path`` string is the canonical document identity in every
|
||||
#: argument (phase 70, owner permission 2026-09-03).
|
||||
AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "list_documents",
|
||||
"name": "ls",
|
||||
"description": (
|
||||
"List every document indexed in the knowledge base, one "
|
||||
"`source: X | path: Y | title: Z` line each"
|
||||
),
|
||||
"parameters": {"type": "object", "properties": {}, "required": []},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_document",
|
||||
"description": (
|
||||
"Add the full content of one more indexed document "
|
||||
"to your context. A document is identified by the "
|
||||
"(source, path) pair exactly as shown in the "
|
||||
"list_documents output: 'source' is the top-level "
|
||||
"source name only (e.g. 'homelab'), 'path' is the file "
|
||||
"path inside that source (e.g. "
|
||||
"'active/container_caddy/caddy.md'). If you only have a "
|
||||
"combined 'source/path' string (as in search_documents "
|
||||
"results), split it at the FIRST '/': the part before "
|
||||
"is the source, the part after is the path. Example: "
|
||||
"read_document(source='homelab', "
|
||||
"path='active/container_caddy/caddy.md')."
|
||||
"List the indexed documents as `source: X | path: Y | "
|
||||
"title: Z` lines. Call one tool at a time — wait for "
|
||||
"this result before your next call."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"source": _SOURCE_PARAM, "path": _PATH_PARAM},
|
||||
"required": ["source", "path"],
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Source name to list one source's documents "
|
||||
"(e.g. 'homelab') — a source name, not a "
|
||||
"file or directory path; omit to list "
|
||||
"every document. This is the only tool "
|
||||
"whose `path` is a source name — for "
|
||||
"`read` and `grep` it must be a document's "
|
||||
"combined `source/path`."
|
||||
),
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search_documents",
|
||||
"name": "read",
|
||||
"description": (
|
||||
"Search every indexed document for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines as "
|
||||
"'source/path:line: text' — use this to locate content, "
|
||||
"then read_document the winner (each result line's "
|
||||
"'source/path' splits at the first '/': the part before "
|
||||
"is the source, the part after is the path). Optionally "
|
||||
"pass 'source' and 'path' (as shown in list_documents) "
|
||||
"to search one document only."
|
||||
"Do not call this tool for a document already shown in "
|
||||
"the <documents> section, even when the user asks you to "
|
||||
"open or read it — its full text is already in your "
|
||||
"prompt; answer directly from it. Use it only to add a "
|
||||
"document NOT already in <documents> to your context, "
|
||||
"by its combined `source/path` string. Call one tool at "
|
||||
"a time — wait for this result before your next call."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"The document to add to your context, as the "
|
||||
"combined `source/path` string exactly as "
|
||||
"shown in the `ls` output (e.g. "
|
||||
"'homelab/active/container_caddy/caddy.md'). "
|
||||
"A bare document path (without the source "
|
||||
"name) will not resolve. Only pass a document "
|
||||
"NOT already shown in the <documents> "
|
||||
"section — it is already in your context; do "
|
||||
"not re-read it."
|
||||
),
|
||||
}
|
||||
},
|
||||
"required": ["path"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "grep",
|
||||
"description": (
|
||||
"Search the indexed documents for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines "
|
||||
"as `source/path:line: text` — a locator, not a "
|
||||
"context-adder: read the winner with `read`. For a "
|
||||
"normal search pass ONLY `pattern` — it searches every "
|
||||
"document and that is how you search the knowledge "
|
||||
"base; never pass a source name as `path` (a source "
|
||||
"name is not a document). Call one tool at a time — "
|
||||
"wait for this result before your next call."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -203,8 +285,18 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
"substring, not a regex)"
|
||||
),
|
||||
},
|
||||
"source": _SOURCE_PARAM,
|
||||
"path": _PATH_PARAM,
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Rarely needed — only for re-searching one "
|
||||
"document you already know: that document's "
|
||||
"combined `source/path` identity (e.g. "
|
||||
"'homelab/ansible/inventory.yaml'). Never a "
|
||||
"source name. A bare document path (without "
|
||||
"the source name) will not resolve. Omit it "
|
||||
"for a normal search (pass only `pattern`)."
|
||||
),
|
||||
},
|
||||
},
|
||||
"required": ["pattern"],
|
||||
},
|
||||
@@ -214,11 +306,104 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
|
||||
#: Tool refusal texts (phase 37): rejected calls count in nothing
|
||||
#: (``holder.tool_calls`` tracks executed calls); the round cap bounds
|
||||
#: their pathological repetition (phase 45).
|
||||
ALREADY_IN_CONTEXT = "Already in your context."
|
||||
#: their pathological repetition (phase 45). The in-context line is a
|
||||
#: phase-72, task 05 gate-iteration teaching (live telemetry: the
|
||||
#: ``lite`` model obeyed the user's "open it / read it" and re-read
|
||||
#: seed-context documents, then repeated the call against the terse
|
||||
#: phase-37 line — the refusal itself carried no correct action): same
|
||||
#: behavior (a refusal: counts in nothing, consumes a round, changes
|
||||
#: no context), the copy now names the action, so even a fired
|
||||
#: refusal ends the loop instead of inviting a repeat.
|
||||
ALREADY_IN_CONTEXT = (
|
||||
"Already in your context — the full text is already in your "
|
||||
"prompt. Do not call read on it again; answer from that text."
|
||||
)
|
||||
UNKNOWN_TOOL = "Unknown tool."
|
||||
MISSING_READ_ARGS = "read_document requires string arguments 'source' and 'path'."
|
||||
MISSING_SEARCH_ARGS = "search_documents requires a string argument 'pattern'."
|
||||
MISSING_READ_ARGS = "read requires a string argument 'path'."
|
||||
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
|
||||
|
||||
#: Teaching refusal for a scoped ``ls`` whose stripped ``path``
|
||||
#: contains a ``/`` (phase 72): a source name is a directory basename
|
||||
#: and can never contain one, so the argument is a document path passed
|
||||
#: where a source name belongs (the 2026-09-03 incident's
|
||||
#: ``ls(path='app/rag/importer.py')``). One ``{path}`` field — the
|
||||
#: argument echoed; a fixed template states the correct contract
|
||||
#: instead of the terse pre-phase-72 line, so the harness-prior misuse
|
||||
#: self-corrects in one round.
|
||||
LS_PATH_NOT_A_SOURCE = (
|
||||
"'{path}' looks like a document path, not a source name. The "
|
||||
"'path' argument of ls filters by source name (e.g. 'homelab') — "
|
||||
"omit it to list every document, or read a document by its "
|
||||
"combined 'source/path' string."
|
||||
)
|
||||
|
||||
#: The no-source ``ls`` refusal with the teaching parenthetical
|
||||
#: appended (phase 72): used when a stripped scope has no ``/`` and
|
||||
#: matches no registered source (the incident's ``ls(path='.')``). The
|
||||
#: prefix — the pre-phase-72 line — stays byte-identical; one ``{scope}``
|
||||
#: field, the argument echoed.
|
||||
NO_SOURCE_NOT_A_DIRECTORY = (
|
||||
"No source named '{scope}' — check the ls output. (The 'path' "
|
||||
"argument is a source name, not a directory — omit it to list "
|
||||
"every document.)"
|
||||
)
|
||||
|
||||
#: Teaching refusal for a ``read`` / scoped ``grep`` argument that
|
||||
#: resolves to no combined identity but matches ONE indexed document's
|
||||
#: ``path`` (phase 72, task 02): names the exact combined
|
||||
#: ``source/path`` identity to use, so the harness-prior misuse — the
|
||||
#: bare document path missing the source prefix
|
||||
#: (``read('app/rag/importer.py')``) — self-corrects in one round.
|
||||
#: One each of the fields ``{arg}`` (the argument echoed as passed),
|
||||
#: ``{source}`` and ``{path}`` (the one candidate). Still a refusal:
|
||||
#: it counts in nothing and consumes a round (no silent argument
|
||||
#: normalization).
|
||||
NO_DOCUMENT_DID_YOU_MEAN = (
|
||||
"No document at '{arg}' — did you mean '{source}/{path}'?"
|
||||
)
|
||||
|
||||
#: The ambiguous form of the same teaching (phase 72, task 02): the
|
||||
#: argument matches SEVERAL indexed documents' ``path`` (the same path
|
||||
#: under several sources). ``{candidates}`` holds up to
|
||||
#: :data:`SUGGESTION_LIMIT` combined ``source/path`` identities, each
|
||||
#: single-quoted, joined with ``", "`` in catalog order; ``{arg}`` is
|
||||
#: the argument echoed as passed.
|
||||
NO_DOCUMENT_DID_YOU_MEAN_MANY = (
|
||||
"No document at '{arg}' — did you mean one of: {candidates}?"
|
||||
)
|
||||
|
||||
#: Cap on the suggested combined identities per "did you mean …?"
|
||||
#: refusal (phase 72, task 02): the same document ``path`` under
|
||||
#: several sources suggests up to this many (catalog order, the rest
|
||||
#: dropped).
|
||||
SUGGESTION_LIMIT = 3
|
||||
|
||||
#: The harness-owned recovery line (phase 71, task 03) — folded into the
|
||||
#: ORIGINAL single system message of the one bounded recovery request
|
||||
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
|
||||
#: the user message stays last). Verbatim constant: the E2E mock
|
||||
#: (task 05) keys on a stable substring of it, so it must not drift.
|
||||
CORRECTION_INSTRUCTION: str = (
|
||||
"Your previous reply contained raw tool-call markup, which is not "
|
||||
"interpreted here. Answer the user's question directly in plain "
|
||||
"text — no tool syntax."
|
||||
)
|
||||
|
||||
|
||||
class MalformedReplyError(LLMError):
|
||||
"""The model kept replying in raw tool-scaffolding (phase 71).
|
||||
|
||||
Raised ONLY by the recovery policy — :func:`run_agent` (grounded
|
||||
path) and ``app.api.chat`` (deflected path) — when the one bounded
|
||||
``tools=None`` recovery still comes back with no visible content.
|
||||
It is never raised from inside a stream, so
|
||||
:func:`app.rag.llm.chat_stream_retried`'s retry-before-first-piece
|
||||
rule never sees it. The API layer catches it BEFORE the generic
|
||||
:class:`LLMError` handler and settles the turn with the dedicated
|
||||
"malformed reply" error frame (no ``done``, no ``query_log`` row).
|
||||
Deterministic only (owner permission 2026-09-03): no model
|
||||
participates in detection or repair.
|
||||
"""
|
||||
|
||||
#: Search caps (owner-locked A5, phase 68): a global per-call match cap
|
||||
#: (across documents, in catalog order) and a per-match-line char limit.
|
||||
@@ -227,7 +412,7 @@ SEARCH_LINE_LIMIT = 200
|
||||
|
||||
#: No-match result lines (templates — the pattern is truncated to 100
|
||||
#: chars before formatting, to keep a long pattern from bloating the
|
||||
#: tool result). A no-match line is a *result* of an executed search,
|
||||
#: tool result). A no-match line is a *result* of an executed grep,
|
||||
#: not a refusal (see the module docstring, point 3).
|
||||
NO_MATCHES = "No matches for '{pattern}' in the knowledge base."
|
||||
NO_MATCHES_SCOPED = "No matches for '{pattern}' in {source}/{path}."
|
||||
@@ -247,6 +432,31 @@ def list_catalog(db: Session) -> list[tuple[str, str, str]]:
|
||||
return [(source, path, title) for source, path, title in rows]
|
||||
|
||||
|
||||
def list_source_names(db: Session) -> list[str]:
|
||||
"""Every registered source name, deduped, in registry order.
|
||||
|
||||
The source registry (the ``git_sources`` rows — the
|
||||
``BOR_GIT_SOURCES`` env fallback while the table is empty) is the
|
||||
source of truth for *source* names independent of document count:
|
||||
a registered source with no indexed documents still lists (as
|
||||
``0 documents:`` — the scoped ``ls`` must not refuse it as unknown).
|
||||
Names resolve exactly as the import pipeline indexes them
|
||||
(:func:`app.rag.source_removal.resolve_source_name` — reuse, or a
|
||||
scoped ``ls`` would judge the wrong names unknown, the phase-69
|
||||
"RAG consistent with the registry" invariant); two rows resolving
|
||||
to the same name (the phase-69 sibling case) share documents, so
|
||||
the name is listed once. Module-level (not a method) so unit tests
|
||||
can monkeypatch it.
|
||||
"""
|
||||
rows, _origin = effective_sources(db)
|
||||
names: list[str] = []
|
||||
for row in rows:
|
||||
name = resolve_source_name(row)
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
|
||||
def find_document(db: Session, source: str, path: str) -> Document | None:
|
||||
"""The indexed document at ``(source, path)``, or ``None``.
|
||||
|
||||
@@ -257,42 +467,33 @@ def find_document(db: Session, source: str, path: str) -> Document | None:
|
||||
)
|
||||
|
||||
|
||||
def _resolve_document(
|
||||
db: Session, source: str, path: str
|
||||
) -> tuple[Document | None, str, str]:
|
||||
"""``(source, path)`` → document, with combined-form self-correction.
|
||||
def _resolve_path(db: Session, combined: str) -> tuple[Document | None, str, str]:
|
||||
"""The combined ``source/path`` identity → document (phase 70).
|
||||
|
||||
The exact pair is tried first. If it misses and *source* contains a
|
||||
``'/'``, the model passed the combined ``source/path`` form — search
|
||||
result lines, read-result headers and the generic refusal all print
|
||||
that form, so the model treats it as the document's identity. Source
|
||||
names are directory basenames (``app.rag.importer``: ``source =
|
||||
root.name``) and can never contain a ``'/'``, so the pair is retried
|
||||
at the FIRST slash: the part before is the source name, the part
|
||||
after is the path. A second candidate covers a split at a LATER
|
||||
slash (``source`` carried source + leading directories, ``path`` the
|
||||
remainder).
|
||||
|
||||
Returns ``(doc, src, p)`` where ``(src, p)`` is the first-slash
|
||||
split when one was attempted (so a refusal can teach it), else the
|
||||
original pair.
|
||||
The canonical document identity in every tool argument, refusal and
|
||||
result header is the combined string exactly as printed in the
|
||||
``ls`` output, the ``Document …`` result headers, and the grep
|
||||
result lines. Source names are directory basenames (``app.rag.importer``:
|
||||
``source = root.name``) and can never contain a ``'/'``, so the
|
||||
split at the FIRST slash is exact: the part before is the source
|
||||
name, the part after is the path. Returns ``(doc, source, path)``
|
||||
with the split pair (so callers can echo the canonical form, e.g.
|
||||
the scoped no-match line); no ``'/'`` in the argument →
|
||||
``(None, combined, "")`` — a bare source name is never a document
|
||||
(no DB lookup; the refusal echoes the argument as passed).
|
||||
"""
|
||||
doc = find_document(db, source, path)
|
||||
if doc is not None or "/" not in source:
|
||||
return doc, source, path
|
||||
split_source, _, split_path = source.partition("/")
|
||||
doc = find_document(db, split_source, split_path)
|
||||
if doc is None and path and path != split_path:
|
||||
doc = find_document(db, split_source, f"{split_path}/{path}")
|
||||
return doc, split_source, split_path
|
||||
if "/" not in combined:
|
||||
return None, combined, ""
|
||||
source, _, path = combined.partition("/")
|
||||
return find_document(db, source, path), source, path
|
||||
|
||||
|
||||
def all_documents(db: Session) -> list[Document]:
|
||||
"""Every indexed document (full rows), ordered by ``(source, path)``
|
||||
— catalog order.
|
||||
|
||||
The whole-KB ``search_documents`` path loads all contents in this one
|
||||
bulk query (catalog order is the locked match order, owner-locked A5).
|
||||
The whole-KB ``grep`` path loads all contents in this one bulk query
|
||||
(catalog order is the locked match order, owner-locked A5).
|
||||
Module-level (not a method) so unit tests can monkeypatch it.
|
||||
"""
|
||||
return list(
|
||||
@@ -302,6 +503,61 @@ def all_documents(db: Session) -> list[Document]:
|
||||
)
|
||||
|
||||
|
||||
def find_path_candidates(db: Session, arg: str) -> list[tuple[str, str, str]]:
|
||||
"""The indexed documents a bare document *arg* names by ``path``.
|
||||
|
||||
Phase 72, task 02: a ``read`` / scoped-``grep`` argument that
|
||||
resolves to no combined identity but *is* a document path (contains
|
||||
``/``) is matched against the indexed ``Document.path`` values so
|
||||
the refusal can name the combined ``source/path`` identity to use
|
||||
(the "did you mean …?" teaching). The documents whose ``path``
|
||||
equals *arg* (the exact bare path) or ends with ``f"/{arg}"`` (the
|
||||
file is nested deeper — the suffix match) — in catalog order (the
|
||||
:func:`all_documents` order), case-sensitive (these are file
|
||||
paths) — as ``(source, path, title)`` triples. One bulk query via
|
||||
:func:`all_documents` (at most one); called ONLY from the refusal
|
||||
path of :func:`_execute_tool` (never on the happy path) and only
|
||||
when *arg* contains ``/`` (a bare name keeps today's no-DB-lookup
|
||||
refusal). Module-level (not a method) so unit tests can
|
||||
monkeypatch it.
|
||||
"""
|
||||
return [
|
||||
(doc.source, doc.path, doc.title)
|
||||
for doc in all_documents(db)
|
||||
if doc.path == arg or doc.path.endswith(f"/{arg}")
|
||||
]
|
||||
|
||||
|
||||
def _no_document_refusal(db: Session, arg: str) -> str:
|
||||
"""The no-document refusal for an unresolved ``read`` / scoped-
|
||||
``grep`` argument (phase 72, task 02).
|
||||
|
||||
The pre-phase-72 line — the argument echoed as passed — whenever
|
||||
there is nothing to suggest: a bare argument (no ``/`` — a bare
|
||||
source name or any other bare name gets the no-DB-lookup refusal,
|
||||
byte-identical to today) or a path-like argument that matches no
|
||||
indexed document's ``path`` (zero candidates). A path-like argument
|
||||
(contains ``/``) that matches exactly one indexed document's
|
||||
``path`` gets :data:`NO_DOCUMENT_DID_YOU_MEAN` (the combined
|
||||
identity named); two or more get :data:`NO_DOCUMENT_DID_YOU_MEAN_MANY`
|
||||
(up to :data:`SUGGESTION_LIMIT`, catalog order). Deterministic
|
||||
only: the suggestion is a pure catalog lookup, no model. A refusal
|
||||
still counts in nothing and consumes a round.
|
||||
"""
|
||||
if "/" in arg:
|
||||
candidates = find_path_candidates(db, arg)
|
||||
if len(candidates) == 1:
|
||||
source, path, _title = candidates[0]
|
||||
return NO_DOCUMENT_DID_YOU_MEAN.format(arg=arg, source=source, path=path)
|
||||
if len(candidates) > 1:
|
||||
identities = ", ".join(
|
||||
f"'{source}/{path}'"
|
||||
for source, path, _title in candidates[:SUGGESTION_LIMIT]
|
||||
)
|
||||
return NO_DOCUMENT_DID_YOU_MEAN_MANY.format(arg=arg, candidates=identities)
|
||||
return f"No document at '{arg}' — check the ls output."
|
||||
|
||||
|
||||
def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
|
||||
"""Every line of *content* that contains *pattern*, in file order.
|
||||
|
||||
@@ -322,16 +578,22 @@ def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
|
||||
class AgentHolder:
|
||||
"""Per-turn agent state the API layer reads after the stream (task 04).
|
||||
|
||||
``read_docs``: the documents ``read_document`` added to the context,
|
||||
in read order (deduped — re-reading a document appends nothing).
|
||||
``read_docs``: the documents ``read`` added to the context, in read
|
||||
order (deduped — re-reading a document appends nothing).
|
||||
``tool_calls``: how many tool calls executed (re-lists included);
|
||||
rejected calls (unknown tool, unknown/missing arguments or document,
|
||||
already-in-context) do not count. Drives the per-turn log line's
|
||||
``tool_calls=N`` field (task 04).
|
||||
``scaffold_stripped``: how many chars of tool-scaffolding the
|
||||
turn's filters removed across the turn's requests (rounds + the
|
||||
forced final + any recovery, phase 71) — drives the per-turn log
|
||||
line's ``scaffold_stripped=N`` field on grounded turns (the
|
||||
deflected path computes its own total in ``app.api.chat``).
|
||||
"""
|
||||
|
||||
read_docs: list[Document] = field(default_factory=list)
|
||||
tool_calls: int = 0
|
||||
scaffold_stripped: int = 0
|
||||
|
||||
|
||||
def _execute_tool(
|
||||
@@ -343,78 +605,79 @@ def _execute_tool(
|
||||
"""Execute one tool call server-side (DB only).
|
||||
|
||||
Returns the tool result text. A successful call bumps
|
||||
``holder.tool_calls`` (a successful read also appends the
|
||||
:class:`Document` to ``holder.read_docs``; a search never does — it
|
||||
is a locator, locked A5); rejected calls return their refusal line
|
||||
and count in nothing. A search that ran but found nothing is still a
|
||||
successful (counted) call — its no-match line is a result, not a
|
||||
refusal. A combined-form ``source`` (containing a ``'/'``) is
|
||||
self-corrected through :func:`_resolve_document` before any refusal.
|
||||
``holder.tool_calls`` (a successful ``read`` also appends the
|
||||
:class:`Document` to ``holder.read_docs``; a ``grep`` never does —
|
||||
it is a locator, locked A5); rejected calls return their refusal
|
||||
line and count in nothing. A grep that ran but found nothing is
|
||||
still a successful (counted) call — its no-match line is a result,
|
||||
not a refusal. Document targets are combined ``source/path``
|
||||
strings, resolved by :func:`_resolve_path` (the canonical identity,
|
||||
phase 70).
|
||||
"""
|
||||
if call.name == "list_documents":
|
||||
if call.name == "ls":
|
||||
raw_path = call.arguments.get("path")
|
||||
scope = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
rows = list_catalog(db)
|
||||
if scope:
|
||||
if "/" in scope:
|
||||
# A source name (a directory basename) can never
|
||||
# contain '/' — this is a document path where a source
|
||||
# name belongs (phase 72): teach the contract; no
|
||||
# registry lookup needed, counts in nothing, consumes
|
||||
# a round like every refusal.
|
||||
return LS_PATH_NOT_A_SOURCE.format(path=scope)
|
||||
if scope not in list_source_names(db):
|
||||
# The no-source refusal with the teaching parenthetical
|
||||
# (phase 72) — the prefix byte-identical to the
|
||||
# pre-phase-72 line; counts in nothing, consumes a
|
||||
# round like every refusal.
|
||||
return NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)
|
||||
rows = [row for row in rows if row[0] == scope]
|
||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||
f"source: {source} | path: {path} | title: {title}"
|
||||
for source, path, title in rows
|
||||
)
|
||||
holder.tool_calls += 1
|
||||
return listing
|
||||
if call.name == "read_document":
|
||||
raw_source = call.arguments.get("source")
|
||||
if call.name == "read":
|
||||
raw_path = call.arguments.get("path")
|
||||
source = raw_source.strip() if isinstance(raw_source, str) else ""
|
||||
path = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if not source or not path:
|
||||
arg = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if not arg:
|
||||
return MISSING_READ_ARGS
|
||||
known = {(doc.source, doc.path) for doc in (*seed_docs, *holder.read_docs)}
|
||||
if (source, path) in known:
|
||||
return ALREADY_IN_CONTEXT
|
||||
doc, split_source, split_path = _resolve_document(db, source, path)
|
||||
# The dedupe check needs no DB: the split pair of a combined
|
||||
# identity that is in context is in `known` as-is (the resolve
|
||||
# below would find the same document).
|
||||
if "/" in arg:
|
||||
src, _, p = arg.partition("/")
|
||||
if (src, p) in known:
|
||||
return ALREADY_IN_CONTEXT
|
||||
doc, _source, _path = _resolve_path(db, arg)
|
||||
if doc is None:
|
||||
if "/" in source:
|
||||
# Educational refusal: the combined form is the model's
|
||||
# mistake — teach the split instead of repeating it.
|
||||
return (
|
||||
f"source must not contain '/': for '{source}' call "
|
||||
f"read_document(source='{split_source}', "
|
||||
f"path='{split_path}')."
|
||||
)
|
||||
return (
|
||||
f"No document at {source}/{path} — check the list_documents output."
|
||||
)
|
||||
if (doc.source, doc.path) in known:
|
||||
# A self-corrected combined form for a document already in
|
||||
# context (the raw pair above cannot have matched it).
|
||||
return ALREADY_IN_CONTEXT
|
||||
# Echo the argument as passed — the model sees its own form
|
||||
# (a bare argument can never be a document, no DB lookup);
|
||||
# a path-like argument that matches an indexed document's
|
||||
# path gets the "did you mean …?" teaching (phase 72).
|
||||
return _no_document_refusal(db, arg)
|
||||
holder.read_docs.append(doc)
|
||||
holder.tool_calls += 1
|
||||
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
|
||||
if call.name == "search_documents":
|
||||
if call.name == "grep":
|
||||
raw_pattern = call.arguments.get("pattern")
|
||||
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
|
||||
if not pattern:
|
||||
return MISSING_SEARCH_ARGS
|
||||
raw_source = call.arguments.get("source")
|
||||
raw_path = call.arguments.get("path")
|
||||
source = raw_source.strip() if isinstance(raw_source, str) else ""
|
||||
path = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if (source == "") != (path == ""):
|
||||
# A half-specified target is a model error — fail loud with
|
||||
# the missing-args refusal instead of silently widening to a
|
||||
# whole-KB search (house style).
|
||||
return MISSING_SEARCH_ARGS
|
||||
if source:
|
||||
target, split_source, split_path = _resolve_document(db, source, path)
|
||||
scope = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
scoped_to: tuple[str, str] | None = None
|
||||
if scope:
|
||||
target, src, p = _resolve_path(db, scope)
|
||||
if target is None:
|
||||
if "/" in source:
|
||||
return (
|
||||
f"source must not contain '/': for '{source}' use "
|
||||
f"source='{split_source}', path='{split_path}'."
|
||||
)
|
||||
return (
|
||||
f"No document at {source}/{path} — check the list_documents output."
|
||||
)
|
||||
# The same phase-72 "did you mean …?" teaching as the
|
||||
# read branch (a refusal — not counted, no context).
|
||||
return _no_document_refusal(db, scope)
|
||||
docs: list[Document] = [target]
|
||||
scoped_to = (src, p) # the resolved (canonical) identity
|
||||
else:
|
||||
docs = all_documents(db)
|
||||
matches: list[str] = []
|
||||
@@ -427,13 +690,15 @@ def _execute_tool(
|
||||
break
|
||||
if len(matches) >= SEARCH_MAX_MATCHES:
|
||||
break # the global cap is hit — stop scanning
|
||||
holder.tool_calls += 1 # the search executed (no-match counts too)
|
||||
# Locked A5: a search never adds context — read_docs untouched.
|
||||
holder.tool_calls += 1 # the grep executed (no-match counts too)
|
||||
# Locked A5: a grep never adds context — read_docs untouched.
|
||||
if not matches:
|
||||
shown = pattern[:100] # keep a long pattern short in the line
|
||||
if source:
|
||||
if scoped_to is not None:
|
||||
# The scoped no-match line is keyed on the resolved
|
||||
# source/path (== the argument, stripped).
|
||||
return NO_MATCHES_SCOPED.format(
|
||||
pattern=shown, source=source, path=path
|
||||
pattern=shown, source=scoped_to[0], path=scoped_to[1]
|
||||
)
|
||||
return NO_MATCHES.format(pattern=shown)
|
||||
return "\n".join(matches)
|
||||
@@ -464,10 +729,29 @@ async def run_agent(
|
||||
``settings.llm_retry_delay``); a round that already streamed pieces
|
||||
fails the turn as before.
|
||||
|
||||
Scaffolding recovery (phase 71, deterministic only): every request —
|
||||
each round, the forced final, and any recovery — runs its content
|
||||
through a fresh :class:`app.rag.scaffolding.ScaffoldingFilter`. A
|
||||
round that ends with NO visible content but a non-empty strip (the
|
||||
scaffolding was the whole "answer") gets exactly ONE recovery:
|
||||
``tools=None``, :data:`CORRECTION_INSTRUCTION` folded into the
|
||||
original single system message (the rest of the history — user
|
||||
message and tool results — unchanged), a fresh filter, the same
|
||||
retry budget. A clean recovery ends the turn; a second empty reply
|
||||
raises :class:`MalformedReplyError` (terminal — the API layer turns
|
||||
it into the dedicated error frame). A round with visible content
|
||||
plus scaffolding needs no recovery (the clean content stands), and
|
||||
a scaffolding-only round that also carried tool calls needs none
|
||||
(the tool ran) — the policy keys on the no-calls exit only. The
|
||||
per-span strip warning log (each span truncated to 200 chars) is
|
||||
the capture mechanism for new registry entries; *holder* accumulates
|
||||
the turn's ``scaffold_stripped`` total for the API layer's log line.
|
||||
|
||||
``seed_docs`` are the documents the retrieval already put in context
|
||||
(they shape the *system_prompt* the caller built); re-reading one of
|
||||
them is rejected as "Already in your context." — the rejection counts
|
||||
in nothing, but it still consumes a round.
|
||||
them is rejected with :data:`ALREADY_IN_CONTEXT` (the phase-72
|
||||
teaching line — answer from the text already in the prompt) — the
|
||||
rejection counts in nothing, but it still consumes a round.
|
||||
"""
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "system", "content": system_prompt},
|
||||
@@ -482,6 +766,12 @@ async def run_agent(
|
||||
rounds = 0
|
||||
while True:
|
||||
calls: list[ToolCallPiece] = []
|
||||
# Phase 71: one fresh filter per round (one per model request —
|
||||
# the retry attempts of this logical request share it: a restart
|
||||
# only happens while the filter was never fed). Content-only:
|
||||
# thinking pieces pass through raw.
|
||||
round_filter = ScaffoldingFilter()
|
||||
round_content = 0 # visible (clean) content chars this round
|
||||
# Phase 48: bind the round's stream so a consumer abandon
|
||||
# (GeneratorExit into the yield below) tears down the in-flight
|
||||
# model stream deterministically — not GC-dependent. Phase 67:
|
||||
@@ -499,16 +789,97 @@ async def run_agent(
|
||||
tools=tools,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=round_filter,
|
||||
)
|
||||
try:
|
||||
async for piece in stream:
|
||||
if isinstance(piece, ToolCallPiece):
|
||||
calls.append(piece)
|
||||
elif isinstance(piece, StreamPiece) and piece.kind == "content":
|
||||
round_content += len(piece.text)
|
||||
yield piece
|
||||
finally:
|
||||
await stream.aclose()
|
||||
# Phase 71: the per-strip-event capture log — one warning per
|
||||
# stripped span, truncated to 200 chars (how a new scaffolding
|
||||
# format gets captured and added to the registry) — and the turn
|
||||
# total for the API layer's ``scaffold_stripped=N`` log field.
|
||||
holder.scaffold_stripped += round_filter.stripped_chars
|
||||
for span in round_filter.stripped_spans:
|
||||
logger.warning(
|
||||
"agent: stripped %d chars of tool-scaffolding in round %d: %r",
|
||||
len(span),
|
||||
rounds + 1,
|
||||
span[:200],
|
||||
)
|
||||
if not calls:
|
||||
return # the answer was streamed
|
||||
if round_content > 0:
|
||||
return # the answer was streamed (the clean content stands)
|
||||
if round_filter.stripped_chars == 0:
|
||||
# Empty/thinking-only answer — today's behavior, unchanged
|
||||
# (the UI handles it); the guardrail keys on a strip.
|
||||
return
|
||||
# Phase 71: the scaffolding was the whole "answer" — the ONE
|
||||
# bounded recovery (a fixed policy, not a conversation):
|
||||
# ``tools=None``, the correction folded into the ORIGINAL
|
||||
# single system message (provider-safe — the user message and
|
||||
# any tool history stay in place), a fresh filter, the same
|
||||
# phase-67 retry budget.
|
||||
logger.warning(
|
||||
"agent: round %d was pure tool-scaffolding (%d chars stripped) "
|
||||
"— running the one bounded recovery",
|
||||
rounds + 1,
|
||||
round_filter.stripped_chars,
|
||||
)
|
||||
messages_recovered = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt + "\n" + CORRECTION_INSTRUCTION,
|
||||
},
|
||||
*messages[1:],
|
||||
]
|
||||
recovery_filter = ScaffoldingFilter()
|
||||
recovered = chat_stream_retried(
|
||||
llm,
|
||||
cast("list[dict[str, str]]", messages_recovered),
|
||||
tools=None,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=recovery_filter,
|
||||
)
|
||||
recovery_content = 0
|
||||
try:
|
||||
async for piece in recovered:
|
||||
if isinstance(piece, StreamPiece) and piece.kind == "content":
|
||||
recovery_content += len(piece.text)
|
||||
yield piece
|
||||
finally:
|
||||
await recovered.aclose()
|
||||
holder.scaffold_stripped += recovery_filter.stripped_chars
|
||||
for span in recovery_filter.stripped_spans:
|
||||
logger.warning(
|
||||
"agent: stripped %d chars of tool-scaffolding in the "
|
||||
"recovery after round %d: %r",
|
||||
len(span),
|
||||
rounds + 1,
|
||||
span[:200],
|
||||
)
|
||||
if recovery_content > 0:
|
||||
return # the recovery answered — the turn ends
|
||||
# The second empty reply is terminal (at most one recovery per
|
||||
# turn). Raised OUTSIDE the stream, so chat_stream_retried's
|
||||
# retry rule never sees it; the API layer catches it before
|
||||
# the generic LLMError handler.
|
||||
logger.warning(
|
||||
"agent: the recovery reply was still empty "
|
||||
"(scaffold_stripped=%d) — settling with a malformed-reply error",
|
||||
holder.scaffold_stripped,
|
||||
)
|
||||
raise MalformedReplyError(
|
||||
f"the model answered in raw tool-scaffolding twice in a row "
|
||||
f"(round {rounds + 1} plus one recovery) — no clean answer "
|
||||
"to stream"
|
||||
)
|
||||
call = calls[0] # a stream can carry several calls; run the first
|
||||
result = _execute_tool(db, call, seed_docs, holder)
|
||||
rounds += 1 # every call the model emits consumes a round
|
||||
@@ -546,17 +917,46 @@ async def run_agent(
|
||||
# teardown as the loop rounds (consumer abandon mid-final
|
||||
# answer must still close the model's stream). Phase 67: the
|
||||
# forced call retries under the same locked-A2 rule as the
|
||||
# loop rounds.
|
||||
# loop rounds. Phase 71: the forced final runs through a
|
||||
# fresh filter too — raw scaffolding can never reach the
|
||||
# user from ANY grounded request.
|
||||
final_filter = ScaffoldingFilter()
|
||||
final = chat_stream_retried(
|
||||
llm,
|
||||
cast("list[dict[str, str]]", messages),
|
||||
tools=None,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=final_filter,
|
||||
)
|
||||
final_content = 0
|
||||
try:
|
||||
async for piece in final:
|
||||
if isinstance(piece, StreamPiece) and piece.kind == "content":
|
||||
final_content += len(piece.text)
|
||||
yield piece
|
||||
finally:
|
||||
await final.aclose()
|
||||
# Phase 71: the same capture log + turn total; a
|
||||
# scaffolding-only forced final (this turn used no recovery,
|
||||
# so nothing is doubled up) settles with the same terminal
|
||||
# malformed-reply error rather than a silently empty answer.
|
||||
holder.scaffold_stripped += final_filter.stripped_chars
|
||||
for span in final_filter.stripped_spans:
|
||||
logger.warning(
|
||||
"agent: stripped %d chars of tool-scaffolding in the "
|
||||
"forced final answer (round %d): %r",
|
||||
len(span),
|
||||
rounds + 1,
|
||||
span[:200],
|
||||
)
|
||||
if final_content == 0 and final_filter.stripped_chars > 0:
|
||||
logger.warning(
|
||||
"agent: the forced final answer was pure tool-scaffolding "
|
||||
"— settling with a malformed-reply error"
|
||||
)
|
||||
raise MalformedReplyError(
|
||||
"the forced final answer was raw tool-scaffolding — no "
|
||||
"clean answer to stream"
|
||||
)
|
||||
return
|
||||
|
||||
+51
-19
@@ -23,13 +23,18 @@ import json
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Literal, cast
|
||||
from typing import TYPE_CHECKING, Any, Literal, cast
|
||||
|
||||
from openai import AsyncOpenAI, AsyncStream
|
||||
from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Phase 71: the filter type is only needed for typing (the module
|
||||
# stays import-graph-clean; callers pass their own instances).
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
|
||||
logger = logging.getLogger("app.llm")
|
||||
|
||||
|
||||
@@ -74,12 +79,12 @@ class ToolCallPiece:
|
||||
``id`` is the model's tool_call id (synthesized as ``call_<index>``
|
||||
when the wire never carried one), ``name`` is the function name
|
||||
(whatever the caller's ``tools`` list names — for the agent loop,
|
||||
``list_documents`` / ``read_document``), and ``arguments`` is the
|
||||
``ls`` / ``read`` / ``grep``, phase 70), and ``arguments`` is the
|
||||
parsed JSON object (``{}`` when the model sent none).
|
||||
"""
|
||||
|
||||
id: str # the model's tool_call id; synthesized "call_<index>" when absent
|
||||
name: str # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
|
||||
name: str # "ls" | "read" | "grep" (whatever AGENT_TOOLS names, phase 70)
|
||||
arguments: dict[str, Any]
|
||||
|
||||
|
||||
@@ -124,7 +129,7 @@ def _materialize_tool_calls(
|
||||
Malformed ``arguments`` JSON raises :class:`LLMError` — a silently
|
||||
dropped tool call would corrupt the agent loop (fail-loud house
|
||||
style). Empty/``null`` arguments become ``{}`` (a no-parameter call
|
||||
such as ``list_documents``).
|
||||
such as an unscoped ``ls``).
|
||||
"""
|
||||
pieces: list[ToolCallPiece] = []
|
||||
for index in sorted(slots):
|
||||
@@ -338,6 +343,7 @@ class LLMClient:
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
scaffolding: ScaffoldingFilter | None = None,
|
||||
) -> AsyncGenerator[StreamPiece | ToolCallPiece, None]:
|
||||
"""Stream assistant pieces from the chat model (PLAN A5/A15, phase 17).
|
||||
|
||||
@@ -365,13 +371,26 @@ class LLMClient:
|
||||
``id`` and ``function.name`` on the first partial and
|
||||
``function.arguments`` in fragments — which are accumulated into
|
||||
one :class:`ToolCallPiece` per call, yielded in index order at
|
||||
stream end (or immediately once a chunk carries
|
||||
``finish_reason="tool_calls"``). Malformed ``arguments`` JSON
|
||||
raises :class:`LLMError`. Wire convention verified live against
|
||||
stream end (after the stream's chunks are exhausted — aipi ends
|
||||
the stream at ``finish_reason="tool_calls"``, so this is the
|
||||
wire's emission point). Malformed ``arguments`` JSON raises
|
||||
:class:`LLMError`. Wire convention verified live against
|
||||
aipi's ``turbo`` on 2026-08-26 via
|
||||
``uv run python -m scripts.llm_probe --tools`` (phase 37, task 01:
|
||||
``probe: turbo tool_calls=supported 2026-08-26``).
|
||||
|
||||
Scaffolding guardrail (phase 71): the caller may pass a
|
||||
``ScaffoldingFilter`` — one per request, caller-owned (this
|
||||
method never creates or resets one). When present, only
|
||||
**content** is filtered: ``delta.content`` is fed through the
|
||||
filter and only the clean text is yielded (an empty clean
|
||||
result yields **no** piece — no empty ``delta`` frames); thinking
|
||||
pieces are never filtered (the scratchpad stays raw, phase 17).
|
||||
At stream end the filter's held tail is flushed to a content
|
||||
piece **before** any tool-call materialization (content-
|
||||
before-tools wire convention). ``None`` (the default) keeps
|
||||
today's byte-identical raw path for callers that opt out.
|
||||
|
||||
Any failure (network, HTTP, malformed stream) surfaces as
|
||||
:class:`LLMError` so the API layer can turn it into an SSE
|
||||
``error`` event instead of a hung request.
|
||||
@@ -405,7 +424,6 @@ class LLMClient:
|
||||
await self._client.chat.completions.create(**kwargs),
|
||||
)
|
||||
calls: dict[int, _ToolCallSlot] = {}
|
||||
emitted = False
|
||||
async for chunk in stream:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
@@ -436,16 +454,22 @@ class LLMClient:
|
||||
yield StreamPiece("thinking", reasoning)
|
||||
content = delta.content
|
||||
if content:
|
||||
yield StreamPiece("content", content)
|
||||
if (
|
||||
calls
|
||||
and not emitted
|
||||
and getattr(choice, "finish_reason", None) == "tool_calls"
|
||||
):
|
||||
for piece in _materialize_tool_calls(calls):
|
||||
yield piece
|
||||
emitted = True
|
||||
if calls and not emitted:
|
||||
if scaffolding is not None:
|
||||
# Phase 71: content only — an empty clean result
|
||||
# yields nothing (no empty delta frames).
|
||||
cleaned = scaffolding.feed(content)
|
||||
if cleaned:
|
||||
yield StreamPiece("content", cleaned)
|
||||
else:
|
||||
yield StreamPiece("content", content)
|
||||
# Phase 71: flush the filter's held tail at stream end, BEFORE
|
||||
# any tool-call materialization — flushed-tail content precedes
|
||||
# ToolCallPieces (content-before-tools wire convention).
|
||||
if scaffolding is not None:
|
||||
tail = scaffolding.flush()
|
||||
if tail:
|
||||
yield StreamPiece("content", tail)
|
||||
if calls:
|
||||
for piece in _materialize_tool_calls(calls):
|
||||
yield piece
|
||||
except LLMError:
|
||||
@@ -469,6 +493,7 @@ async def chat_stream_retried(
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
retries: int = 0,
|
||||
delay: float = 0.0,
|
||||
scaffolding: ScaffoldingFilter | None = None,
|
||||
) -> AsyncGenerator[StreamPiece | ToolCallPiece | RetryPiece, None]:
|
||||
"""Stream a chat turn, retrying a dead endpoint (phase 67).
|
||||
|
||||
@@ -493,6 +518,13 @@ async def chat_stream_retried(
|
||||
The request is restarted byte-identical: ``chat_stream`` is stateless,
|
||||
so every attempt is opened with the SAME *messages*/*tools*.
|
||||
|
||||
Scaffolding guardrail (phase 71): *scaffolding* is passed through to
|
||||
every attempt's ``chat_stream``. The SAME caller-owned filter object
|
||||
across the retry attempts of one logical request is safe by
|
||||
construction: a restarted attempt only happens while no piece was
|
||||
emitted, i.e. the filter was never fed (its pending buffer is still
|
||||
empty).
|
||||
|
||||
Teardown (phase 48, extended): every attempt's stream is explicitly
|
||||
closed in a ``finally`` — normal exhaustion, a terminal
|
||||
:class:`LLMError`, and a consumer abandon (``GeneratorExit`` mid-attempt
|
||||
@@ -502,7 +534,7 @@ async def chat_stream_retried(
|
||||
max_attempts = retries + 1
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
emitted = False
|
||||
stream = llm.chat_stream(messages, tools=tools)
|
||||
stream = llm.chat_stream(messages, tools=tools, scaffolding=scaffolding)
|
||||
try:
|
||||
async for piece in stream:
|
||||
emitted = True
|
||||
|
||||
+118
-20
@@ -24,11 +24,32 @@ the ``<tuning>`` section (order: ``<relevance>`` →
|
||||
roughly what the KB contains before retrieval. With an empty row the
|
||||
prompt is byte-identical to the pre-phase text.
|
||||
|
||||
Agent tools (phase 37): the **HIGH** prompt only carries a ``<tools>``
|
||||
section after the ``<documents>`` body — the grounded turn may call the
|
||||
server-side ``list_documents`` / ``read_document`` tools (round-capped,
|
||||
see :mod:`app.rag.agent`). The LOW/deflection prompt never carries it
|
||||
and stays byte-identical to the pre-phase text.
|
||||
Agent tools (phase 37; phase 70: the copy teaches the harness-aligned
|
||||
``ls`` / ``read`` / ``grep`` shapes; phase 72: the copy states the
|
||||
document-identity contract up front — ``ls``'s ``path`` is a *source
|
||||
name*, not a directory or file path, and ``read``/``grep`` take the
|
||||
combined ``source/path`` string *including the source name* (a bare
|
||||
document path will not resolve) — the same two things the phase-72
|
||||
teaching refusals in :mod:`app.rag.agent` re-state after the fact, so
|
||||
the model carries the contract before it calls a tool): the **HIGH**
|
||||
prompt only carries a ``<tools>`` section after the ``<documents>``
|
||||
body — the grounded turn may extend its context through the three
|
||||
server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is
|
||||
the bound and this section does not re-state it, phase 45). The
|
||||
LOW/deflection prompt never carries it (phase 71: the LOW prompt's
|
||||
only addition is the plain-text line below — it still has no
|
||||
``<tools>`` section).
|
||||
|
||||
Deflection plain-text line (phase 71, owner-permitted 2026-09-03):
|
||||
the otherwise-locked ``LOW`` prompt gains exactly one instruction
|
||||
line — "Reply in plain text only — you have no tools in this mode."
|
||||
— appended to the ``DEFLECT_MODE`` body: a deflected turn offers no
|
||||
tools, so any tool markup there is always wrong, and the line closes
|
||||
the door at the prompt (the deterministic filter + one bounded
|
||||
recovery in :mod:`app.rag.scaffolding` / :mod:`app.rag.agent` is the
|
||||
backstop). The ``DEFLECT_MODE`` marker and everything else in the
|
||||
prompt stay put — the E2E mock LLM keys on the marker's *presence*,
|
||||
not the wording, so that contract is unchanged.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -73,21 +94,71 @@ _KB_INTRO = (
|
||||
)
|
||||
|
||||
#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
|
||||
#: task 03): a grounded turn may extend its context through the two
|
||||
#: server-side tools (round cap: ``BOR_AGENT_MAX_ROUNDS``, see
|
||||
#: :mod:`app.rag.agent`). Appended after
|
||||
#: the mode body (``<documents>``), so the instructions are the last
|
||||
#: thing the model reads. The LOW/deflection prompt never carries it —
|
||||
#: a deflection has no grounded context to extend — and stays
|
||||
#: byte-identical to the pre-phase text. The E2E mock keys off the
|
||||
#: ``<tools>`` marker's *presence*, not this wording.
|
||||
#: task 03; phase 70: the copy is rewritten for the harness-aligned
|
||||
#: ``ls`` / ``read`` / ``grep`` shapes, names/args exactly as the
|
||||
#: ``AGENT_TOOLS`` schemas in :mod:`app.rag.agent`; phase 72: the copy
|
||||
#: states the document-identity contract UP FRONT — ``ls``'s optional
|
||||
#: ``path`` is a *source name* (not a directory or file path) and
|
||||
#: ``read``/``grep`` take the combined ``source/path`` string *including
|
||||
#: the source name* (a bare document path will not resolve) — the same
|
||||
#: two things the phase-72 teaching refusals re-state after the fact):
|
||||
#: a grounded turn may extend its context through the three server-side
|
||||
#: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and
|
||||
#: this section does not re-state it, phase 45). Appended after the mode
|
||||
#: body (``<documents>``), so the instructions are the last thing the
|
||||
#: model reads. The LOW/deflection prompt never carries it — a
|
||||
#: deflection has no grounded context to extend (phase 71: the LOW
|
||||
#: prompt's only addition is the plain-text line in
|
||||
#: :func:`build_deflect_prompt`). The E2E mock keys off the ``<tools>``
|
||||
#: marker's *presence*, not this wording. Task 05 (the live gate's
|
||||
#: iteration loop) keeps the baseline and carries the discipline rules
|
||||
#: the live telemetry motivated, refined across the task-05 re-runs of
|
||||
#: 2026-09-03/04 (run 1: 7/16 executed, 44% — the ``lite`` model obeyed
|
||||
#: the user's "open it / read it" and ``read`` seed-context documents,
|
||||
#: then repeated the refused call; run 2: 8/18, 44% — the repeat is
|
||||
#: gone, but a grep turn batched two calls per reply and the harness
|
||||
#: runs only the first of a batch): the do-not-read rule names the
|
||||
#: user-command scenario (a document already in the ``<documents>``
|
||||
#: section: do not call ``read``, answer from the text already in the
|
||||
#: prompt — an anchor on the concrete ``<document path="...">`` markup
|
||||
#: was tried and REVERTED: it primed the model to latch the seed
|
||||
#: documents' paths as ``ls`` scopes, regressing the incident turn);
|
||||
#: the one-call rule names the consequence (a batched second call is
|
||||
#: discarded — runs only the first); the never-repeat rule says why
|
||||
#: (the refusal already told you the correct form); the ``grep`` clause
|
||||
#: leads with "for a normal search pass only ``pattern``" (the gate's
|
||||
#: live runs showed the model scoping ``grep`` with an ``ls``-style
|
||||
#: source name — the incident shape, but on grep). The behavioral
|
||||
#: contract lives in the ``AGENT_TOOLS`` descriptions as well (the most
|
||||
#: local text at call time): ``read`` must not be called for a
|
||||
#: ``<documents>`` document at all; ``grep`` with only ``pattern``
|
||||
#: searches the whole knowledge base, and a source name is not a
|
||||
#: document.
|
||||
TOOLS_SECTION: str = (
|
||||
"<tools>\n"
|
||||
"If the documents in your context reference other files, or you need "
|
||||
"content that is not included above, call `list_documents` to see what "
|
||||
"is indexed, then `read_document` to pull in exactly one more document. "
|
||||
"Answer as soon as you have what you need — do not read more than one "
|
||||
"extra document.\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
"indexed documents as `source: X | path: Y | title: Z` lines; its "
|
||||
"optional `path` argument is a source name (e.g. 'homelab'), not a "
|
||||
"directory or file path — omit it to list every document. `read` "
|
||||
"pulls in one document by its combined `source/path` string, "
|
||||
"exactly as shown in the `ls` output — including the source name — "
|
||||
"adding its full content to your context. Do not call `read` for a "
|
||||
"document already shown in the <documents> section, even when the "
|
||||
"user asks you to open or read it — its full text is already in "
|
||||
"your prompt; answer directly from it. For `read`, a bare document "
|
||||
"path (without the source name) will not resolve. `grep` locates an "
|
||||
"exact string (case-insensitive) in the indexed documents and "
|
||||
"returns up to 20 matching `source/path:line: text` lines — a "
|
||||
"locator, not a context-adder: read the winner with `read`; for a "
|
||||
"normal search pass only `pattern` — its optional `path` argument "
|
||||
"limits the search to one document you already know, by the same "
|
||||
"combined `source/path` string; never a source name — a bare "
|
||||
"document path (without the source name) will not resolve there "
|
||||
"either. Make exactly one tool call per reply — a reply carrying "
|
||||
"two tool calls runs only the first, the second is discarded — and "
|
||||
"wait for the result before the next call. Never repeat a call that "
|
||||
"was refused or already succeeded — the refusal already told you "
|
||||
"the correct form. Answer as soon as you have what you need.\n"
|
||||
"</tools>"
|
||||
)
|
||||
|
||||
@@ -180,14 +251,34 @@ def build_high_prompt(
|
||||
kb_overview: str | None = None,
|
||||
) -> str:
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + full
|
||||
texts of the top documents + the ``<tools>`` instructions (phase 37).
|
||||
texts of the top documents + the ``<tools>`` instructions (phase 37;
|
||||
phase 70: the harness-aligned ``ls`` / ``read`` / ``grep`` shapes;
|
||||
phase 72: the copy states the document-identity contract — the
|
||||
source-name ``ls`` scope, the combined ``source/path`` identity for
|
||||
``read``/``grep`` — up front).
|
||||
|
||||
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
|
||||
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
|
||||
section. ``<tools>`` is always present in the HIGH prompt (the round
|
||||
cap — not the prompt — decides whether the tools are actually
|
||||
offered to the model, see :mod:`app.rag.agent`).
|
||||
|
||||
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
|
||||
LEADING this section (the document texts are already context — do
|
||||
not ``read`` one the user asked to open) was tried and REVERTED:
|
||||
it never flipped the seed-doc reads (15/15 across gate runs 1-5)
|
||||
and correlated with the incident-turn regression (the model latched
|
||||
the seed documents' paths as ``ls`` scopes — cap reached on the
|
||||
"list the files in this directory" turn) whenever the copy named
|
||||
the ``<document>`` blocks explicitly.
|
||||
"""
|
||||
# 2026-09-04 (controlled tool-calling fast loop): the do-not-read
|
||||
# rule for seed documents lives in TOOLS_SECTION and the ``read``
|
||||
# tool descriptions (the copy levers that stuck — see the gate's
|
||||
# telemetry in TOOL_CALLING_TESTING.md). A per-block instruction
|
||||
# attribute at the ``source``/``path`` copy site was TRIED and
|
||||
# REVERTED the same day (no improvement across runs; the block stays
|
||||
# exactly the document identity + full text).
|
||||
blocks = [
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}">\n'
|
||||
f"{doc.content}\n"
|
||||
@@ -213,7 +304,10 @@ def build_deflect_prompt(
|
||||
|
||||
Section order (phase 31): ``<relevance>`` → ``<knowledge_base>`` →
|
||||
``<tuning>`` → ``DEFLECT_MODE`` body; empty steering/overview omit
|
||||
their section, keeping the prompt byte-identical to the pre-phase text.
|
||||
their section, keeping the prompt byte-identical to the pre-phase
|
||||
text. The body ends with the phase-71 plain-text line (owner-
|
||||
permitted 2026-09-03 — the LOW prompt's only change): a deflected
|
||||
turn offers no tools, so any tool markup there is always wrong.
|
||||
"""
|
||||
weak = "\n".join(f"- {t}" for t in titles) if titles else "(nothing close at all)"
|
||||
mid = "\n".join(
|
||||
@@ -228,5 +322,9 @@ def build_deflect_prompt(
|
||||
+ "DEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||||
"your notes come to the question. They are titles only; do not pretend "
|
||||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||||
# Phase 71 (owner-permitted 2026-09-03): the one plain-text line
|
||||
# — prevention at the prompt. The E2E mock keys on the
|
||||
# DEFLECT_MODE marker's presence, so appending is safe.
|
||||
"Reply in plain text only — you have no tools in this mode.\n"
|
||||
+ weak
|
||||
)
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Deterministic tool-scaffolding guardrail — pattern registry + streaming
|
||||
filter (phase 71, task 01).
|
||||
|
||||
The ``lite`` chat model occasionally emits its own chat-template
|
||||
tool-scaffolding as plain answer text (incident 2026-09-03: a deflected
|
||||
round streamed ``<|tool_call_start|>[read(path='…')]<|tool_call_end|>``
|
||||
into the UI although no tools were offered). This module is the
|
||||
detection half of the guardrail: a fixed registry of *observed*
|
||||
scaffolding forms and a streaming state machine that strips them from
|
||||
``delta.content`` as it flows. Pure Python — no I/O, no logging, no
|
||||
model: the strip *warning* log and the one bounded recovery are emitted
|
||||
by the integration layer (phase 71, tasks 02–03), so this module stays
|
||||
unit-testable in isolation.
|
||||
|
||||
Content only: thinking pieces are the model's raw reasoning by design
|
||||
(phase 17) and are never filtered — the guardrail protects the answer,
|
||||
not the scratchpad.
|
||||
"""
|
||||
import re
|
||||
|
||||
__all__ = ["SCAFFOLD_PATTERNS", "ScaffoldingFilter"]
|
||||
|
||||
#: The known scaffolding forms. The registry is the extension point: a
|
||||
#: new entry needs an observed capture (the strip warning log, task 03)
|
||||
#: + a unit fixture in ``tests/unit/test_scaffolding_filter.py`` — no
|
||||
#: speculative entries. Every entry here traces to the 2026-09-03
|
||||
#: incident (the span form is the deflected round's raw text; the
|
||||
#: standalone siblings are from the same tokenizer family).
|
||||
SCAFFOLD_PATTERNS: tuple[re.Pattern, ...] = (
|
||||
# The observed span (incident 2026-09-03) — non-greedy, so multiple
|
||||
# spans in one buffer each strip to their own end token.
|
||||
re.compile(r"<\|tool_call_start\|>[\s\S]*?<\|tool_call_end\|>"),
|
||||
# Standalone sibling token (same tokenizer family).
|
||||
re.compile(r"<\|tool_calls\|>"),
|
||||
# Standalone sibling token (same tokenizer family).
|
||||
re.compile(r"<\|tool_call\|>"),
|
||||
)
|
||||
|
||||
_SPAN_START = "<|tool_call_start|>"
|
||||
_SPAN_END = "<|tool_call_end|>"
|
||||
#: The literal openings a live (possibly incomplete) match can begin with.
|
||||
_OPENINGS: tuple[str, ...] = (_SPAN_START, "<|tool_calls|>", "<|tool_call|>")
|
||||
#: Boundedness bound: in NORMAL state the held tail is at most one char
|
||||
#: short of the longest opening (an open span is unbounded, but it is
|
||||
#: being *stripped*, never emitted).
|
||||
_MAX_OPENING = max(len(opening) for opening in _OPENINGS)
|
||||
|
||||
|
||||
def _leftmost_match(buf: str) -> re.Match | None:
|
||||
"""The leftmost complete match among :data:`SCAFFOLD_PATTERNS`, or
|
||||
None when the buffer holds no complete scaffolding form.
|
||||
|
||||
(Two patterns cannot match at the same start position — their
|
||||
literals diverge after ``<|tool_call`` — so the leftmost start is
|
||||
unambiguous.)
|
||||
"""
|
||||
best: re.Match | None = None
|
||||
for pattern in SCAFFOLD_PATTERNS:
|
||||
match = pattern.search(buf)
|
||||
if match is not None and (best is None or match.start() < best.start()):
|
||||
best = match
|
||||
return best
|
||||
|
||||
|
||||
def _hold_index(buf: str) -> int:
|
||||
"""Index where the live hold-tail begins; everything from it stays
|
||||
pending (called only after the strip loop found no complete match).
|
||||
|
||||
The tail is the leftmost of two candidates: an **open span** (a
|
||||
start token with no end token after it — the span may continue in
|
||||
future chunks, so everything from that start token onward is held)
|
||||
or a **live prefix** (the longest suffix that is a proper prefix of
|
||||
any opening literal — the token may complete in future chunks).
|
||||
Everything before the leftmost candidate is safe to emit: no
|
||||
complete match remains, and no match can grow from earlier text,
|
||||
since a complete opening earlier in the buffer would already have
|
||||
matched (span) or be a candidate in its own right (standalone).
|
||||
"""
|
||||
if not buf:
|
||||
return 0
|
||||
hold = len(buf)
|
||||
start = buf.find(_SPAN_START)
|
||||
if start != -1 and buf.find(_SPAN_END, start + 1) == -1:
|
||||
hold = start
|
||||
for k in range(min(len(buf), _MAX_OPENING - 1), 0, -1):
|
||||
suffix = buf[-k:]
|
||||
if any(k < len(opening) and opening.startswith(suffix) for opening in _OPENINGS):
|
||||
hold = min(hold, len(buf) - k)
|
||||
break
|
||||
return hold
|
||||
|
||||
|
||||
class ScaffoldingFilter:
|
||||
"""Streaming state machine over a model **content** stream (phase 71).
|
||||
|
||||
One instance per model request (callers create fresh instances —
|
||||
the phase-67 retry-after-dead-attempt case is safe: a dead attempt
|
||||
never fed the filter). Feed the raw ``delta.content`` chunks; each
|
||||
:meth:`feed` returns the clean text safe to emit *now*; :meth:`flush`
|
||||
emits the tail at stream end.
|
||||
|
||||
``feed`` appends the chunk to the internal pending buffer, then
|
||||
(a) repeatedly takes the leftmost complete match among
|
||||
:data:`SCAFFOLD_PATTERNS` — drops it (counting it into
|
||||
:attr:`stripped_chars`) and continues — until none remains; then
|
||||
(b) checks the buffer tail for a **live prefix** via
|
||||
:func:`_hold_index`: the longest suffix that is a proper prefix of
|
||||
any opening literal, or an **open span** (a start token with no end
|
||||
token yet — everything from that start token onward is held, since
|
||||
the span may continue in future chunks). Everything before the held
|
||||
tail is emitted; the held tail becomes the new pending state.
|
||||
Bounded: the held tail in NORMAL state is ≤ the longest opening
|
||||
token minus one char; in an open span it is unbounded but is being
|
||||
*stripped*, never emitted.
|
||||
|
||||
``flush`` (end of stream) emits the pending tail **as-is**: a
|
||||
partial marker at EOF is content, not scaffolding — a documented,
|
||||
pinned choice (a stream that ends mid-``<|tool_call_st`` must not
|
||||
be silently eaten, and a lone ``<|tool_call_end|>`` without a start
|
||||
is prose the user sent or the model produced outside a span).
|
||||
|
||||
The stripped spans themselves are exposed read-only
|
||||
(:attr:`stripped_spans`) so the integration layer can log one
|
||||
warning per strip event (phase 71 task 03) — that log line is how
|
||||
a *new* scaffolding format gets captured and added to the registry.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._pending = ""
|
||||
self._stripped_chars = 0
|
||||
self._stripped_spans: list[str] = []
|
||||
|
||||
@property
|
||||
def stripped_chars(self) -> int:
|
||||
"""Total characters removed so far (read by the caller after the
|
||||
round/turn — the strip warning log, phase 71 task 03)."""
|
||||
return self._stripped_chars
|
||||
|
||||
@property
|
||||
def stripped_spans(self) -> list[str]:
|
||||
"""The raw spans removed so far, in stream order (one entry per
|
||||
strip event — the integration layer's per-span warning log
|
||||
truncates each to 200 chars). Read-only: callers must not
|
||||
mutate the filter's state."""
|
||||
return list(self._stripped_spans)
|
||||
|
||||
def feed(self, chunk: str) -> str:
|
||||
"""Append *chunk* to the pending buffer; return the clean text
|
||||
safe to emit now (possibly ``""`` — e.g. while an open span or a
|
||||
partial marker is still held). An empty chunk is a no-op."""
|
||||
if not chunk:
|
||||
return ""
|
||||
buf = self._strip_complete(self._pending + chunk)
|
||||
hold = _hold_index(buf)
|
||||
self._pending = buf[hold:]
|
||||
return buf[:hold]
|
||||
|
||||
def flush(self) -> str:
|
||||
"""End of stream: emit the pending tail as-is and reset it."""
|
||||
tail = self._pending
|
||||
self._pending = ""
|
||||
return tail
|
||||
|
||||
def _strip_complete(self, buf: str) -> str:
|
||||
"""Drop leftmost-complete matches until none remains (step a)."""
|
||||
while (match := _leftmost_match(buf)) is not None:
|
||||
self._stripped_chars += match.end() - match.start()
|
||||
self._stripped_spans.append(match.group(0))
|
||||
buf = buf[: match.start()] + buf[match.end():]
|
||||
return buf
|
||||
+23
-18
@@ -71,24 +71,26 @@ class ChatThinkingEvent(BaseModel):
|
||||
class ChatToolEvent(BaseModel):
|
||||
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension).
|
||||
|
||||
A15 extension (owner permission 2026-08-26; ``search_documents``
|
||||
added in phase 68): a grounded turn may call the server-side
|
||||
document tools (``list_documents`` / ``read_document`` /
|
||||
``search_documents``, see :mod:`app.rag.agent`); each model-requested
|
||||
A15 extension (owner permission 2026-08-26; the grep added in phase
|
||||
68; phase 70 aligned the surface to the harness-trained
|
||||
``ls`` / ``read`` / ``grep`` — owner permission 2026-09-03): a
|
||||
grounded turn may call the server-side document tools (``ls`` /
|
||||
``read`` / ``grep``, see :mod:`app.rag.agent`); each model-requested
|
||||
call streams as ``{type: "tool", name: str, argument: str | null}``
|
||||
ahead of the answer's ``delta`` frames. ``argument`` is the read
|
||||
document's ``"source/path"`` for ``read_document``, the search
|
||||
pattern for ``search_documents``, and null otherwise (a non-string
|
||||
pattern — a model error the backend refuses — is null). The client
|
||||
renders each frame as a "calling tool" line/state (phase 37 task 05);
|
||||
the ``delta`` / ``done`` shapes are unchanged — the read document is
|
||||
reflected in ``done.sources`` instead (a search adds no source: it is
|
||||
a locator, locked A5).
|
||||
ahead of the answer's ``delta`` frames. ``argument`` is the single
|
||||
string argument the model passed — ``read``'s ``path`` (the combined
|
||||
``source/path``), ``grep``'s ``pattern``, ``ls``'s ``path`` — or
|
||||
null (a non-string value, a model error the backend refuses, and an
|
||||
omitted argument both yield null). The client renders each frame as
|
||||
a "calling tool" line/state (phase 37 task 05); the ``delta`` /
|
||||
``done`` shapes are unchanged — the read document is reflected in
|
||||
``done.sources`` instead (a grep adds no source: it is a locator,
|
||||
locked A5).
|
||||
"""
|
||||
|
||||
type: Literal["tool"] = "tool"
|
||||
name: str # "list_documents" | "read_document" | "search_documents"
|
||||
argument: str | None = None # "source/path" for read_document, pattern for search_documents
|
||||
name: str # "ls" | "read" | "grep" (whatever AGENT_TOOLS names)
|
||||
argument: str | None = None # the single string argument passed, or null
|
||||
|
||||
|
||||
class ChatDoneEvent(BaseModel):
|
||||
@@ -375,10 +377,13 @@ class ToolCall(BaseModel):
|
||||
"""One agent tool-call record (the phase-37 ``tools`` record shape).
|
||||
|
||||
Mirrors the ``{name, argument}`` pair the SSE ``tool`` frames carry
|
||||
(PLAN §4 extension): ``argument`` is the read document's
|
||||
``"source/path"`` for ``read_document`` and null otherwise. Stored
|
||||
inside :class:`ChatMessage.tools` so a saved chat restores the
|
||||
"calling tool" lines pixel-identical (phase 50).
|
||||
(PLAN §4 extension; phase 70): ``argument`` is the single string
|
||||
argument the model passed (``read``'s combined ``source/path``,
|
||||
``grep``'s pattern, ``ls``'s scope) or null. Stored inside
|
||||
:class:`ChatMessage.tools` so a saved chat restores the "calling
|
||||
tool" lines pixel-identical (phase 50). Saved chats persisting the
|
||||
pre-phase-70 tool names still validate — ``name`` is opaque
|
||||
(no migration, locked).
|
||||
"""
|
||||
|
||||
name: str
|
||||
|
||||
+53
-23
@@ -56,21 +56,24 @@
|
||||
* phase 34 task 02) clears the key + the list back to the empty state.
|
||||
*
|
||||
* Agent tool calls (phase 37, PLAN §4 extension; phase 68 added
|
||||
* search_documents): a grounded turn may call the three server-side
|
||||
* document tools (list_documents / read_document / search_documents,
|
||||
* bounded only by the round cap — phases 45/68). Each call streams a
|
||||
* `tool` SSE frame, and the UI shows the "calling tool" state IN
|
||||
* ADDITION to "thinking": the UI state itself stays "thinking" (the
|
||||
* button stays the enabled "Stop" control — phase 48 — never stale,
|
||||
* PLAN §7.4) while the STATUS LABELS change — the #send-status +
|
||||
* typing-indicator labels say what Brain is doing ("…is listing
|
||||
* documents" / "…is reading source/path" / "…is searching for
|
||||
* pattern" — the name prefix resolves from window.BOR_BRAND at call
|
||||
* time, phase 39) — the button no longer relabels to "Calling tool…"
|
||||
* (phase 48, owner-locked: it stays "Stop" for the whole turn) — and a
|
||||
* visible `.tool-call` line (own icon + accent color, distinct from
|
||||
* the brand-ink Thinking block) is appended above the answer, one per
|
||||
* call, in order.
|
||||
* search_documents; phase 70 remapped the surface to the harness
|
||||
* names ls / read(path) / grep(pattern, path?)): a grounded turn may
|
||||
* call the three server-side document tools, bounded only by the
|
||||
* round cap (phases 45/68). Each call streams a `tool` SSE frame, and
|
||||
* the UI shows the "calling tool" state IN ADDITION to "thinking": the
|
||||
* UI state itself stays "thinking" (the button stays the enabled
|
||||
* "Stop" control — phase 48 — never stale, PLAN §7.4) while the STATUS
|
||||
* LABELS change — the #send-status + typing-indicator labels say what
|
||||
* Brain is doing ("…is listing documents" / "…is reading source/path"
|
||||
* / "…is searching for pattern" — the name prefix resolves from
|
||||
* window.BOR_BRAND at call time, phase 39) — the button no longer
|
||||
* relabels to "Calling tool…" (phase 48, owner-locked: it stays
|
||||
* "Stop" for the whole turn) — and a visible `.tool-call` line (own
|
||||
* icon + accent color, distinct from the brand-ink Thinking block) is
|
||||
* appended above the answer, one per call, in order. Phase 70: the
|
||||
* line/label branches key off the NEW names and still carry the legacy
|
||||
* ones (list_documents / read_document / search_documents) — persisted
|
||||
* turns from before the remap render exactly as before (no migration).
|
||||
* Append-only like thinking: frames are tolerated in any interleaving
|
||||
* (a frame after the first delta just appends — the agent loop never
|
||||
* emits one, but it must not crash). The turn record persists an
|
||||
@@ -810,9 +813,22 @@ function closeThinkingBlock(wrap) {
|
||||
* interleaving with thinking frames, even after the first delta (the
|
||||
* agent loop never emits one, but a late frame must not crash) — just
|
||||
* append another line, in order. The SAME helper re-renders the
|
||||
* persisted lines on restore (phase 14 convention): the path/pattern
|
||||
* argument goes through textContent, so nothing HTML-shaped can come
|
||||
* from storage. Lines are not interactive (no focus targets). */
|
||||
* persisted lines on restore (phase 14 convention): every argument
|
||||
* (path / pattern / source scope) goes through textContent, so nothing
|
||||
* HTML-shaped can come from storage. Lines are not interactive (no
|
||||
* focus targets).
|
||||
*
|
||||
* Phase 70 (owner permission 2026-09-03): the server tools were remapped
|
||||
* to the harness-aligned surface — ls / read(path) / grep(pattern,
|
||||
* path?) — so the NEW names get their own lines (read → the Reading
|
||||
* line, grep → the Searching-for line, ls → the Listing-documents line,
|
||||
* a scoped ls → the Listing-documents-in-<scope> line), and the
|
||||
* pre-phase-70 names (list_documents / read_document /
|
||||
* search_documents) still render EXACTLY as before: persisted turns
|
||||
* (phase 14) carry the old names, so both generations render — no
|
||||
* migration. The content marks (the read/search/list glyphs) stay the
|
||||
* exact tool-line template literals — the frontend emoji guard strips
|
||||
* precisely those in this file. */
|
||||
function appendToolLine(wrap, name, argument) {
|
||||
const body = wrap?.querySelector?.(".msg-body");
|
||||
if (!body) return;
|
||||
@@ -829,16 +845,24 @@ function appendToolLine(wrap, name, argument) {
|
||||
const line = document.createElement("span");
|
||||
line.className = "tool-call";
|
||||
line.setAttribute("role", "listitem");
|
||||
if (name === "read_document" && argument) {
|
||||
// Phase 70: new names first, legacy names kept — a restored turn saved
|
||||
// before the remap (read_document / search_documents / list_documents)
|
||||
// renders byte-identical to before (no migration).
|
||||
if ((name === "read" || name === "read_document") && argument) {
|
||||
line.textContent = "📄 Reading ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the path is data, never markup
|
||||
line.appendChild(code);
|
||||
} else if (name === "search_documents" && argument) {
|
||||
} else if ((name === "grep" || name === "search_documents") && argument) {
|
||||
line.textContent = "🔎 Searching for ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the pattern is data, never markup
|
||||
line.appendChild(code);
|
||||
} else if (name === "ls" && argument) {
|
||||
line.textContent = "🔎 Listing documents in ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the source scope is data, never markup
|
||||
line.appendChild(code);
|
||||
} else {
|
||||
line.textContent = "🔎 Listing documents";
|
||||
}
|
||||
@@ -1940,12 +1964,18 @@ async function runTurn(text, { reask = false } = {}) {
|
||||
toolAcc.push({ name, argument });
|
||||
clearTurnTimeout(); // the stream is alive — a frame arrived
|
||||
if (!wrap) wrap = addMessage("brain", "");
|
||||
// Phase 70: the harness-aligned names (read/grep/ls) map to the
|
||||
// same status copy as their legacy counterparts (read_document /
|
||||
// search_documents) — a pre-remap frame keeps its label; the
|
||||
// scoped ls mirrors the scoped tool line.
|
||||
const toolStatus =
|
||||
name === "read_document" && argument
|
||||
(name === "read" || name === "read_document") && argument
|
||||
? `${brand()} is reading ${argument}`
|
||||
: name === "search_documents" && argument
|
||||
: (name === "grep" || name === "search_documents") && argument
|
||||
? `${brand()} is searching for ${argument}`
|
||||
: `${brand()} is listing documents`;
|
||||
: name === "ls" && argument
|
||||
? `${brand()} is listing documents in ${argument}`
|
||||
: `${brand()} is listing documents`;
|
||||
if (uiState === UI_STATE.thinking) {
|
||||
sendStatus.textContent = toolStatus;
|
||||
document
|
||||
|
||||
@@ -133,14 +133,22 @@ function addThinkingBlock(wrap, thinking) {
|
||||
body.insertBefore(block, body.querySelector(".bubble"));
|
||||
}
|
||||
|
||||
/* Tool-call lines (phase 37) — the local copy of the chat page's
|
||||
* appendToolLine: one visible "calling tool" row per saved
|
||||
* {name, argument} record, in saved order, above the answer. The
|
||||
* path argument goes through textContent, so nothing HTML-shaped can
|
||||
* come from storage. Lines are not interactive (no focus targets).
|
||||
* The two content marks (the read glyph / the list glyph) are the
|
||||
* exact app.js template strings — the frontend emoji guard strips
|
||||
* precisely those two literals in this file, as in app.js. */
|
||||
/* Tool-call lines (phase 37; phase 70 remapped the tool names to the
|
||||
* harness surface ls / read(path) / grep(pattern, path?)) — the local
|
||||
* copy of the chat page's appendToolLine: one visible "calling tool"
|
||||
* row per saved {name, argument} record, in saved order, above the
|
||||
* answer. Every argument (path / pattern / source scope) goes through
|
||||
* textContent, so nothing HTML-shaped can come from storage. Lines
|
||||
* are not interactive (no focus targets). Phase 70: the NEW names
|
||||
* render (read → the Reading line, grep → the Searching-for line,
|
||||
* ls → the Listing-documents line, scoped ls → the
|
||||
* Listing-documents-in-<scope> line), and the pre-phase-70 names
|
||||
* (read_document / search_documents / list_documents) still render
|
||||
* exactly as before — a row saved before the remap keeps its exact
|
||||
* line (no migration). The content marks are the exact app.js
|
||||
* template strings — the frontend emoji guard (tests/integration/
|
||||
* test_api.py) strips precisely those literals in this file, as in
|
||||
* app.js. */
|
||||
function addToolLines(wrap, tools) {
|
||||
if (!Array.isArray(tools) || !tools.length) return;
|
||||
const body = wrap.querySelector(".msg-body");
|
||||
@@ -156,11 +164,25 @@ function addToolLines(wrap, tools) {
|
||||
line.setAttribute("role", "listitem");
|
||||
const argument =
|
||||
typeof t.argument === "string" && t.argument ? t.argument : null;
|
||||
if (t.name === "read_document" && argument) {
|
||||
// Phase 70: new names first, legacy names kept — a conversation
|
||||
// saved before the remap renders byte-identical (no migration).
|
||||
if ((t.name === "read" || t.name === "read_document") && argument) {
|
||||
line.textContent = "📄 Reading ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the path is data, never markup
|
||||
line.appendChild(code);
|
||||
} else if (
|
||||
(t.name === "grep" || t.name === "search_documents") && argument
|
||||
) {
|
||||
line.textContent = "🔎 Searching for ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the pattern is data, never markup
|
||||
line.appendChild(code);
|
||||
} else if (t.name === "ls" && argument) {
|
||||
line.textContent = "🔎 Listing documents in ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the source scope is data, never markup
|
||||
line.appendChild(code);
|
||||
} else {
|
||||
line.textContent = "🔎 Listing documents";
|
||||
}
|
||||
|
||||
@@ -0,0 +1,816 @@
|
||||
"""The real-model tool-calling gate (live, the configured chat model).
|
||||
|
||||
The phase-72 pass condition (owner directive 2026-09-03 — "test with the
|
||||
real lite model until tool calls work consistently; don't pass until a
|
||||
sufficient number of tool calls succeed"): this script drives a fixed
|
||||
question battery through the **real** grounded path — the exact mirror
|
||||
of ``app.api.chat`` (embed → retrieve → the honesty gate via
|
||||
``plan_turn`` → the steering notes + KB overview exactly as
|
||||
``app.api.chat`` reads them → ``build_high_prompt`` / deflection prompt →
|
||||
``run_agent`` with the configured chat model and the real Postgres KB, a
|
||||
fresh ``AgentHolder`` per turn; a deflected turn runs the same
|
||||
``tools=None`` + one-bounded-recovery stream the deflected API branch
|
||||
runs) — and applies the four LOCKED pass conditions:
|
||||
|
||||
1. all turns answer (no ``LLMError`` / ``MalformedReplyError``);
|
||||
2. zero turns hit the round cap (the incident's loop signature — hitting
|
||||
the cap means the teaching did not end the loop);
|
||||
3. >=6 of 10 turns emit >=1 tool call (the model keeps USING tools — it
|
||||
does not abandon them and answer from seed context alone, the
|
||||
incident's end state);
|
||||
4. the accuracy bar across the whole run — ``derived`` mode (the
|
||||
phase-72 LOCKED gate): executed / emitted >= 0.90 (refusals count in
|
||||
nothing); ``fixture`` mode (the controlled methodology): contract
|
||||
accuracy — well-formed calls targeting resolvable entities —
|
||||
>= 0.90, with the executed ratio reported alongside (see
|
||||
``classify_call`` and ``TOOL_CALLING_TESTING.md`` for why the two
|
||||
metrics differ: the app's ALREADY_IN_CONTEXT dedupe refusal is an
|
||||
app-semantics choice, not a tool-calling error).
|
||||
|
||||
Batteries (``--mode``):
|
||||
|
||||
* ``derived`` (default, the phase-72 LOCKED battery) — the fixed
|
||||
10-question battery derived from the live catalog's first two documents
|
||||
``D1 = (s1, p1, t1)`` / ``D2 = (s2, p2, t2)`` (catalog order); the
|
||||
grep question's token is the first whitespace-split word of
|
||||
``D2.content`` with length >= 6 (leading/trailing non-alphanumerics
|
||||
stripped, lowercased), falling back to the first word of ``t2``.
|
||||
* ``fixture`` (the controlled fast loop, 2026-09-04 owner directive) —
|
||||
:data:`FIXTURE_BATTERY`, the curated 10 questions pinned to the
|
||||
hand-written fixture KB (``tests/fixtures/agent_kb/``). Combine with
|
||||
``--restore`` so the whole iteration is one command against a known,
|
||||
unguessable, re-embed-free knowledge base (``TOOL_CALLING_TESTING.md``).
|
||||
|
||||
Speed levers (the fast loop): ``--restore`` (restore the fixture dump in
|
||||
one transaction — no git clone, no re-embedding, sub-second); ``--turns
|
||||
N`` (run only the first N questions — the micro-loop for copy
|
||||
iteration; the verdict is then marked ``partial`` and condition 3 is
|
||||
reported, not gated); every turn line carries its wall seconds and the
|
||||
verdict line carries the run's total wall time, so a slow-down is
|
||||
visible in the same line that carries the accuracy.
|
||||
|
||||
House probe pattern (``scripts/llm_probe.py``): ``uv run python -m
|
||||
scripts.agent_realmodel_check`` — argparse, dotenv, plain module, no
|
||||
debugpy. The script never modifies the KB (no commits, no query_log
|
||||
rows — the only writes are the ``--restore`` snapshot restore, which is
|
||||
explicit and transactional).
|
||||
|
||||
Preconditions (exit 2 with an actionable line on failure): the DB is
|
||||
reachable; ``BOR_AGENT_MAX_ROUNDS`` is > 0 (the gate needs tools
|
||||
enabled); ``derived`` mode — the catalog holds >=2 documents and the
|
||||
FIRST TWO catalog documents' ``path``s each contain ``/`` (the
|
||||
bare-path traps need nested paths); ``fixture`` mode — the fixture dump
|
||||
exists (build it with ``uv run python -m scripts.load_test_kb``).
|
||||
|
||||
Exit codes: **0 PASS**, **1 FAIL** (the per-condition breakdown is
|
||||
printed so the copy-lever iteration loop can target the right lever),
|
||||
**2 precondition failure**. For refusal diagnosis, every call is
|
||||
already logged by ``run_agent`` (``agent tool=… args=… round=…/…``) —
|
||||
correlate the logged arguments with the refusal templates in
|
||||
``app/rag/agent.py`` to see which teaching line the model hit.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from app.api.chat import plan_turn
|
||||
from app.api.steering import load_steering_notes
|
||||
from app.config import Settings, get_settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.rag.agent import (
|
||||
CORRECTION_INSTRUCTION,
|
||||
AgentHolder,
|
||||
MalformedReplyError,
|
||||
find_document,
|
||||
list_catalog,
|
||||
list_source_names,
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.llm import (
|
||||
EmbeddingError,
|
||||
LLMClient,
|
||||
LLMError,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
chat_stream_retried,
|
||||
)
|
||||
from app.rag.overview import load_kb_overview
|
||||
from app.rag.retriever import retrieve
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
|
||||
logger = logging.getLogger("agent_realmodel_check")
|
||||
|
||||
#: Repo-relative fixture dump (written by scripts.load_test_kb).
|
||||
DEFAULT_DUMP_PATH = Path("tests/fixtures/test_kb.dump.sql")
|
||||
|
||||
#: The per-turn line truncates the question at this width (the locked
|
||||
#: format prints ``turn NN | emitted=E executed=X cap=Y|N | <question>``).
|
||||
QUESTION_DISPLAY_WIDTH = 40
|
||||
|
||||
#: The controlled fast-loop battery (2026-09-04, owner directive): the
|
||||
#: curated 10 questions pinned to the fixture KB
|
||||
#: (``tests/fixtures/agent_kb/`` — sources ``deployments`` / ``homelab``,
|
||||
#: 8 hand-written documents whose specifics — ``rack7``,
|
||||
#: ``10.77.42.0/24``, VLAN 130, port 18443, ntfy topic
|
||||
#: ``reese-uptime-7``, machine ID ``rbm-8842``, the ``17 2 * * *``
|
||||
#: schedule, image ``ghcr.io/reese/obsidian-bor:2026.7.14``, port
|
||||
#: 18765 — are not guessable by any model).
|
||||
#:
|
||||
#: Design rule (the controlled-test property): every question has ONE
|
||||
#: unambiguously correct tool behavior, verified by the load script's
|
||||
#: retrieval report. The ``read`` targets (Q4/Q5/Q6) are named so the
|
||||
#: question's tokens do NOT lexically seed the target document (the
|
||||
#: filenames carry no topical words the content repeats) — the read
|
||||
#: must actually happen, exactly once, in the combined form. The
|
||||
#: discipline turns (Q7/Q8/Q9) name content that IS seeded — the
|
||||
#: correct behavior there is to answer from the ``<documents>`` context
|
||||
#: (or ``ls``), NOT to re-read. The bare-path traps are the job of the
|
||||
#: locked derived battery, not this one.
|
||||
#:
|
||||
#: 1. the phase-72 incident ("list the files in this directory" — a
|
||||
#: full listing needs ``ls``: 8 docs, 2 in seed context);
|
||||
#: 2. a scoped ``ls`` by the correct source name;
|
||||
#: 3. the no-arg listing;
|
||||
#: 4. a ``read`` of an unseeded document (combined form, given whole);
|
||||
#: 5. a second ``read`` of an unseeded document (explicit "Read …");
|
||||
#: 6. a third ``read`` of an unseeded document;
|
||||
#: 7. the ``grep`` turn (``rbm-8842`` — a string that occurs in exactly
|
||||
#: one fixture document; the grep line alone answers "which ones");
|
||||
#: 8. a title lookup — the target IS seeded; summarize from context;
|
||||
#: 9. a topic lookup — the target IS seeded; answer from context;
|
||||
#: 10. the source name phrased as a directory (the ``ls(path=…)`` scope
|
||||
#: trap).
|
||||
FIXTURE_BATTERY: list[str] = [
|
||||
"List the files in this directory.",
|
||||
"List the documents you have in the homelab source.",
|
||||
"List every document you have indexed.",
|
||||
"Open the document homelab/networking/vela-bridges.md and tell me what it covers.",
|
||||
"Read deployments/quadlet/mimir-service.md and summarize it.",
|
||||
"Open the document homelab/networking/meridian-notes.md and tell me what it covers.",
|
||||
'Find the exact string "rbm-8842" in your documents and tell me which ones '
|
||||
"contain it.",
|
||||
'Which document has the title "Lab Ansible Inventory"? Summarize it.',
|
||||
"What do you know about the qwen 3.8 llama.cpp setup? Give me the exact "
|
||||
"launch arguments.",
|
||||
"List the files in the deployments directory.",
|
||||
]
|
||||
|
||||
|
||||
def _alnum_edge(word: str) -> str:
|
||||
"""Strip leading/trailing non-alphanumerics from *word*."""
|
||||
start = 0
|
||||
end = len(word)
|
||||
while start < end and not word[start].isalnum():
|
||||
start += 1
|
||||
while end > start and not word[end - 1].isalnum():
|
||||
end -= 1
|
||||
return word[start:end]
|
||||
|
||||
|
||||
def derive_token(content: str, title: str) -> str:
|
||||
"""The derived battery's grep token (locked by the phase-72 task
|
||||
file). The first whitespace-split word of *content* whose stripped
|
||||
length is >= 6 (leading/trailing non-alphanumerics stripped,
|
||||
lowercased); the fallback is the first word of *title* (the same
|
||||
cleanup)."""
|
||||
for word in content.split():
|
||||
token = _alnum_edge(word).lower()
|
||||
if len(token) >= 6:
|
||||
return token
|
||||
words = title.split()
|
||||
return _alnum_edge(words[0]).lower() if words else "document"
|
||||
|
||||
|
||||
def build_battery(
|
||||
catalog: list[tuple[str, str, str]], d2_content: str
|
||||
) -> list[str]:
|
||||
"""The fixed 10-question battery (locked by the phase-72 task file —
|
||||
do not swap in easier questions), derived from the live catalog's
|
||||
first two documents ``D1 = (s1, p1, t1)`` / ``D2 = (s2, p2, t2)``
|
||||
(catalog order):
|
||||
|
||||
1. the incident ("list the files in this directory" — the
|
||||
harness-prior ``ls(path='.')`` misuse);
|
||||
2. a scoped ``ls`` by the correct source name (``s1``);
|
||||
3. the no-arg listing;
|
||||
4. a bare-path ``read`` trap (``p1`` without its source prefix);
|
||||
5. the combined form (the correct shape);
|
||||
6. a second bare-path ``read`` trap (``p2``);
|
||||
7. the ``grep`` turn (the derived token — guaranteed to occur in
|
||||
D2's content);
|
||||
8. a title lookup + read (``t2``);
|
||||
9. a title lookup + open (``t1``);
|
||||
10. the source name phrased as a directory (``s2`` — the
|
||||
``ls(path=…)`` scope trap).
|
||||
"""
|
||||
(s1, p1, t1), (s2, p2, t2) = catalog[0], catalog[1]
|
||||
token = derive_token(d2_content, t2)
|
||||
return [
|
||||
"List the files in this directory.",
|
||||
f"List the documents you have in the {s1} source.",
|
||||
"List every document you have indexed.",
|
||||
f"What does the document {p1} contain? Open it and tell me.",
|
||||
f"Read {s1}/{p1} and summarize it.",
|
||||
f"Open the document {p2} and tell me what it covers.",
|
||||
f'Find the exact string "{token}" in your documents and tell me '
|
||||
"which ones contain it.",
|
||||
f'Which document has the title "{t2}"? Read it and summarize.',
|
||||
f"What do you know about {t1}? Open the relevant document and give "
|
||||
"me specifics.",
|
||||
f"List the files in the {s2} directory.",
|
||||
]
|
||||
|
||||
|
||||
def check_preconditions(
|
||||
settings: Settings, mode: str, dump: Path
|
||||
) -> int | None:
|
||||
"""The locked preconditions — ``2`` on failure (an actionable line is
|
||||
printed), ``None`` when all hold: the DB is reachable and
|
||||
``agent_max_rounds`` > 0 (both modes); ``derived`` — the catalog
|
||||
holds >=2 documents and the first two catalog documents' ``path``s
|
||||
each contain ``/`` (the bare-path traps need nested paths);
|
||||
``fixture`` — the fixture dump exists."""
|
||||
if not db_available():
|
||||
print(
|
||||
"precondition failed: database unreachable — start Postgres "
|
||||
"with `podman compose up -d db` and re-run"
|
||||
)
|
||||
return 2
|
||||
if settings.agent_max_rounds <= 0:
|
||||
print(
|
||||
"precondition failed: BOR_AGENT_MAX_ROUNDS is "
|
||||
f"{settings.agent_max_rounds} (the no-tools kill switch) — set "
|
||||
"it to a positive value for the gate"
|
||||
)
|
||||
return 2
|
||||
if mode == "fixture":
|
||||
if not dump.is_file():
|
||||
print(
|
||||
"precondition failed: fixture dump missing "
|
||||
f"({dump}) — build it once: "
|
||||
"`uv run python -m scripts.load_test_kb`"
|
||||
)
|
||||
return 2
|
||||
return None
|
||||
with SessionLocal() as db:
|
||||
catalog = list_catalog(db)
|
||||
if len(catalog) < 2:
|
||||
print(
|
||||
f"precondition failed: catalog holds {len(catalog)} document(s) "
|
||||
"(need >= 2) — import a knowledge base first: "
|
||||
"`uv run python -m scripts.import_docs`"
|
||||
)
|
||||
return 2
|
||||
bad = [(source, path) for source, path, _ in catalog[:2] if "/" not in path]
|
||||
if bad:
|
||||
print(
|
||||
"precondition failed: the first two catalog documents' paths must "
|
||||
f"each contain '/' (the bare-path traps need nested paths) — got "
|
||||
f"{bad!r}; import a source with a nested directory layout"
|
||||
)
|
||||
return 2
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class TurnResult:
|
||||
"""One battery turn's measurements (from the consumed stream + the
|
||||
holder — no app-code changes for measurement)."""
|
||||
|
||||
index: int
|
||||
question: str
|
||||
max_rounds: int # settings.agent_max_rounds for this run
|
||||
emitted: int = 0 # ToolCallPieces the stream yielded
|
||||
executed: int = 0 # holder.tool_calls (refusals count in nothing)
|
||||
answered: bool = True # the stream finished without LLMError
|
||||
error: str = "" # the terminal error (when not answered)
|
||||
deflected: bool = False # the honesty gate deflected (no tools offered)
|
||||
seconds: float = 0.0 # wall time for the whole turn (embed → settled)
|
||||
calls: list[tuple[str, dict[str, Any]]] = field(
|
||||
default_factory=list
|
||||
) # every emitted (name, arguments) — the contract-accuracy input
|
||||
|
||||
@property
|
||||
def cap_reached(self) -> bool:
|
||||
"""Every capped round emitted a call, so the cap implies at
|
||||
least ``max_rounds`` emissions — and never the reverse."""
|
||||
return self.emitted >= self.max_rounds
|
||||
|
||||
def display(self) -> str:
|
||||
"""The per-turn line: ``turn NN | emitted=E executed=X
|
||||
cap=Y|N defl=Y|N | Ss | <question>`` (the locked core plus the
|
||||
2026-09-04 additions — the deflection flag and the turn's wall
|
||||
seconds — so a slow-down is visible on the same line)."""
|
||||
text = self.question
|
||||
if len(text) > QUESTION_DISPLAY_WIDTH:
|
||||
cut = text[:QUESTION_DISPLAY_WIDTH]
|
||||
text = cut.rsplit(" ", 1)[0].rstrip(" ,;:") + " …"
|
||||
return (
|
||||
f"turn {self.index:02d} | emitted={self.emitted} "
|
||||
f"executed={self.executed} cap={'yes' if self.cap_reached else 'no'} "
|
||||
f"defl={'yes' if self.deflected else 'no'} | {self.seconds:5.2f}s "
|
||||
f"| {text}"
|
||||
)
|
||||
|
||||
|
||||
async def run_turn(
|
||||
llm: LLMClient, settings: Settings, index: int, question: str
|
||||
) -> TurnResult:
|
||||
"""One battery question through the REAL grounded path — the exact
|
||||
mirror of ``app.api.chat`` (same prompt the UI gets): embed the
|
||||
question, retrieve, the honesty gate (``plan_turn``), read the
|
||||
steering notes + KB overview the way chat.py reads them, then —
|
||||
grounded — ``run_agent`` with a **fresh** :class:`AgentHolder`, or —
|
||||
deflected — the ``tools=None`` stream with the one bounded
|
||||
scaffolding recovery. Every yielded piece is consumed to the end;
|
||||
``emitted`` counts the yielded :class:`ToolCallPiece` values,
|
||||
``executed`` is the holder's executed-call count (refusals count in
|
||||
nothing). A turn that dies with ``LLMError`` /
|
||||
``MalformedReplyError`` (the latter subclasses the former) or an
|
||||
``EmbeddingError`` is not ``answered``. ``seconds`` is the turn's
|
||||
wall time (embed → settled).
|
||||
"""
|
||||
result = TurnResult(
|
||||
index=index, question=question, max_rounds=settings.agent_max_rounds
|
||||
)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with SessionLocal() as db:
|
||||
steering_notes = load_steering_notes(db)
|
||||
kb_text = (load_kb_overview(db) or "").strip()
|
||||
question_vec = await llm.embed_one(question)
|
||||
chunks = retrieve(db, question, question_vec)
|
||||
plan = plan_turn(chunks, settings, notes=steering_notes, kb_overview=kb_text)
|
||||
if plan.deflected:
|
||||
result.deflected = True
|
||||
await _run_deflected(llm, settings, plan.system_prompt, question, result)
|
||||
else:
|
||||
holder = AgentHolder()
|
||||
stream = run_agent(
|
||||
llm,
|
||||
db,
|
||||
system_prompt=plan.system_prompt,
|
||||
user_message=question,
|
||||
seed_docs=plan.docs,
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
)
|
||||
async for piece in stream:
|
||||
if isinstance(piece, ToolCallPiece):
|
||||
result.emitted += 1
|
||||
result.calls.append((piece.name, dict(piece.arguments)))
|
||||
result.executed = holder.tool_calls
|
||||
except (LLMError, EmbeddingError) as e:
|
||||
result.answered = False
|
||||
result.error = f"{type(e).__name__}: {e}"
|
||||
result.seconds = time.monotonic() - started
|
||||
return result
|
||||
|
||||
|
||||
async def _run_deflected(
|
||||
llm: LLMClient,
|
||||
settings: Settings,
|
||||
system_prompt: str,
|
||||
question: str,
|
||||
result: TurnResult,
|
||||
) -> None:
|
||||
"""The deflected mirror of ``app.api.chat``: one ``tools=None``
|
||||
request through the retry primitive with a caller-owned
|
||||
:class:`ScaffoldingFilter`, and — when the filter wiped the whole
|
||||
reply — exactly ONE bounded recovery (``tools=None``,
|
||||
:data:`CORRECTION_INSTRUCTION` folded into the single system
|
||||
message, a fresh filter). A second empty reply raises
|
||||
:class:`MalformedReplyError` (the turn is not ``answered``). No tool
|
||||
call can be emitted on this path (``tools=None``)."""
|
||||
messages = [
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": question},
|
||||
]
|
||||
first_filter = ScaffoldingFilter()
|
||||
content_chars = 0
|
||||
stream = chat_stream_retried(
|
||||
llm,
|
||||
messages,
|
||||
tools=None,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=first_filter,
|
||||
)
|
||||
try:
|
||||
async for piece in stream:
|
||||
if isinstance(piece, StreamPiece) and piece.kind == "content":
|
||||
content_chars += len(piece.text)
|
||||
finally:
|
||||
await stream.aclose()
|
||||
if content_chars == 0 and first_filter.stripped_chars > 0:
|
||||
recovery_filter = ScaffoldingFilter()
|
||||
recovered = chat_stream_retried(
|
||||
llm,
|
||||
[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt + "\n" + CORRECTION_INSTRUCTION,
|
||||
},
|
||||
*messages[1:],
|
||||
],
|
||||
tools=None,
|
||||
retries=settings.llm_retries,
|
||||
delay=settings.llm_retry_delay,
|
||||
scaffolding=recovery_filter,
|
||||
)
|
||||
recovery_content = 0
|
||||
try:
|
||||
async for piece in recovered:
|
||||
if isinstance(piece, StreamPiece) and piece.kind == "content":
|
||||
recovery_content += len(piece.text)
|
||||
finally:
|
||||
await recovered.aclose()
|
||||
if recovery_content == 0:
|
||||
# Terminal, as in chat.py: the dedicated error, no done.
|
||||
raise MalformedReplyError(
|
||||
"the deflected model answered in raw tool-scaffolding twice "
|
||||
"in a row — no clean answer"
|
||||
)
|
||||
|
||||
|
||||
def classify_call(
|
||||
name: str, args: dict[str, Any], catalog: set[tuple[str, str]], sources: set[str]
|
||||
) -> bool:
|
||||
"""Contract correctness of ONE emitted call (the tool-calling
|
||||
accuracy metric, 2026-09-04 controlled methodology).
|
||||
|
||||
A call is contract-correct when it uses a known tool with well-formed
|
||||
required arguments that target a RESOLVABLE entity — the phase-72
|
||||
incident class (unknown scopes, bare document paths, hallucinated
|
||||
identities, ``ls(path='/')``-style misuse) is exactly what this
|
||||
flags. An ``ALREADY_IN_CONTEXT`` re-read is NOT flagged: the call is
|
||||
well-formed and names a real document — the app's context dedupe
|
||||
refusing a redundant read is an app-semantics choice, not a
|
||||
tool-calling error (the controlled gate's telemetry — the same
|
||||
re-read 15/15 across copy variants — is documented in
|
||||
``TOOL_CALLING_TESTING.md``). The classification mirrors
|
||||
``app.rag.agent._execute_tool``'s resolution rules gate-side (no
|
||||
app-code changes for measurement).
|
||||
"""
|
||||
if name == "ls":
|
||||
raw = args.get("path")
|
||||
scope = raw.strip() if isinstance(raw, str) else ""
|
||||
return scope == "" or scope in sources
|
||||
if name == "read":
|
||||
raw = args.get("path")
|
||||
arg = raw.strip() if isinstance(raw, str) else ""
|
||||
if "/" not in arg:
|
||||
return False # a bare name can never be a document
|
||||
source, _, path = arg.partition("/")
|
||||
return (source, path) in catalog
|
||||
if name == "grep":
|
||||
raw_pattern = args.get("pattern")
|
||||
if not (isinstance(raw_pattern, str) and raw_pattern.strip()):
|
||||
return False
|
||||
raw_path = args.get("path")
|
||||
scope = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if scope:
|
||||
if "/" not in scope:
|
||||
return False
|
||||
source, _, path = scope.partition("/")
|
||||
return (source, path) in catalog
|
||||
return True
|
||||
return False # unknown tool
|
||||
|
||||
|
||||
def score_contract(
|
||||
turns: list[TurnResult],
|
||||
catalog: set[tuple[str, str]],
|
||||
sources: set[str],
|
||||
) -> int:
|
||||
"""Contract accuracy across the run: how many emitted calls are
|
||||
contract-correct (:func:`classify_call`) against the run's catalog
|
||||
and source names. Per-turn counts are attached on each
|
||||
:class:`TurnResult` as ``_contract_ok`` (measurement state, not a
|
||||
dataclass field — the display line stays the locked format)."""
|
||||
ok = 0
|
||||
for turn in turns:
|
||||
turn_ok = sum(
|
||||
1 for name, args in turn.calls if classify_call(name, args, catalog, sources)
|
||||
)
|
||||
turn._contract_ok = turn_ok # type: ignore[attr-defined]
|
||||
ok += turn_ok
|
||||
return ok
|
||||
|
||||
|
||||
def evaluate(
|
||||
turns: list[TurnResult], partial: bool = False, mode: str = "derived"
|
||||
) -> tuple[bool, list[tuple[str, bool, str]]]:
|
||||
"""The pass conditions → ``(passed, [(name, ok, detail)])``.
|
||||
|
||||
``contract_ok`` per turn was attached by :func:`score_contract`
|
||||
before this call (0 while unattached — main always scores first).
|
||||
|
||||
1. all turns ``answered``; 2. zero ``cap_reached`` turns; 3. >=6 of
|
||||
10 turns with ``emitted >= 1`` (on a ``partial`` run — ``--turns N``
|
||||
with N < the battery length — the count is REPORTED but not gated:
|
||||
a short micro-loop exists for copy iteration, not as the gate);
|
||||
4. the accuracy bar — ``derived`` mode (the phase-72 LOCKED gate):
|
||||
``executed / emitted >= 0.90``; ``fixture`` mode (the 2026-09-04
|
||||
controlled methodology): **contract accuracy** (well-formed calls
|
||||
targeting resolvable entities, :func:`classify_call`) >= 0.90 — with
|
||||
the executed ratio REPORTED alongside (a run with zero emitted calls
|
||||
fails condition 3 anyway, so an empty denominator does not sink
|
||||
condition 4).
|
||||
"""
|
||||
n = len(turns)
|
||||
failed = [t for t in turns if not t.answered]
|
||||
caps = [t for t in turns if t.cap_reached]
|
||||
tool_turns = sum(1 for t in turns if t.emitted >= 1)
|
||||
emitted = sum(t.emitted for t in turns)
|
||||
executed = sum(t.executed for t in turns)
|
||||
deflected = sum(1 for t in turns if t.deflected)
|
||||
contract_ok = sum(
|
||||
getattr(t, "_contract_ok", 0) for t in turns # type: ignore[attr-defined]
|
||||
)
|
||||
failed_detail = f"{n - len(failed)}/{n}"
|
||||
if failed:
|
||||
failed_detail += "; " + "; ".join(
|
||||
f"turn {t.index:02d}: {t.error}" for t in failed
|
||||
)
|
||||
conditions: list[tuple[str, bool, str]] = [
|
||||
("all turns answered", not failed, failed_detail),
|
||||
(
|
||||
"zero cap-reached turns",
|
||||
not caps,
|
||||
"0" if not caps else f"{len(caps)} hit the round cap: "
|
||||
+ ", ".join(f"turn {t.index:02d}" for t in caps),
|
||||
),
|
||||
(
|
||||
">= 6 of 10 turns with >= 1 emitted tool call",
|
||||
True if partial else tool_turns >= 6,
|
||||
f"{tool_turns}/{n}"
|
||||
+ (" (partial run — reported, not gated)" if partial else "")
|
||||
+ (
|
||||
f"; {deflected} deflected (no tools offered — the honesty gate)"
|
||||
if deflected
|
||||
else ""
|
||||
),
|
||||
),
|
||||
]
|
||||
if emitted == 0:
|
||||
conditions.append(
|
||||
(
|
||||
"accuracy bar >= 0.90 across the run",
|
||||
True, # no calls emitted — condition 3 already fails
|
||||
"0/0 (no calls emitted — condition 3 fails)",
|
||||
)
|
||||
)
|
||||
elif mode == "fixture":
|
||||
# The controlled methodology's accuracy bar: contract accuracy.
|
||||
# The executed ratio is reported right below it (not gated — it
|
||||
# includes the app's ALREADY_IN_CONTEXT dedupe refusals, which
|
||||
# the controlled telemetry shows are copy-invariant model
|
||||
# behavior, not tool-calling errors).
|
||||
contract_ratio = contract_ok / emitted
|
||||
conditions.append(
|
||||
(
|
||||
"contract accuracy >= 0.90 (well-formed calls, resolvable targets)",
|
||||
contract_ratio >= 0.90,
|
||||
f"{contract_ok}/{emitted} ({round(100 * contract_ratio)}%)",
|
||||
)
|
||||
)
|
||||
conditions.append(
|
||||
(
|
||||
"executed / emitted (reported — includes in-context dedupe refusals)",
|
||||
True,
|
||||
f"{executed}/{emitted} ({round(100 * executed / emitted)}%)",
|
||||
)
|
||||
)
|
||||
else:
|
||||
ratio = executed / emitted
|
||||
conditions.append(
|
||||
(
|
||||
"executed / emitted >= 0.90 across the run (phase-72 locked)",
|
||||
ratio >= 0.90,
|
||||
f"{executed}/{emitted} ({round(100 * ratio)}%) — "
|
||||
f"contract {contract_ok}/{emitted} ({round(100 * contract_ok / emitted)}%)",
|
||||
)
|
||||
)
|
||||
return all(ok for _name, ok, _detail in conditions), conditions
|
||||
|
||||
|
||||
def verdict_line(
|
||||
model: str,
|
||||
turns: list[TurnResult],
|
||||
passed: bool,
|
||||
wall_seconds: float,
|
||||
partial: bool,
|
||||
mode: str = "derived",
|
||||
) -> str:
|
||||
"""The single stable verdict line (model = the configured chat
|
||||
model, date = run date, the run's total wall time since 2026-09-04,
|
||||
both accuracy metrics since the 2026-09-04 controlled methodology)::
|
||||
|
||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=8 calls
|
||||
7/12 executed (58%) contract 12/12 (100%) 2026-09-04 (wall 48.8s)
|
||||
"""
|
||||
answered = sum(1 for t in turns if t.answered)
|
||||
caps = sum(1 for t in turns if t.cap_reached)
|
||||
tool_turns = sum(1 for t in turns if t.emitted >= 1)
|
||||
emitted = sum(t.emitted for t in turns)
|
||||
executed = sum(t.executed for t in turns)
|
||||
contract_ok = sum(
|
||||
getattr(t, "_contract_ok", 0) for t in turns # type: ignore[attr-defined]
|
||||
)
|
||||
pct = round(100 * executed / emitted) if emitted else 0
|
||||
cpct = round(100 * contract_ok / emitted) if emitted else 0
|
||||
deflected = sum(1 for t in turns if t.deflected)
|
||||
return (
|
||||
f"gate: {model} {'PASS' if passed else 'FAIL'} turns={len(turns)}"
|
||||
+ (" (partial)" if partial else "")
|
||||
+ f" answered={answered} caps={caps} tool-turns={tool_turns}"
|
||||
+ (f" deflected={deflected}" if deflected else "")
|
||||
+ f" calls {executed}/{emitted} executed ({pct}%) "
|
||||
f"contract {contract_ok}/{emitted} ({cpct}%) "
|
||||
f"{date.today().isoformat()} (wall {wall_seconds:.1f}s)"
|
||||
)
|
||||
|
||||
|
||||
async def run_battery(
|
||||
llm: LLMClient,
|
||||
settings: Settings,
|
||||
battery: list[str],
|
||||
concurrency: int = 1,
|
||||
) -> list[TurnResult]:
|
||||
"""The whole battery, printing the per-turn line as each turn
|
||||
settles. ``concurrency > 1`` runs that many turns at once against
|
||||
the endpoint (the aggregate verdict is unchanged — the conditions
|
||||
are run-wide sums; the per-turn lines may then print out of order)."""
|
||||
turns: list[TurnResult] = []
|
||||
if concurrency <= 1:
|
||||
for index, question in enumerate(battery, start=1):
|
||||
turns.append(await run_turn(llm, settings, index, question))
|
||||
print(turns[-1].display())
|
||||
return turns
|
||||
semaphore = asyncio.Semaphore(concurrency)
|
||||
|
||||
async def one(index: int, question: str) -> TurnResult:
|
||||
async with semaphore:
|
||||
return await run_turn(llm, settings, index, question)
|
||||
|
||||
gather = [
|
||||
asyncio.create_task(one(index, question))
|
||||
for index, question in enumerate(battery, start=1)
|
||||
]
|
||||
for finished in asyncio.as_completed(gather):
|
||||
result = await finished
|
||||
turns.append(result)
|
||||
print(result.display())
|
||||
turns.sort(key=lambda t: t.index)
|
||||
return turns
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
# CLI-only: pick up .env without side effects on import (the
|
||||
# house probe pattern, cf. scripts/llm_probe.py).
|
||||
load_dotenv()
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"The real-model tool-calling gate: drive a fixed question "
|
||||
"battery through the real grounded path against the live "
|
||||
"endpoint with the configured chat model, and print the "
|
||||
"PASS/FAIL verdict against the four locked conditions "
|
||||
"(exit 0 PASS, 1 FAIL, 2 precondition failure). The fast "
|
||||
"loop: --restore --mode fixture."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=["derived", "fixture"],
|
||||
default="derived",
|
||||
help=(
|
||||
"derived (default, the phase-72 locked battery from the live "
|
||||
"catalog) or fixture (the curated battery pinned to the "
|
||||
"fixture KB — pair with --restore)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--restore",
|
||||
action="store_true",
|
||||
help="restore the fixture KB dump into the database first (one "
|
||||
"transaction — no git clone, no re-embedding)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--turns",
|
||||
type=int,
|
||||
default=None,
|
||||
metavar="N",
|
||||
help="run only the first N battery questions (the micro-loop for "
|
||||
"copy iteration; the verdict is marked partial and condition 3 "
|
||||
"is reported, not gated)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--concurrency",
|
||||
type=int,
|
||||
default=1,
|
||||
metavar="N",
|
||||
help="run up to N turns at once (default 1 — sequential; the "
|
||||
"aggregate verdict is unchanged)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dump",
|
||||
type=Path,
|
||||
default=None,
|
||||
metavar="PATH",
|
||||
help="the fixture dump for --restore / --mode fixture (default: "
|
||||
f"{DEFAULT_DUMP_PATH})",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
logging.basicConfig(
|
||||
level=logging.INFO, format="%(levelname)s %(name)s: %(message)s"
|
||||
)
|
||||
|
||||
settings = get_settings()
|
||||
dump = args.dump or DEFAULT_DUMP_PATH
|
||||
rc = check_preconditions(settings, args.mode, dump)
|
||||
if rc is not None:
|
||||
return rc
|
||||
|
||||
run_started = time.monotonic()
|
||||
if args.restore:
|
||||
from scripts.restore_test_kb import restore_dump
|
||||
|
||||
print(f"restore: {dump} …")
|
||||
t0 = time.monotonic()
|
||||
restored = restore_dump(dump)
|
||||
print(
|
||||
f"restore: ok in {time.monotonic() - t0:.2f}s "
|
||||
f"({restored.docs} docs, {len(restored.sources)} sources)"
|
||||
)
|
||||
|
||||
if args.mode == "fixture":
|
||||
battery = list(FIXTURE_BATTERY)
|
||||
logger.info(
|
||||
"gate: model=%s mode=fixture battery=%d questions (fixture KB)",
|
||||
settings.llm_chat_model,
|
||||
len(battery),
|
||||
)
|
||||
else:
|
||||
with SessionLocal() as db:
|
||||
catalog = list_catalog(db)
|
||||
s2, p2, _t2 = catalog[1]
|
||||
d2 = find_document(db, s2, p2)
|
||||
d2_content = d2.content if d2 is not None else ""
|
||||
battery = build_battery(catalog, d2_content)
|
||||
logger.info(
|
||||
"gate: model=%s kb_docs=%d battery=%d questions",
|
||||
settings.llm_chat_model,
|
||||
len(catalog),
|
||||
len(battery),
|
||||
)
|
||||
|
||||
if args.turns is not None:
|
||||
if args.turns <= 0:
|
||||
print("error: --turns must be >= 1")
|
||||
return 2
|
||||
battery = battery[: args.turns]
|
||||
|
||||
for number, question in enumerate(battery, start=1):
|
||||
print(f" {number:02d}. {question}")
|
||||
|
||||
llm = LLMClient(settings)
|
||||
turns = asyncio.run(
|
||||
run_battery(llm, settings, battery, concurrency=max(1, args.concurrency))
|
||||
)
|
||||
wall = time.monotonic() - run_started
|
||||
|
||||
# The contract-accuracy classification needs the run's catalog +
|
||||
# source names (the KB is static across the run — the restore, when
|
||||
# any, happened before the battery).
|
||||
with SessionLocal() as db:
|
||||
catalog_set = set((s, p) for s, p, _t in list_catalog(db))
|
||||
sources_set = set(list_source_names(db))
|
||||
score_contract(turns, catalog_set, sources_set)
|
||||
passed, conditions = evaluate(
|
||||
turns, partial=args.turns is not None, mode=args.mode
|
||||
)
|
||||
print(
|
||||
verdict_line(
|
||||
settings.llm_chat_model, turns, passed, wall, args.turns is not None, args.mode
|
||||
)
|
||||
)
|
||||
if not passed:
|
||||
print("conditions (the MISS(es) mark the copy lever to iterate):")
|
||||
for name, ok, detail in conditions:
|
||||
print(f" [{'ok ' if ok else 'MISS'}] {name}: {detail}")
|
||||
return 0 if passed else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,347 @@
|
||||
"""Build the controlled tool-calling test KB and snapshot it (one-off).
|
||||
|
||||
The fixture KB lives in ``tests/fixtures/agent_kb/`` — two source
|
||||
directories (``deployments``, ``homelab``) with eight hand-written
|
||||
markdown documents whose specifics (``rack7``, ``10.77.42.0/24``,
|
||||
VLAN 130, port 18443, ntfy topic ``reese-uptime-7``, machine ID
|
||||
``rbm-8842``, the ``17 2 * * *`` schedule, image
|
||||
``ghcr.io/reese/obsidian-bor:2026.7.14``, port 18765, …) are not
|
||||
guessable by any model. This script:
|
||||
|
||||
1. resets the app tables (one TRUNCATE — the dump's table set),
|
||||
2. registers the two fixture directories as ``kind='local'``
|
||||
``git_sources`` rows (so the source registry — and therefore
|
||||
``ls``'s scope names — is self-contained and independent of the
|
||||
``BOR_GIT_SOURCES`` env var),
|
||||
3. imports the fixture documents through the real pipeline
|
||||
(``import_sources`` — real chunking + real ``embed``-model
|
||||
embeddings; this is the ONLY step that burns model calls, and only
|
||||
at build time),
|
||||
4. stores the static KB overview + the sources-version row,
|
||||
5. prints a **retrieval report** for every fixture-battery question
|
||||
(grounded or deflected, which documents would seed) — the battery
|
||||
must be all-grounded for the gate to exercise the tools,
|
||||
6. snapshots the resulting database state into
|
||||
``tests/fixtures/test_kb.dump.sql`` — a data-only SQL script
|
||||
(TRUNCATE + one multi-row ``INSERT`` per app table, generated
|
||||
in-process — the same file runs in psql or psycopg, in one
|
||||
transaction) — and **verifies the snapshot by restoring it and
|
||||
comparing a per-table checksum**.
|
||||
|
||||
Re-run it only when the fixture documents, the chunker, or the
|
||||
embedding model change — everyday iterations restore the dump in
|
||||
sub-second time (``scripts/restore_test_kb`` / the gate's
|
||||
``--restore``), never re-embedding (see ``TOOL_CALLING_TESTING.md``).
|
||||
|
||||
Exit codes: **0** built + verified, **1** build/verification failure,
|
||||
**2** precondition failure.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.chat import plan_turn
|
||||
from app.config import get_settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import (
|
||||
Chunk,
|
||||
DocDraft,
|
||||
Document,
|
||||
GitSource,
|
||||
KbOverview,
|
||||
QueryLog,
|
||||
SavedChat,
|
||||
SourcesMeta,
|
||||
SteeringNote,
|
||||
)
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.retriever import retrieve
|
||||
|
||||
logger = logging.getLogger("scripts.load_test_kb")
|
||||
|
||||
DEFAULT_KB_DIR = Path("tests/fixtures/agent_kb")
|
||||
DEFAULT_DUMP = Path("tests/fixtures/test_kb.dump.sql")
|
||||
|
||||
#: The two fixture source directories (source name = directory basename,
|
||||
#: the importer's rule). Alphabetical — the catalog order the derived
|
||||
#: battery reads.
|
||||
FIXTURE_SOURCES: tuple[str, ...] = ("deployments", "homelab")
|
||||
|
||||
#: The KB overview stored with the fixture (id=1). A plain outline of
|
||||
#: the KB's basic categories — the ``<knowledge_base>`` prompt section
|
||||
#: of every turn. Static on purpose: the dump must be deterministic and
|
||||
#: the gate must not burn a ``lite`` call at restore time.
|
||||
FIXTURE_KB_OVERVIEW: str = (
|
||||
"deployments: the lab Ansible inventory (host addresses and roles), "
|
||||
"the Obsidian BOR quadlet service definition, and the GitLab Runner "
|
||||
"CI setup. homelab: the rack7 Proxmox cluster networking (bridges, "
|
||||
"VLANs, DNS/DHCP), container notes (Uptime Kuma, Qwen 3.8 on "
|
||||
"llama.cpp), and the nightly restic backup configuration."
|
||||
)
|
||||
|
||||
#: (table, model, explicit column list — the generated ``chunks.tsv``
|
||||
#: tsvector column is excluded; Postgres recomputes it).
|
||||
_TABLES: tuple[tuple[str, type, tuple[str, ...]], ...] = (
|
||||
("documents", Document, ("id", "source", "path", "full_path", "title",
|
||||
"content", "content_hash", "indexed_at", "summary")),
|
||||
("chunks", Chunk, ("id", "document_id", "position", "content",
|
||||
"embedding", "is_summary")),
|
||||
("git_sources", GitSource, ("id", "url", "kind", "path", "added_at")),
|
||||
("kb_overview", KbOverview, ("id", "content", "updated_at")),
|
||||
("sources_meta", SourcesMeta, ("id", "version", "updated_at")),
|
||||
("steering_notes", SteeringNote, ("id", "note", "created_at")),
|
||||
("query_log", QueryLog, ("id", "question", "top_score", "fts_hits",
|
||||
"chunk_hits", "deflected", "sources",
|
||||
"latency_ms", "created_at")),
|
||||
("saved_chats", SavedChat, ("id", "title", "messages", "share_token",
|
||||
"sources_version", "created_at", "updated_at")),
|
||||
("doc_drafts", DocDraft, ("id", "token", "title", "path", "body",
|
||||
"status", "branch", "commit_sha", "created_at",
|
||||
"updated_at")),
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# SQL serialization (the dump is plain multi-row INSERTs — the installed
|
||||
# psycopg build exposes no COPY API, and a multi-statement script with
|
||||
# inline ``COPY … FROM stdin`` data cannot be sent through any driver's
|
||||
# simple-protocol execute. INSERT VALUES is the portable form: the same
|
||||
# file runs in psql, psycopg, or anything else that speaks SQL, in one
|
||||
# transaction. With ``standard_conforming_strings`` on (the Postgres
|
||||
# default since 9.1), a string literal needs ONLY single-quote doubling —
|
||||
# backslashes are literal and newlines may be real.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def _sql_value(value: object) -> str:
|
||||
"""One value as a SQL literal (``NULL`` for None)."""
|
||||
if value is None:
|
||||
return "NULL"
|
||||
if isinstance(value, bool):
|
||||
return "TRUE" if value else "FALSE"
|
||||
if isinstance(value, float):
|
||||
return repr(value) # shortest round-trip double
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
if isinstance(value, uuid.UUID):
|
||||
return "'" + str(value) + "'"
|
||||
if isinstance(value, datetime):
|
||||
return "'" + value.isoformat(sep=" ") + "'"
|
||||
if isinstance(value, (list, tuple)) and value and isinstance(value[0], float):
|
||||
# A pgvector vector: the ``[v1, v2, …]`` text literal (pgvector
|
||||
# 0.7+ format; the older ``{…}`` form is rejected). Checked
|
||||
# before the JSONB branch — a JSONB array of dicts never has a
|
||||
# float first element.
|
||||
return "'" + "[" + ",".join(repr(v) for v in value) + "]" + "'"
|
||||
if isinstance(value, (dict, list)):
|
||||
# JSONB columns: the stored JSON text (Postgres re-parses it).
|
||||
text_ = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
else:
|
||||
text_ = str(value)
|
||||
return "'" + text_.replace("'", "''") + "'"
|
||||
|
||||
|
||||
def _dump_table(db: Session, table: str, model: type, columns: tuple[str, ...]) -> str:
|
||||
"""One multi-row ``INSERT INTO <table> (…) VALUES (…), …;`` statement
|
||||
(an empty table emits nothing — there is no row to write)."""
|
||||
rows = db.execute(select(model)).all()
|
||||
value_rows = [
|
||||
"(" + ", ".join(_sql_value(getattr(row[0], name)) for name in columns) + ")"
|
||||
for row in rows
|
||||
]
|
||||
if not value_rows:
|
||||
return ""
|
||||
return (
|
||||
f"INSERT INTO public.{table} ({', '.join(columns)}) VALUES "
|
||||
+ ",\n".join(value_rows)
|
||||
+ ";\n"
|
||||
)
|
||||
|
||||
|
||||
def _table_checksum(db: Session, table: str) -> str:
|
||||
"""An order-independent content checksum for *table* (row::text,
|
||||
sorted aggregation) — the snapshot round-trip check."""
|
||||
return db.execute(
|
||||
text(
|
||||
"select md5(coalesce(string_agg(r, E'\\n' order by r), '')) "
|
||||
f"from (select t::text as r from public.{table} t) s"
|
||||
)
|
||||
).scalar_one()
|
||||
|
||||
|
||||
async def _retrieval_report(llm: LLMClient, battery: list[str]) -> int:
|
||||
"""Print, per battery question, what the real path would do:
|
||||
grounded or deflected, and which documents would seed the context.
|
||||
Returns the number of deflected questions (a loud warning — the
|
||||
gate needs tools offered on its turns)."""
|
||||
settings = get_settings()
|
||||
deflected = 0
|
||||
print("\nretrieval report (the honesty gate per battery question):")
|
||||
with SessionLocal() as db:
|
||||
for number, question in enumerate(battery, start=1):
|
||||
vec = await llm.embed_one(question)
|
||||
chunks = retrieve(db, question, vec)
|
||||
plan = plan_turn(chunks, settings)
|
||||
seed = ", ".join(f"{d.source}/{d.path}" for d in plan.docs) or "—"
|
||||
if plan.deflected:
|
||||
deflected += 1
|
||||
print(
|
||||
f" {number:02d}. {'DEFLECTED ' if plan.deflected else 'grounded '} "
|
||||
f"(best={plan.top_score:.3f} fts={plan.fts_hits}) "
|
||||
f"seed: {seed}\n ← {question}"
|
||||
)
|
||||
return deflected
|
||||
|
||||
|
||||
async def _build(kb_dir: Path, dump_path: Path) -> int:
|
||||
from scripts.agent_realmodel_check import FIXTURE_BATTERY
|
||||
|
||||
started = time.monotonic()
|
||||
if not db_available():
|
||||
print(
|
||||
"load_test_kb: precondition failed — database unreachable; "
|
||||
"start Postgres with `podman compose up -d db`"
|
||||
)
|
||||
return 2
|
||||
|
||||
source_dirs = [kb_dir / name for name in FIXTURE_SOURCES]
|
||||
missing = [str(p) for p in source_dirs if not p.is_dir()]
|
||||
if missing:
|
||||
print(f"load_test_kb: precondition failed — missing source dir(s): {missing}")
|
||||
return 2
|
||||
|
||||
# 1. Reset the dump's table set (one statement — the inter-table FKs
|
||||
# resolve within it).
|
||||
table_list = ", ".join(f"public.{table}" for table, _m, _c in _TABLES)
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(f"TRUNCATE {table_list}"))
|
||||
for directory in source_dirs:
|
||||
absolute = str(directory.resolve())
|
||||
db.add(GitSource(url=absolute, kind="local", path=absolute))
|
||||
db.commit()
|
||||
logger.info("load_test_kb: tables reset; %d local source rows added", len(source_dirs))
|
||||
|
||||
# 2. Import through the real pipeline (the only model-cost step).
|
||||
llm = LLMClient()
|
||||
summary = await import_sources([p.resolve() for p in source_dirs], llm)
|
||||
if summary.errors:
|
||||
print(f"load_test_kb: {summary.errors} file(s) failed to import — aborting")
|
||||
return 1
|
||||
if summary.added == 0:
|
||||
print("load_test_kb: no documents imported — aborting")
|
||||
return 1
|
||||
logger.info(
|
||||
"load_test_kb: imported added=%d chunks=%d embed_batches=%d",
|
||||
summary.added, summary.chunks, summary.embed_batches,
|
||||
)
|
||||
|
||||
# 3. The static overview + sources version.
|
||||
with SessionLocal() as db:
|
||||
overview = db.get(KbOverview, 1) or KbOverview(id=1)
|
||||
overview.content = FIXTURE_KB_OVERVIEW
|
||||
meta = db.get(SourcesMeta, 1) or SourcesMeta(id=1)
|
||||
meta.version = 1
|
||||
db.add(overview)
|
||||
db.add(meta)
|
||||
db.commit()
|
||||
|
||||
# 4. Retrieval report (all battery questions must stay grounded).
|
||||
n_deflected = await _retrieval_report(llm, list(FIXTURE_BATTERY))
|
||||
if n_deflected:
|
||||
print(
|
||||
f"\nload_test_kb: WARNING — {n_deflected} battery question(s) would "
|
||||
"DEFLECT in the real path (no tools offered). Adjust the fixture "
|
||||
"content (a lexical anchor for the question's words) or the "
|
||||
"question before running the gate."
|
||||
)
|
||||
|
||||
# 5. Snapshot (data-only) + round-trip verification.
|
||||
with SessionLocal() as db:
|
||||
before = {table: _table_checksum(db, table) for table, _m, _c in _TABLES}
|
||||
parts = [
|
||||
"-- ============================================================",
|
||||
"-- Brain of Reese — controlled tool-calling test KB (fixture dump)",
|
||||
f"-- Generated by scripts/load_test_kb.py on "
|
||||
f"{datetime.now().astimezone().isoformat(timespec='seconds')}",
|
||||
"-- Data-only snapshot (the schema stays alembic-managed; the",
|
||||
"-- generated chunks.tsv column is recomputed on restore).",
|
||||
f"-- Sources: {', '.join(FIXTURE_SOURCES)} "
|
||||
f"({summary.added} documents, {summary.chunks} chunks).",
|
||||
"-- Restore (one transaction, sub-second):",
|
||||
"-- uv run python -m scripts.restore_test_kb",
|
||||
"-- psql \"$BOR_DATABASE_URL\" --single-transaction -f "
|
||||
"tests/fixtures/test_kb.dump.sql",
|
||||
"-- ============================================================",
|
||||
f"TRUNCATE {table_list};",
|
||||
"",
|
||||
]
|
||||
for table, model, columns in _TABLES:
|
||||
parts.append(_dump_table(db, table, model, columns))
|
||||
script = "\n".join(parts)
|
||||
dump_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
dump_path.write_text(script, encoding="utf-8")
|
||||
logger.info("load_test_kb: dump written: %s (%d KB)", dump_path, len(script) // 1024)
|
||||
|
||||
# Round-trip: restore the dump over the (identical) state and compare
|
||||
# the per-table checksums — a serialization bug must fail the build.
|
||||
from scripts.restore_test_kb import restore_dump
|
||||
|
||||
restore_dump(dump_path) # RuntimeError on failure → the build fails
|
||||
with SessionLocal() as db:
|
||||
after = {table: _table_checksum(db, table) for table, _m, _c in _TABLES}
|
||||
mismatched = [t for t, c in before.items() if after.get(t) != c]
|
||||
if mismatched:
|
||||
print(f"load_test_kb: VERIFICATION FAILED — checksum mismatch: {mismatched}")
|
||||
return 1
|
||||
|
||||
wall = time.monotonic() - started
|
||||
print(
|
||||
f"load_test_kb: ok — docs={summary.added} chunks={summary.chunks} "
|
||||
f"sources={len(FIXTURE_SOURCES)} dump={dump_path} "
|
||||
f"({dump_path.stat().st_size // 1024} KB, verified by round-trip) "
|
||||
f"in {wall:.1f}s"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
# CLI-only: pick up .env without side effects on import (the house
|
||||
# probe pattern, cf. scripts/llm_probe.py).
|
||||
load_dotenv()
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Build the controlled tool-calling test KB from "
|
||||
"tests/fixtures/agent_kb (real embeddings, once) and snapshot "
|
||||
"it to tests/fixtures/test_kb.dump.sql (verified by "
|
||||
"round-trip). Exit 0 built+verified, 1 failure, 2 precondition."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--kb-dir", type=Path, default=DEFAULT_KB_DIR,
|
||||
help=f"the fixture KB root (default: {DEFAULT_KB_DIR})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dump", type=Path, default=DEFAULT_DUMP,
|
||||
help=f"the dump file to write (default: {DEFAULT_DUMP})",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
logging.basicConfig(
|
||||
level=logging.INFO, format="%(levelname)s %(name)s: %(message)s"
|
||||
)
|
||||
return asyncio.run(_build(args.kb_dir, args.dump))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,178 @@
|
||||
"""One-shot restore of the controlled tool-calling test KB (the fast loop).
|
||||
|
||||
The fixture KB (``tests/fixtures/agent_kb/`` — two sources, eight
|
||||
hand-written markdown documents) is built once by
|
||||
:mod:`scripts.load_test_kb`, which embeds the documents and snapshots the
|
||||
resulting database state into ``tests/fixtures/test_kb.dump.sql`` — a
|
||||
data-only SQL script (``TRUNCATE`` + one multi-row ``INSERT`` per app
|
||||
table, generated in-process — the same file runs in psql or psycopg). This
|
||||
script restores that
|
||||
snapshot in **one transaction** through the app's own database URL
|
||||
(``BOR_DATABASE_URL``): no git clone of the homelab repo, no re-embedding,
|
||||
no ``lite``-model calls — the whole known state (documents, chunks +
|
||||
embeddings, the source registry rows, the KB overview, the sources
|
||||
version) lands in a fraction of a second, which is what makes a
|
||||
tool-calling iteration loop fast (see ``TOOL_CALLING_TESTING.md``):
|
||||
|
||||
uv run python -m scripts.restore_test_kb
|
||||
# restore_test_kb: ok in 0.41s (8 docs, 2 sources, 16 chunks)
|
||||
|
||||
The gate runs the same restore inline:
|
||||
``uv run python -m scripts.agent_realmodel_check --restore``.
|
||||
|
||||
The dump is data-only on purpose: the schema stays owned by alembic, and
|
||||
the generated ``chunks.tsv`` tsvector column (``GENERATED ALWAYS AS …
|
||||
STORED``) is recomputed by Postgres, so the restore is safe against schema
|
||||
drift limited to additive columns. Restoring into a database whose schema
|
||||
lacks an app table fails loudly with an actionable line (exit 2).
|
||||
|
||||
Exit codes: **0** restored, **2** precondition failure (DB unreachable,
|
||||
dump missing, schema not applied).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import psycopg
|
||||
from dotenv import load_dotenv
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import SessionLocal, db_available
|
||||
|
||||
#: The app tables the dump covers, TRUNCATE order (one statement —
|
||||
#: Postgres resolves the inter-table FKs within it). ``chunks`` and
|
||||
#: ``documents`` are listed first for readability; the order is
|
||||
#: irrelevant inside a single TRUNCATE.
|
||||
APP_TABLES: tuple[str, ...] = (
|
||||
"chunks",
|
||||
"documents",
|
||||
"git_sources",
|
||||
"kb_overview",
|
||||
"sources_meta",
|
||||
"steering_notes",
|
||||
"query_log",
|
||||
"saved_chats",
|
||||
"doc_drafts",
|
||||
)
|
||||
|
||||
#: Repo-relative default dump location (the load script writes it there).
|
||||
DEFAULT_DUMP = Path("tests/fixtures/test_kb.dump.sql")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RestoreResult:
|
||||
"""What :func:`restore_dump` did — one line of the summary output."""
|
||||
|
||||
seconds: float
|
||||
docs: int
|
||||
sources: tuple[str, ...]
|
||||
chunks: int
|
||||
dump_bytes: int
|
||||
|
||||
|
||||
def _connect():
|
||||
"""A raw psycopg connection on the app's DB URL (psycopg3 speaks the
|
||||
SQLAlchemy URL's driver scheme — ``postgresql+psycopg`` maps to
|
||||
``postgresql`` for psycopg)."""
|
||||
from app.config import get_settings
|
||||
|
||||
url = get_settings().database_url
|
||||
if url.startswith("postgresql+psycopg://"):
|
||||
url = "postgresql://" + url.split("://", 1)[1]
|
||||
return psycopg.connect(url)
|
||||
|
||||
|
||||
def restore_dump(dump: Path) -> RestoreResult:
|
||||
"""Restore *dump* (the data-only SQL script) into the app database.
|
||||
|
||||
One transaction (TRUNCATE + INSERTs + nothing else — a failed restore
|
||||
rolls back and leaves the previous KB intact). Returns the measured
|
||||
result; raises :class:`RuntimeError` with an actionable line on
|
||||
failure (missing table = schema not applied).
|
||||
"""
|
||||
if not dump.is_file():
|
||||
raise RuntimeError(
|
||||
f"dump not found: {dump} — build it first: "
|
||||
"`uv run python -m scripts.load_test_kb`"
|
||||
)
|
||||
script = dump.read_text(encoding="utf-8")
|
||||
started = time.monotonic()
|
||||
conn = _connect()
|
||||
try:
|
||||
with conn.transaction():
|
||||
# A plain multi-statement SQL script (TRUNCATE + INSERTs — no
|
||||
# parameters) runs on psycopg's simple-protocol execute; the
|
||||
# installed stubs type the query parameter as Template-only
|
||||
# (and ``sql.SQL`` wants a LiteralString), hence the ignore.
|
||||
conn.execute(script) # pyright: ignore[reportArgumentType, reportCallIssue]
|
||||
except Exception as e:
|
||||
message = str(e)
|
||||
if "relation" in message and "does not exist" in message:
|
||||
raise RuntimeError(
|
||||
"schema not applied — the dump needs the alembic-managed "
|
||||
f"tables; run `uv run alembic upgrade head` first ({e})"
|
||||
) from None
|
||||
raise RuntimeError(f"restore failed: {e}") from None
|
||||
seconds = time.monotonic() - started
|
||||
with SessionLocal() as db:
|
||||
docs = db.execute(text("select count(*) from documents")).scalar_one()
|
||||
sources = tuple(
|
||||
row[0]
|
||||
for row in db.execute(
|
||||
text("select distinct source from documents order by source")
|
||||
)
|
||||
)
|
||||
chunks = db.execute(text("select count(*) from chunks")).scalar_one()
|
||||
return RestoreResult(
|
||||
seconds=seconds,
|
||||
docs=docs,
|
||||
sources=sources,
|
||||
chunks=chunks,
|
||||
dump_bytes=dump.stat().st_size,
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
# CLI-only: pick up .env without side effects on import (the house
|
||||
# probe pattern, cf. scripts/llm_probe.py).
|
||||
load_dotenv()
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Restore the controlled tool-calling test KB from the fixture "
|
||||
"dump (one transaction, no git clone, no re-embedding). Exit "
|
||||
"0 on success, 2 on precondition failure."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dump",
|
||||
type=Path,
|
||||
default=DEFAULT_DUMP,
|
||||
help=f"the data-only SQL dump to restore (default: {DEFAULT_DUMP})",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if not db_available():
|
||||
print(
|
||||
"restore_test_kb: precondition failed — database unreachable; "
|
||||
"start Postgres with `podman compose up -d db`"
|
||||
)
|
||||
return 2
|
||||
try:
|
||||
result = restore_dump(args.dump)
|
||||
except RuntimeError as e:
|
||||
print(f"restore_test_kb: {e}")
|
||||
return 2
|
||||
print(
|
||||
f"restore_test_kb: ok in {result.seconds:.2f}s "
|
||||
f"({result.docs} docs, {len(result.sources)} sources, "
|
||||
f"{result.chunks} chunks, dump {result.dump_bytes // 1024} KB)"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+336
-52
@@ -56,25 +56,30 @@ Implements just enough of the aipi surface:
|
||||
the echo targets the block itself; its tail still includes the
|
||||
closing tag — same sentinel semantics.)
|
||||
- user message containing ``use your tools`` (phase 37, agent document
|
||||
tools) **and** the system prompt carries the ``<tools>`` section ->
|
||||
the deterministic SINGLE-READ tool flow, discriminated statelessly
|
||||
from the messages (the ``tools`` parameter gates the list/read
|
||||
steps — a no-tools request with no tool results is not the flow):
|
||||
tools; phase 70: the flow emits the harness-aligned names — ``ls``
|
||||
/ ``read`` with the combined ``source/path`` identity) **and** the
|
||||
system prompt carries the ``<tools>`` section -> the deterministic
|
||||
SINGLE-READ tool flow, discriminated statelessly from the messages
|
||||
(the ``tools`` parameter gates the list/read steps — a no-tools
|
||||
request with no tool results is not the flow):
|
||||
* request 1 (``tools`` offered, no tool results yet): stream ONLY
|
||||
``tool_calls`` deltas — ``list_documents`` (synthetic id
|
||||
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
|
||||
content;
|
||||
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
|
||||
arguments), ``finish_reason: "tool_calls"``, no content;
|
||||
* request 2 (a ``tool``-role catalog result in the messages):
|
||||
parse the FIRST catalog line (``source: X | path: Y | title: Z``
|
||||
— the labeled ``source:`` / ``path:`` fields, phase 63) and
|
||||
stream a ``tool_calls`` delta calling ``read_document`` on it
|
||||
(id ``call_1``);
|
||||
* request 3 (a ``tool``-role read result in the messages): a
|
||||
content answer, deterministic: ``Read <source/path>. <first 80
|
||||
chars of the read document's content>`` — so a suite can assert
|
||||
the read document reached the model and landed in the answer.
|
||||
Reached regardless of the ``tools`` parameter (phase 45 keeps
|
||||
the tools offered until the round cap).
|
||||
stream a ``tool_calls`` delta calling ``read`` on the JOINED
|
||||
combined ``source/path`` (the mock joins the two labeled fields
|
||||
— the catalog format is unchanged, so this join is the only
|
||||
parse change, phase 70) (id ``call_1``);
|
||||
* request 3 (a ``tool``-role read result in the messages —
|
||||
content starting with the agent's ``"Document <source/path>:"``
|
||||
header): a content answer, deterministic: ``Read
|
||||
<source/path>. <first 80 chars of the read document's
|
||||
content>`` — so a suite can assert the read document reached
|
||||
the model and landed in the answer. Reached regardless of the
|
||||
``tools`` parameter (phase 45 keeps the tools offered until the
|
||||
round cap).
|
||||
The single-read flow stops at ONE read result; the MULTI-READ
|
||||
variant below reads two.
|
||||
- user message containing BOTH ``use your tools`` AND ``read two
|
||||
@@ -82,15 +87,18 @@ Implements just enough of the aipi surface:
|
||||
system prompt carries the ``<tools>`` section -> the deterministic
|
||||
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
|
||||
the COUNT of ``tool``-role read results (content starting with the
|
||||
agent's ``"Document <source/path>:"`` prefix):
|
||||
* 0 read results, no catalog yet: ``list_documents`` (id
|
||||
``call_0``);
|
||||
* 0 read results, catalog present: ``read_document`` on the FIRST
|
||||
catalog line (id ``call_1``);
|
||||
* 1 read result: ``read_document`` on the SECOND catalog line —
|
||||
the first listing line whose ``source/path`` differs from the
|
||||
one already read (id ``call_2``); a one-document catalog
|
||||
degenerates to the single-read answer (nothing second to read);
|
||||
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
|
||||
flow on the harness-aligned names — ``ls``, then ``read`` on the
|
||||
JOINED combined ``source/path`` of each catalog line:
|
||||
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
|
||||
* 0 read results, catalog present: ``read`` on the JOINED
|
||||
combined ``source/path`` of the FIRST catalog line
|
||||
(id ``call_1``);
|
||||
* 1 read result: ``read`` on the JOINED combined ``source/path``
|
||||
of the SECOND catalog line — the first listing line whose
|
||||
``source/path`` differs from the one already read (id
|
||||
``call_2``); a one-document catalog degenerates to the
|
||||
single-read answer (nothing second to read);
|
||||
* 2 read results: the forced answer, byte-stable: the single-read
|
||||
shape quoting the FIRST read result, plus the line ``I read
|
||||
<sp1> and <sp2>.`` naming both read paths in read order — so a
|
||||
@@ -101,12 +109,13 @@ Implements just enough of the aipi surface:
|
||||
``E2E_REAL_LLM=1`` ignores the mock entirely (the real model does
|
||||
what it does).
|
||||
- user message containing ``search your documents``
|
||||
(``SEARCH_TRIGGER``, phase 68, search tool) **and** the system
|
||||
prompt carries the ``<tools>`` section -> the deterministic SEARCH
|
||||
tool flow, discriminated statelessly from the messages (streaming
|
||||
only):
|
||||
(``SEARCH_TRIGGER``, phase 68 search tool — renamed to the
|
||||
harness-aligned ``grep`` in phase 70, same match/output contract)
|
||||
**and** the system prompt carries the ``<tools>`` section -> the
|
||||
deterministic SEARCH tool flow, discriminated statelessly from the
|
||||
messages (streaming only):
|
||||
* request 1 (``tools`` offered, no search result yet): stream
|
||||
ONLY ``tool_calls`` deltas — ``search_documents`` with
|
||||
ONLY ``tool_calls`` deltas — ``grep`` with
|
||||
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
|
||||
* request 2 (a ``tool``-role search result in the messages —
|
||||
recognizable by its ``source/path:line: text`` match lines or
|
||||
@@ -118,6 +127,62 @@ Implements just enough of the aipi surface:
|
||||
specific phrase — same convention as ``think in paragraphs``); no
|
||||
existing E2E question or fixture file contains the trigger, so
|
||||
every other suite is unaffected.
|
||||
- user message containing ``emit raw tool markup``
|
||||
(``SCAFFOLD_TRIGGER``, phase 71, tool-scaffolding guardrails — the
|
||||
2026-09-03 incident where a deflected round streamed the model's
|
||||
raw ``<|tool_call_start|>…<|tool_call_end|>`` markup into the UI)
|
||||
**or** ``always emit raw tool markup``
|
||||
(``SCAFFOLD_ALWAYS_TRIGGER``, checked FIRST — it contains the
|
||||
former phrase) -> the deterministic SCAFFOLDING flow, independent
|
||||
of the ``<tools>`` marker (both grounded and deflected turns hit
|
||||
it):
|
||||
* ``SCAFFOLD_ALWAYS_TRIGGER``: EVERY request (the one bounded
|
||||
recovery included) streams ONLY ``delta.content`` chunks
|
||||
carrying the incident span ``SCAFFOLD_SPAN`` —
|
||||
``<|tool_call_start|>[read(path='search_docs/reese-notes.md')]
|
||||
<|tool_call_end|>`` — split across the mock's 12-char chunks
|
||||
(the filter's boundary path), ``finish_reason: "stop"``, no
|
||||
structured ``tool_calls``, no reasoning — the terminal
|
||||
malformed-reply path.
|
||||
* ``SCAFFOLD_TRIGGER``: request 1 (no ``CORRECTION_INSTRUCTION``
|
||||
in the system prompt) streams the same scaffolding-only span;
|
||||
request 2 (the system prompt carries the harness constant — a
|
||||
stable substring of ``app.rag.agent.CORRECTION_INSTRUCTION``,
|
||||
IMPORTED into this module so the mock can never drift from
|
||||
it: the one bounded recovery, ``tools=None`` with the
|
||||
correction folded into the single system prompt) streams the
|
||||
clean ``SCAFFOLD_RECOVERY_ANSWER`` — the recovery path.
|
||||
Checked BEFORE the ``SEARCH_TRIGGER`` / ``TOOLS_TRIGGER`` flows
|
||||
(the trigger needs no ``<tools>`` section); no existing E2E
|
||||
question or fixture file contains the phrase, so every other
|
||||
suite is unaffected.
|
||||
- user message containing ``list the files in this directory``
|
||||
(``LS_TEACH_TRIGGER``, phase 72, teaching refusals — the
|
||||
2026-09-03 incident where the harness-prior ``ls(path='.')``
|
||||
misuse met the terse refusal and the model re-reasoned the same
|
||||
paragraphs over and over) **and** the system prompt carries the
|
||||
``<tools>`` section -> the deterministic LS-TEACHING flow,
|
||||
discriminated statelessly from the messages (streaming only):
|
||||
* request 1 (``tools`` offered, no ``tool``-role result in the
|
||||
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
|
||||
with ``{"path": "."}`` (synthetic id ``call_0``),
|
||||
``finish_reason: "tool_calls"``, no content — the incident's
|
||||
misuse, deterministic;
|
||||
* request 2 (a ``tool``-role result present that is NOT a
|
||||
catalog listing — i.e. the teaching refusal): a ``tool_calls``
|
||||
delta — ``ls`` with no arguments (id ``call_1``) — the
|
||||
correction;
|
||||
* request 3 (a ``tool``-role result whose first line matches the
|
||||
``^\\d+ documents:`` catalog header): a deterministic content
|
||||
answer — ``These are the indexed documents: <first catalog
|
||||
line>`` (the ``source: X | path: Y | title: Z`` line, parsed
|
||||
with the ``_CATALOG_LINE_RE`` machinery), ``finish_reason:
|
||||
"stop"`` — the loop ended in ONE correction, not at the round
|
||||
cap.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
|
||||
phrases are disjoint substrings — the phase-71 ordering
|
||||
convention); no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``show me a table`` (phase 44, markdown
|
||||
tables, TODO.md L6) -> the fixed table answer (``TABLE_ANSWER``):
|
||||
a 3-column service table, an ``<img onerror>`` XSS probe line, and
|
||||
@@ -179,6 +244,8 @@ from typing import Any
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import JSONResponse, StreamingResponse
|
||||
|
||||
from app.rag.agent import CORRECTION_INSTRUCTION # phase 71: the harness constant
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
DIM = 768
|
||||
@@ -265,11 +332,12 @@ END_OF_NOTES_TRIGGER = "show the end of your notes"
|
||||
#: phase-24 tail echo targets the block, not the raw message tail).
|
||||
_DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
|
||||
|
||||
#: Phase 37 (agent-document-tools story): a user message containing this
|
||||
#: substring (case-insensitive) — combined with the ``<tools>`` section
|
||||
#: in the system prompt — drives the deterministic tool flow documented
|
||||
#: in the module docstring (list_documents → read_document on the first
|
||||
#: catalog line → the quoted answer). Existing E2E questions do not
|
||||
#: Phase 37 (agent-document-tools story; phase 70: the flow emits the
|
||||
#: harness-aligned names): a user message containing this substring
|
||||
#: (case-insensitive) — combined with the ``<tools>`` section in the
|
||||
#: system prompt — drives the deterministic tool flow documented in the
|
||||
#: module docstring (ls → read on the first catalog line's combined
|
||||
#: ``source/path`` → the quoted answer). Existing E2E questions do not
|
||||
#: contain the phrase, so every other suite is unaffected.
|
||||
TOOLS_TRIGGER = "use your tools"
|
||||
|
||||
@@ -282,11 +350,12 @@ TOOLS_TRIGGER = "use your tools"
|
||||
#: so the 3-step flow is untouched.
|
||||
MULTI_READ_TRIGGER = "read two documents"
|
||||
|
||||
#: Phase 68 (search tool, TODO.md L4): a user message containing this
|
||||
#: substring (case-insensitive) — combined with the ``<tools>`` section
|
||||
#: in the system prompt — drives the deterministic SEARCH tool flow
|
||||
#: (search_documents for ``SEARCH_PATTERN`` → the "Found …" answer),
|
||||
#: documented in the module docstring. Checked BEFORE ``TOOLS_TRIGGER``
|
||||
#: Phase 68 (search tool, TODO.md L4; phase 70: renamed to the
|
||||
#: harness-aligned ``grep``): a user message containing this substring
|
||||
#: (case-insensitive) — combined with the ``<tools>`` section in the
|
||||
#: system prompt — drives the deterministic SEARCH tool flow (grep for
|
||||
#: ``SEARCH_PATTERN`` → the "Found …" answer), documented in the module
|
||||
#: docstring. Checked BEFORE ``TOOLS_TRIGGER``
|
||||
#: (the more specific phrase wins — the same convention as
|
||||
#: ``THINK_PARAS_TRIGGER``); verified 2026-09-01: no existing E2E
|
||||
#: question or fixture file contains the phrase, so every other suite
|
||||
@@ -353,6 +422,71 @@ ALWAYS_FAIL_TRIGGER = "always fail"
|
||||
#: bag-of-words vector — the endpoint's pre-stream embedding retry loop.
|
||||
EMBED_FAIL_TRIGGER = "embed fail once"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 71 (tool-scaffolding guardrails, 2026-09-03 incident):
|
||||
# deterministic raw-markup flows — see the module docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) drives
|
||||
#: the scaffolding flow: request 1 streams ONLY the incident's raw tool
|
||||
#: markup as ``delta.content``; the follow-up request carrying the
|
||||
#: harness correction in the system prompt (the one bounded recovery)
|
||||
#: streams the clean answer. Independent of the ``<tools>`` marker —
|
||||
#: both grounded and deflected turns hit it. Existing E2E questions do
|
||||
#: not contain the phrase, so every other suite is unaffected.
|
||||
SCAFFOLD_TRIGGER = "emit raw tool markup"
|
||||
|
||||
#: A user message containing this substring (checked BEFORE
|
||||
#: ``SCAFFOLD_TRIGGER`` — it contains that phrase) streams the
|
||||
#: scaffolding-only span on EVERY request, recovery included — the
|
||||
#: terminal malformed-reply path (the dedicated error frame, no done).
|
||||
SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup"
|
||||
|
||||
#: The incident span (2026-09-03): the model's chat-template tool
|
||||
#: syntax, emitted as plain ``delta.content`` although no tools were
|
||||
#: offered. Streamed through the mock's 12-char chunking, so it always
|
||||
#: spans ≥2 wire chunks (the filter's boundary path).
|
||||
SCAFFOLD_SPAN = (
|
||||
"<|tool_call_start|>[read(path='search_docs/reese-notes.md')]"
|
||||
"<|tool_call_end|>"
|
||||
)
|
||||
|
||||
#: The clean answer the one bounded recovery produces (byte-stable —
|
||||
#: the dedicated E2E suite asserts the recovered bubble and the wire's
|
||||
#: delta text against it).
|
||||
SCAFFOLD_RECOVERY_ANSWER = "Here is the plain-text answer the recovery produced."
|
||||
|
||||
#: The stable substring of the harness-owned correction constant the
|
||||
#: recovery request carries in its system prompt. Keyed on a substring
|
||||
#: (not the whole constant) so a re-wrap of the constant cannot silently
|
||||
#: re-route the mock; the module-level assert below fails loudly if the
|
||||
#: substring ever leaves the constant (the mock must never drift from
|
||||
#: ``app.rag.agent.CORRECTION_INSTRUCTION``).
|
||||
_CORRECTION_MARKER = "no tool syntax"
|
||||
assert _CORRECTION_MARKER in CORRECTION_INSTRUCTION, (
|
||||
"mock drift: the correction marker left CORRECTION_INSTRUCTION"
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 72 (teaching refusals — the 2026-09-03 incident's ls misuse):
|
||||
# the deterministic LS-TEACH self-correction flow — see the module
|
||||
# docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||
#: the deterministic LS-TEACHING flow (the incident's
|
||||
#: ``ls(path='.')`` misuse → the teaching refusal → the corrected
|
||||
#: no-arg ``ls()`` → the catalog answer). Checked BEFORE the plain
|
||||
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the phase-71
|
||||
#: ordering convention); verified: no existing E2E question or fixture
|
||||
#: file contains the phrase, so every other suite is unaffected.
|
||||
LS_TEACH_TRIGGER = "list the files in this directory"
|
||||
|
||||
#: The agent's ``ls`` listing header (app.rag.agent ``_execute_tool``):
|
||||
#: ``"N documents:"`` — the first line of every catalog tool result.
|
||||
_CATALOG_HEADER_RE = re.compile(r"^\d+ documents:")
|
||||
|
||||
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
|
||||
#: while the endpoint stays down: the openai SDK's default policy
|
||||
#: (max_retries=2 — the app's ``LLMClient`` keeps it) re-POSTs a 500'd
|
||||
@@ -402,11 +536,11 @@ def _chat_dead(key: str, dead_attempts: int) -> bool:
|
||||
return _bump_fail(key) <= dead_attempts * _HTTPS_PER_DEAD_ATTEMPT
|
||||
|
||||
|
||||
#: The agent's ``read_document`` tool-result prefix (app.rag.agent
|
||||
#: The agent's ``read`` tool-result prefix (app.rag.agent
|
||||
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
|
||||
_READ_RESULT_PREFIX = "Document "
|
||||
|
||||
#: One line of the agent's ``list_documents`` output (app.rag.agent
|
||||
#: One line of the agent's ``ls`` output (app.rag.agent
|
||||
#: ``_execute_tool``, phase 63): labeled, pipe-delimited fields —
|
||||
#: ``source: X | path: Y | title: Z`` — unambiguous for LLM parsing even
|
||||
#: when the path contains ``/`` characters.
|
||||
@@ -439,7 +573,7 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
"""Every ``(source, path)`` in the catalog tool result, in listing order.
|
||||
|
||||
Catalog lines are ``source: X | path: Y | title: Z`` (the agent's
|
||||
``list_documents`` output — phase 63: labeled, pipe-delimited
|
||||
``ls`` output — phase 63: labeled, pipe-delimited
|
||||
fields, unambiguous even for paths full of ``/``): the line-level
|
||||
regex recovers the ``source`` and ``path`` fields directly. The
|
||||
``"N documents:"`` header line matches no line and is skipped;
|
||||
@@ -460,8 +594,46 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
return docs
|
||||
|
||||
|
||||
#: One line of the agent's ``search_documents`` output (app.rag.agent
|
||||
#: ``_execute_tool``, phase 68): ``source/path:LINE: text``. The
|
||||
def _tool_results(body: dict[str, Any]) -> list[str]:
|
||||
"""Every ``tool``-role result content in the messages, in order.
|
||||
|
||||
(Phase 72, LS-TEACH flow: the flow is discriminated statelessly
|
||||
from the tool results — a catalog listing vs the teaching
|
||||
refusal vs none yet.)
|
||||
"""
|
||||
return [
|
||||
str(m.get("content") or "")
|
||||
for m in _messages(body)
|
||||
if m.get("role") == "tool"
|
||||
]
|
||||
|
||||
|
||||
def _first_catalog_line(body: dict[str, Any]) -> str | None:
|
||||
"""The first catalog line of a catalog listing in the messages.
|
||||
|
||||
A catalog listing is a ``tool``-role result whose FIRST line is the
|
||||
agent's ``"N documents:"`` header (``_CATALOG_HEADER_RE``); its
|
||||
first ``source: X | path: Y | title: Z`` line (the
|
||||
``_CATALOG_LINE_RE`` machinery) is returned. ``None`` when no
|
||||
catalog listing is in the messages — e.g. while only the teaching
|
||||
refusal is there (the phase-72 LS-TEACH flow's request-2 state).
|
||||
An empty listing (``"0 documents:"`` with no lines) returns
|
||||
``""`` — the listing is present, it is just empty.
|
||||
"""
|
||||
for content in _tool_results(body):
|
||||
lines = content.splitlines()
|
||||
if not lines or not _CATALOG_HEADER_RE.match(lines[0]):
|
||||
continue
|
||||
for line in lines[1:]:
|
||||
if _CATALOG_LINE_RE.match(line):
|
||||
return line
|
||||
return ""
|
||||
return None
|
||||
|
||||
|
||||
#: One line of the agent's ``grep`` output (app.rag.agent
|
||||
#: ``_execute_tool``, phase 68 — phase 70 renamed the tool, the line
|
||||
#: format is unchanged): ``source/path:LINE: text``. The
|
||||
#: non-greedy prefix keeps nested paths (``/`` in the path) intact.
|
||||
_SEARCH_LINE_RE = re.compile(r"^(?P<sp>.+?):(?P<line>\d+): (?P<text>.*)$")
|
||||
|
||||
@@ -472,7 +644,7 @@ def _search_result_line(body: dict[str, Any]) -> str | None:
|
||||
A search result is a ``tool``-role message — never a read result
|
||||
(those start with the agent's ``"Document "`` prefix) — that either
|
||||
carries ``source/path:LINE: text`` match lines (the agent's
|
||||
``search_documents`` output, phase 68) or the sentinel pattern
|
||||
``grep`` output, phase 68) or the sentinel pattern
|
||||
itself (its no-match line quotes the pattern). Returns the first
|
||||
match line's ``text`` part (already 200-char-capped server-side),
|
||||
or the message's first line in the sentinel-only shape, or ``None``
|
||||
@@ -523,6 +695,35 @@ def _search_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("search",)
|
||||
|
||||
|
||||
def _scaffold_flow(body: dict[str, Any]) -> str | None:
|
||||
"""Classify a phase-71 scaffolding request (see the module docstring).
|
||||
|
||||
* ``"scaffold"`` — stream ONLY the incident span
|
||||
(``SCAFFOLD_SPAN``) as ``delta.content`` chunks: ``finish_reason:
|
||||
"stop"``, no structured ``tool_calls``, no reasoning. EVERY
|
||||
request for ``SCAFFOLD_ALWAYS_TRIGGER`` (the recovery included),
|
||||
and the FIRST request of ``SCAFFOLD_TRIGGER`` (no correction in
|
||||
the system prompt yet).
|
||||
* ``"recovery"`` — ``SCAFFOLD_TRIGGER`` whose system prompt carries
|
||||
the harness correction (the one bounded recovery: ``tools=None``,
|
||||
the constant folded into the single system prompt by
|
||||
``app.api.chat`` / ``app.rag.agent``): stream the clean
|
||||
``SCAFFOLD_RECOVERY_ANSWER``.
|
||||
* ``None`` — not the scaffolding flow. The discrimination is
|
||||
stateless, like the other marker flows: the trigger phrase in
|
||||
the user message plus the correction's presence in the system
|
||||
prompt.
|
||||
"""
|
||||
user = _user(body).lower()
|
||||
if SCAFFOLD_ALWAYS_TRIGGER in user: # checked FIRST — it contains SCAFFOLD_TRIGGER
|
||||
return "scaffold"
|
||||
if SCAFFOLD_TRIGGER in user:
|
||||
if _CORRECTION_MARKER in _system(body):
|
||||
return "recovery"
|
||||
return "scaffold"
|
||||
return None
|
||||
|
||||
|
||||
def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a marker request into one step of the tool flow.
|
||||
|
||||
@@ -534,7 +735,9 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
* ``("read", source, path, "call_1")`` — a ``tool``-role catalog
|
||||
result is in the messages: the model reads its FIRST
|
||||
``source: X | path: Y | title: Z`` line (the labeled
|
||||
``source:`` / ``path:`` fields, phase 63).
|
||||
``source:`` / ``path:`` fields, phase 63), emitted as ``read`` on
|
||||
the JOINED combined ``source/path`` (phase 70: the mock joins
|
||||
the two fields — the canonical document identity).
|
||||
* ``("answer", "source/path", content)`` — a ``tool``-role read
|
||||
result (``"Document <source/path>:\n<content>"``) is in the
|
||||
messages: the model answers, quoting the read document. Reached
|
||||
@@ -599,6 +802,40 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("list", "", "")
|
||||
|
||||
|
||||
def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-72 LS-TEACH request (see the module docstring).
|
||||
|
||||
* ``("misuse",)`` — ``tools`` are offered and no ``tool``-role
|
||||
result is in the messages yet: the incident's misuse — ``ls``
|
||||
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
|
||||
"tool_calls"``, no content.
|
||||
* ``("correct",)`` — a ``tool``-role result is in the messages and
|
||||
it is NOT a catalog listing (the teaching refusal): the
|
||||
correction — ``ls`` with no arguments (id ``call_1``).
|
||||
* ``("answer", line)`` — a ``tool``-role result whose first line
|
||||
is the ``"N documents:"`` catalog header: the deterministic
|
||||
content answer ``These are the indexed documents: <line>`` (the
|
||||
first catalog line), ``finish_reason: "stop"`` — the loop
|
||||
settled in ONE correction, not at the round cap.
|
||||
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
||||
section is missing (deflected turns never carry it), or
|
||||
``tools`` are not offered and no tool results are in the
|
||||
messages yet (e.g. ``agent_max_rounds=0``).
|
||||
"""
|
||||
if LS_TEACH_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
line = _first_catalog_line(body)
|
||||
if line is not None:
|
||||
return ("answer", line)
|
||||
if _tool_results(body):
|
||||
return ("correct",)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("misuse",)
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -1047,6 +1284,29 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
if _chat_dead(RETRY_TRIGGER, RETRY_DEAD_ATTEMPTS):
|
||||
return _llm_500(RETRY_TRIGGER)
|
||||
_fail_posts[RETRY_TRIGGER] = 0 # the answer streamed — restart
|
||||
# Phase 71 (tool-scaffolding guardrails): the deterministic raw-
|
||||
# markup flow — checked BEFORE the search/tool marker flows (the
|
||||
# trigger is independent of the ``<tools>`` marker, so both
|
||||
# grounded and deflected turns hit it; SCAFFOLD_ALWAYS_TRIGGER
|
||||
# is checked first inside the classifier — the more specific
|
||||
# phrase wins, same convention as THINK_PARAS_TRIGGER).
|
||||
scaffold_flow = _scaffold_flow(body)
|
||||
if scaffold_flow is not None:
|
||||
# Request 1 (or EVERY request for the ALWAYS trigger): the
|
||||
# incident span as plain delta.content, 12-char chunks
|
||||
# (the span always spans ≥2 chunks — the filter's boundary
|
||||
# path), finish_reason "stop", no tool_calls, no reasoning.
|
||||
# Request 2 of the recovery trigger: the clean answer.
|
||||
answer = (
|
||||
SCAFFOLD_RECOVERY_ANSWER
|
||||
if scaffold_flow == "recovery"
|
||||
else SCAFFOLD_SPAN
|
||||
)
|
||||
return StreamingResponse(
|
||||
_sse_stream(answer, 0.0),
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 68 (search tool): the deterministic search marker flow —
|
||||
# checked BEFORE the phase-37 tool flow (the more specific
|
||||
# trigger phrase wins, same convention as THINK_PARAS_TRIGGER).
|
||||
@@ -1054,7 +1314,7 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
if search_flow is not None:
|
||||
if search_flow[0] == "search":
|
||||
stream = _tool_call_stream(
|
||||
"search_documents", {"pattern": SEARCH_PATTERN}, "call_0"
|
||||
"grep", {"pattern": SEARCH_PATTERN}, "call_0"
|
||||
)
|
||||
else: # "found" — quote the first matched line (80 chars)
|
||||
answer = _apply_max_tokens(
|
||||
@@ -1066,19 +1326,43 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 72 (teaching refusals): the deterministic LS-TEACH
|
||||
# self-correction flow — checked BEFORE the plain
|
||||
# TOOLS_TRIGGER flow (disjoint trigger phrases — the phase-71
|
||||
# ordering convention; the trigger needs the ``<tools>``
|
||||
# section, so deflected turns never hit it).
|
||||
ls_teach = _ls_teach_flow(body)
|
||||
if ls_teach is not None:
|
||||
if ls_teach[0] == "misuse":
|
||||
# The incident's misuse, deterministic: ls(path='.').
|
||||
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
|
||||
elif ls_teach[0] == "correct":
|
||||
# The one-round correction: the no-arg full listing.
|
||||
stream = _tool_call_stream("ls", {}, "call_1")
|
||||
else: # "answer" — quote the first catalog line
|
||||
answer = _apply_max_tokens(
|
||||
f"These are the indexed documents: {ls_teach[1]}",
|
||||
body.get("max_tokens"),
|
||||
)
|
||||
stream = _sse_stream(answer, 0.0)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
flow = _tool_flow(body)
|
||||
if flow is not None:
|
||||
if flow[0] == "list":
|
||||
stream = _tool_call_stream("list_documents", {}, "call_0")
|
||||
stream = _tool_call_stream("ls", {}, "call_0")
|
||||
elif flow[0] == "read":
|
||||
# flow[3] is the synthetic call id — "call_1" for the
|
||||
# single-read flow and the multi-read first read,
|
||||
# "call_2" for the multi-read second read (phase 45,
|
||||
# task 02).
|
||||
# task 02). Phase 70: the harness-aligned shape — one
|
||||
# combined ``source/path`` argument (the mock joins the
|
||||
# two catalog fields; the catalog format is unchanged).
|
||||
stream = _tool_call_stream(
|
||||
"read_document",
|
||||
{"source": flow[1], "path": flow[2]},
|
||||
flow[3],
|
||||
"read", {"path": f"{flow[1]}/{flow[2]}"}, flow[3]
|
||||
)
|
||||
elif flow[0] == "multi_answer":
|
||||
# Phase 45 (task 02): the multi-read forced answer —
|
||||
|
||||
@@ -9,18 +9,20 @@ MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
|
||||
does whatever it does with the tools, while this story's gate is the
|
||||
deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
|
||||
contains ``use your tools`` **and** the system prompt carries the
|
||||
``<tools>`` section of the HIGH prompt):
|
||||
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
|
||||
harness-aligned names — ``ls`` / ``read`` with the combined
|
||||
``source/path`` identity):
|
||||
|
||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``, no
|
||||
arguments, ``finish_reason: "tool_calls"``);
|
||||
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
|
||||
``finish_reason: "tool_calls"``);
|
||||
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
|
||||
``tool_calls`` delta calling ``read_document`` on the FIRST catalog
|
||||
line (id ``call_1``);
|
||||
3. request 3 (no ``tools`` parameter, the read result in the messages) →
|
||||
the content answer ``Read <source/path>. <first 80 chars of the read
|
||||
document's content>`` — so the suite can assert the read document
|
||||
reached the model and landed in the answer.
|
||||
``tool_calls`` delta calling ``read`` on the JOINED combined
|
||||
``source/path`` of the FIRST catalog line (id ``call_1``);
|
||||
3. request 3 (a ``tool``-role read result in the messages) → the content
|
||||
answer ``Read <source/path>. <first 80 chars of the read document's
|
||||
content>`` — so the suite can assert the read document reached the
|
||||
model and landed in the answer.
|
||||
|
||||
KB fixture — reproduces the TODO failure (``aws-route53.md`` references
|
||||
``example-record-file.json`` "for the exact JSON shape of
|
||||
@@ -43,10 +45,11 @@ reseelink.json" but does not include it):
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_marker_question_lists_reads_and_quotes`` — the SSE carries
|
||||
``tool`` frames (list, then read, ahead of any delta), the UI shows
|
||||
the "calling tool" label while a tool runs, the bubble shows both
|
||||
tool lines, the final answer quotes the read document, and the
|
||||
source chips include the read document (viewer link).
|
||||
``tool`` frames (``ls``, then ``read`` with the combined path, ahead
|
||||
of any delta), the UI shows the transient calling-tool status while a
|
||||
tool runs, the bubble shows both tool lines, the final answer quotes
|
||||
the read document, and the source chips include the read document
|
||||
(viewer link).
|
||||
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
|
||||
(phase 14) re-renders the tool lines.
|
||||
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
|
||||
@@ -385,12 +388,13 @@ def test_marker_question_lists_reads_and_quotes(
|
||||
assert i_list is not None and i_read is not None, statuses
|
||||
assert i_list < i_read, statuses
|
||||
|
||||
# Wire level: exactly two `tool` frames — list then read — and both
|
||||
# ahead of the first `delta` frame.
|
||||
# Wire level: exactly two `tool` frames — ``ls`` then ``read`` (the
|
||||
# combined source/path as the model passed it) — and both ahead of
|
||||
# the first `delta` frame.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "list_documents", "argument": None},
|
||||
{"type": "tool", "name": "read_document", "argument": READ_SP},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": READ_SP},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
|
||||
@@ -9,16 +9,20 @@ MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic MULTI-READ marker flow in ``tests/e2e/mock_llm.py`` (user
|
||||
message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
|
||||
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
|
||||
carries the ``<tools>`` section of the HIGH prompt):
|
||||
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
|
||||
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
|
||||
combined ``source/path`` of each catalog line):
|
||||
|
||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``);
|
||||
2. request 2 (the ``tool``-role catalog result) → ``read_document`` on
|
||||
the FIRST catalog line (id ``call_1``);
|
||||
3. request 3 (one ``tool``-role read result) → ``read_document`` on the
|
||||
SECOND catalog line (id ``call_2``) — the pre-phase-45 per-tool
|
||||
budgets would have refused exactly this second read (``No reading
|
||||
budget left — answer with what you have.``);
|
||||
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
|
||||
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
|
||||
JOINED combined ``source/path`` of the FIRST catalog line
|
||||
(id ``call_1``);
|
||||
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
|
||||
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
|
||||
— the pre-phase-45 per-tool budgets would have refused exactly this
|
||||
second read (``No reading budget left — answer with what you
|
||||
have.``);
|
||||
4. request 4 (two read results) → the forced answer, byte-stable: the
|
||||
single-read shape quoting the FIRST read result, plus the line
|
||||
``I read <sp1> and <sp2>.`` naming both read paths in read order.
|
||||
@@ -40,7 +44,8 @@ pattern, grown to three documents):
|
||||
Three documents (not two, as in phase 37) so BOTH reads land on
|
||||
documents outside the seed: with a two-document corpus the second read
|
||||
would be the already-in-context retrieval document and the agent would
|
||||
answer "Already in your context." — a rejection, not the multi-read
|
||||
answer "Already in your context — …" (the in-context refusal) — a
|
||||
rejection, not the multi-read
|
||||
flow this story proves.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
@@ -393,14 +398,15 @@ def test_multi_read_turn(
|
||||
_submit(page, MULTI_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level: exactly THREE `tool` frames — list, read #1, read #2,
|
||||
# in order — and all ahead of the first `delta` frame. This third
|
||||
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
|
||||
# (each read's argument is the JOINED combined source/path), in
|
||||
# order — and all ahead of the first `delta` frame. This third
|
||||
# frame is the one the pre-phase-45 read budget refused.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "list_documents", "argument": None},
|
||||
{"type": "tool", "name": "read_document", "argument": READ1_SP},
|
||||
{"type": "tool", "name": "read_document", "argument": READ2_SP},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": READ1_SP},
|
||||
{"type": "tool", "name": "read", "argument": READ2_SP},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
@@ -518,7 +524,7 @@ def test_relist_allowed(
|
||||
# per-tool budgets would have refused (list budget 1, read budget
|
||||
# 1 — this turn makes one list and TWO reads).
|
||||
frames = _frames(page)
|
||||
assert {"type": "tool", "name": "list_documents", "argument": None} in _tool_frames(
|
||||
assert {"type": "tool", "name": "ls", "argument": None} in _tool_frames(
|
||||
frames
|
||||
)
|
||||
line0 = page.locator(".msg.brain .tool-call").nth(0)
|
||||
@@ -556,12 +562,13 @@ def test_single_tool_flow_regression(
|
||||
_submit(page, SINGLE_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Exactly TWO tool frames — list then ONE read of the first catalog
|
||||
# line — no second read (the marker carries no multi-read trigger).
|
||||
# Exactly TWO tool frames — ls then ONE read of the first catalog
|
||||
# line (the JOINED combined source/path) — no second read (the
|
||||
# marker carries no multi-read trigger).
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "list_documents", "argument": None},
|
||||
{"type": "tool", "name": "read_document", "argument": READ1_SP},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": READ1_SP},
|
||||
]
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
|
||||
@@ -0,0 +1,524 @@
|
||||
"""Phase 70 E2E (Playwright, mock-only): the harness-aligned tool surface
|
||||
(``ls`` / ``read(path)`` / ``grep(pattern, path?)``).
|
||||
|
||||
Story: ``.agent/user_stories/agent-document-tools.md`` (phase 70 reshapes
|
||||
the tools that story delivered — owner decision 2026-09-03: "match
|
||||
existing harnesses as much as possible", the pi.dev tool shapes).
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic marker flows in ``tests/e2e/mock_llm.py`` (phase 70: the
|
||||
flows emit the NEW names with the NEW argument shapes):
|
||||
|
||||
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
|
||||
prompt's ``<tools>`` section): ``ls`` (id ``call_0``, no arguments) →
|
||||
``read`` on the JOINED combined ``source/path`` of the first catalog
|
||||
line (id ``call_1``) → the ``Read <source/path>. <quote>`` answer;
|
||||
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
|
||||
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
|
||||
(id ``call_0``) → the ``Found <matched line>`` answer.
|
||||
|
||||
The combined ``source/path`` string is the canonical document identity:
|
||||
the mock joins the two labeled catalog fields itself (the catalog
|
||||
format is unchanged), and the SSE ``tool`` frames carry exactly what the
|
||||
model "passed" — ``read``'s combined path, ``grep``'s pattern, ``ls``'s
|
||||
scope or null when unscoped (the phase-70 argument rule).
|
||||
|
||||
KB fixtures:
|
||||
|
||||
* READ flow — the ``test_agent_document_tools.py`` two-document pair
|
||||
(TRUNCATE-then-seed): ``Homelab/aws-route53.md`` seeded with one
|
||||
chunk whose embedding is the mock's own bag-of-words vector (the
|
||||
marker question cosines ≈0.69 against it, well past the E2E 0.30
|
||||
threshold, and it FTS-matches too → grounded) and
|
||||
``Deployments/example-record-file.json`` indexed WITHOUT chunks (the
|
||||
catalog-first line the mock reads; never in the retrieval context).
|
||||
* SEARCH flow — the phase-68 fixture (``tests/fixtures/search_docs/``)
|
||||
imported through the real importer, its line 6 carrying the sentinel
|
||||
``reese-sentinel-42`` exactly once (``test_search_tool.py`` pattern).
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
|
||||
grounded READ turn: the UI shows the ``ls`` line (unscoped "🔎
|
||||
Listing documents", no argument) then the "📄 Reading <source/path>"
|
||||
line with the combined path in a ``<code>`` element, the answer
|
||||
streams and quotes the read document, the done-state sources
|
||||
include the read document, and NO raw tool markup (``<|…|>``,
|
||||
``tool_call``) appears anywhere in the DOM — the live incident this
|
||||
phase fixes.
|
||||
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
|
||||
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
|
||||
matched-line answer.
|
||||
3. ``test_wire_argument_rule_across_both_flows`` — the SSE wire across
|
||||
BOTH flows in one session: every ``tool`` frame's name is in
|
||||
{``ls``, ``read``, ``grep``} (no pre-phase-70 name ever reaches the
|
||||
client) and the argument rule holds — ``read`` → the combined path
|
||||
as passed, ``grep`` → the pattern, ``ls`` → null when unscoped.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from tests.e2e.mock_llm import SEARCH_PATTERN, embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "search_docs"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# READ flow — the two-document pair (cf. test_agent_document_tools.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE = "Homelab"
|
||||
SEED_PATH = "aws-route53.md"
|
||||
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
|
||||
|
||||
READ_SOURCE = "Deployments"
|
||||
READ_PATH = "example-record-file.json"
|
||||
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
|
||||
|
||||
#: The retrievable document (the grounded seed context): the repeated
|
||||
#: record-file lines carry the marker question's key tokens — verified
|
||||
#: ≈0.69 cosine against the mock's embeddings (E2E threshold 0.30) plus
|
||||
#: FTS hits.
|
||||
ROUTE53_CONTENT = (
|
||||
"# AWS Route 53 Notes\n\n"
|
||||
"## Record file\n\n"
|
||||
+ (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the exact JSON shape of reeselink.json is "
|
||||
"documented in example-record-file.json.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
|
||||
#: The read document (the catalog-first line the mock reads; no chunks,
|
||||
#: so retrieval never puts it in context). Its FIRST line is longer than
|
||||
#: 80 chars, so the mock's first-80-chars quote is newline-free.
|
||||
RECORD_FILE_CONTENT = (
|
||||
'{ "version": 3, "comment": "ReeseLink hosted zone records — the exact '
|
||||
'JSON shape of reeselink.json",\n'
|
||||
' "hosted_zone_id": "Z0RESEELINK01",\n'
|
||||
' "record_sets": [\n'
|
||||
' { "name": "www.reeselink.example", "type": "A", "ttl": 300,\n'
|
||||
' "resource_records": [ { "value": "10.0.0.20" } ] },\n'
|
||||
' { "name": "api.reeselink.example", "type": "CNAME", "ttl": 300,\n'
|
||||
' "resource_records": [ { "value": "www.reeselink.example" } ] }\n'
|
||||
" ]\n"
|
||||
"}\n"
|
||||
)
|
||||
assert "\n" not in RECORD_FILE_CONTENT[:80] # the quote must stay one line
|
||||
|
||||
#: Carries ``TOOLS_TRIGGER`` (and nothing else — no multi-read, no
|
||||
#: search, no other mock marker).
|
||||
READ_QUESTION = (
|
||||
"Use your tools: what is the exact JSON shape of reeselink.json "
|
||||
"for my aws route53 hosted zone?"
|
||||
)
|
||||
for _other in (
|
||||
"read two documents",
|
||||
"search your documents",
|
||||
"write a long answer",
|
||||
"think in paragraphs",
|
||||
"think out loud",
|
||||
"show the end of your notes",
|
||||
"show me a table",
|
||||
"fail then answer",
|
||||
"always fail",
|
||||
"embed fail once",
|
||||
"pretend to think slowly",
|
||||
):
|
||||
assert _other not in READ_QUESTION.lower(), _other
|
||||
|
||||
READ_ANSWER_PREFIX = f"Read {READ_SP}."
|
||||
READ_ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
|
||||
|
||||
|
||||
def _seed_read_pair(db: Session) -> None:
|
||||
"""The two-document READ-flow KB (see the module docstring)."""
|
||||
md = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=SEED_PATH,
|
||||
full_path=f"/tmp/{SEED_PATH}",
|
||||
title="AWS Route 53 Notes",
|
||||
content=ROUTE53_CONTENT,
|
||||
content_hash=hashlib.sha256(ROUTE53_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
db.add(md)
|
||||
db.flush()
|
||||
# One chunk carrying the mock's own embedding → genuine token
|
||||
# overlap between the marker question and this document (the only
|
||||
# retrievable document).
|
||||
db.add(
|
||||
Chunk(
|
||||
document_id=md.id,
|
||||
position=0,
|
||||
content=ROUTE53_CONTENT,
|
||||
embedding=embed_text(ROUTE53_CONTENT),
|
||||
)
|
||||
)
|
||||
db.add(
|
||||
Document(
|
||||
source=READ_SOURCE,
|
||||
path=READ_PATH,
|
||||
full_path=f"/tmp/{READ_PATH}",
|
||||
title="Example Record File",
|
||||
content=RECORD_FILE_CONTENT,
|
||||
content_hash=hashlib.sha256(RECORD_FILE_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# SEARCH flow — the phase-68 fixture (cf. test_search_tool.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE_S = "search_docs"
|
||||
SEED_PATH_S = "reese-notes.md"
|
||||
SEED_SP_S = f"{SEED_SOURCE_S}/{SEED_PATH_S}"
|
||||
|
||||
#: The fixture's sentinel line (line 6) — the mock's grep matches it
|
||||
#: exactly once; its ``text`` part is what the "Found …" answer quotes.
|
||||
SENTINEL_LINE = f"The offsite vault passphrase marker is {SEARCH_PATTERN}."
|
||||
FOUND_ANSWER = f"Found {SENTINEL_LINE[:80]}"
|
||||
|
||||
#: Carries ``SEARCH_TRIGGER`` and is on-topic (cosine ≈0.51 against the
|
||||
#: fixture + FTS hits → HIGH gate, the ``<tools>`` section rides along).
|
||||
SEARCH_QUESTION = (
|
||||
"Search your documents for the vault passphrase marker in my homelab "
|
||||
"kubernetes backup notes?"
|
||||
)
|
||||
assert SEARCH_PATTERN.lower() not in SEARCH_QUESTION.lower()
|
||||
|
||||
|
||||
def _pin_fixture() -> None:
|
||||
"""The fixture carries the sentinel on line 6, exactly once."""
|
||||
content = (FIXTURES / SEED_PATH_S).read_text(encoding="utf-8")
|
||||
lines = content.split("\n")
|
||||
assert lines[5] == SENTINEL_LINE, lines[5]
|
||||
assert sum(SEARCH_PATTERN in line for line in lines) == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# DB seeding (TRUNCATE-then-seed / TRUNCATE-then-import)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _import_search_fixtures(mock_port: int) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {
|
||||
"_env_file": None,
|
||||
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
|
||||
}
|
||||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return await import_sources([FIXTURES], LLMClient(settings))
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread.
|
||||
|
||||
Playwright's sync API keeps an asyncio loop running on the test
|
||||
thread, so ``asyncio.run`` cannot be called directly from a test
|
||||
body (the established house helper).
|
||||
"""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
t = Thread(target=runner)
|
||||
t.start()
|
||||
t.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _reset_db_read_pair() -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then seed the
|
||||
two-document READ-flow pair. ``steering_notes`` / ``kb_overview``
|
||||
are truncated too, so the HIGH prompt is exactly ``<relevance>`` +
|
||||
``<documents>`` + ``<tools>`` — byte-stable prompts, byte-stable
|
||||
answers."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
_seed_read_pair(db)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _reset_db_search_fixture(mock_port: int) -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then import the
|
||||
phase-68 search fixture through the real importer."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
summary = _run_in_thread(_import_search_fixtures(mock_port))
|
||||
assert summary is not None and summary.added == 1, summary
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_agent_document_tools.py pattern)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_sse_hook(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _drain_frames(page: Page) -> list[dict]:
|
||||
"""One turn's SSE frames: wait for that turn's ``done`` frame, then
|
||||
return EVERY frame captured since the last drain (the hook's
|
||||
background read appends the whole stream at once after it closes, so
|
||||
clearing-and-reading is race-free per turn)."""
|
||||
deadline = time.monotonic() + 10.0
|
||||
while True:
|
||||
raw = page.evaluate(
|
||||
"() => { const f = window.__sseFrames || []; "
|
||||
"window.__sseFrames = []; return f; }"
|
||||
)
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered.
|
||||
|
||||
Phase 48: the label assertion carries the settle wait with an
|
||||
explicit timeout — the in-flight button is the enabled Stop control
|
||||
(never disabled), so ``to_be_enabled`` no longer blocks until the
|
||||
turn settles."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The grounded READ turn: ls line → Reading line → quoted answer,
|
||||
# sources include the read doc, no raw tool markup anywhere in the DOM
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_read_flow_lines_answer_sources_no_raw_markup(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_read_pair()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, READ_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# The UI shows the ls line (UNSCOPED — no argument, no <code>) then
|
||||
# the "📄 Reading <source/path>" line with the COMBINED path in a
|
||||
# <code> element (the path is data, never markup).
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(0).locator("code")).to_have_count(0)
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1).locator("code")).to_have_text(READ_SP)
|
||||
|
||||
# The answer streamed and quotes the read document (the mock's
|
||||
# deterministic echo: "Read <source/path>. <first 80 chars>").
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||
|
||||
# Wire level: ls then read — the phase-70 argument rule (ls
|
||||
# unscoped → null; read → the combined path as passed) — ahead of
|
||||
# the first delta.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": READ_SP},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
# Done-state sources include the read document (retrieval doc first,
|
||||
# the agent's read doc after — the phase-37 extension contract).
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ_SOURCE, READ_PATH),
|
||||
]
|
||||
|
||||
# The live incident this phase fixes: NO raw tool markup anywhere in
|
||||
# the DOM — the model's trained wire shapes (<|tool_call_…|>,
|
||||
# "tool_calls", finish_reason) must never leak into the rendered
|
||||
# conversation.
|
||||
dom = page.locator("#messages").inner_html()
|
||||
for raw in ("<|", "tool_call", "tool_calls", "finish_reason"):
|
||||
assert raw not in dom, f"raw tool markup {raw!r} leaked into the DOM"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The grounded SEARCH turn: the "🔎 Searching for <pattern>" line,
|
||||
# then the matched-line answer
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_grep_flow_line_then_answer(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_pin_fixture()
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_search_fixture(mock_llm)
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, SEARCH_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# ONE tool line above the answer: "🔎 Searching for " + the sentinel
|
||||
# in a <code> element (the pattern is data, never markup).
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text("Searching for")
|
||||
expect(lines.nth(0).locator("code")).to_have_text(SEARCH_PATTERN)
|
||||
|
||||
# The answer quotes the MATCHED LINE — the grep result reached the
|
||||
# model and landed in the answer (the mock's deterministic echo).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(FOUND_ANSWER)
|
||||
|
||||
# Wire level: exactly ONE tool frame — grep carrying the PATTERN as
|
||||
# its argument (the phase-70 argument rule) — ahead of the first
|
||||
# delta; the turn is grounded.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. The SSE wire across BOTH flows: every tool frame carries a
|
||||
# phase-70 name and the single-string argument rule
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_wire_argument_rule_across_both_flows(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_pin_fixture()
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_read_pair()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
# Turn 1 — the READ flow (ls → read on the combined path).
|
||||
_submit(page, READ_QUESTION)
|
||||
_wait_settled(page)
|
||||
read_frames = _drain_frames(page)
|
||||
|
||||
# Turn 2 — re-seed the search fixture, then the SEARCH flow (grep
|
||||
# for the sentinel). The app's chat path is single-turn (system +
|
||||
# user message), so the first turn cannot influence this one.
|
||||
_reset_db_search_fixture(mock_llm)
|
||||
_submit(page, SEARCH_QUESTION)
|
||||
_wait_settled(page)
|
||||
search_frames = _drain_frames(page)
|
||||
|
||||
read_tools = _tool_frames(read_frames)
|
||||
search_tools = _tool_frames(search_frames)
|
||||
# The ordered, combined tool-frame sequence across both flows: the
|
||||
# argument rule end-to-end — read → the combined path as passed,
|
||||
# grep → the pattern, ls → null when unscoped.
|
||||
assert read_tools + search_tools == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": READ_SP},
|
||||
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
|
||||
]
|
||||
# No pre-phase-70 name ever reaches the client.
|
||||
for frame in read_tools + search_tools:
|
||||
assert frame["name"] in {"ls", "read", "grep"}, frame
|
||||
assert frame["argument"] is None or isinstance(frame["argument"], str)
|
||||
|
||||
# And both turns answered (neither flow stalled at a tool round).
|
||||
assert next(f for f in read_frames if f["type"] == "done")["deflected"] is False
|
||||
assert next(f for f in search_frames if f["type"] == "done")["deflected"] is False
|
||||
@@ -1,4 +1,6 @@
|
||||
"""Phase 68 E2E (Playwright, mock-only): the ``search_documents`` tool.
|
||||
"""Phase 68 E2E (Playwright, mock-only): the ``grep`` tool (the
|
||||
phase-68 search tool, renamed to the harness-aligned ``grep`` in
|
||||
phase 70; the A5 match/output contract is unchanged).
|
||||
|
||||
Story: n/a (TODO-derived — the owner roadmap confirmation 2026-09-01,
|
||||
TODO.md L4: "Add a search tool that allows the LLM to grep through the
|
||||
@@ -14,8 +16,8 @@ message contains ``search your documents`` (``SEARCH_TRIGGER``)
|
||||
prompt):
|
||||
|
||||
1. request 1 (``tools`` offered, no search result yet) → streams ONLY
|
||||
``tool_calls`` deltas calling ``search_documents`` with
|
||||
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
|
||||
``tool_calls`` deltas calling ``grep`` with ``{"pattern":
|
||||
SEARCH_PATTERN}`` (id ``call_0``);
|
||||
2. request 2 (a ``tool``-role search result — the
|
||||
``source/path:line: text`` match line) → the content answer
|
||||
``Found <first matched line's content up to 80 chars>`` — so this
|
||||
@@ -40,13 +42,13 @@ shadow the phase-37/45 flows and vice versa).
|
||||
|
||||
Test → phase mapping:
|
||||
1. ``test_search_flow_searches_and_answers_from_match`` — the live
|
||||
search flow: the SSE carries the ``tool`` frame
|
||||
(``search_documents`` with ``argument = <sentinel>``, ahead of any
|
||||
delta), #send-status recorded the transient "… is searching for
|
||||
<sentinel>" state, the bubble shows ONE ``🔎 Searching for``
|
||||
tool line with the sentinel in a ``<code>`` element, the answer
|
||||
quotes the matched line (``Found …`` — the match reached the
|
||||
model), and the turn settles to idle with no error banner.
|
||||
search flow: the SSE carries the ``tool`` frame (``grep`` with
|
||||
``argument = <sentinel>``, ahead of any delta), #send-status
|
||||
recorded the transient "… is searching for <sentinel>" state, the
|
||||
bubble shows ONE ``🔎 Searching for`` tool line with the sentinel in
|
||||
a ``<code>`` element, the answer quotes the matched line
|
||||
(``Found …`` — the match reached the model), and the turn settles to
|
||||
idle with no error banner.
|
||||
2. ``test_search_adds_no_source_by_itself`` — context accounting
|
||||
(locked A5): the search-only flow (no read) leaves
|
||||
``done.sources`` / the source chips / ``query_log.sources`` at the
|
||||
@@ -367,12 +369,12 @@ def test_search_flow_searches_and_answers_from_match(
|
||||
)
|
||||
assert i_think is not None and i_think < i_search, statuses
|
||||
|
||||
# Wire level: exactly ONE `tool` frame — search_documents carrying
|
||||
# the PATTERN as its argument (phase 68 task 02) — ahead of the
|
||||
# first `delta` frame.
|
||||
# Wire level: exactly ONE `tool` frame — grep carrying the PATTERN
|
||||
# as its argument (phase 68 task 02; phase 70 renamed the tool) —
|
||||
# ahead of the first `delta` frame.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "search_documents", "argument": SEARCH_PATTERN}
|
||||
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
@@ -424,7 +426,7 @@ def test_search_adds_no_source_by_itself(
|
||||
# baseline: the one fixture doc, nothing added by the search.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "search_documents", "argument": SEARCH_PATTERN}
|
||||
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
|
||||
]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
|
||||
@@ -0,0 +1,472 @@
|
||||
"""Phase 72 E2E (Playwright, mock-only): the ls-teaching
|
||||
self-correction loop through the real UI.
|
||||
|
||||
Story: ``.agent/user_stories/agent-document-tools.md`` (this phase
|
||||
repairs the model-facing contract the phase-70 tools reshaped — the
|
||||
2026-09-03 incident: the harness-prior ``ls(path='.')`` misuse met the
|
||||
terse refusal, and the model re-reasoned the same paragraphs over and
|
||||
over before answering from the seed documents alone).
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
|
||||
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
|
||||
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
|
||||
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
|
||||
(``No source named '.' — check the ls output. (…)``) → the corrected
|
||||
no-arg ``ls()`` (id ``call_1``) → the deterministic
|
||||
``These are the indexed documents: <first catalog line>`` answer.
|
||||
|
||||
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
|
||||
documents of known ``source``/``path``/``title`` (catalog order =
|
||||
``(source, path)``, so the first catalog line is deterministic):
|
||||
|
||||
* ``Homelab/aws-route53.md`` — the CATALOG-FIRST document, indexed
|
||||
WITHOUT chunks (catalog-only; never in the retrieval context, so
|
||||
the single-read flow's ``read`` of it is NOT deduped as already-in-
|
||||
context). Its FIRST line is longer than 80 chars, so the mock's
|
||||
first-80-chars quote (the single-read regression turn) stays
|
||||
newline-free.
|
||||
* ``Homelab/example-record-file.json`` — the retrievable document:
|
||||
one chunk whose embedding is the mock's own bag-of-words vector
|
||||
(the trigger question cosines well past the E2E 0.30 threshold and
|
||||
FTS-matches too → grounded, the ``<tools>`` section rides along).
|
||||
It is the seed context only — the single-read flow reads the
|
||||
catalog-FIRST document, not the seed.
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
|
||||
LS-TEACH turn: the turn settles (composer re-enables, ``done``
|
||||
observed), the answer bubble carries the first catalog line — the
|
||||
first document's ``source:`` / ``path:`` / title fields (the
|
||||
catalog reached the model and landed in the answer), the UI shows
|
||||
the two tool lines (``🔎 Listing documents in <code>.</code>``
|
||||
then ``🔎 Listing documents``), and no error banner. Wire level:
|
||||
the ``tool`` frames arrive in order — first ``ls`` with
|
||||
``argument: "."``, then ``ls`` with ``argument: null`` — and there
|
||||
is NO third ``tool`` frame (the loop ended in one correction, not
|
||||
at the round cap).
|
||||
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
|
||||
session, the LS-TEACH turn settles and a follow-up question
|
||||
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
|
||||
with the read flow's answer (``ls`` → ``read`` on the first
|
||||
catalog line's combined identity → ``Read <source/path>. <quote>``)
|
||||
— the new flow did not swallow the existing trigger.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from tests.e2e.mock_llm import (
|
||||
LS_TEACH_TRIGGER,
|
||||
TOOLS_TRIGGER,
|
||||
embed_text,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The one-source, two-document fixture (see the module docstring)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE = "Homelab"
|
||||
DOC1_PATH = "aws-route53.md"
|
||||
DOC1_TITLE = "AWS Route 53 Notes"
|
||||
DOC1_SP = f"{SEED_SOURCE}/{DOC1_PATH}"
|
||||
|
||||
DOC2_PATH = "example-record-file.json"
|
||||
DOC2_TITLE = "Example Record File"
|
||||
DOC2_SP = f"{SEED_SOURCE}/{DOC2_PATH}"
|
||||
|
||||
#: The FIRST catalog line (catalog order = (source, path) — DOC1 sorts
|
||||
#: first): the mock's LS-TEACH answer quotes exactly this line.
|
||||
FIRST_CATALOG_LINE = (
|
||||
f"source: {SEED_SOURCE} | path: {DOC1_PATH} | title: {DOC1_TITLE}"
|
||||
)
|
||||
|
||||
#: The catalog-first document (catalog order = (source, path) —
|
||||
#: DOC1 sorts first): the single-read flow reads THIS document, so it
|
||||
#: must NOT be the seed (a seed read dedupes to "Already in your
|
||||
#: context.", which the mock's single-read flow does not model — it
|
||||
#: would loop to the round cap). Indexed WITHOUT chunks: catalog-only,
|
||||
#: never in the retrieval context. Its FIRST line is longer than 80
|
||||
#: chars, so the mock's first-80-chars quote (the single-read
|
||||
#: regression turn) stays newline-free.
|
||||
DOC1_CONTENT = (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the exact JSON shape of reeselink.json is "
|
||||
"documented in the record file below.\n"
|
||||
+ (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the record file shape of reeselink.json is "
|
||||
"the contract every sync job relies on.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
assert "\n" not in DOC1_CONTENT[:80] # the quote must stay one line
|
||||
|
||||
#: The retrievable document (the grounded seed context, the cf.
|
||||
#: test_harness_aligned_tools.py pattern): the repeated record-file
|
||||
#: lines carry the trigger question's key tokens — well past the E2E
|
||||
#: 0.30 cosine threshold, plus FTS hits. Its FIRST line is longer
|
||||
#: than 80 chars too, so the retrieval seed context is one clean
|
||||
#: line.
|
||||
DOC2_CONTENT = (
|
||||
"The ReeseLink hosted zone record file reeselink.json holds every "
|
||||
"aws route53 record for reeselink — the note documents the exact "
|
||||
"JSON shape of reeselink.json for the record file.\n"
|
||||
+ (
|
||||
"The aws route53 record file reeselink.json keeps every record "
|
||||
"for the reeselink hosted zone — the exact JSON shape of the "
|
||||
"record file is the contract every sync job relies on.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
assert "\n" not in DOC2_CONTENT[:80] # the seed context stays one line
|
||||
|
||||
#: Carries ``LS_TEACH_TRIGGER`` and is on-topic (grounded — HIGH, the
|
||||
#: ``<tools>`` section rides along); it carries NO other mock marker.
|
||||
LS_TEACH_QUESTION = (
|
||||
"List the files in this directory — what do my aws route53 notes "
|
||||
"say about the reeselink.json record file?"
|
||||
)
|
||||
assert LS_TEACH_TRIGGER in LS_TEACH_QUESTION.lower()
|
||||
for _other in (
|
||||
"use your tools",
|
||||
"read two documents",
|
||||
"search your documents",
|
||||
"emit raw tool markup",
|
||||
"always emit raw tool markup",
|
||||
"show me a table",
|
||||
"think in paragraphs",
|
||||
"think out loud then hesitate",
|
||||
"think out loud",
|
||||
"show the end of your notes",
|
||||
"write a long answer",
|
||||
"fail then answer",
|
||||
"always fail",
|
||||
"embed fail once",
|
||||
"pretend to think slowly",
|
||||
):
|
||||
assert _other not in LS_TEACH_QUESTION.lower(), _other
|
||||
|
||||
#: Carries ``TOOLS_TRIGGER`` (the single-read flow) and nothing else —
|
||||
#: the no-regression follow-up question in the same session.
|
||||
READ_QUESTION = (
|
||||
"Use your tools: what is the exact JSON shape of reeselink.json "
|
||||
"for my aws route53 hosted zone?"
|
||||
)
|
||||
assert TOOLS_TRIGGER in READ_QUESTION.lower()
|
||||
for _other in (
|
||||
LS_TEACH_TRIGGER,
|
||||
"read two documents",
|
||||
"search your documents",
|
||||
"emit raw tool markup",
|
||||
"always emit raw tool markup",
|
||||
"show me a table",
|
||||
"think in paragraphs",
|
||||
"think out loud then hesitate",
|
||||
"think out loud",
|
||||
"show the end of your notes",
|
||||
"write a long answer",
|
||||
"fail then answer",
|
||||
"always fail",
|
||||
"embed fail once",
|
||||
"pretend to think slowly",
|
||||
):
|
||||
assert _other not in READ_QUESTION.lower(), _other
|
||||
|
||||
#: The mock's single-read answer (the read document reached the model
|
||||
#: and landed in the answer) — DOC1 is the first catalog line, so the
|
||||
#: flow reads ``Homelab/aws-route53.md`` and quotes its first 80 chars.
|
||||
READ_ANSWER_PREFIX = f"Read {DOC1_SP}."
|
||||
READ_ANSWER_QUOTE = DOC1_CONTENT[:80]
|
||||
|
||||
|
||||
def _seed_fixture(db: Session) -> None:
|
||||
"""The one-source, two-document fixture (see the module docstring).
|
||||
|
||||
DOC1 (catalog-first) is indexed WITHOUT chunks; DOC2 carries the
|
||||
single chunk (the mock's own embedding → the trigger question
|
||||
cosines well past the E2E 0.30 threshold and FTS-matches too →
|
||||
grounded). DOC2 is the seed context only — the single-read flow
|
||||
reads the catalog-FIRST document (DOC1), which is not in context.
|
||||
"""
|
||||
db.add(
|
||||
Document(
|
||||
source=SEED_SOURCE,
|
||||
path=DOC1_PATH,
|
||||
full_path=f"/tmp/{DOC1_PATH}",
|
||||
title=DOC1_TITLE,
|
||||
content=DOC1_CONTENT,
|
||||
content_hash=hashlib.sha256(DOC1_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
doc2 = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=DOC2_PATH,
|
||||
full_path=f"/tmp/{DOC2_PATH}",
|
||||
title=DOC2_TITLE,
|
||||
content=DOC2_CONTENT,
|
||||
content_hash=hashlib.sha256(DOC2_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
db.add(doc2)
|
||||
db.flush()
|
||||
# One chunk carrying the mock's own embedding → genuine token
|
||||
# overlap between the trigger question and DOC2 (the only
|
||||
# retrievable document).
|
||||
db.add(
|
||||
Chunk(
|
||||
document_id=doc2.id,
|
||||
position=0,
|
||||
content=DOC2_CONTENT,
|
||||
embedding=embed_text(DOC2_CONTENT),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _reset_db_fixture() -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then seed the
|
||||
one-source, two-document fixture. ``steering_notes`` /
|
||||
``kb_overview`` are truncated too, so the HIGH prompt is exactly
|
||||
``<relevance>`` + ``<documents>`` + ``<tools>`` — byte-stable
|
||||
prompts, byte-stable answers."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
_seed_fixture(db)
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the house pattern — cf. test_harness_aligned_tools.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_sse_hook(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _drain_frames(page: Page) -> list[dict]:
|
||||
"""One turn's SSE frames: wait for that turn's ``done`` frame, then
|
||||
return EVERY frame captured since the last drain (the hook's
|
||||
background read appends the whole stream at once after it closes, so
|
||||
clearing-and-reading is race-free per turn)."""
|
||||
deadline = time.monotonic() + 10.0
|
||||
while True:
|
||||
raw = page.evaluate(
|
||||
"() => { const f = window.__sseFrames || []; "
|
||||
"window.__sseFrames = []; return f; }"
|
||||
)
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered.
|
||||
|
||||
Phase 48: the label assertion carries the settle wait with an
|
||||
explicit timeout — the in-flight button is the enabled Stop control
|
||||
(never disabled), so ``to_be_enabled`` no longer blocks until the
|
||||
turn settles."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _assert_no_error_banner(page: Page) -> None:
|
||||
"""The turn settled through the normal done path — never the red
|
||||
role=alert error banner (the KB-offline banner is a separate,
|
||||
health-driven state the db_ready fixture keeps away)."""
|
||||
banner = page.locator("#kb-banner")
|
||||
expect(banner).to_be_hidden()
|
||||
expect(banner).not_to_have_attribute("role", "alert")
|
||||
expect(banner).not_to_have_class(re.compile(r"is-error"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The grounded LS-TEACH turn: the incident's ls(path='.') misuse →
|
||||
# the teaching refusal → the corrected no-arg ls() → the catalog
|
||||
# answer — the loop settles in ONE correction (two tool rounds),
|
||||
# pinned on the SSE wire
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ls_misuse_self_corrects_to_noarg_listing(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_fixture()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, LS_TEACH_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Self-correction: the answer quotes the FIRST catalog line — the
|
||||
# first document's source: / path: / title fields reached the model
|
||||
# and landed in the answer (the catalog round settled the turn).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("These are the indexed documents:")
|
||||
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# The UI shows the two tool lines in order: the scoped misuse
|
||||
# (🔎 Listing documents in <code>.</code>) then the corrected
|
||||
# unscoped listing (🔎 Listing documents — no <code>).
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(0).locator("code")).to_have_text(".")
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1).locator("code")).to_have_count(0)
|
||||
|
||||
# Two rounds on the wire: the tool frames arrive in order — first
|
||||
# ls with argument "." (the incident's misuse), then ls with
|
||||
# argument null (the correction) — and there is NO third tool
|
||||
# frame: the loop ended in one correction, not at the round cap.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": "."},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert not [f for f in frames if f.get("type") == "error"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. No regression to the plain flow — the SAME session: after the
|
||||
# LS-TEACH turn, the TOOLS_TRIGGER follow-up (the single-read flow)
|
||||
# still settles with the read flow's answer
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_fixture()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
# Turn 1 — the LS-TEACH flow (the incident's misuse → the
|
||||
# correction → the catalog answer).
|
||||
_submit(page, LS_TEACH_QUESTION)
|
||||
_wait_settled(page)
|
||||
teach_frames = _drain_frames(page)
|
||||
assert _tool_frames(teach_frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": "."},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
]
|
||||
expect(
|
||||
page.locator(".msg.brain .bubble").last
|
||||
).to_contain_text(FIRST_CATALOG_LINE)
|
||||
|
||||
# Turn 2 — the SAME session: the single-read flow on
|
||||
# TOOLS_TRIGGER. The new flow must not have swallowed the existing
|
||||
# trigger: the follow-up settles with the read flow's answer.
|
||||
_submit(page, READ_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
second_msg = page.locator(".msg.brain").last
|
||||
# The UI shows the single-read flow's two lines: the unscoped ls
|
||||
# then the read of the first catalog line's COMBINED identity.
|
||||
lines = second_msg.locator(".tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(0).locator("code")).to_have_count(0)
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
|
||||
|
||||
# The answer quotes the read document (the mock's deterministic
|
||||
# echo: "Read <source/path>. <first 80 chars>").
|
||||
bubble = second_msg.locator(".bubble").last
|
||||
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# Wire level for the follow-up: ls (null) → read (the combined
|
||||
# identity) — the single-read flow, unchanged.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": DOC1_SP},
|
||||
]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert not [f for f in frames if f.get("type") == "error"]
|
||||
@@ -0,0 +1,438 @@
|
||||
"""Phase 71 E2E (Playwright, mock-only): tool-scaffolding guardrails —
|
||||
the deterministic strip + one bounded recovery, through the real UI.
|
||||
|
||||
Owner request (chat, 2026-09-03): the same incident as phase 70 — a
|
||||
deflected round streamed the model's raw
|
||||
``<|tool_call_start|>[read(path='…')]<|tool_call_end|>`` chat-template
|
||||
markup into the UI although no tools were offered. The guardrail is
|
||||
DETERMINISTIC ONLY (no model in detection or repair): a streaming
|
||||
filter strips known scaffolding from ``delta.content`` server-side
|
||||
(``app/rag/scaffolding.py``), and a reply whose visible content ends
|
||||
up empty gets exactly ONE bounded recovery (``tools=None``, the
|
||||
harness correction folded into the system prompt); a second empty
|
||||
reply settles with the dedicated error frame.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic scaffolding flows in ``tests/e2e/mock_llm.py`` (phase
|
||||
71), which are independent of the ``<tools>`` marker:
|
||||
|
||||
* ``emit raw tool markup`` (``SCAFFOLD_TRIGGER``): request 1 streams
|
||||
ONLY the incident span as ``delta.content`` (split across the
|
||||
mock's 12-char chunks — the filter's boundary path), no structured
|
||||
``tool_calls``, no reasoning; the follow-up request carrying
|
||||
``CORRECTION_INSTRUCTION`` in the system prompt (the one bounded
|
||||
recovery) streams the clean ``SCAFFOLD_RECOVERY_ANSWER``.
|
||||
* ``always emit raw tool markup`` (``SCAFFOLD_ALWAYS_TRIGGER``): the
|
||||
scaffolding-only span on EVERY request (the recovery included) —
|
||||
the terminal malformed-reply path.
|
||||
|
||||
Both triggers run on an EMPTY knowledge base: with zero chunks the
|
||||
honesty gate is LOW (cosine 0.0 < the E2E 0.30 threshold, no FTS
|
||||
hits), so every turn takes the DEFLECTED path — the exact path the
|
||||
2026-09-03 incident hit — where the filter + recovery live in
|
||||
``app/api/chat.py``.
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_recovery_strips_scaffolding_and_streams_clean_answer`` — the
|
||||
recovery case: the turn settles (the composer re-enables, ``done``
|
||||
on the wire), the final answer bubble carries the recovery's clean
|
||||
text, ``document.body.innerText`` contains NEITHER
|
||||
``tool_call_start`` nor ``tool_call_end`` (nor the raw
|
||||
``[read(path=…`` fragment), and no error banner — and wire-level,
|
||||
no ``delta`` frame ever carries a scaffolding fragment (the strip
|
||||
happens server-side, not in the UI).
|
||||
2. ``test_terminal_scaffolding_lands_on_dedicated_error_app_stays_usable``
|
||||
— the terminal case: the existing error state renders with the
|
||||
dedicated copy ("The model returned a malformed reply — please try
|
||||
again."), no raw tokens in the DOM, no answer bubble, no ``done``
|
||||
and NO ``query_log`` row (the existing terminal-error semantics) —
|
||||
and the app stays usable: a follow-up plain question in the same
|
||||
session gets a normal deflected answer and the banner clears.
|
||||
3. ``test_plain_turn_never_recovers_and_streams_byte_clean`` — no
|
||||
false positive: a plain deflected question streams its
|
||||
first-request answer byte-clean (the concatenated delta text is
|
||||
EXACTLY the mock's deterministic deflection answer — not the
|
||||
recovery's), with no error state and no recovery request visible
|
||||
(the turn settles on the first request).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import QueryLog
|
||||
from tests.e2e.mock_llm import SCAFFOLD_RECOVERY_ANSWER, compose_answer
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Questions + the mock's deterministic expectations
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Carries ``SCAFFOLD_TRIGGER`` (and nothing else — the module-level
|
||||
#: asserts below pin the trigger exclusions).
|
||||
RECOVERY_Q = "Please emit raw tool markup in your reply to this."
|
||||
|
||||
#: Carries ``SCAFFOLD_ALWAYS_TRIGGER`` — the scaffolding-only span on
|
||||
#: EVERY request, recovery included (the terminal path).
|
||||
TERMINAL_Q = "Please always emit raw tool markup in every reply."
|
||||
|
||||
#: A plain question (the phase-67 deflection shape): no trigger at all.
|
||||
PLAIN_Q = "How do I bake sourdough bread?"
|
||||
|
||||
for _q in (RECOVERY_Q, TERMINAL_Q, PLAIN_Q):
|
||||
for _other in (
|
||||
"read two documents",
|
||||
"search your documents",
|
||||
"write a long answer",
|
||||
"think in paragraphs",
|
||||
"think out loud",
|
||||
"show the end of your notes",
|
||||
"show me a table",
|
||||
"fail then answer",
|
||||
"always fail",
|
||||
"embed fail once",
|
||||
"pretend to think slowly",
|
||||
"use your tools",
|
||||
):
|
||||
assert _other not in _q.lower(), f"{_other!r} unexpectedly in {_q!r}"
|
||||
assert "emit raw tool markup" in RECOVERY_Q.lower()
|
||||
assert "always emit raw tool markup" not in RECOVERY_Q.lower()
|
||||
assert "always emit raw tool markup" in TERMINAL_Q.lower()
|
||||
assert "emit raw tool markup" not in PLAIN_Q.lower()
|
||||
|
||||
#: The terminal malformed-reply copy (``app.api.chat`` — the phase-71
|
||||
#: dedicated error frame; the mock's ALWAYS trigger is what makes the
|
||||
#: recovery come back empty).
|
||||
MALFORMED_ERROR_COPY = "The model returned a malformed reply — please try again."
|
||||
|
||||
#: The mock's deterministic deflection phrase (the phase-67 pin).
|
||||
DEFLECT_PHRASE = r"haven't done anything like that"
|
||||
|
||||
#: The raw scaffolding fragments that must NEVER reach the user — the
|
||||
#: span's tokens and the incident's argument fragment (``SCAFFOLD_SPAN``
|
||||
#: in the mock).
|
||||
RAW_TOKENS = ("tool_call_start", "tool_call_end", "[read(path=", "<|")
|
||||
|
||||
|
||||
def _expected_deflect_answer(question: str) -> str:
|
||||
"""The mock's deterministic DEFLECT_MODE answer for *question*.
|
||||
|
||||
Derived from the mock itself (``compose_answer`` on a synthetic
|
||||
body: the mode marker in the system prompt, the question as the
|
||||
user message, no tuning/KB sections — the suite truncates
|
||||
``steering_notes`` / ``kb_overview``, so the real prompt carries
|
||||
none), so the byte-clean pin can never drift from the mock.
|
||||
"""
|
||||
return compose_answer(
|
||||
{
|
||||
"messages": [
|
||||
{"role": "system", "content": "DEFLECT_MODE marker"},
|
||||
{"role": "user", "content": question},
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# DB reset (EMPTY knowledge base → every turn is deterministically
|
||||
# deflected: the LOW gate, the incident's path) + query_log reads
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _reset_db_empty() -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables): an EMPTY
|
||||
knowledge base, so the honesty gate is LOW for every question
|
||||
(no chunks → cosine 0.0 < 0.30, fts_hits 0) — the deflected path
|
||||
where the phase-71 filter + recovery live, and the prompts stay
|
||||
byte-stable regardless of leftovers from other suites."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _query_log_rows() -> list[QueryLog]:
|
||||
with SessionLocal() as db:
|
||||
return list(db.scalars(select(QueryLog)).all())
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page hooks (the SSE capture — the house pattern from
|
||||
# test_agent_document_tools.py / test_llm_retry.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_sse_hook(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _drain_frames(page: Page, terminal: str = "done") -> list[dict]:
|
||||
"""One turn's SSE frames: wait for that turn's *terminal* frame
|
||||
(``done`` — or ``error`` for the terminal case), then return EVERY
|
||||
frame captured since the last drain. The hook's background read
|
||||
appends the whole stream at once after it closes, so
|
||||
clearing-and-reading is race-free per turn: the terminal frame is
|
||||
observed only in the batch that carries the turn's full frame
|
||||
sequence."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate(
|
||||
"() => { const f = window.__sseFrames || []; "
|
||||
"window.__sseFrames = []; return f; }"
|
||||
)
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == terminal for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `{terminal}` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _delta_text(frames: list[dict]) -> str:
|
||||
"""The turn's answer text exactly as the wire carried it."""
|
||||
return "".join(f.get("text", "") for f in frames if f.get("type") == "delta")
|
||||
|
||||
|
||||
def _assert_no_raw_tokens_on_wire(frames: list[dict]) -> None:
|
||||
for frame in frames:
|
||||
if frame.get("type") != "delta":
|
||||
continue
|
||||
text = frame.get("text", "")
|
||||
for raw in RAW_TOKENS:
|
||||
assert raw not in text, f"raw scaffolding {raw!r} on the wire: {text!r}"
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered.
|
||||
|
||||
Phase 48: the label assertion carries the settle wait with an
|
||||
explicit timeout — the in-flight button is the enabled Stop control
|
||||
(never disabled), so ``to_be_enabled`` no longer blocks until the
|
||||
turn settles."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _assert_no_raw_tokens(page: Page) -> None:
|
||||
"""The user-visible contract: no raw scaffolding fragment anywhere
|
||||
in the rendered page (``document.body.innerText``)."""
|
||||
body = page.locator("body").inner_text()
|
||||
for raw in RAW_TOKENS:
|
||||
assert raw not in body, f"raw scaffolding {raw!r} leaked into the DOM"
|
||||
|
||||
|
||||
def _assert_no_error_banner(page: Page) -> None:
|
||||
"""The turn settled through the normal done path — never the red
|
||||
role=alert error banner (the KB-offline banner is a separate,
|
||||
health-driven state the db_ready fixture keeps away)."""
|
||||
banner = page.locator("#kb-banner")
|
||||
expect(banner).to_be_hidden()
|
||||
expect(banner).not_to_have_attribute("role", "alert")
|
||||
expect(banner).not_to_have_class(re.compile(r"is-error"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The recovery case: the span is stripped server-side, the ONE
|
||||
# bounded recovery answers, and no raw token ever reaches the DOM
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_recovery_strips_scaffolding_and_streams_clean_answer(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_empty()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, RECOVERY_Q)
|
||||
_wait_settled(page)
|
||||
|
||||
# The final answer bubble carries the RECOVERY's clean text — the
|
||||
# mock's deterministic recovery answer, proof the one bounded
|
||||
# recovery ran (request 2, with the correction in the system
|
||||
# prompt) and its answer is what the user saw.
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(SCAFFOLD_RECOVERY_ANSWER)
|
||||
# The deflected message state is intact — the turn took the LOW
|
||||
# path, the incident's path.
|
||||
expect(page.locator(".msg.brain").last).to_have_class(re.compile(r"is-deflected"))
|
||||
_assert_no_raw_tokens(page)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# Wire level: the strip happens SERVER-side — no `delta` frame ever
|
||||
# carries a scaffolding fragment, and the concatenated delta text is
|
||||
# EXACTLY the recovery answer (request 1's span produced zero delta
|
||||
# frames). No tool frames, no retry frames, no error frame; the
|
||||
# turn settled with `done` (deflected).
|
||||
frames = _drain_frames(page, terminal="done")
|
||||
assert [f for f in frames if f.get("type") == "delta"], (
|
||||
"the recovery answer must have streamed delta frames"
|
||||
)
|
||||
_assert_no_raw_tokens_on_wire(frames)
|
||||
assert _delta_text(frames) == SCAFFOLD_RECOVERY_ANSWER
|
||||
assert not [f for f in frames if f.get("type") == "tool"]
|
||||
assert not [f for f in frames if f.get("type") == "retry"]
|
||||
assert not [f for f in frames if f.get("type") == "error"]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is True
|
||||
|
||||
# The recovered turn settles durably: exactly one query_log row
|
||||
# (deflected — the incident's path).
|
||||
rows = _query_log_rows()
|
||||
assert len(rows) == 1, rows
|
||||
assert rows[0].question == RECOVERY_Q
|
||||
assert rows[0].deflected is True
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The terminal case: scaffolding twice → the dedicated error frame,
|
||||
# no done, no query_log row — and the app stays usable
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_terminal_scaffolding_lands_on_dedicated_error_app_stays_usable(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_empty()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, TERMINAL_Q)
|
||||
|
||||
# BOTH requests (original + the one bounded recovery) came back
|
||||
# scaffolding-only: no clean content ever streamed, so the turn
|
||||
# settles with the EXISTING terminal error state — the banner
|
||||
# (role=alert) with the DEDICATED malformed-reply copy — and the
|
||||
# send button re-enabled (the banner path settles the state
|
||||
# machine, cf. the phase-67 exhaustion test).
|
||||
expect(page.locator("#kb-banner")).to_have_attribute(
|
||||
"role", "alert", timeout=60_000
|
||||
)
|
||||
expect(page.locator("#kb-banner")).to_contain_text(MALFORMED_ERROR_COPY)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
# No answer bubble was ever rendered (every streamed frame was
|
||||
# stripped server-side) and no raw token is anywhere in the DOM.
|
||||
expect(page.locator("#messages > .msg.brain")).to_have_count(0)
|
||||
_assert_no_raw_tokens(page)
|
||||
|
||||
# Wire: the terminal error frame is LAST — no done, and NO delta
|
||||
# frame at all (both requests' content was pure scaffolding).
|
||||
frames = _drain_frames(page, terminal="error")
|
||||
assert frames[-1]["type"] == "error"
|
||||
assert MALFORMED_ERROR_COPY in frames[-1]["detail"]
|
||||
assert not [f for f in frames if f.get("type") == "done"]
|
||||
assert not [f for f in frames if f.get("type") == "delta"]
|
||||
|
||||
# Terminal semantics (byte-for-byte the existing LLMError shape):
|
||||
# the turn writes no query_log row.
|
||||
assert _query_log_rows() == []
|
||||
|
||||
# The app stays usable: a follow-up plain question (no trigger) in
|
||||
# the SAME session gets a normal streamed deflected answer, the
|
||||
# banner clears, and the wire is clean.
|
||||
_submit(page, PLAIN_Q)
|
||||
_wait_settled(page)
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(re.compile(DEFLECT_PHRASE, re.IGNORECASE))
|
||||
_assert_no_error_banner(page)
|
||||
_assert_no_raw_tokens(page)
|
||||
follow_frames = _drain_frames(page, terminal="done")
|
||||
assert next(f for f in follow_frames if f["type"] == "done")["deflected"] is True
|
||||
assert not [f for f in follow_frames if f.get("type") == "error"]
|
||||
_assert_no_raw_tokens_on_wire(follow_frames)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. No false positive: a plain turn streams its first-request answer
|
||||
# byte-clean — no strip, no recovery request, no error state
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plain_turn_never_recovers_and_streams_byte_clean(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_empty()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, PLAIN_Q)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level: the plain deflected answer streams byte-clean from
|
||||
# the FIRST request — the concatenated delta text is EXACTLY the
|
||||
# mock's deterministic deflection answer (derived from the mock
|
||||
# itself above) and NOT the recovery answer (a recovery request
|
||||
# would have streamed that text instead).
|
||||
frames = _drain_frames(page, terminal="done")
|
||||
delta_text = _delta_text(frames)
|
||||
assert delta_text == _expected_deflect_answer(PLAIN_Q)
|
||||
assert SCAFFOLD_RECOVERY_ANSWER not in delta_text
|
||||
_assert_no_raw_tokens_on_wire(frames)
|
||||
# No recovery request is visible: no error frame, no retry frame
|
||||
# (the recovery is not a phase-67 retry), and the turn settled on
|
||||
# the first request with `done`.
|
||||
assert not [f for f in frames if f.get("type") == "error"]
|
||||
assert not [f for f in frames if f.get("type") == "retry"]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is True
|
||||
|
||||
# UI level: the deflected answer rendered, no error banner, no raw
|
||||
# token in the DOM.
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(re.compile(DEFLECT_PHRASE, re.IGNORECASE))
|
||||
_assert_no_error_banner(page)
|
||||
_assert_no_raw_tokens(page)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user