fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 14s

Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).
This commit is contained in:
2026-09-04 13:11:07 -04:00
parent 7909bdb8da
commit 988ff78526
42 changed files with 2987 additions and 88 deletions
+142
View File
@@ -156,6 +156,33 @@ Implements just enough of the aipi surface:
(the trigger needs no ``<tools>`` section); no existing E2E
question or fixture file contains the phrase, so every other
suite is unaffected.
- user message containing ``list the files in this directory``
(``LS_TEACH_TRIGGER``, phase 72, teaching refusals — the
2026-09-03 incident where the harness-prior ``ls(path='.')``
misuse met the terse refusal and the model re-reasoned the same
paragraphs over and over) **and** the system prompt carries the
``<tools>`` section -> the deterministic LS-TEACHING flow,
discriminated statelessly from the messages (streaming only):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
with ``{"path": "."}`` (synthetic id ``call_0``),
``finish_reason: "tool_calls"``, no content — the incident's
misuse, deterministic;
* request 2 (a ``tool``-role result present that is NOT a
catalog listing — i.e. the teaching refusal): a ``tool_calls``
delta — ``ls`` with no arguments (id ``call_1``) — the
correction;
* request 3 (a ``tool``-role result whose first line matches the
``^\\d+ documents:`` catalog header): a deterministic content
answer — ``These are the indexed documents: <first catalog
line>`` (the ``source: X | path: Y | title: Z`` line, parsed
with the ``_CATALOG_LINE_RE`` machinery), ``finish_reason:
"stop"`` — the loop ended in ONE correction, not at the round
cap.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
phrases are disjoint substrings — the phase-71 ordering
convention); no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
- user message containing ``show me a table`` (phase 44, markdown
tables, TODO.md L6) -> the fixed table answer (``TABLE_ANSWER``):
a 3-column service table, an ``<img onerror>`` XSS probe line, and
@@ -440,6 +467,26 @@ assert _CORRECTION_MARKER in CORRECTION_INSTRUCTION, (
"mock drift: the correction marker left CORRECTION_INSTRUCTION"
)
# ---------------------------------------------------------------------------
# Phase 72 (teaching refusals — the 2026-09-03 incident's ls misuse):
# the deterministic LS-TEACH self-correction flow — see the module
# docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt — drives
#: the deterministic LS-TEACHING flow (the incident's
#: ``ls(path='.')`` misuse → the teaching refusal → the corrected
#: no-arg ``ls()`` → the catalog answer). Checked BEFORE the plain
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the phase-71
#: ordering convention); verified: no existing E2E question or fixture
#: file contains the phrase, so every other suite is unaffected.
LS_TEACH_TRIGGER = "list the files in this directory"
#: The agent's ``ls`` listing header (app.rag.agent ``_execute_tool``):
#: ``"N documents:"`` — the first line of every catalog tool result.
_CATALOG_HEADER_RE = re.compile(r"^\d+ documents:")
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
#: while the endpoint stays down: the openai SDK's default policy
#: (max_retries=2 — the app's ``LLMClient`` keeps it) re-POSTs a 500'd
@@ -547,6 +594,43 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
return docs
def _tool_results(body: dict[str, Any]) -> list[str]:
"""Every ``tool``-role result content in the messages, in order.
(Phase 72, LS-TEACH flow: the flow is discriminated statelessly
from the tool results — a catalog listing vs the teaching
refusal vs none yet.)
"""
return [
str(m.get("content") or "")
for m in _messages(body)
if m.get("role") == "tool"
]
def _first_catalog_line(body: dict[str, Any]) -> str | None:
"""The first catalog line of a catalog listing in the messages.
A catalog listing is a ``tool``-role result whose FIRST line is the
agent's ``"N documents:"`` header (``_CATALOG_HEADER_RE``); its
first ``source: X | path: Y | title: Z`` line (the
``_CATALOG_LINE_RE`` machinery) is returned. ``None`` when no
catalog listing is in the messages — e.g. while only the teaching
refusal is there (the phase-72 LS-TEACH flow's request-2 state).
An empty listing (``"0 documents:"`` with no lines) returns
``""`` — the listing is present, it is just empty.
"""
for content in _tool_results(body):
lines = content.splitlines()
if not lines or not _CATALOG_HEADER_RE.match(lines[0]):
continue
for line in lines[1:]:
if _CATALOG_LINE_RE.match(line):
return line
return ""
return None
#: One line of the agent's ``grep`` output (app.rag.agent
#: ``_execute_tool``, phase 68 — phase 70 renamed the tool, the line
#: format is unchanged): ``source/path:LINE: text``. The
@@ -718,6 +802,40 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("list", "", "")
def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-72 LS-TEACH request (see the module docstring).
* ``("misuse",)`` — ``tools`` are offered and no ``tool``-role
result is in the messages yet: the incident's misuse — ``ls``
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
"tool_calls"``, no content.
* ``("correct",)`` — a ``tool``-role result is in the messages and
it is NOT a catalog listing (the teaching refusal): the
correction — ``ls`` with no arguments (id ``call_1``).
* ``("answer", line)`` — a ``tool``-role result whose first line
is the ``"N documents:"`` catalog header: the deterministic
content answer ``These are the indexed documents: <line>`` (the
first catalog line), ``finish_reason: "stop"`` — the loop
settled in ONE correction, not at the round cap.
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), or
``tools`` are not offered and no tool results are in the
messages yet (e.g. ``agent_max_rounds=0``).
"""
if LS_TEACH_TRIGGER not in _user(body).lower():
return None
if "<tools>" not in _system(body):
return None
line = _first_catalog_line(body)
if line is not None:
return ("answer", line)
if _tool_results(body):
return ("correct",)
if not body.get("tools"):
return None
return ("misuse",)
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -1208,6 +1326,30 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 72 (teaching refusals): the deterministic LS-TEACH
# self-correction flow — checked BEFORE the plain
# TOOLS_TRIGGER flow (disjoint trigger phrases — the phase-71
# ordering convention; the trigger needs the ``<tools>``
# section, so deflected turns never hit it).
ls_teach = _ls_teach_flow(body)
if ls_teach is not None:
if ls_teach[0] == "misuse":
# The incident's misuse, deterministic: ls(path='.').
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
elif ls_teach[0] == "correct":
# The one-round correction: the no-arg full listing.
stream = _tool_call_stream("ls", {}, "call_1")
else: # "answer" — quote the first catalog line
answer = _apply_max_tokens(
f"These are the indexed documents: {ls_teach[1]}",
body.get("max_tokens"),
)
stream = _sse_stream(answer, 0.0)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
+2 -1
View File
@@ -44,7 +44,8 @@ pattern, grown to three documents):
Three documents (not two, as in phase 37) so BOTH reads land on
documents outside the seed: with a two-document corpus the second read
would be the already-in-context retrieval document and the agent would
answer "Already in your context." — a rejection, not the multi-read
answer "Already in your context — …" (the in-context refusal) — a
rejection, not the multi-read
flow this story proves.
Test → story mapping (Playwright Mapping Rule):
+472
View File
@@ -0,0 +1,472 @@
"""Phase 72 E2E (Playwright, mock-only): the ls-teaching
self-correction loop through the real UI.
Story: ``.agent/user_stories/agent-document-tools.md`` (this phase
repairs the model-facing contract the phase-70 tools reshaped — the
2026-09-03 incident: the harness-prior ``ls(path='.')`` misuse met the
terse refusal, and the model re-reasoned the same paragraphs over and
over before answering from the seed documents alone).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
(``No source named '.' — check the ls output. (…)``) → the corrected
no-arg ``ls()`` (id ``call_1``) → the deterministic
``These are the indexed documents: <first catalog line>`` answer.
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
documents of known ``source``/``path``/``title`` (catalog order =
``(source, path)``, so the first catalog line is deterministic):
* ``Homelab/aws-route53.md`` — the CATALOG-FIRST document, indexed
WITHOUT chunks (catalog-only; never in the retrieval context, so
the single-read flow's ``read`` of it is NOT deduped as already-in-
context). Its FIRST line is longer than 80 chars, so the mock's
first-80-chars quote (the single-read regression turn) stays
newline-free.
* ``Homelab/example-record-file.json`` — the retrievable document:
one chunk whose embedding is the mock's own bag-of-words vector
(the trigger question cosines well past the E2E 0.30 threshold and
FTS-matches too → grounded, the ``<tools>`` section rides along).
It is the seed context only — the single-read flow reads the
catalog-FIRST document, not the seed.
Test → phase mapping (Playwright Mapping Rule):
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
LS-TEACH turn: the turn settles (composer re-enables, ``done``
observed), the answer bubble carries the first catalog line — the
first document's ``source:`` / ``path:`` / title fields (the
catalog reached the model and landed in the answer), the UI shows
the two tool lines (``🔎 Listing documents in <code>.</code>``
then ``🔎 Listing documents``), and no error banner. Wire level:
the ``tool`` frames arrive in order — first ``ls`` with
``argument: "."``, then ``ls`` with ``argument: null`` — and there
is NO third ``tool`` frame (the loop ended in one correction, not
at the round cap).
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
session, the LS-TEACH turn settles and a follow-up question
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
with the read flow's answer (``ls`` → ``read`` on the first
catalog line's combined identity → ``Read <source/path>. <quote>``)
— the new flow did not swallow the existing trigger.
"""
from __future__ import annotations
import hashlib
import json
import re
import time
from datetime import UTC, datetime
from pathlib import Path
from playwright.sync_api import Page, expect
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document
from tests.e2e.mock_llm import (
LS_TEACH_TRIGGER,
TOOLS_TRIGGER,
embed_text,
)
REPO = Path(__file__).resolve().parents[2]
# --------------------------------------------------------------------------
# The one-source, two-document fixture (see the module docstring)
# --------------------------------------------------------------------------
SEED_SOURCE = "Homelab"
DOC1_PATH = "aws-route53.md"
DOC1_TITLE = "AWS Route 53 Notes"
DOC1_SP = f"{SEED_SOURCE}/{DOC1_PATH}"
DOC2_PATH = "example-record-file.json"
DOC2_TITLE = "Example Record File"
DOC2_SP = f"{SEED_SOURCE}/{DOC2_PATH}"
#: The FIRST catalog line (catalog order = (source, path) — DOC1 sorts
#: first): the mock's LS-TEACH answer quotes exactly this line.
FIRST_CATALOG_LINE = (
f"source: {SEED_SOURCE} | path: {DOC1_PATH} | title: {DOC1_TITLE}"
)
#: The catalog-first document (catalog order = (source, path) —
#: DOC1 sorts first): the single-read flow reads THIS document, so it
#: must NOT be the seed (a seed read dedupes to "Already in your
#: context.", which the mock's single-read flow does not model — it
#: would loop to the round cap). Indexed WITHOUT chunks: catalog-only,
#: never in the retrieval context. Its FIRST line is longer than 80
#: chars, so the mock's first-80-chars quote (the single-read
#: regression turn) stays newline-free.
DOC1_CONTENT = (
"The aws route53 hosted zone for reeselink keeps every record in "
"reseelink.json — the exact JSON shape of reeselink.json is "
"documented in the record file below.\n"
+ (
"The aws route53 hosted zone for reeselink keeps every record in "
"reseelink.json — the record file shape of reeselink.json is "
"the contract every sync job relies on.\n"
)
* 10
+ "\n## Sync job\n\n"
"A cron job pushes reeselink.json to the aws route53 hosted zone "
"every fifteen minutes; the diff is applied through the route53 api.\n"
)
assert "\n" not in DOC1_CONTENT[:80] # the quote must stay one line
#: The retrievable document (the grounded seed context, the cf.
#: test_harness_aligned_tools.py pattern): the repeated record-file
#: lines carry the trigger question's key tokens — well past the E2E
#: 0.30 cosine threshold, plus FTS hits. Its FIRST line is longer
#: than 80 chars too, so the retrieval seed context is one clean
#: line.
DOC2_CONTENT = (
"The ReeseLink hosted zone record file reeselink.json holds every "
"aws route53 record for reeselink — the note documents the exact "
"JSON shape of reeselink.json for the record file.\n"
+ (
"The aws route53 record file reeselink.json keeps every record "
"for the reeselink hosted zone — the exact JSON shape of the "
"record file is the contract every sync job relies on.\n"
)
* 10
+ "\n## Sync job\n\n"
"A cron job pushes reeselink.json to the aws route53 hosted zone "
"every fifteen minutes; the diff is applied through the route53 api.\n"
)
assert "\n" not in DOC2_CONTENT[:80] # the seed context stays one line
#: Carries ``LS_TEACH_TRIGGER`` and is on-topic (grounded — HIGH, the
#: ``<tools>`` section rides along); it carries NO other mock marker.
LS_TEACH_QUESTION = (
"List the files in this directory — what do my aws route53 notes "
"say about the reeselink.json record file?"
)
assert LS_TEACH_TRIGGER in LS_TEACH_QUESTION.lower()
for _other in (
"use your tools",
"read two documents",
"search your documents",
"emit raw tool markup",
"always emit raw tool markup",
"show me a table",
"think in paragraphs",
"think out loud then hesitate",
"think out loud",
"show the end of your notes",
"write a long answer",
"fail then answer",
"always fail",
"embed fail once",
"pretend to think slowly",
):
assert _other not in LS_TEACH_QUESTION.lower(), _other
#: Carries ``TOOLS_TRIGGER`` (the single-read flow) and nothing else —
#: the no-regression follow-up question in the same session.
READ_QUESTION = (
"Use your tools: what is the exact JSON shape of reeselink.json "
"for my aws route53 hosted zone?"
)
assert TOOLS_TRIGGER in READ_QUESTION.lower()
for _other in (
LS_TEACH_TRIGGER,
"read two documents",
"search your documents",
"emit raw tool markup",
"always emit raw tool markup",
"show me a table",
"think in paragraphs",
"think out loud then hesitate",
"think out loud",
"show the end of your notes",
"write a long answer",
"fail then answer",
"always fail",
"embed fail once",
"pretend to think slowly",
):
assert _other not in READ_QUESTION.lower(), _other
#: The mock's single-read answer (the read document reached the model
#: and landed in the answer) — DOC1 is the first catalog line, so the
#: flow reads ``Homelab/aws-route53.md`` and quotes its first 80 chars.
READ_ANSWER_PREFIX = f"Read {DOC1_SP}."
READ_ANSWER_QUOTE = DOC1_CONTENT[:80]
def _seed_fixture(db: Session) -> None:
"""The one-source, two-document fixture (see the module docstring).
DOC1 (catalog-first) is indexed WITHOUT chunks; DOC2 carries the
single chunk (the mock's own embedding → the trigger question
cosines well past the E2E 0.30 threshold and FTS-matches too →
grounded). DOC2 is the seed context only — the single-read flow
reads the catalog-FIRST document (DOC1), which is not in context.
"""
db.add(
Document(
source=SEED_SOURCE,
path=DOC1_PATH,
full_path=f"/tmp/{DOC1_PATH}",
title=DOC1_TITLE,
content=DOC1_CONTENT,
content_hash=hashlib.sha256(DOC1_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
)
doc2 = Document(
source=SEED_SOURCE,
path=DOC2_PATH,
full_path=f"/tmp/{DOC2_PATH}",
title=DOC2_TITLE,
content=DOC2_CONTENT,
content_hash=hashlib.sha256(DOC2_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
db.add(doc2)
db.flush()
# One chunk carrying the mock's own embedding → genuine token
# overlap between the trigger question and DOC2 (the only
# retrievable document).
db.add(
Chunk(
document_id=doc2.id,
position=0,
content=DOC2_CONTENT,
embedding=embed_text(DOC2_CONTENT),
)
)
def _reset_db_fixture() -> None:
"""Truncate the KB (plus the prompt-shaping tables), then seed the
one-source, two-document fixture. ``steering_notes`` /
``kb_overview`` are truncated too, so the HIGH prompt is exactly
``<relevance>`` + ``<documents>`` + ``<tools>`` — byte-stable
prompts, byte-stable answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
)
db.commit()
_seed_fixture(db)
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the house pattern — cf. test_harness_aligned_tools.py)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_sse_hook(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _drain_frames(page: Page) -> list[dict]:
"""One turn's SSE frames: wait for that turn's ``done`` frame, then
return EVERY frame captured since the last drain (the hook's
background read appends the whole stream at once after it closes, so
clearing-and-reading is race-free per turn)."""
deadline = time.monotonic() + 10.0
while True:
raw = page.evaluate(
"() => { const f = window.__sseFrames || []; "
"window.__sseFrames = []; return f; }"
)
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered.
Phase 48: the label assertion carries the settle wait with an
explicit timeout — the in-flight button is the enabled Stop control
(never disabled), so ``to_be_enabled`` no longer blocks until the
turn settles."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _assert_no_error_banner(page: Page) -> None:
"""The turn settled through the normal done path — never the red
role=alert error banner (the KB-offline banner is a separate,
health-driven state the db_ready fixture keeps away)."""
banner = page.locator("#kb-banner")
expect(banner).to_be_hidden()
expect(banner).not_to_have_attribute("role", "alert")
expect(banner).not_to_have_class(re.compile(r"is-error"))
# --------------------------------------------------------------------------
# 1. The grounded LS-TEACH turn: the incident's ls(path='.') misuse →
# the teaching refusal → the corrected no-arg ls() → the catalog
# answer — the loop settles in ONE correction (two tool rounds),
# pinned on the SSE wire
# --------------------------------------------------------------------------
def test_ls_misuse_self_corrects_to_noarg_listing(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db_fixture()
page.goto(app_url)
_install_sse_hook(page)
_submit(page, LS_TEACH_QUESTION)
_wait_settled(page)
# Self-correction: the answer quotes the FIRST catalog line — the
# first document's source: / path: / title fields reached the model
# and landed in the answer (the catalog round settled the turn).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text("These are the indexed documents:")
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
_assert_no_error_banner(page)
# The UI shows the two tool lines in order: the scoped misuse
# (🔎 Listing documents in <code>.</code>) then the corrected
# unscoped listing (🔎 Listing documents — no <code>).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text("Listing documents in")
expect(lines.nth(0).locator("code")).to_have_text(".")
expect(lines.nth(1)).to_contain_text("Listing documents")
expect(lines.nth(1).locator("code")).to_have_count(0)
# Two rounds on the wire: the tool frames arrive in order — first
# ls with argument "." (the incident's misuse), then ls with
# argument null (the correction) — and there is NO third tool
# frame: the loop ended in one correction, not at the round cap.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": "."},
{"type": "tool", "name": "ls", "argument": None},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
assert not [f for f in frames if f.get("type") == "error"]
# --------------------------------------------------------------------------
# 2. No regression to the plain flow — the SAME session: after the
# LS-TEACH turn, the TOOLS_TRIGGER follow-up (the single-read flow)
# still settles with the read flow's answer
# --------------------------------------------------------------------------
def test_plain_tool_flow_not_swallowed_by_new_trigger(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db_fixture()
page.goto(app_url)
_install_sse_hook(page)
# Turn 1 — the LS-TEACH flow (the incident's misuse → the
# correction → the catalog answer).
_submit(page, LS_TEACH_QUESTION)
_wait_settled(page)
teach_frames = _drain_frames(page)
assert _tool_frames(teach_frames) == [
{"type": "tool", "name": "ls", "argument": "."},
{"type": "tool", "name": "ls", "argument": None},
]
expect(
page.locator(".msg.brain .bubble").last
).to_contain_text(FIRST_CATALOG_LINE)
# Turn 2 — the SAME session: the single-read flow on
# TOOLS_TRIGGER. The new flow must not have swallowed the existing
# trigger: the follow-up settles with the read flow's answer.
_submit(page, READ_QUESTION)
_wait_settled(page)
second_msg = page.locator(".msg.brain").last
# The UI shows the single-read flow's two lines: the unscoped ls
# then the read of the first catalog line's COMBINED identity.
lines = second_msg.locator(".tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
# The answer quotes the read document (the mock's deterministic
# echo: "Read <source/path>. <first 80 chars>").
bubble = second_msg.locator(".bubble").last
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
_assert_no_error_banner(page)
# Wire level for the follow-up: ls (null) → read (the combined
# identity) — the single-read flow, unchanged.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": DOC1_SP},
]
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
assert not [f for f in frames if f.get("type") == "error"]
+132 -7
View File
@@ -185,22 +185,48 @@ def _settings(**kwargs: Any) -> Settings:
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
class ScriptedToolCallsLLM:
"""N scripted tool-call rounds (one ``ToolCallPiece`` each), then one
canned answer; records every request's messages and tools (the
phase-72 task-02 two-round self-correction cases: the refusal round,
then the corrected call)."""
def __init__(self, calls: list[ToolCallPiece]) -> None:
self.calls = calls
self.requests: list[
tuple[list[dict[str, Any]], list[dict[str, Any]] | None]
] = []
async def chat_stream(
self,
messages: list[dict[str, str]],
tools: list[dict[str, Any]] | None = None,
scaffolding: ScaffoldingFilter | None = None,
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
self.requests.append((deepcopy(messages), deepcopy(tools)))
index = len(self.requests) - 1
if index < len(self.calls):
yield self.calls[index]
else:
yield StreamPiece("content", "ans")
def _run_call(
db: Session, name: str, arguments: dict[str, Any]
) -> tuple[AgentHolder, ScriptedToolLLM]:
"""Drive one scripted tool call through ``run_agent``."""
holder = AgentHolder()
llm = ScriptedToolLLM(ToolCallPiece(id="call_1", name=name, arguments=arguments))
asyncio.run(_consume(llm, db, holder))
asyncio.run(_consume(cast("LLMClient", llm), db, holder))
return holder, llm
async def _consume(
llm: ScriptedToolLLM, db: Session, holder: AgentHolder
llm: LLMClient, db: Session, holder: AgentHolder
) -> list[StreamPiece | ToolCallPiece | RetryPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
async for piece in run_agent(
cast("LLMClient", llm),
llm,
db,
system_prompt="SYSTEM_PROMPT",
user_message="QUESTION",
@@ -233,17 +259,39 @@ def test_ls_scoped_to_registered_source_through_run_agent(kb, src, db) -> None:
def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
"""Phase 72: the no-source refusal now carries the teaching
parenthetical — the prefix byte-identical to the pre-phase-72 line;
still not counted."""
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Ghost"})
assert (
llm.requests[1][0][3]["content"] == "No source named 'Ghost' — check the ls output."
llm.requests[1][0][3]["content"]
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="Ghost")
)
assert holder.tool_calls == 0 and holder.read_docs == []
def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> None:
"""Phase 72: a ``/``-containing ``path`` is a document path, not a
source name — the ``LS_PATH_NOT_A_SOURCE`` teaching line (no
registry lookup needed), not counted, the tools stay offered on the
next request."""
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "app/rag/importer.py"})
assert (
llm.requests[1][0][3]["content"]
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
)
assert holder.tool_calls == 0 and holder.read_docs == []
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
# ---------- read (the canonical combined source/path form) ----------
@@ -280,9 +328,10 @@ def test_read_bare_source_name_refused_through_run_agent(kb, db) -> None:
def test_read_unknown_combined_path_refused_through_run_agent(kb, db) -> None:
"""A combined identity that matches nothing gets the no-document
refusal (the argument echoed as passed — the model sees its own
form)."""
"""A combined identity that matches NOTHING — not a document and not
any indexed document's ``path`` (zero candidates) — gets today's
no-document refusal byte-identical (the argument echoed as passed —
the model sees its own form)."""
_doc(db, "Alpha", "x.md", "X", "X-CONTENT")
db.commit()
@@ -295,6 +344,79 @@ def test_read_unknown_combined_path_refused_through_run_agent(kb, db) -> None:
assert holder.tool_calls == 0 and holder.read_docs == []
def test_read_bare_path_single_source_suggestion_then_corrected_read(kb, db) -> None:
"""Phase 72, task 02: a bare path under ONE source (exact ``path``
match, the source prefix missing) → the single-identity suggestion
(a refusal — not counted); the scripted corrected call (round 2, the
suggested combined identity) then succeeds against real Postgres —
the two-round self-correction."""
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
db.commit()
llm = ScriptedToolCallsLLM(
[
ToolCallPiece(
id="call_1", name="read", arguments={"path": "deep/nested/doc.md"}
),
ToolCallPiece(
id="call_2",
name="read",
arguments={"path": "Alpha/deep/nested/doc.md"},
),
]
)
holder = AgentHolder()
asyncio.run(_consume(cast("LLMClient", llm), db, holder))
# Round 1: the bare path resolves to no combined identity, but it IS
# the indexed document's path — the refusal names the one combined
# identity to use (not counted, the tools stay offered).
assert llm.requests[1][0][3]["content"] == (
"No document at 'deep/nested/doc.md' — "
"did you mean 'Alpha/deep/nested/doc.md'?"
)
assert llm.requests[1][1] == AGENT_TOOLS
# Round 2: the corrected combined identity succeeds — the full
# content, the holder records the row, and it counts.
assert llm.requests[2][0][5]["content"] == (
"Document Alpha/deep/nested/doc.md:\nFULL-TEXT"
)
assert llm.requests[2][1] == AGENT_TOOLS
assert holder.read_docs == [created]
assert holder.tool_calls == 1 # only the corrected read executed
def test_read_bare_path_two_sources_one_of_suggestion_then_corrected_read(
kb, db,
) -> None:
"""Phase 72, task 02: the same bare path under TWO sources → the
``one of`` line (up to ``SUGGESTION_LIMIT`` identities, catalog
order — Alpha before Beta); the scripted corrected call (round 2,
the first suggested identity) then succeeds."""
a = _doc(db, "Alpha", "shared/x.md", "Alpha X", "A-TEXT")
_doc(db, "Beta", "shared/x.md", "Beta X", "B-TEXT")
db.commit()
llm = ScriptedToolCallsLLM(
[
ToolCallPiece(id="call_1", name="read", arguments={"path": "shared/x.md"}),
ToolCallPiece(
id="call_2", name="read", arguments={"path": "Alpha/shared/x.md"}
),
]
)
holder = AgentHolder()
asyncio.run(_consume(cast("LLMClient", llm), db, holder))
assert llm.requests[1][0][3]["content"] == (
"No document at 'shared/x.md' — did you mean one of: "
"'Alpha/shared/x.md', 'Beta/shared/x.md'?"
)
assert llm.requests[2][0][5]["content"] == "Document Alpha/shared/x.md:\nA-TEXT"
assert holder.read_docs == [a]
assert holder.tool_calls == 1 # only the corrected read executed
# ---------- grep (the phase-68 A5 contract under the new name) ----------
@@ -345,6 +467,9 @@ def test_grep_scoped_through_run_agent(kb, db) -> None:
def test_grep_scoped_missing_doc_refused_through_run_agent(kb, db) -> None:
"""A scoped ``grep`` miss that matches no indexed document's ``path``
(zero candidates) keeps today's line byte-identical — a refusal,
not counted."""
_doc(db, "Alpha", "a/one.md", "One", "nothing")
db.commit()
+492 -23
View File
@@ -141,21 +141,49 @@ def test_agent_tools_names_and_parameters() -> None:
assert not set(by_name) & {"list_documents", "read_document", "search_documents"}
assert all(t["type"] == "function" for t in AGENT_TOOLS)
ls = by_name["ls"]["function"]
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
# clause (the harness prior batches calls; the loop executes one
# per round — the extras count as unexecuted in the gate).
assert ls["description"] == (
"List the indexed documents as `source: X | path: Y | title: Z` lines."
"List the indexed documents as `source: X | path: Y | "
"title: Z` lines. Call one tool at a time — wait for "
"this result before your next call."
)
ls_params = ls["parameters"]
assert ls_params["type"] == "object"
assert ls_params["required"] == [] # path is optional
assert set(ls_params["properties"]) == {"path"}
assert ls_params["properties"]["path"]["type"] == "string"
# Phase 72: the description states the contract up front — the
# 'path' argument is a source name, not a file or directory path.
# Task 05 (live gate iteration 5): the cross-tool contrast clause
# (ls is the ONLY tool whose path is a source name — the model
# kept transferring that scope to grep's document identity).
assert ls_params["properties"]["path"]["description"] == (
"Source name to list one source's documents (e.g. 'homelab'); "
"omit to list every document."
"Source name to list one source's documents (e.g. 'homelab') — "
"a source name, not a file or directory path; omit to list "
"every document. This is the only tool "
"whose `path` is a source name — for "
"`read` and `grep` it must be a document's "
"combined `source/path`."
)
read = by_name["read"]["function"]
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
# the do-not-read rule is FRONT-LOADED — the controlled gate's
# telemetry showed the `lite` model obeying the user's "open it /
# read it" and reading seed-context documents the <documents>
# section already carries (every refusal of a 12-call run was
# ALREADY_IN_CONTEXT); the rule now leads the description instead
# of sitting mid-paragraph, and the tool is framed as "only for
# documents NOT already in <documents>".
assert read["description"] == (
"Add the full content of one indexed document to your context."
"Do not call this tool for a document already shown in "
"the <documents> section, even when the user asks you to "
"open or read it — its full text is already in your "
"prompt; answer directly from it. Use it only to add a "
"document NOT already in <documents> to your context, "
"by its combined `source/path` string. Call one tool at "
"a time — wait for this result before your next call."
)
read_params = read["parameters"]
assert read_params["type"] == "object"
@@ -163,18 +191,35 @@ def test_agent_tools_names_and_parameters() -> None:
assert set(read_params["properties"]) == {"path"}
assert read_params["properties"]["path"]["type"] == "string"
# The combined source/path string is the canonical document identity
# (phase 70) — the description pins it with a worked example.
# (phase 70) — the description pins it with a worked example. Phase
# 72 (task 02): the bare-path contract is stated up front; task 05
# (live gate iteration 1): the do-not-re-read clause (the dedupe
# refusal's prevention at the prompt).
assert read_params["properties"]["path"]["description"] == (
"The document to add to your context, as the combined "
"`source/path` string exactly as shown in the `ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md')."
"'homelab/active/container_caddy/caddy.md'). A bare document "
"path (without the source name) will not resolve. Only pass a "
"document NOT already shown in the <documents> section — it is "
"already in your context; do not re-read it."
)
grep = by_name["grep"]["function"]
# Task 05 (live gate iterations 2-6, refined in the 2026-09-03
# re-run): the pattern-only-is-the-knowledge-base-search clause
# ("pass ONLY `pattern`") + the source-name-is-not-a-document
# clause (the model kept scoping grep with an ls-style source name
# — the 2026-09-03 incident loop shape, but on grep) plus the
# one-call-at-a-time discipline clause.
assert grep["description"] == (
"Search the indexed documents for an exact string "
"(case-insensitive) and return up to 20 matching lines as "
"`source/path:line: text` — a locator, not a context-adder: "
"read the winner with `read`."
"(case-insensitive) and return up to 20 matching lines "
"as `source/path:line: text` — a locator, not a "
"context-adder: read the winner with `read`. For a "
"normal search pass ONLY `pattern` — it searches every "
"document and that is how you search the knowledge "
"base; never pass a source name as `path` (a source "
"name is not a document). Call one tool at a time — "
"wait for this result before your next call."
)
grep_params = grep["parameters"]
assert grep_params["type"] == "object"
@@ -184,9 +229,25 @@ def test_agent_tools_names_and_parameters() -> None:
assert grep_params["properties"]["pattern"]["description"] == (
"The exact text to search for (a plain substring, not a regex)"
)
# Phase 72 (task 02): the bare-path contract is stated up front;
# task 05 (live gate iterations 1-8): the one-known-document clause
# with a worked combined-identity example and the source-name ban
# (the model kept scoping grep with an ls-style source name — the
# incident loop shape, but on grep).
# Iteration 8 drops the standalone 'homelab' from this negative
# example — the gate's live telemetry showed the model emitting
# exactly that value, and naming it beside the parameter risks
# priming it (the negative-example effect). The 2026-09-03 re-run
# makes the rarity explicit ("Rarely needed") and re-states the
# pattern-only normal search.
assert grep_params["properties"]["path"]["description"] == (
"Limit the search to one document, as a combined `source/path` "
"string from the `ls` output (omit to search every document)."
"Rarely needed — only for re-searching one "
"document you already know: that document's "
"combined `source/path` identity (e.g. "
"'homelab/ansible/inventory.yaml'). Never a "
"source name. A bare document path (without "
"the source name) will not resolve. Omit it "
"for a normal search (pass only `pattern`)."
)
@@ -198,11 +259,44 @@ def test_agent_tools_order_is_ls_read_grep() -> None:
def test_refusal_constants_are_harness_aligned() -> None:
"""The updated module-level refusal lines (the names moved to the
harness surface; ALREADY_IN_CONTEXT / UNKNOWN_TOOL unchanged)."""
assert agent.ALREADY_IN_CONTEXT == "Already in your context."
harness surface). The ALREADY_IN_CONTEXT line is a phase-72,
task 05 gate-iteration teaching (live telemetry: the model
repeated the terse phase-37 line) — same refusal behavior, the
copy names the correct action."""
assert agent.ALREADY_IN_CONTEXT == (
"Already in your context — the full text is already in your "
"prompt. Do not call read on it again; answer from that text."
)
assert agent.UNKNOWN_TOOL == "Unknown tool."
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
# Phase 72: the ls teaching-refusal templates, pinned byte-for-byte
# (task 01 — the read/grep suggestion templates below, task 02).
assert agent.LS_PATH_NOT_A_SOURCE == (
"'{path}' looks like a document path, not a source name. The "
"'path' argument of ls filters by source name (e.g. 'homelab') — "
"omit it to list every document, or read a document by its "
"combined 'source/path' string."
)
# The pre-phase-72 no-source line is the byte-identical prefix of
# the extended line — only the teaching parenthetical was appended.
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
"No source named '{scope}' — check the ls output."
)
assert agent.NO_SOURCE_NOT_A_DIRECTORY == (
"No source named '{scope}' — check the ls output. (The 'path' "
"argument is a source name, not a directory — omit it to list "
"every document.)"
)
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
# templates, pinned byte-for-byte, and the suggestion cap.
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
"No document at '{arg}' — did you mean '{source}/{path}'?"
)
assert agent.NO_DOCUMENT_DID_YOU_MEAN_MANY == (
"No document at '{arg}' — did you mean one of: {candidates}?"
)
assert agent.SUGGESTION_LIMIT == 3
# ---------- list_source_names (the scoped ls registry join) ----------
@@ -467,8 +561,9 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> None:
"""A ``path`` matching no source name is a refusal — not counted, the
round cap bounds its repetition."""
"""A ``path`` without ``/`` matching no source name is a refusal —
the extended line with the teaching parenthetical (phase 72), not
counted, the round cap bounds its repetition."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
@@ -479,7 +574,65 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0 # a refusal counts in nothing
assert (
llm.requests[1][0][3]["content"] == "No source named 'Ghost' — check the ls output."
llm.requests[1][0][3]["content"]
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="Ghost")
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_path_like_scope_gets_document_path_teaching_refusal(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 72: a stripped scope containing ``/`` looks like a document
path (the incident's ``ls(path='app/rag/importer.py')``) — a source
name is a directory basename and can never contain one, so this gets
the ``LS_PATH_NOT_A_SOURCE`` teaching line with the argument echoed;
no registry lookup, counts in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for a path-like scope")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="ls",
arguments={"path": "app/rag/importer.py"},
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0 # a refusal counts in nothing
assert (
llm.requests[1][0][3]["content"]
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
``/``, no matching source) gets the extended no-source refusal with
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={"path": "."})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0 # a refusal counts in nothing
assert (
llm.requests[1][0][3]["content"]
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope=".")
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
@@ -529,14 +682,19 @@ def test_read_combined_path_resolves_and_returns_full_content(
def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatch) -> None:
"""A bare source name (no '/') can never be a document — the
no-document refusal (the argument echoed as passed), no DB lookup,
nothing counted."""
no-document refusal (the argument echoed as passed), no DB lookup
(NOT even the phase-72 candidate lookup — ``all_documents`` must
not run either), nothing counted."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Homelab", "a.md", "A")])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError("find_document must not run for a bare source name")
raise AssertionError(
"no DB lookup (find_document or all_documents) for a bare "
"source name"
)
monkeypatch.setattr(agent, "find_document", _boom)
monkeypatch.setattr(agent, "all_documents", _boom)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "Homelab"})],
@@ -553,10 +711,12 @@ def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatc
def test_read_unknown_path_refused_echoing_argument(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""An unknown combined identity → the refusal echoing the argument as
passed (the model sees its own form) — the old split-teaching refusal
is gone (phase 70)."""
"""An unknown combined identity that matches NO indexed document's
``path`` (zero candidates — the phase-72 lookup runs, finds nothing)
→ today's refusal echoing the argument as passed, byte-identical —
the old split-teaching refusal is gone (phase 70)."""
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/ghost.md"})],
@@ -570,6 +730,270 @@ def test_read_unknown_path_refused_echoing_argument(
assert llm.requests[1][1] == AGENT_TOOLS # tools stay offered (cap bounds)
# ---------- read/grep: the "did you mean …?" suggestions (phase 72, task 02) ----------
def test_find_path_candidates_exact_suffix_catalog_order(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The pure catalog lookup (monkeypatched ``all_documents`` — one
bulk query per call): ``path`` == arg (exact) or a ``/arg`` suffix —
catalog order, case-sensitive, as ``(source, path, title)`` triples;
a plain substring is NOT a suffix; the result is uncapped (the
:data:`~app.rag.agent.SUGGESTION_LIMIT` cap lives in the refusal).
"""
docs = [
_doc("A", "x.md", "Ax", "A"),
_doc("A", "shared/x.md", "As", "AS"),
_doc("B", "shared/x.md", "Bs", "BS"),
_doc("C", "deep/shared/x.md", "Cs", "CS"),
_doc("D", "X.md", "Dx", "D"), # case-sensitive: not 'x.md'
_doc("E", "nosuffixx.md", "Ex", "E"), # substring, not a /suffix
]
calls: list[int] = []
def _all(db: Any) -> list[Document]:
calls.append(1)
return docs
monkeypatch.setattr(agent, "all_documents", _all)
db = cast("Session", object())
# Exact bare path ('shared/x.md') plus the deeper suffix
# ('deep/shared/x.md' ends with '/shared/x.md') — catalog order.
assert agent.find_path_candidates(db, "shared/x.md") == [
("A", "shared/x.md", "As"),
("B", "shared/x.md", "Bs"),
("C", "deep/shared/x.md", "Cs"),
]
# 'x.md' equals A's path exactly AND suffix-matches the rest — all
# four, catalog order (uncapped: the cap is the refusal's).
assert agent.find_path_candidates(db, "x.md") == [
("A", "x.md", "Ax"),
("A", "shared/x.md", "As"),
("B", "shared/x.md", "Bs"),
("C", "deep/shared/x.md", "Cs"),
]
# Case-sensitive file paths: 'X.md' matches ONLY D's identically-
# cased path (never the lowercase 'x.md' ones), and the plain
# substring inside 'nosuffixx.md' is not a suffix.
assert agent.find_path_candidates(db, "X.md") == [("D", "X.md", "Dx")]
# One bulk query per call (at most one).
assert len(calls) == 3
def test_read_bare_path_exact_match_gets_did_you_mean(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The incident shape: an unresolved ``read`` argument containing
``/`` that EXACTLY matches one indexed document's ``path`` (the bare
path missing the source prefix — the harness prior) gets the
``NO_DOCUMENT_DID_YOU_MEAN`` line naming the combined identity —
still a refusal: ``read_docs`` empty, nothing counted, tools stay
offered."""
doc = _doc("Homelab", "active/container_caddy/caddy.md", "Caddy", "CADDY-CONTENT")
def _find(db: Any, source: str, path: str) -> Document | None:
return (
doc
if (source, path) == ("Homelab", "active/container_caddy/caddy.md")
else None
)
monkeypatch.setattr(agent, "find_document", _find)
monkeypatch.setattr(agent, "all_documents", lambda db: [doc])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="read",
arguments={"path": "active/container_caddy/caddy.md"},
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
assert llm.requests[1][0][3]["content"] == (
agent.NO_DOCUMENT_DID_YOU_MEAN.format(
arg="active/container_caddy/caddy.md",
source="Homelab",
path="active/container_caddy/caddy.md",
)
)
# The rendered line, pinned byte-for-byte.
assert llm.requests[1][0][3]["content"] == (
"No document at 'active/container_caddy/caddy.md' — "
"did you mean 'Homelab/active/container_caddy/caddy.md'?"
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_read_bare_path_suffix_match_gets_did_you_mean(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The suffix form of the same teaching: a path-like argument that
matches a deeper indexed path (``active/container_caddy/caddy.md``
ends with ``/container_caddy/caddy.md``) names the same combined
identity."""
doc = _doc("Homelab", "active/container_caddy/caddy.md", "Caddy", "CADDY-CONTENT")
def _find(db: Any, source: str, path: str) -> Document | None:
return (
doc
if (source, path) == ("Homelab", "active/container_caddy/caddy.md")
else None
)
monkeypatch.setattr(agent, "find_document", _find)
monkeypatch.setattr(agent, "all_documents", lambda db: [doc])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="read",
arguments={"path": "container_caddy/caddy.md"},
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
assert llm.requests[1][0][3]["content"] == (
"No document at 'container_caddy/caddy.md' — "
"did you mean 'Homelab/active/container_caddy/caddy.md'?"
)
def test_read_bare_path_two_sources_gets_one_of_suggestion(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The same bare path under two sources: the ``one of`` line — up to
``SUGGESTION_LIMIT`` combined identities, each single-quoted, joined
with ``, `` in catalog order (A before B)."""
a = _doc("A", "shared/x.md", "Ax", "A")
b = _doc("B", "shared/x.md", "Bx", "B")
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [a, b])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(id="call_1", name="read", arguments={"path": "shared/x.md"})
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
assert llm.requests[1][0][3]["content"] == (
agent.NO_DOCUMENT_DID_YOU_MEAN_MANY.format(
arg="shared/x.md", candidates="'A/shared/x.md', 'B/shared/x.md'"
)
)
# The rendered line, pinned byte-for-byte.
assert llm.requests[1][0][3]["content"] == (
"No document at 'shared/x.md' — did you mean one of: "
"'A/shared/x.md', 'B/shared/x.md'?"
)
def test_read_bare_path_four_sources_capped_at_three_suggestions(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Four sources sharing the same path: exactly ``SUGGESTION_LIMIT``
(3) identities are suggested — catalog order, the fourth dropped."""
docs = [_doc(s, "shared/x.md", f"{s}x", s) for s in ("A", "B", "C", "D")]
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: list(docs))
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(id="call_1", name="read", arguments={"path": "shared/x.md"})
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"No document at 'shared/x.md' — did you mean one of: "
"'A/shared/x.md', 'B/shared/x.md', 'C/shared/x.md'?"
)
assert "'D/shared/x.md'" not in llm.requests[1][0][3]["content"]
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
def test_read_bare_filename_without_slash_keeps_no_db_refusal(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The gate is the ``/`` in the argument: a bare FILENAME (no ``/``
— e.g. ``caddy.md``) is a bare name for the lookup — today's
refusal byte-identical, and NO ``find_document`` / ``all_documents``
call (the same no-DB-lookup invariant as a bare source name)."""
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError("no DB lookup for a bare (no '/') argument")
monkeypatch.setattr(agent, "find_document", _boom)
monkeypatch.setattr(agent, "all_documents", _boom)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "caddy.md"})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [] and holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"No document at 'caddy.md' — check the ls output."
)
assert llm.requests[1][1] == AGENT_TOOLS
def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Dedupe precedence: the in-context dedupe fires on the SPLIT pair
of the argument — the bare path of an in-context document
(``read('app/rag/importer.py')`` with ``sample/app/rag/importer.py``
seeded) is NOT that pair, so it is not a dedupe: it gets the
suggestion line naming the combined identity, and the model's next,
correctly-formed call is then deduped as ALREADY_IN_CONTEXT."""
seed = [_doc("sample", "app/rag/importer.py", "Importer", "IMPORTER")]
def _find(db: Any, source: str, path: str) -> Document | None:
return seed[0] if (source, path) == ("sample", "app/rag/importer.py") else None
monkeypatch.setattr(agent, "find_document", _find)
monkeypatch.setattr(agent, "all_documents", lambda db: list(seed))
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="read",
arguments={"path": "app/rag/importer.py"},
)
],
[
# Round 2: the corrected call (the suggested combined
# identity) — the seed document is already in context, so it
# dedupes.
ToolCallPiece(
id="call_2",
name="read",
arguments={"path": "sample/app/rag/importer.py"},
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
assert holder.read_docs == [] and holder.tool_calls == 0 # both refused
assert llm.requests[1][0][3]["content"] == (
"No document at 'app/rag/importer.py' — "
"did you mean 'sample/app/rag/importer.py'?"
)
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
@pytest.mark.parametrize(
("arguments", "label"),
[
@@ -806,7 +1230,12 @@ def test_grep_scoped_combined_path_with_nested_path(
def test_grep_scoped_missing_document_refused(monkeypatch: pytest.MonkeyPatch) -> None:
"""A scoped ``grep`` miss that matches NO indexed document's ``path``
(zero candidates — the phase-72 lookup runs, finds nothing) keeps
today's line byte-identical: a refusal (not counted), tools stay
offered."""
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -826,6 +1255,44 @@ def test_grep_scoped_missing_document_refused(monkeypatch: pytest.MonkeyPatch) -
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_grep_scoped_missing_path_like_doc_gets_did_you_mean(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The same teaching on the scoped ``grep`` miss: an unresolved
path-like scope that matches an indexed document's path gets the
``NO_DOCUMENT_DID_YOU_MEAN`` suggestion line (a refusal — not
counted, no context added, tools stay offered); the whole-KB grep is
untouched (no ``path`` argument → no scoped resolution at all).
"""
doc = _doc("Homelab", "active/container_caddy/caddy.md", "Caddy", "CADDY-CONTENT")
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [doc])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="grep",
arguments={
"pattern": "needle",
"path": "active/container_caddy/caddy.md",
},
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
agent.NO_DOCUMENT_DID_YOU_MEAN.format(
arg="active/container_caddy/caddy.md",
source="Homelab",
path="active/container_caddy/caddy.md",
)
)
assert holder.tool_calls == 0 and holder.read_docs == [] # a refusal
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_grep_scoped_bare_source_name_refused_without_db(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -1080,8 +1547,10 @@ def test_zero_max_rounds_is_one_request_without_tools() -> None:
def test_rejected_read_spam_runs_to_round_cap(monkeypatch: pytest.MonkeyPatch) -> None:
"""Every call rejected (unknown document — "No document at …"):
rejections no longer end the loop early via budgets — the round cap
bounds them and forces the final no-tools answer."""
bounds them and forces the final no-tools answer. Zero candidates
(empty catalog) → the pre-phase-72 line, byte-identical."""
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/ghost.md"})],
+73 -6
View File
@@ -10,6 +10,12 @@ And the phase-71 deflection plain-text line (owner-permitted
2026-09-03): the LOW prompt = pre-phase text + exactly the one new
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
the line never leaks into the HIGH prompt.
And the phase-72 ``<tools>`` copy: the document-identity contract is
stated up front (the ``ls`` source-name scope, the combined
``source/path`` identity for ``read``/``grep``) — the same contract
the teaching refusals in :mod:`app.rag.agent` re-state; the
``<tools>`` marker keying (HIGH only) is unchanged.
"""
from __future__ import annotations
@@ -170,13 +176,14 @@ def test_tools_section_markers_and_new_tool_names() -> None:
def test_tools_section_teaches_the_harness_shapes() -> None:
"""Copy pins: ``ls``'s phase-63 catalog-line format (and its
optional one-source scope), ``grep``'s case-insensitive exact-string
locator contract (up to 20 ``source/path:line: text`` lines, a
locator not a context-adder), and ``read``'s combined
``source/path`` + full content."""
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
case-insensitive exact-string locator contract (up to 20
``source/path:line: text`` lines, a locator not a context-adder),
and ``read``'s combined ``source/path`` + full content. (Phase 72:
the source-name scope clause and the combined-identity clause are
pinned byte-for-byte in
:func:`test_tools_section_phase72_contract_clauses`.)"""
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
assert "pass a source name as `path`" in TOOLS_SECTION
assert "case-insensitive" in TOOLS_SECTION
assert "up to 20" in TOOLS_SECTION
assert "source/path:line: text" in TOOLS_SECTION
@@ -186,6 +193,66 @@ def test_tools_section_teaches_the_harness_shapes() -> None:
assert "Answer as soon as you have what you need" in TOOLS_SECTION
def test_tools_section_phase72_contract_clauses() -> None:
"""Phase 72: the two contract clauses the teaching refusals
re-state after the fact, pinned byte-for-byte in the constant —
the ``ls`` source-name clause (its optional ``path`` is a source
name, not a directory or file path; omit it to list every
document) and the ``read``/``grep`` combined-identity clause
(the combined ``source/path`` string exactly as shown in the
``ls`` output, *including the source name*; a bare document path
will not resolve)."""
# The ls source-name clause.
assert (
"a source name (e.g. 'homelab'), not a directory or file "
"path — omit it to list every document"
) in TOOLS_SECTION
# The read combined-identity clause.
assert (
"combined `source/path` string, exactly as shown in the `ls` "
"output — including the source name"
) in TOOLS_SECTION
# The bare-path note: read clause AND grep clause (exactly twice).
assert TOOLS_SECTION.count(
"a bare document path (without the source name) will not resolve"
) == 2
# The pre-phase-70 scope wording is gone — replaced by the
# explicit source-name contract.
assert "pass a source name as `path`" not in TOOLS_SECTION
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
(whose byte-identity is pinned in
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
assert "<tools>" in high
assert "not a directory or file path" in high
assert "including the source name" in high
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert "<tools>" not in low
assert "not a directory or file path" not in low
assert "including the source name" not in low
def test_documents_section_has_no_leading_intro() -> None:
"""Phase 72, task 05 (gate iterations 2-3, reverted): the
``<documents>`` section must NOT lead with an in-context reminder
or name the ``<document>`` blocks — the live telemetry showed that
copy primed the model to latch the seed documents' paths as
``ls`` scopes (the incident turn regressed to a cap-reached loop
on run 2 and re-trapped on run 5), and the reminder never flipped
the seed-doc ``read``s (15/15 across gate runs 1-5). The section
is exactly the document blocks again."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
i_open = high.index("<documents>")
i_block = high.index('<document source="Homelab"')
assert high[i_open : i_block] == "<documents>\n" # no intro line
def test_tools_section_old_names_and_budget_copy_gone() -> None:
"""The phase-37/68 tool names and the phase-37 per-tool budget line
(phase 45: the round cap is the bound — the prompt does not