fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled methodology (owner directive: stop clearing/re-importing the homelab KB per iteration; measure tool-calling accuracy on a controlled fixture KB, target >90%). Real-model gate verdicts (live, configured chat model 'lite', fixture KB): - Controlled fixture battery (the new methodology's pass condition — contract accuracy >= 90%): PASS, 4 consecutive runs: gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s) - Locked derived battery (phase-72 task 05, executed >= 90% bar, run unchanged on the same fixture KB): gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s) The teaching works — every bare-path trap self-corrects in exactly one round, zero cap hits, zero repeat loops, 10/10 answered. The locked executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the corrected re-reads (the trap question seeds its target, so the correct combined-form read is refused for redundancy) — a copy-invariant model behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04) and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md sections 5 and 7), not a copy lever. Copy changes this phase owns (unit pins updated to follow): - app/rag/agent.py: ls teaching refusals (path-like scope -> document-path line; unknown source -> no-source line with the source-name parenthetical), read/grep 'did you mean source/path?' teaching (find_path_candidates: exact or suffix path match, catalog order, cap 3), ALREADY_IN_CONTEXT naming the correct action (answer from the text already in the prompt), read tool description front-loaded with the do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is the only remaining refusal class; contract accuracy 92-100% across runs) - app/rag/prompts.py: TOOLS_SECTION states the document-identity contract up front (ls path = source name; read/grep = combined source/path including the source name; do-not-read for <documents> documents placed next to the read teaching; one-call-per-reply and never-repeat rules) - tests: refusal pins (unit + integration), new dedicated E2E suite tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in isolation), regression suites green in isolation (harness_aligned_tools, agent_document_tools, agent_unlimited_tools, search_tool, chat_rag). Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff + pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/ move and both phases' .agent/reports/ (AGENTS.md 8).
This commit is contained in:
@@ -156,6 +156,33 @@ Implements just enough of the aipi surface:
|
||||
(the trigger needs no ``<tools>`` section); no existing E2E
|
||||
question or fixture file contains the phrase, so every other
|
||||
suite is unaffected.
|
||||
- user message containing ``list the files in this directory``
|
||||
(``LS_TEACH_TRIGGER``, phase 72, teaching refusals — the
|
||||
2026-09-03 incident where the harness-prior ``ls(path='.')``
|
||||
misuse met the terse refusal and the model re-reasoned the same
|
||||
paragraphs over and over) **and** the system prompt carries the
|
||||
``<tools>`` section -> the deterministic LS-TEACHING flow,
|
||||
discriminated statelessly from the messages (streaming only):
|
||||
* request 1 (``tools`` offered, no ``tool``-role result in the
|
||||
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
|
||||
with ``{"path": "."}`` (synthetic id ``call_0``),
|
||||
``finish_reason: "tool_calls"``, no content — the incident's
|
||||
misuse, deterministic;
|
||||
* request 2 (a ``tool``-role result present that is NOT a
|
||||
catalog listing — i.e. the teaching refusal): a ``tool_calls``
|
||||
delta — ``ls`` with no arguments (id ``call_1``) — the
|
||||
correction;
|
||||
* request 3 (a ``tool``-role result whose first line matches the
|
||||
``^\\d+ documents:`` catalog header): a deterministic content
|
||||
answer — ``These are the indexed documents: <first catalog
|
||||
line>`` (the ``source: X | path: Y | title: Z`` line, parsed
|
||||
with the ``_CATALOG_LINE_RE`` machinery), ``finish_reason:
|
||||
"stop"`` — the loop ended in ONE correction, not at the round
|
||||
cap.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
|
||||
phrases are disjoint substrings — the phase-71 ordering
|
||||
convention); no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``show me a table`` (phase 44, markdown
|
||||
tables, TODO.md L6) -> the fixed table answer (``TABLE_ANSWER``):
|
||||
a 3-column service table, an ``<img onerror>`` XSS probe line, and
|
||||
@@ -440,6 +467,26 @@ assert _CORRECTION_MARKER in CORRECTION_INSTRUCTION, (
|
||||
"mock drift: the correction marker left CORRECTION_INSTRUCTION"
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 72 (teaching refusals — the 2026-09-03 incident's ls misuse):
|
||||
# the deterministic LS-TEACH self-correction flow — see the module
|
||||
# docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||
#: the deterministic LS-TEACHING flow (the incident's
|
||||
#: ``ls(path='.')`` misuse → the teaching refusal → the corrected
|
||||
#: no-arg ``ls()`` → the catalog answer). Checked BEFORE the plain
|
||||
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the phase-71
|
||||
#: ordering convention); verified: no existing E2E question or fixture
|
||||
#: file contains the phrase, so every other suite is unaffected.
|
||||
LS_TEACH_TRIGGER = "list the files in this directory"
|
||||
|
||||
#: The agent's ``ls`` listing header (app.rag.agent ``_execute_tool``):
|
||||
#: ``"N documents:"`` — the first line of every catalog tool result.
|
||||
_CATALOG_HEADER_RE = re.compile(r"^\d+ documents:")
|
||||
|
||||
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
|
||||
#: while the endpoint stays down: the openai SDK's default policy
|
||||
#: (max_retries=2 — the app's ``LLMClient`` keeps it) re-POSTs a 500'd
|
||||
@@ -547,6 +594,43 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
return docs
|
||||
|
||||
|
||||
def _tool_results(body: dict[str, Any]) -> list[str]:
|
||||
"""Every ``tool``-role result content in the messages, in order.
|
||||
|
||||
(Phase 72, LS-TEACH flow: the flow is discriminated statelessly
|
||||
from the tool results — a catalog listing vs the teaching
|
||||
refusal vs none yet.)
|
||||
"""
|
||||
return [
|
||||
str(m.get("content") or "")
|
||||
for m in _messages(body)
|
||||
if m.get("role") == "tool"
|
||||
]
|
||||
|
||||
|
||||
def _first_catalog_line(body: dict[str, Any]) -> str | None:
|
||||
"""The first catalog line of a catalog listing in the messages.
|
||||
|
||||
A catalog listing is a ``tool``-role result whose FIRST line is the
|
||||
agent's ``"N documents:"`` header (``_CATALOG_HEADER_RE``); its
|
||||
first ``source: X | path: Y | title: Z`` line (the
|
||||
``_CATALOG_LINE_RE`` machinery) is returned. ``None`` when no
|
||||
catalog listing is in the messages — e.g. while only the teaching
|
||||
refusal is there (the phase-72 LS-TEACH flow's request-2 state).
|
||||
An empty listing (``"0 documents:"`` with no lines) returns
|
||||
``""`` — the listing is present, it is just empty.
|
||||
"""
|
||||
for content in _tool_results(body):
|
||||
lines = content.splitlines()
|
||||
if not lines or not _CATALOG_HEADER_RE.match(lines[0]):
|
||||
continue
|
||||
for line in lines[1:]:
|
||||
if _CATALOG_LINE_RE.match(line):
|
||||
return line
|
||||
return ""
|
||||
return None
|
||||
|
||||
|
||||
#: One line of the agent's ``grep`` output (app.rag.agent
|
||||
#: ``_execute_tool``, phase 68 — phase 70 renamed the tool, the line
|
||||
#: format is unchanged): ``source/path:LINE: text``. The
|
||||
@@ -718,6 +802,40 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("list", "", "")
|
||||
|
||||
|
||||
def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-72 LS-TEACH request (see the module docstring).
|
||||
|
||||
* ``("misuse",)`` — ``tools`` are offered and no ``tool``-role
|
||||
result is in the messages yet: the incident's misuse — ``ls``
|
||||
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
|
||||
"tool_calls"``, no content.
|
||||
* ``("correct",)`` — a ``tool``-role result is in the messages and
|
||||
it is NOT a catalog listing (the teaching refusal): the
|
||||
correction — ``ls`` with no arguments (id ``call_1``).
|
||||
* ``("answer", line)`` — a ``tool``-role result whose first line
|
||||
is the ``"N documents:"`` catalog header: the deterministic
|
||||
content answer ``These are the indexed documents: <line>`` (the
|
||||
first catalog line), ``finish_reason: "stop"`` — the loop
|
||||
settled in ONE correction, not at the round cap.
|
||||
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
||||
section is missing (deflected turns never carry it), or
|
||||
``tools`` are not offered and no tool results are in the
|
||||
messages yet (e.g. ``agent_max_rounds=0``).
|
||||
"""
|
||||
if LS_TEACH_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
line = _first_catalog_line(body)
|
||||
if line is not None:
|
||||
return ("answer", line)
|
||||
if _tool_results(body):
|
||||
return ("correct",)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("misuse",)
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -1208,6 +1326,30 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 72 (teaching refusals): the deterministic LS-TEACH
|
||||
# self-correction flow — checked BEFORE the plain
|
||||
# TOOLS_TRIGGER flow (disjoint trigger phrases — the phase-71
|
||||
# ordering convention; the trigger needs the ``<tools>``
|
||||
# section, so deflected turns never hit it).
|
||||
ls_teach = _ls_teach_flow(body)
|
||||
if ls_teach is not None:
|
||||
if ls_teach[0] == "misuse":
|
||||
# The incident's misuse, deterministic: ls(path='.').
|
||||
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
|
||||
elif ls_teach[0] == "correct":
|
||||
# The one-round correction: the no-arg full listing.
|
||||
stream = _tool_call_stream("ls", {}, "call_1")
|
||||
else: # "answer" — quote the first catalog line
|
||||
answer = _apply_max_tokens(
|
||||
f"These are the indexed documents: {ls_teach[1]}",
|
||||
body.get("max_tokens"),
|
||||
)
|
||||
stream = _sse_stream(answer, 0.0)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
flow = _tool_flow(body)
|
||||
if flow is not None:
|
||||
if flow[0] == "list":
|
||||
|
||||
@@ -44,7 +44,8 @@ pattern, grown to three documents):
|
||||
Three documents (not two, as in phase 37) so BOTH reads land on
|
||||
documents outside the seed: with a two-document corpus the second read
|
||||
would be the already-in-context retrieval document and the agent would
|
||||
answer "Already in your context." — a rejection, not the multi-read
|
||||
answer "Already in your context — …" (the in-context refusal) — a
|
||||
rejection, not the multi-read
|
||||
flow this story proves.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
|
||||
@@ -0,0 +1,472 @@
|
||||
"""Phase 72 E2E (Playwright, mock-only): the ls-teaching
|
||||
self-correction loop through the real UI.
|
||||
|
||||
Story: ``.agent/user_stories/agent-document-tools.md`` (this phase
|
||||
repairs the model-facing contract the phase-70 tools reshaped — the
|
||||
2026-09-03 incident: the harness-prior ``ls(path='.')`` misuse met the
|
||||
terse refusal, and the model re-reasoned the same paragraphs over and
|
||||
over before answering from the seed documents alone).
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_tool_path_teaching.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
|
||||
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
|
||||
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
|
||||
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
|
||||
(``No source named '.' — check the ls output. (…)``) → the corrected
|
||||
no-arg ``ls()`` (id ``call_1``) → the deterministic
|
||||
``These are the indexed documents: <first catalog line>`` answer.
|
||||
|
||||
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
|
||||
documents of known ``source``/``path``/``title`` (catalog order =
|
||||
``(source, path)``, so the first catalog line is deterministic):
|
||||
|
||||
* ``Homelab/aws-route53.md`` — the CATALOG-FIRST document, indexed
|
||||
WITHOUT chunks (catalog-only; never in the retrieval context, so
|
||||
the single-read flow's ``read`` of it is NOT deduped as already-in-
|
||||
context). Its FIRST line is longer than 80 chars, so the mock's
|
||||
first-80-chars quote (the single-read regression turn) stays
|
||||
newline-free.
|
||||
* ``Homelab/example-record-file.json`` — the retrievable document:
|
||||
one chunk whose embedding is the mock's own bag-of-words vector
|
||||
(the trigger question cosines well past the E2E 0.30 threshold and
|
||||
FTS-matches too → grounded, the ``<tools>`` section rides along).
|
||||
It is the seed context only — the single-read flow reads the
|
||||
catalog-FIRST document, not the seed.
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
|
||||
LS-TEACH turn: the turn settles (composer re-enables, ``done``
|
||||
observed), the answer bubble carries the first catalog line — the
|
||||
first document's ``source:`` / ``path:`` / title fields (the
|
||||
catalog reached the model and landed in the answer), the UI shows
|
||||
the two tool lines (``🔎 Listing documents in <code>.</code>``
|
||||
then ``🔎 Listing documents``), and no error banner. Wire level:
|
||||
the ``tool`` frames arrive in order — first ``ls`` with
|
||||
``argument: "."``, then ``ls`` with ``argument: null`` — and there
|
||||
is NO third ``tool`` frame (the loop ended in one correction, not
|
||||
at the round cap).
|
||||
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
|
||||
session, the LS-TEACH turn settles and a follow-up question
|
||||
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
|
||||
with the read flow's answer (``ls`` → ``read`` on the first
|
||||
catalog line's combined identity → ``Read <source/path>. <quote>``)
|
||||
— the new flow did not swallow the existing trigger.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from tests.e2e.mock_llm import (
|
||||
LS_TEACH_TRIGGER,
|
||||
TOOLS_TRIGGER,
|
||||
embed_text,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The one-source, two-document fixture (see the module docstring)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE = "Homelab"
|
||||
DOC1_PATH = "aws-route53.md"
|
||||
DOC1_TITLE = "AWS Route 53 Notes"
|
||||
DOC1_SP = f"{SEED_SOURCE}/{DOC1_PATH}"
|
||||
|
||||
DOC2_PATH = "example-record-file.json"
|
||||
DOC2_TITLE = "Example Record File"
|
||||
DOC2_SP = f"{SEED_SOURCE}/{DOC2_PATH}"
|
||||
|
||||
#: The FIRST catalog line (catalog order = (source, path) — DOC1 sorts
|
||||
#: first): the mock's LS-TEACH answer quotes exactly this line.
|
||||
FIRST_CATALOG_LINE = (
|
||||
f"source: {SEED_SOURCE} | path: {DOC1_PATH} | title: {DOC1_TITLE}"
|
||||
)
|
||||
|
||||
#: The catalog-first document (catalog order = (source, path) —
|
||||
#: DOC1 sorts first): the single-read flow reads THIS document, so it
|
||||
#: must NOT be the seed (a seed read dedupes to "Already in your
|
||||
#: context.", which the mock's single-read flow does not model — it
|
||||
#: would loop to the round cap). Indexed WITHOUT chunks: catalog-only,
|
||||
#: never in the retrieval context. Its FIRST line is longer than 80
|
||||
#: chars, so the mock's first-80-chars quote (the single-read
|
||||
#: regression turn) stays newline-free.
|
||||
DOC1_CONTENT = (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the exact JSON shape of reeselink.json is "
|
||||
"documented in the record file below.\n"
|
||||
+ (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the record file shape of reeselink.json is "
|
||||
"the contract every sync job relies on.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
assert "\n" not in DOC1_CONTENT[:80] # the quote must stay one line
|
||||
|
||||
#: The retrievable document (the grounded seed context, the cf.
|
||||
#: test_harness_aligned_tools.py pattern): the repeated record-file
|
||||
#: lines carry the trigger question's key tokens — well past the E2E
|
||||
#: 0.30 cosine threshold, plus FTS hits. Its FIRST line is longer
|
||||
#: than 80 chars too, so the retrieval seed context is one clean
|
||||
#: line.
|
||||
DOC2_CONTENT = (
|
||||
"The ReeseLink hosted zone record file reeselink.json holds every "
|
||||
"aws route53 record for reeselink — the note documents the exact "
|
||||
"JSON shape of reeselink.json for the record file.\n"
|
||||
+ (
|
||||
"The aws route53 record file reeselink.json keeps every record "
|
||||
"for the reeselink hosted zone — the exact JSON shape of the "
|
||||
"record file is the contract every sync job relies on.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
assert "\n" not in DOC2_CONTENT[:80] # the seed context stays one line
|
||||
|
||||
#: Carries ``LS_TEACH_TRIGGER`` and is on-topic (grounded — HIGH, the
|
||||
#: ``<tools>`` section rides along); it carries NO other mock marker.
|
||||
LS_TEACH_QUESTION = (
|
||||
"List the files in this directory — what do my aws route53 notes "
|
||||
"say about the reeselink.json record file?"
|
||||
)
|
||||
assert LS_TEACH_TRIGGER in LS_TEACH_QUESTION.lower()
|
||||
for _other in (
|
||||
"use your tools",
|
||||
"read two documents",
|
||||
"search your documents",
|
||||
"emit raw tool markup",
|
||||
"always emit raw tool markup",
|
||||
"show me a table",
|
||||
"think in paragraphs",
|
||||
"think out loud then hesitate",
|
||||
"think out loud",
|
||||
"show the end of your notes",
|
||||
"write a long answer",
|
||||
"fail then answer",
|
||||
"always fail",
|
||||
"embed fail once",
|
||||
"pretend to think slowly",
|
||||
):
|
||||
assert _other not in LS_TEACH_QUESTION.lower(), _other
|
||||
|
||||
#: Carries ``TOOLS_TRIGGER`` (the single-read flow) and nothing else —
|
||||
#: the no-regression follow-up question in the same session.
|
||||
READ_QUESTION = (
|
||||
"Use your tools: what is the exact JSON shape of reeselink.json "
|
||||
"for my aws route53 hosted zone?"
|
||||
)
|
||||
assert TOOLS_TRIGGER in READ_QUESTION.lower()
|
||||
for _other in (
|
||||
LS_TEACH_TRIGGER,
|
||||
"read two documents",
|
||||
"search your documents",
|
||||
"emit raw tool markup",
|
||||
"always emit raw tool markup",
|
||||
"show me a table",
|
||||
"think in paragraphs",
|
||||
"think out loud then hesitate",
|
||||
"think out loud",
|
||||
"show the end of your notes",
|
||||
"write a long answer",
|
||||
"fail then answer",
|
||||
"always fail",
|
||||
"embed fail once",
|
||||
"pretend to think slowly",
|
||||
):
|
||||
assert _other not in READ_QUESTION.lower(), _other
|
||||
|
||||
#: The mock's single-read answer (the read document reached the model
|
||||
#: and landed in the answer) — DOC1 is the first catalog line, so the
|
||||
#: flow reads ``Homelab/aws-route53.md`` and quotes its first 80 chars.
|
||||
READ_ANSWER_PREFIX = f"Read {DOC1_SP}."
|
||||
READ_ANSWER_QUOTE = DOC1_CONTENT[:80]
|
||||
|
||||
|
||||
def _seed_fixture(db: Session) -> None:
|
||||
"""The one-source, two-document fixture (see the module docstring).
|
||||
|
||||
DOC1 (catalog-first) is indexed WITHOUT chunks; DOC2 carries the
|
||||
single chunk (the mock's own embedding → the trigger question
|
||||
cosines well past the E2E 0.30 threshold and FTS-matches too →
|
||||
grounded). DOC2 is the seed context only — the single-read flow
|
||||
reads the catalog-FIRST document (DOC1), which is not in context.
|
||||
"""
|
||||
db.add(
|
||||
Document(
|
||||
source=SEED_SOURCE,
|
||||
path=DOC1_PATH,
|
||||
full_path=f"/tmp/{DOC1_PATH}",
|
||||
title=DOC1_TITLE,
|
||||
content=DOC1_CONTENT,
|
||||
content_hash=hashlib.sha256(DOC1_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
doc2 = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=DOC2_PATH,
|
||||
full_path=f"/tmp/{DOC2_PATH}",
|
||||
title=DOC2_TITLE,
|
||||
content=DOC2_CONTENT,
|
||||
content_hash=hashlib.sha256(DOC2_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
db.add(doc2)
|
||||
db.flush()
|
||||
# One chunk carrying the mock's own embedding → genuine token
|
||||
# overlap between the trigger question and DOC2 (the only
|
||||
# retrievable document).
|
||||
db.add(
|
||||
Chunk(
|
||||
document_id=doc2.id,
|
||||
position=0,
|
||||
content=DOC2_CONTENT,
|
||||
embedding=embed_text(DOC2_CONTENT),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _reset_db_fixture() -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then seed the
|
||||
one-source, two-document fixture. ``steering_notes`` /
|
||||
``kb_overview`` are truncated too, so the HIGH prompt is exactly
|
||||
``<relevance>`` + ``<documents>`` + ``<tools>`` — byte-stable
|
||||
prompts, byte-stable answers."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
_seed_fixture(db)
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the house pattern — cf. test_harness_aligned_tools.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_sse_hook(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _drain_frames(page: Page) -> list[dict]:
|
||||
"""One turn's SSE frames: wait for that turn's ``done`` frame, then
|
||||
return EVERY frame captured since the last drain (the hook's
|
||||
background read appends the whole stream at once after it closes, so
|
||||
clearing-and-reading is race-free per turn)."""
|
||||
deadline = time.monotonic() + 10.0
|
||||
while True:
|
||||
raw = page.evaluate(
|
||||
"() => { const f = window.__sseFrames || []; "
|
||||
"window.__sseFrames = []; return f; }"
|
||||
)
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered.
|
||||
|
||||
Phase 48: the label assertion carries the settle wait with an
|
||||
explicit timeout — the in-flight button is the enabled Stop control
|
||||
(never disabled), so ``to_be_enabled`` no longer blocks until the
|
||||
turn settles."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _assert_no_error_banner(page: Page) -> None:
|
||||
"""The turn settled through the normal done path — never the red
|
||||
role=alert error banner (the KB-offline banner is a separate,
|
||||
health-driven state the db_ready fixture keeps away)."""
|
||||
banner = page.locator("#kb-banner")
|
||||
expect(banner).to_be_hidden()
|
||||
expect(banner).not_to_have_attribute("role", "alert")
|
||||
expect(banner).not_to_have_class(re.compile(r"is-error"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The grounded LS-TEACH turn: the incident's ls(path='.') misuse →
|
||||
# the teaching refusal → the corrected no-arg ls() → the catalog
|
||||
# answer — the loop settles in ONE correction (two tool rounds),
|
||||
# pinned on the SSE wire
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ls_misuse_self_corrects_to_noarg_listing(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_fixture()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, LS_TEACH_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Self-correction: the answer quotes the FIRST catalog line — the
|
||||
# first document's source: / path: / title fields reached the model
|
||||
# and landed in the answer (the catalog round settled the turn).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("These are the indexed documents:")
|
||||
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# The UI shows the two tool lines in order: the scoped misuse
|
||||
# (🔎 Listing documents in <code>.</code>) then the corrected
|
||||
# unscoped listing (🔎 Listing documents — no <code>).
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(0).locator("code")).to_have_text(".")
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1).locator("code")).to_have_count(0)
|
||||
|
||||
# Two rounds on the wire: the tool frames arrive in order — first
|
||||
# ls with argument "." (the incident's misuse), then ls with
|
||||
# argument null (the correction) — and there is NO third tool
|
||||
# frame: the loop ended in one correction, not at the round cap.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": "."},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert not [f for f in frames if f.get("type") == "error"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. No regression to the plain flow — the SAME session: after the
|
||||
# LS-TEACH turn, the TOOLS_TRIGGER follow-up (the single-read flow)
|
||||
# still settles with the read flow's answer
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db_fixture()
|
||||
page.goto(app_url)
|
||||
_install_sse_hook(page)
|
||||
|
||||
# Turn 1 — the LS-TEACH flow (the incident's misuse → the
|
||||
# correction → the catalog answer).
|
||||
_submit(page, LS_TEACH_QUESTION)
|
||||
_wait_settled(page)
|
||||
teach_frames = _drain_frames(page)
|
||||
assert _tool_frames(teach_frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": "."},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
]
|
||||
expect(
|
||||
page.locator(".msg.brain .bubble").last
|
||||
).to_contain_text(FIRST_CATALOG_LINE)
|
||||
|
||||
# Turn 2 — the SAME session: the single-read flow on
|
||||
# TOOLS_TRIGGER. The new flow must not have swallowed the existing
|
||||
# trigger: the follow-up settles with the read flow's answer.
|
||||
_submit(page, READ_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
second_msg = page.locator(".msg.brain").last
|
||||
# The UI shows the single-read flow's two lines: the unscoped ls
|
||||
# then the read of the first catalog line's COMBINED identity.
|
||||
lines = second_msg.locator(".tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(0).locator("code")).to_have_count(0)
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
|
||||
|
||||
# The answer quotes the read document (the mock's deterministic
|
||||
# echo: "Read <source/path>. <first 80 chars>").
|
||||
bubble = second_msg.locator(".bubble").last
|
||||
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# Wire level for the follow-up: ls (null) → read (the combined
|
||||
# identity) — the single-read flow, unchanged.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "read", "argument": DOC1_SP},
|
||||
]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert not [f for f in frames if f.get("type") == "error"]
|
||||
@@ -185,22 +185,48 @@ def _settings(**kwargs: Any) -> Settings:
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
class ScriptedToolCallsLLM:
|
||||
"""N scripted tool-call rounds (one ``ToolCallPiece`` each), then one
|
||||
canned answer; records every request's messages and tools (the
|
||||
phase-72 task-02 two-round self-correction cases: the refusal round,
|
||||
then the corrected call)."""
|
||||
|
||||
def __init__(self, calls: list[ToolCallPiece]) -> None:
|
||||
self.calls = calls
|
||||
self.requests: list[
|
||||
tuple[list[dict[str, Any]], list[dict[str, Any]] | None]
|
||||
] = []
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
scaffolding: ScaffoldingFilter | None = None,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
self.requests.append((deepcopy(messages), deepcopy(tools)))
|
||||
index = len(self.requests) - 1
|
||||
if index < len(self.calls):
|
||||
yield self.calls[index]
|
||||
else:
|
||||
yield StreamPiece("content", "ans")
|
||||
|
||||
|
||||
def _run_call(
|
||||
db: Session, name: str, arguments: dict[str, Any]
|
||||
) -> tuple[AgentHolder, ScriptedToolLLM]:
|
||||
"""Drive one scripted tool call through ``run_agent``."""
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedToolLLM(ToolCallPiece(id="call_1", name=name, arguments=arguments))
|
||||
asyncio.run(_consume(llm, db, holder))
|
||||
asyncio.run(_consume(cast("LLMClient", llm), db, holder))
|
||||
return holder, llm
|
||||
|
||||
|
||||
async def _consume(
|
||||
llm: ScriptedToolLLM, db: Session, holder: AgentHolder
|
||||
llm: LLMClient, db: Session, holder: AgentHolder
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
llm,
|
||||
db,
|
||||
system_prompt="SYSTEM_PROMPT",
|
||||
user_message="QUESTION",
|
||||
@@ -233,17 +259,39 @@ def test_ls_scoped_to_registered_source_through_run_agent(kb, src, db) -> None:
|
||||
|
||||
|
||||
def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
|
||||
"""Phase 72: the no-source refusal now carries the teaching
|
||||
parenthetical — the prefix byte-identical to the pre-phase-72 line;
|
||||
still not counted."""
|
||||
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
||||
db.commit()
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Ghost"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"] == "No source named 'Ghost' — check the ls output."
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="Ghost")
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
|
||||
|
||||
def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> None:
|
||||
"""Phase 72: a ``/``-containing ``path`` is a document path, not a
|
||||
source name — the ``LS_PATH_NOT_A_SOURCE`` teaching line (no
|
||||
registry lookup needed), not counted, the tools stay offered on the
|
||||
next request."""
|
||||
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
||||
db.commit()
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "app/rag/importer.py"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
|
||||
# ---------- read (the canonical combined source/path form) ----------
|
||||
|
||||
|
||||
@@ -280,9 +328,10 @@ def test_read_bare_source_name_refused_through_run_agent(kb, db) -> None:
|
||||
|
||||
|
||||
def test_read_unknown_combined_path_refused_through_run_agent(kb, db) -> None:
|
||||
"""A combined identity that matches nothing gets the no-document
|
||||
refusal (the argument echoed as passed — the model sees its own
|
||||
form)."""
|
||||
"""A combined identity that matches NOTHING — not a document and not
|
||||
any indexed document's ``path`` (zero candidates) — gets today's
|
||||
no-document refusal byte-identical (the argument echoed as passed —
|
||||
the model sees its own form)."""
|
||||
_doc(db, "Alpha", "x.md", "X", "X-CONTENT")
|
||||
db.commit()
|
||||
|
||||
@@ -295,6 +344,79 @@ def test_read_unknown_combined_path_refused_through_run_agent(kb, db) -> None:
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
|
||||
|
||||
def test_read_bare_path_single_source_suggestion_then_corrected_read(kb, db) -> None:
|
||||
"""Phase 72, task 02: a bare path under ONE source (exact ``path``
|
||||
match, the source prefix missing) → the single-identity suggestion
|
||||
(a refusal — not counted); the scripted corrected call (round 2, the
|
||||
suggested combined identity) then succeeds against real Postgres —
|
||||
the two-round self-correction."""
|
||||
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
|
||||
db.commit()
|
||||
|
||||
llm = ScriptedToolCallsLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read", arguments={"path": "deep/nested/doc.md"}
|
||||
),
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
arguments={"path": "Alpha/deep/nested/doc.md"},
|
||||
),
|
||||
]
|
||||
)
|
||||
holder = AgentHolder()
|
||||
asyncio.run(_consume(cast("LLMClient", llm), db, holder))
|
||||
|
||||
# Round 1: the bare path resolves to no combined identity, but it IS
|
||||
# the indexed document's path — the refusal names the one combined
|
||||
# identity to use (not counted, the tools stay offered).
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'deep/nested/doc.md' — "
|
||||
"did you mean 'Alpha/deep/nested/doc.md'?"
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
# Round 2: the corrected combined identity succeeds — the full
|
||||
# content, the holder records the row, and it counts.
|
||||
assert llm.requests[2][0][5]["content"] == (
|
||||
"Document Alpha/deep/nested/doc.md:\nFULL-TEXT"
|
||||
)
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
assert holder.read_docs == [created]
|
||||
assert holder.tool_calls == 1 # only the corrected read executed
|
||||
|
||||
|
||||
def test_read_bare_path_two_sources_one_of_suggestion_then_corrected_read(
|
||||
kb, db,
|
||||
) -> None:
|
||||
"""Phase 72, task 02: the same bare path under TWO sources → the
|
||||
``one of`` line (up to ``SUGGESTION_LIMIT`` identities, catalog
|
||||
order — Alpha before Beta); the scripted corrected call (round 2,
|
||||
the first suggested identity) then succeeds."""
|
||||
a = _doc(db, "Alpha", "shared/x.md", "Alpha X", "A-TEXT")
|
||||
_doc(db, "Beta", "shared/x.md", "Beta X", "B-TEXT")
|
||||
db.commit()
|
||||
|
||||
llm = ScriptedToolCallsLLM(
|
||||
[
|
||||
ToolCallPiece(id="call_1", name="read", arguments={"path": "shared/x.md"}),
|
||||
ToolCallPiece(
|
||||
id="call_2", name="read", arguments={"path": "Alpha/shared/x.md"}
|
||||
),
|
||||
]
|
||||
)
|
||||
holder = AgentHolder()
|
||||
asyncio.run(_consume(cast("LLMClient", llm), db, holder))
|
||||
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'shared/x.md' — did you mean one of: "
|
||||
"'Alpha/shared/x.md', 'Beta/shared/x.md'?"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == "Document Alpha/shared/x.md:\nA-TEXT"
|
||||
assert holder.read_docs == [a]
|
||||
assert holder.tool_calls == 1 # only the corrected read executed
|
||||
|
||||
|
||||
# ---------- grep (the phase-68 A5 contract under the new name) ----------
|
||||
|
||||
|
||||
@@ -345,6 +467,9 @@ def test_grep_scoped_through_run_agent(kb, db) -> None:
|
||||
|
||||
|
||||
def test_grep_scoped_missing_doc_refused_through_run_agent(kb, db) -> None:
|
||||
"""A scoped ``grep`` miss that matches no indexed document's ``path``
|
||||
(zero candidates) keeps today's line byte-identical — a refusal,
|
||||
not counted."""
|
||||
_doc(db, "Alpha", "a/one.md", "One", "nothing")
|
||||
db.commit()
|
||||
|
||||
|
||||
+492
-23
@@ -141,21 +141,49 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
assert not set(by_name) & {"list_documents", "read_document", "search_documents"}
|
||||
assert all(t["type"] == "function" for t in AGENT_TOOLS)
|
||||
ls = by_name["ls"]["function"]
|
||||
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
|
||||
# clause (the harness prior batches calls; the loop executes one
|
||||
# per round — the extras count as unexecuted in the gate).
|
||||
assert ls["description"] == (
|
||||
"List the indexed documents as `source: X | path: Y | title: Z` lines."
|
||||
"List the indexed documents as `source: X | path: Y | "
|
||||
"title: Z` lines. Call one tool at a time — wait for "
|
||||
"this result before your next call."
|
||||
)
|
||||
ls_params = ls["parameters"]
|
||||
assert ls_params["type"] == "object"
|
||||
assert ls_params["required"] == [] # path is optional
|
||||
assert set(ls_params["properties"]) == {"path"}
|
||||
assert ls_params["properties"]["path"]["type"] == "string"
|
||||
# Phase 72: the description states the contract up front — the
|
||||
# 'path' argument is a source name, not a file or directory path.
|
||||
# Task 05 (live gate iteration 5): the cross-tool contrast clause
|
||||
# (ls is the ONLY tool whose path is a source name — the model
|
||||
# kept transferring that scope to grep's document identity).
|
||||
assert ls_params["properties"]["path"]["description"] == (
|
||||
"Source name to list one source's documents (e.g. 'homelab'); "
|
||||
"omit to list every document."
|
||||
"Source name to list one source's documents (e.g. 'homelab') — "
|
||||
"a source name, not a file or directory path; omit to list "
|
||||
"every document. This is the only tool "
|
||||
"whose `path` is a source name — for "
|
||||
"`read` and `grep` it must be a document's "
|
||||
"combined `source/path`."
|
||||
)
|
||||
read = by_name["read"]["function"]
|
||||
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
|
||||
# the do-not-read rule is FRONT-LOADED — the controlled gate's
|
||||
# telemetry showed the `lite` model obeying the user's "open it /
|
||||
# read it" and reading seed-context documents the <documents>
|
||||
# section already carries (every refusal of a 12-call run was
|
||||
# ALREADY_IN_CONTEXT); the rule now leads the description instead
|
||||
# of sitting mid-paragraph, and the tool is framed as "only for
|
||||
# documents NOT already in <documents>".
|
||||
assert read["description"] == (
|
||||
"Add the full content of one indexed document to your context."
|
||||
"Do not call this tool for a document already shown in "
|
||||
"the <documents> section, even when the user asks you to "
|
||||
"open or read it — its full text is already in your "
|
||||
"prompt; answer directly from it. Use it only to add a "
|
||||
"document NOT already in <documents> to your context, "
|
||||
"by its combined `source/path` string. Call one tool at "
|
||||
"a time — wait for this result before your next call."
|
||||
)
|
||||
read_params = read["parameters"]
|
||||
assert read_params["type"] == "object"
|
||||
@@ -163,18 +191,35 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
assert set(read_params["properties"]) == {"path"}
|
||||
assert read_params["properties"]["path"]["type"] == "string"
|
||||
# The combined source/path string is the canonical document identity
|
||||
# (phase 70) — the description pins it with a worked example.
|
||||
# (phase 70) — the description pins it with a worked example. Phase
|
||||
# 72 (task 02): the bare-path contract is stated up front; task 05
|
||||
# (live gate iteration 1): the do-not-re-read clause (the dedupe
|
||||
# refusal's prevention at the prompt).
|
||||
assert read_params["properties"]["path"]["description"] == (
|
||||
"The document to add to your context, as the combined "
|
||||
"`source/path` string exactly as shown in the `ls` output (e.g. "
|
||||
"'homelab/active/container_caddy/caddy.md')."
|
||||
"'homelab/active/container_caddy/caddy.md'). A bare document "
|
||||
"path (without the source name) will not resolve. Only pass a "
|
||||
"document NOT already shown in the <documents> section — it is "
|
||||
"already in your context; do not re-read it."
|
||||
)
|
||||
grep = by_name["grep"]["function"]
|
||||
# Task 05 (live gate iterations 2-6, refined in the 2026-09-03
|
||||
# re-run): the pattern-only-is-the-knowledge-base-search clause
|
||||
# ("pass ONLY `pattern`") + the source-name-is-not-a-document
|
||||
# clause (the model kept scoping grep with an ls-style source name
|
||||
# — the 2026-09-03 incident loop shape, but on grep) plus the
|
||||
# one-call-at-a-time discipline clause.
|
||||
assert grep["description"] == (
|
||||
"Search the indexed documents for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines as "
|
||||
"`source/path:line: text` — a locator, not a context-adder: "
|
||||
"read the winner with `read`."
|
||||
"(case-insensitive) and return up to 20 matching lines "
|
||||
"as `source/path:line: text` — a locator, not a "
|
||||
"context-adder: read the winner with `read`. For a "
|
||||
"normal search pass ONLY `pattern` — it searches every "
|
||||
"document and that is how you search the knowledge "
|
||||
"base; never pass a source name as `path` (a source "
|
||||
"name is not a document). Call one tool at a time — "
|
||||
"wait for this result before your next call."
|
||||
)
|
||||
grep_params = grep["parameters"]
|
||||
assert grep_params["type"] == "object"
|
||||
@@ -184,9 +229,25 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
assert grep_params["properties"]["pattern"]["description"] == (
|
||||
"The exact text to search for (a plain substring, not a regex)"
|
||||
)
|
||||
# Phase 72 (task 02): the bare-path contract is stated up front;
|
||||
# task 05 (live gate iterations 1-8): the one-known-document clause
|
||||
# with a worked combined-identity example and the source-name ban
|
||||
# (the model kept scoping grep with an ls-style source name — the
|
||||
# incident loop shape, but on grep).
|
||||
# Iteration 8 drops the standalone 'homelab' from this negative
|
||||
# example — the gate's live telemetry showed the model emitting
|
||||
# exactly that value, and naming it beside the parameter risks
|
||||
# priming it (the negative-example effect). The 2026-09-03 re-run
|
||||
# makes the rarity explicit ("Rarely needed") and re-states the
|
||||
# pattern-only normal search.
|
||||
assert grep_params["properties"]["path"]["description"] == (
|
||||
"Limit the search to one document, as a combined `source/path` "
|
||||
"string from the `ls` output (omit to search every document)."
|
||||
"Rarely needed — only for re-searching one "
|
||||
"document you already know: that document's "
|
||||
"combined `source/path` identity (e.g. "
|
||||
"'homelab/ansible/inventory.yaml'). Never a "
|
||||
"source name. A bare document path (without "
|
||||
"the source name) will not resolve. Omit it "
|
||||
"for a normal search (pass only `pattern`)."
|
||||
)
|
||||
|
||||
|
||||
@@ -198,11 +259,44 @@ def test_agent_tools_order_is_ls_read_grep() -> None:
|
||||
|
||||
def test_refusal_constants_are_harness_aligned() -> None:
|
||||
"""The updated module-level refusal lines (the names moved to the
|
||||
harness surface; ALREADY_IN_CONTEXT / UNKNOWN_TOOL unchanged)."""
|
||||
assert agent.ALREADY_IN_CONTEXT == "Already in your context."
|
||||
harness surface). The ALREADY_IN_CONTEXT line is a phase-72,
|
||||
task 05 gate-iteration teaching (live telemetry: the model
|
||||
repeated the terse phase-37 line) — same refusal behavior, the
|
||||
copy names the correct action."""
|
||||
assert agent.ALREADY_IN_CONTEXT == (
|
||||
"Already in your context — the full text is already in your "
|
||||
"prompt. Do not call read on it again; answer from that text."
|
||||
)
|
||||
assert agent.UNKNOWN_TOOL == "Unknown tool."
|
||||
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
|
||||
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
|
||||
# Phase 72: the ls teaching-refusal templates, pinned byte-for-byte
|
||||
# (task 01 — the read/grep suggestion templates below, task 02).
|
||||
assert agent.LS_PATH_NOT_A_SOURCE == (
|
||||
"'{path}' looks like a document path, not a source name. The "
|
||||
"'path' argument of ls filters by source name (e.g. 'homelab') — "
|
||||
"omit it to list every document, or read a document by its "
|
||||
"combined 'source/path' string."
|
||||
)
|
||||
# The pre-phase-72 no-source line is the byte-identical prefix of
|
||||
# the extended line — only the teaching parenthetical was appended.
|
||||
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
|
||||
"No source named '{scope}' — check the ls output."
|
||||
)
|
||||
assert agent.NO_SOURCE_NOT_A_DIRECTORY == (
|
||||
"No source named '{scope}' — check the ls output. (The 'path' "
|
||||
"argument is a source name, not a directory — omit it to list "
|
||||
"every document.)"
|
||||
)
|
||||
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
|
||||
# templates, pinned byte-for-byte, and the suggestion cap.
|
||||
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
|
||||
"No document at '{arg}' — did you mean '{source}/{path}'?"
|
||||
)
|
||||
assert agent.NO_DOCUMENT_DID_YOU_MEAN_MANY == (
|
||||
"No document at '{arg}' — did you mean one of: {candidates}?"
|
||||
)
|
||||
assert agent.SUGGESTION_LIMIT == 3
|
||||
|
||||
|
||||
# ---------- list_source_names (the scoped ls registry join) ----------
|
||||
@@ -467,8 +561,9 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
|
||||
|
||||
|
||||
def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A ``path`` matching no source name is a refusal — not counted, the
|
||||
round cap bounds its repetition."""
|
||||
"""A ``path`` without ``/`` matching no source name is a refusal —
|
||||
the extended line with the teaching parenthetical (phase 72), not
|
||||
counted, the round cap bounds its repetition."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||
holder = AgentHolder()
|
||||
@@ -479,7 +574,65 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.tool_calls == 0 # a refusal counts in nothing
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"] == "No source named 'Ghost' — check the ls output."
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="Ghost")
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
|
||||
def test_ls_path_like_scope_gets_document_path_teaching_refusal(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 72: a stripped scope containing ``/`` looks like a document
|
||||
path (the incident's ``ls(path='app/rag/importer.py')``) — a source
|
||||
name is a directory basename and can never contain one, so this gets
|
||||
the ``LS_PATH_NOT_A_SOURCE`` teaching line with the argument echoed;
|
||||
no registry lookup, counts in nothing, tools stay offered."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
||||
|
||||
def _boom_sources(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("no registry lookup for a path-like scope")
|
||||
|
||||
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="ls",
|
||||
arguments={"path": "app/rag/importer.py"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.tool_calls == 0 # a refusal counts in nothing
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
|
||||
def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
|
||||
``/``, no matching source) gets the extended no-source refusal with
|
||||
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
|
||||
offered."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="ls", arguments={"path": "."})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.tool_calls == 0 # a refusal counts in nothing
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope=".")
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
@@ -529,14 +682,19 @@ def test_read_combined_path_resolves_and_returns_full_content(
|
||||
|
||||
def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A bare source name (no '/') can never be a document — the
|
||||
no-document refusal (the argument echoed as passed), no DB lookup,
|
||||
nothing counted."""
|
||||
no-document refusal (the argument echoed as passed), no DB lookup
|
||||
(NOT even the phase-72 candidate lookup — ``all_documents`` must
|
||||
not run either), nothing counted."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Homelab", "a.md", "A")])
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("find_document must not run for a bare source name")
|
||||
raise AssertionError(
|
||||
"no DB lookup (find_document or all_documents) for a bare "
|
||||
"source name"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
monkeypatch.setattr(agent, "all_documents", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "Homelab"})],
|
||||
@@ -553,10 +711,12 @@ def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatc
|
||||
def test_read_unknown_path_refused_echoing_argument(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""An unknown combined identity → the refusal echoing the argument as
|
||||
passed (the model sees its own form) — the old split-teaching refusal
|
||||
is gone (phase 70)."""
|
||||
"""An unknown combined identity that matches NO indexed document's
|
||||
``path`` (zero candidates — the phase-72 lookup runs, finds nothing)
|
||||
→ today's refusal echoing the argument as passed, byte-identical —
|
||||
the old split-teaching refusal is gone (phase 70)."""
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/ghost.md"})],
|
||||
@@ -570,6 +730,270 @@ def test_read_unknown_path_refused_echoing_argument(
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # tools stay offered (cap bounds)
|
||||
|
||||
|
||||
# ---------- read/grep: the "did you mean …?" suggestions (phase 72, task 02) ----------
|
||||
|
||||
|
||||
def test_find_path_candidates_exact_suffix_catalog_order(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The pure catalog lookup (monkeypatched ``all_documents`` — one
|
||||
bulk query per call): ``path`` == arg (exact) or a ``/arg`` suffix —
|
||||
catalog order, case-sensitive, as ``(source, path, title)`` triples;
|
||||
a plain substring is NOT a suffix; the result is uncapped (the
|
||||
:data:`~app.rag.agent.SUGGESTION_LIMIT` cap lives in the refusal).
|
||||
"""
|
||||
docs = [
|
||||
_doc("A", "x.md", "Ax", "A"),
|
||||
_doc("A", "shared/x.md", "As", "AS"),
|
||||
_doc("B", "shared/x.md", "Bs", "BS"),
|
||||
_doc("C", "deep/shared/x.md", "Cs", "CS"),
|
||||
_doc("D", "X.md", "Dx", "D"), # case-sensitive: not 'x.md'
|
||||
_doc("E", "nosuffixx.md", "Ex", "E"), # substring, not a /suffix
|
||||
]
|
||||
calls: list[int] = []
|
||||
|
||||
def _all(db: Any) -> list[Document]:
|
||||
calls.append(1)
|
||||
return docs
|
||||
|
||||
monkeypatch.setattr(agent, "all_documents", _all)
|
||||
db = cast("Session", object())
|
||||
|
||||
# Exact bare path ('shared/x.md') plus the deeper suffix
|
||||
# ('deep/shared/x.md' ends with '/shared/x.md') — catalog order.
|
||||
assert agent.find_path_candidates(db, "shared/x.md") == [
|
||||
("A", "shared/x.md", "As"),
|
||||
("B", "shared/x.md", "Bs"),
|
||||
("C", "deep/shared/x.md", "Cs"),
|
||||
]
|
||||
# 'x.md' equals A's path exactly AND suffix-matches the rest — all
|
||||
# four, catalog order (uncapped: the cap is the refusal's).
|
||||
assert agent.find_path_candidates(db, "x.md") == [
|
||||
("A", "x.md", "Ax"),
|
||||
("A", "shared/x.md", "As"),
|
||||
("B", "shared/x.md", "Bs"),
|
||||
("C", "deep/shared/x.md", "Cs"),
|
||||
]
|
||||
# Case-sensitive file paths: 'X.md' matches ONLY D's identically-
|
||||
# cased path (never the lowercase 'x.md' ones), and the plain
|
||||
# substring inside 'nosuffixx.md' is not a suffix.
|
||||
assert agent.find_path_candidates(db, "X.md") == [("D", "X.md", "Dx")]
|
||||
# One bulk query per call (at most one).
|
||||
assert len(calls) == 3
|
||||
|
||||
|
||||
def test_read_bare_path_exact_match_gets_did_you_mean(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The incident shape: an unresolved ``read`` argument containing
|
||||
``/`` that EXACTLY matches one indexed document's ``path`` (the bare
|
||||
path missing the source prefix — the harness prior) gets the
|
||||
``NO_DOCUMENT_DID_YOU_MEAN`` line naming the combined identity —
|
||||
still a refusal: ``read_docs`` empty, nothing counted, tools stay
|
||||
offered."""
|
||||
doc = _doc("Homelab", "active/container_caddy/caddy.md", "Caddy", "CADDY-CONTENT")
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
return (
|
||||
doc
|
||||
if (source, path) == ("Homelab", "active/container_caddy/caddy.md")
|
||||
else None
|
||||
)
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [doc])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "active/container_caddy/caddy.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
agent.NO_DOCUMENT_DID_YOU_MEAN.format(
|
||||
arg="active/container_caddy/caddy.md",
|
||||
source="Homelab",
|
||||
path="active/container_caddy/caddy.md",
|
||||
)
|
||||
)
|
||||
# The rendered line, pinned byte-for-byte.
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'active/container_caddy/caddy.md' — "
|
||||
"did you mean 'Homelab/active/container_caddy/caddy.md'?"
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
|
||||
def test_read_bare_path_suffix_match_gets_did_you_mean(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The suffix form of the same teaching: a path-like argument that
|
||||
matches a deeper indexed path (``active/container_caddy/caddy.md``
|
||||
ends with ``/container_caddy/caddy.md``) names the same combined
|
||||
identity."""
|
||||
doc = _doc("Homelab", "active/container_caddy/caddy.md", "Caddy", "CADDY-CONTENT")
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
return (
|
||||
doc
|
||||
if (source, path) == ("Homelab", "active/container_caddy/caddy.md")
|
||||
else None
|
||||
)
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [doc])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "container_caddy/caddy.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'container_caddy/caddy.md' — "
|
||||
"did you mean 'Homelab/active/container_caddy/caddy.md'?"
|
||||
)
|
||||
|
||||
|
||||
def test_read_bare_path_two_sources_gets_one_of_suggestion(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The same bare path under two sources: the ``one of`` line — up to
|
||||
``SUGGESTION_LIMIT`` combined identities, each single-quoted, joined
|
||||
with ``, `` in catalog order (A before B)."""
|
||||
a = _doc("A", "shared/x.md", "Ax", "A")
|
||||
b = _doc("B", "shared/x.md", "Bx", "B")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [a, b])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(id="call_1", name="read", arguments={"path": "shared/x.md"})
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
agent.NO_DOCUMENT_DID_YOU_MEAN_MANY.format(
|
||||
arg="shared/x.md", candidates="'A/shared/x.md', 'B/shared/x.md'"
|
||||
)
|
||||
)
|
||||
# The rendered line, pinned byte-for-byte.
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'shared/x.md' — did you mean one of: "
|
||||
"'A/shared/x.md', 'B/shared/x.md'?"
|
||||
)
|
||||
|
||||
|
||||
def test_read_bare_path_four_sources_capped_at_three_suggestions(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Four sources sharing the same path: exactly ``SUGGESTION_LIMIT``
|
||||
(3) identities are suggested — catalog order, the fourth dropped."""
|
||||
docs = [_doc(s, "shared/x.md", f"{s}x", s) for s in ("A", "B", "C", "D")]
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: list(docs))
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(id="call_1", name="read", arguments={"path": "shared/x.md"})
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'shared/x.md' — did you mean one of: "
|
||||
"'A/shared/x.md', 'B/shared/x.md', 'C/shared/x.md'?"
|
||||
)
|
||||
assert "'D/shared/x.md'" not in llm.requests[1][0][3]["content"]
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # a refusal
|
||||
|
||||
|
||||
def test_read_bare_filename_without_slash_keeps_no_db_refusal(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The gate is the ``/`` in the argument: a bare FILENAME (no ``/``
|
||||
— e.g. ``caddy.md``) is a bare name for the lookup — today's
|
||||
refusal byte-identical, and NO ``find_document`` / ``all_documents``
|
||||
call (the same no-DB-lookup invariant as a bare source name)."""
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("no DB lookup for a bare (no '/') argument")
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
monkeypatch.setattr(agent, "all_documents", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "caddy.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'caddy.md' — check the ls output."
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Dedupe precedence: the in-context dedupe fires on the SPLIT pair
|
||||
of the argument — the bare path of an in-context document
|
||||
(``read('app/rag/importer.py')`` with ``sample/app/rag/importer.py``
|
||||
seeded) is NOT that pair, so it is not a dedupe: it gets the
|
||||
suggestion line naming the combined identity, and the model's next,
|
||||
correctly-formed call is then deduped as ALREADY_IN_CONTEXT."""
|
||||
seed = [_doc("sample", "app/rag/importer.py", "Importer", "IMPORTER")]
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
return seed[0] if (source, path) == ("sample", "app/rag/importer.py") else None
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: list(seed))
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "app/rag/importer.py"},
|
||||
)
|
||||
],
|
||||
[
|
||||
# Round 2: the corrected call (the suggested combined
|
||||
# identity) — the seed document is already in context, so it
|
||||
# dedupes.
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
arguments={"path": "sample/app/rag/importer.py"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # both refused
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'app/rag/importer.py' — "
|
||||
"did you mean 'sample/app/rag/importer.py'?"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("arguments", "label"),
|
||||
[
|
||||
@@ -806,7 +1230,12 @@ def test_grep_scoped_combined_path_with_nested_path(
|
||||
|
||||
|
||||
def test_grep_scoped_missing_document_refused(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A scoped ``grep`` miss that matches NO indexed document's ``path``
|
||||
(zero candidates — the phase-72 lookup runs, finds nothing) keeps
|
||||
today's line byte-identical: a refusal (not counted), tools stay
|
||||
offered."""
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
@@ -826,6 +1255,44 @@ def test_grep_scoped_missing_document_refused(monkeypatch: pytest.MonkeyPatch) -
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
|
||||
def test_grep_scoped_missing_path_like_doc_gets_did_you_mean(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The same teaching on the scoped ``grep`` miss: an unresolved
|
||||
path-like scope that matches an indexed document's path gets the
|
||||
``NO_DOCUMENT_DID_YOU_MEAN`` suggestion line (a refusal — not
|
||||
counted, no context added, tools stay offered); the whole-KB grep is
|
||||
untouched (no ``path`` argument → no scoped resolution at all).
|
||||
"""
|
||||
doc = _doc("Homelab", "active/container_caddy/caddy.md", "Caddy", "CADDY-CONTENT")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [doc])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="grep",
|
||||
arguments={
|
||||
"pattern": "needle",
|
||||
"path": "active/container_caddy/caddy.md",
|
||||
},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
agent.NO_DOCUMENT_DID_YOU_MEAN.format(
|
||||
arg="active/container_caddy/caddy.md",
|
||||
source="Homelab",
|
||||
path="active/container_caddy/caddy.md",
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == [] # a refusal
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
|
||||
def test_grep_scoped_bare_source_name_refused_without_db(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
@@ -1080,8 +1547,10 @@ def test_zero_max_rounds_is_one_request_without_tools() -> None:
|
||||
def test_rejected_read_spam_runs_to_round_cap(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Every call rejected (unknown document — "No document at …"):
|
||||
rejections no longer end the loop early via budgets — the round cap
|
||||
bounds them and forces the final no-tools answer."""
|
||||
bounds them and forces the final no-tools answer. Zero candidates
|
||||
(empty catalog) → the pre-phase-72 line, byte-identical."""
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/ghost.md"})],
|
||||
|
||||
@@ -10,6 +10,12 @@ And the phase-71 deflection plain-text line (owner-permitted
|
||||
2026-09-03): the LOW prompt = pre-phase text + exactly the one new
|
||||
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
|
||||
the line never leaks into the HIGH prompt.
|
||||
|
||||
And the phase-72 ``<tools>`` copy: the document-identity contract is
|
||||
stated up front (the ``ls`` source-name scope, the combined
|
||||
``source/path`` identity for ``read``/``grep``) — the same contract
|
||||
the teaching refusals in :mod:`app.rag.agent` re-state; the
|
||||
``<tools>`` marker keying (HIGH only) is unchanged.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -170,13 +176,14 @@ def test_tools_section_markers_and_new_tool_names() -> None:
|
||||
|
||||
|
||||
def test_tools_section_teaches_the_harness_shapes() -> None:
|
||||
"""Copy pins: ``ls``'s phase-63 catalog-line format (and its
|
||||
optional one-source scope), ``grep``'s case-insensitive exact-string
|
||||
locator contract (up to 20 ``source/path:line: text`` lines, a
|
||||
locator not a context-adder), and ``read``'s combined
|
||||
``source/path`` + full content."""
|
||||
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
|
||||
case-insensitive exact-string locator contract (up to 20
|
||||
``source/path:line: text`` lines, a locator not a context-adder),
|
||||
and ``read``'s combined ``source/path`` + full content. (Phase 72:
|
||||
the source-name scope clause and the combined-identity clause are
|
||||
pinned byte-for-byte in
|
||||
:func:`test_tools_section_phase72_contract_clauses`.)"""
|
||||
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
|
||||
assert "pass a source name as `path`" in TOOLS_SECTION
|
||||
assert "case-insensitive" in TOOLS_SECTION
|
||||
assert "up to 20" in TOOLS_SECTION
|
||||
assert "source/path:line: text" in TOOLS_SECTION
|
||||
@@ -186,6 +193,66 @@ def test_tools_section_teaches_the_harness_shapes() -> None:
|
||||
assert "Answer as soon as you have what you need" in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_phase72_contract_clauses() -> None:
|
||||
"""Phase 72: the two contract clauses the teaching refusals
|
||||
re-state after the fact, pinned byte-for-byte in the constant —
|
||||
the ``ls`` source-name clause (its optional ``path`` is a source
|
||||
name, not a directory or file path; omit it to list every
|
||||
document) and the ``read``/``grep`` combined-identity clause
|
||||
(the combined ``source/path`` string exactly as shown in the
|
||||
``ls`` output, *including the source name*; a bare document path
|
||||
will not resolve)."""
|
||||
# The ls source-name clause.
|
||||
assert (
|
||||
"a source name (e.g. 'homelab'), not a directory or file "
|
||||
"path — omit it to list every document"
|
||||
) in TOOLS_SECTION
|
||||
# The read combined-identity clause.
|
||||
assert (
|
||||
"combined `source/path` string, exactly as shown in the `ls` "
|
||||
"output — including the source name"
|
||||
) in TOOLS_SECTION
|
||||
# The bare-path note: read clause AND grep clause (exactly twice).
|
||||
assert TOOLS_SECTION.count(
|
||||
"a bare document path (without the source name) will not resolve"
|
||||
) == 2
|
||||
# The pre-phase-70 scope wording is gone — replaced by the
|
||||
# explicit source-name contract.
|
||||
assert "pass a source name as `path`" not in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
||||
"""Phase 72: the contract clauses ride the HIGH prompt with the
|
||||
rest of the section and never leak into the LOW/deflection prompt
|
||||
(whose byte-identity is pinned in
|
||||
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
high = build_high_prompt([doc])
|
||||
assert "<tools>" in high
|
||||
assert "not a directory or file path" in high
|
||||
assert "including the source name" in high
|
||||
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
|
||||
assert "<tools>" not in low
|
||||
assert "not a directory or file path" not in low
|
||||
assert "including the source name" not in low
|
||||
|
||||
|
||||
def test_documents_section_has_no_leading_intro() -> None:
|
||||
"""Phase 72, task 05 (gate iterations 2-3, reverted): the
|
||||
``<documents>`` section must NOT lead with an in-context reminder
|
||||
or name the ``<document>`` blocks — the live telemetry showed that
|
||||
copy primed the model to latch the seed documents' paths as
|
||||
``ls`` scopes (the incident turn regressed to a cap-reached loop
|
||||
on run 2 and re-trapped on run 5), and the reminder never flipped
|
||||
the seed-doc ``read``s (15/15 across gate runs 1-5). The section
|
||||
is exactly the document blocks again."""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
high = build_high_prompt([doc])
|
||||
i_open = high.index("<documents>")
|
||||
i_block = high.index('<document source="Homelab"')
|
||||
assert high[i_open : i_block] == "<documents>\n" # no intro line
|
||||
|
||||
|
||||
def test_tools_section_old_names_and_budget_copy_gone() -> None:
|
||||
"""The phase-37/68 tool names and the phase-37 per-tool budget line
|
||||
(phase 45: the round cap is the bound — the prompt does not
|
||||
|
||||
Reference in New Issue
Block a user