Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)

Browse Source
This commit is contained in:
ducoterra
2026-09-03 11:17:47 -04:00
parent 16f1cfbcaf
commit 801639efcc
55 changed files with 4031 additions and 1466 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+66 -52
View File
@@ -56,25 +56,30 @@ Implements just enough of the aipi surface:
the echo targets the block itself; its tail still includes the
closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document
tools) **and** the system prompt carries the ``<tools>`` section ->
the deterministic SINGLE-READ tool flow, discriminated statelessly
from the messages (the ``tools`` parameter gates the list/read
steps — a no-tools request with no tool results is not the flow):
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
SINGLE-READ tool flow, discriminated statelessly from the messages
(the ``tools`` parameter gates the list/read steps — a no-tools
request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``list_documents`` (synthetic id
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
content;
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
arguments), ``finish_reason: "tool_calls"``, no content;
* request 2 (a ``tool``-role catalog result in the messages):
parse the FIRST catalog line (``source: X | path: Y | title: Z``
— the labeled ``source:`` / ``path:`` fields, phase 63) and
stream a ``tool_calls`` delta calling ``read_document`` on it
(id ``call_1``);
* request 3 (a ``tool``-role read result in the messages): a
content answer, deterministic: ``Read <source/path>. <first 80
chars of the read document's content>`` — so a suite can assert
the read document reached the model and landed in the answer.
Reached regardless of the ``tools`` parameter (phase 45 keeps
the tools offered until the round cap).
stream a ``tool_calls`` delta calling ``read`` on the JOINED
combined ``source/path`` (the mock joins the two labeled fields
— the catalog format is unchanged, so this join is the only
parse change, phase 70) (id ``call_1``);
* request 3 (a ``tool``-role read result in the messages —
content starting with the agent's ``"Document <source/path>:"``
header): a content answer, deterministic: ``Read
<source/path>. <first 80 chars of the read document's
content>`` — so a suite can assert the read document reached
the model and landed in the answer. Reached regardless of the
``tools`` parameter (phase 45 keeps the tools offered until the
round cap).
The single-read flow stops at ONE read result; the MULTI-READ
variant below reads two.
- user message containing BOTH ``use your tools`` AND ``read two
@@ -82,15 +87,18 @@ Implements just enough of the aipi surface:
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix):
* 0 read results, no catalog yet: ``list_documents`` (id
``call_0``);
* 0 read results, catalog present: ``read_document`` on the FIRST
catalog line (id ``call_1``);
* 1 read result: ``read_document`` on the SECOND catalog line —
the first listing line whose ``source/path`` differs from the
one already read (id ``call_2``); a one-document catalog
degenerates to the single-read answer (nothing second to read);
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
flow on the harness-aligned names — ``ls``, then ``read`` on the
JOINED combined ``source/path`` of each catalog line:
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
* 0 read results, catalog present: ``read`` on the JOINED
combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
* 1 read result: ``read`` on the JOINED combined ``source/path``
of the SECOND catalog line — the first listing line whose
``source/path`` differs from the one already read (id
``call_2``); a one-document catalog degenerates to the
single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
<sp1> and <sp2>.`` naming both read paths in read order — so a
@@ -101,12 +109,13 @@ Implements just enough of the aipi surface:
``E2E_REAL_LLM=1`` ignores the mock entirely (the real model does
what it does).
- user message containing ``search your documents``
(``SEARCH_TRIGGER``, phase 68, search tool) **and** the system
prompt carries the ``<tools>`` section -> the deterministic SEARCH
tool flow, discriminated statelessly from the messages (streaming
only):
(``SEARCH_TRIGGER``, phase 68 search tool — renamed to the
harness-aligned ``grep`` in phase 70, same match/output contract)
**and** the system prompt carries the ``<tools>`` section -> the
deterministic SEARCH tool flow, discriminated statelessly from the
messages (streaming only):
* request 1 (``tools`` offered, no search result yet): stream
ONLY ``tool_calls`` deltas — ``search_documents`` with
ONLY ``tool_calls`` deltas — ``grep`` with
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
* request 2 (a ``tool``-role search result in the messages —
recognizable by its ``source/path:line: text`` match lines or
@@ -265,11 +274,12 @@ END_OF_NOTES_TRIGGER = "show the end of your notes"
#: phase-24 tail echo targets the block, not the raw message tail).
_DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
#: Phase 37 (agent-document-tools story): a user message containing this
#: substring (case-insensitive) — combined with the ``<tools>`` section
#: in the system prompt — drives the deterministic tool flow documented
#: in the module docstring (list_documents → read_document on the first
#: catalog line → the quoted answer). Existing E2E questions do not
#: Phase 37 (agent-document-tools story; phase 70: the flow emits the
#: harness-aligned names): a user message containing this substring
#: (case-insensitive) — combined with the ``<tools>`` section in the
#: system prompt — drives the deterministic tool flow documented in the
#: module docstring (ls → read on the first catalog line's combined
#: ``source/path`` → the quoted answer). Existing E2E questions do not
#: contain the phrase, so every other suite is unaffected.
TOOLS_TRIGGER = "use your tools"
@@ -282,11 +292,12 @@ TOOLS_TRIGGER = "use your tools"
#: so the 3-step flow is untouched.
MULTI_READ_TRIGGER = "read two documents"
#: Phase 68 (search tool, TODO.md L4): a user message containing this
#: substring (case-insensitive) — combined with the ``<tools>`` section
#: in the system prompt — drives the deterministic SEARCH tool flow
#: (search_documents for ``SEARCH_PATTERN`` → the "Found …" answer),
#: documented in the module docstring. Checked BEFORE ``TOOLS_TRIGGER``
#: Phase 68 (search tool, TODO.md L4; phase 70: renamed to the
#: harness-aligned ``grep``): a user message containing this substring
#: (case-insensitive) — combined with the ``<tools>`` section in the
#: system prompt — drives the deterministic SEARCH tool flow (grep for
#: ``SEARCH_PATTERN`` → the "Found …" answer), documented in the module
#: docstring. Checked BEFORE ``TOOLS_TRIGGER``
#: (the more specific phrase wins — the same convention as
#: ``THINK_PARAS_TRIGGER``); verified 2026-09-01: no existing E2E
#: question or fixture file contains the phrase, so every other suite
@@ -402,11 +413,11 @@ def _chat_dead(key: str, dead_attempts: int) -> bool:
return _bump_fail(key) <= dead_attempts * _HTTPS_PER_DEAD_ATTEMPT
#: The agent's ``read_document`` tool-result prefix (app.rag.agent
#: The agent's ``read`` tool-result prefix (app.rag.agent
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
_READ_RESULT_PREFIX = "Document "
#: One line of the agent's ``list_documents`` output (app.rag.agent
#: One line of the agent's ``ls`` output (app.rag.agent
#: ``_execute_tool``, phase 63): labeled, pipe-delimited fields —
#: ``source: X | path: Y | title: Z`` — unambiguous for LLM parsing even
#: when the path contains ``/`` characters.
@@ -439,7 +450,7 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
"""Every ``(source, path)`` in the catalog tool result, in listing order.
Catalog lines are ``source: X | path: Y | title: Z`` (the agent's
``list_documents`` output — phase 63: labeled, pipe-delimited
``ls`` output — phase 63: labeled, pipe-delimited
fields, unambiguous even for paths full of ``/``): the line-level
regex recovers the ``source`` and ``path`` fields directly. The
``"N documents:"`` header line matches no line and is skipped;
@@ -460,8 +471,9 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
return docs
#: One line of the agent's ``search_documents`` output (app.rag.agent
#: ``_execute_tool``, phase 68): ``source/path:LINE: text``. The
#: One line of the agent's ``grep`` output (app.rag.agent
#: ``_execute_tool``, phase 68 — phase 70 renamed the tool, the line
#: format is unchanged): ``source/path:LINE: text``. The
#: non-greedy prefix keeps nested paths (``/`` in the path) intact.
_SEARCH_LINE_RE = re.compile(r"^(?P<sp>.+?):(?P<line>\d+): (?P<text>.*)$")
@@ -472,7 +484,7 @@ def _search_result_line(body: dict[str, Any]) -> str | None:
A search result is a ``tool``-role message — never a read result
(those start with the agent's ``"Document "`` prefix) — that either
carries ``source/path:LINE: text`` match lines (the agent's
``search_documents`` output, phase 68) or the sentinel pattern
``grep`` output, phase 68) or the sentinel pattern
itself (its no-match line quotes the pattern). Returns the first
match line's ``text`` part (already 200-char-capped server-side),
or the message's first line in the sentinel-only shape, or ``None``
@@ -534,7 +546,9 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
* ``("read", source, path, "call_1")`` — a ``tool``-role catalog
result is in the messages: the model reads its FIRST
``source: X | path: Y | title: Z`` line (the labeled
``source:`` / ``path:`` fields, phase 63).
``source:`` / ``path:`` fields, phase 63), emitted as ``read`` on
the JOINED combined ``source/path`` (phase 70: the mock joins
the two fields — the canonical document identity).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document. Reached
@@ -1054,7 +1068,7 @@ def chat_completions(body: dict[str, Any]) -> Any:
if search_flow is not None:
if search_flow[0] == "search":
stream = _tool_call_stream(
"search_documents", {"pattern": SEARCH_PATTERN}, "call_0"
"grep", {"pattern": SEARCH_PATTERN}, "call_0"
)
else: # "found" — quote the first matched line (80 chars)
answer = _apply_max_tokens(
@@ -1069,16 +1083,16 @@ def chat_completions(body: dict[str, Any]) -> Any:
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
stream = _tool_call_stream("list_documents", {}, "call_0")
stream = _tool_call_stream("ls", {}, "call_0")
elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
# "call_2" for the multi-read second read (phase 45,
# task 02).
# task 02). Phase 70: the harness-aligned shape — one
# combined ``source/path`` argument (the mock joins the
# two catalog fields; the catalog format is unchanged).
stream = _tool_call_stream(
"read_document",
{"source": flow[1], "path": flow[2]},
flow[3],
"read", {"path": f"{flow[1]}/{flow[2]}"}, flow[3]
)
elif flow[0] == "multi_answer":
# Phase 45 (task 02): the multi-read forced answer —
tests/e2e/test_agent_document_tools.py
+21 -17
View File
@@ -9,18 +9,20 @@ MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
does whatever it does with the tools, while this story's gate is the
deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
contains ``use your tools`` **and** the system prompt carries the
``<tools>`` section of the HIGH prompt):
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
harness-aligned names — ``ls`` / ``read`` with the combined
``source/path`` identity):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``, no
arguments, ``finish_reason: "tool_calls"``);
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
``finish_reason: "tool_calls"``);
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
``tool_calls`` delta calling ``read_document`` on the FIRST catalog
line (id ``call_1``);
3. request 3 (no ``tools`` parameter, the read result in the messages) →
the content answer ``Read <source/path>. <first 80 chars of the read
document's content>`` — so the suite can assert the read document
reached the model and landed in the answer.
``tool_calls`` delta calling ``read`` on the JOINED combined
``source/path`` of the FIRST catalog line (id ``call_1``);
3. request 3 (a ``tool``-role read result in the messages) → the content
answer ``Read <source/path>. <first 80 chars of the read document's
content>`` — so the suite can assert the read document reached the
model and landed in the answer.
KB fixture — reproduces the TODO failure (``aws-route53.md`` references
``example-record-file.json`` "for the exact JSON shape of
@@ -43,10 +45,11 @@ reseelink.json" but does not include it):
Test → story mapping (Playwright Mapping Rule):
1. ``test_marker_question_lists_reads_and_quotes`` — the SSE carries
``tool`` frames (list, then read, ahead of any delta), the UI shows
the "calling tool" label while a tool runs, the bubble shows both
tool lines, the final answer quotes the read document, and the
source chips include the read document (viewer link).
``tool`` frames (``ls``, then ``read`` with the combined path, ahead
of any delta), the UI shows the transient calling-tool status while a
tool runs, the bubble shows both tool lines, the final answer quotes
the read document, and the source chips include the read document
(viewer link).
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
(phase 14) re-renders the tool lines.
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
@@ -385,12 +388,13 @@ def test_marker_question_lists_reads_and_quotes(
assert i_list is not None and i_read is not None, statuses
assert i_list < i_read, statuses
# Wire level: exactly two `tool` frames — list then read — and both
# ahead of the first `delta` frame.
# Wire level: exactly two `tool` frames — ``ls`` then ``read`` (the
# combined source/path as the model passed it) — and both ahead of
# the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ_SP},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
tests/e2e/test_agent_unlimited_tools.py
+24 -18
View File
@@ -9,16 +9,20 @@ MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic MULTI-READ marker flow in ``tests/e2e/mock_llm.py`` (user
message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
carries the ``<tools>`` section of the HIGH prompt):
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
combined ``source/path`` of each catalog line):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read_document`` on
the FIRST catalog line (id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read_document`` on the
SECOND catalog line (id ``call_2``) — the pre-phase-45 per-tool
budgets would have refused exactly this second read (``No reading
budget left — answer with what you have.``);
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
JOINED combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
— the pre-phase-45 per-tool budgets would have refused exactly this
second read (``No reading budget left — answer with what you
have.``);
4. request 4 (two read results) → the forced answer, byte-stable: the
single-read shape quoting the FIRST read result, plus the line
``I read <sp1> and <sp2>.`` naming both read paths in read order.
@@ -393,14 +397,15 @@ def test_multi_read_turn(
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: exactly THREE `tool` frames — list, read #1, read #2,
# in order — and all ahead of the first `delta` frame. This third
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
# (each read's argument is the JOINED combined source/path), in
# order — and all ahead of the first `delta` frame. This third
# frame is the one the pre-phase-45 read budget refused.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ1_SP},
{"type": "tool", "name": "read_document", "argument": READ2_SP},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ1_SP},
{"type": "tool", "name": "read", "argument": READ2_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
@@ -518,7 +523,7 @@ def test_relist_allowed(
# per-tool budgets would have refused (list budget 1, read budget
# 1 — this turn makes one list and TWO reads).
frames = _frames(page)
assert {"type": "tool", "name": "list_documents", "argument": None} in _tool_frames(
assert {"type": "tool", "name": "ls", "argument": None} in _tool_frames(
frames
)
line0 = page.locator(".msg.brain .tool-call").nth(0)
@@ -556,12 +561,13 @@ def test_single_tool_flow_regression(
_submit(page, SINGLE_QUESTION)
_wait_settled(page)
# Exactly TWO tool frames — list then ONE read of the first catalog
# line — no second read (the marker carries no multi-read trigger).
# Exactly TWO tool frames — ls then ONE read of the first catalog
# line (the JOINED combined source/path) — no second read (the
# marker carries no multi-read trigger).
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ1_SP},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ1_SP},
]
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
tests/e2e/test_harness_aligned_tools.py
+524
View File
@@ -0,0 +1,524 @@
"""Phase 70 E2E (Playwright, mock-only): the harness-aligned tool surface
(``ls`` / ``read(path)`` / ``grep(pattern, path?)``).
Story: ``.agent/user_stories/agent-document-tools.md`` (phase 70 reshapes
the tools that story delivered — owner decision 2026-09-03: "match
existing harnesses as much as possible", the pi.dev tool shapes).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic marker flows in ``tests/e2e/mock_llm.py`` (phase 70: the
flows emit the NEW names with the NEW argument shapes):
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
prompt's ``<tools>`` section): ``ls`` (id ``call_0``, no arguments) →
``read`` on the JOINED combined ``source/path`` of the first catalog
line (id ``call_1``) → the ``Read <source/path>. <quote>`` answer;
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
(id ``call_0``) → the ``Found <matched line>`` answer.
The combined ``source/path`` string is the canonical document identity:
the mock joins the two labeled catalog fields itself (the catalog
format is unchanged), and the SSE ``tool`` frames carry exactly what the
model "passed" — ``read``'s combined path, ``grep``'s pattern, ``ls``'s
scope or null when unscoped (the phase-70 argument rule).
KB fixtures:
* READ flow — the ``test_agent_document_tools.py`` two-document pair
(TRUNCATE-then-seed): ``Homelab/aws-route53.md`` seeded with one
chunk whose embedding is the mock's own bag-of-words vector (the
marker question cosines ≈0.69 against it, well past the E2E 0.30
threshold, and it FTS-matches too → grounded) and
``Deployments/example-record-file.json`` indexed WITHOUT chunks (the
catalog-first line the mock reads; never in the retrieval context).
* SEARCH flow — the phase-68 fixture (``tests/fixtures/search_docs/``)
imported through the real importer, its line 6 carrying the sentinel
``reese-sentinel-42`` exactly once (``test_search_tool.py`` pattern).
Test → phase mapping (Playwright Mapping Rule):
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
grounded READ turn: the UI shows the ``ls`` line (unscoped "🔎
Listing documents", no argument) then the "📄 Reading <source/path>"
line with the combined path in a ``<code>`` element, the answer
streams and quotes the read document, the done-state sources
include the read document, and NO raw tool markup (``<|…|>``,
``tool_call``) appears anywhere in the DOM — the live incident this
phase fixes.
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
matched-line answer.
3. ``test_wire_argument_rule_across_both_flows`` — the SSE wire across
BOTH flows in one session: every ``tool`` frame's name is in
{``ls``, ``read``, ``grep``} (no pre-phase-70 name ever reaches the
client) and the argument rule holds — ``read`` → the combined path
as passed, ``grep`` → the pattern, ``ls`` → null when unscoped.
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import time
from datetime import UTC, datetime
from pathlib import Path
from threading import Thread
from typing import Any
from playwright.sync_api import Page, expect
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from tests.e2e.mock_llm import SEARCH_PATTERN, embed_text
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "search_docs"
# --------------------------------------------------------------------------
# READ flow — the two-document pair (cf. test_agent_document_tools.py)
# --------------------------------------------------------------------------
SEED_SOURCE = "Homelab"
SEED_PATH = "aws-route53.md"
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
READ_SOURCE = "Deployments"
READ_PATH = "example-record-file.json"
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
#: The retrievable document (the grounded seed context): the repeated
#: record-file lines carry the marker question's key tokens — verified
#: ≈0.69 cosine against the mock's embeddings (E2E threshold 0.30) plus
#: FTS hits.
ROUTE53_CONTENT = (
"# AWS Route 53 Notes\n\n"
"## Record file\n\n"
+ (
"The aws route53 hosted zone for reeselink keeps every record in "
"reseelink.json — the exact JSON shape of reeselink.json is "
"documented in example-record-file.json.\n"
)
* 10
+ "\n## Sync job\n\n"
"A cron job pushes reeselink.json to the aws route53 hosted zone "
"every fifteen minutes; the diff is applied through the route53 api.\n"
)
#: The read document (the catalog-first line the mock reads; no chunks,
#: so retrieval never puts it in context). Its FIRST line is longer than
#: 80 chars, so the mock's first-80-chars quote is newline-free.
RECORD_FILE_CONTENT = (
'{ "version": 3, "comment": "ReeseLink hosted zone records — the exact '
'JSON shape of reeselink.json",\n'
' "hosted_zone_id": "Z0RESEELINK01",\n'
' "record_sets": [\n'
' { "name": "www.reeselink.example", "type": "A", "ttl": 300,\n'
' "resource_records": [ { "value": "10.0.0.20" } ] },\n'
' { "name": "api.reeselink.example", "type": "CNAME", "ttl": 300,\n'
' "resource_records": [ { "value": "www.reeselink.example" } ] }\n'
" ]\n"
"}\n"
)
assert "\n" not in RECORD_FILE_CONTENT[:80] # the quote must stay one line
#: Carries ``TOOLS_TRIGGER`` (and nothing else — no multi-read, no
#: search, no other mock marker).
READ_QUESTION = (
"Use your tools: what is the exact JSON shape of reeselink.json "
"for my aws route53 hosted zone?"
)
for _other in (
"read two documents",
"search your documents",
"write a long answer",
"think in paragraphs",
"think out loud",
"show the end of your notes",
"show me a table",
"fail then answer",
"always fail",
"embed fail once",
"pretend to think slowly",
):
assert _other not in READ_QUESTION.lower(), _other
READ_ANSWER_PREFIX = f"Read {READ_SP}."
READ_ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
def _seed_read_pair(db: Session) -> None:
"""The two-document READ-flow KB (see the module docstring)."""
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
full_path=f"/tmp/{SEED_PATH}",
title="AWS Route 53 Notes",
content=ROUTE53_CONTENT,
content_hash=hashlib.sha256(ROUTE53_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
db.add(md)
db.flush()
# One chunk carrying the mock's own embedding → genuine token
# overlap between the marker question and this document (the only
# retrievable document).
db.add(
Chunk(
document_id=md.id,
position=0,
content=ROUTE53_CONTENT,
embedding=embed_text(ROUTE53_CONTENT),
)
)
db.add(
Document(
source=READ_SOURCE,
path=READ_PATH,
full_path=f"/tmp/{READ_PATH}",
title="Example Record File",
content=RECORD_FILE_CONTENT,
content_hash=hashlib.sha256(RECORD_FILE_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
)
# --------------------------------------------------------------------------
# SEARCH flow — the phase-68 fixture (cf. test_search_tool.py)
# --------------------------------------------------------------------------
SEED_SOURCE_S = "search_docs"
SEED_PATH_S = "reese-notes.md"
SEED_SP_S = f"{SEED_SOURCE_S}/{SEED_PATH_S}"
#: The fixture's sentinel line (line 6) — the mock's grep matches it
#: exactly once; its ``text`` part is what the "Found …" answer quotes.
SENTINEL_LINE = f"The offsite vault passphrase marker is {SEARCH_PATTERN}."
FOUND_ANSWER = f"Found {SENTINEL_LINE[:80]}"
#: Carries ``SEARCH_TRIGGER`` and is on-topic (cosine ≈0.51 against the
#: fixture + FTS hits → HIGH gate, the ``<tools>`` section rides along).
SEARCH_QUESTION = (
"Search your documents for the vault passphrase marker in my homelab "
"kubernetes backup notes?"
)
assert SEARCH_PATTERN.lower() not in SEARCH_QUESTION.lower()
def _pin_fixture() -> None:
"""The fixture carries the sentinel on line 6, exactly once."""
content = (FIXTURES / SEED_PATH_S).read_text(encoding="utf-8")
lines = content.split("\n")
assert lines[5] == SENTINEL_LINE, lines[5]
assert sum(SEARCH_PATTERN in line for line in lines) == 1
# --------------------------------------------------------------------------
# DB seeding (TRUNCATE-then-seed / TRUNCATE-then-import)
# --------------------------------------------------------------------------
async def _import_search_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {
"_env_file": None,
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return await import_sources([FIXTURES], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread.
Playwright's sync API keeps an asyncio loop running on the test
thread, so ``asyncio.run`` cannot be called directly from a test
body (the established house helper).
"""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _reset_db_read_pair() -> None:
"""Truncate the KB (plus the prompt-shaping tables), then seed the
two-document READ-flow pair. ``steering_notes`` / ``kb_overview``
are truncated too, so the HIGH prompt is exactly ``<relevance>`` +
``<documents>`` + ``<tools>`` — byte-stable prompts, byte-stable
answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
)
db.commit()
_seed_read_pair(db)
db.commit()
def _reset_db_search_fixture(mock_port: int) -> None:
"""Truncate the KB (plus the prompt-shaping tables), then import the
phase-68 search fixture through the real importer."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
)
db.commit()
summary = _run_in_thread(_import_search_fixtures(mock_port))
assert summary is not None and summary.added == 1, summary
# --------------------------------------------------------------------------
# Page helpers (the test_agent_document_tools.py pattern)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_sse_hook(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _drain_frames(page: Page) -> list[dict]:
"""One turn's SSE frames: wait for that turn's ``done`` frame, then
return EVERY frame captured since the last drain (the hook's
background read appends the whole stream at once after it closes, so
clearing-and-reading is race-free per turn)."""
deadline = time.monotonic() + 10.0
while True:
raw = page.evaluate(
"() => { const f = window.__sseFrames || []; "
"window.__sseFrames = []; return f; }"
)
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered.
Phase 48: the label assertion carries the settle wait with an
explicit timeout — the in-flight button is the enabled Stop control
(never disabled), so ``to_be_enabled`` no longer blocks until the
turn settles."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
# --------------------------------------------------------------------------
# 1. The grounded READ turn: ls line → Reading line → quoted answer,
# sources include the read doc, no raw tool markup anywhere in the DOM
# --------------------------------------------------------------------------
def test_read_flow_lines_answer_sources_no_raw_markup(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db_read_pair()
page.goto(app_url)
_install_sse_hook(page)
_submit(page, READ_QUESTION)
_wait_settled(page)
# The UI shows the ls line (UNSCOPED — no argument, no <code>) then
# the "📄 Reading <source/path>" line with the COMBINED path in a
# <code> element (the path is data, never markup).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(READ_SP)
# The answer streamed and quotes the read document (the mock's
# deterministic echo: "Read <source/path>. <first 80 chars>").
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
# Wire level: ls then read — the phase-70 argument rule (ls
# unscoped → null; read → the combined path as passed) — ahead of
# the first delta.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# Done-state sources include the read document (retrieval doc first,
# the agent's read doc after — the phase-37 extension contract).
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH),
(READ_SOURCE, READ_PATH),
]
# The live incident this phase fixes: NO raw tool markup anywhere in
# the DOM — the model's trained wire shapes (<|tool_call_…|>,
# "tool_calls", finish_reason) must never leak into the rendered
# conversation.
dom = page.locator("#messages").inner_html()
for raw in ("<|", "tool_call", "tool_calls", "finish_reason"):
assert raw not in dom, f"raw tool markup {raw!r} leaked into the DOM"
# --------------------------------------------------------------------------
# 2. The grounded SEARCH turn: the "🔎 Searching for <pattern>" line,
# then the matched-line answer
# --------------------------------------------------------------------------
def test_grep_flow_line_then_answer(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
_pin_fixture()
page.set_default_timeout(30_000)
_reset_db_search_fixture(mock_llm)
page.goto(app_url)
_install_sse_hook(page)
_submit(page, SEARCH_QUESTION)
_wait_settled(page)
# ONE tool line above the answer: "🔎 Searching for " + the sentinel
# in a <code> element (the pattern is data, never markup).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text("Searching for")
expect(lines.nth(0).locator("code")).to_have_text(SEARCH_PATTERN)
# The answer quotes the MATCHED LINE — the grep result reached the
# model and landed in the answer (the mock's deterministic echo).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(FOUND_ANSWER)
# Wire level: exactly ONE tool frame — grep carrying the PATTERN as
# its argument (the phase-70 argument rule) — ahead of the first
# delta; the turn is grounded.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# --------------------------------------------------------------------------
# 3. The SSE wire across BOTH flows: every tool frame carries a
# phase-70 name and the single-string argument rule
# --------------------------------------------------------------------------
def test_wire_argument_rule_across_both_flows(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
_pin_fixture()
page.set_default_timeout(30_000)
_reset_db_read_pair()
page.goto(app_url)
_install_sse_hook(page)
# Turn 1 — the READ flow (ls → read on the combined path).
_submit(page, READ_QUESTION)
_wait_settled(page)
read_frames = _drain_frames(page)
# Turn 2 — re-seed the search fixture, then the SEARCH flow (grep
# for the sentinel). The app's chat path is single-turn (system +
# user message), so the first turn cannot influence this one.
_reset_db_search_fixture(mock_llm)
_submit(page, SEARCH_QUESTION)
_wait_settled(page)
search_frames = _drain_frames(page)
read_tools = _tool_frames(read_frames)
search_tools = _tool_frames(search_frames)
# The ordered, combined tool-frame sequence across both flows: the
# argument rule end-to-end — read → the combined path as passed,
# grep → the pattern, ls → null when unscoped.
assert read_tools + search_tools == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ_SP},
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
]
# No pre-phase-70 name ever reaches the client.
for frame in read_tools + search_tools:
assert frame["name"] in {"ls", "read", "grep"}, frame
assert frame["argument"] is None or isinstance(frame["argument"], str)
# And both turns answered (neither flow stalled at a tool round).
assert next(f for f in read_frames if f["type"] == "done")["deflected"] is False
assert next(f for f in search_frames if f["type"] == "done")["deflected"] is False
tests/e2e/test_search_tool.py
+17 -15
View File
@@ -1,4 +1,6 @@
"""Phase 68 E2E (Playwright, mock-only): the ``search_documents`` tool.
"""Phase 68 E2E (Playwright, mock-only): the ``grep`` tool (the
phase-68 search tool, renamed to the harness-aligned ``grep`` in
phase 70; the A5 match/output contract is unchanged).
Story: n/a (TODO-derived — the owner roadmap confirmation 2026-09-01,
TODO.md L4: "Add a search tool that allows the LLM to grep through the
@@ -14,8 +16,8 @@ message contains ``search your documents`` (``SEARCH_TRIGGER``)
prompt):
1. request 1 (``tools`` offered, no search result yet) → streams ONLY
``tool_calls`` deltas calling ``search_documents`` with
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
``tool_calls`` deltas calling ``grep`` with ``{"pattern":
SEARCH_PATTERN}`` (id ``call_0``);
2. request 2 (a ``tool``-role search result — the
``source/path:line: text`` match line) → the content answer
``Found <first matched line's content up to 80 chars>`` — so this
@@ -40,13 +42,13 @@ shadow the phase-37/45 flows and vice versa).
Test → phase mapping:
1. ``test_search_flow_searches_and_answers_from_match`` — the live
search flow: the SSE carries the ``tool`` frame
(``search_documents`` with ``argument = <sentinel>``, ahead of any
delta), #send-status recorded the transient "… is searching for
<sentinel>" state, the bubble shows ONE ``🔎 Searching for``
tool line with the sentinel in a ``<code>`` element, the answer
quotes the matched line (``Found …`` — the match reached the
model), and the turn settles to idle with no error banner.
search flow: the SSE carries the ``tool`` frame (``grep`` with
``argument = <sentinel>``, ahead of any delta), #send-status
recorded the transient "… is searching for <sentinel>" state, the
bubble shows ONE ``🔎 Searching for`` tool line with the sentinel in
a ``<code>`` element, the answer quotes the matched line
(``Found …`` — the match reached the model), and the turn settles to
idle with no error banner.
2. ``test_search_adds_no_source_by_itself`` — context accounting
(locked A5): the search-only flow (no read) leaves
``done.sources`` / the source chips / ``query_log.sources`` at the
@@ -367,12 +369,12 @@ def test_search_flow_searches_and_answers_from_match(
)
assert i_think is not None and i_think < i_search, statuses
# Wire level: exactly ONE `tool` frame — search_documents carrying
# the PATTERN as its argument (phase 68 task 02) — ahead of the
# first `delta` frame.
# Wire level: exactly ONE `tool` frame — grep carrying the PATTERN
# as its argument (phase 68 task 02; phase 70 renamed the tool) —
# ahead of the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "search_documents", "argument": SEARCH_PATTERN}
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
@@ -424,7 +426,7 @@ def test_search_adds_no_source_by_itself(
# baseline: the one fixture doc, nothing added by the search.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "search_documents", "argument": SEARCH_PATTERN}
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
tests/integration/test_agent_tools.py
+182 -135
View File
@@ -1,17 +1,19 @@
"""Integration: the agent DB accessors against real Postgres (phase 37).
"""Integration: the agent DB accessors against real Postgres (phase 37;
the harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
``list_catalog`` must order rows by ``(source, path)`` — the same order as
``GET /api/docs`` — and ``find_document`` must resolve a hit to the full
document row (content included, for the never-truncated read) and return
``None`` for unknown ``source``/``path`` pairs. Phase 68: the
``search_documents`` tool is pinned here too — its locked parameter
shape in ``AGENT_TOOLS``, and a scripted ``ToolCallPiece`` executed
through ``run_agent`` against the real DB (``all_documents`` for a
whole-KB search, ``find_document`` for a scoped one). The
combined-form self-correction (a ``source`` argument carrying
``source/path``) is pinned here as well, through ``run_agent``:
the split read executes against the real table, and a still-unknown
split gets the educational refusal.
``list_catalog`` must order rows by ``(source, path)`` — the same order
as ``GET /api/docs`` — ``list_source_names`` must resolve the
registered source names (the scoped ``ls`` join), and ``find_document``
must resolve a hit to the full document row (content included, for the
never-truncated read) and return ``None`` for unknown pairs. Phase 70:
the ``ls``/``read``/``grep`` tools are pinned here too — the locked
parameter shape in ``AGENT_TOOLS``, and scripted ``ToolCallPiece``s
executed through ``run_agent`` against the real DB: ``ls`` scoped to a
registered source name (unknown name → refusal), ``read`` on the
canonical combined ``source/path`` form (first-slash split; a bare
source name and an unknown identity get the no-document refusal), and
``grep`` (``all_documents`` for a whole-KB search, ``find_document`` for
a scoped one).
Requires: podman compose up -d db
"""
@@ -24,11 +26,11 @@ from copy import deepcopy
from typing import Any, cast
import pytest
from sqlalchemy import text
from sqlalchemy import delete, text
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import Document
from app.models import Document, GitSource
from app.rag import agent
from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
@@ -58,6 +60,19 @@ def kb(db) -> Iterator[None]:
db.commit()
@pytest.fixture()
def src(db) -> Iterator[GitSource]:
"""One registered git source — the scoped ``ls`` source-name check
reads the real registry, so the row is inserted and deleted around
the tests (``repo_name`` resolves the URL to ``Homelab``)."""
row = GitSource(url="https://github.com/reese/Homelab.git", kind="git")
db.add(row)
db.commit()
yield row
db.execute(delete(GitSource).where(GitSource.id == row.id))
db.commit()
def test_list_catalog_orders_by_source_then_path(kb, db) -> None:
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
@@ -75,6 +90,23 @@ def test_list_catalog_is_empty_without_rows(kb, db) -> None:
assert agent.list_catalog(db) == []
def test_list_source_names_resolves_registry_rows(db) -> None:
"""The real registry: git names resolve through the import pipeline's
``repo_name`` (trailing ``.git`` stripped); a second row resolving to
the same name (the phase-69 sibling case) is listed once."""
a = GitSource(url="https://github.com/reese/Homelab.git", kind="git")
b = GitSource(url="https://github.com/reese/Homelab", kind="git") # sibling
c = GitSource(url="https://e.com/deployments", kind="git")
db.add_all([a, b, c])
db.commit()
try:
assert agent.list_source_names(db).count("Homelab") == 1 # deduped
assert "deployments" in agent.list_source_names(db)
finally:
db.execute(delete(GitSource).where(GitSource.id.in_([a.id, b.id, c.id])))
db.commit()
def test_find_document_hit_returns_full_row(kb, db) -> None:
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
db.commit()
@@ -97,36 +129,29 @@ def test_find_document_none_for_unknown_pairs(kb, db) -> None:
assert agent.find_document(db, "nope", "nope.md") is None # nothing at all
# ---------- search_documents (phase 68) ----------
# ---------- AGENT_TOOLS surface (phase 70: ls / read / grep) ----------
def test_all_documents_orders_by_source_then_path(kb, db) -> None:
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
db.commit()
docs = agent.all_documents(db)
assert [(d.source, d.path) for d in docs] == [
("Alpha", "c/third.md"),
("Zeta", "a/first.md"),
("Zeta", "b/second.md"),
]
assert [d.content for d in docs] == ["AC", "ZA", "ZB"] # full rows
def test_agent_tools_offers_search_documents_with_locked_shape() -> None:
def test_agent_tools_offers_the_harness_aligned_surface() -> None:
by_name = {t["function"]["name"]: t for t in AGENT_TOOLS}
assert list(by_name) == [ # the third tool, in order
"list_documents",
"read_document",
"search_documents",
assert list(by_name) == [ # the harness order, phase 70
"ls",
"read",
"grep",
]
search = by_name["search_documents"]["function"]["parameters"]
assert search["type"] == "object"
assert search["required"] == ["pattern"]
assert set(search["properties"]) == {"pattern", "source", "path"}
assert all(p["type"] == "string" for p in search["properties"].values())
ls = by_name["ls"]["function"]["parameters"]
assert ls["type"] == "object"
assert ls["required"] == [] # path is optional
assert set(ls["properties"]) == {"path"}
read = by_name["read"]["function"]["parameters"]
assert read["type"] == "object"
assert read["required"] == ["path"]
assert set(read["properties"]) == {"path"}
grep = by_name["grep"]["function"]["parameters"]
assert grep["type"] == "object"
assert grep["required"] == ["pattern"]
assert set(grep["properties"]) == {"pattern", "path"}
assert all(p["type"] == "string" for p in grep["properties"].values())
class ScriptedToolLLM:
@@ -156,26 +181,12 @@ def _settings(**kwargs: Any) -> Settings:
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
def _run_search(
db: Session, arguments: dict[str, Any]
def _run_call(
db: Session, name: str, arguments: dict[str, Any]
) -> tuple[AgentHolder, ScriptedToolLLM]:
"""Drive one scripted ``search_documents`` call through ``run_agent``."""
"""Drive one scripted tool call through ``run_agent``."""
holder = AgentHolder()
llm = ScriptedToolLLM(
ToolCallPiece(id="call_1", name="search_documents", arguments=arguments)
)
asyncio.run(_consume(llm, db, holder))
return holder, llm
def _run_read(
db: Session, arguments: dict[str, Any]
) -> tuple[AgentHolder, ScriptedToolLLM]:
"""Drive one scripted ``read_document`` call through ``run_agent``."""
holder = AgentHolder()
llm = ScriptedToolLLM(
ToolCallPiece(id="call_1", name="read_document", arguments=arguments)
)
llm = ScriptedToolLLM(ToolCallPiece(id="call_1", name=name, arguments=arguments))
asyncio.run(_consume(llm, db, holder))
return holder, llm
@@ -197,12 +208,113 @@ async def _consume(
return out
def test_search_whole_kb_through_run_agent(kb, db) -> None:
# ---------- ls (scoped through the real registry) ----------
def test_ls_scoped_to_registered_source_through_run_agent(kb, src, db) -> None:
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
_doc(db, "Other", "b.md", "B", "B-CONTENT")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
# Offered: the first request carries AGENT_TOOLS (the 3-tool list).
assert llm.requests[0][1] == AGENT_TOOLS
# Executed against the real DB: the listing filtered to the source.
assert llm.requests[1][0][3]["content"] == (
"1 documents:\nsource: Homelab | path: a.md | title: A"
)
assert holder.tool_calls == 1
assert holder.read_docs == []
def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Ghost"})
assert (
llm.requests[1][0][3]["content"] == "No source named 'Ghost' — check the ls output."
)
assert holder.tool_calls == 0 and holder.read_docs == []
# ---------- read (the canonical combined source/path form) ----------
def test_read_combined_path_through_run_agent(kb, db) -> None:
"""The combined ``source/path`` identity resolves at the FIRST slash
against the REAL table (a path with further slashes included): the
read executes, the holder records the row, the result header carries
the true source/path."""
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
db.commit()
holder, llm = _run_call(db, "read", {"path": "Alpha/deep/nested/doc.md"})
assert llm.requests[1][0][3]["content"] == (
"Document Alpha/deep/nested/doc.md:\nFULL-TEXT"
)
assert holder.tool_calls == 1
assert holder.read_docs == [created]
def test_read_bare_source_name_refused_through_run_agent(kb, db) -> None:
"""A bare source name (no '/') can never be a document — the
no-document refusal echoing the argument as passed; the old
split-teaching refusal is gone (phase 70)."""
_doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
db.commit()
holder, llm = _run_call(db, "read", {"path": "Alpha"})
assert (
llm.requests[1][0][3]["content"] == "No document at 'Alpha' — check the ls output."
)
assert holder.tool_calls == 0 and holder.read_docs == []
def test_read_unknown_combined_path_refused_through_run_agent(kb, db) -> None:
"""A combined identity that matches nothing gets the no-document
refusal (the argument echoed as passed — the model sees its own
form)."""
_doc(db, "Alpha", "x.md", "X", "X-CONTENT")
db.commit()
holder, llm = _run_call(db, "read", {"path": "Alpha/nope/deep.md"})
assert (
llm.requests[1][0][3]["content"]
== "No document at 'Alpha/nope/deep.md' — check the ls output."
)
assert holder.tool_calls == 0 and holder.read_docs == []
# ---------- grep (the phase-68 A5 contract under the new name) ----------
def test_all_documents_orders_by_source_then_path(kb, db) -> None:
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
db.commit()
docs = agent.all_documents(db)
assert [(d.source, d.path) for d in docs] == [
("Alpha", "c/third.md"),
("Zeta", "a/first.md"),
("Zeta", "b/second.md"),
]
assert [d.content for d in docs] == ["AC", "ZA", "ZB"] # full rows
def test_grep_whole_kb_through_run_agent(kb, db) -> None:
_doc(db, "Beta", "b/two.md", "Two", "no hit\nNEEDLE in two\nlast")
_doc(db, "Alpha", "a/one.md", "One", "first\nneedle in one\nthird")
db.commit()
holder, llm = _run_search(db, {"pattern": "needle"})
holder, llm = _run_call(db, "grep", {"pattern": "needle"})
# Offered: the first request carries AGENT_TOOLS (the 3-tool list).
assert llm.requests[0][1] == AGENT_TOOLS
@@ -212,17 +324,15 @@ def test_search_whole_kb_through_run_agent(kb, db) -> None:
"Beta/b/two.md:2: NEEDLE in two"
)
assert holder.tool_calls == 1
assert holder.read_docs == [] # locked A5: search adds no context
assert holder.read_docs == [] # locked A5: grep adds no context
def test_search_scoped_through_run_agent(kb, db) -> None:
def test_grep_scoped_through_run_agent(kb, db) -> None:
_doc(db, "Alpha", "a/one.md", "One", "first\nNeedle here\nthird")
_doc(db, "Beta", "b/two.md", "Two", "NEEDLE too")
db.commit()
holder, llm = _run_search(
db, {"pattern": "needle", "source": "Alpha", "path": "a/one.md"}
)
holder, llm = _run_call(db, "grep", {"pattern": "needle", "path": "Alpha/a/one.md"})
# Only the named document is searched — the other one's hit is absent.
assert llm.requests[1][0][3]["content"] == "Alpha/a/one.md:2: Needle here"
@@ -230,90 +340,27 @@ def test_search_scoped_through_run_agent(kb, db) -> None:
assert holder.read_docs == []
def test_search_scoped_missing_doc_refused_through_run_agent(kb, db) -> None:
def test_grep_scoped_missing_doc_refused_through_run_agent(kb, db) -> None:
_doc(db, "Alpha", "a/one.md", "One", "nothing")
db.commit()
holder, llm = _run_search(
db, {"pattern": "needle", "source": "Alpha", "path": "ghost.md"}
)
holder, llm = _run_call(db, "grep", {"pattern": "needle", "path": "Alpha/ghost.md"})
assert (
llm.requests[1][0][3]["content"]
== "No document at Alpha/ghost.md — check the list_documents output."
== "No document at 'Alpha/ghost.md' — check the ls output."
)
assert holder.tool_calls == 0 and holder.read_docs == []
# ---------- combined 'source/path' self-correction (read_document) ----------
def test_read_combined_source_self_corrects_through_run_agent(kb, db) -> None:
"""The model's combined 'source' ('Alpha/deep/nested/doc.md') resolves
through the first-slash split against the REAL table: the read
executes, the holder records the row, the result header carries the
true source/path."""
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
db.commit()
holder, llm = _run_read(
db,
{
"source": "Alpha/deep/nested/doc.md",
"path": "deep/nested/doc.md",
},
)
assert llm.requests[1][0][3]["content"] == (
"Document Alpha/deep/nested/doc.md:\nFULL-TEXT"
)
assert holder.tool_calls == 1
assert holder.read_docs == [created]
def test_read_combined_source_later_slash_split_through_run_agent(kb, db) -> None:
"""source='Alpha/deep' + path='nested/doc.md' (a split at a LATER
slash) resolves via the continuation candidate against the real
table."""
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
db.commit()
holder, llm = _run_read(
db, {"source": "Alpha/deep", "path": "nested/doc.md"}
)
assert llm.requests[1][0][3]["content"] == (
"Document Alpha/deep/nested/doc.md:\nFULL-TEXT"
)
assert holder.tool_calls == 1
assert holder.read_docs == [created]
def test_read_combined_source_refusal_teaches_split(kb, db) -> None:
"""A combined source that matches nothing (even split) gets the
educational refusal naming the corrected arguments."""
_doc(db, "Alpha", "x.md", "X", "X-CONTENT")
db.commit()
holder, llm = _run_read(
db, {"source": "Alpha/nope/deep.md", "path": "nope/deep.md"}
)
assert llm.requests[1][0][3]["content"] == (
"source must not contain '/': for 'Alpha/nope/deep.md' call "
"read_document(source='Alpha', path='nope/deep.md')."
)
assert holder.tool_calls == 0 and holder.read_docs == []
def test_search_no_matches_through_run_agent(kb, db) -> None:
def test_grep_no_matches_through_run_agent(kb, db) -> None:
_doc(db, "Alpha", "a/one.md", "One", "nothing matching")
db.commit()
holder, llm = _run_search(db, {"pattern": "zebra"})
holder, llm = _run_call(db, "grep", {"pattern": "zebra"})
assert llm.requests[1][0][3]["content"] == (
"No matches for 'zebra' in the knowledge base."
)
assert holder.tool_calls == 1 # an executed search with zero hits
assert holder.tool_calls == 1 # an executed grep with zero hits
assert holder.read_docs == []
tests/integration/test_api.py
+7 -5
View File
@@ -300,11 +300,12 @@ def test_ui_chrome_has_no_emoji(client, path: str) -> None:
Phase 37 revision (owner permission 2026-08-26, PLAN §4): the agent's
``.tool-call`` line carries the CONTENT marks — 🔎 (list) and 📄
(read) — the only emoji in the whole frontend, and only as the exact
tool-line template strings in app.js. Phase 68 revision: the
``search_documents`` tool line adds the third template literal
("🔎 Searching for "). The guard strips precisely those three
literals; any other emoji, or those marks anywhere else, still
fails."""
tool-line template strings in app.js. Phase 68 revision: the search
tool line (the ``grep`` tool, phase 70) adds the third template
literal ("🔎 Searching for "). Phase 70 revision: the scoped ``ls``
tool line adds the fourth ("🔎 Listing documents in "). The guard strips
precisely those four literals; any other emoji, or those marks
anywhere else, still fails."""
r = client.get(path)
assert r.status_code == 200
text = r.text
@@ -312,6 +313,7 @@ def test_ui_chrome_has_no_emoji(client, path: str) -> None:
text = text.replace('"🔎 Listing documents"', "")
text = text.replace('"📄 Reading "', "")
text = text.replace('"🔎 Searching for "', "")
text = text.replace('"🔎 Listing documents in "', "")
assert _find_emoji(text) == [], f"emoji found in {path}: {_find_emoji(text)!r}"
tests/integration/test_chat_api.py
+94 -31
View File
@@ -22,12 +22,12 @@ from typing import Any
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import func, select, text
from sqlalchemy import delete, func, select, text
from app.api import chat as chat_api
from app.config import Settings, get_settings
from app.main import app as fastapi_app
from app.models import Chunk, QueryLog
from app.models import Chunk, GitSource, QueryLog
from app.rag import agent
from app.rag.agent import AGENT_TOOLS
from app.rag.importer import import_sources
@@ -550,13 +550,13 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
tool_script=[
[
StreamPiece("thinking", "Let me list what is indexed…"),
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
ToolCallPiece(id="call_1", name="ls", arguments={}),
],
[
ToolCallPiece(
id="call_2",
name="read_document",
arguments={"source": "docs", "path": "homelab/backups.md"},
name="read",
arguments={"path": "docs/homelab/backups.md"},
)
],
# the answer request still carries the tools (2 rounds < the
@@ -580,10 +580,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
list_frame, read_frame = frames[1], frames[2]
assert set(list_frame) == {"type", "name", "argument"}
assert list_frame["name"] == "list_documents"
assert list_frame["argument"] is None # the tool takes no parameters
assert list_frame["name"] == "ls"
assert list_frame["argument"] is None # no ``path`` argument was passed
assert set(read_frame) == {"type", "name", "argument"}
assert read_frame["name"] == "read_document"
assert read_frame["name"] == "read"
# Phase 70: the frame's argument is the single string the model
# passed — the combined ``source/path``.
assert read_frame["argument"] == "docs/homelab/backups.md"
deltas = [f for f in frames if f["type"] == "delta"]
@@ -621,28 +623,24 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
assert "'docs/homelab/backups.md'" in lines[-1]
def test_grounded_turn_streams_search_tool_frames(
def test_grounded_turn_streams_grep_tool_frames(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""Phase 68: a scripted ``search_documents`` call streams as
``{type: "tool", name: "search_documents", argument: <pattern>}`` —
"""Phase 68 (renamed ``grep`` in phase 70): a scripted ``grep`` call
streams as ``{type: "tool", name: "grep", argument: <pattern>}`` —
the raw pattern is the frame's ``argument`` (the UI renders the
"searching for" line from it). A non-string pattern — a model error
the backend refuses — yields ``argument: null``. A search adds no
the backend refuses — yields ``argument: null``. A grep adds no
source: ``done.sources`` stays the retrieval docs (locked A5)."""
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1",
name="search_documents",
arguments={"pattern": "Cilium"},
),
ToolCallPiece(id="call_1", name="grep", arguments={"pattern": "Cilium"}),
],
[
ToolCallPiece(
id="call_2",
name="search_documents",
name="grep",
arguments={"pattern": 42}, # model error: non-string
),
],
@@ -659,25 +657,90 @@ def test_grounded_turn_streams_search_tool_frames(
types = [f["type"] for f in frames]
assert "error" not in types
assert len(scripted.seen_tools) == 3 # both searches executed (rounds)
assert len(scripted.seen_tools) == 3 # both greps executed (rounds)
tool_frames = [f for f in frames if f["type"] == "tool"]
assert len(tool_frames) == 2
first, second = tool_frames
assert set(first) == {"type", "name", "argument"}
assert first["name"] == "search_documents"
assert first["name"] == "grep"
assert first["argument"] == "Cilium" # the raw pattern
assert set(second) == {"type", "name", "argument"}
assert second["name"] == "search_documents"
assert second["name"] == "grep"
assert second["argument"] is None # the non-string pattern → null
# The searches still answered: deltas, then a grounded done.
# The greps still answered: deltas, then a grounded done.
assert [f for f in frames if f["type"] == "delta"]
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/backups.md" not in paths # a search adds no source
assert "homelab/backups.md" not in paths # a grep adds no source
def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
) -> None:
"""Phase 70 pins: the frame's ``argument`` is the single string
argument the model passed — an ``ls`` frame carries the scope when
the model gave one (null only when it is omitted, pinned above) —
and frame emission is execution-independent: a rejected call (an
unknown ``read`` path) still streams its frame with the model's
argument as-is. The rejected read adds no source (``done.sources``
stays the retrieval docs), and rejected calls count nothing
(``tool_calls=1`` — only the executed scoped ``ls``)."""
# The scoped ``ls`` source-name check reads the registry — insert a
# row resolving to ``docs`` (the fixture's source name) and delete
# it again afterwards.
src = GitSource(url="https://github.com/reese/docs.git", kind="git")
db.add(src)
db.commit()
try:
scripted = FakeRagLLM(
tool_script=[
[ToolCallPiece(id="call_1", name="ls", arguments={"path": "docs"})],
[
ToolCallPiece(
id="call_2", name="read", arguments={"path": "docs/homelab/nope.md"}
)
],
]
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
caplog.set_level(logging.INFO, logger="app.chat")
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
finally:
db.execute(delete(GitSource).where(GitSource.id == src.id))
db.commit()
types = [f["type"] for f in frames]
assert "error" not in types
# Both calls stream a frame — the rejected read included.
tool_frames = [f for f in frames if f["type"] == "tool"]
assert len(tool_frames) == 2
ls_frame, read_frame = tool_frames
assert set(ls_frame) == {"type", "name", "argument"}
assert ls_frame["name"] == "ls"
assert ls_frame["argument"] == "docs" # the model's scope, as passed
assert set(read_frame) == {"type", "name", "argument"}
assert read_frame["name"] == "read"
# The rejected call's frame still carries the model's argument as
# passed — frame emission is execution-independent.
assert read_frame["argument"] == "docs/homelab/nope.md"
# The rejected read adds no source — done.sources stays retrieval.
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/nope.md" not in paths # the refused read cites nothing
# The rejected call counts nothing — only the executed scoped ls.
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
assert lines and "tool_calls=1" in lines[-1]
def test_deflected_turn_stays_byte_identical_without_tools(
@@ -690,12 +753,12 @@ def test_deflected_turn_stays_byte_identical_without_tools(
``tools`` key."""
scripted = FakeRagLLM(
tool_script=[
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[
ToolCallPiece(
id="call_2",
name="read_document",
arguments={"source": "docs", "path": "homelab/backups.md"},
name="read",
arguments={"path": "docs/homelab/backups.md"},
)
],
[StreamPiece("content", "never used — the agent never runs")],
@@ -743,12 +806,12 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request(
the kill switch survives the phase-45 budget removal."""
scripted = FakeRagLLM(
tool_script=[
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[
ToolCallPiece(
id="call_2",
name="read_document",
arguments={"source": "docs", "path": "homelab/backups.md"},
name="read",
arguments={"path": "docs/homelab/backups.md"},
)
],
]
@@ -799,7 +862,7 @@ def test_tool_execution_db_failure_yields_error_event(
``error`` event as the pre-stream retrieval path — never a severed
stream (the "never stale" contract, PLAN §7.4)."""
scripted = FakeRagLLM(
tool_script=[[ToolCallPiece(id="call_1", name="list_documents", arguments={})]]
tool_script=[[ToolCallPiece(id="call_1", name="ls", arguments={})]]
)
def boom(*_a: Any, **_k: Any) -> Any:
@@ -815,7 +878,7 @@ def test_tool_execution_db_failure_yields_error_event(
# The ``tool`` frame went out first (the model requested the call);
# the failed execution ends the turn with the structured error event.
assert [f["type"] for f in frames] == ["tool", "error"]
assert frames[0]["name"] == "list_documents"
assert frames[0]["name"] == "ls"
assert "offline mid-question" in frames[1]["detail"]
assert db.scalars(select(QueryLog)).all() == [] # no row for a failed turn
tests/integration/test_chats_api.py
+6 -2
View File
@@ -64,8 +64,12 @@ FULL_BRAIN: dict[str, Any] = {
"suggestions": ["What ports does Traefik expose?"],
"thinking": "The kubernetes doc covers the cluster layout…",
"tools": [
{"name": "read_document", "argument": "Homelab/kubernetes.md"},
{"name": "list_documents", "argument": None},
{"name": "read", "argument": "Homelab/kubernetes.md"},
{"name": "ls", "argument": None},
# Saved chats persisting the pre-phase-70 tool names still
# validate — ``name`` is opaque to the API (no migration,
# locked: old chats render fine).
{"name": "read_document", "argument": "Homelab/legacy-notes.md"},
],
"stopped": False,
}
tests/unit/test_agent.py
+839 -812
View File
File diff suppressed because it is too large Load Diff
tests/unit/test_chat_gate.py
+10 -1
View File
@@ -578,10 +578,16 @@ def test_endpoint_grounded_turn_runs_agent_loop_with_tools(
assert not any(f["type"] == "tool" for f in frames)
assert len(llm.seen) == 1
assert llm.seen_tools == [AGENT_TOOLS] # one request, tools offered
# The system prompt is the HIGH prompt with the <tools> instructions.
# The system prompt is the HIGH prompt with the <tools> instructions
# (phase 70: the harness-aligned ls/read/grep copy — new names in,
# old phase-37/68 names out).
(system, _user) = llm.seen[0][0], llm.seen[0][1]
assert "<relevance>HIGH</relevance>" in system["content"]
assert "<tools>" in system["content"]
for tool in ("`ls`", "`grep`", "`read`"):
assert tool in system["content"]
for old in ("list_documents", "read_document", "search_documents"):
assert old not in system["content"]
def test_endpoint_deflected_turn_never_offers_tools(
@@ -605,6 +611,9 @@ def test_endpoint_deflected_turn_never_offers_tools(
assert llm.seen_tools == [None]
(system, _user) = llm.seen[0][0], llm.seen[0][1]
assert "<tools>" not in system["content"] # the LOW prompt never carries it
# Phase 70: the rewritten <tools> copy stays out of the deflected path
# (the LOW prompt is byte-identical to the pre-phase text).
assert "You may extend your context with three tools" not in system["content"]
def test_endpoint_score_at_threshold_answers(
tests/unit/test_frontend_tool_states.py
+97 -16
View File
@@ -6,7 +6,12 @@ suite (task 06). Like the other frontend-adjacent unit files, this module
pins the JS/CSS markers the story depends on, so a silent regression in
the tool branch, the persistence shape, or the tool-line styling is
catched without a browser. Phase 68 extends the pins with the
``search_documents`` status/line contract.
``search_documents`` status/line contract. Phase 70 extends the pins to
the harness-aligned names (``ls`` / ``read`` / ``grep``) in both
``app.js`` and the shared page's local copy (``shared.js``) — the legacy
names (``list_documents`` / ``read_document`` / ``search_documents``)
must keep rendering exactly as before for persisted turns (no
migration).
"""
from __future__ import annotations
@@ -15,6 +20,7 @@ from pathlib import Path
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
APP_JS = FRONTEND / "assets" / "app.js"
SHARED_JS = FRONTEND / "assets" / "shared.js"
STYLES_CSS = FRONTEND / "assets" / "styles.css"
@@ -22,6 +28,10 @@ def _js() -> str:
return APP_JS.read_text(encoding="utf-8")
def _shared_js() -> str:
return SHARED_JS.read_text(encoding="utf-8")
def _css() -> str:
return STYLES_CSS.read_text(encoding="utf-8")
@@ -66,7 +76,10 @@ def test_calling_tool_label_strings() -> None:
status lives in #send-status + the typing-indicator aria-label only.
Phase 39 centralizes the brand prefix: the name resolves from
window.BOR_BRAND at call time via brand() (the default name renders
the same bytes)."""
the same bytes). Phase 70: the ternary keys off the harness-aligned
names (read / grep / ls) and still carries the legacy names
(read_document / search_documents) — a pre-remap label stays
accurate."""
js = _js()
tool_idx = js.find('ev.type === "tool"')
delta_idx = js.find('ev.type === "delta"')
@@ -74,12 +87,19 @@ def test_calling_tool_label_strings() -> None:
assert "sendLabel" not in branch, "phase 48: the button keeps its Stop label"
assert "`${brand()} is listing documents`" in branch
assert "`${brand()} is reading ${argument}`" in branch
# Phase 68: the search status — locked name+argument gate, sitting
# BETWEEN the read branch and the listing fallback in the ternary.
assert "name === \"search_documents\" && argument" in branch, (
"the search status requires the name AND a string argument"
# Phase 70: the read status — new + legacy name, locked
# name+argument gate, first in the ternary.
assert 'name === "read" || name === "read_document") && argument' in branch, (
"the read status requires the name (new or legacy) AND a string argument"
)
# The search status — new + legacy name, sitting BETWEEN the read
# branch and the listing fallback in the ternary.
assert 'name === "grep" || name === "search_documents") && argument' in branch
assert "`${brand()} is searching for ${argument}`" in branch
# Phase 70: the scoped ls status mirrors the scoped tool line; the
# unscoped listing stays the final fallback.
assert 'name === "ls" && argument' in branch
assert "`${brand()} is listing documents in ${argument}`" in branch
read = branch.find("is reading")
search = branch.find("is searching for")
listing = branch.find("is listing documents")
@@ -120,21 +140,41 @@ def test_tool_lines_render_into_the_bubble_wrap() -> None:
assert "code.textContent = argument" in body, (
"the path is data — textContent, never innerHTML"
)
assert "name === \"read_document\" && argument" in body
# Phase 68: the search branch mirrors the read branch — the same
# name+argument gate, a <code> element, and the pattern through
# textContent (never markup); the listing stays the final else.
assert "name === \"search_documents\" && argument" in body
# Phase 70: the harness-aligned names key the branches, with the
# legacy names kept — a persisted turn from before the remap
# (read_document / search_documents / list_documents) renders
# unchanged (no migration).
assert '(name === "read" || name === "read_document") && argument' in body, (
"read (new) and read_document (legacy) both render the Reading line"
)
assert '(name === "grep" || name === "search_documents") && argument' in body, (
"grep (new) and search_documents (legacy) both render the Searching line"
)
assert 'line.textContent = "🔎 Searching for "' in body
search_part = body.split('name === "search_documents"', 1)[1]
assert 'document.createElement("code")' in search_part, (
grep_part = body.split('name === "grep"', 1)[1]
assert 'document.createElement("code")' in grep_part, (
"the pattern gets the same <code> treatment as the read path"
)
assert "code.textContent = argument" in search_part, (
assert "code.textContent = argument" in grep_part, (
"the pattern is data — textContent, never innerHTML"
)
assert 'line.textContent = "🔎 Listing documents"' in search_part, (
"the listing fallback remains the final else"
# Phase 70: the scoped ls line — the scope through textContent, and
# the unscoped "Listing documents" stays the final else (legacy
# list_documents, and a nameless/unknown frame, land there too).
assert 'name === "ls" && argument' in body
assert 'line.textContent = "🔎 Listing documents in "' in body
ls_part = body.split('name === "ls" && argument', 1)[1]
assert 'document.createElement("code")' in ls_part, (
"the scope gets the same <code> treatment as the read path"
)
assert "code.textContent = argument" in ls_part, (
"the scope is data — textContent, never innerHTML"
)
assert 'line.textContent = "🔎 Listing documents"' in ls_part, (
"the unscoped listing fallback remains the final else"
)
assert "innerHTML" not in body, (
"no HTML injection surface on tool lines — textContent only"
)
@@ -231,6 +271,47 @@ def test_tool_call_style_is_accent_and_contrast_safe() -> None:
)
def test_shared_page_tool_lines_cover_new_and_legacy_names() -> None:
"""Phase 70: the shared page's local copy (``addToolLines``) renders
the harness-aligned names — read → Reading, grep → Searching for,
ls → Listing documents, scoped ls → Listing documents in <scope> —
and keeps the legacy branches (read_document / search_documents), so
a conversation saved before the remap renders exactly as before (no
migration). Every argument through textContent; the lines carry no
innerHTML at all."""
js = _shared_js()
fn = js.find("function addToolLines")
assert fn != -1, "addToolLines must exist in shared.js"
body = js[fn : js.find("\n}\n", fn)]
assert '(t.name === "read" || t.name === "read_document") && argument' in body, (
"read (new) and read_document (legacy) both render the Reading line"
)
assert '(t.name === "grep" || t.name === "search_documents") && argument' in body, (
"grep (new) and search_documents (legacy) both render the Searching line"
)
assert 'line.textContent = "📄 Reading "' in body
assert 'line.textContent = "🔎 Searching for "' in body
assert 't.name === "ls" && argument' in body
assert 'line.textContent = "🔎 Listing documents in "' in body
ls_part = body.split('t.name === "ls" && argument', 1)[1]
assert 'document.createElement("code")' in ls_part, (
"the scope gets the same <code> treatment as the read path"
)
assert "code.textContent = argument" in ls_part, (
"the scope is data — textContent, never innerHTML"
)
assert 'line.textContent = "🔎 Listing documents"' in ls_part, (
"the unscoped listing fallback remains the final else (legacy"
" list_documents renders unchanged)"
)
assert body.count("code.textContent = argument") == 3, (
"all three argument-bearing lines (read / grep / ls) are textContent-only"
)
assert "innerHTML" not in body, (
"no HTML injection surface on shared tool lines — textContent only"
)
def test_no_cdn_added() -> None:
"""AGENTS.md rule 6: the tool state adds no external script/link."""
index = (FRONTEND / "index.html").read_text(encoding="utf-8")
tests/unit/test_llm_client.py
+30 -32
View File
@@ -500,29 +500,27 @@ def test_chat_stream_llm_error_passes_through_unwrapped() -> None:
# ---------- tool-call streaming (phase 37, task 02) ----------
#: The agent's tool list (phase 37) — the exact wire shape AGENT_TOOLS will
#: pass through (the names are whatever the caller's tools list names).
#: The agent's tool list (phase 70: the harness-aligned surface) — the
#: exact wire shape AGENT_TOOLS passes through (the names are whatever
#: the caller's tools list names).
_AGENT_TOOLS: list[dict[str, Any]] = [
{
"type": "function",
"function": {
"name": "list_documents",
"name": "ls",
"description": "List the indexed documents.",
"parameters": {"type": "object", "properties": {}},
"parameters": {"type": "object", "properties": {}, "required": []},
},
},
{
"type": "function",
"function": {
"name": "read_document",
"name": "read",
"description": "Add one indexed document's full text to the context.",
"parameters": {
"type": "object",
"properties": {
"source": {"type": "string"},
"path": {"type": "string"},
},
"required": ["source", "path"],
"properties": {"path": {"type": "string"}},
"required": ["path"],
},
},
},
@@ -557,12 +555,12 @@ def test_chat_stream_accumulates_tool_call_across_chunk_partials() -> None:
_tool_call(
0,
id="call_abc",
name="read_document",
arguments='{"source": "Homelab", "pa',
name="read",
arguments='{"path": "Homelab/ku',
)
],
),
_chunk(None, tool_calls=[_tool_call(0, arguments='th": "kubernetes.md"}')]),
_chunk(None, tool_calls=[_tool_call(0, arguments='bernetes.md"}')]),
_chunk(None, finish_reason="tool_calls"),
]
)
@@ -572,8 +570,8 @@ def test_chat_stream_accumulates_tool_call_across_chunk_partials() -> None:
assert pieces == [
ToolCallPiece(
id="call_abc",
name="read_document",
arguments={"source": "Homelab", "path": "kubernetes.md"},
name="read",
arguments={"path": "Homelab/kubernetes.md"},
)
]
@@ -586,14 +584,14 @@ def test_chat_stream_two_tool_calls_yielded_in_index_order() -> None:
_chunk(
None,
tool_calls=[
_tool_call(1, id="call_b", name="read_document", arguments='{"sou')
_tool_call(1, id="call_b", name="read", arguments='{"pa')
],
),
_chunk(
None,
tool_calls=[
_tool_call(0, id="call_a", name="list_documents"),
_tool_call(1, arguments='rce": "Homelab", "path": "a.md"}')
_tool_call(0, id="call_a", name="ls"),
_tool_call(1, arguments='th": "Homelab/a.md"}')
],
),
_chunk(None, finish_reason="tool_calls"),
@@ -603,11 +601,11 @@ def test_chat_stream_two_tool_calls_yielded_in_index_order() -> None:
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
)
assert pieces == [
ToolCallPiece(id="call_a", name="list_documents", arguments={}),
ToolCallPiece(id="call_a", name="ls", arguments={}),
ToolCallPiece(
id="call_b",
name="read_document",
arguments={"source": "Homelab", "path": "a.md"},
name="read",
arguments={"path": "Homelab/a.md"},
),
]
@@ -619,21 +617,21 @@ def test_chat_stream_tool_calls_yielded_at_stream_end_without_finish_reason() ->
[
_chunk(
None,
tool_calls=[_tool_call(0, id="call_z", name="list_documents")],
tool_calls=[_tool_call(0, id="call_z", name="ls")],
)
]
)
pieces = _collect_with_tools(
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
)
assert pieces == [ToolCallPiece(id="call_z", name="list_documents", arguments={})]
assert pieces == [ToolCallPiece(id="call_z", name="ls", arguments={})]
def test_chat_stream_synthesizes_call_id_when_absent() -> None:
"""Wire never carried the call id ⇒ synthesized "call_<index>"."""
llm, _ = _make_stream_client(
[
_chunk(None, tool_calls=[_tool_call(2, name="read_document", arguments="{}")]),
_chunk(None, tool_calls=[_tool_call(2, name="read", arguments="{}")]),
_chunk(None, finish_reason="tool_calls"),
]
)
@@ -643,7 +641,7 @@ def test_chat_stream_synthesizes_call_id_when_absent() -> None:
assert pieces == [
ToolCallPiece(
id="call_2",
name="read_document",
name="read",
arguments={},
)
]
@@ -656,7 +654,7 @@ def test_chat_stream_null_arguments_become_empty_dict() -> None:
_chunk(
None,
tool_calls=[
_tool_call(0, id="call_n", name="list_documents", arguments="null")
_tool_call(0, id="call_n", name="ls", arguments="null")
],
),
_chunk(None, finish_reason="tool_calls"),
@@ -665,7 +663,7 @@ def test_chat_stream_null_arguments_become_empty_dict() -> None:
pieces = _collect_with_tools(
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
)
assert pieces == [ToolCallPiece(id="call_n", name="list_documents", arguments={})]
assert pieces == [ToolCallPiece(id="call_n", name="ls", arguments={})]
def test_chat_stream_malformed_tool_arguments_raise_llm_error() -> None:
@@ -679,8 +677,8 @@ def test_chat_stream_malformed_tool_arguments_raise_llm_error() -> None:
_tool_call(
0,
id="call_x",
name="read_document",
arguments='{"source": "Homelab",',
name="read",
arguments='{"path": "Homelab",',
)
],
),
@@ -706,7 +704,7 @@ def test_chat_stream_non_object_tool_arguments_raise_llm_error() -> None:
_chunk(
None,
tool_calls=[
_tool_call(0, id="call_y", name="read_document", arguments='[1, 2]')
_tool_call(0, id="call_y", name="grep", arguments='[1, 2]')
],
),
_chunk(None, finish_reason="tool_calls"),
@@ -987,7 +985,7 @@ def test_retried_healthy_stream_is_untouched(
a healthy turn is byte-identical to the plain chat_stream."""
answer = [
StreamPiece("thinking", "hmm"),
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
ToolCallPiece(id="call_1", name="ls", arguments={}),
StreamPiece("content", "Talos."),
]
client = _ScriptedClient([(answer, None)])
@@ -995,7 +993,7 @@ def test_retried_healthy_stream_is_untouched(
tools = [
{
"type": "function",
"function": {"name": "list_documents", "parameters": {}},
"function": {"name": "ls", "parameters": {}},
}
]
pieces = _collect_retried(
tests/unit/test_llm_stream_teardown.py
+2 -2
View File
@@ -39,7 +39,7 @@ def _chunk(content: str) -> SimpleNamespace:
def _tool_chunk() -> SimpleNamespace:
"""One chunk carrying a malformed-arguments tool call (index 0)."""
fn = SimpleNamespace(name="read_document", arguments='{"source": "Homelab",')
fn = SimpleNamespace(name="read", arguments='{"path": "Homelab",')
tc = SimpleNamespace(index=0, id="call_x", function=fn)
delta = SimpleNamespace(content=None, tool_calls=[tc])
return SimpleNamespace(choices=[SimpleNamespace(delta=delta)])
@@ -194,7 +194,7 @@ def test_llm_error_materialization_passes_through_and_closes() -> None:
async def drain() -> None:
async for _ in llm.chat_stream(
[{"role": "user", "content": "q"}],
tools=[{"type": "function", "function": {"name": "read_document"}}],
tools=[{"type": "function", "function": {"name": "read"}}],
):
pass
tests/unit/test_mock_tool_flow.py
+7 -5
View File
@@ -32,10 +32,11 @@ from tests.e2e.mock_llm import (
SYSTEM_HIGH = "<relevance>HIGH</relevance>\n<documents>\n</documents>\n<tools>\n…\n</tools>"
SYSTEM_LOW = "<relevance>LOW</relevance>\n"
#: A minimal truthy ``tools`` parameter (the mock only checks presence).
TOOLS = [{"type": "function", "function": {"name": "list_documents"}}]
#: A minimal truthy ``tools`` parameter (the mock only checks presence;
#: the phase-70 harness-aligned names).
TOOLS = [{"type": "function", "function": {"name": "ls"}}]
#: The agent's ``list_documents`` output for a two-document KB
#: The agent's ``ls`` output for a two-document KB
#: (``app/rag/agent.py`` ``_execute_tool``): one
#: ``source: X | path: Y | title: Z`` line per document (phase 63: labeled,
#: unambiguous fields), ``(source, path)`` order.
@@ -103,7 +104,7 @@ def _body(
{
"id": f"call_{i}",
"type": "function",
"function": {"name": "list_documents", "arguments": "{}"},
"function": {"name": "ls", "arguments": "{}"},
}
],
}
@@ -274,7 +275,8 @@ SEARCH_USER = (
assert SEARCH_TRIGGER in SEARCH_USER.lower()
assert TOOLS_TRIGGER not in SEARCH_USER.lower()
#: The agent's ``search_documents`` result for the e2e fixture
#: The agent's ``grep`` result for the e2e fixture (phase 70 renamed
#: the phase-68 tool; the line format is unchanged)
#: (``app/rag/agent.py`` ``_execute_tool``): one ``source/path:LINE: text``
#: match line (the sentinel line, 200-char-capped server-side).
SEARCH_RESULT = (
tests/unit/test_prompts.py
+62
View File
@@ -138,6 +138,68 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
)
assert "<tuning>" not in build_high_prompt([doc])
assert "<tuning>" not in build_deflect_prompt([])
# Phase 70: the rewritten <tools> copy stays out of the LOW path —
# the byte-identical equality above already proves it; this names
# the contract (no <tools>, no new copy) on both empty/non-empty LOW
# builds.
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert "<tools>" not in low
assert TOOLS_SECTION not in low
# ---------- <tools> section copy (phase 70: ls / read / grep) ----------
def test_tools_section_markers_and_new_tool_names() -> None:
"""Phase 70: the section keeps the ``<tools>``/``</tools>`` markers
the E2E mock keys on and teaches the harness-aligned tool names
(backticked, exactly as the ``AGENT_TOOLS`` schemas name them)."""
assert TOOLS_SECTION.startswith("<tools>\n")
assert TOOLS_SECTION.rstrip().endswith("</tools>")
for tool in ("`ls`", "`grep`", "`read`"):
assert tool in TOOLS_SECTION
def test_tools_section_teaches_the_harness_shapes() -> None:
"""Copy pins: ``ls``'s phase-63 catalog-line format (and its
optional one-source scope), ``grep``'s case-insensitive exact-string
locator contract (up to 20 ``source/path:line: text`` lines, a
locator not a context-adder), and ``read``'s combined
``source/path`` + full content."""
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
assert "pass a source name as `path`" in TOOLS_SECTION
assert "case-insensitive" in TOOLS_SECTION
assert "up to 20" in TOOLS_SECTION
assert "source/path:line: text" in TOOLS_SECTION
assert "locator, not a context-adder" in TOOLS_SECTION
assert "combined `source/path`" in TOOLS_SECTION
assert "full content" in TOOLS_SECTION
assert "Answer as soon as you have what you need" in TOOLS_SECTION
def test_tools_section_old_names_and_budget_copy_gone() -> None:
"""The phase-37/68 tool names and the phase-37 per-tool budget line
(phase 45: the round cap is the bound — the prompt does not
re-state it) are out of the copy."""
for old in ("list_documents", "read_document", "search_documents"):
assert old not in TOOLS_SECTION
assert "more than one" not in TOOLS_SECTION
assert "extra document" not in TOOLS_SECTION
def test_high_prompt_still_ends_with_tools_section() -> None:
"""Mock keying intact: the HIGH prompt still ends with the
``<tools>`` section after ``</documents>``, now in the phase-70
copy — new names in, old names out."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
prompt = build_high_prompt([doc])
assert TOOLS_SECTION in prompt
assert prompt.index("</documents>") < prompt.index("<tools>")
assert prompt.rstrip().endswith("</tools>")
for tool in ("`ls`", "`grep`", "`read`"):
assert tool in prompt
for old in ("list_documents", "read_document", "search_documents"):
assert old not in prompt
def test_relevance_placeholder_rejected_for_garbage() -> None:
tests/unit/test_sse_events.py
+9 -8
View File
@@ -88,21 +88,22 @@ def test_tool_frame_serializes_exactly() -> None:
``{type: "tool", name: str, argument: str | null}`` — one per
model-requested document tool call, streamed ahead of the ``delta``
frames of the answer."""
frame = sse_event(ChatToolEvent(name="read_document", argument="S/p.md").model_dump())
assert frame == 'data: {"type": "tool", "name": "read_document", "argument": "S/p.md"}\n\n'
assert _payload(frame) == {"type": "tool", "name": "read_document", "argument": "S/p.md"}
frame = sse_event(ChatToolEvent(name="read", argument="S/p.md").model_dump())
assert frame == 'data: {"type": "tool", "name": "read", "argument": "S/p.md"}\n\n'
assert _payload(frame) == {"type": "tool", "name": "read", "argument": "S/p.md"}
def test_tool_frame_argument_is_null_for_parameterless_tools() -> None:
"""``list_documents`` takes no parameters, so its frame's ``argument``
serializes as JSON null (the client renders the name alone)."""
dumped = ChatToolEvent(name="list_documents").model_dump()
assert dumped == {"type": "tool", "name": "list_documents", "argument": None}
"""``ls`` (unscoped) carries no string argument, so its frame's
``argument`` serializes as JSON null (the client renders the name
alone)."""
dumped = ChatToolEvent(name="ls").model_dump()
assert dumped == {"type": "tool", "name": "ls", "argument": None}
assert _payload(sse_event(dumped))["argument"] is None
def test_tool_event_shape_is_type_name_argument_only() -> None:
dumped = ChatToolEvent(name="read_document", argument="S/p.md").model_dump()
dumped = ChatToolEvent(name="read", argument="S/p.md").model_dump()
assert set(dumped.keys()) == {"type", "name", "argument"}
assert dumped["type"] == "tool" # default — call sites never spell it out