Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)

Browse Source
This commit is contained in:
ducoterra
2026-09-03 11:17:47 -04:00
parent 16f1cfbcaf
commit 801639efcc
55 changed files with 4031 additions and 1466 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+66 -52
View File
@@ -56,25 +56,30 @@ Implements just enough of the aipi surface:
the echo targets the block itself; its tail still includes the
closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document
tools) **and** the system prompt carries the ``<tools>`` section ->
the deterministic SINGLE-READ tool flow, discriminated statelessly
from the messages (the ``tools`` parameter gates the list/read
steps — a no-tools request with no tool results is not the flow):
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
SINGLE-READ tool flow, discriminated statelessly from the messages
(the ``tools`` parameter gates the list/read steps — a no-tools
request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``list_documents`` (synthetic id
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
content;
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
arguments), ``finish_reason: "tool_calls"``, no content;
* request 2 (a ``tool``-role catalog result in the messages):
parse the FIRST catalog line (``source: X | path: Y | title: Z``
— the labeled ``source:`` / ``path:`` fields, phase 63) and
stream a ``tool_calls`` delta calling ``read_document`` on it
(id ``call_1``);
* request 3 (a ``tool``-role read result in the messages): a
content answer, deterministic: ``Read <source/path>. <first 80
chars of the read document's content>`` — so a suite can assert
the read document reached the model and landed in the answer.
Reached regardless of the ``tools`` parameter (phase 45 keeps
the tools offered until the round cap).
stream a ``tool_calls`` delta calling ``read`` on the JOINED
combined ``source/path`` (the mock joins the two labeled fields
— the catalog format is unchanged, so this join is the only
parse change, phase 70) (id ``call_1``);
* request 3 (a ``tool``-role read result in the messages —
content starting with the agent's ``"Document <source/path>:"``
header): a content answer, deterministic: ``Read
<source/path>. <first 80 chars of the read document's
content>`` — so a suite can assert the read document reached
the model and landed in the answer. Reached regardless of the
``tools`` parameter (phase 45 keeps the tools offered until the
round cap).
The single-read flow stops at ONE read result; the MULTI-READ
variant below reads two.
- user message containing BOTH ``use your tools`` AND ``read two
@@ -82,15 +87,18 @@ Implements just enough of the aipi surface:
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix):
* 0 read results, no catalog yet: ``list_documents`` (id
``call_0``);
* 0 read results, catalog present: ``read_document`` on the FIRST
catalog line (id ``call_1``);
* 1 read result: ``read_document`` on the SECOND catalog line —
the first listing line whose ``source/path`` differs from the
one already read (id ``call_2``); a one-document catalog
degenerates to the single-read answer (nothing second to read);
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
flow on the harness-aligned names — ``ls``, then ``read`` on the
JOINED combined ``source/path`` of each catalog line:
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
* 0 read results, catalog present: ``read`` on the JOINED
combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
* 1 read result: ``read`` on the JOINED combined ``source/path``
of the SECOND catalog line — the first listing line whose
``source/path`` differs from the one already read (id
``call_2``); a one-document catalog degenerates to the
single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
<sp1> and <sp2>.`` naming both read paths in read order — so a
@@ -101,12 +109,13 @@ Implements just enough of the aipi surface:
``E2E_REAL_LLM=1`` ignores the mock entirely (the real model does
what it does).
- user message containing ``search your documents``
(``SEARCH_TRIGGER``, phase 68, search tool) **and** the system
prompt carries the ``<tools>`` section -> the deterministic SEARCH
tool flow, discriminated statelessly from the messages (streaming
only):
(``SEARCH_TRIGGER``, phase 68 search tool — renamed to the
harness-aligned ``grep`` in phase 70, same match/output contract)
**and** the system prompt carries the ``<tools>`` section -> the
deterministic SEARCH tool flow, discriminated statelessly from the
messages (streaming only):
* request 1 (``tools`` offered, no search result yet): stream
ONLY ``tool_calls`` deltas — ``search_documents`` with
ONLY ``tool_calls`` deltas — ``grep`` with
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
* request 2 (a ``tool``-role search result in the messages —
recognizable by its ``source/path:line: text`` match lines or
@@ -265,11 +274,12 @@ END_OF_NOTES_TRIGGER = "show the end of your notes"
#: phase-24 tail echo targets the block, not the raw message tail).
_DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
#: Phase 37 (agent-document-tools story): a user message containing this
#: substring (case-insensitive) — combined with the ``<tools>`` section
#: in the system prompt — drives the deterministic tool flow documented
#: in the module docstring (list_documents → read_document on the first
#: catalog line → the quoted answer). Existing E2E questions do not
#: Phase 37 (agent-document-tools story; phase 70: the flow emits the
#: harness-aligned names): a user message containing this substring
#: (case-insensitive) — combined with the ``<tools>`` section in the
#: system prompt — drives the deterministic tool flow documented in the
#: module docstring (ls → read on the first catalog line's combined
#: ``source/path`` → the quoted answer). Existing E2E questions do not
#: contain the phrase, so every other suite is unaffected.
TOOLS_TRIGGER = "use your tools"
@@ -282,11 +292,12 @@ TOOLS_TRIGGER = "use your tools"
#: so the 3-step flow is untouched.
MULTI_READ_TRIGGER = "read two documents"
#: Phase 68 (search tool, TODO.md L4): a user message containing this
#: substring (case-insensitive) — combined with the ``<tools>`` section
#: in the system prompt — drives the deterministic SEARCH tool flow
#: (search_documents for ``SEARCH_PATTERN`` → the "Found …" answer),
#: documented in the module docstring. Checked BEFORE ``TOOLS_TRIGGER``
#: Phase 68 (search tool, TODO.md L4; phase 70: renamed to the
#: harness-aligned ``grep``): a user message containing this substring
#: (case-insensitive) — combined with the ``<tools>`` section in the
#: system prompt — drives the deterministic SEARCH tool flow (grep for
#: ``SEARCH_PATTERN`` → the "Found …" answer), documented in the module
#: docstring. Checked BEFORE ``TOOLS_TRIGGER``
#: (the more specific phrase wins — the same convention as
#: ``THINK_PARAS_TRIGGER``); verified 2026-09-01: no existing E2E
#: question or fixture file contains the phrase, so every other suite
@@ -402,11 +413,11 @@ def _chat_dead(key: str, dead_attempts: int) -> bool:
return _bump_fail(key) <= dead_attempts * _HTTPS_PER_DEAD_ATTEMPT
#: The agent's ``read_document`` tool-result prefix (app.rag.agent
#: The agent's ``read`` tool-result prefix (app.rag.agent
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
_READ_RESULT_PREFIX = "Document "
#: One line of the agent's ``list_documents`` output (app.rag.agent
#: One line of the agent's ``ls`` output (app.rag.agent
#: ``_execute_tool``, phase 63): labeled, pipe-delimited fields —
#: ``source: X | path: Y | title: Z`` — unambiguous for LLM parsing even
#: when the path contains ``/`` characters.
@@ -439,7 +450,7 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
"""Every ``(source, path)`` in the catalog tool result, in listing order.
Catalog lines are ``source: X | path: Y | title: Z`` (the agent's
``list_documents`` output — phase 63: labeled, pipe-delimited
``ls`` output — phase 63: labeled, pipe-delimited
fields, unambiguous even for paths full of ``/``): the line-level
regex recovers the ``source`` and ``path`` fields directly. The
``"N documents:"`` header line matches no line and is skipped;
@@ -460,8 +471,9 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
return docs
#: One line of the agent's ``search_documents`` output (app.rag.agent
#: ``_execute_tool``, phase 68): ``source/path:LINE: text``. The
#: One line of the agent's ``grep`` output (app.rag.agent
#: ``_execute_tool``, phase 68 — phase 70 renamed the tool, the line
#: format is unchanged): ``source/path:LINE: text``. The
#: non-greedy prefix keeps nested paths (``/`` in the path) intact.
_SEARCH_LINE_RE = re.compile(r"^(?P<sp>.+?):(?P<line>\d+): (?P<text>.*)$")
@@ -472,7 +484,7 @@ def _search_result_line(body: dict[str, Any]) -> str | None:
A search result is a ``tool``-role message — never a read result
(those start with the agent's ``"Document "`` prefix) — that either
carries ``source/path:LINE: text`` match lines (the agent's
``search_documents`` output, phase 68) or the sentinel pattern
``grep`` output, phase 68) or the sentinel pattern
itself (its no-match line quotes the pattern). Returns the first
match line's ``text`` part (already 200-char-capped server-side),
or the message's first line in the sentinel-only shape, or ``None``
@@ -534,7 +546,9 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
* ``("read", source, path, "call_1")`` — a ``tool``-role catalog
result is in the messages: the model reads its FIRST
``source: X | path: Y | title: Z`` line (the labeled
``source:`` / ``path:`` fields, phase 63).
``source:`` / ``path:`` fields, phase 63), emitted as ``read`` on
the JOINED combined ``source/path`` (phase 70: the mock joins
the two fields — the canonical document identity).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document. Reached
@@ -1054,7 +1068,7 @@ def chat_completions(body: dict[str, Any]) -> Any:
if search_flow is not None:
if search_flow[0] == "search":
stream = _tool_call_stream(
"search_documents", {"pattern": SEARCH_PATTERN}, "call_0"
"grep", {"pattern": SEARCH_PATTERN}, "call_0"
)
else: # "found" — quote the first matched line (80 chars)
answer = _apply_max_tokens(
@@ -1069,16 +1083,16 @@ def chat_completions(body: dict[str, Any]) -> Any:
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
stream = _tool_call_stream("list_documents", {}, "call_0")
stream = _tool_call_stream("ls", {}, "call_0")
elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
# "call_2" for the multi-read second read (phase 45,
# task 02).
# task 02). Phase 70: the harness-aligned shape — one
# combined ``source/path`` argument (the mock joins the
# two catalog fields; the catalog format is unchanged).
stream = _tool_call_stream(
"read_document",
{"source": flow[1], "path": flow[2]},
flow[3],
"read", {"path": f"{flow[1]}/{flow[2]}"}, flow[3]
)
elif flow[0] == "multi_answer":
# Phase 45 (task 02): the multi-read forced answer —
tests/e2e/test_agent_document_tools.py
+21 -17
View File
@@ -9,18 +9,20 @@ MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
does whatever it does with the tools, while this story's gate is the
deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
contains ``use your tools`` **and** the system prompt carries the
``<tools>`` section of the HIGH prompt):
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
harness-aligned names — ``ls`` / ``read`` with the combined
``source/path`` identity):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``, no
arguments, ``finish_reason: "tool_calls"``);
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
``finish_reason: "tool_calls"``);
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
``tool_calls`` delta calling ``read_document`` on the FIRST catalog
line (id ``call_1``);
3. request 3 (no ``tools`` parameter, the read result in the messages) →
the content answer ``Read <source/path>. <first 80 chars of the read
document's content>`` — so the suite can assert the read document
reached the model and landed in the answer.
``tool_calls`` delta calling ``read`` on the JOINED combined
``source/path`` of the FIRST catalog line (id ``call_1``);
3. request 3 (a ``tool``-role read result in the messages) → the content
answer ``Read <source/path>. <first 80 chars of the read document's
content>`` — so the suite can assert the read document reached the
model and landed in the answer.
KB fixture — reproduces the TODO failure (``aws-route53.md`` references
``example-record-file.json`` "for the exact JSON shape of
@@ -43,10 +45,11 @@ reseelink.json" but does not include it):
Test → story mapping (Playwright Mapping Rule):
1. ``test_marker_question_lists_reads_and_quotes`` — the SSE carries
``tool`` frames (list, then read, ahead of any delta), the UI shows
the "calling tool" label while a tool runs, the bubble shows both
tool lines, the final answer quotes the read document, and the
source chips include the read document (viewer link).
``tool`` frames (``ls``, then ``read`` with the combined path, ahead
of any delta), the UI shows the transient calling-tool status while a
tool runs, the bubble shows both tool lines, the final answer quotes
the read document, and the source chips include the read document
(viewer link).
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
(phase 14) re-renders the tool lines.
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
@@ -385,12 +388,13 @@ def test_marker_question_lists_reads_and_quotes(
assert i_list is not None and i_read is not None, statuses
assert i_list < i_read, statuses
# Wire level: exactly two `tool` frames — list then read — and both
# ahead of the first `delta` frame.
# Wire level: exactly two `tool` frames — ``ls`` then ``read`` (the
# combined source/path as the model passed it) — and both ahead of
# the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ_SP},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
tests/e2e/test_agent_unlimited_tools.py
+24 -18
View File
@@ -9,16 +9,20 @@ MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic MULTI-READ marker flow in ``tests/e2e/mock_llm.py`` (user
message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
carries the ``<tools>`` section of the HIGH prompt):
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
combined ``source/path`` of each catalog line):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read_document`` on
the FIRST catalog line (id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read_document`` on the
SECOND catalog line (id ``call_2``) — the pre-phase-45 per-tool
budgets would have refused exactly this second read (``No reading
budget left — answer with what you have.``);
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
JOINED combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
— the pre-phase-45 per-tool budgets would have refused exactly this
second read (``No reading budget left — answer with what you
have.``);
4. request 4 (two read results) → the forced answer, byte-stable: the
single-read shape quoting the FIRST read result, plus the line
``I read <sp1> and <sp2>.`` naming both read paths in read order.
@@ -393,14 +397,15 @@ def test_multi_read_turn(
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: exactly THREE `tool` frames — list, read #1, read #2,
# in order — and all ahead of the first `delta` frame. This third
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
# (each read's argument is the JOINED combined source/path), in
# order — and all ahead of the first `delta` frame. This third
# frame is the one the pre-phase-45 read budget refused.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ1_SP},
{"type": "tool", "name": "read_document", "argument": READ2_SP},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ1_SP},
{"type": "tool", "name": "read", "argument": READ2_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
@@ -518,7 +523,7 @@ def test_relist_allowed(
# per-tool budgets would have refused (list budget 1, read budget
# 1 — this turn makes one list and TWO reads).
frames = _frames(page)
assert {"type": "tool", "name": "list_documents", "argument": None} in _tool_frames(
assert {"type": "tool", "name": "ls", "argument": None} in _tool_frames(
frames
)
line0 = page.locator(".msg.brain .tool-call").nth(0)
@@ -556,12 +561,13 @@ def test_single_tool_flow_regression(
_submit(page, SINGLE_QUESTION)
_wait_settled(page)
# Exactly TWO tool frames — list then ONE read of the first catalog
# line — no second read (the marker carries no multi-read trigger).
# Exactly TWO tool frames — ls then ONE read of the first catalog
# line (the JOINED combined source/path) — no second read (the
# marker carries no multi-read trigger).
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ1_SP},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ1_SP},
]
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
tests/e2e/test_harness_aligned_tools.py
+524
View File
@@ -0,0 +1,524 @@
"""Phase 70 E2E (Playwright, mock-only): the harness-aligned tool surface
(``ls`` / ``read(path)`` / ``grep(pattern, path?)``).
Story: ``.agent/user_stories/agent-document-tools.md`` (phase 70 reshapes
the tools that story delivered — owner decision 2026-09-03: "match
existing harnesses as much as possible", the pi.dev tool shapes).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_harness_aligned_tools.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic marker flows in ``tests/e2e/mock_llm.py`` (phase 70: the
flows emit the NEW names with the NEW argument shapes):
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
prompt's ``<tools>`` section): ``ls`` (id ``call_0``, no arguments) →
``read`` on the JOINED combined ``source/path`` of the first catalog
line (id ``call_1``) → the ``Read <source/path>. <quote>`` answer;
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
(id ``call_0``) → the ``Found <matched line>`` answer.
The combined ``source/path`` string is the canonical document identity:
the mock joins the two labeled catalog fields itself (the catalog
format is unchanged), and the SSE ``tool`` frames carry exactly what the
model "passed" — ``read``'s combined path, ``grep``'s pattern, ``ls``'s
scope or null when unscoped (the phase-70 argument rule).
KB fixtures:
* READ flow — the ``test_agent_document_tools.py`` two-document pair
(TRUNCATE-then-seed): ``Homelab/aws-route53.md`` seeded with one
chunk whose embedding is the mock's own bag-of-words vector (the
marker question cosines ≈0.69 against it, well past the E2E 0.30
threshold, and it FTS-matches too → grounded) and
``Deployments/example-record-file.json`` indexed WITHOUT chunks (the
catalog-first line the mock reads; never in the retrieval context).
* SEARCH flow — the phase-68 fixture (``tests/fixtures/search_docs/``)
imported through the real importer, its line 6 carrying the sentinel
``reese-sentinel-42`` exactly once (``test_search_tool.py`` pattern).
Test → phase mapping (Playwright Mapping Rule):
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
grounded READ turn: the UI shows the ``ls`` line (unscoped "🔎
Listing documents", no argument) then the "📄 Reading <source/path>"
line with the combined path in a ``<code>`` element, the answer
streams and quotes the read document, the done-state sources
include the read document, and NO raw tool markup (``<|…|>``,
``tool_call``) appears anywhere in the DOM — the live incident this
phase fixes.
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
matched-line answer.
3. ``test_wire_argument_rule_across_both_flows`` — the SSE wire across
BOTH flows in one session: every ``tool`` frame's name is in
{``ls``, ``read``, ``grep``} (no pre-phase-70 name ever reaches the
client) and the argument rule holds — ``read`` → the combined path
as passed, ``grep`` → the pattern, ``ls`` → null when unscoped.
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import time
from datetime import UTC, datetime
from pathlib import Path
from threading import Thread
from typing import Any
from playwright.sync_api import Page, expect
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from tests.e2e.mock_llm import SEARCH_PATTERN, embed_text
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "search_docs"
# --------------------------------------------------------------------------
# READ flow — the two-document pair (cf. test_agent_document_tools.py)
# --------------------------------------------------------------------------
SEED_SOURCE = "Homelab"
SEED_PATH = "aws-route53.md"
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
READ_SOURCE = "Deployments"
READ_PATH = "example-record-file.json"
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
#: The retrievable document (the grounded seed context): the repeated
#: record-file lines carry the marker question's key tokens — verified
#: ≈0.69 cosine against the mock's embeddings (E2E threshold 0.30) plus
#: FTS hits.
ROUTE53_CONTENT = (
"# AWS Route 53 Notes\n\n"
"## Record file\n\n"
+ (
"The aws route53 hosted zone for reeselink keeps every record in "
"reseelink.json — the exact JSON shape of reeselink.json is "
"documented in example-record-file.json.\n"
)
* 10
+ "\n## Sync job\n\n"
"A cron job pushes reeselink.json to the aws route53 hosted zone "
"every fifteen minutes; the diff is applied through the route53 api.\n"
)
#: The read document (the catalog-first line the mock reads; no chunks,
#: so retrieval never puts it in context). Its FIRST line is longer than
#: 80 chars, so the mock's first-80-chars quote is newline-free.
RECORD_FILE_CONTENT = (
'{ "version": 3, "comment": "ReeseLink hosted zone records — the exact '
'JSON shape of reeselink.json",\n'
' "hosted_zone_id": "Z0RESEELINK01",\n'
' "record_sets": [\n'
' { "name": "www.reeselink.example", "type": "A", "ttl": 300,\n'
' "resource_records": [ { "value": "10.0.0.20" } ] },\n'
' { "name": "api.reeselink.example", "type": "CNAME", "ttl": 300,\n'
' "resource_records": [ { "value": "www.reeselink.example" } ] }\n'
" ]\n"
"}\n"
)
assert "\n" not in RECORD_FILE_CONTENT[:80] # the quote must stay one line
#: Carries ``TOOLS_TRIGGER`` (and nothing else — no multi-read, no
#: search, no other mock marker).
READ_QUESTION = (
"Use your tools: what is the exact JSON shape of reeselink.json "
"for my aws route53 hosted zone?"
)
for _other in (
"read two documents",
"search your documents",
"write a long answer",
"think in paragraphs",
"think out loud",
"show the end of your notes",
"show me a table",
"fail then answer",
"always fail",
"embed fail once",
"pretend to think slowly",
):
assert _other not in READ_QUESTION.lower(), _other
READ_ANSWER_PREFIX = f"Read {READ_SP}."
READ_ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
def _seed_read_pair(db: Session) -> None:
"""The two-document READ-flow KB (see the module docstring)."""
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
full_path=f"/tmp/{SEED_PATH}",
title="AWS Route 53 Notes",
content=ROUTE53_CONTENT,
content_hash=hashlib.sha256(ROUTE53_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
db.add(md)
db.flush()
# One chunk carrying the mock's own embedding → genuine token
# overlap between the marker question and this document (the only
# retrievable document).
db.add(
Chunk(
document_id=md.id,
position=0,
content=ROUTE53_CONTENT,
embedding=embed_text(ROUTE53_CONTENT),
)
)
db.add(
Document(
source=READ_SOURCE,
path=READ_PATH,
full_path=f"/tmp/{READ_PATH}",
title="Example Record File",
content=RECORD_FILE_CONTENT,
content_hash=hashlib.sha256(RECORD_FILE_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
)
# --------------------------------------------------------------------------
# SEARCH flow — the phase-68 fixture (cf. test_search_tool.py)
# --------------------------------------------------------------------------
SEED_SOURCE_S = "search_docs"
SEED_PATH_S = "reese-notes.md"
SEED_SP_S = f"{SEED_SOURCE_S}/{SEED_PATH_S}"
#: The fixture's sentinel line (line 6) — the mock's grep matches it
#: exactly once; its ``text`` part is what the "Found …" answer quotes.
SENTINEL_LINE = f"The offsite vault passphrase marker is {SEARCH_PATTERN}."
FOUND_ANSWER = f"Found {SENTINEL_LINE[:80]}"
#: Carries ``SEARCH_TRIGGER`` and is on-topic (cosine ≈0.51 against the
#: fixture + FTS hits → HIGH gate, the ``<tools>`` section rides along).
SEARCH_QUESTION = (
"Search your documents for the vault passphrase marker in my homelab "
"kubernetes backup notes?"
)
assert SEARCH_PATTERN.lower() not in SEARCH_QUESTION.lower()
def _pin_fixture() -> None:
"""The fixture carries the sentinel on line 6, exactly once."""
content = (FIXTURES / SEED_PATH_S).read_text(encoding="utf-8")
lines = content.split("\n")
assert lines[5] == SENTINEL_LINE, lines[5]
assert sum(SEARCH_PATTERN in line for line in lines) == 1
# --------------------------------------------------------------------------
# DB seeding (TRUNCATE-then-seed / TRUNCATE-then-import)
# --------------------------------------------------------------------------
async def _import_search_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {
"_env_file": None,
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return await import_sources([FIXTURES], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread.
Playwright's sync API keeps an asyncio loop running on the test
thread, so ``asyncio.run`` cannot be called directly from a test
body (the established house helper).
"""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _reset_db_read_pair() -> None:
"""Truncate the KB (plus the prompt-shaping tables), then seed the
two-document READ-flow pair. ``steering_notes`` / ``kb_overview``
are truncated too, so the HIGH prompt is exactly ``<relevance>`` +
``<documents>`` + ``<tools>`` — byte-stable prompts, byte-stable
answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
)
db.commit()
_seed_read_pair(db)
db.commit()
def _reset_db_search_fixture(mock_port: int) -> None:
"""Truncate the KB (plus the prompt-shaping tables), then import the
phase-68 search fixture through the real importer."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
)
db.commit()
summary = _run_in_thread(_import_search_fixtures(mock_port))
assert summary is not None and summary.added == 1, summary
# --------------------------------------------------------------------------
# Page helpers (the test_agent_document_tools.py pattern)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_sse_hook(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _drain_frames(page: Page) -> list[dict]:
"""One turn's SSE frames: wait for that turn's ``done`` frame, then
return EVERY frame captured since the last drain (the hook's
background read appends the whole stream at once after it closes, so
clearing-and-reading is race-free per turn)."""
deadline = time.monotonic() + 10.0
while True:
raw = page.evaluate(
"() => { const f = window.__sseFrames || []; "
"window.__sseFrames = []; return f; }"
)
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered.
Phase 48: the label assertion carries the settle wait with an
explicit timeout — the in-flight button is the enabled Stop control
(never disabled), so ``to_be_enabled`` no longer blocks until the
turn settles."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
# --------------------------------------------------------------------------
# 1. The grounded READ turn: ls line → Reading line → quoted answer,
# sources include the read doc, no raw tool markup anywhere in the DOM
# --------------------------------------------------------------------------
def test_read_flow_lines_answer_sources_no_raw_markup(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db_read_pair()
page.goto(app_url)
_install_sse_hook(page)
_submit(page, READ_QUESTION)
_wait_settled(page)
# The UI shows the ls line (UNSCOPED — no argument, no <code>) then
# the "📄 Reading <source/path>" line with the COMBINED path in a
# <code> element (the path is data, never markup).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(READ_SP)
# The answer streamed and quotes the read document (the mock's
# deterministic echo: "Read <source/path>. <first 80 chars>").
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
# Wire level: ls then read — the phase-70 argument rule (ls
# unscoped → null; read → the combined path as passed) — ahead of
# the first delta.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# Done-state sources include the read document (retrieval doc first,
# the agent's read doc after — the phase-37 extension contract).
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH),
(READ_SOURCE, READ_PATH),
]
# The live incident this phase fixes: NO raw tool markup anywhere in
# the DOM — the model's trained wire shapes (<|tool_call_…|>,
# "tool_calls", finish_reason) must never leak into the rendered
# conversation.
dom = page.locator("#messages").inner_html()
for raw in ("<|", "tool_call", "tool_calls", "finish_reason"):
assert raw not in dom, f"raw tool markup {raw!r} leaked into the DOM"
# --------------------------------------------------------------------------
# 2. The grounded SEARCH turn: the "🔎 Searching for <pattern>" line,
# then the matched-line answer
# --------------------------------------------------------------------------
def test_grep_flow_line_then_answer(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
_pin_fixture()
page.set_default_timeout(30_000)
_reset_db_search_fixture(mock_llm)
page.goto(app_url)
_install_sse_hook(page)
_submit(page, SEARCH_QUESTION)
_wait_settled(page)
# ONE tool line above the answer: "🔎 Searching for " + the sentinel
# in a <code> element (the pattern is data, never markup).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text("Searching for")
expect(lines.nth(0).locator("code")).to_have_text(SEARCH_PATTERN)
# The answer quotes the MATCHED LINE — the grep result reached the
# model and landed in the answer (the mock's deterministic echo).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(FOUND_ANSWER)
# Wire level: exactly ONE tool frame — grep carrying the PATTERN as
# its argument (the phase-70 argument rule) — ahead of the first
# delta; the turn is grounded.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# --------------------------------------------------------------------------
# 3. The SSE wire across BOTH flows: every tool frame carries a
# phase-70 name and the single-string argument rule
# --------------------------------------------------------------------------
def test_wire_argument_rule_across_both_flows(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
_pin_fixture()
page.set_default_timeout(30_000)
_reset_db_read_pair()
page.goto(app_url)
_install_sse_hook(page)
# Turn 1 — the READ flow (ls → read on the combined path).
_submit(page, READ_QUESTION)
_wait_settled(page)
read_frames = _drain_frames(page)
# Turn 2 — re-seed the search fixture, then the SEARCH flow (grep
# for the sentinel). The app's chat path is single-turn (system +
# user message), so the first turn cannot influence this one.
_reset_db_search_fixture(mock_llm)
_submit(page, SEARCH_QUESTION)
_wait_settled(page)
search_frames = _drain_frames(page)
read_tools = _tool_frames(read_frames)
search_tools = _tool_frames(search_frames)
# The ordered, combined tool-frame sequence across both flows: the
# argument rule end-to-end — read → the combined path as passed,
# grep → the pattern, ls → null when unscoped.
assert read_tools + search_tools == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "read", "argument": READ_SP},
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
]
# No pre-phase-70 name ever reaches the client.
for frame in read_tools + search_tools:
assert frame["name"] in {"ls", "read", "grep"}, frame
assert frame["argument"] is None or isinstance(frame["argument"], str)
# And both turns answered (neither flow stalled at a tool round).
assert next(f for f in read_frames if f["type"] == "done")["deflected"] is False
assert next(f for f in search_frames if f["type"] == "done")["deflected"] is False
tests/e2e/test_search_tool.py
+17 -15
View File
@@ -1,4 +1,6 @@
"""Phase 68 E2E (Playwright, mock-only): the ``search_documents`` tool.
"""Phase 68 E2E (Playwright, mock-only): the ``grep`` tool (the
phase-68 search tool, renamed to the harness-aligned ``grep`` in
phase 70; the A5 match/output contract is unchanged).
Story: n/a (TODO-derived — the owner roadmap confirmation 2026-09-01,
TODO.md L4: "Add a search tool that allows the LLM to grep through the
@@ -14,8 +16,8 @@ message contains ``search your documents`` (``SEARCH_TRIGGER``)
prompt):
1. request 1 (``tools`` offered, no search result yet) → streams ONLY
``tool_calls`` deltas calling ``search_documents`` with
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
``tool_calls`` deltas calling ``grep`` with ``{"pattern":
SEARCH_PATTERN}`` (id ``call_0``);
2. request 2 (a ``tool``-role search result — the
``source/path:line: text`` match line) → the content answer
``Found <first matched line's content up to 80 chars>`` — so this
@@ -40,13 +42,13 @@ shadow the phase-37/45 flows and vice versa).
Test → phase mapping:
1. ``test_search_flow_searches_and_answers_from_match`` — the live
search flow: the SSE carries the ``tool`` frame
(``search_documents`` with ``argument = <sentinel>``, ahead of any
delta), #send-status recorded the transient "… is searching for
<sentinel>" state, the bubble shows ONE ``🔎 Searching for``
tool line with the sentinel in a ``<code>`` element, the answer
quotes the matched line (``Found …`` — the match reached the
model), and the turn settles to idle with no error banner.
search flow: the SSE carries the ``tool`` frame (``grep`` with
``argument = <sentinel>``, ahead of any delta), #send-status
recorded the transient "… is searching for <sentinel>" state, the
bubble shows ONE ``🔎 Searching for`` tool line with the sentinel in
a ``<code>`` element, the answer quotes the matched line
(``Found …`` — the match reached the model), and the turn settles to
idle with no error banner.
2. ``test_search_adds_no_source_by_itself`` — context accounting
(locked A5): the search-only flow (no read) leaves
``done.sources`` / the source chips / ``query_log.sources`` at the
@@ -367,12 +369,12 @@ def test_search_flow_searches_and_answers_from_match(
)
assert i_think is not None and i_think < i_search, statuses
# Wire level: exactly ONE `tool` frame — search_documents carrying
# the PATTERN as its argument (phase 68 task 02) — ahead of the
# first `delta` frame.
# Wire level: exactly ONE `tool` frame — grep carrying the PATTERN
# as its argument (phase 68 task 02; phase 70 renamed the tool) —
# ahead of the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "search_documents", "argument": SEARCH_PATTERN}
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
@@ -424,7 +426,7 @@ def test_search_adds_no_source_by_itself(
# baseline: the one fixture doc, nothing added by the search.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "search_documents", "argument": SEARCH_PATTERN}
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False