Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?)

Browse Source
This commit is contained in:
ducoterra
2026-09-03 11:17:47 -04:00
parent 16f1cfbcaf
commit 801639efcc
55 changed files with 4031 additions and 1466 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+66 -52
View File
@@ -56,25 +56,30 @@ Implements just enough of the aipi surface:
the echo targets the block itself; its tail still includes the
closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document
tools) **and** the system prompt carries the ``<tools>`` section ->
the deterministic SINGLE-READ tool flow, discriminated statelessly
from the messages (the ``tools`` parameter gates the list/read
steps — a no-tools request with no tool results is not the flow):
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
SINGLE-READ tool flow, discriminated statelessly from the messages
(the ``tools`` parameter gates the list/read steps — a no-tools
request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``list_documents`` (synthetic id
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
content;
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
arguments), ``finish_reason: "tool_calls"``, no content;
* request 2 (a ``tool``-role catalog result in the messages):
parse the FIRST catalog line (``source: X | path: Y | title: Z``
— the labeled ``source:`` / ``path:`` fields, phase 63) and
stream a ``tool_calls`` delta calling ``read_document`` on it
(id ``call_1``);
* request 3 (a ``tool``-role read result in the messages): a
content answer, deterministic: ``Read <source/path>. <first 80
chars of the read document's content>`` — so a suite can assert
the read document reached the model and landed in the answer.
Reached regardless of the ``tools`` parameter (phase 45 keeps
the tools offered until the round cap).
stream a ``tool_calls`` delta calling ``read`` on the JOINED
combined ``source/path`` (the mock joins the two labeled fields
— the catalog format is unchanged, so this join is the only
parse change, phase 70) (id ``call_1``);
* request 3 (a ``tool``-role read result in the messages —
content starting with the agent's ``"Document <source/path>:"``
header): a content answer, deterministic: ``Read
<source/path>. <first 80 chars of the read document's
content>`` — so a suite can assert the read document reached
the model and landed in the answer. Reached regardless of the
``tools`` parameter (phase 45 keeps the tools offered until the
round cap).
The single-read flow stops at ONE read result; the MULTI-READ
variant below reads two.
- user message containing BOTH ``use your tools`` AND ``read two
@@ -82,15 +87,18 @@ Implements just enough of the aipi surface:
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix):
* 0 read results, no catalog yet: ``list_documents`` (id
``call_0``);
* 0 read results, catalog present: ``read_document`` on the FIRST
catalog line (id ``call_1``);
* 1 read result: ``read_document`` on the SECOND catalog line —
the first listing line whose ``source/path`` differs from the
one already read (id ``call_2``); a one-document catalog
degenerates to the single-read answer (nothing second to read);
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
flow on the harness-aligned names — ``ls``, then ``read`` on the
JOINED combined ``source/path`` of each catalog line:
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
* 0 read results, catalog present: ``read`` on the JOINED
combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
* 1 read result: ``read`` on the JOINED combined ``source/path``
of the SECOND catalog line — the first listing line whose
``source/path`` differs from the one already read (id
``call_2``); a one-document catalog degenerates to the
single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
<sp1> and <sp2>.`` naming both read paths in read order — so a
@@ -101,12 +109,13 @@ Implements just enough of the aipi surface:
``E2E_REAL_LLM=1`` ignores the mock entirely (the real model does
what it does).
- user message containing ``search your documents``
(``SEARCH_TRIGGER``, phase 68, search tool) **and** the system
prompt carries the ``<tools>`` section -> the deterministic SEARCH
tool flow, discriminated statelessly from the messages (streaming
only):
(``SEARCH_TRIGGER``, phase 68 search tool — renamed to the
harness-aligned ``grep`` in phase 70, same match/output contract)
**and** the system prompt carries the ``<tools>`` section -> the
deterministic SEARCH tool flow, discriminated statelessly from the
messages (streaming only):
* request 1 (``tools`` offered, no search result yet): stream
ONLY ``tool_calls`` deltas — ``search_documents`` with
ONLY ``tool_calls`` deltas — ``grep`` with
``{"pattern": SEARCH_PATTERN}`` (id ``call_0``);
* request 2 (a ``tool``-role search result in the messages —
recognizable by its ``source/path:line: text`` match lines or
@@ -265,11 +274,12 @@ END_OF_NOTES_TRIGGER = "show the end of your notes"
#: phase-24 tail echo targets the block, not the raw message tail).
_DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
#: Phase 37 (agent-document-tools story): a user message containing this
#: substring (case-insensitive) — combined with the ``<tools>`` section
#: in the system prompt — drives the deterministic tool flow documented
#: in the module docstring (list_documents → read_document on the first
#: catalog line → the quoted answer). Existing E2E questions do not
#: Phase 37 (agent-document-tools story; phase 70: the flow emits the
#: harness-aligned names): a user message containing this substring
#: (case-insensitive) — combined with the ``<tools>`` section in the
#: system prompt — drives the deterministic tool flow documented in the
#: module docstring (ls → read on the first catalog line's combined
#: ``source/path`` → the quoted answer). Existing E2E questions do not
#: contain the phrase, so every other suite is unaffected.
TOOLS_TRIGGER = "use your tools"
@@ -282,11 +292,12 @@ TOOLS_TRIGGER = "use your tools"
#: so the 3-step flow is untouched.
MULTI_READ_TRIGGER = "read two documents"
#: Phase 68 (search tool, TODO.md L4): a user message containing this
#: substring (case-insensitive) — combined with the ``<tools>`` section
#: in the system prompt — drives the deterministic SEARCH tool flow
#: (search_documents for ``SEARCH_PATTERN`` → the "Found …" answer),
#: documented in the module docstring. Checked BEFORE ``TOOLS_TRIGGER``
#: Phase 68 (search tool, TODO.md L4; phase 70: renamed to the
#: harness-aligned ``grep``): a user message containing this substring
#: (case-insensitive) — combined with the ``<tools>`` section in the
#: system prompt — drives the deterministic SEARCH tool flow (grep for
#: ``SEARCH_PATTERN`` → the "Found …" answer), documented in the module
#: docstring. Checked BEFORE ``TOOLS_TRIGGER``
#: (the more specific phrase wins — the same convention as
#: ``THINK_PARAS_TRIGGER``); verified 2026-09-01: no existing E2E
#: question or fixture file contains the phrase, so every other suite
@@ -402,11 +413,11 @@ def _chat_dead(key: str, dead_attempts: int) -> bool:
return _bump_fail(key) <= dead_attempts * _HTTPS_PER_DEAD_ATTEMPT
#: The agent's ``read_document`` tool-result prefix (app.rag.agent
#: The agent's ``read`` tool-result prefix (app.rag.agent
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
_READ_RESULT_PREFIX = "Document "
#: One line of the agent's ``list_documents`` output (app.rag.agent
#: One line of the agent's ``ls`` output (app.rag.agent
#: ``_execute_tool``, phase 63): labeled, pipe-delimited fields —
#: ``source: X | path: Y | title: Z`` — unambiguous for LLM parsing even
#: when the path contains ``/`` characters.
@@ -439,7 +450,7 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
"""Every ``(source, path)`` in the catalog tool result, in listing order.
Catalog lines are ``source: X | path: Y | title: Z`` (the agent's
``list_documents`` output — phase 63: labeled, pipe-delimited
``ls`` output — phase 63: labeled, pipe-delimited
fields, unambiguous even for paths full of ``/``): the line-level
regex recovers the ``source`` and ``path`` fields directly. The
``"N documents:"`` header line matches no line and is skipped;
@@ -460,8 +471,9 @@ def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
return docs
#: One line of the agent's ``search_documents`` output (app.rag.agent
#: ``_execute_tool``, phase 68): ``source/path:LINE: text``. The
#: One line of the agent's ``grep`` output (app.rag.agent
#: ``_execute_tool``, phase 68 — phase 70 renamed the tool, the line
#: format is unchanged): ``source/path:LINE: text``. The
#: non-greedy prefix keeps nested paths (``/`` in the path) intact.
_SEARCH_LINE_RE = re.compile(r"^(?P<sp>.+?):(?P<line>\d+): (?P<text>.*)$")
@@ -472,7 +484,7 @@ def _search_result_line(body: dict[str, Any]) -> str | None:
A search result is a ``tool``-role message — never a read result
(those start with the agent's ``"Document "`` prefix) — that either
carries ``source/path:LINE: text`` match lines (the agent's
``search_documents`` output, phase 68) or the sentinel pattern
``grep`` output, phase 68) or the sentinel pattern
itself (its no-match line quotes the pattern). Returns the first
match line's ``text`` part (already 200-char-capped server-side),
or the message's first line in the sentinel-only shape, or ``None``
@@ -534,7 +546,9 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
* ``("read", source, path, "call_1")`` — a ``tool``-role catalog
result is in the messages: the model reads its FIRST
``source: X | path: Y | title: Z`` line (the labeled
``source:`` / ``path:`` fields, phase 63).
``source:`` / ``path:`` fields, phase 63), emitted as ``read`` on
the JOINED combined ``source/path`` (phase 70: the mock joins
the two fields — the canonical document identity).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document. Reached
@@ -1054,7 +1068,7 @@ def chat_completions(body: dict[str, Any]) -> Any:
if search_flow is not None:
if search_flow[0] == "search":
stream = _tool_call_stream(
"search_documents", {"pattern": SEARCH_PATTERN}, "call_0"
"grep", {"pattern": SEARCH_PATTERN}, "call_0"
)
else: # "found" — quote the first matched line (80 chars)
answer = _apply_max_tokens(
@@ -1069,16 +1083,16 @@ def chat_completions(body: dict[str, Any]) -> Any:
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
stream = _tool_call_stream("list_documents", {}, "call_0")
stream = _tool_call_stream("ls", {}, "call_0")
elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
# "call_2" for the multi-read second read (phase 45,
# task 02).
# task 02). Phase 70: the harness-aligned shape — one
# combined ``source/path`` argument (the mock joins the
# two catalog fields; the catalog format is unchanged).
stream = _tool_call_stream(
"read_document",
{"source": flow[1], "path": flow[2]},
flow[3],
"read", {"path": f"{flow[1]}/{flow[2]}"}, flow[3]
)
elif flow[0] == "multi_answer":
# Phase 45 (task 02): the multi-read forced answer —