Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(rag): unbounded agent tool calls behind a round cap (owner revision)

Browse Source 
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:

- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
  negative rejected) replaces agent_list_calls / agent_read_calls;
  .env.example + README document the single knob; app/rag/prompts.py
  docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
  rounds >= max_rounds, at which point it forces one final no-tools
  answer (the cap is the only forced exit); 0 = no tools — exactly one
  tools=None request, byte-identical to the pre-phase-37 path (the
  kill switch). Rejected calls (unknown tool / missing args /
  already-in-context / unknown path) still consume a round, so
  pathological rejected-call streams are bounded by the cap. The
  per-call log line is now tool/args/round=N/M; the per-turn
  tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
  the deterministic list -> read #1 -> read #2 -> forced-answer flow
  (byte-stable "I read <sp1> and <sp2>." line), classified by the
  count of tool-role read results; the phase-37 single-read flow stays
  byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
  mock-only): three tool frames/lines in order (one list, two reads —
  the second read is what the old read budget refused) + the
  both-named non-deflected answer; done.sources + chips = retrieval
  doc + both reads, deduped; no budget refusal rendered; the
  single-read marker flow regression (exactly one read, single tool
  pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
  the only PLAN edit this phase; the phase-37 note's budget clause is
  marked removed.

Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.

uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.

Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
This commit is contained in:
ducoterra
2026-08-28 04:50:56 -04:00
parent bc70ce36e0
commit b855d0aef9
16 changed files with 1311 additions and 278 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+158 -38
View File
@@ -46,8 +46,9 @@ Implements just enough of the aipi surface:
closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document
tools) **and** the system prompt carries the ``<tools>`` section ->
the deterministic tool-calling flow, discriminated statelessly from
the messages + the ``tools`` parameter:
the deterministic SINGLE-READ tool flow, discriminated statelessly
from the messages (the ``tools`` parameter gates the list/read
steps — a no-tools request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``list_documents`` (synthetic id
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
@@ -56,15 +57,37 @@ Implements just enough of the aipi surface:
parse the FIRST catalog line (``source/path — title`` → split on
``" — "`` → ``rsplit("/", 1)``) and stream a ``tool_calls`` delta
calling ``read_document`` on it (id ``call_1``);
* request 3 (the read result in the messages, no ``tools``
parameter): a content answer, deterministic: ``Read
<source/path>. <first 80 chars of the read document's content>``
— so a suite can assert the read document reached the model and
landed in the answer.
* request 3 (a ``tool``-role read result in the messages): a
content answer, deterministic: ``Read <source/path>. <first 80
chars of the read document's content>`` — so a suite can assert
the read document reached the model and landed in the answer.
Reached regardless of the ``tools`` parameter (phase 45 keeps
the tools offered until the round cap).
The single-read flow stops at ONE read result; the MULTI-READ
variant below reads two.
- user message containing BOTH ``use your tools`` AND ``read two
documents`` (``MULTI_READ_TRIGGER``, phase 45 task 02) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix):
* 0 read results, no catalog yet: ``list_documents`` (id
``call_0``);
* 0 read results, catalog present: ``read_document`` on the FIRST
catalog line (id ``call_1``);
* 1 read result: ``read_document`` on the SECOND catalog line —
the first listing line whose ``source/path`` differs from the
one already read (id ``call_2``); a one-document catalog
degenerates to the single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
<sp1> and <sp2>.`` naming both read paths in read order — so a
suite can assert the model used BOTH documents.
All other requests (including the marker without a ``<tools>``
section, or with the tool conversation not yet started and no tools
offered — e.g. budgets 0/0) behave exactly as today. ``E2E_REAL_LLM=1``
ignores the mock entirely (the real model does what it does).
offered — e.g. ``agent_max_rounds=0``) behave exactly as today.
``E2E_REAL_LLM=1`` ignores the mock entirely (the real model does
what it does).
- user message containing ``show me a table`` (phase 44, markdown
tables, TODO.md L6) -> the fixed table answer (``TABLE_ANSWER``):
a 3-column service table, an ``<img onerror>`` XSS probe line, and
@@ -174,6 +197,15 @@ _DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
#: contain the phrase, so every other suite is unaffected.
TOOLS_TRIGGER = "use your tools"
#: Phase 45 (agent-unlimited-tools story, task 02): a user message
#: containing BOTH ``TOOLS_TRIGGER`` and this substring (case-insensitive
#: — the check lowercases the user message) drives the deterministic
#: MULTI-READ tool flow (list → read #1 → read #2 → the forced answer
#: naming both read paths) — see the module docstring. The existing
#: phase-37 E2E question carries ``TOOLS_TRIGGER`` but not this phrase,
#: so the 3-step flow is untouched.
MULTI_READ_TRIGGER = "read two documents"
#: Phase 44 (markdown-tables story, TODO.md L6): a user message
#: containing this substring (case-insensitive) gets the fixed table
#: answer (``TABLE_ANSWER`` below) — a 3-column table, an XSS probe
@@ -211,46 +243,126 @@ TABLE_ANSWER = (
_READ_RESULT_PREFIX = "Document "
def _tool_flow(body: dict[str, Any]) -> tuple[str, str, str] | None:
"""Classify a marker request into one step of the tool flow (phase 37).
def _read_results(body: dict[str, Any]) -> list[tuple[str, str]]:
"""The read results in the messages, in order: ``(source/path, content)``.
Returns one of:
* ``("list", "", "")`` — ``tools`` are offered and no tool results
are in the messages yet: the model lists the catalog.
* ``("read", source, path)`` — a ``tool``-role catalog result is in
the messages: the model reads its FIRST ``source/path — title``
line (split on ``" — "``, then ``rsplit("/", 1)``).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document.
* ``None`` — not the marker flow: the request behaves exactly as
today (marker absent, no ``<tools>`` section, or a no-tools first
request — the budgets-0/0 path).
A read result is a ``tool``-role message whose content starts with
the agent's read-result prefix (``app.rag.agent`` ``_execute_tool``):
``"Document <source/path>:\n<content>"``. The header is stripped of
the prefix AND the trailing colon so the path stays clean.
"""
if TOOLS_TRIGGER not in _user(body).lower():
return None
if "<tools>" not in _system(body):
return None
tool_msgs = [m for m in _messages(body) if m.get("role") == "tool"]
for m in tool_msgs: # a read result means the forced-answer request
out: list[tuple[str, str]] = []
for m in _messages(body):
if m.get("role") != "tool":
continue
content = str(m.get("content") or "")
if content.startswith(_READ_RESULT_PREFIX):
# The header is "Document <source/path>:" — drop the prefix
# AND the trailing colon so the answer quotes a clean path.
header, _, doc_content = content.partition("\n")
sp = header[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
return ("answer", sp, doc_content)
if not body.get("tools"):
return None
for m in tool_msgs: # a catalog result means the read request
out.append((sp, doc_content))
return out
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
"""Every ``source/path`` in the catalog tool result, in listing order.
Catalog lines are ``source/path — title`` (the agent's
``list_documents`` output): split on ``" — "``, keep the head, and
recover ``(source, path)`` with ``rsplit("/", 1)`` (``rpartition``)
— the same convention the single-read flow's read step uses. The
``"N documents:"`` header line carries no ``/`` and is skipped; read-
result messages are full documents, not listings, and are skipped
too.
"""
docs: list[tuple[str, str]] = []
for m in _messages(body):
if m.get("role") != "tool":
continue
content = str(m.get("content") or "")
if content.startswith(_READ_RESULT_PREFIX):
continue
for line in content.splitlines():
head = line.split(" — ", 1)[0].strip()
if "/" in head:
source, _, path = head.rpartition("/")
if source and path:
return ("read", source, path)
docs.append((source, path))
return docs
def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a marker request into one step of the tool flow.
Single-read (phase 37 — the user message carries ``TOOLS_TRIGGER``
only):
* ``("list", "", "")`` — ``tools`` are offered and no tool results
are in the messages yet: the model lists the catalog.
* ``("read", source, path, "call_1")`` — a ``tool``-role catalog
result is in the messages: the model reads its FIRST
``source/path — title`` line (split on ``" — "``, then
``rsplit("/", 1)``).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document. Reached
regardless of the ``tools`` parameter (phase 45 keeps the tools
offered until the round cap).
Multi-read (phase 45 task 02 — the user message carries BOTH
``TOOLS_TRIGGER`` and ``MULTI_READ_TRIGGER``), classified by the
count of ``tool``-role read results:
* 0 read results: ``("list", "", "")`` (no catalog yet) or
``("read", source, path, "call_1")`` on the FIRST catalog doc.
* 1 read result: ``("read", source, path, "call_2")`` on the SECOND
catalog doc — the first listing line whose ``source/path``
differs from the one already read. A one-document catalog
degenerates to the single-read ``("answer", ...)`` shape (nothing
second to read).
* 2 read results: ``("multi_answer", "", text)`` — the forced
answer, byte-stable: the single-read shape quoting the FIRST read
result, plus ``I read <sp1> and <sp2>.`` (both read paths, read
order). The second element is unused.
* ``None`` — not the marker flow: the request behaves exactly as
before (marker absent, no ``<tools>`` section, or a no-tools
request with no tool results — e.g. ``agent_max_rounds=0``).
"""
user = _user(body).lower()
if TOOLS_TRIGGER not in user:
return None
if "<tools>" not in _system(body):
return None
reads = _read_results(body)
if MULTI_READ_TRIGGER in user:
if not reads:
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if not docs:
return ("list", "", "")
return ("read", docs[0][0], docs[0][1], "call_1")
if len(reads) == 1:
skip = reads[0][0]
second = next(
(d for d in _catalog_docs(body) if f"{d[0]}/{d[1]}" != skip), None
)
if second is None:
# One-document catalog: nothing second to read — the
# single-read answer shape (deterministic degenerate).
return ("answer", reads[0][0], reads[0][1])
return ("read", second[0], second[1], "call_2")
(sp1, c1), (sp2, _c2) = reads[0], reads[1]
answer = f"Read {sp1}. {c1[:80]} I read {sp1} and {sp2}."
return ("multi_answer", "", answer)
# Phase-37 single-read flow — byte-identical to the original.
if reads:
return ("answer", reads[0][0], reads[0][1])
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if docs:
return ("read", docs[0][0], docs[0][1], "call_1")
return ("list", "", "")
@@ -653,11 +765,19 @@ def chat_completions(body: dict[str, Any]) -> Any:
if flow[0] == "list":
stream = _tool_call_stream("list_documents", {}, "call_0")
elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
# "call_2" for the multi-read second read (phase 45,
# task 02).
stream = _tool_call_stream(
"read_document",
{"source": flow[1], "path": flow[2]},
"call_1",
flow[3],
)
elif flow[0] == "multi_answer":
# Phase 45 (task 02): the multi-read forced answer —
# computed in _tool_flow, byte-stable.
stream = _sse_stream(_apply_max_tokens(flow[2], body.get("max_tokens")), 0.0)
else: # "answer" — quote the read document (first 80 chars)
answer = _apply_max_tokens(
f"Read {flow[1]}. {flow[2][:80]}", body.get("max_tokens")