feat(rag): unbounded agent tool calls behind a round cap (owner revision)

Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:

- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
  negative rejected) replaces agent_list_calls / agent_read_calls;
  .env.example + README document the single knob; app/rag/prompts.py
  docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
  rounds >= max_rounds, at which point it forces one final no-tools
  answer (the cap is the only forced exit); 0 = no tools — exactly one
  tools=None request, byte-identical to the pre-phase-37 path (the
  kill switch). Rejected calls (unknown tool / missing args /
  already-in-context / unknown path) still consume a round, so
  pathological rejected-call streams are bounded by the cap. The
  per-call log line is now tool/args/round=N/M; the per-turn
  tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
  the deterministic list -> read #1 -> read #2 -> forced-answer flow
  (byte-stable "I read <sp1> and <sp2>." line), classified by the
  count of tool-role read results; the phase-37 single-read flow stays
  byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
  mock-only): three tool frames/lines in order (one list, two reads —
  the second read is what the old read budget refused) + the
  both-named non-deflected answer; done.sources + chips = retrieval
  doc + both reads, deduped; no budget refusal rendered; the
  single-read marker flow regression (exactly one read, single tool
  pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
  the only PLAN edit this phase; the phase-37 note's budget clause is
  marked removed.

Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.

uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.

Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
This commit is contained in:
2026-08-28 04:50:56 -04:00
parent bc70ce36e0
commit b855d0aef9
16 changed files with 1311 additions and 278 deletions
+11 -1
View File
@@ -183,11 +183,21 @@ stuck button).
> gains a second event type — `{"type":"tool","name":"…","argument":…}` — > gains a second event type — `{"type":"tool","name":"…","argument":…}` —
> carrying the model's document tool calls on grounded turns (phase 37: > carrying the model's document tool calls on grounded turns (phase 37:
> `list_documents` / `read_document`, budgeted by `BOR_AGENT_LIST_CALLS` > `list_documents` / `read_document`, budgeted by `BOR_AGENT_LIST_CALLS`
> / `BOR_AGENT_READ_CALLS`; `argument` is `"source/path"` for > / `BOR_AGENT_READ_CALLS` (removed in phase 45 — see the revision note
> below); `argument` is `"source/path"` for
> `read_document`, null otherwise). Client rule: render each `tool` frame > `read_document`, null otherwise). Client rule: render each `tool` frame
> as a "calling tool" line/state (task 05); `delta` and `done` shapes are > as a "calling tool" line/state (task 05); `delta` and `done` shapes are
> unchanged — the read document is reflected in `done.sources` instead > unchanged — the read document is reflected in `done.sources` instead
> (deduped) — a recorded extension of A15, not a silent deviation. > (deduped) — a recorded extension of A15, not a silent deviation.
>
> **SSE revision (phase 45, owner permission 2026-08-27):** the phase-37
> per-turn tool budgets are **removed** (owner: "allow the LLM to make
> as many tool calls as it wants — `TODO.md` L8): `BOR_AGENT_LIST_CALLS`
> / `BOR_AGENT_READ_CALLS` no longer exist; `BOR_AGENT_MAX_ROUNDS`
> (default 10) caps the tool rounds and `0` disables the tools
> entirely (the pre-phase-37 path). The `tool` event shape and the
> `done` shape are unchanged — a recorded revision of the phase-37
> note's budget wording, not a silent deviation.
--- ---
+1 -2
View File
@@ -38,8 +38,7 @@ BOR_HYBRID_LEXICAL_CANDIDATES=30 # FTS list width for the fusion
BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
# --- Agent document tools (phase 37: grounded turns may list + read) --- # --- Agent document tools (phase 37: grounded turns may list + read) ---
BOR_AGENT_LIST_CALLS=1 # per-turn list_documents opportunities (0 disables the tool) # BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)
BOR_AGENT_READ_CALLS=1 # per-turn read_document opportunities (0 disables the tool)
# --- Import scope (A9 formats; may only narrow, never widen) --- # --- Import scope (A9 formats; may only narrow, never widen) ---
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py # BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py
+10 -11
View File
@@ -151,7 +151,7 @@ server-side tools — on **grounded** (high-relevance) turns only:
* **`list_documents`** — lists every indexed document, one * **`list_documents`** — lists every indexed document, one
`source/path — title` line each (the same order as the Sources page); `source/path — title` line each (the same order as the Sources page);
* **`read_document(source, path)`** — appends the **full** text of * **`read_document(source, path)`** — appends the **full** text of
exactly one more indexed document to the context (never truncated). one more indexed document to the context (never truncated).
Each call the model requests is executed against Postgres only (no extra Each call the model requests is executed against Postgres only (no extra
LLM round trip) and streamed as an SSE `tool` frame ahead of the answer — LLM round trip) and streamed as an SSE `tool` frame ahead of the answer —
@@ -163,18 +163,18 @@ above the answer, one per call, in order. The tool lines persist with the
message, so a reloaded conversation re-renders them. The read document is message, so a reloaded conversation re-renders them. The read document is
reflected in the answer's **source chips** and in the `query_log` row. reflected in the answer's **source chips** and in the `query_log` row.
The opportunities are budgeted per turn: The tools stay offered for the whole turn — the model may call them as
many times as it needs (re-lists included), bounded only by a round cap
that stops a pathological infinite loop:
| Env | Default | Meaning | | Env | Default | Meaning |
|---|---|---| |---|---|---|
| `BOR_AGENT_LIST_CALLS` | `1` | `list_documents` calls per turn (0 disables the tool) | | `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn — every call the model emits consumes a round; at the cap the loop forces one final no-tools answer |
| `BOR_AGENT_READ_CALLS` | `1` | `read_document` calls per turn (0 disables the tool) |
Once both budgets are spent the tools are dropped from the LLM request `BOR_AGENT_MAX_ROUNDS=0` reproduces the pre-agent chat behavior exactly
and the model must answer. `BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0` (no `tools` in the request, no `tool` frames) — the kill switch.
reproduces the pre-agent chat behavior exactly (no `tools` in the Deflected turns run no tools at all — the low-relevance path is
request, no `tool` frames). Deflected turns run no tools at all — the unchanged.
low-relevance path is unchanged.
## Admin & sign-in ## Admin & sign-in
@@ -643,8 +643,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | cosine list width for the RRF fusion | | `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | cosine list width for the RRF fusion |
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion | | `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) | | `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_AGENT_LIST_CALLS` | `1` | per-turn `list_documents` tool opportunities on grounded turns (0 disables the tool) | | `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn — every call the model emits consumes a round; at the cap the loop forces one final no-tools answer (0 = no tools, the kill switch) |
| `BOR_AGENT_READ_CALLS` | `1` | per-turn `read_document` tool opportunities on grounded turns (0 disables the tool) |
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) | | `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) | | `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) | | `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
+21 -20
View File
@@ -48,26 +48,26 @@ outline (0 when absent) and the per-turn log line records
``kb_chars=N`` after ``tuning=N`` (PLAN §9 line extension). ``kb_chars=N`` after ``tuning=N`` (PLAN §9 line extension).
Agent document tools (phase 37, PLAN §4 extension, owner permission Agent document tools (phase 37, PLAN §4 extension, owner permission
2026-08-26): a **grounded** turn (``not plan.deflected``) no longer 2026-08-26; phase 45 removed the per-tool budgets — owner permission
2026-08-27): a **grounded** turn (``not plan.deflected``) no longer
streams a bare ``chat_stream`` — it runs the agent loop streams a bare ``chat_stream`` — it runs the agent loop
(``app.rag.agent.run_agent``), which offers the model the two (``app.rag.agent.run_agent``), which offers the model the two
server-side tools ``list_documents`` / ``read_document`` while the server-side tools ``list_documents`` / ``read_document`` for the whole
per-turn budgets (``BOR_AGENT_LIST_CALLS`` / ``BOR_AGENT_READ_CALLS``, turn (as many calls as the model wants, re-lists included) until it
default 1 each) last; once both budgets are spent the ``tools`` key is answers or the round cap (``BOR_AGENT_MAX_ROUNDS``, default 10) forces
dropped from the request and the model must answer. Each one final no-tools answer. Each model-requested call streams as an SSE
model-requested call streams as an SSE ``tool`` event — ``tool`` event — ``{"type": "tool", "name": …, "argument":
``{"type": "tool", "name": …, "argument": "source/path" | null}`` — "source/path" | null}`` — ahead of the answer's ``delta`` frames.
ahead of the answer's ``delta`` frames. ``done.sources``, ``done.sources``, ``query_log.sources`` and the per-turn log line all
``query_log.sources`` and the per-turn log line all report the same report the same combined source list (retrieval docs + the agent's
combined source list (retrieval docs + the agent's read docs, deduped read docs, deduped by ``(source, path)``, order preserved), and the log
by ``(source, path)``, order preserved), and the log line records line records ``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9 line
``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9 line extension — extension — ``N`` counts executed tool calls; rejected calls do not
``N`` counts budget-consuming executions; rejected calls do not count). **Deflected turns keep the direct ``chat_stream`` —
count). **Deflected turns keep the direct ``chat_stream`` — byte- byte-identical to the pre-phase path (A8):** the LOW prompt never
identical to the pre-phase path (A8):** the LOW prompt never carries carries tools, and with ``agent_max_rounds`` at **0** ``run_agent``
tools, and with **both budgets at 0** ``run_agent`` makes exactly one makes exactly one ``tools=None`` request, reproducing the pre-phase
``tools=None`` request, reproducing the pre-phase behavior (budgets- behavior (the kill switch).
as-kill-switch).
""" """
from __future__ import annotations from __future__ import annotations
@@ -299,8 +299,9 @@ async def chat(
# bare ``chat_stream`` — its ``ToolCallPiece``s stream as # bare ``chat_stream`` — its ``ToolCallPiece``s stream as
# ``tool`` events ahead of the answer. A deflected turn keeps # ``tool`` events ahead of the answer. A deflected turn keeps
# the direct ``chat_stream`` (byte-identical, A8): the LOW # the direct ``chat_stream`` (byte-identical, A8): the LOW
# prompt never carries tools, and with both budgets at 0 # prompt never carries tools, and with
# ``run_agent`` is a single ``tools=None`` request anyway. # ``agent_max_rounds=0`` ``run_agent`` is a single
# ``tools=None`` request anyway (the kill switch).
holder = AgentHolder() holder = AgentHolder()
answer_stream: AsyncIterator[StreamPiece | ToolCallPiece] answer_stream: AsyncIterator[StreamPiece | ToolCallPiece]
if plan.deflected: if plan.deflected:
+17 -7
View File
@@ -87,13 +87,15 @@ class Settings(BaseSettings):
#: ``app.rag.overview``). Overflow is cut at the cap and the shared #: ``app.rag.overview``). Overflow is cut at the cap and the shared
#: ``[…truncated…]`` marker is appended (summarizer convention). #: ``[…truncated…]`` marker is appended (summarizer convention).
overview_input_max_chars: int = 40_000 overview_input_max_chars: int = 40_000
#: Per-turn opportunities to call the ``list_documents`` agent tool #: Hard cap on the agent tool rounds per grounded turn (phase 45,
#: (phase 37, ``app.rag.agent``); 0 disables the tool entirely #: revising phase 37's per-tool budgets — owner permission
#: (pre-phase behavior with both budgets at 0). #: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls
agent_list_calls: int = 1 #: as it wants"). Every tool call the model emits consumes a
#: Per-turn opportunities to call the ``read_document`` agent tool #: round; at the cap the loop forces one final no-tools answer.
#: (phase 37, ``app.rag.agent``); 0 disables the tool entirely. #: ``0`` disables the tools entirely — the turn is a single
agent_read_calls: int = 1 #: request with ``tools=None`` (the pre-phase-37 path — the kill
#: switch). Negative values are rejected at startup (validator).
agent_max_rounds: int = 10
# --- Hybrid retrieval (A7, revised 2026-08-21) --- # --- Hybrid retrieval (A7, revised 2026-08-21) ---
# cosine top-N ∪ Postgres FTS top-N, fused with Reciprocal Rank Fusion # cosine top-N ∪ Postgres FTS top-N, fused with Reciprocal Rank Fusion
@@ -159,6 +161,14 @@ class Settings(BaseSettings):
) )
return v return v
@field_validator("agent_max_rounds")
@classmethod
def _agent_max_rounds_non_negative(cls, v: int) -> int:
"""``0`` is the no-tools kill switch — a negative value is a typo."""
if v < 0:
raise ValueError("agent_max_rounds must be >= 0 (0 = no tools)")
return v
# Suggested questions (onboarding + empty state). # Suggested questions (onboarding + empty state).
suggestions: list[str] = [ suggestions: list[str] = [
"How is my Kubernetes cluster set up?", "How is my Kubernetes cluster set up?",
+61 -67
View File
@@ -15,38 +15,42 @@ probe came back "supported".
Loop contract (one grounded chat turn; the API layer wires this in, Loop contract (one grounded chat turn; the API layer wires this in,
task 04): task 04):
1. While budget remains the model is offered the two OpenAI functions in 1. The model is offered the two OpenAI functions in :data:`AGENT_TOOLS`
:data:`AGENT_TOOLS`: up to ``settings.agent_list_calls`` for the whole turn — phase 45 removed the phase-37 per-tool budgets
(``BOR_AGENT_LIST_CALLS``, default 1) ``list_documents`` calls and up (owner permission 2026-08-27, ``TODO.md`` L8: "allow the LLM to make
to ``settings.agent_read_calls`` (``BOR_AGENT_READ_CALLS``, default 1) as many tool calls as it wants"): ``list_documents`` and
``read_document`` calls. With both budgets at 0 the loop makes exactly ``read_document`` can each be called as many times as the model needs,
one request with ``tools=None`` — byte-identical to the pre-phase chat re-lists included. With ``settings.agent_max_rounds``
path (budgets-as-kill-switch, phase 37 locked decision). (``BOR_AGENT_MAX_ROUNDS``, default 10) at 0 the loop makes exactly one
request with ``tools=None`` — byte-identical to the pre-phase-37 chat
path (the kill switch).
2. Each tool call the model emits is executed server-side against 2. Each tool call the model emits is executed server-side against
Postgres only (no LLM, no network): ``list_documents`` returns the Postgres only (no LLM, no network): ``list_documents`` returns the
indexed catalog — one ``source/path — title`` line per document, indexed catalog — one ``source/path — title`` line per document,
``GET /api/docs`` order (uncapped in v1; the UI never shows it, only ``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
the model does) — and ``read_document`` returns the document's **full** the model does) — and ``read_document`` returns the document's **full**
content (A7-revised contract: never truncated). content (A7-revised contract: never truncated).
3. Rejected calls consume **no** budget and get a one-line refusal: 3. Rejected calls get a one-line refusal and count in nothing
unknown tool name → ``"Unknown tool."``; missing ``source``/``path`` (``holder.tool_calls`` tracks executed calls only): unknown tool name
arguments; a document already in context (seed or previously read) → → ``"Unknown tool."``; missing ``source``/``path`` arguments; a
``"Already in your context."``; an unknown ``source/path`` → document already in context (seed or previously read) → ``"Already in
``"No document at …"``; an exhausted list/read budget → the matching your context."``; an unknown ``source/path`` → ``"No document at …"``.
``"No … budget left"`` refusal. A rejected call still consumes a *round* in the loop, so a
4. Every executed call is appended back to the message history as the pathological stream that keeps emitting rejected calls is bounded by
assistant tool-call message + the tool result, and the model is called the cap (point 4).
again. Once **both** budgets are spent, ``tools`` is dropped from the 4. Every call the model emits is appended back to the message history as
request and the model must answer. Belt-and-braces round cap: the assistant tool-call message + the tool result (refusals included),
``max_rounds = 2 + agent_list_calls + agent_read_calls`` (every tool consumes one round, and the model is called again. At the round cap —
round consumes a budget, so the cap only catches pathological streams ``max_rounds = settings.agent_max_rounds`` (``BOR_AGENT_MAX_ROUNDS``,
that keep calling rejected tools) — at the cap the loop forces one default 10) — the loop forces one final ``chat_stream(messages,
final ``chat_stream(messages, tools=None)`` and returns. tools=None)`` and returns: the cap is the **only** forced exit
(besides "the stream carried no calls"), and it bounds pathological
rejected-call streams.
5. A rare stream that carries both content and a tool call keeps the 5. A rare stream that carries both content and a tool call keeps the
content (it was already emitted) **and** still runs the tool. content (it was already emitted) **and** still runs the tool.
6. *holder* (an :class:`AgentHolder`) records the read documents and the 6. *holder* (an :class:`AgentHolder`) records the read documents and the
number of budget-consuming tool executions; the API layer (task 04) number of executed tool calls (re-lists included); the API layer
reads it after the stream to extend ``done.sources`` / (task 04) reads it after the stream to extend ``done.sources`` /
``query_log.sources`` and the per-turn log line (``tool_calls=N``). ``query_log.sources`` and the per-turn log line (``tool_calls=N``).
The DB accessors (:func:`list_catalog`, :func:`find_document`) are The DB accessors (:func:`list_catalog`, :func:`find_document`) are
@@ -71,8 +75,9 @@ from app.rag.llm import LLMClient, StreamPiece, ToolCallPiece
logger = logging.getLogger("app.agent") logger = logging.getLogger("app.agent")
#: The two agent tools (phase 37): OpenAI function definitions passed as #: The two agent tools (phase 37): OpenAI function definitions passed as
#: ``tools=AGENT_TOOLS`` to ``chat_stream`` while the per-turn budgets #: ``tools=AGENT_TOOLS`` to ``chat_stream`` for the whole grounded turn —
#: (``BOR_AGENT_LIST_CALLS`` / ``BOR_AGENT_READ_CALLS``) remain. #: phase 45 removed the per-tool budgets; the round cap
#: (``BOR_AGENT_MAX_ROUNDS``) is the only bound.
AGENT_TOOLS: list[dict[str, Any]] = [ AGENT_TOOLS: list[dict[str, Any]] = [
{ {
"type": "function", "type": "function",
@@ -90,7 +95,7 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"function": { "function": {
"name": "read_document", "name": "read_document",
"description": ( "description": (
"Add the full content of exactly one more indexed document " "Add the full content of one more indexed document "
"to your context" "to your context"
), ),
"parameters": { "parameters": {
@@ -117,9 +122,9 @@ AGENT_TOOLS: list[dict[str, Any]] = [
}, },
] ]
#: Tool refusal texts (phase 37): rejected calls consume no budget. #: Tool refusal texts (phase 37): rejected calls count in nothing
LIST_EXHAUSTED = "No listing budget left — answer with what you have." #: (``holder.tool_calls`` tracks executed calls); the round cap bounds
READ_EXHAUSTED = "No reading budget left — answer with what you have." #: their pathological repetition (phase 45).
ALREADY_IN_CONTEXT = "Already in your context." ALREADY_IN_CONTEXT = "Already in your context."
UNKNOWN_TOOL = "Unknown tool." UNKNOWN_TOOL = "Unknown tool."
MISSING_READ_ARGS = "read_document requires string arguments 'source' and 'path'." MISSING_READ_ARGS = "read_document requires string arguments 'source' and 'path'."
@@ -155,8 +160,8 @@ class AgentHolder:
``read_docs``: the documents ``read_document`` added to the context, ``read_docs``: the documents ``read_document`` added to the context,
in read order (deduped — re-reading a document appends nothing). in read order (deduped — re-reading a document appends nothing).
``tool_calls``: how many tool executions consumed budget; rejected ``tool_calls``: how many tool calls executed (re-lists included);
calls (unknown tool, unknown/missing document, exhausted budget, rejected calls (unknown tool, unknown/missing arguments or document,
already-in-context) do not count. Drives the per-turn log line's already-in-context) do not count. Drives the per-turn log line's
``tool_calls=N`` field (task 04). ``tool_calls=N`` field (task 04).
""" """
@@ -170,47 +175,40 @@ def _execute_tool(
call: ToolCallPiece, call: ToolCallPiece,
seed_docs: Sequence[Document], seed_docs: Sequence[Document],
holder: AgentHolder, holder: AgentHolder,
list_left: int, ) -> str:
read_left: int,
) -> tuple[str, int, int]:
"""Execute one tool call server-side (DB only). """Execute one tool call server-side (DB only).
Returns ``(result, list_left, read_left)``. Rejected calls consume no Returns the tool result text. A successful call bumps
budget; a successful read appends the :class:`Document` to ``holder.tool_calls`` (a successful read also appends the
``holder.read_docs`` and bumps ``holder.tool_calls``. :class:`Document` to ``holder.read_docs``); rejected calls return
their refusal line and count in nothing.
""" """
if call.name == "list_documents": if call.name == "list_documents":
if list_left <= 0:
return LIST_EXHAUSTED, list_left, read_left
rows = list_catalog(db) rows = list_catalog(db)
listing = f"{len(rows)} documents:\n" + "\n".join( listing = f"{len(rows)} documents:\n" + "\n".join(
f"{source}/{path} — {title}" for source, path, title in rows f"{source}/{path} — {title}" for source, path, title in rows
) )
holder.tool_calls += 1 holder.tool_calls += 1
return listing, list_left - 1, read_left return listing
if call.name == "read_document": if call.name == "read_document":
raw_source = call.arguments.get("source") raw_source = call.arguments.get("source")
raw_path = call.arguments.get("path") raw_path = call.arguments.get("path")
source = raw_source.strip() if isinstance(raw_source, str) else "" source = raw_source.strip() if isinstance(raw_source, str) else ""
path = raw_path.strip() if isinstance(raw_path, str) else "" path = raw_path.strip() if isinstance(raw_path, str) else ""
if not source or not path: if not source or not path:
return MISSING_READ_ARGS, list_left, read_left return MISSING_READ_ARGS
known = {(doc.source, doc.path) for doc in (*seed_docs, *holder.read_docs)} known = {(doc.source, doc.path) for doc in (*seed_docs, *holder.read_docs)}
if (source, path) in known: if (source, path) in known:
return ALREADY_IN_CONTEXT, list_left, read_left return ALREADY_IN_CONTEXT
if read_left <= 0:
return READ_EXHAUSTED, list_left, read_left
doc = find_document(db, source, path) doc = find_document(db, source, path)
if doc is None: if doc is None:
return ( return (
f"No document at {source}/{path} — check the list_documents output.", f"No document at {source}/{path} — check the list_documents output."
list_left,
read_left,
) )
holder.read_docs.append(doc) holder.read_docs.append(doc)
holder.tool_calls += 1 holder.tool_calls += 1
return f"Document {source}/{path}:\n{doc.content}", list_left, read_left - 1 return f"Document {source}/{path}:\n{doc.content}"
return UNKNOWN_TOOL, list_left, read_left return UNKNOWN_TOOL
async def run_agent( async def run_agent(
@@ -228,24 +226,23 @@ async def run_agent(
Every piece (``thinking`` / ``content`` / tool calls) is yielded as it Every piece (``thinking`` / ``content`` / tool calls) is yielded as it
arrives; the API layer (task 04) turns tool-call pieces into SSE arrives; the API layer (task 04) turns tool-call pieces into SSE
``tool`` events. After the loop finishes, *holder* carries the read ``tool`` events. After the loop finishes, *holder* carries the read
documents and the budget-consuming tool count. documents and the executed tool-call count (re-lists included).
``seed_docs`` are the documents the retrieval already put in context ``seed_docs`` are the documents the retrieval already put in context
(they shape the *system_prompt* the caller built); re-reading one of (they shape the *system_prompt* the caller built); re-reading one of
them is rejected as "Already in your context." without spending budget. them is rejected as "Already in your context." — the rejection counts
in nothing, but it still consumes a round.
""" """
messages: list[dict[str, Any]] = [ messages: list[dict[str, Any]] = [
{"role": "system", "content": system_prompt}, {"role": "system", "content": system_prompt},
{"role": "user", "content": user_message}, {"role": "user", "content": user_message},
] ]
list_left = settings.agent_list_calls # Phase 45: no per-tool budgets — the tools stay offered for the
read_left = settings.agent_read_calls # whole turn, bounded by the round cap. ``0`` is the no-tools kill
tools: list[dict[str, Any]] | None = AGENT_TOOLS if (list_left or read_left) else None # switch: exactly one request with ``tools=None`` (the pre-phase-37
# Every tool round consumes a budget, so this cap only catches # path).
# pathological streams that keep calling rejected tools (belt and max_rounds = settings.agent_max_rounds
# braces — the budgets already force the answer after tools: list[dict[str, Any]] | None = AGENT_TOOLS if max_rounds > 0 else None
# list + read rounds).
max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls
rounds = 0 rounds = 0
while True: while True:
calls: list[ToolCallPiece] = [] calls: list[ToolCallPiece] = []
@@ -258,15 +255,14 @@ async def run_agent(
if not calls: if not calls:
return # the answer was streamed return # the answer was streamed
call = calls[0] # a stream can carry several calls; run the first call = calls[0] # a stream can carry several calls; run the first
result, list_left, read_left = _execute_tool( result = _execute_tool(db, call, seed_docs, holder)
db, call, seed_docs, holder, list_left, read_left rounds += 1 # every call the model emits consumes a round
)
logger.info( logger.info(
"agent tool=%s args=%s budget list_left=%d read_left=%d", "agent tool=%s args=%s round=%d/%d",
call.name, call.name,
json.dumps(call.arguments, ensure_ascii=False)[:200], json.dumps(call.arguments, ensure_ascii=False)[:200],
list_left, rounds,
read_left, max_rounds,
) )
messages.append( messages.append(
{ {
@@ -285,8 +281,6 @@ async def run_agent(
} }
) )
messages.append({"role": "tool", "tool_call_id": call.id, "content": result}) messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS
rounds += 1
if rounds >= max_rounds: if rounds >= max_rounds:
logger.warning( logger.warning(
"agent round cap reached (rounds=%d) — forcing a final " "agent round cap reached (rounds=%d) — forcing a final "
+7 -7
View File
@@ -26,9 +26,9 @@ prompt is byte-identical to the pre-phase text.
Agent tools (phase 37): the **HIGH** prompt only carries a ``<tools>`` Agent tools (phase 37): the **HIGH** prompt only carries a ``<tools>``
section after the ``<documents>`` body — the grounded turn may call the section after the ``<documents>`` body — the grounded turn may call the
server-side ``list_documents`` / ``read_document`` tools (budgeted, see server-side ``list_documents`` / ``read_document`` tools (round-capped,
:mod:`app.rag.agent`). The LOW/deflection prompt never carries it and see :mod:`app.rag.agent`). The LOW/deflection prompt never carries it
stays byte-identical to the pre-phase text. and stays byte-identical to the pre-phase text.
""" """
from __future__ import annotations from __future__ import annotations
@@ -74,8 +74,8 @@ _KB_INTRO = (
#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37, #: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
#: task 03): a grounded turn may extend its context through the two #: task 03): a grounded turn may extend its context through the two
#: server-side tools (budgets: ``BOR_AGENT_LIST_CALLS`` / #: server-side tools (round cap: ``BOR_AGENT_MAX_ROUNDS``, see
#: ``BOR_AGENT_READ_CALLS``, see :mod:`app.rag.agent`). Appended after #: :mod:`app.rag.agent`). Appended after
#: the mode body (``<documents>``), so the instructions are the last #: the mode body (``<documents>``), so the instructions are the last
#: thing the model reads. The LOW/deflection prompt never carries it — #: thing the model reads. The LOW/deflection prompt never carries it —
#: a deflection has no grounded context to extend — and stays #: a deflection has no grounded context to extend — and stays
@@ -184,8 +184,8 @@ texts of the top documents + the ``<tools>`` instructions (phase 37).
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>`` Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their → ``<documents>`` → ``<tools>``; empty steering/overview omit their
section. ``<tools>`` is always present in the HIGH prompt (the section. ``<tools>`` is always present in the HIGH prompt (the round
budgets — not the prompt — decide whether the tools are actually cap — not the prompt — decides whether the tools are actually
offered to the model, see :mod:`app.rag.agent`). offered to the model, see :mod:`app.rag.agent`).
""" """
blocks = [ blocks = [
+158 -38
View File
@@ -46,8 +46,9 @@ Implements just enough of the aipi surface:
closing tag — same sentinel semantics.) closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document - user message containing ``use your tools`` (phase 37, agent document
tools) **and** the system prompt carries the ``<tools>`` section -> tools) **and** the system prompt carries the ``<tools>`` section ->
the deterministic tool-calling flow, discriminated statelessly from the deterministic SINGLE-READ tool flow, discriminated statelessly
the messages + the ``tools`` parameter: from the messages (the ``tools`` parameter gates the list/read
steps — a no-tools request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY * request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``list_documents`` (synthetic id ``tool_calls`` deltas — ``list_documents`` (synthetic id
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no ``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
@@ -56,15 +57,37 @@ Implements just enough of the aipi surface:
parse the FIRST catalog line (``source/path — title`` → split on parse the FIRST catalog line (``source/path — title`` → split on
``" — "`` → ``rsplit("/", 1)``) and stream a ``tool_calls`` delta ``" — "`` → ``rsplit("/", 1)``) and stream a ``tool_calls`` delta
calling ``read_document`` on it (id ``call_1``); calling ``read_document`` on it (id ``call_1``);
* request 3 (the read result in the messages, no ``tools`` * request 3 (a ``tool``-role read result in the messages): a
parameter): a content answer, deterministic: ``Read content answer, deterministic: ``Read <source/path>. <first 80
<source/path>. <first 80 chars of the read document's content>`` chars of the read document's content>`` — so a suite can assert
— so a suite can assert the read document reached the model and the read document reached the model and landed in the answer.
landed in the answer. Reached regardless of the ``tools`` parameter (phase 45 keeps
the tools offered until the round cap).
The single-read flow stops at ONE read result; the MULTI-READ
variant below reads two.
- user message containing BOTH ``use your tools`` AND ``read two
documents`` (``MULTI_READ_TRIGGER``, phase 45 task 02) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix):
* 0 read results, no catalog yet: ``list_documents`` (id
``call_0``);
* 0 read results, catalog present: ``read_document`` on the FIRST
catalog line (id ``call_1``);
* 1 read result: ``read_document`` on the SECOND catalog line —
the first listing line whose ``source/path`` differs from the
one already read (id ``call_2``); a one-document catalog
degenerates to the single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
<sp1> and <sp2>.`` naming both read paths in read order — so a
suite can assert the model used BOTH documents.
All other requests (including the marker without a ``<tools>`` All other requests (including the marker without a ``<tools>``
section, or with the tool conversation not yet started and no tools section, or with the tool conversation not yet started and no tools
offered — e.g. budgets 0/0) behave exactly as today. ``E2E_REAL_LLM=1`` offered — e.g. ``agent_max_rounds=0``) behave exactly as today.
ignores the mock entirely (the real model does what it does). ``E2E_REAL_LLM=1`` ignores the mock entirely (the real model does
what it does).
- user message containing ``show me a table`` (phase 44, markdown - user message containing ``show me a table`` (phase 44, markdown
tables, TODO.md L6) -> the fixed table answer (``TABLE_ANSWER``): tables, TODO.md L6) -> the fixed table answer (``TABLE_ANSWER``):
a 3-column service table, an ``<img onerror>`` XSS probe line, and a 3-column service table, an ``<img onerror>`` XSS probe line, and
@@ -174,6 +197,15 @@ _DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
#: contain the phrase, so every other suite is unaffected. #: contain the phrase, so every other suite is unaffected.
TOOLS_TRIGGER = "use your tools" TOOLS_TRIGGER = "use your tools"
#: Phase 45 (agent-unlimited-tools story, task 02): a user message
#: containing BOTH ``TOOLS_TRIGGER`` and this substring (case-insensitive
#: — the check lowercases the user message) drives the deterministic
#: MULTI-READ tool flow (list → read #1 → read #2 → the forced answer
#: naming both read paths) — see the module docstring. The existing
#: phase-37 E2E question carries ``TOOLS_TRIGGER`` but not this phrase,
#: so the 3-step flow is untouched.
MULTI_READ_TRIGGER = "read two documents"
#: Phase 44 (markdown-tables story, TODO.md L6): a user message #: Phase 44 (markdown-tables story, TODO.md L6): a user message
#: containing this substring (case-insensitive) gets the fixed table #: containing this substring (case-insensitive) gets the fixed table
#: answer (``TABLE_ANSWER`` below) — a 3-column table, an XSS probe #: answer (``TABLE_ANSWER`` below) — a 3-column table, an XSS probe
@@ -211,46 +243,126 @@ TABLE_ANSWER = (
_READ_RESULT_PREFIX = "Document " _READ_RESULT_PREFIX = "Document "
def _tool_flow(body: dict[str, Any]) -> tuple[str, str, str] | None: def _read_results(body: dict[str, Any]) -> list[tuple[str, str]]:
"""Classify a marker request into one step of the tool flow (phase 37). """The read results in the messages, in order: ``(source/path, content)``.
Returns one of: A read result is a ``tool``-role message whose content starts with
the agent's read-result prefix (``app.rag.agent`` ``_execute_tool``):
* ``("list", "", "")`` — ``tools`` are offered and no tool results ``"Document <source/path>:\n<content>"``. The header is stripped of
are in the messages yet: the model lists the catalog. the prefix AND the trailing colon so the path stays clean.
* ``("read", source, path)`` — a ``tool``-role catalog result is in
the messages: the model reads its FIRST ``source/path — title``
line (split on ``" — "``, then ``rsplit("/", 1)``).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document.
* ``None`` — not the marker flow: the request behaves exactly as
today (marker absent, no ``<tools>`` section, or a no-tools first
request — the budgets-0/0 path).
""" """
if TOOLS_TRIGGER not in _user(body).lower(): out: list[tuple[str, str]] = []
return None for m in _messages(body):
if "<tools>" not in _system(body): if m.get("role") != "tool":
return None continue
tool_msgs = [m for m in _messages(body) if m.get("role") == "tool"]
for m in tool_msgs: # a read result means the forced-answer request
content = str(m.get("content") or "") content = str(m.get("content") or "")
if content.startswith(_READ_RESULT_PREFIX): if content.startswith(_READ_RESULT_PREFIX):
# The header is "Document <source/path>:" — drop the prefix
# AND the trailing colon so the answer quotes a clean path.
header, _, doc_content = content.partition("\n") header, _, doc_content = content.partition("\n")
sp = header[len(_READ_RESULT_PREFIX):].strip().removesuffix(":") sp = header[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
return ("answer", sp, doc_content) out.append((sp, doc_content))
if not body.get("tools"): return out
return None
for m in tool_msgs: # a catalog result means the read request
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
"""Every ``source/path`` in the catalog tool result, in listing order.
Catalog lines are ``source/path — title`` (the agent's
``list_documents`` output): split on ``" — "``, keep the head, and
recover ``(source, path)`` with ``rsplit("/", 1)`` (``rpartition``)
— the same convention the single-read flow's read step uses. The
``"N documents:"`` header line carries no ``/`` and is skipped; read-
result messages are full documents, not listings, and are skipped
too.
"""
docs: list[tuple[str, str]] = []
for m in _messages(body):
if m.get("role") != "tool":
continue
content = str(m.get("content") or "") content = str(m.get("content") or "")
if content.startswith(_READ_RESULT_PREFIX):
continue
for line in content.splitlines(): for line in content.splitlines():
head = line.split(" — ", 1)[0].strip() head = line.split(" — ", 1)[0].strip()
if "/" in head: if "/" in head:
source, _, path = head.rpartition("/") source, _, path = head.rpartition("/")
if source and path: if source and path:
return ("read", source, path) docs.append((source, path))
return docs
def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a marker request into one step of the tool flow.
Single-read (phase 37 — the user message carries ``TOOLS_TRIGGER``
only):
* ``("list", "", "")`` — ``tools`` are offered and no tool results
are in the messages yet: the model lists the catalog.
* ``("read", source, path, "call_1")`` — a ``tool``-role catalog
result is in the messages: the model reads its FIRST
``source/path — title`` line (split on ``" — "``, then
``rsplit("/", 1)``).
* ``("answer", "source/path", content)`` — a ``tool``-role read
result (``"Document <source/path>:\n<content>"``) is in the
messages: the model answers, quoting the read document. Reached
regardless of the ``tools`` parameter (phase 45 keeps the tools
offered until the round cap).
Multi-read (phase 45 task 02 — the user message carries BOTH
``TOOLS_TRIGGER`` and ``MULTI_READ_TRIGGER``), classified by the
count of ``tool``-role read results:
* 0 read results: ``("list", "", "")`` (no catalog yet) or
``("read", source, path, "call_1")`` on the FIRST catalog doc.
* 1 read result: ``("read", source, path, "call_2")`` on the SECOND
catalog doc — the first listing line whose ``source/path``
differs from the one already read. A one-document catalog
degenerates to the single-read ``("answer", ...)`` shape (nothing
second to read).
* 2 read results: ``("multi_answer", "", text)`` — the forced
answer, byte-stable: the single-read shape quoting the FIRST read
result, plus ``I read <sp1> and <sp2>.`` (both read paths, read
order). The second element is unused.
* ``None`` — not the marker flow: the request behaves exactly as
before (marker absent, no ``<tools>`` section, or a no-tools
request with no tool results — e.g. ``agent_max_rounds=0``).
"""
user = _user(body).lower()
if TOOLS_TRIGGER not in user:
return None
if "<tools>" not in _system(body):
return None
reads = _read_results(body)
if MULTI_READ_TRIGGER in user:
if not reads:
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if not docs:
return ("list", "", "")
return ("read", docs[0][0], docs[0][1], "call_1")
if len(reads) == 1:
skip = reads[0][0]
second = next(
(d for d in _catalog_docs(body) if f"{d[0]}/{d[1]}" != skip), None
)
if second is None:
# One-document catalog: nothing second to read — the
# single-read answer shape (deterministic degenerate).
return ("answer", reads[0][0], reads[0][1])
return ("read", second[0], second[1], "call_2")
(sp1, c1), (sp2, _c2) = reads[0], reads[1]
answer = f"Read {sp1}. {c1[:80]} I read {sp1} and {sp2}."
return ("multi_answer", "", answer)
# Phase-37 single-read flow — byte-identical to the original.
if reads:
return ("answer", reads[0][0], reads[0][1])
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if docs:
return ("read", docs[0][0], docs[0][1], "call_1")
return ("list", "", "") return ("list", "", "")
@@ -653,11 +765,19 @@ def chat_completions(body: dict[str, Any]) -> Any:
if flow[0] == "list": if flow[0] == "list":
stream = _tool_call_stream("list_documents", {}, "call_0") stream = _tool_call_stream("list_documents", {}, "call_0")
elif flow[0] == "read": elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
# "call_2" for the multi-read second read (phase 45,
# task 02).
stream = _tool_call_stream( stream = _tool_call_stream(
"read_document", "read_document",
{"source": flow[1], "path": flow[2]}, {"source": flow[1], "path": flow[2]},
"call_1", flow[3],
) )
elif flow[0] == "multi_answer":
# Phase 45 (task 02): the multi-read forced answer —
# computed in _tool_flow, byte-stable.
stream = _sse_stream(_apply_max_tokens(flow[2], body.get("max_tokens")), 0.0)
else: # "answer" — quote the read document (first 80 chars) else: # "answer" — quote the read document (first 80 chars)
answer = _apply_max_tokens( answer = _apply_max_tokens(
f"Read {flow[1]}. {flow[2][:80]}", body.get("max_tokens") f"Read {flow[1]}. {flow[2][:80]}", body.get("max_tokens")
+585
View File
@@ -0,0 +1,585 @@
"""Phase 45 E2E (Playwright, mock-only): as many tool calls as the model wants.
Story: ``.agent/user_stories/agent-unlimited-tools.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic MULTI-READ marker flow in ``tests/e2e/mock_llm.py`` (user
message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
carries the ``<tools>`` section of the HIGH prompt):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read_document`` on
the FIRST catalog line (id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read_document`` on the
SECOND catalog line (id ``call_2``) — the pre-phase-45 per-tool
budgets would have refused exactly this second read (``No reading
budget left — answer with what you have.``);
4. request 4 (two read results) → the forced answer, byte-stable: the
single-read shape quoting the FIRST read result, plus the line
``I read <sp1> and <sp2>.`` naming both read paths in read order.
KB fixture (the ``test_agent_document_tools.py`` TRUNCATE-then-seed
pattern, grown to three documents):
* ``Deployments/aaa-record-shape.json`` — read #1: indexed (in the
agent's catalog, readable) but seeded WITHOUT chunks, so retrieval
never puts it in context; sorts FIRST in the catalog;
* ``Deployments/bbb-zone-sync.yaml`` — read #2: same shape; sorts
SECOND;
* ``Homelab/route53-notes.md`` — the ONLY retrievable document: one
chunk whose embedding is the mock's own bag-of-words vector (genuine
token overlap: the marker questions cosine ≈0.65/≈0.71 against it,
well past the E2E 0.30 threshold, and they FTS-match too) → the
grounded seed context.
Three documents (not two, as in phase 37) so BOTH reads land on
documents outside the seed: with a two-document corpus the second read
would be the already-in-context retrieval document and the agent would
answer "Already in your context." — a rejection, not the multi-read
flow this story proves.
Test → story mapping (Playwright Mapping Rule):
1. ``test_multi_read_turn`` — the turn streams THREE ``tool`` frames /
``.tool-call`` lines in order (one list — "is listing documents" —
and two reads — "is reading <source/path>" — the #send-status
transition recorded deterministically via MutationObserver), then a
final non-deflected answer containing the mock's byte-stable
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
the turn, no budget refusal anywhere.
2. ``test_done_sources_include_reads`` — the source chips under the
answer list the retrieval doc PLUS both read documents, deduped
(the phase-37 ``done.sources`` extension contract, now with 2
reads); the same combined list lands in ``query_log.sources``.
3. ``test_relist_allowed`` — the listing tool ran (its line rendered)
and no pre-phase-45 budget refusal ("… budget left") appears
anywhere in the message bubble or tool lines: the old
``LIST_EXHAUSTED`` / ``READ_EXHAUSTED`` refusal strings are gone
from the product (the source-level grep was task 01's job).
4. ``test_single_tool_flow_regression`` (phase 37) — the original
marker WITHOUT the multi-read trigger still answers after exactly
ONE read with its single tool pair (list + one read). The full
phase-37 suite runs unmodified in the regression pass.
"""
from __future__ import annotations
import hashlib
import json
import re
import time
from collections.abc import Callable
from datetime import UTC, datetime
from playwright.sync_api import Page, expect
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document, QueryLog
from tests.e2e.mock_llm import embed_text
# --------------------------------------------------------------------------
# Fixture documents (deterministic, token-controlled)
# --------------------------------------------------------------------------
READ1_SOURCE = "Deployments"
READ1_PATH = "aaa-record-shape.json"
READ1_SP = f"{READ1_SOURCE}/{READ1_PATH}"
READ2_SOURCE = "Deployments"
READ2_PATH = "bbb-zone-sync.yaml"
READ2_SP = f"{READ2_SOURCE}/{READ2_PATH}"
SEED_SOURCE = "Homelab"
SEED_PATH = "route53-notes.md"
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
#: The retrievable document: references the record shape "for the exact
#: JSON shape of reeselink.json" (the TODO failure, same story as the
#: phase-37 fixture). The repeated record-file lines carry the marker
#: questions' key tokens (aws, route53, hosted, zone, reeselink, json,
#: exact, shape) — verified ≈0.65 (multi question) / ≈0.71 (single
#: question) cosine against the mock's embeddings (E2E threshold 0.30)
#: plus FTS hits, so both turns are solidly grounded.
ROUTE53_CONTENT = (
"# AWS Route 53 Notes\n\n"
"## Record file\n\n"
+ (
"The aws route53 hosted zone for reeselink keeps every record in "
"reseelink.json — the exact JSON shape of reeselink.json is "
"documented in aaa-record-shape.json.\n"
)
* 10
+ "\n## Sync job\n\n"
"A cron job pushes reeselink.json to the aws route53 hosted zone "
"every fifteen minutes; the diff is applied through the route53 api.\n"
)
#: Read #1: the JSON shape. Its FIRST line is longer than 80 chars, so
#: the mock's first-80-chars quote is newline-free (the rendered-text
#: assertion matches it verbatim). Pinned by the assert below.
RECORD_CONTENT = (
'{"version": 4, "comment": "ReeseLink hosted zone records — the exact '
'JSON shape of reeselink.json",\n'
' "hosted_zone_id": "Z0RESEELINK45",\n'
' "record_sets": [\n'
' { "name": "www.reeselink.example", "type": "A", "ttl": 300 }\n'
' ]\n'
"}\n"
)
assert "\n" not in RECORD_CONTENT[:80] # the quote must stay one line
#: Read #2: the sync runbook.
RUNBOOK_CONTENT = (
"sync:\n"
" schedule: every fifteen minutes\n"
" target: reeselink.json\n"
" engine: aws route53 api\n"
" note: the diff is applied through the route53 api\n"
)
#: Carries BOTH markers — ``use your tools`` (phase 37) and ``read two
#: documents`` (phase 45 ``MULTI_READ_TRIGGER``).
MULTI_QUESTION = (
"Use your tools and read two documents: what is the exact JSON shape "
"of reeselink.json for my aws route53 hosted zone?"
)
#: The phase-37 marker WITHOUT the multi-read trigger — the original
#: 3-step single-read flow (regression test 4).
SINGLE_QUESTION = (
"Use your tools: what is the exact JSON shape of reeselink.json "
"for my aws route53 hosted zone?"
)
assert "use your tools" in MULTI_QUESTION.lower()
assert "read two documents" in MULTI_QUESTION.lower()
assert "read two documents" not in SINGLE_QUESTION.lower()
#: The mock's byte-stable multi-read answer pieces (mock_llm
#: ``_tool_flow``): the single-read shape quoting the FIRST read result,
#: plus both read paths in read order.
ANSWER_PREFIX = f"Read {READ1_SP}."
ANSWER_QUOTE = RECORD_CONTENT[:80]
BOTH_READS_LINE = f"I read {READ1_SP} and {READ2_SP}."
#: The pre-phase-45 budget refusals (phase 37 ``LIST_EXHAUSTED`` /
#: ``READ_EXHAUSTED``) — gone from the app (task 01) and never rendered
#: (test 3). The generic "budget left" fragment covers both exact
#: strings.
BUDGET_REFUSAL_FRAGMENTS = (
"No listing budget left — answer with what you have.",
"No reading budget left — answer with what you have.",
"budget left",
)
# The combined source list the app reports (app/api/chat.py): retrieval
# docs first, then the agent's read docs, deduped by (source, path).
EXPECTED_SOURCES = [
(SEED_SOURCE, SEED_PATH),
(READ1_SOURCE, READ1_PATH),
(READ2_SOURCE, READ2_PATH),
]
EXPECTED_SOURCES_LINE = ", ".join(f"{s}/{p}" for s, p in EXPECTED_SOURCES)
# --------------------------------------------------------------------------
# DB seeding (TRUNCATE-then-seed, cf. test_agent_document_tools.py)
# --------------------------------------------------------------------------
def _doc(source: str, path: str, title: str, content: str) -> Document:
return Document(
source=source,
path=path,
full_path=f"/tmp/{path}",
title=title,
content=content,
content_hash=hashlib.sha256(content.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
)
def _seed(db: Session) -> None:
"""The three-document KB from the module docstring."""
md = _doc(SEED_SOURCE, SEED_PATH, "AWS Route 53 Notes", ROUTE53_CONTENT)
db.add(md)
db.flush()
# One chunk carrying the mock's own embedding → genuine token
# overlap between the marker questions and this document (the only
# retrievable document — the grounded seed context).
db.add(
Chunk(
document_id=md.id,
position=0,
content=ROUTE53_CONTENT,
embedding=embed_text(ROUTE53_CONTENT),
)
)
# The two read documents: indexed, catalogued, readable — but NO
# chunks, so retrieval never puts them in context.
db.add(_doc(READ1_SOURCE, READ1_PATH, "Record Shape", RECORD_CONTENT))
db.add(_doc(READ2_SOURCE, READ2_PATH, "Zone Sync Runbook", RUNBOOK_CONTENT))
def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""Truncate the KB (plus the prompt-shaping tables), then re-seed.
``steering_notes`` / ``kb_overview`` are truncated too, so the HIGH
prompt is exactly ``<relevance>`` + ``<documents>`` + ``<tools>``
regardless of leftovers from other suites — byte-stable prompts,
byte-stable answers.
"""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
)
db.commit()
if seed is not None:
seed(db)
db.commit()
def _last_query_log() -> QueryLog:
with SessionLocal() as db:
rows = db.scalars(select(QueryLog)).all()
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
return rows[0]
# --------------------------------------------------------------------------
# Page helpers
# --------------------------------------------------------------------------
#: Records every value #send-label takes during the turn (a
#: MutationObserver on the element), so the transient "Calling tool…"
#: state is captured deterministically — no polling race (the phase-37
#: flake fix, phase 44 task 03).
LABEL_RECORDER = """
() => {
if (window.__labelsInstalled) return;
window.__labelsInstalled = true;
window.__labels = [];
const el = document.querySelector('#send-label');
if (!el) return;
const rec = (v) => {
const l = window.__labels;
if (!l.length || l[l.length - 1] !== v) l.push(v);
};
rec(el.textContent);
new MutationObserver(() => rec(el.textContent)).observe(el, {
childList: true,
subtree: true,
});
}
"""
#: Records every value #send-status takes during the turn — the
#: "… is listing documents" / "… is reading <source/path>" tool states
#: are transient (the first delta switches the status to the streaming
#: state), so the pre-submit observer is the deterministic source of
#: truth for their order.
STATUS_RECORDER = """
() => {
if (window.__statusesInstalled) return;
window.__statusesInstalled = true;
window.__statuses = [];
const el = document.querySelector('#send-status');
if (!el) return;
const rec = (v) => {
const l = window.__statuses;
if (!l.length || l[l.length - 1] !== v) l.push(v);
};
rec(el.textContent);
new MutationObserver(() => rec(el.textContent)).observe(el, {
childList: true,
subtree: true,
});
}
"""
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions for
#: the ``tool`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
"""Install all hooks on the loaded page (post-goto, pre-submit)."""
page.evaluate(SSE_HOOK)
page.evaluate(LABEL_RECORDER)
page.evaluate(STATUS_RECORDER)
def _frames(page: Page) -> list[dict]:
"""The captured SSE frames, once the hook's background read settles."""
deadline = time.monotonic() + 10.0
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send")
# --------------------------------------------------------------------------
# 1. The multi-read turn: list → read #1 → read #2 → both-named answer
# --------------------------------------------------------------------------
def test_multi_read_turn(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db(_seed)
page.goto(app_url)
_install_page_hooks(page)
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: exactly THREE `tool` frames — list, read #1, read #2,
# in order — and all ahead of the first `delta` frame. This third
# frame is the one the pre-phase-45 read budget refused.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ1_SP},
{"type": "tool", "name": "read_document", "argument": READ2_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# The transient "calling tool" states, recorded deterministically:
# the label shows "Calling tool…" and #send-status walked through
# "… is listing documents" then "… is reading <sp>" for BOTH reads,
# in order.
labels = page.evaluate("() => window.__labels")
assert "Calling tool…" in labels, labels
assert labels.index("Calling tool…") > labels.index("Thinking…")
statuses = page.evaluate("() => window.__statuses")
i_list = next(
(i for i, s in enumerate(statuses) if "is listing documents" in s), None
)
i_read1 = next(
(i for i, s in enumerate(statuses) if f"is reading {READ1_SP}" in s), None
)
i_read2 = next(
(i for i, s in enumerate(statuses) if f"is reading {READ2_SP}" in s), None
)
assert (
i_list is not None and i_read1 is not None and i_read2 is not None
), statuses
assert i_list < i_read1 < i_read2, statuses
# Three visible tool lines, in order, above the answer.
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ1_SP)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ2_SP)
# The final answer is non-deflected, quotes the FIRST read result,
# and names BOTH read paths (the mock's byte-stable line).
last = page.locator(".msg.brain").last
expect(last).not_to_have_class(re.compile(r"is-deflected"))
bubble = last.locator(".bubble")
expect(bubble).to_contain_text(ANSWER_PREFIX)
expect(bubble).to_contain_text(ANSWER_QUOTE)
expect(bubble).to_contain_text(BOTH_READS_LINE)
# Durable record: grounded, combined sources (retrieval + both
# reads).
row = _last_query_log()
assert row.question == MULTI_QUESTION
assert row.deflected is False
assert row.sources == EXPECTED_SOURCES_LINE
# --------------------------------------------------------------------------
# 2. done.sources / source chips: retrieval doc + BOTH reads, deduped
# --------------------------------------------------------------------------
def test_done_sources_include_reads(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db(_seed)
page.goto(app_url)
_install_page_hooks(page)
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: done.sources is the retrieval doc FIRST, then both
# read documents — deduped (the retrieval doc was never read, the
# reads are each read once; nothing appears twice).
frames = _frames(page)
done = next(f for f in frames if f.get("type") == "done")
assert [(s["source"], s["path"]) for s in done["sources"]] == EXPECTED_SOURCES
pairs = [(s["source"], s["path"]) for s in done["sources"]]
assert len(pairs) == len(set(pairs)), "done.sources must be deduped"
# UI: exactly three source chips under the answer, in the same
# order, each a viewer link — no duplicated chip.
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(3)
expect(chips.nth(0)).to_contain_text(SEED_SP)
expect(chips.nth(1)).to_contain_text(READ1_SP)
expect(chips.nth(2)).to_contain_text(READ2_SP)
for i, (source, path) in enumerate(EXPECTED_SOURCES):
expect(chips.nth(i)).to_have_attribute(
"href", f"/document.html?source={source}&path={path}&back=%2F"
)
# --------------------------------------------------------------------------
# 3. No budget refusal: the listing ran, and the pre-phase-45 refusal
# strings are nowhere in the rendered message
# --------------------------------------------------------------------------
def test_relist_allowed(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db(_seed)
page.goto(app_url)
_install_page_hooks(page)
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# The listing tool actually ran (its line rendered, its wire frame
# present) — and the turn completed past the point where the old
# per-tool budgets would have refused (list budget 1, read budget
# 1 — this turn makes one list and TWO reads).
frames = _frames(page)
assert {"type": "tool", "name": "list_documents", "argument": None} in _tool_frames(
frames
)
line0 = page.locator(".msg.brain .tool-call").nth(0)
expect(line0).to_contain_text("Listing documents")
# No pre-phase-45 budget refusal anywhere in the message — neither
# the exact old strings nor the generic fragment — not in the
# bubble, not in any tool line.
msg_text = page.locator(".msg.brain").last.text_content() or ""
for fragment in BUDGET_REFUSAL_FRAGMENTS:
assert fragment not in msg_text, (
f"budget refusal {fragment!r} rendered: {msg_text!r}"
)
# And it answered (a refusal would have left the model stuck — the
# turn settled with a non-deflected, both-named answer).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(BOTH_READS_LINE)
# --------------------------------------------------------------------------
# 4. Phase-37 regression: the single-read marker flow still answers
# after exactly ONE read with its single tool pair
# --------------------------------------------------------------------------
def test_single_tool_flow_regression(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_reset_db(_seed)
page.goto(app_url)
_install_page_hooks(page)
_submit(page, SINGLE_QUESTION)
_wait_settled(page)
# Exactly TWO tool frames — list then ONE read of the first catalog
# line — no second read (the marker carries no multi-read trigger).
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "list_documents", "argument": None},
{"type": "tool", "name": "read_document", "argument": READ1_SP},
]
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ1_SP)
# The single-read answer shape: quotes the read document; it does
# NOT carry the multi-read both-named line (READ2 was never read).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(ANSWER_PREFIX)
expect(bubble).to_contain_text(ANSWER_QUOTE)
expect(bubble).not_to_contain_text(BOTH_READS_LINE)
expect(bubble).not_to_contain_text(READ2_SP)
# done: non-deflected; sources = retrieval doc + the single read
# (READ2 absent — it was never read).
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH),
(READ1_SOURCE, READ1_PATH),
]
row = _last_query_log()
assert row.question == SINGLE_QUESTION
assert row.deflected is False
assert row.sources == f"{SEED_SP}, {READ1_SP}"
+20 -19
View File
@@ -78,10 +78,10 @@ class FakeRagLLM:
#: is yielded for the *i*-th request that carries a non-None #: is yielded for the *i*-th request that carries a non-None
#: ``tools`` parameter (a request the agent loop is offering tools #: ``tools`` parameter (a request the agent loop is offering tools
#: on). A request without tools — the deflected direct path, the #: on). A request without tools — the deflected direct path, the
#: post-budget answer request, or the 0/0 single-request path — #: cap-forced answer request, or the kill-switch
#: always yields the thinking + answer stream below, so a #: (``agent_max_rounds=0``) single-request path — always yields the
#: deflected turn through this fake is byte-identical to the #: thinking + answer stream below, so a deflected turn through this
#: plain fake's output. #: fake is byte-identical to the plain fake's output.
self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or []) self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or [])
async def embed(self, texts: list[str]) -> list[list[float]]: async def embed(self, texts: list[str]) -> list[list[float]]:
@@ -503,8 +503,8 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the ``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
``query_log`` row include the read document (deduped, order ``query_log`` row include the read document (deduped, order
preserved); the per-turn log line carries ``tool_calls=2``. preserved); the per-turn log line carries ``tool_calls=2``.
The agent loop offers tools while budgets last and drops them Phase 45: the agent loop keeps offering the tools for the whole
(``tools=None``) once both are spent.""" turn — the round cap (not per-tool budgets) is the bound."""
scripted = FakeRagLLM( scripted = FakeRagLLM(
tool_script=[ tool_script=[
[ [
@@ -518,8 +518,9 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
arguments={"source": "docs", "path": "homelab/backups.md"}, arguments={"source": "docs", "path": "homelab/backups.md"},
) )
], ],
# the post-budget answer request (tools=None) falls back to the # the answer request still carries the tools (2 rounds < the
# fake's thinking + answer stream # default cap of 10); the fake's tool_script is exhausted, so
# it falls back to the thinking + answer stream
] ]
) )
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
@@ -557,12 +558,13 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
assert len(sources) == len(set(sources)) # deduped by (source, path) assert len(sources) == len(set(sources)) # deduped by (source, path)
assert done["sources"][-1]["title"] == "Backup Strategy" assert done["sources"][-1]["title"] == "Backup Strategy"
# The agent loop offered the tools while any budget remained and # Phase 45: the tools stay offered on every request — the round cap
# dropped them once both were spent (single post-budget request). # (not spent budgets) bounds the loop, and the model answered while
# still being offered the tools (2 rounds < default cap 10).
assert len(scripted.seen_messages) == 3 assert len(scripted.seen_messages) == 3
assert scripted.seen_tools[0] == AGENT_TOOLS assert scripted.seen_tools[0] == AGENT_TOOLS
assert scripted.seen_tools[1] == AGENT_TOOLS # the read budget was still open assert scripted.seen_tools[1] == AGENT_TOOLS
assert scripted.seen_tools[2] is None assert scripted.seen_tools[2] == AGENT_TOOLS
# The query_log row carries the same combined source list. # The query_log row carries the same combined source list.
(row,) = db.scalars(select(QueryLog)).all() (row,) = db.scalars(select(QueryLog)).all()
@@ -628,17 +630,17 @@ def test_deflected_turn_stays_byte_identical_without_tools(
assert "backups.md" not in row.sources assert "backups.md" not in row.sources
def test_zero_agent_budgets_reproduce_pre_phase_single_request( def test_zero_max_rounds_reproduce_pre_phase_single_request(
client, client,
db, db,
seeded_kb: FakeRagLLM, seeded_kb: FakeRagLLM,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture, caplog: pytest.LogCaptureFixture,
) -> None: ) -> None:
"""(c) ``BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0``: no ``tool`` """(c) ``BOR_AGENT_MAX_ROUNDS=0``: no ``tool`` frames, exactly one
frames, exactly one request **without** a ``tools`` key (the request **without** a ``tools`` key (the pre-phase request shape),
pre-phase request shape), ``done.sources`` unchanged, and ``done.sources`` unchanged, and ``tool_calls=0`` in the log line —
``tool_calls=0`` in the log line — budgets-as-kill-switch.""" the kill switch survives the phase-45 budget removal."""
scripted = FakeRagLLM( scripted = FakeRagLLM(
tool_script=[ tool_script=[
[ToolCallPiece(id="call_1", name="list_documents", arguments={})], [ToolCallPiece(id="call_1", name="list_documents", arguments={})],
@@ -658,8 +660,7 @@ def test_zero_agent_budgets_reproduce_pre_phase_single_request(
lambda: Settings( lambda: Settings(
_env_file=None, # pyright: ignore[reportCallIssue] _env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=live.relevance_threshold, relevance_threshold=live.relevance_threshold,
agent_list_calls=0, agent_max_rounds=0,
agent_read_calls=0,
), ),
) )
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
+155 -103
View File
@@ -3,10 +3,13 @@
A scripted fake LLM (canned stream sequences) + monkeypatched A scripted fake LLM (canned stream sequences) + monkeypatched
``list_catalog`` / ``find_document`` — no database, no network. Covers ``list_catalog`` / ``find_document`` — no database, no network. Covers
the loop mechanics: the list → read → answer happy path (event order, the loop mechanics: the list → read → answer happy path (event order,
holder state, the ``tools=None`` request after the budgets are spent, holder state, the tools staying offered on every request — phase 45
the assistant/tool message history), the 0/0 single-call path, budget removed the per-tool budgets, the assistant/tool message history), the
exhaustion, dedupe, unknown tool / missing args / unknown path, the kill switch (``agent_max_rounds=0`` single-call path), the round cap
round cap, and the ``<tools>`` prompt section (HIGH only). forcing a final no-tools answer (an always-calling stream and an
always-rejected stream), re-lists and multi-reads executing without
budgets, dedupe, unknown tool / missing args / unknown path, and the
``<tools>`` prompt section (HIGH only).
""" """
from __future__ import annotations from __future__ import annotations
@@ -102,6 +105,11 @@ def test_agent_tools_names_and_parameters() -> None:
read_params = by_name["read_document"]["function"]["parameters"] read_params = by_name["read_document"]["function"]["parameters"]
assert read_params["required"] == ["source", "path"] assert read_params["required"] == ["source", "path"]
assert set(read_params["properties"]) == {"source", "path"} assert set(read_params["properties"]) == {"source", "path"}
# Phase 45: the per-tool budgets are gone — "exactly one more"
# dropped out of the read_document description.
assert by_name["read_document"]["function"]["description"] == (
"Add the full content of one more indexed document to your context"
)
# ---------- happy path: list → read → answer ---------- # ---------- happy path: list → read → answer ----------
@@ -148,11 +156,12 @@ def test_list_then_read_then_answer(
assert holder.read_docs == [target] assert holder.read_docs == [target]
assert holder.tool_calls == 2 assert holder.tool_calls == 2
# Default budgets (1/1): tools offered while any budget remains… # Phase 45: no per-tool budgets — the tools stay offered on every
# request (the round cap, not spent budgets, bounds the loop), so
# the answer request still carries them (2 rounds < default cap 10).
assert llm.requests[0][1] == AGENT_TOOLS assert llm.requests[0][1] == AGENT_TOOLS
assert llm.requests[1][1] == AGENT_TOOLS assert llm.requests[1][1] == AGENT_TOOLS
# …and dropped (tools=None) once both are spent. assert llm.requests[2][1] == AGENT_TOOLS
assert llm.requests[2][1] is None
assert len(llm.requests) == 3 assert len(llm.requests) == 3
# The follow-up request carries the assistant tool-call + tool result. # The follow-up request carries the assistant tool-call + tool result.
@@ -226,17 +235,53 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
assert llm.requests[1][0][3]["content"] == "0 documents:\n" assert llm.requests[1][0][3]["content"] == "0 documents:\n"
# ---------- budgets ---------- # ---------- round cap (phase 45: replaces the per-tool budgets) ----------
def test_zero_budgets_is_one_request_without_tools() -> None: def test_always_list_bounded_by_round_cap(monkeypatch: pytest.MonkeyPatch) -> None:
"""BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0 → byte-identical """A model that keeps calling ``list_documents`` gets exactly
single-call path: exactly one request, tools=None, no history growth.""" ``agent_max_rounds`` tool rounds, then one forced ``tools=None``
request streams the answer — the cap is the only forced exit."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
listing = "1 documents:\nS/a.md — A"
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
[ToolCallPiece(id="call_3", name="list_documents", arguments={})],
[StreamPiece("content", "forced answer")],
)
pieces = asyncio.run(_run(llm, holder, _settings(agent_max_rounds=3)))
assert [type(p) for p in pieces] == [
ToolCallPiece,
ToolCallPiece,
ToolCallPiece,
StreamPiece,
]
assert len(llm.requests) == 4 # 3 tool rounds + the forced answer
# The three tool rounds were offered the tools…
assert llm.requests[0][1] == AGENT_TOOLS
assert llm.requests[1][1] == AGENT_TOOLS
assert llm.requests[2][1] == AGENT_TOOLS
# …and the forced final request carries no tools, whatever is left.
assert llm.requests[3][1] is None
# Every re-list executed and counted.
assert holder.tool_calls == 3
# The final request carries all three executed listings as history.
final_msgs = llm.requests[3][0]
assert len(final_msgs) == 8 # 2 + 3 rounds × (assistant + tool)
assert final_msgs[3]["content"] == listing
assert final_msgs[5]["content"] == listing
assert final_msgs[7]["content"] == listing
def test_zero_max_rounds_is_one_request_without_tools() -> None:
"""``agent_max_rounds=0`` — the kill switch: exactly one request,
``tools=None``, no tool lines, no history growth (byte-identical to
the pre-phase-37 path)."""
holder = AgentHolder() holder = AgentHolder()
llm = ScriptedLLM([StreamPiece("thinking", "t "), StreamPiece("content", "direct answer")]) llm = ScriptedLLM([StreamPiece("thinking", "t "), StreamPiece("content", "direct answer")])
pieces = asyncio.run( pieces = asyncio.run(_run(llm, holder, _settings(agent_max_rounds=0)))
_run(llm, holder, _settings(agent_list_calls=0, agent_read_calls=0))
)
assert [type(p) for p in pieces] == [StreamPiece, StreamPiece] assert [type(p) for p in pieces] == [StreamPiece, StreamPiece]
assert len(llm.requests) == 1 assert len(llm.requests) == 1
assert llm.requests[0][1] is None assert llm.requests[0][1] is None
@@ -247,12 +292,92 @@ def test_zero_budgets_is_one_request_without_tools() -> None:
assert holder.read_docs == [] and holder.tool_calls == 0 assert holder.read_docs == [] and holder.tool_calls == 0
def test_read_budget_exhausted_refuses_and_appends_nothing( def test_rejected_read_spam_runs_to_round_cap(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
"""Every call rejected (unknown path — "No document at …"): rejections
no longer end the loop early via budgets — the round cap bounds them
and forces the final no-tools answer."""
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="read_document",
arguments={"source": "S", "path": "ghost.md"},
)
],
[
ToolCallPiece(
id="call_2",
name="read_document",
arguments={"source": "S", "path": "ghost.md"},
)
],
[
ToolCallPiece(
id="call_3",
name="read_document",
arguments={"source": "S", "path": "ghost.md"},
)
],
[StreamPiece("content", "forced answer")],
)
asyncio.run(_run(llm, holder, _settings(agent_max_rounds=3)))
assert len(llm.requests) == 4 # 3 rejected rounds + the forced answer
assert llm.requests[0][1] == AGENT_TOOLS
assert llm.requests[1][1] == AGENT_TOOLS
assert llm.requests[2][1] == AGENT_TOOLS
assert llm.requests[3][1] is None # the forced final request: no tools
assert holder.read_docs == [] and holder.tool_calls == 0 # nothing executed
refusal = "No document at S/ghost.md — check the list_documents output."
assert llm.requests[1][0][3]["content"] == refusal
assert llm.requests[2][0][5]["content"] == refusal
assert llm.requests[3][0][7]["content"] == refusal
# ---------- unlimited calls: re-lists and multi-reads (phase 45) ----------
def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
"""Re-lists execute — a second ``list_documents`` in one turn returns
the catalog again and counts in ``tool_calls`` (no budget to
exhaust)."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 2 # both re-lists executed and counted
listing = (
"2 documents:\n"
"Deployments/backups.md — Backup Strategy\n"
"Homelab/aws-route53.md — AWS Route53 Records"
)
# The answer request carries the catalog a second time as a tool result.
assert llm.requests[2][0][3]["content"] == listing # first listing
assert llm.requests[2][0][5]["content"] == listing # the re-list
assert llm.requests[2][1] == AGENT_TOOLS # still offered (no budgets)
def test_multi_read_executes_without_budgets(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Reads are no longer budgeted either — two different documents can
be read in one turn (re-reading the same one is still deduped via
ALREADY_IN_CONTEXT — see the rejection tests)."""
a = _doc("S", "a.md", "A", "A-CONTENT") a = _doc("S", "a.md", "A", "A-CONTENT")
b = _doc("S", "b.md", "B", "B-CONTENT")
monkeypatch.setattr( monkeypatch.setattr(
agent, "find_document", lambda db, source, path: a if path == "a.md" else None agent, "find_document", lambda db, source, path: {"a.md": a, "b.md": b}[path]
) )
holder = AgentHolder() holder = AgentHolder()
llm = ScriptedLLM( llm = ScriptedLLM(
@@ -268,38 +393,15 @@ def test_read_budget_exhausted_refuses_and_appends_nothing(
], ],
[StreamPiece("content", "ans")], [StreamPiece("content", "ans")],
) )
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1))) asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [a, b] # both reads appended, in order
assert holder.read_docs == [a] # the refused read appended nothing assert holder.tool_calls == 2
assert holder.tool_calls == 1 # …and consumed no budget assert llm.requests[1][0][3]["content"] == "Document S/a.md:\nA-CONTENT"
refusal = llm.requests[2][0][5] assert llm.requests[2][0][5]["content"] == "Document S/b.md:\nB-CONTENT"
assert refusal == { assert llm.requests[2][1] == AGENT_TOOLS # the second read was still offered
"role": "tool",
"tool_call_id": "call_2",
"content": agent.READ_EXHAUSTED,
}
# The list budget is still open, so tools stay offered after the refusal.
assert llm.requests[2][1] == AGENT_TOOLS
def test_list_budget_exhausted_refuses_with_its_own_message( # ---------- rejections (non-budget; the round cap bounds their repetition) ----------
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
assert holder.tool_calls == 1
assert llm.requests[2][0][5]["content"] == agent.LIST_EXHAUSTED
# The read budget is still open, so tools stay offered after the refusal.
assert llm.requests[2][1] == AGENT_TOOLS
# ---------- rejections (no budget consumed) ----------
def test_reading_a_seed_doc_is_already_in_context( def test_reading_a_seed_doc_is_already_in_context(
@@ -326,7 +428,8 @@ def test_reading_a_seed_doc_is_already_in_context(
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed)) asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
assert holder.read_docs == [] and holder.tool_calls == 0 assert holder.read_docs == [] and holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
# No budget consumed → tools are still offered on the next request. # Rejected → the tools are still offered on the next request (the
# round cap is the only bound).
assert llm.requests[1][1] == AGENT_TOOLS assert llm.requests[1][1] == AGENT_TOOLS
@@ -349,15 +452,15 @@ def test_reading_an_already_read_doc_is_deduped(
], ],
[StreamPiece("content", "ans")], [StreamPiece("content", "ans")],
) )
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1))) asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [doc] # appended exactly once assert holder.read_docs == [doc] # appended exactly once
assert holder.tool_calls == 1 assert holder.tool_calls == 1
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
# The read budget is intact after the deduped refusal… # Rejected → the tools are still offered on the next request…
assert llm.requests[2][1] == AGENT_TOOLS assert llm.requests[2][1] == AGENT_TOOLS
def test_unknown_path_refused_without_budget( def test_unknown_path_refused(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None) monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
@@ -378,7 +481,7 @@ def test_unknown_path_refused_without_budget(
llm.requests[1][0][3]["content"] llm.requests[1][0][3]["content"]
== "No document at S/ghost.md — check the list_documents output." == "No document at S/ghost.md — check the list_documents output."
) )
assert llm.requests[1][1] == AGENT_TOOLS # budget intact assert llm.requests[1][1] == AGENT_TOOLS # tools stay offered (cap bounds)
def test_unknown_tool_name_refused( def test_unknown_tool_name_refused(
@@ -393,7 +496,7 @@ def test_unknown_tool_name_refused(
asyncio.run(_run(llm, holder, _settings())) asyncio.run(_run(llm, holder, _settings()))
assert holder.read_docs == [] and holder.tool_calls == 0 assert holder.read_docs == [] and holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == agent.UNKNOWN_TOOL assert llm.requests[1][0][3]["content"] == agent.UNKNOWN_TOOL
assert llm.requests[1][1] == AGENT_TOOLS # nothing was consumed assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
@pytest.mark.parametrize( @pytest.mark.parametrize(
@@ -426,57 +529,6 @@ def test_read_document_missing_arguments_refused(
assert llm.requests[1][1] == AGENT_TOOLS assert llm.requests[1][1] == AGENT_TOOLS
# ---------- round cap (pathological stream) ----------
def test_round_cap_forces_a_final_no_tools_answer(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A model that keeps calling a budget-exhausted tool must be forced
to answer at ``max_rounds = 2 + list + read`` (= 4 for 1/1)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
[ToolCallPiece(id="call_3", name="list_documents", arguments={})],
[ToolCallPiece(id="call_4", name="list_documents", arguments={})],
[StreamPiece("content", "forced answer")],
)
pieces = asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
assert [type(p) for p in pieces] == [
ToolCallPiece,
ToolCallPiece,
ToolCallPiece,
ToolCallPiece,
StreamPiece,
]
assert len(llm.requests) == 5
# The forced final request carries no tools, whatever is left.
assert llm.requests[4][1] is None
# Only the first call consumed budget; the three rejections did not.
assert holder.tool_calls == 1
# The 4th rejection sits at messages[2 + 4*2 - 1] of the final request.
assert llm.requests[4][0][9]["content"] == agent.LIST_EXHAUSTED
# ---------- settings ----------
def test_agent_budget_settings_default_to_one_each() -> None:
s = _settings()
assert s.agent_list_calls == 1
assert s.agent_read_calls == 1
def test_agent_budget_settings_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("BOR_AGENT_LIST_CALLS", "0")
monkeypatch.setenv("BOR_AGENT_READ_CALLS", "2")
s = _settings()
assert s.agent_list_calls == 0
assert s.agent_read_calls == 2
# ---------- prompts: <tools> section (HIGH only) ---------- # ---------- prompts: <tools> section (HIGH only) ----------
+4 -3
View File
@@ -563,9 +563,10 @@ def test_endpoint_grounded_turn_runs_agent_loop_with_tools(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
"""Phase 37: a grounded endpoint turn runs the agent loop — the """Phase 37: a grounded endpoint turn runs the agent loop — the
single no-tool-call request carries ``AGENT_TOOLS`` (default 1/1 single no-tool-call request carries ``AGENT_TOOLS`` (the default
budgets), no ``tool`` frames stream, and the ``done`` event is the round cap keeps the tools offered), no ``tool`` frames stream, and
plain retrieval shape (the tool-free answer is byte-identical).""" the ``done`` event is the plain retrieval shape (the tool-free
answer is byte-identical)."""
_session, llm = gate_env _session, llm = gate_env
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT") doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.90)])) monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.90)]))
+20
View File
@@ -78,6 +78,26 @@ def test_max_output_tokens_env_override(monkeypatch) -> None:
assert s.max_output_tokens == 1234 assert s.max_output_tokens == 1234
def test_agent_max_rounds_default_and_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 45: the per-tool budgets are gone — ``BOR_AGENT_MAX_ROUNDS``
(default 10) is the single agent-loop knob; ``0`` is the no-tools
kill switch."""
monkeypatch.delenv("BOR_AGENT_MAX_ROUNDS", raising=False)
assert _settings().agent_max_rounds == 10
monkeypatch.setenv("BOR_AGENT_MAX_ROUNDS", "5")
assert _settings().agent_max_rounds == 5
monkeypatch.setenv("BOR_AGENT_MAX_ROUNDS", "0")
assert _settings().agent_max_rounds == 0
def test_agent_max_rounds_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None:
"""``0`` is the kill switch — a negative value is a typo, so the
validator fails loudly at startup."""
monkeypatch.setenv("BOR_AGENT_MAX_ROUNDS", "-1")
with pytest.raises(ValidationError, match="agent_max_rounds"):
_settings()
def test_stream_thinking_default_true_and_env_parse(monkeypatch: pytest.MonkeyPatch) -> None: def test_stream_thinking_default_true_and_env_parse(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 17 kill-switch (``BOR_STREAM_THINKING``): on by default, """Phase 17 kill-switch (``BOR_STREAM_THINKING``): on by default,
``0``/``false`` turn the ``thinking`` SSE frames off.""" ``0``/``false`` turn the ``thinking`` SSE frames off."""
+241
View File
@@ -0,0 +1,241 @@
"""Unit tests for the E2E mock's tool-flow classifier (phase 45, task 02).
The mock (``tests/e2e/mock_llm.py``) classifies marker requests
statelessly into one step of the agent tool flow. This file pins the
classification at unit speed — no Playwright, no LLM process:
* the phase-37 SINGLE-READ flow (``TOOLS_TRIGGER`` only) stays
byte-identical: list → read (first catalog line, ``call_1``) → answer;
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` + ``MULTI_READ_TRIGGER``)
classifies by the count of ``tool``-role read results: list → read #1
(``call_1``) → read #2 (second catalog line, ``call_2``) → the
byte-stable ``multi_answer`` naming both read paths.
"""
from __future__ import annotations
from typing import Any
from tests.e2e.mock_llm import (
MULTI_READ_TRIGGER,
TOOLS_TRIGGER,
_tool_flow,
)
# --------------------------------------------------------------------------
# Wire fixtures — byte-identical to what app/rag/agent.py produces
# --------------------------------------------------------------------------
#: The ``<tools>`` section marks the HIGH prompt (app/rag/prompts.py).
SYSTEM_HIGH = "<relevance>HIGH</relevance>\n<documents>\n</documents>\n<tools>\n…\n</tools>"
SYSTEM_LOW = "<relevance>LOW</relevance>\n"
#: A minimal truthy ``tools`` parameter (the mock only checks presence).
TOOLS = [{"type": "function", "function": {"name": "list_documents"}}]
#: The agent's ``list_documents`` output for a two-document KB
# (``app/rag/agent.py`` ``_execute_tool``): one ``source/path — title``
#: line per document, ``(source, path)`` order.
CATALOG_2 = (
"2 documents:\n"
"Deployments/example-record-file.json — Example Record File\n"
"Homelab/aws-route53.md — AWS Route 53 Notes"
)
CATALOG_1 = "1 documents:\nDeployments/example-record-file.json — Example Record File"
CATALOG_3 = (
"3 documents:\n"
"Deployments/aaa.md — AAA\n"
"Deployments/bbb.md — BBB\n"
"Homelab/ccc.md — CCC"
)
DOC1_SP = "Deployments/example-record-file.json"
DOC1_CONTENT = (
"The record file keeps every hosted zone record — first line is longer "
"than eighty characters so the quote truncation below is observable.\n"
"second line of the document content"
)
assert len(DOC1_CONTENT) > 80
DOC2_SP = "Homelab/aws-route53.md"
DOC2_CONTENT = "Route 53 notes — the second read, short on purpose."
SINGLE_USER = "Use your tools: what is the exact shape of the record file?"
#: Carries BOTH markers — ``use your tools`` then ``read two documents``.
MULTI_USER = "Use your tools and read two documents: compare the zone notes with the record file."
#: The multi marker alone — no ``use your tools``.
MULTI_ONLY_USER = "Please read two documents and compare them."
PLAIN_USER = "How does the sync job push records to the zone?"
assert TOOLS_TRIGGER in SINGLE_USER.lower() and MULTI_READ_TRIGGER not in SINGLE_USER.lower()
assert TOOLS_TRIGGER in MULTI_USER.lower() and MULTI_READ_TRIGGER in MULTI_USER.lower()
def _read_result(sp: str, content: str) -> str:
"""The agent's read-result text (``_execute_tool`` prefix)."""
return f"Document {sp}:\n{content}"
def _body(
user: str,
tool_msgs: tuple[str, ...] = (),
tools: Any = TOOLS,
system: str = SYSTEM_HIGH,
) -> dict[str, Any]:
"""A chat-completion body: system + user + the tool results in order."""
messages: list[dict[str, Any]] = [
{"role": "system", "content": system},
{"role": "user", "content": user},
]
for i, content in enumerate(tool_msgs):
messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": f"call_{i}",
"type": "function",
"function": {"name": "list_documents", "arguments": "{}"},
}
],
}
)
messages.append({"role": "tool", "tool_call_id": f"call_{i}", "content": content})
return {"messages": messages, "tools": tools}
# --------------------------------------------------------------------------
# Phase-37 single-read flow — must stay byte-identical
# --------------------------------------------------------------------------
def test_single_flow_list_step() -> None:
assert _tool_flow(_body(SINGLE_USER)) == ("list", "", "")
def test_single_flow_read_step_first_catalog_line() -> None:
flow = _tool_flow(_body(SINGLE_USER, (CATALOG_3,)))
# The FIRST listing line (Deployments/aaa.md), rsplit convention.
assert flow == ("read", "Deployments", "aaa.md", "call_1")
def test_single_flow_answer_step_with_tools_offered() -> None:
# Phase 45: the round cap keeps the tools offered until it is hit —
# the answer step fires regardless of the ``tools`` parameter.
flow = _tool_flow(
_body(SINGLE_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
def test_single_flow_answer_step_without_tools() -> None:
flow = _tool_flow(
_body(
SINGLE_USER,
(CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)),
tools=None,
)
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
def test_single_flow_no_tools_no_results_is_not_the_flow() -> None:
# agent_max_rounds=0 path: marker + <tools> prompt, but the request
# carries no tools and no tool results — regular answer, not a flow.
assert _tool_flow(_body(SINGLE_USER, tools=None)) is None
def test_single_flow_marker_without_tools_section_is_none() -> None:
assert _tool_flow(_body(SINGLE_USER, system=SYSTEM_LOW)) is None
def test_single_flow_plain_question_is_none() -> None:
assert _tool_flow(_body(PLAIN_USER)) is None
# --------------------------------------------------------------------------
# Phase-45 multi-read flow (task 02)
# --------------------------------------------------------------------------
def test_multi_flow_list_step() -> None:
assert _tool_flow(_body(MULTI_USER)) == ("list", "", "")
def test_multi_flow_read_first_step() -> None:
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2,)))
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_1")
def test_multi_flow_read_second_step_skips_already_read() -> None:
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
# The second catalog line — the first line differing from DOC1.
assert flow == ("read", "Homelab", "aws-route53.md", "call_2")
def test_multi_flow_read_second_is_listing_order_not_last() -> None:
# Three-doc catalog, first doc read: read #2 is the SECOND line
# (Deployments/bbb.md), not the last one.
flow = _tool_flow(_body(MULTI_USER, (CATALOG_3, _read_result("Deployments/aaa.md", "x"))))
assert flow == ("read", "Deployments", "bbb.md", "call_2")
def test_multi_flow_answer_step_names_both_paths() -> None:
flow = _tool_flow(
_body(
MULTI_USER,
(
CATALOG_2,
_read_result(DOC1_SP, DOC1_CONTENT),
_read_result(DOC2_SP, DOC2_CONTENT),
),
)
)
assert flow is not None
assert flow[0] == "multi_answer"
# Byte-stable: the single-read shape quoting the FIRST read result
# (first 80 chars), plus both read paths in read order.
assert flow[2] == f"Read {DOC1_SP}. {DOC1_CONTENT[:80]} I read {DOC1_SP} and {DOC2_SP}."
def test_multi_flow_answer_step_without_tools_offered() -> None:
# The forced answer is content, not a tool call — it must not be
# gated on the ``tools`` parameter.
flow = _tool_flow(
_body(
MULTI_USER,
(
CATALOG_2,
_read_result(DOC1_SP, DOC1_CONTENT),
_read_result(DOC2_SP, DOC2_CONTENT),
),
tools=None,
)
)
assert flow is not None
assert flow[0] == "multi_answer"
def test_multi_flow_one_document_catalog_degenerates_to_single_answer() -> None:
# Nothing second to read — the single-read answer shape, quoting the
# only read result.
flow = _tool_flow(
_body(MULTI_USER, (CATALOG_1, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
def test_multi_flow_no_tools_no_results_is_not_the_flow() -> None:
assert _tool_flow(_body(MULTI_USER, tools=None)) is None
def test_multi_trigger_without_tools_trigger_is_none() -> None:
# The multi marker alone (no ``use your tools``) is not the flow.
assert MULTI_READ_TRIGGER in MULTI_ONLY_USER
assert TOOLS_TRIGGER not in MULTI_ONLY_USER.lower()
assert _tool_flow(_body(MULTI_ONLY_USER)) is None
def test_multi_flow_requires_tools_section() -> None:
assert _tool_flow(_body(MULTI_USER, system=SYSTEM_LOW)) is None