feat(rag): unbounded agent tool calls behind a round cap (owner revision)
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:
- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
negative rejected) replaces agent_list_calls / agent_read_calls;
.env.example + README document the single knob; app/rag/prompts.py
docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
rounds >= max_rounds, at which point it forces one final no-tools
answer (the cap is the only forced exit); 0 = no tools — exactly one
tools=None request, byte-identical to the pre-phase-37 path (the
kill switch). Rejected calls (unknown tool / missing args /
already-in-context / unknown path) still consume a round, so
pathological rejected-call streams are bounded by the cap. The
per-call log line is now tool/args/round=N/M; the per-turn
tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
the deterministic list -> read #1 -> read #2 -> forced-answer flow
(byte-stable "I read <sp1> and <sp2>." line), classified by the
count of tool-role read results; the phase-37 single-read flow stays
byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
mock-only): three tool frames/lines in order (one list, two reads —
the second read is what the old read budget refused) + the
both-named non-deflected answer; done.sources + chips = retrieval
doc + both reads, deduped; no budget refusal rendered; the
single-read marker flow regression (exactly one read, single tool
pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
the only PLAN edit this phase; the phase-37 note's budget clause is
marked removed.
Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.
uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.
Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
This commit is contained in:
@@ -78,10 +78,10 @@ class FakeRagLLM:
|
||||
#: is yielded for the *i*-th request that carries a non-None
|
||||
#: ``tools`` parameter (a request the agent loop is offering tools
|
||||
#: on). A request without tools — the deflected direct path, the
|
||||
#: post-budget answer request, or the 0/0 single-request path —
|
||||
#: always yields the thinking + answer stream below, so a
|
||||
#: deflected turn through this fake is byte-identical to the
|
||||
#: plain fake's output.
|
||||
#: cap-forced answer request, or the kill-switch
|
||||
#: (``agent_max_rounds=0``) single-request path — always yields the
|
||||
#: thinking + answer stream below, so a deflected turn through this
|
||||
#: fake is byte-identical to the plain fake's output.
|
||||
self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or [])
|
||||
|
||||
async def embed(self, texts: list[str]) -> list[list[float]]:
|
||||
@@ -503,8 +503,8 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
The agent loop offers tools while budgets last and drops them
|
||||
(``tools=None``) once both are spent."""
|
||||
Phase 45: the agent loop keeps offering the tools for the whole
|
||||
turn — the round cap (not per-tool budgets) is the bound."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -518,8 +518,9 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
# the post-budget answer request (tools=None) falls back to the
|
||||
# fake's thinking + answer stream
|
||||
# the answer request still carries the tools (2 rounds < the
|
||||
# default cap of 10); the fake's tool_script is exhausted, so
|
||||
# it falls back to the thinking + answer stream
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
@@ -557,12 +558,13 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
assert len(sources) == len(set(sources)) # deduped by (source, path)
|
||||
assert done["sources"][-1]["title"] == "Backup Strategy"
|
||||
|
||||
# The agent loop offered the tools while any budget remained and
|
||||
# dropped them once both were spent (single post-budget request).
|
||||
# Phase 45: the tools stay offered on every request — the round cap
|
||||
# (not spent budgets) bounds the loop, and the model answered while
|
||||
# still being offered the tools (2 rounds < default cap 10).
|
||||
assert len(scripted.seen_messages) == 3
|
||||
assert scripted.seen_tools[0] == AGENT_TOOLS
|
||||
assert scripted.seen_tools[1] == AGENT_TOOLS # the read budget was still open
|
||||
assert scripted.seen_tools[2] is None
|
||||
assert scripted.seen_tools[1] == AGENT_TOOLS
|
||||
assert scripted.seen_tools[2] == AGENT_TOOLS
|
||||
|
||||
# The query_log row carries the same combined source list.
|
||||
(row,) = db.scalars(select(QueryLog)).all()
|
||||
@@ -628,17 +630,17 @@ def test_deflected_turn_stays_byte_identical_without_tools(
|
||||
assert "backups.md" not in row.sources
|
||||
|
||||
|
||||
def test_zero_agent_budgets_reproduce_pre_phase_single_request(
|
||||
def test_zero_max_rounds_reproduce_pre_phase_single_request(
|
||||
client,
|
||||
db,
|
||||
seeded_kb: FakeRagLLM,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""(c) ``BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0``: no ``tool``
|
||||
frames, exactly one request **without** a ``tools`` key (the
|
||||
pre-phase request shape), ``done.sources`` unchanged, and
|
||||
``tool_calls=0`` in the log line — budgets-as-kill-switch."""
|
||||
"""(c) ``BOR_AGENT_MAX_ROUNDS=0``: no ``tool`` frames, exactly one
|
||||
request **without** a ``tools`` key (the pre-phase request shape),
|
||||
``done.sources`` unchanged, and ``tool_calls=0`` in the log line —
|
||||
the kill switch survives the phase-45 budget removal."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
@@ -658,8 +660,7 @@ def test_zero_agent_budgets_reproduce_pre_phase_single_request(
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
agent_list_calls=0,
|
||||
agent_read_calls=0,
|
||||
agent_max_rounds=0,
|
||||
),
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
|
||||
Reference in New Issue
Block a user