Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 160_tool_budget_telemetry
Build and Push Containers / build-and-push-app (push) Successful in 2m27s
Details
Build and Push Containers / build-and-push-db (push) Failing after 2m8s
Details

Browse Source 
All green. Final report:

**Phase 160 completion pass — verified (final verification; all 5 tasks already in `complete/`)**

- **Verified, no code defects found:** budget trailers (`TOOL_BUDGET_REMAINING/LAST/SPENT`), cap-path rejection + `ROUND_CAP_FALLBACK_ANSWER` + `ROUND_CAP_WRAPUP_INSTRUCTION`, `done.budget_exhausted` (ABSENT-when-false), `rounds=/cap_hit=` log fields, `agent_max_rounds=10` default, frontend Continue button + restore + shared-page exclusion, mock markers.
- **Tests/lint:** `uv run pytest` 153/153 green (EXIT 0); `uv run pytest --cov=app --cov-report=term-missing` TOTAL **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors/0 warnings.
- **Criteria:** ① decreasing trailer pinned by unit tests (9→…→spent) ✅ ② cap path ends readable, no tool lines after cap (E2E) ✅ ③ `budget_exhausted` true-only-on-cap, absent otherwise (integration) ✅ ④ Continue button → one fresh `continue` turn, fresh budget, record intact (E2E) ✅ ⑤ flag survives refresh; shared chat never renders it (`shared.js` has no `budget_exhausted` path) ✅ ⑥ `max_rounds=0` byte-identical single `tools=None` request; default still 10 ✅ ⑦ full suite/coverage/lint green ✅ ⑧ `test_tool_budget_telemetry.py` **7/7** in isolation; all 6 regression E2E suites green in isolation ✅ ⑨ commit + phase move = harness (I make no git history/index changes).
- **Docs:** README.md — fixed: the "Continue researching" button and cap behavior were undocumented (added a Chat-section paragraph + an Agent-tools-section paragraph, incl. the phase-71 recorded-revision note); env-var table, commands, and architecture all verified consistent otherwise. AGENTS.md — verified consistent, no changes needed.
- **Decisions:** no code changes required — phase was fully implemented; only README docs added. No live-infrastructure touched.
- **Next pending phase:** `161_reasoning_bubble`.
This commit is contained in:
ducoterra
2026-10-06 01:42:44 -04:00
parent cb8767512a
commit 8172956c15
55 changed files with 4124 additions and 344 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/agent.py
+200 -15
View File
@@ -211,12 +211,34 @@ task 04):
that keeps emitting rejected calls is bounded by the cap (point 4).
4. Every call the model emits is appended back to the message history as
the assistant tool-call message + the tool result (refusals included),
consumes one round, and the model is called again. At the round cap —
consumes one round, and the model is called again. Every tool result
that goes on the wire carries the budget-telemetry trailer (phase 160,
task 01) — :data:`TOOL_BUDGET_REMAINING` (≥ 2 left),
:data:`TOOL_BUDGET_LAST` (1 left) or :data:`TOOL_BUDGET_SPENT`
(0 left) via :func:`_budget_note` — the count is what the model has
LEFT after the call it is reading (so the first result of a 10-round
budget reads 9, and the boundary call's result reads the spent line).
At the round cap —
``max_rounds = settings.agent_max_rounds`` (``BOR_AGENT_MAX_ROUNDS``,
default 10) — the loop forces one final retried no-tools request
(``chat_stream_retried`` with ``tools=None``) and returns: the cap is
the **only** forced exit (besides "the stream carried no calls"), and
it bounds pathological rejected-call streams.
it bounds pathological rejected-call streams. Phase 160 (task 02):
that forced final's single system message carries
:data:`ROUND_CAP_WRAPUP_INSTRUCTION` folded in (the phase-71
:data:`CORRECTION_INSTRUCTION` pattern), coaching the model that no
tool calls remain — do not emit any (they now have no effect), answer
from the results already in the conversation, and, if incomplete,
tell the user they can say "continue" to keep researching. A tool
call the model attempts on that ``tools=None`` final is **rejected**
— never executed, never yielded (no ``tool`` frame, no visible
markup), and counted in ``holder.rejected_calls`` (``tool_calls``
stays executed-only). If that forced final streams no visible content
(only calls, only scaffolding, or nothing), the turn settles with the
deterministic :data:`ROUND_CAP_FALLBACK_ANSWER` — an honest
"budget spent" line — instead of a terminal
``MalformedReplyError`` (the recorded revision of the phase-71
cap-path terminal; the non-cap recovery path's error is unchanged).
5. A rare stream that carries both content and a tool call keeps the
content (it was already emitted) **and** still runs the tool.
6. *holder* (an :class:`AgentHolder`) records the read documents and the
@@ -724,6 +746,45 @@ LS_MAX_FILE_LINES = 50
#: round (no silent argument normalization).
NOT_A_FOLDER = "'{arg}' is not a folder — {parent} has: {subfolders}"
#: The tool-call budget telemetry (phase 160, task 01; TODO L4 — "the
#: LLM needs to know how many tool calls it has left"): appended to
#: EVERY tool result the model receives — the raw result plus the note
#: — so the model plans its last reads and wraps up on its own instead
#: of hitting the cap mid-research. Appended in :func:`run_agent`
#: (after ``rounds += 1``), NEVER inside :func:`_execute_tool` (the
#: pure seam — its raw return stays byte-identical, the phase-95
#: ``ToolResultPiece`` reads ``holder.read_truncations`` untouched, and
#: a rejection's refusal text carries the note too — a refusal
#: consumes a round, so the count is exactly as true there).
#: **The count is what the model has LEFT after the call it is
#: reading** — the first result of a 10-round budget reads 9, the
#: boundary call's result reads the spent line (0). The three forms
#: (``n ≥ 2`` / ``n == 1`` / ``n ≤ 0``) live in the one pure helper
#: :func:`_budget_note` so the wording sits in exactly one place.
TOOL_BUDGET_REMAINING: str = "\n\n[tool calls remaining: {n}]"
TOOL_BUDGET_LAST: str = "\n\n[1 tool call remaining — make it count, then answer]"
TOOL_BUDGET_SPENT: str = (
'\n\n[No tool calls remain — answer now from what you have, or '
'tell the user to say "continue" to keep researching.]'
)
def _budget_note(remaining: int) -> str:
"""The phase-160 budget-telemetry trailer for a tool result.
*remaining* is ``max_rounds - rounds`` after the call's own round
was consumed — what the model has LEFT: ``≥ 2`` → the formatted
:data:`TOOL_BUDGET_REMAINING` line, ``1`` → :data:`TOOL_BUDGET_LAST`,
``≤ 0`` → :data:`TOOL_BUDGET_SPENT`. Pure (the unit tests hit it
directly); the wording lives in the three constants above.
"""
if remaining >= 2:
return TOOL_BUDGET_REMAINING.format(n=remaining)
if remaining == 1:
return TOOL_BUDGET_LAST
return TOOL_BUDGET_SPENT
#: The harness-owned recovery line (phase 71, task 03) — folded into the
#: ORIGINAL single system message of the one bounded recovery request
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
@@ -735,6 +796,40 @@ CORRECTION_INSTRUCTION: str = (
"text — no tool syntax."
)
#: The round-cap wrap-up coaching (phase 160, task 02) — folded into the
#: ORIGINAL single system message of the one forced final ``tools=None``
#: request (``system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION``, the
#: phase-71 :data:`CORRECTION_INSTRUCTION` pattern — the user message and
#: the tool history stay in place). In the :data:`CORRECTION_INSTRUCTION`
#: voice it says exactly four things: no tool calls remain; do NOT emit a
#: tool call, tool-call JSON, or any ``<|tool_call…|>``-style markup (a
#: tool call now has no effect at all); answer from the results already in
#: the conversation, stating plainly what is still unchecked; and, if the
#: answer is incomplete, end by telling the user they can say "continue"
#: to let the model keep researching.
ROUND_CAP_WRAPUP_INSTRUCTION: str = (
"You have no tool calls left — the research budget for this turn is "
"spent. Do NOT emit a tool call, tool-call JSON, or any "
"<|tool_call…|>-style markup: a tool call now has no effect at all. "
"Answer from the results already in this conversation, stating "
"plainly what you checked and what is still unchecked. If your "
'answer is incomplete, end by telling the user they can say "continue"'
" to let you keep researching."
)
#: The round-cap fallback answer (phase 160, task 02) — the deterministic
#: backstop :func:`run_agent` streams when the forced final carries NO
#: visible content (only rejected calls, only scaffolding, or nothing).
#: One short sentence in the brand voice: the budget ran out before an
#: answer came together, what was gathered above still stands, say
#: "continue" to keep digging. Free of markdown decoration (the UI renders
#: it through the normal markdown path).
ROUND_CAP_FALLBACK_ANSWER: str = (
"I've run out of research budget for this turn before the answer came "
"together, but what I gathered above still stands — say \"continue\" "
"to let me keep digging."
)
class MalformedReplyError(LLMError):
"""The model kept replying in raw tool-scaffolding (phase 71).
@@ -1488,6 +1583,32 @@ class AgentHolder:
rejected calls (unknown tool, unknown/missing arguments or document,
already-in-context) do not count. Drives the per-turn log line's
``tool_calls=N`` field (task 04).
``rejected_calls`` (phase 160, task 02): how many tool calls the
model attempted AFTER the round cap — on the forced final
``tools=None`` request a model (the documented ``lite`` behavior) can
still emit structured ``tool_calls``; those are REJECTED (never
executed, never shown — no ``tool`` frame, no visible markup) and
counted HERE only. They count in ``tool_calls`` in nothing (the
executed-only counter stays clean), the refusal style: "counts in
nothing". Drives the cap-path rejection warning log.
``rounds`` (phase 160, task 03): the number of rounds CONSUMED by
the agent loop — one per call the model emitted on a tools-
offered round (an executed call or a rejection; both consume the
round — a call rejected past the cap on the forced final is NOT
a round, it counts in ``rejected_calls``). Mirrored from the
loop's local counter (``holder.rounds = rounds`` beside
``rounds += 1``, and once before the forced final, so a turn that
ends at the cap reports the full count) — the API layer reads
it, never re-derives it. Drives the per-turn log line's
``rounds=N`` field.
``cap_reached`` (phase 160, task 03): ``True`` exactly where the
round cap forces the final no-tools answer (the ``run_agent``
round-cap block); ``False`` on every turn that answered inside
the cap (and every ``agent_max_rounds=0`` kill-switch turn — the
loop never runs). Drives the per-turn log line's
``cap_hit=0|1`` field and the done frame's ``budget_exhausted``
key (the UI's Continue-researching affordance keys on it, task
04).
``scaffold_stripped``: how many chars of tool-scaffolding the
turn's filters removed across the turn's requests (rounds + the
forced final + any recovery, phase 71) — drives the per-turn log
@@ -1538,6 +1659,9 @@ class AgentHolder:
read_docs: list[Document] = field(default_factory=list)
tool_calls: int = 0
rejected_calls: int = 0
rounds: int = 0
cap_reached: bool = False
scaffold_stripped: int = 0
read_truncations: list[tuple[str, int, int]] = field(default_factory=list)
shown_images: list[Document] = field(default_factory=list)
@@ -2230,6 +2354,7 @@ async def run_agent(
with db_factory() as tool_db:
result = _execute_tool(tool_db, call, seed_docs, holder, settings)
rounds += 1 # every call the model emits consumes a round
holder.rounds = rounds # phase 160 (task 03): mirror for the API layer
logger.info(
"agent tool=%s args=%s round=%d/%d",
call.name,
@@ -2253,7 +2378,20 @@ async def run_agent(
],
}
)
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
# Phase 160 (task 01): the budget-telemetry trailer — the model
# sees how many tool rounds it has LEFT after this call (appended
# here, never inside :func:`_execute_tool`, which stays the pure
# ``→ str`` seam: the raw result, the phase-95 ToolResultPiece and
# the read_coverage bookkeeping are untouched; a rejected call's
# refusal carries the note too — it consumed a round, so the
# count is exactly as true there).
messages.append(
{
"role": "tool",
"tool_call_id": call.id,
"content": result + _budget_note(max_rounds - rounds),
}
)
# Phase 95: surface this round's truncation(s) to the API layer.
# The piece lands AFTER the round's ``tool`` frame (the matching
# ``ToolCallPiece`` was already streamed, above) and BEFORE the
@@ -2273,17 +2411,41 @@ async def run_agent(
"no-tools answer",
rounds,
)
# Phase 160 (task 03): report the cap — ``cap_reached``
# is set exactly here (the forced no-tools final is about
# to run) and ``rounds`` is mirrored once more before it,
# so a turn that ends at the cap reports the FULL count
# (the API layer reads the holder; it never re-derives).
holder.cap_reached = True
holder.rounds = rounds
# Phase 48: the forced final answer gets the same explicit
# teardown as the loop rounds (consumer abandon mid-final
# answer must still close the model's stream). Phase 67: the
# forced call retries under the same locked-A2 rule as the
# loop rounds. Phase 71: the forced final runs through a
# fresh filter too — raw scaffolding can never reach the
# user from ANY grounded request.
# user from ANY grounded request. Phase 160 (task 02): the
# single system message carries the wrap-up coaching
# (``system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION``,
# the phase-71 :data:`CORRECTION_INSTRUCTION` pattern — the
# user message and the tool history stay in place) so the
# model is told the budget is spent and to answer or hand
# off; a ``tools=None`` request can still come back with
# structured ``tool_calls`` (the documented ``lite``
# behavior), and those are REJECTED in the loop below —
# never executed, never yielded (no ``tool`` frame, no
# visible markup), counted in ``holder.rejected_calls``.
messages_capped = [
{
"role": "system",
"content": system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION,
},
*messages[1:],
]
final_filter = ScaffoldingFilter()
final = chat_stream_retried(
llm,
messages,
messages_capped,
tools=None,
retries=settings.llm_retries,
delay=settings.llm_retry_delay,
@@ -2292,15 +2454,22 @@ async def run_agent(
final_content = 0
try:
async for piece in final:
if isinstance(piece, ToolCallPiece):
# Phase 160 (task 02): the budget is spent — a
# call the model attempts now has no effect.
# Reject it: never executed, never yielded (no
# ``tool`` frame, no visible markup), counted
# here only (``tool_calls`` stays executed-only).
holder.rejected_calls += 1
continue
if isinstance(piece, StreamPiece) and piece.kind == "content":
final_content += len(piece.text)
yield piece
finally:
await final.aclose()
# Phase 71: the same capture log + turn total; a
# scaffolding-only forced final (this turn used no recovery,
# so nothing is doubled up) settles with the same terminal
# malformed-reply error rather than a silently empty answer.
# Phase 71: the same capture log + turn total (the per-span
# strip warning is the capture mechanism for new registry
# entries — unchanged).
holder.scaffold_stripped += final_filter.stripped_chars
for span in final_filter.stripped_spans:
logger.warning(
@@ -2310,13 +2479,29 @@ async def run_agent(
rounds + 1,
span[:200],
)
if final_content == 0 and final_filter.stripped_chars > 0:
if holder.rejected_calls:
logger.warning(
"agent: the forced final answer was pure tool-scaffolding "
"— settling with a malformed-reply error"
"agent: %d tool call(s) attempted after the round cap — "
"rejected, not shown",
holder.rejected_calls,
)
raise MalformedReplyError(
"the forced final answer was raw tool-scaffolding — no "
"clean answer to stream"
# Phase 160 (task 02 — recorded revision of the phase-71
# cap-path terminal, AGENTS.md rule 3, owner-confirmed
# 2026-10-05): a forced final with NO visible content (only
# rejected calls, only scaffolding, or nothing) settles with
# the deterministic budget-fallback answer — an honest
# "budget spent" line — instead of a terminal
# ``MalformedReplyError`` (an error frame is the wrong
# signal for the exact symptom this TODO reports). The
# NON-cap recovery path's ``MalformedReplyError`` is
# unchanged.
if final_content == 0:
logger.warning(
"agent: round cap hit with no visible reply "
"(rejected_calls=%d, scaffold_stripped=%d) — "
"answering with the budget fallback",
holder.rejected_calls,
holder.scaffold_stripped,
)
yield StreamPiece(kind="content", text=ROUND_CAP_FALLBACK_ANSWER)
return