phase: 160_tool_budget_telemetry
All green. Final report: **Phase 160 completion pass — verified (final verification; all 5 tasks already in `complete/`)** - **Verified, no code defects found:** budget trailers (`TOOL_BUDGET_REMAINING/LAST/SPENT`), cap-path rejection + `ROUND_CAP_FALLBACK_ANSWER` + `ROUND_CAP_WRAPUP_INSTRUCTION`, `done.budget_exhausted` (ABSENT-when-false), `rounds=/cap_hit=` log fields, `agent_max_rounds=10` default, frontend Continue button + restore + shared-page exclusion, mock markers. - **Tests/lint:** `uv run pytest` 153/153 green (EXIT 0); `uv run pytest --cov=app --cov-report=term-missing` TOTAL **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors/0 warnings. - **Criteria:** ① decreasing trailer pinned by unit tests (9→…→spent) ✅ ② cap path ends readable, no tool lines after cap (E2E) ✅ ③ `budget_exhausted` true-only-on-cap, absent otherwise (integration) ✅ ④ Continue button → one fresh `continue` turn, fresh budget, record intact (E2E) ✅ ⑤ flag survives refresh; shared chat never renders it (`shared.js` has no `budget_exhausted` path) ✅ ⑥ `max_rounds=0` byte-identical single `tools=None` request; default still 10 ✅ ⑦ full suite/coverage/lint green ✅ ⑧ `test_tool_budget_telemetry.py` **7/7** in isolation; all 6 regression E2E suites green in isolation ✅ ⑨ commit + phase move = harness (I make no git history/index changes). - **Docs:** README.md — fixed: the "Continue researching" button and cap behavior were undocumented (added a Chat-section paragraph + an Agent-tools-section paragraph, incl. the phase-71 recorded-revision note); env-var table, commands, and architecture all verified consistent otherwise. AGENTS.md — verified consistent, no changes needed. - **Decisions:** no code changes required — phase was fully implemented; only README docs added. No live-infrastructure touched. - **Next pending phase:** `161_reasoning_bubble`.
This commit is contained in:
@@ -842,6 +842,57 @@ Implements just enough of the aipi surface:
|
||||
deflected turns never hit it); verified 2026-10-01: no
|
||||
existing E2E question or fixture file contains the phrase, so
|
||||
every other suite is unaffected.
|
||||
- user message containing ``run out of tool calls``
|
||||
(``ROUND_CAP_TRIGGER``, phase 160 task 05 — the
|
||||
tool-call-budget telemetry's dedicated suite
|
||||
``tests/e2e/test_tool_budget_telemetry.py``) — detected on
|
||||
the LAST user message only (the mock's existing invariant:
|
||||
the last ``user`` message is the current question, the
|
||||
earlier non-system messages are prior turns):
|
||||
* while ``tools`` are offered (the request carries
|
||||
``tools`` — the app issues such a request only while the
|
||||
round budget is unspent, i.e. no tool result answers the
|
||||
last emitted call yet) and the flow is active: stream
|
||||
ONLY ``tool_calls`` deltas — ``ls`` with no arguments,
|
||||
incrementing synthetic ids ``call_cap_1``,
|
||||
``call_cap_2``, … (the next id is derived from the
|
||||
``call_cap_N`` ids already in the messages' ``assistant``
|
||||
tool calls), ``finish_reason: "tool_calls"``, no content —
|
||||
the model NEVER stops asking, so the turn is guaranteed
|
||||
to hit ``BOR_AGENT_MAX_ROUNDS``;
|
||||
* on the request WITHOUT ``tools`` (the forced final — the
|
||||
round-cap path; the request still carries the full
|
||||
history, so the trigger stays the last user message):
|
||||
stream a ``tool_calls`` delta ANYWAY — ``read`` on the
|
||||
SEED path (the system prompt's ``<documents>`` section's
|
||||
FIRST ``<document>`` block, the joined ``source/path`` —
|
||||
the ``read`` tool's combined identity, phase 70), id
|
||||
``call_cap_dump``, NO content — the EXACT dumped-call
|
||||
shape the phase kills (and the shape ``SCAFFOLD_TRIGGER``
|
||||
does not cover — that one emits textual markup, this one
|
||||
is structured). The app REJECTS it (never executed, never
|
||||
shown, counted in ``AgentHolder.rejected_calls``);
|
||||
* the ``continue`` question (the last user message is
|
||||
EXACTLY ``continue`` — the UI's ``CONTINUE_QUESTION`` —
|
||||
and an EARLIER user message carries the trigger; the
|
||||
client history rides in the stateless A10 request): the
|
||||
stable ``ROUND_CAP_CONTINUE_ANSWER`` (a fixed sentence
|
||||
naming the flow's seed document — the
|
||||
``SCOPED_SEARCH_PATH`` suite-constant convention: the
|
||||
literal "continue" shares no tokens with the KB, so the
|
||||
continuation turn deflects and its prompt carries no
|
||||
``<documents>`` block; the sentence names the document
|
||||
directly), so a suite asserts a real continuation answer
|
||||
rather than a tool call.
|
||||
Streaming only (the app's chat path is the only streaming
|
||||
consumer; a non-streaming marker request never issues and
|
||||
falls through to the regular answer, like the other marker
|
||||
flows). Checked BEFORE the plain ``TOOLS_TRIGGER`` flow
|
||||
(disjoint trigger phrases — the phase-71/72/94/118/131/
|
||||
139/141/145 ordering convention); verified 2026-10-05: no
|
||||
existing E2E question or fixture file contains the phrase
|
||||
(or sends a ``continue`` question with the trigger in the
|
||||
history), so every other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -1193,6 +1244,42 @@ HISTORY_TRIGGER = "echo my history"
|
||||
#: phrase, so every other suite is unaffected.
|
||||
FOLDER_MAP_TRIGGER = "repeat your folder map"
|
||||
|
||||
#: Phase 160 (task 05, tool-call budget telemetry, TODO.md L4 —
|
||||
#: the dedicated suite
|
||||
#: ``tests/e2e/test_tool_budget_telemetry.py``): a user message
|
||||
#: containing this substring (case-insensitive), detected on the
|
||||
#: LAST user message only (the mock's existing invariant — the
|
||||
#: last ``user`` message is the current question, the earlier
|
||||
#: non-system messages are prior turns), drives the deterministic
|
||||
#: ROUND-CAP flow: while ``tools`` are offered the mock NEVER
|
||||
#: answers (``ls`` calls, incrementing ``call_cap_N`` ids — the
|
||||
#: turn is guaranteed to hit ``BOR_AGENT_MAX_ROUNDS``), and the
|
||||
#: forced final (no ``tools``) streams a structured ``read`` call
|
||||
#: anyway (id ``call_cap_dump``, the seed path, NO content) — the
|
||||
#: exact dumped-call shape the phase kills. The ``continue``
|
||||
#: question (exactly ``continue``, the trigger in an EARLIER user
|
||||
#: message) gets :data:`ROUND_CAP_CONTINUE_ANSWER`. Disjoint from
|
||||
#: every other trigger phrase; verified 2026-10-05: no existing
|
||||
#: E2E question or fixture file contains the phrase, so every
|
||||
#: other suite is unaffected.
|
||||
ROUND_CAP_TRIGGER = "run out of tool calls"
|
||||
|
||||
#: Phase 160 (task 05): the ROUND-CAP flow's CONTINUATION answer
|
||||
#: — the fixed sentence the ``continue`` question gets. The
|
||||
#: literal "continue" shares no tokens with the KB, so the
|
||||
#: continuation turn DEFLECTS (A8 LOW) and its prompt carries no
|
||||
#: ``<documents>`` block — the sentence therefore names the
|
||||
#: flow's seed document DIRECTLY (the
|
||||
#: :data:`SCOPED_SEARCH_PATH` suite-constant convention; the
|
||||
#: suite seeds exactly that document, so the name IS the
|
||||
#: seed's). Byte-stable on purpose: the suite asserts on it, so
|
||||
#: it must not drift.
|
||||
ROUND_CAP_CONTINUE_ANSWER = (
|
||||
"Picking up where the tool budget ran out — still digging into "
|
||||
"docs/homelab/kubernetes.md: it says the cluster runs Talos "
|
||||
"Linux on three nodes, and Cilium handles networking."
|
||||
)
|
||||
|
||||
#: Phase 122 (image documents, LOCKED A3): the fixed description the
|
||||
#: mock's vision (CHAT) model returns for an ``IMAGE_DESCRIPTION_MODE``
|
||||
#: request — the multimodal user message
|
||||
@@ -3079,6 +3166,106 @@ def _origin_link_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("call", target, "call_0")
|
||||
|
||||
|
||||
#: Phase 160 (task 05): the round-cap flow's incrementing call ids
|
||||
#: — ``call_cap_1``, ``call_cap_2``, … The NEXT id is derived from
|
||||
#: the ids already in the messages' ``assistant`` tool calls (the
|
||||
#: flow is stateless over the message history — a retry that
|
||||
#: re-issues the same request re-derives the same next id).
|
||||
_CAP_CALL_ID_RE = re.compile(r"^call_cap_(\d+)$")
|
||||
|
||||
|
||||
def _round_cap_next_call_id(body: dict[str, Any]) -> str:
|
||||
"""The next incrementing ``call_cap_N`` id (stateless).
|
||||
|
||||
The mock's ``ls`` calls in the round-cap flow carry
|
||||
incrementing synthetic ids — the NEXT id is one past the
|
||||
highest ``call_cap_N`` id already in the messages'
|
||||
``assistant`` tool calls (``call_cap_1`` on the first
|
||||
request — no ids yet).
|
||||
"""
|
||||
n = 0
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "assistant":
|
||||
continue
|
||||
for call in m.get("tool_calls") or []:
|
||||
if not isinstance(call, dict):
|
||||
continue
|
||||
match = _CAP_CALL_ID_RE.match(str(call.get("id") or ""))
|
||||
if match:
|
||||
n = max(n, int(match.group(1)))
|
||||
return f"call_cap_{n + 1}"
|
||||
|
||||
|
||||
def _round_cap_seed_path(body: dict[str, Any]) -> str:
|
||||
"""The seed document's combined ``source/path`` — or the
|
||||
sentinel (the premise broke: no ``<documents>`` block; the
|
||||
suite fails loudly on the sentinel in the dump call).
|
||||
|
||||
Phase 160 (task 05): the FORCED FINAL's request (the
|
||||
round-cap path) still carries the grounded system prompt —
|
||||
the ``<documents>`` section's FIRST ``<document>`` block is
|
||||
the seed document (the suite seeds exactly one). The mock
|
||||
joins the labeled ``source`` / ``path`` fields (the
|
||||
``read`` tool's combined identity, phase 70).
|
||||
"""
|
||||
block = _DOCUMENT_BLOCK_RE.search(_system(body))
|
||||
if not block:
|
||||
return "seed/unknown.md"
|
||||
return f"{block.group('source')}/{block.group('path')}"
|
||||
|
||||
|
||||
def _round_cap_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Phase 160 (task 05): the ROUND-CAP marker flow — or ``None``.
|
||||
|
||||
Discriminated on the LAST user message (the mock's existing
|
||||
invariant: the last ``user`` message is the current
|
||||
question, the earlier non-system messages are prior turns —
|
||||
the ``_history_echo`` ``last_user`` scan convention):
|
||||
|
||||
* the last user message carries :data:`ROUND_CAP_TRIGGER`:
|
||||
- the request offers ``tools`` (the app issues such a
|
||||
request only while the round budget is unspent — no
|
||||
tool result answers the last emitted call yet):
|
||||
``("call", call_cap_N)`` — the model NEVER stops
|
||||
asking (``ls``, no arguments, the next incrementing
|
||||
id) — the turn is guaranteed to hit
|
||||
``BOR_AGENT_MAX_ROUNDS``;
|
||||
- the request carries NO ``tools`` (the forced final —
|
||||
the round-cap path; the trigger is still the last user
|
||||
message): ``("dump", <seed path>)`` — a ``read`` on
|
||||
the seed path (id ``call_cap_dump``, NO content) — the
|
||||
exact dumped-call shape the phase kills;
|
||||
* the last user message is EXACTLY ``continue`` (the UI's
|
||||
``CONTINUE_QUESTION``) and an EARLIER user message
|
||||
carries the trigger (the client history rides in the
|
||||
stateless A10 request): ``("continue",)`` — the stable
|
||||
:data:`ROUND_CAP_CONTINUE_ANSWER` (a fixed sentence
|
||||
naming the flow's seed document — the
|
||||
:data:`SCOPED_SEARCH_PATH` suite-constant convention: the
|
||||
literal "continue" shares no tokens with the KB, so the
|
||||
continuation turn deflects and its prompt carries no
|
||||
``<documents>`` block).
|
||||
"""
|
||||
users = [
|
||||
_content_text(m.get("content"))
|
||||
for m in _messages(body)
|
||||
if m.get("role") == "user"
|
||||
]
|
||||
if not users:
|
||||
return None
|
||||
last_user = users[-1].strip().lower()
|
||||
if ROUND_CAP_TRIGGER in last_user:
|
||||
if body.get("tools"):
|
||||
return ("call", _round_cap_next_call_id(body))
|
||||
return ("dump", _round_cap_seed_path(body))
|
||||
if (
|
||||
last_user == "continue"
|
||||
and any(ROUND_CAP_TRIGGER in u.lower() for u in users[:-1])
|
||||
):
|
||||
return ("continue",)
|
||||
return None
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -4563,6 +4750,42 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 160 (task 05, tool-call budget telemetry,
|
||||
# TODO.md L4): the deterministic ROUND-CAP marker flow
|
||||
# (the budget-telemetry suite's dedicated flow — see the
|
||||
# module docstring). Checked BEFORE the plain
|
||||
# TOOLS_TRIGGER flow (disjoint trigger phrases — the
|
||||
# phase-71/72/94/118/131/139/141/145 ordering
|
||||
# convention).
|
||||
round_cap = _round_cap_flow(body)
|
||||
if round_cap is not None:
|
||||
if round_cap[0] == "call":
|
||||
# The model never stops asking: ``ls`` with no
|
||||
# arguments, the next incrementing id, NO content
|
||||
# — the turn is guaranteed to hit
|
||||
# ``BOR_AGENT_MAX_ROUNDS``.
|
||||
stream = _tool_call_stream("ls", {}, round_cap[1])
|
||||
elif round_cap[0] == "dump":
|
||||
# The forced final (no ``tools``) — the model
|
||||
# dumps a structured ``read`` call anyway (the
|
||||
# seed path, id ``call_cap_dump``, NO content) —
|
||||
# the exact shape the phase kills (the app
|
||||
# REJECTS it — never executed, never shown).
|
||||
stream = _tool_call_stream(
|
||||
"read", {"path": round_cap[1]}, "call_cap_dump"
|
||||
)
|
||||
else: # "continue" — the stable continuation answer
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
ROUND_CAP_CONTINUE_ANSWER, body.get("max_tokens")
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
|
||||
Reference in New Issue
Block a user