Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery

Browse Source
This commit is contained in:
ducoterra
2026-09-03 13:39:15 -04:00
parent 801639efcc
commit 575d6c88d0
38 changed files with 2793 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/agent.py
+192 -2
View File
@@ -95,6 +95,27 @@ task 04):
exactly one round. With ``settings.llm_retries=0`` every request is a
single plain attempt (the pre-phase-67 path).
Scaffolding guardrail (phase 71, deterministic only — owner permission
2026-09-03: "deterministic guardrails only right now, forget using a
model for that"): every model request (each round, the forced final,
and any recovery) runs its ``delta.content`` through a fresh caller-
owned :class:`app.rag.scaffolding.ScaffoldingFilter`, so raw
``<|tool_call_start|>…<|tool_call_end|>`` tokens can never reach the
user as answer text. A round that ends with NO visible content AND a
non-empty strip (the scaffolding was the whole "answer") gets exactly
ONE bounded recovery: one extra request with ``tools=None``, the same
messages with :data:`CORRECTION_INSTRUCTION` folded into the original
single system message, a fresh filter, and the same phase-67 retry
budget. A recovery that also comes back empty — or a round with no
strip and no content (today's empty/thinking-only answer) — settles as
before; a second empty reply raises :class:`MalformedReplyError` (the
API layer turns it into the dedicated error frame). A round with real
visible content plus scaffolding needs no recovery (the clean content
stands), and a scaffolding-only round that also carried tool calls
needs none either (the tool ran) — the policy keys on the no-calls
exit only. No model participates in detection or repair: the
registry + the fixed retry policy are the whole guardrail.
The DB accessors (:func:`list_catalog`, :func:`list_source_names`,
:func:`find_document`, :func:`all_documents`) and the
:func:`grep_document` line matcher are module-level functions so unit
@@ -116,11 +137,13 @@ from app.models import Document
from app.rag.git_sources import effective_sources
from app.rag.llm import (
LLMClient,
LLMError,
RetryPiece,
StreamPiece,
ToolCallPiece,
chat_stream_retried,
)
from app.rag.scaffolding import ScaffoldingFilter
from app.rag.source_removal import resolve_source_name
logger = logging.getLogger("app.agent")
@@ -228,6 +251,33 @@ UNKNOWN_TOOL = "Unknown tool."
MISSING_READ_ARGS = "read requires a string argument 'path'."
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
#: The harness-owned recovery line (phase 71, task 03) — folded into the
#: ORIGINAL single system message of the one bounded recovery request
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
#: the user message stays last). Verbatim constant: the E2E mock
#: (task 05) keys on a stable substring of it, so it must not drift.
CORRECTION_INSTRUCTION: str = (
"Your previous reply contained raw tool-call markup, which is not "
"interpreted here. Answer the user's question directly in plain "
"text — no tool syntax."
)
class MalformedReplyError(LLMError):
"""The model kept replying in raw tool-scaffolding (phase 71).
Raised ONLY by the recovery policy — :func:`run_agent` (grounded
path) and ``app.api.chat`` (deflected path) — when the one bounded
``tools=None`` recovery still comes back with no visible content.
It is never raised from inside a stream, so
:func:`app.rag.llm.chat_stream_retried`'s retry-before-first-piece
rule never sees it. The API layer catches it BEFORE the generic
:class:`LLMError` handler and settles the turn with the dedicated
"malformed reply" error frame (no ``done``, no ``query_log`` row).
Deterministic only (owner permission 2026-09-03): no model
participates in detection or repair.
"""
#: Search caps (owner-locked A5, phase 68): a global per-call match cap
#: (across documents, in catalog order) and a per-match-line char limit.
SEARCH_MAX_MATCHES = 20
@@ -352,10 +402,16 @@ class AgentHolder:
rejected calls (unknown tool, unknown/missing arguments or document,
already-in-context) do not count. Drives the per-turn log line's
``tool_calls=N`` field (task 04).
``scaffold_stripped``: how many chars of tool-scaffolding the
turn's filters removed across the turn's requests (rounds + the
forced final + any recovery, phase 71) — drives the per-turn log
line's ``scaffold_stripped=N`` field on grounded turns (the
deflected path computes its own total in ``app.api.chat``).
"""
read_docs: list[Document] = field(default_factory=list)
tool_calls: int = 0
scaffold_stripped: int = 0
def _execute_tool(
@@ -476,6 +532,24 @@ async def run_agent(
``settings.llm_retry_delay``); a round that already streamed pieces
fails the turn as before.
Scaffolding recovery (phase 71, deterministic only): every request —
each round, the forced final, and any recovery — runs its content
through a fresh :class:`app.rag.scaffolding.ScaffoldingFilter`. A
round that ends with NO visible content but a non-empty strip (the
scaffolding was the whole "answer") gets exactly ONE recovery:
``tools=None``, :data:`CORRECTION_INSTRUCTION` folded into the
original single system message (the rest of the history — user
message and tool results — unchanged), a fresh filter, the same
retry budget. A clean recovery ends the turn; a second empty reply
raises :class:`MalformedReplyError` (terminal — the API layer turns
it into the dedicated error frame). A round with visible content
plus scaffolding needs no recovery (the clean content stands), and
a scaffolding-only round that also carried tool calls needs none
(the tool ran) — the policy keys on the no-calls exit only. The
per-span strip warning log (each span truncated to 200 chars) is
the capture mechanism for new registry entries; *holder* accumulates
the turn's ``scaffold_stripped`` total for the API layer's log line.
``seed_docs`` are the documents the retrieval already put in context
(they shape the *system_prompt* the caller built); re-reading one of
them is rejected as "Already in your context." — the rejection counts
@@ -494,6 +568,12 @@ async def run_agent(
rounds = 0
while True:
calls: list[ToolCallPiece] = []
# Phase 71: one fresh filter per round (one per model request —
# the retry attempts of this logical request share it: a restart
# only happens while the filter was never fed). Content-only:
# thinking pieces pass through raw.
round_filter = ScaffoldingFilter()
round_content = 0 # visible (clean) content chars this round
# Phase 48: bind the round's stream so a consumer abandon
# (GeneratorExit into the yield below) tears down the in-flight
# model stream deterministically — not GC-dependent. Phase 67:
@@ -511,16 +591,97 @@ async def run_agent(
tools=tools,
retries=settings.llm_retries,
delay=settings.llm_retry_delay,
scaffolding=round_filter,
)
try:
async for piece in stream:
if isinstance(piece, ToolCallPiece):
calls.append(piece)
elif isinstance(piece, StreamPiece) and piece.kind == "content":
round_content += len(piece.text)
yield piece
finally:
await stream.aclose()
# Phase 71: the per-strip-event capture log — one warning per
# stripped span, truncated to 200 chars (how a new scaffolding
# format gets captured and added to the registry) — and the turn
# total for the API layer's ``scaffold_stripped=N`` log field.
holder.scaffold_stripped += round_filter.stripped_chars
for span in round_filter.stripped_spans:
logger.warning(
"agent: stripped %d chars of tool-scaffolding in round %d: %r",
len(span),
rounds + 1,
span[:200],
)
if not calls:
return # the answer was streamed
if round_content > 0:
return # the answer was streamed (the clean content stands)
if round_filter.stripped_chars == 0:
# Empty/thinking-only answer — today's behavior, unchanged
# (the UI handles it); the guardrail keys on a strip.
return
# Phase 71: the scaffolding was the whole "answer" — the ONE
# bounded recovery (a fixed policy, not a conversation):
# ``tools=None``, the correction folded into the ORIGINAL
# single system message (provider-safe — the user message and
# any tool history stay in place), a fresh filter, the same
# phase-67 retry budget.
logger.warning(
"agent: round %d was pure tool-scaffolding (%d chars stripped) "
"— running the one bounded recovery",
rounds + 1,
round_filter.stripped_chars,
)
messages_recovered = [
{
"role": "system",
"content": system_prompt + "\n" + CORRECTION_INSTRUCTION,
},
*messages[1:],
]
recovery_filter = ScaffoldingFilter()
recovered = chat_stream_retried(
llm,
cast("list[dict[str, str]]", messages_recovered),
tools=None,
retries=settings.llm_retries,
delay=settings.llm_retry_delay,
scaffolding=recovery_filter,
)
recovery_content = 0
try:
async for piece in recovered:
if isinstance(piece, StreamPiece) and piece.kind == "content":
recovery_content += len(piece.text)
yield piece
finally:
await recovered.aclose()
holder.scaffold_stripped += recovery_filter.stripped_chars
for span in recovery_filter.stripped_spans:
logger.warning(
"agent: stripped %d chars of tool-scaffolding in the "
"recovery after round %d: %r",
len(span),
rounds + 1,
span[:200],
)
if recovery_content > 0:
return # the recovery answered — the turn ends
# The second empty reply is terminal (at most one recovery per
# turn). Raised OUTSIDE the stream, so chat_stream_retried's
# retry rule never sees it; the API layer catches it before
# the generic LLMError handler.
logger.warning(
"agent: the recovery reply was still empty "
"(scaffold_stripped=%d) — settling with a malformed-reply error",
holder.scaffold_stripped,
)
raise MalformedReplyError(
f"the model answered in raw tool-scaffolding twice in a row "
f"(round {rounds + 1} plus one recovery) — no clean answer "
"to stream"
)
call = calls[0] # a stream can carry several calls; run the first
result = _execute_tool(db, call, seed_docs, holder)
rounds += 1 # every call the model emits consumes a round
@@ -558,17 +719,46 @@ async def run_agent(
# teardown as the loop rounds (consumer abandon mid-final
# answer must still close the model's stream). Phase 67: the
# forced call retries under the same locked-A2 rule as the
# loop rounds.
# loop rounds. Phase 71: the forced final runs through a
# fresh filter too — raw scaffolding can never reach the
# user from ANY grounded request.
final_filter = ScaffoldingFilter()
final = chat_stream_retried(
llm,
cast("list[dict[str, str]]", messages),
tools=None,
retries=settings.llm_retries,
delay=settings.llm_retry_delay,
scaffolding=final_filter,
)
final_content = 0
try:
async for piece in final:
if isinstance(piece, StreamPiece) and piece.kind == "content":
final_content += len(piece.text)
yield piece
finally:
await final.aclose()
# Phase 71: the same capture log + turn total; a
# scaffolding-only forced final (this turn used no recovery,
# so nothing is doubled up) settles with the same terminal
# malformed-reply error rather than a silently empty answer.
holder.scaffold_stripped += final_filter.stripped_chars
for span in final_filter.stripped_spans:
logger.warning(
"agent: stripped %d chars of tool-scaffolding in the "
"forced final answer (round %d): %r",
len(span),
rounds + 1,
span[:200],
)
if final_content == 0 and final_filter.stripped_chars > 0:
logger.warning(
"agent: the forced final answer was pure tool-scaffolding "
"— settling with a malformed-reply error"
)
raise MalformedReplyError(
"the forced final answer was raw tool-scaffolding — no "
"clean answer to stream"
)
return