phase: 124_image_chips_and_answers
All green. Final report: **Phase 124 final verification pass — all criteria verified, no defects found (nothing to fix)** - Verified the implemented phase (all 7 tasks done, changes in working tree): `BOR_ANSWER_IMAGES` toggle, `show_image` tool + refusals + `holder.shown_images`, toggle-gated prompt clause, `answer_image` frame field (None-omitting serializer) + `ChatMessage` persistence, `merge_cited_docs` auto-cite bug fix, answer-figure UI (live/restore/shared/CSS), mock triggers + E2E suite. Also confirmed the bytes route → public change matches the phase-123 shared-chat precedent (task 06 design). - `uv run pytest --cov=app --cov-report=term-missing` → **2848 passed**, TOTAL **99%** (>90% gate) - `bash .agents/validate.sh` → **validation OK** (tests + coverage + ruff + pyright) - `uv run ruff check . && uv run pyright` → clean, 0 errors - `uv run pytest tests/e2e/test_answer_images.py -v --no-cov` (isolated) → **5/5 passed** - Mock-inert spot checks: `tests/e2e/test_image_documents.py` 4/4, `tests/e2e/test_agent_document_tools.py` 4/4 **Completion criteria:** ① zero-read suggested-tier image doc chips with figure — verified (unit matrix, integration `test_zero_read_grounded_turn_auto_cites_suggested_image_doc`, E2E bug-fix flows); ② show_image → answer figure below text/above chips, second call refused, persists + restore + shared — verified (unit handler matrix, integration round-trip, E2E feature flow); ③ default-off byte-identical (3 tools, no clause, no key) — verified (unit + integration + E2E toggle-off); ④ full suite green / >90% / lint clean — verified above; ⑤ commit + phase move — harness responsibility (changes left uncommitted in working tree, no live-infrastructure changes made). **Next pending phase:** none — `todo/` holds only this phase; the pipeline is clear once the harness moves it.
This commit is contained in:
+143
-2
@@ -390,6 +390,71 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
},
|
||||
]
|
||||
|
||||
#: The 4th agent tool (phase 124, task 02; LOCKED A3/A5) — offered
|
||||
#: ONLY when ``settings.answer_images`` (``BOR_ANSWER_IMAGES``) is true:
|
||||
#: :func:`agent_tools` composes the offer list (the three base tools
|
||||
#: stay :data:`AGENT_TOOLS`, byte-identical). The description IS the
|
||||
#: relevance gate (A5 — "we don't want to just show every image that
|
||||
#: seems relevant"): call it only when the image itself contains the
|
||||
#: answer, at most ONE image per answer, the text answer is always
|
||||
#: given in full (the image supplements, never replaces). Same OpenAI
|
||||
#: function shape as the other three (one required string arg ``path``,
|
||||
#: the combined ``source/path`` identity — the bare-path-will-not-
|
||||
#: resolve teaching mirrored from ``read``).
|
||||
SHOW_IMAGE_TOOL: dict[str, Any] = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "show_image",
|
||||
"description": (
|
||||
"Show a standalone image document to the user as part of "
|
||||
"your answer. Call it only when the image itself contains "
|
||||
"the answer — a diagram, screenshot, or photo of the thing "
|
||||
"the user asked about — never for a merely related image. "
|
||||
"Show at most one image per answer. The image supplements "
|
||||
"your text answer — answer fully in text either way. Its "
|
||||
"`path` is the combined `source/path` string, exactly as "
|
||||
"shown in the `ls` output or the `<documents>` blocks — a "
|
||||
"bare document path will not resolve. Call one tool at a "
|
||||
"time — wait for this result before your next call."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"The image document to show, as the combined "
|
||||
"`source/path` string exactly as shown in the "
|
||||
"`ls` output (e.g. "
|
||||
"'homelab/docs/architecture.png'). A bare "
|
||||
"document path (without the source name) will "
|
||||
"not resolve. Show at most one image per "
|
||||
"answer — a second call is refused."
|
||||
),
|
||||
}
|
||||
},
|
||||
"required": ["path"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
|
||||
"""The grounded turn's tool offer list (phase 124, task 02).
|
||||
|
||||
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus
|
||||
:data:`SHOW_IMAGE_TOOL` when ``settings.answer_images``
|
||||
(``BOR_ANSWER_IMAGES``) is true — otherwise exactly the three base
|
||||
tools (the pre-phase-124 offer, byte-identical: no 4th tool in the
|
||||
wire ``tools`` array). A FRESH list on every call — callers never
|
||||
mutate the result or the constants (``run_agent`` passes it straight
|
||||
to the LLM request).
|
||||
"""
|
||||
tools: list[dict[str, Any]] = list(AGENT_TOOLS)
|
||||
if settings.answer_images:
|
||||
tools.append(SHOW_IMAGE_TOOL)
|
||||
return tools
|
||||
|
||||
#: Tool refusal texts (phase 37): rejected calls count in nothing
|
||||
#: (``holder.tool_calls`` tracks executed calls); the round cap bounds
|
||||
#: their pathological repetition (phase 45). The in-context line is a
|
||||
@@ -407,6 +472,24 @@ ALREADY_IN_CONTEXT = (
|
||||
UNKNOWN_TOOL = "Unknown tool."
|
||||
MISSING_READ_ARGS = "read requires a string argument 'path'."
|
||||
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
|
||||
MISSING_SHOW_ARGS = "show_image requires a string argument 'path'."
|
||||
|
||||
#: The ``show_image`` refusals (phase 124, task 02): the same contract
|
||||
#: as every other refusal — counts in nothing, consumes a round,
|
||||
#: changes no context. ``NOT_AN_IMAGE_DOC``: one ``{arg}`` field, the
|
||||
#: argument echoed as passed — a text doc is refused, never silently
|
||||
#: converted to a read (LOCKED A5). ``ALREADY_SHOWING``: the LOCKED A3
|
||||
#: one-image-per-answer gate, enforced SERVER-side (the model is taught
|
||||
#: it in the tool description + the prompt clause; the handler is the
|
||||
#: actual gate).
|
||||
NOT_AN_IMAGE_DOC = (
|
||||
"'{arg}' is not an image document — only standalone image "
|
||||
"documents can be shown as answers."
|
||||
)
|
||||
ALREADY_SHOWING = (
|
||||
"Already showing an image this answer — do not show another one. "
|
||||
"Answer from what you have."
|
||||
)
|
||||
|
||||
#: The read-truncation notice (phase 95, task 01) — appended, after the
|
||||
#: shared :data:`TRUNCATION_MARKER`, to a ``read`` result whose document
|
||||
@@ -1166,12 +1249,19 @@ class AgentHolder:
|
||||
as today — this list only carries the truncation signal the agent
|
||||
loop turns into :class:`app.rag.llm.ToolResultPiece` values (task 02
|
||||
surfaces them to the UI).
|
||||
``shown_images`` (phase 124, task 02): the documents ``show_image``
|
||||
showed, in call order — at most one in practice (LOCKED A3: a
|
||||
second call is refused with :data:`ALREADY_SHOWING`, which appends
|
||||
nothing). Read by ``app.api.chat`` after the stream for the done
|
||||
frame's ``answer_image`` field (task 04) — the ``read_docs``
|
||||
precedent (an API-layer read-after-stream signal).
|
||||
"""
|
||||
|
||||
read_docs: list[Document] = field(default_factory=list)
|
||||
tool_calls: int = 0
|
||||
scaffold_stripped: int = 0
|
||||
read_truncations: list[tuple[str, int, int]] = field(default_factory=list)
|
||||
shown_images: list[Document] = field(default_factory=list)
|
||||
|
||||
|
||||
def _execute_tool(
|
||||
@@ -1327,6 +1417,51 @@ def _execute_tool(
|
||||
f"{marker}"
|
||||
f"{doc.content}"
|
||||
)
|
||||
if call.name == "show_image":
|
||||
# Phase 124 (task 02; LOCKED A3/A5): the 4th agent tool — the
|
||||
# model shows a standalone image document as part of its
|
||||
# answer. Offered only when ``settings.answer_images``
|
||||
# (``agent_tools``); the handler enforces the contract
|
||||
# server-side — image docs only, ONE image per answer. Every
|
||||
# refusal below counts in nothing and consumes a round, like
|
||||
# every other refusal.
|
||||
raw_path = call.arguments.get("path")
|
||||
arg = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if not arg:
|
||||
return MISSING_SHOW_ARGS
|
||||
doc, _source, _path = _resolve_path(db, arg)
|
||||
if doc is None:
|
||||
# The same phase-72 "did you mean …?" teaching as the read
|
||||
# branch (a refusal — not counted, no context).
|
||||
return _no_document_refusal(db, arg)
|
||||
if not doc.is_image:
|
||||
# LOCKED A5: a text doc is refused, never silently
|
||||
# converted to a read — the tool shows IMAGE documents
|
||||
# only.
|
||||
return NOT_AN_IMAGE_DOC.format(arg=arg)
|
||||
if holder.shown_images:
|
||||
# LOCKED A3: one image per answer — server-enforced, not
|
||||
# just taught.
|
||||
return ALREADY_SHOWING
|
||||
holder.shown_images.append(doc)
|
||||
holder.tool_calls += 1
|
||||
# The confirming result: a NEW first line (absent from ``read``
|
||||
# — the model must know the show registered) followed by the
|
||||
# exact document block ``read`` returns for an image doc at-or-
|
||||
# under-cap (the ``Document …`` header + phase-106 date line +
|
||||
# :data:`IMAGE_DOC_MARKER` — the header contract the E2E mock
|
||||
# pins on ``read`` results is deliberately reused here). An
|
||||
# image doc's content is the summary-capped vision description
|
||||
# — short by construction (phase 122, task 03), so NO
|
||||
# truncation machinery; the plain block stays correct even if
|
||||
# it ever exceeded the cap.
|
||||
return (
|
||||
"Image shown to the user — it is now part of your answer.\n"
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{IMAGE_DOC_MARKER}\n"
|
||||
f"{doc.content}"
|
||||
)
|
||||
if call.name == "grep":
|
||||
raw_pattern = call.arguments.get("pattern")
|
||||
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
|
||||
@@ -1485,9 +1620,15 @@ async def run_agent(
|
||||
# Phase 45: no per-tool budgets — the tools stay offered for the
|
||||
# whole turn, bounded by the round cap. ``0`` is the no-tools kill
|
||||
# switch: exactly one request with ``tools=None`` (the pre-phase-37
|
||||
# path).
|
||||
# path). Phase 124 (task 02): the offer list is :func:`agent_tools`
|
||||
# — the three base tools (byte-identical) plus ``show_image`` only
|
||||
# when ``settings.answer_images`` (``BOR_ANSWER_IMAGES``) is on;
|
||||
# off (the default) the offer is element-for-element the
|
||||
# pre-phase-124 list.
|
||||
max_rounds = settings.agent_max_rounds
|
||||
tools: list[dict[str, Any]] | None = AGENT_TOOLS if max_rounds > 0 else None
|
||||
tools: list[dict[str, Any]] | None = (
|
||||
agent_tools(settings) if max_rounds > 0 else None
|
||||
)
|
||||
rounds = 0
|
||||
while True:
|
||||
calls: list[ToolCallPiece] = []
|
||||
|
||||
Reference in New Issue
Block a user