phase: 124_image_chips_and_answers
All green. Final report: **Phase 124 final verification pass — all criteria verified, no defects found (nothing to fix)** - Verified the implemented phase (all 7 tasks done, changes in working tree): `BOR_ANSWER_IMAGES` toggle, `show_image` tool + refusals + `holder.shown_images`, toggle-gated prompt clause, `answer_image` frame field (None-omitting serializer) + `ChatMessage` persistence, `merge_cited_docs` auto-cite bug fix, answer-figure UI (live/restore/shared/CSS), mock triggers + E2E suite. Also confirmed the bytes route → public change matches the phase-123 shared-chat precedent (task 06 design). - `uv run pytest --cov=app --cov-report=term-missing` → **2848 passed**, TOTAL **99%** (>90% gate) - `bash .agents/validate.sh` → **validation OK** (tests + coverage + ruff + pyright) - `uv run ruff check . && uv run pyright` → clean, 0 errors - `uv run pytest tests/e2e/test_answer_images.py -v --no-cov` (isolated) → **5/5 passed** - Mock-inert spot checks: `tests/e2e/test_image_documents.py` 4/4, `tests/e2e/test_agent_document_tools.py` 4/4 **Completion criteria:** ① zero-read suggested-tier image doc chips with figure — verified (unit matrix, integration `test_zero_read_grounded_turn_auto_cites_suggested_image_doc`, E2E bug-fix flows); ② show_image → answer figure below text/above chips, second call refused, persists + restore + shared — verified (unit handler matrix, integration round-trip, E2E feature flow); ③ default-off byte-identical (3 tools, no clause, no key) — verified (unit + integration + E2E toggle-off); ④ full suite green / >90% / lint clean — verified above; ⑤ commit + phase move — harness responsibility (changes left uncommitted in working tree, no live-infrastructure changes made). **Next pending phase:** none — `todo/` holds only this phase; the pipeline is clear once the harness moves it.
This commit is contained in:
@@ -467,6 +467,56 @@ Implements just enough of the aipi surface:
|
||||
the trigger phrase is disjoint from every other trigger's; verified
|
||||
2026-09-16: no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``show me the picture of``
|
||||
(``SHOW_IMAGE_TRIGGER``, phase 124 task 07 — the image-chips-and-
|
||||
answers suite ``tests/e2e/test_answer_images.py``) **and** the
|
||||
system prompt contains ``show_image`` (the tool clause's presence
|
||||
— the ``BOR_ANSWER_IMAGES`` toggle's server-side marker: the
|
||||
clause rides inside the HIGH prompt's ``<tools>`` section ONLY
|
||||
when the toggle is on, so on a default-OFF app the trigger NEVER
|
||||
fires and the mock behaves exactly as pre-phase for every other
|
||||
suite) -> the deterministic SHOW_IMAGE flow, discriminated
|
||||
statelessly from the tool results (streaming only): the mock
|
||||
picks the IMAGE document seeded in the ``<documents>`` blocks —
|
||||
the block whose ``path`` attribute ends in an image extension
|
||||
(``app.rag.summarizer.IMAGE_MIMES``' family — the suite's
|
||||
single-PNG fixture guarantees exactly one such block):
|
||||
* request 1 (``tools`` offered, no confirmed show yet): stream
|
||||
ONLY ``tool_calls`` deltas — ``show_image`` with
|
||||
``{"path": <combined source/path>}`` (synthetic id
|
||||
``call_show``, ``finish_reason: "tool_calls"``, no content) —
|
||||
NO ``read`` (the ZERO-READ turn: the image doc's seed block
|
||||
already carries its entire content, so a real model never
|
||||
reads it — the zero-read shape is what makes the bug-fix's
|
||||
chip assertion meaningful);
|
||||
* the follow-up request (a ``tool``-role result carrying the
|
||||
agent's confirming line ``Image shown to the user`` —
|
||||
``_SHOW_CONFIRM_MARKER``) in the messages: the deterministic
|
||||
FINAL answer — a short sentence ending with the ``<documents>``
|
||||
block's tail echo (the phase-24 END_OF_NOTES echo convention —
|
||||
it proves the seed was there), NO further tool calls.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||
phrase — the phase-71/72/94/118 ordering convention); verified
|
||||
2026-09-26: no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``which diagram is in your notes``
|
||||
(``DIAGRAM_TRIGGER``, phase 124 task 07 — the BUG-FIX-ONLY
|
||||
trigger, same suite) -> the deterministic ZERO-READ answer: the
|
||||
mock answers straight from the seed with NO tool calls at all
|
||||
(no ``show_image``, no ``read`` — unlike the ``SHOW_IMAGE_
|
||||
TRIGGER`` flow it needs NO clause, so it also fires on a default-
|
||||
OFF app: the chips fix is not gated by ``BOR_ANSWER_IMAGES``,
|
||||
it belongs to the ``BOR_IMAGES``-indexed docs) and QUOTES THE
|
||||
TAIL of the seeded image doc's ``<document>`` block (the same
|
||||
image-doc pick as above): proving the description WAS in the
|
||||
prompt (for an image doc the seed block IS its entire content)
|
||||
and that its chip appears with NO tool calls at all (the phase-
|
||||
124 auto-cite — the fix for the owner's report). Checked BEFORE
|
||||
the ``DEFLECT_MODE`` branch (the marker lives in the user message
|
||||
— the ``TABLE_TRIGGER``/``HISTORY_TRIGGER`` convention), so a
|
||||
marker question always gets the echo whatever the honesty gate
|
||||
says. Verified 2026-09-26: no existing E2E question or fixture
|
||||
file contains the phrase, so every other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -580,6 +630,7 @@ from fastapi.responses import JSONResponse, StreamingResponse
|
||||
|
||||
from app.rag.agent import ALREADY_IN_CONTEXT, CORRECTION_INSTRUCTION
|
||||
from app.rag.folder_summaries import FOLDER_HEADER_PREFIX # phase 94: the mock's key
|
||||
from app.rag.summarizer import IMAGE_MIMES # phase 124: the image-doc extension family
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
@@ -1756,6 +1807,117 @@ _SUMMARY_SEED_READ_CALL_RE = re.compile(
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the scripted
|
||||
# SHOW_IMAGE flow + the zero-read bug-fix trigger — see the module
|
||||
# docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``show_image`` prompt clause in the system
|
||||
#: prompt (the ``BOR_ANSWER_IMAGES`` toggle's server-side marker —
|
||||
#: the clause rides inside the HIGH prompt's ``<tools>`` section ONLY
|
||||
#: when the toggle is on) — drives the deterministic SHOW_IMAGE flow
|
||||
#: (the mock shows the seeded image document and answers straight
|
||||
#: from the seed — ZERO ``read`` calls). On a default-OFF app the
|
||||
#: clause is absent, so the trigger NEVER fires there and the mock
|
||||
#: behaves exactly as pre-phase for every other suite. Disjoint from
|
||||
#: every other trigger phrase — the phase-71/72/94/118 ordering
|
||||
#: convention; verified 2026-09-26: no existing E2E question or
|
||||
#: fixture file contains the phrase, so every other suite is
|
||||
#: unaffected.
|
||||
SHOW_IMAGE_TRIGGER = "show me the picture of"
|
||||
|
||||
#: The phase-124 BUG-FIX-ONLY trigger (case-insensitive): the mock
|
||||
#: answers straight from the seed with ZERO tool calls and QUOTES THE
|
||||
#: TAIL of the seeded image doc's ``<document>`` block — proving the
|
||||
#: description was in the prompt (for an image doc the seed block IS
|
||||
#: its entire content) and that its chip appears with NO tool calls
|
||||
#: at all (the phase-124 auto-cite). Unlike ``SHOW_IMAGE_TRIGGER``
|
||||
#: it needs NO clause — the chips fix is not gated by
|
||||
#: ``BOR_ANSWER_IMAGES`` (it belongs to the ``BOR_IMAGES``-indexed
|
||||
#: docs), so it also fires on a default-OFF app. Checked BEFORE the
|
||||
#: ``DEFLECT_MODE`` branch in :func:`compose_answer` (the marker
|
||||
#: lives in the user message — the TABLE/HISTORY convention), so a
|
||||
#: marker question always gets the echo whatever the honesty gate
|
||||
#: says.
|
||||
DIAGRAM_TRIGGER = "which diagram is in your notes"
|
||||
|
||||
#: The agent's ``show_image`` SUCCESS result's confirming first line
|
||||
#: (``app.rag.agent``'s ``show_image`` branch — the line ``read``'s
|
||||
#: result never carries): the SHOW_IMAGE flow's follow-up request
|
||||
#: keys on it in a ``tool``-role result (stateless discrimination —
|
||||
#: the house marker-flow convention).
|
||||
_SHOW_CONFIRM_MARKER = "Image shown to the user"
|
||||
|
||||
#: The image-doc extension family (``app.rag.summarizer.IMAGE_MIMES``
|
||||
#: — phase 122's ext→mime map, imported so the mock can never drift
|
||||
#: from the app's image family): a seeded ``<documents>`` block is
|
||||
#: the IMAGE document when its ``path`` attribute ends in one of
|
||||
#: these (the suite's single-PNG fixture guarantees exactly one such
|
||||
#: block — the pick is deterministic).
|
||||
_IMAGE_PATH_SUFFIXES = tuple(IMAGE_MIMES)
|
||||
|
||||
|
||||
def _seeded_image_doc(system: str) -> tuple[str, str, str] | None:
|
||||
"""The seeded image doc's ``(source, path, content)`` — or ``None``.
|
||||
|
||||
Phase 124 (task 07): the ``<documents>`` block (the
|
||||
:data:`_DOCUMENT_BLOCK_RE` machinery) whose ``path`` attribute
|
||||
ends in an image extension (:data:`_IMAGE_PATH_SUFFIXES`) is the
|
||||
seeded image document — for an image doc the block's content IS
|
||||
the document's entire text (the vision description — content ==
|
||||
summary, phase 122 task 03). Both of this phase's triggers pick
|
||||
the doc this way (the suite's single-PNG fixture guarantees
|
||||
exactly one such block; ``None`` = a text-only KB, the premise
|
||||
broke — the caller degrades loudly).
|
||||
"""
|
||||
for block in _DOCUMENT_BLOCK_RE.finditer(system):
|
||||
if block.group("path").lower().endswith(_IMAGE_PATH_SUFFIXES):
|
||||
return (block.group("source"), block.group("path"), block.group("content"))
|
||||
return None
|
||||
|
||||
|
||||
def _show_image_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-124 SHOW_IMAGE request (see the module
|
||||
docstring). Stateless over the messages, like the other marker
|
||||
flows:
|
||||
|
||||
* ``("call", combined)`` — the system prompt carries the
|
||||
``show_image`` clause (the toggle-ON gate) and no ``tool``-role
|
||||
result has confirmed the show yet: stream ONLY the
|
||||
``show_image`` tool-call delta on the seeded image doc's
|
||||
combined ``source/path`` (synthetic id ``call_show``,
|
||||
``finish_reason: "tool_calls"``, no content) — NO ``read`` (the
|
||||
zero-read turn).
|
||||
* ``("answer",)`` — a ``tool``-role result carrying the agent's
|
||||
confirming line (:data:`_SHOW_CONFIRM_MARKER`) is in the
|
||||
messages: the deterministic FINAL answer — a short sentence
|
||||
ending with the ``<documents>`` block's tail echo (the phase-24
|
||||
END_OF_NOTES echo convention — it proves the seed was there),
|
||||
NO further tool calls.
|
||||
* ``None`` — not the flow: the trigger phrase is absent, the
|
||||
clause is absent (a default-OFF app — the mock behaves exactly
|
||||
as pre-phase), the turn is deflected (no ``<documents>`` seed
|
||||
— a deflection prompt never carries one), or no seeded image
|
||||
doc exists (a text-only KB — the premise broke; the suite fails
|
||||
loudly on the missing figure).
|
||||
"""
|
||||
if SHOW_IMAGE_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
system = _system(body)
|
||||
if "show_image" not in system:
|
||||
return None
|
||||
doc = _seeded_image_doc(system)
|
||||
if doc is None:
|
||||
return None
|
||||
if any(_SHOW_CONFIRM_MARKER in r for r in _tool_results(body)):
|
||||
return ("answer",)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("call", f"{doc[0]}/{doc[1]}")
|
||||
|
||||
|
||||
def _summary_seed_read_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-118 scripted summary-read request (see the
|
||||
module docstring). The question carries the scripted call (``read
|
||||
@@ -2280,6 +2442,31 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
# message, a deflection prompt never carries it, so a marker
|
||||
# question always gets the echo whatever the gate says.
|
||||
answer = _history_echo(body)
|
||||
elif DIAGRAM_TRIGGER in user.lower():
|
||||
# Phase 124 (task 07): the BUG-FIX-ONLY trigger — the mock
|
||||
# answers straight from the seed with ZERO tool calls (no
|
||||
# ``show_image``, no ``read`` — the clause is NOT required,
|
||||
# so the flow also fires on a default-OFF app: the chips fix
|
||||
# belongs to the ``BOR_IMAGES``-indexed docs) and quotes the
|
||||
# TAIL of the seeded image doc's ``<document>`` block (the
|
||||
# same pick as the SHOW_IMAGE flow — the block's content is
|
||||
# the image doc's entire text, the vision description):
|
||||
# proving the description WAS in the prompt and that the chip
|
||||
# appears with NO tool calls at all (the phase-124 auto-cite
|
||||
# — the fix for the owner's report). Checked BEFORE the
|
||||
# DEFLECT_MODE branch (the marker lives in the user message —
|
||||
# the TABLE/HISTORY convention), so a marker question always
|
||||
# gets the echo whatever the honesty gate says. A text-only KB
|
||||
# (no seeded image doc) degrades to the plain line — the
|
||||
# suite's premise broke; it fails loudly on the missing quote.
|
||||
image_doc = _seeded_image_doc(system)
|
||||
answer = (
|
||||
f"Your notes carry that diagram — it reads: “{image_doc[2][-160:]}” "
|
||||
"(Deterministic mock answer for E2E.)"
|
||||
if image_doc
|
||||
else "Your notes don't carry that diagram — there is nothing to show. "
|
||||
"(Deterministic mock answer for E2E.)"
|
||||
)
|
||||
elif "DEFLECT_MODE" in system:
|
||||
answer = (
|
||||
"Ah — I haven't done anything like that, so I don't want to make stuff up! "
|
||||
@@ -2911,6 +3098,42 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
# extension path) and answers straight from the seed: ZERO
|
||||
# ``read`` calls (the zero-read turn that makes the bug-fix's
|
||||
# chip assertion meaningful). Clause-gated — the ``show_image``
|
||||
# teaching line in the system prompt is the
|
||||
# ``BOR_ANSWER_IMAGES`` toggle's presence marker, so on a
|
||||
# default-OFF app the trigger never fires and the mock stays
|
||||
# byte-identical to pre-phase for every other suite. Checked
|
||||
# BEFORE the plain TOOLS_TRIGGER flow (disjoint trigger
|
||||
# phrase — the phase-71/72/94/118 ordering convention; the
|
||||
# trigger needs the clause, so deflected turns never hit it).
|
||||
show_image = _show_image_flow(body)
|
||||
if show_image is not None:
|
||||
if show_image[0] == "call":
|
||||
stream = _tool_call_stream(
|
||||
"show_image", {"path": show_image[1]}, "call_show"
|
||||
)
|
||||
else: # "answer" — the final answer + the tail echo
|
||||
block = _DOCUMENTS_BLOCK_RE.search(_system(body))
|
||||
tail = (block.group(0) if block else _context(body))[-160:]
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
"Here is the picture you asked about — …and the very "
|
||||
f"end of my notes reads: “{tail}” "
|
||||
"(Deterministic mock answer for E2E.)",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 109 (task 03, never-frozen-turn story suite): the
|
||||
# deterministic REPORTED-REPRO turn (delta → tool →
|
||||
# thinking-after-delta — TODO.md L3): the scripted first answer,
|
||||
|
||||
Reference in New Issue
Block a user