Files

T

ducoterra 0ff1f8c4d6 chore(agent): phase roadmap from TODO.md — 4 phases (120–123)

Protocol B append: failed-turn retry (L3–4), git source tokens (L5),
image documents (L6 ingest), chat image questions (L6 chat side).
TODO.md items now live in .agents/phases/todo/ and the file is cleared.

LLM-Generated: true

2026-09-24 16:27:44 -04:00

3.3 KiB

Raw Blame History

Task 05 — RAG display: image docs in the chat sources + the agent read marker

Phase: 122_image_documents · Source: TODO.md:6 — "Images retrieved by the RAG should be shown in the chat nicely."

Objective

When a retrieved/agent-read document is an image, the chat shows it: the sources block renders a compact inline image with its summary as caption/alt, and the agent's read tool tells the model it is reading a generated image description.

Work

app/rag/retriever.py / app/api/chat.py — the sources frames the chat bubble renders (the SSE sources/related-doc frames and the agent-sourced doc list): add an OPTIONAL image_url field to the per-doc ref shape — populated (the /api/documents/{id}/image path) iff the doc row has is_image, absent otherwise (the omission rule — text-doc frames stay byte-identical). The retriever already has the Document row; the agent's doc refs (the read-tool results / source list) do too — set it at the frame-build sites (grep for the source-ref construction in both modules; one shared helper source_ref_with_image(doc, …) keeps the two sites in lockstep).
frontend/assets/app.js — the chat's sources block renderer: when a source ref carries image_url, render a compact inline <img> (max-height ~96px, object-fit: contain, the theme's surface, alt + visible caption = the doc summary — the "shown nicely" requirement) in place of / beside the existing doc chip text (keep the title + the existing chip affordance — the image is additive, not a replacement). A failed image load collapses to the plain chip (never a broken-image icon).
app/rag/agent.py — the read tool's result for an image doc: prefix the description with the marker line Image document — the text below is a description generated from the image: (a module constant) so the model reasons about what it is reading; non-image docs' results are byte-identical.
ASSUMPTION: the chat QUESTION side (users submitting images) is phase 123 — this task only covers RETRIEVED images in the answer's sources.
ASSUMPTION: the sources-frame image_url is the only new frame field — no doc-id leak beyond what the frame already carries (the path encodes the doc id, same as the content endpoint).

Testing & Quality

Integration: tests/integration/test_chat_api.py (extend, task 06) — a mocked grounded answer that includes an image doc in its sources → the SSE frame carries image_url for that ref only; a text-only grounding has NO image_url key anywhere (byte check).
Unit: tests/unit/test_image_documents.py (task 06) — the frame-helper (present/absent), the agent marker (image vs non-image result), house-style source assertions: the sources renderer reads image_url, sets alt, and falls back on image error.
E2E: test_image_documents.py scenario (task 06) — ask a question the mock LLM grounds on the fixture image doc → the chat sources block shows the inline image with its caption.
Coverage: >90% on the touched modules.

Completion Criteria

A chat answer grounded on an image doc shows the inline image + caption in its sources block; text-doc answers render byte-identically to before.
The agent read result for an image doc carries the marker; the model sees the description, not raw bytes.
uv run pytest green; uv run ruff check . && uv run pyright clean.

3.3 KiB Raw Blame History