Files
brain-of-reese/.agents/phases/todo/122_image_documents/03_image_description.md
T
ducoterra 0ff1f8c4d6 chore(agent): phase roadmap from TODO.md — 4 phases (120–123)
Protocol B append: failed-turn retry (L3–4), git source tokens (L5),
image documents (L6 ingest), chat image questions (L6 chat side).
TODO.md items now live in .agents/phases/todo/ and the file is cleared.

LLM-Generated: true
2026-09-24 16:27:44 -04:00

4.5 KiB
Raw Blame History

Task 03 — Image description: the vision model writes the only embedded text

Phase: 122_image_documents · Source: TODO.md:6 — "the only embedded part of an image will be the summary generated by the model."

Objective

describe_image generates the image's description with the CHAT model (vision), the description becomes BOTH Document.content and Document.summary (so the chunk pipeline embeds exactly that text — and only that text), and a failed description fails soft (skip + count + log, sync continues).

Work

  1. app/rag/summarizer.py — new async def describe_image(llm, data: bytes, mime: str, settings=None) -> str | None (the summarizer module owns model-text generation; follow the existing summary-call conventions — client, model, timeout, the summary_max_chars cap):
    • ONE chat-model call (settings.llm_chat_model — LOCKED A3; the lite summary model is not assumed vision-capable) with messages [{role: "user", content: [{type: "text", text: <DESCRIBE_PROMPT>}, {type: "image_url", image_url: {url: f"data:{mime};base64,{b64}"}}]}] — the multimodal content-list shape the OpenAI-compatible API expects.
    • DESCRIBE_PROMPT (a module constant, pinned by a unit test): a faithful, retrieval-oriented description — what is depicted, any visible text/labels/titles, diagram/table structure, salient details; 2–4 sentences of substance (the description is the ONLY retrievable text of the doc, so it must carry the image's meaning).
    • Return the stripped text capped at settings.summary_max_chars (the phase-30 cap — the description IS the summary); return None on any client error, empty response, or non-2xx (the caller fails soft). No retries beyond the SDK's own — a description failure must not stall a sync.
  2. app/rag/importer.py — wire the task-02 seam: the image branch's content/summary come from describe_image —
    • description None → skip the doc entirely (no row, no image_dir copy kept — delete the copy if it was made, or make the copy AFTER a successful description so a failure never leaves an orphan), summary.images_failed += 1, logger.warning("import: image description failed source=%s path=%s", source, rel) — the fail-soft skip (LOCKED A3).
    • success → content = description, then the existing _store_summary path (L582) runs with the description as the summary (the is_summary position −1 chunk mirrors it — phase-30 behavior, unchanged), and the normal content chunks embed the description (for a short description that is typically ONE content chunk + the summary chunk — the chunker's existing behavior, no special case).
    • the phase-118 backfill branch (unchanged image doc, summary is None) calls the SAME path — a description failure there keeps the doc as-is and logs (no row mutation).
  3. app/rag/llm.py — no new client: describe_image reuses the existing llm.chat-equivalent client the summarizer already uses for text summaries (verify the exact client method name in app/rag/summarizer.py and match it — the multimodal payload is a plain list[dict] message, so no client change is needed; IF the existing client hard-codes text-only content: str typing, extend its signature to accept content: str | list — pyright-clean).
  4. ASSUMPTION (A3 re-stated): the CHAT model describes; if the owner's chat model lacks vision, describe_image returns None (the SDK errors) and every image doc is skipped + logged — honest, visible failure (the images_failed counter in the sync log is the signal).

Testing & Quality

  • Unit: tests/unit/test_image_documents.py (task 06) — with a MOCK client: the prompt shape (text part + image_url data-URL part, correct model), the cap is applied, whitespace stripped; None on mock error / empty string / client exception; the importer's skip path (no row, images_failed == 1, warning logged, no orphan copy) and the success path (content == summary == description, is_summary chunk present, embedding called with the description text — the ONLY text embedded).
  • Integration: the mock-vision import_sources end-to-end (task 06).
  • Coverage: >90% on the touched modules.

Completion Criteria

  • A fixture PNG through the mock vision client yields a doc whose content == summary == the description, with its embedding(s) derived from that text only.
  • A failing mock client skips the doc, bumps images_failed, logs, and the sync completes with the other docs indexed.
  • uv run pytest green; uv run ruff check . && uv run pyright clean.