brain-of-reese/.agents/phases/todo/122_image_documents/00_phase.md

Phase 122 — Image documents: standalone images become first-class, retrievable documents

Source: TODO.md L6 — "Need to support images. Images uploaded as part of documents or as standalone images should be read, summarized, and retrieved like any other document. Note that the embedding model won't support images, so the only embedded part of an image will be the summary generated by the model. The user should be able to turn on and off image support in their .env depending on whether their model supports it. Images retrieved by the RAG should be shown in the chat nicely and users should be able to submit images as part of their question in brain of reese." Story: n/a (feature request; extends the phase-28/30/38 import pipeline, phase-90 no-scan uploads, and the RAG/chat assets). Context: app/config.py — Settings (BOR_ prefix; upload_dir L361, sources_dir L355, upload_max_mb L370 — raw-string/expanduser house convention). app/rag/importer.py — iter_importable_files (L235, extension filter via llm.settings.import_extension_set + match_extension L217), import_sources (L279, prune L283), _index_file (L435 — read_text L453, sha256 over text, Document upsert, the unchanged/hash path with the phase-118 summary backfill), _store_summary (L582 — lite-model summary + the position −1 is_summary chunk), _prune (L649 — deletes docs of the imported sources not in seen). app/models.py — Document (L102: content, content_hash, summary L140, created_at/created_at_manual), Chunk (L148: is_summary L161). app/rag/summarizer.py + app/rag/llm.py — the lite summary path + the chat-model client (Settings model names, check_models). app/rag/archive_upload.py — archive unpack into upload_dir (image members land on disk today, then get filtered out by the extension walk). app/api/git_sources.py:432 — the upload endpoint. app/api/docs.py — the document content endpoint (document viewer). app/rag/retriever.py / app/rag/agent.py (read tool, message build L1446–1448) / app/api/chat.py — RAG + the SSE sources frames. app/api/config.py:30 — GET /api/config public flags dict (task 01 of phase 123 extends it). Frontend: frontend/assets/app.js (chat source chips), frontend/assets/document.js + frontend/document.html (#doc-content L144), frontend/assets/sources.js (Sources page). alembic/ — migrations.

Objective

With BOR_IMAGES=true, a standalone image file — arriving as a direct upload, inside an uploaded archive, or as a file in a git/local source — becomes a first-class document: the vision model (the chat model) describes it, the description is the document's content AND summary, only the description is embedded (the embedding model never sees pixels), the image bytes persist and are served, and the image shows up in the Sources page, the document viewer, and the chat — with retrieved image docs rendered inline in the answer's sources.

Dependencies

• 121_git_source_tokens (todo) — pipeline predecessor (execution order) only; no code dependency.
• Code dependencies (all complete): phase 30 summary pipeline (_store_summary, is_summary chunk), phase 90 no-scan upload, phase 89/105 per-source walk options, the RAG agent + SSE sources frames.

Design (shared by all tasks — the executor reads this, not the chat)

• Toggle (task 01, LOCKED A3): three new Settings fields — images: bool = False (BOR_IMAGES, 0/false off — the phase-67 llm_retries bool style), image_extensions: str = "png,jpg,jpeg,webp,gif,bmp" (BOR_IMAGE_EXTENSIONS, comma-separated, lowercased into a frozenset by the importer — the import_extension_set property precedent), and image_dir: str = "~/bor-sources/images" (BOR_IMAGE_DIR, raw-string/expanduser convention — the persistent home for image bytes, deliberately separate from sources_dir/upload_dir). .env.example gets all three with a comment: off by default — enable only when your chat model supports vision, because image descriptions are generated by the chat model. GET /api/config (task 01) gains images: bool so the UI can gate affordances (consumed by phase 123; the Sources page can show an "images off" hint — optional, not required).
• Why the bytes are copied (task 02): upload dirs are REPLACED on every upload (archive_upload.swap_in), git checkouts are re-cloned, and local dirs are user-edited — a served image must outlive its source file. The importer copies each ingested image to image_dir/<doc-uuid>.<ext> (created on demand) and stores that path in Document.image_path. The copy happens ONLY when the doc is new or its hash changes; a replaced image deletes the stale copy; pruned docs delete their copy.
• Storage (task 02): Document.is_image: bool (server default false — every pre-phase-122 row is a text doc) + Document.image_path: str | None (NULL for text docs). One migration, one downgrade.
• Ingest (task 02): the walk: iter_importable_files/import_sources accept the image frozenset IN ADDITION to import_extension_set, ONLY when settings.images is true (images are never user-configurable via BOR_IMPORT_EXTENSIONS — the toggle is the single knob, LOCKED A3/A4). _index_file branches on image extension: read BYTES (not read_text), sha256 over the bytes (the digest rule is unchanged — content identity), copy to image_dir, set is_image + image_path, and content = the vision description (task 03). The normal chunk pipeline then embeds the content (= the description) — that is exactly the TODO's "the only embedded part of an image will be the summary generated by the model"; the phase-30 summary chunk (is_summary, position −1) mirrors Document.summary, which equals the description too. Title = the file stem (the non-markdown rule at L505–509). The unchanged/hash path works unmodified (byte digest → "unchanged" skips re-describing; the phase-118 backfill path re-describes a NULL-summary image doc on its next sync — same fail-soft). Prune guard: _prune (L649) must NOT delete is_image docs while settings.images is false (an image doc is invisible to an images-off walk, not a deleted file — otherwise turning the toggle off and syncing would silently destroy the image documents). Toggle ON → normal prune semantics (a deleted image file prunes its doc + copy).
• Description (task 03, LOCKED A3): describe_image in app/rag/summarizer.py (one function, the summarizer module owns model-text generation): a SINGLE chat-model call (Settings.llm_chat_model — the vision model; the lite summary model is NOT assumed vision-capable, LOCKED A3) with a multimodal user message [{type: "text", text: <fixed describe prompt>}, {type: "image_url", image_url: {url: <data URL from the bytes + mime>}}]; the prompt asks for a faithful, retrieval-oriented description (what is shown, any text/labels/diagram content, salient details — the description is the ONLY thing retrievable, so it must carry the image's meaning). Output capped at settings.summary_max_chars (the description IS the doc's summary; the phase-30 cap keeps it uniform). Stored: Document.summary = Document.content = description. Fail-soft: a failed/empty description → the doc is SKIPPED (no row, ImportSummary counts it in a new images_failed counter + a logger.warning with source/path) — an undescribed image is unsearchable noise; the sync continues (the importer's existing fail-soft convention).
• Serve + display (task 04): GET /api/documents/{doc_id}/image (new route in app/api/docs.py) — 404 for missing docs and non-image docs; serves image_path bytes with the correct Content-Type (ext → mime map: png/jpeg/webp/gif/bmp) — PUBLIC like the document content itself (this app's document content is already anonymous-readable; the image is part of that content). The document content endpoint (the one frontend/assets/document.js boots against) gains is_image: bool + image_url (the new route's path, absent for text docs) so document.html renders <img src> (max-width 100%, the theme's image treatment) with the summary/description text below it instead of the markdown content; the Sources page row for an image doc shows a small thumbnail (lazy-loaded, loading="lazy", aspect-ratio box) or the existing doc icon when the fetch is not yet possible offline — the thumbnail is a progressive enhancement (a fetch failure falls back to the icon).
• RAG display (task 05): the SSE sources frames (and any sources-list shape the chat bubble renders from) carry an OPTIONAL image_url on image docs (the retriever/agent know the Document row — add the field where SourceRef-shaped frames are built in app/api/chat.py/app/rag/retriever.py); the chat's sources block renders a compact inline <img> (capped height, the summary as caption/alt) for image docs — "shown in the chat nicely" (TODO L6). The agent's read tool on an image doc returns its description prefixed with a one-line marker (e.g. Image document — description generated from the image:) so the model knows what it is reading. alt text = the summary everywhere (WCAG).
• NOT touched (this phase): chat-side image submission (phase 123), the lite summary path for TEXT docs, archive unpacking (image members already land on disk — only the walk filter changes), and git/local sync scheduling.
• Locked assumptions: A3 — descriptions use the CHAT model (BOR_LLM_CHAT_MODEL, must be vision-capable); BOR_IMAGES defaults to false; generation failure → doc skipped + logged. A4 — "images uploaded as part of documents" = standalone image files arriving via direct upload / uploaded archives / source walks — NOT embedded-image extraction from PDFs/DOCX.

Tasks

1. 01_image_toggle.md — BOR_IMAGES / BOR_IMAGE_EXTENSIONS / BOR_IMAGE_DIR settings + .env.example + GET /api/config flag; off = byte-identical behavior.
2. 02_image_ingest.md — Document.is_image/image_path + migration; walk accepts image extensions when on; _index_file binary branch + persistent copy; prune guard when off.
3. 03_image_description.md — describe_image (chat-model vision), content = summary = description, fail-soft skip + counter.
4. 04_serve_and_display.md — GET /api/documents/{id}/image; document viewer + Sources page rendering.
5. 05_rag_display.md — image_url on chat source frames + inline image in the chat sources block + the agent read marker.
6. 06_image_tests.md — unit + integration + isolated E2E test_image_documents.py.

Testing & Quality

• Unit: tests/unit/test_image_documents.py (new, task 06) — settings parsing (toggle off by default, extensions frozenset, mime map), the _index_file image branch (bytes digest, copy to image_dir, is_image/image_path set, text content never read for an image), the prune guard (toggle off → image docs survive; toggle on → deleted image prunes), describe_image prompt shape (multimodal content list, chat model, cap) with a mock client, and the fail-soft skip path.
• Integration: tests/integration/test_docs_api.py (extend) — the image route (200 + correct Content-Type for a seeded image doc; 404 for text docs and missing ids); the content endpoint exposes is_image/image_url for image docs and omits them for text docs (byte-identical text-doc responses); import_sources end-to-end with images=True and a mock vision client (a fixture PNG → doc row with description content + is_summary chunk embedding; images=False → the file is ignored, pre-existing image doc survives prune).
• E2E: tests/e2e/test_image_documents.py (new, task 06) — isolated run per AGENTS.md §4, BOR_IMAGES=true for this suite's app instance: upload a small fixture PNG (via the existing upload endpoint's UI or page.request) → sync → the Sources page lists it (thumbnail or icon) → open the document viewer → the image renders with its description → ask a question the mock LLM grounds on the image doc → the chat's sources block shows the inline image.
• Coverage: >90% on app/ (validate.sh gate).

Completion Criteria

• With BOR_IMAGES=true: an uploaded standalone image (direct or in an archive) and an image file in a git/local source become documents whose content/summary is the vision description and whose ONLY embedded text is that description.
• With BOR_IMAGES=false (the default): every request, walk, and response is byte-identical to pre-phase; existing image docs (if any) survive a sync.
• The image renders in the document viewer and in the chat's sources block (inline, with alt text); a failed description skips the doc and logs — the sync completes.
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run ruff check . && uv run pyright clean.
• One --no-gpg-sign commit; phase dir moved to .agents/phases/complete/ by the pipeline gate.

Locked decisions

• A3 — image descriptions are generated by the CHAT model (BOR_LLM_CHAT_MODEL, vision-capable); BOR_IMAGES defaults to false; a failed/empty description skips the doc and logs (owner-confirmed 2026-09-24, roadmap confirmation).
• A4 — "images uploaded as part of documents" = standalone image files via direct upload / uploaded archives / source walks — no embedded-image extraction from PDFs/DOCX (owner-confirmed 2026-09-24).
• Prune guard (derived from A3/A4, same confirmation): images-off syncs never prune is_image docs — turning the toggle off must not destroy image documents.

Commit

git add app/ alembic/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): index standalone images as documents — described, embedded, and displayed via the vision model"