phase: 122_image_documents
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**
**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs
**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)
**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).
**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.
**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
@@ -0,0 +1,54 @@
|
||||
# Phase 122 — Image documents: standalone images become first-class, retrievable documents
|
||||
|
||||
**Source:** `TODO.md` L6 — "Need to support images. Images uploaded as part of documents or as standalone images should be read, summarized, and retrieved like any other document. Note that the embedding model won't support images, so the only embedded part of an image will be the summary generated by the model. The user should be able to turn on and off image support in their .env depending on whether their model supports it. Images retrieved by the RAG should be shown in the chat nicely and users should be able to submit images as part of their question in brain of reese."
|
||||
**Story:** n/a (feature request; extends the phase-28/30/38 import pipeline, phase-90 no-scan uploads, and the RAG/chat assets).
|
||||
**Context:** `app/config.py` — `Settings` (`BOR_` prefix; `upload_dir` L361, `sources_dir` L355, `upload_max_mb` L370 — raw-string/`expanduser` house convention). `app/rag/importer.py` — `iter_importable_files` (L235, extension filter via `llm.settings.import_extension_set` + `match_extension` L217), `import_sources` (L279, `prune` L283), `_index_file` (L435 — `read_text` L453, sha256 over text, Document upsert, the unchanged/hash path with the phase-118 summary backfill), `_store_summary` (L582 — lite-model summary + the position −1 `is_summary` chunk), `_prune` (L649 — deletes docs of the imported sources not in `seen`). `app/models.py` — `Document` (L102: `content`, `content_hash`, `summary` L140, `created_at`/`created_at_manual`), `Chunk` (L148: `is_summary` L161). `app/rag/summarizer.py` + `app/rag/llm.py` — the lite summary path + the chat-model client (`Settings` model names, `check_models`). `app/rag/archive_upload.py` — archive unpack into `upload_dir` (image members land on disk today, then get filtered out by the extension walk). `app/api/git_sources.py:432` — the upload endpoint. `app/api/docs.py` — the document content endpoint (document viewer). `app/rag/retriever.py` / `app/rag/agent.py` (read tool, message build L1446–1448) / `app/api/chat.py` — RAG + the SSE sources frames. `app/api/config.py:30` — `GET /api/config` public flags dict (task 01 of phase 123 extends it). Frontend: `frontend/assets/app.js` (chat source chips), `frontend/assets/document.js` + `frontend/document.html` (`#doc-content` L144), `frontend/assets/sources.js` (Sources page). `alembic/` — migrations.
|
||||
|
||||
## Objective
|
||||
With `BOR_IMAGES=true`, a standalone image file — arriving as a direct upload, inside an uploaded archive, or as a file in a git/local source — becomes a first-class document: the vision model (the chat model) describes it, the description is the document's content AND summary, only the description is embedded (the embedding model never sees pixels), the image bytes persist and are served, and the image shows up in the Sources page, the document viewer, and the chat — with retrieved image docs rendered inline in the answer's sources.
|
||||
|
||||
## Dependencies
|
||||
- `121_git_source_tokens` (todo) — pipeline predecessor (execution order) only; no code dependency.
|
||||
- Code dependencies (all complete): phase 30 summary pipeline (`_store_summary`, `is_summary` chunk), phase 90 no-scan upload, phase 89/105 per-source walk options, the RAG agent + SSE sources frames.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Toggle (task 01, LOCKED A3):** three new `Settings` fields — `images: bool = False` (`BOR_IMAGES`, `0`/`false` off — the phase-67 `llm_retries` bool style), `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` (`BOR_IMAGE_EXTENSIONS`, comma-separated, lowercased into a frozenset by the importer — the `import_extension_set` property precedent), and `image_dir: str = "~/bor-sources/images"` (`BOR_IMAGE_DIR`, raw-string/`expanduser` convention — the persistent home for image bytes, deliberately separate from `sources_dir`/`upload_dir`). `.env.example` gets all three with a comment: **off by default — enable only when your chat model supports vision, because image descriptions are generated by the chat model.** `GET /api/config` (task 01) gains `images: bool` so the UI can gate affordances (consumed by phase 123; the Sources page can show an "images off" hint — optional, not required).
|
||||
- **Why the bytes are copied (task 02):** upload dirs are REPLACED on every upload (`archive_upload.swap_in`), git checkouts are re-cloned, and local dirs are user-edited — a served image must outlive its source file. The importer copies each ingested image to `image_dir/<doc-uuid>.<ext>` (created on demand) and stores that path in `Document.image_path`. The copy happens ONLY when the doc is new or its hash changes; a replaced image deletes the stale copy; pruned docs delete their copy.
|
||||
- **Storage (task 02):** `Document.is_image: bool` (server default `false` — every pre-phase-122 row is a text doc) + `Document.image_path: str | None` (NULL for text docs). One migration, one downgrade.
|
||||
- **Ingest (task 02):** the walk: `iter_importable_files`/`import_sources` accept the image frozenset IN ADDITION to `import_extension_set`, ONLY when `settings.images` is true (images are never user-configurable via `BOR_IMPORT_EXTENSIONS` — the toggle is the single knob, LOCKED A3/A4). `_index_file` branches on image extension: read BYTES (not `read_text`), sha256 over the bytes (the digest rule is unchanged — content identity), copy to `image_dir`, set `is_image` + `image_path`, and `content` = the vision description (task 03). The normal chunk pipeline then embeds the content (= the description) — that is exactly the TODO's "the only embedded part of an image will be the summary generated by the model"; the phase-30 summary chunk (`is_summary`, position −1) mirrors `Document.summary`, which equals the description too. Title = the file stem (the non-markdown rule at L505–509). The unchanged/hash path works unmodified (byte digest → "unchanged" skips re-describing; the phase-118 backfill path re-describes a NULL-summary image doc on its next sync — same fail-soft). **Prune guard:** `_prune` (L649) must NOT delete `is_image` docs while `settings.images` is false (an image doc is invisible to an images-off walk, not a deleted file — otherwise turning the toggle off and syncing would silently destroy the image documents). Toggle ON → normal prune semantics (a deleted image file prunes its doc + copy).
|
||||
- **Description (task 03, LOCKED A3):** `describe_image` in `app/rag/summarizer.py` (one function, the summarizer module owns model-text generation): a SINGLE chat-model call (`Settings.llm_chat_model` — the vision model; the lite summary model is NOT assumed vision-capable, LOCKED A3) with a multimodal user message `[{type: "text", text: <fixed describe prompt>}, {type: "image_url", image_url: {url: <data URL from the bytes + mime>}}]`; the prompt asks for a faithful, retrieval-oriented description (what is shown, any text/labels/diagram content, salient details — the description is the ONLY thing retrievable, so it must carry the image's meaning). Output capped at `settings.summary_max_chars` (the description IS the doc's summary; the phase-30 cap keeps it uniform). Stored: `Document.summary = Document.content = description`. **Fail-soft:** a failed/empty description → the doc is SKIPPED (no row, `ImportSummary` counts it in a new `images_failed` counter + a `logger.warning` with source/path) — an undescribed image is unsearchable noise; the sync continues (the importer's existing fail-soft convention).
|
||||
- **Serve + display (task 04):** `GET /api/documents/{doc_id}/image` (new route in `app/api/docs.py`) — 404 for missing docs and non-image docs; serves `image_path` bytes with the correct `Content-Type` (ext → mime map: png/jpeg/webp/gif/bmp) — PUBLIC like the document content itself (this app's document content is already anonymous-readable; the image is part of that content). The document content endpoint (the one `frontend/assets/document.js` boots against) gains `is_image: bool` + `image_url` (the new route's path, absent for text docs) so `document.html` renders `<img src>` (max-width 100%, the theme's image treatment) with the summary/description text below it instead of the markdown content; the Sources page row for an image doc shows a small thumbnail (lazy-loaded, `loading="lazy"`, aspect-ratio box) or the existing doc icon when the fetch is not yet possible offline — the thumbnail is a progressive enhancement (a fetch failure falls back to the icon).
|
||||
- **RAG display (task 05):** the SSE sources frames (and any sources-list shape the chat bubble renders from) carry an OPTIONAL `image_url` on image docs (the retriever/agent know the `Document` row — add the field where `SourceRef`-shaped frames are built in `app/api/chat.py`/`app/rag/retriever.py`); the chat's sources block renders a compact inline `<img>` (capped height, the summary as caption/alt) for image docs — "shown in the chat nicely" (TODO L6). The agent's `read` tool on an image doc returns its description prefixed with a one-line marker (e.g. `Image document — description generated from the image:`) so the model knows what it is reading. `alt` text = the summary everywhere (WCAG).
|
||||
- **NOT touched (this phase):** chat-side image submission (phase 123), the lite summary path for TEXT docs, archive unpacking (image members already land on disk — only the walk filter changes), and git/local sync scheduling.
|
||||
- **Locked assumptions:** **A3** — descriptions use the CHAT model (`BOR_LLM_CHAT_MODEL`, must be vision-capable); `BOR_IMAGES` defaults to **false**; generation failure → doc skipped + logged. **A4** — "images uploaded as part of documents" = standalone image files arriving via direct upload / uploaded archives / source walks — NOT embedded-image extraction from PDFs/DOCX.
|
||||
|
||||
## Tasks
|
||||
1. `01_image_toggle.md` — `BOR_IMAGES` / `BOR_IMAGE_EXTENSIONS` / `BOR_IMAGE_DIR` settings + `.env.example` + `GET /api/config` flag; off = byte-identical behavior.
|
||||
2. `02_image_ingest.md` — `Document.is_image`/`image_path` + migration; walk accepts image extensions when on; `_index_file` binary branch + persistent copy; prune guard when off.
|
||||
3. `03_image_description.md` — `describe_image` (chat-model vision), content = summary = description, fail-soft skip + counter.
|
||||
4. `04_serve_and_display.md` — `GET /api/documents/{id}/image`; document viewer + Sources page rendering.
|
||||
5. `05_rag_display.md` — `image_url` on chat source frames + inline image in the chat sources block + the agent `read` marker.
|
||||
6. `06_image_tests.md` — unit + integration + isolated E2E `test_image_documents.py`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_documents.py` (new, task 06) — settings parsing (toggle off by default, extensions frozenset, mime map), the `_index_file` image branch (bytes digest, copy to `image_dir`, `is_image`/`image_path` set, text `content` never read for an image), the prune guard (toggle off → image docs survive; toggle on → deleted image prunes), `describe_image` prompt shape (multimodal content list, chat model, cap) with a mock client, and the fail-soft skip path.
|
||||
- Integration: `tests/integration/test_docs_api.py` (extend) — the image route (200 + correct Content-Type for a seeded image doc; 404 for text docs and missing ids); the content endpoint exposes `is_image`/`image_url` for image docs and omits them for text docs (byte-identical text-doc responses); `import_sources` end-to-end with `images=True` and a mock vision client (a fixture PNG → doc row with description content + `is_summary` chunk embedding; `images=False` → the file is ignored, pre-existing image doc survives prune).
|
||||
- E2E: `tests/e2e/test_image_documents.py` (new, task 06) — isolated run per AGENTS.md §4, `BOR_IMAGES=true` for this suite's app instance: upload a small fixture PNG (via the existing upload endpoint's UI or `page.request`) → sync → the Sources page lists it (thumbnail or icon) → open the document viewer → the image renders with its description → ask a question the mock LLM grounds on the image doc → the chat's sources block shows the inline image.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] With `BOR_IMAGES=true`: an uploaded standalone image (direct or in an archive) and an image file in a git/local source become documents whose content/summary is the vision description and whose ONLY embedded text is that description.
|
||||
- [ ] With `BOR_IMAGES=false` (the default): every request, walk, and response is byte-identical to pre-phase; existing image docs (if any) survive a sync.
|
||||
- [ ] The image renders in the document viewer and in the chat's sources block (inline, with alt text); a failed description skips the doc and logs — the sync completes.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A3 — image descriptions are generated by the CHAT model (`BOR_LLM_CHAT_MODEL`, vision-capable); `BOR_IMAGES` defaults to false; a failed/empty description skips the doc and logs (owner-confirmed 2026-09-24, roadmap confirmation).**
|
||||
- **A4 — "images uploaded as part of documents" = standalone image files via direct upload / uploaded archives / source walks — no embedded-image extraction from PDFs/DOCX (owner-confirmed 2026-09-24).**
|
||||
- **Prune guard (derived from A3/A4, same confirmation):** images-off syncs never prune `is_image` docs — turning the toggle off must not destroy image documents.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ alembic/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): index standalone images as documents — described, embedded, and displayed via the vision model"
|
||||
```
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 01 — Image toggle: BOR_IMAGES + extensions + dir, off by default
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "The user should be able to turn on and off image support in their .env depending on whether their model supports it."
|
||||
|
||||
## Objective
|
||||
The single env knob for image support exists and is surfaced — `BOR_IMAGES` (default **false**), `BOR_IMAGE_EXTENSIONS`, `BOR_IMAGE_DIR` — with `GET /api/config` exposing the flag for UI gating. Toggle off = byte-identical behavior to pre-phase.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — three new `Settings` fields (house docstring style, the `upload_dir`/`llm_retries` precedents):
|
||||
- `images: bool = False` — `BOR_IMAGES`, `0`/`false` off (LOCKED A3 default). Docstring: master switch for image-document indexing (phase 122) — off by default, enable only when the chat model supports vision (descriptions come from it).
|
||||
- `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` — `BOR_IMAGE_EXTENSIONS`, comma-separated, case-insensitive; a property/parse into a lowercased-dotted frozenset (the `import_extension_set` precedent) — the image set is SEPARATE from `import_extension_set` (images are never user-added via `BOR_IMPORT_EXTENSIONS`).
|
||||
- `image_dir: str = "~/bor-sources/images"` — `BOR_IMAGE_DIR`, raw string, `Path.expanduser()` applied by the importer (the `sources_dir`/`upload_dir` convention) — the persistent home for image bytes (uploads are replaced, checkouts re-cloned — the copy must outlive the source file).
|
||||
2. `.env.example` — the three entries with the comment block: off by default + the vision-model dependency note (LOCKED A3).
|
||||
3. `app/api/config.py:30` — the `app_config` dict gains `"images": settings.images` (the dict is `str | bool`-valued — bools already allowed). Extend the docstring: consumed by the chat composer (phase 123) to show/hide the attach control, optionally by the Sources page.
|
||||
4. ASSUMPTION: `GET /api/config` is already anonymous-readable (the UI gates on it pre-login in phase 123 — no auth change here).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_documents.py` (task 06 finalizes) — defaults (`images` False, extensions frozenset `{".png", …}` with the dotted form the matchers expect, dir default), env overrides, the frozenset parse is case-insensitive and trims spaces.
|
||||
- Integration: the existing `GET /api/config` test asserts the new `images` key (default false in the test env).
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `Settings()` with no env: `images is False`, the extension set is the six defaults, `image_dir` is the default path.
|
||||
- [ ] `GET /api/config` returns `images: false` in the default test env (byte-check the other keys unchanged).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 02 — Image ingest: model fields, walk filter, binary index path, prune guard
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Images uploaded as part of documents or as standalone images should be read, summarized, and retrieved like any other document."
|
||||
|
||||
## Objective
|
||||
When `BOR_IMAGES=true`, standalone image files in ANY ingest path (direct upload, uploaded archive, git/local source walk) become `Document` rows — bytes persisted to `image_dir`, `is_image`/`image_path` set, `content` = the vision description (task 03) — and an images-off sync never prunes existing image docs.
|
||||
|
||||
## Work
|
||||
1. `app/models.py` — `Document` (L102): add, with house docstrings (the `summary` L140 / `created_at_manual` precedent):
|
||||
- `is_image: Mapped[bool] = mapped_column(Boolean, default=False, server_default=text("false"), nullable=False)` — True iff the doc's content is a vision description of an image (phase 122); the image bytes live at `image_path`.
|
||||
- `image_path: Mapped[str | None] = mapped_column(Text, default=None)` — absolute path of the persistent copy in `settings.image_dir`; NULL for text docs.
|
||||
2. `alembic/versions/` — new revision: both columns (`is_image` NOT NULL server_default 'false'; `image_path` nullable) + downgrade.
|
||||
3. `app/rag/importer.py`:
|
||||
- `iter_importable_files` (L235) / the walk in `import_sources` (L279): when `llm.settings.images`, accept a file iff its extension matches `import_extension_set` OR the image frozenset (task 01) — pass the image set in (the function takes explicit extension sets; the image set is NOT merged into `import_extension_set`).
|
||||
- `_index_file` (L435): image branch FIRST (before the `read_text` at L453) — if the path's extension is in the image set: `data = full_path.read_bytes()`, `digest = sha256(data)`, and on new/changed: copy `data` to `image_dir/<doc-id or uuid4>.<ext>` (dir created with `mkdir(parents=True, exist_ok=True)`), set `is_image=True` + `image_path` on the `Document` row, `content` = the description (task 03's `describe_image` — this task wires the call; the function lands in task 03, so for THIS task store `content = ""` placeholder ONLY if task 03 is not yet merged — the phases run task-ordered, so in practice task 03's function exists; wire it directly and let task 03 implement it. If implementing strictly per task: this task stores `content` via a `_describe_or_skip` hook that task 03 fills — keep the seam single and commented).
|
||||
- A CHANGED image (hash differs) deletes the stale `image_path` copy before replacing it.
|
||||
- The unchanged/hash path (L470+) works unmodified for images (byte digest); the phase-118 summary-backfill branch (L480) re-describes an image doc whose `summary` is NULL on the next sync (same fail-soft).
|
||||
- `_prune` (L649): the prune guard (LOCKED derived decision) — when `settings.images` is FALSE, skip every `is_image` doc (invisible to the walk ≠ deleted); toggle TRUE → normal prune + delete the `image_path` copy of each pruned image doc (also on the normal prune path when the file is gone).
|
||||
- `ImportSummary` (L100): new `images_failed: int = 0` counter + its slot in `format_counts`/`log` (L135–151) — task 03 increments it; add it now so the log shape is stable.
|
||||
4. ASSUMPTION (A4 re-stated): only standalone image FILES are ingested — no archive-of-documents extraction, no PDF/DOCX embedded-image pulls (the archive unpacker already places image members on disk; the walk now just accepts them).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_documents.py` (task 06) — the walk accepts `.png` only when `images=True` (off → ignored, the byte-identical default), the binary branch (digest over bytes, copy made, fields set, `read_text` never called for an image), the changed-image stale-copy delete, the prune guard (off → image doc survives; on + file gone → pruned + copy deleted), `images_failed` in the log line.
|
||||
- Integration: `tests/integration/test_docs_api.py` (task 06) — the `import_sources` end-to-end cases.
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `alembic upgrade head` applies; a seeded image walk with `images=True` creates the doc row + `image_dir` copy; `images=False` ignores the file entirely.
|
||||
- [ ] A sync with `images=False` leaves a pre-existing image doc untouched (prune guard).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 03 — Image description: the vision model writes the only embedded text
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "the only embedded part of an image will be the summary generated by the model."
|
||||
|
||||
## Objective
|
||||
`describe_image` generates the image's description with the CHAT model (vision), the description becomes BOTH `Document.content` and `Document.summary` (so the chunk pipeline embeds exactly that text — and only that text), and a failed description fails soft (skip + count + log, sync continues).
|
||||
|
||||
## Work
|
||||
1. `app/rag/summarizer.py` — new `async def describe_image(llm, data: bytes, mime: str, settings=None) -> str | None` (the summarizer module owns model-text generation; follow the existing summary-call conventions — client, model, timeout, the `summary_max_chars` cap):
|
||||
- ONE chat-model call (`settings.llm_chat_model` — LOCKED A3; the lite summary model is not assumed vision-capable) with messages `[{role: "user", content: [{type: "text", text: <DESCRIBE_PROMPT>}, {type: "image_url", image_url: {url: f"data:{mime};base64,{b64}"}}]}]` — the multimodal content-list shape the OpenAI-compatible API expects.
|
||||
- `DESCRIBE_PROMPT` (a module constant, pinned by a unit test): a faithful, retrieval-oriented description — what is depicted, any visible text/labels/titles, diagram/table structure, salient details; 2–4 sentences of substance (the description is the ONLY retrievable text of the doc, so it must carry the image's meaning).
|
||||
- Return the stripped text capped at `settings.summary_max_chars` (the phase-30 cap — the description IS the summary); return `None` on any client error, empty response, or non-2xx (the caller fails soft). No retries beyond the SDK's own — a description failure must not stall a sync.
|
||||
2. `app/rag/importer.py` — wire the task-02 seam: the image branch's `content`/`summary` come from `describe_image` —
|
||||
- description `None` → **skip the doc entirely** (no row, no `image_dir` copy kept — delete the copy if it was made, or make the copy AFTER a successful description so a failure never leaves an orphan), `summary.images_failed += 1`, `logger.warning("import: image description failed source=%s path=%s", source, rel)` — the fail-soft skip (LOCKED A3).
|
||||
- success → `content = description`, then the existing `_store_summary` path (L582) runs with the description as the summary (the `is_summary` position −1 chunk mirrors it — phase-30 behavior, unchanged), and the normal content chunks embed the description (for a short description that is typically ONE content chunk + the summary chunk — the chunker's existing behavior, no special case).
|
||||
- the phase-118 backfill branch (unchanged image doc, `summary is None`) calls the SAME path — a description failure there keeps the doc as-is and logs (no row mutation).
|
||||
3. `app/rag/llm.py` — no new client: `describe_image` reuses the existing `llm.chat`-equivalent client the summarizer already uses for text summaries (verify the exact client method name in `app/rag/summarizer.py` and match it — the multimodal payload is a plain `list[dict]` message, so no client change is needed; IF the existing client hard-codes text-only `content: str` typing, extend its signature to accept `content: str | list` — pyright-clean).
|
||||
4. ASSUMPTION (A3 re-stated): the CHAT model describes; if the owner's chat model lacks vision, `describe_image` returns `None` (the SDK errors) and every image doc is skipped + logged — honest, visible failure (the `images_failed` counter in the sync log is the signal).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_documents.py` (task 06) — with a MOCK client: the prompt shape (text part + `image_url` data-URL part, correct model), the cap is applied, whitespace stripped; `None` on mock error / empty string / client exception; the importer's skip path (no row, `images_failed == 1`, warning logged, no orphan copy) and the success path (content == summary == description, `is_summary` chunk present, embedding called with the description text — the ONLY text embedded).
|
||||
- Integration: the mock-vision `import_sources` end-to-end (task 06).
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A fixture PNG through the mock vision client yields a doc whose `content` == `summary` == the description, with its embedding(s) derived from that text only.
|
||||
- [ ] A failing mock client skips the doc, bumps `images_failed`, logs, and the sync completes with the other docs indexed.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — Serve the image + render it in the document viewer and Sources page
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Images … should be read, summarized, and retrieved like any other document."
|
||||
|
||||
## Objective
|
||||
Image bytes are served through a dedicated document route, the document viewer renders the image with its description, and the Sources page shows an image affordance — image docs read like first-class documents in every existing surface.
|
||||
|
||||
## Work
|
||||
1. `app/api/docs.py` — new route `GET /api/documents/{doc_id}/image`:
|
||||
- 404 (the router's existing "unknown document" shape) for a missing doc and for a doc with `is_image` false / `image_path` NULL;
|
||||
- 404 if the file is missing on disk (defensive — the row exists but the copy was lost);
|
||||
- otherwise `FileResponse` (or a `Response` with the bytes) with `Content-Type` from an ext→mime map (`png`→`image/png`, `jpg`/`jpeg`→`image/jpeg`, `webp`→`image/webp`, `gif`→`image/gif`, `bmp`→`image/bmp` — the map lives in `app/rag/importer.py` or a small shared spot the unit tests can import; default `application/octet-stream` for an unexpected ext) and `Cache-Control: private, max-age=3600` (the image bytes are content-hashed — long enough, bustable by re-upload).
|
||||
- PUBLIC, like the document content endpoint (this app serves document content to anonymous visitors — the image is part of that content).
|
||||
- The document CONTENT endpoint the viewer boots against (same module): response gains `is_image: bool` (always present) + `image_url` (the `/api/documents/{id}/image` path — ABSENT for text docs, the `_drop_absent_share_url` omission precedent; never `null`). Text-doc responses gain only `is_image: false` — one new key, documented in the response schema's docstring.
|
||||
2. `frontend/assets/document.js` + `frontend/document.html`:
|
||||
- boot reads `is_image`; when true, `#doc-content` renders `<img src="{image_url}" alt="{summary}">` (block, `max-width: 100%`, the theme's surface treatment) with the description/summary text in the normal content slot below it (the document's readable content IS the description — no markdown render of a non-markdown string is needed; render it as the existing plain-content path).
|
||||
- an `<img>` error fallback: on `onerror` the image area shows a small "image unavailable" note (the 404-on-missing-file case) — the page still shows the description.
|
||||
3. `frontend/assets/sources.js` — the Sources page row for an image doc: a small thumbnail (48px box, `object-fit: cover`, `loading="lazy"`, `alt = summary`) where the doc icon sits; the thumbnail is a PROGRESSIVE enhancement — a failed fetch (or the row rendered before the fetch resolves) falls back to the existing icon (no layout shift beyond the fixed box). The doc title/path columns are unchanged.
|
||||
4. `frontend/assets/styles.css` — the viewer image block + the Sources thumbnail box (theme tokens; WCAG: alt text everywhere, no contrast concerns for decorative images).
|
||||
5. ASSUMPTION: the thumbnail uses the SAME full-size route (no separate thumb route) — a KB-scale image set makes a thumb pipeline unjustified; lazy loading keeps the Sources page fast.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: `tests/integration/test_docs_api.py` (task 06) — the image route (200 + exact `Content-Type` per ext for a seeded doc; 404 for a text doc; 404 for a missing id; 404 for a row whose file is deleted); the content endpoint: `is_image` present in ALL responses, `image_url` absent for text docs and present for image docs.
|
||||
- Unit: `tests/unit/test_image_documents.py` (task 06) — the ext→mime map (all six + the octet-stream default); house-style source assertions: the viewer renders the `img` from `image_url` with `alt = summary`, the sources row falls back to the icon on image error, no `null` in the text-doc content response.
|
||||
- E2E: `test_image_documents.py` scenarios (task 06) cover viewer + Sources rendering.
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/documents/{id}/image` serves the exact uploaded bytes with the right Content-Type; text docs 404.
|
||||
- [ ] The document viewer shows the image + its description; the Sources page shows the thumbnail (or the icon fallback).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Task 05 — RAG display: image docs in the chat sources + the agent read marker
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Images retrieved by the RAG should be shown in the chat nicely."
|
||||
|
||||
## Objective
|
||||
When a retrieved/agent-read document is an image, the chat shows it: the sources block renders a compact inline image with its summary as caption/alt, and the agent's `read` tool tells the model it is reading a generated image description.
|
||||
|
||||
## Work
|
||||
1. `app/rag/retriever.py` / `app/api/chat.py` — the sources frames the chat bubble renders (the SSE `sources`/related-doc frames and the agent-sourced doc list): add an OPTIONAL `image_url` field to the per-doc ref shape — populated (the `/api/documents/{id}/image` path) iff the doc row has `is_image`, absent otherwise (the omission rule — text-doc frames stay byte-identical). The retriever already has the `Document` row; the agent's doc refs (the read-tool results / source list) do too — set it at the frame-build sites (grep for the source-ref construction in both modules; one shared helper `source_ref_with_image(doc, …)` keeps the two sites in lockstep).
|
||||
2. `frontend/assets/app.js` — the chat's sources block renderer: when a source ref carries `image_url`, render a compact inline `<img>` (max-height ~96px, `object-fit: contain`, the theme's surface, `alt` + visible caption = the doc summary — the "shown nicely" requirement) in place of / beside the existing doc chip text (keep the title + the existing chip affordance — the image is additive, not a replacement). A failed image load collapses to the plain chip (never a broken-image icon).
|
||||
3. `app/rag/agent.py` — the `read` tool's result for an image doc: prefix the description with the marker line `Image document — the text below is a description generated from the image:` (a module constant) so the model reasons about what it is reading; non-image docs' results are byte-identical.
|
||||
4. ASSUMPTION: the chat QUESTION side (users submitting images) is phase 123 — this task only covers RETRIEVED images in the answer's sources.
|
||||
5. ASSUMPTION: the sources-frame `image_url` is the only new frame field — no doc-id leak beyond what the frame already carries (the path encodes the doc id, same as the content endpoint).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: `tests/integration/test_chat_api.py` (extend, task 06) — a mocked grounded answer that includes an image doc in its sources → the SSE frame carries `image_url` for that ref only; a text-only grounding has NO `image_url` key anywhere (byte check).
|
||||
- Unit: `tests/unit/test_image_documents.py` (task 06) — the frame-helper (present/absent), the agent marker (image vs non-image result), house-style source assertions: the sources renderer reads `image_url`, sets `alt`, and falls back on image error.
|
||||
- E2E: `test_image_documents.py` scenario (task 06) — ask a question the mock LLM grounds on the fixture image doc → the chat sources block shows the inline image with its caption.
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A chat answer grounded on an image doc shows the inline image + caption in its sources block; text-doc answers render byte-identically to before.
|
||||
- [ ] The agent `read` result for an image doc carries the marker; the model sees the description, not raw bytes.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 06 — Image tests: unit + integration + isolated E2E
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Need to support images …"
|
||||
|
||||
## Objective
|
||||
Pin the whole image-document contract: the off-by-default byte-identity, the ingest/description/serve pipeline, the RAG display, and the end-to-end user path (upload → Sources → viewer → chat) as an isolated Playwright suite.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_image_documents.py` (new) — consolidates the per-task unit cases (the tasks ship code; this task ships the full pin):
|
||||
- settings: defaults (`images` False, six-extension frozenset, dir default), env overrides, case-insensitive parse (task 01);
|
||||
- the walk: image accepted iff `images=True`; off → the file is ignored (the default byte-identity);
|
||||
- `_index_file` image branch: digest over BYTES, copy to `image_dir`, `is_image`/`image_path` set, changed-image stale-copy delete, prune guard (off → survives; on + gone → pruned + copy deleted), `images_failed` in the log line (task 02);
|
||||
- `describe_image`: mock-client prompt shape (multimodal parts, chat model), cap, `None` on error/empty, the importer skip path (no row, no orphan, counter, warning) and the success path (content == summary == description; embedding called with the description only) (task 03);
|
||||
- the ext→mime map (task 04);
|
||||
- the source-frame `image_url` helper (present/absent) + the agent `read` marker + house-style frontend assertions (viewer `img` + alt + fallback; sources thumbnail fallback; chat sources inline image + alt) (task 05).
|
||||
2. `tests/integration/test_docs_api.py` (extend, task 04's cases) — the image route (200 + Content-Type per ext; 404 text doc / missing id / missing file), the content endpoint's `is_image`/`image_url` omission rules; `tests/integration/test_chat_api.py` (extend, task 05's case) — the SSE `image_url` frame; `tests/integration/` (new file `test_image_import.py` or the existing import test file — follow whichever exists) — `import_sources` end-to-end: `images=True` + mock vision → the fixture PNG becomes a doc (description content, `is_summary` chunk, one content chunk); `images=False` → ignored + a pre-seeded image doc survives prune; a failing mock → `images_failed == 1`, no row, other docs indexed.
|
||||
- Fixtures: a tiny valid PNG (a few bytes, generated in-test or a committed fixture under `tests/` — check the existing fixture conventions), a mock vision client (the existing mock-LLM test patterns in `tests/`).
|
||||
3. `tests/e2e/test_image_documents.py` (new — isolated run per AGENTS.md §4: `uv run pytest tests/e2e/test_image_documents.py -v --no-cov`). The suite's app instance runs with `BOR_IMAGES=true` (env override in the E2E fixture — the `conftest.py` pattern for per-suite app env):
|
||||
- upload a fixture PNG (the Sources-page upload flow or `page.request` against the upload endpoint, then trigger the sync through the UI as the Sources page does);
|
||||
- the Sources page lists the image doc (thumbnail or icon fallback);
|
||||
- open the document viewer → the image renders + the description text below it;
|
||||
- ask a question the mock LLM grounds on the image doc (the existing mock-LLM grounding pattern) → the chat's sources block shows the inline image with its caption;
|
||||
- negative: with the DEFAULT env (`BOR_IMAGES` unset/false), the same upload produces NO image doc (the default-off contract).
|
||||
4. Run the full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%), the isolated E2E file, `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Testing & Quality
|
||||
- This task IS the phase's test suite (see Work).
|
||||
- Coverage: **>90%** on `app/` — the phase's `app/` surface (config, importer, summarizer, docs API, chat frames, agent marker) is fully exercised.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] All test artifacts exist and pass; the isolated E2E file passes standalone.
|
||||
- [ ] The default-off byte-identity is asserted (unit + integration + the E2E negative case).
|
||||
- [ ] `uv run pytest --cov=app` TOTAL >90%; lint + types clean.
|
||||
Reference in New Issue
Block a user