phase: 122_image_documents
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**
**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs
**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)
**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).
**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.
**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
@@ -0,0 +1,28 @@
|
||||
# Task 03 — Image description: the vision model writes the only embedded text
|
||||
|
||||
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "the only embedded part of an image will be the summary generated by the model."
|
||||
|
||||
## Objective
|
||||
`describe_image` generates the image's description with the CHAT model (vision), the description becomes BOTH `Document.content` and `Document.summary` (so the chunk pipeline embeds exactly that text — and only that text), and a failed description fails soft (skip + count + log, sync continues).
|
||||
|
||||
## Work
|
||||
1. `app/rag/summarizer.py` — new `async def describe_image(llm, data: bytes, mime: str, settings=None) -> str | None` (the summarizer module owns model-text generation; follow the existing summary-call conventions — client, model, timeout, the `summary_max_chars` cap):
|
||||
- ONE chat-model call (`settings.llm_chat_model` — LOCKED A3; the lite summary model is not assumed vision-capable) with messages `[{role: "user", content: [{type: "text", text: <DESCRIBE_PROMPT>}, {type: "image_url", image_url: {url: f"data:{mime};base64,{b64}"}}]}]` — the multimodal content-list shape the OpenAI-compatible API expects.
|
||||
- `DESCRIBE_PROMPT` (a module constant, pinned by a unit test): a faithful, retrieval-oriented description — what is depicted, any visible text/labels/titles, diagram/table structure, salient details; 2–4 sentences of substance (the description is the ONLY retrievable text of the doc, so it must carry the image's meaning).
|
||||
- Return the stripped text capped at `settings.summary_max_chars` (the phase-30 cap — the description IS the summary); return `None` on any client error, empty response, or non-2xx (the caller fails soft). No retries beyond the SDK's own — a description failure must not stall a sync.
|
||||
2. `app/rag/importer.py` — wire the task-02 seam: the image branch's `content`/`summary` come from `describe_image` —
|
||||
- description `None` → **skip the doc entirely** (no row, no `image_dir` copy kept — delete the copy if it was made, or make the copy AFTER a successful description so a failure never leaves an orphan), `summary.images_failed += 1`, `logger.warning("import: image description failed source=%s path=%s", source, rel)` — the fail-soft skip (LOCKED A3).
|
||||
- success → `content = description`, then the existing `_store_summary` path (L582) runs with the description as the summary (the `is_summary` position −1 chunk mirrors it — phase-30 behavior, unchanged), and the normal content chunks embed the description (for a short description that is typically ONE content chunk + the summary chunk — the chunker's existing behavior, no special case).
|
||||
- the phase-118 backfill branch (unchanged image doc, `summary is None`) calls the SAME path — a description failure there keeps the doc as-is and logs (no row mutation).
|
||||
3. `app/rag/llm.py` — no new client: `describe_image` reuses the existing `llm.chat`-equivalent client the summarizer already uses for text summaries (verify the exact client method name in `app/rag/summarizer.py` and match it — the multimodal payload is a plain `list[dict]` message, so no client change is needed; IF the existing client hard-codes text-only `content: str` typing, extend its signature to accept `content: str | list` — pyright-clean).
|
||||
4. ASSUMPTION (A3 re-stated): the CHAT model describes; if the owner's chat model lacks vision, `describe_image` returns `None` (the SDK errors) and every image doc is skipped + logged — honest, visible failure (the `images_failed` counter in the sync log is the signal).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_documents.py` (task 06) — with a MOCK client: the prompt shape (text part + `image_url` data-URL part, correct model), the cap is applied, whitespace stripped; `None` on mock error / empty string / client exception; the importer's skip path (no row, `images_failed == 1`, warning logged, no orphan copy) and the success path (content == summary == description, `is_summary` chunk present, embedding called with the description text — the ONLY text embedded).
|
||||
- Integration: the mock-vision `import_sources` end-to-end (task 06).
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A fixture PNG through the mock vision client yields a doc whose `content` == `summary` == the description, with its embedding(s) derived from that text only.
|
||||
- [ ] A failing mock client skips the doc, bumps `images_failed`, logs, and the sync completes with the other docs indexed.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
Reference in New Issue
Block a user