phase: 122_image_documents
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**
**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs
**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)
**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).
**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.
**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
+131
-6
@@ -1,4 +1,5 @@
|
||||
"""Document summarizer (phase 30, task 03).
|
||||
"""Document summarizer (phase 30, task 03) + image descriptions
|
||||
(phase 122, task 03).
|
||||
|
||||
Builds the ``SUMMARY_MODE`` prompt for one document, calls the aipi
|
||||
``lite`` model through the one-shot ``LLMClient.chat`` (phase 30,
|
||||
@@ -22,18 +23,33 @@ Quality contracts enforced here:
|
||||
summarizer re-asserts defensively and never hands the importer a
|
||||
pointer-only row).
|
||||
|
||||
The ``SUMMARY_MODE`` marker follows the ``DEFLECT_MODE`` convention:
|
||||
the deterministic E2E mock LLM keys on it in the system prompt
|
||||
(``tests/e2e/mock_llm.py`` — wired in task 06).
|
||||
Image descriptions (phase 122, LOCKED A3): :func:`describe_image` is
|
||||
this module's second one-shot generation path — a SINGLE CHAT-model
|
||||
(vision) call describing one image's bytes as a base64 data URL. The
|
||||
description becomes the image document's ``content`` AND ``summary``
|
||||
(it is the ONLY embedded text of the doc — the embedding model never
|
||||
sees pixels, and the ``lite`` summary model is deliberately NOT used:
|
||||
it is not assumed vision-capable). Fail-soft by contract: any client
|
||||
error, empty reply, or non-2xx yields ``None`` — the importer skips
|
||||
the doc, counts ``images_failed``, and the sync continues.
|
||||
|
||||
The ``SUMMARY_MODE`` / ``IMAGE_DESCRIPTION_MODE`` markers follow the
|
||||
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on
|
||||
them (``tests/e2e/mock_llm.py`` — the image branch is wired by task
|
||||
06's story suite).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Protocol
|
||||
import base64
|
||||
import logging
|
||||
from typing import Any, Protocol
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
from app.rag.llm import LLMError
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
|
||||
logger = logging.getLogger("app.summarizer")
|
||||
|
||||
#: System-prompt marker for summary generation — the E2E mock LLM keys on
|
||||
#: it (same convention as ``DEFLECT_MODE``, PLAN §6).
|
||||
SUMMARY_MODE = "SUMMARY_MODE"
|
||||
@@ -51,6 +67,55 @@ SUMMARY_INSTRUCTION = (
|
||||
#: Full system prompt: marker first (the mock's key), then the instruction.
|
||||
SYSTEM_PROMPT = f"{SUMMARY_MODE}: {SUMMARY_INSTRUCTION}"
|
||||
|
||||
#: Image-description marker (phase 122, task 03) — the deterministic
|
||||
#: E2E mock LLM keys on it (same convention as ``SUMMARY_MODE`` /
|
||||
#: ``DEFLECT_MODE``, PLAN §6; the story suite wires the mock's branch
|
||||
#: in task 06). It heads the text part of the multimodal describe
|
||||
#: message, so it rides the user message, not a system prompt.
|
||||
IMAGE_DESCRIPTION_MODE = "IMAGE_DESCRIPTION_MODE"
|
||||
|
||||
#: Locked instruction for the CHAT (vision) model (phase 122, LOCKED
|
||||
#: A3): the description is the ONLY retrievable text of the image
|
||||
#: document (the embedding model never sees pixels), so it must be a
|
||||
#: faithful, retrieval-oriented account that carries the image's full
|
||||
#: meaning — what is depicted, any visible text/labels/titles,
|
||||
#: diagram/table structure, salient details.
|
||||
DESCRIBE_INSTRUCTION = (
|
||||
"Describe this image faithfully, in plain text, for a search index. "
|
||||
"State what is depicted, transcribe any visible text, labels, or "
|
||||
"titles, describe the structure of any diagram, table, or layout, and "
|
||||
"call out the most salient details. Write 2-4 sentences of substance. "
|
||||
"Do not use markdown. Do not invent anything that is not visible in "
|
||||
"the image. Your description is the ONLY text that will ever be "
|
||||
"retrieved for this image — it must carry the image's full meaning."
|
||||
)
|
||||
|
||||
#: The full describe prompt: marker first (the mock's key), then the
|
||||
#: instruction — the single text part of the multimodal user message.
|
||||
DESCRIBE_PROMPT = f"{IMAGE_DESCRIPTION_MODE}: {DESCRIBE_INSTRUCTION}"
|
||||
|
||||
#: Extension → MIME type for the image family (phase 122). Dotted,
|
||||
#: lowercase keys — the ``image_extension_set`` shape. Task 03 uses it
|
||||
#: for the describe call's data-URL mime; task 04's serve route reuses
|
||||
#: it for the image bytes' ``Content-Type`` (one map, one truth). A
|
||||
#: ``BOR_IMAGE_EXTENSIONS`` token outside this map (a custom format)
|
||||
#: takes the :data:`IMAGE_FALLBACK_MIME` data-URL mime in the describe
|
||||
#: call — the vision endpoint may reject it, and the fail-soft skip
|
||||
#: (``images_failed``) is the honest outcome.
|
||||
IMAGE_MIMES: dict[str, str] = {
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".webp": "image/webp",
|
||||
".gif": "image/gif",
|
||||
".bmp": "image/bmp",
|
||||
}
|
||||
|
||||
#: The data-URL mime for an image extension :data:`IMAGE_MIMES` does not
|
||||
#: name (phase 122) — the best-effort generic, never a guess at a
|
||||
#: specific type.
|
||||
IMAGE_FALLBACK_MIME = "application/octet-stream"
|
||||
|
||||
|
||||
class SummaryLLM(Protocol):
|
||||
"""The one-shot chat surface the summarizer needs.
|
||||
@@ -58,12 +123,17 @@ class SummaryLLM(Protocol):
|
||||
:class:`app.rag.llm.LLMClient` satisfies it; unit tests pass a
|
||||
duck-typed fake (``chat`` + ``settings``) instead — same pattern as
|
||||
the importer's ``Embedder`` protocol.
|
||||
|
||||
``content`` may be a string (text calls — ``generate_summary``) or
|
||||
a list of OpenAI-compatible parts (phase 122 multimodal image
|
||||
descriptions — ``{type: "text", …}`` + ``{type: "image_url", …}``);
|
||||
the client passes message dicts through untouched.
|
||||
"""
|
||||
|
||||
settings: Settings
|
||||
|
||||
async def chat(
|
||||
self, messages: list[dict[str, str]], model: str | None = None
|
||||
self, messages: list[dict[str, Any]], model: str | None = None
|
||||
) -> str: ...
|
||||
|
||||
|
||||
@@ -121,3 +191,58 @@ async def generate_summary(
|
||||
"refusing to store a silent summary"
|
||||
)
|
||||
return f"{summary}\nSource: {source}/{path}"
|
||||
|
||||
|
||||
async def describe_image(
|
||||
llm: SummaryLLM,
|
||||
*,
|
||||
data: bytes,
|
||||
mime: str,
|
||||
settings: Settings | None = None,
|
||||
) -> str | None:
|
||||
"""One-shot CHAT-model (vision) description of one image (phase 122,
|
||||
LOCKED A3) — the text that becomes the image document's ``content``
|
||||
AND ``summary`` (the ONLY embedded text of the doc; the embedding
|
||||
model never sees pixels).
|
||||
|
||||
ONE chat-model call against ``settings.llm_chat_model`` (the vision
|
||||
model — the ``lite`` summary model is NOT assumed vision-capable)
|
||||
with the multimodal user message the OpenAI-compatible API expects:
|
||||
``[{type: "text", text: DESCRIBE_PROMPT}, {type: "image_url",
|
||||
image_url: {url: <data URL from *data* + *mime*>}}]`` — no system
|
||||
prompt, no tools, no app-level retries beyond the client's own
|
||||
(SDK-level + the house one-shot empty-content policy) — a
|
||||
description failure must not stall a sync.
|
||||
|
||||
Returns the stripped reply cut exactly at
|
||||
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap —
|
||||
the description IS the summary, so it keeps the same uniform
|
||||
ceiling). Returns ``None`` on any client error, empty reply, or
|
||||
non-2xx (the client raises :class:`LLMError` for all three classes)
|
||||
— the caller (the importer's ``_describe_or_skip`` seam) fails soft:
|
||||
the doc is skipped, counted in ``images_failed``, and the sync
|
||||
continues (LOCKED A3).
|
||||
"""
|
||||
data_url = f"data:{mime};base64,{base64.b64encode(data).decode('ascii')}"
|
||||
messages: list[dict[str, Any]] = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": DESCRIBE_PROMPT},
|
||||
{"type": "image_url", "image_url": {"url": data_url}},
|
||||
],
|
||||
}
|
||||
]
|
||||
model = llm.settings.llm_chat_model
|
||||
try:
|
||||
raw = await llm.chat(messages, model=model)
|
||||
except LLMError as e:
|
||||
logger.warning("image description failed (model=%s): %s", model, e)
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
# The client already rejects empty content; this is the
|
||||
# defensive re-assert (the duck-typed fakes may return it).
|
||||
return None
|
||||
limit = (settings or llm.settings).summary_max_chars
|
||||
return text[:limit]
|
||||
|
||||
Reference in New Issue
Block a user