phase: 122_image_documents
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Failing after 13s

**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**

**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs

**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)

**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).

**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.

**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
2026-09-25 01:54:23 -04:00
parent 0f77e9a876
commit a19d78d284
63 changed files with 5484 additions and 111 deletions
+131 -6
View File
@@ -1,4 +1,5 @@
"""Document summarizer (phase 30, task 03).
"""Document summarizer (phase 30, task 03) + image descriptions
(phase 122, task 03).
Builds the ``SUMMARY_MODE`` prompt for one document, calls the aipi
``lite`` model through the one-shot ``LLMClient.chat`` (phase 30,
@@ -22,18 +23,33 @@ Quality contracts enforced here:
summarizer re-asserts defensively and never hands the importer a
pointer-only row).
The ``SUMMARY_MODE`` marker follows the ``DEFLECT_MODE`` convention:
the deterministic E2E mock LLM keys on it in the system prompt
(``tests/e2e/mock_llm.py`` — wired in task 06).
Image descriptions (phase 122, LOCKED A3): :func:`describe_image` is
this module's second one-shot generation path — a SINGLE CHAT-model
(vision) call describing one image's bytes as a base64 data URL. The
description becomes the image document's ``content`` AND ``summary``
(it is the ONLY embedded text of the doc — the embedding model never
sees pixels, and the ``lite`` summary model is deliberately NOT used:
it is not assumed vision-capable). Fail-soft by contract: any client
error, empty reply, or non-2xx yields ``None`` — the importer skips
the doc, counts ``images_failed``, and the sync continues.
The ``SUMMARY_MODE`` / ``IMAGE_DESCRIPTION_MODE`` markers follow the
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on
them (``tests/e2e/mock_llm.py`` — the image branch is wired by task
06's story suite).
"""
from __future__ import annotations
from typing import Protocol
import base64
import logging
from typing import Any, Protocol
from app.config import Settings, get_settings
from app.rag.llm import LLMError
from app.rag.retriever import TRUNCATION_MARKER
logger = logging.getLogger("app.summarizer")
#: System-prompt marker for summary generation — the E2E mock LLM keys on
#: it (same convention as ``DEFLECT_MODE``, PLAN §6).
SUMMARY_MODE = "SUMMARY_MODE"
@@ -51,6 +67,55 @@ SUMMARY_INSTRUCTION = (
#: Full system prompt: marker first (the mock's key), then the instruction.
SYSTEM_PROMPT = f"{SUMMARY_MODE}: {SUMMARY_INSTRUCTION}"
#: Image-description marker (phase 122, task 03) — the deterministic
#: E2E mock LLM keys on it (same convention as ``SUMMARY_MODE`` /
#: ``DEFLECT_MODE``, PLAN §6; the story suite wires the mock's branch
#: in task 06). It heads the text part of the multimodal describe
#: message, so it rides the user message, not a system prompt.
IMAGE_DESCRIPTION_MODE = "IMAGE_DESCRIPTION_MODE"
#: Locked instruction for the CHAT (vision) model (phase 122, LOCKED
#: A3): the description is the ONLY retrievable text of the image
#: document (the embedding model never sees pixels), so it must be a
#: faithful, retrieval-oriented account that carries the image's full
#: meaning — what is depicted, any visible text/labels/titles,
#: diagram/table structure, salient details.
DESCRIBE_INSTRUCTION = (
"Describe this image faithfully, in plain text, for a search index. "
"State what is depicted, transcribe any visible text, labels, or "
"titles, describe the structure of any diagram, table, or layout, and "
"call out the most salient details. Write 2-4 sentences of substance. "
"Do not use markdown. Do not invent anything that is not visible in "
"the image. Your description is the ONLY text that will ever be "
"retrieved for this image — it must carry the image's full meaning."
)
#: The full describe prompt: marker first (the mock's key), then the
#: instruction — the single text part of the multimodal user message.
DESCRIBE_PROMPT = f"{IMAGE_DESCRIPTION_MODE}: {DESCRIBE_INSTRUCTION}"
#: Extension → MIME type for the image family (phase 122). Dotted,
#: lowercase keys — the ``image_extension_set`` shape. Task 03 uses it
#: for the describe call's data-URL mime; task 04's serve route reuses
#: it for the image bytes' ``Content-Type`` (one map, one truth). A
#: ``BOR_IMAGE_EXTENSIONS`` token outside this map (a custom format)
#: takes the :data:`IMAGE_FALLBACK_MIME` data-URL mime in the describe
#: call — the vision endpoint may reject it, and the fail-soft skip
#: (``images_failed``) is the honest outcome.
IMAGE_MIMES: dict[str, str] = {
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".webp": "image/webp",
".gif": "image/gif",
".bmp": "image/bmp",
}
#: The data-URL mime for an image extension :data:`IMAGE_MIMES` does not
#: name (phase 122) — the best-effort generic, never a guess at a
#: specific type.
IMAGE_FALLBACK_MIME = "application/octet-stream"
class SummaryLLM(Protocol):
"""The one-shot chat surface the summarizer needs.
@@ -58,12 +123,17 @@ class SummaryLLM(Protocol):
:class:`app.rag.llm.LLMClient` satisfies it; unit tests pass a
duck-typed fake (``chat`` + ``settings``) instead — same pattern as
the importer's ``Embedder`` protocol.
``content`` may be a string (text calls — ``generate_summary``) or
a list of OpenAI-compatible parts (phase 122 multimodal image
descriptions — ``{type: "text", …}`` + ``{type: "image_url", …}``);
the client passes message dicts through untouched.
"""
settings: Settings
async def chat(
self, messages: list[dict[str, str]], model: str | None = None
self, messages: list[dict[str, Any]], model: str | None = None
) -> str: ...
@@ -121,3 +191,58 @@ async def generate_summary(
"refusing to store a silent summary"
)
return f"{summary}\nSource: {source}/{path}"
async def describe_image(
llm: SummaryLLM,
*,
data: bytes,
mime: str,
settings: Settings | None = None,
) -> str | None:
"""One-shot CHAT-model (vision) description of one image (phase 122,
LOCKED A3) — the text that becomes the image document's ``content``
AND ``summary`` (the ONLY embedded text of the doc; the embedding
model never sees pixels).
ONE chat-model call against ``settings.llm_chat_model`` (the vision
model — the ``lite`` summary model is NOT assumed vision-capable)
with the multimodal user message the OpenAI-compatible API expects:
``[{type: "text", text: DESCRIBE_PROMPT}, {type: "image_url",
image_url: {url: <data URL from *data* + *mime*>}}]`` — no system
prompt, no tools, no app-level retries beyond the client's own
(SDK-level + the house one-shot empty-content policy) — a
description failure must not stall a sync.
Returns the stripped reply cut exactly at
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap —
the description IS the summary, so it keeps the same uniform
ceiling). Returns ``None`` on any client error, empty reply, or
non-2xx (the client raises :class:`LLMError` for all three classes)
— the caller (the importer's ``_describe_or_skip`` seam) fails soft:
the doc is skipped, counted in ``images_failed``, and the sync
continues (LOCKED A3).
"""
data_url = f"data:{mime};base64,{base64.b64encode(data).decode('ascii')}"
messages: list[dict[str, Any]] = [
{
"role": "user",
"content": [
{"type": "text", "text": DESCRIBE_PROMPT},
{"type": "image_url", "image_url": {"url": data_url}},
],
}
]
model = llm.settings.llm_chat_model
try:
raw = await llm.chat(messages, model=model)
except LLMError as e:
logger.warning("image description failed (model=%s): %s", model, e)
return None
text = raw.strip()
if not text:
# The client already rejects empty content; this is the
# defensive re-assert (the duck-typed fakes may return it).
return None
limit = (settings or llm.settings).summary_max_chars
return text[:limit]