phase: 122_image_documents
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**
**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs
**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)
**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).
**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.
**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
+73
-5
@@ -26,6 +26,14 @@ Implements just enough of the aipi surface:
|
||||
``SUMMARY_MODE`` branch: the folder marker CONTAINS the summary
|
||||
marker as a substring, so the summary branch would otherwise
|
||||
shadow every folder-summary call
|
||||
- ``IMAGE_DESCRIPTION_MODE`` (phase 122, image documents) -> the
|
||||
fixed ``IMAGE_DESCRIPTION_ANSWER`` description: the CHAT model's
|
||||
(vision) reply to ``summarizer.describe_image``'s multimodal user
|
||||
message (the marker is its first text part; the call is
|
||||
non-streaming, so only this answer path serves it). The reply is
|
||||
the image document's whole content + summary, so it is byte-
|
||||
stable and token-dense (its cosine against the story suite's
|
||||
question clears the mock-calibrated gate)
|
||||
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
|
||||
- otherwise -> upbeat answer quoting the provided document context
|
||||
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
|
||||
@@ -592,19 +600,46 @@ def _messages(body: dict[str, Any]) -> list[dict[str, str]]:
|
||||
return body.get("messages", [])
|
||||
|
||||
|
||||
def _content_text(content: Any) -> str:
|
||||
"""The TEXT of one message's content (phase 122 list safety).
|
||||
|
||||
String content passes through byte-identical (as does an absent
|
||||
value or an explicit ``None`` — the ``or ""`` semantics the
|
||||
:func:`_context` docstring pins). A multimodal part LIST — the
|
||||
phase-122 image description's ``[{type: "text", …},
|
||||
{type: "image_url", …}]``, the only list-content message the app
|
||||
produces (``app.rag.summarizer.describe_image``) — contributes its
|
||||
text parts joined (the ``image_url`` part carries no text): the
|
||||
trigger checks and marker branches read the text, and no existing
|
||||
string-content request is affected.
|
||||
"""
|
||||
if isinstance(content, list):
|
||||
return " ".join(
|
||||
part.get("text", "")
|
||||
for part in content
|
||||
if isinstance(part, dict) and part.get("type") == "text"
|
||||
)
|
||||
return content or ""
|
||||
|
||||
|
||||
def _system(body: dict[str, Any]) -> str:
|
||||
return " ".join(m.get("content", "") for m in _messages(body) if m.get("role") == "system")
|
||||
return " ".join(
|
||||
_content_text(m.get("content"))
|
||||
for m in _messages(body)
|
||||
if m.get("role") == "system"
|
||||
)
|
||||
|
||||
|
||||
def _user(body: dict[str, Any]) -> str:
|
||||
parts = [m.get("content", "") for m in _messages(body) if m.get("role") == "user"]
|
||||
parts = [_content_text(m.get("content")) for m in _messages(body) if m.get("role") == "user"]
|
||||
return parts[-1] if parts else ""
|
||||
|
||||
|
||||
def _context(body: dict[str, Any]) -> str:
|
||||
"""The document context is the longest system/user message in practice.
|
||||
|
||||
``m.get("content") or ""`` (NOT ``m.get("content", "")``): a well-formed
|
||||
``_content_text(m.get("content"))`` (the ``or ""`` semantics, NOT
|
||||
``m.get("content", "")``): a well-formed
|
||||
OpenAI tool-call message carries ``content: None`` EXPLICITLY (the app's
|
||||
agent loop appends exactly that — ``app/rag/agent.py``), and a forced
|
||||
final answer after a tool round (the round-cap path) reaches this helper
|
||||
@@ -612,9 +647,12 @@ def _context(body: dict[str, Any]) -> str:
|
||||
an explicit ``None`` and crashes ``len()`` with a 500 (phase 93 task 04
|
||||
caught it via the deterministic single-read flow's ALREADY_IN_CONTEXT
|
||||
loop); ``or ""`` treats absent and explicit-None alike, so the fallback
|
||||
composes deterministically instead of traceback-ing."""
|
||||
composes deterministically instead of traceback-ing. Phase 122: a
|
||||
multimodal part list maps to its text parts (see
|
||||
:func:`_content_text`), so ``len()``/slicing never meet a list.
|
||||
"""
|
||||
msgs = _messages(body)
|
||||
return max((m.get("content") or "" for m in msgs), key=len)
|
||||
return max((_content_text(m.get("content")) for m in msgs), key=len)
|
||||
|
||||
|
||||
LONG_ANSWER_TRIGGER = "write a long answer"
|
||||
@@ -739,6 +777,25 @@ HISTORY_TRIGGER = "echo my history"
|
||||
#: phrase, so every other suite is unaffected.
|
||||
FOLDER_MAP_TRIGGER = "repeat your folder map"
|
||||
|
||||
#: Phase 122 (image documents, LOCKED A3): the fixed description the
|
||||
#: mock's vision (CHAT) model returns for an ``IMAGE_DESCRIPTION_MODE``
|
||||
#: request — the multimodal user message
|
||||
#: ``[{type: "text", …IMAGE_DESCRIPTION_MODE…}, {type: "image_url",
|
||||
#: …}]`` (``app.rag.summarizer.describe_image``, the app's only
|
||||
#: list-content message). It becomes the image document's WHOLE
|
||||
#: ``content`` AND ``summary`` (the only embedded text — the embedding
|
||||
#: model never sees pixels), so the text is byte-stable across runs and
|
||||
#: token-dense: the story suite's question ("What is shown in the
|
||||
#: homelab network diagram?", cosine ≈0.38 against the mock's
|
||||
#: token-overlap embeddings) clears the mock-calibrated gate (0.30)
|
||||
#: and the citation usefulness floor (0.15).
|
||||
IMAGE_DESCRIPTION_ANSWER = (
|
||||
"A network diagram of the homelab server room: a core router on top, "
|
||||
"a core switch in the middle, and three labeled subnets at the bottom — "
|
||||
"VLAN 10 office, VLAN 20 lab, and VLAN 30 storage — with a legend of "
|
||||
"cable runs. Title: Homelab Network Map."
|
||||
)
|
||||
|
||||
TABLE_ANSWER = (
|
||||
"Here's the shape, in a table:\n"
|
||||
"\n"
|
||||
@@ -2178,6 +2235,17 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
answer = "Knowledge base outline:\n- " + " ".join(
|
||||
TOKEN_RE.findall(user.lower())[:8]
|
||||
)
|
||||
elif "IMAGE_DESCRIPTION_MODE" in user:
|
||||
# Phase 122 (image documents, LOCKED A3): the CHAT model's
|
||||
# (vision) image description — a NON-STREAMING multimodal user
|
||||
# message, so only this path ever sees it. ``_content_text``
|
||||
# joins the text parts, so the marker (the first text part)
|
||||
# lands in ``user``. Checked before the user-trigger branches:
|
||||
# the marker is a fixed app constant, and a real question is
|
||||
# never expected to type it (the other user triggers are owner
|
||||
# phrasings a question might legitimately contain — this one
|
||||
# is an app-to-app marker).
|
||||
answer = IMAGE_DESCRIPTION_ANSWER
|
||||
elif TABLE_TRIGGER in user.lower():
|
||||
# Markdown tables (phase 44, TODO.md L6): the story E2E's
|
||||
# deterministic table answer — a 3-column table, the
|
||||
|
||||
Reference in New Issue
Block a user