phase: 140_pictorial_pdf_pages
Build and Push Containers / build-and-push-app (push) Successful in 2m26s
Build and Push Containers / build-and-push-db (push) Successful in 12s

Everything is verified green. Final report:

## Phase 140 final verification pass — report

- Verified all 4 tasks already complete (code in working tree, task files in `complete/`): `pictorial_pages` gate + 2 settings (`BOR_PDF_PICTORIAL`, `BOR_PDF_PICTORIAL_MIN_CHARS` + validator), `PICTORIAL_PAGE_MODE`/`describe_pdf_page` seam + `render_pdf_pages` page selector + mock branch, `_pictorial_pdf_pass` in `_index_pdf_file` with log-only counters, dedicated E2E. No code changes needed.
- Test results (exact commands):
  - `uv run pytest --cov=app --cov-report=term-missing` → **3320 passed, 11 skipped, 0 failed; TOTAL 99%** (>90% gate)
  - `uv run pytest tests/e2e/test_pictorial_pdf.py -v --no-cov` → **3 passed in isolation** (Playwright, mock-only)
  - Pre-existing suites unedited: `test_pdf_support.py` 6 passed; `test_garbage_pdf.py` 2 passed + 1 skipped (owner fixture); `test_pdf_image_context.py` 4 passed
  - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
- Completion criteria:
  1. IKEA tar import repro — pinned by 2 integration tests; **skipped loudly here** (`test_sources/ikea_instructions.tar` absent on this machine — verified repo-wide; same house convention as completed phase-133 HNB tests; runs real on the owner's machine). Equivalent logic fully covered by unit pins on synthetic IKEA-class (vector-drawn, zero-raster) PDFs.
  2. Chat grounded on described page — **green** (E2E test b: phrase-only-in-illustration question, PDF in `done.sources`, answer echoes the block)
  3. Clean-text no-op (zero vision calls) + garbage precedence pin — **green** (`test_zero_candidates_is_a_noop` unit; `test_garbage_pdf_with_pictorial_on_takes_the_fallback_not_the_pass` integration, verified alone)
  4. `BOR_PDF_PICTORIAL=0` byte-identical — **green** (unit pin on synthetic PDF; IKEA variant runs on owner's machine)
  5. Full gate + E2E isolation + pre-existing suites — **green** (above)
  6. ruff + pyright — **clean**
  7. Commit/phase-move — delegated to harness per run rules (left in working tree: 16 modified + 8 new files, uncommitted)
- Deviations: none beyond the expected owner-fixture skips (11, all pre-existing fixture-missing skips, including 2 IKEA).
- Next pending phase: **none** — `todo/` holds only 140; 139 is already complete.
This commit is contained in:
2026-09-30 19:16:35 -04:00
parent a86544a083
commit 9633eb7653
45 changed files with 3895 additions and 36 deletions
+40
View File
@@ -396,6 +396,34 @@ class Settings(BaseSettings):
#: upload): the served copy must outlive the source file.
image_dir: str = "~/bor-sources/images"
# --- Pictorial PDF pages (phase 140: sparse pages rendered + vision-described) ---
#: Master switch for the pictorial PDF pass (phase 140, LOCKED A3,
#: ``BOR_PDF_PICTORIAL``; ``0``/``false`` = off). While on, a PDF
#: whose text layer passes the phase-133 usability gate ALSO gets the
#: sparse-page pass: every page whose stripped text is under
#: :attr:`pdf_pictorial_min_chars` chars (``app.rag.pdf.pictorial_pages``
#: — the deterministic gate, LOCKED A1) is rendered to a PNG and
#: vision-described, and the description appends to that page's
#: indexed content block (``[Page N illustration: …]``, LOCKED A4) —
#: the embeddings then carry what the thin text layer couldn't (the
#: IKEA assembly-instruction class: 20 of 24 pages are 4–73 chars of
#: markers around vector pictograms). NO page cap: every candidate
#: page is described (PLAN A21 — the LLM calls are free; a slower
#: sync is acceptable, a lower-quality KB is not). While off, a
#: PDF's indexed content is byte-identical to pre-phase-140 (the
#: ``BOR_IMAGES`` naming pattern — the kill switch).
pdf_pictorial: bool = True
#: Sparseness bar for the pictorial pass (phase 140, LOCKED A1,
#: ``BOR_PDF_PICTORIAL_MIN_CHARS``): a page whose STRIPPED text is
#: STRICTLY shorter than this many characters is a pictorial
#: candidate — a page with exactly this many stripped chars is NOT
#: sparse (the boundary is pinned in the unit tests). Default 100 —
#: the IKEA fixture's sparse pages run 4–73 chars and its text-heavy
#: pages 3,091+, so 100 sits between the classes. ``0`` = every page
#: is a candidate (PLAN A21); a negative value is a typo — the
#: validator fails startup loudly (the ``agent_max_rounds`` pattern).
pdf_pictorial_min_chars: int = 100
# --- Chat image questions (phase 123: attach an image to a question) ---
#: Where a user's question image bytes are stored (phase 123,
#: ``BOR_CHAT_IMAGE_DIR`` — the ``image_dir`` convention: a sibling of
@@ -572,6 +600,18 @@ class Settings(BaseSettings):
raise ValueError("read_max_chars must be >= 0 (chars)")
return v
@field_validator("pdf_pictorial_min_chars")
@classmethod
def _pdf_pictorial_min_chars_non_negative(cls, v: int) -> int:
"""``0`` is the every-page-is-a-candidate mode (PLAN A21 — the
LLM calls are free) — a negative threshold is a typo (the
``agent_max_rounds`` pattern, phase 140)."""
if v < 0:
raise ValueError(
"pdf_pictorial_min_chars must be >= 0 (0 = every page is a candidate)"
)
return v
@field_validator("embed_question_max_chars")
@classmethod
def _embed_question_max_chars_positive(cls, v: int) -> int:
+191 -1
View File
@@ -129,6 +129,28 @@ as unchanged (no re-chunk/re-embed/re-summary, no KB change; the
vision calls are re-issued each sync — the no-cache cost, documented
in :func:`_fallback_page_screenshots`).
Pictorial sparse pages (phase 140, task 03, LOCKED A1–A4): AFTER the
phase-133 gate, a PDF whose text layer is USABLE gets the sparse-page
pass (:func:`_pictorial_pdf_pass`, gated by the ``pdf_pictorial``
setting — ``BOR_PDF_PICTORIAL``, the house kill-switch): the
deterministic gate (:func:`app.rag.pdf.pictorial_pages` — NO LLM
call, LOCKED A1) selects the pages whose stripped text is under
``pdf_pictorial_min_chars``; EVERY candidate is rendered (PLAN A21 —
no page cap, no call budget) and vision-described through the phase-
140 seam (:func:`app.rag.summarizer.describe_pdf_page` — the
``PICTORIAL_PAGE_MODE`` one-shot, fail-soft per page), and the
description appends to the page's content block as
``[Page N illustration: …]`` (LOCKED A2 — no LLM judge: every
described page joins the content; the owner's L5 rationale: more
information is better for the embeddings). The precedence is locked:
the pass runs on a USABLE layer ONLY — an unusable layer takes the
phase-133 full-page screenshot fallback and never burns a second
vision pass. Zero candidates → a no-op (no render, no vision call —
a clean-text PDF's path is byte-identical to pre-phase); the
counters ``pdfs_pictorial`` / ``pdf_pictorial_pages`` are log-only
(the sync ``detail`` API shape is untouched, the stats log-only
convention).
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
description, the same source's importable markdown files are scanned for
``![alt](path)`` references that name the image (the task-02
@@ -250,6 +272,7 @@ from app.rag.pdf import (
extract_pdf_page_texts,
extract_pdf_text,
is_usable_pdf_text,
pictorial_pages,
render_pdf_pages,
)
from app.rag.retriever import TRUNCATION_MARKER
@@ -257,6 +280,7 @@ from app.rag.summarizer import (
IMAGE_FALLBACK_MIME,
IMAGE_MIMES,
describe_image,
describe_pdf_page,
generate_summary,
)
@@ -341,6 +365,20 @@ class ImportSummary:
#: ``---PAGE-N---``-anchored descriptions, or ``""`` when the
#: fallback produced nothing — the phase-126 empty-content path).
pdfs_screenshotted: int = 0
#: PDFs (phase 140, task 03, LOCKED A2/A4) whose USABLE text layer
#: carried ≥1 pictorial candidate page (``app.rag.pdf.
#: pictorial_pages`` — the deterministic sparse-page gate, LOCKED
#: A1) whose vision description was appended to the page block
#: (``[Page N illustration: …]``) — the sparse-page pass
#: (:func:`_pictorial_pdf_pass`). Log-only (the sync ``detail``
#: API shape is untouched — the stats log-only convention,
#: phase 133); ``pdf_pictorial_pages`` counts the described pages.
pdfs_pictorial: int = 0
#: Pictorial candidate pages described across the run (phase 140,
#: task 03) — one PDF with three described sparse pages counts
#: ``pdfs_pictorial=1`` + ``pdf_pictorial_pages=3`` (A21: no page
#: cap — the count may be large, that is the point).
pdf_pictorial_pages: int = 0
#: PDF-embedded image docs whose stored ``image_context`` was
#: backfilled with the document-context block (phase 137, task 03)
#: — the one-time best-effort pass; a success never repeats, a
@@ -367,7 +405,8 @@ class ImportSummary:
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"summary_backfilled=%d dates_updated=%d images_failed=%d "
"image_context_backfilled=%d pdfs_screenshotted=%d formats=%s",
"image_context_backfilled=%d pdfs_screenshotted=%d "
"pdfs_pictorial=%d pdf_pictorial_pages=%d formats=%s",
self.files,
self.added,
self.updated,
@@ -383,6 +422,8 @@ class ImportSummary:
self.images_failed,
self.image_context_backfilled,
self.pdfs_screenshotted,
self.pdfs_pictorial,
self.pdf_pictorial_pages,
self.format_counts(),
)
@@ -1165,6 +1206,19 @@ async def _index_pdf_file(
An unopenable PDF — or one whose fallback also produced nothing
— yields ``""`` (still a document — it may carry embedded images
— with no content chunks; a warning is the signal).
* Phase 140 (task 03, LOCKED A1–A4): when the layer is USABLE and
``settings.pdf_pictorial`` is on, the sparse-page pass
(:func:`_pictorial_pdf_pass`) renders + vision-describes every
page whose stripped text is under
``settings.pdf_pictorial_min_chars`` (the deterministic gate —
NO LLM call, A1; every candidate, no page cap — A21/A3) and
appends the description to the page block as
``[Page N illustration: …]`` (A2/A4). The precedence is locked:
the pass sits AFTER the phase-133 gate, so an unusable layer
(the fallback's full-page screenshots) never burns a second
vision pass; zero candidates → a no-op (the extracted text
stays byte-identical to the pre-phase-140 path — no render, no
vision call).
* the sha256 digest is over the FINAL content (the digest rule is
content identity — the same text is the same document version;
the fallback's constructed page descriptions digest exactly the
@@ -1213,6 +1267,20 @@ async def _index_pdf_file(
content = await _fallback_page_screenshots(
session, source=source, rel=rel, full_path=full_path, llm=llm
)
elif llm.settings.pdf_pictorial:
# Phase 140 (task 03, LOCKED A1–A4): the pictorial pass —
# USABLE layers only (the precedence is locked: the phase-133
# fallback above already screenshotted an unusable layer, no
# double vision). Zero candidates → a no-op (the pass returns
# None — the extracted text stays byte-identical to the
# pre-phase-140 path); otherwise the FINAL content carries the
# described candidates' illustration blocks (A4: the digest
# below runs over it, the phase-126/133 convention).
passed = await _pictorial_pdf_pass(
source=source, rel=rel, full_path=full_path, llm=llm, summary=summary
)
if passed is not None:
content = passed
if not content:
logger.warning(
"import: PDF has no extractable text source=%s path=%s", source, rel
@@ -1406,6 +1474,128 @@ async def _fallback_page_screenshots(
)
async def _pictorial_pdf_pass(
*,
source: str,
rel: str,
full_path: Path,
llm: Embedder,
summary: ImportSummary,
) -> str | None:
"""The phase-140 (task 03) pictorial pass (LOCKED A1–A4) — the
IKEA assembly-instruction class (TODO L5): a PDF whose text layer
is USABLE but SPARSE (mostly pictorial pages — 4–73 chars of
markers around vector-drawn pictograms) gets every sparse page
rendered to a PNG and vision-described, and the description
APPENDS to that page's indexed content block
(``[Page N illustration: …]``) — the page's text stays, so the
embeddings carry what the thin text layer couldn't (the owner's
L5 rationale: "More information is better when generating the
text embeddings").
* the gate is :func:`app.rag.pdf.pictorial_pages` over
:func:`app.rag.pdf.extract_pdf_page_texts` — deterministic, NO
LLM call (LOCKED A1): the 1-based page numbers whose stripped
text is under ``settings.pdf_pictorial_min_chars``, ascending.
(The kill-switch itself — ``settings.pdf_pictorial`` — gates the
whole pass in the caller: while off, this function is never
called and the content is byte-identical to pre-phase.)
* ZERO candidates → a NO-OP (returns ``None``): no render, no
vision call — the caller keeps its :func:`extract_pdf_text`
output untouched, so a clean-text PDF's path is byte-identical
to the pre-phase-140 path (the pin).
* a page that could not be rendered (the zero-byte
:func:`render_pdf_pages` placeholder — the index never shifts,
so it stays aligned with the candidates) is skipped with a
warning (the phase-133 handling — the page keeps its text only,
no vision call is burned for it); a page whose vision call
failed or came back blank (:func:`describe_pdf_page` →
``None``) is skipped with a warning and keeps its text only —
one dead page never kills the pass (LOCKED A2 fail-soft).
* the assembly (LOCKED A4): every page of the
:func:`extract_pdf_page_texts` map (the SAME pages, order and
stripped text as :func:`extract_pdf_text`) as its
:data:`PAGE_SEPARATOR` + the page text + — for the described
candidates — ``"\\n\\n[Page {n} illustration: {description}]"``.
The assembly with NO descriptions is byte-identical to
:func:`extract_pdf_text`'s output (the unit pin).
* the caller's digest runs over this FINAL content (the
phase-126/133 convention — A4): a stable file + a stable vision
reply re-syncs as ``unchanged``; a non-deterministic re-describe
re-updates the row — a cost, not a bug (the phase-133 docstring
documents it, A21 endorses it).
* stats (log-only — the sync ``detail`` API shape is untouched,
the phase-133 convention): ``summary.pdfs_pictorial`` (+1 when
≥1 page was described) and ``summary.pdf_pictorial_pages``
(+the described page count), plus the ONE loud log line (PLAN
§9): ``pdf: pictorial pass source=… path=… — N candidate pages,
M described (min_chars=…)``.
"""
settings = llm.settings
page_texts = extract_pdf_page_texts(full_path)
candidates = pictorial_pages(
page_texts, min_chars=settings.pdf_pictorial_min_chars
)
if not candidates:
return None
# ONE render for the whole candidate set (the phase-140 selector —
# a 500-page PDF with 30 sparse pages renders 30 PNGs, not 500);
# the result is aligned with *candidates* (ascending, one slot
# each — a zero-byte placeholder never shifts the positions).
pngs = render_pdf_pages(full_path, pages=candidates)
descriptions: dict[int, str] = {}
# strict: the render's result is aligned with the candidate list by
# contract (one slot per named page, ascending) — a length drift is
# a parser regression, fail loud.
for page, png in zip(candidates, pngs, strict=True):
if not png:
# The phase-133 zero-byte placeholder: an unreadable page.
logger.warning(
"pdf: skipping pictorial page %d of %s/%s — the render is "
"empty (unreadable page)",
page, source, rel,
)
continue
desc = await describe_pdf_page(
llm,
data=png,
mime="image/png",
page=page,
page_text=page_texts[page],
)
if desc is None:
# Fail-soft (LOCKED A2): one dead page never kills the
# pass — the page keeps its text only, the sync continues.
logger.warning(
"pdf: pictorial page %d of %s/%s has no description — "
"the page keeps its text only",
page, source, rel,
)
continue
descriptions[page] = desc
# The ONE loud log line (PLAN §9): N candidate pages, M described.
logger.info(
"pdf: pictorial pass source=%s path=%s — %d candidate pages, %d "
"described (min_chars=%d)",
source, rel, len(candidates), len(descriptions),
settings.pdf_pictorial_min_chars,
)
# The stats (log-only — the sync ``detail`` API shape is untouched).
if descriptions:
summary.pdfs_pictorial += 1
summary.pdf_pictorial_pages += len(descriptions)
# The assembly: the extracted text's page blocks, the described
# candidates carrying their illustration block (A4 — the page's
# text STAYS; the description appends after it).
parts: list[str] = []
for page, text in page_texts.items():
block = text
if page in descriptions:
block += f"\n\n[Page {page} illustration: {descriptions[page]}]"
parts.append(f"{PAGE_SEPARATOR.format(page=page)}\n{block}")
return "\n\n".join(parts)
async def _index_pdf_embedded_images(
session: Session,
*,
+82 -17
View File
@@ -29,11 +29,20 @@ package, imported as ``fitz`` — its stable API name).
I/O) — REAL content vs. garbage (the pre-126 XML/binary soup, the
empty text layer). It is the single trigger for the page-screenshot
vision fallback (phase 133 task 03).
* :func:`pictorial_pages` — the phase-140 sparse-page gate: the 1-based
page numbers of a :func:`extract_pdf_page_texts` map whose stripped
text is STRICTLY under *min_chars* chars (ascending) — pure,
deterministic, NO LLM call (LOCKED A1): the "way to determine when
that's necessary" (TODO L5) for the pictorial pass.
* :func:`render_pdf_pages` — the fallback's visual input (phase 133
task 03): one PNG per page (150 DPI, page order); an unreadable page
yields ``b""`` (the caller's placeholder — the page index never
shifts), an unopenable file yields ``[]`` (the caller's existing
empty-content path).
task 03): one PNG per page (150 DPI, page order); the phase-140
optional 1-based *pages* selector renders ONLY the named pages
(ascending — a 500-page PDF with 30 sparse pages renders 30 PNGs,
not 500), ``None`` = every page (the phase-133 caller's result is
byte-identical); an unreadable (or out-of-range named) page yields
``b""`` at its position (the caller's placeholder — the result
stays aligned with the requested pages), an unopenable file yields
``[]`` (the caller's existing empty-content path).
Every failure mode is fail-soft: an unreadable file returns an empty
result, an unreadable page or image is logged and skipped — this module
@@ -43,7 +52,7 @@ from __future__ import annotations
import logging
import re
from collections.abc import Iterator
from collections.abc import Iterator, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Any
@@ -264,15 +273,60 @@ def is_usable_pdf_text(text: str) -> bool:
return non_printable / len(text) < NONPRINT_CHAR_FRACTION
def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
"""Render every page of *pdf_path* to a PNG (page order — list index
*i* is page *i*+1).
def pictorial_pages(page_texts: dict[int, str], *, min_chars: int) -> list[int]:
"""The 1-based page numbers of *page_texts* that are PICTORIAL
candidates: ``len(text.strip()) < min_chars`` — STRICTLY less than
(a page with exactly *min_chars* stripped chars is NOT sparse; the
boundary is pinned in the unit tests), ascending.
The deterministic sparse-page gate (phase 140, task 01, LOCKED A1)
— the "way to determine when that's necessary" for the pictorial PDF
pass (TODO L5): NO LLM call, pure, deterministic (no PDF, no I/O —
the input is a :func:`extract_pdf_page_texts` map, so the gate never
re-opens the file). A page that carries no text at all has NO key in
the map and is therefore not a candidate here (the importer's pass
owns the page-total handling); an empty map yields ``[]`` — a
clean-text PDF's path never burns a vision call.
The IKEA example (the phase-140 fixture, ``min_chars=100``):
``{1: "BESTÅ", 2: <7 KB of warnings>, 6: "Illustration 6"}`` →
``[1, 6, …]`` — page 1 (the title page: the logo + product name) IS
a candidate: describing it is useful (the logo + product name) and
never harmful; page 2's 7 KB of multilingual warnings is not.
``min_chars=0`` is the documented PLAN A21 special case — **every
page in the map is a candidate** (the "describe everything" mode).
The strict-``<`` test cannot express this (no length is below 0), so
0 is not "a stricter bar": it is the A21 no-cap endpoint, and a
negative value — unreachable through ``Settings`` (the validator
fails startup loudly) — takes the same branch defensively.
"""
if min_chars <= 0:
return sorted(page_texts)
return sorted(
page for page, text in page_texts.items() if len(text.strip()) < min_chars
)
def render_pdf_pages(
pdf_path: Path, dpi: int = 150, pages: Sequence[int] | None = None
) -> list[bytes]:
"""Render the selected pages of *pdf_path* to PNGs.
*pages* (phase 140, task 02): the optional 1-based selector —
render ONLY the named pages, in ascending order, one PNG each
(a 500-page PDF with 30 pictorial candidates renders 30 PNGs, not
500). ``None`` (the default) = every page in page order — the
phase-133 caller's result is byte-identical, so list index *i* is
page *i*+1 (with a selector, list index *i* is the *i*-th smallest
NAMED page).
The fallback's visual input (phase 133 task 03, ASSUMPTION A8:
fixed 150 DPI — the balance of vision legibility vs data-URL
payload). A page that cannot be rendered yields ``b""`` (the
caller's placeholder — the page index NEVER shifts, so
``---PAGE-N---`` stays truthful; the failure is logged, the
payload). A page that cannot be rendered — including a named page
outside the document's page range — yields ``b""`` at ITS position
(the caller's placeholder — the result's order NEVER shifts, so it
stays aligned with the requested pages; the failure is logged, the
:func:`extract_pdf_text` fail-soft style). A file that cannot be
OPENED yields ``[]`` (the caller's existing empty-content path —
the doc lands with the no-extractable-text behavior, unchanged).
@@ -283,18 +337,29 @@ def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
logger.warning("pdf: could not open %s for page rendering — %s", pdf_path, exc)
return []
pages: list[bytes] = []
# 1-based page numbers: the default walks every page in page order
# (the phase-133 contract — list index *i* is page *i*+1); a
# selector renders the named pages, ascending (de-duplicated — a
# repeated page number would render it twice for no reason).
numbers = (
range(1, len(doc) + 1)
if pages is None
else sorted(set(pages))
)
rendered: list[bytes] = []
with doc:
for page_num in range(len(doc)):
for page_number in numbers:
try:
pages.append(doc[page_num].get_pixmap(dpi=dpi).tobytes("png"))
rendered.append(
doc[page_number - 1].get_pixmap(dpi=dpi).tobytes("png")
)
except Exception as exc:
logger.warning(
"pdf: could not render page %d of %s — %s",
page_num + 1, pdf_path, exc,
page_number, pdf_path, exc,
)
pages.append(b"")
return pages
rendered.append(b"")
return rendered
def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
+143 -9
View File
@@ -33,10 +33,24 @@ it is not assumed vision-capable). Fail-soft by contract: any client
error, empty reply, or non-2xx yields ``None`` — the importer skips
the doc, counts ``images_failed``, and the sync continues.
The ``SUMMARY_MODE`` / ``IMAGE_DESCRIPTION_MODE`` markers follow the
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on
them (``tests/e2e/mock_llm.py`` — the image branch is wired by task
06's story suite).
Pictorial PDF pages (phase 140, LOCKED A2): :func:`describe_pdf_page`
is this module's third one-shot generation path — a SINGLE CHAT-model
(vision) call describing ONE rendered sparse PDF page, with the
page's OWN extracted text as context so the model describes what the
text does NOT carry (the IKEA assembly-instruction class: vector
pictograms around a 4–73-char text layer). The description is
APPENDED to the page's indexed content block by the importer (the
``[Page N illustration: …]`` line, LOCKED A4) — the page's text stays;
more information is better for the embeddings (the owner's L5
rationale). Fail-soft by contract: any client error or empty reply
yields ``None`` — the importer skips that page with a warning and
continues (one dead page never stalls a sync).
The ``SUMMARY_MODE`` / ``IMAGE_DESCRIPTION_MODE`` /
``PICTORIAL_PAGE_MODE`` markers follow the ``DEFLECT_MODE``
convention: the deterministic E2E mock LLM keys on them
(``tests/e2e/mock_llm.py`` — the image branch is wired by task
06's story suite, the pictorial branch by phase 140 task 02).
"""
from __future__ import annotations
@@ -92,8 +106,41 @@ DESCRIBE_INSTRUCTION = (
#: The full describe prompt: marker first (the mock's key), then the
#: instruction — the single text part of the multimodal user message.
#: BYTE-IDENTICAL CONTRACT: the phase 122/126/137 suites pin this
#: string untouched — pictorial-page descriptions go through their
#: own marker (:data:`PICTORIAL_PAGE_MODE`), never a re-wrap of this
#: one.
DESCRIBE_PROMPT = f"{IMAGE_DESCRIPTION_MODE}: {DESCRIBE_INSTRUCTION}"
#: Pictorial-page marker (phase 140, task 02, LOCKED A1/A2) — the
#: deterministic E2E mock LLM keys on it (same convention as
#: ``IMAGE_DESCRIPTION_MODE``, PLAN §6). It heads the text part of
#: the multimodal describe message, so it rides the user message, not
#: a system prompt.
PICTORIAL_PAGE_MODE = "PICTORIAL_PAGE_MODE"
#: Locked instruction for the CHAT (vision) model (phase 140, LOCKED
#: A2): the description APPENDS to the page's indexed content block
#: (``[Page N illustration: …]``) — the embeddings then carry what
#: the thin text layer couldn't (the owner's L5 rationale) — so it
#: must be a faithful, retrieval-oriented account of what the page
#: SHOWS: the assembly steps, part names, arrows, tool icons, and
#: before/after states — ESPECIALLY the information the extracted
#: text does not contain.
PICTORIAL_INSTRUCTION = (
"Describe in detail what this page of the PDF shows, in plain text, "
"for a search index. State the assembly steps, part names, arrows, "
"tool icons, and before/after states it illustrates — ESPECIALLY the "
"information the extracted text does not contain. Write 2-4 sentences "
"of substance. Do not use markdown. Do not invent anything that is "
"not visible on the page."
)
#: The full pictorial-page prompt: marker first (the mock's key), then
#: the instruction — the head of the multimodal user message's text
#: part (the page block is appended by :func:`describe_pdf_page`).
PICTORIAL_PROMPT = f"{PICTORIAL_PAGE_MODE}: {PICTORIAL_INSTRUCTION}"
#: Extension → MIME type for the image family (phase 122). Dotted,
#: lowercase keys — the ``image_extension_set`` shape. Task 03 uses it
#: for the describe call's data-URL mime; task 04's serve route reuses
@@ -266,11 +313,43 @@ async def describe_image(
if context
else ""
)
return await _one_shot_vision_reply(
llm,
data_url=data_url,
text=DESCRIBE_PROMPT + context_prefix,
settings=settings,
what="image description",
)
async def _one_shot_vision_reply(
llm: SummaryLLM,
*,
data_url: str,
text: str,
settings: Settings | None,
what: str,
) -> str | None:
"""The shared one-chat-call multimodal contract (phase 122, reused
by phase 140): ONE user message
``[{type: "text", text: <text>},
{type: "image_url", image_url: {url: <data_url>}}]`` against
``llm.settings.llm_chat_model`` — no system prompt, no tools, no
app-level retry.
Returns the stripped reply cut exactly at
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap);
``None`` on any client error (the client raises :class:`LLMError`
for a failure, an empty reply, or a non-2xx) or an empty/blank
reply after trimming (the house one-shot empty-content policy — the
duck-typed fakes may return one even though the real client
rejects it). *what* names the caller for the warning log line.
"""
messages: list[dict[str, Any]] = [
{
"role": "user",
"content": [
{"type": "text", "text": DESCRIBE_PROMPT + context_prefix},
{"type": "text", "text": text},
{"type": "image_url", "image_url": {"url": data_url}},
],
}
@@ -279,12 +358,67 @@ async def describe_image(
try:
raw = await llm.chat(messages, model=model)
except LLMError as e:
logger.warning("image description failed (model=%s): %s", model, e)
logger.warning("%s failed (model=%s): %s", what, model, e)
return None
text = raw.strip()
if not text:
text_out = raw.strip()
if not text_out:
# The client already rejects empty content; this is the
# defensive re-assert (the duck-typed fakes may return it).
return None
limit = (settings or llm.settings).summary_max_chars
return text[:limit]
return text_out[:limit]
async def describe_pdf_page(
llm: SummaryLLM,
*,
data: bytes,
mime: str,
page: int,
page_text: str | None,
settings: Settings | None = None,
) -> str | None:
"""One-shot CHAT-model (vision) description of ONE rendered PDF page
(phase 140, task 02, LOCKED A2) — the text the importer appends to
that page's indexed content block as
``[Page N illustration: …]`` (the page's own text stays; the
embeddings then carry what the thin text layer couldn't — the
owner's L5 rationale).
ONE chat-model call against ``llm.settings.llm_chat_model`` (the
vision model — the ``lite`` summary model is NOT assumed
vision-capable) with the same multimodal contract as
:func:`describe_image` (via :func:`_one_shot_vision_reply`): no
system prompt, no tools, no app-level retry. The message's text
part is :data:`PICTORIAL_PROMPT` (the ``PICTORIAL_PAGE_MODE``
marker + the locked instruction) followed by the page block — the
page's OWN extracted text as context, so the model describes what
the text does NOT contain::
PICTORIAL_PROMPT
+ "\\n\\nPage {page} of the PDF. Text extracted from this page:\n"
+ (page_text.strip() or "(none)")
``data`` is the rendered page's PNG bytes (phase 133's
``render_pdf_pages`` output), ``mime`` its MIME type (``image/png``
for the rendered page).
Returns the stripped reply cut exactly at
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap).
Returns ``None`` on any client error, empty reply, or non-2xx
(fail-soft — the importer skips THAT page with a warning and
continues: one dead page never stalls a sync, LOCKED A2).
"""
data_url = image_data_url(data, mime)
extracted = page_text.strip() if page_text is not None else ""
return await _one_shot_vision_reply(
llm,
data_url=data_url,
text=(
PICTORIAL_PROMPT
+ f"\n\nPage {page} of the PDF. Text extracted from this page:\n"
+ (extracted or "(none)")
),
settings=settings,
what=f"pictorial page description (page={page})",
)