phase: 140_pictorial_pdf_pages
Everything is verified green. Final report: ## Phase 140 final verification pass — report - Verified all 4 tasks already complete (code in working tree, task files in `complete/`): `pictorial_pages` gate + 2 settings (`BOR_PDF_PICTORIAL`, `BOR_PDF_PICTORIAL_MIN_CHARS` + validator), `PICTORIAL_PAGE_MODE`/`describe_pdf_page` seam + `render_pdf_pages` page selector + mock branch, `_pictorial_pdf_pass` in `_index_pdf_file` with log-only counters, dedicated E2E. No code changes needed. - Test results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → **3320 passed, 11 skipped, 0 failed; TOTAL 99%** (>90% gate) - `uv run pytest tests/e2e/test_pictorial_pdf.py -v --no-cov` → **3 passed in isolation** (Playwright, mock-only) - Pre-existing suites unedited: `test_pdf_support.py` 6 passed; `test_garbage_pdf.py` 2 passed + 1 skipped (owner fixture); `test_pdf_image_context.py` 4 passed - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors - Completion criteria: 1. IKEA tar import repro — pinned by 2 integration tests; **skipped loudly here** (`test_sources/ikea_instructions.tar` absent on this machine — verified repo-wide; same house convention as completed phase-133 HNB tests; runs real on the owner's machine). Equivalent logic fully covered by unit pins on synthetic IKEA-class (vector-drawn, zero-raster) PDFs. 2. Chat grounded on described page — **green** (E2E test b: phrase-only-in-illustration question, PDF in `done.sources`, answer echoes the block) 3. Clean-text no-op (zero vision calls) + garbage precedence pin — **green** (`test_zero_candidates_is_a_noop` unit; `test_garbage_pdf_with_pictorial_on_takes_the_fallback_not_the_pass` integration, verified alone) 4. `BOR_PDF_PICTORIAL=0` byte-identical — **green** (unit pin on synthetic PDF; IKEA variant runs on owner's machine) 5. Full gate + E2E isolation + pre-existing suites — **green** (above) 6. ruff + pyright — **clean** 7. Commit/phase-move — delegated to harness per run rules (left in working tree: 16 modified + 8 new files, uncommitted) - Deviations: none beyond the expected owner-fixture skips (11, all pre-existing fixture-missing skips, including 2 IKEA). - Next pending phase: **none** — `todo/` holds only 140; 139 is already complete.
This commit is contained in:
+191
-1
@@ -129,6 +129,28 @@ as unchanged (no re-chunk/re-embed/re-summary, no KB change; the
|
||||
vision calls are re-issued each sync — the no-cache cost, documented
|
||||
in :func:`_fallback_page_screenshots`).
|
||||
|
||||
Pictorial sparse pages (phase 140, task 03, LOCKED A1–A4): AFTER the
|
||||
phase-133 gate, a PDF whose text layer is USABLE gets the sparse-page
|
||||
pass (:func:`_pictorial_pdf_pass`, gated by the ``pdf_pictorial``
|
||||
setting — ``BOR_PDF_PICTORIAL``, the house kill-switch): the
|
||||
deterministic gate (:func:`app.rag.pdf.pictorial_pages` — NO LLM
|
||||
call, LOCKED A1) selects the pages whose stripped text is under
|
||||
``pdf_pictorial_min_chars``; EVERY candidate is rendered (PLAN A21 —
|
||||
no page cap, no call budget) and vision-described through the phase-
|
||||
140 seam (:func:`app.rag.summarizer.describe_pdf_page` — the
|
||||
``PICTORIAL_PAGE_MODE`` one-shot, fail-soft per page), and the
|
||||
description appends to the page's content block as
|
||||
``[Page N illustration: …]`` (LOCKED A2 — no LLM judge: every
|
||||
described page joins the content; the owner's L5 rationale: more
|
||||
information is better for the embeddings). The precedence is locked:
|
||||
the pass runs on a USABLE layer ONLY — an unusable layer takes the
|
||||
phase-133 full-page screenshot fallback and never burns a second
|
||||
vision pass. Zero candidates → a no-op (no render, no vision call —
|
||||
a clean-text PDF's path is byte-identical to pre-phase); the
|
||||
counters ``pdfs_pictorial`` / ``pdf_pictorial_pages`` are log-only
|
||||
(the sync ``detail`` API shape is untouched, the stats log-only
|
||||
convention).
|
||||
|
||||
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
|
||||
description, the same source's importable markdown files are scanned for
|
||||
```` references that name the image (the task-02
|
||||
@@ -250,6 +272,7 @@ from app.rag.pdf import (
|
||||
extract_pdf_page_texts,
|
||||
extract_pdf_text,
|
||||
is_usable_pdf_text,
|
||||
pictorial_pages,
|
||||
render_pdf_pages,
|
||||
)
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
@@ -257,6 +280,7 @@ from app.rag.summarizer import (
|
||||
IMAGE_FALLBACK_MIME,
|
||||
IMAGE_MIMES,
|
||||
describe_image,
|
||||
describe_pdf_page,
|
||||
generate_summary,
|
||||
)
|
||||
|
||||
@@ -341,6 +365,20 @@ class ImportSummary:
|
||||
#: ``---PAGE-N---``-anchored descriptions, or ``""`` when the
|
||||
#: fallback produced nothing — the phase-126 empty-content path).
|
||||
pdfs_screenshotted: int = 0
|
||||
#: PDFs (phase 140, task 03, LOCKED A2/A4) whose USABLE text layer
|
||||
#: carried ≥1 pictorial candidate page (``app.rag.pdf.
|
||||
#: pictorial_pages`` — the deterministic sparse-page gate, LOCKED
|
||||
#: A1) whose vision description was appended to the page block
|
||||
#: (``[Page N illustration: …]``) — the sparse-page pass
|
||||
#: (:func:`_pictorial_pdf_pass`). Log-only (the sync ``detail``
|
||||
#: API shape is untouched — the stats log-only convention,
|
||||
#: phase 133); ``pdf_pictorial_pages`` counts the described pages.
|
||||
pdfs_pictorial: int = 0
|
||||
#: Pictorial candidate pages described across the run (phase 140,
|
||||
#: task 03) — one PDF with three described sparse pages counts
|
||||
#: ``pdfs_pictorial=1`` + ``pdf_pictorial_pages=3`` (A21: no page
|
||||
#: cap — the count may be large, that is the point).
|
||||
pdf_pictorial_pages: int = 0
|
||||
#: PDF-embedded image docs whose stored ``image_context`` was
|
||||
#: backfilled with the document-context block (phase 137, task 03)
|
||||
#: — the one-time best-effort pass; a success never repeats, a
|
||||
@@ -367,7 +405,8 @@ class ImportSummary:
|
||||
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
|
||||
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
|
||||
"summary_backfilled=%d dates_updated=%d images_failed=%d "
|
||||
"image_context_backfilled=%d pdfs_screenshotted=%d formats=%s",
|
||||
"image_context_backfilled=%d pdfs_screenshotted=%d "
|
||||
"pdfs_pictorial=%d pdf_pictorial_pages=%d formats=%s",
|
||||
self.files,
|
||||
self.added,
|
||||
self.updated,
|
||||
@@ -383,6 +422,8 @@ class ImportSummary:
|
||||
self.images_failed,
|
||||
self.image_context_backfilled,
|
||||
self.pdfs_screenshotted,
|
||||
self.pdfs_pictorial,
|
||||
self.pdf_pictorial_pages,
|
||||
self.format_counts(),
|
||||
)
|
||||
|
||||
@@ -1165,6 +1206,19 @@ async def _index_pdf_file(
|
||||
An unopenable PDF — or one whose fallback also produced nothing
|
||||
— yields ``""`` (still a document — it may carry embedded images
|
||||
— with no content chunks; a warning is the signal).
|
||||
* Phase 140 (task 03, LOCKED A1–A4): when the layer is USABLE and
|
||||
``settings.pdf_pictorial`` is on, the sparse-page pass
|
||||
(:func:`_pictorial_pdf_pass`) renders + vision-describes every
|
||||
page whose stripped text is under
|
||||
``settings.pdf_pictorial_min_chars`` (the deterministic gate —
|
||||
NO LLM call, A1; every candidate, no page cap — A21/A3) and
|
||||
appends the description to the page block as
|
||||
``[Page N illustration: …]`` (A2/A4). The precedence is locked:
|
||||
the pass sits AFTER the phase-133 gate, so an unusable layer
|
||||
(the fallback's full-page screenshots) never burns a second
|
||||
vision pass; zero candidates → a no-op (the extracted text
|
||||
stays byte-identical to the pre-phase-140 path — no render, no
|
||||
vision call).
|
||||
* the sha256 digest is over the FINAL content (the digest rule is
|
||||
content identity — the same text is the same document version;
|
||||
the fallback's constructed page descriptions digest exactly the
|
||||
@@ -1213,6 +1267,20 @@ async def _index_pdf_file(
|
||||
content = await _fallback_page_screenshots(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm
|
||||
)
|
||||
elif llm.settings.pdf_pictorial:
|
||||
# Phase 140 (task 03, LOCKED A1–A4): the pictorial pass —
|
||||
# USABLE layers only (the precedence is locked: the phase-133
|
||||
# fallback above already screenshotted an unusable layer, no
|
||||
# double vision). Zero candidates → a no-op (the pass returns
|
||||
# None — the extracted text stays byte-identical to the
|
||||
# pre-phase-140 path); otherwise the FINAL content carries the
|
||||
# described candidates' illustration blocks (A4: the digest
|
||||
# below runs over it, the phase-126/133 convention).
|
||||
passed = await _pictorial_pdf_pass(
|
||||
source=source, rel=rel, full_path=full_path, llm=llm, summary=summary
|
||||
)
|
||||
if passed is not None:
|
||||
content = passed
|
||||
if not content:
|
||||
logger.warning(
|
||||
"import: PDF has no extractable text source=%s path=%s", source, rel
|
||||
@@ -1406,6 +1474,128 @@ async def _fallback_page_screenshots(
|
||||
)
|
||||
|
||||
|
||||
async def _pictorial_pdf_pass(
|
||||
*,
|
||||
source: str,
|
||||
rel: str,
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
) -> str | None:
|
||||
"""The phase-140 (task 03) pictorial pass (LOCKED A1–A4) — the
|
||||
IKEA assembly-instruction class (TODO L5): a PDF whose text layer
|
||||
is USABLE but SPARSE (mostly pictorial pages — 4–73 chars of
|
||||
markers around vector-drawn pictograms) gets every sparse page
|
||||
rendered to a PNG and vision-described, and the description
|
||||
APPENDS to that page's indexed content block
|
||||
(``[Page N illustration: …]``) — the page's text stays, so the
|
||||
embeddings carry what the thin text layer couldn't (the owner's
|
||||
L5 rationale: "More information is better when generating the
|
||||
text embeddings").
|
||||
|
||||
* the gate is :func:`app.rag.pdf.pictorial_pages` over
|
||||
:func:`app.rag.pdf.extract_pdf_page_texts` — deterministic, NO
|
||||
LLM call (LOCKED A1): the 1-based page numbers whose stripped
|
||||
text is under ``settings.pdf_pictorial_min_chars``, ascending.
|
||||
(The kill-switch itself — ``settings.pdf_pictorial`` — gates the
|
||||
whole pass in the caller: while off, this function is never
|
||||
called and the content is byte-identical to pre-phase.)
|
||||
* ZERO candidates → a NO-OP (returns ``None``): no render, no
|
||||
vision call — the caller keeps its :func:`extract_pdf_text`
|
||||
output untouched, so a clean-text PDF's path is byte-identical
|
||||
to the pre-phase-140 path (the pin).
|
||||
* a page that could not be rendered (the zero-byte
|
||||
:func:`render_pdf_pages` placeholder — the index never shifts,
|
||||
so it stays aligned with the candidates) is skipped with a
|
||||
warning (the phase-133 handling — the page keeps its text only,
|
||||
no vision call is burned for it); a page whose vision call
|
||||
failed or came back blank (:func:`describe_pdf_page` →
|
||||
``None``) is skipped with a warning and keeps its text only —
|
||||
one dead page never kills the pass (LOCKED A2 fail-soft).
|
||||
* the assembly (LOCKED A4): every page of the
|
||||
:func:`extract_pdf_page_texts` map (the SAME pages, order and
|
||||
stripped text as :func:`extract_pdf_text`) as its
|
||||
:data:`PAGE_SEPARATOR` + the page text + — for the described
|
||||
candidates — ``"\\n\\n[Page {n} illustration: {description}]"``.
|
||||
The assembly with NO descriptions is byte-identical to
|
||||
:func:`extract_pdf_text`'s output (the unit pin).
|
||||
* the caller's digest runs over this FINAL content (the
|
||||
phase-126/133 convention — A4): a stable file + a stable vision
|
||||
reply re-syncs as ``unchanged``; a non-deterministic re-describe
|
||||
re-updates the row — a cost, not a bug (the phase-133 docstring
|
||||
documents it, A21 endorses it).
|
||||
* stats (log-only — the sync ``detail`` API shape is untouched,
|
||||
the phase-133 convention): ``summary.pdfs_pictorial`` (+1 when
|
||||
≥1 page was described) and ``summary.pdf_pictorial_pages``
|
||||
(+the described page count), plus the ONE loud log line (PLAN
|
||||
§9): ``pdf: pictorial pass source=… path=… — N candidate pages,
|
||||
M described (min_chars=…)``.
|
||||
"""
|
||||
settings = llm.settings
|
||||
page_texts = extract_pdf_page_texts(full_path)
|
||||
candidates = pictorial_pages(
|
||||
page_texts, min_chars=settings.pdf_pictorial_min_chars
|
||||
)
|
||||
if not candidates:
|
||||
return None
|
||||
# ONE render for the whole candidate set (the phase-140 selector —
|
||||
# a 500-page PDF with 30 sparse pages renders 30 PNGs, not 500);
|
||||
# the result is aligned with *candidates* (ascending, one slot
|
||||
# each — a zero-byte placeholder never shifts the positions).
|
||||
pngs = render_pdf_pages(full_path, pages=candidates)
|
||||
descriptions: dict[int, str] = {}
|
||||
# strict: the render's result is aligned with the candidate list by
|
||||
# contract (one slot per named page, ascending) — a length drift is
|
||||
# a parser regression, fail loud.
|
||||
for page, png in zip(candidates, pngs, strict=True):
|
||||
if not png:
|
||||
# The phase-133 zero-byte placeholder: an unreadable page.
|
||||
logger.warning(
|
||||
"pdf: skipping pictorial page %d of %s/%s — the render is "
|
||||
"empty (unreadable page)",
|
||||
page, source, rel,
|
||||
)
|
||||
continue
|
||||
desc = await describe_pdf_page(
|
||||
llm,
|
||||
data=png,
|
||||
mime="image/png",
|
||||
page=page,
|
||||
page_text=page_texts[page],
|
||||
)
|
||||
if desc is None:
|
||||
# Fail-soft (LOCKED A2): one dead page never kills the
|
||||
# pass — the page keeps its text only, the sync continues.
|
||||
logger.warning(
|
||||
"pdf: pictorial page %d of %s/%s has no description — "
|
||||
"the page keeps its text only",
|
||||
page, source, rel,
|
||||
)
|
||||
continue
|
||||
descriptions[page] = desc
|
||||
# The ONE loud log line (PLAN §9): N candidate pages, M described.
|
||||
logger.info(
|
||||
"pdf: pictorial pass source=%s path=%s — %d candidate pages, %d "
|
||||
"described (min_chars=%d)",
|
||||
source, rel, len(candidates), len(descriptions),
|
||||
settings.pdf_pictorial_min_chars,
|
||||
)
|
||||
# The stats (log-only — the sync ``detail`` API shape is untouched).
|
||||
if descriptions:
|
||||
summary.pdfs_pictorial += 1
|
||||
summary.pdf_pictorial_pages += len(descriptions)
|
||||
# The assembly: the extracted text's page blocks, the described
|
||||
# candidates carrying their illustration block (A4 — the page's
|
||||
# text STAYS; the description appends after it).
|
||||
parts: list[str] = []
|
||||
for page, text in page_texts.items():
|
||||
block = text
|
||||
if page in descriptions:
|
||||
block += f"\n\n[Page {page} illustration: {descriptions[page]}]"
|
||||
parts.append(f"{PAGE_SEPARATOR.format(page=page)}\n{block}")
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
async def _index_pdf_embedded_images(
|
||||
session: Session,
|
||||
*,
|
||||
|
||||
+82
-17
@@ -29,11 +29,20 @@ package, imported as ``fitz`` — its stable API name).
|
||||
I/O) — REAL content vs. garbage (the pre-126 XML/binary soup, the
|
||||
empty text layer). It is the single trigger for the page-screenshot
|
||||
vision fallback (phase 133 task 03).
|
||||
* :func:`pictorial_pages` — the phase-140 sparse-page gate: the 1-based
|
||||
page numbers of a :func:`extract_pdf_page_texts` map whose stripped
|
||||
text is STRICTLY under *min_chars* chars (ascending) — pure,
|
||||
deterministic, NO LLM call (LOCKED A1): the "way to determine when
|
||||
that's necessary" (TODO L5) for the pictorial pass.
|
||||
* :func:`render_pdf_pages` — the fallback's visual input (phase 133
|
||||
task 03): one PNG per page (150 DPI, page order); an unreadable page
|
||||
yields ``b""`` (the caller's placeholder — the page index never
|
||||
shifts), an unopenable file yields ``[]`` (the caller's existing
|
||||
empty-content path).
|
||||
task 03): one PNG per page (150 DPI, page order); the phase-140
|
||||
optional 1-based *pages* selector renders ONLY the named pages
|
||||
(ascending — a 500-page PDF with 30 sparse pages renders 30 PNGs,
|
||||
not 500), ``None`` = every page (the phase-133 caller's result is
|
||||
byte-identical); an unreadable (or out-of-range named) page yields
|
||||
``b""`` at its position (the caller's placeholder — the result
|
||||
stays aligned with the requested pages), an unopenable file yields
|
||||
``[]`` (the caller's existing empty-content path).
|
||||
|
||||
Every failure mode is fail-soft: an unreadable file returns an empty
|
||||
result, an unreadable page or image is logged and skipped — this module
|
||||
@@ -43,7 +52,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from collections.abc import Iterator, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -264,15 +273,60 @@ def is_usable_pdf_text(text: str) -> bool:
|
||||
return non_printable / len(text) < NONPRINT_CHAR_FRACTION
|
||||
|
||||
|
||||
def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
|
||||
"""Render every page of *pdf_path* to a PNG (page order — list index
|
||||
*i* is page *i*+1).
|
||||
def pictorial_pages(page_texts: dict[int, str], *, min_chars: int) -> list[int]:
|
||||
"""The 1-based page numbers of *page_texts* that are PICTORIAL
|
||||
candidates: ``len(text.strip()) < min_chars`` — STRICTLY less than
|
||||
(a page with exactly *min_chars* stripped chars is NOT sparse; the
|
||||
boundary is pinned in the unit tests), ascending.
|
||||
|
||||
The deterministic sparse-page gate (phase 140, task 01, LOCKED A1)
|
||||
— the "way to determine when that's necessary" for the pictorial PDF
|
||||
pass (TODO L5): NO LLM call, pure, deterministic (no PDF, no I/O —
|
||||
the input is a :func:`extract_pdf_page_texts` map, so the gate never
|
||||
re-opens the file). A page that carries no text at all has NO key in
|
||||
the map and is therefore not a candidate here (the importer's pass
|
||||
owns the page-total handling); an empty map yields ``[]`` — a
|
||||
clean-text PDF's path never burns a vision call.
|
||||
|
||||
The IKEA example (the phase-140 fixture, ``min_chars=100``):
|
||||
``{1: "BESTÅ", 2: <7 KB of warnings>, 6: "Illustration 6"}`` →
|
||||
``[1, 6, …]`` — page 1 (the title page: the logo + product name) IS
|
||||
a candidate: describing it is useful (the logo + product name) and
|
||||
never harmful; page 2's 7 KB of multilingual warnings is not.
|
||||
|
||||
``min_chars=0`` is the documented PLAN A21 special case — **every
|
||||
page in the map is a candidate** (the "describe everything" mode).
|
||||
The strict-``<`` test cannot express this (no length is below 0), so
|
||||
0 is not "a stricter bar": it is the A21 no-cap endpoint, and a
|
||||
negative value — unreachable through ``Settings`` (the validator
|
||||
fails startup loudly) — takes the same branch defensively.
|
||||
"""
|
||||
if min_chars <= 0:
|
||||
return sorted(page_texts)
|
||||
return sorted(
|
||||
page for page, text in page_texts.items() if len(text.strip()) < min_chars
|
||||
)
|
||||
|
||||
|
||||
def render_pdf_pages(
|
||||
pdf_path: Path, dpi: int = 150, pages: Sequence[int] | None = None
|
||||
) -> list[bytes]:
|
||||
"""Render the selected pages of *pdf_path* to PNGs.
|
||||
|
||||
*pages* (phase 140, task 02): the optional 1-based selector —
|
||||
render ONLY the named pages, in ascending order, one PNG each
|
||||
(a 500-page PDF with 30 pictorial candidates renders 30 PNGs, not
|
||||
500). ``None`` (the default) = every page in page order — the
|
||||
phase-133 caller's result is byte-identical, so list index *i* is
|
||||
page *i*+1 (with a selector, list index *i* is the *i*-th smallest
|
||||
NAMED page).
|
||||
|
||||
The fallback's visual input (phase 133 task 03, ASSUMPTION A8:
|
||||
fixed 150 DPI — the balance of vision legibility vs data-URL
|
||||
payload). A page that cannot be rendered yields ``b""`` (the
|
||||
caller's placeholder — the page index NEVER shifts, so
|
||||
``---PAGE-N---`` stays truthful; the failure is logged, the
|
||||
payload). A page that cannot be rendered — including a named page
|
||||
outside the document's page range — yields ``b""`` at ITS position
|
||||
(the caller's placeholder — the result's order NEVER shifts, so it
|
||||
stays aligned with the requested pages; the failure is logged, the
|
||||
:func:`extract_pdf_text` fail-soft style). A file that cannot be
|
||||
OPENED yields ``[]`` (the caller's existing empty-content path —
|
||||
the doc lands with the no-extractable-text behavior, unchanged).
|
||||
@@ -283,18 +337,29 @@ def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
|
||||
logger.warning("pdf: could not open %s for page rendering — %s", pdf_path, exc)
|
||||
return []
|
||||
|
||||
pages: list[bytes] = []
|
||||
# 1-based page numbers: the default walks every page in page order
|
||||
# (the phase-133 contract — list index *i* is page *i*+1); a
|
||||
# selector renders the named pages, ascending (de-duplicated — a
|
||||
# repeated page number would render it twice for no reason).
|
||||
numbers = (
|
||||
range(1, len(doc) + 1)
|
||||
if pages is None
|
||||
else sorted(set(pages))
|
||||
)
|
||||
rendered: list[bytes] = []
|
||||
with doc:
|
||||
for page_num in range(len(doc)):
|
||||
for page_number in numbers:
|
||||
try:
|
||||
pages.append(doc[page_num].get_pixmap(dpi=dpi).tobytes("png"))
|
||||
rendered.append(
|
||||
doc[page_number - 1].get_pixmap(dpi=dpi).tobytes("png")
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"pdf: could not render page %d of %s — %s",
|
||||
page_num + 1, pdf_path, exc,
|
||||
page_number, pdf_path, exc,
|
||||
)
|
||||
pages.append(b"")
|
||||
return pages
|
||||
rendered.append(b"")
|
||||
return rendered
|
||||
|
||||
|
||||
def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
|
||||
|
||||
+143
-9
@@ -33,10 +33,24 @@ it is not assumed vision-capable). Fail-soft by contract: any client
|
||||
error, empty reply, or non-2xx yields ``None`` — the importer skips
|
||||
the doc, counts ``images_failed``, and the sync continues.
|
||||
|
||||
The ``SUMMARY_MODE`` / ``IMAGE_DESCRIPTION_MODE`` markers follow the
|
||||
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on
|
||||
them (``tests/e2e/mock_llm.py`` — the image branch is wired by task
|
||||
06's story suite).
|
||||
Pictorial PDF pages (phase 140, LOCKED A2): :func:`describe_pdf_page`
|
||||
is this module's third one-shot generation path — a SINGLE CHAT-model
|
||||
(vision) call describing ONE rendered sparse PDF page, with the
|
||||
page's OWN extracted text as context so the model describes what the
|
||||
text does NOT carry (the IKEA assembly-instruction class: vector
|
||||
pictograms around a 4–73-char text layer). The description is
|
||||
APPENDED to the page's indexed content block by the importer (the
|
||||
``[Page N illustration: …]`` line, LOCKED A4) — the page's text stays;
|
||||
more information is better for the embeddings (the owner's L5
|
||||
rationale). Fail-soft by contract: any client error or empty reply
|
||||
yields ``None`` — the importer skips that page with a warning and
|
||||
continues (one dead page never stalls a sync).
|
||||
|
||||
The ``SUMMARY_MODE`` / ``IMAGE_DESCRIPTION_MODE`` /
|
||||
``PICTORIAL_PAGE_MODE`` markers follow the ``DEFLECT_MODE``
|
||||
convention: the deterministic E2E mock LLM keys on them
|
||||
(``tests/e2e/mock_llm.py`` — the image branch is wired by task
|
||||
06's story suite, the pictorial branch by phase 140 task 02).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -92,8 +106,41 @@ DESCRIBE_INSTRUCTION = (
|
||||
|
||||
#: The full describe prompt: marker first (the mock's key), then the
|
||||
#: instruction — the single text part of the multimodal user message.
|
||||
#: BYTE-IDENTICAL CONTRACT: the phase 122/126/137 suites pin this
|
||||
#: string untouched — pictorial-page descriptions go through their
|
||||
#: own marker (:data:`PICTORIAL_PAGE_MODE`), never a re-wrap of this
|
||||
#: one.
|
||||
DESCRIBE_PROMPT = f"{IMAGE_DESCRIPTION_MODE}: {DESCRIBE_INSTRUCTION}"
|
||||
|
||||
#: Pictorial-page marker (phase 140, task 02, LOCKED A1/A2) — the
|
||||
#: deterministic E2E mock LLM keys on it (same convention as
|
||||
#: ``IMAGE_DESCRIPTION_MODE``, PLAN §6). It heads the text part of
|
||||
#: the multimodal describe message, so it rides the user message, not
|
||||
#: a system prompt.
|
||||
PICTORIAL_PAGE_MODE = "PICTORIAL_PAGE_MODE"
|
||||
|
||||
#: Locked instruction for the CHAT (vision) model (phase 140, LOCKED
|
||||
#: A2): the description APPENDS to the page's indexed content block
|
||||
#: (``[Page N illustration: …]``) — the embeddings then carry what
|
||||
#: the thin text layer couldn't (the owner's L5 rationale) — so it
|
||||
#: must be a faithful, retrieval-oriented account of what the page
|
||||
#: SHOWS: the assembly steps, part names, arrows, tool icons, and
|
||||
#: before/after states — ESPECIALLY the information the extracted
|
||||
#: text does not contain.
|
||||
PICTORIAL_INSTRUCTION = (
|
||||
"Describe in detail what this page of the PDF shows, in plain text, "
|
||||
"for a search index. State the assembly steps, part names, arrows, "
|
||||
"tool icons, and before/after states it illustrates — ESPECIALLY the "
|
||||
"information the extracted text does not contain. Write 2-4 sentences "
|
||||
"of substance. Do not use markdown. Do not invent anything that is "
|
||||
"not visible on the page."
|
||||
)
|
||||
|
||||
#: The full pictorial-page prompt: marker first (the mock's key), then
|
||||
#: the instruction — the head of the multimodal user message's text
|
||||
#: part (the page block is appended by :func:`describe_pdf_page`).
|
||||
PICTORIAL_PROMPT = f"{PICTORIAL_PAGE_MODE}: {PICTORIAL_INSTRUCTION}"
|
||||
|
||||
#: Extension → MIME type for the image family (phase 122). Dotted,
|
||||
#: lowercase keys — the ``image_extension_set`` shape. Task 03 uses it
|
||||
#: for the describe call's data-URL mime; task 04's serve route reuses
|
||||
@@ -266,11 +313,43 @@ async def describe_image(
|
||||
if context
|
||||
else ""
|
||||
)
|
||||
return await _one_shot_vision_reply(
|
||||
llm,
|
||||
data_url=data_url,
|
||||
text=DESCRIBE_PROMPT + context_prefix,
|
||||
settings=settings,
|
||||
what="image description",
|
||||
)
|
||||
|
||||
|
||||
async def _one_shot_vision_reply(
|
||||
llm: SummaryLLM,
|
||||
*,
|
||||
data_url: str,
|
||||
text: str,
|
||||
settings: Settings | None,
|
||||
what: str,
|
||||
) -> str | None:
|
||||
"""The shared one-chat-call multimodal contract (phase 122, reused
|
||||
by phase 140): ONE user message
|
||||
``[{type: "text", text: <text>},
|
||||
{type: "image_url", image_url: {url: <data_url>}}]`` against
|
||||
``llm.settings.llm_chat_model`` — no system prompt, no tools, no
|
||||
app-level retry.
|
||||
|
||||
Returns the stripped reply cut exactly at
|
||||
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap);
|
||||
``None`` on any client error (the client raises :class:`LLMError`
|
||||
for a failure, an empty reply, or a non-2xx) or an empty/blank
|
||||
reply after trimming (the house one-shot empty-content policy — the
|
||||
duck-typed fakes may return one even though the real client
|
||||
rejects it). *what* names the caller for the warning log line.
|
||||
"""
|
||||
messages: list[dict[str, Any]] = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": DESCRIBE_PROMPT + context_prefix},
|
||||
{"type": "text", "text": text},
|
||||
{"type": "image_url", "image_url": {"url": data_url}},
|
||||
],
|
||||
}
|
||||
@@ -279,12 +358,67 @@ async def describe_image(
|
||||
try:
|
||||
raw = await llm.chat(messages, model=model)
|
||||
except LLMError as e:
|
||||
logger.warning("image description failed (model=%s): %s", model, e)
|
||||
logger.warning("%s failed (model=%s): %s", what, model, e)
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
text_out = raw.strip()
|
||||
if not text_out:
|
||||
# The client already rejects empty content; this is the
|
||||
# defensive re-assert (the duck-typed fakes may return it).
|
||||
return None
|
||||
limit = (settings or llm.settings).summary_max_chars
|
||||
return text[:limit]
|
||||
return text_out[:limit]
|
||||
|
||||
|
||||
async def describe_pdf_page(
|
||||
llm: SummaryLLM,
|
||||
*,
|
||||
data: bytes,
|
||||
mime: str,
|
||||
page: int,
|
||||
page_text: str | None,
|
||||
settings: Settings | None = None,
|
||||
) -> str | None:
|
||||
"""One-shot CHAT-model (vision) description of ONE rendered PDF page
|
||||
(phase 140, task 02, LOCKED A2) — the text the importer appends to
|
||||
that page's indexed content block as
|
||||
``[Page N illustration: …]`` (the page's own text stays; the
|
||||
embeddings then carry what the thin text layer couldn't — the
|
||||
owner's L5 rationale).
|
||||
|
||||
ONE chat-model call against ``llm.settings.llm_chat_model`` (the
|
||||
vision model — the ``lite`` summary model is NOT assumed
|
||||
vision-capable) with the same multimodal contract as
|
||||
:func:`describe_image` (via :func:`_one_shot_vision_reply`): no
|
||||
system prompt, no tools, no app-level retry. The message's text
|
||||
part is :data:`PICTORIAL_PROMPT` (the ``PICTORIAL_PAGE_MODE``
|
||||
marker + the locked instruction) followed by the page block — the
|
||||
page's OWN extracted text as context, so the model describes what
|
||||
the text does NOT contain::
|
||||
|
||||
PICTORIAL_PROMPT
|
||||
+ "\\n\\nPage {page} of the PDF. Text extracted from this page:\n"
|
||||
+ (page_text.strip() or "(none)")
|
||||
|
||||
``data`` is the rendered page's PNG bytes (phase 133's
|
||||
``render_pdf_pages`` output), ``mime`` its MIME type (``image/png``
|
||||
for the rendered page).
|
||||
|
||||
Returns the stripped reply cut exactly at
|
||||
``(settings or llm.settings).summary_max_chars`` (the phase-30 cap).
|
||||
Returns ``None`` on any client error, empty reply, or non-2xx
|
||||
(fail-soft — the importer skips THAT page with a warning and
|
||||
continues: one dead page never stalls a sync, LOCKED A2).
|
||||
"""
|
||||
data_url = image_data_url(data, mime)
|
||||
extracted = page_text.strip() if page_text is not None else ""
|
||||
return await _one_shot_vision_reply(
|
||||
llm,
|
||||
data_url=data_url,
|
||||
text=(
|
||||
PICTORIAL_PROMPT
|
||||
+ f"\n\nPage {page} of the PDF. Text extracted from this page:\n"
|
||||
+ (extracted or "(none)")
|
||||
),
|
||||
settings=settings,
|
||||
what=f"pictorial page description (page={page})",
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user