phase: 140_pictorial_pdf_pages

Everything is verified green. Final report: ## Phase 140 final verification pass — report - Verified all 4 tasks already complete (code in working tree, task files in `complete/`): `pictorial_pages` gate + 2 settings (`BOR_PDF_PICTORIAL`, `BOR_PDF_PICTORIAL_MIN_CHARS` + validator), `PICTORIAL_PAGE_MODE`/`describe_pdf_page` seam + `render_pdf_pages` page selector + mock branch, `_pictorial_pdf_pass` in `_index_pdf_file` with log-only counters, dedicated E2E. No code changes needed. - Test results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → **3320 passed, 11 skipped, 0 failed; TOTAL 99%** (>90% gate) - `uv run pytest tests/e2e/test_pictorial_pdf.py -v --no-cov` → **3 passed in isolation** (Playwright, mock-only) - Pre-existing suites unedited: `test_pdf_support.py` 6 passed; `test_garbage_pdf.py` 2 passed + 1 skipped (owner fixture); `test_pdf_image_context.py` 4 passed - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors - Completion criteria: 1. IKEA tar import repro — pinned by 2 integration tests; **skipped loudly here** (`test_sources/ikea_instructions.tar` absent on this machine — verified repo-wide; same house convention as completed phase-133 HNB tests; runs real on the owner's machine). Equivalent logic fully covered by unit pins on synthetic IKEA-class (vector-drawn, zero-raster) PDFs. 2. Chat grounded on described page — **green** (E2E test b: phrase-only-in-illustration question, PDF in `done.sources`, answer echoes the block) 3. Clean-text no-op (zero vision calls) + garbage precedence pin — **green** (`test_zero_candidates_is_a_noop` unit; `test_garbage_pdf_with_pictorial_on_takes_the_fallback_not_the_pass` integration, verified alone) 4. `BOR_PDF_PICTORIAL=0` byte-identical — **green** (unit pin on synthetic PDF; IKEA variant runs on owner's machine) 5. Full gate + E2E isolation + pre-existing suites — **green** (above) 6. ruff + pyright — **clean** 7. Commit/phase-move — delegated to harness per run rules (left in working tree: 16 modified + 8 new files, uncommitted) - Deviations: none beyond the expected owner-fixture skips (11, all pre-existing fixture-missing skips, including 2 IKEA). - Next pending phase: **none** — `todo/` holds only 140; 139 is already complete.
2026-09-30 19:16:35 -04:00
parent a86544a083
commit 9633eb7653
45 changed files with 3895 additions and 36 deletions
@@ -29,11 +29,20 @@ package, imported as ``fitz`` — its stable API name).
  I/O) — REAL content vs. garbage (the pre-126 XML/binary soup, the
  empty text layer). It is the single trigger for the page-screenshot
  vision fallback (phase 133 task 03).
+* :func:`pictorial_pages` — the phase-140 sparse-page gate: the 1-based
+  page numbers of a :func:`extract_pdf_page_texts` map whose stripped
+  text is STRICTLY under *min_chars* chars (ascending) — pure,
+  deterministic, NO LLM call (LOCKED A1): the "way to determine when
+  that's necessary" (TODO L5) for the pictorial pass.
 * :func:`render_pdf_pages` — the fallback's visual input (phase 133
-  task 03): one PNG per page (150 DPI, page order); an unreadable page
-  yields ``b""`` (the caller's placeholder — the page index never
-  shifts), an unopenable file yields ``[]`` (the caller's existing
-  empty-content path).
+  task 03): one PNG per page (150 DPI, page order); the phase-140
+  optional 1-based *pages* selector renders ONLY the named pages
+  (ascending — a 500-page PDF with 30 sparse pages renders 30 PNGs,
+  not 500), ``None`` = every page (the phase-133 caller's result is
+  byte-identical); an unreadable (or out-of-range named) page yields
+  ``b""`` at its position (the caller's placeholder — the result
+  stays aligned with the requested pages), an unopenable file yields
+  ``[]`` (the caller's existing empty-content path).

 Every failure mode is fail-soft: an unreadable file returns an empty
 result, an unreadable page or image is logged and skipped — this module
@@ -43,7 +52,7 @@ from __future__ import annotations

 import logging
 import re
-from collections.abc import Iterator
+from collections.abc import Iterator, Sequence
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
@@ -264,15 +273,60 @@ def is_usable_pdf_text(text: str) -> bool:
    return non_printable / len(text) < NONPRINT_CHAR_FRACTION


-def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
-    """Render every page of *pdf_path* to a PNG (page order — list index
-    *i* is page *i*+1).
+def pictorial_pages(page_texts: dict[int, str], *, min_chars: int) -> list[int]:
+    """The 1-based page numbers of *page_texts* that are PICTORIAL
+    candidates: ``len(text.strip()) < min_chars`` — STRICTLY less than
+    (a page with exactly *min_chars* stripped chars is NOT sparse; the
+    boundary is pinned in the unit tests), ascending.
+
+    The deterministic sparse-page gate (phase 140, task 01, LOCKED A1)
+    — the "way to determine when that's necessary" for the pictorial PDF
+    pass (TODO L5): NO LLM call, pure, deterministic (no PDF, no I/O —
+    the input is a :func:`extract_pdf_page_texts` map, so the gate never
+    re-opens the file). A page that carries no text at all has NO key in
+    the map and is therefore not a candidate here (the importer's pass
+    owns the page-total handling); an empty map yields ``[]`` — a
+    clean-text PDF's path never burns a vision call.
+
+    The IKEA example (the phase-140 fixture, ``min_chars=100``):
+    ``{1: "BESTÅ", 2: <7 KB of warnings>, 6: "Illustration 6"}`` →
+    ``[1, 6, …]`` — page 1 (the title page: the logo + product name) IS
+    a candidate: describing it is useful (the logo + product name) and
+    never harmful; page 2's 7 KB of multilingual warnings is not.
+
+    ``min_chars=0`` is the documented PLAN A21 special case — **every
+    page in the map is a candidate** (the "describe everything" mode).
+    The strict-``<`` test cannot express this (no length is below 0), so
+    0 is not "a stricter bar": it is the A21 no-cap endpoint, and a
+    negative value — unreachable through ``Settings`` (the validator
+    fails startup loudly) — takes the same branch defensively.
+    """
+    if min_chars <= 0:
+        return sorted(page_texts)
+    return sorted(
+        page for page, text in page_texts.items() if len(text.strip()) < min_chars
+    )
+
+
+def render_pdf_pages(
+    pdf_path: Path, dpi: int = 150, pages: Sequence[int] | None = None
+) -> list[bytes]:
+    """Render the selected pages of *pdf_path* to PNGs.
+
+    *pages* (phase 140, task 02): the optional 1-based selector —
+    render ONLY the named pages, in ascending order, one PNG each
+    (a 500-page PDF with 30 pictorial candidates renders 30 PNGs, not
+    500). ``None`` (the default) = every page in page order — the
+    phase-133 caller's result is byte-identical, so list index *i* is
+    page *i*+1 (with a selector, list index *i* is the *i*-th smallest
+    NAMED page).

    The fallback's visual input (phase 133 task 03, ASSUMPTION A8:
    fixed 150 DPI — the balance of vision legibility vs data-URL
-    payload). A page that cannot be rendered yields ``b""`` (the
-    caller's placeholder — the page index NEVER shifts, so
-    ``---PAGE-N---`` stays truthful; the failure is logged, the
+    payload). A page that cannot be rendered — including a named page
+    outside the document's page range — yields ``b""`` at ITS position
+    (the caller's placeholder — the result's order NEVER shifts, so it
+    stays aligned with the requested pages; the failure is logged, the
    :func:`extract_pdf_text` fail-soft style). A file that cannot be
    OPENED yields ``[]`` (the caller's existing empty-content path —
    the doc lands with the no-extractable-text behavior, unchanged).
@@ -283,18 +337,29 @@ def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
        logger.warning("pdf: could not open %s for page rendering — %s", pdf_path, exc)
        return []

-    pages: list[bytes] = []
+    # 1-based page numbers: the default walks every page in page order
+    # (the phase-133 contract — list index *i* is page *i*+1); a
+    # selector renders the named pages, ascending (de-duplicated — a
+    # repeated page number would render it twice for no reason).
+    numbers = (
+        range(1, len(doc) + 1)
+        if pages is None
+        else sorted(set(pages))
+    )
+    rendered: list[bytes] = []
    with doc:
-        for page_num in range(len(doc)):
+        for page_number in numbers:
            try:
-                pages.append(doc[page_num].get_pixmap(dpi=dpi).tobytes("png"))
+                rendered.append(
+                    doc[page_number - 1].get_pixmap(dpi=dpi).tobytes("png")
+                )
            except Exception as exc:
                logger.warning(
                    "pdf: could not render page %d of %s — %s",
-                    page_num + 1, pdf_path, exc,
+                    page_number, pdf_path, exc,
                )
-                pages.append(b"")
-    return pages
+                rendered.append(b"")
+    return rendered


 def extract_pdf_images(pdf_path: Path) -> list[PdfImage]: