phase: 151_pdf_image_extraction

The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.

All verification is complete. Final state:

**Phase 151 — final verification pass: complete, project 100% green.**

- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`

app/rag/importer.py

@@ -317,7 +317,7 @@ from app.rag.md_images import ImageRef, extract_image_refs
 from app.rag.pdf import (
     PAGE_SEPARATOR,
     PdfImage,
-    extract_pdf_images,
+    extract_pdf_images_with_stats,
     extract_pdf_page_texts,
     extract_pdf_text,
     is_usable_pdf_text,
@@ -433,6 +433,24 @@ class ImportSummary:
     #: — the one-time best-effort pass; a success never repeats, a
     #: failure retries next sync.
     image_context_backfilled: int = 0
+    #: Image OCCURRENCES the PDF parser's dedupe dropped (phase 151,
+    #: tasks 01/03) — repeated placements + byte-identical copies
+    #: stored under distinct xrefs (one :class:`app.rag.pdf.PdfImage`
+    #: per DISTINCT image, first placement wins). Log-only (the sync
+    #: ``detail`` API shape is untouched — the stats log-only
+    #: convention, phase 133): the owner reads it in the sync log.
+    pdf_images_deduped: int = 0
+    #: DISTINCT PDF-embedded images the parser's blank/decorative gate
+    #: dropped (phase 151, task 03 —
+    #: :func:`app.rag.pdf.is_blank_image` on the base samples): a
+    #: near-white, nearly-uniform underlay (the TODO L4 "blank white
+    #: images") is skipped ENTIRELY — no row, no ``image_dir`` copy,
+    #: no vision call (ASSUMPTION A1). Log-only (the sync ``detail``
+    #: API shape is untouched — the stats log-only convention,
+    #: phase 133); while ``Settings.pdf_image_skip_blank`` (phase 151
+    #: task 04, the kill switch) is ``False`` the gate is off and this
+    #: count stays 0 (pre-phase behavior).
+    pdf_images_blank_skipped: int = 0
     #: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
     #: tuples in encounter order (added/updated in walk order, pruned in
     #: _prune order). Drives the sync's incremental folder-summary
@@ -455,7 +473,8 @@ class ImportSummary:
             "errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
             "summary_backfilled=%d dates_updated=%d images_failed=%d "
             "image_context_backfilled=%d pdfs_screenshotted=%d "
-            "pdfs_pictorial=%d pdf_pictorial_pages=%d formats=%s",
+            "pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
+            "pdf_images_blank_skipped=%d formats=%s",
             self.files,
             self.added,
             self.updated,
@@ -473,6 +492,8 @@ class ImportSummary:
             self.pdfs_screenshotted,
             self.pdfs_pictorial,
             self.pdf_pictorial_pages,
+            self.pdf_images_deduped,
+            self.pdf_images_blank_skipped,
             self.format_counts(),
         )
 
@@ -1934,10 +1955,21 @@ async def _index_pdf_embedded_images(
     """The phase-126 (A3) embedded-image pass — one call per (re)indexed
     PDF, after its document commit.
 
-    Each embedded image (page order, 0-based index on the page — the
-    parser's order) is indexed as a STANDALONE image document through
-    the phase-122 mechanics, inline (no temp file, no
-    ``_index_image_file`` refactor):
+    Each DISTINCT embedded image (the parser dedupes repeated
+    placements and byte-identical copies — phase 151 task 01 — first
+    placement wins; page order, 0-based index on the page — the
+    parser's order; the parser's blank/decorative gate — phase 151
+    task 03, :func:`app.rag.pdf.is_blank_image` on the base samples —
+    drops near-white underlays BEFORE they reach this pass: no row,
+    no copy, no vision call) is indexed as a STANDALONE image document
+    through the phase-122 mechanics, inline (no temp file, no
+    ``_index_image_file`` refactor): the gate is ON by default and
+    ``Settings.pdf_image_skip_blank`` (``BOR_PDF_IMAGE_SKIP_BLANK`` —
+    phase 151 task 04, the kill switch, read HERE — the one place —
+    and passed down as the extractor's ``skip_blank`` keyword) is its
+    single knob: ``false`` restores indexing every extracted image
+    (the pre-phase-151 behavior); dedupe and soft-mask compositing are
+    UNAFFECTED by the knob (they are correctness fixes).
 
     * gated by the phase-122 ``images`` toggle (the single knob, LOCKED
       A3): while it is OFF the whole pass is skipped (one debug line —
@@ -1947,12 +1979,19 @@ async def _index_pdf_embedded_images(
       (:data:`_MIME_TO_EXT`) is skipped with a warning (the unsupported-
       extension rule, same as a standalone image's);
     * the document's synthetic ``path`` is
-      ``<pdf path>#page=<N>#img=<index>.<ext>`` — unique per image
-      occurrence, stable across re-indexes (the upsert key); the title
-      is ``Embedded image from <pdf stem> page <N>``;
+      ``<pdf path>#page=<N>#img=<index>.<ext>`` — N/index are the
+      FIRST placement's (the parser's dedupe, phase 151 task 01),
+      unique per distinct image, stable across re-indexes (the upsert
+      key); the title is ``Embedded image from <pdf stem> page <N>``;
     * the digest is sha256 over the image BYTES (the digest rule is
       content identity); an image doc whose digest matches is skipped
-      (no re-describe, no re-embed — unchanged);
+      (no re-describe, no re-embed — unchanged). Phase 151 (task 02)
+      consequence, verified: a MASKED image's bytes change with the
+      smask compositing (base stream → alpha PNG), so pre-phase rows
+      for masked images re-describe ONCE on the first post-phase sync
+      (their sha256 digest differs) — expected, self-limiting, no
+      migration; unmasked images' bytes are untouched, so their rows
+      stay digest-skipped as before;
     * the description comes through the SAME single seam
       (:func:`_describe_or_skip` — the bytes as a data URL whose mime
       comes from the copy path's extension); a failed/empty description
@@ -2018,7 +2057,18 @@ async def _index_pdf_embedded_images(
     # task-03 backfill probes the parent with the SAME helper when it
     # is still in the source, so the two paths can never drift.
     page_texts, text_usable = _pdf_image_context_inputs(full_path)
-    images = extract_pdf_images(full_path)
+    # Phase 151 (task 03): the stats-carrying variant — the parser's
+    # per-PDF drop counts (dedupe + blank gate) land on the run's
+    # summary (log-only: the sync ``detail`` API shape is untouched).
+    # Phase 151 (task 04): the knob is read HERE — the one place —
+    # and passed down as the extractor's `skip_blank` keyword
+    # (default True; `false` = every extracted image indexed, the
+    # pre-phase behavior; dedupe + compositing are unaffected).
+    images, stats = extract_pdf_images_with_stats(
+        full_path, skip_blank=settings.pdf_image_skip_blank
+    )
+    summary.pdf_images_deduped += stats.duplicates
+    summary.pdf_images_blank_skipped += stats.blank_skipped
     for img in images:
         await _index_one_embedded_image(
             session, img, source=source, rel=rel, full_path=full_path,