brain-of-reese/.agents/phases/complete/151_pdf_image_extraction/00_phase.md

Phase 151 — no more blank white image documents from PDFs (dedupe + soft masks + a blank gate)

Source: TODO.md L4 — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?" Story: n/a (TODO-derived phase, Protocol B append). Context: app/rag/pdf.py — extract_pdf_images() (the module's image extractor: page.get_image_info(xrefs=True) per page via _page_image_entries, then doc.extract_image(xref) per entry, one PdfImage(page, index, width, height, data, mime) per occurrence) and its consumer app/rag/importer.py _index_pdf_embedded_images() (L1924 — the per-PDF embedded-image pass: _pdf_image_context_inputs once, then extract_pdf_images(full_path) and one _index_one_embedded_image() (L2030) per image: mime→ext gate through _MIME_TO_EXT (L1340 — png/jpg/gif/webp/bmp/tiff; anything else maps to .bin and is skipped), sha256-digest skip, one vision describe call (_describe_or_skip, L2907), the image_dir copy, chunk + embed, one transaction per image). The images toggle (BOR_IMAGES, settings.images, default OFF) gates the whole pass (LOCKED A3). ImportSummary (L390–475) carries the log-only counters (images_failed, pdfs_screenshotted, pdfs_pictorial, pdf_pictorial_pages, image_context_backfilled) — the phase-133 stats log-only convention: counters land in the import: summary … line, never in the sync detail API shape.

Reproduced on the owner's file (test_sources/podman-systemd.pdf, 50 pages) — three distinct defects:

1. Duplicates: get_image_info reports one entry per placement, so the file yields 46 occurrences of only 15 unique xrefs (each image is placed 2–4 times). Every occurrence becomes its own document (…#page=N#img=i.<ext>), so the same image is vision-described and embedded up to 4×.
2. Soft masks ignored: every one of the 15 images carries an /SMask (extract_image reports smask=<xref>, get_image_info reports has-mask=True). extract_image() returns the base sample stream only — the mask (which carries the visible shape) is dropped, so the stored bytes are not what a PDF viewer shows.
3. Decorative underlays indexed: those images are table-shading underlays — DeviceGray, mean=238/255, 8 unique sample values, 99.5% of samples within ±8 of the mean (verified: composited with their mask they are still a near-white block; the table text a viewer shows is real PDF text drawn on top). They are indexed as documents, vision-described, and returned as image sources.

Measured comparison (the same probe, fitz.Pixmap(...).samples): the podman underlays uniq=8, near-uniform(±8) 0.95–0.995; test_sources/pdf_horse_test.pdf's photo uniq=256, 0.14; test_sources/players_handbook.pdf page art uniq≥188, 0.10–0.59. The gate separates them cleanly.

Locked (owner decisions, chat): blank/decorative images are skipped entirely (never indexed, never described — not stored-and-flagged); JPEG2000 (jpx) images stay skipped as they are today (the unsupported-extension gate keeps running on the RAW embedded format, so compositing to PNG must not newly admit the 586 jpx images in players_handbook.pdf).

Objective

A PDF's embedded-image pass yields one document per genuinely distinct, non-decorative image, with the bytes a viewer would show (soft mask applied) — no blank white documents, no triplicate images, and far fewer vision calls per PDF.

Dependencies

• 150_drop_sync_result (todo) — pipeline order only (that phase touches the Sources-page sync UI; this phase touches app/rag/pdf.py + the importer's image pass). No shared files.

Design (shared by all tasks — the executor reads this, not the chat)

• Dedupe (task 01): in extract_pdf_images, keep the FIRST occurrence of each distinct image and skip the repeats — key = the xref when non-zero, else the sha256 of the final data (a digest set also catches byte-identical images stored under different xrefs). The kept PdfImage carries the first occurrence's page/index (the synthetic path …#page=N#img=i.<ext> is therefore stable for the first placement). One INFO line per PDF when duplicates were dropped (pdf: deduped N duplicate image occurrence(s) in <path>), DEBUG per image.
• Soft masks (task 02): when the info/extract_image dict reports a non-zero smask, build the VISIBLE image instead of the raw base: base = fitz.Pixmap(doc, xref), mask = fitz.Pixmap(doc, smask_xref), visible = fitz.Pixmap(base, mask) (PyMuPDF's alpha-combining constructor) → data = visible.tobytes("png"), mime = "image/png". Fail-soft: any exception (or a Pixmap with a colorspace PyMuPDF cannot combine) falls back to the raw extract_image bytes + the raw mime and logs a warning — the image is still indexed, exactly as today. Ordering rule (owner-locked A4): the raw-format gate runs FIRST — an image whose raw format is unsupported (jpx → .bin) is left exactly as today (raw bytes, raw mime, skipped downstream by the importer's ext gate); compositing happens only for images whose raw mime already maps through _MIME_TO_EXT. A jpx image with an smask therefore stays skipped.
• Blank gate (task 03): a NEW pure predicate in app/rag/pdf.py, is_blank_image(samples: bytes, *, mean_min: int = 235, tolerance: int = 12, fraction: float = 0.98) -> bool — pure (no PDF, no I/O — the is_usable_pdf_text / pictorial_pages house style, so it is unit-testable with hand-built sample buffers): blank when len(samples) is non-empty AND mean(samples) >= mean_min AND at least fraction of the samples lie within ±tolerance of the mean. Applied in extract_pdf_images to the samples of the image that will actually be stored (the composited pixmap when a mask was applied, else the base pixmap): a blank image is skipped (not returned), with one INFO line per skipped image (pdf: skipped blank/decorative image page=%d index=%d of %s — mean=%.0f uniform=%.3f) and one summary line per PDF. Fail-soft: a pixmap that cannot be built skips the gate (the image is kept — never lose a real image to a decode problem).
• The knob (task 04): Settings.pdf_image_skip_blank: bool = True (BOR_PDF_IMAGE_SKIP_BLANK) — the kill switch, exactly the pdf_pictorial (L433) pattern: a bool with a #: docstring naming the env var, read by the importer and passed down, False restores the pre-phase behavior (every image kept, gate off). Dedupe and mask compositing are NOT knobed (they are correctness fixes). .env.example gains the documented entry next to the other PDF/image vars; README.md gains one line in the PDF/image settings section.
• Counters (task 03): two log-only ImportSummary fields — pdf_images_deduped and pdf_images_blank_skipped — appended to ImportSummary.log()'s line (the phase-133 stats log-only convention: the sync detail API shape is untouched). They count occurrences dropped per run across all PDFs.
• ASSUMPTION A1 (owner-locked): blank images are skipped entirely (no row, no copy, no vision call) rather than indexed-and-flagged.
• ASSUMPTION A2 (owner-locked): thresholds mean ≥ 235, ≥98% of samples within ±12 — pinned by unit tests against the three real PDFs in test_sources/ (podman underlays skipped; the horse photo and the handbook page art kept).
• ASSUMPTION A3: the dedupe changes the synthetic path for repeat occurrences, so image docs created pre-phase for a 2nd/3rd placement are pruned on the next sync and the first placement's doc is kept (idempotent from then on). No migration code.
• ASSUMPTION A4 (owner-locked): jpx (JPEG2000) images stay unsupported/skipped; compositing never newly admits them.
• ASSUMPTION A5: the gate lives in the parser (extract_pdf_images), not the importer, so every caller of the parser gets the fix and the parser stays the single place that decides what "an embedded image" is.

Tasks

1. 01_dedupe_image_occurrences.md — one PdfImage per distinct image (xref, else digest); duplicate-drop log + counter.
2. 02_composite_soft_masks.md — /SMask composited to an alpha PNG (raw-format gate first, fail-soft to raw bytes).
3. 03_blank_image_gate.md — the pure is_blank_image() predicate + the skip in the extractor + the two log-only counters.
4. 04_settings_knob.md — BOR_PDF_IMAGE_SKIP_BLANK (default on) threaded parser → importer, .env.example + README.
5. 05_unit_tests.md — PyMuPDF-built fixtures for dedupe/mask/blank + the real-PDF regression pins.
6. 06_e2e_pdf_image_extraction.md — the dedicated Playwright suite (upload a PDF with a duplicate placement + a blank underlay + a real image → exactly one image doc, no blank docs).

Testing & Quality

• Unit (tests/unit/test_pdf.py + a new tests/unit/test_pdf_image_filter.py): dedupe (N placements of one image → 1 result, first occurrence's page/index kept, counter), digest-level dedupe across xrefs, smask compositing (an RGBA image round-trips with alpha; the raw-format gate leaves jpx-style unsupported formats untouched; the fail-soft path returns raw bytes), is_blank_image as a pure function (uniform white → blank; noise → not blank; dark-uniform → NOT blank because the mean gate is about near-white underlays; empty samples → not blank), the knob off → every image kept.
• Real-file pins (skip-if-missing, the HANDBOOK/HORSE_PDF pattern already in tests/unit/test_pdf.py L1734–1737): podman-systemd.pdf → 15 unique images in, 0 image docs out (all are decorative underlays); pdf_horse_test.pdf → its photo kept; players_handbook.pdf → its images stay skipped by the ext gate (unchanged, jpx).
• Integration: the importer's embedded-image pass indexes exactly the extractor's returned list (a blank/duplicate image never reaches _index_one_embedded_image — assert with a fake Embedder counting vision calls, the existing image-test idiom).
• Coverage: >90% on new/modified code (app/rag/pdf.py, the importer's pass, app/config.py).
• E2E: one dedicated Playwright file, run in isolation (uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov).

Completion Criteria

• syncing a PDF like podman-systemd.pdf creates no blank white image documents and no duplicate image documents (verified by the E2E and the real-file unit pins)
• an image with an /SMask is stored as the composited image (alpha PNG), not the blank base; an unsupported raw format (jpx) is still skipped
• BOR_PDF_IMAGE_SKIP_BLANK=false restores the pre-phase behavior (every extracted image indexed) — pinned by a unit test
• import: summary … pdf_images_deduped=N pdf_images_blank_skipped=M … appears in the sync log; the sync detail API shape is unchanged
• no behavior change in completed phases (the phase-137 document-context block, the phase-125 markdown enrichment, the digest skip, the images toggle semantics, the phase-133 fallback, the phase-140 pictorial pass)
• uv run pytest green; the dedicated E2E green in isolation; uv run ruff check . && uv run pyright clean; app/ coverage >90%
• one atomic commit per AGENTS.md rule 8 (--no-gpg-sign)