# Phase 151 — no more blank white image documents from PDFs (dedupe + soft masks + a blank gate)

**Source:** `TODO.md` L4 — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a (TODO-derived phase, Protocol B append).
**Context:** `app/rag/pdf.py` — `extract_pdf_images()` (the module's image extractor: `page.get_image_info(xrefs=True)` per page via `_page_image_entries`, then `doc.extract_image(xref)` per entry, one `PdfImage(page, index, width, height, data, mime)` per **occurrence**) and its consumer `app/rag/importer.py` `_index_pdf_embedded_images()` (L1924 — the per-PDF embedded-image pass: `_pdf_image_context_inputs` once, then `extract_pdf_images(full_path)` and one `_index_one_embedded_image()` (L2030) per image: mime→ext gate through `_MIME_TO_EXT` (L1340 — png/jpg/gif/webp/bmp/tiff; anything else maps to `.bin` and is skipped), sha256-digest skip, one vision describe call (`_describe_or_skip`, L2907), the `image_dir` copy, chunk + embed, one transaction per image). The `images` toggle (`BOR_IMAGES`, `settings.images`, default OFF) gates the whole pass (LOCKED A3). `ImportSummary` (L390–475) carries the log-only counters (`images_failed`, `pdfs_screenshotted`, `pdfs_pictorial`, `pdf_pictorial_pages`, `image_context_backfilled`) — the phase-133 stats log-only convention: counters land in the `import: summary …` line, never in the sync `detail` API shape.

**Reproduced on the owner's file (`test_sources/podman-systemd.pdf`, 50 pages) — three distinct defects:**
1. **Duplicates:** `get_image_info` reports one entry per *placement*, so the file yields **46 occurrences of only 15 unique xrefs** (each image is placed 2–4 times). Every occurrence becomes its own document (`…#page=N#img=i.<ext>`), so the same image is vision-described and embedded up to 4×.
2. **Soft masks ignored:** every one of the 15 images carries an `/SMask` (`extract_image` reports `smask=<xref>`, `get_image_info` reports `has-mask=True`). `extract_image()` returns the **base** sample stream only — the mask (which carries the visible shape) is dropped, so the stored bytes are not what a PDF viewer shows.
3. **Decorative underlays indexed:** those images are table-shading underlays — `DeviceGray`, `mean=238/255`, **8 unique sample values**, **99.5% of samples within ±8 of the mean** (verified: composited with their mask they are still a near-white block; the table text a viewer shows is real PDF text drawn on top). They are indexed as documents, vision-described, and returned as image sources.

Measured comparison (the same probe, `fitz.Pixmap(...).samples`): the podman underlays `uniq=8`, near-uniform(±8) `0.95–0.995`; `test_sources/pdf_horse_test.pdf`'s photo `uniq=256`, `0.14`; `test_sources/players_handbook.pdf` page art `uniq≥188`, `0.10–0.59`. The gate separates them cleanly.

**Locked (owner decisions, chat):** blank/decorative images are **skipped entirely** (never indexed, never described — not stored-and-flagged); JPEG2000 (`jpx`) images stay **skipped** as they are today (the unsupported-extension gate keeps running on the RAW embedded format, so compositing to PNG must not newly admit the 586 `jpx` images in `players_handbook.pdf`).

## Objective
A PDF's embedded-image pass yields one document per genuinely distinct, non-decorative image, with the bytes a viewer would show (soft mask applied) — no blank white documents, no triplicate images, and far fewer vision calls per PDF.

## Dependencies
- `150_drop_sync_result` (todo) — pipeline order only (that phase touches the Sources-page sync UI; this phase touches `app/rag/pdf.py` + the importer's image pass). No shared files.

## Design (shared by all tasks — the executor reads this, not the chat)
- **Dedupe (task 01):** in `extract_pdf_images`, keep the FIRST occurrence of each distinct image and skip the repeats — key = the `xref` when non-zero, else the sha256 of the final `data` (a digest set also catches byte-identical images stored under different xrefs). The kept `PdfImage` carries the first occurrence's `page`/`index` (the synthetic path `…#page=N#img=i.<ext>` is therefore stable for the first placement). One INFO line per PDF when duplicates were dropped (`pdf: deduped N duplicate image occurrence(s) in <path>`), DEBUG per image.
- **Soft masks (task 02):** when the info/`extract_image` dict reports a non-zero `smask`, build the VISIBLE image instead of the raw base: `base = fitz.Pixmap(doc, xref)`, `mask = fitz.Pixmap(doc, smask_xref)`, `visible = fitz.Pixmap(base, mask)` (PyMuPDF's alpha-combining constructor) → `data = visible.tobytes("png")`, `mime = "image/png"`. Fail-soft: any exception (or a `Pixmap` with a colorspace PyMuPDF cannot combine) falls back to the raw `extract_image` bytes + the raw mime and logs a warning — the image is still indexed, exactly as today. **Ordering rule (owner-locked A4):** the raw-format gate runs FIRST — an image whose raw format is unsupported (`jpx` → `.bin`) is left exactly as today (raw bytes, raw mime, skipped downstream by the importer's ext gate); compositing happens only for images whose raw mime already maps through `_MIME_TO_EXT`. A `jpx` image with an smask therefore stays skipped.
- **Blank gate (task 03):** a NEW pure predicate in `app/rag/pdf.py`, `is_blank_image(samples: bytes, *, mean_min: int = 235, tolerance: int = 12, fraction: float = 0.98) -> bool` — pure (no PDF, no I/O — the `is_usable_pdf_text` / `pictorial_pages` house style, so it is unit-testable with hand-built sample buffers): blank when `len(samples)` is non-empty AND `mean(samples) >= mean_min` AND at least `fraction` of the samples lie within `±tolerance` of the mean. Applied in `extract_pdf_images` to the samples of the image that will actually be stored (the composited pixmap when a mask was applied, else the base pixmap): a blank image is **skipped** (not returned), with one INFO line per skipped image (`pdf: skipped blank/decorative image page=%d index=%d of %s — mean=%.0f uniform=%.3f`) and one summary line per PDF. Fail-soft: a pixmap that cannot be built skips the gate (the image is kept — never lose a real image to a decode problem).
- **The knob (task 04):** `Settings.pdf_image_skip_blank: bool = True` (`BOR_PDF_IMAGE_SKIP_BLANK`) — the kill switch, exactly the `pdf_pictorial` (L433) pattern: a `bool` with a `#:` docstring naming the env var, read by the importer and passed down, `False` restores the pre-phase behavior (every image kept, gate off). Dedupe and mask compositing are NOT knobed (they are correctness fixes). `.env.example` gains the documented entry next to the other PDF/image vars; `README.md` gains one line in the PDF/image settings section.
- **Counters (task 03):** two log-only `ImportSummary` fields — `pdf_images_deduped` and `pdf_images_blank_skipped` — appended to `ImportSummary.log()`'s line (the phase-133 stats log-only convention: the sync `detail` API shape is untouched). They count occurrences dropped per run across all PDFs.
- **ASSUMPTION A1 (owner-locked):** blank images are skipped entirely (no row, no copy, no vision call) rather than indexed-and-flagged.
- **ASSUMPTION A2 (owner-locked):** thresholds `mean ≥ 235`, `≥98%` of samples within `±12` — pinned by unit tests against the three real PDFs in `test_sources/` (podman underlays skipped; the horse photo and the handbook page art kept).
- **ASSUMPTION A3:** the dedupe changes the synthetic path for repeat occurrences, so image docs created pre-phase for a 2nd/3rd placement are pruned on the next sync and the first placement's doc is kept (idempotent from then on). No migration code.
- **ASSUMPTION A4 (owner-locked):** `jpx` (JPEG2000) images stay unsupported/skipped; compositing never newly admits them.
- **ASSUMPTION A5:** the gate lives in the parser (`extract_pdf_images`), not the importer, so every caller of the parser gets the fix and the parser stays the single place that decides what "an embedded image" is.

## Tasks
1. `01_dedupe_image_occurrences.md` — one `PdfImage` per distinct image (xref, else digest); duplicate-drop log + counter.
2. `02_composite_soft_masks.md` — `/SMask` composited to an alpha PNG (raw-format gate first, fail-soft to raw bytes).
3. `03_blank_image_gate.md` — the pure `is_blank_image()` predicate + the skip in the extractor + the two log-only counters.
4. `04_settings_knob.md` — `BOR_PDF_IMAGE_SKIP_BLANK` (default on) threaded parser → importer, `.env.example` + README.
5. `05_unit_tests.md` — PyMuPDF-built fixtures for dedupe/mask/blank + the real-PDF regression pins.
6. `06_e2e_pdf_image_extraction.md` — the dedicated Playwright suite (upload a PDF with a duplicate placement + a blank underlay + a real image → exactly one image doc, no blank docs).

## Testing & Quality
- Unit (`tests/unit/test_pdf.py` + a new `tests/unit/test_pdf_image_filter.py`): dedupe (N placements of one image → 1 result, first occurrence's page/index kept, counter), digest-level dedupe across xrefs, smask compositing (an RGBA image round-trips with alpha; the raw-format gate leaves `jpx`-style unsupported formats untouched; the fail-soft path returns raw bytes), `is_blank_image` as a pure function (uniform white → blank; noise → not blank; dark-uniform → NOT blank because the mean gate is about near-white underlays; empty samples → not blank), the knob off → every image kept.
- Real-file pins (skip-if-missing, the `HANDBOOK`/`HORSE_PDF` pattern already in `tests/unit/test_pdf.py` L1734–1737): `podman-systemd.pdf` → 15 unique images in, **0 image docs out** (all are decorative underlays); `pdf_horse_test.pdf` → its photo kept; `players_handbook.pdf` → its images stay skipped by the ext gate (unchanged, `jpx`).
- Integration: the importer's embedded-image pass indexes exactly the extractor's returned list (a blank/duplicate image never reaches `_index_one_embedded_image` — assert with a fake `Embedder` counting vision calls, the existing image-test idiom).
- Coverage: **>90%** on new/modified code (`app/rag/pdf.py`, the importer's pass, `app/config.py`).
- E2E: one dedicated Playwright file, run in isolation (`uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov`).

## Completion Criteria
- [ ] syncing a PDF like `podman-systemd.pdf` creates **no** blank white image documents and **no** duplicate image documents (verified by the E2E and the real-file unit pins)
- [ ] an image with an `/SMask` is stored as the composited image (alpha PNG), not the blank base; an unsupported raw format (`jpx`) is still skipped
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` restores the pre-phase behavior (every extracted image indexed) — pinned by a unit test
- [ ] `import: summary … pdf_images_deduped=N pdf_images_blank_skipped=M …` appears in the sync log; the sync `detail` API shape is unchanged
- [ ] no behavior change in completed phases (the phase-137 document-context block, the phase-125 markdown enrichment, the digest skip, the `images` toggle semantics, the phase-133 fallback, the phase-140 pictorial pass)
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)