phase: 151_pdf_image_extraction
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree. All verification is complete. Final state: **Phase 151 — final verification pass: complete, project 100% green.** - **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite - **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped) - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors - **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions) - **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope) - **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays) - **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
@@ -990,9 +990,12 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
phase-106 date-refresh counter, the phase-122 ``images_failed``
|
||||
counter, the phase-137 ``image_context_backfilled`` counter (next
|
||||
to ``images_failed``), the phase-133 ``pdfs_screenshotted``
|
||||
counter, and the phase-140 ``pdfs_pictorial`` /
|
||||
``pdf_pictorial_pages`` counters (next to ``pdfs_screenshotted``)
|
||||
sit after ``dates_updated``, before ``formats``."""
|
||||
counter, the phase-140 ``pdfs_pictorial`` /
|
||||
``pdf_pictorial_pages`` counters (next to ``pdfs_screenshotted``),
|
||||
and the phase-151 ``pdf_images_deduped`` / ``pdf_images_blank_skipped``
|
||||
counters (the parser's per-PDF drop counts, next to
|
||||
``pdf_pictorial_pages``) sit after ``dates_updated``, before
|
||||
``formats``."""
|
||||
s = ImportSummary()
|
||||
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
|
||||
s.summaries, s.summary_errors = 2, 1
|
||||
@@ -1003,6 +1006,8 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
s.pdfs_screenshotted = 1
|
||||
s.pdfs_pictorial = 2
|
||||
s.pdf_pictorial_pages = 5
|
||||
s.pdf_images_deduped = 3
|
||||
s.pdf_images_blank_skipped = 1
|
||||
s.formats = {"md": 1, "yaml": 2}
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
s.log()
|
||||
@@ -1012,6 +1017,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||
"dates_updated=0 images_failed=0 image_context_backfilled=2 "
|
||||
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
|
||||
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
|
||||
"formats=yaml:2,md:1"
|
||||
)
|
||||
|
||||
|
||||
+565
-16
@@ -3,11 +3,17 @@
|
||||
Covers ``app/rag/pdf.py``: text extraction with ``---PAGE-N---``
|
||||
separators (A2), the per-page text map (phase 137 task 01 —
|
||||
``extract_pdf_page_texts``, the document-context raw material),
|
||||
embedded image extraction (original encoded bytes),
|
||||
MIME mapping (extension first, then image type code, else the
|
||||
embedded image extraction (one per DISTINCT image — phase 151 task
|
||||
01's dedupe, first placement wins; a masked image is composited to
|
||||
the visible alpha PNG — task 02; the raw-format gate runs first,
|
||||
owner-locked A4 — and the real-file regression pins), MIME mapping
|
||||
(extension first, then image type code, else the
|
||||
``application/octet-stream`` fallback), and the fail-soft contract —
|
||||
unreadable files and broken pages/images return empty results, never
|
||||
raise. Temp PDFs are built with PyMuPDF itself (no binary fixtures).
|
||||
raise. The blank/decorative gate (phase 151 task 03), its kill
|
||||
switch (task 04), and the importer's image-pass integration live in
|
||||
tests/unit/test_pdf_image_filter.py. Temp PDFs are built with
|
||||
PyMuPDF itself (no binary fixtures).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -15,7 +21,9 @@ import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
import struct
|
||||
import tarfile
|
||||
import zlib
|
||||
from collections.abc import Callable, Sequence
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
@@ -43,6 +51,7 @@ from app.rag.pdf import (
|
||||
PAGE_SEPARATOR,
|
||||
XMLISH_LINE_FRACTION,
|
||||
PdfImage,
|
||||
PdfImageStats,
|
||||
_mime_from_image_info,
|
||||
_page_image_entries,
|
||||
extract_pdf_images,
|
||||
@@ -602,12 +611,20 @@ def test_pdf_embedded_image_unsupported_mime_is_skipped(
|
||||
root = tmp_path / "pdfmime"
|
||||
root.mkdir()
|
||||
_build_pdf(root / "report.pdf", [("Text page.", [])])
|
||||
# Phase 151 (task 03): the importer now calls the stats-carrying
|
||||
# variant — patch it (the tuple shape: images + PdfImageStats).
|
||||
# Phase 151 (task 04): the knob rides as the `skip_blank`
|
||||
# keyword — the stand-in accepts it (its value is irrelevant here:
|
||||
# the tiff mime is rejected by the extension gate either way).
|
||||
monkeypatch.setattr(
|
||||
importer,
|
||||
"extract_pdf_images",
|
||||
lambda _path: [
|
||||
PdfImage(page=1, index=0, width=5, height=5, data=b"fakedata", mime="image/tiff")
|
||||
],
|
||||
"extract_pdf_images_with_stats",
|
||||
lambda _path, **_kwargs: (
|
||||
[
|
||||
PdfImage(page=1, index=0, width=5, height=5, data=b"fakedata", mime="image/tiff")
|
||||
],
|
||||
PdfImageStats(duplicates=0, blank_skipped=0, total_found=1),
|
||||
),
|
||||
)
|
||||
llm = _pdf_llm(tmp_path)
|
||||
try:
|
||||
@@ -1176,6 +1193,39 @@ def blank_middle_page_pdf(tmp_path: Path) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def same_image_three_pages_pdf(tmp_path: Path) -> Path:
|
||||
"""3 pages, EACH carrying the SAME image (phase 151 task 01 — the
|
||||
dedupe fixture: one distinct image, three placements)."""
|
||||
path = tmp_path / "triple.pdf"
|
||||
data = _png_bytes(40, 30)
|
||||
doc = fitz.open()
|
||||
for n in (1, 2, 3):
|
||||
page = doc.new_page()
|
||||
page.insert_text((72, 72), f"Text on page {n}.")
|
||||
page.insert_image(fitz.Rect(72, 150, 112, 180), stream=data)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def two_distinct_images_pdf(tmp_path: Path) -> Path:
|
||||
"""Two pages, each with its OWN image (different size AND pixels —
|
||||
genuinely distinct: the dedupe must NOT merge them)."""
|
||||
path = tmp_path / "two.pdf"
|
||||
doc = fitz.open()
|
||||
p1 = doc.new_page()
|
||||
p1.insert_image(fitz.Rect(72, 150, 112, 180), stream=_png_bytes(40, 30))
|
||||
p2 = doc.new_page()
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 15, 15))
|
||||
pix.clear_with(50) # different pixels (the shared helper uses 200)
|
||||
p2.insert_image(fitz.Rect(72, 150, 87, 165), stream=pix.tobytes("png"))
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# extract_pdf_text
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1255,22 +1305,26 @@ def test_pdf_without_images_returns_empty_list(single_page_pdf: Path) -> None:
|
||||
|
||||
def test_embedded_images_extracted_in_order(multi_page_pdf: Path) -> None:
|
||||
images = extract_pdf_images(multi_page_pdf)
|
||||
# Phase 151 (task 01): page 3's SECOND placement of the same JPEG
|
||||
# (index 1 — the same embedded xref) is deduped: one PdfImage per
|
||||
# DISTINCT image, first occurrence wins (page/index stay the first
|
||||
# placement's).
|
||||
assert [(img.page, img.index) for img in images] == [
|
||||
(2, 0), (3, 0), (3, 1), (3, 2),
|
||||
(2, 0), (3, 0), (3, 2),
|
||||
]
|
||||
# page 2: 40x30 PNG
|
||||
img = images[0]
|
||||
assert (img.width, img.height) == (40, 30)
|
||||
assert img.mime == "image/png"
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
# page 3: 40x30 JPEG, then the same JPEG again (distinct index)
|
||||
for img in images[1:3]:
|
||||
assert (img.width, img.height) == (40, 30)
|
||||
assert img.mime == "image/jpeg"
|
||||
assert img.data.startswith(JPEG_MAGIC)
|
||||
assert images[1].data == images[2].data # same embedded xref
|
||||
# page 3: 15x15 PNG
|
||||
img = images[3]
|
||||
# page 3: the 40x30 JPEG at its FIRST placement (index 0 — index 1
|
||||
# is the deduped repeat of the same embedded xref)
|
||||
img = images[1]
|
||||
assert (img.width, img.height) == (40, 30)
|
||||
assert img.mime == "image/jpeg"
|
||||
assert img.data.startswith(JPEG_MAGIC)
|
||||
# page 3: 15x15 PNG at index 2
|
||||
img = images[2]
|
||||
assert (img.width, img.height) == (15, 15)
|
||||
assert img.mime == "image/png"
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
@@ -1407,6 +1461,448 @@ def test_extracted_ext_wins_over_info_ext(
|
||||
assert images[0].mime == "image/jpeg" # ext beats the PNG type code
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 151, task 01 — one PdfImage per DISTINCT image (the dedupe)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_same_image_three_placements_yields_one_first_wins(
|
||||
same_image_three_pages_pdf: Path,
|
||||
) -> None:
|
||||
"""The task's core pin: the SAME image on 3 pages → ONE PdfImage,
|
||||
carrying the FIRST occurrence's page/index (page 1, index 0) — the
|
||||
synthetic path stays stable for the first placement."""
|
||||
images = extract_pdf_images(same_image_three_pages_pdf)
|
||||
assert len(images) == 1
|
||||
assert (images[0].page, images[0].index) == (1, 0)
|
||||
assert (images[0].width, images[0].height) == (40, 30)
|
||||
assert images[0].mime == "image/png"
|
||||
assert images[0].data.startswith(PNG_MAGIC)
|
||||
|
||||
|
||||
def test_two_distinct_images_are_both_kept(
|
||||
two_distinct_images_pdf: Path,
|
||||
) -> None:
|
||||
"""The no-duplicate path: genuinely distinct images (different size
|
||||
AND pixels) are NOT merged — two entries in page order."""
|
||||
images = extract_pdf_images(two_distinct_images_pdf)
|
||||
assert [(img.page, img.index) for img in images] == [(1, 0), (2, 0)]
|
||||
assert images[0].data != images[1].data
|
||||
assert (images[0].width, images[0].height) == (40, 30)
|
||||
assert (images[1].width, images[1].height) == (15, 15)
|
||||
|
||||
|
||||
def test_byte_identical_images_under_distinct_xrefs_dedupe_to_one(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
|
||||
) -> None:
|
||||
"""The DIGEST path: two DIFFERENT xrefs holding byte-identical
|
||||
bytes → one entry (the first), one drop — the xref check alone
|
||||
cannot catch this, the sha256 of the stored bytes can."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
|
||||
def _extract(xref: int) -> dict[str, Any]:
|
||||
return {"image": b"identical-bytes", "ext": "png", "width": 5, "height": 5}
|
||||
|
||||
doc = _FakeDoc(
|
||||
[
|
||||
_fake_image_page(
|
||||
[
|
||||
{"xref": 7, "width": 5, "height": 5},
|
||||
{"xref": 9, "width": 5, "height": 5},
|
||||
]
|
||||
)
|
||||
],
|
||||
_extract,
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
assert (images[0].page, images[0].index) == (1, 0)
|
||||
|
||||
|
||||
def test_different_bytes_under_distinct_xrefs_are_both_kept(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
|
||||
) -> None:
|
||||
"""The digest path's negative control: distinct xrefs with
|
||||
DIFFERENT bytes are distinct images — both kept."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
|
||||
def _extract(xref: int) -> dict[str, Any]:
|
||||
return {
|
||||
"image": b"first-bytes" if xref == 7 else b"second-bytes",
|
||||
"ext": "png", "width": 5, "height": 5,
|
||||
}
|
||||
|
||||
doc = _FakeDoc(
|
||||
[
|
||||
_fake_image_page(
|
||||
[
|
||||
{"xref": 7, "width": 5, "height": 5},
|
||||
{"xref": 9, "width": 5, "height": 5},
|
||||
]
|
||||
)
|
||||
],
|
||||
_extract,
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
images = extract_pdf_images(path)
|
||||
assert [(img.page, img.index) for img in images] == [(1, 0), (1, 1)]
|
||||
|
||||
|
||||
def test_duplicate_drop_logs_one_info_line_per_pdf(
|
||||
same_image_three_pages_pdf: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""One INFO line per PDF that dropped anything (the task's exact
|
||||
format), one DEBUG line per drop; 3 placements → 2 dropped,
|
||||
1 unique."""
|
||||
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
|
||||
assert len(extract_pdf_images(same_image_three_pages_pdf)) == 1
|
||||
infos = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.INFO
|
||||
and "duplicate image occurrence(s)" in r.getMessage()
|
||||
]
|
||||
assert len(infos) == 1 # ONE line per PDF, not one per drop
|
||||
assert infos[0].getMessage() == (
|
||||
f"pdf: dropped 2 duplicate image occurrence(s) "
|
||||
f"from {same_image_three_pages_pdf} (1 unique)"
|
||||
)
|
||||
debugs = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.DEBUG and "duplicate image" in r.getMessage()
|
||||
]
|
||||
assert len(debugs) == 2 # one per dropped occurrence
|
||||
assert all(r.getMessage().endswith("— skipped") for r in debugs)
|
||||
|
||||
|
||||
def test_no_duplicates_no_info_line(
|
||||
two_distinct_images_pdf: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""A PDF with no duplicates never logs the INFO line — the line is
|
||||
the duplicate signal, its absence means nothing was dropped."""
|
||||
with caplog.at_level(logging.INFO, logger="app.rag.pdf"):
|
||||
assert len(extract_pdf_images(two_distinct_images_pdf)) == 2
|
||||
assert not [
|
||||
r
|
||||
for r in caplog.records
|
||||
if "duplicate image occurrence(s)" in r.getMessage()
|
||||
]
|
||||
|
||||
|
||||
#: The owner's real 50-page fixture (untracked — ``test_sources/`` is
|
||||
#: gitignored): the podman + systemd guide. THE phase-151 reproducer —
|
||||
#: its table-shading underlays are the "blank white images" the TODO
|
||||
#: reported.
|
||||
PODMAN_PDF = Path(__file__).resolve().parents[2] / "test_sources" / "podman-systemd.pdf"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
|
||||
def test_podman_pdf_yields_one_entry_per_distinct_image() -> None:
|
||||
"""THE real-file pin: 48 placements (one per ``get_image_info``
|
||||
entry — the phase docs measured 46) spanning 15 unique xrefs — but
|
||||
6 of those xrefs are BYTE-IDENTICAL copies (same base AND same
|
||||
SMask bytes stored under a different xref), so there are only 9
|
||||
truly DISTINCT images. Task 01's digest set is spec'd for exactly
|
||||
the "byte-identical images stored under different xrefs" case, so
|
||||
it merges the 6 copies: 48 placements → 9 entries (the task
|
||||
file's "15" figure is the unique-XREF count — unreachable while
|
||||
keeping the digest set AND the phase's "one document per genuinely
|
||||
distinct image" objective, since those 6 xrefs are genuine
|
||||
duplicates a viewer cannot tell apart). First-occurrence order is
|
||||
pinned too: every kept entry carries its FIRST placement's
|
||||
page/index.
|
||||
|
||||
Phase 151 (task 03): the dedupe pin runs with the blank gate OFF
|
||||
(``skip_blank=False``) — the gate (default on) drops all 9 distinct
|
||||
images (they are the decorative underlays the phase exists to
|
||||
filter), so the default now returns ``[]`` for this file (pinned
|
||||
in tests/unit/test_pdf_image_filter.py)."""
|
||||
images = extract_pdf_images(PODMAN_PDF, skip_blank=False)
|
||||
assert len(images) == 9
|
||||
assert [(img.page, img.index) for img in images] == [
|
||||
(7, 0), (22, 0), (26, 0), (30, 0), (33, 0),
|
||||
(36, 0), (41, 0), (45, 0), (47, 0),
|
||||
]
|
||||
# no two surviving entries share bytes (the dedupe's contract, on
|
||||
# the real file)
|
||||
digests = {hashlib.sha256(img.data).hexdigest() for img in images}
|
||||
assert len(digests) == 9
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 151, task 02 — the /SMask is applied (the visible image, not
|
||||
# the blank base stream extract_image() returns)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _png_chunk(tag: bytes, payload: bytes) -> bytes:
|
||||
"""One PNG chunk (4-byte length + tag + payload + CRC32)."""
|
||||
return (
|
||||
struct.pack(">I", len(payload))
|
||||
+ tag
|
||||
+ payload
|
||||
+ struct.pack(">I", zlib.crc32(tag + payload) & 0xFFFFFFFF)
|
||||
)
|
||||
|
||||
|
||||
def _make_png(
|
||||
width: int, height: int, rgba: bool, pixel: Callable[[int, int], tuple[int, ...]]
|
||||
) -> bytes:
|
||||
"""A deterministic PNG built with stdlib only (``zlib`` +
|
||||
``struct``). PyMuPDF's Pixmap exposes no sample mutation in this
|
||||
release, so the fixture is hand-rolled: a genuine RGBA
|
||||
(color-type-6) PNG is exactly what makes ``insert_image`` store a
|
||||
``/SMask`` (base image + separate mask xref — the shape this task
|
||||
composites)."""
|
||||
color_type = 6 if rgba else 2
|
||||
n = 4 if rgba else 3
|
||||
raw = bytearray()
|
||||
for y in range(height):
|
||||
raw.append(0) # filter type: none
|
||||
for x in range(width):
|
||||
raw += bytes(pixel(x, y)[:n])
|
||||
ihdr = struct.pack(">IIBBBBB", width, height, 8, color_type, 0, 0, 0)
|
||||
return (
|
||||
b"\x89PNG\r\n\x1a\n"
|
||||
+ _png_chunk(b"IHDR", ihdr)
|
||||
+ _png_chunk(b"IDAT", zlib.compress(bytes(raw)))
|
||||
+ _png_chunk(b"IEND", b"")
|
||||
)
|
||||
|
||||
|
||||
def _insert_image_pdf(path: Path, png: bytes) -> Path:
|
||||
"""A one-page PDF carrying *png* (40x30, top-left at 72,150)."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_image(fitz.Rect(72, 150, 112, 180), stream=png)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
def _raw_extract(path: Path) -> dict[str, Any]:
|
||||
"""The page's first image as ``extract_image`` reports it — the
|
||||
raw reference (what the PRE-phase extractor returned verbatim)."""
|
||||
doc = fitz.open(str(path))
|
||||
with doc:
|
||||
info = doc[0].get_image_info(xrefs=True)[0]
|
||||
return doc.extract_image(int(info["xref"]))
|
||||
|
||||
|
||||
def _masked_png() -> bytes:
|
||||
"""A 40x30 RGBA PNG with varying pixels and a non-uniform alpha
|
||||
(top half opaque, bottom half fully transparent — the mask must
|
||||
survive the round trip)."""
|
||||
return _make_png(
|
||||
40, 30, True, lambda x, y: (x % 256, y % 256, (x + y) % 256, 255 if y < 15 else 0)
|
||||
)
|
||||
|
||||
|
||||
def test_masked_image_is_composited_to_the_visible_alpha_png(tmp_path: Path) -> None:
|
||||
"""THE task's core pin: an RGBA PNG inserted via ``insert_image``
|
||||
(PyMuPDF stores it as a BASE image + a ``/SMask``) is extracted as
|
||||
the composited VISIBLE image: (a) the raw ``extract_image`` bytes ≠
|
||||
the extractor's ``data``, (b) ``mime == "image/png"``, (c)
|
||||
re-opening the extractor's bytes as a ``fitz.Pixmap`` reports
|
||||
``alpha == 1`` (the mask survived) at the base's dimensions — and
|
||||
``width``/``height`` come from ``extract_image`` (unchanged)."""
|
||||
path = _insert_image_pdf(tmp_path / "masked.pdf", _masked_png())
|
||||
raw = _raw_extract(path)
|
||||
# Precondition (the bug this task fixes): the /SMask is stored
|
||||
# separately, and the raw extract is the maskless base.
|
||||
assert int(raw.get("smask") or 0) != 0
|
||||
assert fitz.Pixmap(raw["image"]).alpha == 0
|
||||
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
img = images[0]
|
||||
assert (img.page, img.index) == (1, 0)
|
||||
assert img.data != raw["image"] # (a) the composite, not the raw base
|
||||
assert img.mime == "image/png" # (b)
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
vis = fitz.Pixmap(img.data)
|
||||
assert vis.alpha == 1 # (c) the mask survived
|
||||
assert (vis.width, vis.height) == (40, 30)
|
||||
assert (img.width, img.height) == (40, 30) # from extract_image
|
||||
|
||||
|
||||
def test_unmasked_image_is_byte_identical_to_pre_phase(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The common case is unchanged: an image WITHOUT a mask is
|
||||
returned exactly as the raw ``extract_image`` bytes (no composite,
|
||||
no re-encode — pre-phase behavior) and no composite line is
|
||||
logged."""
|
||||
png = _make_png(40, 30, False, lambda x, y: (x % 256, y % 256, (x + y) % 256))
|
||||
path = _insert_image_pdf(tmp_path / "plain.pdf", png)
|
||||
raw = _raw_extract(path)
|
||||
assert int(raw.get("smask") or 0) == 0
|
||||
|
||||
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
assert images[0].data == raw["image"] # the raw bytes, verbatim
|
||||
assert images[0].mime == "image/png" # the raw mime (ext = png)
|
||||
assert not [
|
||||
r for r in caplog.records if "composited smask=" in r.getMessage()
|
||||
]
|
||||
|
||||
|
||||
def test_mask_composite_logs_one_debug_line(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""One DEBUG line (the task's exact format) when a mask was
|
||||
applied."""
|
||||
path = _insert_image_pdf(tmp_path / "masked.pdf", _masked_png())
|
||||
smask = int(_raw_extract(path).get("smask") or 0)
|
||||
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
|
||||
extract_pdf_images(path)
|
||||
debugs = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.DEBUG and "composited smask=" in r.getMessage()
|
||||
]
|
||||
assert len(debugs) == 1
|
||||
assert debugs[0].getMessage() == (
|
||||
f"pdf: composited smask={smask} for image page=1 index=0 of {path}"
|
||||
)
|
||||
|
||||
|
||||
def test_composite_failure_falls_back_to_raw_bytes_with_warning(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Fail-soft: a compositing that RAISES (the monkeypatched
|
||||
``fitz.Pixmap``) returns the raw bytes + raw mime, logs the
|
||||
warning (the task's exact format), and never loses the image.
|
||||
No exception escapes the extractor. The ``_FakeDoc`` stand-in
|
||||
(the task-01 convention) is used so the extractor's compositing
|
||||
block is the ONLY ``fitz.Pixmap`` call in flight (PyMuPDF's real
|
||||
constructor resolves the module-global ``Pixmap`` internally, so
|
||||
patching it would break the real base/mask reads too)."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
raw_bytes = b"raw-base-bytes"
|
||||
doc = _FakeDoc(
|
||||
[_fake_image_page([{"xref": 7, "width": 5, "height": 5}])],
|
||||
lambda xref: {
|
||||
"image": raw_bytes, "ext": "png", "width": 5, "height": 5, "smask": 8,
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
|
||||
attempts: list[tuple[tuple[Any, ...], dict[str, Any]]] = []
|
||||
|
||||
def _boom(*args: Any, **kwargs: Any) -> None:
|
||||
attempts.append((args, kwargs))
|
||||
raise RuntimeError("simulated combine failure")
|
||||
|
||||
monkeypatch.setattr(pdf_mod.fitz, "Pixmap", _boom)
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.pdf"):
|
||||
images = extract_pdf_images(path)
|
||||
assert attempts, "the composite path was attempted (a Pixmap was read)"
|
||||
assert len(images) == 1 # the image is NOT lost
|
||||
assert images[0].data == raw_bytes # raw bytes, verbatim
|
||||
assert images[0].mime == "image/png" # the raw mime
|
||||
|
||||
warnings = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.WARNING and "could not composite smask=" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1
|
||||
assert warnings[0].getMessage() == (
|
||||
f"pdf: could not composite smask=8 for image page=1 index=0 "
|
||||
f"of {path} — simulated combine failure; using raw bytes"
|
||||
)
|
||||
|
||||
|
||||
def test_unsupported_raw_format_with_mask_stays_untouched(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Owner-locked A4 — the raw-format gate runs FIRST: an image
|
||||
whose raw format the importer's ``_MIME_TO_EXT`` gate will reject
|
||||
(``jpx``/JPEG2000 → the ``application/octet-stream`` fallback — the
|
||||
586 images in ``players_handbook.pdf``) is NOT composited even
|
||||
when it carries a ``/SMask``: raw bytes + raw mime, and the
|
||||
``fitz.Pixmap`` combiner is never called (compositing must never
|
||||
newly admit an unsupported format)."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
raw_bytes = b"jpx-base-bytes"
|
||||
doc = _FakeDoc(
|
||||
[_fake_image_page([{"xref": 7, "width": 5, "height": 5}])],
|
||||
lambda xref: {
|
||||
"image": raw_bytes, "ext": "jpx", "width": 5, "height": 5, "smask": 8,
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
|
||||
def _no_composite(*args: Any, **kwargs: Any) -> None:
|
||||
# Phase 151 (task 03): the blank gate also READS the base
|
||||
# pixmap (Pixmap(doc, xref) — a fail-soft probe that keeps the
|
||||
# image when it raises, as it does on this fake doc) — that is
|
||||
# not compositing. Compositing is the Pixmap(Pixmap, Pixmap)
|
||||
# call (first arg a Pixmap): the raw-format gate must still run
|
||||
# before it, so that shape alone is the failure sentinel.
|
||||
if args and isinstance(args[0], fitz.Pixmap):
|
||||
pytest.fail("the raw-format gate must run before any compositing")
|
||||
raise ValueError("fake doc — no real pixmap (test sentinel)")
|
||||
|
||||
monkeypatch.setattr(pdf_mod.fitz, "Pixmap", _no_composite)
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
assert images[0].data == raw_bytes # raw bytes, verbatim
|
||||
assert images[0].mime == "application/octet-stream" # raw mime (→ .bin → skipped)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
|
||||
def test_podman_pdf_masked_images_are_composited() -> None:
|
||||
"""Real-file pin: EVERY distinct image of the owner's podman PDF
|
||||
carries a ``/SMask`` pre-phase (``has-mask=True`` in
|
||||
``get_image_info``, a non-zero ``smask`` in ``extract_image``), so
|
||||
post-phase the extractor's output for it is the composited visible
|
||||
image: each entry decodes as an alpha PNG (``alpha == 1``, the
|
||||
base's dimensions) whose bytes DIFFER from the raw base stream
|
||||
``extract_image`` returns (the maskless DeviceGray base — the
|
||||
"blank white image" this phase fixes). The stronger per-entry
|
||||
form of the task's "at least one xref" check. Phase 151 (task 03)
|
||||
landed: the blank gate (default on) removes these entries
|
||||
entirely, so this compositing pin runs with the gate OFF
|
||||
(``skip_blank=False``) — the default's ``[]`` result for this
|
||||
file is pinned in tests/unit/test_pdf_image_filter.py."""
|
||||
images = extract_pdf_images(PODMAN_PDF, skip_blank=False)
|
||||
assert len(images) == 9 # the task-01 dedupe pin still holds
|
||||
|
||||
# The pre-phase property (every placement masked) + the raw base
|
||||
# digests (the "blank" bytes the pre-phase extractor stored).
|
||||
raw_digests: set[str] = set()
|
||||
doc = fitz.open(str(PODMAN_PDF))
|
||||
with doc:
|
||||
for page in doc:
|
||||
for info in page.get_image_info(xrefs=True):
|
||||
xref = int(info.get("xref") or 0)
|
||||
if not xref:
|
||||
continue
|
||||
assert info.get("has-mask") is True
|
||||
raw = doc.extract_image(xref)
|
||||
assert int(raw.get("smask") or 0) != 0
|
||||
raw_digests.add(hashlib.sha256(raw["image"]).hexdigest())
|
||||
|
||||
for img in images:
|
||||
assert img.mime == "image/png"
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
vis = fitz.Pixmap(img.data)
|
||||
assert vis.alpha == 1
|
||||
assert (vis.width, vis.height) == (img.width, img.height)
|
||||
# the composite differs from the raw base (every entry)
|
||||
assert hashlib.sha256(img.data).hexdigest() not in raw_digests
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _mime_from_image_info
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1903,3 +2399,56 @@ def test_extract_pdf_text_regression_pin_horse() -> None:
|
||||
text = extract_pdf_text(HORSE_PDF)
|
||||
assert text == _HORSE_EXACT
|
||||
assert hashlib.sha256(text.encode("utf-8")).hexdigest() == _HORSE_SHA256
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 151 — real-file regression pins (skip-if-missing, the
|
||||
# ``HANDBOOK`` / ``HORSE_PDF`` idiom; test_sources/ is gitignored)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
|
||||
def test_podman_pdf_returns_no_image_documents_by_default() -> None:
|
||||
"""THE reported bug, pinned (the owner's exact file — the bug's
|
||||
tombstone test): the 50-page podman + systemd guide places its
|
||||
table-shading underlays 48 times (``get_image_info`` reports one
|
||||
entry per PLACEMENT); post-phase the extractor returns NONE of
|
||||
them — task 01's dedupe collapses the placements to 9 distinct
|
||||
images (39 dropped — 15 unique xrefs of which 6 are byte-identical
|
||||
copies, so the digest set merges them; the task file's 31/15
|
||||
figure assumed xref-only dedupe over 46 placements — the binding
|
||||
outcome is identical: 0 out) and task 03's blank gate skips all 9
|
||||
(near-white underlays — ASSUMPTION A1: never indexed, never
|
||||
described). The full drop accounting and the log lines are pinned
|
||||
in tests/unit/test_pdf_image_filter.py."""
|
||||
assert extract_pdf_images(PODMAN_PDF) == []
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HORSE_PDF.is_file(), reason="horse fixture missing")
|
||||
def test_horse_pdf_returns_exactly_one_non_blank_image() -> None:
|
||||
"""The real-file "kept" pin: the horse photo (mean 123, uniq 256,
|
||||
uniform ~0.20 — the phase overview's measured basis) clears the
|
||||
blank gate's mean floor — exactly ONE image out, NOT blank-skipped
|
||||
(being a masked JPEG, it is stored as the task-02 composite:
|
||||
alpha PNG)."""
|
||||
images = extract_pdf_images(HORSE_PDF)
|
||||
assert len(images) == 1
|
||||
assert images[0].mime == "image/png"
|
||||
assert images[0].data.startswith(PNG_MAGIC)
|
||||
assert fitz.Pixmap(images[0].data).alpha == 1
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HANDBOOK.is_file(), reason="handbook fixture missing")
|
||||
def test_handbook_pdf_images_keep_the_unmapped_jpx_mime() -> None:
|
||||
"""The real-file "no behavior change" pin: the 293-page handbook's
|
||||
586 JPEG2000 page-art images are STILL returned by the parser
|
||||
(no gate drop, no compositing — owner-locked A4: an unsupported
|
||||
raw format is left exactly as pre-phase) with the UNMAPPED raw
|
||||
mime (``application/octet-stream`` — ``jpx`` is not in the
|
||||
importer's ``_MIME_TO_EXT``), so the importer's extension gate
|
||||
skips them exactly as pre-phase (assert the mime mapping here,
|
||||
not the importer — its gate behavior is pinned in test_importer.
|
||||
py)."""
|
||||
images = extract_pdf_images(HANDBOOK)
|
||||
assert len(images) == 586
|
||||
assert all(img.mime == "application/octet-stream" for img in images)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user