phase: 151_pdf_image_extraction
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 13s

The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.

All verification is complete. Final state:

**Phase 151 — final verification pass: complete, project 100% green.**

- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
2026-10-02 07:16:08 -04:00
parent edf6048a0d
commit c7cfcd26b5
41 changed files with 3952 additions and 85 deletions
+39 -18
View File
@@ -64,9 +64,9 @@ Fixture provenance:
garbage line repeated 20× (``<illustrator:StartupProfile>Web</…>``
— 100% of the lines look like tags, so the task-02 gate fails
deterministically), plus ONE small embedded raster image per page
(a 1×1 PNG via ``insert_image`` — the fallback's embedded-image leg
runs too: the screenshots are the CONTENT, the extracted images are
the citation figures — both, no conflict).
(a colorful noise PNG via ``insert_image`` — the fallback's
embedded-image leg runs too: the screenshots are the CONTENT, the
extracted images are the citation figures — both, no conflict).
* ``cleandoc/`` — ONE small clean PDF (PyMuPDF, real prose text, no
images) — the gate's no-false-negative subject.
@@ -101,7 +101,6 @@ Test → observable mapping (Playwright Mapping Rule):
"""
from __future__ import annotations
import base64
import json
import os
import re
@@ -181,13 +180,33 @@ CLEAN_PROSE = (
"The clean reference page covers the homelab firewall rule audit."
)
#: A real 1×1 transparent PNG (the E2E suites' fixture — the pipeline
#: is content-agnostic; the well-formed bytes keep the extraction +
#: copy + serve path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _embedded_png(seed: int) -> bytes:
"""A small deterministic colorful noise PNG (channel means ~97 —
the "real image" stand-in that CLEARS the phase-151 blank/
decorative gate's near-white floor with room to spare). The old
1×1 transparent fixture decomposed in PyMuPDF to a near-white
1-sample base — blank by the owner-locked gate — and is now
(correctly) skipped as a decorative underlay, so the suite's
pipeline pins ride on a real non-blank image instead (the
pipeline is content-agnostic; the well-formed bytes keep the
extraction + copy + serve path honest). *seed* offsets the pixel
formula — the two pages carry DISTINCT images (the same stream
twice would be one placement pair: the phase-151 dedupe keeps the
first and drops the repeat, and this suite pins one image DOC PER
PAGE)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
for y in range(18):
for x in range(24):
pix.set_pixel(
x, y,
((x * 7 + seed) % 256, (y * 13 + seed) % 256, ((x + y) * 5 + seed) % 256),
)
return pix.tobytes("png")
#: The two embedded fixture images (24×18 colorful noise — non-blank
#: by the phase-151 gate, distinct per page — see :func:`_embedded_png`).
EMBED_PNGS = (_embedded_png(0), _embedded_png(1))
def _build_garbage_pdf(path: Path) -> None:
@@ -196,16 +215,18 @@ def _build_garbage_pdf(path: Path) -> None:
= the owner's exact garbage line × 20 (the gate's XML signal fails
deterministically — 100% of the non-empty lines look like tags;
the ``---PAGE-N---`` separators are not tag-shaped), plus ONE small
embedded raster image per page (the 1×1 PNG — the fallback's
embedded-image leg runs too). The text layer is real EXTRACTABLE
text — this is the FINDING's class (the pre-126 raw-XMP soup), not
the empty-layer class."""
embedded raster image per page (a DISTINCT noise PNG per page —
the fallback's embedded-image leg runs too). The text layer is real
EXTRACTABLE text — this is the FINDING's class (the pre-126 raw-XMP
soup), not the empty-layer class."""
doc = fitz.open()
for _page_num in range(1, N_GARBAGE_PAGES + 1):
for page_num in range(1, N_GARBAGE_PAGES + 1):
page = doc.new_page()
for i in range(GARBAGE_LINES_PER_PAGE):
page.insert_text((72, 72 + i * 14), GARBAGE_LINE)
page.insert_image(fitz.Rect(72, 500, 73, 501), stream=PNG_1X1)
page.insert_image(
fitz.Rect(72, 500, 96, 518), stream=EMBED_PNGS[page_num - 1]
)
doc.save(path)
doc.close()
@@ -648,7 +669,7 @@ def test_garbage_pdf_takes_the_screenshot_path(
assert "<illustrator" not in content
# The fallback's embedded-image leg: both extracted images (one per
# page, the 1×1 PNG — ``#page=N#img=0`` paths) indexed — and phase
# page, the noise PNG — ``#page=N#img=0`` paths) indexed — and phase
# 137 (task 02, A7) gives each the ORIGIN-LINE-ONLY document
# context: the garbage text layer failed the gate, so the "page
# text" is dropped (the soup never reaches the embedder), while the