phase: 151_pdf_image_extraction
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree. All verification is complete. Final state: **Phase 151 — final verification pass: complete, project 100% green.** - **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite - **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped) - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors - **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions) - **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope) - **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays) - **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
@@ -16,7 +16,7 @@ The suite's single module app runs with ``BOR_IMAGES=true`` + the
|
||||
pattern — the conftest env with the leak guards):
|
||||
|
||||
* **Seed 1** (source ``e2e-pdf-imgctx``): ONE fixture PDF at the
|
||||
source root (two text pages + one embedded 1×1 PNG on page 2 — the
|
||||
source root (two text pages + one embedded noise PNG on page 2 — the
|
||||
phase-126 PyMuPDF recipe, built in the fixture) arrives through the
|
||||
REAL admin upload flow + the RAG page's Sync button. The forward
|
||||
path (phase 137 task 02) is the story: the embedded image's
|
||||
@@ -48,7 +48,6 @@ not duplicated here).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
@@ -95,13 +94,26 @@ PDF_TITLE = "Homelab Report" # the metadata title
|
||||
PAGE_ONE = "The quarterly homelab report covers the core switch firmware upgrade."
|
||||
PAGE_TWO = "The homelab backup schedule runs nightly at 02:00 and keeps seven rotations."
|
||||
|
||||
#: A real 1×1 transparent PNG (the house fixture — the pipeline is
|
||||
#: content-agnostic; the well-formed bytes keep the extraction + copy
|
||||
#: path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
def _embedded_png() -> bytes:
|
||||
"""A small deterministic colorful noise PNG (channel means ~97 —
|
||||
the "real image" stand-in that CLEARS the phase-151 blank/
|
||||
decorative gate's near-white floor with room to spare). The old
|
||||
1×1 transparent fixture decomposed in PyMuPDF to a near-white
|
||||
1-sample base — blank by the owner-locked gate — and is now
|
||||
(correctly) skipped as a decorative underlay, so the suite's
|
||||
pipeline pins ride on a real non-blank image instead (the
|
||||
pipeline is content-agnostic; the well-formed bytes keep the
|
||||
extraction + copy path honest)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
|
||||
for y in range(18):
|
||||
for x in range(24):
|
||||
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
#: The embedded fixture image (24×18 colorful noise — non-blank by the
|
||||
#: phase-151 gate; see :func:`_embedded_png`).
|
||||
EMBED_PNG = _embedded_png()
|
||||
|
||||
# Seed 1's stored document block (NO markdown reference exists in that
|
||||
# source — the doc block alone: the provenance's origin line + the
|
||||
@@ -241,14 +253,14 @@ def app_url(app_server: str) -> str:
|
||||
|
||||
def _build_pdf(root: Path) -> Path:
|
||||
"""The fixture PDF (the phase-126 recipe, built LOCALLY): two
|
||||
text pages + one embedded 1×1 PNG on page 2 + the metadata
|
||||
text pages + one embedded noise PNG on page 2 + the metadata
|
||||
title."""
|
||||
pdf_path = root / DOC_PATH
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), PAGE_ONE)
|
||||
page2 = doc.new_page()
|
||||
page2.insert_text((72, 72), PAGE_TWO)
|
||||
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
|
||||
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
|
||||
doc.set_metadata({"title": PDF_TITLE})
|
||||
doc.save(pdf_path)
|
||||
doc.close()
|
||||
|
||||
Reference in New Issue
Block a user