phase: 151_pdf_image_extraction
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 13s

The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.

All verification is complete. Final state:

**Phase 151 — final verification pass: complete, project 100% green.**

- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
2026-10-02 07:16:08 -04:00
parent edf6048a0d
commit c7cfcd26b5
41 changed files with 3952 additions and 85 deletions
+23 -11
View File
@@ -16,7 +16,7 @@ The suite's single module app runs with ``BOR_IMAGES=true`` + the
pattern — the conftest env with the leak guards):
* **Seed 1** (source ``e2e-pdf-imgctx``): ONE fixture PDF at the
source root (two text pages + one embedded 1×1 PNG on page 2 — the
source root (two text pages + one embedded noise PNG on page 2 — the
phase-126 PyMuPDF recipe, built in the fixture) arrives through the
REAL admin upload flow + the RAG page's Sync button. The forward
path (phase 137 task 02) is the story: the embedded image's
@@ -48,7 +48,6 @@ not duplicated here).
"""
from __future__ import annotations
import base64
import json
import os
import re
@@ -95,13 +94,26 @@ PDF_TITLE = "Homelab Report" # the metadata title
PAGE_ONE = "The quarterly homelab report covers the core switch firmware upgrade."
PAGE_TWO = "The homelab backup schedule runs nightly at 02:00 and keeps seven rotations."
#: A real 1×1 transparent PNG (the house fixture — the pipeline is
#: content-agnostic; the well-formed bytes keep the extraction + copy
#: path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _embedded_png() -> bytes:
"""A small deterministic colorful noise PNG (channel means ~97 —
the "real image" stand-in that CLEARS the phase-151 blank/
decorative gate's near-white floor with room to spare). The old
1×1 transparent fixture decomposed in PyMuPDF to a near-white
1-sample base — blank by the owner-locked gate — and is now
(correctly) skipped as a decorative underlay, so the suite's
pipeline pins ride on a real non-blank image instead (the
pipeline is content-agnostic; the well-formed bytes keep the
extraction + copy path honest)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
for y in range(18):
for x in range(24):
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
return pix.tobytes("png")
#: The embedded fixture image (24×18 colorful noise — non-blank by the
#: phase-151 gate; see :func:`_embedded_png`).
EMBED_PNG = _embedded_png()
# Seed 1's stored document block (NO markdown reference exists in that
# source — the doc block alone: the provenance's origin line + the
@@ -241,14 +253,14 @@ def app_url(app_server: str) -> str:
def _build_pdf(root: Path) -> Path:
"""The fixture PDF (the phase-126 recipe, built LOCALLY): two
text pages + one embedded 1×1 PNG on page 2 + the metadata
text pages + one embedded noise PNG on page 2 + the metadata
title."""
pdf_path = root / DOC_PATH
doc = fitz.open()
doc.new_page().insert_text((72, 72), PAGE_ONE)
page2 = doc.new_page()
page2.insert_text((72, 72), PAGE_TWO)
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
doc.set_metadata({"title": PDF_TITLE})
doc.save(pdf_path)
doc.close()