phase: 151_pdf_image_extraction
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 13s

The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.

All verification is complete. Final state:

**Phase 151 — final verification pass: complete, project 100% green.**

- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
2026-10-02 07:16:08 -04:00
parent edf6048a0d
commit c7cfcd26b5
41 changed files with 3952 additions and 85 deletions
+21 -9
View File
@@ -38,7 +38,6 @@ either direction, and without the token the walk is blind to the file):
"""
from __future__ import annotations
import base64
import json
import os
import re
@@ -87,13 +86,26 @@ QUESTION = (
"what does the homelab report say about the backup schedule?"
)
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
#: the pipeline is content-agnostic; the well-formed bytes keep the
#: extraction + copy + serve + render path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _embedded_png() -> bytes:
"""A small deterministic colorful noise PNG (channel means ~97 —
the "real image" stand-in that CLEARS the phase-151 blank/
decorative gate's near-white floor with room to spare). The old
1×1 transparent fixture decomposed in PyMuPDF to a near-white
1-sample base — blank by the owner-locked gate — and is now
(correctly) skipped as a decorative underlay, so the suite's
pipeline pins ride on a real non-blank image instead (the
pipeline is content-agnostic; the well-formed bytes keep the
extraction + copy + serve + render path honest)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
for y in range(18):
for x in range(24):
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
return pix.tobytes("png")
#: The embedded fixture image (24×18 colorful noise — non-blank by the
#: phase-151 gate; see :func:`_embedded_png`).
EMBED_PNG = _embedded_png()
UPLOAD_TIMEOUT_MS = 30_000
SYNC_TIMEOUT_MS = 60_000
@@ -231,7 +243,7 @@ def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
doc.new_page().insert_text((72, 72), PAGE_ONE)
page2 = doc.new_page()
page2.insert_text((72, 72), PAGE_TWO)
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
doc.set_metadata({"title": PDF_TITLE})
doc.save(pdf_path)
doc.close()