Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 151_pdf_image_extraction
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.

All verification is complete. Final state:

**Phase 151 — final verification pass: complete, project 100% green.**

- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
ducoterra
2026-10-02 07:16:08 -04:00
parent edf6048a0d
commit c7cfcd26b5
41 changed files with 3952 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/151_pdf_image_extraction/00_phase.md → .agents/phases/complete/151_pdf_image_extraction/00_phase.md
View File
.agents/phases/todo/151_pdf_image_extraction/01_dedupe_image_occurrences.md → .agents/phases/complete/151_pdf_image_extraction/01_dedupe_image_occurrences.md
View File
.agents/phases/todo/151_pdf_image_extraction/02_composite_soft_masks.md → .agents/phases/complete/151_pdf_image_extraction/02_composite_soft_masks.md
View File
.agents/phases/todo/151_pdf_image_extraction/03_blank_image_gate.md → .agents/phases/complete/151_pdf_image_extraction/03_blank_image_gate.md
View File
.agents/phases/todo/151_pdf_image_extraction/04_settings_knob.md → .agents/phases/complete/151_pdf_image_extraction/04_settings_knob.md
View File
.agents/phases/todo/151_pdf_image_extraction/05_unit_tests.md → .agents/phases/complete/151_pdf_image_extraction/05_unit_tests.md
View File
.agents/phases/todo/151_pdf_image_extraction/06_e2e_pdf_image_extraction.md → .agents/phases/complete/151_pdf_image_extraction/06_e2e_pdf_image_extraction.md
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__00_phase.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__00_phase.a1.md
+13
View File
@@ -0,0 +1,13 @@
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.
All verification is complete. Final state:
**Phase 151 — final verification pass: complete, project 100% green.**
- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__00_phase.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 748 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5682 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__01_dedupe_image_occurrences.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__01_dedupe_image_occurrences.a1.md
+17
View File
@@ -0,0 +1,17 @@
All green. Final report:
**Implemented (phase 151, task 01 — dedupe image occurrences)**
- `app/rag/pdf.py`: `extract_pdf_images()` now returns one `PdfImage` per DISTINCT image — `seen_xrefs` (repeated placements) then `seen_digests` (sha256 of stored bytes under a different xref), first occurrence's `page`/`index` wins; one INFO line per PDF (`pdf: dropped N duplicate image occurrence(s) from … (M unique)`) + DEBUG per drop; `import hashlib` added; module + function docstrings updated
- `app/rag/importer.py`: `_index_pdf_embedded_images()` docstring gained the dedupe note (no counter plumbing — that's task 03's)
- `tests/unit/test_pdf.py`: 6 new tests (3×-placement→1 first-wins, 2 distinct kept, digest path across xrefs + negative control, INFO once per PDF / DEBUG per drop, no-dup no-line) + real-file podman pin; existing order test updated to dedupe semantics
**Tests / lint / coverage**
- `uv run pytest tests/unit/test_pdf.py -q` → green (96 passed)
- `uv run pytest --cov=app` (full suite) → **3470 passed, 0 failed** in 93s; `app/` coverage **99%** (`app/rag/pdf.py` 100% — both dedupe branches + no-dup path covered)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
**Docs**: no doc impact (README/AGENTS.md make no per-occurrence claims; the README line lands with task 04's knob)
**Notable deviation**: task pinned the podman file at 15 entries — measured reality: 48 placements (phase doc said 46) over 15 unique xrefs, but 6 of those xrefs are **byte-identical copies** (same base AND SMask bytes), so the task's own digest set ("catches byte-identical images under different xrefs") merges them → **9** truly distinct images. I implemented the specified algorithm and pinned 9 with a documented rationale; the phase objective ("one document per genuinely distinct image", "no duplicate image documents") requires the merge — the "15" was the unique-xref count.
**Next pending task**: `02_composite_soft_masks.md`
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__01_dedupe_image_occurrences.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.............. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 265 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 744 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 141 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5626 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__02_composite_soft_masks.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__02_composite_soft_masks.a1.md
+19
View File
@@ -0,0 +1,19 @@
All green. Task 02 is complete.
**Implemented (phase 151, task 02 — /SMask compositing):**
- `app/rag/pdf.py` `extract_pdf_images()`: after the raw-format decision, non-zero `smask` + supported raw mime → `fitz.Pixmap(base, mask)` composite → `tobytes("png")` + `mime="image/png"`; raw-format gate FIRST (owner-locked A4: `application/octet-stream`/`jpx` left byte-identical, never newly admitted); fail-soft `try/except` → raw bytes + warning `pdf: could not composite smask=… — …; using raw bytes`; DEBUG line on success; width/height unchanged
- Docstrings: module `extract_pdf_images` bullet, `PdfImage.data`/`mime` comments, function contract extended
- `app/rag/importer.py`: digest-skip consequence noted in `_index_pdf_embedded_images` docstring (masked rows re-describe once post-phase — expected, self-limiting, no migration)
- Tests: 6 new in `tests/unit/test_pdf.py` — stdlib-zlib RGBA-PNG fixture (insert_image writes the /SMask), composite core pin (raw≠data, mime, alpha round-trip), unmasked byte-identical pin, DEBUG-line pin, fail-soft pin, `jpx`+mask untouched pin (combiner never called), plus the real-file pin: all 9 podman distinct underlays returned as composited alpha PNGs differing from every raw base digest
**Results:**
- `uv run pytest tests/unit/test_pdf.py -q` → 102 passed
- `uv run pytest` → **3476 passed** (100% green)
- `uv run pytest --cov=app` → `app/rag/pdf.py` 100%, `importer.py` 99%, TOTAL 99% (>90%)
- `uv run ruff check . && uv run pyright` → clean, 0 errors
**Docs:** no doc impact (no new env vars/commands; the knob + README entry belongs to task 04)
**Notable:** fail-soft test uses the `_FakeDoc` convention — PyMuPDF's real `Pixmap.__init__` resolves the module-global `Pixmap` internally, so replacing that attribute breaks the real constructors; the fake doc isolates the composite block as the only Pixmap call.
**Next pending task:** `03_blank_image_gate.md`
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__02_composite_soft_masks.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 265 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 744 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 152 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5637 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__03_blank_image_gate.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__03_blank_image_gate.a1.md
+11
View File
@@ -0,0 +1,11 @@
All checks pass. The task is complete.
**Report — phase 151, task 03 (blank image gate)**
- `app/rag/pdf.py`: pure `is_blank_image()` predicate (+ `BLANK_IMAGE_*` constants), `PdfImageStats` dataclass, `extract_pdf_images_with_stats()` (gate after dedupe+mask: base samples, alpha dropped first, skip with INFO lines, fail-soft keep), thin `extract_pdf_images()` wrapper with `skip_blank: bool = True` kill switch.
- `app/rag/importer.py`: `ImportSummary.pdf_images_deduped` / `.pdf_images_blank_skipped` log-only counters in the `import: summary` line; `_index_pdf_embedded_images()` calls the `_with_stats` variant. Sync `detail` API shape untouched (verified explicit construction).
- Tests: new `tests/unit/test_pdf_image_filter.py` (22 tests — predicate boundaries pinned exactly at mean/fraction/tolerance, extractor-level skip/keep/kill-switch/fail-soft/alpha-drop, real-file pins: podman → `[]` + stats (39 dup, 9 blank, 48 found), horse photo kept, 586 jpx unchanged; importer integration: 1 blank + 1 real → exactly 1 doc + 1 vision call + summary line). Updated 3 existing pins (2 podman tests → `skip_blank=False` per task-02's own anticipation; jpx sentinel now allows the gate's base read; `test_importer.py` log-line format).
- Full suite: `uv run pytest --cov=app` → **3498 passed, 0 failed**; `app/` coverage **99%** (`pdf.py` 100%, `importer.py` 99%); `uv run ruff check .` clean; `uv run pyright` 0 errors.
- Docs: README PDF embedded-images bullet updated (dedupe + compositing + blank gate + new log counters). AGENTS.md: no impact.
- **Notable deviations (flagged per AGENTS.md rule 3):** (1) tolerance pinned at **17**, not the overview's ±12 — the podman file's 238/255-blend underlay (xref 1633) has its 255 level 16.2 above its mean, so ±12 leaves it in (0.954 < 0.98) and violates the binding "returns no images" criterion; ±17 is the smallest fix, all other pins unchanged. (2) Gate measures **base** samples, not the composited pixmap — PyMuPDF's `Pixmap(base, mask)` is premultiplied (composite mean ≈231–234 / uniform ≈0.95, misses every underlay; base mean ≈238 / ≈0.99, catches all). (3) Real-file stats are 39/9 (shipped digest-level dedupe over 48 placements), not the task file's 31/15 (which assumed xref-only dedupe of 46) — outcome identical: 0 out.
- Next pending task: `04_settings_knob.md` (`BOR_PDF_IMAGE_SKIP_BLANK`).
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__03_blank_image_gate.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
.......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 265 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 748 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5681 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__04_settings_knob.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__04_settings_knob.a1.md
+11
View File
@@ -0,0 +1,11 @@
**Report — phase 151, task 04 (`BOR_PDF_IMAGE_SKIP_BLANK` knob)**
- `app/config.py`: `pdf_image_skip_blank: bool = True` added next to `pdf_pictorial`/`pdf_pictorial_min_chars`, `#:` docstring naming `BOR_PDF_IMAGE_SKIP_BLANK`, default, and the off-behavior (dedupe/compositing unaffected).
- `app/rag/importer.py`: `_index_pdf_embedded_images()` now reads the knob in the one place and passes `skip_blank=settings.pdf_image_skip_blank` to `extract_pdf_images_with_stats`; pass docstring bullet updated (gate + knob).
- Tests: new Settings default/env-override unit test + an end-to-end kill-switch integration test (same blank+real PDF, knob off → BOTH image docs, 2 vision calls, `pdf_images_blank_skipped=0`) in `tests/unit/test_pdf_image_filter.py`.
- Fixed a defect this wiring exposed: `tests/unit/test_pdf.py::test_pdf_embedded_image_unsupported_mime_is_skipped` patched the extractor with `lambda _path:` (no `skip_blank` kwarg) — now accepts `**_kwargs`.
- `uv run pytest --cov=app`: **3500 passed, 0 failed**; `app/` coverage **99%** (>90% gate).
- `uv run ruff check .`: clean; `uv run pyright`: **0 errors, 0 warnings**.
- Docs: `README.md` gained the `BOR_PDF_IMAGE_SKIP_BLANK` row in the env-var table; `.env.example` gained the documented PDF-embedded-images block; `AGENTS.md`: no impact.
- Deviations: none — knob is a plain bool exactly per the `pdf_pictorial` pattern, no admin-UI toggle (task ASSUMPTION).
- Next pending task: `05_unit_tests.md` (PyMuPDF fixtures + real-PDF regression pins), then `06_e2e_pdf_image_extraction.md`.
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__04_settings_knob.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
............................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 748 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5682 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__05_unit_tests.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__05_unit_tests.a1.md
+11
View File
@@ -0,0 +1,11 @@
All gates green. Task 05 complete.
**Report — phase 151, task 05 (unit + integration tests for dedupe, masks, blank gate)**
- **Gap audit:** tasks 01–04 had already landed most of the matrix; I added what task 05's spec still assigned:
- `tests/unit/test_pdf_image_filter.py` (+8 tests): dedupe matrix with `stats.duplicates` counting (3 placements → 1 at first page/index + INFO line; 2 distinct → 2; byte-identical bytes under two xrefs → 1, digest path); soft-mask pins (inserted RGBA → `image/png`, bytes ≠ raw base, `fitz.Pixmap(...).alpha == 1`; unmasked → byte-identical to raw; `fitz.Pixmap` raising → raw bytes + warning, fail-soft); owner-locked A4 pin (`ext="jpx"` + smask → raw bytes + unmapped mime, combiner never called); integration: blank + real + **duplicate placement** → exactly 1 image doc, 1 vision call, `pdf_images_deduped=1`/`pdf_images_blank_skipped=1` in the `import: summary` line. Module docstring now covers the full phase-151 matrix.
- `tests/unit/test_pdf.py` (+3 real-file pins, skip-if-missing idiom): `podman-systemd.pdf` → `extract_pdf_images(...) == []` (the bug's tombstone); `pdf_horse_test.pdf` → exactly 1 non-blank image kept; `players_handbook.pdf` → 586 images, all unmapped `jpx` mime (importer's ext gate unchanged). Stale module docstring line ("original encoded bytes") fixed.
- **Results:** `uv run pytest tests/unit/test_pdf_image_filter.py tests/unit/test_pdf.py -v` → 137 passed (0 skipped — real fixtures present, pins executed); `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, 0 failed**, TOTAL 99% (`app/rag/pdf.py` 100%, `app/rag/importer.py` 99%, `app/config.py` 100%); `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings.
- **Docs:** no doc impact (test-only task; the `BOR_PDF_IMAGE_SKIP_BLANK` README/`.env.example` lines landed in task 04).
- **Deviations (pre-existing, documented in code + task reports):** podman stats pinned at measured 39 dup / 9 blank of 48 placements, not the task file's 31/15 of 46 — the shipped digest-level dedupe (task 01's own spec) merges 6 byte-identical xrefs; binding outcome identical (0 out). Tolerance ±17 (calibrated in task 03, owner-locked formula/floor/fraction unchanged).
- **Next pending task:** `06_e2e_pdf_image_extraction.md` (the dedicated Playwright suite).
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__05_unit_tests.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 748 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5682 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__06_e2e_pdf_image_extraction.a1.err
View File
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__06_e2e_pdf_image_extraction.a1.md
+10
View File
@@ -0,0 +1,10 @@
Task 06 is complete. Everything is green.
**Report — phase 151, task 06 (E2E: no blank, no duplicate image docs)**
- **Implemented:** `tests/e2e/test_pdf_image_extraction.py` — 2 module apps (knob ON default + `BOR_PDF_IMAGE_SKIP_BLANK=false` negative control), PyMuPDF-built fixture (near-white 200×200 underlay + same real image placed 3×, cross-page duplicate = the podman case), seeded through the real upload + Sync UI path; 5 tests: exactly one image doc (first placement `#page=1#img=1`, no duplicate/blank rows), stored bytes checked with the app's own `is_blank_image` (imported), exactly 1 vision call via the mock's `/v1/e2e/captured` window, idempotent re-sync (`0 added · 1 unchanged`, 0 vision calls), knob-off app yields the blank doc again (2 vision calls, dedupe still on)
- **Defect fix (step 8):** phase-151's gate correctly skips the house 1×1-transparent-PNG fixture (its PyMuPDF base is a near-white 1-sample base → blank), which had broken `test_pdf_support.py`, `test_pdf_image_context.py`, `test_garbage_pdf.py`; swapped their embedded fixture to a small non-blank noise PNG (garbage PDF: distinct image per page — same-stream placements now dedupe by design); all behavioral pins untouched
- **Results:** dedicated E2E green in isolation (`uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov` → 5 passed); all six required PDF/image E2Es green in isolation (6+4+3+4+4+5 passed) + `test_garbage_pdf`/`test_pdf_pages_in_sources` green; `uv run pytest --cov=app` → 3511 passed, `app/` coverage 99% (>90%); `uv run ruff check . && uv run pyright` clean
- **Docs:** no doc impact (the `BOR_PDF_IMAGE_SKIP_BLANK` README/`.env.example` entries landed with task 04; task 06 is test-only)
- **Deviation:** the task's "six pre-existing E2Es green unedited" was unsatisfiable — the gate (owner-locked, unit-pinned) legitimately drops the 1×1 transparent fixture, so the three suites' fixture image was updated (behavior pins unchanged); the task's "(c) genuinely distinct colorful image" is realized as the single real image (distinct from the underlay), since the task's own pins fix exactly one image doc and one vision call per PDF
- **Next pending task:** none in `151_pdf_image_extraction/` — phase 151 is complete (harness commits + moves the dir)
.agents/reports/151_pdf_image_extraction/151_pdf_image_extraction__06_e2e_pdf_image_extraction.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 748 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5682 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/screenshots/doc_modal_scrollbar_1280.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 41 KiB

After

Width:  |  Height:  |  Size: 42 KiB

.env.example
+12
View File
@@ -167,6 +167,18 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
# 0 = every page (A21); negative
# fails startup loudly
# --- PDF embedded images (phase 151: the blank/decorative gate) ---
# Gated by the SAME BOR_IMAGES toggle as the image docs above (the
# embedded-image pass runs only while it is on). While on (the
# DEFAULT), the pass skips near-white, nearly-uniform embedded images
# entirely — the table-shading "blank white" underlays (TODO L4):
# no row, no copy, no vision call. 0/false = off: EVERY extracted
# image is indexed again (the pre-phase-151 behavior — for a KB of
# scanned/decorative pages where a near-white image still matters).
# Dedupe + soft-mask compositing are unaffected (correctness fixes,
# not toggled).
# BOR_PDF_IMAGE_SKIP_BLANK=1 # 0 = index blank images too
# --- Chat image questions (phase 123: attach an image to a question) ---
# Gated by the SAME BOR_IMAGES toggle above (enable only when the chat
# model supports vision). The question image is stored on the server —
README.md
+10 -1
View File
@@ -636,7 +636,15 @@ uploaded archives):
persistent copy under `BOR_IMAGE_DIR`, retrievable + displayable);
with the toggle off the pass is skipped and the PDF text indexes
normally. Embedded images in a format outside `BOR_IMAGE_EXTENSIONS`
are skipped with a warning.
are skipped with a warning. The extraction is de-duplicated and
viewer-faithful (phase 151): repeated placements of the same image
yield ONE document (first placement wins), an image with a soft
mask (`/SMask`) is stored as the composited image a PDF viewer
shows (alpha PNG — not the blank base stream), and near-white,
nearly-uniform images (the table-shading "blank white" underlays)
are skipped entirely — no row, no copy, no vision call. The run
counts the drops log-only on the import summary line
(`pdf_images_deduped=N` `pdf_images_blank_skipped=N`).
- **Document context (phase 137).** Every embedded image is described
*with* the document it came from: the vision prompt — and the stored
`image_context` + enriched content — carry an
@@ -830,6 +838,7 @@ to `.env` and adjust.
| `BOR_IMPORT_EXTENSIONS` | A9 family | Importable text formats — the A9 default is `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2`; any well-formed extension is allowed. Add the `pdf` token to make PDFs first-class, retrievable documents (phase 126, opt-in). Images are **not** added here — see `BOR_IMAGES` |
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
| `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) |
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view |
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
app/config.py
+17
View File
@@ -442,6 +442,23 @@ class Settings(BaseSettings):
#: validator fails startup loudly (the ``agent_max_rounds`` pattern).
pdf_pictorial_min_chars: int = 100
# --- PDF embedded images (phase 151: the blank/decorative gate) ---
#: Skip blank/decorative PDF-embedded images (phase 151, task 04,
#: ``BOR_PDF_IMAGE_SKIP_BLANK``; ``0``/``false`` = off — the kill
#: switch, the ``pdf_pictorial``/``images`` naming pattern). While
#: on (the DEFAULT) the parser's blank/decorative gate
#: (``app.rag.pdf.is_blank_image`` — near-white AND nearly uniform,
#: owner-locked A2) drops the near-white underlays (the table-
#: shading "blank white" images, TODO L4) from the embedded-image
#: pass: no row, no copy, no vision call. While off, EVERY
#: extracted image is indexed again — the pre-phase-151 behavior
#: (a KB of scanned/decorative pages where a near-white image
#: still matters). Dedupe (task 01) and soft-mask compositing
#: (task 02) are UNAFFECTED by this knob — they are correctness
#: fixes. A plain bool like ``pdf_pictorial``/``images`` — no
#: validator.
pdf_image_skip_blank: bool = True
# --- Chat image questions (phase 123: attach an image to a question) ---
#: Where a user's question image bytes are stored (phase 123,
#: ``BOR_CHAT_IMAGE_DIR`` — the ``image_dir`` convention: a sibling of
app/rag/importer.py
+61 -11
View File
@@ -317,7 +317,7 @@ from app.rag.md_images import ImageRef, extract_image_refs
from app.rag.pdf import (
PAGE_SEPARATOR,
PdfImage,
extract_pdf_images,
extract_pdf_images_with_stats,
extract_pdf_page_texts,
extract_pdf_text,
is_usable_pdf_text,
@@ -433,6 +433,24 @@ class ImportSummary:
#: — the one-time best-effort pass; a success never repeats, a
#: failure retries next sync.
image_context_backfilled: int = 0
#: Image OCCURRENCES the PDF parser's dedupe dropped (phase 151,
#: tasks 01/03) — repeated placements + byte-identical copies
#: stored under distinct xrefs (one :class:`app.rag.pdf.PdfImage`
#: per DISTINCT image, first placement wins). Log-only (the sync
#: ``detail`` API shape is untouched — the stats log-only
#: convention, phase 133): the owner reads it in the sync log.
pdf_images_deduped: int = 0
#: DISTINCT PDF-embedded images the parser's blank/decorative gate
#: dropped (phase 151, task 03 —
#: :func:`app.rag.pdf.is_blank_image` on the base samples): a
#: near-white, nearly-uniform underlay (the TODO L4 "blank white
#: images") is skipped ENTIRELY — no row, no ``image_dir`` copy,
#: no vision call (ASSUMPTION A1). Log-only (the sync ``detail``
#: API shape is untouched — the stats log-only convention,
#: phase 133); while ``Settings.pdf_image_skip_blank`` (phase 151
#: task 04, the kill switch) is ``False`` the gate is off and this
#: count stays 0 (pre-phase behavior).
pdf_images_blank_skipped: int = 0
#: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
#: tuples in encounter order (added/updated in walk order, pruned in
#: _prune order). Drives the sync's incremental folder-summary
@@ -455,7 +473,8 @@ class ImportSummary:
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"summary_backfilled=%d dates_updated=%d images_failed=%d "
"image_context_backfilled=%d pdfs_screenshotted=%d "
"pdfs_pictorial=%d pdf_pictorial_pages=%d formats=%s",
"pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
"pdf_images_blank_skipped=%d formats=%s",
self.files,
self.added,
self.updated,
@@ -473,6 +492,8 @@ class ImportSummary:
self.pdfs_screenshotted,
self.pdfs_pictorial,
self.pdf_pictorial_pages,
self.pdf_images_deduped,
self.pdf_images_blank_skipped,
self.format_counts(),
)
@@ -1934,10 +1955,21 @@ async def _index_pdf_embedded_images(
"""The phase-126 (A3) embedded-image pass — one call per (re)indexed
PDF, after its document commit.
Each embedded image (page order, 0-based index on the page — the
parser's order) is indexed as a STANDALONE image document through
the phase-122 mechanics, inline (no temp file, no
``_index_image_file`` refactor):
Each DISTINCT embedded image (the parser dedupes repeated
placements and byte-identical copies — phase 151 task 01 — first
placement wins; page order, 0-based index on the page — the
parser's order; the parser's blank/decorative gate — phase 151
task 03, :func:`app.rag.pdf.is_blank_image` on the base samples —
drops near-white underlays BEFORE they reach this pass: no row,
no copy, no vision call) is indexed as a STANDALONE image document
through the phase-122 mechanics, inline (no temp file, no
``_index_image_file`` refactor): the gate is ON by default and
``Settings.pdf_image_skip_blank`` (``BOR_PDF_IMAGE_SKIP_BLANK`` —
phase 151 task 04, the kill switch, read HERE — the one place —
and passed down as the extractor's ``skip_blank`` keyword) is its
single knob: ``false`` restores indexing every extracted image
(the pre-phase-151 behavior); dedupe and soft-mask compositing are
UNAFFECTED by the knob (they are correctness fixes).
* gated by the phase-122 ``images`` toggle (the single knob, LOCKED
A3): while it is OFF the whole pass is skipped (one debug line —
@@ -1947,12 +1979,19 @@ async def _index_pdf_embedded_images(
(:data:`_MIME_TO_EXT`) is skipped with a warning (the unsupported-
extension rule, same as a standalone image's);
* the document's synthetic ``path`` is
``<pdf path>#page=<N>#img=<index>.<ext>`` — unique per image
occurrence, stable across re-indexes (the upsert key); the title
is ``Embedded image from <pdf stem> page <N>``;
``<pdf path>#page=<N>#img=<index>.<ext>`` — N/index are the
FIRST placement's (the parser's dedupe, phase 151 task 01),
unique per distinct image, stable across re-indexes (the upsert
key); the title is ``Embedded image from <pdf stem> page <N>``;
* the digest is sha256 over the image BYTES (the digest rule is
content identity); an image doc whose digest matches is skipped
(no re-describe, no re-embed — unchanged);
(no re-describe, no re-embed — unchanged). Phase 151 (task 02)
consequence, verified: a MASKED image's bytes change with the
smask compositing (base stream → alpha PNG), so pre-phase rows
for masked images re-describe ONCE on the first post-phase sync
(their sha256 digest differs) — expected, self-limiting, no
migration; unmasked images' bytes are untouched, so their rows
stay digest-skipped as before;
* the description comes through the SAME single seam
(:func:`_describe_or_skip` — the bytes as a data URL whose mime
comes from the copy path's extension); a failed/empty description
@@ -2018,7 +2057,18 @@ async def _index_pdf_embedded_images(
# task-03 backfill probes the parent with the SAME helper when it
# is still in the source, so the two paths can never drift.
page_texts, text_usable = _pdf_image_context_inputs(full_path)
images = extract_pdf_images(full_path)
# Phase 151 (task 03): the stats-carrying variant — the parser's
# per-PDF drop counts (dedupe + blank gate) land on the run's
# summary (log-only: the sync ``detail`` API shape is untouched).
# Phase 151 (task 04): the knob is read HERE — the one place —
# and passed down as the extractor's `skip_blank` keyword
# (default True; `false` = every extracted image indexed, the
# pre-phase behavior; dedupe + compositing are unaffected).
images, stats = extract_pdf_images_with_stats(
full_path, skip_blank=settings.pdf_image_skip_blank
)
summary.pdf_images_deduped += stats.duplicates
summary.pdf_images_blank_skipped += stats.blank_skipped
for img in images:
await _index_one_embedded_image(
session, img, source=source, rel=rel, full_path=full_path,
app/rag/pdf.py
+400 -16
View File
@@ -16,10 +16,39 @@ package, imported as ``fitz`` — its stable API name).
value for page N is exactly that page's separator-block body). The
phase-137 image-context block is built from this map (page N's entry
is the document context for an image embedded on page N).
* :func:`extract_pdf_images` — one :class:`PdfImage` per embedded-image
occurrence, in page order then 0-based index on the page. ``data`` is
the image's original encoded bytes as stored in the PDF (no
re-encoding), ``mime`` is derived via :func:`_mime_from_image_info`.
* :func:`extract_pdf_images` — one :class:`PdfImage` per DISTINCT
embedded image (page order, first placement wins) — repeated
placements and byte-identical copies are deduplicated (phase 151
task 01). ``data`` is the image's original encoded bytes as stored in
the PDF (no re-encoding), ``mime`` is derived via
:func:`_mime_from_image_info` — EXCEPT an image stored with a soft
mask (``/SMask``): ``extract_image`` returns the BASE sample stream
only (the mask carries the visible shape), so masked images are
composited to the visible alpha PNG (``mime="image/png"``) — the
bytes a PDF viewer would show (phase 151 task 02); an image whose
raw format is unsupported (the ``application/octet-stream``
fallback, e.g. JPEG2000 ``jpx``) is left untouched (raw bytes, raw
mime) so the importer's extension gate still decides. THEN the
blank/decorative gate (phase 151 task 03, ``skip_blank=True`` by
default — the ``skip_blank=False`` keyword is the function-level
kill switch): a distinct image whose base samples are near-white AND
nearly uniform (:func:`is_blank_image`) is SKIPPED entirely — no
row, no copy, no vision call (ASSUMPTION A1).
* :func:`extract_pdf_images_with_stats` — the stats-carrying variant
the importer calls (phase 151 task 03): the same extraction plus a
:class:`PdfImageStats` (``duplicates`` — occurrences dropped by the
dedupe, ``blank_skipped`` — distinct images dropped by the gate,
``total_found`` — occurrences found) so the run's
``ImportSummary`` can record them log-only (``pdf_images_deduped`` /
``pdf_images_blank_skipped`` — the phase-133 stats convention, the
sync ``detail`` API shape untouched).
* :func:`is_blank_image` — the phase-151 (task 03) blank/decorative
predicate: pure, deterministic, no PDF, no I/O (the
:func:`is_usable_pdf_text` / :func:`pictorial_pages` house style)
— near-white (mean ≥ 235) AND nearly uniform (≥98% of the samples
within ±17 of the mean). A dark uniform block is NOT blank (the gate
targets near-white page underlays); empty samples are not blank
(never blank a thing we cannot measure).
* :func:`_mime_from_image_info` — maps a PyMuPDF image-info dict to a
MIME type: the embedded file extension (``ext``) first, then the
image type code (``type``: 2=JPEG, 3=PNG, 4=GIF, 8=TIFF, 9=WebP),
@@ -50,6 +79,7 @@ never raises for malformed input (the importer owns the loud failures).
"""
from __future__ import annotations
import hashlib
import logging
import re
from collections.abc import Iterator, Sequence
@@ -86,6 +116,45 @@ _EXT_TO_MIME: dict[str, str] = {
"tiff": "image/tiff",
}
#: Phase 151 (task 03) — the blank/decorative image gate's thresholds
#: (ASSUMPTION A2, owner-locked): a PDF-embedded image is
#: blank/decorative when its COLOR samples are NEAR-WHITE AND NEARLY
#: UNIFORM — :func:`is_blank_image`'s three numbers.
#: Measured basis (probe: ``fitz.Pixmap(...).samples`` on the image's
#: BASE sample stream, the three real PDFs in ``test_sources/``):
#
#: * ``podman-systemd.pdf``'s 9 table-shading underlays: mean 238,
#: uniq 8, uniform(±8) 0.95–0.995 → blank (the TODO L4 bug — the
#: "blank white images" the owner saw in the KB);
#: * ``pdf_horse_test.pdf``'s photo: mean 123, uniq 256, uniform ~0.20
#: → kept (the mean gate);
#: * ``players_handbook.pdf``'s page art: mean 24–221, uniq ≥188,
#: uniform 0.10–0.59 → kept (the mean gate).
BLANK_IMAGE_MEAN = 235 #: near-white floor (owner-locked A2)
#: "same as the mean" band (owner-locked A2 — see the CALIBRATED
#: DEVIATION note below: the phase overview's ±12 misses the podman
#: PDF's 238/255-blend underlay, ±17 is the smallest fix).
BLANK_IMAGE_TOLERANCE = 17
#: fraction of samples that must sit in the band (owner-locked A2).
BLANK_IMAGE_FRACTION = 0.98
# CALIBRATED DEVIATION (flagged per AGENTS.md rule 3 — never silent):
# the phase overview pinned the band at ±12, but the podman PDF's
# smallest underlay (page 47, 5 300 samples) is a 238/255 blend whose
# 255 level sits 16.2 above its mean (238.81) — at ±12, 4.5% of its
# samples fall out-of-band (uniform 0.954 < 0.98) and the image
# SURVIVES the gate, violating the phase's completion criterion
# (the extractor must return NO images for that file — the owner's
# file is the phase's reason for existing; a 238/255 strip is the
# very "blank white" document the TODO reported, the table text a
# viewer shows is real PDF text drawn on top). ±17 is the smallest
# integer band that keeps every measured real-file pin: all 9 podman
# underlays blank (99.4–99.9% in-band at ±17), the horse photo kept
# (mean 123 < 235 — the mean gate, tolerance irrelevant), and all 586
# players_handbook images kept (means 17–225 — the mean gate). The
# predicate's formula, the mean floor (235), and the fraction (0.98)
# are the owner-locked values, unchanged.
@dataclass
class PdfImage:
@@ -95,8 +164,33 @@ class PdfImage:
index: int #: image index on the page (0-based, occurrence order)
width: int #: native pixel width
height: int #: native pixel height
data: bytes #: original encoded bytes as stored in the PDF
mime: str #: MIME type (e.g. ``image/png``)
data: bytes #: encoded bytes — the original stored bytes, or the
#: composited (base + soft mask) alpha PNG for masked images
#: (phase 151 task 02)
mime: str #: MIME type (e.g. ``image/png`` — always for composited
#: masked images, phase 151 task 02)
@dataclass(frozen=True)
class PdfImageStats:
"""The per-PDF drop counts of one
:func:`extract_pdf_images_with_stats` call (phase 151, task 03).
The importer records :attr:`duplicates` / :attr:`blank_skipped`
on the run's :class:`app.rag.importer.ImportSummary` (the
phase-133 stats log-only convention — the sync ``detail`` API
shape is untouched).
"""
duplicates: int #: image OCCURRENCES dropped by the dedupe (task
#: 01) — repeated placements + byte-identical copies under
#: distinct xrefs; per run: sum across the PDFs
blank_skipped: int #: DISTINCT images dropped by the
#: blank/decorative gate (task 03) — no row, no image_dir copy,
#: no vision call
total_found: int #: image occurrences FOUND in the PDF (one per
#: placement — before dedupe and gate); hence
#: ``len(images) == total_found - duplicates - blank_skipped``
def _mime_from_image_info(info: dict[str, Any]) -> str:
@@ -308,6 +402,54 @@ def pictorial_pages(page_texts: dict[int, str], *, min_chars: int) -> list[int]:
)
def is_blank_image(
samples: bytes,
*,
mean_min: int = BLANK_IMAGE_MEAN,
tolerance: int = BLANK_IMAGE_TOLERANCE,
fraction: float = BLANK_IMAGE_FRACTION,
) -> bool:
"""Whether *samples* (raw image sample bytes — one byte per sample,
COLOR channels only, no alpha) are a BLANK/DECORATIVE near-white
image (phase 151, task 03).
In the house style of :func:`is_usable_pdf_text` /
:func:`pictorial_pages`: pure, deterministic, NO PDF, NO I/O, NO
LLM — unit-testable with hand-built sample buffers.
Blank when ALL of:
* ``samples`` is non-empty — empty samples → ``False`` (never blank
a thing we cannot measure);
* ``sum(samples) / len(samples) >= mean_min`` — the near-white
floor (default :data:`BLANK_IMAGE_MEAN` = 235). A DARK uniform
block is NOT blank by this rule: the gate targets near-white
PAGE UNDERLAYS (the table-shading rectangles the TODO L4 bug
indexed) — a solid dark block is unusual and cheaper to keep
than to explain away;
* at least ``fraction`` of the samples lie within ``±tolerance``
of the mean (defaults :data:`BLANK_IMAGE_FRACTION` = 0.98,
:data:`BLANK_IMAGE_TOLERANCE` = 17) — the "carries no visible
content" test: a real photo scatters its samples far outside
the band, a decorative underlay sits on one (near-white) level.
Measured basis (the constants' docstring carries the full numbers):
the podman underlays (mean 238, uniq 8, uniform(±8) 0.95–0.995)
are blank; the horse photo (mean 123, uniq 256, uniform ~0.20) and
the handbook page art (mean 24–221, uniq ≥188, uniform 0.10–0.59)
are kept — the mean gate alone carries both real-file "kept" pins,
the fraction+tolerance carry the "near-white but textured" class.
"""
if not samples:
return False
n = len(samples)
mean = sum(samples) / n
if mean < mean_min:
return False
uniform = sum(1 for s in samples if abs(s - mean) <= tolerance) / n
return uniform >= fraction
def render_pdf_pages(
pdf_path: Path, dpi: int = 150, pages: Sequence[int] | None = None
) -> list[bytes]:
@@ -362,23 +504,111 @@ def render_pdf_pages(
return rendered
def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
"""Extract all embedded images from *pdf_path*.
def extract_pdf_images_with_stats(
pdf_path: Path, *, skip_blank: bool = True
) -> tuple[list[PdfImage], PdfImageStats]:
"""Extract all DISTINCT embedded images from *pdf_path*, with the
per-PDF drop counts (phase 151, task 03 — the stats-carrying
variant the importer calls; :func:`extract_pdf_images` is the thin
list-only wrapper for every other caller).
Returns one :class:`PdfImage` per image occurrence, in page order
then 0-based index on the page. ``data`` is the original encoded
bytes as stored in the PDF; ``mime`` is derived from the image's
extension/type via :func:`_mime_from_image_info`.
Returns ``(images, stats)`` — one :class:`PdfImage` per DISTINCT
embedded image, in page order then 0-based index on the page, plus
the :class:`PdfImageStats` of the drops (``len(images) ==
stats.total_found - stats.duplicates - stats.blank_skipped``). The
FIRST occurrence wins: the kept entry carries the first
placement's ``page``/``index`` (the importer's synthetic path
``…#page=N#img=i.<ext>`` is therefore stable for the first
placement). Two kinds of duplicate are dropped (phase 151, task
01) — PyMuPDF reports one info entry per PLACEMENT, so a PDF that
places the same image N× yields N entries:
Fail-soft: an unreadable file returns ``[]``; an image without an
``xref`` or that cannot be extracted is logged and skipped.
* the same ``xref`` seen again (a repeated placement) — the
``seen_xrefs`` set;
* byte-identical stored bytes under a DIFFERENT ``xref`` (the same
sha256 of ``data``) — the ``seen_digests`` set (real documents DO
store the same image under several xrefs — the digest set is what
catches those; unique-xref count overstates the distinct-image
count).
When anything was dropped, ONE INFO line per PDF after the page
loop (``pdf: dropped N duplicate image occurrence(s) from <path>
(M unique)``) plus one DEBUG line per drop (``pdf: duplicate image
xref=… page=… index=… of <path> — skipped``). Dedupe is PER-PDF —
the seen-sets are local to this call, never across files (two PDFs
sharing a logo each keep their own image doc: the synthetic path is
per-PDF, so cross-PDF dedupe would break the upsert key).
``data`` is the original encoded bytes as stored in the PDF;
``mime`` is derived from the image's extension/type via
:func:`_mime_from_image_info` — with one exception (phase 151,
task 02): an image stored with a soft mask (``/SMask``). PyMuPDF's
``extract_image`` returns the BASE sample stream only — the mask
(which carries the visible shape — e.g. the table-shading underlay
the owner's podman PDF stores as a near-white base) is dropped — so
for such images the extractor composites base + mask into the
VISIBLE image (``fitz.Pixmap(base, mask)`` → ``data =
visible.tobytes("png")``, ``mime = "image/png"``): the bytes a PDF
viewer shows, not the blank base. Two rules:
* the RAW-format gate runs FIRST (owner-locked A4): an image whose
raw format the importer's ``_MIME_TO_EXT`` gate rejects (the
``application/octet-stream`` fallback — e.g. JPEG2000 ``jpx``,
the 586 images in ``players_handbook.pdf``) is NOT composited —
raw bytes + raw mime, byte-identical to pre-phase; compositing
must never newly admit an unsupported format (a ``jpx`` image
with a mask stays skipped);
* the compositing is fail-soft: any exception (a colorspace the
combiner cannot merge, a broken mask xref) falls back to the raw
bytes + raw mime with a warning — the image is still indexed,
exactly as pre-phase (a decode problem never loses a real
image). ``width``/``height`` are unchanged either way (the
composited pixmap has the base's dimensions).
THEN the blank/decorative gate (phase 151, task 03, gated by
*skip_blank* — the kill switch at the function level; task 04's
``BOR_PDF_IMAGE_SKIP_BLANK`` setting lands here, dedupe and mask
compositing are NOT knobed): a distinct image whose samples are
near-white AND nearly uniform (:func:`is_blank_image`) is SKIPPED
entirely — not returned (ASSUMPTION A1: never indexed, never
described — no row, no ``image_dir`` copy, no vision call) — with
one INFO line per skipped image (``pdf: skipped blank/decorative
image page=… index=… of <path> — mean=… uniform=…``) and one INFO
line per PDF when anything was skipped (``pdf: skipped N
blank/decorative image(s) in <path>``). The gate measures the
image's BASE sample stream (``fitz.Pixmap(doc, xref)``, the alpha
channel dropped first when present — the mean is over COLOR
samples only, an alpha channel would drag the mean and produce
false blanks) for every image — masked or not: for a masked image
the base is the correct "visible content" signal, because the
``/SMask`` only modulates the base's VISIBILITY (a near-white base
renders near-white over the white page under ANY mask, so a
near-white base is decorative no matter what the mask says) — and
the stored composite's color channels are PyMuPDF's
PREMULTIPLIED base×mask/255, which darkens the transparent regions
and would miss the very near-white underlays this gate targets
(measured on the podman underlays: alpha-stripped composite
mean ≈ 231–234 / uniform ≈ 0.95 vs base mean ≈ 238 / uniform
≈ 0.99 — only the base clears the pinned thresholds on every
underlay). The gate is fail-soft: a pixmap that cannot be built
skips the gate (the image is KEPT, DEBUG why) — a decode problem
must never delete a real image.
Fail-soft: an unreadable file returns ``([], PdfImageStats(0, 0,
0))``; an image without an ``xref`` or that cannot be extracted is
logged and skipped.
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as exc:
logger.warning("pdf: could not open %s for image extraction — %s", pdf_path, exc)
return []
return [], PdfImageStats(duplicates=0, blank_skipped=0, total_found=0)
seen_xrefs: set[int] = set()
seen_digests: set[str] = set()
dropped = 0
blank_skipped = 0
total_found = 0
images: list[PdfImage] = []
with doc:
for page_num in range(len(doc)):
@@ -413,10 +643,133 @@ def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
page_num + 1, index, pdf_path,
)
continue
# One image occurrence found (valid xref, extracted,
# non-empty data) — the input to dedupe + gate (task
# 03's total_found; len(images) = total_found -
# duplicates - blank_skipped).
total_found += 1
# Phase 151 (task 01): one PdfImage per DISTINCT image.
# Repeated placements share the xref (the first check);
# byte-identical bytes stored under a different xref are
# caught by the digest of the stored bytes (the second).
if xref in seen_xrefs:
dropped += 1
logger.debug(
"pdf: duplicate image xref=%d page=%d index=%d of %s — skipped",
xref, page_num + 1, index, pdf_path,
)
continue
digest = hashlib.sha256(data).hexdigest()
if digest in seen_digests:
dropped += 1
logger.debug(
"pdf: duplicate image xref=%d page=%d index=%d of %s — skipped",
xref, page_num + 1, index, pdf_path,
)
continue
seen_xrefs.add(xref)
seen_digests.add(digest)
# The image info's own `ext` is unreliable in some
# PyMuPDF versions (None); `extract_image` reports the
# actual format of the returned bytes, so it wins.
mime = _mime_from_image_info({**info, "ext": extracted.get("ext") or ""})
# Phase 151 (task 02): a /SMask carries the visible
# shape — `extract_image` returns the BASE sample
# stream only (the mask is dropped), so the raw bytes
# are not what a PDF viewer shows. Composite base +
# mask to the visible alpha PNG.
#
# Ordering rule (owner-locked A4): the raw-format gate
# runs FIRST — an image whose raw format the importer's
# `_MIME_TO_EXT` gate will reject (the
# `application/octet-stream` fallback — e.g. the 586
# JPEG2000 `jpx` images in players_handbook.pdf) is
# left EXACTLY as today (raw bytes, raw mime, skipped
# downstream): compositing must never newly admit an
# unsupported format.
smask = int(extracted.get("smask") or 0)
# The image's BASE pixmap, kept for the blank gate
# (task 03) when the compositing built it (a second
# decode of the same xref for the gate would be wasted
# work; when the compositing never ran — no mask or an
# unsupported raw format — the gate builds it itself).
base_pixmap: Any = None
if smask and mime != "application/octet-stream":
try:
base = fitz.Pixmap(doc, xref)
base_pixmap = base
mask = fitz.Pixmap(doc, smask)
visible = fitz.Pixmap(base, mask) # base + alpha-from-mask
data = visible.tobytes("png")
mime = "image/png"
logger.debug(
"pdf: composited smask=%d for image page=%d index=%d of %s",
smask, page_num + 1, index, pdf_path,
)
except Exception as exc:
# Fail-soft: `data`/`mime` keep their raw
# values — the image is still indexed, exactly
# as pre-phase (a decode problem never loses a
# real image).
logger.warning(
"pdf: could not composite smask=%d for image page=%d "
"index=%d of %s — %s; using raw bytes",
smask, page_num + 1, index, pdf_path, exc,
)
# Phase 151 (task 03): the blank/decorative gate — a
# near-white, nearly-uniform image (a table-shading
# underlay) carries no visible content: skip it
# entirely (ASSUMPTION A1 — no row, no copy, no vision
# call). Gated by `skip_blank` (the function-level kill
# switch; task 04's settings knob lands here) — dedupe
# and mask compositing above are NOT knobed.
if skip_blank:
try:
# The BASE sample stream (the docstring's gate-
# input rule): the base's color content is what
# the image SHOWS (a mask only modulates its
# visibility), and PyMuPDF's stored composite
# is premultiplied (base×mask/255) — darkened
# in the transparent regions, where the gate
# would miss the very underlays it targets.
gate_pixmap = (
base_pixmap
if base_pixmap is not None
else fitz.Pixmap(doc, xref)
)
if gate_pixmap.alpha:
# Drop the alpha FIRST: the mean is over
# COLOR samples only (an alpha channel
# would drag the mean and produce false
# blanks).
gate_pixmap = fitz.Pixmap(gate_pixmap, 0)
samples = gate_pixmap.samples
if is_blank_image(samples):
blank_skipped += 1
mean = sum(samples) / len(samples)
uniform = (
sum(
1
for s in samples
if abs(s - mean) <= BLANK_IMAGE_TOLERANCE
)
/ len(samples)
)
logger.info(
"pdf: skipped blank/decorative image page=%d "
"index=%d of %s — mean=%.1f uniform=%.3f",
page_num + 1, index, pdf_path, mean, uniform,
)
continue
except Exception as exc:
# Fail-soft: a pixmap that cannot be built
# skips the gate (the image is KEPT — a decode
# problem must never delete a real image).
logger.debug(
"pdf: blank gate could not build pixmap page=%d "
"index=%d of %s — %s; keeping image",
page_num + 1, index, pdf_path, exc,
)
images.append(
PdfImage(
page=page_num + 1,
@@ -427,4 +780,35 @@ def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
mime=mime,
)
)
return images
if dropped:
logger.info(
"pdf: dropped %d duplicate image occurrence(s) from %s (%d unique)",
dropped, pdf_path, len(images),
)
if blank_skipped:
logger.info(
"pdf: skipped %d blank/decorative image(s) in %s",
blank_skipped, pdf_path,
)
return images, PdfImageStats(
duplicates=dropped, blank_skipped=blank_skipped, total_found=total_found
)
def extract_pdf_images(
pdf_path: Path, *, skip_blank: bool = True
) -> list[PdfImage]:
"""Extract all DISTINCT embedded images from *pdf_path*.
Thin wrapper over :func:`extract_pdf_images_with_stats` (phase 151,
task 03) returning the image list only — the pre-task-03 contract
for every other caller. *skip_blank* (default ON) passes straight
through: the kill switch at the function level —
``extract_pdf_images(path, skip_blank=False)`` restores the
pre-phase result (every distinct image returned, gate off; the
dedupe and mask compositing stay on — they are correctness fixes,
not knobed). The importer calls the ``_with_stats`` variant so the
run's :class:`~app.rag.importer.ImportSummary` can record the drop
counts (the phase-133 stats log-only convention).
"""
return extract_pdf_images_with_stats(pdf_path, skip_blank=skip_blank)[0]
tests/e2e/test_garbage_pdf.py
+38 -17
View File
@@ -64,9 +64,9 @@ Fixture provenance:
garbage line repeated 20× (``<illustrator:StartupProfile>Web</…>``
— 100% of the lines look like tags, so the task-02 gate fails
deterministically), plus ONE small embedded raster image per page
(a 1×1 PNG via ``insert_image`` — the fallback's embedded-image leg
runs too: the screenshots are the CONTENT, the extracted images are
the citation figures — both, no conflict).
(a colorful noise PNG via ``insert_image`` — the fallback's
embedded-image leg runs too: the screenshots are the CONTENT, the
extracted images are the citation figures — both, no conflict).
* ``cleandoc/`` — ONE small clean PDF (PyMuPDF, real prose text, no
images) — the gate's no-false-negative subject.
@@ -101,7 +101,6 @@ Test → observable mapping (Playwright Mapping Rule):
"""
from __future__ import annotations
import base64
import json
import os
import re
@@ -181,13 +180,33 @@ CLEAN_PROSE = (
"The clean reference page covers the homelab firewall rule audit."
)
#: A real 1×1 transparent PNG (the E2E suites' fixture — the pipeline
#: is content-agnostic; the well-formed bytes keep the extraction +
#: copy + serve path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
def _embedded_png(seed: int) -> bytes:
"""A small deterministic colorful noise PNG (channel means ~97 —
the "real image" stand-in that CLEARS the phase-151 blank/
decorative gate's near-white floor with room to spare). The old
1×1 transparent fixture decomposed in PyMuPDF to a near-white
1-sample base — blank by the owner-locked gate — and is now
(correctly) skipped as a decorative underlay, so the suite's
pipeline pins ride on a real non-blank image instead (the
pipeline is content-agnostic; the well-formed bytes keep the
extraction + copy + serve path honest). *seed* offsets the pixel
formula — the two pages carry DISTINCT images (the same stream
twice would be one placement pair: the phase-151 dedupe keeps the
first and drops the repeat, and this suite pins one image DOC PER
PAGE)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
for y in range(18):
for x in range(24):
pix.set_pixel(
x, y,
((x * 7 + seed) % 256, (y * 13 + seed) % 256, ((x + y) * 5 + seed) % 256),
)
return pix.tobytes("png")
#: The two embedded fixture images (24×18 colorful noise — non-blank
#: by the phase-151 gate, distinct per page — see :func:`_embedded_png`).
EMBED_PNGS = (_embedded_png(0), _embedded_png(1))
def _build_garbage_pdf(path: Path) -> None:
@@ -196,16 +215,18 @@ def _build_garbage_pdf(path: Path) -> None:
= the owner's exact garbage line × 20 (the gate's XML signal fails
deterministically — 100% of the non-empty lines look like tags;
the ``---PAGE-N---`` separators are not tag-shaped), plus ONE small
embedded raster image per page (the 1×1 PNG — the fallback's
embedded-image leg runs too). The text layer is real EXTRACTABLE
text — this is the FINDING's class (the pre-126 raw-XMP soup), not
the empty-layer class."""
embedded raster image per page (a DISTINCT noise PNG per page —
the fallback's embedded-image leg runs too). The text layer is real
EXTRACTABLE text — this is the FINDING's class (the pre-126 raw-XMP
soup), not the empty-layer class."""
doc = fitz.open()
for _page_num in range(1, N_GARBAGE_PAGES + 1):
for page_num in range(1, N_GARBAGE_PAGES + 1):
page = doc.new_page()
for i in range(GARBAGE_LINES_PER_PAGE):
page.insert_text((72, 72 + i * 14), GARBAGE_LINE)
page.insert_image(fitz.Rect(72, 500, 73, 501), stream=PNG_1X1)
page.insert_image(
fitz.Rect(72, 500, 96, 518), stream=EMBED_PNGS[page_num - 1]
)
doc.save(path)
doc.close()
@@ -648,7 +669,7 @@ def test_garbage_pdf_takes_the_screenshot_path(
assert "<illustrator" not in content
# The fallback's embedded-image leg: both extracted images (one per
# page, the 1×1 PNG — ``#page=N#img=0`` paths) indexed — and phase
# page, the noise PNG — ``#page=N#img=0`` paths) indexed — and phase
# 137 (task 02, A7) gives each the ORIGIN-LINE-ONLY document
# context: the garbage text layer failed the gate, so the "page
# text" is dropped (the soup never reaches the embedder), while the
tests/e2e/test_pdf_image_context.py
+23 -11
View File
@@ -16,7 +16,7 @@ The suite's single module app runs with ``BOR_IMAGES=true`` + the
pattern — the conftest env with the leak guards):
* **Seed 1** (source ``e2e-pdf-imgctx``): ONE fixture PDF at the
source root (two text pages + one embedded 1×1 PNG on page 2 — the
source root (two text pages + one embedded noise PNG on page 2 — the
phase-126 PyMuPDF recipe, built in the fixture) arrives through the
REAL admin upload flow + the RAG page's Sync button. The forward
path (phase 137 task 02) is the story: the embedded image's
@@ -48,7 +48,6 @@ not duplicated here).
"""
from __future__ import annotations
import base64
import json
import os
import re
@@ -95,13 +94,26 @@ PDF_TITLE = "Homelab Report" # the metadata title
PAGE_ONE = "The quarterly homelab report covers the core switch firmware upgrade."
PAGE_TWO = "The homelab backup schedule runs nightly at 02:00 and keeps seven rotations."
#: A real 1×1 transparent PNG (the house fixture — the pipeline is
#: content-agnostic; the well-formed bytes keep the extraction + copy
#: path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _embedded_png() -> bytes:
"""A small deterministic colorful noise PNG (channel means ~97 —
the "real image" stand-in that CLEARS the phase-151 blank/
decorative gate's near-white floor with room to spare). The old
1×1 transparent fixture decomposed in PyMuPDF to a near-white
1-sample base — blank by the owner-locked gate — and is now
(correctly) skipped as a decorative underlay, so the suite's
pipeline pins ride on a real non-blank image instead (the
pipeline is content-agnostic; the well-formed bytes keep the
extraction + copy path honest)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
for y in range(18):
for x in range(24):
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
return pix.tobytes("png")
#: The embedded fixture image (24×18 colorful noise — non-blank by the
#: phase-151 gate; see :func:`_embedded_png`).
EMBED_PNG = _embedded_png()
# Seed 1's stored document block (NO markdown reference exists in that
# source — the doc block alone: the provenance's origin line + the
@@ -241,14 +253,14 @@ def app_url(app_server: str) -> str:
def _build_pdf(root: Path) -> Path:
"""The fixture PDF (the phase-126 recipe, built LOCALLY): two
text pages + one embedded 1×1 PNG on page 2 + the metadata
text pages + one embedded noise PNG on page 2 + the metadata
title."""
pdf_path = root / DOC_PATH
doc = fitz.open()
doc.new_page().insert_text((72, 72), PAGE_ONE)
page2 = doc.new_page()
page2.insert_text((72, 72), PAGE_TWO)
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
doc.set_metadata({"title": PDF_TITLE})
doc.save(pdf_path)
doc.close()
tests/e2e/test_pdf_image_extraction.py
+802
View File
@@ -0,0 +1,802 @@
"""Phase 151 E2E (Playwright): a PDF sync produces no blank, no duplicate
image documents — the owner's complaint, proven fixed in the browser
(TODO L4: "it extracted a series of blank white images … there don't
appear to be any blank images in the document, is this a bug?").
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the vision call
counts are read from the mock's request capture (the
``/v1/e2e/captured`` pair), which only the deterministic mock exposes.
Two module apps (the ``test_pdf_support.py`` per-suite-app pattern —
the conftest env with the leak guards, ``BOR_IMAGES=true`` + the
``pdf`` import token pinned EXPLICITLY on both, the mock LLM only):
* **the story app** (``BOR_PDF_IMAGE_SKIP_BLANK=true`` — the default,
pinned so an operator's local gitignored ``.env`` cannot leak the
premise off): the fixture PDF (below) arrives through the REAL admin
upload flow + the RAG page's **Sync** button (the same user path the
owner used). Through the product path:
- the catalog carries EXACTLY ONE image document for the PDF — the
distinct real image at its FIRST placement (``…#page=1#img=1.png``
— the dedupe's stability contract: the same-page duplicate at
``#img=2`` and the cross-page duplicate on page 2 — the podman
case — never produced their own documents);
- NO image document's stored bytes are a blank white image — the
stored copy is checked with the APP'S OWN predicate
(``app.rag.pdf.is_blank_image`` imported, not re-derived, so the
test cannot drift from the product);
- the mock's vision call count for image description is 1 (the
duplicate placements and the blank underlay never reached the
model — the cost the owner paid in "several minutes");
- a follow-up re-sync (nothing changed) settles on
``0 added · 1 unchanged`` and creates nothing new — idempotence
after the fix.
* **the knob-off app** (``BOR_PDF_IMAGE_SKIP_BLANK=false`` — the
negative control): the SAME PDF through the SAME user path now
yields the blank underlay document too (the knob is proven live in
the product, not just in unit tests): two image documents, the
underlay's stored bytes blank per the app's own predicate — the
exact class the story app skipped — and two vision calls (the
pre-phase cost). The knob only gates the blank gate: the duplicate
placements still produce no documents (dedupe is a correctness
fix, not knobed).
The fixture PDF (built with PyMuPDF in the fixture — the house
convention, no binary committed): two text pages; page 1 carries the
near-white 200×200 "decorative underlay" (image index 0 — the blank
gate's target), then the real image — genuinely distinct (colorful
noise) from the underlay — inserted TWICE (indices 1 and 2, the
same-page duplicate placement); page 2 carries one more placement of
the real image (index 0 — the cross-page duplicate, the podman case,
where each image was placed 2–4 times). Four placements, two distinct
images, so the extractor must index exactly ONE image document and
make exactly ONE vision call (the task's "exactly one image document
for the PDF" and "vision call count is 1" pins fix the distinct
non-blank total at one — the colorful real image IS the genuinely
distinct image the task's fixture list names).
Test-order note: both apps share the E2E Postgres — the story seed
runs first, the knob-off seed truncates and re-seeds (the phase-126
isolation pattern), so every story test runs before the knob-off
test, which truncates at the end.
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import zipfile
from collections.abc import Iterator
from pathlib import Path
import httpx
import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (house convention)
import pytest
from playwright.sync_api import Browser, Page, expect
from sqlalchemy import select, text
from app.config import Settings
from app.db import SessionLocal
from app.models import Document
from app.rag.pdf import is_blank_image
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_CONTEXT_ANSWER
REPO = Path(__file__).resolve().parents[2]
SOURCE_ON = "e2e-pdf151" # the story app's archive stem (archive_source_name)
SOURCE_OFF = "e2e-pdf151-off" # the knob-off app's archive stem
DOC_PATH = "guide.pdf"
#: The synthetic embedded-image paths (the importer's
#: ``<pdf path>#page=<N>#img=<index>.<ext>`` rule). The underlay sits
#: at page 1 index 0; the real image's FIRST placement at page 1
#: index 1; the duplicates at page 1 index 2 and page 2 index 0 (they
#: must never appear as documents).
BLANK_REL = f"{DOC_PATH}#page=1#img=0.png" # the underlay (skipped by default)
REAL_REL = f"{DOC_PATH}#page=1#img=1.png" # the real image, first placement
DUP_RELS = (
f"{DOC_PATH}#page=1#img=2.png", # the same-page duplicate
f"{DOC_PATH}#page=2#img=0.png", # the cross-page duplicate (podman case)
)
PDF_TITLE = "Podman Systemd Guide" # the metadata title
#: The fixture PDF's page texts (the image's page — page 1 — carries
#: the document context the phase-137 block stores on the image doc).
PAGE_ONE = "The podman systemd guide covers container unit files and their restart policies."
PAGE_TWO = "The guide's appendix lists the journal log locations and the default timeouts."
#: The kept image's stored document block (no markdown reference exists
#: in the source — the doc block alone: provenance origin line + the
#: page's text).
DOC_BLOCK = f"Extracted from {SOURCE_ON}/{DOC_PATH} (page 1):\n{PAGE_ONE}"
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
UPLOAD_TIMEOUT_MS = 30_000
SYNC_TIMEOUT_MS = 60_000
# ---------------------------------------------------------------------------
# Fixture PDF: the images (stdlib-free, PyMuPDF-built — deterministic)
# ---------------------------------------------------------------------------
def _solid_png(width: int, height: int, value: int) -> bytes:
"""A uniform RGB PNG (one ``value`` per channel). At 250 the
underlay's samples are mean 250 (≥ the 235 near-white floor) and
100% within any tolerance band — blank by
:func:`app.rag.pdf.is_blank_image` with room to spare."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height))
pix.clear_with(value)
return pix.tobytes("png")
def _noise_png(width: int, height: int) -> bytes:
"""A deterministic colorful noise PNG (channel means ~127 — the
"real photo" stand-in: it clears the mean gate with room to spare,
and its scattered samples clear the uniformity gate). Genuinely
distinct from the underlay in both channels AND structure."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height))
for y in range(height):
for x in range(width):
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
return pix.tobytes("png")
#: The two fixture images: the near-white 200×200 decorative underlay
#: (the TODO L4 "blank white" class) and the genuinely distinct
#: colorful real image.
BLANK_PNG = _solid_png(200, 200, 250)
REAL_PNG = _noise_png(96, 64)
def _build_fixture_pdf(path: Path) -> None:
"""The phase-151 fixture: two text pages; page 1 carries the
underlay (index 0), the real image (index 1), and a SECOND
placement of the same real image (index 2 — the same-page
duplicate); page 2 carries one more placement of the real image
(index 0 — the cross-page duplicate, the podman case). PyMuPDF
stores the repeated insertions under one xref (the
repeated-placement path — the digest set is the backstop for
byte-identical copies under distinct xrefs)."""
doc = fitz.open()
page1 = doc.new_page()
page1.insert_text((72, 72), PAGE_ONE)
y = 150
for data in (BLANK_PNG, REAL_PNG, REAL_PNG):
pix = fitz.Pixmap(data)
page1.insert_image(
fitz.Rect(72, y, 72 + pix.width, y + pix.height), stream=data
)
y += pix.height + 10
page2 = doc.new_page()
page2.insert_text((72, 72), PAGE_TWO)
pix = fitz.Pixmap(REAL_PNG)
page2.insert_image(
fitz.Rect(72, 150, 72 + pix.width, 150 + pix.height), stream=REAL_PNG
)
doc.set_metadata({"title": PDF_TITLE})
doc.save(path)
doc.close()
# ---------------------------------------------------------------------------
# Module apps: the story (knob ON — the default) and the knob-off
# negative control. Each owns its port + its scratch upload/source/
# image dirs. Both pin BOR_IMPORT_EXTENSIONS WITH the ``pdf`` token
# and BOR_IMAGES=true explicitly; the knob under test is pinned per
# app (a process-wide lru_cached setting — the two apps are the only
# way to prove it live in the product).
# ---------------------------------------------------------------------------
def _app_env(
mock_llm: int, app_port: int, *, skip_blank: bool, scratch: Path
) -> dict[str, str]:
"""The conftest app env (leak guards included) with the phase-151
knobs: ``BOR_IMPORT_EXTENSIONS`` pinned to a set carrying the
``pdf`` token, ``BOR_IMAGES`` pinned true (the embedded-image pass
must run on BOTH apps), ``BOR_PDF_IMAGE_SKIP_BLANK`` pinned per
app (true for the story app, false for the negative control —
the contract under test cannot leak in either direction), and the
image/upload homes in the suite's scratch dir (the app under test
must not write copies into the owner's real ``~/bor-sources``).
``BOR_PDF_PICTORIAL`` rides at the conftest pin (``0``) through
``dict(os.environ)`` — the fixture's sparse pages stay pre-phase-
140 content, byte-stable across the re-sync."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated gate (conftest pattern) — the suite drives no
# chat turn, but the app boots with the same values every other
# module app gets (the startup validator's premise).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
# Phase 67: instant retry waits + the code-default budget.
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# Leak guards (conftest pattern).
env["BOR_GIT_SOURCES"] = ""
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
# Phase 151: the import scope WITH the ``pdf`` token, the images
# toggle ON (the embedded-image pass runs on both apps), the knob
# under test pinned per app, and the suite-private homes.
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
env["BOR_IMAGES"] = "true"
env["BOR_PDF_IMAGE_SKIP_BLANK"] = "true" if skip_blank else "false"
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_IMAGE_DIR"] = str(scratch / "images")
return env
@pytest.fixture(scope="module")
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
"""The story app — ``BOR_PDF_IMAGE_SKIP_BLANK=true`` (the default
ON gate; the module env override — the conftest session app is
never started in this isolated run, so no port clash)."""
scratch = tmp_path_factory.mktemp("bor_pdf151_on")
port = int(os.environ.get("E2E_APP_PORT_PDF151", "8168"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, skip_blank=True, scratch=scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
@pytest.fixture(scope="module")
def off_app_server(
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The negative-control app — ``BOR_PDF_IMAGE_SKIP_BLANK=false``
(the kill switch OFF: every extracted image is indexed again —
the pre-phase-151 behavior; only the knob-off test starts it)."""
scratch = tmp_path_factory.mktemp("bor_pdf151_off")
port = int(os.environ.get("E2E_APP_PORT_PDF151_OFF", "8169"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, skip_blank=False, scratch=scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def off_app_url(off_app_server: str) -> str:
return off_app_server
# ---------------------------------------------------------------------------
# The fixture archives: ONE PDF, two archive stems (two source names —
# a leftover row from either seed is then visibly foreign in the other
# app's catalog)
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def fixture_pdf(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The fixture PDF (two text pages; the underlay + the real image
twice on page 1, the real image once more on page 2 — built with
PyMuPDF, the house convention)."""
root = tmp_path_factory.mktemp("bor_pdf151_pdf")
pdf_path = root / DOC_PATH
_build_fixture_pdf(pdf_path)
return pdf_path
def _zip(fixture_pdf: Path, tmp_path_factory: pytest.TempPathFactory, stem: str) -> Path:
root = tmp_path_factory.mktemp(f"bor_pdf151_zip_{stem}")
archive = root / f"{stem}.zip"
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
zf.write(fixture_pdf, arcname=DOC_PATH)
return archive
@pytest.fixture(scope="module")
def zip_on(fixture_pdf: Path, tmp_path_factory: pytest.TempPathFactory) -> Path:
"""Source ``e2e-pdf151`` — the story app's archive (ONE PDF at the
source root, document path ``guide.pdf``)."""
return _zip(fixture_pdf, tmp_path_factory, SOURCE_ON)
@pytest.fixture(scope="module")
def zip_off(fixture_pdf: Path, tmp_path_factory: pytest.TempPathFactory) -> Path:
"""Source ``e2e-pdf151-off`` — the SAME PDF for the knob-off app."""
return _zip(fixture_pdf, tmp_path_factory, SOURCE_OFF)
# ---------------------------------------------------------------------------
# DB + UI + mock-capture helpers
# ---------------------------------------------------------------------------
def _truncate_all() -> None:
"""Fresh KB + registry per test group (the E2E isolation pattern):
the suites share one Postgres, and a leftover row would corrupt the
counts. ``sources_meta`` (the KB generation counter) resets with
the KB — the sync re-creates the row."""
with SessionLocal() as db:
db.execute(text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources, sources_meta, folder_summaries"
))
db.commit()
def _health_db(app_url: str) -> bool:
try:
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
except Exception: # noqa: BLE001 — unreachable is the skip case
return False
def _upload_zip(page: Page, app_url: str, archive: Path, source_name: str) -> None:
"""The real admin upload flow: form login → the git-sources page →
pick the archive → submit → the terminal result line (phase 90:
"Uploaded <source> — press Sync sources to import it.")."""
login(page, app_url, next="/git-sources.html")
expect(page).to_have_url(app_url + "/git-sources.html", timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result")
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text(
f"Uploaded {source_name} — press Sync sources to import it.",
timeout=UPLOAD_TIMEOUT_MS,
)
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
"""The RAG page's "Sync sources" button (the scan — the same user
path the owner used): click → "Syncing…" → the terminal
"Synced HH:MM" label + the counts line settled in
``#sync-progress-text`` (the phase-150 single sync text surface —
the wait barrier, no sleeps)."""
page.goto(app_url + "/sources.html")
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text(
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
)
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
def _drill_to_source(page: Page, name: str) -> None:
"""Top level → the source (the file table then holds its direct
files — ours are at the source root)."""
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
def _reset_capture(mock_port: int) -> None:
"""Clear the mock's request capture (the phase-123 pair) so the
next sync's window holds ONLY that sync's LLM traffic."""
httpx.post(f"http://127.0.0.1:{mock_port}/v1/e2e/captured/reset", timeout=5)
def _image_description_calls(mock_port: int) -> int:
"""The vision describe calls in the mock's current capture — the
``IMAGE_DESCRIPTION_MODE`` marker is the app's fixed
``DESCRIBE_PROMPT`` head (``app.rag.summarizer``), a marker a real
question is never expected to type, so it identifies the image
describe calls exactly (no other sync traffic carries it)."""
bodies = httpx.get(f"http://127.0.0.1:{mock_port}/v1/e2e/captured", timeout=5).json()
return sum(1 for body in bodies if "IMAGE_DESCRIPTION_MODE" in json.dumps(body))
def _stored_image_is_blank(image_path: str) -> bool:
"""The APP'S OWN blank predicate (``app.rag.pdf.is_blank_image`` —
imported, not re-derived, so the test cannot drift from the
product) over the stored copy's samples: the exact gate input the
extractor measures — alpha dropped first when present, so the mean
runs over COLOR samples only."""
pm = fitz.Pixmap(Path(image_path).read_bytes())
if pm.alpha:
pm = fitz.Pixmap(pm, 0)
return is_blank_image(pm.samples)
# ---------------------------------------------------------------------------
# The module seeds: upload + sync through the real UI, once per app
# (the story E2E truncate/re-import-per-module pattern)
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def seeded(
app_url: str,
browser: Browser,
zip_on: Path,
mock_llm: int,
) -> Iterator[dict[str, object]]:
"""Upload the fixture zip through the admin UI on the story app
(knob ON), then scan it with the RAG page's Sync button (the
mock's vision model describes the embedded image in-process).
Yields the kept image doc's identity + the first sync's vision
call count. The module app's ``/api/config`` flag is asserted
first — the env override must have reached the app under test, or
the whole premise of the suite is void."""
if not _health_db(app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
assert cfg["images"] is True, "the story app must run with BOR_IMAGES=true"
_truncate_all()
page = browser.new_page(viewport={"width": 1280, "height": 800})
try:
_upload_zip(page, app_url, zip_on, SOURCE_ON)
# The capture window: ONLY the first sync's LLM traffic.
_reset_capture(mock_llm)
# The PDF + its ONE embedded image (the underlay and both
# duplicate placements are dropped by the parser — the gate
# and the dedupe are pre-importer): both count added.
_sync_via_ui(page, app_url, expected_result="2 added")
first_sync_vision = _image_description_calls(mock_llm)
_reset_capture(mock_llm) # a clean window for the re-sync test
finally:
page.close()
with SessionLocal() as db:
image_docs = db.scalars(
select(Document).where(
Document.source == SOURCE_ON, Document.is_image.is_(True)
)
).all()
# The seed's sanity pin (the rest — first placement, the
# no-blank bytes, the vision count — is asserted in the tests
# below against this state).
assert [doc.path for doc in image_docs] == [REAL_REL]
img = image_docs[0]
assert img.is_image is True and img.image_path is not None
pdf = db.scalar(
select(Document).where(
Document.source == SOURCE_ON, Document.path == DOC_PATH
)
)
assert pdf is not None and pdf.is_image is False
assert pdf.title == PDF_TITLE
assert "---PAGE-1---" in pdf.content and PAGE_ONE in pdf.content
yield {
"image_url": f"/api/documents/{img.id}/image",
"image_doc_id": img.id,
"image_doc_hash": img.content_hash,
"first_sync_vision_calls": first_sync_vision,
}
@pytest.fixture(scope="module")
def seeded_off(
off_app_url: str,
browser: Browser,
zip_off: Path,
mock_llm: int,
) -> Iterator[dict[str, object]]:
"""The SAME fixture PDF through the knob-off app (the negative
control): truncate + re-seed (the shared-Postgres pattern — this
runs LAST, after every story test has consumed the story seed),
upload + sync via the same user path. Yields the sync's vision
call count (2 — the blank underlay IS described now, the
pre-phase cost)."""
if not _health_db(off_app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = httpx.get(f"{off_app_url}/api/config", timeout=5).json()
assert cfg["images"] is True, "the knob-off app must run with BOR_IMAGES=true"
_truncate_all()
page = browser.new_page(viewport={"width": 1280, "height": 800})
try:
_upload_zip(page, off_app_url, zip_off, SOURCE_OFF)
_reset_capture(mock_llm)
# The PDF + BOTH embedded images (the blank gate is OFF — the
# underlay is indexed again; the duplicate placements are
# STILL dropped — dedupe is a correctness fix, not knobed):
# three count added.
_sync_via_ui(page, off_app_url, expected_result="3 added")
sync_vision = _image_description_calls(mock_llm)
finally:
page.close()
with SessionLocal() as db:
image_docs = db.scalars(
select(Document).where(
Document.source == SOURCE_OFF, Document.is_image.is_(True)
)
).all()
assert sorted(doc.path for doc in image_docs) == sorted([BLANK_REL, REAL_REL])
yield {"vision_calls": sync_vision}
# ---------------------------------------------------------------------------
# 1. Sources page (story app): exactly one image doc, no blank row
# ---------------------------------------------------------------------------
def test_sources_page_lists_exactly_one_image_doc(
page: Page, app_url: str, seeded: dict[str, object]
) -> None:
"""The owner's complaint, the browser lens: the KB total is 2 docs
/ 5 chunks (the PDF: 2 page-anchored content chunks + the summary
chunk; the one image: its description chunk + summary chunk — the
underlay contributes NOTHING, no row, no chunk) and the
drill-down table lists the PDF (a plain text row, no thumbnail) +
the ONE image document (the thumbnail served from the persistent
copy) — and NO row for the underlay's synthetic path
(``#page=1#img=0``)."""
login(page, app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("2")
expect(page.locator("#stat-chunks")).to_have_text("5")
expect(page.locator("#sources-empty")).to_be_hidden()
_drill_to_source(page, SOURCE_ON)
expect(page.locator("#docs-tbody tr")).to_have_count(2)
# The PDF — a plain text row (no thumbnail box at all). Pinned by
# the EXACT doc-link text (the image row's path carries
# ``guide.pdf`` as a prefix — a substring match would hit both).
pdf_row = page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{DOC_PATH}")'),
)
expect(pdf_row).to_have_count(1)
expect(pdf_row.locator(".kb-doc-thumb")).to_have_count(0)
# The ONE image document — the real image's thumbnail row (the
# bytes route's lazy <img>, alt = the description; visible = the
# served copy resolved).
img_row = page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{REAL_REL}")'),
)
expect(img_row).to_have_count(1)
img = img_row.locator(".kb-doc-thumb-img")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000) # the fetch resolves — no glyph
expect(img).to_have_attribute("src", str(seeded["image_url"]))
# The blank underlay's row — the TODO L4 artifact — is ABSENT.
blank_rows = page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{BLANK_REL}")'),
)
expect(blank_rows).to_have_count(0)
# ---------------------------------------------------------------------------
# 2. The kept doc is the FIRST placement's; exactly ONE vision call
# ---------------------------------------------------------------------------
def test_kept_doc_is_the_first_placement_with_one_vision_call(
app_url: str, seeded: dict[str, object]
) -> None:
"""The dedupe's stability contract + the cost pin: the kept image
doc's synthetic path is the FIRST placement's
(``…#page=1#img=1.png`` — the real image's first insertion; the
underlay consumed index 0 and was skipped), the duplicate
placements' paths (``#page=1#img=2`` and ``#page=2#img=0``) exist
for NO document, and the first sync's capture held EXACTLY ONE
vision describe call — the duplicate placements and the blank
underlay never reached the model (the cost the owner paid in
"several minutes" — pre-phase, every placement was described)."""
with SessionLocal() as db:
img = db.scalar(
select(Document).where(
Document.source == SOURCE_ON, Document.path == REAL_REL
)
)
assert img is not None, "the real image's first placement must be a document"
# Phase 137 regression pin (no behavior change): the document
# context (provenance origin line + page 1's text) reaches the
# vision model (the mock's context-aware reply is the stored
# content), summary === content (the image convention).
assert img.title == "Embedded image from guide page 1"
assert img.image_context == DOC_BLOCK
assert img.content == f"{DOC_BLOCK}\n\n{IMAGE_DESCRIPTION_CONTEXT_ANSWER}"
assert img.summary == img.content
for dup_rel in DUP_RELS:
assert (
db.scalar(
select(Document.id).where(
Document.source == SOURCE_ON, Document.path == dup_rel
)
)
is None
), f"the duplicate placement {dup_rel} must not be a document"
# ONE vision call in the first sync's window (the PDF's text
# summary and the KB overview are different markers — the
# IMAGE_DESCRIPTION_MODE count is the describe calls exactly).
assert seeded["first_sync_vision_calls"] == 1
# ---------------------------------------------------------------------------
# 3. No image document's stored bytes are a blank white image
# ---------------------------------------------------------------------------
def test_no_stored_image_doc_is_blank(app_url: str, seeded: dict[str, object]) -> None:
"""The bytes lens on "no blank white documents": every image
document's PERSISTENT copy (the row's ``image_path`` — the same
bytes the bytes route serves) is checked with the app's own
``is_blank_image`` predicate (imported — the test cannot drift
from the product): NOT blank. The single doc's bytes are the
colorful real image — the underlay's near-white bytes never made
it into the KB (no row, no copy)."""
with SessionLocal() as db:
image_docs = db.scalars(
select(Document).where(
Document.source == SOURCE_ON, Document.is_image.is_(True)
)
).all()
assert len(image_docs) == 1 # the catalog lens (test 1) says the same
for doc in image_docs:
assert doc.image_path is not None and doc.image_path != ""
assert _stored_image_is_blank(doc.image_path) is False, (
f"image doc {doc.path}'s stored copy is blank by the app's own "
"predicate — the TODO L4 artifact is back"
)
# ---------------------------------------------------------------------------
# 4. Re-sync (nothing changed): `unchanged`, creates nothing new
# ---------------------------------------------------------------------------
def test_resync_is_idempotent(
page: Page, app_url: str, seeded: dict[str, object], mock_llm: int
) -> None:
"""Idempotence after the fix: a second Sync (the PDF is
byte-identical — the fast path, no image pass, no vision) settles
on ``0 added · 1 unchanged`` (the PDF doc only — the image doc is
a synthetic path the walk never sees), burns ZERO vision calls in
its window, and the KB is exactly what it was before — the same
two documents, the same image doc row (id + content hash + path
unchanged), the same counts."""
login(page, app_url) # the test's fresh page is a fresh context (no session)
_reset_capture(mock_llm)
_sync_via_ui(page, app_url, expected_result="0 added · 1 unchanged")
# The re-sync's window: no describe call (the image pass never ran
# — the PDF's digest matched), no vision at all.
assert _image_description_calls(mock_llm) == 0
with SessionLocal() as db:
docs = db.scalars(select(Document).where(Document.source == SOURCE_ON)).all()
assert len(docs) == 2 # nothing new, nothing pruned
img = db.scalar(
select(Document).where(
Document.source == SOURCE_ON, Document.path == REAL_REL
)
)
assert img is not None
assert img.id == seeded["image_doc_id"]
assert img.content_hash == seeded["image_doc_hash"]
# The counts surface is unchanged.
expect(page.locator("#stat-docs")).to_have_text("2")
expect(page.locator("#stat-chunks")).to_have_text("5")
# ---------------------------------------------------------------------------
# 5. Negative control (knob OFF): the blank underlay is indexed again
# ---------------------------------------------------------------------------
def test_knob_off_indexes_the_blank_underlay(
page: Page, off_app_url: str, seeded_off: dict[str, object]
) -> None:
"""The kill switch, proven live in the product (not just in unit
tests): the SAME PDF through the SAME user path on
``BOR_PDF_IMAGE_SKIP_BLANK=false`` yields the blank underlay
document TOO — 3 docs / 7 chunks, the underlay's row present in
the drill-down table (its near-white thumbnail served), TWO vision
calls (the pre-phase cost — the blank IS described now), and the
underlay's stored bytes blank per the app's own predicate — the
exact class the story app's gate dropped. The knob gates ONLY the
blank gate: the duplicate placements still produce no documents
(dedupe is a correctness fix), so the catalog is the real image +
the underlay — not the four placements."""
try:
login(page, off_app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("3")
expect(page.locator("#stat-chunks")).to_have_text("7")
_drill_to_source(page, SOURCE_OFF)
expect(page.locator("#docs-tbody tr")).to_have_count(3)
# The underlay's row — absent on the story app (test 1) — is
# BACK: the knob off restored its indexing (the pre-phase-151
# behavior the owner's bug report describes).
blank_row = page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{BLANK_REL}")'),
)
expect(blank_row).to_have_count(1)
blank_img = blank_row.locator(".kb-doc-thumb-img")
expect(blank_img).to_have_count(1)
expect(blank_img).to_be_visible(timeout=15_000) # the near-white copy serves
# The duplicate placements: still no documents (not knobed).
for dup_rel in DUP_RELS:
expect(
page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{dup_rel}")'),
)
).to_have_count(0)
# The bytes lens: the underlay's stored copy is blank by the
# app's own predicate — the exact class the story app skipped
# (test 3 proved its absence there); the real image's copy is
# not blank (unchanged by the knob).
with SessionLocal() as db:
blank = db.scalar(
select(Document).where(
Document.source == SOURCE_OFF, Document.path == BLANK_REL
)
)
real = db.scalar(
select(Document).where(
Document.source == SOURCE_OFF, Document.path == REAL_REL
)
)
assert blank is not None and blank.image_path is not None
assert _stored_image_is_blank(blank.image_path) is True
assert real is not None and real.image_path is not None
assert _stored_image_is_blank(real.image_path) is False
# The pre-phase cost: BOTH images described (the blank too).
assert seeded_off["vision_calls"] == 2
finally:
_truncate_all()
tests/e2e/test_pdf_support.py
+21 -9
View File
@@ -38,7 +38,6 @@ either direction, and without the token the walk is blind to the file):
"""
from __future__ import annotations
import base64
import json
import os
import re
@@ -87,13 +86,26 @@ QUESTION = (
"what does the homelab report say about the backup schedule?"
)
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
#: the pipeline is content-agnostic; the well-formed bytes keep the
#: extraction + copy + serve + render path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _embedded_png() -> bytes:
"""A small deterministic colorful noise PNG (channel means ~97 —
the "real image" stand-in that CLEARS the phase-151 blank/
decorative gate's near-white floor with room to spare). The old
1×1 transparent fixture decomposed in PyMuPDF to a near-white
1-sample base — blank by the owner-locked gate — and is now
(correctly) skipped as a decorative underlay, so the suite's
pipeline pins ride on a real non-blank image instead (the
pipeline is content-agnostic; the well-formed bytes keep the
extraction + copy + serve + render path honest)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
for y in range(18):
for x in range(24):
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
return pix.tobytes("png")
#: The embedded fixture image (24×18 colorful noise — non-blank by the
#: phase-151 gate; see :func:`_embedded_png`).
EMBED_PNG = _embedded_png()
UPLOAD_TIMEOUT_MS = 30_000
SYNC_TIMEOUT_MS = 60_000
@@ -231,7 +243,7 @@ def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
doc.new_page().insert_text((72, 72), PAGE_ONE)
page2 = doc.new_page()
page2.insert_text((72, 72), PAGE_TWO)
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
doc.set_metadata({"title": PDF_TITLE})
doc.save(pdf_path)
doc.close()
tests/unit/test_importer.py
+9 -3
View File
@@ -990,9 +990,12 @@ def test_import_summary_log_line_includes_summary_counters(
phase-106 date-refresh counter, the phase-122 ``images_failed``
counter, the phase-137 ``image_context_backfilled`` counter (next
to ``images_failed``), the phase-133 ``pdfs_screenshotted``
counter, and the phase-140 ``pdfs_pictorial`` /
``pdf_pictorial_pages`` counters (next to ``pdfs_screenshotted``)
sit after ``dates_updated``, before ``formats``."""
counter, the phase-140 ``pdfs_pictorial`` /
``pdf_pictorial_pages`` counters (next to ``pdfs_screenshotted``),
and the phase-151 ``pdf_images_deduped`` / ``pdf_images_blank_skipped``
counters (the parser's per-PDF drop counts, next to
``pdf_pictorial_pages``) sit after ``dates_updated``, before
``formats``."""
s = ImportSummary()
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
s.summaries, s.summary_errors = 2, 1
@@ -1003,6 +1006,8 @@ def test_import_summary_log_line_includes_summary_counters(
s.pdfs_screenshotted = 1
s.pdfs_pictorial = 2
s.pdf_pictorial_pages = 5
s.pdf_images_deduped = 3
s.pdf_images_blank_skipped = 1
s.formats = {"md": 1, "yaml": 2}
with caplog.at_level(logging.INFO, logger="app.importer"):
s.log()
@@ -1012,6 +1017,7 @@ def test_import_summary_log_line_includes_summary_counters(
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
"dates_updated=0 images_failed=0 image_context_backfilled=2 "
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
"formats=yaml:2,md:1"
)
tests/unit/test_pdf.py
+560 -11
View File
@@ -3,11 +3,17 @@
Covers ``app/rag/pdf.py``: text extraction with ``---PAGE-N---``
separators (A2), the per-page text map (phase 137 task 01 —
``extract_pdf_page_texts``, the document-context raw material),
embedded image extraction (original encoded bytes),
MIME mapping (extension first, then image type code, else the
embedded image extraction (one per DISTINCT image — phase 151 task
01's dedupe, first placement wins; a masked image is composited to
the visible alpha PNG — task 02; the raw-format gate runs first,
owner-locked A4 — and the real-file regression pins), MIME mapping
(extension first, then image type code, else the
``application/octet-stream`` fallback), and the fail-soft contract —
unreadable files and broken pages/images return empty results, never
raise. Temp PDFs are built with PyMuPDF itself (no binary fixtures).
raise. The blank/decorative gate (phase 151 task 03), its kill
switch (task 04), and the importer's image-pass integration live in
tests/unit/test_pdf_image_filter.py. Temp PDFs are built with
PyMuPDF itself (no binary fixtures).
"""
from __future__ import annotations
@@ -15,7 +21,9 @@ import asyncio
import hashlib
import logging
import re
import struct
import tarfile
import zlib
from collections.abc import Callable, Sequence
from datetime import timedelta
from pathlib import Path
@@ -43,6 +51,7 @@ from app.rag.pdf import (
PAGE_SEPARATOR,
XMLISH_LINE_FRACTION,
PdfImage,
PdfImageStats,
_mime_from_image_info,
_page_image_entries,
extract_pdf_images,
@@ -602,12 +611,20 @@ def test_pdf_embedded_image_unsupported_mime_is_skipped(
root = tmp_path / "pdfmime"
root.mkdir()
_build_pdf(root / "report.pdf", [("Text page.", [])])
# Phase 151 (task 03): the importer now calls the stats-carrying
# variant — patch it (the tuple shape: images + PdfImageStats).
# Phase 151 (task 04): the knob rides as the `skip_blank`
# keyword — the stand-in accepts it (its value is irrelevant here:
# the tiff mime is rejected by the extension gate either way).
monkeypatch.setattr(
importer,
"extract_pdf_images",
lambda _path: [
"extract_pdf_images_with_stats",
lambda _path, **_kwargs: (
[
PdfImage(page=1, index=0, width=5, height=5, data=b"fakedata", mime="image/tiff")
],
PdfImageStats(duplicates=0, blank_skipped=0, total_found=1),
),
)
llm = _pdf_llm(tmp_path)
try:
@@ -1176,6 +1193,39 @@ def blank_middle_page_pdf(tmp_path: Path) -> Path:
return path
@pytest.fixture()
def same_image_three_pages_pdf(tmp_path: Path) -> Path:
"""3 pages, EACH carrying the SAME image (phase 151 task 01 — the
dedupe fixture: one distinct image, three placements)."""
path = tmp_path / "triple.pdf"
data = _png_bytes(40, 30)
doc = fitz.open()
for n in (1, 2, 3):
page = doc.new_page()
page.insert_text((72, 72), f"Text on page {n}.")
page.insert_image(fitz.Rect(72, 150, 112, 180), stream=data)
doc.save(path)
doc.close()
return path
@pytest.fixture()
def two_distinct_images_pdf(tmp_path: Path) -> Path:
"""Two pages, each with its OWN image (different size AND pixels —
genuinely distinct: the dedupe must NOT merge them)."""
path = tmp_path / "two.pdf"
doc = fitz.open()
p1 = doc.new_page()
p1.insert_image(fitz.Rect(72, 150, 112, 180), stream=_png_bytes(40, 30))
p2 = doc.new_page()
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 15, 15))
pix.clear_with(50) # different pixels (the shared helper uses 200)
p2.insert_image(fitz.Rect(72, 150, 87, 165), stream=pix.tobytes("png"))
doc.save(path)
doc.close()
return path
# ---------------------------------------------------------------------------
# extract_pdf_text
# ---------------------------------------------------------------------------
@@ -1255,22 +1305,26 @@ def test_pdf_without_images_returns_empty_list(single_page_pdf: Path) -> None:
def test_embedded_images_extracted_in_order(multi_page_pdf: Path) -> None:
images = extract_pdf_images(multi_page_pdf)
# Phase 151 (task 01): page 3's SECOND placement of the same JPEG
# (index 1 — the same embedded xref) is deduped: one PdfImage per
# DISTINCT image, first occurrence wins (page/index stay the first
# placement's).
assert [(img.page, img.index) for img in images] == [
(2, 0), (3, 0), (3, 1), (3, 2),
(2, 0), (3, 0), (3, 2),
]
# page 2: 40x30 PNG
img = images[0]
assert (img.width, img.height) == (40, 30)
assert img.mime == "image/png"
assert img.data.startswith(PNG_MAGIC)
# page 3: 40x30 JPEG, then the same JPEG again (distinct index)
for img in images[1:3]:
# page 3: the 40x30 JPEG at its FIRST placement (index 0 — index 1
# is the deduped repeat of the same embedded xref)
img = images[1]
assert (img.width, img.height) == (40, 30)
assert img.mime == "image/jpeg"
assert img.data.startswith(JPEG_MAGIC)
assert images[1].data == images[2].data # same embedded xref
# page 3: 15x15 PNG
img = images[3]
# page 3: 15x15 PNG at index 2
img = images[2]
assert (img.width, img.height) == (15, 15)
assert img.mime == "image/png"
assert img.data.startswith(PNG_MAGIC)
@@ -1407,6 +1461,448 @@ def test_extracted_ext_wins_over_info_ext(
assert images[0].mime == "image/jpeg" # ext beats the PNG type code
# ---------------------------------------------------------------------------
# phase 151, task 01 — one PdfImage per DISTINCT image (the dedupe)
# ---------------------------------------------------------------------------
def test_same_image_three_placements_yields_one_first_wins(
same_image_three_pages_pdf: Path,
) -> None:
"""The task's core pin: the SAME image on 3 pages → ONE PdfImage,
carrying the FIRST occurrence's page/index (page 1, index 0) — the
synthetic path stays stable for the first placement."""
images = extract_pdf_images(same_image_three_pages_pdf)
assert len(images) == 1
assert (images[0].page, images[0].index) == (1, 0)
assert (images[0].width, images[0].height) == (40, 30)
assert images[0].mime == "image/png"
assert images[0].data.startswith(PNG_MAGIC)
def test_two_distinct_images_are_both_kept(
two_distinct_images_pdf: Path,
) -> None:
"""The no-duplicate path: genuinely distinct images (different size
AND pixels) are NOT merged — two entries in page order."""
images = extract_pdf_images(two_distinct_images_pdf)
assert [(img.page, img.index) for img in images] == [(1, 0), (2, 0)]
assert images[0].data != images[1].data
assert (images[0].width, images[0].height) == (40, 30)
assert (images[1].width, images[1].height) == (15, 15)
def test_byte_identical_images_under_distinct_xrefs_dedupe_to_one(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
) -> None:
"""The DIGEST path: two DIFFERENT xrefs holding byte-identical
bytes → one entry (the first), one drop — the xref check alone
cannot catch this, the sha256 of the stored bytes can."""
path = tmp_path / "fake.pdf"
path.write_bytes(b"%PDF-1.4 fake")
def _extract(xref: int) -> dict[str, Any]:
return {"image": b"identical-bytes", "ext": "png", "width": 5, "height": 5}
doc = _FakeDoc(
[
_fake_image_page(
[
{"xref": 7, "width": 5, "height": 5},
{"xref": 9, "width": 5, "height": 5},
]
)
],
_extract,
)
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
images = extract_pdf_images(path)
assert len(images) == 1
assert (images[0].page, images[0].index) == (1, 0)
def test_different_bytes_under_distinct_xrefs_are_both_kept(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
) -> None:
"""The digest path's negative control: distinct xrefs with
DIFFERENT bytes are distinct images — both kept."""
path = tmp_path / "fake.pdf"
path.write_bytes(b"%PDF-1.4 fake")
def _extract(xref: int) -> dict[str, Any]:
return {
"image": b"first-bytes" if xref == 7 else b"second-bytes",
"ext": "png", "width": 5, "height": 5,
}
doc = _FakeDoc(
[
_fake_image_page(
[
{"xref": 7, "width": 5, "height": 5},
{"xref": 9, "width": 5, "height": 5},
]
)
],
_extract,
)
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
images = extract_pdf_images(path)
assert [(img.page, img.index) for img in images] == [(1, 0), (1, 1)]
def test_duplicate_drop_logs_one_info_line_per_pdf(
same_image_three_pages_pdf: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""One INFO line per PDF that dropped anything (the task's exact
format), one DEBUG line per drop; 3 placements → 2 dropped,
1 unique."""
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
assert len(extract_pdf_images(same_image_three_pages_pdf)) == 1
infos = [
r
for r in caplog.records
if r.levelno == logging.INFO
and "duplicate image occurrence(s)" in r.getMessage()
]
assert len(infos) == 1 # ONE line per PDF, not one per drop
assert infos[0].getMessage() == (
f"pdf: dropped 2 duplicate image occurrence(s) "
f"from {same_image_three_pages_pdf} (1 unique)"
)
debugs = [
r
for r in caplog.records
if r.levelno == logging.DEBUG and "duplicate image" in r.getMessage()
]
assert len(debugs) == 2 # one per dropped occurrence
assert all(r.getMessage().endswith("— skipped") for r in debugs)
def test_no_duplicates_no_info_line(
two_distinct_images_pdf: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A PDF with no duplicates never logs the INFO line — the line is
the duplicate signal, its absence means nothing was dropped."""
with caplog.at_level(logging.INFO, logger="app.rag.pdf"):
assert len(extract_pdf_images(two_distinct_images_pdf)) == 2
assert not [
r
for r in caplog.records
if "duplicate image occurrence(s)" in r.getMessage()
]
#: The owner's real 50-page fixture (untracked — ``test_sources/`` is
#: gitignored): the podman + systemd guide. THE phase-151 reproducer —
#: its table-shading underlays are the "blank white images" the TODO
#: reported.
PODMAN_PDF = Path(__file__).resolve().parents[2] / "test_sources" / "podman-systemd.pdf"
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
def test_podman_pdf_yields_one_entry_per_distinct_image() -> None:
"""THE real-file pin: 48 placements (one per ``get_image_info``
entry — the phase docs measured 46) spanning 15 unique xrefs — but
6 of those xrefs are BYTE-IDENTICAL copies (same base AND same
SMask bytes stored under a different xref), so there are only 9
truly DISTINCT images. Task 01's digest set is spec'd for exactly
the "byte-identical images stored under different xrefs" case, so
it merges the 6 copies: 48 placements → 9 entries (the task
file's "15" figure is the unique-XREF count — unreachable while
keeping the digest set AND the phase's "one document per genuinely
distinct image" objective, since those 6 xrefs are genuine
duplicates a viewer cannot tell apart). First-occurrence order is
pinned too: every kept entry carries its FIRST placement's
page/index.
Phase 151 (task 03): the dedupe pin runs with the blank gate OFF
(``skip_blank=False``) — the gate (default on) drops all 9 distinct
images (they are the decorative underlays the phase exists to
filter), so the default now returns ``[]`` for this file (pinned
in tests/unit/test_pdf_image_filter.py)."""
images = extract_pdf_images(PODMAN_PDF, skip_blank=False)
assert len(images) == 9
assert [(img.page, img.index) for img in images] == [
(7, 0), (22, 0), (26, 0), (30, 0), (33, 0),
(36, 0), (41, 0), (45, 0), (47, 0),
]
# no two surviving entries share bytes (the dedupe's contract, on
# the real file)
digests = {hashlib.sha256(img.data).hexdigest() for img in images}
assert len(digests) == 9
# ---------------------------------------------------------------------------
# phase 151, task 02 — the /SMask is applied (the visible image, not
# the blank base stream extract_image() returns)
# ---------------------------------------------------------------------------
def _png_chunk(tag: bytes, payload: bytes) -> bytes:
"""One PNG chunk (4-byte length + tag + payload + CRC32)."""
return (
struct.pack(">I", len(payload))
+ tag
+ payload
+ struct.pack(">I", zlib.crc32(tag + payload) & 0xFFFFFFFF)
)
def _make_png(
width: int, height: int, rgba: bool, pixel: Callable[[int, int], tuple[int, ...]]
) -> bytes:
"""A deterministic PNG built with stdlib only (``zlib`` +
``struct``). PyMuPDF's Pixmap exposes no sample mutation in this
release, so the fixture is hand-rolled: a genuine RGBA
(color-type-6) PNG is exactly what makes ``insert_image`` store a
``/SMask`` (base image + separate mask xref — the shape this task
composites)."""
color_type = 6 if rgba else 2
n = 4 if rgba else 3
raw = bytearray()
for y in range(height):
raw.append(0) # filter type: none
for x in range(width):
raw += bytes(pixel(x, y)[:n])
ihdr = struct.pack(">IIBBBBB", width, height, 8, color_type, 0, 0, 0)
return (
b"\x89PNG\r\n\x1a\n"
+ _png_chunk(b"IHDR", ihdr)
+ _png_chunk(b"IDAT", zlib.compress(bytes(raw)))
+ _png_chunk(b"IEND", b"")
)
def _insert_image_pdf(path: Path, png: bytes) -> Path:
"""A one-page PDF carrying *png* (40x30, top-left at 72,150)."""
doc = fitz.open()
doc.new_page().insert_image(fitz.Rect(72, 150, 112, 180), stream=png)
doc.save(path)
doc.close()
return path
def _raw_extract(path: Path) -> dict[str, Any]:
"""The page's first image as ``extract_image`` reports it — the
raw reference (what the PRE-phase extractor returned verbatim)."""
doc = fitz.open(str(path))
with doc:
info = doc[0].get_image_info(xrefs=True)[0]
return doc.extract_image(int(info["xref"]))
def _masked_png() -> bytes:
"""A 40x30 RGBA PNG with varying pixels and a non-uniform alpha
(top half opaque, bottom half fully transparent — the mask must
survive the round trip)."""
return _make_png(
40, 30, True, lambda x, y: (x % 256, y % 256, (x + y) % 256, 255 if y < 15 else 0)
)
def test_masked_image_is_composited_to_the_visible_alpha_png(tmp_path: Path) -> None:
"""THE task's core pin: an RGBA PNG inserted via ``insert_image``
(PyMuPDF stores it as a BASE image + a ``/SMask``) is extracted as
the composited VISIBLE image: (a) the raw ``extract_image`` bytes ≠
the extractor's ``data``, (b) ``mime == "image/png"``, (c)
re-opening the extractor's bytes as a ``fitz.Pixmap`` reports
``alpha == 1`` (the mask survived) at the base's dimensions — and
``width``/``height`` come from ``extract_image`` (unchanged)."""
path = _insert_image_pdf(tmp_path / "masked.pdf", _masked_png())
raw = _raw_extract(path)
# Precondition (the bug this task fixes): the /SMask is stored
# separately, and the raw extract is the maskless base.
assert int(raw.get("smask") or 0) != 0
assert fitz.Pixmap(raw["image"]).alpha == 0
images = extract_pdf_images(path)
assert len(images) == 1
img = images[0]
assert (img.page, img.index) == (1, 0)
assert img.data != raw["image"] # (a) the composite, not the raw base
assert img.mime == "image/png" # (b)
assert img.data.startswith(PNG_MAGIC)
vis = fitz.Pixmap(img.data)
assert vis.alpha == 1 # (c) the mask survived
assert (vis.width, vis.height) == (40, 30)
assert (img.width, img.height) == (40, 30) # from extract_image
def test_unmasked_image_is_byte_identical_to_pre_phase(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The common case is unchanged: an image WITHOUT a mask is
returned exactly as the raw ``extract_image`` bytes (no composite,
no re-encode — pre-phase behavior) and no composite line is
logged."""
png = _make_png(40, 30, False, lambda x, y: (x % 256, y % 256, (x + y) % 256))
path = _insert_image_pdf(tmp_path / "plain.pdf", png)
raw = _raw_extract(path)
assert int(raw.get("smask") or 0) == 0
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
images = extract_pdf_images(path)
assert len(images) == 1
assert images[0].data == raw["image"] # the raw bytes, verbatim
assert images[0].mime == "image/png" # the raw mime (ext = png)
assert not [
r for r in caplog.records if "composited smask=" in r.getMessage()
]
def test_mask_composite_logs_one_debug_line(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""One DEBUG line (the task's exact format) when a mask was
applied."""
path = _insert_image_pdf(tmp_path / "masked.pdf", _masked_png())
smask = int(_raw_extract(path).get("smask") or 0)
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
extract_pdf_images(path)
debugs = [
r
for r in caplog.records
if r.levelno == logging.DEBUG and "composited smask=" in r.getMessage()
]
assert len(debugs) == 1
assert debugs[0].getMessage() == (
f"pdf: composited smask={smask} for image page=1 index=0 of {path}"
)
def test_composite_failure_falls_back_to_raw_bytes_with_warning(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Fail-soft: a compositing that RAISES (the monkeypatched
``fitz.Pixmap``) returns the raw bytes + raw mime, logs the
warning (the task's exact format), and never loses the image.
No exception escapes the extractor. The ``_FakeDoc`` stand-in
(the task-01 convention) is used so the extractor's compositing
block is the ONLY ``fitz.Pixmap`` call in flight (PyMuPDF's real
constructor resolves the module-global ``Pixmap`` internally, so
patching it would break the real base/mask reads too)."""
path = tmp_path / "fake.pdf"
path.write_bytes(b"%PDF-1.4 fake")
raw_bytes = b"raw-base-bytes"
doc = _FakeDoc(
[_fake_image_page([{"xref": 7, "width": 5, "height": 5}])],
lambda xref: {
"image": raw_bytes, "ext": "png", "width": 5, "height": 5, "smask": 8,
},
)
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
attempts: list[tuple[tuple[Any, ...], dict[str, Any]]] = []
def _boom(*args: Any, **kwargs: Any) -> None:
attempts.append((args, kwargs))
raise RuntimeError("simulated combine failure")
monkeypatch.setattr(pdf_mod.fitz, "Pixmap", _boom)
with caplog.at_level(logging.WARNING, logger="app.rag.pdf"):
images = extract_pdf_images(path)
assert attempts, "the composite path was attempted (a Pixmap was read)"
assert len(images) == 1 # the image is NOT lost
assert images[0].data == raw_bytes # raw bytes, verbatim
assert images[0].mime == "image/png" # the raw mime
warnings = [
r
for r in caplog.records
if r.levelno == logging.WARNING and "could not composite smask=" in r.getMessage()
]
assert len(warnings) == 1
assert warnings[0].getMessage() == (
f"pdf: could not composite smask=8 for image page=1 index=0 "
f"of {path} — simulated combine failure; using raw bytes"
)
def test_unsupported_raw_format_with_mask_stays_untouched(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Owner-locked A4 — the raw-format gate runs FIRST: an image
whose raw format the importer's ``_MIME_TO_EXT`` gate will reject
(``jpx``/JPEG2000 → the ``application/octet-stream`` fallback — the
586 images in ``players_handbook.pdf``) is NOT composited even
when it carries a ``/SMask``: raw bytes + raw mime, and the
``fitz.Pixmap`` combiner is never called (compositing must never
newly admit an unsupported format)."""
path = tmp_path / "fake.pdf"
path.write_bytes(b"%PDF-1.4 fake")
raw_bytes = b"jpx-base-bytes"
doc = _FakeDoc(
[_fake_image_page([{"xref": 7, "width": 5, "height": 5}])],
lambda xref: {
"image": raw_bytes, "ext": "jpx", "width": 5, "height": 5, "smask": 8,
},
)
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
def _no_composite(*args: Any, **kwargs: Any) -> None:
# Phase 151 (task 03): the blank gate also READS the base
# pixmap (Pixmap(doc, xref) — a fail-soft probe that keeps the
# image when it raises, as it does on this fake doc) — that is
# not compositing. Compositing is the Pixmap(Pixmap, Pixmap)
# call (first arg a Pixmap): the raw-format gate must still run
# before it, so that shape alone is the failure sentinel.
if args and isinstance(args[0], fitz.Pixmap):
pytest.fail("the raw-format gate must run before any compositing")
raise ValueError("fake doc — no real pixmap (test sentinel)")
monkeypatch.setattr(pdf_mod.fitz, "Pixmap", _no_composite)
images = extract_pdf_images(path)
assert len(images) == 1
assert images[0].data == raw_bytes # raw bytes, verbatim
assert images[0].mime == "application/octet-stream" # raw mime (→ .bin → skipped)
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
def test_podman_pdf_masked_images_are_composited() -> None:
"""Real-file pin: EVERY distinct image of the owner's podman PDF
carries a ``/SMask`` pre-phase (``has-mask=True`` in
``get_image_info``, a non-zero ``smask`` in ``extract_image``), so
post-phase the extractor's output for it is the composited visible
image: each entry decodes as an alpha PNG (``alpha == 1``, the
base's dimensions) whose bytes DIFFER from the raw base stream
``extract_image`` returns (the maskless DeviceGray base — the
"blank white image" this phase fixes). The stronger per-entry
form of the task's "at least one xref" check. Phase 151 (task 03)
landed: the blank gate (default on) removes these entries
entirely, so this compositing pin runs with the gate OFF
(``skip_blank=False``) — the default's ``[]`` result for this
file is pinned in tests/unit/test_pdf_image_filter.py."""
images = extract_pdf_images(PODMAN_PDF, skip_blank=False)
assert len(images) == 9 # the task-01 dedupe pin still holds
# The pre-phase property (every placement masked) + the raw base
# digests (the "blank" bytes the pre-phase extractor stored).
raw_digests: set[str] = set()
doc = fitz.open(str(PODMAN_PDF))
with doc:
for page in doc:
for info in page.get_image_info(xrefs=True):
xref = int(info.get("xref") or 0)
if not xref:
continue
assert info.get("has-mask") is True
raw = doc.extract_image(xref)
assert int(raw.get("smask") or 0) != 0
raw_digests.add(hashlib.sha256(raw["image"]).hexdigest())
for img in images:
assert img.mime == "image/png"
assert img.data.startswith(PNG_MAGIC)
vis = fitz.Pixmap(img.data)
assert vis.alpha == 1
assert (vis.width, vis.height) == (img.width, img.height)
# the composite differs from the raw base (every entry)
assert hashlib.sha256(img.data).hexdigest() not in raw_digests
# ---------------------------------------------------------------------------
# _mime_from_image_info
# ---------------------------------------------------------------------------
@@ -1903,3 +2399,56 @@ def test_extract_pdf_text_regression_pin_horse() -> None:
text = extract_pdf_text(HORSE_PDF)
assert text == _HORSE_EXACT
assert hashlib.sha256(text.encode("utf-8")).hexdigest() == _HORSE_SHA256
# ---------------------------------------------------------------------------
# phase 151 — real-file regression pins (skip-if-missing, the
# ``HANDBOOK`` / ``HORSE_PDF`` idiom; test_sources/ is gitignored)
# ---------------------------------------------------------------------------
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
def test_podman_pdf_returns_no_image_documents_by_default() -> None:
"""THE reported bug, pinned (the owner's exact file — the bug's
tombstone test): the 50-page podman + systemd guide places its
table-shading underlays 48 times (``get_image_info`` reports one
entry per PLACEMENT); post-phase the extractor returns NONE of
them — task 01's dedupe collapses the placements to 9 distinct
images (39 dropped — 15 unique xrefs of which 6 are byte-identical
copies, so the digest set merges them; the task file's 31/15
figure assumed xref-only dedupe over 46 placements — the binding
outcome is identical: 0 out) and task 03's blank gate skips all 9
(near-white underlays — ASSUMPTION A1: never indexed, never
described). The full drop accounting and the log lines are pinned
in tests/unit/test_pdf_image_filter.py."""
assert extract_pdf_images(PODMAN_PDF) == []
@pytest.mark.skipif(not HORSE_PDF.is_file(), reason="horse fixture missing")
def test_horse_pdf_returns_exactly_one_non_blank_image() -> None:
"""The real-file "kept" pin: the horse photo (mean 123, uniq 256,
uniform ~0.20 — the phase overview's measured basis) clears the
blank gate's mean floor — exactly ONE image out, NOT blank-skipped
(being a masked JPEG, it is stored as the task-02 composite:
alpha PNG)."""
images = extract_pdf_images(HORSE_PDF)
assert len(images) == 1
assert images[0].mime == "image/png"
assert images[0].data.startswith(PNG_MAGIC)
assert fitz.Pixmap(images[0].data).alpha == 1
@pytest.mark.skipif(not HANDBOOK.is_file(), reason="handbook fixture missing")
def test_handbook_pdf_images_keep_the_unmapped_jpx_mime() -> None:
"""The real-file "no behavior change" pin: the 293-page handbook's
586 JPEG2000 page-art images are STILL returned by the parser
(no gate drop, no compositing — owner-locked A4: an unsupported
raw format is left exactly as pre-phase) with the UNMAPPED raw
mime (``application/octet-stream`` — ``jpx`` is not in the
importer's ``_MIME_TO_EXT``), so the importer's extension gate
skips them exactly as pre-phase (assert the mime mapping here,
not the importer — its gate behavior is pinned in test_importer.
py)."""
images = extract_pdf_images(HANDBOOK)
assert len(images) == 586
assert all(img.mime == "application/octet-stream" for img in images)
tests/unit/test_pdf_image_filter.py
+1061
View File
File diff suppressed because it is too large Load Diff