phase: 151_pdf_image_extraction
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree. All verification is complete. Final state: **Phase 151 — final verification pass: complete, project 100% green.** - **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite - **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped) - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors - **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions) - **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope) - **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays) - **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
@@ -64,9 +64,9 @@ Fixture provenance:
|
||||
garbage line repeated 20× (``<illustrator:StartupProfile>Web</…>``
|
||||
— 100% of the lines look like tags, so the task-02 gate fails
|
||||
deterministically), plus ONE small embedded raster image per page
|
||||
(a 1×1 PNG via ``insert_image`` — the fallback's embedded-image leg
|
||||
runs too: the screenshots are the CONTENT, the extracted images are
|
||||
the citation figures — both, no conflict).
|
||||
(a colorful noise PNG via ``insert_image`` — the fallback's
|
||||
embedded-image leg runs too: the screenshots are the CONTENT, the
|
||||
extracted images are the citation figures — both, no conflict).
|
||||
* ``cleandoc/`` — ONE small clean PDF (PyMuPDF, real prose text, no
|
||||
images) — the gate's no-false-negative subject.
|
||||
|
||||
@@ -101,7 +101,6 @@ Test → observable mapping (Playwright Mapping Rule):
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
@@ -181,13 +180,33 @@ CLEAN_PROSE = (
|
||||
"The clean reference page covers the homelab firewall rule audit."
|
||||
)
|
||||
|
||||
#: A real 1×1 transparent PNG (the E2E suites' fixture — the pipeline
|
||||
#: is content-agnostic; the well-formed bytes keep the extraction +
|
||||
#: copy + serve path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
def _embedded_png(seed: int) -> bytes:
|
||||
"""A small deterministic colorful noise PNG (channel means ~97 —
|
||||
the "real image" stand-in that CLEARS the phase-151 blank/
|
||||
decorative gate's near-white floor with room to spare). The old
|
||||
1×1 transparent fixture decomposed in PyMuPDF to a near-white
|
||||
1-sample base — blank by the owner-locked gate — and is now
|
||||
(correctly) skipped as a decorative underlay, so the suite's
|
||||
pipeline pins ride on a real non-blank image instead (the
|
||||
pipeline is content-agnostic; the well-formed bytes keep the
|
||||
extraction + copy + serve path honest). *seed* offsets the pixel
|
||||
formula — the two pages carry DISTINCT images (the same stream
|
||||
twice would be one placement pair: the phase-151 dedupe keeps the
|
||||
first and drops the repeat, and this suite pins one image DOC PER
|
||||
PAGE)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
|
||||
for y in range(18):
|
||||
for x in range(24):
|
||||
pix.set_pixel(
|
||||
x, y,
|
||||
((x * 7 + seed) % 256, (y * 13 + seed) % 256, ((x + y) * 5 + seed) % 256),
|
||||
)
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
#: The two embedded fixture images (24×18 colorful noise — non-blank
|
||||
#: by the phase-151 gate, distinct per page — see :func:`_embedded_png`).
|
||||
EMBED_PNGS = (_embedded_png(0), _embedded_png(1))
|
||||
|
||||
|
||||
def _build_garbage_pdf(path: Path) -> None:
|
||||
@@ -196,16 +215,18 @@ def _build_garbage_pdf(path: Path) -> None:
|
||||
= the owner's exact garbage line × 20 (the gate's XML signal fails
|
||||
deterministically — 100% of the non-empty lines look like tags;
|
||||
the ``---PAGE-N---`` separators are not tag-shaped), plus ONE small
|
||||
embedded raster image per page (the 1×1 PNG — the fallback's
|
||||
embedded-image leg runs too). The text layer is real EXTRACTABLE
|
||||
text — this is the FINDING's class (the pre-126 raw-XMP soup), not
|
||||
the empty-layer class."""
|
||||
embedded raster image per page (a DISTINCT noise PNG per page —
|
||||
the fallback's embedded-image leg runs too). The text layer is real
|
||||
EXTRACTABLE text — this is the FINDING's class (the pre-126 raw-XMP
|
||||
soup), not the empty-layer class."""
|
||||
doc = fitz.open()
|
||||
for _page_num in range(1, N_GARBAGE_PAGES + 1):
|
||||
for page_num in range(1, N_GARBAGE_PAGES + 1):
|
||||
page = doc.new_page()
|
||||
for i in range(GARBAGE_LINES_PER_PAGE):
|
||||
page.insert_text((72, 72 + i * 14), GARBAGE_LINE)
|
||||
page.insert_image(fitz.Rect(72, 500, 73, 501), stream=PNG_1X1)
|
||||
page.insert_image(
|
||||
fitz.Rect(72, 500, 96, 518), stream=EMBED_PNGS[page_num - 1]
|
||||
)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
@@ -648,7 +669,7 @@ def test_garbage_pdf_takes_the_screenshot_path(
|
||||
assert "<illustrator" not in content
|
||||
|
||||
# The fallback's embedded-image leg: both extracted images (one per
|
||||
# page, the 1×1 PNG — ``#page=N#img=0`` paths) indexed — and phase
|
||||
# page, the noise PNG — ``#page=N#img=0`` paths) indexed — and phase
|
||||
# 137 (task 02, A7) gives each the ORIGIN-LINE-ONLY document
|
||||
# context: the garbage text layer failed the gate, so the "page
|
||||
# text" is dropped (the soup never reaches the embedder), while the
|
||||
|
||||
@@ -16,7 +16,7 @@ The suite's single module app runs with ``BOR_IMAGES=true`` + the
|
||||
pattern — the conftest env with the leak guards):
|
||||
|
||||
* **Seed 1** (source ``e2e-pdf-imgctx``): ONE fixture PDF at the
|
||||
source root (two text pages + one embedded 1×1 PNG on page 2 — the
|
||||
source root (two text pages + one embedded noise PNG on page 2 — the
|
||||
phase-126 PyMuPDF recipe, built in the fixture) arrives through the
|
||||
REAL admin upload flow + the RAG page's Sync button. The forward
|
||||
path (phase 137 task 02) is the story: the embedded image's
|
||||
@@ -48,7 +48,6 @@ not duplicated here).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
@@ -95,13 +94,26 @@ PDF_TITLE = "Homelab Report" # the metadata title
|
||||
PAGE_ONE = "The quarterly homelab report covers the core switch firmware upgrade."
|
||||
PAGE_TWO = "The homelab backup schedule runs nightly at 02:00 and keeps seven rotations."
|
||||
|
||||
#: A real 1×1 transparent PNG (the house fixture — the pipeline is
|
||||
#: content-agnostic; the well-formed bytes keep the extraction + copy
|
||||
#: path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
def _embedded_png() -> bytes:
|
||||
"""A small deterministic colorful noise PNG (channel means ~97 —
|
||||
the "real image" stand-in that CLEARS the phase-151 blank/
|
||||
decorative gate's near-white floor with room to spare). The old
|
||||
1×1 transparent fixture decomposed in PyMuPDF to a near-white
|
||||
1-sample base — blank by the owner-locked gate — and is now
|
||||
(correctly) skipped as a decorative underlay, so the suite's
|
||||
pipeline pins ride on a real non-blank image instead (the
|
||||
pipeline is content-agnostic; the well-formed bytes keep the
|
||||
extraction + copy path honest)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
|
||||
for y in range(18):
|
||||
for x in range(24):
|
||||
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
#: The embedded fixture image (24×18 colorful noise — non-blank by the
|
||||
#: phase-151 gate; see :func:`_embedded_png`).
|
||||
EMBED_PNG = _embedded_png()
|
||||
|
||||
# Seed 1's stored document block (NO markdown reference exists in that
|
||||
# source — the doc block alone: the provenance's origin line + the
|
||||
@@ -241,14 +253,14 @@ def app_url(app_server: str) -> str:
|
||||
|
||||
def _build_pdf(root: Path) -> Path:
|
||||
"""The fixture PDF (the phase-126 recipe, built LOCALLY): two
|
||||
text pages + one embedded 1×1 PNG on page 2 + the metadata
|
||||
text pages + one embedded noise PNG on page 2 + the metadata
|
||||
title."""
|
||||
pdf_path = root / DOC_PATH
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), PAGE_ONE)
|
||||
page2 = doc.new_page()
|
||||
page2.insert_text((72, 72), PAGE_TWO)
|
||||
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
|
||||
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
|
||||
doc.set_metadata({"title": PDF_TITLE})
|
||||
doc.save(pdf_path)
|
||||
doc.close()
|
||||
|
||||
@@ -0,0 +1,802 @@
|
||||
"""Phase 151 E2E (Playwright): a PDF sync produces no blank, no duplicate
|
||||
image documents — the owner's complaint, proven fixed in the browser
|
||||
(TODO L4: "it extracted a series of blank white images … there don't
|
||||
appear to be any blank images in the document, is this a bug?").
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the vision call
|
||||
counts are read from the mock's request capture (the
|
||||
``/v1/e2e/captured`` pair), which only the deterministic mock exposes.
|
||||
|
||||
Two module apps (the ``test_pdf_support.py`` per-suite-app pattern —
|
||||
the conftest env with the leak guards, ``BOR_IMAGES=true`` + the
|
||||
``pdf`` import token pinned EXPLICITLY on both, the mock LLM only):
|
||||
|
||||
* **the story app** (``BOR_PDF_IMAGE_SKIP_BLANK=true`` — the default,
|
||||
pinned so an operator's local gitignored ``.env`` cannot leak the
|
||||
premise off): the fixture PDF (below) arrives through the REAL admin
|
||||
upload flow + the RAG page's **Sync** button (the same user path the
|
||||
owner used). Through the product path:
|
||||
- the catalog carries EXACTLY ONE image document for the PDF — the
|
||||
distinct real image at its FIRST placement (``…#page=1#img=1.png``
|
||||
— the dedupe's stability contract: the same-page duplicate at
|
||||
``#img=2`` and the cross-page duplicate on page 2 — the podman
|
||||
case — never produced their own documents);
|
||||
- NO image document's stored bytes are a blank white image — the
|
||||
stored copy is checked with the APP'S OWN predicate
|
||||
(``app.rag.pdf.is_blank_image`` imported, not re-derived, so the
|
||||
test cannot drift from the product);
|
||||
- the mock's vision call count for image description is 1 (the
|
||||
duplicate placements and the blank underlay never reached the
|
||||
model — the cost the owner paid in "several minutes");
|
||||
- a follow-up re-sync (nothing changed) settles on
|
||||
``0 added · 1 unchanged`` and creates nothing new — idempotence
|
||||
after the fix.
|
||||
* **the knob-off app** (``BOR_PDF_IMAGE_SKIP_BLANK=false`` — the
|
||||
negative control): the SAME PDF through the SAME user path now
|
||||
yields the blank underlay document too (the knob is proven live in
|
||||
the product, not just in unit tests): two image documents, the
|
||||
underlay's stored bytes blank per the app's own predicate — the
|
||||
exact class the story app skipped — and two vision calls (the
|
||||
pre-phase cost). The knob only gates the blank gate: the duplicate
|
||||
placements still produce no documents (dedupe is a correctness
|
||||
fix, not knobed).
|
||||
|
||||
The fixture PDF (built with PyMuPDF in the fixture — the house
|
||||
convention, no binary committed): two text pages; page 1 carries the
|
||||
near-white 200×200 "decorative underlay" (image index 0 — the blank
|
||||
gate's target), then the real image — genuinely distinct (colorful
|
||||
noise) from the underlay — inserted TWICE (indices 1 and 2, the
|
||||
same-page duplicate placement); page 2 carries one more placement of
|
||||
the real image (index 0 — the cross-page duplicate, the podman case,
|
||||
where each image was placed 2–4 times). Four placements, two distinct
|
||||
images, so the extractor must index exactly ONE image document and
|
||||
make exactly ONE vision call (the task's "exactly one image document
|
||||
for the PDF" and "vision call count is 1" pins fix the distinct
|
||||
non-blank total at one — the colorful real image IS the genuinely
|
||||
distinct image the task's fixture list names).
|
||||
|
||||
Test-order note: both apps share the E2E Postgres — the story seed
|
||||
runs first, the knob-off seed truncates and re-seeds (the phase-126
|
||||
isolation pattern), so every story test runs before the knob-off
|
||||
test, which truncates at the end.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import zipfile
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (house convention)
|
||||
import pytest
|
||||
from playwright.sync_api import Browser, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.pdf import is_blank_image
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import IMAGE_DESCRIPTION_CONTEXT_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
SOURCE_ON = "e2e-pdf151" # the story app's archive stem (archive_source_name)
|
||||
SOURCE_OFF = "e2e-pdf151-off" # the knob-off app's archive stem
|
||||
DOC_PATH = "guide.pdf"
|
||||
|
||||
#: The synthetic embedded-image paths (the importer's
|
||||
#: ``<pdf path>#page=<N>#img=<index>.<ext>`` rule). The underlay sits
|
||||
#: at page 1 index 0; the real image's FIRST placement at page 1
|
||||
#: index 1; the duplicates at page 1 index 2 and page 2 index 0 (they
|
||||
#: must never appear as documents).
|
||||
BLANK_REL = f"{DOC_PATH}#page=1#img=0.png" # the underlay (skipped by default)
|
||||
REAL_REL = f"{DOC_PATH}#page=1#img=1.png" # the real image, first placement
|
||||
DUP_RELS = (
|
||||
f"{DOC_PATH}#page=1#img=2.png", # the same-page duplicate
|
||||
f"{DOC_PATH}#page=2#img=0.png", # the cross-page duplicate (podman case)
|
||||
)
|
||||
|
||||
PDF_TITLE = "Podman Systemd Guide" # the metadata title
|
||||
|
||||
#: The fixture PDF's page texts (the image's page — page 1 — carries
|
||||
#: the document context the phase-137 block stores on the image doc).
|
||||
PAGE_ONE = "The podman systemd guide covers container unit files and their restart policies."
|
||||
PAGE_TWO = "The guide's appendix lists the journal log locations and the default timeouts."
|
||||
|
||||
#: The kept image's stored document block (no markdown reference exists
|
||||
#: in the source — the doc block alone: provenance origin line + the
|
||||
#: page's text).
|
||||
DOC_BLOCK = f"Extracted from {SOURCE_ON}/{DOC_PATH} (page 1):\n{PAGE_ONE}"
|
||||
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture PDF: the images (stdlib-free, PyMuPDF-built — deterministic)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _solid_png(width: int, height: int, value: int) -> bytes:
|
||||
"""A uniform RGB PNG (one ``value`` per channel). At 250 the
|
||||
underlay's samples are mean 250 (≥ the 235 near-white floor) and
|
||||
100% within any tolerance band — blank by
|
||||
:func:`app.rag.pdf.is_blank_image` with room to spare."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height))
|
||||
pix.clear_with(value)
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
def _noise_png(width: int, height: int) -> bytes:
|
||||
"""A deterministic colorful noise PNG (channel means ~127 — the
|
||||
"real photo" stand-in: it clears the mean gate with room to spare,
|
||||
and its scattered samples clear the uniformity gate). Genuinely
|
||||
distinct from the underlay in both channels AND structure."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height))
|
||||
for y in range(height):
|
||||
for x in range(width):
|
||||
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
#: The two fixture images: the near-white 200×200 decorative underlay
|
||||
#: (the TODO L4 "blank white" class) and the genuinely distinct
|
||||
#: colorful real image.
|
||||
BLANK_PNG = _solid_png(200, 200, 250)
|
||||
REAL_PNG = _noise_png(96, 64)
|
||||
|
||||
|
||||
def _build_fixture_pdf(path: Path) -> None:
|
||||
"""The phase-151 fixture: two text pages; page 1 carries the
|
||||
underlay (index 0), the real image (index 1), and a SECOND
|
||||
placement of the same real image (index 2 — the same-page
|
||||
duplicate); page 2 carries one more placement of the real image
|
||||
(index 0 — the cross-page duplicate, the podman case). PyMuPDF
|
||||
stores the repeated insertions under one xref (the
|
||||
repeated-placement path — the digest set is the backstop for
|
||||
byte-identical copies under distinct xrefs)."""
|
||||
doc = fitz.open()
|
||||
page1 = doc.new_page()
|
||||
page1.insert_text((72, 72), PAGE_ONE)
|
||||
y = 150
|
||||
for data in (BLANK_PNG, REAL_PNG, REAL_PNG):
|
||||
pix = fitz.Pixmap(data)
|
||||
page1.insert_image(
|
||||
fitz.Rect(72, y, 72 + pix.width, y + pix.height), stream=data
|
||||
)
|
||||
y += pix.height + 10
|
||||
page2 = doc.new_page()
|
||||
page2.insert_text((72, 72), PAGE_TWO)
|
||||
pix = fitz.Pixmap(REAL_PNG)
|
||||
page2.insert_image(
|
||||
fitz.Rect(72, 150, 72 + pix.width, 150 + pix.height), stream=REAL_PNG
|
||||
)
|
||||
doc.set_metadata({"title": PDF_TITLE})
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module apps: the story (knob ON — the default) and the knob-off
|
||||
# negative control. Each owns its port + its scratch upload/source/
|
||||
# image dirs. Both pin BOR_IMPORT_EXTENSIONS WITH the ``pdf`` token
|
||||
# and BOR_IMAGES=true explicitly; the knob under test is pinned per
|
||||
# app (a process-wide lru_cached setting — the two apps are the only
|
||||
# way to prove it live in the product).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(
|
||||
mock_llm: int, app_port: int, *, skip_blank: bool, scratch: Path
|
||||
) -> dict[str, str]:
|
||||
"""The conftest app env (leak guards included) with the phase-151
|
||||
knobs: ``BOR_IMPORT_EXTENSIONS`` pinned to a set carrying the
|
||||
``pdf`` token, ``BOR_IMAGES`` pinned true (the embedded-image pass
|
||||
must run on BOTH apps), ``BOR_PDF_IMAGE_SKIP_BLANK`` pinned per
|
||||
app (true for the story app, false for the negative control —
|
||||
the contract under test cannot leak in either direction), and the
|
||||
image/upload homes in the suite's scratch dir (the app under test
|
||||
must not write copies into the owner's real ``~/bor-sources``).
|
||||
``BOR_PDF_PICTORIAL`` rides at the conftest pin (``0``) through
|
||||
``dict(os.environ)`` — the fixture's sparse pages stay pre-phase-
|
||||
140 content, byte-stable across the re-sync."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern) — the suite drives no
|
||||
# chat turn, but the app boots with the same values every other
|
||||
# module app gets (the startup validator's premise).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Leak guards (conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
# Phase 151: the import scope WITH the ``pdf`` token, the images
|
||||
# toggle ON (the embedded-image pass runs on both apps), the knob
|
||||
# under test pinned per app, and the suite-private homes.
|
||||
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
|
||||
env["BOR_IMAGES"] = "true"
|
||||
env["BOR_PDF_IMAGE_SKIP_BLANK"] = "true" if skip_blank else "false"
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
|
||||
"""The story app — ``BOR_PDF_IMAGE_SKIP_BLANK=true`` (the default
|
||||
ON gate; the module env override — the conftest session app is
|
||||
never started in this isolated run, so no port clash)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_pdf151_on")
|
||||
port = int(os.environ.get("E2E_APP_PORT_PDF151", "8168"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, skip_blank=True, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def off_app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The negative-control app — ``BOR_PDF_IMAGE_SKIP_BLANK=false``
|
||||
(the kill switch OFF: every extracted image is indexed again —
|
||||
the pre-phase-151 behavior; only the knob-off test starts it)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_pdf151_off")
|
||||
port = int(os.environ.get("E2E_APP_PORT_PDF151_OFF", "8169"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, skip_blank=False, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def off_app_url(off_app_server: str) -> str:
|
||||
return off_app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The fixture archives: ONE PDF, two archive stems (two source names —
|
||||
# a leftover row from either seed is then visibly foreign in the other
|
||||
# app's catalog)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fixture_pdf(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The fixture PDF (two text pages; the underlay + the real image
|
||||
twice on page 1, the real image once more on page 2 — built with
|
||||
PyMuPDF, the house convention)."""
|
||||
root = tmp_path_factory.mktemp("bor_pdf151_pdf")
|
||||
pdf_path = root / DOC_PATH
|
||||
_build_fixture_pdf(pdf_path)
|
||||
return pdf_path
|
||||
|
||||
|
||||
def _zip(fixture_pdf: Path, tmp_path_factory: pytest.TempPathFactory, stem: str) -> Path:
|
||||
root = tmp_path_factory.mktemp(f"bor_pdf151_zip_{stem}")
|
||||
archive = root / f"{stem}.zip"
|
||||
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.write(fixture_pdf, arcname=DOC_PATH)
|
||||
return archive
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zip_on(fixture_pdf: Path, tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""Source ``e2e-pdf151`` — the story app's archive (ONE PDF at the
|
||||
source root, document path ``guide.pdf``)."""
|
||||
return _zip(fixture_pdf, tmp_path_factory, SOURCE_ON)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zip_off(fixture_pdf: Path, tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""Source ``e2e-pdf151-off`` — the SAME PDF for the knob-off app."""
|
||||
return _zip(fixture_pdf, tmp_path_factory, SOURCE_OFF)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB + UI + mock-capture helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry per test group (the E2E isolation pattern):
|
||||
the suites share one Postgres, and a leftover row would corrupt the
|
||||
counts. ``sources_meta`` (the KB generation counter) resets with
|
||||
the KB — the sync re-creates the row."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _health_db(app_url: str) -> bool:
|
||||
try:
|
||||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||
return False
|
||||
|
||||
|
||||
def _upload_zip(page: Page, app_url: str, archive: Path, source_name: str) -> None:
|
||||
"""The real admin upload flow: form login → the git-sources page →
|
||||
pick the archive → submit → the terminal result line (phase 90:
|
||||
"Uploaded <source> — press Sync sources to import it.")."""
|
||||
login(page, app_url, next="/git-sources.html")
|
||||
expect(page).to_have_url(app_url + "/git-sources.html", timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
page.set_input_files("#archive-upload-file", str(archive))
|
||||
page.click("#archive-upload-btn")
|
||||
result = page.locator("#archive-upload-result")
|
||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(result).to_have_text(
|
||||
f"Uploaded {source_name} — press Sync sources to import it.",
|
||||
timeout=UPLOAD_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
|
||||
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
|
||||
"""The RAG page's "Sync sources" button (the scan — the same user
|
||||
path the owner used): click → "Syncing…" → the terminal
|
||||
"Synced HH:MM" label + the counts line settled in
|
||||
``#sync-progress-text`` (the phase-150 single sync text surface —
|
||||
the wait barrier, no sleeps)."""
|
||||
page.goto(app_url + "/sources.html")
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
|
||||
)
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
|
||||
|
||||
|
||||
def _drill_to_source(page: Page, name: str) -> None:
|
||||
"""Top level → the source (the file table then holds its direct
|
||||
files — ours are at the source root)."""
|
||||
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
|
||||
|
||||
|
||||
def _reset_capture(mock_port: int) -> None:
|
||||
"""Clear the mock's request capture (the phase-123 pair) so the
|
||||
next sync's window holds ONLY that sync's LLM traffic."""
|
||||
httpx.post(f"http://127.0.0.1:{mock_port}/v1/e2e/captured/reset", timeout=5)
|
||||
|
||||
|
||||
def _image_description_calls(mock_port: int) -> int:
|
||||
"""The vision describe calls in the mock's current capture — the
|
||||
``IMAGE_DESCRIPTION_MODE`` marker is the app's fixed
|
||||
``DESCRIBE_PROMPT`` head (``app.rag.summarizer``), a marker a real
|
||||
question is never expected to type, so it identifies the image
|
||||
describe calls exactly (no other sync traffic carries it)."""
|
||||
bodies = httpx.get(f"http://127.0.0.1:{mock_port}/v1/e2e/captured", timeout=5).json()
|
||||
return sum(1 for body in bodies if "IMAGE_DESCRIPTION_MODE" in json.dumps(body))
|
||||
|
||||
|
||||
def _stored_image_is_blank(image_path: str) -> bool:
|
||||
"""The APP'S OWN blank predicate (``app.rag.pdf.is_blank_image`` —
|
||||
imported, not re-derived, so the test cannot drift from the
|
||||
product) over the stored copy's samples: the exact gate input the
|
||||
extractor measures — alpha dropped first when present, so the mean
|
||||
runs over COLOR samples only."""
|
||||
pm = fitz.Pixmap(Path(image_path).read_bytes())
|
||||
if pm.alpha:
|
||||
pm = fitz.Pixmap(pm, 0)
|
||||
return is_blank_image(pm.samples)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The module seeds: upload + sync through the real UI, once per app
|
||||
# (the story E2E truncate/re-import-per-module pattern)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded(
|
||||
app_url: str,
|
||||
browser: Browser,
|
||||
zip_on: Path,
|
||||
mock_llm: int,
|
||||
) -> Iterator[dict[str, object]]:
|
||||
"""Upload the fixture zip through the admin UI on the story app
|
||||
(knob ON), then scan it with the RAG page's Sync button (the
|
||||
mock's vision model describes the embedded image in-process).
|
||||
Yields the kept image doc's identity + the first sync's vision
|
||||
call count. The module app's ``/api/config`` flag is asserted
|
||||
first — the env override must have reached the app under test, or
|
||||
the whole premise of the suite is void."""
|
||||
if not _health_db(app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is True, "the story app must run with BOR_IMAGES=true"
|
||||
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_zip(page, app_url, zip_on, SOURCE_ON)
|
||||
# The capture window: ONLY the first sync's LLM traffic.
|
||||
_reset_capture(mock_llm)
|
||||
# The PDF + its ONE embedded image (the underlay and both
|
||||
# duplicate placements are dropped by the parser — the gate
|
||||
# and the dedupe are pre-importer): both count added.
|
||||
_sync_via_ui(page, app_url, expected_result="2 added")
|
||||
first_sync_vision = _image_description_calls(mock_llm)
|
||||
_reset_capture(mock_llm) # a clean window for the re-sync test
|
||||
finally:
|
||||
page.close()
|
||||
|
||||
with SessionLocal() as db:
|
||||
image_docs = db.scalars(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_ON, Document.is_image.is_(True)
|
||||
)
|
||||
).all()
|
||||
# The seed's sanity pin (the rest — first placement, the
|
||||
# no-blank bytes, the vision count — is asserted in the tests
|
||||
# below against this state).
|
||||
assert [doc.path for doc in image_docs] == [REAL_REL]
|
||||
img = image_docs[0]
|
||||
assert img.is_image is True and img.image_path is not None
|
||||
pdf = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_ON, Document.path == DOC_PATH
|
||||
)
|
||||
)
|
||||
assert pdf is not None and pdf.is_image is False
|
||||
assert pdf.title == PDF_TITLE
|
||||
assert "---PAGE-1---" in pdf.content and PAGE_ONE in pdf.content
|
||||
yield {
|
||||
"image_url": f"/api/documents/{img.id}/image",
|
||||
"image_doc_id": img.id,
|
||||
"image_doc_hash": img.content_hash,
|
||||
"first_sync_vision_calls": first_sync_vision,
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded_off(
|
||||
off_app_url: str,
|
||||
browser: Browser,
|
||||
zip_off: Path,
|
||||
mock_llm: int,
|
||||
) -> Iterator[dict[str, object]]:
|
||||
"""The SAME fixture PDF through the knob-off app (the negative
|
||||
control): truncate + re-seed (the shared-Postgres pattern — this
|
||||
runs LAST, after every story test has consumed the story seed),
|
||||
upload + sync via the same user path. Yields the sync's vision
|
||||
call count (2 — the blank underlay IS described now, the
|
||||
pre-phase cost)."""
|
||||
if not _health_db(off_app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = httpx.get(f"{off_app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is True, "the knob-off app must run with BOR_IMAGES=true"
|
||||
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_zip(page, off_app_url, zip_off, SOURCE_OFF)
|
||||
_reset_capture(mock_llm)
|
||||
# The PDF + BOTH embedded images (the blank gate is OFF — the
|
||||
# underlay is indexed again; the duplicate placements are
|
||||
# STILL dropped — dedupe is a correctness fix, not knobed):
|
||||
# three count added.
|
||||
_sync_via_ui(page, off_app_url, expected_result="3 added")
|
||||
sync_vision = _image_description_calls(mock_llm)
|
||||
finally:
|
||||
page.close()
|
||||
|
||||
with SessionLocal() as db:
|
||||
image_docs = db.scalars(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_OFF, Document.is_image.is_(True)
|
||||
)
|
||||
).all()
|
||||
assert sorted(doc.path for doc in image_docs) == sorted([BLANK_REL, REAL_REL])
|
||||
yield {"vision_calls": sync_vision}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Sources page (story app): exactly one image doc, no blank row
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sources_page_lists_exactly_one_image_doc(
|
||||
page: Page, app_url: str, seeded: dict[str, object]
|
||||
) -> None:
|
||||
"""The owner's complaint, the browser lens: the KB total is 2 docs
|
||||
/ 5 chunks (the PDF: 2 page-anchored content chunks + the summary
|
||||
chunk; the one image: its description chunk + summary chunk — the
|
||||
underlay contributes NOTHING, no row, no chunk) and the
|
||||
drill-down table lists the PDF (a plain text row, no thumbnail) +
|
||||
the ONE image document (the thumbnail served from the persistent
|
||||
copy) — and NO row for the underlay's synthetic path
|
||||
(``#page=1#img=0``)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("2")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("5")
|
||||
expect(page.locator("#sources-empty")).to_be_hidden()
|
||||
|
||||
_drill_to_source(page, SOURCE_ON)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
||||
|
||||
# The PDF — a plain text row (no thumbnail box at all). Pinned by
|
||||
# the EXACT doc-link text (the image row's path carries
|
||||
# ``guide.pdf`` as a prefix — a substring match would hit both).
|
||||
pdf_row = page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{DOC_PATH}")'),
|
||||
)
|
||||
expect(pdf_row).to_have_count(1)
|
||||
expect(pdf_row.locator(".kb-doc-thumb")).to_have_count(0)
|
||||
|
||||
# The ONE image document — the real image's thumbnail row (the
|
||||
# bytes route's lazy <img>, alt = the description; visible = the
|
||||
# served copy resolved).
|
||||
img_row = page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{REAL_REL}")'),
|
||||
)
|
||||
expect(img_row).to_have_count(1)
|
||||
img = img_row.locator(".kb-doc-thumb-img")
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_be_visible(timeout=15_000) # the fetch resolves — no glyph
|
||||
expect(img).to_have_attribute("src", str(seeded["image_url"]))
|
||||
|
||||
# The blank underlay's row — the TODO L4 artifact — is ABSENT.
|
||||
blank_rows = page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{BLANK_REL}")'),
|
||||
)
|
||||
expect(blank_rows).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The kept doc is the FIRST placement's; exactly ONE vision call
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_kept_doc_is_the_first_placement_with_one_vision_call(
|
||||
app_url: str, seeded: dict[str, object]
|
||||
) -> None:
|
||||
"""The dedupe's stability contract + the cost pin: the kept image
|
||||
doc's synthetic path is the FIRST placement's
|
||||
(``…#page=1#img=1.png`` — the real image's first insertion; the
|
||||
underlay consumed index 0 and was skipped), the duplicate
|
||||
placements' paths (``#page=1#img=2`` and ``#page=2#img=0``) exist
|
||||
for NO document, and the first sync's capture held EXACTLY ONE
|
||||
vision describe call — the duplicate placements and the blank
|
||||
underlay never reached the model (the cost the owner paid in
|
||||
"several minutes" — pre-phase, every placement was described)."""
|
||||
with SessionLocal() as db:
|
||||
img = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_ON, Document.path == REAL_REL
|
||||
)
|
||||
)
|
||||
assert img is not None, "the real image's first placement must be a document"
|
||||
# Phase 137 regression pin (no behavior change): the document
|
||||
# context (provenance origin line + page 1's text) reaches the
|
||||
# vision model (the mock's context-aware reply is the stored
|
||||
# content), summary === content (the image convention).
|
||||
assert img.title == "Embedded image from guide page 1"
|
||||
assert img.image_context == DOC_BLOCK
|
||||
assert img.content == f"{DOC_BLOCK}\n\n{IMAGE_DESCRIPTION_CONTEXT_ANSWER}"
|
||||
assert img.summary == img.content
|
||||
for dup_rel in DUP_RELS:
|
||||
assert (
|
||||
db.scalar(
|
||||
select(Document.id).where(
|
||||
Document.source == SOURCE_ON, Document.path == dup_rel
|
||||
)
|
||||
)
|
||||
is None
|
||||
), f"the duplicate placement {dup_rel} must not be a document"
|
||||
# ONE vision call in the first sync's window (the PDF's text
|
||||
# summary and the KB overview are different markers — the
|
||||
# IMAGE_DESCRIPTION_MODE count is the describe calls exactly).
|
||||
assert seeded["first_sync_vision_calls"] == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. No image document's stored bytes are a blank white image
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_stored_image_doc_is_blank(app_url: str, seeded: dict[str, object]) -> None:
|
||||
"""The bytes lens on "no blank white documents": every image
|
||||
document's PERSISTENT copy (the row's ``image_path`` — the same
|
||||
bytes the bytes route serves) is checked with the app's own
|
||||
``is_blank_image`` predicate (imported — the test cannot drift
|
||||
from the product): NOT blank. The single doc's bytes are the
|
||||
colorful real image — the underlay's near-white bytes never made
|
||||
it into the KB (no row, no copy)."""
|
||||
with SessionLocal() as db:
|
||||
image_docs = db.scalars(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_ON, Document.is_image.is_(True)
|
||||
)
|
||||
).all()
|
||||
assert len(image_docs) == 1 # the catalog lens (test 1) says the same
|
||||
for doc in image_docs:
|
||||
assert doc.image_path is not None and doc.image_path != ""
|
||||
assert _stored_image_is_blank(doc.image_path) is False, (
|
||||
f"image doc {doc.path}'s stored copy is blank by the app's own "
|
||||
"predicate — the TODO L4 artifact is back"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. Re-sync (nothing changed): `unchanged`, creates nothing new
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_resync_is_idempotent(
|
||||
page: Page, app_url: str, seeded: dict[str, object], mock_llm: int
|
||||
) -> None:
|
||||
"""Idempotence after the fix: a second Sync (the PDF is
|
||||
byte-identical — the fast path, no image pass, no vision) settles
|
||||
on ``0 added · 1 unchanged`` (the PDF doc only — the image doc is
|
||||
a synthetic path the walk never sees), burns ZERO vision calls in
|
||||
its window, and the KB is exactly what it was before — the same
|
||||
two documents, the same image doc row (id + content hash + path
|
||||
unchanged), the same counts."""
|
||||
login(page, app_url) # the test's fresh page is a fresh context (no session)
|
||||
_reset_capture(mock_llm)
|
||||
_sync_via_ui(page, app_url, expected_result="0 added · 1 unchanged")
|
||||
# The re-sync's window: no describe call (the image pass never ran
|
||||
# — the PDF's digest matched), no vision at all.
|
||||
assert _image_description_calls(mock_llm) == 0
|
||||
|
||||
with SessionLocal() as db:
|
||||
docs = db.scalars(select(Document).where(Document.source == SOURCE_ON)).all()
|
||||
assert len(docs) == 2 # nothing new, nothing pruned
|
||||
img = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_ON, Document.path == REAL_REL
|
||||
)
|
||||
)
|
||||
assert img is not None
|
||||
assert img.id == seeded["image_doc_id"]
|
||||
assert img.content_hash == seeded["image_doc_hash"]
|
||||
# The counts surface is unchanged.
|
||||
expect(page.locator("#stat-docs")).to_have_text("2")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("5")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. Negative control (knob OFF): the blank underlay is indexed again
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_knob_off_indexes_the_blank_underlay(
|
||||
page: Page, off_app_url: str, seeded_off: dict[str, object]
|
||||
) -> None:
|
||||
"""The kill switch, proven live in the product (not just in unit
|
||||
tests): the SAME PDF through the SAME user path on
|
||||
``BOR_PDF_IMAGE_SKIP_BLANK=false`` yields the blank underlay
|
||||
document TOO — 3 docs / 7 chunks, the underlay's row present in
|
||||
the drill-down table (its near-white thumbnail served), TWO vision
|
||||
calls (the pre-phase cost — the blank IS described now), and the
|
||||
underlay's stored bytes blank per the app's own predicate — the
|
||||
exact class the story app's gate dropped. The knob gates ONLY the
|
||||
blank gate: the duplicate placements still produce no documents
|
||||
(dedupe is a correctness fix), so the catalog is the real image +
|
||||
the underlay — not the four placements."""
|
||||
try:
|
||||
login(page, off_app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("3")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("7")
|
||||
|
||||
_drill_to_source(page, SOURCE_OFF)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(3)
|
||||
|
||||
# The underlay's row — absent on the story app (test 1) — is
|
||||
# BACK: the knob off restored its indexing (the pre-phase-151
|
||||
# behavior the owner's bug report describes).
|
||||
blank_row = page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{BLANK_REL}")'),
|
||||
)
|
||||
expect(blank_row).to_have_count(1)
|
||||
blank_img = blank_row.locator(".kb-doc-thumb-img")
|
||||
expect(blank_img).to_have_count(1)
|
||||
expect(blank_img).to_be_visible(timeout=15_000) # the near-white copy serves
|
||||
# The duplicate placements: still no documents (not knobed).
|
||||
for dup_rel in DUP_RELS:
|
||||
expect(
|
||||
page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{dup_rel}")'),
|
||||
)
|
||||
).to_have_count(0)
|
||||
|
||||
# The bytes lens: the underlay's stored copy is blank by the
|
||||
# app's own predicate — the exact class the story app skipped
|
||||
# (test 3 proved its absence there); the real image's copy is
|
||||
# not blank (unchanged by the knob).
|
||||
with SessionLocal() as db:
|
||||
blank = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_OFF, Document.path == BLANK_REL
|
||||
)
|
||||
)
|
||||
real = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_OFF, Document.path == REAL_REL
|
||||
)
|
||||
)
|
||||
assert blank is not None and blank.image_path is not None
|
||||
assert _stored_image_is_blank(blank.image_path) is True
|
||||
assert real is not None and real.image_path is not None
|
||||
assert _stored_image_is_blank(real.image_path) is False
|
||||
# The pre-phase cost: BOTH images described (the blank too).
|
||||
assert seeded_off["vision_calls"] == 2
|
||||
finally:
|
||||
_truncate_all()
|
||||
@@ -38,7 +38,6 @@ either direction, and without the token the walk is blind to the file):
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
@@ -87,13 +86,26 @@ QUESTION = (
|
||||
"what does the homelab report say about the backup schedule?"
|
||||
)
|
||||
|
||||
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
|
||||
#: the pipeline is content-agnostic; the well-formed bytes keep the
|
||||
#: extraction + copy + serve + render path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
def _embedded_png() -> bytes:
|
||||
"""A small deterministic colorful noise PNG (channel means ~97 —
|
||||
the "real image" stand-in that CLEARS the phase-151 blank/
|
||||
decorative gate's near-white floor with room to spare). The old
|
||||
1×1 transparent fixture decomposed in PyMuPDF to a near-white
|
||||
1-sample base — blank by the owner-locked gate — and is now
|
||||
(correctly) skipped as a decorative underlay, so the suite's
|
||||
pipeline pins ride on a real non-blank image instead (the
|
||||
pipeline is content-agnostic; the well-formed bytes keep the
|
||||
extraction + copy + serve + render path honest)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 24, 18))
|
||||
for y in range(18):
|
||||
for x in range(24):
|
||||
pix.set_pixel(x, y, ((x * 7) % 256, (y * 13) % 256, ((x + y) * 5) % 256))
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
#: The embedded fixture image (24×18 colorful noise — non-blank by the
|
||||
#: phase-151 gate; see :func:`_embedded_png`).
|
||||
EMBED_PNG = _embedded_png()
|
||||
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
@@ -231,7 +243,7 @@ def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
doc.new_page().insert_text((72, 72), PAGE_ONE)
|
||||
page2 = doc.new_page()
|
||||
page2.insert_text((72, 72), PAGE_TWO)
|
||||
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
|
||||
page2.insert_image(fitz.Rect(72, 150, 96, 168), stream=EMBED_PNG)
|
||||
doc.set_metadata({"title": PDF_TITLE})
|
||||
doc.save(pdf_path)
|
||||
doc.close()
|
||||
|
||||
@@ -990,9 +990,12 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
phase-106 date-refresh counter, the phase-122 ``images_failed``
|
||||
counter, the phase-137 ``image_context_backfilled`` counter (next
|
||||
to ``images_failed``), the phase-133 ``pdfs_screenshotted``
|
||||
counter, and the phase-140 ``pdfs_pictorial`` /
|
||||
``pdf_pictorial_pages`` counters (next to ``pdfs_screenshotted``)
|
||||
sit after ``dates_updated``, before ``formats``."""
|
||||
counter, the phase-140 ``pdfs_pictorial`` /
|
||||
``pdf_pictorial_pages`` counters (next to ``pdfs_screenshotted``),
|
||||
and the phase-151 ``pdf_images_deduped`` / ``pdf_images_blank_skipped``
|
||||
counters (the parser's per-PDF drop counts, next to
|
||||
``pdf_pictorial_pages``) sit after ``dates_updated``, before
|
||||
``formats``."""
|
||||
s = ImportSummary()
|
||||
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
|
||||
s.summaries, s.summary_errors = 2, 1
|
||||
@@ -1003,6 +1006,8 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
s.pdfs_screenshotted = 1
|
||||
s.pdfs_pictorial = 2
|
||||
s.pdf_pictorial_pages = 5
|
||||
s.pdf_images_deduped = 3
|
||||
s.pdf_images_blank_skipped = 1
|
||||
s.formats = {"md": 1, "yaml": 2}
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
s.log()
|
||||
@@ -1012,6 +1017,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||
"dates_updated=0 images_failed=0 image_context_backfilled=2 "
|
||||
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
|
||||
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
|
||||
"formats=yaml:2,md:1"
|
||||
)
|
||||
|
||||
|
||||
+565
-16
@@ -3,11 +3,17 @@
|
||||
Covers ``app/rag/pdf.py``: text extraction with ``---PAGE-N---``
|
||||
separators (A2), the per-page text map (phase 137 task 01 —
|
||||
``extract_pdf_page_texts``, the document-context raw material),
|
||||
embedded image extraction (original encoded bytes),
|
||||
MIME mapping (extension first, then image type code, else the
|
||||
embedded image extraction (one per DISTINCT image — phase 151 task
|
||||
01's dedupe, first placement wins; a masked image is composited to
|
||||
the visible alpha PNG — task 02; the raw-format gate runs first,
|
||||
owner-locked A4 — and the real-file regression pins), MIME mapping
|
||||
(extension first, then image type code, else the
|
||||
``application/octet-stream`` fallback), and the fail-soft contract —
|
||||
unreadable files and broken pages/images return empty results, never
|
||||
raise. Temp PDFs are built with PyMuPDF itself (no binary fixtures).
|
||||
raise. The blank/decorative gate (phase 151 task 03), its kill
|
||||
switch (task 04), and the importer's image-pass integration live in
|
||||
tests/unit/test_pdf_image_filter.py. Temp PDFs are built with
|
||||
PyMuPDF itself (no binary fixtures).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -15,7 +21,9 @@ import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
import struct
|
||||
import tarfile
|
||||
import zlib
|
||||
from collections.abc import Callable, Sequence
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
@@ -43,6 +51,7 @@ from app.rag.pdf import (
|
||||
PAGE_SEPARATOR,
|
||||
XMLISH_LINE_FRACTION,
|
||||
PdfImage,
|
||||
PdfImageStats,
|
||||
_mime_from_image_info,
|
||||
_page_image_entries,
|
||||
extract_pdf_images,
|
||||
@@ -602,12 +611,20 @@ def test_pdf_embedded_image_unsupported_mime_is_skipped(
|
||||
root = tmp_path / "pdfmime"
|
||||
root.mkdir()
|
||||
_build_pdf(root / "report.pdf", [("Text page.", [])])
|
||||
# Phase 151 (task 03): the importer now calls the stats-carrying
|
||||
# variant — patch it (the tuple shape: images + PdfImageStats).
|
||||
# Phase 151 (task 04): the knob rides as the `skip_blank`
|
||||
# keyword — the stand-in accepts it (its value is irrelevant here:
|
||||
# the tiff mime is rejected by the extension gate either way).
|
||||
monkeypatch.setattr(
|
||||
importer,
|
||||
"extract_pdf_images",
|
||||
lambda _path: [
|
||||
PdfImage(page=1, index=0, width=5, height=5, data=b"fakedata", mime="image/tiff")
|
||||
],
|
||||
"extract_pdf_images_with_stats",
|
||||
lambda _path, **_kwargs: (
|
||||
[
|
||||
PdfImage(page=1, index=0, width=5, height=5, data=b"fakedata", mime="image/tiff")
|
||||
],
|
||||
PdfImageStats(duplicates=0, blank_skipped=0, total_found=1),
|
||||
),
|
||||
)
|
||||
llm = _pdf_llm(tmp_path)
|
||||
try:
|
||||
@@ -1176,6 +1193,39 @@ def blank_middle_page_pdf(tmp_path: Path) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def same_image_three_pages_pdf(tmp_path: Path) -> Path:
|
||||
"""3 pages, EACH carrying the SAME image (phase 151 task 01 — the
|
||||
dedupe fixture: one distinct image, three placements)."""
|
||||
path = tmp_path / "triple.pdf"
|
||||
data = _png_bytes(40, 30)
|
||||
doc = fitz.open()
|
||||
for n in (1, 2, 3):
|
||||
page = doc.new_page()
|
||||
page.insert_text((72, 72), f"Text on page {n}.")
|
||||
page.insert_image(fitz.Rect(72, 150, 112, 180), stream=data)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def two_distinct_images_pdf(tmp_path: Path) -> Path:
|
||||
"""Two pages, each with its OWN image (different size AND pixels —
|
||||
genuinely distinct: the dedupe must NOT merge them)."""
|
||||
path = tmp_path / "two.pdf"
|
||||
doc = fitz.open()
|
||||
p1 = doc.new_page()
|
||||
p1.insert_image(fitz.Rect(72, 150, 112, 180), stream=_png_bytes(40, 30))
|
||||
p2 = doc.new_page()
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 15, 15))
|
||||
pix.clear_with(50) # different pixels (the shared helper uses 200)
|
||||
p2.insert_image(fitz.Rect(72, 150, 87, 165), stream=pix.tobytes("png"))
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# extract_pdf_text
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1255,22 +1305,26 @@ def test_pdf_without_images_returns_empty_list(single_page_pdf: Path) -> None:
|
||||
|
||||
def test_embedded_images_extracted_in_order(multi_page_pdf: Path) -> None:
|
||||
images = extract_pdf_images(multi_page_pdf)
|
||||
# Phase 151 (task 01): page 3's SECOND placement of the same JPEG
|
||||
# (index 1 — the same embedded xref) is deduped: one PdfImage per
|
||||
# DISTINCT image, first occurrence wins (page/index stay the first
|
||||
# placement's).
|
||||
assert [(img.page, img.index) for img in images] == [
|
||||
(2, 0), (3, 0), (3, 1), (3, 2),
|
||||
(2, 0), (3, 0), (3, 2),
|
||||
]
|
||||
# page 2: 40x30 PNG
|
||||
img = images[0]
|
||||
assert (img.width, img.height) == (40, 30)
|
||||
assert img.mime == "image/png"
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
# page 3: 40x30 JPEG, then the same JPEG again (distinct index)
|
||||
for img in images[1:3]:
|
||||
assert (img.width, img.height) == (40, 30)
|
||||
assert img.mime == "image/jpeg"
|
||||
assert img.data.startswith(JPEG_MAGIC)
|
||||
assert images[1].data == images[2].data # same embedded xref
|
||||
# page 3: 15x15 PNG
|
||||
img = images[3]
|
||||
# page 3: the 40x30 JPEG at its FIRST placement (index 0 — index 1
|
||||
# is the deduped repeat of the same embedded xref)
|
||||
img = images[1]
|
||||
assert (img.width, img.height) == (40, 30)
|
||||
assert img.mime == "image/jpeg"
|
||||
assert img.data.startswith(JPEG_MAGIC)
|
||||
# page 3: 15x15 PNG at index 2
|
||||
img = images[2]
|
||||
assert (img.width, img.height) == (15, 15)
|
||||
assert img.mime == "image/png"
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
@@ -1407,6 +1461,448 @@ def test_extracted_ext_wins_over_info_ext(
|
||||
assert images[0].mime == "image/jpeg" # ext beats the PNG type code
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 151, task 01 — one PdfImage per DISTINCT image (the dedupe)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_same_image_three_placements_yields_one_first_wins(
|
||||
same_image_three_pages_pdf: Path,
|
||||
) -> None:
|
||||
"""The task's core pin: the SAME image on 3 pages → ONE PdfImage,
|
||||
carrying the FIRST occurrence's page/index (page 1, index 0) — the
|
||||
synthetic path stays stable for the first placement."""
|
||||
images = extract_pdf_images(same_image_three_pages_pdf)
|
||||
assert len(images) == 1
|
||||
assert (images[0].page, images[0].index) == (1, 0)
|
||||
assert (images[0].width, images[0].height) == (40, 30)
|
||||
assert images[0].mime == "image/png"
|
||||
assert images[0].data.startswith(PNG_MAGIC)
|
||||
|
||||
|
||||
def test_two_distinct_images_are_both_kept(
|
||||
two_distinct_images_pdf: Path,
|
||||
) -> None:
|
||||
"""The no-duplicate path: genuinely distinct images (different size
|
||||
AND pixels) are NOT merged — two entries in page order."""
|
||||
images = extract_pdf_images(two_distinct_images_pdf)
|
||||
assert [(img.page, img.index) for img in images] == [(1, 0), (2, 0)]
|
||||
assert images[0].data != images[1].data
|
||||
assert (images[0].width, images[0].height) == (40, 30)
|
||||
assert (images[1].width, images[1].height) == (15, 15)
|
||||
|
||||
|
||||
def test_byte_identical_images_under_distinct_xrefs_dedupe_to_one(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
|
||||
) -> None:
|
||||
"""The DIGEST path: two DIFFERENT xrefs holding byte-identical
|
||||
bytes → one entry (the first), one drop — the xref check alone
|
||||
cannot catch this, the sha256 of the stored bytes can."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
|
||||
def _extract(xref: int) -> dict[str, Any]:
|
||||
return {"image": b"identical-bytes", "ext": "png", "width": 5, "height": 5}
|
||||
|
||||
doc = _FakeDoc(
|
||||
[
|
||||
_fake_image_page(
|
||||
[
|
||||
{"xref": 7, "width": 5, "height": 5},
|
||||
{"xref": 9, "width": 5, "height": 5},
|
||||
]
|
||||
)
|
||||
],
|
||||
_extract,
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
assert (images[0].page, images[0].index) == (1, 0)
|
||||
|
||||
|
||||
def test_different_bytes_under_distinct_xrefs_are_both_kept(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
|
||||
) -> None:
|
||||
"""The digest path's negative control: distinct xrefs with
|
||||
DIFFERENT bytes are distinct images — both kept."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
|
||||
def _extract(xref: int) -> dict[str, Any]:
|
||||
return {
|
||||
"image": b"first-bytes" if xref == 7 else b"second-bytes",
|
||||
"ext": "png", "width": 5, "height": 5,
|
||||
}
|
||||
|
||||
doc = _FakeDoc(
|
||||
[
|
||||
_fake_image_page(
|
||||
[
|
||||
{"xref": 7, "width": 5, "height": 5},
|
||||
{"xref": 9, "width": 5, "height": 5},
|
||||
]
|
||||
)
|
||||
],
|
||||
_extract,
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
images = extract_pdf_images(path)
|
||||
assert [(img.page, img.index) for img in images] == [(1, 0), (1, 1)]
|
||||
|
||||
|
||||
def test_duplicate_drop_logs_one_info_line_per_pdf(
|
||||
same_image_three_pages_pdf: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""One INFO line per PDF that dropped anything (the task's exact
|
||||
format), one DEBUG line per drop; 3 placements → 2 dropped,
|
||||
1 unique."""
|
||||
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
|
||||
assert len(extract_pdf_images(same_image_three_pages_pdf)) == 1
|
||||
infos = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.INFO
|
||||
and "duplicate image occurrence(s)" in r.getMessage()
|
||||
]
|
||||
assert len(infos) == 1 # ONE line per PDF, not one per drop
|
||||
assert infos[0].getMessage() == (
|
||||
f"pdf: dropped 2 duplicate image occurrence(s) "
|
||||
f"from {same_image_three_pages_pdf} (1 unique)"
|
||||
)
|
||||
debugs = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.DEBUG and "duplicate image" in r.getMessage()
|
||||
]
|
||||
assert len(debugs) == 2 # one per dropped occurrence
|
||||
assert all(r.getMessage().endswith("— skipped") for r in debugs)
|
||||
|
||||
|
||||
def test_no_duplicates_no_info_line(
|
||||
two_distinct_images_pdf: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""A PDF with no duplicates never logs the INFO line — the line is
|
||||
the duplicate signal, its absence means nothing was dropped."""
|
||||
with caplog.at_level(logging.INFO, logger="app.rag.pdf"):
|
||||
assert len(extract_pdf_images(two_distinct_images_pdf)) == 2
|
||||
assert not [
|
||||
r
|
||||
for r in caplog.records
|
||||
if "duplicate image occurrence(s)" in r.getMessage()
|
||||
]
|
||||
|
||||
|
||||
#: The owner's real 50-page fixture (untracked — ``test_sources/`` is
|
||||
#: gitignored): the podman + systemd guide. THE phase-151 reproducer —
|
||||
#: its table-shading underlays are the "blank white images" the TODO
|
||||
#: reported.
|
||||
PODMAN_PDF = Path(__file__).resolve().parents[2] / "test_sources" / "podman-systemd.pdf"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
|
||||
def test_podman_pdf_yields_one_entry_per_distinct_image() -> None:
|
||||
"""THE real-file pin: 48 placements (one per ``get_image_info``
|
||||
entry — the phase docs measured 46) spanning 15 unique xrefs — but
|
||||
6 of those xrefs are BYTE-IDENTICAL copies (same base AND same
|
||||
SMask bytes stored under a different xref), so there are only 9
|
||||
truly DISTINCT images. Task 01's digest set is spec'd for exactly
|
||||
the "byte-identical images stored under different xrefs" case, so
|
||||
it merges the 6 copies: 48 placements → 9 entries (the task
|
||||
file's "15" figure is the unique-XREF count — unreachable while
|
||||
keeping the digest set AND the phase's "one document per genuinely
|
||||
distinct image" objective, since those 6 xrefs are genuine
|
||||
duplicates a viewer cannot tell apart). First-occurrence order is
|
||||
pinned too: every kept entry carries its FIRST placement's
|
||||
page/index.
|
||||
|
||||
Phase 151 (task 03): the dedupe pin runs with the blank gate OFF
|
||||
(``skip_blank=False``) — the gate (default on) drops all 9 distinct
|
||||
images (they are the decorative underlays the phase exists to
|
||||
filter), so the default now returns ``[]`` for this file (pinned
|
||||
in tests/unit/test_pdf_image_filter.py)."""
|
||||
images = extract_pdf_images(PODMAN_PDF, skip_blank=False)
|
||||
assert len(images) == 9
|
||||
assert [(img.page, img.index) for img in images] == [
|
||||
(7, 0), (22, 0), (26, 0), (30, 0), (33, 0),
|
||||
(36, 0), (41, 0), (45, 0), (47, 0),
|
||||
]
|
||||
# no two surviving entries share bytes (the dedupe's contract, on
|
||||
# the real file)
|
||||
digests = {hashlib.sha256(img.data).hexdigest() for img in images}
|
||||
assert len(digests) == 9
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 151, task 02 — the /SMask is applied (the visible image, not
|
||||
# the blank base stream extract_image() returns)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _png_chunk(tag: bytes, payload: bytes) -> bytes:
|
||||
"""One PNG chunk (4-byte length + tag + payload + CRC32)."""
|
||||
return (
|
||||
struct.pack(">I", len(payload))
|
||||
+ tag
|
||||
+ payload
|
||||
+ struct.pack(">I", zlib.crc32(tag + payload) & 0xFFFFFFFF)
|
||||
)
|
||||
|
||||
|
||||
def _make_png(
|
||||
width: int, height: int, rgba: bool, pixel: Callable[[int, int], tuple[int, ...]]
|
||||
) -> bytes:
|
||||
"""A deterministic PNG built with stdlib only (``zlib`` +
|
||||
``struct``). PyMuPDF's Pixmap exposes no sample mutation in this
|
||||
release, so the fixture is hand-rolled: a genuine RGBA
|
||||
(color-type-6) PNG is exactly what makes ``insert_image`` store a
|
||||
``/SMask`` (base image + separate mask xref — the shape this task
|
||||
composites)."""
|
||||
color_type = 6 if rgba else 2
|
||||
n = 4 if rgba else 3
|
||||
raw = bytearray()
|
||||
for y in range(height):
|
||||
raw.append(0) # filter type: none
|
||||
for x in range(width):
|
||||
raw += bytes(pixel(x, y)[:n])
|
||||
ihdr = struct.pack(">IIBBBBB", width, height, 8, color_type, 0, 0, 0)
|
||||
return (
|
||||
b"\x89PNG\r\n\x1a\n"
|
||||
+ _png_chunk(b"IHDR", ihdr)
|
||||
+ _png_chunk(b"IDAT", zlib.compress(bytes(raw)))
|
||||
+ _png_chunk(b"IEND", b"")
|
||||
)
|
||||
|
||||
|
||||
def _insert_image_pdf(path: Path, png: bytes) -> Path:
|
||||
"""A one-page PDF carrying *png* (40x30, top-left at 72,150)."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_image(fitz.Rect(72, 150, 112, 180), stream=png)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
def _raw_extract(path: Path) -> dict[str, Any]:
|
||||
"""The page's first image as ``extract_image`` reports it — the
|
||||
raw reference (what the PRE-phase extractor returned verbatim)."""
|
||||
doc = fitz.open(str(path))
|
||||
with doc:
|
||||
info = doc[0].get_image_info(xrefs=True)[0]
|
||||
return doc.extract_image(int(info["xref"]))
|
||||
|
||||
|
||||
def _masked_png() -> bytes:
|
||||
"""A 40x30 RGBA PNG with varying pixels and a non-uniform alpha
|
||||
(top half opaque, bottom half fully transparent — the mask must
|
||||
survive the round trip)."""
|
||||
return _make_png(
|
||||
40, 30, True, lambda x, y: (x % 256, y % 256, (x + y) % 256, 255 if y < 15 else 0)
|
||||
)
|
||||
|
||||
|
||||
def test_masked_image_is_composited_to_the_visible_alpha_png(tmp_path: Path) -> None:
|
||||
"""THE task's core pin: an RGBA PNG inserted via ``insert_image``
|
||||
(PyMuPDF stores it as a BASE image + a ``/SMask``) is extracted as
|
||||
the composited VISIBLE image: (a) the raw ``extract_image`` bytes ≠
|
||||
the extractor's ``data``, (b) ``mime == "image/png"``, (c)
|
||||
re-opening the extractor's bytes as a ``fitz.Pixmap`` reports
|
||||
``alpha == 1`` (the mask survived) at the base's dimensions — and
|
||||
``width``/``height`` come from ``extract_image`` (unchanged)."""
|
||||
path = _insert_image_pdf(tmp_path / "masked.pdf", _masked_png())
|
||||
raw = _raw_extract(path)
|
||||
# Precondition (the bug this task fixes): the /SMask is stored
|
||||
# separately, and the raw extract is the maskless base.
|
||||
assert int(raw.get("smask") or 0) != 0
|
||||
assert fitz.Pixmap(raw["image"]).alpha == 0
|
||||
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
img = images[0]
|
||||
assert (img.page, img.index) == (1, 0)
|
||||
assert img.data != raw["image"] # (a) the composite, not the raw base
|
||||
assert img.mime == "image/png" # (b)
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
vis = fitz.Pixmap(img.data)
|
||||
assert vis.alpha == 1 # (c) the mask survived
|
||||
assert (vis.width, vis.height) == (40, 30)
|
||||
assert (img.width, img.height) == (40, 30) # from extract_image
|
||||
|
||||
|
||||
def test_unmasked_image_is_byte_identical_to_pre_phase(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The common case is unchanged: an image WITHOUT a mask is
|
||||
returned exactly as the raw ``extract_image`` bytes (no composite,
|
||||
no re-encode — pre-phase behavior) and no composite line is
|
||||
logged."""
|
||||
png = _make_png(40, 30, False, lambda x, y: (x % 256, y % 256, (x + y) % 256))
|
||||
path = _insert_image_pdf(tmp_path / "plain.pdf", png)
|
||||
raw = _raw_extract(path)
|
||||
assert int(raw.get("smask") or 0) == 0
|
||||
|
||||
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
assert images[0].data == raw["image"] # the raw bytes, verbatim
|
||||
assert images[0].mime == "image/png" # the raw mime (ext = png)
|
||||
assert not [
|
||||
r for r in caplog.records if "composited smask=" in r.getMessage()
|
||||
]
|
||||
|
||||
|
||||
def test_mask_composite_logs_one_debug_line(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""One DEBUG line (the task's exact format) when a mask was
|
||||
applied."""
|
||||
path = _insert_image_pdf(tmp_path / "masked.pdf", _masked_png())
|
||||
smask = int(_raw_extract(path).get("smask") or 0)
|
||||
with caplog.at_level(logging.DEBUG, logger="app.rag.pdf"):
|
||||
extract_pdf_images(path)
|
||||
debugs = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.DEBUG and "composited smask=" in r.getMessage()
|
||||
]
|
||||
assert len(debugs) == 1
|
||||
assert debugs[0].getMessage() == (
|
||||
f"pdf: composited smask={smask} for image page=1 index=0 of {path}"
|
||||
)
|
||||
|
||||
|
||||
def test_composite_failure_falls_back_to_raw_bytes_with_warning(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Fail-soft: a compositing that RAISES (the monkeypatched
|
||||
``fitz.Pixmap``) returns the raw bytes + raw mime, logs the
|
||||
warning (the task's exact format), and never loses the image.
|
||||
No exception escapes the extractor. The ``_FakeDoc`` stand-in
|
||||
(the task-01 convention) is used so the extractor's compositing
|
||||
block is the ONLY ``fitz.Pixmap`` call in flight (PyMuPDF's real
|
||||
constructor resolves the module-global ``Pixmap`` internally, so
|
||||
patching it would break the real base/mask reads too)."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
raw_bytes = b"raw-base-bytes"
|
||||
doc = _FakeDoc(
|
||||
[_fake_image_page([{"xref": 7, "width": 5, "height": 5}])],
|
||||
lambda xref: {
|
||||
"image": raw_bytes, "ext": "png", "width": 5, "height": 5, "smask": 8,
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
|
||||
attempts: list[tuple[tuple[Any, ...], dict[str, Any]]] = []
|
||||
|
||||
def _boom(*args: Any, **kwargs: Any) -> None:
|
||||
attempts.append((args, kwargs))
|
||||
raise RuntimeError("simulated combine failure")
|
||||
|
||||
monkeypatch.setattr(pdf_mod.fitz, "Pixmap", _boom)
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.pdf"):
|
||||
images = extract_pdf_images(path)
|
||||
assert attempts, "the composite path was attempted (a Pixmap was read)"
|
||||
assert len(images) == 1 # the image is NOT lost
|
||||
assert images[0].data == raw_bytes # raw bytes, verbatim
|
||||
assert images[0].mime == "image/png" # the raw mime
|
||||
|
||||
warnings = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.levelno == logging.WARNING and "could not composite smask=" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1
|
||||
assert warnings[0].getMessage() == (
|
||||
f"pdf: could not composite smask=8 for image page=1 index=0 "
|
||||
f"of {path} — simulated combine failure; using raw bytes"
|
||||
)
|
||||
|
||||
|
||||
def test_unsupported_raw_format_with_mask_stays_untouched(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Owner-locked A4 — the raw-format gate runs FIRST: an image
|
||||
whose raw format the importer's ``_MIME_TO_EXT`` gate will reject
|
||||
(``jpx``/JPEG2000 → the ``application/octet-stream`` fallback — the
|
||||
586 images in ``players_handbook.pdf``) is NOT composited even
|
||||
when it carries a ``/SMask``: raw bytes + raw mime, and the
|
||||
``fitz.Pixmap`` combiner is never called (compositing must never
|
||||
newly admit an unsupported format)."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
raw_bytes = b"jpx-base-bytes"
|
||||
doc = _FakeDoc(
|
||||
[_fake_image_page([{"xref": 7, "width": 5, "height": 5}])],
|
||||
lambda xref: {
|
||||
"image": raw_bytes, "ext": "jpx", "width": 5, "height": 5, "smask": 8,
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
|
||||
def _no_composite(*args: Any, **kwargs: Any) -> None:
|
||||
# Phase 151 (task 03): the blank gate also READS the base
|
||||
# pixmap (Pixmap(doc, xref) — a fail-soft probe that keeps the
|
||||
# image when it raises, as it does on this fake doc) — that is
|
||||
# not compositing. Compositing is the Pixmap(Pixmap, Pixmap)
|
||||
# call (first arg a Pixmap): the raw-format gate must still run
|
||||
# before it, so that shape alone is the failure sentinel.
|
||||
if args and isinstance(args[0], fitz.Pixmap):
|
||||
pytest.fail("the raw-format gate must run before any compositing")
|
||||
raise ValueError("fake doc — no real pixmap (test sentinel)")
|
||||
|
||||
monkeypatch.setattr(pdf_mod.fitz, "Pixmap", _no_composite)
|
||||
images = extract_pdf_images(path)
|
||||
assert len(images) == 1
|
||||
assert images[0].data == raw_bytes # raw bytes, verbatim
|
||||
assert images[0].mime == "application/octet-stream" # raw mime (→ .bin → skipped)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
|
||||
def test_podman_pdf_masked_images_are_composited() -> None:
|
||||
"""Real-file pin: EVERY distinct image of the owner's podman PDF
|
||||
carries a ``/SMask`` pre-phase (``has-mask=True`` in
|
||||
``get_image_info``, a non-zero ``smask`` in ``extract_image``), so
|
||||
post-phase the extractor's output for it is the composited visible
|
||||
image: each entry decodes as an alpha PNG (``alpha == 1``, the
|
||||
base's dimensions) whose bytes DIFFER from the raw base stream
|
||||
``extract_image`` returns (the maskless DeviceGray base — the
|
||||
"blank white image" this phase fixes). The stronger per-entry
|
||||
form of the task's "at least one xref" check. Phase 151 (task 03)
|
||||
landed: the blank gate (default on) removes these entries
|
||||
entirely, so this compositing pin runs with the gate OFF
|
||||
(``skip_blank=False``) — the default's ``[]`` result for this
|
||||
file is pinned in tests/unit/test_pdf_image_filter.py."""
|
||||
images = extract_pdf_images(PODMAN_PDF, skip_blank=False)
|
||||
assert len(images) == 9 # the task-01 dedupe pin still holds
|
||||
|
||||
# The pre-phase property (every placement masked) + the raw base
|
||||
# digests (the "blank" bytes the pre-phase extractor stored).
|
||||
raw_digests: set[str] = set()
|
||||
doc = fitz.open(str(PODMAN_PDF))
|
||||
with doc:
|
||||
for page in doc:
|
||||
for info in page.get_image_info(xrefs=True):
|
||||
xref = int(info.get("xref") or 0)
|
||||
if not xref:
|
||||
continue
|
||||
assert info.get("has-mask") is True
|
||||
raw = doc.extract_image(xref)
|
||||
assert int(raw.get("smask") or 0) != 0
|
||||
raw_digests.add(hashlib.sha256(raw["image"]).hexdigest())
|
||||
|
||||
for img in images:
|
||||
assert img.mime == "image/png"
|
||||
assert img.data.startswith(PNG_MAGIC)
|
||||
vis = fitz.Pixmap(img.data)
|
||||
assert vis.alpha == 1
|
||||
assert (vis.width, vis.height) == (img.width, img.height)
|
||||
# the composite differs from the raw base (every entry)
|
||||
assert hashlib.sha256(img.data).hexdigest() not in raw_digests
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _mime_from_image_info
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1903,3 +2399,56 @@ def test_extract_pdf_text_regression_pin_horse() -> None:
|
||||
text = extract_pdf_text(HORSE_PDF)
|
||||
assert text == _HORSE_EXACT
|
||||
assert hashlib.sha256(text.encode("utf-8")).hexdigest() == _HORSE_SHA256
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 151 — real-file regression pins (skip-if-missing, the
|
||||
# ``HANDBOOK`` / ``HORSE_PDF`` idiom; test_sources/ is gitignored)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(not PODMAN_PDF.is_file(), reason="podman fixture missing")
|
||||
def test_podman_pdf_returns_no_image_documents_by_default() -> None:
|
||||
"""THE reported bug, pinned (the owner's exact file — the bug's
|
||||
tombstone test): the 50-page podman + systemd guide places its
|
||||
table-shading underlays 48 times (``get_image_info`` reports one
|
||||
entry per PLACEMENT); post-phase the extractor returns NONE of
|
||||
them — task 01's dedupe collapses the placements to 9 distinct
|
||||
images (39 dropped — 15 unique xrefs of which 6 are byte-identical
|
||||
copies, so the digest set merges them; the task file's 31/15
|
||||
figure assumed xref-only dedupe over 46 placements — the binding
|
||||
outcome is identical: 0 out) and task 03's blank gate skips all 9
|
||||
(near-white underlays — ASSUMPTION A1: never indexed, never
|
||||
described). The full drop accounting and the log lines are pinned
|
||||
in tests/unit/test_pdf_image_filter.py."""
|
||||
assert extract_pdf_images(PODMAN_PDF) == []
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HORSE_PDF.is_file(), reason="horse fixture missing")
|
||||
def test_horse_pdf_returns_exactly_one_non_blank_image() -> None:
|
||||
"""The real-file "kept" pin: the horse photo (mean 123, uniq 256,
|
||||
uniform ~0.20 — the phase overview's measured basis) clears the
|
||||
blank gate's mean floor — exactly ONE image out, NOT blank-skipped
|
||||
(being a masked JPEG, it is stored as the task-02 composite:
|
||||
alpha PNG)."""
|
||||
images = extract_pdf_images(HORSE_PDF)
|
||||
assert len(images) == 1
|
||||
assert images[0].mime == "image/png"
|
||||
assert images[0].data.startswith(PNG_MAGIC)
|
||||
assert fitz.Pixmap(images[0].data).alpha == 1
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HANDBOOK.is_file(), reason="handbook fixture missing")
|
||||
def test_handbook_pdf_images_keep_the_unmapped_jpx_mime() -> None:
|
||||
"""The real-file "no behavior change" pin: the 293-page handbook's
|
||||
586 JPEG2000 page-art images are STILL returned by the parser
|
||||
(no gate drop, no compositing — owner-locked A4: an unsupported
|
||||
raw format is left exactly as pre-phase) with the UNMAPPED raw
|
||||
mime (``application/octet-stream`` — ``jpx`` is not in the
|
||||
importer's ``_MIME_TO_EXT``), so the importer's extension gate
|
||||
skips them exactly as pre-phase (assert the mime mapping here,
|
||||
not the importer — its gate behavior is pinned in test_importer.
|
||||
py)."""
|
||||
images = extract_pdf_images(HANDBOOK)
|
||||
assert len(images) == 586
|
||||
assert all(img.mime == "application/octet-stream" for img in images)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user