phase: 133_pdf_screenshot_fallback
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).
## Phase 133 final verification report
- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.
This commit is contained in:
@@ -0,0 +1,698 @@
|
||||
"""Phase 133 task 04 E2E (Playwright, mock-only): the real guide stays
|
||||
clean, a synthetic garbage PDF takes the page-screenshot path.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov
|
||||
|
||||
The dedicated suite for ``133_pdf_screenshot_fallback`` (A16 — one
|
||||
Playwright file per phase, run in isolation): it pins both legs of the
|
||||
phase's guarantee at the USER SURFACE. (a) The owner's REAL guide
|
||||
(``test_sources/hnb_2026_guide.tar.gz`` — the file whose PRE-phase-126
|
||||
import stored raw Illustrator XMP lines, the TODO.md L9 complaint)
|
||||
imports through the admin sync and PREVIEWs CLEAN text — the exact
|
||||
complaint, negated on the served content AND the rendered viewer page.
|
||||
(b) A synthetic PDF whose text layer is pure XML garbage
|
||||
(``<illustrator:StartupProfile>Web</…>`` — the owner's exact line
|
||||
shape) takes the page-screenshot fallback end to end — its stored
|
||||
content is the mock's CANNED vision descriptions, one per rendered
|
||||
page, and the garbage text layer is NOT what got stored. (c) A clean
|
||||
PDF never burns the fallback — its stored content is byte-faithful to
|
||||
the text layer (the gate's no-false-negative pin at the E2E level).
|
||||
|
||||
House pattern: one module app over the shared E2E Postgres (its own
|
||||
port, module env — the conftest session app is never started in this
|
||||
isolated run, so no port clash), each test registers its HOST TEMP DIR
|
||||
as a ``kind=local`` source through the admin git-sources API (the
|
||||
phase-49/phase-127 API-registration pattern — adding a plain directory
|
||||
is an API-only operation), runs the real admin sync
|
||||
(``POST /api/sync`` + the status poll, the surface the UI button
|
||||
wraps), and an autouse truncate keeps the shared registry + KB fresh
|
||||
per test (the phase-96 cleanup pattern, with the no-running-sync wait
|
||||
so a leaked run cannot race the truncate).
|
||||
|
||||
App env: ``BOR_IMPORT_EXTENSIONS`` pinned WITH the ``pdf`` token (the
|
||||
owner's live config — the phase-126/127 suites' pattern: without the
|
||||
token the walk is blind to the file, and an operator's local
|
||||
gitignored ``.env`` cannot leak the scope) and ``BOR_IMAGES=1`` (the
|
||||
owner's live toggle — the embedded-image pipeline is live, so the
|
||||
fallback PDF's extracted images join the KB exactly as a clean PDF's
|
||||
do).
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the fallback's
|
||||
vision surface is the mock's deterministic ``IMAGE_DESCRIPTION_MODE``
|
||||
canned reply (``IMAGE_DESCRIPTION_ANSWER``, the same reply the
|
||||
phase-122 image docs get), which makes the fallback's stored content a
|
||||
PURE FUNCTION of the fixture — the byte-exact content assertions below
|
||||
are exact only against that canned text.
|
||||
|
||||
Fixture provenance:
|
||||
* ``hnb/`` — the REAL guide: ``test_sources/hnb_2026_guide.tar.gz``
|
||||
(UNTRACKED owner fixture — the stable reference is the tarball,
|
||||
never the extracted PDF) extracted into a module temp dir named
|
||||
``hnb`` (the directory's basename is the ``kind=local`` source name,
|
||||
phase 38; one 6.2 MB / 48-page Illustrator PDF — the extraction is
|
||||
seconds, module-scoped, once).
|
||||
* ``garbagedoc/`` — a synthetic garbage PDF built IN the test with
|
||||
PyMuPDF (the fixture pattern from phase 133 task 03's unit test,
|
||||
lifted here): 2 pages, each page's ONLY text = the owner's exact
|
||||
garbage line repeated 20× (``<illustrator:StartupProfile>Web</…>``
|
||||
— 100% of the lines look like tags, so the task-02 gate fails
|
||||
deterministically), plus ONE small embedded raster image per page
|
||||
(a 1×1 PNG via ``insert_image`` — the fallback's embedded-image leg
|
||||
runs too: the screenshots are the CONTENT, the extracted images are
|
||||
the citation figures — both, no conflict).
|
||||
* ``cleandoc/`` — ONE small clean PDF (PyMuPDF, real prose text, no
|
||||
images) — the gate's no-false-negative subject.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_real_guide_previews_clean_text`` — the admin sync over
|
||||
``hnb/`` settles ``success`` (the terminal status body);
|
||||
``GET /api/documents/content?source=hnb&path=<the guide's pdf>``
|
||||
(the PREVIEW surface — the endpoint the document viewer's content
|
||||
uses) carries the real first-page text ("2026 U.S. Benefits
|
||||
Guide") and NO line matching the XML-ish regex (the owner's
|
||||
complaint, negated on the SERVED content, not the DB); the
|
||||
document viewer page (``/document.html?source=hnb&path=…``)
|
||||
renders that content — real text, page markers, the stem title,
|
||||
never ``<illustrator:…>`` lines.
|
||||
2. ``test_garbage_pdf_takes_the_screenshot_path`` — the admin sync
|
||||
over ``garbagedoc/`` settles ``success``; the stored content (the
|
||||
same content endpoint) starts with ``---PAGE-1---`` and is the
|
||||
mock's CANNED vision description for BOTH pages (2 separators, 2
|
||||
descriptions — the fallback fired end to end, mock-deterministic)
|
||||
and contains NO ``<illustrator:StartupProfile>`` line (the garbage
|
||||
text layer is NOT what got stored); the two embedded images
|
||||
(``#page=N#img=0.png``) indexed with the canned description too,
|
||||
and the sync's ``detail`` is the UNCHANGED shape (``added`` counts
|
||||
the PDF + its image docs per the phase-126 A3 convention — the
|
||||
fallback is invisible in the status; its counter is log-only,
|
||||
PLAN §9 — ``pdfs_screenshotted`` is NOT a ``detail`` key).
|
||||
3. ``test_clean_pdf_never_burns_the_fallback`` — the admin sync over
|
||||
``cleandoc/`` settles ``success``; the stored content is the
|
||||
PROSE, byte-faithful to the extraction (``---PAGE-1---\n<prose>``
|
||||
exactly) and NOT the canned vision description (the fallback did
|
||||
not fire on a clean layer).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pymupdf as fitz # PyMuPDF — the synthetic-fixture builder (LOCKED A1)
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The owner's untracked fixture (the stable reference — the tarball,
|
||||
#: never the extracted PDF, is what this suite starts from).
|
||||
TARBALL = REPO / "test_sources" / "hnb_2026_guide.tar.gz"
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_GARBPDF", "8149"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture constants (deterministic, pinned)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The real guide — the source dir's basename is the source name
|
||||
#: (``kind=local``, phase 38) and the inner file keeps its name.
|
||||
HNB_SOURCE = "hnb"
|
||||
GUIDE_PDF = "HBI-2026-Guide-Year-Round_260704_131221.pdf"
|
||||
|
||||
#: The pinned fixture text (the real first page — phase 133 task 01's
|
||||
#: pin). If the owner ever re-cuts the fixture, the failure names the
|
||||
#: regression — the point.
|
||||
FIRST_PAGE_TEXT = "2026 U.S. Benefits Guide"
|
||||
|
||||
#: The title: the stem fallback — the guide's metadata carries no
|
||||
#: title (only the PDFium creator/producer), so the non-markdown rule
|
||||
#: applies (phase 126 A2 — one document per PDF, metadata-or-stem).
|
||||
GUIDE_TITLE = GUIDE_PDF[:-4]
|
||||
|
||||
#: The phase-133 XML-ish line heuristic (the task-02 gate's regex):
|
||||
#: ``<tag>``, ``</tag>``, ``<tag attr=``, and the PDF-structure
|
||||
#: ``<</Key…`` noise all match. The preview is clean when NO line
|
||||
#: matches (the real extraction is 0/2,409 XML-ish lines).
|
||||
XMLISH_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
|
||||
|
||||
#: The owner's exact garbage line (the TODO.md L9 preview shape).
|
||||
GARBAGE_SOURCE = "garbagedoc"
|
||||
GARBAGE_PDF = "garbage.pdf"
|
||||
GARBAGE_LINE = "<illustrator:StartupProfile>Web</illustrator:StartupProfile>"
|
||||
GARBAGE_LINES_PER_PAGE = 20 # the page's ONLY text (100% XML-ish)
|
||||
N_GARBAGE_PAGES = 2
|
||||
|
||||
#: The clean PDF — real prose (no tag-shaped lines, no images): the
|
||||
#: gate's no-false-negative subject (the stored content must be
|
||||
#: byte-faithful to the extraction).
|
||||
CLEAN_SOURCE = "cleandoc"
|
||||
CLEAN_PDF = "clean.pdf"
|
||||
CLEAN_PROSE = (
|
||||
"The clean reference page covers the homelab firewall rule audit."
|
||||
)
|
||||
|
||||
#: A real 1×1 transparent PNG (the E2E suites' fixture — the pipeline
|
||||
#: is content-agnostic; the well-formed bytes keep the extraction +
|
||||
#: copy + serve path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
|
||||
def _build_garbage_pdf(path: Path) -> None:
|
||||
"""A REAL tiny garbage PDF (the fixture pattern from phase 133
|
||||
task 03's unit test, lifted here): 2 pages, each page's ONLY text
|
||||
= the owner's exact garbage line × 20 (the gate's XML signal fails
|
||||
deterministically — 100% of the non-empty lines look like tags;
|
||||
the ``---PAGE-N---`` separators are not tag-shaped), plus ONE small
|
||||
embedded raster image per page (the 1×1 PNG — the fallback's
|
||||
embedded-image leg runs too). The text layer is real EXTRACTABLE
|
||||
text — this is the FINDING's class (the pre-126 raw-XMP soup), not
|
||||
the empty-layer class."""
|
||||
doc = fitz.open()
|
||||
for _page_num in range(1, N_GARBAGE_PAGES + 1):
|
||||
page = doc.new_page()
|
||||
for i in range(GARBAGE_LINES_PER_PAGE):
|
||||
page.insert_text((72, 72 + i * 14), GARBAGE_LINE)
|
||||
page.insert_image(fitz.Rect(72, 500, 73, 501), stream=PNG_1X1)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _build_clean_pdf(path: Path) -> None:
|
||||
"""ONE small clean PDF: a single page of real prose (no
|
||||
tag-shaped lines, no embedded images) — the extracted text layer
|
||||
is exactly :data:`CLEAN_PROSE`, so the byte-faithful content
|
||||
assertion is exact."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), CLEAN_PROSE)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def hnb_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The REAL guide (module-scoped, once — the 6 MB extraction is
|
||||
seconds): the tarball is extracted into a temp source dir named
|
||||
``hnb`` (the basename is the source name; the inner file keeps its
|
||||
name). ``test_sources/`` is untracked — the extraction never
|
||||
touches the repo, so ``git status`` stays clean of binaries."""
|
||||
if not TARBALL.is_file():
|
||||
pytest.skip(
|
||||
"test_sources/hnb_2026_guide.tar.gz is missing — the "
|
||||
"untracked owner fixture (2.6 MB, one 48-page Illustrator PDF)"
|
||||
)
|
||||
src = tmp_path_factory.mktemp("bor_garb_hnb") / HNB_SOURCE
|
||||
src.mkdir()
|
||||
with tarfile.open(TARBALL, "r:gz") as tar:
|
||||
tar.extractall(src, filter="data")
|
||||
assert (src / GUIDE_PDF).is_file()
|
||||
return src
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def garbage_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The synthetic garbage source (module-scoped): ``garbagedoc/``
|
||||
with the 2-page garbage PDF (see :func:`_build_garbage_pdf`)."""
|
||||
src = tmp_path_factory.mktemp("bor_garb_garbage") / GARBAGE_SOURCE
|
||||
src.mkdir()
|
||||
_build_garbage_pdf(src / GARBAGE_PDF)
|
||||
assert (src / GARBAGE_PDF).is_file()
|
||||
return src
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def clean_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The clean-PDF source (module-scoped): ``cleandoc/`` with the
|
||||
one-page prose PDF (see :func:`_build_clean_pdf`)."""
|
||||
src = tmp_path_factory.mktemp("bor_garb_clean") / CLEAN_SOURCE
|
||||
src.mkdir()
|
||||
_build_clean_pdf(src / CLEAN_PDF)
|
||||
assert (src / CLEAN_PDF).is_file()
|
||||
return src
|
||||
|
||||
|
||||
def _app_env(
|
||||
base_url: str, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> dict[str, str]:
|
||||
"""The per-module app env (the conftest pattern, cf.
|
||||
``test_pdf_support.py`` / ``test_incremental_folder_summaries.py``):
|
||||
the ``base_url`` is the mock LLM directly (the deterministic
|
||||
canned replies are what the byte-exact assertions key on),
|
||||
``BOR_GIT_SOURCES`` forced empty (the sync sources are this
|
||||
suite's own DB-registered local rows — an operator's local
|
||||
gitignored ``.env`` cannot leak the owner's corpus), the
|
||||
leak-guarded code defaults, and the phase-133 knobs:
|
||||
``BOR_IMPORT_EXTENSIONS`` pinned WITH the ``pdf`` token (the
|
||||
owner's live config — the feature's gate; without it the walk is
|
||||
blind to the PDFs) and ``BOR_IMAGES=1`` (the owner's live toggle —
|
||||
the embedded-image pipeline is live on BOTH the clean and the
|
||||
fallback PDFs), with the image/upload homes in the suite's scratch
|
||||
dir (the app under test must not write copies into the owner's
|
||||
real ``~/bor-sources``)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern) — no chat turn is ever
|
||||
# sent in this suite, but the app boots with the same env shape.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directories this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
# Phase 133: the owner's live import scope WITH the ``pdf`` token +
|
||||
# the images toggle under test + the suite-private homes.
|
||||
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
|
||||
env["BOR_IMAGES"] = "1"
|
||||
scratch = tmp_path_factory.mktemp("bor_garb_scratch")
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The app under test (all three tests): one module app with the
|
||||
phase-133 env (see :func:`_app_env`). The temp source dirs are NOT
|
||||
dependencies — the app reads them only at sync time, and the
|
||||
``hnb_dir`` fixture skips (not the whole module) when the untracked
|
||||
owner fixture is missing."""
|
||||
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
||||
E2E suites share one Postgres, so a leftover git_sources row or
|
||||
document would pollute the sync's counts and the content the
|
||||
assertions pin. ``sources_meta`` (the KB generation counter)
|
||||
resets with the KB — the sync re-creates the row."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries, sources_meta"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
|
||||
"""No background sync may leak across tests (the run state lives in
|
||||
the app's memory, and a still-running run would keep importing into
|
||||
the NEXT test's truncated KB): wait for the app's sync status to
|
||||
be non-running BEFORE the truncate. Own admin session (the
|
||||
endpoint is admin-only) — usually a no-op: the tests settle only
|
||||
after their run's terminal state."""
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
||||
"""Per-test KB isolation (the phase-96 cleanup pattern): the
|
||||
shared E2E Postgres is truncated before AND after every test —
|
||||
each test registers its own source and runs its own sync, so the
|
||||
KB rows are exactly this test's own. The pre-wait keeps a leaked
|
||||
run from the previous test (the module app is shared) from racing
|
||||
the truncate."""
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
yield
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers (the phase-127 house pattern: API-driven source registration
|
||||
# + sync over plain httpx — the admin cookie rides the client jar)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _register_source(base_url: str, path: str) -> None:
|
||||
"""Register the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` / phase-94 pattern — a plain
|
||||
directory is an API/DB-only operation; the page form is gone)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/git-sources", json={"kind": "local", "path": path})
|
||||
assert r.status_code == 201, r.text
|
||||
|
||||
|
||||
def _started_at(body: dict[str, Any]) -> float:
|
||||
"""The status body's ``started_at`` as a wall-clock epoch (the
|
||||
server stamps it in UTC; this process runs on the same host —
|
||||
same clock, the phase-64 recorder's freshness convention)."""
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return 0.0
|
||||
try:
|
||||
return datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
|
||||
run reaches a terminal state (the ``test_local_directory_sources``
|
||||
pattern, over plain httpx — with the stale-terminal freshness gate:
|
||||
the module app is shared, so the PREVIOUS run's terminal lives in
|
||||
the app's memory until this run supersedes it; a terminal is
|
||||
accepted only after a running tick was observed or its
|
||||
``started_at`` post-dates this POST)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
t0 = time.time()
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
saw_running = False
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
state = body["state"]
|
||||
if state == "running":
|
||||
saw_running = True
|
||||
elif state in ("success", "failed") and (
|
||||
saw_running
|
||||
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
|
||||
):
|
||||
return body
|
||||
time.sleep(0.1)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _doc_content(base_url: str, source: str, path: str) -> dict[str, Any]:
|
||||
"""``GET /api/documents/content`` (the PREVIEW surface — the
|
||||
endpoint the document viewer's content uses) as the signed-in
|
||||
admin. The wire shape (``DocContent``): ``content`` is the stored
|
||||
document content verbatim; ``is_image`` is always present."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.get(
|
||||
"/api/documents/content", params={"source": source, "path": path}
|
||||
)
|
||||
assert r.status_code == 200, f"content lookup failed: {r.text}"
|
||||
return r.json()
|
||||
|
||||
|
||||
def _image_docs(source: str) -> list[Document]:
|
||||
"""The source's stored ``is_image`` documents, in path order (the
|
||||
test process's direct DB access — the E2E's other established
|
||||
lens)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(
|
||||
select(Document)
|
||||
.where(Document.source == source, Document.is_image.is_(True))
|
||||
.order_by(Document.path)
|
||||
).all()
|
||||
return list(rows)
|
||||
|
||||
|
||||
def _xmlish_lines(content: str) -> list[str]:
|
||||
"""The content's non-empty lines matching the phase-133 XML-ish
|
||||
regex (the owner's complaint — the pre-126 raw-XMP lines)."""
|
||||
return [
|
||||
line
|
||||
for line in content.splitlines()
|
||||
if line.strip() and XMLISH_RE.search(line)
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The REAL guide (the owner's exact file) previews CLEAN text
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_real_guide_previews_clean_text(
|
||||
app_url: str, hnb_dir: Path, page: Page, db_ready: None
|
||||
) -> None:
|
||||
"""The owner's exact file, through the real admin sync: the
|
||||
preview (``GET /api/documents/content`` — the endpoint the
|
||||
document viewer's content uses) carries the real first-page text
|
||||
and NO XML-ish line (the TODO.md L9 complaint, negated on the
|
||||
SERVED content), and the document viewer page renders that content
|
||||
— real text, never ``<illustrator:…>`` lines. The pre-126
|
||||
binary-text import (the raw-XMP soup) can never come back: if the
|
||||
PDF were ever routed through the text branch again, the preview
|
||||
would be ~46% XML-ish and both assertions break."""
|
||||
_register_source(app_url, str(hnb_dir))
|
||||
# 48 pages + the embedded-image pass against the mock — the
|
||||
# generous budget (the in-process regression pin's 120 s bar plus
|
||||
# the HTTP round-trips over the mock).
|
||||
body = _run_sync(app_url, timeout_s=180.0)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
# The walk admitted exactly the one file (the ``pdf`` token — the
|
||||
# PDF branch, never the text branch), no errors, nothing pruned.
|
||||
assert detail["files"] == 1, detail
|
||||
assert detail["errors"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
|
||||
# The preview surface (the endpoint the document viewer's content
|
||||
# uses): the body is the CLEAN extracted text.
|
||||
doc = _doc_content(app_url, HNB_SOURCE, GUIDE_PDF)
|
||||
assert doc["is_image"] is False # it is a PDF document, not an image doc
|
||||
assert doc["title"] == GUIDE_TITLE # the stem fallback (no metadata title)
|
||||
content = doc["content"]
|
||||
assert FIRST_PAGE_TEXT in content, (
|
||||
"the real first-page text is missing from the served content — "
|
||||
"the stored content is not the extraction"
|
||||
)
|
||||
xml_lines = _xmlish_lines(content)
|
||||
assert xml_lines == [], (
|
||||
f"the owner's complaint returned on the served content — "
|
||||
f"{len(xml_lines)} XML-ish lines, e.g. {xml_lines[:3]!r}"
|
||||
)
|
||||
|
||||
# The phase-126 A3 leg: the guide's embedded images joined the KB
|
||||
# with the canned descriptions — the ``added`` accounting is the
|
||||
# one file PLUS its image docs (the phase-126 convention the
|
||||
# task-01 regression pin documents).
|
||||
imgs = _image_docs(HNB_SOURCE)
|
||||
assert imgs, "the guide's embedded images did not index"
|
||||
assert detail["added"] == 1 + len(imgs), detail
|
||||
|
||||
# The document viewer PAGE renders that content (the user surface):
|
||||
# the real text + the page-1 marker, the stem title, and the owner's
|
||||
# line is ABSENT from the rendered preview.
|
||||
login(page, app_url)
|
||||
page.goto(
|
||||
app_url + f"/document.html?source={HNB_SOURCE}&path={GUIDE_PDF}"
|
||||
)
|
||||
expect(page.locator("#doc-not-found")).to_be_hidden(timeout=30_000)
|
||||
expect(page.locator("#doc-title")).to_have_text(GUIDE_TITLE)
|
||||
raw = page.locator("pre.doc-raw")
|
||||
expect(raw).to_be_visible(timeout=30_000)
|
||||
expect(raw).to_contain_text(FIRST_PAGE_TEXT)
|
||||
expect(raw).to_contain_text("---PAGE-1---")
|
||||
rendered = raw.inner_text()
|
||||
assert GARBAGE_LINE not in rendered, (
|
||||
"the viewer renders the owner's <illustrator:StartupProfile> line — "
|
||||
"the pre-126 garbage preview came back"
|
||||
)
|
||||
assert _xmlish_lines(rendered) == [], (
|
||||
"the viewer renders XML-ish lines — the clean extraction is not what "
|
||||
"the preview shows"
|
||||
)
|
||||
# The non-image doc with a non-empty summary renders the Summary
|
||||
# panel (the phase-36 shape, above the content).
|
||||
expect(page.locator(".doc-summary")).to_have_count(1)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The synthetic garbage PDF takes the screenshot fallback end to end
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_garbage_pdf_takes_the_screenshot_path(
|
||||
app_url: str, garbage_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""The owner's specified path, end to end: a PDF whose text layer
|
||||
is pure XML garbage fails the task-02 gate, every page is rendered
|
||||
to a screenshot and read by the (mock) vision model, and the stored
|
||||
content is the CANNED page descriptions — 2 separators, 2
|
||||
descriptions, in page order — with the garbage text layer NOT what
|
||||
got stored. The fallback is invisible in the sync's status shape
|
||||
(``detail`` is the phase-126 shape — the counter is log-only,
|
||||
PLAN §9), and the extracted embedded images join the KB exactly as
|
||||
a clean PDF's do (the screenshots are the CONTENT; the extracted
|
||||
images are the citation figures — both, no conflict)."""
|
||||
_register_source(app_url, str(garbage_dir))
|
||||
body = _run_sync(app_url, timeout_s=120.0)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["files"] == 1, detail
|
||||
assert detail["errors"] == 0, detail
|
||||
|
||||
# The stored content (the same content endpoint the preview uses):
|
||||
# the fallback's page-anchored canned descriptions, byte-exact.
|
||||
doc = _doc_content(app_url, GARBAGE_SOURCE, GARBAGE_PDF)
|
||||
assert doc["is_image"] is False # a PDF whose content is vision text
|
||||
content = doc["content"]
|
||||
assert content.startswith("---PAGE-1---")
|
||||
assert content.count("---PAGE-1---") == 1
|
||||
assert content.count("---PAGE-2---") == 1
|
||||
# The mock's CANNED vision description for BOTH pages (the
|
||||
# ``IMAGE_DESCRIPTION_MODE`` reply — deterministic). The first-40
|
||||
# char pin is the task's lens; the count + the byte-exact shape
|
||||
# below make it airtight.
|
||||
assert IMAGE_DESCRIPTION_ANSWER[:40] in content
|
||||
assert content.count(IMAGE_DESCRIPTION_ANSWER) == 2
|
||||
expected = (
|
||||
f"---PAGE-1---\n{IMAGE_DESCRIPTION_ANSWER}\n\n"
|
||||
f"---PAGE-2---\n{IMAGE_DESCRIPTION_ANSWER}"
|
||||
)
|
||||
assert content == expected, (
|
||||
"the stored content is not the fallback's canned page descriptions"
|
||||
)
|
||||
# The garbage text layer is NOT what got stored — the screenshot
|
||||
# path replaced it (the owner's exact line, negated).
|
||||
assert GARBAGE_LINE not in content
|
||||
assert "<illustrator" not in content
|
||||
|
||||
# The fallback's embedded-image leg: both extracted images (one per
|
||||
# page, the 1×1 PNG — ``#page=N#img=0`` paths) indexed with the
|
||||
# canned description (the phase-122 image convention).
|
||||
imgs = _image_docs(GARBAGE_SOURCE)
|
||||
assert [i.path for i in imgs] == [
|
||||
f"{GARBAGE_PDF}#page=1#img=0.png",
|
||||
f"{GARBAGE_PDF}#page=2#img=0.png",
|
||||
]
|
||||
for img in imgs:
|
||||
assert img.content == IMAGE_DESCRIPTION_ANSWER
|
||||
assert img.summary == img.content # the image convention
|
||||
|
||||
# The sync's ``detail`` is the UNCHANGED shape: the garbage PDF is
|
||||
# a NORMAL import (the one file + its two image docs per the
|
||||
# phase-126 A3 convention) and the fallback's counter is
|
||||
# log-only — ``pdfs_screenshotted`` is NOT a ``detail`` key.
|
||||
assert detail["added"] == 1 + len(imgs), detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert "pdfs_screenshotted" not in detail
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. A clean PDF never burns the fallback (the gate's no-false-
|
||||
# negative pin at the E2E level)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_clean_pdf_never_burns_the_fallback(
|
||||
app_url: str, clean_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""The gate is the ONLY gate: a clean text layer never reaches the
|
||||
fallback — the stored content is the PROSE, byte-faithful to the
|
||||
extraction (``---PAGE-1---\n<prose>`` exactly — the phase-126
|
||||
happy path, unchanged) and NOT the canned vision description (no
|
||||
screenshot, no vision burn on the happy path)."""
|
||||
_register_source(app_url, str(clean_dir))
|
||||
body = _run_sync(app_url, timeout_s=60.0)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
# One file, one added document — no embedded images in the clean
|
||||
# fixture, so the ``added`` count is the PDF alone.
|
||||
assert detail["files"] == 1, detail
|
||||
assert detail["added"] == 1, detail
|
||||
assert detail["errors"] == 0, detail
|
||||
|
||||
doc = _doc_content(app_url, CLEAN_SOURCE, CLEAN_PDF)
|
||||
assert doc["is_image"] is False
|
||||
# Byte-faithful to the text layer (the extraction's exact shape —
|
||||
# the page marker + the stripped page text).
|
||||
assert doc["content"] == f"---PAGE-1---\n{CLEAN_PROSE}"
|
||||
# The fallback did NOT fire: no canned vision text anywhere in the
|
||||
# stored content.
|
||||
assert IMAGE_DESCRIPTION_ANSWER[:40] not in doc["content"]
|
||||
assert _image_docs(CLEAN_SOURCE) == []
|
||||
@@ -636,13 +636,18 @@ def test_pdf_without_embedded_images_and_unchanged_rerun(db, tmp_path) -> None:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
"""A PDF with no extractable text (image-only, ``images`` OFF — the
|
||||
embedded image is invisible) → the document is still created, with
|
||||
EMPTY content (the empty byte string's digest), NO content chunks
|
||||
and NO content embedding (there is no text to embed — the only
|
||||
chunk is the best-effort ``lite`` summary's ``is_summary`` row), and
|
||||
the warning line is the signal."""
|
||||
def test_pdf_without_text_takes_the_screenshot_fallback(db, tmp_path) -> None:
|
||||
"""Phase 133 (task 03): a PDF with no extractable text layer
|
||||
(image-only, ``images`` OFF — the embedded image is invisible to
|
||||
the phase-126 A3 pass, but the FALLBACK is not gated by the
|
||||
toggle) → the gate's empty-layer target fires: the single page is
|
||||
rendered to a PNG and vision-described (the mock's fixed
|
||||
:data:`DESCRIPTION`), and the document lands with the
|
||||
``---PAGE-N---``-anchored description as content (one page-anchored
|
||||
content chunk + the ``lite`` summary chunk, both embedded). The
|
||||
phase-126 empty-content behavior is retired for layers that CAN be
|
||||
screenshotted — it survives only for an unopenable file, where the
|
||||
render fails too (the unit suite pins it)."""
|
||||
source_root = tmp_path / "pdfemptykb"
|
||||
source_root.mkdir()
|
||||
_build_pdf(source_root / "pics.pdf", [("", [_png_bytes(40, 30)])])
|
||||
@@ -650,6 +655,7 @@ def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
@@ -657,18 +663,19 @@ def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
)
|
||||
)
|
||||
assert doc is not None and doc.is_image is False
|
||||
assert doc.content == ""
|
||||
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
|
||||
assert doc.content == f"---PAGE-1---\n{DESCRIPTION}"
|
||||
assert doc.content_hash == hashlib.sha256(doc.content.encode("utf-8")).hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c for c in chunks if not c.is_summary] == [], (
|
||||
"no text → no content chunks (and no content embedding)"
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [f"Page 1:\n{DESCRIPTION}"], (
|
||||
"the page-anchored fallback content is chunked like any PDF"
|
||||
)
|
||||
# The phase-30/118 path still runs best-effort on the empty
|
||||
# content — the summary is the only chunk, and it is embedded.
|
||||
assert len(chunks) == 1 and chunks[0].is_summary
|
||||
assert chunks[0].embedding is not None and len(chunks[0].embedding) == 768
|
||||
# The phase-30/118 path runs on the fallback content — the
|
||||
# summary chunk mirrors the stored summary, both embedded.
|
||||
assert len(chunks) == 2
|
||||
assert all(c.embedding is not None and len(c.embedding) == 768 for c in chunks)
|
||||
assert doc.summary is not None
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
@@ -0,0 +1,331 @@
|
||||
"""Integration: phase 133 (task 01) — the owner's HNB guide regression pin.
|
||||
|
||||
FINDING (the two-line root cause, full evidence in the phase overview):
|
||||
the garbage the owner saw is the PRE-PHASE-126 import — ``pdf`` was
|
||||
already in the owner's ``BOR_IMPORT_EXTENSIONS`` (local, untracked
|
||||
``.env``) before the PDF parser shipped, so the generic text branch
|
||||
read the RAW PDF bytes (``read_text(errors="replace")``) and the
|
||||
uncompressed Illustrator XMP/structure blocks (222 ×
|
||||
``<illustrator:StartupProfile>`` + 138 × ``<x:xmpmeta>`` blocks) were
|
||||
chunked and embedded as literal text. Phase 126+ extracts the same
|
||||
file correctly; the stale production row self-heals on re-sync
|
||||
(content-hash delta → ``updated`` with clean content).
|
||||
|
||||
This suite pins the owner's EXACT file as a permanent regression: the
|
||||
archive's 48-page Illustrator PDF is imported through the REAL
|
||||
importer — the real walk, the real PDF branch (extraction → digest →
|
||||
page-anchored chunking → 768-dim embeds → ``lite`` summary → the
|
||||
phase-126 embedded-image pass, whose vision seam runs against the
|
||||
multimodal-capable fake below) — and must land CLEAN content: the
|
||||
XML-ish line fraction stays far under the task-02 gate's 0.30
|
||||
(0.0 in the real extraction — asserted < 0.05 to be strict), the
|
||||
non-printable fraction stays < 0.01, and the real first-page text is
|
||||
present. If the PDF is ever routed through the text branch again, the
|
||||
stored content is the raw bytes again (~46% XML-ish) and the pin
|
||||
breaks — the pin has teeth.
|
||||
|
||||
Run:
|
||||
uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v
|
||||
Requires: podman compose up -d db
|
||||
|
||||
Fixture provenance: ``test_sources/hnb_2026_guide.tar.gz`` — the
|
||||
UNTRACKED owner fixture (2.6 MB compressed; one 6.2 MB, 48-page
|
||||
PDFium/Illustrator export, ``HBI-2026-Guide-Year-Round_260704_131221.pdf``
|
||||
inside). The test extracts it into ``tmp_path`` (the test's local
|
||||
source dir) and never touches the repo — ``git status`` stays clean of
|
||||
binaries. The pinned fixture text (the first-page title) fails LOUDLY
|
||||
if the owner ever re-cuts the fixture: that IS the regression signal.
|
||||
|
||||
Config note (why the settings are built the way they are):
|
||||
* ``_env_file=".env"`` — the REAL extension set: ``pdf`` is in the
|
||||
owner's live ``BOR_IMPORT_EXTENSIONS``. A ``_env_file=None`` fake
|
||||
would use the code defaults (no ``pdf`` token) and SILENTLY skip the
|
||||
PDF — the exact trap that hid this class before (FINDING, point 3).
|
||||
* ``images=True`` — the owner's live toggle (``BOR_IMAGES=1`` in the
|
||||
same ``.env``). The suite forces ``BOR_IMAGES`` to the code default
|
||||
(off) in the process environment (the conftest leak class, phase
|
||||
122), and process env beats the ``.env`` file — so the live value is
|
||||
restored with an init kwarg (init kwargs beat both). The embedded-
|
||||
image pass is the phase-126 A3 seam this pin covers.
|
||||
* ``image_dir`` under ``tmp_path`` — the house hermetic pattern
|
||||
(``test_importer.py``'s ``_image_llm``): the embedded-image copies
|
||||
never touch the owner's persistent image home.
|
||||
|
||||
``summary.added`` accounting: the walked file contributes EXACTLY ONE
|
||||
added document (the guide PDF); the phase-126 A3 convention counts
|
||||
each embedded image doc ``added`` too, so the assertion is
|
||||
``added == 1 + len(image_docs)`` together with ``files == 1`` (the
|
||||
walk admitted the one file) — the strict generalization of the naive
|
||||
``added == 1`` for a PDF whose images join the KB.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import tarfile
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.pdf import PAGE_SEPARATOR
|
||||
from app.rag.summarizer import DESCRIBE_PROMPT
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The repo root (this file lives in ``tests/integration/``).
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The owner's untracked fixture (the stable reference — the tarball,
|
||||
#: never the extracted PDF, is what the test starts from).
|
||||
TARBALL = REPO_ROOT / "test_sources" / "hnb_2026_guide.tar.gz"
|
||||
|
||||
#: The inner file keeps its name; the temp source dir is named
|
||||
#: ``test_sources`` (the importer's ``source = root.name`` convention —
|
||||
#: the same identity the handbook pin uses).
|
||||
SOURCE = "test_sources"
|
||||
PDF_NAME = "HBI-2026-Guide-Year-Round_260704_131221.pdf"
|
||||
PDF_REL = f"{SOURCE}/{PDF_NAME}"
|
||||
|
||||
#: The pinned fixture text (the real first page). If the owner ever
|
||||
#: re-cuts the fixture, the failure names the regression — the point.
|
||||
FIRST_PAGE_TEXT = "2026 U.S. Benefits Guide"
|
||||
|
||||
#: The title: the stem fallback — the PDF's metadata carries no title
|
||||
#: (only the PDFium creator/producer), so the non-markdown rule applies.
|
||||
TITLE = "HBI-2026-Guide-Year-Round_260704_131221"
|
||||
|
||||
#: The phase-133 XML-ish line heuristic (the task-02 gate's regex):
|
||||
#: ``<tag>``, ``</tag>``, ``<tag attr=``, and the PDF-structure
|
||||
#: ``<</Key…`` noise all match.
|
||||
XMLISH_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
|
||||
|
||||
#: Clean-content bars (stricter than the task-02 gate on purpose — the
|
||||
#: real extraction is 0.0 XML-ish / ~0.0005 non-printable, the
|
||||
#: pre-126 garbage read is ~0.46 XML-ish AND heavy on U+FFFD).
|
||||
XML_FRACTION_BAR = 0.05
|
||||
NONPRINT_FRACTION_BAR = 0.01
|
||||
|
||||
#: The task's wall-time guard: 48 pages × in-process fakes must stay
|
||||
#: fast — a drift past this means the test is too heavy.
|
||||
WALL_TIME_BAR_S = 120.0
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not TARBALL.is_file(),
|
||||
reason=(
|
||||
"test_sources/hnb_2026_guide.tar.gz is missing — the untracked "
|
||||
"owner fixture (2.6 MB, one 48-page Illustrator PDF)"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class GuideVisionLLM(FakeEmbedder):
|
||||
"""The task-03-style vision mock the ``fakes.py`` comment
|
||||
anticipates: the base ``chat`` is string-only — the MULTIMODAL user
|
||||
content ``describe_image`` sends (the ``list``-of-parts shape)
|
||||
would crash it — so ``list`` content gets the deterministic canned
|
||||
reply instead. String (``lite``) calls keep the ``FakeEmbedder``
|
||||
behaviour. Mirrors ``test_importer.py``'s ``_MockVisionLLM``
|
||||
pattern; the REAL ``import_sources`` → ``_describe_or_skip`` →
|
||||
``describe_image`` chain runs against it (no seam patch)."""
|
||||
|
||||
VISION_REPLY = "A rendered page of the guide (test vision description)."
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
#: The text part of every multimodal (vision) call received.
|
||||
self.vision_texts: list[str] = []
|
||||
|
||||
async def chat(
|
||||
self, messages: list[dict[str, Any]], model: str | None = None
|
||||
) -> str:
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
self.vision_texts.append(
|
||||
next((p["text"] for p in user if p.get("type") == "text"), "")
|
||||
)
|
||||
return self.VISION_REPLY
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
def _settings(tmp_path: Path) -> Settings:
|
||||
"""The owner's live import config (the point of the pin — see the
|
||||
module docstring's Config note) + the hermetic image dir."""
|
||||
kwargs: dict[str, Any] = {
|
||||
"_env_file": ".env",
|
||||
"images": True,
|
||||
"image_dir": str(tmp_path / "images"),
|
||||
}
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _xml_frac(content: str) -> float:
|
||||
"""The phase-133 XML-ish line fraction: non-empty lines matching
|
||||
:data:`XMLISH_RE`, over the non-empty lines."""
|
||||
lines = [line for line in content.splitlines() if line.strip()]
|
||||
if not lines:
|
||||
return 0.0
|
||||
return sum(1 for line in lines if XMLISH_RE.search(line)) / len(lines)
|
||||
|
||||
|
||||
def _nonprint_frac(content: str) -> float:
|
||||
"""The binary-soup signal: U+FFFD replacement chars + C0 control
|
||||
chars over the whole text — the whitespace controls (``\\n``/``\\t``/
|
||||
``\\r``) are prose line breaks, not garbage (counting them would
|
||||
flag every document), so they are excluded."""
|
||||
if not content:
|
||||
return 0.0
|
||||
bad = sum(
|
||||
1
|
||||
for ch in content
|
||||
if ch == "\ufffd" or (ord(ch) < 32 and ch not in "\r\n\t")
|
||||
)
|
||||
return bad / len(content)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def guide_kb(
|
||||
db: Session, tmp_path: Path
|
||||
) -> Iterator[tuple[GuideVisionLLM, ImportSummary, float]]:
|
||||
"""The real guide, imported ONCE per test through the REAL importer
|
||||
(the house KB pattern — ``test_read_line_ranges_handbook.py``):
|
||||
``chunks``/``documents``/``folder_summaries`` are truncated around
|
||||
the test, the tarball is extracted into a temp ``test_sources``
|
||||
source dir, and the deterministic in-process
|
||||
:class:`GuideVisionLLM` stands in for the embed/summary/vision
|
||||
calls (the whole 48-page index build stays fast and offline).
|
||||
Yields ``(llm, summary, elapsed_s)``."""
|
||||
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||
db.commit()
|
||||
root = tmp_path / SOURCE
|
||||
root.mkdir()
|
||||
with tarfile.open(TARBALL, "r:gz") as tar:
|
||||
tar.extractall(root, filter="data")
|
||||
llm = GuideVisionLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
started = time.monotonic()
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
elapsed = time.monotonic() - started
|
||||
yield llm, summary, elapsed
|
||||
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||
db.commit()
|
||||
db.expire_all()
|
||||
|
||||
|
||||
def test_hnb_guide_imports_clean_through_the_real_pipeline(
|
||||
db: Session,
|
||||
guide_kb: tuple[GuideVisionLLM, ImportSummary, float],
|
||||
) -> None:
|
||||
"""The owner's exact file lands CLEAN through the real pipeline:
|
||||
the walk admits the one file (``pdf`` token — the PDF branch, never
|
||||
the text branch), the stored content is clean + carries the real
|
||||
first-page text, the page-anchored chunks are embedded, the ``lite``
|
||||
summary is stored, and every embedded-image doc got the fake vision
|
||||
description (the multimodal ``chat`` was called)."""
|
||||
llm, summary, elapsed = guide_kb
|
||||
|
||||
# The task's wall-time guard — the suite must stay fast.
|
||||
assert elapsed < WALL_TIME_BAR_S, (
|
||||
f"the guide import took {elapsed:.1f}s (> {WALL_TIME_BAR_S:.0f}s) — "
|
||||
"the test is too heavy, the suite must stay fast"
|
||||
)
|
||||
|
||||
# The walk ADMITTED the guide as one file with the ``pdf`` token —
|
||||
# the PDF branch (the text branch is unreachable for ``.pdf`` after
|
||||
# phase 126; if it ever ran on this file again, the content bars
|
||||
# below break — the pin has teeth).
|
||||
assert summary.files == 1
|
||||
assert summary.formats == {"pdf": 1}
|
||||
assert summary.errors == 0
|
||||
assert summary.unchanged == 0
|
||||
assert summary.pruned == 0
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == SOURCE, Document.path == PDF_NAME)
|
||||
)
|
||||
assert doc is not None, "the guide did not import"
|
||||
assert doc.is_image is False
|
||||
assert doc.title == TITLE # the stem fallback (no metadata title)
|
||||
|
||||
content = doc.content
|
||||
# The stored content is CLEAN — the pre-126 XML-garbage import can
|
||||
# never come back (the garbage read is ~0.46 XML-ish AND heavy on
|
||||
# U+FFFD; the real extraction is 0.0 / ~0.0005).
|
||||
assert _xml_frac(content) < XML_FRACTION_BAR, (
|
||||
f"the guide's content is {_xml_frac(content):.0%} XML-ish — the "
|
||||
"pre-126 raw-bytes import came back (the PDF must not be routed "
|
||||
"through the text branch)"
|
||||
)
|
||||
assert _nonprint_frac(content) < NONPRINT_FRACTION_BAR, (
|
||||
f"the guide's content is {_nonprint_frac(content):.2%} non-printable — "
|
||||
"binary soup, not extracted text"
|
||||
)
|
||||
# The real first-page text is present (the pinned fixture text) and
|
||||
# the content opens with the page-1 marker (the phase-126 A2 shape).
|
||||
assert FIRST_PAGE_TEXT in content
|
||||
assert content.startswith(PAGE_SEPARATOR.format(page=1))
|
||||
|
||||
# The real pipeline ran to the end: page-anchored content chunks
|
||||
# with 768-dim embeddings + exactly one ``is_summary`` chunk at
|
||||
# position −1 (the ``lite`` summary, stored with the pointer line).
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert content_chunks, "the guide's content produced no chunks"
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||
assert doc.summary is not None
|
||||
assert doc.summary.endswith(f"Source: {PDF_REL}")
|
||||
|
||||
# The embedded-image docs (the phase-126 A3 ``#page=…#img=…``
|
||||
# rows): every one of them got the FAKE vision description — the
|
||||
# multimodal ``chat`` was called (the task's literal assertion),
|
||||
# exactly once per image doc, with the locked bare prompt.
|
||||
image_docs = db.scalars(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE,
|
||||
Document.is_image.is_(True),
|
||||
Document.path.like(f"{PDF_NAME}#page=%"),
|
||||
)
|
||||
).all()
|
||||
assert image_docs, "the guide's embedded images did not index"
|
||||
for img in image_docs:
|
||||
assert img.content == GuideVisionLLM.VISION_REPLY
|
||||
assert img.summary == img.content # the image convention
|
||||
assert img.image_path is not None
|
||||
copy = Path(img.image_path)
|
||||
assert copy.parent == Path(llm.settings.image_dir).expanduser()
|
||||
assert copy.exists(), "the embedded image's copy is missing"
|
||||
multimodal_calls = [
|
||||
messages
|
||||
for messages in llm.chat_calls
|
||||
if any(
|
||||
isinstance(m.get("content"), list) for m in messages if m.get("role") == "user"
|
||||
)
|
||||
]
|
||||
assert len(multimodal_calls) >= 1, "the multimodal chat was never called"
|
||||
assert len(llm.vision_texts) == len(image_docs), (
|
||||
"one vision call per embedded image doc (no failures)"
|
||||
)
|
||||
# The bare locked prompt on every call (no markdown reference
|
||||
# exists in this PDF-only source — the phase-126 A3 shape).
|
||||
assert all(t == DESCRIBE_PROMPT for t in llm.vision_texts)
|
||||
assert summary.images_failed == 0
|
||||
|
||||
# The added accounting (see the module docstring): the walked file
|
||||
# is EXACTLY ONE added document (the guide PDF); every other added
|
||||
# doc is an embedded-image byproduct (the phase-126 A3 convention
|
||||
# counts image docs added/updated) — nothing else touched the KB.
|
||||
assert summary.added == 1 + len(image_docs)
|
||||
assert summary.updated == 0
|
||||
+422
-3
@@ -14,6 +14,8 @@ for the fail-soft path.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
@@ -985,14 +987,16 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
"""PLAN §9 summary line: the phase-30 counters sit between
|
||||
``embed_batches`` and ``formats``; the phase-118 backfill counter
|
||||
sits between ``summary_errors`` and ``dates_updated``; the
|
||||
phase-106 date-refresh counter and the phase-122 ``images_failed``
|
||||
counter sit after ``dates_updated``, before ``formats``."""
|
||||
phase-106 date-refresh counter, the phase-122 ``images_failed``
|
||||
counter, and the phase-133 ``pdfs_screenshotted`` counter sit after
|
||||
``dates_updated``, before ``formats``."""
|
||||
s = ImportSummary()
|
||||
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
|
||||
s.summaries, s.summary_errors = 2, 1
|
||||
s.summary_backfilled = 1
|
||||
s.dates_updated = 0
|
||||
s.images_failed = 0
|
||||
s.pdfs_screenshotted = 1
|
||||
s.formats = {"md": 1, "yaml": 2}
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
s.log()
|
||||
@@ -1000,7 +1004,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
assert line == (
|
||||
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||
"dates_updated=0 images_failed=0 formats=yaml:2,md:1"
|
||||
"dates_updated=0 images_failed=0 pdfs_screenshotted=1 formats=yaml:2,md:1"
|
||||
)
|
||||
|
||||
|
||||
@@ -1644,3 +1648,418 @@ def test_on_chunk_progress_pdf_branch_reports_content_chunks(
|
||||
# The content embed (the 2 page chunks, one batch) + the excluded
|
||||
# summary-chunk embed — the hook fired for the first only.
|
||||
assert [len(c) for c in llm.calls] == [2, 1]
|
||||
|
||||
|
||||
# ---------- phase 133, task 03: the page-screenshot fallback ----------
|
||||
|
||||
|
||||
class _PageVisionLLM(FakeEmbedder):
|
||||
"""The phase-133 mock VISION client — a STABLE vision model: the
|
||||
MULTIMODAL describe call (``user`` content = a part list — the
|
||||
shape ``describe_image`` sends) gets a canned reply DETERMINED BY
|
||||
THE IMAGE'S BYTES (a short sha256 of the data URL's base64 — the
|
||||
same rendered page always gets the same reply, ACROSS RUNS; the
|
||||
pages' bytes differ, so the replies differ and the page order
|
||||
stays checkable, and the digest-stability test's "stable vision
|
||||
model" premise holds). Text (``lite``) calls keep the
|
||||
``FakeEmbedder`` behaviour. Every vision call's model is recorded
|
||||
(``vision_models``)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.vision_calls = 0
|
||||
self.vision_models: list[str | None] = []
|
||||
|
||||
@staticmethod
|
||||
def reply_for(image_bytes: bytes) -> str:
|
||||
"""The canned reply a vision call for *image_bytes* gets —
|
||||
deterministic in the bytes (tests build the expected content
|
||||
with the same helper)."""
|
||||
return f"Page {hashlib.sha256(image_bytes).hexdigest()[:8]} described."
|
||||
|
||||
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
self.vision_calls += 1
|
||||
self.vision_models.append(model)
|
||||
for part in user:
|
||||
if part.get("type") == "image_url":
|
||||
b64 = part["image_url"]["url"].rsplit(",", 1)[1]
|
||||
return self.reply_for(base64.b64decode(b64))
|
||||
raise LLMError("no image part in the multimodal message (test bug)")
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
class _FirstPageFailsVisionLLM(FakeEmbedder):
|
||||
"""A vision client that REJECTS the first multimodal call (a dead
|
||||
page's vision request) and then describes — drives the per-page
|
||||
fail-soft (the dead page becomes the placeholder, the next page
|
||||
still gets its description)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.vision_calls = 0
|
||||
|
||||
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
self.vision_calls += 1
|
||||
if self.vision_calls == 1:
|
||||
raise LLMError("simulated dead vision call (test sentinel)")
|
||||
return "Page two described fine."
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
class _AlwaysFailsVisionLLM(FakeEmbedder):
|
||||
"""A vision client that REJECTS every multimodal call (the dead
|
||||
vision endpoint — the per-page fail-soft's total-failure case:
|
||||
every page becomes the placeholder and the doc lands empty)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.vision_calls = 0
|
||||
|
||||
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
self.vision_calls += 1
|
||||
raise LLMError("simulated dead vision endpoint (test sentinel)")
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
def _pdf_settings(tmp_path: Path, images: bool = False) -> Settings:
|
||||
"""Settings admitting the ``pdf`` import token (the code DEFAULT set
|
||||
has no ``pdf`` — the phase-133 task-01 trap: a default fake would
|
||||
silently skip the PDF) with the image knobs at their test values
|
||||
(``images`` off unless pinned otherwise — the fallback's own vision
|
||||
calls need no image copy, only the phase-122 embedded-image pass
|
||||
does)."""
|
||||
return Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
|
||||
import_extensions="md,txt,pdf",
|
||||
images=images,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
)
|
||||
|
||||
|
||||
def _build_garbage_pdf(path: Path, pages: int = 2) -> None:
|
||||
"""A REAL tiny garbage PDF: every page carries ONLY XML-ish lines
|
||||
(``<foo:pN>i</foo:pN>`` — page-specific tags, so the rendered page
|
||||
PNGs differ and the stable fake's replies do too) — the gate's XML
|
||||
signal fails deterministically (100% of the text lines look like
|
||||
tags; the ``---PAGE-N---`` separators are not tag-shaped). The text
|
||||
layer is real EXTRACTABLE text — this is the FINDING's class (the
|
||||
pre-126 raw-XMP soup), not the empty-layer class."""
|
||||
doc = fitz.open()
|
||||
for p in range(1, pages + 1):
|
||||
page = doc.new_page()
|
||||
for i in range(10):
|
||||
page.insert_text((72, 72 + i * 14), f"<foo:p{p}>{i}</foo:p{p}>")
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _pdf_doc(db, source: str, rel: str) -> Document:
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == rel)
|
||||
)
|
||||
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
|
||||
return doc
|
||||
|
||||
|
||||
def test_garbage_pdf_takes_the_page_screenshot_fallback(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The owner's specified path end to end: a PDF whose text layer is
|
||||
XML garbage fails the gate, every page is rendered to a PNG and
|
||||
read by the vision model, and the doc lands with the
|
||||
``---PAGE-N---``-anchored page descriptions (PAGE ORDER pinned by
|
||||
the differing canned replies) — the page-aware chunker works
|
||||
unchanged on the separators, the digest is over the FINAL content,
|
||||
and the counter + the loud warning (the FINDING's signature) fire.
|
||||
``is_image`` stays False (it is a PDF document whose content
|
||||
happens to be vision text)."""
|
||||
root = tmp_path / "garbage"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "garbage.pdf")
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path)
|
||||
# The stable fake's expected replies (rendered page bytes → reply;
|
||||
# page order pinned: page i's bytes carry page i's description).
|
||||
pngs = importer.render_pdf_pages(root / "garbage.pdf")
|
||||
d1, d2 = _PageVisionLLM.reply_for(pngs[0]), _PageVisionLLM.reply_for(pngs[1])
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
assert llm.vision_calls == 2 # one vision call per page
|
||||
|
||||
doc = _pdf_doc(db, "garbage", "garbage.pdf")
|
||||
assert doc.is_image is False and doc.image_path is None
|
||||
assert doc.content == f"---PAGE-1---\n{d1}\n\n---PAGE-2---\n{d2}"
|
||||
assert d1 != d2, "the pages' bytes differ — so do their replies"
|
||||
# The digest runs over the FINAL (fallback) content — the
|
||||
# phase-126 digest rule, unchanged.
|
||||
assert doc.content_hash == hashlib.sha256(
|
||||
doc.content.encode("utf-8")
|
||||
).hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [
|
||||
f"Page 1:\n{d1}",
|
||||
f"Page 2:\n{d2}",
|
||||
]
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||
assert doc.summary is not None
|
||||
|
||||
# The loud warning (PLAN §9, the FINDING's signature) — the
|
||||
# required line, exact.
|
||||
warnings = [
|
||||
r for r in caplog.records if "text layer unusable" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||
assert warnings[0].getMessage() == (
|
||||
"pdf: text layer unusable for garbage/garbage.pdf — "
|
||||
"falling back to 2 page screenshots"
|
||||
)
|
||||
# The vision calls went to the CHAT model (the phase-122 seam).
|
||||
assert llm.vision_models == [llm.settings.llm_chat_model] * 2
|
||||
finally:
|
||||
_cleanup_source(db, "garbage")
|
||||
|
||||
|
||||
def test_clean_pdf_never_burns_a_fallback_vision_call(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The gate is the ONLY gate: a clean text layer never reaches the
|
||||
fallback — ``pdfs_screenshotted`` stays 0 and the multimodal chat
|
||||
is NEVER called (the images toggle is off, so the only vision seam
|
||||
in play is the fallback itself; the phase-126 happy path is
|
||||
byte-identical — the extracted text is the content, no fallback
|
||||
warning, no ``Page N —`` placeholders)."""
|
||||
root = tmp_path / "clean"
|
||||
root.mkdir()
|
||||
_build_one_page_pdf(root / "clean.pdf", "Clean prose about the homelab.")
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 0
|
||||
assert llm.vision_calls == 0
|
||||
doc = _pdf_doc(db, "clean", "clean.pdf")
|
||||
assert doc.content == "---PAGE-1---\nClean prose about the homelab."
|
||||
assert not [
|
||||
r for r in caplog.records if "text layer unusable" in r.getMessage()
|
||||
]
|
||||
finally:
|
||||
_cleanup_source(db, "clean")
|
||||
|
||||
|
||||
def test_fallback_is_fail_soft_per_page(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""A dead vision call on ONE page → the deterministic placeholder
|
||||
line (``page N — description unavailable``) for that page, the
|
||||
other page still gets its description — the doc still lands (a
|
||||
dead vision endpoint must not sink the doc, the phase-122
|
||||
fail-soft spirit). The vision seam logged its failure (the
|
||||
phase-122 warning); the fallback counted once."""
|
||||
root = tmp_path / "partfail"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "partfail.pdf", pages=2)
|
||||
llm = _FirstPageFailsVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.summarizer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
assert llm.vision_calls == 2 # both pages were attempted
|
||||
doc = _pdf_doc(db, "partfail", "partfail.pdf")
|
||||
assert doc.content == (
|
||||
"---PAGE-1---\npage 1 — description unavailable\n\n"
|
||||
"---PAGE-2---\nPage two described fine."
|
||||
)
|
||||
# The phase-122 seam's failure warning (the dead page's signal).
|
||||
assert [
|
||||
r for r in caplog.records if "image description failed" in r.getMessage()
|
||||
]
|
||||
finally:
|
||||
_cleanup_source(db, "partfail")
|
||||
|
||||
|
||||
def test_fallback_all_pages_failed_lands_empty_content(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""A garbage PDF whose vision calls ALL fail (the dead vision
|
||||
endpoint) → the existing EMPTY-CONTENT path: the doc lands with
|
||||
``""`` (the empty byte string's digest, no content chunks), the
|
||||
fallback was still taken (``pdfs_screenshotted == 1`` — the gate
|
||||
fired), and the phase-126 "no extractable text" warning fires —
|
||||
the doc really is empty. A dead endpoint must not sink the sync."""
|
||||
root = tmp_path / "allfail"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "allfail.pdf", pages=1)
|
||||
llm = _AlwaysFailsVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
assert llm.vision_calls == 1
|
||||
doc = _pdf_doc(db, "allfail", "allfail.pdf")
|
||||
assert doc.content == ""
|
||||
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c for c in chunks if not c.is_summary] == []
|
||||
warnings = [
|
||||
r for r in caplog.records if "no extractable text" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||
finally:
|
||||
_cleanup_source(db, "allfail")
|
||||
|
||||
|
||||
def test_fallback_unrenderable_page_gets_the_placeholder(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A page that cannot be RENDERED (``b""`` at its index — the
|
||||
:func:`render_pdf_pages` fail-soft slot) gets the deterministic
|
||||
placeholder while the good pages keep their descriptions — the
|
||||
page index never shifts (``---PAGE-N---`` stays truthful), and the
|
||||
unrenderable page burns no vision call."""
|
||||
root = tmp_path / "halfblind"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "halfblind.pdf", pages=3)
|
||||
monkeypatch.setattr(
|
||||
importer,
|
||||
"render_pdf_pages",
|
||||
lambda _path, dpi=150: [b"\x89PNG-page-1", b"", b"\x89PNG-page-3"],
|
||||
)
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path)
|
||||
d1 = _PageVisionLLM.reply_for(b"\x89PNG-page-1")
|
||||
d3 = _PageVisionLLM.reply_for(b"\x89PNG-page-3")
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
assert llm.vision_calls == 2 # the blank page burned no vision call
|
||||
doc = _pdf_doc(db, "halfblind", "halfblind.pdf")
|
||||
assert doc.content == (
|
||||
f"---PAGE-1---\n{d1}\n\n"
|
||||
"---PAGE-2---\npage 2 — description unavailable\n\n"
|
||||
f"---PAGE-3---\n{d3}"
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "halfblind")
|
||||
|
||||
|
||||
def test_fallback_rerun_lands_unchanged_no_reembed(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Digest stability (the task's re-burn guard): the digest runs
|
||||
over the constructed fallback content — a re-sync of an unchanged
|
||||
garbage PDF (same pages + the deterministic fake's same replies)
|
||||
re-imports as UNCHANGED: the fallback re-runs (the gate + the
|
||||
render are cheap; the vision calls are re-issued — the documented
|
||||
no-cache cost), but the digest MATCHES, so the row is skipped —
|
||||
no re-chunk, no re-embed, no re-summary, no KB change (the row is
|
||||
not ``updated`` and records no changed path). The doc's content
|
||||
and digest are byte-stable across the re-sync."""
|
||||
root = tmp_path / "stable"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "stable.pdf")
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path)
|
||||
try:
|
||||
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s1.added, s1.pdfs_screenshotted) == (1, 1)
|
||||
assert llm.vision_calls == 2
|
||||
embeds1 = len(llm.calls)
|
||||
doc1 = _pdf_doc(db, "stable", "stable.pdf")
|
||||
content1, digest1 = doc1.content, doc1.content_hash
|
||||
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
# The fallback re-ran (gate re-fired, vision re-issued) — but
|
||||
# the row itself re-imported as unchanged (no re-embed, no
|
||||
# re-chunk, no KB change: the stable replies give the same
|
||||
# digest, so the upsert is skipped entirely).
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 0, 1)
|
||||
assert s2.pdfs_screenshotted == 1 # the gate re-fired
|
||||
assert llm.vision_calls == 4 # the vision calls were re-issued
|
||||
assert len(llm.calls) == embeds1, "unchanged: no re-embed"
|
||||
assert s2.changed_paths == [], "unchanged: no KB change recorded"
|
||||
|
||||
doc2 = _pdf_doc(db, "stable", "stable.pdf")
|
||||
assert doc2.content == content1 # byte-stable content
|
||||
assert doc2.content_hash == digest1 # byte-stable digest
|
||||
finally:
|
||||
_cleanup_source(db, "stable")
|
||||
|
||||
|
||||
def test_fallback_embedded_images_still_join_the_pipeline(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Both, no conflict (the phase design): a fallback PDF with the
|
||||
images toggle ON ALSO runs the phase-126 embedded-image pass — the
|
||||
screenshots are the CONTENT (the fallback's vision calls: one per
|
||||
page), the extracted images are the citation figures (the
|
||||
phase-122 pass's vision calls: one per image). The content keeps
|
||||
the page descriptions; the image doc is separate."""
|
||||
root = tmp_path / "both"
|
||||
root.mkdir()
|
||||
# Garbage text (the gate fires) + an embedded image on page 2
|
||||
# (the phase-126 A3 pass indexes it).
|
||||
doc = fitz.open()
|
||||
page1 = doc.new_page()
|
||||
for i in range(10):
|
||||
page1.insert_text((72, 72 + i * 14), f"<foo:bar>{i}</foo:bar>")
|
||||
page2 = doc.new_page()
|
||||
for i in range(10):
|
||||
page2.insert_text((72, 72 + i * 14), f"<foo:baz>{i}</foo:baz>")
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
|
||||
pix.clear_with(200)
|
||||
page2.insert_image(fitz.Rect(72, 220, 112, 250), stream=pix.tobytes("png"))
|
||||
doc.save(root / "both.pdf")
|
||||
doc.close()
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path, images=True)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
# The PDF (screenshot content) + its one embedded image doc.
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 2, 0)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
assert summary.images_failed == 0
|
||||
# 2 fallback page descriptions + 1 embedded-image description.
|
||||
assert llm.vision_calls == 3
|
||||
|
||||
pdf_doc = _pdf_doc(db, "both", "both.pdf")
|
||||
pngs = importer.render_pdf_pages(root / "both.pdf")
|
||||
d1 = _PageVisionLLM.reply_for(pngs[0])
|
||||
d2 = _PageVisionLLM.reply_for(pngs[1])
|
||||
assert d1 != d2
|
||||
assert pdf_doc.content == f"---PAGE-1---\n{d1}\n\n---PAGE-2---\n{d2}"
|
||||
# The embedded image (the clear_with(200) 40x30 PNG) gets its
|
||||
# OWN stable reply — the third vision call, the citation figure.
|
||||
image_doc = _pdf_doc(db, "both", "both.pdf#page=2#img=0.png")
|
||||
assert image_doc.is_image is True
|
||||
assert image_doc.content != d1 and image_doc.content != d2
|
||||
finally:
|
||||
_cleanup_source(db, "both")
|
||||
|
||||
+297
-16
@@ -12,6 +12,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import tarfile
|
||||
from collections.abc import Callable, Sequence
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
@@ -29,12 +30,17 @@ from app.rag import pdf as pdf_mod
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import EmbeddingError, LLMError
|
||||
from app.rag.pdf import (
|
||||
_XMLISH_LINE_RE,
|
||||
NONPRINT_CHAR_FRACTION,
|
||||
PAGE_SEPARATOR,
|
||||
XMLISH_LINE_FRACTION,
|
||||
PdfImage,
|
||||
_mime_from_image_info,
|
||||
_page_image_entries,
|
||||
extract_pdf_images,
|
||||
extract_pdf_text,
|
||||
is_usable_pdf_text,
|
||||
render_pdf_pages,
|
||||
)
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
@@ -475,13 +481,20 @@ def test_pdf_text_change_updates_doc_and_keeps_unchanged_images(
|
||||
_cleanup_source(db, "pdfchg")
|
||||
|
||||
|
||||
def test_pdf_without_text_still_becomes_a_document(
|
||||
def test_pdf_without_text_takes_the_screenshot_fallback(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""An image-only PDF (no extractable text) still becomes a document
|
||||
— with empty content (the empty byte string's digest), no content
|
||||
chunks, and a warning (the signal). The summary path still runs
|
||||
(best-effort on the empty content)."""
|
||||
"""Phase 133 (task 03): an image-only PDF (an EMPTY text layer — the
|
||||
gate's other target) takes the page-screenshot fallback: the single
|
||||
page is rendered to a PNG and vision-described (the mock's canned
|
||||
:data:`DESCRIPTION`), and the doc lands with the ``---PAGE-N---``-
|
||||
anchored description as content (chunked + summarized as any other
|
||||
PDF). The phase-126 empty-content behavior is retired for layers
|
||||
that CAN be screenshotted — it survives only for an unopenable
|
||||
file (``test_corrupt_pdf_file_becomes_an_empty_document``), where
|
||||
the render fails too. The loud fallback warning fires; the phase-
|
||||
126 "no extractable text" warning does NOT (the doc is not empty).
|
||||
"""
|
||||
root = tmp_path / "pdfempty"
|
||||
root.mkdir()
|
||||
_build_pdf(root / "pics.pdf", [("", [_png_bytes(40, 30)])])
|
||||
@@ -490,41 +503,61 @@ def test_pdf_without_text_still_becomes_a_document(
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added) == (1, 1)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
doc = _doc(db, "pdfempty", "pics.pdf")
|
||||
assert doc.content == ""
|
||||
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
|
||||
assert doc.content == f"{PAGE_SEPARATOR.format(page=1)}\n{DESCRIPTION}"
|
||||
assert doc.content_hash == hashlib.sha256(doc.content.encode("utf-8")).hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c for c in chunks if not c.is_summary] == []
|
||||
assert doc.summary is not None # best-effort on the empty content
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [f"Page 1:\n{DESCRIPTION}"]
|
||||
assert doc.summary is not None
|
||||
# The loud fallback warning (the FINDING's signature) fired —
|
||||
# and the phase-126 "no extractable text" warning did NOT (the
|
||||
# doc is not empty — the screenshots ARE the content).
|
||||
warnings = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if "no extractable text" in r.getMessage()
|
||||
r for r in caplog.records if "text layer unusable" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||
assert "falling back to 1 page screenshots" in warnings[0].getMessage()
|
||||
assert not [
|
||||
r for r in caplog.records if "no extractable text" in r.getMessage()
|
||||
]
|
||||
finally:
|
||||
_cleanup_source(db, "pdfempty")
|
||||
|
||||
|
||||
def test_corrupt_pdf_file_becomes_an_empty_document(db, tmp_path: Path) -> None:
|
||||
def test_corrupt_pdf_file_becomes_an_empty_document(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""A .pdf file that is not a PDF at all: the parser's fail-soft
|
||||
contract (no text, no images) — the document lands with empty
|
||||
content and no chunks, and the sync stays green."""
|
||||
contract (no text, no images) AND the phase-133 fallback's
|
||||
render-less branch (``render_pdf_pages`` → ``[]`` → the helper's
|
||||
``""``) — the document lands with empty content (the empty byte
|
||||
string's digest) and no chunks, the fallback is still counted
|
||||
(the gate fired, the render produced nothing), and the phase-126
|
||||
"no extractable text" warning fires — the doc really is empty."""
|
||||
root = tmp_path / "pdfbad"
|
||||
root.mkdir()
|
||||
(root / "broken.pdf").write_bytes(b"this is not a pdf at all")
|
||||
llm = _pdf_llm(tmp_path)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1 # the gate fired
|
||||
doc = _doc(db, "pdfbad", "broken.pdf")
|
||||
assert doc.content == "" and doc.title == "broken"
|
||||
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c for c in chunks if not c.is_summary] == []
|
||||
warnings = [
|
||||
r for r in caplog.records if "no extractable text" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||
finally:
|
||||
_cleanup_source(db, "pdfbad")
|
||||
|
||||
@@ -1074,3 +1107,251 @@ def test_page_image_entries_falls_back_on_old_pymupdf() -> None:
|
||||
)
|
||||
entries = _page_image_entries(page)
|
||||
assert entries == [{"xref": 7, "width": 40, "height": 30}]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# is_usable_pdf_text (phase 133 task 02 — the deterministic garbage gate)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The owner's untracked fixture (the stable reference — the same
|
||||
#: provenance as the phase-133 task-01 integration regression pin:
|
||||
#: the tarball, never the extracted PDF, is what the test starts
|
||||
#: from, and it never touches the repo).
|
||||
TARBALL = Path(__file__).resolve().parents[2] / "test_sources" / "hnb_2026_guide.tar.gz"
|
||||
|
||||
#: The inner file's name (it keeps its name through extraction — the
|
||||
#: task-01 convention).
|
||||
PDF_NAME = "HBI-2026-Guide-Year-Round_260704_131221.pdf"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def pre126_garbage_text(tmp_path_factory: pytest.TempPathFactory) -> str:
|
||||
"""THE REAL GARBAGE (the task's airtight pin, module-scoped):
|
||||
the owner's exact PDF read with the PRE-PHASE-126 recipe —
|
||||
``read_text(encoding="utf-8", errors="replace")`` +
|
||||
``.replace("\\x00", "")`` — the 46%-XML-ish, U+FFFD-heavy soup
|
||||
that was chunked and embedded before the parser shipped (the
|
||||
FINDING: 138,170 non-empty lines, ~46% XML-ish, the owner's
|
||||
``<illustrator:StartupProfile>`` preview). The tarball's single
|
||||
member is extracted into a temp dir (the task-01 integration
|
||||
pin's pattern, shared — the fixture stays out of the repo)."""
|
||||
if not TARBALL.is_file():
|
||||
pytest.skip(
|
||||
"test_sources/hnb_2026_guide.tar.gz is missing — the "
|
||||
"untracked owner fixture (2.6 MB, one 48-page Illustrator PDF)"
|
||||
)
|
||||
root = tmp_path_factory.mktemp("pre126")
|
||||
with tarfile.open(TARBALL, "r:gz") as tar:
|
||||
tar.extractall(root, filter="data")
|
||||
member = root / PDF_NAME
|
||||
return member.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||||
|
||||
|
||||
def test_gate_clean_text_with_page_separators_is_usable() -> None:
|
||||
"""Real prose + the phase-126 ``---PAGE-N---`` separators: the
|
||||
happy path the gate must NEVER block (a clean PDF never burns a
|
||||
vision call — the gate is the only gate, and the happy path stays
|
||||
byte-identical to phase 126)."""
|
||||
clean = (
|
||||
f"{PAGE_SEPARATOR.format(page=1)}\n2026 U.S. Benefits Guide\n"
|
||||
"\n"
|
||||
"Covered employees receive comprehensive medical, dental, and\n"
|
||||
"vision benefits under this plan document.\n"
|
||||
"\n"
|
||||
f"{PAGE_SEPARATOR.format(page=2)}\nPremiums are deducted from\n"
|
||||
"each paycheck; the plan year runs January 1 through\n"
|
||||
"December 31."
|
||||
)
|
||||
assert is_usable_pdf_text(clean) is True
|
||||
|
||||
|
||||
def test_gate_empty_text_layer_is_unusable() -> None:
|
||||
"""The empty text layer — a scanned/image-only PDF has nothing to
|
||||
read — is the fallback's other target."""
|
||||
assert is_usable_pdf_text("") is False
|
||||
assert is_usable_pdf_text(" \n\t\n \r\n") is False
|
||||
|
||||
|
||||
def test_gate_xml_heavy_layer_is_unusable() -> None:
|
||||
"""The raw-XMP / binary-soup signature: ≥ 30% of the non-empty
|
||||
lines look like tags (``<tag>``, ``</tag>``, ``<tag attr=`` — the
|
||||
regex covers them all, including the PDF-structure ``<</Key…``
|
||||
noise)."""
|
||||
tags = [f"<xmp:Prop{i}>value</xmp:Prop{i}>" for i in range(30)]
|
||||
prose = [f"Line {i} of ordinary prose." for i in range(70)]
|
||||
assert is_usable_pdf_text("\n".join(tags + prose)) is False # 30/100
|
||||
|
||||
|
||||
def test_gate_borderline_document_about_xml_stays_usable() -> None:
|
||||
"""ASSUMPTION A8's edge from below: a real document ABOUT XML
|
||||
whose tag-like lines stay UNDER the 0.30 bar (29%) is usable —
|
||||
the documented false-positive cost is paid only at/above the bar."""
|
||||
tags = [f"<example:{i}>text</example:{i}>" for i in range(29)]
|
||||
prose = [
|
||||
f"Paragraph {i} explains the markup in plain words."
|
||||
for i in range(71)
|
||||
]
|
||||
assert is_usable_pdf_text("\n".join(tags + prose)) is True # 29/100
|
||||
|
||||
|
||||
def test_gate_at_and_above_the_xml_bar_is_unusable() -> None:
|
||||
"""The threshold's edges from above: EXACTLY 30% tag-like lines is
|
||||
already unusable (the bar is ≥, not >), and 31% is too — a document
|
||||
ABOUT XML that reaches the bar pays the documented ASSUMPTION A8
|
||||
cost (the fallback still extracts it correctly via vision)."""
|
||||
for n in (30, 31):
|
||||
tags = [f"<example:{i}>text</example:{i}>" for i in range(n)]
|
||||
prose = [
|
||||
f"Paragraph {i} of ordinary explanation."
|
||||
for i in range(100 - n)
|
||||
]
|
||||
assert is_usable_pdf_text("\n".join(tags + prose)) is False
|
||||
|
||||
|
||||
def test_gate_replacement_char_soup_is_unusable() -> None:
|
||||
"""The SECOND signal, alone: a ≥ 5% U+FFFD load (the
|
||||
``errors="replace"`` binary-soup signal) is unusable even with few
|
||||
tag-like lines — the XML-ish gate does not mask it, and this soup
|
||||
has none (the two failure signals fire independently)."""
|
||||
soup = "Benefits summary paragraph. " * 24 + "\ufffd" * 60 # ~7.9%
|
||||
assert is_usable_pdf_text(soup) is False
|
||||
|
||||
|
||||
def test_gate_structural_newlines_and_tabs_are_printable() -> None:
|
||||
"""The documented exclusion: the ``\\n``/``\\t``/``\\r`` the
|
||||
extractor legitimately emits are NOT non-printable — a
|
||||
newline/tab/indent-heavy document is printable (counting them
|
||||
would flag every prose document)."""
|
||||
text = "\n\t".join(f"Indented line {i} with a tab." for i in range(20))
|
||||
assert is_usable_pdf_text(text) is True
|
||||
|
||||
|
||||
def test_gate_real_pre126_garbage_is_unusable(
|
||||
pre126_garbage_text: str,
|
||||
) -> None:
|
||||
"""THE REAL GARBAGE (the airtight pin): the owner's exact file,
|
||||
read with the pre-126 recipe (the module fixture) — the
|
||||
reproduced 46%-XML-ish, U+FFFD-heavy soup — must be flagged
|
||||
unusable. Both signals fire on the REAL string (the FINDING's
|
||||
evidence, pinned against the named thresholds: if the owner ever
|
||||
re-cuts the fixture, the drift shows up here — the pin has teeth,
|
||||
the same contract as the task-01 suite)."""
|
||||
assert is_usable_pdf_text(pre126_garbage_text) is False
|
||||
# The owner's exact preview line is in the reproduced soup.
|
||||
assert "<illustrator:StartupProfile>" in pre126_garbage_text
|
||||
lines = [line for line in pre126_garbage_text.splitlines() if line.strip()]
|
||||
xml_frac = sum(1 for line in lines if _XMLISH_LINE_RE.search(line)) / len(lines)
|
||||
nonprint_frac = sum(
|
||||
1
|
||||
for ch in pre126_garbage_text
|
||||
if ch == "\ufffd" or (ord(ch) < 32 and ch not in "\t\n\r")
|
||||
) / len(pre126_garbage_text)
|
||||
# Both signals independently clear their named bars on the real
|
||||
# string — neither signal can silently die.
|
||||
assert xml_frac >= XMLISH_LINE_FRACTION
|
||||
assert nonprint_frac >= NONPRINT_CHAR_FRACTION
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# render_pdf_pages (phase 133 task 03 — the fallback's visual input)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _two_page_text_pdf(path: Path) -> Path:
|
||||
"""A 2-page temp PDF with EXPLICIT letter pages (612x792pt —
|
||||
deterministic render sizes: 1275x1650 px at 150 DPI, 612x792 at
|
||||
72 DPI) and page-specific text (the render must be page-specific,
|
||||
so the pages' PNG bytes must differ)."""
|
||||
doc = fitz.open()
|
||||
doc.new_page(width=612, height=792).insert_text((72, 72), "First page marker text.")
|
||||
doc.new_page(width=612, height=792).insert_text((72, 72), "Second page marker text.")
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
def test_render_pdf_pages_returns_one_png_per_page_in_order(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""A 2-page PDF → 2 PNGs in PAGE ORDER (list index i is page i+1),
|
||||
each a genuine decodable PNG at the default 150-DPI render (612x792pt
|
||||
letter → 1275x1650 px), and page-specific (page 1's bytes != page
|
||||
2's — different text on the page)."""
|
||||
path = _two_page_text_pdf(tmp_path / "two.pdf")
|
||||
pages = render_pdf_pages(path)
|
||||
assert len(pages) == 2
|
||||
for png in pages:
|
||||
assert png.startswith(PNG_MAGIC)
|
||||
pix = fitz.Pixmap(png)
|
||||
assert (pix.width, pix.height) == (1275, 1650)
|
||||
assert pages[0] != pages[1] # the render is page-specific
|
||||
|
||||
|
||||
def test_render_pdf_pages_honors_the_dpi_argument(tmp_path: Path) -> None:
|
||||
"""ASSUMPTION A8's fixed 150 DPI is the DEFAULT; an explicit *dpi*
|
||||
changes the render (72 DPI → the page's natural 612x792 px)."""
|
||||
path = _two_page_text_pdf(tmp_path / "two.pdf")
|
||||
pix = fitz.Pixmap(render_pdf_pages(path, dpi=72)[0])
|
||||
assert (pix.width, pix.height) == (612, 792)
|
||||
|
||||
|
||||
def test_render_pdf_pages_unopenable_file_returns_empty_list(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""A file that cannot be OPENED yields ``[]`` (the caller's existing
|
||||
empty-content path — the doc lands with the no-extractable-text
|
||||
behavior, unchanged) + a warning (the fail-soft style)."""
|
||||
bad = tmp_path / "bad.pdf"
|
||||
bad.write_bytes(b"this is not a pdf at all")
|
||||
with caplog.at_level(logging.WARNING, logger="app.rag.pdf"):
|
||||
assert render_pdf_pages(bad) == []
|
||||
assert render_pdf_pages(tmp_path / "missing.pdf") == []
|
||||
warnings = [
|
||||
r for r in caplog.records if "could not open" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 2 and all(w.levelno == logging.WARNING for w in warnings)
|
||||
|
||||
|
||||
def test_render_pdf_pages_zero_page_doc_returns_empty_list(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
|
||||
) -> None:
|
||||
"""A zero-page PDF (openable, no pages — this PyMuPDF refuses to
|
||||
SAVE such a file, so it is stubbed) yields ``[]`` — the loop runs
|
||||
zero times, nothing renders."""
|
||||
path = tmp_path / "zero.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: _FakeDoc([]))
|
||||
assert render_pdf_pages(path) == []
|
||||
|
||||
|
||||
def test_render_pdf_pages_broken_page_is_zero_bytes_at_its_index(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
|
||||
) -> None:
|
||||
"""A page whose get_pixmap raises → ``b""`` at ITS index (the
|
||||
caller's placeholder slot — the page index NEVER shifts, so
|
||||
``---PAGE-N---`` stays truthful); the good pages keep their PNGs
|
||||
at their own indices, and the failure is logged (the
|
||||
:func:`extract_pdf_text` fail-soft style)."""
|
||||
path = tmp_path / "fake.pdf"
|
||||
path.write_bytes(b"%PDF-1.4 fake")
|
||||
|
||||
class _RenderPage:
|
||||
def __init__(self, data: bytes | None) -> None:
|
||||
self._data = data
|
||||
|
||||
def get_pixmap(self, dpi: int) -> Any:
|
||||
if self._data is None:
|
||||
raise RuntimeError("boom: page render failed")
|
||||
return SimpleNamespace(tobytes=lambda _fmt: self._data)
|
||||
|
||||
doc = _FakeDoc(
|
||||
[
|
||||
_RenderPage(b"\x89PNG-fake-page-1"),
|
||||
_RenderPage(None),
|
||||
_RenderPage(b"\x89PNG-fake-page-3"),
|
||||
]
|
||||
)
|
||||
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
|
||||
assert render_pdf_pages(path) == [
|
||||
b"\x89PNG-fake-page-1", b"", b"\x89PNG-fake-page-3",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user