phase: 133_pdf_screenshot_fallback
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).
## Phase 133 final verification report
- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.
This commit is contained in:
@@ -1,43 +0,0 @@
|
||||
# Phase 133 — Garbage PDF text is detected; the page-screenshot vision fallback extracts it
|
||||
|
||||
**Source:** `TODO.md` L9 — "test_sources/hnb_2026_guide.tar.gz contains a pdf that is being read as binary. Figure out why. It contains images, tables, and text but none of that is being extracted correctly. Instead, the raw xml is being embedded and the preview shows the entire document as lines like `<illustrator:StartupProfile>Web</illustrator:StartupProfile>`. This project should support this kind of PDF and should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working."
|
||||
|
||||
**FINDING (diagnosed 2026-09-27, before phase authoring — the evidence, verified against the real fixture):**
|
||||
1. The archive holds ONE file: `HBI-2026-Guide-Year-Round_260704_131221.pdf` (6.2 MB, 48 pages, PDFium-produced, **Adobe Illustrator export** — the raw bytes carry 222 × `illustrator:StartupProfile` + 138 × `<x:xmpmeta` uncompressed XMP blocks).
|
||||
2. The CURRENT code (phase 126 onward) extracts it CORRECTLY: `extract_pdf_text` returns 2,585 lines, **0 XML-ish**; a full real-importer run stores clean content (`---PAGE-N---` blocks, "2026 U.S. Benefits Guide" first page), the title, and vision-described embedded images.
|
||||
3. The garbage the owner saw is the **pre-126 import**: `pdf` was already in `BOR_IMPORT_EXTENSIONS` in the local `.env` (untracked — predates the parser) BEFORE `126_pdf_support` shipped (commit `e0f11e7`, 2026-09-27 09:36). The generic text path (`importer.py` L689: `read_text(encoding="utf-8", errors="replace").replace("\x00", "")`) read the raw PDF bytes — the uncompressed XMP/structure blocks survive as literal text (reproduced: 138,170 non-empty lines, **46% XML-ish** — `<</Pages 2 0 R /Type/Catalog>>`, `<illustrator:StartupProfile>Web</…>`, exactly the owner's preview) — and that garbage was chunked + embedded.
|
||||
4. **Production fix is a re-sync**: the importer's content-hash delta sees the old garbage row's digest ≠ the newly extracted text's digest → the next sync re-indexes the file as `updated` with clean content (deploy phase 126 first — it is already on `main`). No migration needed. This is documented here, not a code task.
|
||||
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/pdf.py` (phase 126 — `extract_pdf_text` (L109, `page.get_text("text")` per page, `PAGE_SEPARATOR` blocks, fail-soft `""`), `extract_pdf_images` (L145), the fitz import); `app/rag/importer.py` — `_index_pdf_file` (the phase-126 PDF branch: extract → doc row → digest over extracted text → chunk + embed → lite summary → embedded images via `_describe_or_skip` (L1779)); `app/rag/summarizer.py` — `describe_image` (L208: ONE chat-model call, multimodal user message, `data: bytes` + `mime`, returns `str | None`, no app-level retry); `ImportSummary` (L212, the stats counters + `log()` line); `tests/e2e/mock_llm.py` — the `IMAGE_DESCRIPTION_MODE` canned vision reply (deterministic for the E2E); `test_sources/hnb_2026_guide.tar.gz` (the untracked fixture, 2.6 MB compressed).
|
||||
|
||||
## Objective
|
||||
Two guarantees, one for today and one for the class of problem: (a) the owner's guide is pinned as a regression — it imports through the real pipeline with CLEAN content (the root cause is documented above: a pre-126 binary-text import, self-healed by a re-sync); (b) any future PDF whose extracted text is GARBAGE (XML/binary soup, empty text layer) is DETECTED by a deterministic gate and falls back to the owner's specified path — every page rendered to a screenshot and read by the vision model, preserving what the text layer couldn't.
|
||||
|
||||
## Dependencies
|
||||
- `132_image_upload_feedback` (todo) — pipeline order only; code dependencies are the completed phase-126 PDF pipeline + the phase-122 vision path.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The gate (task 02):** `app/rag/pdf.py` gains `is_usable_pdf_text(text: str) -> bool` — pure, deterministic, unit-testable. Unusable when ANY of: `text.strip() == ""` (the empty text layer — the fallback's other target); the fraction of non-empty lines that look like XML is ≥ 0.30 (a line is XML-ish: `re.search(r"</?[A-Za-z_][\w:.-]*[\s/>=]", line)` — covers `<tag>`, `</tag>`, `<tag attr=`, and the PDF-structure `<</Key…` noise); the non-printable fraction (U+FFFD replacement chars + C0 control chars, over the whole text) is ≥ 0.05 (the binary-soup signal — the pre-126 read is 46% XML-ish AND heavy on U+FFFD; a real document about XML with ≥ 30% tag-like lines is a documented false-positive cost — ASSUMPTION A8). Applied in `_index_pdf_file` AFTER `extract_pdf_text` (the extractor stays pure).
|
||||
- **The fallback (task 03):** when the gate fails, `_index_pdf_file` takes the owner's specified path — "screenshotting the pages and reading each one as an image": `app/rag/pdf.py` gains `render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]` (one PNG per page, PyMuPDF `page.get_pixmap(dpi=dpi).tobytes("png")`, in page order; an unreadable page → a zero-byte entry the caller replaces with the placeholder — no page index may shift); for each page, `describe_image(llm, data=png, mime="image/png")` — the phase-122 vision call, fail-soft per page (`None` → the placeholder line). The stored `content` = `PAGE_SEPARATOR` blocks of the page descriptions (`---PAGE-N---\n<description or "page N — description unavailable">`) — the SAME separator convention as the text path, so the chunker's page-boundary handling works unchanged. `is_image` stays `False` (it is a PDF document whose content happens to be vision text). The digest is over the constructed content (the phase-126 convention — the fallback output is stable for a stable file, so re-syncs don't re-burn vision: a cached row re-imports as unchanged). ONE loud log line (PLAN §9): `pdf: text layer unusable for <source>/<path> (xml_frac=… nonprint=…) — falling back to N page screenshots`. `ImportSummary` gains `pdfs_screenshotted: int = 0` (the counter + the `log()` field; the sync `detail` shape stays untouched — the stats log-only convention).
|
||||
- **The regression pin (task 01):** the real guide, imported through the REAL importer (the fake LLM with a multimodal `chat` override for the embedded images), must land clean content — 0/low XML-ish fraction + the real first-page text present. The fixture stays the tar.gz (the test extracts it into `tmp_path` — never the repo, `test_sources/` is untracked).
|
||||
|
||||
## Tasks
|
||||
1. `01_diagnosis_regression_test.md` — the FINDING is documented in this overview; the task pins the real guide's clean import as a regression integration test.
|
||||
2. `02_pdf_text_gate.md` — `is_usable_pdf_text` (the heuristic) + the pre-126-garbage reproduction fixture.
|
||||
3. `03_page_screenshot_fallback.md` — `render_pdf_pages` + `_index_pdf_file`'s fallback branch + the stats counter.
|
||||
4. `04_e2e_garbage_pdf.md` — dedicated Playwright E2E, run in isolation (the real guide's happy path + a synthetic garbage PDF's fallback path).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the gate (clean text, the REAL pre-126 garbage string — reproduced by reading the fixture with the old `errors="replace"` recipe, XML-heavy text, empty text, a borderline document about XML); `render_pdf_pages` (page count, PNG magic bytes, the zero-byte unreadable-page placeholder); the fallback content assembly (separators, placeholders, the digest stability).
|
||||
- Integration: the real-guide regression (task 01).
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only (the `IMAGE_DESCRIPTION_MODE` canned reply is the fallback's vision surface).
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the real guide imports with clean content through the real pipeline (the regression test green — the owner's exact file)
|
||||
- [ ] a PDF whose text layer is XML/binary garbage (synthetic) lands with the vision-described page content — the owner's specified fallback fires end to end
|
||||
- [ ] a clean PDF NEVER burns a single vision call for the fallback (the gate is the only gate — the happy path is byte-identical to phase 126)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the phase-126 clean-PDF path, the image pipeline, the sync)
|
||||
- [ ] the production stale row is fixed by a re-sync (documented above — verify post-deploy: the guide's content no longer shows XML lines)
|
||||
@@ -1,30 +0,0 @@
|
||||
# Task 01 — The regression pin: the real guide imports CLEAN through the real pipeline
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "test_sources/hnb_2026_guide.tar.gz contains a pdf that is being read as binary. Figure out why. ... This project should support this kind of PDF and should be able to extract it correctly"
|
||||
|
||||
## Objective
|
||||
The owner's exact file is a permanent regression: importing `test_sources/hnb_2026_guide.tar.gz`'s PDF through the REAL importer (extraction, digest, chunking, summary, embedded images) lands CLEAN content — the pre-126 XML-garbage import can never come back, and the phase-126 happy path on this 48-page Illustrator PDF is pinned.
|
||||
|
||||
## Work
|
||||
1. `tests/integration/test_pdf_hnb_guide_regression.py` (new) — the house integration pattern (the local compose Postgres, the `tmp_path` fixture, the `tests/fakes.py` `FakeEmbedder`):
|
||||
- Fixture: extract `test_sources/hnb_2026_guide.tar.gz` into `tmp_path` (the test's local source dir — `test_sources/` is untracked, the tar.gz is the stable reference; the extraction is ~6 MB, seconds). The inner file keeps its name (`HBI-2026-Guide-Year-Round_260704_131221.pdf`).
|
||||
- The fake LLM: `FakeEmbedder` with (a) `settings = Settings(_env_file=".env")` (the real extension set — `pdf` is in the owner's live config; a `_env_file=None` fake would silently skip the PDF — the exact trap that hid this class before) and (b) a `chat` override that returns a deterministic string for MULTIMODAL user content (the `list`-of-parts shape `describe_image` sends — the fake's string-only `chat` crashes on it; the override: `"A rendered page of the guide (test vision description)."` — the docstring names it as the task-03-style vision mock the `fakes.py` comment anticipates).
|
||||
- Import with `import_sources([dir], llm)` (the real walk + the real PDF branch).
|
||||
2. Assertions (the observable contract):
|
||||
- `summary.added == 1` (the guide PDF) — the file is ADMITTED by the walk and indexed through the PDF branch (not the text branch: if the text branch ever ran on it again, the content assertions below fail);
|
||||
- the stored `content` is CLEAN: the XML-ish line fraction (< the task-02 gate's 0.30 threshold — assert < 0.05 to be strict: the real extraction is 0/2,585) AND the non-printable fraction < 0.01;
|
||||
- the real first-page text is present: `"2026 U.S. Benefits Guide"` in `content` (pinned fixture text — the file is a stable untracked fixture; if the owner ever re-cuts it, the test failure names the regression, which is the point);
|
||||
- `is_image is False`, `title == "HBI-2026-Guide-Year-Round_260704_131221"` (the stem fallback — the PDF's metadata carries no title, only PDFium creator/producer), the content starts with `---PAGE-1---`;
|
||||
- the embedded-image docs (the phase-126 `#page=…#img=…` rows) got the FAKE vision description (the multimodal `chat` was called — assert `llm.chat_calls` contains ≥ 1 multimodal message) or are absent when the fake "fails" (not this test's scope — the fail-soft is the phase-126 contract).
|
||||
- Wall-time guard: the test completes in < 120 s (48 pages × in-process fakes; if it drifts, the test is too heavy — the suite must stay fast).
|
||||
3. Header docstring: the FINDING summary (the pre-126 root cause in two lines — see the phase overview's FINDING block), the run command (`uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v`), the fixture provenance.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: this file IS the task's suite.
|
||||
- Coverage: **>90%** on new/modified code (test-only task — the bar is carried by the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v` green (DB up: `podman compose up -d db`)
|
||||
- [ ] the test fails if the PDF is ever routed through the text branch again (a temporary experiment moving the branch order must break it — the pin has teeth)
|
||||
- [ ] the test does not commit the extracted PDF (it lives in `tmp_path`; `git status` stays clean of binaries)
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -1,54 +0,0 @@
|
||||
# Task 02 — `is_usable_pdf_text`: the deterministic garbage gate
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "...even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working."
|
||||
|
||||
## Objective
|
||||
A pure, deterministic predicate decides whether an extracted PDF text layer is real content or garbage (the pre-126 XML/binary soup class, the empty text layer class) — so the screenshot fallback has exactly one trigger, unit-testable without any PDF.
|
||||
|
||||
## Work
|
||||
1. `app/rag/pdf.py` — new module-level predicate (place it after `extract_pdf_text`, with a docstring citing the FINDING — the pre-126 `read_text(errors="replace")` soup, the 46%-XML-ish reproduction, the owner's `<illustrator:StartupProfile>` preview):
|
||||
```python
|
||||
#: A line "looks like XML" when it carries a tag-shaped token — the
|
||||
#: pre-126 garbage's signature (raw XMP blocks, PDF-structure noise).
|
||||
_XMLISH_LINE_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
|
||||
|
||||
def is_usable_pdf_text(text: str) -> bool:
|
||||
"""Whether an extracted PDF text layer is REAL content (not garbage).
|
||||
|
||||
Unusable (→ the page-screenshot fallback) when ANY of:
|
||||
* ``text.strip() == ""`` — the empty text layer (a scanned/image-
|
||||
only PDF has nothing to read);
|
||||
* ≥ :data:`XMLISH_LINE_FRACTION` (0.30) of the non-empty lines
|
||||
match :data:`_XMLISH_LINE_RE` — the raw-XMP / binary-soup
|
||||
signature (the FINDING's reproduced pre-126 read is 46%);
|
||||
* ≥ :data:`NONPRINT_CHAR_FRACTION` (0.05) of all characters are
|
||||
non-printable (U+FFFD replacement chars + C0 controls) — the
|
||||
``errors="replace"`` binary-soup signal.
|
||||
|
||||
Pure + deterministic (no PDF, no I/O) — the unit tests pin the
|
||||
thresholds against the REAL reproduced garbage string, not a
|
||||
hand-drawn one. A real document ABOUT xml whose tag-like lines
|
||||
reach 30% is a documented false positive (ASSUMPTION A8 — the
|
||||
fallback still extracts it correctly via vision; the cost is a
|
||||
few vision calls).
|
||||
"""
|
||||
```
|
||||
with the two threshold constants (`XMLISH_LINE_FRACTION = 0.30`, `NONPRINT_CHAR_FRACTION = 0.05`) — named, documented, not magic. Non-empty-line counting: `line.strip()` truthy (house convention); the non-printable set: `"\ufffd"` + `unicodedata.category(ch).startswith("C")` (or an explicit C0 range check — pick one, document it; exclude the structural `\\n`/`\\t` the extractor legitimately emits — Cc category would catch them, so use the explicit set: `"\ufffd"` + control chars EXCEPT `\\t\\n\\r` — document the exclusion).
|
||||
2. `tests/unit/test_pdf.py` (extend — the phase-126 suite) — the gate's pins:
|
||||
- clean text (a few real paragraphs + the `---PAGE-N---` separators) → `True`;
|
||||
- THE REAL GARBAGE: reproduce the pre-126 read in the test — read `test_sources/hnb_2026_guide.tar.gz`'s PDF (extract to `tmp_path` — share a module-scoped fixture with the integration test's pattern, or a smaller inline reproduction: the test may instead read the tar.gz's PDF bytes with `read_text(errors="replace").replace("\\x00", "")` and pin `is_usable_pdf_text(garbage) is False` — the REAL string, the owner's exact file, the airtight pin);
|
||||
- the empty text layer (`""` / whitespace) → `False`;
|
||||
- an XML-heavy layer (≥ 30% tag-like lines) → `False`;
|
||||
- a borderline document ABOUT xml (29% tag-like lines, printable) → `True` (the threshold's edge, from below) and 31% → `False` (from above);
|
||||
- the U+FFFD-heavy soup (5%+ replacement chars, few tags) → `False` (the second signal fires alone);
|
||||
- the structural `\\n`/`\\t` exclusion (text full of newlines/indents is printable).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above — this task's suite.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `is_usable_pdf_text` is pure (no PDF dependency) and pinned on the REAL reproduced garbage string (the owner's file, not a lookalike)
|
||||
- [ ] both failure signals (XML-ish fraction, non-printable fraction) are tested independently
|
||||
- [ ] the thresholds are named constants with the false-positive cost documented
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -1,51 +0,0 @@
|
||||
# Task 03 — The screenshot fallback: unusable text layer → render pages, read each by vision
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "...should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working."
|
||||
|
||||
## Objective
|
||||
When the gate (task 02) says the text layer is garbage or empty, `_index_pdf_file` takes the owner's specified path: every page is rendered to a PNG screenshot and read by the vision model — the stored content is the page-anchored descriptions, so tables/images/whatever the text layer lost are preserved as the model saw them on the page.
|
||||
|
||||
## Work
|
||||
1. `app/rag/pdf.py` — new pure renderer:
|
||||
```python
|
||||
def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
|
||||
"""Render every page of *pdf_path* to a PNG (page order, 1-based
|
||||
meaning — list index i is page i+1).
|
||||
|
||||
The fallback's visual input (ASSUMPTION A8: fixed 150 DPI — the
|
||||
balance of vision legibility vs data-URL payload; a page that
|
||||
cannot be rendered yields b"" (the caller's placeholder — the
|
||||
page index NEVER shifts, so ---PAGE-N--- stays truthful).
|
||||
A file that cannot be OPENED yields [] (the caller's existing
|
||||
empty-content path — the doc lands with the no-extractable-text
|
||||
behavior, unchanged).
|
||||
"""
|
||||
```
|
||||
PyMuPDF: `page.get_pixmap(dpi=dpi).tobytes("png")` per page, per-page try/except (the `extract_pdf_text` fail-soft style — log a warning per skipped page).
|
||||
2. `app/rag/importer.py` — `_index_pdf_file`: after `content = extract_pdf_text(full_path)`, insert the gate branch (BEFORE the doc-row upsert, so the digest/chunk/summary pipeline runs on the FINAL content exactly once):
|
||||
```python
|
||||
if not is_usable_pdf_text(content):
|
||||
summary.pdfs_screenshotted += 1
|
||||
content = await _fallback_page_screenshots(session, source=source, rel=rel,
|
||||
full_path=full_path, llm=llm)
|
||||
```
|
||||
with the new module-level helper `_fallback_page_screenshots` (place it near `_index_pdf_file`):
|
||||
- `pages = render_pdf_pages(full_path)`;
|
||||
- the loud one-line log (PLAN §9, the FINDING's signature): `logger.warning("pdf: text layer unusable for %s/%s — falling back to %d page screenshots", source, rel, len(pages))` (the gate's fractions ride the log at DEBUG: `logger.debug("pdf: gate xml_frac=%.2f nonprint=%.2f …")` — computed by a small `_gate_fractions(text)` helper the unit tests pin too, if the warning wants them — optional, keep the required line exact);
|
||||
- per page (index i, page = i+1): `png = pages[i]`; if `png == b""` → `block_text = f"page {page} — description unavailable"`; else `desc = await describe_image(llm, data=png, mime="image/png")` (the phase-122 call — `settings` default, `context` None); `desc is None` or blank → the same placeholder (fail-soft per page — a dead vision endpoint must not sink the doc, the phase-122 fail-soft spirit); else `block_text = desc.strip()`;
|
||||
- `content = "\n\n".join(f"{PAGE_SEPARATOR.format(page=p)}\n{block}" for p, block in ...)` — the SAME separator convention as `extract_pdf_text` (the chunker's page-boundary handling works unchanged);
|
||||
- an ALL-failed render (`pages == []` or every page placeholder) → return `""` (the existing empty-content path — the doc lands as a zero-text doc, the phase-126 no-extractable-text behavior);
|
||||
- the embedded-image extraction (`extract_pdf_images` + `_describe_or_skip`) still runs AFTER the gate branch on BOTH paths — the fallback doc's embedded images join the pipeline exactly as a clean PDF's do (the screenshots are the CONTENT; the extracted images are the citation figures — both, no conflict).
|
||||
3. `app/rag/importer.py` — `ImportSummary`: add `pdfs_screenshotted: int = 0` (documented: "PDFs whose text layer failed the gate and took the page-screenshot vision fallback (phase 133)") + the field in `log()`'s format string (append to the existing line — the house ample-logging shape).
|
||||
4. `app/rag/importer.py` — the digest: the fallback's constructed content is what the sha256 runs over (the existing `digest = hashlib.sha256(content.encode(...))` line — no change needed, just verify it runs AFTER the gate branch so a fallback doc's row is stable across re-syncs: same pages + same vision model → same descriptions → same digest → unchanged on the next sync, no vision re-burn. If `describe_image`'s reply is non-deterministic in production, note it in the docstring — the re-burn is then a cost, not a bug).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_pdf.py` (extend) — `render_pdf_pages`: a 2-page temp PDF (built with fitz in the test — the phase-126 fixture pattern) → 2 PNGs (magic bytes `\\x89PNG`), page order; an unreadable path → `[]`; a corrupted page (if constructible — else the b"" path via a monkeypatched `get_pixmap` raising) → the zero-byte entry at its index, the index intact.
|
||||
- Unit: `tests/unit/test_importer.py` (extend) — the fallback branch, with a REAL tiny garbage PDF (build one in the test: a fitz page whose inserted text is ONLY XML-ish lines — `<foo:bar>x</foo:bar>` repeated — so the gate fails deterministically) + a fake LLM whose multimodal `chat` returns a canned page description: the stored doc's content = `---PAGE-1---\\n<canned>` (the separator convention), `summary.pdfs_screenshotted == 1`, the loud warning logged (the caplog pin); a CLEAN tiny PDF (the phase-126 fixtures) → `pdfs_screenshotted == 0` and the vision multimodal `chat` is NEVER called for the fallback (only the embedded-image describes — the gate never burns vision on a clean layer).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a garbage-text PDF lands with the page-screenshot content (the synthetic test: `---PAGE-N---` + the canned descriptions)
|
||||
- [ ] a clean PDF's path is byte-identical to phase 126 (no fallback, no extra vision calls, `pdfs_screenshotted == 0`)
|
||||
- [ ] the fallback is fail-soft per page (a dead vision call → the placeholder line, the doc still lands)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -1,29 +0,0 @@
|
||||
# Task 04 — E2E: the real guide stays clean, a synthetic garbage PDF takes the screenshot path
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "...the raw xml is being embedded and the preview shows the entire document as lines like `<illustrator:StartupProfile>Web</illustrator:StartupProfile>`. This project should support this kind of PDF and should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image..."
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) pins both legs of the guarantee at the user surface: (a) the owner's REAL guide, imported through the admin sync, previews CLEAN text (no XML lines — the exact complaint, negated); (b) a synthetic PDF whose text layer is pure XML garbage takes the screenshot fallback end to end — its content is the mock's canned vision descriptions, and the preview shows them.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_garbage_pdf.py` (new) — house pattern (module app fixture over the shared E2E Postgres, host temp-dir `kind=local` source registered via the admin git-sources API, autouse truncate cleanup, mock-only — `E2E_REAL_LLM=1` unsupported: the fallback's vision surface is the deterministic `IMAGE_DESCRIPTION_MODE` canned reply).
|
||||
App env: `BOR_IMPORT_EXTENSIONS` must include `pdf` (the owner's live config — the app fixture's env, the phase-126/127 suites' pattern) and `BOR_IMAGES=1` (the embedded-image pipeline is live).
|
||||
2. Fixtures (two local sources, two tests):
|
||||
- `hnb/` — the REAL guide: extract `test_sources/hnb_2026_guide.tar.gz` into the temp dir (module-scoped, once — the 6 MB extraction is seconds).
|
||||
- `garbagedoc/` — a synthetic garbage PDF built IN the test with PyMuPDF (the fixture pattern from task 03's unit test, lifted here): 2 pages, each page's ONLY text = repeated XML-ish lines (`<illustrator:StartupProfile>Web</illustrator:StartupProfile>` × ~20 — the owner's exact line shape, so the test is a faithful reproduction of the report), plus one small embedded raster image per page (fitz `insert_image` from a tiny PNG built in the test — the fallback's embedded-image leg runs too).
|
||||
3. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_real_guide_previews_clean_text` — admin sync over `hnb/` settles `success`; `GET /api/documents/content?source=hnb&path=<the guide's pdf>` (the PREVIEW surface — the endpoint the document viewer's content uses): the body contains `"2026 U.S. Benefits Guide"` (the real first-page text) and NO line matching the XML-ish regex (the owner's complaint, asserted on the served content, not the DB); the document viewer page (`/document.html?source=hnb&path=…`) renders that content (the preview shows real text, not `<illustrator:…>` lines).
|
||||
- `test_garbage_pdf_takes_the_screenshot_path` — admin sync over `garbagedoc/` settles `success`; the stored doc's content (the same content endpoint): starts with `---PAGE-1---`, contains the mock's CANNED vision description (the `IMAGE_DESCRIPTION_MODE` reply's deterministic text — assert the first 40 chars) for BOTH pages (2 separators, 2 descriptions), and contains NO `<illustrator:StartupProfile>` line (the garbage text layer is NOT what got stored — the screenshot path replaced it); the sync's `detail` is `success` with `added == 1` (the garbage PDF is a normal import — the fallback is invisible in the status shape; the loud log line is server-side, PLAN §9).
|
||||
- `test_clean_pdf_never_burns_the_fallback` — a third source `cleandoc/` with ONE small clean PDF (fitz, real prose text): sync → the stored content is the PROSE (the text layer, byte-faithful to the extraction) — NOT the canned vision description (the fallback did not fire on a clean layer — the gate's no-false-negative pin at the E2E level).
|
||||
4. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`), the mock-only gate + why (the canned vision reply makes the fallback deterministic), the fixture provenance (the real tar.gz + the synthetic builder), the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% `app/` bar is carried by tasks 02–03; the real-guide regression by task 01; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the real guide's PREVIEW (the owner's exact surface) is clean — the `<illustrator:StartupProfile>` line is asserted ABSENT on the served content
|
||||
- [ ] the synthetic garbage PDF's stored content is the canned vision descriptions (the fallback fired end to end, mock-deterministic)
|
||||
- [ ] the clean PDF is byte-faithful to its text layer (no false-positive fallback)
|
||||
- [ ] no behavior change in completed phases (the phase-126 pipeline, the sync, the document viewer)
|
||||
Reference in New Issue
Block a user