phase: 133_pdf_screenshot_fallback

All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).

## Phase 133 final verification report

- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
  1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
  2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
  3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
  4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
  5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
  6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.

app/rag/importer.py

@@ -109,6 +109,26 @@ any existing image docs; an unsupported image mime (outside
 ``BOR_IMAGE_EXTENSIONS``) is skipped with a warning. A PDF's viewer
 presentation is its extracted text (A4 — nothing binary is served).
 
+Text-layer garbage gate + page-screenshot fallback (phase 133,
+task 03): AFTER extraction and BEFORE the digest/chunk/summary
+pipeline, the extracted text passes the deterministic garbage gate
+(:func:`app.rag.pdf.is_usable_pdf_text` — empty text layer or
+XML/binary soup, the pre-126 raw-bytes class). On failure the content
+becomes the page-screenshot fallback (:func:
+`_fallback_page_screenshots` — every page rendered to a PNG via
+:func:`app.rag.pdf.render_pdf_pages` and read by the phase-122 vision
+call, the SAME ``---PAGE-N---`` separators, the loud
+``pdf: text layer unusable for …`` warning, the
+``pdfs_screenshotted`` counter) — the owner's specified path for the
+class of PDF whose text layer is garbage. A CLEAN text layer NEVER
+burns a fallback vision call (the gate is the only gate — the
+phase-126 happy path is byte-identical), and the digest runs over the
+FINAL content exactly once — for a stable file + stable vision model
+the digest is byte-stable, so an unchanged re-sync re-imports the row
+as unchanged (no re-chunk/re-embed/re-summary, no KB change; the
+vision calls are re-issued each sync — the no-cache cost, documented
+in :func:`_fallback_page_screenshots`).
+
 Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
 description, the same source's importable markdown files are scanned for
 ``![alt](path)`` references that name the image (the task-02
@@ -173,7 +193,14 @@ from app.rag.chunker import chunk_document, extract_title
 from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
 from app.rag.llm import EmbeddingError, LLMError
 from app.rag.md_images import ImageRef, extract_image_refs
-from app.rag.pdf import PdfImage, extract_pdf_images, extract_pdf_text
+from app.rag.pdf import (
+    PAGE_SEPARATOR,
+    PdfImage,
+    extract_pdf_images,
+    extract_pdf_text,
+    is_usable_pdf_text,
+    render_pdf_pages,
+)
 from app.rag.summarizer import (
     IMAGE_FALLBACK_MIME,
     IMAGE_MIMES,
@@ -254,6 +281,14 @@ class ImportSummary:
     #: Fail-soft: the sync continues, this counter + the warning line
     #: are the signal.
     images_failed: int = 0
+    #: PDFs (phase 133, task 03) whose text layer failed the garbage
+    #: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
+    #: XML/binary soup) and took the page-screenshot vision fallback
+    #: (:func:`_fallback_page_screenshots` — every page rendered to a
+    #: PNG and read by the phase-122 vision call; the content is the
+    #: ``---PAGE-N---``-anchored descriptions, or ``""`` when the
+    #: fallback produced nothing — the phase-126 empty-content path).
+    pdfs_screenshotted: int = 0
     #: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
     #: tuples in encounter order (added/updated in walk order, pruned in
     #: _prune order). Drives the sync's incremental folder-summary
@@ -274,7 +309,8 @@ class ImportSummary:
         logger.info(
             "import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
             "errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
-            "summary_backfilled=%d dates_updated=%d images_failed=%d formats=%s",
+            "summary_backfilled=%d dates_updated=%d images_failed=%d "
+            "pdfs_screenshotted=%d formats=%s",
             self.files,
             self.added,
             self.updated,
@@ -288,6 +324,7 @@ class ImportSummary:
             self.summary_backfilled,
             self.dates_updated,
             self.images_failed,
+            self.pdfs_screenshotted,
             self.format_counts(),
         )
 
@@ -1034,12 +1071,24 @@ async def _index_pdf_file(
     * ``content`` is the parser's text — every text-bearing page
       prefixed with the ``---PAGE-N---`` marker (A2 — one document per
       PDF, the markers are the page anchors the page-aware chunker
-      (task 03) splits on); an unopenable/empty PDF yields ``""``
-      (still a document — it may carry embedded images — with no
-      content chunks; a warning is the signal).
-    * the sha256 digest is over the extracted text (the digest rule is
+      (task 03) splits on). Phase 133 (task 03): if that text fails
+      the garbage gate (:func:`is_usable_pdf_text` — empty layer or
+      XML/binary soup), the content becomes the page-screenshot
+      fallback (:func:`_fallback_page_screenshots` — the rendered
+      pages read by the vision model, the same separators) BEFORE the
+      digest/chunk/summary pipeline runs; the gate never fires for a
+      clean layer (no fallback vision calls on the happy path).
+      An unopenable PDF — or one whose fallback also produced nothing
+      — yields ``""`` (still a document — it may carry embedded images
+      — with no content chunks; a warning is the signal).
+    * the sha256 digest is over the FINAL content (the digest rule is
       content identity — the same text is the same document version;
-      the empty case digests the empty byte string).
+      the fallback's constructed page descriptions digest exactly the
+      same way, so an unchanged fallback doc re-syncs as unchanged —
+      no re-chunk/re-embed/re-summary, no KB change — while the vision
+      model's replies stay stable; a non-deterministic vision model
+      re-describes the row each sync, a cost, not a bug; the empty
+      case digests the empty byte string).
     * the title is the metadata title or the file stem
       (:func:`_extract_pdf_title`); the date follows the text path
       exactly (the phase-106 D2 mtime fallback — PDFs carry no
@@ -1071,12 +1120,22 @@ async def _index_pdf_file(
     # Extract the text (fail-soft: an unopenable PDF yields "" — the
     # document may still carry embedded images worth indexing).
     content = extract_pdf_text(full_path)
+    # Phase 133 (task 03): the garbage gate — BEFORE the digest/chunk/
+    # summary pipeline, so it runs on the FINAL content exactly once.
+    # A clean layer never reaches the fallback (the gate is the only
+    # gate — no fallback vision calls on the happy path).
+    if not is_usable_pdf_text(content):
+        summary.pdfs_screenshotted += 1
+        content = await _fallback_page_screenshots(
+            session, source=source, rel=rel, full_path=full_path, llm=llm
+        )
     if not content:
         logger.warning(
             "import: PDF has no extractable text source=%s path=%s", source, rel
         )
-    # Content identity: the digest is over the extracted text (the
-    # empty case is the empty byte string's digest).
+    # Content identity: the digest is over the FINAL content — the
+    # extracted text, or the fallback's constructed page descriptions
+    # (the empty case is the empty byte string's digest).
     digest = (
         hashlib.sha256(content.encode("utf-8")).hexdigest()
         if content
@@ -1174,13 +1233,95 @@ async def _index_pdf_file(
     # Phase 126 (A3): the embedded images join the phase-122 pipeline
     # (gated by the ``images`` toggle — skipped, debug-logged, while it
     # is off). Runs after the document commit, so a per-image failure
-    # never loses the PDF's text index.
+    # never loses the PDF's text index. Runs on BOTH the clean and the
+    # fallback content (phase 133: the screenshots are the CONTENT; the
+    # extracted images are the citation figures — both, no conflict).
     await _index_pdf_embedded_images(
         session, source=source, rel=rel, full_path=full_path,
         llm=llm, summary=summary, raw_date=raw_date,
     )
 
 
+async def _fallback_page_screenshots(
+    session: Session,
+    *,
+    source: str,
+    rel: str,
+    full_path: Path,
+    llm: Embedder,
+) -> str:
+    """The phase-133 (task 03) page-screenshot fallback — the owner's
+    specified path ("screenshotting the pages and reading each one as
+    an image") for a PDF whose text layer failed
+    :func:`is_usable_pdf_text` (XML/binary garbage, or an empty text
+    layer): every page is rendered to a PNG screenshot and read by the
+    vision model, so tables/images/whatever the text layer lost are
+    preserved as the model saw them on the page.
+
+    * ``content`` is the :data:`PAGE_SEPARATOR`-anchored page
+      descriptions (``---PAGE-N---\n<description>`` — the SAME
+      separator convention as :func:`extract_pdf_text`, so the
+      page-aware chunker's page-boundary handling works unchanged);
+      the doc stays a PDF document (``is_image`` untouched — it is a
+      PDF whose content happens to be vision text).
+    * one vision call per page (the phase-122
+      :func:`describe_image` — ONE CHAT-model call, the PNG bytes as a
+      data URL, ``context`` None); FAIL-SOFT PER PAGE: a page that
+      could not be rendered (``b""`` from :func:`render_pdf_pages` —
+      the index never shifts) or whose vision call failed/came back
+      blank gets the deterministic placeholder
+      ``page N — description unavailable`` (a dead vision endpoint
+      must not sink the doc — the phase-122 fail-soft spirit).
+    * an ALL-failed render (no pages rendered — an unopenable file —
+      or every page a placeholder) returns ``""``: the existing
+      empty-content path (the doc lands as a zero-text doc, the
+      phase-126 no-extractable-text behavior).
+    * the digest the caller computes is over the constructed content —
+      for a stable file + stable vision model the content (and thus
+      the digest) is byte-stable across re-syncs, so the row
+      re-imports as UNCHANGED: no re-chunk, no re-embed, no
+      re-summary, no KB change. The fallback itself RE-RUNS on every
+      sync of a garbage PDF (the gate is a cheap pure predicate and
+      the render needs no LLM; the vision calls are re-issued before
+      the unchanged check can skip them — a documented cost of the
+      no-cache design, bounded by the page count of the garbage
+      PDFs). If the production vision model's reply is
+      non-deterministic, the row is re-described each sync (updated —
+      a cost, not a bug; the doc is still correct).
+
+    *session* is part of the signature for symmetry with the other
+    per-file index helpers (the fallback itself needs no DB access —
+    the caller's upsert pipeline owns the row).
+    """
+    pages = render_pdf_pages(full_path)
+    # The loud one-line log (PLAN §9, the FINDING's signature).
+    logger.warning(
+        "pdf: text layer unusable for %s/%s — falling back to %d page screenshots",
+        source, rel, len(pages),
+    )
+    if not pages:
+        return ""
+    blocks: list[str] = []
+    got_any = False
+    for i, png in enumerate(pages):
+        page = i + 1
+        if not png:
+            blocks.append(f"page {page} — description unavailable")
+            continue
+        desc = await describe_image(llm, data=png, mime="image/png")
+        if desc is None or not desc.strip():
+            blocks.append(f"page {page} — description unavailable")
+        else:
+            blocks.append(desc.strip())
+            got_any = True
+    if not got_any:
+        return ""
+    return "\n\n".join(
+        f"{PAGE_SEPARATOR.format(page=page)}\n{block}"
+        for page, block in enumerate(blocks, start=1)
+    )
+
+
 async def _index_pdf_embedded_images(
     session: Session,
     *,

app/rag/pdf.py

@@ -18,6 +18,16 @@ package, imported as ``fitz`` — its stable API name).
   MIME type: the embedded file extension (``ext``) first, then the
   image type code (``type``: 2=JPEG, 3=PNG, 4=GIF, 8=TIFF, 9=WebP),
   else the ``application/octet-stream`` fallback.
+* :func:`is_usable_pdf_text` — the phase-133 garbage gate: a pure,
+  deterministic predicate over the extracted text layer (no PDF, no
+  I/O) — REAL content vs. garbage (the pre-126 XML/binary soup, the
+  empty text layer). It is the single trigger for the page-screenshot
+  vision fallback (phase 133 task 03).
+* :func:`render_pdf_pages` — the fallback's visual input (phase 133
+  task 03): one PNG per page (150 DPI, page order); an unreadable page
+  yields ``b""`` (the caller's placeholder — the page index never
+  shifts), an unopenable file yields ``[]`` (the caller's existing
+  empty-content path).
 
 Every failure mode is fail-soft: an unreadable file returns an empty
 result, an unreadable page or image is logged and skipped — this module
@@ -26,6 +36,7 @@ never raises for malformed input (the importer owns the loud failures).
 from __future__ import annotations
 
 import logging
+import re
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
@@ -142,6 +153,101 @@ def extract_pdf_text(pdf_path: Path) -> str:
     return "\n\n".join(parts)
 
 
+#: A line "looks like XML" when it carries a tag-shaped token — the
+#: pre-126 garbage's signature (raw XMP blocks, PDF-structure noise).
+_XMLISH_LINE_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
+
+#: Garbage-gate thresholds (phase 133 task 02): an extracted text
+#: layer is UNUSABLE when its XML-ish line fraction reaches this bar.
+#: The FINDING's reproduced pre-126 read of the owner's file is 46%
+#: XML-ish; a real document ABOUT XML that reaches the bar is a
+#: documented false positive (ASSUMPTION A8 — the fallback still
+#: extracts it via vision; the cost is a few vision calls).
+XMLISH_LINE_FRACTION = 0.30
+
+#: Garbage-gate thresholds (phase 133 task 02): the non-printable
+#: fraction (U+FFFD replacement chars + C0 controls over the whole
+#: text) at which the layer is UNUSABLE — the ``errors="replace"``
+#: binary-soup signal (the pre-126 read is ~19% non-printable; real
+#: extracted text is ~0.05%).
+NONPRINT_CHAR_FRACTION = 0.05
+
+
+def is_usable_pdf_text(text: str) -> bool:
+    """Whether an extracted PDF text layer is REAL content (not garbage).
+
+    Unusable (→ the page-screenshot fallback) when ANY of:
+
+    * ``text.strip() == ""`` — the empty text layer (a scanned/image-
+      only PDF has nothing to read);
+    * ≥ :data:`XMLISH_LINE_FRACTION` (0.30) of the non-empty lines
+      match :data:`_XMLISH_LINE_RE` — the raw-XMP / binary-soup
+      signature (the FINDING's reproduced pre-126 read is 46% —
+      ``read_text(errors="replace")`` on the owner's file left
+      ``<illustrator:StartupProfile>Web</illustrator:StartupProfile>``
+      lines in the preview);
+    * ≥ :data:`NONPRINT_CHAR_FRACTION` (0.05) of all characters are
+      non-printable — the ``errors="replace"`` binary-soup signal.
+      The non-printable set is ``U+FFFD`` + C0 control chars
+      (0x00–0x1F) EXCEPT the structural ``\\t``/``\\n``/``\\r`` the
+      extractor legitimately emits (the explicit set is deliberate: a
+      ``unicodedata`` Cc-category check would catch those too, and
+      counting them would flag every prose document).
+
+    Pure + deterministic (no PDF, no I/O) — the unit tests pin the
+    thresholds against the REAL reproduced garbage string, not a
+    hand-drawn one. A real document ABOUT XML whose tag-like lines
+    reach 30% is a documented false positive (ASSUMPTION A8 — the
+    fallback still extracts it correctly via vision; the cost is a
+    few vision calls).
+    """
+    if not text.strip():
+        return False
+    non_empty_lines = [line for line in text.splitlines() if line.strip()]
+    xml_ish = sum(1 for line in non_empty_lines if _XMLISH_LINE_RE.search(line))
+    if xml_ish / len(non_empty_lines) >= XMLISH_LINE_FRACTION:
+        return False
+    non_printable = sum(
+        1
+        for ch in text
+        if ch == "\ufffd" or (ord(ch) < 32 and ch not in "\t\n\r")
+    )
+    return non_printable / len(text) < NONPRINT_CHAR_FRACTION
+
+
+def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
+    """Render every page of *pdf_path* to a PNG (page order — list index
+    *i* is page *i*+1).
+
+    The fallback's visual input (phase 133 task 03, ASSUMPTION A8:
+    fixed 150 DPI — the balance of vision legibility vs data-URL
+    payload). A page that cannot be rendered yields ``b""`` (the
+    caller's placeholder — the page index NEVER shifts, so
+    ``---PAGE-N---`` stays truthful; the failure is logged, the
+    :func:`extract_pdf_text` fail-soft style). A file that cannot be
+    OPENED yields ``[]`` (the caller's existing empty-content path —
+    the doc lands with the no-extractable-text behavior, unchanged).
+    """
+    try:
+        doc = fitz.open(str(pdf_path))
+    except Exception as exc:
+        logger.warning("pdf: could not open %s for page rendering — %s", pdf_path, exc)
+        return []
+
+    pages: list[bytes] = []
+    with doc:
+        for page_num in range(len(doc)):
+            try:
+                pages.append(doc[page_num].get_pixmap(dpi=dpi).tobytes("png"))
+            except Exception as exc:
+                logger.warning(
+                    "pdf: could not render page %d of %s — %s",
+                    page_num + 1, pdf_path, exc,
+                )
+                pages.append(b"")
+    return pages
+
+
 def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
     """Extract all embedded images from *pdf_path*.