phase: 133_pdf_screenshot_fallback
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).
## Phase 133 final verification report
- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.
This commit is contained in:
@@ -636,13 +636,18 @@ def test_pdf_without_embedded_images_and_unchanged_rerun(db, tmp_path) -> None:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
"""A PDF with no extractable text (image-only, ``images`` OFF — the
|
||||
embedded image is invisible) → the document is still created, with
|
||||
EMPTY content (the empty byte string's digest), NO content chunks
|
||||
and NO content embedding (there is no text to embed — the only
|
||||
chunk is the best-effort ``lite`` summary's ``is_summary`` row), and
|
||||
the warning line is the signal."""
|
||||
def test_pdf_without_text_takes_the_screenshot_fallback(db, tmp_path) -> None:
|
||||
"""Phase 133 (task 03): a PDF with no extractable text layer
|
||||
(image-only, ``images`` OFF — the embedded image is invisible to
|
||||
the phase-126 A3 pass, but the FALLBACK is not gated by the
|
||||
toggle) → the gate's empty-layer target fires: the single page is
|
||||
rendered to a PNG and vision-described (the mock's fixed
|
||||
:data:`DESCRIPTION`), and the document lands with the
|
||||
``---PAGE-N---``-anchored description as content (one page-anchored
|
||||
content chunk + the ``lite`` summary chunk, both embedded). The
|
||||
phase-126 empty-content behavior is retired for layers that CAN be
|
||||
screenshotted — it survives only for an unopenable file, where the
|
||||
render fails too (the unit suite pins it)."""
|
||||
source_root = tmp_path / "pdfemptykb"
|
||||
source_root.mkdir()
|
||||
_build_pdf(source_root / "pics.pdf", [("", [_png_bytes(40, 30)])])
|
||||
@@ -650,6 +655,7 @@ def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
@@ -657,18 +663,19 @@ def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
)
|
||||
)
|
||||
assert doc is not None and doc.is_image is False
|
||||
assert doc.content == ""
|
||||
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
|
||||
assert doc.content == f"---PAGE-1---\n{DESCRIPTION}"
|
||||
assert doc.content_hash == hashlib.sha256(doc.content.encode("utf-8")).hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c for c in chunks if not c.is_summary] == [], (
|
||||
"no text → no content chunks (and no content embedding)"
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [f"Page 1:\n{DESCRIPTION}"], (
|
||||
"the page-anchored fallback content is chunked like any PDF"
|
||||
)
|
||||
# The phase-30/118 path still runs best-effort on the empty
|
||||
# content — the summary is the only chunk, and it is embedded.
|
||||
assert len(chunks) == 1 and chunks[0].is_summary
|
||||
assert chunks[0].embedding is not None and len(chunks[0].embedding) == 768
|
||||
# The phase-30/118 path runs on the fallback content — the
|
||||
# summary chunk mirrors the stored summary, both embedded.
|
||||
assert len(chunks) == 2
|
||||
assert all(c.embedding is not None and len(c.embedding) == 768 for c in chunks)
|
||||
assert doc.summary is not None
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
Reference in New Issue
Block a user