Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 133_pdf_screenshot_fallback
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).

## Phase 133 final verification report

- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
  1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
  2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
  3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
  4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
  5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
  6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.
This commit is contained in:
ducoterra
2026-09-28 15:39:52 -04:00
parent 4cb85e5baa
commit 25de518dc9
28 changed files with 2701 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/133_pdf_screenshot_fallback/00_phase.md → .agents/phases/complete/133_pdf_screenshot_fallback/00_phase.md
View File
.agents/phases/todo/133_pdf_screenshot_fallback/01_diagnosis_regression_test.md → .agents/phases/complete/133_pdf_screenshot_fallback/01_diagnosis_regression_test.md
View File
.agents/phases/todo/133_pdf_screenshot_fallback/02_pdf_text_gate.md → .agents/phases/complete/133_pdf_screenshot_fallback/02_pdf_text_gate.md
View File
.agents/phases/todo/133_pdf_screenshot_fallback/03_page_screenshot_fallback.md → .agents/phases/complete/133_pdf_screenshot_fallback/03_page_screenshot_fallback.md
View File
.agents/phases/todo/133_pdf_screenshot_fallback/04_e2e_garbage_pdf.md → .agents/phases/complete/133_pdf_screenshot_fallback/04_e2e_garbage_pdf.md
View File
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__00_phase.a1.err
View File
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__00_phase.a1.md
+14
View File
@@ -0,0 +1,14 @@
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).
## Phase 133 final verification report
- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__00_phase.a1.validate
+115
View File
@@ -0,0 +1,115 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 595 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 109 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 410 0 100%
--------------------------------------------------
TOTAL 5258 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__01_diagnosis_regression_test.a1.err
View File
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__01_diagnosis_regression_test.a1.md
+19
View File
@@ -0,0 +1,19 @@
Task 01 complete — all gates green on the final state.
**Implemented**
- `tests/integration/test_pdf_hnb_guide_regression.py` — pins the owner's exact `test_sources/hnb_2026_guide.tar.gz` 48-page Illustrator PDF through the REAL importer: clean-content bars (XML-ish < 0.05, non-printable < 0.01), pinned first-page text + `---PAGE-1---` + stem title, 91 embedded chunks + lite summary, all 27 embedded-image docs got the fake vision reply (multimodal `chat` ≥ 1, locked bare `DESCRIBE_PROMPT`), 120 s wall-time guard; FINDING + run command + provenance in the docstring; fixture extracted to `tmp_path` only.
**Results**
- `uv run pytest --cov=app --cov-report=term-missing` → 3109 passed, `app/` coverage 99% (>90%)
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors
- `uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v --no-cov` → 1 passed (0.71 s)
- Pin has teeth: pre-126 text-branch recipe on the file measures 46.2% XML-ish / 18.6% non-printable — both bars break
**Docs:** no doc impact (test-only task, no user-visible change)
**Decisions** (deviations from the task's literal text, documented in the test docstring)
- `summary.added == 1 + len(image_docs)` instead of `== 1`: phase-126 A3 counts each embedded-image doc `added` (code + phase-126 test confirm); `summary.files == 1` pins the walk admitted the one file. The required multimodal assertion forces `images` on, which makes image docs added.
- `Settings(_env_file=".env", images=True, image_dir=<tmp>)`: conftest forces `BOR_IMAGES` off process-wide, so the plain form would silently skip the embedded-image pass; the kwarg restores the owner's live `BOR_IMAGES=1`, `image_dir` is hermetic per the house `_image_llm` pattern.
- Non-printable bar excludes `\n`/`\t`/`\r` (counting them flags every document; ~2.5% of the real guide is newlines).
**Next pending task:** `.agents/phases/todo/133_pdf_screenshot_fallback/02_pdf_text_gate.md`
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__01_diagnosis_regression_test.a1.validate
+115
View File
@@ -0,0 +1,115 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
............. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 410 0 100%
--------------------------------------------------
TOTAL 5206 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__02_pdf_text_gate.a1.err
View File
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__02_pdf_text_gate.a1.md
+16
View File
@@ -0,0 +1,16 @@
Task 02 complete — everything 100% green.
**Implemented**
- `app/rag/pdf.py`: new pure, deterministic gate `is_usable_pdf_text(text) -> bool` (placed after `extract_pdf_text`) — unusable when empty text layer, ≥ `XMLISH_LINE_FRACTION` (0.30) XML-ish non-empty lines, or ≥ `NONPRINT_CHAR_FRACTION` (0.05) non-printable chars (U+FFFD + C0 controls, explicitly excluding structural `\t`/`\n`/`\r` — documented). Constants named/documented; docstring cites the FINDING (pre-126 `errors="replace"` soup, 46% XML-ish, the owner's `<illustrator:StartupProfile>` preview, ASSUMPTION A8 false-positive cost).
- `tests/unit/test_pdf.py`: 8 new pins — clean text+separators → True; empty/whitespace → False; **the REAL pre-126 garbage** (module-scoped fixture reading the owner's tar.gz PDF with the exact old recipe, pins `is False` + both signals clearing their named bars + the `<illustrator:StartupProfile>` line present, skip-if-missing like task 01); 30% and 31% tag-like → False; 29% → True (A8 edge); U+FFFD ~7.9% soup with no tags → False (second signal alone); tab/newline-heavy text → True (exclusion).
**Results**
- `uv run pytest --cov=app --cov-report=term-missing` → **3117 passed**, 0 failed (69 s); `app/rag/pdf.py` 100%, `app/` TOTAL 99% (>90%)
- `uv run pytest tests/unit/test_pdf.py` → 64 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
**Docs:** no doc impact (internal predicate; not wired to the importer until task 03 — README/AGENTS.md stay accurate).
**Decisions:** explicit C0-range check (`ord(ch) < 32`, minus `\t\n\r`) over `unicodedata` Cc — matches the task-01 integration pin's `_nonprint_frac` convention; real-garbage fixture extracts the tar member to a module-scoped temp dir (never the repo).
**Next pending task:** `.agents/phases/todo/133_pdf_screenshot_fallback/03_page_screenshot_fallback.md`
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__02_pdf_text_gate.a1.validate
+115
View File
@@ -0,0 +1,115 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
..................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 94 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 410 0 100%
--------------------------------------------------
TOTAL 5219 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__03_page_screenshot_fallback.a1.err
View File
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__03_page_screenshot_fallback.a1.md
+21
View File
@@ -0,0 +1,21 @@
Everything is green and verified. Final report:
## Task 03 — page-screenshot fallback: complete
**Implemented**
- `app/rag/pdf.py`: `render_pdf_pages(pdf_path, dpi=150) -> list[bytes]` — one PNG per page in page order; unreadable page → `b""` at its index (index never shifts); unopenable file → `[]`.
- `app/rag/importer.py`: gate branch in `_index_pdf_file` after extraction (before digest/upsert) — `summary.pdfs_screenshotted += 1` + `_fallback_page_screenshots`; new helper renders pages, logs the exact loud line (`pdf: text layer unusable for <s>/<p> — falling back to N page screenshots`), vision-reads each page via phase-122 `describe_image` (fail-soft → `page N — description unavailable`), all-failed → `""` (empty-content path), `---PAGE-N---` separators; `ImportSummary.pdfs_screenshotted` + `log()` field.
- Stale phase-126 pins updated (image-only PDF now takes the fallback; corrupt file stays empty with the fallback counted); "no extractable text" warning now fires only when the doc truly lands empty.
**Tests**
- `uv run pytest --cov=app --cov-report=term-missing` → **3129 passed**, TOTAL **99%** (`app/rag/pdf.py` 100%, `importer.py` 98% — only pre-existing uncovered lines; all new code covered)
- `uv run ruff check . && uv run pyright` → clean
- Isolated E2E spot-checks: `test_pdf_support.py` 6 passed, `test_image_anchor_context.py` 4 passed, `test_import_documents.py` 3 passed; task-01 real-guide regression passed
**Docs**
- README: PDF section gained the phase-133 fallback bullet (`pdfs_screenshotted=N`); AGENTS.md — no doc impact.
**Notable decision**
- The mandated gate sits before the row lookup, so a garbage PDF's vision calls re-issue each sync before the digest check marks the row unchanged (the row itself re-imports unchanged — no re-chunk/embed/KB change). Eliminating the vision re-issue would need a schema change (out of scope); the cost is documented honestly in the docstrings instead of the overview's "no vision re-burn" phrasing. The task's optional `_gate_fractions` DEBUG line was skipped (required warning line kept exact).
**Next pending task:** `.agents/phases/todo/133_pdf_screenshot_fallback/04_e2e_garbage_pdf.md`
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__03_page_screenshot_fallback.a1.validate
+115
View File
@@ -0,0 +1,115 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 595 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 109 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 410 0 100%
--------------------------------------------------
TOTAL 5258 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__04_e2e_garbage_pdf.a1.err
View File
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__04_e2e_garbage_pdf.a1.md
+19
View File
@@ -0,0 +1,19 @@
All green. Task 04 is complete.
**Implemented**
- `tests/e2e/test_garbage_pdf.py` (new) — the phase-133 dedicated Playwright suite, house pattern: one module app (port 8149, module env with `BOR_IMPORT_EXTENSIONS` pinned incl. `pdf`, `BOR_IMAGES=1`, leak guards, scratch image/upload homes), `kind=local` host temp dirs registered via the admin git-sources API, autouse truncate with no-running-sync wait, mock-only.
- Fixtures: `hnb/` (real `hnb_2026_guide.tar.gz` extracted module-scoped, skips only this test if the untracked fixture is missing), `garbagedoc/` (2-page PyMuPDF garbage PDF — owner's exact `<illustrator:StartupProfile>Web</…>` line ×20/page + one 1×1 PNG embedded per page), `cleandoc/` (1-page prose PDF).
- Tests: (1) real guide syncs `success` — content endpoint carries "2026 U.S. Benefits Guide" with **zero** XML-ish lines, and `/document.html` renders that content (stem title, page markers, `<illustrator:…>` absent); (2) garbage PDF content is byte-exactly the two pages of the mock's canned `IMAGE_DESCRIPTION_MODE` description (first-40-char pin, count, exact shape), garbage text not stored, both `#page=N#img=0.png` image docs indexed, `detail` shape unchanged (`pdfs_screenshotted` asserted absent — log-only); (3) clean PDF byte-faithful to its text layer, no fallback vision text.
**Results (all commands)**
- `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` → **3 passed** (in isolation, DB up)
- `uv run pytest --cov=app --cov-report=term-missing` → **3129 passed**, `app/` coverage **99%** (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
**Docs**: no doc impact (test-only task; phase-133 behavior already documented in README by tasks 01–03).
**Notable decisions**
- Task mapping's `added == 1` for the garbage sync is superseded by the phase-126 A3 convention (documented in the task-01 pin's docstring): with the mandated embedded images + `BOR_IMAGES=1`, `added` counts the PDF + its 2 image docs → asserted `added == 1 + len(image_docs)` (verified live: 3), plus `files == 1` and the "invisible in the status shape" intent pinned via the absent `pdfs_screenshotted` key.
- Verified via an instrumented probe that the real-guide sync genuinely imports (48 pages, 27 image docs, clean content, `pdfs_screenshotted=0` in the log line) — the fast runtime is the mock, not skipped work.
**Next pending task**: none — this was the final task (04) of phase `133_pdf_screenshot_fallback`.
.agents/reports/133_pdf_screenshot_fallback/133_pdf_screenshot_fallback__04_e2e_garbage_pdf.a1.validate
+115
View File
@@ -0,0 +1,115 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 595 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 109 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 410 0 100%
--------------------------------------------------
TOTAL 5258 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
README.md
+10
View File
@@ -556,6 +556,16 @@ uploaded archives):
with the toggle off the pass is skipped and the PDF text indexes
normally. Embedded images in a format outside `BOR_IMAGE_EXTENSIONS`
are skipped with a warning.
- **Garbage text layer → page-screenshot fallback (phase 133).** Before
indexing, the extracted text passes a deterministic garbage gate
(empty text layer, or ≥ 30% tag-looking lines, or ≥ 5% non-printable
chars — the class of PDF that reads as raw XML/binary soup). On
failure every page is rendered to a screenshot and read by the vision
model; the stored content is the `---PAGE-N---`-anchored page
descriptions (a failed page gets a `page N — description unavailable`
placeholder — fail-soft, the doc still lands). Clean text layers
never burn a fallback vision call. The run counts the affected PDFs
on the import summary line (`pdfs_screenshotted=N`).
- **Served as text.** The document viewer renders the extracted text —
never the binary PDF.
app/rag/importer.py
+151 -10
View File
@@ -109,6 +109,26 @@ any existing image docs; an unsupported image mime (outside
``BOR_IMAGE_EXTENSIONS``) is skipped with a warning. A PDF's viewer
presentation is its extracted text (A4 — nothing binary is served).
Text-layer garbage gate + page-screenshot fallback (phase 133,
task 03): AFTER extraction and BEFORE the digest/chunk/summary
pipeline, the extracted text passes the deterministic garbage gate
(:func:`app.rag.pdf.is_usable_pdf_text` — empty text layer or
XML/binary soup, the pre-126 raw-bytes class). On failure the content
becomes the page-screenshot fallback (:func:
`_fallback_page_screenshots` — every page rendered to a PNG via
:func:`app.rag.pdf.render_pdf_pages` and read by the phase-122 vision
call, the SAME ``---PAGE-N---`` separators, the loud
``pdf: text layer unusable for …`` warning, the
``pdfs_screenshotted`` counter) — the owner's specified path for the
class of PDF whose text layer is garbage. A CLEAN text layer NEVER
burns a fallback vision call (the gate is the only gate — the
phase-126 happy path is byte-identical), and the digest runs over the
FINAL content exactly once — for a stable file + stable vision model
the digest is byte-stable, so an unchanged re-sync re-imports the row
as unchanged (no re-chunk/re-embed/re-summary, no KB change; the
vision calls are re-issued each sync — the no-cache cost, documented
in :func:`_fallback_page_screenshots`).
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
description, the same source's importable markdown files are scanned for
``![alt](path)`` references that name the image (the task-02
@@ -173,7 +193,14 @@ from app.rag.chunker import chunk_document, extract_title
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
from app.rag.llm import EmbeddingError, LLMError
from app.rag.md_images import ImageRef, extract_image_refs
from app.rag.pdf import PdfImage, extract_pdf_images, extract_pdf_text
from app.rag.pdf import (
PAGE_SEPARATOR,
PdfImage,
extract_pdf_images,
extract_pdf_text,
is_usable_pdf_text,
render_pdf_pages,
)
from app.rag.summarizer import (
IMAGE_FALLBACK_MIME,
IMAGE_MIMES,
@@ -254,6 +281,14 @@ class ImportSummary:
#: Fail-soft: the sync continues, this counter + the warning line
#: are the signal.
images_failed: int = 0
#: PDFs (phase 133, task 03) whose text layer failed the garbage
#: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
#: XML/binary soup) and took the page-screenshot vision fallback
#: (:func:`_fallback_page_screenshots` — every page rendered to a
#: PNG and read by the phase-122 vision call; the content is the
#: ``---PAGE-N---``-anchored descriptions, or ``""`` when the
#: fallback produced nothing — the phase-126 empty-content path).
pdfs_screenshotted: int = 0
#: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
#: tuples in encounter order (added/updated in walk order, pruned in
#: _prune order). Drives the sync's incremental folder-summary
@@ -274,7 +309,8 @@ class ImportSummary:
logger.info(
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"summary_backfilled=%d dates_updated=%d images_failed=%d formats=%s",
"summary_backfilled=%d dates_updated=%d images_failed=%d "
"pdfs_screenshotted=%d formats=%s",
self.files,
self.added,
self.updated,
@@ -288,6 +324,7 @@ class ImportSummary:
self.summary_backfilled,
self.dates_updated,
self.images_failed,
self.pdfs_screenshotted,
self.format_counts(),
)
@@ -1034,12 +1071,24 @@ async def _index_pdf_file(
* ``content`` is the parser's text — every text-bearing page
prefixed with the ``---PAGE-N---`` marker (A2 — one document per
PDF, the markers are the page anchors the page-aware chunker
(task 03) splits on); an unopenable/empty PDF yields ``""``
(still a document — it may carry embedded images — with no
content chunks; a warning is the signal).
* the sha256 digest is over the extracted text (the digest rule is
(task 03) splits on). Phase 133 (task 03): if that text fails
the garbage gate (:func:`is_usable_pdf_text` — empty layer or
XML/binary soup), the content becomes the page-screenshot
fallback (:func:`_fallback_page_screenshots` — the rendered
pages read by the vision model, the same separators) BEFORE the
digest/chunk/summary pipeline runs; the gate never fires for a
clean layer (no fallback vision calls on the happy path).
An unopenable PDF — or one whose fallback also produced nothing
— yields ``""`` (still a document — it may carry embedded images
— with no content chunks; a warning is the signal).
* the sha256 digest is over the FINAL content (the digest rule is
content identity — the same text is the same document version;
the empty case digests the empty byte string).
the fallback's constructed page descriptions digest exactly the
same way, so an unchanged fallback doc re-syncs as unchanged —
no re-chunk/re-embed/re-summary, no KB change — while the vision
model's replies stay stable; a non-deterministic vision model
re-describes the row each sync, a cost, not a bug; the empty
case digests the empty byte string).
* the title is the metadata title or the file stem
(:func:`_extract_pdf_title`); the date follows the text path
exactly (the phase-106 D2 mtime fallback — PDFs carry no
@@ -1071,12 +1120,22 @@ async def _index_pdf_file(
# Extract the text (fail-soft: an unopenable PDF yields "" — the
# document may still carry embedded images worth indexing).
content = extract_pdf_text(full_path)
# Phase 133 (task 03): the garbage gate — BEFORE the digest/chunk/
# summary pipeline, so it runs on the FINAL content exactly once.
# A clean layer never reaches the fallback (the gate is the only
# gate — no fallback vision calls on the happy path).
if not is_usable_pdf_text(content):
summary.pdfs_screenshotted += 1
content = await _fallback_page_screenshots(
session, source=source, rel=rel, full_path=full_path, llm=llm
)
if not content:
logger.warning(
"import: PDF has no extractable text source=%s path=%s", source, rel
)
# Content identity: the digest is over the extracted text (the
# empty case is the empty byte string's digest).
# Content identity: the digest is over the FINAL content — the
# extracted text, or the fallback's constructed page descriptions
# (the empty case is the empty byte string's digest).
digest = (
hashlib.sha256(content.encode("utf-8")).hexdigest()
if content
@@ -1174,13 +1233,95 @@ async def _index_pdf_file(
# Phase 126 (A3): the embedded images join the phase-122 pipeline
# (gated by the ``images`` toggle — skipped, debug-logged, while it
# is off). Runs after the document commit, so a per-image failure
# never loses the PDF's text index.
# never loses the PDF's text index. Runs on BOTH the clean and the
# fallback content (phase 133: the screenshots are the CONTENT; the
# extracted images are the citation figures — both, no conflict).
await _index_pdf_embedded_images(
session, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
)
async def _fallback_page_screenshots(
session: Session,
*,
source: str,
rel: str,
full_path: Path,
llm: Embedder,
) -> str:
"""The phase-133 (task 03) page-screenshot fallback — the owner's
specified path ("screenshotting the pages and reading each one as
an image") for a PDF whose text layer failed
:func:`is_usable_pdf_text` (XML/binary garbage, or an empty text
layer): every page is rendered to a PNG screenshot and read by the
vision model, so tables/images/whatever the text layer lost are
preserved as the model saw them on the page.
* ``content`` is the :data:`PAGE_SEPARATOR`-anchored page
descriptions (``---PAGE-N---\n<description>`` — the SAME
separator convention as :func:`extract_pdf_text`, so the
page-aware chunker's page-boundary handling works unchanged);
the doc stays a PDF document (``is_image`` untouched — it is a
PDF whose content happens to be vision text).
* one vision call per page (the phase-122
:func:`describe_image` — ONE CHAT-model call, the PNG bytes as a
data URL, ``context`` None); FAIL-SOFT PER PAGE: a page that
could not be rendered (``b""`` from :func:`render_pdf_pages` —
the index never shifts) or whose vision call failed/came back
blank gets the deterministic placeholder
``page N — description unavailable`` (a dead vision endpoint
must not sink the doc — the phase-122 fail-soft spirit).
* an ALL-failed render (no pages rendered — an unopenable file —
or every page a placeholder) returns ``""``: the existing
empty-content path (the doc lands as a zero-text doc, the
phase-126 no-extractable-text behavior).
* the digest the caller computes is over the constructed content —
for a stable file + stable vision model the content (and thus
the digest) is byte-stable across re-syncs, so the row
re-imports as UNCHANGED: no re-chunk, no re-embed, no
re-summary, no KB change. The fallback itself RE-RUNS on every
sync of a garbage PDF (the gate is a cheap pure predicate and
the render needs no LLM; the vision calls are re-issued before
the unchanged check can skip them — a documented cost of the
no-cache design, bounded by the page count of the garbage
PDFs). If the production vision model's reply is
non-deterministic, the row is re-described each sync (updated —
a cost, not a bug; the doc is still correct).
*session* is part of the signature for symmetry with the other
per-file index helpers (the fallback itself needs no DB access —
the caller's upsert pipeline owns the row).
"""
pages = render_pdf_pages(full_path)
# The loud one-line log (PLAN §9, the FINDING's signature).
logger.warning(
"pdf: text layer unusable for %s/%s — falling back to %d page screenshots",
source, rel, len(pages),
)
if not pages:
return ""
blocks: list[str] = []
got_any = False
for i, png in enumerate(pages):
page = i + 1
if not png:
blocks.append(f"page {page} — description unavailable")
continue
desc = await describe_image(llm, data=png, mime="image/png")
if desc is None or not desc.strip():
blocks.append(f"page {page} — description unavailable")
else:
blocks.append(desc.strip())
got_any = True
if not got_any:
return ""
return "\n\n".join(
f"{PAGE_SEPARATOR.format(page=page)}\n{block}"
for page, block in enumerate(blocks, start=1)
)
async def _index_pdf_embedded_images(
session: Session,
*,
app/rag/pdf.py
+106
View File
@@ -18,6 +18,16 @@ package, imported as ``fitz`` — its stable API name).
MIME type: the embedded file extension (``ext``) first, then the
image type code (``type``: 2=JPEG, 3=PNG, 4=GIF, 8=TIFF, 9=WebP),
else the ``application/octet-stream`` fallback.
* :func:`is_usable_pdf_text` — the phase-133 garbage gate: a pure,
deterministic predicate over the extracted text layer (no PDF, no
I/O) — REAL content vs. garbage (the pre-126 XML/binary soup, the
empty text layer). It is the single trigger for the page-screenshot
vision fallback (phase 133 task 03).
* :func:`render_pdf_pages` — the fallback's visual input (phase 133
task 03): one PNG per page (150 DPI, page order); an unreadable page
yields ``b""`` (the caller's placeholder — the page index never
shifts), an unopenable file yields ``[]`` (the caller's existing
empty-content path).
Every failure mode is fail-soft: an unreadable file returns an empty
result, an unreadable page or image is logged and skipped — this module
@@ -26,6 +36,7 @@ never raises for malformed input (the importer owns the loud failures).
from __future__ import annotations
import logging
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Any
@@ -142,6 +153,101 @@ def extract_pdf_text(pdf_path: Path) -> str:
return "\n\n".join(parts)
#: A line "looks like XML" when it carries a tag-shaped token — the
#: pre-126 garbage's signature (raw XMP blocks, PDF-structure noise).
_XMLISH_LINE_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
#: Garbage-gate thresholds (phase 133 task 02): an extracted text
#: layer is UNUSABLE when its XML-ish line fraction reaches this bar.
#: The FINDING's reproduced pre-126 read of the owner's file is 46%
#: XML-ish; a real document ABOUT XML that reaches the bar is a
#: documented false positive (ASSUMPTION A8 — the fallback still
#: extracts it via vision; the cost is a few vision calls).
XMLISH_LINE_FRACTION = 0.30
#: Garbage-gate thresholds (phase 133 task 02): the non-printable
#: fraction (U+FFFD replacement chars + C0 controls over the whole
#: text) at which the layer is UNUSABLE — the ``errors="replace"``
#: binary-soup signal (the pre-126 read is ~19% non-printable; real
#: extracted text is ~0.05%).
NONPRINT_CHAR_FRACTION = 0.05
def is_usable_pdf_text(text: str) -> bool:
"""Whether an extracted PDF text layer is REAL content (not garbage).
Unusable (→ the page-screenshot fallback) when ANY of:
* ``text.strip() == ""`` — the empty text layer (a scanned/image-
only PDF has nothing to read);
* ≥ :data:`XMLISH_LINE_FRACTION` (0.30) of the non-empty lines
match :data:`_XMLISH_LINE_RE` — the raw-XMP / binary-soup
signature (the FINDING's reproduced pre-126 read is 46% —
``read_text(errors="replace")`` on the owner's file left
``<illustrator:StartupProfile>Web</illustrator:StartupProfile>``
lines in the preview);
* ≥ :data:`NONPRINT_CHAR_FRACTION` (0.05) of all characters are
non-printable — the ``errors="replace"`` binary-soup signal.
The non-printable set is ``U+FFFD`` + C0 control chars
(0x00–0x1F) EXCEPT the structural ``\\t``/``\\n``/``\\r`` the
extractor legitimately emits (the explicit set is deliberate: a
``unicodedata`` Cc-category check would catch those too, and
counting them would flag every prose document).
Pure + deterministic (no PDF, no I/O) — the unit tests pin the
thresholds against the REAL reproduced garbage string, not a
hand-drawn one. A real document ABOUT XML whose tag-like lines
reach 30% is a documented false positive (ASSUMPTION A8 — the
fallback still extracts it correctly via vision; the cost is a
few vision calls).
"""
if not text.strip():
return False
non_empty_lines = [line for line in text.splitlines() if line.strip()]
xml_ish = sum(1 for line in non_empty_lines if _XMLISH_LINE_RE.search(line))
if xml_ish / len(non_empty_lines) >= XMLISH_LINE_FRACTION:
return False
non_printable = sum(
1
for ch in text
if ch == "\ufffd" or (ord(ch) < 32 and ch not in "\t\n\r")
)
return non_printable / len(text) < NONPRINT_CHAR_FRACTION
def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
"""Render every page of *pdf_path* to a PNG (page order — list index
*i* is page *i*+1).
The fallback's visual input (phase 133 task 03, ASSUMPTION A8:
fixed 150 DPI — the balance of vision legibility vs data-URL
payload). A page that cannot be rendered yields ``b""`` (the
caller's placeholder — the page index NEVER shifts, so
``---PAGE-N---`` stays truthful; the failure is logged, the
:func:`extract_pdf_text` fail-soft style). A file that cannot be
OPENED yields ``[]`` (the caller's existing empty-content path —
the doc lands with the no-extractable-text behavior, unchanged).
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as exc:
logger.warning("pdf: could not open %s for page rendering — %s", pdf_path, exc)
return []
pages: list[bytes] = []
with doc:
for page_num in range(len(doc)):
try:
pages.append(doc[page_num].get_pixmap(dpi=dpi).tobytes("png"))
except Exception as exc:
logger.warning(
"pdf: could not render page %d of %s — %s",
page_num + 1, pdf_path, exc,
)
pages.append(b"")
return pages
def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
"""Extract all embedded images from *pdf_path*.
tests/e2e/test_garbage_pdf.py
+698
View File
@@ -0,0 +1,698 @@
"""Phase 133 task 04 E2E (Playwright, mock-only): the real guide stays
clean, a synthetic garbage PDF takes the page-screenshot path.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov
The dedicated suite for ``133_pdf_screenshot_fallback`` (A16 — one
Playwright file per phase, run in isolation): it pins both legs of the
phase's guarantee at the USER SURFACE. (a) The owner's REAL guide
(``test_sources/hnb_2026_guide.tar.gz`` — the file whose PRE-phase-126
import stored raw Illustrator XMP lines, the TODO.md L9 complaint)
imports through the admin sync and PREVIEWs CLEAN text — the exact
complaint, negated on the served content AND the rendered viewer page.
(b) A synthetic PDF whose text layer is pure XML garbage
(``<illustrator:StartupProfile>Web</…>`` — the owner's exact line
shape) takes the page-screenshot fallback end to end — its stored
content is the mock's CANNED vision descriptions, one per rendered
page, and the garbage text layer is NOT what got stored. (c) A clean
PDF never burns the fallback — its stored content is byte-faithful to
the text layer (the gate's no-false-negative pin at the E2E level).
House pattern: one module app over the shared E2E Postgres (its own
port, module env — the conftest session app is never started in this
isolated run, so no port clash), each test registers its HOST TEMP DIR
as a ``kind=local`` source through the admin git-sources API (the
phase-49/phase-127 API-registration pattern — adding a plain directory
is an API-only operation), runs the real admin sync
(``POST /api/sync`` + the status poll, the surface the UI button
wraps), and an autouse truncate keeps the shared registry + KB fresh
per test (the phase-96 cleanup pattern, with the no-running-sync wait
so a leaked run cannot race the truncate).
App env: ``BOR_IMPORT_EXTENSIONS`` pinned WITH the ``pdf`` token (the
owner's live config — the phase-126/127 suites' pattern: without the
token the walk is blind to the file, and an operator's local
gitignored ``.env`` cannot leak the scope) and ``BOR_IMAGES=1`` (the
owner's live toggle — the embedded-image pipeline is live, so the
fallback PDF's extracted images join the KB exactly as a clean PDF's
do).
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the fallback's
vision surface is the mock's deterministic ``IMAGE_DESCRIPTION_MODE``
canned reply (``IMAGE_DESCRIPTION_ANSWER``, the same reply the
phase-122 image docs get), which makes the fallback's stored content a
PURE FUNCTION of the fixture — the byte-exact content assertions below
are exact only against that canned text.
Fixture provenance:
* ``hnb/`` — the REAL guide: ``test_sources/hnb_2026_guide.tar.gz``
(UNTRACKED owner fixture — the stable reference is the tarball,
never the extracted PDF) extracted into a module temp dir named
``hnb`` (the directory's basename is the ``kind=local`` source name,
phase 38; one 6.2 MB / 48-page Illustrator PDF — the extraction is
seconds, module-scoped, once).
* ``garbagedoc/`` — a synthetic garbage PDF built IN the test with
PyMuPDF (the fixture pattern from phase 133 task 03's unit test,
lifted here): 2 pages, each page's ONLY text = the owner's exact
garbage line repeated 20× (``<illustrator:StartupProfile>Web</…>``
— 100% of the lines look like tags, so the task-02 gate fails
deterministically), plus ONE small embedded raster image per page
(a 1×1 PNG via ``insert_image`` — the fallback's embedded-image leg
runs too: the screenshots are the CONTENT, the extracted images are
the citation figures — both, no conflict).
* ``cleandoc/`` — ONE small clean PDF (PyMuPDF, real prose text, no
images) — the gate's no-false-negative subject.
Test → observable mapping (Playwright Mapping Rule):
1. ``test_real_guide_previews_clean_text`` — the admin sync over
``hnb/`` settles ``success`` (the terminal status body);
``GET /api/documents/content?source=hnb&path=<the guide's pdf>``
(the PREVIEW surface — the endpoint the document viewer's content
uses) carries the real first-page text ("2026 U.S. Benefits
Guide") and NO line matching the XML-ish regex (the owner's
complaint, negated on the SERVED content, not the DB); the
document viewer page (``/document.html?source=hnb&path=…``)
renders that content — real text, page markers, the stem title,
never ``<illustrator:…>`` lines.
2. ``test_garbage_pdf_takes_the_screenshot_path`` — the admin sync
over ``garbagedoc/`` settles ``success``; the stored content (the
same content endpoint) starts with ``---PAGE-1---`` and is the
mock's CANNED vision description for BOTH pages (2 separators, 2
descriptions — the fallback fired end to end, mock-deterministic)
and contains NO ``<illustrator:StartupProfile>`` line (the garbage
text layer is NOT what got stored); the two embedded images
(``#page=N#img=0.png``) indexed with the canned description too,
and the sync's ``detail`` is the UNCHANGED shape (``added`` counts
the PDF + its image docs per the phase-126 A3 convention — the
fallback is invisible in the status; its counter is log-only,
PLAN §9 — ``pdfs_screenshotted`` is NOT a ``detail`` key).
3. ``test_clean_pdf_never_burns_the_fallback`` — the admin sync over
``cleandoc/`` settles ``success``; the stored content is the
PROSE, byte-faithful to the extraction (``---PAGE-1---\n<prose>``
exactly) and NOT the canned vision description (the fallback did
not fire on a clean layer).
"""
from __future__ import annotations
import base64
import json
import os
import re
import subprocess
import sys
import tarfile
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import httpx
import pymupdf as fitz # PyMuPDF — the synthetic-fixture builder (LOCKED A1)
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import Document
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
REPO = Path(__file__).resolve().parents[2]
#: The owner's untracked fixture (the stable reference — the tarball,
#: never the extracted PDF, is what this suite starts from).
TARBALL = REPO / "test_sources" / "hnb_2026_guide.tar.gz"
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_GARBPDF", "8149"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
# ---------------------------------------------------------------------------
# Fixture constants (deterministic, pinned)
# ---------------------------------------------------------------------------
#: The real guide — the source dir's basename is the source name
#: (``kind=local``, phase 38) and the inner file keeps its name.
HNB_SOURCE = "hnb"
GUIDE_PDF = "HBI-2026-Guide-Year-Round_260704_131221.pdf"
#: The pinned fixture text (the real first page — phase 133 task 01's
#: pin). If the owner ever re-cuts the fixture, the failure names the
#: regression — the point.
FIRST_PAGE_TEXT = "2026 U.S. Benefits Guide"
#: The title: the stem fallback — the guide's metadata carries no
#: title (only the PDFium creator/producer), so the non-markdown rule
#: applies (phase 126 A2 — one document per PDF, metadata-or-stem).
GUIDE_TITLE = GUIDE_PDF[:-4]
#: The phase-133 XML-ish line heuristic (the task-02 gate's regex):
#: ``<tag>``, ``</tag>``, ``<tag attr=``, and the PDF-structure
#: ``<</Key…`` noise all match. The preview is clean when NO line
#: matches (the real extraction is 0/2,409 XML-ish lines).
XMLISH_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
#: The owner's exact garbage line (the TODO.md L9 preview shape).
GARBAGE_SOURCE = "garbagedoc"
GARBAGE_PDF = "garbage.pdf"
GARBAGE_LINE = "<illustrator:StartupProfile>Web</illustrator:StartupProfile>"
GARBAGE_LINES_PER_PAGE = 20 # the page's ONLY text (100% XML-ish)
N_GARBAGE_PAGES = 2
#: The clean PDF — real prose (no tag-shaped lines, no images): the
#: gate's no-false-negative subject (the stored content must be
#: byte-faithful to the extraction).
CLEAN_SOURCE = "cleandoc"
CLEAN_PDF = "clean.pdf"
CLEAN_PROSE = (
"The clean reference page covers the homelab firewall rule audit."
)
#: A real 1×1 transparent PNG (the E2E suites' fixture — the pipeline
#: is content-agnostic; the well-formed bytes keep the extraction +
#: copy + serve path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _build_garbage_pdf(path: Path) -> None:
"""A REAL tiny garbage PDF (the fixture pattern from phase 133
task 03's unit test, lifted here): 2 pages, each page's ONLY text
= the owner's exact garbage line × 20 (the gate's XML signal fails
deterministically — 100% of the non-empty lines look like tags;
the ``---PAGE-N---`` separators are not tag-shaped), plus ONE small
embedded raster image per page (the 1×1 PNG — the fallback's
embedded-image leg runs too). The text layer is real EXTRACTABLE
text — this is the FINDING's class (the pre-126 raw-XMP soup), not
the empty-layer class."""
doc = fitz.open()
for _page_num in range(1, N_GARBAGE_PAGES + 1):
page = doc.new_page()
for i in range(GARBAGE_LINES_PER_PAGE):
page.insert_text((72, 72 + i * 14), GARBAGE_LINE)
page.insert_image(fitz.Rect(72, 500, 73, 501), stream=PNG_1X1)
doc.save(path)
doc.close()
def _build_clean_pdf(path: Path) -> None:
"""ONE small clean PDF: a single page of real prose (no
tag-shaped lines, no embedded images) — the extracted text layer
is exactly :data:`CLEAN_PROSE`, so the byte-faithful content
assertion is exact."""
doc = fitz.open()
doc.new_page().insert_text((72, 72), CLEAN_PROSE)
doc.save(path)
doc.close()
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def hnb_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The REAL guide (module-scoped, once — the 6 MB extraction is
seconds): the tarball is extracted into a temp source dir named
``hnb`` (the basename is the source name; the inner file keeps its
name). ``test_sources/`` is untracked — the extraction never
touches the repo, so ``git status`` stays clean of binaries."""
if not TARBALL.is_file():
pytest.skip(
"test_sources/hnb_2026_guide.tar.gz is missing — the "
"untracked owner fixture (2.6 MB, one 48-page Illustrator PDF)"
)
src = tmp_path_factory.mktemp("bor_garb_hnb") / HNB_SOURCE
src.mkdir()
with tarfile.open(TARBALL, "r:gz") as tar:
tar.extractall(src, filter="data")
assert (src / GUIDE_PDF).is_file()
return src
@pytest.fixture(scope="module")
def garbage_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The synthetic garbage source (module-scoped): ``garbagedoc/``
with the 2-page garbage PDF (see :func:`_build_garbage_pdf`)."""
src = tmp_path_factory.mktemp("bor_garb_garbage") / GARBAGE_SOURCE
src.mkdir()
_build_garbage_pdf(src / GARBAGE_PDF)
assert (src / GARBAGE_PDF).is_file()
return src
@pytest.fixture(scope="module")
def clean_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The clean-PDF source (module-scoped): ``cleandoc/`` with the
one-page prose PDF (see :func:`_build_clean_pdf`)."""
src = tmp_path_factory.mktemp("bor_garb_clean") / CLEAN_SOURCE
src.mkdir()
_build_clean_pdf(src / CLEAN_PDF)
assert (src / CLEAN_PDF).is_file()
return src
def _app_env(
base_url: str, tmp_path_factory: pytest.TempPathFactory
) -> dict[str, str]:
"""The per-module app env (the conftest pattern, cf.
``test_pdf_support.py`` / ``test_incremental_folder_summaries.py``):
the ``base_url`` is the mock LLM directly (the deterministic
canned replies are what the byte-exact assertions key on),
``BOR_GIT_SOURCES`` forced empty (the sync sources are this
suite's own DB-registered local rows — an operator's local
gitignored ``.env`` cannot leak the owner's corpus), the
leak-guarded code defaults, and the phase-133 knobs:
``BOR_IMPORT_EXTENSIONS`` pinned WITH the ``pdf`` token (the
owner's live config — the feature's gate; without it the walk is
blind to the PDFs) and ``BOR_IMAGES=1`` (the owner's live toggle —
the embedded-image pipeline is live on BOTH the clean and the
fallback PDFs), with the image/upload homes in the suite's scratch
dir (the app under test must not write copies into the owner's
real ``~/bor-sources``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
)
# Mock-calibrated gate (conftest pattern) — no chat turn is ever
# sent in this suite, but the app boots with the same env shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
# Phase 67: instant retry waits + the code-default budget.
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directories this suite registers.
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
# Phase 133: the owner's live import scope WITH the ``pdf`` token +
# the images toggle under test + the suite-private homes.
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
env["BOR_IMAGES"] = "1"
scratch = tmp_path_factory.mktemp("bor_garb_scratch")
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_IMAGE_DIR"] = str(scratch / "images")
return env
@pytest.fixture(scope="module")
def app_server(
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The app under test (all three tests): one module app with the
phase-133 env (see :func:`_app_env`). The temp source dirs are NOT
dependencies — the app reads them only at sync time, and the
``hnb_dir`` fixture skips (not the whole module) when the untracked
owner fixture is missing."""
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern): the
E2E suites share one Postgres, so a leftover git_sources row or
document would pollute the sync's counts and the content the
assertions pin. ``sources_meta`` (the KB generation counter)
resets with the KB — the sync re-creates the row."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries, sources_meta"
)
)
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No background sync may leak across tests (the run state lives in
the app's memory, and a still-running run would keep importing into
the NEXT test's truncated KB): wait for the app's sync status to
be non-running BEFORE the truncate. Own admin session (the
endpoint is admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(base_url=base_url, timeout=5.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation (the phase-96 cleanup pattern): the
shared E2E Postgres is truncated before AND after every test —
each test registers its own source and runs its own sync, so the
KB rows are exactly this test's own. The pre-wait keeps a leaked
run from the previous test (the module app is shared) from racing
the truncate."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers (the phase-127 house pattern: API-driven source registration
# + sync over plain httpx — the admin cookie rides the client jar)
# ---------------------------------------------------------------------------
def _register_source(base_url: str, path: str) -> None:
"""Register the temp directory through the authenticated API (the
``test_local_directory_sources.py`` / phase-94 pattern — a plain
directory is an API/DB-only operation; the page form is gone)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post("/api/git-sources", json={"kind": "local", "path": path})
assert r.status_code == 201, r.text
def _started_at(body: dict[str, Any]) -> float:
"""The status body's ``started_at`` as a wall-clock epoch (the
server stamps it in UTC; this process runs on the same host —
same clock, the phase-64 recorder's freshness convention)."""
started = body.get("started_at")
if not started:
return 0.0
try:
return datetime.fromisoformat(str(started)).timestamp()
except ValueError:
return 0.0
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
run reaches a terminal state (the ``test_local_directory_sources``
pattern, over plain httpx — with the stale-terminal freshness gate:
the module app is shared, so the PREVIOUS run's terminal lives in
the app's memory until this run supersedes it; a terminal is
accepted only after a running tick was observed or its
``started_at`` post-dates this POST)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
t0 = time.time()
r = client.post("/api/sync")
assert r.status_code == 202, r.text
saw_running = False
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
state = body["state"]
if state == "running":
saw_running = True
elif state in ("success", "failed") and (
saw_running
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
):
return body
time.sleep(0.1)
raise AssertionError(f"sync did not reach a terminal state: {body}")
def _doc_content(base_url: str, source: str, path: str) -> dict[str, Any]:
"""``GET /api/documents/content`` (the PREVIEW surface — the
endpoint the document viewer's content uses) as the signed-in
admin. The wire shape (``DocContent``): ``content`` is the stored
document content verbatim; ``is_image`` is always present."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.get(
"/api/documents/content", params={"source": source, "path": path}
)
assert r.status_code == 200, f"content lookup failed: {r.text}"
return r.json()
def _image_docs(source: str) -> list[Document]:
"""The source's stored ``is_image`` documents, in path order (the
test process's direct DB access — the E2E's other established
lens)."""
with SessionLocal() as db:
rows = db.scalars(
select(Document)
.where(Document.source == source, Document.is_image.is_(True))
.order_by(Document.path)
).all()
return list(rows)
def _xmlish_lines(content: str) -> list[str]:
"""The content's non-empty lines matching the phase-133 XML-ish
regex (the owner's complaint — the pre-126 raw-XMP lines)."""
return [
line
for line in content.splitlines()
if line.strip() and XMLISH_RE.search(line)
]
# ---------------------------------------------------------------------------
# 1. The REAL guide (the owner's exact file) previews CLEAN text
# ---------------------------------------------------------------------------
def test_real_guide_previews_clean_text(
app_url: str, hnb_dir: Path, page: Page, db_ready: None
) -> None:
"""The owner's exact file, through the real admin sync: the
preview (``GET /api/documents/content`` — the endpoint the
document viewer's content uses) carries the real first-page text
and NO XML-ish line (the TODO.md L9 complaint, negated on the
SERVED content), and the document viewer page renders that content
— real text, never ``<illustrator:…>`` lines. The pre-126
binary-text import (the raw-XMP soup) can never come back: if the
PDF were ever routed through the text branch again, the preview
would be ~46% XML-ish and both assertions break."""
_register_source(app_url, str(hnb_dir))
# 48 pages + the embedded-image pass against the mock — the
# generous budget (the in-process regression pin's 120 s bar plus
# the HTTP round-trips over the mock).
body = _run_sync(app_url, timeout_s=180.0)
assert body["state"] == "success", body
detail = body["detail"]
# The walk admitted exactly the one file (the ``pdf`` token — the
# PDF branch, never the text branch), no errors, nothing pruned.
assert detail["files"] == 1, detail
assert detail["errors"] == 0, detail
assert detail["pruned"] == 0, detail
# The preview surface (the endpoint the document viewer's content
# uses): the body is the CLEAN extracted text.
doc = _doc_content(app_url, HNB_SOURCE, GUIDE_PDF)
assert doc["is_image"] is False # it is a PDF document, not an image doc
assert doc["title"] == GUIDE_TITLE # the stem fallback (no metadata title)
content = doc["content"]
assert FIRST_PAGE_TEXT in content, (
"the real first-page text is missing from the served content — "
"the stored content is not the extraction"
)
xml_lines = _xmlish_lines(content)
assert xml_lines == [], (
f"the owner's complaint returned on the served content — "
f"{len(xml_lines)} XML-ish lines, e.g. {xml_lines[:3]!r}"
)
# The phase-126 A3 leg: the guide's embedded images joined the KB
# with the canned descriptions — the ``added`` accounting is the
# one file PLUS its image docs (the phase-126 convention the
# task-01 regression pin documents).
imgs = _image_docs(HNB_SOURCE)
assert imgs, "the guide's embedded images did not index"
assert detail["added"] == 1 + len(imgs), detail
# The document viewer PAGE renders that content (the user surface):
# the real text + the page-1 marker, the stem title, and the owner's
# line is ABSENT from the rendered preview.
login(page, app_url)
page.goto(
app_url + f"/document.html?source={HNB_SOURCE}&path={GUIDE_PDF}"
)
expect(page.locator("#doc-not-found")).to_be_hidden(timeout=30_000)
expect(page.locator("#doc-title")).to_have_text(GUIDE_TITLE)
raw = page.locator("pre.doc-raw")
expect(raw).to_be_visible(timeout=30_000)
expect(raw).to_contain_text(FIRST_PAGE_TEXT)
expect(raw).to_contain_text("---PAGE-1---")
rendered = raw.inner_text()
assert GARBAGE_LINE not in rendered, (
"the viewer renders the owner's <illustrator:StartupProfile> line — "
"the pre-126 garbage preview came back"
)
assert _xmlish_lines(rendered) == [], (
"the viewer renders XML-ish lines — the clean extraction is not what "
"the preview shows"
)
# The non-image doc with a non-empty summary renders the Summary
# panel (the phase-36 shape, above the content).
expect(page.locator(".doc-summary")).to_have_count(1)
# ---------------------------------------------------------------------------
# 2. The synthetic garbage PDF takes the screenshot fallback end to end
# ---------------------------------------------------------------------------
def test_garbage_pdf_takes_the_screenshot_path(
app_url: str, garbage_dir: Path, db_ready: None
) -> None:
"""The owner's specified path, end to end: a PDF whose text layer
is pure XML garbage fails the task-02 gate, every page is rendered
to a screenshot and read by the (mock) vision model, and the stored
content is the CANNED page descriptions — 2 separators, 2
descriptions, in page order — with the garbage text layer NOT what
got stored. The fallback is invisible in the sync's status shape
(``detail`` is the phase-126 shape — the counter is log-only,
PLAN §9), and the extracted embedded images join the KB exactly as
a clean PDF's do (the screenshots are the CONTENT; the extracted
images are the citation figures — both, no conflict)."""
_register_source(app_url, str(garbage_dir))
body = _run_sync(app_url, timeout_s=120.0)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["files"] == 1, detail
assert detail["errors"] == 0, detail
# The stored content (the same content endpoint the preview uses):
# the fallback's page-anchored canned descriptions, byte-exact.
doc = _doc_content(app_url, GARBAGE_SOURCE, GARBAGE_PDF)
assert doc["is_image"] is False # a PDF whose content is vision text
content = doc["content"]
assert content.startswith("---PAGE-1---")
assert content.count("---PAGE-1---") == 1
assert content.count("---PAGE-2---") == 1
# The mock's CANNED vision description for BOTH pages (the
# ``IMAGE_DESCRIPTION_MODE`` reply — deterministic). The first-40
# char pin is the task's lens; the count + the byte-exact shape
# below make it airtight.
assert IMAGE_DESCRIPTION_ANSWER[:40] in content
assert content.count(IMAGE_DESCRIPTION_ANSWER) == 2
expected = (
f"---PAGE-1---\n{IMAGE_DESCRIPTION_ANSWER}\n\n"
f"---PAGE-2---\n{IMAGE_DESCRIPTION_ANSWER}"
)
assert content == expected, (
"the stored content is not the fallback's canned page descriptions"
)
# The garbage text layer is NOT what got stored — the screenshot
# path replaced it (the owner's exact line, negated).
assert GARBAGE_LINE not in content
assert "<illustrator" not in content
# The fallback's embedded-image leg: both extracted images (one per
# page, the 1×1 PNG — ``#page=N#img=0`` paths) indexed with the
# canned description (the phase-122 image convention).
imgs = _image_docs(GARBAGE_SOURCE)
assert [i.path for i in imgs] == [
f"{GARBAGE_PDF}#page=1#img=0.png",
f"{GARBAGE_PDF}#page=2#img=0.png",
]
for img in imgs:
assert img.content == IMAGE_DESCRIPTION_ANSWER
assert img.summary == img.content # the image convention
# The sync's ``detail`` is the UNCHANGED shape: the garbage PDF is
# a NORMAL import (the one file + its two image docs per the
# phase-126 A3 convention) and the fallback's counter is
# log-only — ``pdfs_screenshotted`` is NOT a ``detail`` key.
assert detail["added"] == 1 + len(imgs), detail
assert detail["pruned"] == 0, detail
assert "pdfs_screenshotted" not in detail
# ---------------------------------------------------------------------------
# 3. A clean PDF never burns the fallback (the gate's no-false-
# negative pin at the E2E level)
# ---------------------------------------------------------------------------
def test_clean_pdf_never_burns_the_fallback(
app_url: str, clean_dir: Path, db_ready: None
) -> None:
"""The gate is the ONLY gate: a clean text layer never reaches the
fallback — the stored content is the PROSE, byte-faithful to the
extraction (``---PAGE-1---\n<prose>`` exactly — the phase-126
happy path, unchanged) and NOT the canned vision description (no
screenshot, no vision burn on the happy path)."""
_register_source(app_url, str(clean_dir))
body = _run_sync(app_url, timeout_s=60.0)
assert body["state"] == "success", body
detail = body["detail"]
# One file, one added document — no embedded images in the clean
# fixture, so the ``added`` count is the PDF alone.
assert detail["files"] == 1, detail
assert detail["added"] == 1, detail
assert detail["errors"] == 0, detail
doc = _doc_content(app_url, CLEAN_SOURCE, CLEAN_PDF)
assert doc["is_image"] is False
# Byte-faithful to the text layer (the extraction's exact shape —
# the page marker + the stripped page text).
assert doc["content"] == f"---PAGE-1---\n{CLEAN_PROSE}"
# The fallback did NOT fire: no canned vision text anywhere in the
# stored content.
assert IMAGE_DESCRIPTION_ANSWER[:40] not in doc["content"]
assert _image_docs(CLEAN_SOURCE) == []
tests/integration/test_importer.py
+22 -15
View File
@@ -636,13 +636,18 @@ def test_pdf_without_embedded_images_and_unchanged_rerun(db, tmp_path) -> None:
_cleanup_source(db, source_root.name)
def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
"""A PDF with no extractable text (image-only, ``images`` OFF — the
embedded image is invisible) → the document is still created, with
EMPTY content (the empty byte string's digest), NO content chunks
and NO content embedding (there is no text to embed — the only
chunk is the best-effort ``lite`` summary's ``is_summary`` row), and
the warning line is the signal."""
def test_pdf_without_text_takes_the_screenshot_fallback(db, tmp_path) -> None:
"""Phase 133 (task 03): a PDF with no extractable text layer
(image-only, ``images`` OFF — the embedded image is invisible to
the phase-126 A3 pass, but the FALLBACK is not gated by the
toggle) → the gate's empty-layer target fires: the single page is
rendered to a PNG and vision-described (the mock's fixed
:data:`DESCRIPTION`), and the document lands with the
``---PAGE-N---``-anchored description as content (one page-anchored
content chunk + the ``lite`` summary chunk, both embedded). The
phase-126 empty-content behavior is retired for layers that CAN be
screenshotted — it survives only for an unopenable file, where the
render fails too (the unit suite pins it)."""
source_root = tmp_path / "pdfemptykb"
source_root.mkdir()
_build_pdf(source_root / "pics.pdf", [("", [_png_bytes(40, 30)])])
@@ -650,6 +655,7 @@ def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
try:
summary = asyncio.run(import_sources([source_root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
assert summary.pdfs_screenshotted == 1
doc = db.scalar(
select(Document).where(
@@ -657,18 +663,19 @@ def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
)
)
assert doc is not None and doc.is_image is False
assert doc.content == ""
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
assert doc.content == f"---PAGE-1---\n{DESCRIPTION}"
assert doc.content_hash == hashlib.sha256(doc.content.encode("utf-8")).hexdigest()
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
assert [c for c in chunks if not c.is_summary] == [], (
"no text → no content chunks (and no content embedding)"
content_chunks = [c for c in chunks if not c.is_summary]
assert [c.content for c in content_chunks] == [f"Page 1:\n{DESCRIPTION}"], (
"the page-anchored fallback content is chunked like any PDF"
)
# The phase-30/118 path still runs best-effort on the empty
# content — the summary is the only chunk, and it is embedded.
assert len(chunks) == 1 and chunks[0].is_summary
assert chunks[0].embedding is not None and len(chunks[0].embedding) == 768
# The phase-30/118 path runs on the fallback content — the
# summary chunk mirrors the stored summary, both embedded.
assert len(chunks) == 2
assert all(c.embedding is not None and len(c.embedding) == 768 for c in chunks)
assert doc.summary is not None
finally:
_cleanup_source(db, source_root.name)
tests/integration/test_pdf_hnb_guide_regression.py
+331
View File
@@ -0,0 +1,331 @@
"""Integration: phase 133 (task 01) — the owner's HNB guide regression pin.
FINDING (the two-line root cause, full evidence in the phase overview):
the garbage the owner saw is the PRE-PHASE-126 import — ``pdf`` was
already in the owner's ``BOR_IMPORT_EXTENSIONS`` (local, untracked
``.env``) before the PDF parser shipped, so the generic text branch
read the RAW PDF bytes (``read_text(errors="replace")``) and the
uncompressed Illustrator XMP/structure blocks (222 ×
``<illustrator:StartupProfile>`` + 138 × ``<x:xmpmeta>`` blocks) were
chunked and embedded as literal text. Phase 126+ extracts the same
file correctly; the stale production row self-heals on re-sync
(content-hash delta → ``updated`` with clean content).
This suite pins the owner's EXACT file as a permanent regression: the
archive's 48-page Illustrator PDF is imported through the REAL
importer — the real walk, the real PDF branch (extraction → digest →
page-anchored chunking → 768-dim embeds → ``lite`` summary → the
phase-126 embedded-image pass, whose vision seam runs against the
multimodal-capable fake below) — and must land CLEAN content: the
XML-ish line fraction stays far under the task-02 gate's 0.30
(0.0 in the real extraction — asserted < 0.05 to be strict), the
non-printable fraction stays < 0.01, and the real first-page text is
present. If the PDF is ever routed through the text branch again, the
stored content is the raw bytes again (~46% XML-ish) and the pin
breaks — the pin has teeth.
Run:
uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v
Requires: podman compose up -d db
Fixture provenance: ``test_sources/hnb_2026_guide.tar.gz`` — the
UNTRACKED owner fixture (2.6 MB compressed; one 6.2 MB, 48-page
PDFium/Illustrator export, ``HBI-2026-Guide-Year-Round_260704_131221.pdf``
inside). The test extracts it into ``tmp_path`` (the test's local
source dir) and never touches the repo — ``git status`` stays clean of
binaries. The pinned fixture text (the first-page title) fails LOUDLY
if the owner ever re-cuts the fixture: that IS the regression signal.
Config note (why the settings are built the way they are):
* ``_env_file=".env"`` — the REAL extension set: ``pdf`` is in the
owner's live ``BOR_IMPORT_EXTENSIONS``. A ``_env_file=None`` fake
would use the code defaults (no ``pdf`` token) and SILENTLY skip the
PDF — the exact trap that hid this class before (FINDING, point 3).
* ``images=True`` — the owner's live toggle (``BOR_IMAGES=1`` in the
same ``.env``). The suite forces ``BOR_IMAGES`` to the code default
(off) in the process environment (the conftest leak class, phase
122), and process env beats the ``.env`` file — so the live value is
restored with an init kwarg (init kwargs beat both). The embedded-
image pass is the phase-126 A3 seam this pin covers.
* ``image_dir`` under ``tmp_path`` — the house hermetic pattern
(``test_importer.py``'s ``_image_llm``): the embedded-image copies
never touch the owner's persistent image home.
``summary.added`` accounting: the walked file contributes EXACTLY ONE
added document (the guide PDF); the phase-126 A3 convention counts
each embedded image doc ``added`` too, so the assertion is
``added == 1 + len(image_docs)`` together with ``files == 1`` (the
walk admitted the one file) — the strict generalization of the naive
``added == 1`` for a PDF whose images join the KB.
"""
from __future__ import annotations
import asyncio
import re
import tarfile
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import Chunk, Document
from app.rag.importer import ImportSummary, import_sources
from app.rag.pdf import PAGE_SEPARATOR
from app.rag.summarizer import DESCRIBE_PROMPT
from tests.fakes import FakeEmbedder
#: The repo root (this file lives in ``tests/integration/``).
REPO_ROOT = Path(__file__).resolve().parents[2]
#: The owner's untracked fixture (the stable reference — the tarball,
#: never the extracted PDF, is what the test starts from).
TARBALL = REPO_ROOT / "test_sources" / "hnb_2026_guide.tar.gz"
#: The inner file keeps its name; the temp source dir is named
#: ``test_sources`` (the importer's ``source = root.name`` convention —
#: the same identity the handbook pin uses).
SOURCE = "test_sources"
PDF_NAME = "HBI-2026-Guide-Year-Round_260704_131221.pdf"
PDF_REL = f"{SOURCE}/{PDF_NAME}"
#: The pinned fixture text (the real first page). If the owner ever
#: re-cuts the fixture, the failure names the regression — the point.
FIRST_PAGE_TEXT = "2026 U.S. Benefits Guide"
#: The title: the stem fallback — the PDF's metadata carries no title
#: (only the PDFium creator/producer), so the non-markdown rule applies.
TITLE = "HBI-2026-Guide-Year-Round_260704_131221"
#: The phase-133 XML-ish line heuristic (the task-02 gate's regex):
#: ``<tag>``, ``</tag>``, ``<tag attr=``, and the PDF-structure
#: ``<</Key…`` noise all match.
XMLISH_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
#: Clean-content bars (stricter than the task-02 gate on purpose — the
#: real extraction is 0.0 XML-ish / ~0.0005 non-printable, the
#: pre-126 garbage read is ~0.46 XML-ish AND heavy on U+FFFD).
XML_FRACTION_BAR = 0.05
NONPRINT_FRACTION_BAR = 0.01
#: The task's wall-time guard: 48 pages × in-process fakes must stay
#: fast — a drift past this means the test is too heavy.
WALL_TIME_BAR_S = 120.0
pytestmark = pytest.mark.skipif(
not TARBALL.is_file(),
reason=(
"test_sources/hnb_2026_guide.tar.gz is missing — the untracked "
"owner fixture (2.6 MB, one 48-page Illustrator PDF)"
),
)
class GuideVisionLLM(FakeEmbedder):
"""The task-03-style vision mock the ``fakes.py`` comment
anticipates: the base ``chat`` is string-only — the MULTIMODAL user
content ``describe_image`` sends (the ``list``-of-parts shape)
would crash it — so ``list`` content gets the deterministic canned
reply instead. String (``lite``) calls keep the ``FakeEmbedder``
behaviour. Mirrors ``test_importer.py``'s ``_MockVisionLLM``
pattern; the REAL ``import_sources`` → ``_describe_or_skip`` →
``describe_image`` chain runs against it (no seam patch)."""
VISION_REPLY = "A rendered page of the guide (test vision description)."
def __init__(self) -> None:
super().__init__()
#: The text part of every multimodal (vision) call received.
self.vision_texts: list[str] = []
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.vision_texts.append(
next((p["text"] for p in user if p.get("type") == "text"), "")
)
return self.VISION_REPLY
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def _settings(tmp_path: Path) -> Settings:
"""The owner's live import config (the point of the pin — see the
module docstring's Config note) + the hermetic image dir."""
kwargs: dict[str, Any] = {
"_env_file": ".env",
"images": True,
"image_dir": str(tmp_path / "images"),
}
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
def _xml_frac(content: str) -> float:
"""The phase-133 XML-ish line fraction: non-empty lines matching
:data:`XMLISH_RE`, over the non-empty lines."""
lines = [line for line in content.splitlines() if line.strip()]
if not lines:
return 0.0
return sum(1 for line in lines if XMLISH_RE.search(line)) / len(lines)
def _nonprint_frac(content: str) -> float:
"""The binary-soup signal: U+FFFD replacement chars + C0 control
chars over the whole text — the whitespace controls (``\\n``/``\\t``/
``\\r``) are prose line breaks, not garbage (counting them would
flag every document), so they are excluded."""
if not content:
return 0.0
bad = sum(
1
for ch in content
if ch == "\ufffd" or (ord(ch) < 32 and ch not in "\r\n\t")
)
return bad / len(content)
@pytest.fixture()
def guide_kb(
db: Session, tmp_path: Path
) -> Iterator[tuple[GuideVisionLLM, ImportSummary, float]]:
"""The real guide, imported ONCE per test through the REAL importer
(the house KB pattern — ``test_read_line_ranges_handbook.py``):
``chunks``/``documents``/``folder_summaries`` are truncated around
the test, the tarball is extracted into a temp ``test_sources``
source dir, and the deterministic in-process
:class:`GuideVisionLLM` stands in for the embed/summary/vision
calls (the whole 48-page index build stays fast and offline).
Yields ``(llm, summary, elapsed_s)``."""
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
db.commit()
root = tmp_path / SOURCE
root.mkdir()
with tarfile.open(TARBALL, "r:gz") as tar:
tar.extractall(root, filter="data")
llm = GuideVisionLLM()
llm.settings = _settings(tmp_path)
started = time.monotonic()
summary = asyncio.run(import_sources([root], llm, session=db))
elapsed = time.monotonic() - started
yield llm, summary, elapsed
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
db.commit()
db.expire_all()
def test_hnb_guide_imports_clean_through_the_real_pipeline(
db: Session,
guide_kb: tuple[GuideVisionLLM, ImportSummary, float],
) -> None:
"""The owner's exact file lands CLEAN through the real pipeline:
the walk admits the one file (``pdf`` token — the PDF branch, never
the text branch), the stored content is clean + carries the real
first-page text, the page-anchored chunks are embedded, the ``lite``
summary is stored, and every embedded-image doc got the fake vision
description (the multimodal ``chat`` was called)."""
llm, summary, elapsed = guide_kb
# The task's wall-time guard — the suite must stay fast.
assert elapsed < WALL_TIME_BAR_S, (
f"the guide import took {elapsed:.1f}s (> {WALL_TIME_BAR_S:.0f}s) — "
"the test is too heavy, the suite must stay fast"
)
# The walk ADMITTED the guide as one file with the ``pdf`` token —
# the PDF branch (the text branch is unreachable for ``.pdf`` after
# phase 126; if it ever ran on this file again, the content bars
# below break — the pin has teeth).
assert summary.files == 1
assert summary.formats == {"pdf": 1}
assert summary.errors == 0
assert summary.unchanged == 0
assert summary.pruned == 0
doc = db.scalar(
select(Document).where(Document.source == SOURCE, Document.path == PDF_NAME)
)
assert doc is not None, "the guide did not import"
assert doc.is_image is False
assert doc.title == TITLE # the stem fallback (no metadata title)
content = doc.content
# The stored content is CLEAN — the pre-126 XML-garbage import can
# never come back (the garbage read is ~0.46 XML-ish AND heavy on
# U+FFFD; the real extraction is 0.0 / ~0.0005).
assert _xml_frac(content) < XML_FRACTION_BAR, (
f"the guide's content is {_xml_frac(content):.0%} XML-ish — the "
"pre-126 raw-bytes import came back (the PDF must not be routed "
"through the text branch)"
)
assert _nonprint_frac(content) < NONPRINT_FRACTION_BAR, (
f"the guide's content is {_nonprint_frac(content):.2%} non-printable — "
"binary soup, not extracted text"
)
# The real first-page text is present (the pinned fixture text) and
# the content opens with the page-1 marker (the phase-126 A2 shape).
assert FIRST_PAGE_TEXT in content
assert content.startswith(PAGE_SEPARATOR.format(page=1))
# The real pipeline ran to the end: page-anchored content chunks
# with 768-dim embeddings + exactly one ``is_summary`` chunk at
# position −1 (the ``lite`` summary, stored with the pointer line).
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
content_chunks = [c for c in chunks if not c.is_summary]
assert content_chunks, "the guide's content produced no chunks"
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert doc.summary is not None
assert doc.summary.endswith(f"Source: {PDF_REL}")
# The embedded-image docs (the phase-126 A3 ``#page=…#img=…``
# rows): every one of them got the FAKE vision description — the
# multimodal ``chat`` was called (the task's literal assertion),
# exactly once per image doc, with the locked bare prompt.
image_docs = db.scalars(
select(Document).where(
Document.source == SOURCE,
Document.is_image.is_(True),
Document.path.like(f"{PDF_NAME}#page=%"),
)
).all()
assert image_docs, "the guide's embedded images did not index"
for img in image_docs:
assert img.content == GuideVisionLLM.VISION_REPLY
assert img.summary == img.content # the image convention
assert img.image_path is not None
copy = Path(img.image_path)
assert copy.parent == Path(llm.settings.image_dir).expanduser()
assert copy.exists(), "the embedded image's copy is missing"
multimodal_calls = [
messages
for messages in llm.chat_calls
if any(
isinstance(m.get("content"), list) for m in messages if m.get("role") == "user"
)
]
assert len(multimodal_calls) >= 1, "the multimodal chat was never called"
assert len(llm.vision_texts) == len(image_docs), (
"one vision call per embedded image doc (no failures)"
)
# The bare locked prompt on every call (no markdown reference
# exists in this PDF-only source — the phase-126 A3 shape).
assert all(t == DESCRIBE_PROMPT for t in llm.vision_texts)
assert summary.images_failed == 0
# The added accounting (see the module docstring): the walked file
# is EXACTLY ONE added document (the guide PDF); every other added
# doc is an embedded-image byproduct (the phase-126 A3 convention
# counts image docs added/updated) — nothing else touched the KB.
assert summary.added == 1 + len(image_docs)
assert summary.updated == 0
tests/unit/test_importer.py
+422 -3
View File
@@ -14,6 +14,8 @@ for the fail-soft path.
from __future__ import annotations
import asyncio
import base64
import hashlib
import logging
import os
from collections.abc import Callable
@@ -985,14 +987,16 @@ def test_import_summary_log_line_includes_summary_counters(
"""PLAN §9 summary line: the phase-30 counters sit between
``embed_batches`` and ``formats``; the phase-118 backfill counter
sits between ``summary_errors`` and ``dates_updated``; the
phase-106 date-refresh counter and the phase-122 ``images_failed``
counter sit after ``dates_updated``, before ``formats``."""
phase-106 date-refresh counter, the phase-122 ``images_failed``
counter, and the phase-133 ``pdfs_screenshotted`` counter sit after
``dates_updated``, before ``formats``."""
s = ImportSummary()
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
s.summaries, s.summary_errors = 2, 1
s.summary_backfilled = 1
s.dates_updated = 0
s.images_failed = 0
s.pdfs_screenshotted = 1
s.formats = {"md": 1, "yaml": 2}
with caplog.at_level(logging.INFO, logger="app.importer"):
s.log()
@@ -1000,7 +1004,7 @@ def test_import_summary_log_line_includes_summary_counters(
assert line == (
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
"dates_updated=0 images_failed=0 formats=yaml:2,md:1"
"dates_updated=0 images_failed=0 pdfs_screenshotted=1 formats=yaml:2,md:1"
)
@@ -1644,3 +1648,418 @@ def test_on_chunk_progress_pdf_branch_reports_content_chunks(
# The content embed (the 2 page chunks, one batch) + the excluded
# summary-chunk embed — the hook fired for the first only.
assert [len(c) for c in llm.calls] == [2, 1]
# ---------- phase 133, task 03: the page-screenshot fallback ----------
class _PageVisionLLM(FakeEmbedder):
"""The phase-133 mock VISION client — a STABLE vision model: the
MULTIMODAL describe call (``user`` content = a part list — the
shape ``describe_image`` sends) gets a canned reply DETERMINED BY
THE IMAGE'S BYTES (a short sha256 of the data URL's base64 — the
same rendered page always gets the same reply, ACROSS RUNS; the
pages' bytes differ, so the replies differ and the page order
stays checkable, and the digest-stability test's "stable vision
model" premise holds). Text (``lite``) calls keep the
``FakeEmbedder`` behaviour. Every vision call's model is recorded
(``vision_models``)."""
def __init__(self) -> None:
super().__init__()
self.vision_calls = 0
self.vision_models: list[str | None] = []
@staticmethod
def reply_for(image_bytes: bytes) -> str:
"""The canned reply a vision call for *image_bytes* gets —
deterministic in the bytes (tests build the expected content
with the same helper)."""
return f"Page {hashlib.sha256(image_bytes).hexdigest()[:8]} described."
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.vision_calls += 1
self.vision_models.append(model)
for part in user:
if part.get("type") == "image_url":
b64 = part["image_url"]["url"].rsplit(",", 1)[1]
return self.reply_for(base64.b64decode(b64))
raise LLMError("no image part in the multimodal message (test bug)")
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
class _FirstPageFailsVisionLLM(FakeEmbedder):
"""A vision client that REJECTS the first multimodal call (a dead
page's vision request) and then describes — drives the per-page
fail-soft (the dead page becomes the placeholder, the next page
still gets its description)."""
def __init__(self) -> None:
super().__init__()
self.vision_calls = 0
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.vision_calls += 1
if self.vision_calls == 1:
raise LLMError("simulated dead vision call (test sentinel)")
return "Page two described fine."
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
class _AlwaysFailsVisionLLM(FakeEmbedder):
"""A vision client that REJECTS every multimodal call (the dead
vision endpoint — the per-page fail-soft's total-failure case:
every page becomes the placeholder and the doc lands empty)."""
def __init__(self) -> None:
super().__init__()
self.vision_calls = 0
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.vision_calls += 1
raise LLMError("simulated dead vision endpoint (test sentinel)")
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def _pdf_settings(tmp_path: Path, images: bool = False) -> Settings:
"""Settings admitting the ``pdf`` import token (the code DEFAULT set
has no ``pdf`` — the phase-133 task-01 trap: a default fake would
silently skip the PDF) with the image knobs at their test values
(``images`` off unless pinned otherwise — the fallback's own vision
calls need no image copy, only the phase-122 embedded-image pass
does)."""
return Settings(
_env_file=None, # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
import_extensions="md,txt,pdf",
images=images,
image_dir=str(tmp_path / "images"),
)
def _build_garbage_pdf(path: Path, pages: int = 2) -> None:
"""A REAL tiny garbage PDF: every page carries ONLY XML-ish lines
(``<foo:pN>i</foo:pN>`` — page-specific tags, so the rendered page
PNGs differ and the stable fake's replies do too) — the gate's XML
signal fails deterministically (100% of the text lines look like
tags; the ``---PAGE-N---`` separators are not tag-shaped). The text
layer is real EXTRACTABLE text — this is the FINDING's class (the
pre-126 raw-XMP soup), not the empty-layer class."""
doc = fitz.open()
for p in range(1, pages + 1):
page = doc.new_page()
for i in range(10):
page.insert_text((72, 72 + i * 14), f"<foo:p{p}>{i}</foo:p{p}>")
doc.save(path)
doc.close()
def _pdf_doc(db, source: str, rel: str) -> Document:
doc = db.scalar(
select(Document).where(Document.source == source, Document.path == rel)
)
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
return doc
def test_garbage_pdf_takes_the_page_screenshot_fallback(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The owner's specified path end to end: a PDF whose text layer is
XML garbage fails the gate, every page is rendered to a PNG and
read by the vision model, and the doc lands with the
``---PAGE-N---``-anchored page descriptions (PAGE ORDER pinned by
the differing canned replies) — the page-aware chunker works
unchanged on the separators, the digest is over the FINAL content,
and the counter + the loud warning (the FINDING's signature) fire.
``is_image`` stays False (it is a PDF document whose content
happens to be vision text)."""
root = tmp_path / "garbage"
root.mkdir()
_build_garbage_pdf(root / "garbage.pdf")
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path)
# The stable fake's expected replies (rendered page bytes → reply;
# page order pinned: page i's bytes carry page i's description).
pngs = importer.render_pdf_pages(root / "garbage.pdf")
d1, d2 = _PageVisionLLM.reply_for(pngs[0]), _PageVisionLLM.reply_for(pngs[1])
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert summary.pdfs_screenshotted == 1
assert llm.vision_calls == 2 # one vision call per page
doc = _pdf_doc(db, "garbage", "garbage.pdf")
assert doc.is_image is False and doc.image_path is None
assert doc.content == f"---PAGE-1---\n{d1}\n\n---PAGE-2---\n{d2}"
assert d1 != d2, "the pages' bytes differ — so do their replies"
# The digest runs over the FINAL (fallback) content — the
# phase-126 digest rule, unchanged.
assert doc.content_hash == hashlib.sha256(
doc.content.encode("utf-8")
).hexdigest()
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
content_chunks = [c for c in chunks if not c.is_summary]
assert [c.content for c in content_chunks] == [
f"Page 1:\n{d1}",
f"Page 2:\n{d2}",
]
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert doc.summary is not None
# The loud warning (PLAN §9, the FINDING's signature) — the
# required line, exact.
warnings = [
r for r in caplog.records if "text layer unusable" in r.getMessage()
]
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
assert warnings[0].getMessage() == (
"pdf: text layer unusable for garbage/garbage.pdf — "
"falling back to 2 page screenshots"
)
# The vision calls went to the CHAT model (the phase-122 seam).
assert llm.vision_models == [llm.settings.llm_chat_model] * 2
finally:
_cleanup_source(db, "garbage")
def test_clean_pdf_never_burns_a_fallback_vision_call(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The gate is the ONLY gate: a clean text layer never reaches the
fallback — ``pdfs_screenshotted`` stays 0 and the multimodal chat
is NEVER called (the images toggle is off, so the only vision seam
in play is the fallback itself; the phase-126 happy path is
byte-identical — the extracted text is the content, no fallback
warning, no ``Page N —`` placeholders)."""
root = tmp_path / "clean"
root.mkdir()
_build_one_page_pdf(root / "clean.pdf", "Clean prose about the homelab.")
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert summary.pdfs_screenshotted == 0
assert llm.vision_calls == 0
doc = _pdf_doc(db, "clean", "clean.pdf")
assert doc.content == "---PAGE-1---\nClean prose about the homelab."
assert not [
r for r in caplog.records if "text layer unusable" in r.getMessage()
]
finally:
_cleanup_source(db, "clean")
def test_fallback_is_fail_soft_per_page(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A dead vision call on ONE page → the deterministic placeholder
line (``page N — description unavailable``) for that page, the
other page still gets its description — the doc still lands (a
dead vision endpoint must not sink the doc, the phase-122
fail-soft spirit). The vision seam logged its failure (the
phase-122 warning); the fallback counted once."""
root = tmp_path / "partfail"
root.mkdir()
_build_garbage_pdf(root / "partfail.pdf", pages=2)
llm = _FirstPageFailsVisionLLM()
llm.settings = _pdf_settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.summarizer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert summary.pdfs_screenshotted == 1
assert llm.vision_calls == 2 # both pages were attempted
doc = _pdf_doc(db, "partfail", "partfail.pdf")
assert doc.content == (
"---PAGE-1---\npage 1 — description unavailable\n\n"
"---PAGE-2---\nPage two described fine."
)
# The phase-122 seam's failure warning (the dead page's signal).
assert [
r for r in caplog.records if "image description failed" in r.getMessage()
]
finally:
_cleanup_source(db, "partfail")
def test_fallback_all_pages_failed_lands_empty_content(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A garbage PDF whose vision calls ALL fail (the dead vision
endpoint) → the existing EMPTY-CONTENT path: the doc lands with
``""`` (the empty byte string's digest, no content chunks), the
fallback was still taken (``pdfs_screenshotted == 1`` — the gate
fired), and the phase-126 "no extractable text" warning fires —
the doc really is empty. A dead endpoint must not sink the sync."""
root = tmp_path / "allfail"
root.mkdir()
_build_garbage_pdf(root / "allfail.pdf", pages=1)
llm = _AlwaysFailsVisionLLM()
llm.settings = _pdf_settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert summary.pdfs_screenshotted == 1
assert llm.vision_calls == 1
doc = _pdf_doc(db, "allfail", "allfail.pdf")
assert doc.content == ""
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
assert [c for c in chunks if not c.is_summary] == []
warnings = [
r for r in caplog.records if "no extractable text" in r.getMessage()
]
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
finally:
_cleanup_source(db, "allfail")
def test_fallback_unrenderable_page_gets_the_placeholder(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A page that cannot be RENDERED (``b""`` at its index — the
:func:`render_pdf_pages` fail-soft slot) gets the deterministic
placeholder while the good pages keep their descriptions — the
page index never shifts (``---PAGE-N---`` stays truthful), and the
unrenderable page burns no vision call."""
root = tmp_path / "halfblind"
root.mkdir()
_build_garbage_pdf(root / "halfblind.pdf", pages=3)
monkeypatch.setattr(
importer,
"render_pdf_pages",
lambda _path, dpi=150: [b"\x89PNG-page-1", b"", b"\x89PNG-page-3"],
)
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path)
d1 = _PageVisionLLM.reply_for(b"\x89PNG-page-1")
d3 = _PageVisionLLM.reply_for(b"\x89PNG-page-3")
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert summary.pdfs_screenshotted == 1
assert llm.vision_calls == 2 # the blank page burned no vision call
doc = _pdf_doc(db, "halfblind", "halfblind.pdf")
assert doc.content == (
f"---PAGE-1---\n{d1}\n\n"
"---PAGE-2---\npage 2 — description unavailable\n\n"
f"---PAGE-3---\n{d3}"
)
finally:
_cleanup_source(db, "halfblind")
def test_fallback_rerun_lands_unchanged_no_reembed(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""Digest stability (the task's re-burn guard): the digest runs
over the constructed fallback content — a re-sync of an unchanged
garbage PDF (same pages + the deterministic fake's same replies)
re-imports as UNCHANGED: the fallback re-runs (the gate + the
render are cheap; the vision calls are re-issued — the documented
no-cache cost), but the digest MATCHES, so the row is skipped —
no re-chunk, no re-embed, no re-summary, no KB change (the row is
not ``updated`` and records no changed path). The doc's content
and digest are byte-stable across the re-sync."""
root = tmp_path / "stable"
root.mkdir()
_build_garbage_pdf(root / "stable.pdf")
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path)
try:
s1 = asyncio.run(import_sources([root], llm, session=db))
assert (s1.added, s1.pdfs_screenshotted) == (1, 1)
assert llm.vision_calls == 2
embeds1 = len(llm.calls)
doc1 = _pdf_doc(db, "stable", "stable.pdf")
content1, digest1 = doc1.content, doc1.content_hash
s2 = asyncio.run(import_sources([root], llm, session=db))
# The fallback re-ran (gate re-fired, vision re-issued) — but
# the row itself re-imported as unchanged (no re-embed, no
# re-chunk, no KB change: the stable replies give the same
# digest, so the upsert is skipped entirely).
assert (s2.added, s2.updated, s2.unchanged) == (0, 0, 1)
assert s2.pdfs_screenshotted == 1 # the gate re-fired
assert llm.vision_calls == 4 # the vision calls were re-issued
assert len(llm.calls) == embeds1, "unchanged: no re-embed"
assert s2.changed_paths == [], "unchanged: no KB change recorded"
doc2 = _pdf_doc(db, "stable", "stable.pdf")
assert doc2.content == content1 # byte-stable content
assert doc2.content_hash == digest1 # byte-stable digest
finally:
_cleanup_source(db, "stable")
def test_fallback_embedded_images_still_join_the_pipeline(
db, tmp_path: Path
) -> None:
"""Both, no conflict (the phase design): a fallback PDF with the
images toggle ON ALSO runs the phase-126 embedded-image pass — the
screenshots are the CONTENT (the fallback's vision calls: one per
page), the extracted images are the citation figures (the
phase-122 pass's vision calls: one per image). The content keeps
the page descriptions; the image doc is separate."""
root = tmp_path / "both"
root.mkdir()
# Garbage text (the gate fires) + an embedded image on page 2
# (the phase-126 A3 pass indexes it).
doc = fitz.open()
page1 = doc.new_page()
for i in range(10):
page1.insert_text((72, 72 + i * 14), f"<foo:bar>{i}</foo:bar>")
page2 = doc.new_page()
for i in range(10):
page2.insert_text((72, 72 + i * 14), f"<foo:baz>{i}</foo:baz>")
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
pix.clear_with(200)
page2.insert_image(fitz.Rect(72, 220, 112, 250), stream=pix.tobytes("png"))
doc.save(root / "both.pdf")
doc.close()
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=True)
try:
summary = asyncio.run(import_sources([root], llm, session=db))
# The PDF (screenshot content) + its one embedded image doc.
assert (summary.files, summary.added, summary.errors) == (1, 2, 0)
assert summary.pdfs_screenshotted == 1
assert summary.images_failed == 0
# 2 fallback page descriptions + 1 embedded-image description.
assert llm.vision_calls == 3
pdf_doc = _pdf_doc(db, "both", "both.pdf")
pngs = importer.render_pdf_pages(root / "both.pdf")
d1 = _PageVisionLLM.reply_for(pngs[0])
d2 = _PageVisionLLM.reply_for(pngs[1])
assert d1 != d2
assert pdf_doc.content == f"---PAGE-1---\n{d1}\n\n---PAGE-2---\n{d2}"
# The embedded image (the clear_with(200) 40x30 PNG) gets its
# OWN stable reply — the third vision call, the citation figure.
image_doc = _pdf_doc(db, "both", "both.pdf#page=2#img=0.png")
assert image_doc.is_image is True
assert image_doc.content != d1 and image_doc.content != d2
finally:
_cleanup_source(db, "both")
tests/unit/test_pdf.py
+296 -15
View File
@@ -12,6 +12,7 @@ from __future__ import annotations
import asyncio
import hashlib
import logging
import tarfile
from collections.abc import Callable, Sequence
from datetime import timedelta
from pathlib import Path
@@ -29,12 +30,17 @@ from app.rag import pdf as pdf_mod
from app.rag.importer import import_sources
from app.rag.llm import EmbeddingError, LLMError
from app.rag.pdf import (
_XMLISH_LINE_RE,
NONPRINT_CHAR_FRACTION,
PAGE_SEPARATOR,
XMLISH_LINE_FRACTION,
PdfImage,
_mime_from_image_info,
_page_image_entries,
extract_pdf_images,
extract_pdf_text,
is_usable_pdf_text,
render_pdf_pages,
)
from tests.fakes import FakeEmbedder
@@ -475,13 +481,20 @@ def test_pdf_text_change_updates_doc_and_keeps_unchanged_images(
_cleanup_source(db, "pdfchg")
def test_pdf_without_text_still_becomes_a_document(
def test_pdf_without_text_takes_the_screenshot_fallback(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""An image-only PDF (no extractable text) still becomes a document
— with empty content (the empty byte string's digest), no content
chunks, and a warning (the signal). The summary path still runs
(best-effort on the empty content)."""
"""Phase 133 (task 03): an image-only PDF (an EMPTY text layer — the
gate's other target) takes the page-screenshot fallback: the single
page is rendered to a PNG and vision-described (the mock's canned
:data:`DESCRIPTION`), and the doc lands with the ``---PAGE-N---``-
anchored description as content (chunked + summarized as any other
PDF). The phase-126 empty-content behavior is retired for layers
that CAN be screenshotted — it survives only for an unopenable
file (``test_corrupt_pdf_file_becomes_an_empty_document``), where
the render fails too. The loud fallback warning fires; the phase-
126 "no extractable text" warning does NOT (the doc is not empty).
"""
root = tmp_path / "pdfempty"
root.mkdir()
_build_pdf(root / "pics.pdf", [("", [_png_bytes(40, 30)])])
@@ -490,41 +503,61 @@ def test_pdf_without_text_still_becomes_a_document(
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added) == (1, 1)
assert summary.pdfs_screenshotted == 1
doc = _doc(db, "pdfempty", "pics.pdf")
assert doc.content == ""
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
assert doc.content == f"{PAGE_SEPARATOR.format(page=1)}\n{DESCRIPTION}"
assert doc.content_hash == hashlib.sha256(doc.content.encode("utf-8")).hexdigest()
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
assert [c for c in chunks if not c.is_summary] == []
assert doc.summary is not None # best-effort on the empty content
content_chunks = [c for c in chunks if not c.is_summary]
assert [c.content for c in content_chunks] == [f"Page 1:\n{DESCRIPTION}"]
assert doc.summary is not None
# The loud fallback warning (the FINDING's signature) fired —
# and the phase-126 "no extractable text" warning did NOT (the
# doc is not empty — the screenshots ARE the content).
warnings = [
r
for r in caplog.records
if "no extractable text" in r.getMessage()
r for r in caplog.records if "text layer unusable" in r.getMessage()
]
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
assert "falling back to 1 page screenshots" in warnings[0].getMessage()
assert not [
r for r in caplog.records if "no extractable text" in r.getMessage()
]
finally:
_cleanup_source(db, "pdfempty")
def test_corrupt_pdf_file_becomes_an_empty_document(db, tmp_path: Path) -> None:
def test_corrupt_pdf_file_becomes_an_empty_document(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A .pdf file that is not a PDF at all: the parser's fail-soft
contract (no text, no images) — the document lands with empty
content and no chunks, and the sync stays green."""
contract (no text, no images) AND the phase-133 fallback's
render-less branch (``render_pdf_pages`` → ``[]`` → the helper's
``""``) — the document lands with empty content (the empty byte
string's digest) and no chunks, the fallback is still counted
(the gate fired, the render produced nothing), and the phase-126
"no extractable text" warning fires — the doc really is empty."""
root = tmp_path / "pdfbad"
root.mkdir()
(root / "broken.pdf").write_bytes(b"this is not a pdf at all")
llm = _pdf_llm(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert summary.pdfs_screenshotted == 1 # the gate fired
doc = _doc(db, "pdfbad", "broken.pdf")
assert doc.content == "" and doc.title == "broken"
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
assert [c for c in chunks if not c.is_summary] == []
warnings = [
r for r in caplog.records if "no extractable text" in r.getMessage()
]
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
finally:
_cleanup_source(db, "pdfbad")
@@ -1074,3 +1107,251 @@ def test_page_image_entries_falls_back_on_old_pymupdf() -> None:
)
entries = _page_image_entries(page)
assert entries == [{"xref": 7, "width": 40, "height": 30}]
# ---------------------------------------------------------------------------
# is_usable_pdf_text (phase 133 task 02 — the deterministic garbage gate)
# ---------------------------------------------------------------------------
#: The owner's untracked fixture (the stable reference — the same
#: provenance as the phase-133 task-01 integration regression pin:
#: the tarball, never the extracted PDF, is what the test starts
#: from, and it never touches the repo).
TARBALL = Path(__file__).resolve().parents[2] / "test_sources" / "hnb_2026_guide.tar.gz"
#: The inner file's name (it keeps its name through extraction — the
#: task-01 convention).
PDF_NAME = "HBI-2026-Guide-Year-Round_260704_131221.pdf"
@pytest.fixture(scope="module")
def pre126_garbage_text(tmp_path_factory: pytest.TempPathFactory) -> str:
"""THE REAL GARBAGE (the task's airtight pin, module-scoped):
the owner's exact PDF read with the PRE-PHASE-126 recipe —
``read_text(encoding="utf-8", errors="replace")`` +
``.replace("\\x00", "")`` — the 46%-XML-ish, U+FFFD-heavy soup
that was chunked and embedded before the parser shipped (the
FINDING: 138,170 non-empty lines, ~46% XML-ish, the owner's
``<illustrator:StartupProfile>`` preview). The tarball's single
member is extracted into a temp dir (the task-01 integration
pin's pattern, shared — the fixture stays out of the repo)."""
if not TARBALL.is_file():
pytest.skip(
"test_sources/hnb_2026_guide.tar.gz is missing — the "
"untracked owner fixture (2.6 MB, one 48-page Illustrator PDF)"
)
root = tmp_path_factory.mktemp("pre126")
with tarfile.open(TARBALL, "r:gz") as tar:
tar.extractall(root, filter="data")
member = root / PDF_NAME
return member.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
def test_gate_clean_text_with_page_separators_is_usable() -> None:
"""Real prose + the phase-126 ``---PAGE-N---`` separators: the
happy path the gate must NEVER block (a clean PDF never burns a
vision call — the gate is the only gate, and the happy path stays
byte-identical to phase 126)."""
clean = (
f"{PAGE_SEPARATOR.format(page=1)}\n2026 U.S. Benefits Guide\n"
"\n"
"Covered employees receive comprehensive medical, dental, and\n"
"vision benefits under this plan document.\n"
"\n"
f"{PAGE_SEPARATOR.format(page=2)}\nPremiums are deducted from\n"
"each paycheck; the plan year runs January 1 through\n"
"December 31."
)
assert is_usable_pdf_text(clean) is True
def test_gate_empty_text_layer_is_unusable() -> None:
"""The empty text layer — a scanned/image-only PDF has nothing to
read — is the fallback's other target."""
assert is_usable_pdf_text("") is False
assert is_usable_pdf_text(" \n\t\n \r\n") is False
def test_gate_xml_heavy_layer_is_unusable() -> None:
"""The raw-XMP / binary-soup signature: ≥ 30% of the non-empty
lines look like tags (``<tag>``, ``</tag>``, ``<tag attr=`` — the
regex covers them all, including the PDF-structure ``<</Key…``
noise)."""
tags = [f"<xmp:Prop{i}>value</xmp:Prop{i}>" for i in range(30)]
prose = [f"Line {i} of ordinary prose." for i in range(70)]
assert is_usable_pdf_text("\n".join(tags + prose)) is False # 30/100
def test_gate_borderline_document_about_xml_stays_usable() -> None:
"""ASSUMPTION A8's edge from below: a real document ABOUT XML
whose tag-like lines stay UNDER the 0.30 bar (29%) is usable —
the documented false-positive cost is paid only at/above the bar."""
tags = [f"<example:{i}>text</example:{i}>" for i in range(29)]
prose = [
f"Paragraph {i} explains the markup in plain words."
for i in range(71)
]
assert is_usable_pdf_text("\n".join(tags + prose)) is True # 29/100
def test_gate_at_and_above_the_xml_bar_is_unusable() -> None:
"""The threshold's edges from above: EXACTLY 30% tag-like lines is
already unusable (the bar is ≥, not >), and 31% is too — a document
ABOUT XML that reaches the bar pays the documented ASSUMPTION A8
cost (the fallback still extracts it correctly via vision)."""
for n in (30, 31):
tags = [f"<example:{i}>text</example:{i}>" for i in range(n)]
prose = [
f"Paragraph {i} of ordinary explanation."
for i in range(100 - n)
]
assert is_usable_pdf_text("\n".join(tags + prose)) is False
def test_gate_replacement_char_soup_is_unusable() -> None:
"""The SECOND signal, alone: a ≥ 5% U+FFFD load (the
``errors="replace"`` binary-soup signal) is unusable even with few
tag-like lines — the XML-ish gate does not mask it, and this soup
has none (the two failure signals fire independently)."""
soup = "Benefits summary paragraph. " * 24 + "\ufffd" * 60 # ~7.9%
assert is_usable_pdf_text(soup) is False
def test_gate_structural_newlines_and_tabs_are_printable() -> None:
"""The documented exclusion: the ``\\n``/``\\t``/``\\r`` the
extractor legitimately emits are NOT non-printable — a
newline/tab/indent-heavy document is printable (counting them
would flag every prose document)."""
text = "\n\t".join(f"Indented line {i} with a tab." for i in range(20))
assert is_usable_pdf_text(text) is True
def test_gate_real_pre126_garbage_is_unusable(
pre126_garbage_text: str,
) -> None:
"""THE REAL GARBAGE (the airtight pin): the owner's exact file,
read with the pre-126 recipe (the module fixture) — the
reproduced 46%-XML-ish, U+FFFD-heavy soup — must be flagged
unusable. Both signals fire on the REAL string (the FINDING's
evidence, pinned against the named thresholds: if the owner ever
re-cuts the fixture, the drift shows up here — the pin has teeth,
the same contract as the task-01 suite)."""
assert is_usable_pdf_text(pre126_garbage_text) is False
# The owner's exact preview line is in the reproduced soup.
assert "<illustrator:StartupProfile>" in pre126_garbage_text
lines = [line for line in pre126_garbage_text.splitlines() if line.strip()]
xml_frac = sum(1 for line in lines if _XMLISH_LINE_RE.search(line)) / len(lines)
nonprint_frac = sum(
1
for ch in pre126_garbage_text
if ch == "\ufffd" or (ord(ch) < 32 and ch not in "\t\n\r")
) / len(pre126_garbage_text)
# Both signals independently clear their named bars on the real
# string — neither signal can silently die.
assert xml_frac >= XMLISH_LINE_FRACTION
assert nonprint_frac >= NONPRINT_CHAR_FRACTION
# ---------------------------------------------------------------------------
# render_pdf_pages (phase 133 task 03 — the fallback's visual input)
# ---------------------------------------------------------------------------
def _two_page_text_pdf(path: Path) -> Path:
"""A 2-page temp PDF with EXPLICIT letter pages (612x792pt —
deterministic render sizes: 1275x1650 px at 150 DPI, 612x792 at
72 DPI) and page-specific text (the render must be page-specific,
so the pages' PNG bytes must differ)."""
doc = fitz.open()
doc.new_page(width=612, height=792).insert_text((72, 72), "First page marker text.")
doc.new_page(width=612, height=792).insert_text((72, 72), "Second page marker text.")
doc.save(path)
doc.close()
return path
def test_render_pdf_pages_returns_one_png_per_page_in_order(
tmp_path: Path,
) -> None:
"""A 2-page PDF → 2 PNGs in PAGE ORDER (list index i is page i+1),
each a genuine decodable PNG at the default 150-DPI render (612x792pt
letter → 1275x1650 px), and page-specific (page 1's bytes != page
2's — different text on the page)."""
path = _two_page_text_pdf(tmp_path / "two.pdf")
pages = render_pdf_pages(path)
assert len(pages) == 2
for png in pages:
assert png.startswith(PNG_MAGIC)
pix = fitz.Pixmap(png)
assert (pix.width, pix.height) == (1275, 1650)
assert pages[0] != pages[1] # the render is page-specific
def test_render_pdf_pages_honors_the_dpi_argument(tmp_path: Path) -> None:
"""ASSUMPTION A8's fixed 150 DPI is the DEFAULT; an explicit *dpi*
changes the render (72 DPI → the page's natural 612x792 px)."""
path = _two_page_text_pdf(tmp_path / "two.pdf")
pix = fitz.Pixmap(render_pdf_pages(path, dpi=72)[0])
assert (pix.width, pix.height) == (612, 792)
def test_render_pdf_pages_unopenable_file_returns_empty_list(
tmp_path: Path, caplog: pytest.LogCaptureFixture,
) -> None:
"""A file that cannot be OPENED yields ``[]`` (the caller's existing
empty-content path — the doc lands with the no-extractable-text
behavior, unchanged) + a warning (the fail-soft style)."""
bad = tmp_path / "bad.pdf"
bad.write_bytes(b"this is not a pdf at all")
with caplog.at_level(logging.WARNING, logger="app.rag.pdf"):
assert render_pdf_pages(bad) == []
assert render_pdf_pages(tmp_path / "missing.pdf") == []
warnings = [
r for r in caplog.records if "could not open" in r.getMessage()
]
assert len(warnings) == 2 and all(w.levelno == logging.WARNING for w in warnings)
def test_render_pdf_pages_zero_page_doc_returns_empty_list(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
) -> None:
"""A zero-page PDF (openable, no pages — this PyMuPDF refuses to
SAVE such a file, so it is stubbed) yields ``[]`` — the loop runs
zero times, nothing renders."""
path = tmp_path / "zero.pdf"
path.write_bytes(b"%PDF-1.4 fake")
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: _FakeDoc([]))
assert render_pdf_pages(path) == []
def test_render_pdf_pages_broken_page_is_zero_bytes_at_its_index(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path,
) -> None:
"""A page whose get_pixmap raises → ``b""`` at ITS index (the
caller's placeholder slot — the page index NEVER shifts, so
``---PAGE-N---`` stays truthful); the good pages keep their PNGs
at their own indices, and the failure is logged (the
:func:`extract_pdf_text` fail-soft style)."""
path = tmp_path / "fake.pdf"
path.write_bytes(b"%PDF-1.4 fake")
class _RenderPage:
def __init__(self, data: bytes | None) -> None:
self._data = data
def get_pixmap(self, dpi: int) -> Any:
if self._data is None:
raise RuntimeError("boom: page render failed")
return SimpleNamespace(tobytes=lambda _fmt: self._data)
doc = _FakeDoc(
[
_RenderPage(b"\x89PNG-fake-page-1"),
_RenderPage(None),
_RenderPage(b"\x89PNG-fake-page-3"),
]
)
monkeypatch.setattr(pdf_mod.fitz, "open", lambda _p: doc)
assert render_pdf_pages(path) == [
b"\x89PNG-fake-page-1", b"", b"\x89PNG-fake-page-3",
]