phase: 137_pdf_image_context
Build and Push Containers / build-and-push-app (push) Successful in 2m51s
Build and Push Containers / build-and-push-db (push) Successful in 13s

All verification complete — everything is 100% green. Final report:

**Phase 137 final verification pass — all tasks already complete, all criteria verified**

- **Full suite:** `uv run pytest --cov=app --cov-report=term-missing` → 3272 passed, exit 0; `app/` coverage **99%** (>90% ✓)
- **Phase E2E in isolation:** `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov` → **4 passed** (forward context, display line, standalone parity, both-blocks markdown case)
- **Pre-existing suites in isolation:** `test_pdf_support.py` + `test_image_anchor_context.py` + `test_image_documents.py` → **14 passed**; ripple-edited `test_garbage_pdf.py` → **3 passed**
- **Lint/types:** `uv run ruff check .` → all passed; `uv run pyright` → 0 errors

**Completion criteria:** (1) forward-path context in vision msg + stored `image_context`/summary — ✓ (unit builder/seam + integration fake-LLM + E2E); (2) both blocks document-first + standalone describe byte-identical — ✓ (combiner + standalone pins); (3) backfill with context + zero-vision idempotent re-sync + parent-gone origin-only — ✓ (6 integration tests); (4) suggestion block leads with `From <source>/<pdf> (page N)` — ✓ (renderer units + E2E); (5) all gates green — ✓ above; (6) no behavior change in phases 122/125/126 + sync `detail` shape untouched — ✓ (verified in code: new counter is log-only); (7) ruff+pyright — ✓; (8) commit — left to the harness per pipeline rules (working tree untouched by me).

**Documentation:** README.md audited in full — the new phase-137 bullet verified against code (6 000-char cap, log-only `image_context_backfilled=N`, `<document>`-body locator placement), all env tables/defaults cross-checked vs `app/config.py` and `.env.example` — consistent, no changes needed. AGENTS.md: every rule/command verified — consistent, no changes needed.

**Notable deviations (pre-existing, documented in task-05 report, forced by design):** `test_pdf_support.py`/`test_garbage_pdf.py` got minimal ripple edits (the "UNEDITED" pin was unsatisfiable once describe calls carry context by design); the unit `test_image_anchor_context.py` CONTEXT_PREFIX pin generalized per ASSUMPTION A5; one extra mock line (`_seed_context_line` accepts `From …`).

**Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-29 02:14:46 -04:00
parent 0035bdf533
commit 5e87622b30
39 changed files with 3565 additions and 167 deletions
+64 -15
View File
@@ -10,6 +10,12 @@ package, imported as ``fitz`` — its stable API name).
these markers so every chunk keeps its page anchor. Pages without
text (image-only pages) contribute nothing, so a PDF with no
extractable text returns ``""``.
* :func:`extract_pdf_page_texts` — one entry per page that has text:
1-indexed page number → the page's stripped text (the same page read
as :func:`extract_pdf_text`, so the two agree page-for-page; the
value for page N is exactly that page's separator-block body). The
phase-137 image-context block is built from this map (page N's entry
is the document context for an image embedded on page N).
* :func:`extract_pdf_images` — one :class:`PdfImage` per embedded-image
occurrence, in page order then 0-based index on the page. ``data`` is
the image's original encoded bytes as stored in the PDF (no
@@ -37,6 +43,7 @@ from __future__ import annotations
import logging
import re
from collections.abc import Iterator
from dataclasses import dataclass
from pathlib import Path
from typing import Any
@@ -117,6 +124,32 @@ def _page_image_entries(page: Any) -> list[dict[str, Any]]:
return [dict(info) for info in infos]
def _iter_page_texts(doc: Any, pdf_path: Path) -> Iterator[tuple[int, str]]:
"""Yield ``(page, text)`` for every page of *doc* that has text.
``page`` is the 1-indexed page number and ``text`` the page's
stripped text (the ``get_text("text")`` read — it always returns a
str; the union return type comes from pymupdf's other flavors,
dict/words/blocks). A page that cannot be read is logged and
skipped; a page without text contributes nothing. The shared page
iteration behind :func:`extract_pdf_text` and
:func:`extract_pdf_page_texts` — the two callers share the exact
same read and cannot drift.
"""
for page_num in range(len(doc)):
try:
raw = doc[page_num].get_text("text")
except Exception as exc:
logger.warning(
"pdf: could not read text of page %d of %s — %s",
page_num + 1, pdf_path, exc,
)
continue
text = raw.strip() if isinstance(raw, str) else ""
if text:
yield page_num + 1, text
def extract_pdf_text(pdf_path: Path) -> str:
"""Extract the text of all pages of *pdf_path*.
@@ -134,25 +167,41 @@ def extract_pdf_text(pdf_path: Path) -> str:
logger.warning("pdf: could not open %s — %s", pdf_path, exc)
return ""
parts: list[str] = []
with doc:
for page_num in range(len(doc)):
try:
raw = doc[page_num].get_text("text")
except Exception as exc:
logger.warning(
"pdf: could not read text of page %d of %s — %s",
page_num + 1, pdf_path, exc,
)
continue
# get_text("text") always returns a str; the union return type
# comes from pymupdf's other flavors (dict/words/blocks).
text = raw.strip() if isinstance(raw, str) else ""
if text:
parts.append(f"{PAGE_SEPARATOR.format(page=page_num + 1)}\n{text}")
parts = [
f"{PAGE_SEPARATOR.format(page=page)}\n{text}"
for page, text in _iter_page_texts(doc, pdf_path)
]
return "\n\n".join(parts)
def extract_pdf_page_texts(pdf_path: Path) -> dict[int, str]:
"""One entry per page of *pdf_path* that has text.
Keys are the 1-indexed page numbers, values the page's stripped
text — the shared :func:`_iter_page_texts` read, so this map agrees
with :func:`extract_pdf_text` page-for-page: the value for page N
is exactly the text after that page's ``---PAGE-N---`` separator.
Pages without text (image-only pages) contribute nothing — no key.
The phase-137 document context for an embedded image is built from
this map: one image on page N gets page N's text as the context
block's body.
Fail-soft: an unreadable file returns ``{}`` (one warning — the
:func:`extract_pdf_text` open contract); a page that cannot be
read is logged and skipped.
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as exc:
logger.warning("pdf: could not open %s for page texts — %s", pdf_path, exc)
return {}
with doc:
return dict(_iter_page_texts(doc, pdf_path))
#: A line "looks like XML" when it carries a tag-shaped token — the
#: pre-126 garbage's signature (raw XMP blocks, PDF-structure noise).
_XMLISH_LINE_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")