phase: 137_pdf_image_context
All verification complete — everything is 100% green. Final report: **Phase 137 final verification pass — all tasks already complete, all criteria verified** - **Full suite:** `uv run pytest --cov=app --cov-report=term-missing` → 3272 passed, exit 0; `app/` coverage **99%** (>90% ✓) - **Phase E2E in isolation:** `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov` → **4 passed** (forward context, display line, standalone parity, both-blocks markdown case) - **Pre-existing suites in isolation:** `test_pdf_support.py` + `test_image_anchor_context.py` + `test_image_documents.py` → **14 passed**; ripple-edited `test_garbage_pdf.py` → **3 passed** - **Lint/types:** `uv run ruff check .` → all passed; `uv run pyright` → 0 errors **Completion criteria:** (1) forward-path context in vision msg + stored `image_context`/summary — ✓ (unit builder/seam + integration fake-LLM + E2E); (2) both blocks document-first + standalone describe byte-identical — ✓ (combiner + standalone pins); (3) backfill with context + zero-vision idempotent re-sync + parent-gone origin-only — ✓ (6 integration tests); (4) suggestion block leads with `From <source>/<pdf> (page N)` — ✓ (renderer units + E2E); (5) all gates green — ✓ above; (6) no behavior change in phases 122/125/126 + sync `detail` shape untouched — ✓ (verified in code: new counter is log-only); (7) ruff+pyright — ✓; (8) commit — left to the harness per pipeline rules (working tree untouched by me). **Documentation:** README.md audited in full — the new phase-137 bullet verified against code (6 000-char cap, log-only `image_context_backfilled=N`, `<document>`-body locator placement), all env tables/defaults cross-checked vs `app/config.py` and `.env.example` — consistent, no changes needed. AGENTS.md: every rule/command verified — consistent, no changes needed. **Notable deviations (pre-existing, documented in task-05 report, forced by design):** `test_pdf_support.py`/`test_garbage_pdf.py` got minimal ripple edits (the "UNEDITED" pin was unsatisfiable once describe calls carry context by design); the unit `test_image_anchor_context.py` CONTEXT_PREFIX pin generalized per ASSUMPTION A5; one extra mock line (`_seed_context_line` accepts `From …`). **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+64
-15
@@ -10,6 +10,12 @@ package, imported as ``fitz`` — its stable API name).
|
||||
these markers so every chunk keeps its page anchor. Pages without
|
||||
text (image-only pages) contribute nothing, so a PDF with no
|
||||
extractable text returns ``""``.
|
||||
* :func:`extract_pdf_page_texts` — one entry per page that has text:
|
||||
1-indexed page number → the page's stripped text (the same page read
|
||||
as :func:`extract_pdf_text`, so the two agree page-for-page; the
|
||||
value for page N is exactly that page's separator-block body). The
|
||||
phase-137 image-context block is built from this map (page N's entry
|
||||
is the document context for an image embedded on page N).
|
||||
* :func:`extract_pdf_images` — one :class:`PdfImage` per embedded-image
|
||||
occurrence, in page order then 0-based index on the page. ``data`` is
|
||||
the image's original encoded bytes as stored in the PDF (no
|
||||
@@ -37,6 +43,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -117,6 +124,32 @@ def _page_image_entries(page: Any) -> list[dict[str, Any]]:
|
||||
return [dict(info) for info in infos]
|
||||
|
||||
|
||||
def _iter_page_texts(doc: Any, pdf_path: Path) -> Iterator[tuple[int, str]]:
|
||||
"""Yield ``(page, text)`` for every page of *doc* that has text.
|
||||
|
||||
``page`` is the 1-indexed page number and ``text`` the page's
|
||||
stripped text (the ``get_text("text")`` read — it always returns a
|
||||
str; the union return type comes from pymupdf's other flavors,
|
||||
dict/words/blocks). A page that cannot be read is logged and
|
||||
skipped; a page without text contributes nothing. The shared page
|
||||
iteration behind :func:`extract_pdf_text` and
|
||||
:func:`extract_pdf_page_texts` — the two callers share the exact
|
||||
same read and cannot drift.
|
||||
"""
|
||||
for page_num in range(len(doc)):
|
||||
try:
|
||||
raw = doc[page_num].get_text("text")
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"pdf: could not read text of page %d of %s — %s",
|
||||
page_num + 1, pdf_path, exc,
|
||||
)
|
||||
continue
|
||||
text = raw.strip() if isinstance(raw, str) else ""
|
||||
if text:
|
||||
yield page_num + 1, text
|
||||
|
||||
|
||||
def extract_pdf_text(pdf_path: Path) -> str:
|
||||
"""Extract the text of all pages of *pdf_path*.
|
||||
|
||||
@@ -134,25 +167,41 @@ def extract_pdf_text(pdf_path: Path) -> str:
|
||||
logger.warning("pdf: could not open %s — %s", pdf_path, exc)
|
||||
return ""
|
||||
|
||||
parts: list[str] = []
|
||||
with doc:
|
||||
for page_num in range(len(doc)):
|
||||
try:
|
||||
raw = doc[page_num].get_text("text")
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"pdf: could not read text of page %d of %s — %s",
|
||||
page_num + 1, pdf_path, exc,
|
||||
)
|
||||
continue
|
||||
# get_text("text") always returns a str; the union return type
|
||||
# comes from pymupdf's other flavors (dict/words/blocks).
|
||||
text = raw.strip() if isinstance(raw, str) else ""
|
||||
if text:
|
||||
parts.append(f"{PAGE_SEPARATOR.format(page=page_num + 1)}\n{text}")
|
||||
parts = [
|
||||
f"{PAGE_SEPARATOR.format(page=page)}\n{text}"
|
||||
for page, text in _iter_page_texts(doc, pdf_path)
|
||||
]
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def extract_pdf_page_texts(pdf_path: Path) -> dict[int, str]:
|
||||
"""One entry per page of *pdf_path* that has text.
|
||||
|
||||
Keys are the 1-indexed page numbers, values the page's stripped
|
||||
text — the shared :func:`_iter_page_texts` read, so this map agrees
|
||||
with :func:`extract_pdf_text` page-for-page: the value for page N
|
||||
is exactly the text after that page's ``---PAGE-N---`` separator.
|
||||
Pages without text (image-only pages) contribute nothing — no key.
|
||||
|
||||
The phase-137 document context for an embedded image is built from
|
||||
this map: one image on page N gets page N's text as the context
|
||||
block's body.
|
||||
|
||||
Fail-soft: an unreadable file returns ``{}`` (one warning — the
|
||||
:func:`extract_pdf_text` open contract); a page that cannot be
|
||||
read is logged and skipped.
|
||||
"""
|
||||
try:
|
||||
doc = fitz.open(str(pdf_path))
|
||||
except Exception as exc:
|
||||
logger.warning("pdf: could not open %s for page texts — %s", pdf_path, exc)
|
||||
return {}
|
||||
|
||||
with doc:
|
||||
return dict(_iter_page_texts(doc, pdf_path))
|
||||
|
||||
|
||||
#: A line "looks like XML" when it carries a tag-shaped token — the
|
||||
#: pre-126 garbage's signature (raw XMP blocks, PDF-structure noise).
|
||||
_XMLISH_LINE_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
|
||||
|
||||
Reference in New Issue
Block a user