phase: 126_pdf_support
Build and Push Containers / build-and-push-app (push) Failing after 2m36s
Build and Push Containers / build-and-push-db (push) Successful in 1m39s

All gates verified green — no code changes were needed on this pass. Final report:

## Phase 126 — PDF support: final verification pass

**Verified (all tasks 01–05 already complete in working tree):**
- `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token
- `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk

**Test / lint / coverage results:**
- `uv run pytest` → 2997 passed
- `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:**
1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests)
2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test)
3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites)
4. PDF renders as extracted text in viewer — ✅ (E2E)
5. Test/coverage/lint gates — ✅ (above)
6. Commit + phase move — harness handles (per protocol, no `git` on this pass)

**Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree.

**Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
2026-09-27 09:36:40 -04:00
parent e0307fe3ff
commit e0f11e7aa8
40 changed files with 4092 additions and 0 deletions
+49
View File
@@ -26,6 +26,11 @@ per-format policies:
endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and
Jinja templates; plain-text paragraph packing (``chunk_text``) — no
format-specific splitter (owner decision).
* **pdf** (phase 126, LOCKED A2) — page-aware: the extracted text's
``---PAGE-N---`` markers are the primary split boundary (one logical
unit per text-bearing page; a long page sub-splits through the
paragraph packer), every chunk keeps its ``Page N:`` anchor, and the
markers themselves never appear in a chunk.
Every format honors :data:`HARD_MAX_CHARS` (1200 — the aipi ~1024-token
request cap) and the target/overlap settings; oversized blocks are split
@@ -324,6 +329,47 @@ def chunk_text(content: str, target_chars: int = 2000, overlap_chars: int = 200)
return _pack_blocks(_paragraph_blocks(lines, [False] * len(lines)), target, overlap)
#: The page marker of the extracted PDF text (phase 126, A2 —
#: ``app.rag.pdf.extract_pdf_text`` prefixes every text-bearing page
#: block with it; the number is the 1-indexed page number).
_PDF_PAGE_MARKER_RE = re.compile(r"---PAGE-(\d+)---")
def _chunk_pdf(content: str, target_chars: int = 2000, overlap_chars: int = 200) -> list[str]:
"""Page-aware chunking of one PDF's extracted text (phase 126, A2).
A page is the primary split boundary: each ``---PAGE-N---`` block is
packed on its own (the plain-text paragraph packer — PDFs have no
code fences, so paragraph boundaries = blank lines) into chunks of
at most ``target_chars`` with the ``overlap_chars`` carry-over, and
every chunk of the page is prefixed with the ``Page N:`` anchor so a
hit is readable in context. The marker number is the page number
(blank pages carry no marker — the numbering follows the markers,
not the chunk count). The markers themselves never appear in a
chunk, and the anchor-aware budget keeps the anchored chunk under
the target — :data:`HARD_MAX_CHARS` included.
"""
target, overlap = _normalize_target_overlap(target_chars, overlap_chars)
# re.split with a capturing group keeps the markers: the pieces are
# [before-first, page, text, page, text, …].
pieces = _PDF_PAGE_MARKER_RE.split(content)
chunks: list[str] = []
for i in range(1, len(pieces) - 1, 2):
page = int(pieces[i])
text = pieces[i + 1].strip()
if not text:
continue
anchor = f"Page {page}:"
# Budget the anchor out of the page's packing target so the
# anchored chunk cannot exceed ``target`` (the markdown
# section-anchor precedent, the ``budget`` rule).
budget = max(1, target - len(anchor) - 1)
chunks.extend(
f"{anchor}\n{part}" for part in chunk_text(text, budget, overlap)
)
return chunks
def _normalize_target_overlap(target_chars: int, overlap_chars: int) -> tuple[int, int]:
"""Validate + clamp the size policy (shared by every format)."""
if target_chars <= 0:
@@ -358,6 +404,9 @@ _FORMAT_CHUNKERS = {
".os": chunk_text,
".endpoint": chunk_text,
".j2": chunk_text,
# Phase 126 (LOCKED A2): the page-aware PDF chunker — the extracted
# text's ``---PAGE-N---`` markers are the primary split boundary.
".pdf": _chunk_pdf,
}