phase: 126_pdf_support
All gates verified green — no code changes were needed on this pass. Final report: ## Phase 126 — PDF support: final verification pass **Verified (all tasks 01–05 already complete in working tree):** - `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token - `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk **Test / lint / coverage results:** - `uv run pytest` → 2997 passed - `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** 1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests) 2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test) 3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites) 4. PDF renders as extracted text in viewer — ✅ (E2E) 5. Test/coverage/lint gates — ✅ (above) 6. Commit + phase move — harness handles (per protocol, no `git` on this pass) **Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree. **Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
@@ -26,6 +26,11 @@ per-format policies:
|
||||
endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and
|
||||
Jinja templates; plain-text paragraph packing (``chunk_text``) — no
|
||||
format-specific splitter (owner decision).
|
||||
* **pdf** (phase 126, LOCKED A2) — page-aware: the extracted text's
|
||||
``---PAGE-N---`` markers are the primary split boundary (one logical
|
||||
unit per text-bearing page; a long page sub-splits through the
|
||||
paragraph packer), every chunk keeps its ``Page N:`` anchor, and the
|
||||
markers themselves never appear in a chunk.
|
||||
|
||||
Every format honors :data:`HARD_MAX_CHARS` (1200 — the aipi ~1024-token
|
||||
request cap) and the target/overlap settings; oversized blocks are split
|
||||
@@ -324,6 +329,47 @@ def chunk_text(content: str, target_chars: int = 2000, overlap_chars: int = 200)
|
||||
return _pack_blocks(_paragraph_blocks(lines, [False] * len(lines)), target, overlap)
|
||||
|
||||
|
||||
#: The page marker of the extracted PDF text (phase 126, A2 —
|
||||
#: ``app.rag.pdf.extract_pdf_text`` prefixes every text-bearing page
|
||||
#: block with it; the number is the 1-indexed page number).
|
||||
_PDF_PAGE_MARKER_RE = re.compile(r"---PAGE-(\d+)---")
|
||||
|
||||
|
||||
def _chunk_pdf(content: str, target_chars: int = 2000, overlap_chars: int = 200) -> list[str]:
|
||||
"""Page-aware chunking of one PDF's extracted text (phase 126, A2).
|
||||
|
||||
A page is the primary split boundary: each ``---PAGE-N---`` block is
|
||||
packed on its own (the plain-text paragraph packer — PDFs have no
|
||||
code fences, so paragraph boundaries = blank lines) into chunks of
|
||||
at most ``target_chars`` with the ``overlap_chars`` carry-over, and
|
||||
every chunk of the page is prefixed with the ``Page N:`` anchor so a
|
||||
hit is readable in context. The marker number is the page number
|
||||
(blank pages carry no marker — the numbering follows the markers,
|
||||
not the chunk count). The markers themselves never appear in a
|
||||
chunk, and the anchor-aware budget keeps the anchored chunk under
|
||||
the target — :data:`HARD_MAX_CHARS` included.
|
||||
"""
|
||||
target, overlap = _normalize_target_overlap(target_chars, overlap_chars)
|
||||
# re.split with a capturing group keeps the markers: the pieces are
|
||||
# [before-first, page, text, page, text, …].
|
||||
pieces = _PDF_PAGE_MARKER_RE.split(content)
|
||||
chunks: list[str] = []
|
||||
for i in range(1, len(pieces) - 1, 2):
|
||||
page = int(pieces[i])
|
||||
text = pieces[i + 1].strip()
|
||||
if not text:
|
||||
continue
|
||||
anchor = f"Page {page}:"
|
||||
# Budget the anchor out of the page's packing target so the
|
||||
# anchored chunk cannot exceed ``target`` (the markdown
|
||||
# section-anchor precedent, the ``budget`` rule).
|
||||
budget = max(1, target - len(anchor) - 1)
|
||||
chunks.extend(
|
||||
f"{anchor}\n{part}" for part in chunk_text(text, budget, overlap)
|
||||
)
|
||||
return chunks
|
||||
|
||||
|
||||
def _normalize_target_overlap(target_chars: int, overlap_chars: int) -> tuple[int, int]:
|
||||
"""Validate + clamp the size policy (shared by every format)."""
|
||||
if target_chars <= 0:
|
||||
@@ -358,6 +404,9 @@ _FORMAT_CHUNKERS = {
|
||||
".os": chunk_text,
|
||||
".endpoint": chunk_text,
|
||||
".j2": chunk_text,
|
||||
# Phase 126 (LOCKED A2): the page-aware PDF chunker — the extracted
|
||||
# text's ``---PAGE-N---`` markers are the primary split boundary.
|
||||
".pdf": _chunk_pdf,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user