Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 126_pdf_support
Build and Push Containers / build-and-push-app (push) Failing after 2m36s
Details
Build and Push Containers / build-and-push-db (push) Successful in 1m39s
Details

Browse Source 
All gates verified green — no code changes were needed on this pass. Final report:

## Phase 126 — PDF support: final verification pass

**Verified (all tasks 01–05 already complete in working tree):**
- `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token
- `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk

**Test / lint / coverage results:**
- `uv run pytest` → 2997 passed
- `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:**
1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests)
2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test)
3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites)
4. PDF renders as extracted text in viewer — ✅ (E2E)
5. Test/coverage/lint gates — ✅ (above)
6. Commit + phase move — harness handles (per protocol, no `git` on this pass)

**Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree.

**Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
ducoterra
2026-09-27 09:36:40 -04:00
parent e0307fe3ff
commit e0f11e7aa8
40 changed files with 4092 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/chunker.py
+49
View File
@@ -26,6 +26,11 @@ per-format policies:
endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and
Jinja templates; plain-text paragraph packing (``chunk_text``) — no
format-specific splitter (owner decision).
* **pdf** (phase 126, LOCKED A2) — page-aware: the extracted text's
``---PAGE-N---`` markers are the primary split boundary (one logical
unit per text-bearing page; a long page sub-splits through the
paragraph packer), every chunk keeps its ``Page N:`` anchor, and the
markers themselves never appear in a chunk.
Every format honors :data:`HARD_MAX_CHARS` (1200 — the aipi ~1024-token
request cap) and the target/overlap settings; oversized blocks are split
@@ -324,6 +329,47 @@ def chunk_text(content: str, target_chars: int = 2000, overlap_chars: int = 200)
return _pack_blocks(_paragraph_blocks(lines, [False] * len(lines)), target, overlap)
#: The page marker of the extracted PDF text (phase 126, A2 —
#: ``app.rag.pdf.extract_pdf_text`` prefixes every text-bearing page
#: block with it; the number is the 1-indexed page number).
_PDF_PAGE_MARKER_RE = re.compile(r"---PAGE-(\d+)---")
def _chunk_pdf(content: str, target_chars: int = 2000, overlap_chars: int = 200) -> list[str]:
"""Page-aware chunking of one PDF's extracted text (phase 126, A2).
A page is the primary split boundary: each ``---PAGE-N---`` block is
packed on its own (the plain-text paragraph packer — PDFs have no
code fences, so paragraph boundaries = blank lines) into chunks of
at most ``target_chars`` with the ``overlap_chars`` carry-over, and
every chunk of the page is prefixed with the ``Page N:`` anchor so a
hit is readable in context. The marker number is the page number
(blank pages carry no marker — the numbering follows the markers,
not the chunk count). The markers themselves never appear in a
chunk, and the anchor-aware budget keeps the anchored chunk under
the target — :data:`HARD_MAX_CHARS` included.
"""
target, overlap = _normalize_target_overlap(target_chars, overlap_chars)
# re.split with a capturing group keeps the markers: the pieces are
# [before-first, page, text, page, text, …].
pieces = _PDF_PAGE_MARKER_RE.split(content)
chunks: list[str] = []
for i in range(1, len(pieces) - 1, 2):
page = int(pieces[i])
text = pieces[i + 1].strip()
if not text:
continue
anchor = f"Page {page}:"
# Budget the anchor out of the page's packing target so the
# anchored chunk cannot exceed ``target`` (the markdown
# section-anchor precedent, the ``budget`` rule).
budget = max(1, target - len(anchor) - 1)
chunks.extend(
f"{anchor}\n{part}" for part in chunk_text(text, budget, overlap)
)
return chunks
def _normalize_target_overlap(target_chars: int, overlap_chars: int) -> tuple[int, int]:
"""Validate + clamp the size policy (shared by every format)."""
if target_chars <= 0:
@@ -358,6 +404,9 @@ _FORMAT_CHUNKERS = {
".os": chunk_text,
".endpoint": chunk_text,
".j2": chunk_text,
# Phase 126 (LOCKED A2): the page-aware PDF chunker — the extracted
# text's ``---PAGE-N---`` markers are the primary split boundary.
".pdf": _chunk_pdf,
}
app/rag/importer.py
+516
View File
@@ -85,6 +85,30 @@ summary (stored verbatim, no ``lite`` call, no pointer line), so the
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
equals ``Document.content``.
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
AFTER the image branch, BEFORE any text read — a PDF is never
``read_text``-ed; the binary is owned by the phase-126 parser). The
document's ``content`` is the extracted text (every text-bearing page
prefixed with the ``---PAGE-N---`` marker — A2, one document per PDF),
the digest is sha256 over that text (the digest rule is unchanged —
content identity), the title is the PDF's embedded metadata title or
the file stem, and the page-aware chunker (task 03) chunks it with a
``Page N:`` anchor. With the phase-122 ``images`` toggle ON, the
embedded images join the image pipeline (A3, :func:
`_index_pdf_embedded_images`): each is described through the SAME
``_describe_or_skip`` seam (the bytes as a data URL), copied to
``settings.image_dir`` (the copy is written only AFTER a successful
description — a failure never leaves an orphan), and indexed as a
standalone ``is_image`` document under the synthetic path
``<pdf path>#page=<N>#img=<index>.<ext>`` (the description IS the
content and summary — the phase-122 convention). Toggle OFF → the
embedded images are skipped (debug-logged) and the prune guard keeps
any existing image docs; an unsupported image mime (outside
``BOR_IMAGE_EXTENSIONS``) is skipped with a warning. A PDF's viewer
presentation is its extracted text (A4 — nothing binary is served).
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
description, the same source's importable markdown files are scanned for
``![alt](path)`` references that name the image (the task-02
@@ -135,6 +159,7 @@ from datetime import UTC, datetime
from pathlib import Path, PurePosixPath
from typing import Any, Protocol
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
from sqlalchemy import select
from sqlalchemy.orm import Session
@@ -145,6 +170,7 @@ from app.rag.chunker import chunk_document, extract_title
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
from app.rag.llm import EmbeddingError, LLMError
from app.rag.md_images import ImageRef, extract_image_refs
from app.rag.pdf import PdfImage, extract_pdf_images, extract_pdf_text
from app.rag.summarizer import (
IMAGE_FALLBACK_MIME,
IMAGE_MIMES,
@@ -652,6 +678,14 @@ async def _index_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date,
)
# Phase 126: PDF files take the PDF index path — AFTER the image
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
# owns it and ``read_text`` is never called for one).
if full_path.suffix.lower() == ".pdf":
return await _index_pdf_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date,
)
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
@@ -855,6 +889,461 @@ async def _store_summary(
logger.error("import: summary failed source=%s path=%s — %s", source, rel, e)
# ---------------------------------------------------------------------------
# PDF indexing (phase 126, LOCKED A1/A2/A3/A4).
# ---------------------------------------------------------------------------
#: The ``Document.title`` column width (``app.models``) — the metadata
#: title is capped to it (an over-long PDF title must not blow up the
#: upsert).
_PDF_TITLE_MAX_CHARS = 500
#: MIME → extension for the phase-122 image family (phase 126, task 04)
#: — the reverse of the ``app.rag.summarizer.IMAGE_MIMES`` values for the
#: formats PyMuPDF reports. An unlisted mime takes ``.bin`` (never a
#: member of ``BOR_IMAGE_EXTENSIONS`` — the image is skipped with a
#: warning, the honest fail-soft outcome).
_MIME_TO_EXT: dict[str, str] = {
"image/png": ".png",
"image/jpeg": ".jpg",
"image/gif": ".gif",
"image/webp": ".webp",
"image/bmp": ".bmp",
"image/tiff": ".tiff",
}
def _extract_pdf_title(full_path: Path, fallback: str) -> str:
"""The title of one PDF document (phase 126, task 04).
The embedded metadata title (``fitz``'s ``doc.metadata["title"]``)
— trimmed and capped at the :data:`_PDF_TITLE_MAX_CHARS` column
width. An unreadable PDF, an absent metadata dict, or an
empty/whitespace-only title falls back to *fallback* (the file stem
— the non-markdown title rule; the metadata title is a convenience,
never a requirement).
"""
try:
doc = fitz.open(str(full_path))
except Exception:
return fallback
with doc:
meta = doc.metadata
title = str(meta.get("title") or "").strip() if meta else ""
if not title:
return fallback
return title[:_PDF_TITLE_MAX_CHARS]
async def _index_pdf_file(
session: Session,
*,
source: str,
rel: str,
full_path: Path,
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
) -> None:
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
for the document, then the embedded images (A3):
* ``content`` is the parser's text — every text-bearing page
prefixed with the ``---PAGE-N---`` marker (A2 — one document per
PDF, the markers are the page anchors the page-aware chunker
(task 03) splits on); an unopenable/empty PDF yields ``""``
(still a document — it may carry embedded images — with no
content chunks; a warning is the signal).
* the sha256 digest is over the extracted text (the digest rule is
content identity — the same text is the same document version;
the empty case digests the empty byte string).
* the title is the metadata title or the file stem
(:func:`_extract_pdf_title`); the date follows the text path
exactly (the phase-106 D2 mtime fallback — PDFs carry no
filesystem-visible creation date — plus the D1 manual-date lock
and the D4 refresh on the unchanged path).
* chunk + embed runs through the shared
:func:`_replace_content_chunks` helper (the endpoint token-cap
retry lives there), and the phase-30/118 (A2) ``lite`` summary
path runs on the text — PDFs are no more excluded than any other
non-markdown format.
* the unchanged path (same text digest) skips everything — no
re-extract, no re-embed, no image pass — and runs the same
backfill/manual-date/date-refresh dance as the text path.
* THEN the embedded images: :func:`_index_pdf_embedded_images`
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
``raw_date`` follows the text path: the caller's git last-commit
date or ``None`` (the mtime fallback, read before the unchanged
early-return — the unchanged path refreshes from the same source).
"""
# Extract the text (fail-soft: an unopenable PDF yields "" — the
# document may still carry embedded images worth indexing).
content = extract_pdf_text(full_path)
if not content:
logger.warning(
"import: PDF has no extractable text source=%s path=%s", source, rel
)
# Content identity: the digest is over the extracted text (the
# empty case is the empty byte string's digest).
digest = (
hashlib.sha256(content.encode("utf-8")).hexdigest()
if content
else hashlib.sha256(b"").hexdigest()
)
title = _extract_pdf_title(full_path, fallback=full_path.stem)
if raw_date is None:
# D2 fallback (same as the text path): no source date in the
# map → the file's mtime (one stat). Read before the unchanged
# early-return — the unchanged path refreshes from it.
raw_date = file_mtime_datetime(full_path)
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
if doc is not None and doc.content_hash == digest:
# Unchanged PDF (text digest) — the text path's unchanged branch,
# unmodified in shape: no re-extract, no re-embed, no image pass.
summary.unchanged += 1
logger.info("import: unchanged source=%s path=%s", source, rel)
# Phase 118 (A2): an unchanged doc whose summary is still NULL
# gets the same best-effort summary pass (see the text path).
if doc.summary is None:
await _store_summary(
session, doc=doc, source=source, rel=rel, content=content,
llm=llm, summary=summary, backfill=True,
)
if doc.created_at_manual:
# D1/D4: the owner's correction survives the sync — no write
# at all (the text path's manual-date early-return).
return
# D4: the date refreshes on every sync and may go OLDER (no
# monotonic guard); a date-only refresh counts unchanged.
target = normalize_doc_date(raw_date)
if target != doc.created_at:
doc.created_at = target
session.commit()
summary.dates_updated += 1
logger.info(
"import: date-refreshed source=%s path=%s date=%s",
source, rel, doc.created_at.isoformat(),
)
return
verb = "updated" if doc is not None else "added"
if doc is None:
doc = Document(
source=source,
path=rel,
full_path=str(full_path),
title=title,
content=content,
content_hash=digest,
indexed_at=datetime.now(UTC),
created_at=normalize_doc_date(raw_date),
)
session.add(doc)
else:
doc.full_path = str(full_path)
doc.title = title
doc.content = content
doc.content_hash = digest
doc.indexed_at = datetime.now(UTC)
# Phase 106 (D4): a content change is a new document version —
# the date is re-sourced and a manual correction reset.
doc.created_at = normalize_doc_date(raw_date)
doc.created_at_manual = False
session.flush() # guarantees doc.id even for brand-new rows
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
# aware chunker dispatches on the ``.pdf`` suffix; empty content
# clears the chunk rows and returns 0), committed atomically with
# the document.
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
)
session.commit()
if verb == "added":
summary.added += 1
else:
summary.updated += 1
summary.chunks += chunks_n
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
# Phase 30; phase 118 (A2): the ``lite`` summary of the extracted
# text (best-effort — a failure leaves the PDF indexed without a
# summary, counted in ``summary_errors``).
await _store_summary(
session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
)
# Phase 126 (A3): the embedded images join the phase-122 pipeline
# (gated by the ``images`` toggle — skipped, debug-logged, while it
# is off). Runs after the document commit, so a per-image failure
# never loses the PDF's text index.
await _index_pdf_embedded_images(
session, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
)
async def _index_pdf_embedded_images(
session: Session,
*,
source: str,
rel: str,
full_path: Path,
llm: Embedder,
summary: ImportSummary,
raw_date: datetime,
) -> None:
"""The phase-126 (A3) embedded-image pass — one call per (re)indexed
PDF, after its document commit.
Each embedded image (page order, 0-based index on the page — the
parser's order) is indexed as a STANDALONE image document through
the phase-122 mechanics, inline (no temp file, no
``_index_image_file`` refactor):
* gated by the phase-122 ``images`` toggle (the single knob, LOCKED
A3): while it is OFF the whole pass is skipped (one debug line —
the extraction itself is not worth running) and the prune guard
keeps any existing image docs;
* an image whose mime maps to no ``BOR_IMAGE_EXTENSIONS`` member
(:data:`_MIME_TO_EXT`) is skipped with a warning (the unsupported-
extension rule, same as a standalone image's);
* the document's synthetic ``path`` is
``<pdf path>#page=<N>#img=<index>.<ext>`` — unique per image
occurrence, stable across re-indexes (the upsert key); the title
is ``Embedded image from <pdf stem> page <N>``;
* the digest is sha256 over the image BYTES (the digest rule is
content identity); an image doc whose digest matches is skipped
(no re-describe, no re-embed — unchanged);
* the description comes through the SAME single seam
(:func:`_describe_or_skip` — the bytes as a data URL whose mime
comes from the copy path's extension); a failed/empty description
SKIPS the image entirely (no row, no copy) — ``images_failed`` +
a warning, the sync continues (LOCKED A3 fail-soft). The
``image_dir`` copy (``<doc-id>.<ext>`` — a new image's id is the
uuid4 chosen here; a changed image keeps its id and its stale
copy is deleted before replacement) is written only AFTER a
successful description (a failure never leaves an orphan);
* phase 125 parity (the standalone image's task-03 scan): a same-
source markdown file that references the image by its synthetic
path (``![alt](<pdf path>#page=<N>#img=<index>.<ext>)``) enriches
the description — the combined context goes to the vision call
and is stored on the doc's ``image_context``, and ``content``
becomes ``<context>\\n\\n<description>`` (as does ``summary`` —
the image convention); no reference keeps the bare shape
(byte-identical to the un-enriched case). The markdown-FIRST case
(a markdown indexed before the PDF recorded its reference on its
own row) is closed by the shared post-index pass
(:func:`_enrich_pending_images`), exactly as for a standalone
image;
* the description IS the content and the summary (the phase-122
image convention — :func:`_store_summary` stores it verbatim),
chunked + embedded through :func:`_replace_content_chunks` (one
transaction per image); a per-image embed failure rolls back that
image only (``errors`` + the error line) and the pass moves on —
the PDF's text index and the other images are already committed.
* a new/updated image doc counts ``added``/``updated`` (it really
changed the KB — the sources-version bump fires on it, like a
standalone image's).
``raw_date`` is the PDF's resolved source date (the pass runs only
for (re)indexed PDFs, so it is always concrete) — the image docs
inherit the PDF's date (they are part of it).
"""
settings = llm.settings
if not settings.images:
# Phase-122 toggle off: embedded images are skipped (debug-
# logged) — and the prune guard keeps any existing image docs
# (they are invisible to the walk, not deleted files).
logger.debug(
"import: PDF embedded images skipped (images toggle off) "
"source=%s path=%s",
source, rel,
)
return
images = extract_pdf_images(full_path)
for img in images:
await _index_one_embedded_image(
session, img, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
)
async def _index_one_embedded_image(
session: Session,
img: PdfImage,
*,
source: str,
rel: str,
full_path: Path,
llm: Embedder,
summary: ImportSummary,
raw_date: datetime,
) -> None:
"""One embedded image of one PDF → the phase-122 image pipeline,
with the phase-125 task-03 markdown context scan (parity with the
standalone image's :func:`_index_image_file` — see
:func:`_index_pdf_embedded_images` for the full contract)."""
settings = llm.settings
ext = _MIME_TO_EXT.get(img.mime, ".bin")
if ext not in settings.image_extension_set:
logger.warning(
"import: PDF embedded image skipped (unsupported extension) "
"source=%s path=%s page=%d index=%d mime=%s",
source, rel, img.page, img.index, img.mime,
)
return
image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
digest = hashlib.sha256(img.data).hexdigest()
image_doc = session.scalar(
select(Document).where(Document.source == source, Document.path == image_rel)
)
if image_doc is not None and image_doc.content_hash == digest:
# Unchanged embedded image (byte digest) — no re-describe, no
# re-embed (the standalone image's unchanged rule).
logger.debug(
"import: PDF embedded image unchanged source=%s path=%s",
source, image_rel,
)
return
verb = "updated" if image_doc is not None else "added"
# Named by the doc id (the phase-122 copy rule): a new image's id is
# the uuid4 chosen here (row and copy agree); a changed image keeps
# its id (the copy path is stable, the stale copy is deleted below).
doc_id = image_doc.id if image_doc is not None else uuid.uuid4()
image_dir = Path(settings.image_dir).expanduser()
copy_path = image_dir / f"{doc_id}{ext}"
# Phase 125 (task 03, LOCKED A1/A2) — the embedded image's parity
# with the standalone image's path (phase 126): BEFORE the
# description, the markdown reference context for THIS image — a
# ``![alt](<pdf path>#page=<N>#img=<index>.<ext>)`` reference in a
# same-source markdown file (the synthetic path is the doc's
# ``path``, and the reference resolves to it through the shared
# :func:`_resolve_ref_path` rule) — or ``None`` (no reference —
# the description call and the row stay byte-identical to the
# un-enriched shape). It goes to the vision model (a context-aware
# description, LOCKED A3) AND is stored on the doc (``image_context``
# — the retrieval display's context line, task 05). A re-indexed
# image re-scans: the context follows the current source state
# (a deleted referencing markdown drops it back to ``None``).
image_context = _markdown_image_context(
_source_root_for(full_path, rel), llm, source=source, image_rel=image_rel
)
# The description through the single seam (the mime resolves from
# the copy path's extension — the extension was derived from the
# mime, so the round-trip is lossless for the default family). The
# description is generated BEFORE anything is written, so a failure
# skips the image with no row and no copy.
description = await _describe_or_skip(
llm, data=img.data, source=source, rel=image_rel, full_path=copy_path,
context=image_context,
)
if description is None:
# LOCKED A3 fail-soft: an undescribed image is unsearchable
# noise — skip it (no row, no copy).
summary.images_failed += 1
logger.warning(
"import: PDF embedded image description failed source=%s path=%s "
"page=%d index=%d",
source, image_rel, img.page, img.index,
)
return
if image_doc is not None:
# A CHANGED image: the stale copy is deleted before replacement.
_delete_image_copy(image_doc.image_path)
image_dir.mkdir(parents=True, exist_ok=True)
copy_path.write_bytes(img.data)
# Phase 125 (task 03): the enriched content — the reference context
# FIRST (for retrieval), then the description (the standalone image's
# rule; ``summary = content`` below keeps the image convention).
content = (
f"{image_context}\n\n{description}" if image_context is not None else description
)
title = f"Embedded image from {full_path.stem} page {img.page}"
if image_doc is None:
image_doc = Document(
id=doc_id,
source=source,
path=image_rel,
full_path=str(copy_path),
title=title,
content=content,
content_hash=digest,
indexed_at=datetime.now(UTC),
created_at=normalize_doc_date(raw_date),
is_image=True,
image_path=str(copy_path),
image_context=image_context,
)
session.add(image_doc)
else:
image_doc.full_path = str(copy_path)
image_doc.title = title
image_doc.content = content
image_doc.content_hash = digest
image_doc.indexed_at = datetime.now(UTC)
# Phase 106 (D4): a content change is a new document version —
# the date is re-sourced and a manual correction reset.
image_doc.created_at = normalize_doc_date(raw_date)
image_doc.created_at_manual = False
image_doc.is_image = True
image_doc.image_path = str(copy_path)
# Phase 125 (task 03): a re-indexed image re-scans its markdown
# references — the context follows the current source state.
image_doc.image_context = image_context
session.flush() # guarantees doc.id even for brand-new rows
# Phase 1+2 — the shared pipeline on the description (one
# transaction per image; a per-image embed failure rolls back THIS
# image only — the PDF doc and the other images are already
# committed — and the pass moves on).
try:
chunks_n = await _replace_content_chunks(
session, doc=image_doc, llm=llm, content=content, rel=image_rel
)
session.commit()
except EmbeddingError as e:
session.rollback()
# The description succeeded (the copy was written), but the
# embed gave up: delete the copy so a failure never leaves an
# orphan (the phase-122 no-orphan rule).
_delete_image_copy(str(copy_path))
summary.errors += 1
logger.error(
"import: PDF embedded image embed failed source=%s path=%s "
"page=%d index=%d — %s",
source, image_rel, img.page, img.index, e,
)
return
if verb == "added":
summary.added += 1
else:
summary.updated += 1
summary.chunks += chunks_n
logger.info(
"import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
)
# Phase 122 (task 03) shape on the description — image-aware: the
# description IS the summary (stored verbatim, no ``lite`` call), so
# ``doc.summary`` == ``doc.content`` and the ``is_summary`` position
# −1 chunk mirrors it. With a markdown reference (phase 125) the
# summary is the ENRICHED content — the same value the standalone
# image's path stores.
await _store_summary(
session, doc=image_doc, source=source, rel=image_rel,
content=content, llm=llm, summary=summary,
)
def _source_root_for(full_path: Path, rel: str) -> Path:
"""The source root of one walked file (phase 125, task 03).
@@ -1517,6 +2006,12 @@ async def _index_image_file(
)
#: The synthetic path marker of a PDF-embedded image doc (phase 126,
#: LOCKED A3 — ``<pdf path>#page=<N>#img=<index>.<ext>``):
#: :data:`_EMBEDDED_IMAGE_RE` is the prune's parent rule below.
_EMBEDDED_IMAGE_RE = re.compile(r"#page=\d+#img=\d+")
def _prune(
session: Session,
source_names: set[str],
@@ -1532,6 +2027,19 @@ def _prune(
first images-off sync. Toggle ON → normal semantics: a deleted image
file prunes its doc, and the pruned image's ``image_dir`` copy is
deleted with it.
Phase 126 (LOCKED A3) — the PDF-embedded image's parent rule: an
embedded image doc's synthetic path (``#page=<N>#img=<index>``)
NEVER appears in the walk's *seen* — the image is derived from its
parent PDF, whose on-disk file is the walked file. Pruning it on the
plain rule would destroy it in the very sync that (re)indexes it,
so it is skipped while its parent PDF's rel is still in *seen* (the
image is invisible to the walk, not deleted). A pruned/absent parent
PDF prunes its embedded image docs with it (row + ``image_dir`` copy
— no orphans when the PDF leaves the source). The rule holds in BOTH
toggle states (toggle off, the phase-122 guard already skips every
``is_image`` doc; this one is the toggle-on half of the same
"invisible, not deleted" contract).
"""
if not source_names:
return 0
@@ -1542,6 +2050,14 @@ def _prune(
if doc.is_image and not images:
# Prune guard: invisible to the walk, not deleted.
continue
if doc.is_image and _EMBEDDED_IMAGE_RE.search(doc.path):
# Phase 126 (A3): the parent PDF's rel is the path minus
# its ``#page=…#img=…`` fragment — the walked file.
parent_rel = doc.path.split("#", 1)[0]
if (doc.source, parent_rel) in seen:
# Parent PDF still in the source — the image is
# invisible to the walk, not deleted.
continue
_delete_image_copy(doc.image_path)
session.delete(doc)
pruned += 1
app/rag/pdf.py
+210
View File
@@ -0,0 +1,210 @@
"""PDF parsing utilities (phase 126 — PDF support, locked decision A1).
Pure functions (I/O = reading the PDF file only) for extracting text
and embedded images from PDF files using PyMuPDF (the ``pymupdf``
package, imported as ``fitz`` — its stable API name).
* :func:`extract_pdf_text` — the text of every page that has text,
each prefixed with :data:`PAGE_SEPARATOR` (``---PAGE-N---``, N is the
1-indexed page number). The PDF chunker (phase 126 task 03) splits on
these markers so every chunk keeps its page anchor. Pages without
text (image-only pages) contribute nothing, so a PDF with no
extractable text returns ``""``.
* :func:`extract_pdf_images` — one :class:`PdfImage` per embedded-image
occurrence, in page order then 0-based index on the page. ``data`` is
the image's original encoded bytes as stored in the PDF (no
re-encoding), ``mime`` is derived via :func:`_mime_from_image_info`.
* :func:`_mime_from_image_info` — maps a PyMuPDF image-info dict to a
MIME type: the embedded file extension (``ext``) first, then the
image type code (``type``: 2=JPEG, 3=PNG, 4=GIF, 8=TIFF, 9=WebP),
else the ``application/octet-stream`` fallback.
Every failure mode is fail-soft: an unreadable file returns an empty
result, an unreadable page or image is logged and skipped — this module
never raises for malformed input (the importer owns the loud failures).
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import pymupdf as fitz # PyMuPDF (locked A1); `fitz` is the stable API name
logger = logging.getLogger("app.rag.pdf")
#: Page separator used in extracted text (the chunker splits on this).
#: ``{page}`` is the 1-indexed page number.
PAGE_SEPARATOR = "---PAGE-{page}---"
#: Mapping from PyMuPDF image type codes to MIME types
#: (2=JPEG, 3=PNG, 4=GIF, 8=TIFF, 9=WebP — phase-126 design).
_TYPE_TO_MIME: dict[int, str] = {
2: "image/jpeg",
3: "image/png",
4: "image/gif",
8: "image/tiff",
9: "image/webp",
}
#: Mapping from embedded-image file extensions to MIME types.
_EXT_TO_MIME: dict[str, str] = {
"png": "image/png",
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"tif": "image/tiff",
"tiff": "image/tiff",
}
@dataclass
class PdfImage:
"""One embedded image from a PDF page."""
page: int #: 1-indexed page number
index: int #: image index on the page (0-based, occurrence order)
width: int #: native pixel width
height: int #: native pixel height
data: bytes #: original encoded bytes as stored in the PDF
mime: str #: MIME type (e.g. ``image/png``)
def _mime_from_image_info(info: dict[str, Any]) -> str:
"""Map a PyMuPDF image-info dict to a MIME type.
The embedded file extension (``ext``) wins over the image type code
(``type``); unknown/missing both fall back to
``application/octet-stream``.
"""
ext = str(info.get("ext") or "").lower()
if ext in _EXT_TO_MIME:
return _EXT_TO_MIME[ext]
type_code = info.get("type")
if isinstance(type_code, int):
return _TYPE_TO_MIME.get(type_code, "application/octet-stream")
return "application/octet-stream"
def _page_image_entries(page: Any) -> list[dict[str, Any]]:
"""One info dict per embedded-image occurrence on *page*.
Prefers ``get_image_info(xrefs=True)`` (one entry per occurrence,
with the ``xref`` object number); older PyMuPDF releases without the
``xrefs`` keyword are served from ``get_images(full=True)`` (one
entry per unique image).
"""
try:
infos: list[dict[str, Any]] = page.get_image_info(xrefs=True)
except TypeError:
infos = [
{"xref": entry[0], "width": entry[2], "height": entry[3]}
for entry in page.get_images(full=True)
]
return [dict(info) for info in infos]
def extract_pdf_text(pdf_path: Path) -> str:
"""Extract the text of all pages of *pdf_path*.
Every page that has text is prefixed with :data:`PAGE_SEPARATOR`
(``---PAGE-N---``, N 1-indexed) and the page blocks are joined with
a blank line. Pages without text contribute nothing, so a PDF with
no extractable text (or a zero-page file) returns ``""``.
Fail-soft: an unreadable file returns ``""``; a page that cannot be
read is logged and skipped.
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as exc:
logger.warning("pdf: could not open %s — %s", pdf_path, exc)
return ""
parts: list[str] = []
with doc:
for page_num in range(len(doc)):
try:
raw = doc[page_num].get_text("text")
except Exception as exc:
logger.warning(
"pdf: could not read text of page %d of %s — %s",
page_num + 1, pdf_path, exc,
)
continue
# get_text("text") always returns a str; the union return type
# comes from pymupdf's other flavors (dict/words/blocks).
text = raw.strip() if isinstance(raw, str) else ""
if text:
parts.append(f"{PAGE_SEPARATOR.format(page=page_num + 1)}\n{text}")
return "\n\n".join(parts)
def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
"""Extract all embedded images from *pdf_path*.
Returns one :class:`PdfImage` per image occurrence, in page order
then 0-based index on the page. ``data`` is the original encoded
bytes as stored in the PDF; ``mime`` is derived from the image's
extension/type via :func:`_mime_from_image_info`.
Fail-soft: an unreadable file returns ``[]``; an image without an
``xref`` or that cannot be extracted is logged and skipped.
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as exc:
logger.warning("pdf: could not open %s for image extraction — %s", pdf_path, exc)
return []
images: list[PdfImage] = []
with doc:
for page_num in range(len(doc)):
try:
entries = _page_image_entries(doc[page_num])
except Exception as exc:
logger.warning(
"pdf: could not list images on page %d of %s — %s",
page_num + 1, pdf_path, exc,
)
continue
for index, info in enumerate(entries):
xref = int(info.get("xref") or 0)
if not xref:
logger.warning(
"pdf: skipping image page=%d index=%d of %s — no xref",
page_num + 1, index, pdf_path,
)
continue
try:
extracted = doc.extract_image(xref)
except Exception as exc:
logger.warning(
"pdf: could not extract image page=%d index=%d of %s — %s",
page_num + 1, index, pdf_path, exc,
)
continue
data = extracted.get("image") or b""
if not data:
logger.warning(
"pdf: skipping image page=%d index=%d of %s — empty data",
page_num + 1, index, pdf_path,
)
continue
# The image info's own `ext` is unreliable in some
# PyMuPDF versions (None); `extract_image` reports the
# actual format of the returned bytes, so it wins.
mime = _mime_from_image_info({**info, "ext": extracted.get("ext") or ""})
images.append(
PdfImage(
page=page_num + 1,
index=index,
width=int(extracted.get("width") or info.get("width") or 0),
height=int(extracted.get("height") or info.get("height") or 0),
data=data,
mime=mime,
)
)
return images