# Phase 126 — PDF support: PDFs become first-class, retrievable documents

**Source:** `TODO.md` — "add PDF support" (implied by the image handling assessment: "PDFs: Not supported. No PDF parsing library is in pyproject.toml.")
**Story:** n/a (new file format support).
**Context:** `app/config.py` — `Settings` (`import_extensions` L343, `import_extension_set` L733 — the extension filter; `images` L379, `image_extensions` L390, `image_dir` L413 — the image pipeline); `app/rag/importer.py` — `iter_importable_files` (L235, extension walk via `match_extension`), `import_sources` (L279, `prune` L283), `_index_file` (L435 — text branch: `read_text`, sha256 over text, Document upsert, chunk + embed pipeline), `_index_image_file` (L801 — binary branch: bytes digest, persistent copy, `describe_image`), `_store_summary` (L582), `_prune` (L649). `app/rag/chunker.py` — format-aware chunker (markdown, yaml, json, py, txt dispatch). `app/rag/summarizer.py` — `describe_image` (L208, vision model), `generate_summary` (L182, lite model). `app/rag/retriever.py` — `_selection_order` (L666), `select_suggested` (L864), `source_ref_with_image` (L945). `app/models.py` — `Document` (L102), `Chunk` (L171). `app/rag/llm.py` — the LLM client. `app/api/docs.py` — document content endpoint. `app/rag/archive_upload.py` — archive unpack (PDFs in archives should also be indexed).

## Objective
PDF files become first-class, retrievable documents: text is extracted page by page, embedded images are extracted and described (joining the standalone image pipeline), and the full PDF content is chunked, embedded, and summarized — all behind a new `BOR_IMPORT_EXTENSIONS` token (`pdf`).

## Dependencies
- `122_image_documents` (complete) — the image-doc pipeline: `describe_image`, `Document.is_image`/`image_path`, the vision model description path. Embedded PDF images reuse this pipeline.
- `125_image_anchor_context` (complete) — the `image_context` field and enrichment pipeline (PDF-embedded images can be enriched if referenced by a markdown file in the same source).
- Code dependencies (all complete): phase 30 summary pipeline, phase 118 summary seeding, the chunker, the LLM client.

## Design (shared by all tasks — the executor reads this, not the chat)
- **Dependency (task 01):** add `pymupdf` (PyMuPDF, the `fitz` package) to `pyproject.toml` dependencies. PyMuPDF is chosen over `pdfplumber` because it natively supports both text extraction AND embedded image extraction in a single API, and it's a well-maintained package with good performance. The `fitz` module is the import name.
- **PDF parsing module (task 02):** `app/rag/pdf.py` — new module with:
  - `def extract_pdf_text(pdf_path: Path) -> str`: extracts text from all pages, joined with `---PAGE-N---` separators (where N is 1-indexed). The separators enable the chunker to respect page boundaries.
  - `def extract_pdf_images(pdf_path: Path) -> list[PdfImage]`: extracts embedded images from all pages. Returns a list of `PdfImage` dataclasses: `page: int`, `index: int` (image index on the page), `width: int`, `height: int`, `data: bytes`, `mime: str` (derived from the image type code via PyMuPDF's `page.get_image_info()`).
  - `_mime_from_image_info(info: dict) -> str`: maps PyMuPDF's image type codes to MIME types (e.g., `2` → `image/jpeg`, `3` → `image/png`, `4` → `image/gif`, `8` → `image/webp`, `9` → `image/tiff`). Falls back to `application/octet-stream` for unknown types.
  - All functions are pure (except I/O for reading the PDF file) — fully unit-testable with a temp PDF fixture.
- **PDF chunking (task 03):** extend `app/rag/chunker.py`'s `chunk_document` dispatch to handle `.pdf` files. The PDF chunker:
  - Splits on `---PAGE-N---` separators first (one chunk per page, or sub-split if a page is too large).
  - Each chunk keeps its page anchor (e.g., "Page 3:" prefix) so hits are readable in context.
  - Code blocks within PDFs are handled by the existing paragraph packing (PDFs don't have code fences, so paragraph boundaries = blank lines).
  - The `HARD_MAX_CHARS` cap (1200) still applies.
  - For very large PDFs (>50 pages), the chunk target is reduced to keep individual chunks manageable.
- **PDF indexing (task 04):** in `app/rag/importer.py`'s `_index_file`:
  - Add a new branch for `.pdf` files (checked AFTER the image branch, BEFORE the text branch):
    ```python
    if full_path.suffix.lower() == ".pdf":
        return await _index_pdf_file(session, source, rel, full_path, llm, summary, raw_date)
    ```
  - `_index_pdf_file`:
    1. Extract text via `extract_pdf_text` (task 02).
    2. Extract embedded images via `extract_pdf_images` (task 02).
    3. Create the PDF document row with the extracted text as `content`, the file's sha256 (over the text) as `content_hash`, and the title from the PDF's metadata (if available) or the file stem.
    4. Chunk and embed the text (task 03's chunker handles `.pdf`).
    5. Generate a lite summary of the PDF content.
    6. For each embedded image:
       - If the image extension matches `BOR_IMAGE_EXTENSIONS` and `BOR_IMAGES` is on: create a standalone image document (reuse `_index_image_file`'s logic, but the image bytes come from the PDF, not a file on disk). The image's `path` is `<pdf_source>/<pdf_path>#page=<N>#img=<index>` (a synthetic path that uniquely identifies the embedded image).
       - If the image extension doesn't match or `BOR_IMAGES` is off: log a warning and skip the image.
       - The embedded image's description becomes the image doc's content+summary (the phase-122 pipeline).
    7. The PDF document's content includes inline references to embedded images: after each page's text, append `[Embedded image: <image_type> on page <N>]` (a placeholder that the LLM can use to know images exist, even if they're not described).
- **PDF in archives (task 05):** PDFs inside uploaded archives should be indexed too. The archive unpack (`app/rag/archive_upload.py`) already extracts all files matching the import extensions. Add `pdf` to the archive's accepted file types (archives already unpack everything; the extension filter in `iter_importable_files` handles the rest).
- **NOT touched:** the `BOR_IMAGES` toggle (PDF embedded images reuse the existing image pipeline, gated by it), the `show_image` tool (unchanged), the image serve route (unchanged), the document viewer (PDFs are served as text content, not as binary PDFs — the viewer renders the extracted text), and the prune logic (unchanged).
- **Locked assumptions:** **A1** — `pymupdf` is the PDF parser (text + image extraction). **A2** — one document per PDF (not per page); pages are separated by `---PAGE-N---` markers. **A3** — embedded images become standalone image documents (reusing the phase-122 pipeline). **A4** — PDFs are served as extracted text in the document viewer (not as binary PDFs).

## Tasks
1. `01_pdf_dependency.md` — add `pymupdf` to `pyproject.toml`.
2. `02_pdf_parser.md` — `app/rag/pdf.py`: text extraction, image extraction, MIME mapping.
3. `03_pdf_chunking.md` — extend the chunker to handle `.pdf` files.
4. `04_pdf_indexing.md` — `_index_pdf_file` in the importer: text + embedded images + summary.
5. `05_pdf_archive_and_tests.md` — PDFs in archives, unit + integration + E2E tests.

## Testing & Quality
- Unit: `tests/unit/test_pdf.py` (new, task 05) — `extract_pdf_text` (multi-page PDF → text with page separators; single-page PDF → text without extra separators; empty PDF → empty string); `extract_pdf_images` (PDF with embedded images → list of `PdfImage` with correct page/index/size/data/mime; PDF with no images → empty list); `_mime_from_image_info` (known types → correct MIME; unknown type → fallback).
- Integration: `tests/integration/test_importer.py` (extend) — seed a source with a PDF file → sync → the PDF document has extracted text, chunks, and embedding; embedded images (if `BOR_IMAGES=true`) become image documents; a PDF without embedded images → no image docs.
- E2E: `tests/e2e/test_pdf_support.py` (new, task 05) — isolated run: upload a PDF via the admin upload flow → sync → the PDF appears on the Sources page → open the document viewer → the extracted text renders; ask a question the PDF answers → the chat grounds on the PDF content.
- Coverage: **>90%** on `app/` (validate.sh gate).

## Completion Criteria
- [ ] A PDF file in a git/local source or uploaded archive becomes a document with extracted text, chunks, embedding, and summary.
- [ ] With `BOR_IMAGES=true`: embedded images in the PDF become standalone image documents (described by the vision model).
- [ ] With `BOR_IMAGES=false`: embedded images are skipped (logged), the PDF text is indexed normally.
- [ ] The PDF renders as extracted text in the document viewer.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.

## Locked decisions
- **A1 — `pymupdf` (PyMuPDF) is the PDF parser; it handles both text and embedded image extraction (owner-confirmed 2026-09-26, roadmap confirmation).**
- **A2 — one document per PDF; pages are separated by `---PAGE-N---` markers in the content (owner-confirmed 2026-09-26, roadmap confirmation).**
- **A3 — embedded images become standalone image documents, reusing the phase-122 pipeline (gated by `BOR_IMAGES`) (owner-confirmed 2026-09-26, roadmap confirmation).**
- **A4 — PDFs are served as extracted text in the document viewer (not as binary PDFs) (owner-confirmed 2026-09-26, roadmap confirmation).**

## Commit
```bash
git add app/ pyproject.toml tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(import): add PDF support — text extraction, embedded images, chunking"
```