phase: 126_pdf_support
All gates verified green — no code changes were needed on this pass. Final report: ## Phase 126 — PDF support: final verification pass **Verified (all tasks 01–05 already complete in working tree):** - `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token - `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk **Test / lint / coverage results:** - `uv run pytest` → 2997 passed - `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** 1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests) 2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test) 3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites) 4. PDF renders as extracted text in viewer — ✅ (E2E) 5. Test/coverage/lint gates — ✅ (above) 6. Commit + phase move — harness handles (per protocol, no `git` on this pass) **Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree. **Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# Phase 126 — PDF support: PDFs become first-class, retrievable documents
|
||||
|
||||
**Source:** `TODO.md` — "add PDF support" (implied by the image handling assessment: "PDFs: Not supported. No PDF parsing library is in pyproject.toml.")
|
||||
**Story:** n/a (new file format support).
|
||||
**Context:** `app/config.py` — `Settings` (`import_extensions` L343, `import_extension_set` L733 — the extension filter; `images` L379, `image_extensions` L390, `image_dir` L413 — the image pipeline); `app/rag/importer.py` — `iter_importable_files` (L235, extension walk via `match_extension`), `import_sources` (L279, `prune` L283), `_index_file` (L435 — text branch: `read_text`, sha256 over text, Document upsert, chunk + embed pipeline), `_index_image_file` (L801 — binary branch: bytes digest, persistent copy, `describe_image`), `_store_summary` (L582), `_prune` (L649). `app/rag/chunker.py` — format-aware chunker (markdown, yaml, json, py, txt dispatch). `app/rag/summarizer.py` — `describe_image` (L208, vision model), `generate_summary` (L182, lite model). `app/rag/retriever.py` — `_selection_order` (L666), `select_suggested` (L864), `source_ref_with_image` (L945). `app/models.py` — `Document` (L102), `Chunk` (L171). `app/rag/llm.py` — the LLM client. `app/api/docs.py` — document content endpoint. `app/rag/archive_upload.py` — archive unpack (PDFs in archives should also be indexed).
|
||||
|
||||
## Objective
|
||||
PDF files become first-class, retrievable documents: text is extracted page by page, embedded images are extracted and described (joining the standalone image pipeline), and the full PDF content is chunked, embedded, and summarized — all behind a new `BOR_IMPORT_EXTENSIONS` token (`pdf`).
|
||||
|
||||
## Dependencies
|
||||
- `122_image_documents` (complete) — the image-doc pipeline: `describe_image`, `Document.is_image`/`image_path`, the vision model description path. Embedded PDF images reuse this pipeline.
|
||||
- `125_image_anchor_context` (complete) — the `image_context` field and enrichment pipeline (PDF-embedded images can be enriched if referenced by a markdown file in the same source).
|
||||
- Code dependencies (all complete): phase 30 summary pipeline, phase 118 summary seeding, the chunker, the LLM client.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Dependency (task 01):** add `pymupdf` (PyMuPDF, the `fitz` package) to `pyproject.toml` dependencies. PyMuPDF is chosen over `pdfplumber` because it natively supports both text extraction AND embedded image extraction in a single API, and it's a well-maintained package with good performance. The `fitz` module is the import name.
|
||||
- **PDF parsing module (task 02):** `app/rag/pdf.py` — new module with:
|
||||
- `def extract_pdf_text(pdf_path: Path) -> str`: extracts text from all pages, joined with `---PAGE-N---` separators (where N is 1-indexed). The separators enable the chunker to respect page boundaries.
|
||||
- `def extract_pdf_images(pdf_path: Path) -> list[PdfImage]`: extracts embedded images from all pages. Returns a list of `PdfImage` dataclasses: `page: int`, `index: int` (image index on the page), `width: int`, `height: int`, `data: bytes`, `mime: str` (derived from the image type code via PyMuPDF's `page.get_image_info()`).
|
||||
- `_mime_from_image_info(info: dict) -> str`: maps PyMuPDF's image type codes to MIME types (e.g., `2` → `image/jpeg`, `3` → `image/png`, `4` → `image/gif`, `8` → `image/webp`, `9` → `image/tiff`). Falls back to `application/octet-stream` for unknown types.
|
||||
- All functions are pure (except I/O for reading the PDF file) — fully unit-testable with a temp PDF fixture.
|
||||
- **PDF chunking (task 03):** extend `app/rag/chunker.py`'s `chunk_document` dispatch to handle `.pdf` files. The PDF chunker:
|
||||
- Splits on `---PAGE-N---` separators first (one chunk per page, or sub-split if a page is too large).
|
||||
- Each chunk keeps its page anchor (e.g., "Page 3:" prefix) so hits are readable in context.
|
||||
- Code blocks within PDFs are handled by the existing paragraph packing (PDFs don't have code fences, so paragraph boundaries = blank lines).
|
||||
- The `HARD_MAX_CHARS` cap (1200) still applies.
|
||||
- For very large PDFs (>50 pages), the chunk target is reduced to keep individual chunks manageable.
|
||||
- **PDF indexing (task 04):** in `app/rag/importer.py`'s `_index_file`:
|
||||
- Add a new branch for `.pdf` files (checked AFTER the image branch, BEFORE the text branch):
|
||||
```python
|
||||
if full_path.suffix.lower() == ".pdf":
|
||||
return await _index_pdf_file(session, source, rel, full_path, llm, summary, raw_date)
|
||||
```
|
||||
- `_index_pdf_file`:
|
||||
1. Extract text via `extract_pdf_text` (task 02).
|
||||
2. Extract embedded images via `extract_pdf_images` (task 02).
|
||||
3. Create the PDF document row with the extracted text as `content`, the file's sha256 (over the text) as `content_hash`, and the title from the PDF's metadata (if available) or the file stem.
|
||||
4. Chunk and embed the text (task 03's chunker handles `.pdf`).
|
||||
5. Generate a lite summary of the PDF content.
|
||||
6. For each embedded image:
|
||||
- If the image extension matches `BOR_IMAGE_EXTENSIONS` and `BOR_IMAGES` is on: create a standalone image document (reuse `_index_image_file`'s logic, but the image bytes come from the PDF, not a file on disk). The image's `path` is `<pdf_source>/<pdf_path>#page=<N>#img=<index>` (a synthetic path that uniquely identifies the embedded image).
|
||||
- If the image extension doesn't match or `BOR_IMAGES` is off: log a warning and skip the image.
|
||||
- The embedded image's description becomes the image doc's content+summary (the phase-122 pipeline).
|
||||
7. The PDF document's content includes inline references to embedded images: after each page's text, append `[Embedded image: <image_type> on page <N>]` (a placeholder that the LLM can use to know images exist, even if they're not described).
|
||||
- **PDF in archives (task 05):** PDFs inside uploaded archives should be indexed too. The archive unpack (`app/rag/archive_upload.py`) already extracts all files matching the import extensions. Add `pdf` to the archive's accepted file types (archives already unpack everything; the extension filter in `iter_importable_files` handles the rest).
|
||||
- **NOT touched:** the `BOR_IMAGES` toggle (PDF embedded images reuse the existing image pipeline, gated by it), the `show_image` tool (unchanged), the image serve route (unchanged), the document viewer (PDFs are served as text content, not as binary PDFs — the viewer renders the extracted text), and the prune logic (unchanged).
|
||||
- **Locked assumptions:** **A1** — `pymupdf` is the PDF parser (text + image extraction). **A2** — one document per PDF (not per page); pages are separated by `---PAGE-N---` markers. **A3** — embedded images become standalone image documents (reusing the phase-122 pipeline). **A4** — PDFs are served as extracted text in the document viewer (not as binary PDFs).
|
||||
|
||||
## Tasks
|
||||
1. `01_pdf_dependency.md` — add `pymupdf` to `pyproject.toml`.
|
||||
2. `02_pdf_parser.md` — `app/rag/pdf.py`: text extraction, image extraction, MIME mapping.
|
||||
3. `03_pdf_chunking.md` — extend the chunker to handle `.pdf` files.
|
||||
4. `04_pdf_indexing.md` — `_index_pdf_file` in the importer: text + embedded images + summary.
|
||||
5. `05_pdf_archive_and_tests.md` — PDFs in archives, unit + integration + E2E tests.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_pdf.py` (new, task 05) — `extract_pdf_text` (multi-page PDF → text with page separators; single-page PDF → text without extra separators; empty PDF → empty string); `extract_pdf_images` (PDF with embedded images → list of `PdfImage` with correct page/index/size/data/mime; PDF with no images → empty list); `_mime_from_image_info` (known types → correct MIME; unknown type → fallback).
|
||||
- Integration: `tests/integration/test_importer.py` (extend) — seed a source with a PDF file → sync → the PDF document has extracted text, chunks, and embedding; embedded images (if `BOR_IMAGES=true`) become image documents; a PDF without embedded images → no image docs.
|
||||
- E2E: `tests/e2e/test_pdf_support.py` (new, task 05) — isolated run: upload a PDF via the admin upload flow → sync → the PDF appears on the Sources page → open the document viewer → the extracted text renders; ask a question the PDF answers → the chat grounds on the PDF content.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A PDF file in a git/local source or uploaded archive becomes a document with extracted text, chunks, embedding, and summary.
|
||||
- [ ] With `BOR_IMAGES=true`: embedded images in the PDF become standalone image documents (described by the vision model).
|
||||
- [ ] With `BOR_IMAGES=false`: embedded images are skipped (logged), the PDF text is indexed normally.
|
||||
- [ ] The PDF renders as extracted text in the document viewer.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — `pymupdf` (PyMuPDF) is the PDF parser; it handles both text and embedded image extraction (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
- **A2 — one document per PDF; pages are separated by `---PAGE-N---` markers in the content (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
- **A3 — embedded images become standalone image documents, reusing the phase-122 pipeline (gated by `BOR_IMAGES`) (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
- **A4 — PDFs are served as extracted text in the document viewer (not as binary PDFs) (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ pyproject.toml tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(import): add PDF support — text extraction, embedded images, chunking"
|
||||
```
|
||||
Reference in New Issue
Block a user