Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 125_image_anchor_context
Build and Push Containers / build-and-push-app (push) Successful in 2m8s
Details
Build and Push Containers / build-and-push-db (push) Failing after 1m38s
Details

Browse Source 
All completion criteria verified. The repository is left functional with all changes in the working tree (no commits made, per the harness protocol).

## Final report — phase 125 `image_anchor_context` (verification pass)

All 6 tasks were already complete; this pass verified every criterion and ran the full gates:

- **Migration 0023** `documents.image_context` (TEXT nullable, reversible) — applied (`alembic current` = 0023 head); round-trip test (4 passed) exercises real upgrade → downgrade 0022 → upgrade.
- **Criterion 1** (referenced image enriched + context line in retrieval): integration `test_markdown_referenced_image_gets_context_and_enriched_chunks` + E2E `test_sync_enriches_the_image_doc_with_markdown_context`, `test_chat_answer_quotes_the_context_line` (wire prompt pinned: `<document>` block leads with the one-line `Referenced in …: "alt"` line) — verified present and passing.
- **Criterion 2** (standalone image byte-identical): E2E `test_standalone_*` (NULL `image_context`, block body exactly the bare description, no `Referenced in` anywhere in the prompt) + unit `test_describe_image_without_context_is_byte_identical` — passing.
- **Criterion 3** (markdown-first: record stored on markdown doc, image picks up context later): integration `test_markdown_indexed_before_image_stores_record_and_image_picks_it_up` + post-index enrichment pass test (markdown file gone from disk) — passing.
- **Gates:** `uv run pytest` → 2938 passed, exit 0 · `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings.
- **Phase E2E in isolation:** `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov` → 4 passed.
- **Mock-LLM change regression check:** neighboring image E2E suites green in isolation — `test_smoke` (3), `test_image_documents` (16), `test_chat_image_questions` (17), `test_answer_images` (17).
- No defects found; no code changes needed. No live-infrastructure changes (local dev DB only). No commit made (harness commits).
- Next pending phase: check `.agents/phases/todo/` after the harness moves this phase.
This commit is contained in:
ducoterra
2026-09-27 05:53:12 -04:00
parent 62e4f6b572
commit e0307fe3ff
60 changed files with 6583 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/126_pdf_support/00_phase.md
+79
View File
@@ -0,0 +1,79 @@
# Phase 126 — PDF support: PDFs become first-class, retrievable documents
**Source:** `TODO.md` — "add PDF support" (implied by the image handling assessment: "PDFs: Not supported. No PDF parsing library is in pyproject.toml.")
**Story:** n/a (new file format support).
**Context:** `app/config.py` — `Settings` (`import_extensions` L343, `import_extension_set` L733 — the extension filter; `images` L379, `image_extensions` L390, `image_dir` L413 — the image pipeline); `app/rag/importer.py` — `iter_importable_files` (L235, extension walk via `match_extension`), `import_sources` (L279, `prune` L283), `_index_file` (L435 — text branch: `read_text`, sha256 over text, Document upsert, chunk + embed pipeline), `_index_image_file` (L801 — binary branch: bytes digest, persistent copy, `describe_image`), `_store_summary` (L582), `_prune` (L649). `app/rag/chunker.py` — format-aware chunker (markdown, yaml, json, py, txt dispatch). `app/rag/summarizer.py` — `describe_image` (L208, vision model), `generate_summary` (L182, lite model). `app/rag/retriever.py` — `_selection_order` (L666), `select_suggested` (L864), `source_ref_with_image` (L945). `app/models.py` — `Document` (L102), `Chunk` (L171). `app/rag/llm.py` — the LLM client. `app/api/docs.py` — document content endpoint. `app/rag/archive_upload.py` — archive unpack (PDFs in archives should also be indexed).
## Objective
PDF files become first-class, retrievable documents: text is extracted page by page, embedded images are extracted and described (joining the standalone image pipeline), and the full PDF content is chunked, embedded, and summarized — all behind a new `BOR_IMPORT_EXTENSIONS` token (`pdf`).
## Dependencies
- `122_image_documents` (complete) — the image-doc pipeline: `describe_image`, `Document.is_image`/`image_path`, the vision model description path. Embedded PDF images reuse this pipeline.
- `125_image_anchor_context` (complete) — the `image_context` field and enrichment pipeline (PDF-embedded images can be enriched if referenced by a markdown file in the same source).
- Code dependencies (all complete): phase 30 summary pipeline, phase 118 summary seeding, the chunker, the LLM client.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Dependency (task 01):** add `pymupdf` (PyMuPDF, the `fitz` package) to `pyproject.toml` dependencies. PyMuPDF is chosen over `pdfplumber` because it natively supports both text extraction AND embedded image extraction in a single API, and it's a well-maintained package with good performance. The `fitz` module is the import name.
- **PDF parsing module (task 02):** `app/rag/pdf.py` — new module with:
- `def extract_pdf_text(pdf_path: Path) -> str`: extracts text from all pages, joined with `---PAGE-N---` separators (where N is 1-indexed). The separators enable the chunker to respect page boundaries.
- `def extract_pdf_images(pdf_path: Path) -> list[PdfImage]`: extracts embedded images from all pages. Returns a list of `PdfImage` dataclasses: `page: int`, `index: int` (image index on the page), `width: int`, `height: int`, `data: bytes`, `mime: str` (derived from the image type code via PyMuPDF's `page.get_image_info()`).
- `_mime_from_image_info(info: dict) -> str`: maps PyMuPDF's image type codes to MIME types (e.g., `2` → `image/jpeg`, `3` → `image/png`, `4` → `image/gif`, `8` → `image/webp`, `9` → `image/tiff`). Falls back to `application/octet-stream` for unknown types.
- All functions are pure (except I/O for reading the PDF file) — fully unit-testable with a temp PDF fixture.
- **PDF chunking (task 03):** extend `app/rag/chunker.py`'s `chunk_document` dispatch to handle `.pdf` files. The PDF chunker:
- Splits on `---PAGE-N---` separators first (one chunk per page, or sub-split if a page is too large).
- Each chunk keeps its page anchor (e.g., "Page 3:" prefix) so hits are readable in context.
- Code blocks within PDFs are handled by the existing paragraph packing (PDFs don't have code fences, so paragraph boundaries = blank lines).
- The `HARD_MAX_CHARS` cap (1200) still applies.
- For very large PDFs (>50 pages), the chunk target is reduced to keep individual chunks manageable.
- **PDF indexing (task 04):** in `app/rag/importer.py`'s `_index_file`:
- Add a new branch for `.pdf` files (checked AFTER the image branch, BEFORE the text branch):
```python
if full_path.suffix.lower() == ".pdf":
return await _index_pdf_file(session, source, rel, full_path, llm, summary, raw_date)
```
- `_index_pdf_file`:
1. Extract text via `extract_pdf_text` (task 02).
2. Extract embedded images via `extract_pdf_images` (task 02).
3. Create the PDF document row with the extracted text as `content`, the file's sha256 (over the text) as `content_hash`, and the title from the PDF's metadata (if available) or the file stem.
4. Chunk and embed the text (task 03's chunker handles `.pdf`).
5. Generate a lite summary of the PDF content.
6. For each embedded image:
- If the image extension matches `BOR_IMAGE_EXTENSIONS` and `BOR_IMAGES` is on: create a standalone image document (reuse `_index_image_file`'s logic, but the image bytes come from the PDF, not a file on disk). The image's `path` is `<pdf_source>/<pdf_path>#page=<N>#img=<index>` (a synthetic path that uniquely identifies the embedded image).
- If the image extension doesn't match or `BOR_IMAGES` is off: log a warning and skip the image.
- The embedded image's description becomes the image doc's content+summary (the phase-122 pipeline).
7. The PDF document's content includes inline references to embedded images: after each page's text, append `[Embedded image: <image_type> on page <N>]` (a placeholder that the LLM can use to know images exist, even if they're not described).
- **PDF in archives (task 05):** PDFs inside uploaded archives should be indexed too. The archive unpack (`app/rag/archive_upload.py`) already extracts all files matching the import extensions. Add `pdf` to the archive's accepted file types (archives already unpack everything; the extension filter in `iter_importable_files` handles the rest).
- **NOT touched:** the `BOR_IMAGES` toggle (PDF embedded images reuse the existing image pipeline, gated by it), the `show_image` tool (unchanged), the image serve route (unchanged), the document viewer (PDFs are served as text content, not as binary PDFs — the viewer renders the extracted text), and the prune logic (unchanged).
- **Locked assumptions:** **A1** — `pymupdf` is the PDF parser (text + image extraction). **A2** — one document per PDF (not per page); pages are separated by `---PAGE-N---` markers. **A3** — embedded images become standalone image documents (reusing the phase-122 pipeline). **A4** — PDFs are served as extracted text in the document viewer (not as binary PDFs).
## Tasks
1. `01_pdf_dependency.md` — add `pymupdf` to `pyproject.toml`.
2. `02_pdf_parser.md` — `app/rag/pdf.py`: text extraction, image extraction, MIME mapping.
3. `03_pdf_chunking.md` — extend the chunker to handle `.pdf` files.
4. `04_pdf_indexing.md` — `_index_pdf_file` in the importer: text + embedded images + summary.
5. `05_pdf_archive_and_tests.md` — PDFs in archives, unit + integration + E2E tests.
## Testing & Quality
- Unit: `tests/unit/test_pdf.py` (new, task 05) — `extract_pdf_text` (multi-page PDF → text with page separators; single-page PDF → text without extra separators; empty PDF → empty string); `extract_pdf_images` (PDF with embedded images → list of `PdfImage` with correct page/index/size/data/mime; PDF with no images → empty list); `_mime_from_image_info` (known types → correct MIME; unknown type → fallback).
- Integration: `tests/integration/test_importer.py` (extend) — seed a source with a PDF file → sync → the PDF document has extracted text, chunks, and embedding; embedded images (if `BOR_IMAGES=true`) become image documents; a PDF without embedded images → no image docs.
- E2E: `tests/e2e/test_pdf_support.py` (new, task 05) — isolated run: upload a PDF via the admin upload flow → sync → the PDF appears on the Sources page → open the document viewer → the extracted text renders; ask a question the PDF answers → the chat grounds on the PDF content.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A PDF file in a git/local source or uploaded archive becomes a document with extracted text, chunks, embedding, and summary.
- [ ] With `BOR_IMAGES=true`: embedded images in the PDF become standalone image documents (described by the vision model).
- [ ] With `BOR_IMAGES=false`: embedded images are skipped (logged), the PDF text is indexed normally.
- [ ] The PDF renders as extracted text in the document viewer.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — `pymupdf` (PyMuPDF) is the PDF parser; it handles both text and embedded image extraction (owner-confirmed 2026-09-26, roadmap confirmation).**
- **A2 — one document per PDF; pages are separated by `---PAGE-N---` markers in the content (owner-confirmed 2026-09-26, roadmap confirmation).**
- **A3 — embedded images become standalone image documents, reusing the phase-122 pipeline (gated by `BOR_IMAGES`) (owner-confirmed 2026-09-26, roadmap confirmation).**
- **A4 — PDFs are served as extracted text in the document viewer (not as binary PDFs) (owner-confirmed 2026-09-26, roadmap confirmation).**
## Commit
```bash
git add app/ pyproject.toml tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(import): add PDF support — text extraction, embedded images, chunking"
```
.agents/phases/todo/126_pdf_support/01_pdf_dependency.md
+19
View File
@@ -0,0 +1,19 @@
# Task 01 — Add `pymupdf` dependency
**Phase:** `126_pdf_support` · **Source:** `00_phase.md` — the `pymupdf` dependency
## Objective
Add `pymupdf` to `pyproject.toml` dependencies. PyMuPDF provides both PDF text extraction and embedded image extraction via the `fitz` module.
## Work
1. `pyproject.toml` — add `pymupdf>=1.24,<2.0` to the `dependencies` list (after the existing dependencies, before the dev group). Version pin: `1.24` is a stable release with good text and image extraction support; `<2.0` follows the project's convention of pinning major versions.
2. ASSUMPTION: none. `pymupdf` is a pure-Python package with optional C extensions (the binary wheels are available for common platforms).
## Testing & Quality
- No code tests needed — this is a dependency addition.
- Verify: `uv sync` completes without errors; `import fitz` works in a Python shell.
## Completion Criteria
- [ ] `uv sync` completes successfully.
- [ ] `import fitz` works; `fitz.VersionBind` shows the installed version.
- [ ] `uv run pytest` still green (no regression from the dependency addition).
.agents/phases/todo/126_pdf_support/02_pdf_parser.md
+173
View File
@@ -0,0 +1,173 @@
# Task 02 — PDF parser module: `app/rag/pdf.py`
**Phase:** `126_pdf_support` · **Source:** `00_phase.md` — text extraction, image extraction, MIME mapping
## Objective
A new `app/rag/pdf.py` module with pure functions for PDF text extraction, embedded image extraction, and MIME type mapping. All functions are I/O-bound only (read a PDF file, return parsed data) — fully unit-testable with temp PDF fixtures.
## Work
1. `app/rag/pdf.py` — new module:
```python
"""PDF parsing utilities (phase 126).
Pure functions for extracting text and embedded images from PDF files
using PyMuPDF (the ``fitz`` module).
"""
from __future__ import annotations
import hashlib
import logging
from dataclasses import dataclass
from pathlib import Path
import fitz # pymupdf
logger = logging.getLogger("app.pdf")
#: Page separator used in extracted text (chunker splits on this).
PAGE_SEPARATOR = "---PAGE-{page}---"
@dataclass
class PdfImage:
"""One embedded image from a PDF page."""
page: int #: 1-indexed page number
index: int #: image index on the page (0-based, from fitz)
width: int
height: int
data: bytes #: raw image bytes
mime: str #: MIME type (e.g., "image/png")
#: Mapping from PyMuPDF image type codes to MIME types.
#: Type codes: 2=JPEG, 3=PNG, 4=GIF, 8=TIFF, 9=WebP (per fitz docs).
_TYPE_TO_MIME: dict[int, str] = {
2: "image/jpeg",
3: "image/png",
4: "image/gif",
8: "image/tiff",
9: "image/webp",
}
def _mime_from_image_info(info: dict) -> str:
"""Map a PyMuPDF image info dict to a MIME type.
Falls back to ``application/octet-stream`` for unknown types.
"""
# Try the 'ext' key first (file extension), then the 'type' code.
ext = info.get("ext", "").lower()
if ext == "png":
return "image/png"
if ext in ("jpg", "jpeg"):
return "image/jpeg"
if ext == "gif":
return "image/gif"
if ext == "webp":
return "image/webp"
if ext == "tiff" or ext == "tif":
return "image/tiff"
# Fallback to type code.
type_code = info.get("type")
if type_code is not None:
return _TYPE_TO_MIME.get(type_code, "application/octet-stream")
return "application/octet-stream"
def extract_pdf_text(pdf_path: Path) -> str:
"""Extract text from all pages of a PDF, joined with page separators.
Returns the full text with ``---PAGE-N---`` separators between pages.
Returns an empty string if the PDF has no text or cannot be opened.
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as e:
logger.warning("pdf: could not open %s — %s", pdf_path, e)
return ""
parts: list[str] = []
for page_num in range(len(doc)):
page = doc[page_num]
text = page.get_text("text").strip()
if text:
parts.append(f"{PAGE_SEPARATOR.format(page=page_num + 1)}\n{text}")
else:
# Even empty pages get a separator (for chunker consistency).
parts.append(f"{PAGE_SEPARATOR.format(page=page_num + 1)}")
doc.close()
return "\n\n".join(parts)
def extract_pdf_images(pdf_path: Path) -> list[PdfImage]:
"""Extract all embedded images from a PDF.
Returns a list of ``PdfImage`` dataclasses, one per embedded image.
Images are extracted in page order, then by their index on the page.
Returns an empty list if the PDF has no embedded images or cannot
be opened.
"""
try:
doc = fitz.open(str(pdf_path))
except Exception as e:
logger.warning("pdf: could not open %s for image extraction — %s", pdf_path, e)
return []
images: list[PdfImage] = []
for page_num in range(len(doc)):
page = doc[page_num]
image_list = page.get_images(full=True)
for img_index, img_info in enumerate(image_list):
try:
xref = img_info[0]
# Get the raw image bytes.
pix = fitz.Pixmap(doc, xref)
# Convert to RGB if needed (CMYK → RGB).
if pix.n - pix.alpha > 3:
pix = fitz.Pixmap(fitz.csRGB, pix)
data = pix.tobytes("png") if pix.n == 4 else pix.tobytes("jpeg")
# Determine the best bytes representation.
# If the image is grayscale or has alpha, use PNG.
# Otherwise, use JPEG for smaller size.
if pix.alpha > 0 or pix.n <= 2:
data = pix.tobytes("png")
mime = "image/png"
else:
data = pix.tobytes("jpeg")
mime = "image/jpeg"
# Fallback: if tobytes fails, use the raw data.
if not data:
data = pix.tobytes()
mime = _mime_from_image_info(img_info)
images.append(PdfImage(
page=page_num + 1,
index=img_index,
width=pix.width,
height=pix.height,
data=data,
mime=mime,
))
pix = None # free the pixmap
except Exception as e:
logger.warning(
"pdf: could not extract image page=%d index=%d — %s",
page_num + 1, img_index, e,
)
doc.close()
return images
```
2. ASSUMPTION: none. The module uses only `fitz` (from `pymupdf`) and stdlib.
## Testing & Quality
- Unit: `tests/unit/test_pdf.py` (task 05) — create temp PDF fixtures using `fitz` itself:
- A single-page PDF with text → `extract_pdf_text` returns the text with one page separator.
- A multi-page PDF → text with multiple page separators.
- A PDF with no text (image-only) → empty string.
- A PDF with embedded images → `extract_pdf_images` returns the correct list.
- A PDF with no embedded images → empty list.
- `_mime_from_image_info` (known types → correct MIME; unknown → fallback).
- Error handling: non-existent PDF → empty string / empty list (no exception).
- Coverage: **>90%** on the new module.
## Completion Criteria
- [ ] `extract_pdf_text` returns correct text with page separators for a multi-page PDF.
- [ ] `extract_pdf_images` returns correct image data for a PDF with embedded images.
- [ ] `_mime_from_image_info` maps known type codes correctly; unknown types get the fallback.
- [ ] Non-existent or unreadable PDFs return empty results (no exceptions).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
.agents/phases/todo/126_pdf_support/03_pdf_chunking.md
+56
View File
@@ -0,0 +1,56 @@
# Task 03 — PDF chunking: extend the chunker for `.pdf` files
**Phase:** `126_pdf_support` · **Source:** `00_phase.md` — extend the chunker to handle `.pdf` files
## Objective
Extend `app/rag/chunker.py`'s `chunk_document` dispatch to handle `.pdf` files. PDF chunks are split on page separators (`---PAGE-N---`), with each page as a logical unit.
## Work
1. `app/rag/chunker.py` — extend the `chunk_document` dispatch:
- Add `.pdf` to the suffix check that dispatches to the paragraph packer (PDFs don't have a format-specific splitter like markdown's heading-based splitting).
- The PDF chunker uses the same `chunk_text` function as `txt` (paragraph packing), but with page-aware splitting:
```python
def _chunk_pdf(content: str, rel: str, target: int, overlap: int) -> list[str]:
"""Chunk PDF content, respecting page separators.
Pages are the primary split boundary. Each page's text is packed
into chunks of ``target`` chars with ``overlap`` chars carry-over.
"""
# Split on page separators.
pages = re.split(r'---PAGE-\d+---', content)
# pages[0] is empty (content starts with the separator).
result: list[str] = []
for i, page_text in enumerate(pages[1:], start=1):
page_text = page_text.strip()
if not page_text:
continue
# Prepend the page anchor for readability.
anchored = f"Page {i}:\n{page_text}"
# Chunk the anchored text using paragraph packing.
chunks = chunk_text(anchored, target=target, overlap=overlap)
result.extend(chunks)
return result
```
- Register `.pdf` in the `chunk_document` dispatch table:
```python
CHUNKERS: dict[str, Callable] = {
...
".pdf": _chunk_pdf,
}
```
2. ASSUMPTION: none. The PDF chunker reuses the existing `chunk_text` paragraph packer — it only adds page-aware splitting.
## Testing & Quality
- Unit: `tests/unit/test_chunker.py` (extend) — add PDF-specific tests:
- A single-page PDF → one or more chunks with "Page 1:" prefix.
- A multi-page PDF → chunks with correct "Page N:" prefixes.
- An empty PDF → empty list.
- A very large page (>HARD_MAX_CHARS) → split by line (the `chunk_text` behavior).
- Verify that page separators in the input are removed from the output chunks.
- Coverage: **>90%** on the touched code.
## Completion Criteria
- [ ] `chunk_document` dispatches `.pdf` files to the page-aware chunker.
- [ ] Chunks have "Page N:" prefixes and no page separator markers.
- [ ] Multi-page PDFs produce chunks with correct page anchors.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
.agents/phases/todo/126_pdf_support/04_pdf_indexing.md
+262
View File
@@ -0,0 +1,262 @@
# Task 04 — PDF indexing: `_index_pdf_file` in the importer
**Phase:** `126_pdf_support` · **Source:** `00_phase.md` — text + embedded images + summary
## Objective
Add PDF indexing to the importer: `_index_pdf_file` extracts text, creates the document, chunks + embeds, generates a summary, and handles embedded images.
## Work
1. `app/rag/importer.py` — add the import:
```python
from app.rag.pdf import extract_pdf_text, extract_pdf_images, PdfImage
```
2. `app/rag/importer.py` — in `_index_file`, add the PDF branch (AFTER the image branch, BEFORE the text branch):
```python
# Phase 126: PDF files take the PDF index path.
if full_path.suffix.lower() == ".pdf":
return await _index_pdf_file(
session, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
)
```
3. `app/rag/importer.py` — new `_index_pdf_file` function (follow the `_index_file` text branch pattern):
```python
async def _index_pdf_file(
session: Session,
*,
source: str,
rel: str,
full_path: Path,
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
) -> None:
"""Index one PDF file: extract text, create doc, chunk + embed,
summarize, and extract embedded images.
"""
settings = llm.settings
# Extract text.
content = extract_pdf_text(full_path)
if not content:
# No text in the PDF — still create a doc (it might have
# embedded images), but with empty content.
logger.warning("import: PDF has no extractable text source=%s path=%s", source, rel)
# Digest over the extracted text (content identity).
digest = hashlib.sha256(content.encode("utf-8")).hexdigest() if content else hashlib.sha256(b"").hexdigest()
# Get PDF metadata for the title.
title = _extract_pdf_title(full_path, fallback=full_path.stem)
# Date: use the file's mtime (D2 fallback) — PDFs don't have
# standard creation dates accessible via the filesystem.
if raw_date is None:
raw_date = file_mtime_datetime(full_path)
# Upsert the document row.
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
if doc is None:
doc = Document(
source=source,
path=rel,
full_path=str(full_path),
title=title,
content=content,
content_hash=digest,
indexed_at=datetime.now(UTC),
created_at=normalize_doc_date(raw_date),
)
session.add(doc)
else:
# Check for content change.
if doc.content_hash == digest and doc.content == content:
summary.unchanged += 1
logger.info("import: unchanged source=%s path=%s", source, rel)
# Summary backfill (phase 118, A2).
if doc.summary is None:
await _store_summary(
session, doc=doc, source=source, rel=rel,
content=content, llm=llm, summary=summary, backfill=True,
)
if doc.created_at_manual:
return
target = normalize_doc_date(raw_date)
if target != doc.created_at:
doc.created_at = target
session.commit()
summary.dates_updated += 1
return
doc.full_path = str(full_path)
doc.title = title
doc.content = content
doc.content_hash = digest
doc.indexed_at = datetime.now(UTC)
doc.created_at = normalize_doc_date(raw_date)
doc.created_at_manual = False
session.flush()
# Chunk and embed.
if content:
target = max(400, settings.chunk_target_chars)
while True:
chunks_text = chunk_document(content, rel, target, settings.chunk_overlap_chars)
doc.chunks = [
Chunk(document_id=doc.id, position=i, content=c)
for i, c in enumerate(chunks_text)
]
session.flush()
if not doc.chunks:
break
try:
vectors = await llm.embed([c.content for c in doc.chunks])
for row, vec in zip(doc.chunks, vectors, strict=True):
row.embedding = vec
break
except EmbeddingError as e:
if "token cap" not in str(e) or target <= 400:
raise
logger.info(
"import: re-chunking PDF at %d chars after endpoint token cap: %s",
target // 2, rel,
)
target //= 2
session.commit()
# Verb count.
if doc.content_hash == digest and doc.content == content and doc.id in [d.id for d in []]:
# This is a bit awkward — we need to track whether this was
# new or updated. Use the session's identity map.
pass # handled by the unchanged path above
else:
summary.added += 1 if doc.id not in session.identity_map else 0
# Actually, let's just use the verb tracking from _index_file.
# The simplest approach: track added/updated before the upsert.
# Generate summary (phase 30, A2).
await _store_summary(
session, doc=doc, source=source, rel=rel,
content=content, llm=llm, summary=summary, backfill=False,
)
# Extract and index embedded images.
await _index_pdf_embedded_images(
session, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
)
logger.info("import: %s source=%s path=%s chunks=%d",
"added" if newly_created else "updated", source, rel, len(doc.chunks))
```
Wait — the above is getting complex. Let me simplify. The key insight is that `_index_pdf_file` should follow the same pattern as `_index_file` but with PDF-specific extraction. Let me refactor.
Actually, let me simplify this significantly. The `_index_pdf_file` function should:
1. Extract text
2. Upsert the document
3. Chunk + embed
4. Generate summary
5. Extract embedded images
6. Log the result
The unchanged/hash path is the same as `_index_file` — check the hash, skip if unchanged.
4. `app/rag/importer.py` — `_extract_pdf_title(full_path, fallback) -> str`:
- Open the PDF with `fitz`, try to get the metadata title (`doc.metadata.get("title", "")`).
- If no title in metadata, return the file stem (the fallback).
- Strip whitespace and limit to 500 chars (the `Document.title` column width).
5. `app/rag/importer.py` — `_index_pdf_embedded_images(session, source, rel, full_path, llm, summary, raw_date)`:
- Extract embedded images via `extract_pdf_images(full_path)`.
- For each image:
- If `settings.images` is False → skip (log a debug message).
- If the image extension doesn't match `settings.image_extension_set` → skip (log a warning).
- Create a synthetic path for the embedded image: `<rel>#page=<N>#img=<index>.<ext>` (e.g., `report.pdf#page=3#img=0.png`).
- Call `_index_image_file` with the synthetic path and the image bytes (the `_index_image_file` function expects a `full_path` on disk — we need to either write a temp file or refactor `_index_image_file` to accept bytes directly).
- **Alternative approach:** instead of calling `_index_image_file`, handle the embedded image inline: copy the bytes to `settings.image_dir`, describe them via `describe_image`, create the image doc row, chunk + embed the description. This avoids the temp file and the `_index_image_file` refactoring.
Let me go with the inline approach — it's simpler and avoids refactoring `_index_image_file`:
```python
async def _index_pdf_embedded_images(...):
images = extract_pdf_images(full_path)
for img in images:
ext = _ext_from_mime(img.mime) # "image/png" → ".png"
if ext not in settings.image_extension_set:
logger.warning("import: PDF embedded image has unsupported extension: %s", img.mime)
continue
# Write to image_dir.
img_id = uuid.uuid4()
img_filename = f"{img_id}{ext}"
img_dir_path = Path(settings.image_dir)
img_dir_path.mkdir(parents=True, exist_ok=True)
img_path = str(img_dir_path / img_filename)
Path(img_path).write_bytes(img.data)
# Describe the image.
description = await _describe_or_skip(
llm, data=img.data, source=source, rel=f"{rel}#page={img.page}#img={img.index}",
mime=img.mime, full_path=Path(img_path),
)
if description is None:
summary.images_failed += 1
Path(img_path).unlink(missing_ok=True)
continue
# Create the image doc row.
img_doc = Document(
source=source,
path=f"{rel}#page={img.page}#img={img.index}{ext}",
full_path=img_path,
title=f"Embedded image from {full_path.stem} page {img.page}",
content=description,
content_hash=hashlib.sha256(img.data).hexdigest(),
indexed_at=datetime.now(UTC),
created_at=normalize_doc_date(raw_date),
is_image=True,
image_path=img_path,
)
session.add(img_doc)
session.flush()
# Chunk + embed the description.
chunks_text = chunk_document(description, img_doc.path,
settings.chunk_target_chars, settings.chunk_overlap_chars)
img_doc.chunks = [
Chunk(document_id=img_doc.id, position=i, content=c)
for i, c in enumerate(chunks_text)
]
session.flush()
try:
vectors = await llm.embed([c.content for c in img_doc.chunks])
for row, vec in zip(img_doc.chunks, vectors, strict=True):
row.embedding = vec
except EmbeddingError:
pass # fail soft, like the text path
# Summary = description (phase-122 convention for image docs).
img_doc.summary = description
is_summary_chunk = Chunk(
document_id=img_doc.id, position=-1, content=description,
is_summary=True,
)
img_doc.chunks.append(is_summary_chunk)
session.flush()
session.commit()
```
6. ASSUMPTION: none. The embedded image indexing reuses the same patterns as standalone image indexing (phase 122).
## Testing & Quality
- Unit: `tests/unit/test_pdf.py` (extend) — `_extract_pdf_title` (PDF with metadata title → returns it; PDF without → returns fallback).
- Integration: `tests/integration/test_importer.py` (extend) — seed a source with a PDF file → sync → the PDF document has extracted text, chunks, and embedding; embedded images (if `BOR_IMAGES=true`) become image documents.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] A PDF file in a source → the PDF document has extracted text, chunks, embedding, and summary.
- [ ] With `BOR_IMAGES=true`: embedded images become image documents.
- [ ] With `BOR_IMAGES=false`: embedded images are skipped.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
.agents/phases/todo/126_pdf_support/05_pdf_archive_and_tests.md
+60
View File
@@ -0,0 +1,60 @@
# Task 05 — PDFs in archives and full test suite
**Phase:** `126_pdf_support` · **Source:** `00_phase.md` — PDFs in archives, unit + integration + E2E tests
## Objective
Ensure PDFs inside uploaded archives are indexed, and write the full test suite (unit, integration, E2E).
## Work
1. `app/rag/archive_upload.py` — no code changes needed. Archives are unpacked into `upload_dir`, and the extension walk (`iter_importable_files`) already filters by `import_extension_set`. Adding `pdf` to the set means PDFs in archives are automatically picked up. Verify that the archive unpack doesn't skip `.pdf` files (it shouldn't — archives unpack all files).
2. `app/config.py` — no code changes needed. `pdf` is added to `BOR_IMPORT_EXTENSIONS` by the user (the config already accepts any well-formed extension via `import_extensions`). Document it in `.env.example` alongside the existing extension list:
```
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,pdf
```
3. `tests/unit/test_pdf.py` (new) — comprehensive unit tests:
- Create temp PDF fixtures using `fitz`:
- Single-page PDF with text → `extract_pdf_text` returns text with one page separator.
- Multi-page PDF (3 pages) → text with 3 page separators.
- PDF with no text (blank pages) → empty string.
- PDF with embedded images (draw shapes, add text) → `extract_pdf_images` returns the images.
- PDF with no embedded images → empty list.
- `_mime_from_image_info` (all known type codes → correct MIME; unknown → fallback).
- `_extract_pdf_title` (PDF with metadata → returns title; PDF without → returns fallback).
- Error handling: non-existent PDF → empty results.
- `tests/unit/test_chunker.py` (extend) — PDF-specific chunking tests:
- Single-page PDF → correct chunks with "Page 1:" prefix.
- Multi-page PDF → correct chunks with "Page N:" prefixes.
- Empty PDF → empty list.
- Large page → split by line.
4. `tests/integration/test_importer.py` (extend) — integration tests:
- Seed a source with a PDF file (create a temp PDF fixture) → sync → the PDF document has extracted text, chunks, and embedding.
- A PDF with embedded images + `BOR_IMAGES=true` → embedded images become image documents.
- A PDF with embedded images + `BOR_IMAGES=false` → embedded images are skipped.
- A PDF with no text → document is created with empty content (no chunks, no embedding).
- A PDF referenced in a markdown file → the markdown doc stores the reference info (phase 125 enrichment path).
5. `tests/e2e/test_pdf_support.py` (new) — isolated E2E suite:
- Module apps: one app with `BOR_IMAGES=true`, one with `BOR_IMAGES=false`.
- KB fixture: create a test PDF with text and an embedded image (using `fitz` in the test setup). Upload via the admin upload flow → sync.
- Verify: the PDF appears on the Sources page → open the document viewer → the extracted text renders → ask a question the PDF answers → the chat grounds on the PDF content.
- With `BOR_IMAGES=true`: the embedded image appears as a source chip (if retrieved).
- With `BOR_IMAGES=false`: no image chip, the PDF text is indexed normally.
- Run in isolation: `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov`.
6. Run the full test suite: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- All new logic covered by unit tests (PDF parsing, chunking, title extraction).
- Integration tests cover the full import flow with seeded PDF data.
- E2E tests cover the real user path: upload → sync → view → ask.
- No regression in existing tests (run `test_image_documents.py` and `test_image_anchor_context.py` to verify).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_pdf.py -v --no-cov` passes.
- [ ] `uv run pytest tests/integration/test_importer.py -v --no-cov` passes (extended).
- [ ] `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` passes in isolation.
- [ ] `uv run pytest` green; coverage TOTAL >90%; `ruff` + `pyright` clean.
- [ ] No regression in existing E2E suites.