brain-of-reese/.agents/phases/todo/137_pdf_image_context/00_phase.md

Phase 137 — PDF-embedded images always carry the document context in their summary

Source: TODO.md L3 — "Images extracted from PDFs still don't have any relevant context in their summaries provided by the PDF. They're still just getting summarized 'as-is'. … Image summaries should always include the document context if the images come from a document (this includes markdown files) so the embedder can more effectively retrieve image sources." Story: n/a (TODO-derived phase, Protocol B append). Context: app/rag/pdf.py (phase 126 — extract_pdf_text (L87, per-page get_text("text") with ---PAGE-N--- separators, fail-soft ""), extract_pdf_images (L145, one PdfImage per occurrence: page 1-indexed, index 0-based, data, mime)); app/rag/importer.py — _index_pdf_embedded_images (L1091, the per-PDF embedded-image pass after its document commit), _index_one_embedded_image (L1182, the phase-122 pipeline inline: mime gate → synthetic path <pdf>#page=N#img=I.<ext> → byte-digest skip → _markdown_image_context (L1403, the phase-125 reference scan) → _describe_or_skip (L1742, the single describe seam) → copy write → chunk + embed → _store_summary), _markdown_image_context (L1403) + _md_ref_block (L1436, the Referenced in <source>/<md>: block writer), _enrich_pending_images (L1627, the phase-125 task-04 post-index pass — the shape of the new backfill pass), ImportSummary (L212, stats counters + log() — stats are log-only, the sync detail API shape never changes); app/rag/summarizer.py — describe_image (L208, ONE chat-model vision call, context appended after the locked DESCRIBE_PROMPT as a \n\nThis image is referenced in documentation with the following context:\n prefix — the wording phase 137 generalizes), DESCRIBE_PROMPT / IMAGE_DESCRIPTION_MODE (L78/L57 — the mock's key; byte-identity criteria below); app/rag/prompts.py — _extract_context_line (L470, renders only Referenced in …: blocks, everything else degrades to None), _image_context_line (L536), _document_body (L548, the phase-125 context-line lead); app/models.py — Document.image_context (L170, TEXT nullable, migration 0023); tests/e2e/mock_llm.py — the IMAGE_DESCRIPTION_MODE canned vision reply (L2472 branch, IMAGE_DESCRIPTION_ANSWER L865) and the phase-125 CONTEXT_LINE_TRIGGER pattern (the model for this phase's mock trigger); tests/e2e/test_pdf_support.py (the synthetic PDF fixture recipe: PyMuPDF build, page2.insert_image(…, stream=PNG_1X1), images-on/off module apps); test_sources/players_handbook.pdf (293 pages, 586 embedded images — the real multi-page fixture), test_sources/hnb_2026_guide.tar.gz (the owner's example — stock-photo descriptions with no document context).

Objective

Every image that comes from a document carries the document context in the text that gets embedded: the vision prompt for a PDF-embedded image includes WHERE it came from (source + PDF path + page) and the surrounding page text, so the model writes a description that names the document and what the image illustrates; the context is stored on image_context (idempotent, re-scanned on re-index) and rendered as a short locator line in the chat suggestion block; and every image document already in the KB is backfilled on the next sync. Markdown-referenced images (phase 125) keep working byte-for-byte; PDFs are the only importable format that embeds images, so this closes the TODO's gap completely.

Dependencies

• 136_force_push_reset (todo) — pipeline order only.
• Code: the completed phase-126 PDF pipeline + the phase-122 vision seam + the phase-125 context machinery (all on main); and pending 133_pdf_screenshot_fallback — its is_usable_pdf_text gate (task 02 of that phase) runs BEFORE this phase in the pipeline and is reused to suppress garbage page text in the context (ASSUMPTION A7).

Design (shared by all tasks — the executor reads this, not the chat)

• The context block (task 02, ASSUMPTIONS A2/A3): one block per embedded image, built from the PDF's own text —

  Extracted from <source>/<pdf path> (page <N>):
  <page text, stripped, capped at PDF_IMAGE_CONTEXT_MAX_CHARS = 6000>
  

  The origin line is ALWAYS present (even when the page has no text — the owner's core complaint is "no mention of the document it came from"); the page-text line is omitted for a text-less page. If phase 133's gate exists (app.rag.pdf.is_usable_pdf_text) and says the PDF's text layer is unusable, the page text is dropped (origin line only — ASSUMPTION A7). When the image ALSO has phase-125 markdown reference blocks, the stored/passed context is the document block FIRST, then the markdown blocks (provenance → purpose — ASSUMPTION A2); the combining helper is one shared function used by the forward path AND the backfill (task 03) so the two can never drift.
• The seam (task 02, ASSUMPTION A5): describe_image's context prefix wording generalizes from "This image is referenced in documentation with the following context:" to neutral text that covers both provenance and reference (e.g. "This image is part of a document. The following context describes where it comes from and what it is used for:"). DESCRIBE_PROMPT (marker + locked instruction) stays byte-identical; context=None (standalone image) sends the bare prompt — byte-identical to today, so the phase-122 standalone path and every pre-existing E2E stay green.
• The forward path (task 02): _index_pdf_embedded_images calls the new extract_pdf_page_texts ONCE per PDF (a dict[int, str] page → stripped text — task 01) and passes it to _index_one_embedded_image, which builds the combined context, passes it as context to _describe_or_skip, and stores the SAME value on image_context (the phase-125 task-03 rule: context follows the current source state on re-index — a re-indexed image re-scans; the digest-skip for unchanged images is UNCHANGED: an unchanged image keeps its (now context-carrying) content).
• The backfill (task 03, ASSUMPTION A4): a post-index pass in import_sources next to _enrich_pending_images — the phase-118 A2 "best-effort until it sticks" shape. Candidates: is_image=True docs whose path matches the embedded-image pattern (*.pdf#page=<N>#img=<I>.<ext>) AND whose image_context does NOT start with Extracted from (NULL or markdown-only) AND whose image_path copy still exists on disk. Per candidate: bytes from the copy, page text re-extracted from the parent PDF if it is still in the source (parent gone → origin line only, parsed from the doc's own synthetic path — no disk dependency), re-describe through the SAME seam, re-chunk + re-embed (_replace_content_chunks, one transaction per image), store the combined context. Idempotent by the stored-context gate: a success writes a context starting with Extracted from → the doc never qualifies again. Fail-soft per image (a None description leaves the row untouched, counts images_failed, sync continues). New ImportSummary.image_context_backfilled: int = 0 counter + log() field ONLY (the stats log-only convention — the sync detail API shape is untouched). One loud log line per backfilled image (PLAN §9).
• The display line (task 04, ASSUMPTION A6): _extract_context_line learns the Extracted from <source>/<pdf path> (page <N>): header (regex; the page-text body is NEVER rendered — same rule as the Context: body) and renders it as From <source>/<pdf path> (page <N>). A value carrying BOTH block kinds renders the clauses in stored order (From …; Referenced in …). A value with no parsable block degrades to None exactly as today (pre-phase rows, markdown IMAGE_CONTEXT: records on text docs — the is_image guard already excludes those).
• The E2E (task 05): the mock gains ONE new trigger, following the phase-125 CONTEXT_LINE_TRIGGER pattern exactly: when the IMAGE_DESCRIPTION_MODE message's text part contains Extracted from (the new provenance header), the canned reply becomes a context-aware description (fixed string, e.g. naming the document + page) instead of IMAGE_DESCRIPTION_ANSWER — the deterministic observable that the context REACHED the vision model. The suite uploads a synthetic PDF (text on page 1, one embedded PNG on page 2 — the test_pdf_support.py recipe) through the images-on module app, syncs, and asserts: the image doc's content/summary is the context-aware canned reply; its image_context starts with Extracted from <source>/report.pdf (page 2); a chat question that retrieves the image shows the From <source>/report.pdf (page 2) line in the suggestion. A markdown-referencing case (a .md in the same archive referencing the synthetic path) asserts BOTH blocks, document first.

Tasks

1. 01_pdf_page_texts.md — extract_pdf_page_texts in app/rag/pdf.py (the shared per-page iteration, fail-soft) + unit tests on the real fixture PDFs.
2. 02_embedded_image_document_context.md — the document-context block builder + the forward path (build once per PDF, combine with markdown blocks, pass through the seam, store on image_context) + the generalized seam prefix + unit tests with a capturing fake LLM.
3. 03_backfill_image_context.md — the per-sync idempotent backfill pass + the image_context_backfilled counter + integration tests (existing row backfilled, idempotent no-op, parent-gone origin-only, standalone never touched).
4. 04_retrieval_display_line.md — _extract_context_line renders the Extracted from block as the one-line From … (page N) locator + unit tests (both-blocks ordering, degradation, pre-phase byte-identity).
5. 05_e2e_pdf_image_context.md — the mock's Extracted from trigger + the dedicated Playwright suite tests/e2e/test_pdf_image_context.py, run in isolation.

Testing & Quality

• Unit: task 01 — extract_pdf_page_texts on test_sources/players_handbook.pdf (293 pages, spot-check known page text, page count, 1-indexed keys) and pdf_horse_test.pdf; an unreadable path → {} (fail-soft); extract_pdf_text output byte-identical after the shared-iteration refactor (the phase-126 regression pin). Task 02 — the block builder (page text capped at 6000, text-less page → origin line only, garbage gate drops the page text when 133's gate exists); the combining order (document first, markdown second; markdown-only and document-only cases); the seam message (context present → neutral prefix + combined text AFTER the byte-identical DESCRIBE_PROMPT; context=None → byte-identical pre-137 message); the forward path via a capturing fake LLM (the stored image_context == the context passed; the title/unchanged-skip rules untouched). Task 04 — the renderer (Extracted from alone, both blocks in stored order, unparseable → None, the is_image guard). Task 05 — the mock trigger (context-aware reply only when Extracted from is in the text part; the bare IMAGE_DESCRIPTION_ANSWER otherwise — the phase-122/126 suites stay green without edits).
• Integration: task 03 — a seeded PDF-embedded image row without context is backfilled through the real import_sources (the fake LLM records the context-bearing describe call; content/chunks/image_context updated); a second sync is a NO-OP (zero describe calls — the idempotency gate); a row whose parent PDF was deleted from the source gets the origin-line-only context; a standalone image (no #page=) is NEVER touched; a markdown-enriched row (context Referenced in …) gets the combined context (document block prepended).
• E2E: one dedicated Playwright file, run in isolation (AGENTS.md §4/§9), mock-only.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).

Completion Criteria

• a (re)indexed PDF's embedded images are described with the document context: the vision message carries Extracted from <source>/<pdf> (page N): + the capped page text (the unit/integration proofs above), and the stored image_context/summary carry it too
• a markdown-referenced PDF image carries BOTH blocks, document first — and a standalone image's describe message is byte-identical to pre-phase (the phase-122 regression pin)
• an existing context-free image doc (e.g. one of the HNB-guide stock photos) is re-described WITH context on the next sync, and the following sync burns zero vision calls for it (the idempotency test)
• the chat suggestion block for a backfilled/forward image leads with From <source>/<pdf> (page N)
• uv run pytest --cov=app --cov-report=term-missing green, app/ coverage >90%; uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov green in isolation; the pre-existing test_pdf_support.py, test_image_anchor_context.py, test_image_documents.py suites stay green UNEDITED
• no behavior change in completed phases (the phase-122 standalone path, the phase-125 markdown enrichment, the phase-126 PDF text path, the sync API shape)
• uv run ruff check . && uv run pyright green
• one atomic Conventional Commits commit for the phase (feat(rag): …), --no-gpg-sign