brain-of-reese/app/rag/importer.py

"""Knowledge-base importer (PLAN §5 / §9 / §11).

Walks the in-scope files (the A9 family by default — the original seven
plus the quadlet family and ``j2`` — ``BOR_IMPORT_EXTENSIONS``, which may
name any well-formed extension; case-insensitive), diffs by sha256
against ``documents.content_hash`` and, for every new or changed file, runs
the two-phase upsert:

1. upsert the document row and replace its chunk rows (embeddings NULL)
2. embed the new chunks in batches and attach the vectors
3. commit — one transaction per file, so a failed embedding leaves the
   database untouched and the file is simply retried on the next run
4. every file (phase 30; phase 118, A2: markdown included — the
   non-markdown-only scope is retired): generate a ``lite``-model summary
   and, best-effort, store it on ``documents.summary`` plus one extra
   embedded chunk (``is_summary``, position −1). The document row and its
   content chunks are already committed at this point, so a summary
   failure only means the file is indexed without a summary (logged and
   counted in ``summary_errors``) — it is never lost.

Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
or hidden files) is skipped, plus the well-known exclusion list — UNLESS
the source's phase-105 hidden-folders flag admits dot-prefixed paths; the
exclusion list always applies. A token may also name extensionless files
by their exact lowercased full filename (``Dockerfile`` under the
``dockerfile`` token — phase 102).

``prune=True`` deletes documents (of the imported sources only) whose files
no longer exist **or no longer match the format filter** — this is how
previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
summary line with per-format counts (PLAN §9). Phase 122 prune guard
(LOCKED, derived from A3/A4): while the ``images`` toggle is OFF, an
``is_image`` doc is INVISIBLE to the walk, not a deleted file — prune
skips it (turning the toggle off and syncing must never destroy image
documents); a toggle-ON run prunes a deleted image file normally and
deletes its ``image_dir`` copy with the row.

Document dates (phase 106, D2/D4): every import sources
``documents.created_at`` from the file's source — the per-file git
last-commit date when a ``doc_dates_by_root`` entry names the file,
else the file's mtime — normalized by
:func:`app.rag.doc_dates.normalize_doc_date` (undetermined or future →
today, D3) on every add and update. On the unchanged path the stored
date is REFRESHED from the same source (it may go OLDER — no monotonic
guard) and counted in ``summary.dates_updated`` — unless the row
carries the owner's manual correction (``created_at_manual``, D1), which
the sync never touches.

NULL-summary backfill (phase 118, A2): an UNCHANGED file (same
``content_hash``) whose ``documents.summary`` is still NULL — a
pre-phase-30 row, or an earlier fail-soft miss — gets the same
best-effort summary pass on every sync until it sticks. A success counts
``summary_backfilled`` (never ``summaries``) and touches nothing else: no
content re-embed, no added/updated/pruned count — so no
``sources_meta`` bump, no KB-overview/folder-summary regeneration. The
backfill runs BEFORE the ``created_at_manual`` early-return (the manual
flag protects the DATE only, D1) and the strict ``is None`` check leaves
owner-set summaries (even empty strings, phase 57) alone.

Unchanged fast path (phase 147, task 02): the sync's single change
signal is the sha256 of the RAW source file's bytes
(:func:`_file_hash` — streamed in 1 MiB chunks, a large PDF is never
loaded fully), computed ONCE at the top of :func:`_index_file` BEFORE
the image/text/PDF dispatch and compared against
``documents.file_hash`` (migration 0025 — nullable, no index, internal
bookkeeping; DISTINCT from ``content_hash``, the FINAL indexed
content's digest, which stays the retriever's dedupe/caching identity).
A match returns through :func:`_unchanged_doc` — the ONE shared
unchanged path (the three branches' near-identical unchanged blocks,
one code site): the ``unchanged`` count + log line, the phase-118 (A2)
summary-NULL backfill from the STORED ``doc.content`` (never a
re-read/re-extract/re-describe — the digest-equality invariant), the
phase-106 (D1) manual-date early-return, and the (D4) date refresh
(may go older, counts ``dates_updated``) — with ZERO file read beyond
the streamed hash, ZERO PDF extraction, ZERO vision call, and ZERO
embed. Every create/update STAMPS ``file_hash`` (all three branches),
and the shared helper stamps too (a pre-phase NULL row, or a
bytes-changed/content-same file whose content digest still matches),
so the NEXT sync is always fast. Pre-phase rows (``file_hash`` NULL)
fall through to the branches' content-digest logic exactly as
pre-phase on the first post-upgrade sync (one full pass), then stamp
and go fast. The previous re-burn costs (a garbage PDF's fallback
vision re-issued each sync, a non-deterministic vision model flipping
a pictorial PDF's content digest) are gone: the file bytes decide.

Standalone images (phase 122, LOCKED A3): with the ``images`` toggle
(``BOR_IMAGES``) ON, the walk also admits the image extension set
(``BOR_IMAGE_EXTENSIONS`` — a SEPARATE set from ``import_extension_set``;
images are never user-added via ``BOR_IMPORT_EXTENSIONS``, the toggle is
the single knob). Such a file takes the binary index path
(:func:`_index_image_file`): the sha256 digest is over the raw BYTES
(content identity — the digest rule is unchanged), the bytes are copied
to the persistent home ``settings.image_dir/<doc-id>.<ext>`` (dir created
on demand; the copy is written ONLY after a successful description, so a
failure never leaves an orphan; a changed image deletes the stale copy
first; a pruned image doc deletes its copy), the row carries
``is_image=True`` + ``image_path``, and ``content`` is the vision
description — the ONLY embedded text of the document (the embedding model
never sees pixels; ``read_text`` is never called for an image). The
description comes through the single seam :func:`_describe_or_skip`
(task 03: :func:`app.rag.summarizer.describe_image` — ONE CHAT-model
(vision) call with the image bytes as a base64 data URL; the ``lite``
summary model is NOT assumed vision-capable, LOCKED A3); a failed/empty
description SKIPS the doc entirely (no row, no copy) — counted in
``images_failed`` + a warning, the sync continues (fail-soft). The normal
chunk pipeline then embeds ``content`` and the phase-30 summary path runs
on it — image-aware (task 03): for an image doc the description IS the
summary (stored verbatim, no ``lite`` call, no pointer line), so the
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
equals ``Document.content``.

PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
AFTER the image branch, BEFORE any text read — a PDF is never
``read_text``-ed; the binary is owned by the phase-126 parser). The
document's ``content`` is the extracted text (every text-bearing page
prefixed with the ``---PAGE-N---`` marker — A2, one document per PDF),
the digest is sha256 over that text (the digest rule is unchanged —
content identity), the title is the PDF's embedded metadata title or
the file stem, and the page-aware chunker (task 03) chunks it with a
``Page N:`` anchor. With the phase-122 ``images`` toggle ON, the
embedded images join the image pipeline (A3, :func:
`_index_pdf_embedded_images`): each is described through the SAME
``_describe_or_skip`` seam (the bytes as a data URL), copied to
``settings.image_dir`` (the copy is written only AFTER a successful
description — a failure never leaves an orphan), and indexed as a
standalone ``is_image`` document under the synthetic path
``<pdf path>#page=<N>#img=<index>.<ext>`` (the description IS the
content and summary — the phase-122 convention). Toggle OFF → the
embedded images are skipped (debug-logged) and the prune guard keeps
any existing image docs; an unsupported image mime (outside
``BOR_IMAGE_EXTENSIONS``) is skipped with a warning. A PDF's viewer
presentation is its extracted text (A4 — nothing binary is served).

Text-layer garbage gate + page-screenshot fallback (phase 133,
task 03): AFTER extraction and BEFORE the digest/chunk/summary
pipeline, the extracted text passes the deterministic garbage gate
(:func:`app.rag.pdf.is_usable_pdf_text` — empty text layer or
XML/binary soup, the pre-126 raw-bytes class). On failure the content
becomes the page-screenshot fallback (:func:
`_fallback_page_screenshots` — every page rendered to a PNG via
:func:`app.rag.pdf.render_pdf_pages` and read by the phase-122 vision
call, the SAME ``---PAGE-N---`` separators, the loud
``pdf: text layer unusable for …`` warning, the
``pdfs_screenshotted`` counter) — the owner's specified path for the
class of PDF whose text layer is garbage. A CLEAN text layer NEVER
burns a fallback vision call (the gate is the only gate — the
phase-126 happy path is byte-identical), and the digest runs over the
FINAL content exactly once — and an UNCHANGED re-sync is caught by the
phase-147 fast path BEFORE this pipeline runs at all (the raw file
bytes are the change signal: no re-extract, no re-render, no
re-issued vision call, no re-chunk/re-embed/re-summary, no KB change —
see :func:`_index_file`).

Pictorial sparse pages (phase 140, task 03, LOCKED A1–A4): AFTER the
phase-133 gate, a PDF whose text layer is USABLE gets the sparse-page
pass (:func:`_pictorial_pdf_pass`, gated by the ``pdf_pictorial``
setting — ``BOR_PDF_PICTORIAL``, the house kill-switch): the
deterministic gate (:func:`app.rag.pdf.pictorial_pages` — NO LLM
call, LOCKED A1) selects the pages whose stripped text is under
``pdf_pictorial_min_chars``; EVERY candidate is rendered (PLAN A21 —
no page cap, no call budget) and vision-described through the phase-
140 seam (:func:`app.rag.summarizer.describe_pdf_page` — the
``PICTORIAL_PAGE_MODE`` one-shot, fail-soft per page), and the
description appends to the page's content block as
``[Page N illustration: …]`` (LOCKED A2 — no LLM judge: every
described page joins the content; the owner's L5 rationale: more
information is better for the embeddings). The precedence is locked:
the pass runs on a USABLE layer ONLY — an unusable layer takes the
phase-133 full-page screenshot fallback and never burns a second
vision pass. Zero candidates → a no-op (no render, no vision call —
a clean-text PDF's path is byte-identical to pre-phase); the
counters ``pdfs_pictorial`` / ``pdf_pictorial_pages`` are log-only
(the sync ``detail`` API shape is untouched, the stats log-only
convention).

Persistent page copies (phase 148, task 01): in the PDF branch's
NEW/CHANGED path — after the final content is decided (the
fallback/pictorial passes have run) and after the row's flush (the
doc has its id) — a page-QUALIFYING PDF (the phase-133 fallback
fired, or the phase-140 pictorial pass described ≥1 page — the
sync's own decisions: text-heavy PDFs stay ineligible, and the
``pdf_pictorial`` kill switch keeps its semantics) gets EVERY page
rendered at index time (:func:`app.rag.pdf.render_pdf_pages` — 150
dpi, every page) and persisted to ``settings.pdf_page_dir``
(``BOR_PDF_PAGE_DIR``, the ``image_dir`` convention) as
``<doc-id>-p{N}.png`` (N 1-based), with ``documents.page_count``
(migration 0026) recording the page count — a per-page render
failure leaves a GAP (no file — the route 404s that N, the viewer's
note covers it). WRITE-FIRST (phase 163, task 02, LOCKED A4): the
fresh render's bytes land ATOMICALLY per file (``<name>.png.tmp`` +
``os.replace``) BEFORE any old copy is removed — only the old-extent
files the fresh render did NOT write (a shrunk PDF's ``p{N}`` tail)
go, so a rollback or crash can never restore a stale ``page_count``
over ZERO files. The row is stamped with the strip's CLASS beside the
count (``documents.page_strip_mode``, migration 0027 — ``primary`` for
the fallback class, ``additive`` for the pictorial class, phase 163);
a document that STOPS qualifying is cleared (both columns NULL +
copies deleted — deletion-first is correct where there is no
replacement set), and the prune deletes the copies with the row
(:func:`_delete_page_copies`). FAIL-SOFT: a render or write failure
leaves ``page_count``/``page_strip_mode`` NULL and logs a warning —
the document indexes EXACTLY as pre-phase (the render is a display
concern, it never gates the indexing).

PDF strip re-gate (phase 163, task 03, LOCKED A5): the UNCHANGED paths
never assume "the copies persist" any more — a row that still claims a
strip (``page_count`` set, a ``.pdf`` path) is re-gated
deterministically from the CURRENT bytes + CURRENT settings
(:func:`_regate_pdf_strip`, run at the tail of
:func:`_unchanged_doc`): the class is re-stamped (``primary`` for the
unusable-layer fallback class, ``additive`` for the sparse-page
class), any missing ``<doc-id>-p{N}.png`` is RE-RENDERED through the
same write-first atomic writer, and a strip that no longer qualifies
(lowered ``pdf_pictorial_min_chars``, the kill switch off) is cleared
with its copies. ZERO vision/LLM calls (the render is pure CPU, PLAN
A21-safe), one greppable INFO line only when the re-gate ACTS, and
fail-soft on every exception — the loss state (an external wipe, a
pre-phase delete-first rollback) cannot survive one sync. Every other
file's fast path still performs ZERO PDF parsing (the guard).

Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
description, the same source's importable markdown files are scanned for
``![alt](path)`` references that name the image (the task-02
:func:`app.rag.md_images.extract_image_refs`); when found, the combined
context string (``Referenced in <source>/<md>:`` + alt text + surrounding
paragraphs, see :func:`_markdown_image_context`) is (a) passed to the
vision call as ``context`` (appended AFTER the locked describe prompt —
the model learns what the image is for) and (b) stored on the doc's
``image_context`` (the retrieval display's context line, task 05), and
the stored ``content`` is ``<context>\n\n<description>`` (context first
for retrieval; ``summary`` mirrors it). A standalone image (no markdown
reference) gets ``image_context = NULL`` and the bare description —
byte-identical to pre-phase-125.

Markdown-FIRST image context (phase 125, task 04, LOCKED A1): a
markdown file (re)indexed while a reference's image has NO document in
the source yet records those references on ITS OWN ``image_context``
(the ``IMAGE_CONTEXT:`` record — :func:`_format_missing_image_refs`;
references are resolved exactly as the task-03 scan, and a reference
escaping the source is not recorded). A reference that DOES name an
existing image doc records nothing (the task-03 path owns that case —
enrichment at image-index time — and the markdown doc's field stays
NULL). After ALL of a source's files are indexed, the post-index
enrichment pass (:func:`_enrich_pending_images`) enriches the still-
``image_context IS NULL`` image docs that a markdown doc's record
references — the case where the markdown was indexed first and its
on-disk file is gone by the time the image is indexed (the task-03
disk scan cannot see it) — with the task-03 mechanics (re-describe
with context, ``content = <context>\n\n<description>``, the context
stored, re-chunk + re-embed). The pass is idempotent (an image doc
with ``image_context`` set is never touched; a markdown doc's record
is NOT cleared on success — it is the retrieval display's record
also, task 05).

PDF-embedded image document context (phase 137, task 02): a PDF's
embedded images are ALWAYS described with their document context —
the ``Extracted from <source>/<pdf path> (page <N>):`` provenance
block (see :func:`_pdf_image_context_block`: the origin line is
ALWAYS present — even for a text-less page; the page's own text
follows, stripped, cut at :data:`PDF_IMAGE_CONTEXT_MAX_CHARS` with
the shared :data:`~app.rag.retriever.TRUNCATION_MARKER` on its own
line; the page text is dropped entirely for a text-less page OR when
the phase-133 garbage gate (:func:`app.rag.pdf.is_usable_pdf_text`)
marks the PDF's text layer unusable — origin line only, A7) —
combined with any phase-125 markdown reference blocks (document block
FIRST — provenance before purpose — the single
:func:`_combine_image_contexts` combiner, shared with the task-03
backfill so the two paths can never drift). The per-page text map
(:func:`app.rag.pdf.extract_pdf_page_texts`) is extracted ONCE per
PDF for the whole image pass. The combined context is the vision
call's ``context`` (the generalized phase-125 prefix — the model
names the document and page in its description), the stored
``image_context`` (re-scanned on re-index — the context follows the
current source state; the unchanged-image digest skip is untouched),
and the ``content``/``summary`` enrichment
(``<context>\n\n<description>`` — the same text the embedder indexes).
Standalone images (no PDF) are untouched: no document block, the
phase-125 markdown shape unchanged, and the reference-free image's
bare prompt stays byte-identical.

Document-context backfill (phase 137, task 03): the per-sync,
idempotent, best-effort pass (:func:`_backfill_image_contexts`) runs
for each source immediately after the phase-125 enrichment pass, and
re-describes every PDF-embedded image doc that is still missing the
document block (``image_context`` NULL — a pre-phase-137 row with no
context at all — or markdown-only: the phase-125 ``Referenced in …``
value with the document block still missing). The bytes come from the
``image_dir`` copy, the page text is re-extracted from the parent PDF
while it is still in the source (the parent gone for good → the
origin line only — the origin is still valuable and needs no disk),
the fresh description goes through the SAME seam
(:func:`_describe_or_skip`), and the row's ``content``/``summary`` /
``image_context`` + chunks are replaced (one transaction per image).
A describe ``None`` or an embed failure leaves the row untouched and
retries next sync. A success writes the ``Extracted from …`` header
into ``image_context`` — the idempotency gate — so the doc never
qualifies again (the following sync burns zero vision calls for it).
Deliberately unchanged on the row: ``content_hash`` (the image
identity), ``created_at``/``created_at_manual`` (the SOURCE did not
change — the phase-106 date rules do not apply), ``title``, and
``image_path``. The new ``image_context_backfilled`` counter is
log-only (the sync ``detail`` shape is untouched — the stats log-only
convention, phase 133).

``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now, an
optional ``on_chunk_progress`` callback (phase 129, task 02)
reporting the in-flight file's embedding-chunk position at batch
granularity, and an optional ``on_image_progress`` callback (phase
152, task 02) reporting the in-flight PDF's vision-image position —
immediately before each of the PDF's vision describe calls, with the
image's 1-based position in the file's ONE continuous vision counter
and the span's count (see the function docstring).
"""
from __future__ import annotations

import hashlib
import logging
import os
import posixpath
import re
import uuid
from collections.abc import Callable, Sequence
from dataclasses import dataclass, field
from datetime import UTC, datetime
from pathlib import Path, PurePosixPath
from typing import Any, Protocol

import pymupdf as fitz  # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
from sqlalchemy import select
from sqlalchemy.orm import Session

from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.rag.chunker import chunk_document, extract_title
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
from app.rag.llm import EmbeddingError, LLMError
from app.rag.md_images import ImageRef, extract_image_refs
from app.rag.pdf import (
    PAGE_SEPARATOR,
    PdfImage,
    extract_pdf_images_with_stats,
    extract_pdf_page_texts,
    extract_pdf_text,
    is_usable_pdf_text,
    pictorial_pages,
    render_pdf_pages,
)
from app.rag.retriever import TRUNCATION_MARKER
from app.rag.summarizer import (
    IMAGE_FALLBACK_MIME,
    IMAGE_MIMES,
    describe_image,
    describe_pdf_page,
    generate_summary,
)

logger = logging.getLogger("app.importer")

#: Non-content directories never imported (PLAN anchor A9).
EXCLUDED_DIRS: frozenset[str] = frozenset(
    {".venv", "node_modules", ".git", "__pycache__", ".pytest_cache", "dist", "build"}
)

#: Markdown extensions that can carry ``![alt](path)`` image references
#: (phase 125, task 03) — the two markdown tokens of the A9 family. The
#: scan (see :func:`_markdown_image_context`) only reads these, and only
#: among the source's importable files.
_MARKDOWN_EXTENSIONS: frozenset[str] = frozenset({".md", ".markdown"})


class Embedder(Protocol):
    """Everything the importer needs from the LLM client (duck-typed for tests)."""

    settings: Settings
    embed_batches: int

    async def embed(
        self,
        texts: list[str],
        *,
        # Phase 129 (task 01): the per-batch position hook — ``None``
        # (every pre-phase caller) is a zero-cost guarded no-op.
        on_progress: Callable[[int, int], None] | None = None,
    ) -> list[list[float]]: ...

    async def chat(self, messages: list[dict[str, Any]], model: str | None = None) -> str: ...
    # ^ the one-shot completion the summarizer uses — the ``lite`` model
    #   for text summaries (phase 30, task 01) and the CHAT (vision)
    #   model for the phase-122 image description (multimodal content:
    #   a string or a list of OpenAI-compatible parts); :class:`app.rag.
    #   llm.LLMClient` satisfies it.


@dataclass
class ImportSummary:
    """Counts for one import run (also printed by the CLI)."""

    files: int = 0
    added: int = 0
    updated: int = 0
    unchanged: int = 0
    pruned: int = 0
    errors: int = 0
    chunks: int = 0
    embed_batches: int = 0
    #: Files whose lite summary was generated + indexed (phase 30; phase
    #: 118, A2: every A9 format, markdown included). One ``is_summary``
    #: chunk per success.
    summaries: int = 0
    #: Files whose summary generation failed (best-effort — the document
    #: is still indexed, without a summary).
    summary_errors: int = 0
    #: Unchanged docs whose NULL summary was backfilled (phase 118, A2) —
    #: one ``is_summary`` chunk per success; the content is untouched, so
    #: a backfill NEVER counts added/updated/pruned (no
    #: ``sources_meta`` bump, no overview/folder-summary regeneration).
    summary_backfilled: int = 0
    #: Files whose ``created_at`` was refreshed on the UNCHANGED path —
    #: content untouched, date re-sourced (phase 106, D4: the date may
    #: go OLDER; a date-only refresh NEVER counts added/updated/pruned,
    #: so no ``sources_meta`` bump, no overview/folder-summary
    #: regeneration).
    dates_updated: int = 0
    #: Image docs (phase 122, LOCKED A3) whose vision description failed
    #: or came back empty — the doc is SKIPPED entirely (no row, no
    #: ``image_dir`` copy): an undescribed image is unsearchable noise.
    #: Fail-soft: the sync continues, this counter + the warning line
    #: are the signal.
    images_failed: int = 0
    #: PDFs (phase 133, task 03) whose text layer failed the garbage
    #: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
    #: XML/binary soup) and took the page-screenshot vision fallback
    #: (:func:`_fallback_page_screenshots` — every page rendered to a
    #: PNG and read by the phase-122 vision call; the content is the
    #: ``---PAGE-N---``-anchored descriptions, or ``""`` when the
    #: fallback produced nothing — the phase-126 empty-content path).
    pdfs_screenshotted: int = 0
    #: PDFs (phase 140, task 03, LOCKED A2/A4) whose USABLE text layer
    #: carried ≥1 pictorial candidate page (``app.rag.pdf.
    #: pictorial_pages`` — the deterministic sparse-page gate, LOCKED
    #: A1) whose vision description was appended to the page block
    #: (``[Page N illustration: …]``) — the sparse-page pass
    #: (:func:`_pictorial_pdf_pass`). Log-only (the sync ``detail``
    #: API shape is untouched — the stats log-only convention,
    #: phase 133); ``pdf_pictorial_pages`` counts the described pages.
    pdfs_pictorial: int = 0
    #: Pictorial candidate pages described across the run (phase 140,
    #: task 03) — one PDF with three described sparse pages counts
    #: ``pdfs_pictorial=1`` + ``pdf_pictorial_pages=3`` (A21: no page
    #: cap — the count may be large, that is the point).
    pdf_pictorial_pages: int = 0
    #: PDF-embedded image docs whose stored ``image_context`` was
    #: backfilled with the document-context block (phase 137, task 03)
    #: — the one-time best-effort pass; a success never repeats, a
    #: failure retries next sync.
    image_context_backfilled: int = 0
    #: Image OCCURRENCES the PDF parser's dedupe dropped (phase 151,
    #: tasks 01/03) — repeated placements + byte-identical copies
    #: stored under distinct xrefs (one :class:`app.rag.pdf.PdfImage`
    #: per DISTINCT image, first placement wins). Log-only (the sync
    #: ``detail`` API shape is untouched — the stats log-only
    #: convention, phase 133): the owner reads it in the sync log.
    pdf_images_deduped: int = 0
    #: DISTINCT PDF-embedded images the parser's blank/decorative gate
    #: dropped (phase 151, task 03 —
    #: :func:`app.rag.pdf.is_blank_image` on the base samples): a
    #: near-white, nearly-uniform underlay (the TODO L4 "blank white
    #: images") is skipped ENTIRELY — no row, no ``image_dir`` copy,
    #: no vision call (ASSUMPTION A1). Log-only (the sync ``detail``
    #: API shape is untouched — the stats log-only convention,
    #: phase 133); while ``Settings.pdf_image_skip_blank`` (phase 151
    #: task 04, the kill switch) is ``False`` the gate is off and this
    #: count stays 0 (pre-phase behavior).
    pdf_images_blank_skipped: int = 0
    #: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
    #: tuples in encounter order (added/updated in walk order, pruned in
    #: _prune order). Drives the sync's incremental folder-summary
    #: regeneration (phase 127): len(changed_paths) == added + updated +
    #: pruned. Unchanged / backfilled / date-only refreshes NEVER record.
    changed_paths: list[tuple[str, str]] = field(default_factory=list)
    #: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
    formats: dict[str, int] = field(default_factory=dict)

    def format_counts(self) -> str:
        """``md:203,yaml:267,py:14`` — highest count first (PLAN §9)."""
        if not self.formats:
            return "none"
        ordered = sorted(self.formats.items(), key=lambda kv: (-kv[1], kv[0]))
        return ",".join(f"{ext}:{count}" for ext, count in ordered)

    def log(self) -> None:
        logger.info(
            "import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
            "errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
            "summary_backfilled=%d dates_updated=%d images_failed=%d "
            "image_context_backfilled=%d pdfs_screenshotted=%d "
            "pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
            "pdf_images_blank_skipped=%d formats=%s",
            self.files,
            self.added,
            self.updated,
            self.unchanged,
            self.pruned,
            self.errors,
            self.chunks,
            self.embed_batches,
            self.summaries,
            self.summary_errors,
            self.summary_backfilled,
            self.dates_updated,
            self.images_failed,
            self.image_context_backfilled,
            self.pdfs_screenshotted,
            self.pdfs_pictorial,
            self.pdf_pictorial_pages,
            self.pdf_images_deduped,
            self.pdf_images_blank_skipped,
            self.format_counts(),
        )


def normalize_ignore_path(entry: str) -> str:
    """One ignore-path entry → canonical form (phase 89, A1).

    Trim surrounding whitespace, then strip ALL leading/trailing
    ``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
    all become ``"my/files"``. ``""`` / ``"//"``,
    ``"  "`` normalize to ``""`` (callers drop empties).
    """
    return entry.strip().strip("/")


def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
    """Phase 89, A1 — the pure prefix rule, nothing else.

    ``rel`` is the source-relative POSIX path WITHOUT a leading
    slash (the ``documents.path`` string). Match = ``rel`` STARTS
    WITH a normalized entry: raw string prefix — deliberately NO
    component-boundary check (``"my/files"`` also matches
    ``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
    matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
    """
    return any(rel.startswith(p) for p in prefixes)


def _ignore_for_root(
    root: Path, ignore_by_root: dict[str, list[str]] | None
) -> tuple[str, ...]:
    """The normalized, non-empty prefix tuple for one root (phase 89).

    Keyed by ``str(root)`` — the root string exactly as the caller
    passed it in ``sources`` (unambiguous when two rows share a
    source *name* but different dirs). Callers may pass RAW box
    lines: the importer normalizes + drops empties here, the single
    choke point — stored lists (already normalized) normalize to
    themselves.
    """
    raw = (ignore_by_root or {}).get(str(root)) or []
    return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)


def _include_hidden_for_root(
    root: Path, include_hidden_by_root: dict[str, bool] | None
) -> bool:
    """The per-root hidden-folders flag (phase 105, A1).

    Keyed by ``str(root)`` — the root string exactly as the caller
    passed it in ``sources`` (the ``_ignore_for_root`` convention,
    phase 89): ``True`` only for roots the caller lists as True;
    unlisted/``None`` roots are ``False`` — every existing caller
    behaves byte-identically (A4).
    """
    return bool((include_hidden_by_root or {}).get(str(root), False))


def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
    """The bare lowercased token *path* imports under, or ``None``.

    1. Non-empty lowercased dotted suffix in *extensions* (the A9 rule —
       ``kubernetes.md`` → ``md``).
    2. No suffix: the lowercased FULL filename equals a bare token of
       *extensions* (``Dockerfile`` → ``dockerfile``) — the phase-102
       extensionless rule. Exact name only: ``mydockerfile`` never
       matches the ``dockerfile`` token.
    """
    suffix = path.suffix.lower()
    if suffix and suffix in extensions:
        return suffix.lstrip(".")
    if not suffix and path.name.lower() in {e.lstrip(".") for e in extensions}:
        return path.name.lower()
    return None


def iter_importable_files(
    root: Path,
    extensions: frozenset[str],
    excluded: frozenset[str] = EXCLUDED_DIRS,
    ignore: tuple[str, ...] = (),
    include_hidden: bool = False,
    image_extensions: frozenset[str] = frozenset(),
) -> list[Path]:
    """All importable files under *root* (sorted), per the A9 scope rules.

    *extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
    Skips: when ``include_hidden`` is False (the default), any path with a
    dot-prefixed component (hidden dirs/files — vendored caches like
    ``.esphome/.espressif/**``); when True, dot-prefixed components are
    ADMITTED (files inside hidden folders, and hidden files) and only
    *excluded* is consulted (A1 — caches/VCS internals are never content).
    The well-known non-content directories in *excluded* are skipped in
    BOTH states. *ignore* (phase 89, A1) is a
    tuple of ALREADY-normalized, non-empty source-relative path prefixes
    (the importer's ``_ignore_for_root`` is the normalization choke point
    — raw box lines never reach this function): a file is skipped when its
    source-relative POSIX path starts with any entry; the default ``()``
    keeps every existing caller byte-identical. The *ignore* tuple
    composes additively in both states.

    *image_extensions* (phase 122) is the lowercased dotted
    image-extension set admitted IN ADDITION to *extensions* — passed by
    :func:`import_sources` only while the ``images`` toggle is on (it
    reads ``llm.settings``; the image set is never merged into
    *extensions*). The empty default admits nothing: every existing caller
    (and the toggle-off walk) stays byte-identical to pre-phase.
    """
    if not root.is_dir():
        return []
    files: list[Path] = []
    for path in sorted(root.rglob("*")):
        if not path.is_file():
            continue
        rel = path.relative_to(root)
        if any(
            (not include_hidden and part.startswith(".")) or part in excluded
            for part in rel.parts
        ):
            continue
        if ignore and is_ignored(rel.as_posix(), ignore):
            continue
        matched = match_extension(path, extensions)
        if matched is None and image_extensions:
            # Phase 122: the image set is admitted IN ADDITION to the
            # import set (toggle on only — the caller passes it in).
            matched = match_extension(path, image_extensions)
        if matched is None:
            continue
        files.append(path)
    return files


#: The streamed-hash read size (phase 147, task 02) — 1 MiB: a
#: multi-hundred-MB PDF is hashed in a bounded stream, never loaded
#: fully into memory.
_HASH_CHUNK_SIZE = 1024 * 1024


def _file_hash(path: Path) -> str:
    """Phase 147 (task 02): the sha256 of *path*'s RAW bytes, streamed
    in 1 MiB chunks (the :data:`_HASH_CHUNK_SIZE` loop — a large file
    is never loaded fully into memory; the ``hashlib`` incremental
    pattern).

    The sync's SINGLE change signal — computed ONCE at the top of
    :func:`_index_file`, before the image/text/PDF dispatch, and
    compared against the stamped ``documents.file_hash`` (the fast
    path). Raw bytes on purpose (the owner's "hash the document
    contents"): every class hashes the same way — images, text, PDFs
    alike — and a bytes-identical file is an unchanged file, whatever
    its extraction/vision pipeline would produce.
    """
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while chunk := f.read(_HASH_CHUNK_SIZE):
            h.update(chunk)
    return h.hexdigest()


async def import_sources(
    sources: list[Path],
    llm: Embedder,
    *,
    prune: bool = False,
    limit: int | None = None,
    session: Session | None = None,
    progress: Callable[[str, str, int, int], None] | None = None,
    on_chunk_progress: Callable[[int, int], None] | None = None,
    on_image_progress: Callable[[int, int], None] | None = None,
    ignore_by_root: dict[str, list[str]] | None = None,
    include_hidden_by_root: dict[str, bool] | None = None,
    doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
) -> ImportSummary:
    """Import every A9-format file under *sources* (see module docstring).

    ``session`` may be supplied (tests); a private one is opened and closed
    otherwise. ``limit`` caps the number of files processed (debug only) and
    disables pruning, since an incomplete walk must not drive deletions.

    ``progress`` (phase 64, task 01) is an optional per-file hook called
    once per importable file, immediately before that file's
    ``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
    same POSIX *rel* the document rows use, ``done`` = the 1-based index of
    the current file **across all sources**, and ``total`` = the combined
    pre-walk count of importable files across all *sources* roots. The
    pre-walk (same extension/exclusion rules, directory stats only, no file
    reads) happens **only when *progress* is provided**: callers passing
    nothing pay no extra walk and behave exactly as before. Under
    ``limit``, the hook still fires per processed file only — ``done`` never
    exceeds the limit, but ``total`` stays the full pre-walk count (an
    incomplete walk must not misreport the denominator).

    ``on_chunk_progress`` (phase 129, task 02) is an optional
    chunk-position hook for the file currently being indexed: while that
    file's content chunks embed, it is called — through
    :meth:`app.rag.llm.LLMClient.embed`'s ``on_progress`` — before EACH
    embed batch with ``(first_chunk_1based, total_chunks)``: the 1-based
    position of the first chunk of the in-flight batch and the file's
    total content-chunk count (batch granularity, the phase-129 A6
    assumption — the transport embeds token-budgeted batches per HTTP
    call). It carries the in-flight FILE's chunk position only, never
    the file identity — the per-file ``progress`` hook owns that: the
    caller pairs them (a ``progress`` call names the file, and every
    chunk position reported until the next ``progress`` call belongs to
    it; the position climbs through the file's batches and restarts at
    1 for the next file). Same "only when provided" rule as
    ``progress``: ``None`` (every existing caller — the CLI, the chat
    path) is a zero-cost guarded no-op, and the run is byte-identical
    to a hook-less run. Only the content-chunk embeds report (the
    shared :func:`_replace_content_chunks` — text, image, and PDF
    branches; the image branch's one description chunk reports
    ``1/1``, which IS its only chunk): the document's summary-chunk
    embed (:func:`_store_summary`) NEVER receives the hook, so the
    counter counts content chunks only (no ``1/1`` noise after the
    real progress ends), and the post-index passes (the phase-125
    enrichment, the phase-137 document-context backfill) do not report
    either — the PDF's vision describe calls report through the
    separate ``on_image_progress`` hook instead (phase 152, below).

    ``on_image_progress`` (phase 152, task 02) is an optional
    vision-image-position hook for the PDF file currently being
    indexed: while the PDF's vision spans run — the phase-133
    page-screenshot fallback, the phase-140 pictorial sparse-page
    pass, and the phase-126 embedded-image pass — it is called
    immediately before each of their vision describe calls with
    ``(position_1based, total)``: the 1-based position of the image
    about to be described and the count for that span — the extracted
    image count (post-dedupe, post-blank-gate, phase 151: the work
    actually about to happen) for the embedded pass, the candidate-
    page count for the pictorial pass, the rendered-page count for the
    screenshot fallback. The spans of ONE file are reported as ONE
    continuous counter: each span is called with the count already
    reported for this file (*base*) and returns the number of images
    it reported, and the caller adds them — the pass that runs second
    starts where the first ended, so the position climbs across the
    file's whole vision span instead of restarting (the fallback and
    the pictorial pass are mutually exclusive — the phase-140
    precedence over the garbage gate — and both precede the embedded
    pass, which runs after the document commit). A skipped span (the
    ``images`` toggle off, zero candidates, a clean text layer) reports
    nothing and advances the counter not at all. Standalone image
    files get NO sub-position (phase 152 A4 — one image per file,
    ``1/1`` per file would be noise; the file count already covers
    them), and the hook carries counts only, never the image identity
    (A2 — the per-file ``progress`` hook names the file). Same "only
    when provided" rule as ``progress`` and ``on_chunk_progress``:
    ``None`` (every existing caller — the CLI, the chat path) is a
    zero-cost guarded no-op, and the run is byte-identical to a
    hook-less run.

    ``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
    string exactly as passed in *sources* — to that source's RAW ignore-path
    lines (the importer normalizes them via ``_ignore_for_root``, the single
    choke point): matching files are never walked, so they are never
    embedded and never summarized, and the progress pre-walk uses the same
    per-root tuple as the processing loop, so ``total`` never counts them.
    A file that newly matches a pattern simply never enters ``seen``, so the
    next ``prune=True`` run deletes its row automatically (A2 — the A9
    junk-precedent). ``None`` (the default) changes nothing: the map is read
    per root, unlisted roots get an empty tuple, and every existing caller
    behaves byte-identically.

    ``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the
    stored flag: ``True`` admits dot-prefixed components for that root
    (``EXCLUDED_DIRS`` and the extension filter still apply; the ignore
    tuple composes additively). Unlisted/``None`` roots are ``False`` —
    byte-identical to pre-phase-105. A file that was indexed with the
    flag ON and is walked again with it OFF simply never enters
    ``seen``, so the next ``prune=True`` run deletes its row
    automatically (A2 — the A9/phase-89 precedent).

    ``doc_dates_by_root`` (phase 106, D2/D4) maps ``str(root)`` — the
    root path string exactly as passed in *sources* — to that source's
    RAW per-file source dates: source-relative POSIX path → the git
    last-commit datetime (task 03's ``file_commit_dates``). ONLY git
    roots are listed — unlisted roots (local dirs, unpacked uploads)
    take the mtime fallback, and a path missing from its root's map
    does too. The map entry beats the file's mtime when present. The
    progress pre-walk is untouched (dates change no file count).
    ``None`` (the default) changes nothing for existing callers: the
    mtime fallback applies to every file — which IS the behavior
    change, D4: an unchanged file now refreshes its stored date from
    its source on every run (the backfill-correction case).

    Images (phase 122): when ``llm.settings.images`` is on, BOTH walks
    (the progress pre-walk and the processing loop — same rules, so
    ``total`` counts images) also admit ``llm.settings.image_extension_set``
    files, each indexed through the binary image path (see the module
    docstring). ``prune=True`` with the toggle ON prunes a deleted image
    file normally (row + ``image_dir`` copy); with the toggle OFF the
    prune skips ``is_image`` docs (the prune guard — the image is
    invisible to the walk, not a deleted file).

    Markdown-FIRST image context (phase 125, task 04): after ALL of a
    source's files are indexed, the post-index enrichment pass
    (:func:`_enrich_pending_images`) runs for that source — it enriches
    the still-unenriched image docs (``image_context IS NULL``) that a
    markdown doc's ``IMAGE_CONTEXT:`` record references (the case where
    the markdown was indexed first and its on-disk file is gone by the
    time the image is indexed — the task-03 disk scan cannot see it).
    The pass is driven by committed doc rows, so it is safe under
    ``limit`` too (it deletes nothing), and a per-image failure
    (missing copy, failed re-description, embed error) keeps that image
    as-is and moves on.

    Phase 137 (task 03): the document-context backfill
    (:func:`_backfill_image_contexts`) runs for that source immediately
    AFTER the enrichment pass (enrichment first: it owns the
    markdown-FIRST case and may have just set a markdown-only context,
    which the backfill then prepends the document block to). It
    re-describes the PDF-embedded image docs a pre-phase-137 sync left
    with ``image_context`` NULL or markdown-only — the bytes from the
    persistent copy, the page text re-extracted from the parent PDF
    while it is still in the source (gone → the origin line only), the
    SAME describe seam, and the row's ``content``/``summary`` /
    ``image_context``/chunks replaced. Idempotent (a success writes the
    ``Extracted from …`` header — the doc never qualifies again, the
    following sync burns zero vision calls for it) and fail-soft (a
    failure leaves the row untouched — the next sync retries it).
    Driven by committed doc rows, so it is safe under ``limit`` too
    (it deletes nothing).
    """
    if limit is not None and limit <= 0:
        raise ValueError("limit must be >= 1")
    summary = ImportSummary()
    # Phase 122: the image set is admitted by the walks ONLY while the
    # toggle is on — the empty set admits nothing, so the toggle-off run
    # (walk, counts, prune) stays byte-identical to pre-phase.
    image_exts = llm.settings.image_extension_set if llm.settings.images else frozenset()
    owns_session = session is None
    if session is None:
        session = SessionLocal()
    seen: set[tuple[str, str]] = set()
    source_names: set[str] = set()
    # phase 64 (task 01): the hook's combined denominator, walked with the
    # exact same rules as the processing loop below (directory stats only,
    # no file reads). Skipped entirely for ``progress=None`` callers — no
    # extra pass, byte-identical behaviour and cost.
    total = 0
    if progress is not None:
        for root in sources:
            total += len(
                iter_importable_files(
                    root,
                    llm.settings.import_extension_set,
                    ignore=_ignore_for_root(root, ignore_by_root),
                    include_hidden=_include_hidden_for_root(
                        root, include_hidden_by_root
                    ),
                    image_extensions=image_exts,
                )
            )
    try:
        for root in sources:
            if not root.is_dir():
                logger.warning("import: source dir not found, skipping: %s", root)
                continue
            if limit is not None and summary.files >= limit:
                break
            source = root.name
            source_names.add(source)
            ignore = _ignore_for_root(root, ignore_by_root)
            include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
            # Phase 106 (D2): the root's raw source dates (git last-commit
            # for git roots, keyed by the same str(root) convention); {}
            # for unlisted roots — every file then takes the mtime fallback.
            dates_map = (doc_dates_by_root or {}).get(str(root), {})
            for path in iter_importable_files(
                root,
                llm.settings.import_extension_set,
                ignore=ignore,
                include_hidden=include_hidden,
                image_extensions=image_exts,
            ):
                if limit is not None and summary.files >= limit:
                    break
                rel = path.relative_to(root).as_posix()
                seen.add((source, rel))
                summary.files += 1
                # Phase 102: the matched bare token (``dockerfile`` for an
                # extensionless ``Dockerfile``), never ``unknown`` — the
                # file is in scope, so the walk matched it. Phase 122: an
                # image file matches the image set, not the import set.
                ext = (
                    match_extension(path, llm.settings.import_extension_set)
                    or match_extension(path, image_exts)
                    or "unknown"
                )
                summary.formats[ext] = summary.formats.get(ext, 0) + 1
                if progress is not None:
                    # phase 64: report the file *before* indexing it — a
                    # file that then errors or turns out unchanged was
                    # already the "current file". No try/except around the
                    # call: the hooks in this repo only assign fields.
                    progress(source, rel, summary.files, total)
                try:
                    await _index_file(
                        session, source=source, rel=rel, full_path=path, llm=llm,
                        summary=summary, raw_date=dates_map.get(rel),
                        on_chunk=on_chunk_progress, on_image=on_image_progress,
                    )
                except EmbeddingError as e:
                    # A pathological file (e.g. content the embedding endpoint
                    # refuses) must not abort the whole KB: roll back its
                    # uncommitted rows, log loudly, and keep going. The next
                    # run retries it.
                    session.rollback()
                    summary.errors += 1
                    logger.error("import: error source=%s path=%s — %s", source, rel, e)
            # Phase 125 (task 04): the post-index enrichment pass — the
            # markdown-FIRST case (a markdown indexed while its
            # referenced image had no document yet recorded the
            # reference on its doc row; the image was just indexed, but
            # the task-03 disk scan could not see the reference — the
            # markdown file is no longer on disk). Idempotent +
            # fail-soft (see the function).
            await _enrich_pending_images(session, source=source, llm=llm, summary=summary)
            # Phase 137 (task 03): the one-time document-context
            # backfill — the PDF-embedded image docs a pre-phase-137
            # sync left WITHOUT the document block (``image_context``
            # NULL or markdown-only) get it now: re-described with the
            # context, the row's content/summary/context/chunks
            # replaced. Idempotent + fail-soft (see the function).
            await _backfill_image_contexts(
                session, source=source, root=root, llm=llm, summary=summary
            )
        if prune:
            if limit is not None:
                logger.warning("import: --prune ignored because --limit was given")
            else:
                summary.pruned = _prune(
                    session, source_names, seen, images=llm.settings.images,
                    summary=summary,
                    page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
                )
        summary.embed_batches = llm.embed_batches
        summary.log()
        return summary
    finally:
        if owns_session:
            session.close()


async def _replace_content_chunks(
    session: Session,
    *,
    doc: Document,
    llm: Embedder,
    content: str,
    rel: str,
    on_chunk: Callable[[int, int], None] | None = None,
) -> int:
    """Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace
    the doc's content chunk rows (embeddings NULL), and embed the new
    chunks in batches — returning the chunk count. The caller owns the
    transaction (it commits, or rolls back on the :class:`EmbeddingError`
    that propagates when the token-cap retry gives up).

    The endpoint token-cap retry (halve this file's chunk target, floor
    400 — URL-dense paragraphs tokenize at ~1.1 chars/token; the global
    policy stays intact for the rest of the KB) is the house rule, shared
    verbatim by the text path (:func:`_index_file`), the image path
    (task 03), and the task-04 enrichment pass — one copy, three call
    sites.

    ``on_chunk`` (phase 129, task 02) is the in-flight file's
    chunk-position hook, forwarded as ``on_progress`` to the content
    embed: it fires before each top-level embed batch with
    ``(first_chunk_1based, total_chunks)``. The token-cap re-chunk retry
    (the ``while True`` above that halves *target*) re-embeds the NEW
    (smaller-target) chunk list, and the hook reports against the NEW
    total automatically — no special case: the announced total is
    always the chunk list actually in flight. The empty-content
    ``doc.chunks = []`` break never embeds, so the hook never fires for
    it. ``None`` (the default; every pre-phase call site — the CLI, the
    phase-125 enrichment pass, and the PDF's embedded-image docs, which
    index separate documents rather than the walked file) keeps the call
    byte-identical.
    """
    settings = llm.settings
    target = max(400, settings.chunk_target_chars)
    chunks_text: list[str] = []
    while True:
        chunks_text = chunk_document(content, rel, target, settings.chunk_overlap_chars)
        doc.chunks = [
            Chunk(document_id=doc.id, position=i, content=c) for i, c in enumerate(chunks_text)
        ]
        session.flush()  # delete-orphan cascade drops the previous rows
        if not doc.chunks:
            break
        try:
            # Phase 129 (task 02): the in-flight file's chunk-position
            # hook (``on_progress`` — the task-01 contract; ``None`` is a
            # zero-cost no-op).
            vectors = await llm.embed(
                [c.content for c in doc.chunks], on_progress=on_chunk
            )
            for row, vec in zip(doc.chunks, vectors, strict=True):
                row.embedding = vec
            break
        except EmbeddingError as e:
            if "token cap" not in str(e) or target <= 400:
                raise
            logger.info(
                "import: re-chunking at %d chars after endpoint token cap: %s",
                target // 2,
                rel,
            )
            target //= 2
    return len(chunks_text)


async def _unchanged_doc(
    session: Session,
    doc: Document,
    *,
    source: str,
    rel: str,
    llm: Embedder,
    summary: ImportSummary,
    raw_date: datetime,
    file_digest: str,
    full_path: Path | None = None,
) -> None:
    """Phase 147 (task 02): the ONE shared unchanged path — the three
    branches' near-identical unchanged blocks, one code site, reached
    from two places:

    * the FAST PATH at the top of :func:`_index_file` (a
      ``doc.file_hash == file_digest`` match — the raw bytes are what
      was indexed last time): zero file read beyond the streamed
      hash, zero extraction, zero vision, zero embed;
    * each branch's CONTENT-digest hit (a pre-phase NULL-hash row, or
      a bytes-changed/content-same file whose final content digest
      still matches): the branch's own expensive work already ran,
      but the row is skipped exactly as pre-phase.

    The preserved semantics, in order (the per-branch blocks' counts,
    log wordings, and commits — verbatim):

    1. ``summary.unchanged`` + the ``import: unchanged`` log line;
    2. the STAMP: ``doc.file_hash = file_digest`` (a pre-phase NULL
       row — or a bytes-changed/content-same row — learns its hash
       here, so the NEXT sync takes the fast path; a no-op on the
       fast path, where the DB value already equals the digest);
    3. the phase-118 (A2) backfill when ``doc.summary is None`` —
       from the STORED ``doc.content``: EXACTLY what the file's bytes
       produced is already stored (text: the file text; image: the
       vision description; PDF: the final constructed content — the
       digest-equality invariant), so the backfill never re-reads the
       file, never re-extracts, never re-describes (strict
       ``is None`` — an owner-set empty string, phase 57, is never
       touched);
    4. the phase-106 (D1) early-return — ``created_at_manual``
       protects the DATE: no date write, the manual correction
       survives (the phase-97 ``manually_edited`` precedent); the
       stamp, if dirty, is the only write, and it commits so the next
       sync is fast even for a manual-date row;
    5. the phase-106 (D4) date refresh — ``normalize_doc_date`` of
       the raw date, committed + counted in ``dates_updated`` + the
       ``import: date-refreshed`` line when it changes (it may go
       OLDER — no monotonic guard);
    6. the stamp's final commit — when the row is still dirty after
       the date block (the stamp set ``file_hash`` and the date did
       not change): quiet bookkeeping — no count, no log (the
       ``unchanged`` line already fired; a date-change run already
       committed in step 5);
    7. the phase-163 (task 03) PDF strip RE-GATE — a row that still
       claims a strip (``page_count`` set) whose path ends in ``.pdf``
       goes through :func:`_regate_pdf_strip`: the strip is re-derived
       from the CURRENT bytes + CURRENT settings (deterministic, ZERO
       LLM/vision calls) — a missing/changed ``page_strip_mode`` is
       re-stamped, copies that vanished from disk are RE-RENDERED, a
       strip that no longer qualifies is cleared — and whatever it
       wrote commits in this helper's transaction. Unchanged files
       re-gate the PDF strip deterministically (phase 163): the
       pre-phase "the copies persist" assumption is gone, so a wipe of
       ``pdf_page_dir`` or a pre-phase delete-first rollback cannot
       outlive one sync. Every other file (``page_count`` NULL) pays
       ZERO extra read — the guard keeps the fast path's contract, and
       BOTH exits (the D1 manual-date return and the normal tail) run
       the re-gate.
    """
    summary.unchanged += 1
    logger.info("import: unchanged source=%s path=%s", source, rel)
    # 2 — the stamp (a no-op on the fast path; a real write on the
    # branch paths — the next sync is then fast either way).
    doc.file_hash = file_digest
    # 3 — the phase-118 (A2) summary-NULL backfill, from the stored
    # content (see the docstring — never a re-read/re-extract).
    if doc.summary is None:
        await _store_summary(
            session, doc=doc, source=source, rel=rel, content=doc.content,
            llm=llm, summary=summary, backfill=True,
        )

    def _regate_strip() -> None:
        """Step 7 (phase 163, task 03): the GUARDED strip re-gate — only
        a row that carries a strip (``page_count`` set) on a ``.pdf``
        path reaches :func:`_regate_pdf_strip`, so every other file's
        fast path stays byte-identical (zero extra read, zero PDF
        parse). The re-gate never raises (fail-soft inside); its writes
        ride this helper's transaction."""
        if doc.page_count is None or not str(doc.path).lower().endswith(".pdf"):
            return
        _regate_pdf_strip(
            session,
            doc,
            full_path=full_path if full_path is not None else Path(doc.full_path),
            llm=llm,
            summary=summary,
        )
        if session.is_modified(doc):
            session.commit()

    # 4 — the phase-106 (D1) manual-date lock: no date write; the
    # stamp, if dirty, is the only write (committed — see step 2's
    # "next sync takes the fast path") — plus whatever the phase-163
    # re-gate repaired.
    if doc.created_at_manual:
        _regate_strip()
        if session.is_modified(doc):
            session.commit()
        return
    # 5 — the phase-106 (D4) date refresh: the stored date follows the
    # source on every sync and may go OLDER (no monotonic guard); a
    # date-only refresh counts ``unchanged`` — never added/updated/
    # pruned, so no ``sources_meta`` bump and no regeneration.
    target = normalize_doc_date(raw_date)
    if target != doc.created_at:
        doc.created_at = target
        session.commit()
        summary.dates_updated += 1
        logger.info(
            "import: date-refreshed source=%s path=%s date=%s",
            source, rel,
            doc.created_at.isoformat(),
        )
    elif session.is_modified(doc):
        # 6 — the stamp's commit (the date did not move): quiet
        # bookkeeping — the ``unchanged`` line already fired.
        session.commit()
    # 7 — the phase-163 (task 03) strip re-gate (see the local helper).
    _regate_strip()


def _regate_pdf_strip(
    session: Session,
    doc: Document,
    *,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
) -> None:
    """Phase 163 (task 03, LOCKED A5): the DETERMINISTIC re-gate of a
    PDF's display strip — run from the unchanged paths
    (:func:`_unchanged_doc`) whenever a row still claims a strip
    (``page_count`` set), so a strip can never die silently.

    WHY (the 2026-10-07 live state, owner report): the ``Letumdei`` row
    carried ``page_count = 36`` over ZERO files — the pre-phase-163
    :func:`_persist_pdf_pages` deleted the old copies BEFORE rendering,
    OUTSIDE the DB transaction, so a later per-file rollback (the
    import loop's ``EmbeddingError`` handler) restored the stale count
    over an empty directory; an external wipe of ``pdf_page_dir`` lands
    exactly the same state. The phase-147 fast path believed "the
    copies persist" and never re-rendered, so every
    ``GET /api/documents/<id>/pages/N`` 404'd and the viewer showed
    "Page N unavailable." on all 36 pages — on every sync, forever.

    WHAT IT DOES — the changed-path gate mirrored exactly, from the
    CURRENT file bytes + CURRENT settings, with ZERO vision/LLM calls
    (:func:`app.rag.pdf.render_pdf_pages` is pure CPU — PLAN A21-safe,
    no model, no budget):

    * ``not is_usable_pdf_text(extract_pdf_text(full_path))`` → the
      phase-133 fallback class → mode ``"primary"`` (the strip IS the
      view — the same doc as today);
    * else ``settings.pdf_pictorial`` ON and
      :func:`app.rag.pdf.pictorial_pages` NON-EMPTY over
      :func:`app.rag.pdf.extract_pdf_page_texts` → the phase-140
      pictorial class → mode ``"additive"`` (the text view + the strip
      behind a toggle). The candidates are the existing gate's
      VERBATIM (text-carrying pages under ``min_chars``; a page with no
      text has no key in the map and is not a candidate) — the re-gate
      adds NO new eligibility semantics;
    * else the strip no longer qualifies (the owner lowered
      ``pdf_pictorial_min_chars``, the kill switch went off, the layer
      gained substance) → the REPAIR arm: the copies go
      (``_delete_page_copies`` reads ``doc.page_count``, so it runs
      while the count still describes the extent) and BOTH columns are
      cleared.

    For a re-gated-ELIGIBLE doc the copies are VERIFIED: any missing
    ``<doc-id>-p{N}.png`` in 1..``page_count`` re-renders EVERY page
    through the changed path's own write-first atomic writer
    (:func:`_write_page_copies` — tmp + ``os.replace``), then the
    stale tail beyond the new count goes and ``page_count`` follows the
    fresh render's length (a drift names both counts in a WARNING — on
    unchanged bytes a drift is pathological). A fresh render producing
    NO pages (unopenable right after the fast path hashed the file)
    leaves the row exactly as it was + a WARNING: the re-gate never
    clears a strip on a transient.

    The mode re-stamp is SILENT when it only fills a NULL (a pre-phase
    row learning its class is beneficial bookkeeping — it rides the
    same INFO line only when real work followed); a change of an
    ALREADY-SET mode (the kill switch, a class flip) says so
    (``mode <old>-><new>``). The ONE greppable INFO line fires only
    when the re-gate ACTS — never per unchanged doc.

    FAIL-SOFT: the whole body is wrapped — any exception is a WARNING
    and the sync continues (the strip is a display concern). A repair's
    DB writes ride the caller's transaction, so an exception raised
    after them rolls back with the file like any other per-file
    failure. *summary* keeps the unchanged path's signature shape (and
    a future counter site); the re-gate books nothing there — its stats
    are log-only (the phase-133/140 convention).
    """
    source, rel = doc.source, doc.path
    try:
        page_dir = Path(llm.settings.pdf_page_dir).expanduser()
        # The gate, mirrored from the changed path (the phase-133
        # precedence over the phase-140 pass, the kill switch included).
        if not is_usable_pdf_text(extract_pdf_text(full_path)):
            strip_class: str | None = "fallback"
        elif llm.settings.pdf_pictorial and pictorial_pages(
            extract_pdf_page_texts(full_path),
            min_chars=llm.settings.pdf_pictorial_min_chars,
        ):
            strip_class = "pictorial"
        else:
            strip_class = None
        if strip_class is None:
            # The REPAIR arm — the strip no longer qualifies: delete
            # while the count still names the extent, then clear both
            # columns (the same clearing the changed path does for a
            # doc that stopped qualifying).
            _delete_page_copies(doc, page_dir)
            doc.page_count = None
            doc.page_strip_mode = None
            logger.info(
                "import: PDF strip re-gate source=%s path=%s — cleared stale strip",
                source, rel,
            )
            return
        mode = "primary" if strip_class == "fallback" else "additive"
        old_mode = doc.page_strip_mode
        old_count = doc.page_count or 0
        actions: list[str] = []
        missing = [
            n
            for n in range(1, old_count + 1)
            if not (page_dir / f"{doc.id}-p{n}.png").is_file()
        ]
        if missing:
            pages = render_pdf_pages(full_path)
            if not pages or not any(pages):
                # Unopenable (or every page a placeholder) — right
                # after the fast path hashed the file this is
                # pathological: KEEP the row as-is (never clear a strip
                # on a transient) and let the next sync try again.
                logger.warning(
                    "import: PDF strip re-gate source=%s path=%s — re-render "
                    "produced no pages (state kept)",
                    source, rel,
                )
                return
            page_dir.mkdir(parents=True, exist_ok=True)
            written_ns = _write_page_copies(doc, page_dir, pages)
            _delete_stale_page_tail(
                doc_id=doc.id,
                page_dir=page_dir,
                old_extent=old_count,
                written_ns=written_ns,
            )
            doc.page_count = len(pages)
            actions.append(f"healed {len(missing)} pages")
            if doc.page_count != old_count:
                logger.warning(
                    "import: PDF strip re-gate source=%s path=%s — page_count "
                    "drift %s->%s on unchanged bytes",
                    source, rel, old_count, doc.page_count,
                )
        if doc.page_strip_mode != mode:
            # NULL -> stamped stays SILENT (it rides the line only when
            # work followed); a change of an already-set mode is news.
            doc.page_strip_mode = mode
            if old_mode is not None:
                actions.append(f"mode {old_mode}->{mode}")
        if actions:
            logger.info(
                "import: PDF strip re-gate source=%s path=%s — %s",
                source, rel, " | ".join(actions),
            )
    except Exception as e:
        # Fail-soft: the strip never breaks a sync.
        logger.warning(
            "import: PDF strip re-gate failed source=%s path=%s — %s",
            source, rel, e,
        )


async def _index_file(
    session: Session,
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
    raw_date: datetime | None = None,
    on_chunk: Callable[[int, int], None] | None = None,
    on_image: Callable[[int, int], None] | None = None,
) -> None:
    """Upsert one file: doc row + chunk rows + embeddings, one transaction.

    Phase 147 (task 02): the UNCHANGED FAST PATH. BEFORE the
    image/text/PDF dispatch, the file's RAW source bytes are hashed
    (:func:`_file_hash` — streamed), the ``(source, path)`` row is
    looked up, and the D2 mtime fallback for *raw_date* runs (once,
    here — the branches receive the already-computed date) — a
    ``doc.file_hash == file_digest`` match returns through the shared
    unchanged path (:func:`_unchanged_doc`) with ZERO file read beyond
    the streamed hash, ZERO PDF extraction, ZERO vision call, and
    ZERO embed (the preserved bookkeeping: the phase-118 summary-NULL
    backfill from the STORED ``doc.content``, the phase-106 D1
    manual-date lock, the D4 date refresh). A pre-phase row
    (``file_hash`` NULL) or a bytes-changed file falls through to the
    branches below, which keep their own lookups + content-digest
    checks (the content digest then decides changed vs. the same
    shared path) and STAMP ``file_hash`` on every create/update — so
    the next sync is always fast.

    ``raw_date`` (phase 106, D2) is the file's RAW source date — the
    git last-commit datetime from the caller's ``doc_dates_by_root``
    map, or ``None`` (every non-git case): the file's mtime is read
    here, once, and becomes the source date (the D2 fallback).

    ``on_chunk`` (phase 129, task 02) is the in-flight file's
    chunk-position hook (see :func:`import_sources`) — passed through to
    ALL THREE branches' content embeds: the text path's
    :func:`_replace_content_chunks` directly, the image branch via
    :func:`_index_image_file` (its one description chunk reports
    ``1/1`` — it IS the file's only chunk), and the PDF branch via
    :func:`_index_pdf_file` (its content embed is the primary target).
    ``None`` (the default) keeps every branch byte-identical.

    ``on_image`` (phase 152, task 02) is the in-flight file's
    vision-image-position hook (see :func:`import_sources`) — passed
    through to the PDF branch only (:func:`_index_pdf_file` — its
    three vision spans share ONE continuous counter per file): the
    image branch receives NO sub-position (phase 152 A4 — one image
    per file, ``1/1`` would be noise; the file count already covers
    it) and the text branch has no vision span. ``None`` (the
    default) keeps every branch byte-identical.

    Image files (phase 122, toggle on) delegate to
    :func:`_index_image_file` — the binary path (bytes digest,
    persistent copy, ``content`` = the vision description) — BEFORE
    any text read: ``read_text`` is never called for an image.
    """
    settings = llm.settings
    # Phase 147 (task 02): the unchanged fast path — the RAW source-
    # file-bytes digest is the single change signal, computed ONCE
    # here, before the image/text/PDF dispatch, and compared against
    # the stamped ``documents.file_hash``. A match returns through the
    # shared unchanged path with zero further work — no
    # ``read_text``/``read_bytes``, no ``extract_pdf_text``, no
    # fallback/pictorial vision, no re-chunk/re-embed/re-summary.
    file_digest = _file_hash(full_path)
    doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
    if raw_date is None:
        # D2 fallback: no source date in the map → the file's mtime
        # (one stat). Read before the unchanged early-return — the
        # fast path (and the branches' unchanged path) refresh the
        # stored date from the same source.
        raw_date = file_mtime_datetime(full_path)
    if doc is not None and doc.file_hash == file_digest:
        # The fast path: the raw bytes are what was indexed last time
        # — the bookkeeping (backfill / D1 / D4) runs in the shared
        # helper, nothing else does.
        await _unchanged_doc(
            session, doc, source=source, rel=rel, llm=llm, summary=summary,
            raw_date=raw_date, file_digest=file_digest, full_path=full_path,
        )
        return
    # Phase 122 (task 02): the image branch FIRST. Only reachable while
    # the ``images`` toggle is on — the walk never admits image files
    # while it is off, and with it off this check is a no-op (the text
    # path below stays byte-identical to pre-phase). Phase 147: this
    # point is reached only for a pre-phase NULL-hash row or a
    # bytes-changed file — the branch's own digest then decides.
    if settings.images:
        image_set = settings.image_extension_set
        if match_extension(full_path, image_set) is not None:
            return await _index_image_file(
                session, source=source, rel=rel, full_path=full_path, llm=llm,
                summary=summary, raw_date=raw_date, file_digest=file_digest,
                on_chunk=on_chunk,
            )
    # Phase 126: PDF files take the PDF index path — AFTER the image
    # branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
    # owns it and ``read_text`` is never called for one). Phase 147:
    # reached only for a pre-phase NULL-hash row or a bytes-changed
    # file — the branch's own digest then decides.
    if full_path.suffix.lower() == ".pdf":
        return await _index_pdf_file(
            session, source=source, rel=rel, full_path=full_path, llm=llm,
            summary=summary, raw_date=raw_date, file_digest=file_digest,
            on_chunk=on_chunk, on_image=on_image,
        )
    content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
    digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
    if doc is not None and doc.content_hash == digest:
        # Phase 147: the shared unchanged path — the pre-phase
        # content-digest hit (a pre-phase NULL-hash row, or a
        # bytes-changed/content-same file). The helper stamps the new
        # file hash, so the NEXT sync takes the fast path above.
        await _unchanged_doc(
            session, doc, source=source, rel=rel, llm=llm, summary=summary,
            raw_date=raw_date, file_digest=file_digest, full_path=full_path,
        )
        return

    verb = "updated" if doc is not None else "added"
    # A ``#`` line is a real heading in markdown but a comment in every
    # other format — titles for those come from the file stem.
    if full_path.suffix.lower() in (".md", ".markdown"):
        title = extract_title(content, fallback=full_path.stem)
    else:
        title = full_path.stem
    if doc is None:
        doc = Document(
            source=source,
            path=rel,
            full_path=str(full_path),
            title=title,
            content=content,
            content_hash=digest,
            # Phase 147: the raw-bytes stamp (the next sync's fast
            # path).
            file_hash=file_digest,
            indexed_at=datetime.now(UTC),
            # Phase 106 (D2/D3): the sourced creation date, normalized
            # (undetermined or future → today). ``created_at_manual``
            # stays the column default (False) — only the date-edit API
            # (task 05) sets it.
            created_at=normalize_doc_date(raw_date),
        )
        session.add(doc)
    else:
        doc.full_path = str(full_path)
        doc.title = title
        doc.content = content
        doc.content_hash = digest
        # Phase 147: re-stamp (the file's bytes changed).
        doc.file_hash = file_digest
        doc.indexed_at = datetime.now(UTC)
        # Phase 106 (D4): a content change is a new document version —
        # the date is re-sourced and a previous manual correction is
        # reset (it referred to the old content).
        doc.created_at = normalize_doc_date(raw_date)
        doc.created_at_manual = False

    session.flush()  # guarantees doc.id even for brand-new rows

    # Phase 125 (task 04, LOCKED A1): a markdown file's image-reference
    # bookkeeping — references whose image has NO document yet in this
    # source are recorded on the doc's ``image_context`` (the
    # ``IMAGE_CONTEXT:`` record) so the post-index enrichment pass picks
    # them up when the image is later indexed; references that DO name an
    # existing image doc record nothing (the task-03 path owns that case
    # — the field stays NULL). The record rides the file's own
    # transaction (committed by the ``session.commit()`` below).
    if full_path.suffix.lower() in _MARKDOWN_EXTENSIONS:
        doc.image_context = _missing_image_context(
            session, source=source, md_rel=rel, content=content
        )

    # Phase 1+2 — chunk, replace the chunk rows (embeddings NULL), embed,
    # and commit the whole file atomically (the endpoint token-cap retry
    # lives in the shared helper — the global policy stays intact for the
    # rest of the KB).
    chunks_n = await _replace_content_chunks(
        session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
    )
    session.commit()

    if verb == "added":
        summary.added += 1
    else:
        summary.updated += 1
    summary.changed_paths.append((source, rel))
    summary.chunks += chunks_n
    logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)

    # Phase 30; phase 118 (A2, 2026-09-15): EVERY new/changed document
    # gets a ``lite``-model summary — markdown included. Phase 30's
    # "markdown is already natural language" exclusion is retired: the
    # summary is the retrieval seed context (the phase-118 suggestion
    # blocks), not a formatting convenience.
    await _store_summary(
        session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
    )


async def _store_summary(
    session: Session,
    *,
    doc: Document,
    source: str,
    rel: str,
    content: str,
    llm: Embedder,
    summary: ImportSummary,
    backfill: bool = False,
) -> None:
    """Best-effort ``lite`` summary for one already-committed document.

    Generates the summary (task 03), stores it on ``documents.summary``
    and indexes it as one extra embedded chunk (``is_summary``,
    position −1) that hybrid search can hit instead of badly-formatted
    raw text. Replacement is guaranteed: any pre-existing ``is_summary``
    chunk of this document is deleted first, so at most one summary chunk
    exists per document at a time.

    Best-effort by contract: the doc row + content chunks are committed
    by the caller before this runs, so an :class:`LLMError` /
    :class:`EmbeddingError` only rolls back the summary rows — the file
    stays indexed, without a summary, and the failure is counted in
    ``summary_errors`` (PLAN phase 30).

    ``backfill`` (phase 118, A2): the unchanged-doc NULL-summary path —
    a success counts ``summary_backfilled`` instead of ``summaries``
    (the doc content is untouched, so the import's KB-change signal must
    not move); the rest of the mechanics are identical.

    Image docs (phase 122, task 03, LOCKED A3): for an ``is_image`` doc
    the vision description — ``content`` (which equals ``doc.content``
    on the backfill path) — IS the summary: no ``lite`` call, no
    pointer line (the summary mirrors the description verbatim, so
    ``doc.summary`` == ``doc.content``). The phase-30 chunk mechanics
    (one ``is_summary`` position −1 chunk, replacement, best-effort
    rollback) are unchanged; the only remaining failure class is the
    summary chunk's embed (the ``doc`` row + content chunks survive —
    fail-soft, same as the text path).
    """
    try:
        if doc.is_image:
            # Phase 122 (task 03): the description IS the summary —
            # stored verbatim (no ``lite`` call, no pointer line).
            text = content
        else:
            text = await generate_summary(llm, source=source, path=rel, content=content)
        # Replacement: at most one summary chunk per document at a time.
        # Removing from the collection is what the ``delete-orphan``
        # cascade turns into a row delete on flush — and it keeps the
        # in-memory collection consistent (this session runs with
        # ``expire_on_commit=False``).
        for old in [c for c in doc.chunks if c.is_summary]:
            doc.chunks.remove(old)
        chunk = Chunk(document_id=doc.id, position=-1, content=text, is_summary=True)
        vector = (await llm.embed([text]))[0]
        chunk.embedding = vector
        doc.summary = text
        # Append through the relationship (the ``all`` cascade persists the
        # row) so the collection — live in this session because of
        # ``expire_on_commit=False`` — reflects the committed state.
        doc.chunks.append(chunk)
        session.commit()
        if backfill:
            # Phase 118 (A2): the backfill counts itself apart from fresh
            # imports — the doc content is unchanged, so ``summaries``
            # (a KB-change signal) must not move.
            summary.summary_backfilled += 1
            logger.info(
                "import: summary-backfill source=%s path=%s chars=%d",
                source, rel, len(text),
            )
        else:
            summary.summaries += 1
            logger.info("import: summary source=%s path=%s chars=%d", source, rel, len(text))
    except (LLMError, EmbeddingError) as e:
        session.rollback()
        summary.summary_errors += 1
        logger.error("import: summary failed source=%s path=%s — %s", source, rel, e)


# ---------------------------------------------------------------------------
# PDF indexing (phase 126, LOCKED A1/A2/A3/A4).
# ---------------------------------------------------------------------------

#: The ``Document.title`` column width (``app.models``) — the metadata
#: title is capped to it (an over-long PDF title must not blow up the
#: upsert).
_PDF_TITLE_MAX_CHARS = 500

#: MIME → extension for the phase-122 image family (phase 126, task 04)
#: — the reverse of the ``app.rag.summarizer.IMAGE_MIMES`` values for the
#: formats PyMuPDF reports. An unlisted mime takes ``.bin`` (never a
#: member of ``BOR_IMAGE_EXTENSIONS`` — the image is skipped with a
#: warning, the honest fail-soft outcome).
_MIME_TO_EXT: dict[str, str] = {
    "image/png": ".png",
    "image/jpeg": ".jpg",
    "image/gif": ".gif",
    "image/webp": ".webp",
    "image/bmp": ".bmp",
    "image/tiff": ".tiff",
}


def _extract_pdf_title(full_path: Path, fallback: str) -> str:
    """The title of one PDF document (phase 126, task 04).

    The embedded metadata title (``fitz``'s ``doc.metadata["title"]``)
    — trimmed and capped at the :data:`_PDF_TITLE_MAX_CHARS` column
    width. An unreadable PDF, an absent metadata dict, or an
    empty/whitespace-only title falls back to *fallback* (the file stem
    — the non-markdown title rule; the metadata title is a convenience,
    never a requirement).
    """
    try:
        doc = fitz.open(str(full_path))
    except Exception:
        return fallback
    with doc:
        meta = doc.metadata
        title = str(meta.get("title") or "").strip() if meta else ""
    if not title:
        return fallback
    return title[:_PDF_TITLE_MAX_CHARS]


async def _index_pdf_file(
    session: Session,
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
    file_digest: str,
    raw_date: datetime,
    on_chunk: Callable[[int, int], None] | None = None,
    on_image: Callable[[int, int], None] | None = None,
) -> None:
    """The phase-126 PDF branch of :func:`_index_file` — a PDF is
    indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
    for the document, then the embedded images (A3):

    * ``content`` is the parser's text — every text-bearing page
      prefixed with the ``---PAGE-N---`` marker (A2 — one document per
      PDF, the markers are the page anchors the page-aware chunker
      (task 03) splits on). Phase 133 (task 03): if that text fails
      the garbage gate (:func:`is_usable_pdf_text` — empty layer or
      XML/binary soup), the content becomes the page-screenshot
      fallback (:func:`_fallback_page_screenshots` — the rendered
      pages read by the vision model, the same separators) BEFORE the
      digest/chunk/summary pipeline runs; the gate never fires for a
      clean layer (no fallback vision calls on the happy path).
      An unopenable PDF — or one whose fallback also produced nothing
      — yields ``""`` (still a document — it may carry embedded images
      — with no content chunks; a warning is the signal).
    * Phase 140 (task 03, LOCKED A1–A4): when the layer is USABLE and
      ``settings.pdf_pictorial`` is on, the sparse-page pass
      (:func:`_pictorial_pdf_pass`) renders + vision-describes every
      page whose stripped text is under
      ``settings.pdf_pictorial_min_chars`` (the deterministic gate —
      NO LLM call, A1; every candidate, no page cap — A21/A3) and
      appends the description to the page block as
      ``[Page N illustration: …]`` (A2/A4). The precedence is locked:
      the pass sits AFTER the phase-133 gate, so an unusable layer
      (the fallback's full-page screenshots) never burns a second
      vision pass; zero candidates → a no-op (the extracted text
      stays byte-identical to the pre-phase-140 path — no render, no
      vision call).
    * the sha256 digest is over the FINAL content (the digest rule is
      content identity — the same text is the same document version;
      the empty case digests the empty byte string). Phase 147: the
      UNCHANGED detection itself is the caller's RAW-BYTES file hash
      (the fast path — :func:`_index_file`): an unchanged PDF never
      re-extracts, never re-renders, and never re-issues a vision
      call (the previous "non-deterministic vision model re-describes
      the row each sync" cost is gone — a re-describe now happens
      only when the file's BYTES change).
    * the title is the metadata title or the file stem
      (:func:`_extract_pdf_title`); the date follows the text path
      exactly (the phase-106 D2 mtime fallback — PDFs carry no
      filesystem-visible creation date — plus the D1 manual-date lock
      and the D4 refresh on the unchanged path).
    * chunk + embed runs through the shared
      :func:`_replace_content_chunks` helper (the endpoint token-cap
      retry lives there), and the phase-30/118 (A2) ``lite`` summary
      path runs on the text — PDFs are no more excluded than any other
      non-markdown format.
    * the UNCHANGED path (phase 147): the fast path above catches an
      unchanged file before any of this runs; a pre-phase NULL-hash
      row, or a bytes-changed/content-same PDF, that reaches the
      content-digest hit below runs the SAME shared path
      (:func:`_unchanged_doc`) — no re-extract, no re-embed, no image
      pass, the backfill/manual-date/date-refresh semantics in one
      code site — and the helper stamps the new file hash (the next
      sync takes the fast path). This path renders NO pages through
      THIS branch, but a row that carries a strip is re-gated by the
      helper's phase-163 tail (:func:`_regate_pdf_strip` — the class
      re-stamped, missing copies healed, a stale strip cleared; zero
      vision/LLM work).
    * Phase 148 (task 01): in this NEW/CHANGED path, RIGHT AFTER the
      row's flush and BEFORE the chunk pipeline, a page-QUALIFYING
      PDF (the phase-133 fallback fired, or the phase-140 pictorial
      pass described ≥1 page — the signals recorded above) gets its
      every page rendered to ``settings.pdf_page_dir`` as
      ``<doc-id>-p{N}.png`` and ``documents.page_count`` +
      ``documents.page_strip_mode`` set
      (:func:`_persist_pdf_pages` — WRITE-FIRST (phase 163, LOCKED
      A4): the new copies land atomically per file BEFORE any old copy
      is removed, a stopped-qualifier cleared, a per-page ``b""`` a
      gap). FAIL-SOFT — a render/write exception leaves
      ``page_count``/``page_strip_mode`` NULL + a warning; the
      document indexes exactly as pre-phase.
    * THEN the embedded images: :func:`_index_pdf_embedded_images`
      (A3 — the phase-122 pipeline, gated by the ``images`` toggle).

    ``raw_date`` is the caller's git last-commit date, or the file's
    mtime — :func:`_index_file` computes the D2 fallback (and reads
    it before the fast-path check) — and the unchanged path
    refreshes from the same source. ``file_digest`` is the caller's
    raw-bytes sha256 (phase 147 — the fast-path signal and this
    branch's stamp value).

    ``on_chunk`` (phase 129, task 02) is the in-flight file's
    chunk-position hook: it is forwarded to the PDF's CONTENT embed
    (the :func:`_replace_content_chunks` call — the primary target),
    but NOT to the embedded-image pass (those image docs are separate
    documents — their ``1/1`` positions would be noise after the
    file's real progress ends, the A6 exclusion). ``None`` keeps the
    branch byte-identical.

    ``on_image`` (phase 152, task 02) is the in-flight file's
    vision-image-position hook (see :func:`import_sources`): it is
    forwarded to the three vision spans — the phase-133 fallback
    (:func:`_fallback_page_screenshots`), the phase-140 pictorial pass
    (:func:`_pictorial_pdf_pass`), and the embedded-image pass
    (:func:`_index_pdf_embedded_images`) — each firing it immediately
    before its vision describe calls. The spans of this file share
    ONE continuous counter: the local *images_reported* starts at 0,
    each span is called with the count reported so far as its *base*
    (the pass that runs second starts where the first ended), and the
    returned reported count is added — the position climbs across the
    file's whole vision span instead of restarting. The fallback and
    the pictorial pass are mutually exclusive (the phase-140
    precedence over the garbage gate — an unusable layer takes the
    fallback, a usable one the pass), both run BEFORE the document
    commit, and the embedded pass runs after it; a skipped span
    (toggle off, zero candidates, a clean text layer) reports nothing
    and advances the counter not at all. ``None`` keeps the branch
    byte-identical.
    """
    # Extract the text (fail-soft: an unopenable PDF yields "" — the
    # document may still carry embedded images worth indexing).
    content = extract_pdf_text(full_path)
    # Phase 148 (task 01): the page-strip ELIGIBILITY signals — the
    # LOCKED rule, recorded where the sync makes the decision: a PDF
    # gets rendered pages iff the sync ALREADY decided its text is not
    # the useful view — the phase-133 fallback fired (unusable text
    # layer — the screenshot class) OR the phase-140 pictorial pass
    # described ≥1 page (the sparse-page class — the IKEA fixture).
    # Text-heavy PDFs (usable layer, zero candidates) stay ineligible
    # (the text view), and ``pdf_pictorial=0`` keeps its kill-switch
    # semantics (no pictorial pass → the sparse class stays ineligible
    # — the pre-phase-148 viewer).
    fallback_fired = False
    pictorial_described = False
    # Phase 152 (task 02): the file's ONE continuous vision-image
    # counter — 0 at the top of the file (the per-file pairing rule:
    # each file's spans start fresh; the caller's status fields reset
    # at the file boundary). Each span below is called with the count
    # reported so far as its *base* and returns the number of images
    # it reported, which is added — the pass that runs second starts
    # where the first ended. Stays 0 while the hook is None.
    images_reported = 0
    # Phase 133 (task 03): the garbage gate — BEFORE the digest/chunk/
    # summary pipeline, so it runs on the FINAL content exactly once.
    # A clean layer never reaches the fallback (the gate is the only
    # gate — no fallback vision calls on the happy path).
    if not is_usable_pdf_text(content):
        summary.pdfs_screenshotted += 1
        fallback_fired = True
        content, reported = await _fallback_page_screenshots(
            session, source=source, rel=rel, full_path=full_path, llm=llm,
            on_image=on_image, base=images_reported,
        )
        images_reported += reported
    elif llm.settings.pdf_pictorial:
        # Phase 140 (task 03, LOCKED A1–A4): the pictorial pass —
        # USABLE layers only (the precedence is locked: the phase-133
        # fallback above already screenshotted an unusable layer, no
        # double vision). Zero candidates → a no-op (the pass returns
        # None — the extracted text stays byte-identical to the
        # pre-phase-140 path); otherwise the FINAL content carries the
        # described candidates' illustration blocks (A4: the digest
        # below runs over it, the phase-126/133 convention).
        passed, reported = await _pictorial_pdf_pass(
            source=source, rel=rel, full_path=full_path, llm=llm, summary=summary,
            on_image=on_image, base=images_reported,
        )
        images_reported += reported
        if passed is not None:
            pictorial_described = True
            content = passed
    if not content:
        logger.warning(
            "import: PDF has no extractable text source=%s path=%s", source, rel
        )
    # Content identity: the digest is over the FINAL content — the
    # extracted text, or the fallback's constructed page descriptions
    # (the empty case is the empty byte string's digest).
    digest = (
        hashlib.sha256(content.encode("utf-8")).hexdigest()
        if content
        else hashlib.sha256(b"").hexdigest()
    )
    title = _extract_pdf_title(full_path, fallback=full_path.stem)
    doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
    if doc is not None and doc.content_hash == digest:
        # Phase 147: the shared unchanged path — a pre-phase NULL-hash
        # row, or a bytes-changed/content-same PDF (the content digest
        # still matches). No re-extract, no re-embed, no image pass;
        # the helper stamps the new file hash (the next sync takes the
        # fast path).
        await _unchanged_doc(
            session, doc, source=source, rel=rel, llm=llm, summary=summary,
            raw_date=raw_date, file_digest=file_digest, full_path=full_path,
        )
        return

    verb = "updated" if doc is not None else "added"
    if doc is None:
        doc = Document(
            source=source,
            path=rel,
            full_path=str(full_path),
            title=title,
            content=content,
            content_hash=digest,
            # Phase 147: the raw-bytes stamp (the next sync's fast
            # path).
            file_hash=file_digest,
            indexed_at=datetime.now(UTC),
            created_at=normalize_doc_date(raw_date),
        )
        session.add(doc)
    else:
        doc.full_path = str(full_path)
        doc.title = title
        doc.content = content
        doc.content_hash = digest
        # Phase 147: re-stamp (the file's bytes changed).
        doc.file_hash = file_digest
        doc.indexed_at = datetime.now(UTC)
        # Phase 106 (D4): a content change is a new document version —
        # the date is re-sourced and a manual correction reset.
        doc.created_at = normalize_doc_date(raw_date)
        doc.created_at_manual = False

    session.flush()  # guarantees doc.id even for brand-new rows

    # Phase 148 (task 01): the PERSISTENT PAGE COPIES — a qualifying
    # PDF (the eligibility signals above) gets every page rendered to
    # ``settings.pdf_page_dir`` (``<doc-id>-p{N}.png``) and
    # ``documents.page_count`` set. RIGHT AFTER the flush (the doc has
    # its id — the copy names need it) and BEFORE the chunk pipeline
    # (the count commits ATOMICALLY with the doc + chunks). Runs ONLY
    # in this new/changed path: the phase-147 fast path returned
    # before the branch and the content-digest hit above took the
    # shared unchanged path — neither renders here; the phase-163
    # task-03 re-gate (:func:`_regate_pdf_strip`, run by
    # :func:`_unchanged_doc`) owns the repair of an unchanged doc's
    # strip. FAIL-SOFT: the render is a DISPLAY
    # concern — any exception leaves ``page_count``/``page_strip_mode``
    # NULL, the warning is the signal, and the document proceeds to
    # chunk/embed/summary EXACTLY as pre-phase.
    # Phase 163 (task 02): the eligibility decision carried as a CLASS
    # (LOCKED A7 — the truth table is the phase-148 rule verbatim: a
    # non-None class ⇔ fallback_fired or pictorial_described). The
    # class decides the stamped `page_strip_mode` ("fallback" →
    # "primary", "pictorial" → "additive"), nothing else.
    strip_class = (
        "fallback" if fallback_fired else ("pictorial" if pictorial_described else None)
    )
    try:
        _persist_pdf_pages(
            doc,
            full_path=full_path,
            page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
            strip_class=strip_class,
            source=source,
            rel=rel,
        )
    except Exception as e:
        doc.page_count = None
        doc.page_strip_mode = None
        logger.warning(
            "import: PDF page render failed source=%s path=%s — %s",
            source, rel, e,
        )

    # Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
    # aware chunker dispatches on the ``.pdf`` suffix; empty content
    # clears the chunk rows and returns 0), committed atomically with
    # the document. Phase 129 (task 02): the in-flight file's
    # chunk-position hook rides the content embed (the embedded-image
    # pass does not receive it — separate documents, the A6 exclusion).
    chunks_n = await _replace_content_chunks(
        session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
    )
    session.commit()

    if verb == "added":
        summary.added += 1
    else:
        summary.updated += 1
    summary.changed_paths.append((source, rel))
    summary.chunks += chunks_n
    logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)

    # Phase 30; phase 118 (A2): the ``lite`` summary of the extracted
    # text (best-effort — a failure leaves the PDF indexed without a
    # summary, counted in ``summary_errors``).
    await _store_summary(
        session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
    )

    # Phase 126 (A3): the embedded images join the phase-122 pipeline
    # (gated by the ``images`` toggle — skipped, debug-logged, while it
    # is off). Runs after the document commit, so a per-image failure
    # never loses the PDF's text index. Runs on BOTH the clean and the
    # fallback content (phase 133: the screenshots are the CONTENT; the
    # extracted images are the citation figures — both, no conflict).
    await _index_pdf_embedded_images(
        session, source=source, rel=rel, full_path=full_path,
        llm=llm, summary=summary, raw_date=raw_date,
        on_image=on_image, base=images_reported,
    )


async def _fallback_page_screenshots(
    session: Session,
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    on_image: Callable[[int, int], None] | None = None,
    base: int = 0,
) -> tuple[str, int]:
    """The phase-133 (task 03) page-screenshot fallback — the owner's
    specified path ("screenshotting the pages and reading each one as
    an image") for a PDF whose text layer failed
    :func:`is_usable_pdf_text` (XML/binary garbage, or an empty text
    layer): every page is rendered to a PNG screenshot and read by the
    vision model, so tables/images/whatever the text layer lost are
    preserved as the model saw them on the page.

    * ``content`` is the :data:`PAGE_SEPARATOR`-anchored page
      descriptions (``---PAGE-N---\n<description>`` — the SAME
      separator convention as :func:`extract_pdf_text`, so the
      page-aware chunker's page-boundary handling works unchanged);
      the doc stays a PDF document (``is_image`` untouched — it is a
      PDF whose content happens to be vision text).
    * one vision call per page (the phase-122
      :func:`describe_image` — ONE CHAT-model call, the PNG bytes as a
      data URL, ``context`` None); FAIL-SOFT PER PAGE: a page that
      could not be rendered (``b""`` from :func:`render_pdf_pages` —
      the index never shifts) or whose vision call failed/came back
      blank gets the deterministic placeholder
      ``page N — description unavailable`` (a dead vision endpoint
      must not sink the doc — the phase-122 fail-soft spirit).
    * an ALL-failed render (no pages rendered — an unopenable file —
      or every page a placeholder) returns ``""``: the existing
      empty-content path (the doc lands as a zero-text doc, the
      phase-126 no-extractable-text behavior).
    * the digest the caller computes is over the constructed content
      — and phase 147's fast path makes a fallback re-run moot for an
      unchanged file: the raw BYTES are the change signal, so an
      unchanged garbage PDF is caught at the top of :func:`_index_file`
      BEFORE the gate runs at all (no re-extract, no re-render, no
      re-issued vision call — the owner's "really long sync times for
      PDFs that are mostly pictoral" complaint, fixed). The fallback's
      vision cost now applies to a garbage PDF's FIRST index — and to
      a re-index only when the file's bytes change; a
      non-deterministic production vision model no longer flips an
      unchanged row's content digest (the row stays put until the
      bytes move; the doc is still correct either way).

    * phase 152 (task 02): the ``on_image`` hook (the file's
      vision-image position, see :func:`import_sources`) is called
      immediately before EACH rendered page's vision call with
      ``(base + i + 1, base + len(pages))`` — the page's 1-based slot
      in the rendered-page list and the span's total (the rendered-
      page count), at the START of the page's slow call, never after.
      The return is the ``(content, reported)`` pair — *reported* the
      number of images this span reported (0 while the hook is
      ``None``), the count the caller adds to the file's continuous
      counter. A hook-less run is byte-identical (every call site is
      guarded).

    *session* is part of the signature for symmetry with the other
    per-file index helpers (the fallback itself needs no DB access —
    the caller's upsert pipeline owns the row).
    """
    pages = render_pdf_pages(full_path)
    # The loud one-line log (PLAN §9, the FINDING's signature).
    logger.warning(
        "pdf: text layer unusable for %s/%s — falling back to %d page screenshots",
        source, rel, len(pages),
    )
    if not pages:
        return "", 0
    blocks: list[str] = []
    got_any = False
    reported = 0
    for i, png in enumerate(pages):
        page = i + 1
        if not png:
            blocks.append(f"page {page} — description unavailable")
            continue
        # Phase 152 (task 02): the file's vision-image position —
        # this page's 1-based slot and the span's total (the rendered-
        # page count), at the START of the page's slow call, never
        # after.
        if on_image is not None:
            on_image(base + i + 1, base + len(pages))
            reported += 1
        desc = await describe_image(llm, data=png, mime="image/png")
        if desc is None or not desc.strip():
            blocks.append(f"page {page} — description unavailable")
        else:
            blocks.append(desc.strip())
            got_any = True
    if not got_any:
        return "", reported
    return (
        "\n\n".join(
            f"{PAGE_SEPARATOR.format(page=page)}\n{block}"
            for page, block in enumerate(blocks, start=1)
        ),
        reported,
    )


async def _pictorial_pdf_pass(
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
    on_image: Callable[[int, int], None] | None = None,
    base: int = 0,
) -> tuple[str | None, int]:
    """The phase-140 (task 03) pictorial pass (LOCKED A1–A4) — the
    IKEA assembly-instruction class (TODO L5): a PDF whose text layer
    is USABLE but SPARSE (mostly pictorial pages — 4–73 chars of
    markers around vector-drawn pictograms) gets every sparse page
    rendered to a PNG and vision-described, and the description
    APPENDS to that page's indexed content block
    (``[Page N illustration: …]``) — the page's text stays, so the
    embeddings carry what the thin text layer couldn't (the owner's
    L5 rationale: "More information is better when generating the
    text embeddings").

    * the gate is :func:`app.rag.pdf.pictorial_pages` over
      :func:`app.rag.pdf.extract_pdf_page_texts` — deterministic, NO
      LLM call (LOCKED A1): the 1-based page numbers whose stripped
      text is under ``settings.pdf_pictorial_min_chars``, ascending.
      (The kill-switch itself — ``settings.pdf_pictorial`` — gates the
      whole pass in the caller: while off, this function is never
      called and the content is byte-identical to pre-phase.)
    * ZERO candidates → a NO-OP (returns ``None``): no render, no
      vision call — the caller keeps its :func:`extract_pdf_text`
      output untouched, so a clean-text PDF's path is byte-identical
      to the pre-phase-140 path (the pin).
    * a page that could not be rendered (the zero-byte
      :func:`render_pdf_pages` placeholder — the index never shifts,
      so it stays aligned with the candidates) is skipped with a
      warning (the phase-133 handling — the page keeps its text only,
      no vision call is burned for it); a page whose vision call
      failed or came back blank (:func:`describe_pdf_page` →
      ``None``) is skipped with a warning and keeps its text only —
      one dead page never kills the pass (LOCKED A2 fail-soft).
    * the assembly (LOCKED A4): every page of the
      :func:`extract_pdf_page_texts` map (the SAME pages, order and
      stripped text as :func:`extract_pdf_text`) as its
      :data:`PAGE_SEPARATOR` + the page text + — for the described
      candidates — ``"\\n\\n[Page {n} illustration: {description}]"``.
      The assembly with NO descriptions is byte-identical to
      :func:`extract_pdf_text`'s output (the unit pin).
    * the caller's digest runs over this FINAL content (the
      phase-126/133 convention — A4) — and phase 147's fast path
      catches an unchanged file BEFORE the pass runs at all (the raw
      bytes are the change signal): no re-render, no re-issued vision
      call, and a non-deterministic vision model no longer re-updates
      an unchanged row — the pass burns vision only on a file's first
      index and on byte changes.
    * stats (log-only — the sync ``detail`` API shape is untouched,
      the phase-133 convention): ``summary.pdfs_pictorial`` (+1 when
      ≥1 page was described) and ``summary.pdf_pictorial_pages``
      (+the described page count), plus the ONE loud log line (PLAN
      §9): ``pdf: pictorial pass source=… path=… — N candidate pages,
      M described (min_chars=…)``.
    * phase 152 (task 02): the ``on_image`` hook (the file's
      vision-image position, see :func:`import_sources`) is called
      immediately before EACH candidate's vision call with
      ``(base + i + 1, base + len(candidates))`` — the candidate's
      1-based slot in the deterministic gate's list and the span's
      total (the candidate-page count), at the START of the page's
      slow call, never after (an unreadable page — the zero-byte
      render placeholder — burns no vision call and reports nothing).
      The return is the ``(content-or-None, reported)`` pair —
      *reported* the number of images this span reported (0 while the
      hook is ``None`` or there are zero candidates), the count the
      caller adds to the file's continuous counter. A hook-less run
      is byte-identical (every call site is guarded).
    """
    settings = llm.settings
    page_texts = extract_pdf_page_texts(full_path)
    candidates = pictorial_pages(
        page_texts, min_chars=settings.pdf_pictorial_min_chars
    )
    if not candidates:
        return None, 0
    # ONE render for the whole candidate set (the phase-140 selector —
    # a 500-page PDF with 30 sparse pages renders 30 PNGs, not 500);
    # the result is aligned with *candidates* (ascending, one slot
    # each — a zero-byte placeholder never shifts the positions).
    pngs = render_pdf_pages(full_path, pages=candidates)
    descriptions: dict[int, str] = {}
    reported = 0
    # strict: the render's result is aligned with the candidate list by
    # contract (one slot per named page, ascending) — a length drift is
    # a parser regression, fail loud.
    for i, (page, png) in enumerate(zip(candidates, pngs, strict=True)):
        if not png:
            # The phase-133 zero-byte placeholder: an unreadable page.
            logger.warning(
                "pdf: skipping pictorial page %d of %s/%s — the render is "
                "empty (unreadable page)",
                page, source, rel,
            )
            continue
        # Phase 152 (task 02): the file's vision-image position —
        # this candidate's 1-based slot and the span's total (the
        # candidate-page count), at the START of the page's slow
        # call, never after.
        if on_image is not None:
            on_image(base + i + 1, base + len(candidates))
            reported += 1
        desc = await describe_pdf_page(
            llm,
            data=png,
            mime="image/png",
            page=page,
            page_text=page_texts[page],
        )
        if desc is None:
            # Fail-soft (LOCKED A2): one dead page never kills the
            # pass — the page keeps its text only, the sync continues.
            logger.warning(
                "pdf: pictorial page %d of %s/%s has no description — "
                "the page keeps its text only",
                page, source, rel,
            )
            continue
        descriptions[page] = desc
    # The ONE loud log line (PLAN §9): N candidate pages, M described.
    logger.info(
        "pdf: pictorial pass source=%s path=%s — %d candidate pages, %d "
        "described (min_chars=%d)",
        source, rel, len(candidates), len(descriptions),
        settings.pdf_pictorial_min_chars,
    )
    # The stats (log-only — the sync ``detail`` API shape is untouched).
    if descriptions:
        summary.pdfs_pictorial += 1
    summary.pdf_pictorial_pages += len(descriptions)
    # The assembly: the extracted text's page blocks, the described
    # candidates carrying their illustration block (A4 — the page's
    # text STAYS; the description appends after it).
    parts: list[str] = []
    for page, text in page_texts.items():
        block = text
        if page in descriptions:
            block += f"\n\n[Page {page} illustration: {descriptions[page]}]"
        parts.append(f"{PAGE_SEPARATOR.format(page=page)}\n{block}")
    return "\n\n".join(parts), reported


def _persist_pdf_pages(
    doc: Document,
    *,
    full_path: Path,
    page_dir: Path,
    strip_class: str | None,
    source: str,
    rel: str,
) -> None:
    """Phase 148 (task 01) + phase 163 (task 02): the persistent page
    copies of one NEW/CHANGED PDF — the display strip's bytes (task 02's
    public route serves them) — written WRITE-FIRST and stamped with the
    strip's CLASS.

    *strip_class* is the sync's eligibility decision carried as a CLASS
    (phase 163, LOCKED A7): ``"fallback"`` (the phase-133 fallback fired
    — the screenshots ARE the content → stamped mode ``"primary"``),
    ``"pictorial"`` (the phase-140 pass described ≥1 sparse page →
    stamped mode ``"additive"``), or ``None`` (no strip). The truth
    table is the phase-148 eligibility rule VERBATIM: a non-None class
    ⇔ the old ``fallback_fired or pictorial_described`` — this phase
    changes only what the class DOES (the stamp), never who qualifies.

    * ELIGIBLE: EVERY page of the PDF is rendered (150 dpi,
      :func:`render_pdf_pages`' default — the phase-133 constant; the
      ``pages=None`` "list index *i* is page *i*+1" contract) and each
      NON-EMPTY PNG lands in *page_dir* as ``<doc-id>-p{N}.png``
      (N 1-based) through an ATOMIC per-file write — the shared writer
      :func:`_write_page_copies` puts the bytes on ``<name>.png.tmp``
      first and ``os.replace`` moves them onto the final name, so a
      concurrent reader of the route NEVER sees a torn PNG (the phase
      163 task-03 re-gate heals a strip through THE SAME writer) — and
      the success tail stamps ``doc.page_count = len(pages)``
      (one slot per page — the list IS the page count for an openable
      file) and ``doc.page_strip_mode`` from the class. A per-page
      ``b""`` (an unreadable page) is a GAP, not an error: no file is
      written for it (the route 404s that N, the viewer's note covers
      it), and the count still covers the full range.
    * WRITE-FIRST (phase 163, LOCKED A4 — the 2026-10-07 live loss): the
      OLD copies are deleted LAST, never first. The pre-phase order
      (blanket delete → render → write) ran OUTSIDE the DB transaction,
      so any later per-file rollback (the import loop's
      ``EmbeddingError`` handler) — or a crash between delete and write —
      restored the stale ``page_count`` over ZERO files and the strip
      404ed forever. Now only the old-extent files (1..the count read
      BEFORE any change) that the fresh render did NOT write — a shrunk
      PDF's ``p{N}`` tail, a new gap — are removed, and only once every
      replacement byte is durably in place; a file about to be
      overwritten is never deleted at all.
    * NOT eligible (``strip_class is None``), an unopenable file
      (:func:`render_pdf_pages` → ``[]``), or a render with ZERO
      non-empty pages: the existing copies are deleted (the helper reads
      ``doc.page_count``, so it runs BEFORE the nulling) and BOTH
      ``page_count`` and ``page_strip_mode`` are cleared — the
      deletion-first order is correct HERE, there is no replacement set
      to protect; a document that STOPPED qualifying loses its strip
      (e.g. a new version gained a real text layer).
    * FAIL-SOFT: a write failure removes this call's ``.tmp`` files and
      the final-name files it CREATED — but NEVER a file it OVERWROTE
      (the replaced copy keeps standing as the durable new bytes; a
      failed call may change a copy's bytes, it may never remove a
      copy) — and re-raises: the caller (:func:`_index_pdf_file`)
      catches, leaves ``page_count`` AND ``page_strip_mode`` NULL, logs
      the warning, and the document indexes exactly as pre-phase (the
      render is a display concern — it never gates indexing).

    *page_dir* is ``Path(settings.pdf_page_dir).expanduser()`` (the
    ``image_dir`` convention — raw string, expanduser'd by the
    importer); the directory is created on demand, only when a page is
    actually written (an ineligible doc never creates it).
    """
    if strip_class is None:
        # No strip — the DELETION-first order is correct here (there is
        # no replacement set to protect); the helper reads
        # doc.page_count, so it runs BEFORE the nulling (a fresh doc's
        # NULL count is a no-op; missing files are a silent no-op).
        _delete_page_copies(doc, page_dir)
        doc.page_count = None
        doc.page_strip_mode = None
        return
    pages = render_pdf_pages(full_path)
    if not pages or not any(pages):
        # Unopenable file (the render's ``[]``) or every page a
        # placeholder: no strip and NO replacement set — the existing
        # copies go with the cleared row (helper BEFORE the nulling),
        # fail-soft, the index is untouched (the render helper already
        # logged the per-page failures).
        logger.warning(
            "import: PDF page render produced no pages source=%s path=%s",
            source, rel,
        )
        _delete_page_copies(doc, page_dir)
        doc.page_count = None
        doc.page_strip_mode = None
        return
    # The row's OWN old extent, read BEFORE any change — the stale-tail
    # delete below walks it (the write-first contract never deletes a
    # file outside this extent, and never before its replacement is
    # durable).
    old_extent = doc.page_count or 0
    # 1 — WRITE FIRST, atomically per file (the shared writer, phase
    # 163 task 02 — the phase-163 task-03 re-gate heals through it).
    written_ns = _write_page_copies(doc, page_dir, pages)
    # 2 — the stale tail goes LAST.
    _delete_stale_page_tail(
        doc_id=doc.id,
        page_dir=page_dir,
        old_extent=old_extent,
        written_ns=written_ns,
    )
    doc.page_count = len(pages)
    doc.page_strip_mode = "primary" if strip_class == "fallback" else "additive"


def _write_page_copies(doc: Document, page_dir: Path, pages: Sequence[bytes]) -> set[int]:
    """Phase 163 (task 02, LOCKED A4; extracted for the task-03 re-gate,
    which heals a strip through the SAME writer): write every NON-EMPTY
    PNG of *pages* to ``page_dir/<doc-id>-p{N}.png`` (N 1-based) and
    return the set of page numbers written.

    * ATOMIC per file — the bytes land on ``<name>.png.tmp`` and
      ``os.replace`` moves them onto the final name, so a concurrent
      reader of the ``/pages/{n}`` route never sees a torn PNG.
    * A per-page ``b""`` (the render's failure placeholder) is the GAP:
      no file, no entry in the returned set.
    * The directory is created on demand (a doc that never writes never
      creates it).
    * A failure leaves NO orphan: this call's ``.tmp`` files and the
      finals it CREATED are removed — an OVERWRITTEN final keeps
      standing (deleting it would re-open the 2026-10-07 hole with one
      page instead of the whole strip) — and the exception re-raises so
      the caller keeps its fail-soft contract.
    """
    page_dir.mkdir(parents=True, exist_ok=True)
    written_tmps: list[Path] = []
    created_finals: list[Path] = []  # finals that did NOT exist before
    written_ns: set[int] = set()
    try:
        for n, png in enumerate(pages, start=1):
            if not png:
                # A per-page render failure: the GAP (no file — the
                # route 404s that N, the viewer's note covers it).
                continue
            final = page_dir / f"{doc.id}-p{n}.png"
            tmp = page_dir / f"{doc.id}-p{n}.png.tmp"
            # Track the tmp BEFORE the write: a failure mid-write may
            # leave it on disk, and the cleanup must find it either way.
            written_tmps.append(tmp)
            existed = final.exists()
            tmp.write_bytes(png)
            os.replace(tmp, final)  # atomic per file — no torn reads
            written_ns.add(n)
            if not existed:
                created_finals.append(final)
    except Exception:
        # A partial write leaves no orphan: this call's tmp files and
        # the finals it CREATED go (an OVERWRITTEN final keeps standing
        # — deleting it would re-open the 2026-10-07 hole with one page
        # instead of the whole strip), then the re-raise hands the
        # failure to the caller's fail-soft handler (page_count +
        # page_strip_mode NULL + the warning).
        for path in [*written_tmps, *created_finals]:
            path.unlink(missing_ok=True)
        raise
    return written_ns


def _delete_stale_page_tail(
    *, doc_id: uuid.UUID, page_dir: Path, old_extent: int, written_ns: set[int]
) -> None:
    """Phase 163 (task 02, LOCKED A4 — the LAST step of the write-first
    order): remove the files of the row's OLD extent (1..*old_extent*)
    the fresh render did NOT write — a shrunk PDF's ``p{N}`` tail, a new
    gap. Runs only once every replacement byte is durable, never touches
    a file outside the old extent, and never touches a file that was just
    overwritten. Best-effort like :func:`_delete_page_copies`: an
    unreadable file is logged, never raised (the sync survives)."""
    for n in range(1, old_extent + 1):
        if n in written_ns:
            continue
        path = page_dir / f"{doc_id}-p{n}.png"
        try:
            path.unlink(missing_ok=True)
        except OSError as e:
            logger.warning("import: could not delete PDF page copy %s — %s", path, e)


async def _index_pdf_embedded_images(
    session: Session,
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
    raw_date: datetime,
    on_image: Callable[[int, int], None] | None = None,
    base: int = 0,
) -> int:
    """The phase-126 (A3) embedded-image pass — one call per (re)indexed
    PDF, after its document commit.

    Each DISTINCT embedded image (the parser dedupes repeated
    placements and byte-identical copies — phase 151 task 01 — first
    placement wins; page order, 0-based index on the page — the
    parser's order; the parser's blank/decorative gate — phase 151
    task 03, :func:`app.rag.pdf.is_blank_image` on the base samples —
    drops near-white underlays BEFORE they reach this pass: no row,
    no copy, no vision call) is indexed as a STANDALONE image document
    through the phase-122 mechanics, inline (no temp file, no
    ``_index_image_file`` refactor): the gate is ON by default and
    ``Settings.pdf_image_skip_blank`` (``BOR_PDF_IMAGE_SKIP_BLANK`` —
    phase 151 task 04, the kill switch, read HERE — the one place —
    and passed down as the extractor's ``skip_blank`` keyword) is its
    single knob: ``false`` restores indexing every extracted image
    (the pre-phase-151 behavior); dedupe and soft-mask compositing are
    UNAFFECTED by the knob (they are correctness fixes).

    * gated by the phase-122 ``images`` toggle (the single knob, LOCKED
      A3): while it is OFF the whole pass is skipped (one debug line —
      the extraction itself is not worth running) and the prune guard
      keeps any existing image docs;
    * an image whose mime maps to no ``BOR_IMAGE_EXTENSIONS`` member
      (:data:`_MIME_TO_EXT`) is skipped with a warning (the unsupported-
      extension rule, same as a standalone image's);
    * the document's synthetic ``path`` is
      ``<pdf path>#page=<N>#img=<index>.<ext>`` — N/index are the
      FIRST placement's (the parser's dedupe, phase 151 task 01),
      unique per distinct image, stable across re-indexes (the upsert
      key); the title is ``Embedded image from <pdf stem> page <N>``;
    * the digest is sha256 over the image BYTES (the digest rule is
      content identity); an image doc whose digest matches is skipped
      (no re-describe, no re-embed — unchanged). Phase 151 (task 02)
      consequence, verified: a MASKED image's bytes change with the
      smask compositing (base stream → alpha PNG), so pre-phase rows
      for masked images re-describe ONCE on the first post-phase sync
      (their sha256 digest differs) — expected, self-limiting, no
      migration; unmasked images' bytes are untouched, so their rows
      stay digest-skipped as before;
    * the description comes through the SAME single seam
      (:func:`_describe_or_skip` — the bytes as a data URL whose mime
      comes from the copy path's extension); a failed/empty description
      SKIPS the image entirely (no row, no copy) — ``images_failed`` +
      a warning, the sync continues (LOCKED A3 fail-soft). The
      ``image_dir`` copy (``<doc-id>.<ext>`` — a new image's id is the
      uuid4 chosen here; a changed image keeps its id and its stale
      copy is deleted before replacement) is written only AFTER a
      successful description (a failure never leaves an orphan);
    * phase 125 parity (the standalone image's task-03 scan): a same-
      source markdown file that references the image by its synthetic
      path (``![alt](<pdf path>#page=<N>#img=<index>.<ext>)``) enriches
      the description — the combined context goes to the vision call
      and is stored on the doc's ``image_context``, and ``content``
      becomes ``<context>\\n\\n<description>`` (as does ``summary`` —
      the image convention); no reference keeps the bare shape
      (byte-identical to the un-enriched case). The markdown-FIRST case
      (a markdown indexed before the PDF recorded its reference on its
      own row) is closed by the shared post-index pass
      (:func:`_enrich_pending_images`), exactly as for a standalone
      image;
    * the description IS the content and the summary (the phase-122
      image convention — :func:`_store_summary` stores it verbatim),
      chunked + embedded through :func:`_replace_content_chunks` (one
      transaction per image); a per-image embed failure rolls back that
      image only (``errors`` + the error line) and the pass moves on —
      the PDF's text index and the other images are already committed.
    * a new/updated image doc counts ``added``/``updated`` (it really
      changed the KB — the sources-version bump fires on it, like a
      standalone image's);
    * phase 137 (task 02, A2/A7): the document context is built ONCE
      per PDF — :func:`app.rag.pdf.extract_pdf_page_texts` (task 01)
      reads the per-page text map a single time for the whole pass, and
      the phase-133 garbage gate
      (:func:`app.rag.pdf.is_usable_pdf_text`) is run over the PDF's
      extracted text layer: a layer the gate flags as unusable yields
      ``text_usable=False`` → every block is the origin line only
      (the "page text" would be the pre-126 soup — ASSUMPTION A7). An
      unopenable PDF reads as an empty map + an unusable (empty)
      layer: origin-line-only blocks, fail-soft like the rest of the
      parser.

    * phase 152 (task 02): the ``on_image`` hook (the file's
      vision-image position, see :func:`import_sources`) is called at
      the START of EACH image's pipeline step — before the slow
      :func:`_describe_or_skip` seam inside
      :func:`_index_one_embedded_image` — with
      ``(base + i + 1, base + len(images))``: the image's 1-based slot
      in the pass's list (post-dedupe, post-blank-gate, phase 151:
      the work actually about to happen) and the span's total. Returns
      the number of images reported (0 while the hook is ``None`` or
      the toggle is off) — the count the caller adds to the file's
      continuous counter. A hook-less run is byte-identical (every
      call site is guarded).

    ``raw_date`` is the PDF's resolved source date (the pass runs only
    for (re)indexed PDFs, so it is always concrete) — the image docs
    inherit the PDF's date (they are part of it).
    """
    settings = llm.settings
    if not settings.images:
        # Phase-122 toggle off: embedded images are skipped (debug-
        # logged) — and the prune guard keeps any existing image docs
        # (they are invisible to the walk, not deleted files).
        logger.debug(
            "import: PDF embedded images skipped (images toggle off) "
            "source=%s path=%s",
            source, rel,
        )
        return 0
    # Phase 137 (task 02/03): the document context's raw material —
    # the SHARED probe (task 01's per-page text map extracted ONCE per
    # PDF — the pass reads it per image — plus the phase-133 garbage
    # gate's verdict over the layer, A7 — the gate's empty-layer target
    # makes a text-less PDF's blocks origin-line-only, too). The
    # task-03 backfill probes the parent with the SAME helper when it
    # is still in the source, so the two paths can never drift.
    page_texts, text_usable = _pdf_image_context_inputs(full_path)
    # Phase 151 (task 03): the stats-carrying variant — the parser's
    # per-PDF drop counts (dedupe + blank gate) land on the run's
    # summary (log-only: the sync ``detail`` API shape is untouched).
    # Phase 151 (task 04): the knob is read HERE — the one place —
    # and passed down as the extractor's `skip_blank` keyword
    # (default True; `false` = every extracted image indexed, the
    # pre-phase behavior; dedupe + compositing are unaffected).
    images, stats = extract_pdf_images_with_stats(
        full_path, skip_blank=settings.pdf_image_skip_blank
    )
    summary.pdf_images_deduped += stats.duplicates
    summary.pdf_images_blank_skipped += stats.blank_skipped
    reported = 0
    for i, img in enumerate(images):
        # Phase 152 (task 02): the file's vision-image position —
        # this image's 1-based slot in the pass's list (post-dedupe,
        # post-blank-gate: the work actually about to happen) and the
        # span's total, at the START of the image's pipeline step
        # (the slow :func:`_describe_or_skip` seam inside
        # :func:`_index_one_embedded_image`), never after.
        if on_image is not None:
            on_image(base + i + 1, base + len(images))
            reported += 1
        await _index_one_embedded_image(
            session, img, source=source, rel=rel, full_path=full_path,
            llm=llm, summary=summary, raw_date=raw_date,
            page_texts=page_texts, text_usable=text_usable,
        )
    return reported


async def _index_one_embedded_image(
    session: Session,
    img: PdfImage,
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
    raw_date: datetime,
    page_texts: dict[int, str],
    text_usable: bool = True,
) -> None:
    """One embedded image of one PDF → the phase-122 image pipeline,
    with the phase-125 task-03 markdown context scan (parity with the
    standalone image's :func:`_index_image_file`), and the phase-137
    (task 02, A2) document-context block — the combined context
    (document block FIRST, markdown blocks second) is the vision call's
    ``context``, the stored ``image_context``, and the
    ``content``/``summary`` enrichment (see
    :func:`_index_pdf_embedded_images` for the full contract).

    *page_texts* / *text_usable* are the pass's per-PDF context inputs
    (the task-01 page map; the phase-133 gate's verdict — A7).
    """
    settings = llm.settings
    ext = _MIME_TO_EXT.get(img.mime, ".bin")
    if ext not in settings.image_extension_set:
        logger.warning(
            "import: PDF embedded image skipped (unsupported extension) "
            "source=%s path=%s page=%d index=%d mime=%s",
            source, rel, img.page, img.index, img.mime,
        )
        return
    image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
    digest = hashlib.sha256(img.data).hexdigest()
    image_doc = session.scalar(
        select(Document).where(Document.source == source, Document.path == image_rel)
    )
    if image_doc is not None and image_doc.content_hash == digest:
        # Unchanged embedded image (byte digest) — no re-describe, no
        # re-embed (the standalone image's unchanged rule).
        logger.debug(
            "import: PDF embedded image unchanged source=%s path=%s",
            source, image_rel,
        )
        return
    verb = "updated" if image_doc is not None else "added"
    # Named by the doc id (the phase-122 copy rule): a new image's id is
    # the uuid4 chosen here (row and copy agree); a changed image keeps
    # its id (the copy path is stable, the stale copy is deleted below).
    doc_id = image_doc.id if image_doc is not None else uuid.uuid4()
    image_dir = Path(settings.image_dir).expanduser()
    copy_path = image_dir / f"{doc_id}{ext}"
    # Phase 125 (task 03, LOCKED A1/A2) — the embedded image's parity
    # with the standalone image's path (phase 126): BEFORE the
    # description, the markdown reference context for THIS image — a
    # ``![alt](<pdf path>#page=<N>#img=<index>.<ext>)`` reference in a
    # same-source markdown file (the synthetic path is the doc's
    # ``path``, and the reference resolves to it through the shared
    # :func:`_resolve_ref_path` rule) — or ``None`` (no reference).
    # A re-indexed image re-scans: the context follows the current
    # source state (a deleted referencing markdown drops it back to
    # ``None``).
    md_context = _markdown_image_context(
        _source_root_for(full_path, rel), llm, source=source, image_rel=image_rel
    )
    # Phase 137 (task 02, A2) — the document-context block (the
    # provenance: WHERE the image came from — ALWAYS present, even for
    # a text-less page or a garbage-gated text layer — A7) combined
    # with the markdown reference blocks (document FIRST — provenance
    # before purpose). The SAME combined value goes to the vision model
    # (a context-aware description, LOCKED A3), is stored on the doc
    # (``image_context`` — the retrieval display's context line), and
    # enriches ``content``/``summary`` below — the text the embedder
    # indexes carries the context the retrieval needs.
    image_context = _combine_image_contexts(
        _pdf_image_context_block(source, rel, img.page, page_texts, text_usable=text_usable),
        md_context,
    )
    # The description through the single seam (the mime resolves from
    # the copy path's extension — the extension was derived from the
    # mime, so the round-trip is lossless for the default family). The
    # description is generated BEFORE anything is written, so a failure
    # skips the image with no row and no copy.
    description = await _describe_or_skip(
        llm, data=img.data, source=source, rel=image_rel, full_path=copy_path,
        context=image_context,
    )
    if description is None:
        # LOCKED A3 fail-soft: an undescribed image is unsearchable
        # noise — skip it (no row, no copy).
        summary.images_failed += 1
        logger.warning(
            "import: PDF embedded image description failed source=%s path=%s "
            "page=%d index=%d",
            source, image_rel, img.page, img.index,
        )
        return
    if image_doc is not None:
        # A CHANGED image: the stale copy is deleted before replacement.
        _delete_image_copy(image_doc.image_path)
    image_dir.mkdir(parents=True, exist_ok=True)
    copy_path.write_bytes(img.data)

    # Phase 125 (task 03) / phase 137 (task 02): the enriched content —
    # the combined context FIRST (for retrieval), then the description
    # (the image convention; ``summary = content`` below). For embedded
    # images the context is ALWAYS present (the document block is the
    # provenance — never empty), so the bare-description shape is
    # standalone-only.
    content = f"{image_context}\n\n{description}"

    title = f"Embedded image from {full_path.stem} page {img.page}"
    if image_doc is None:
        image_doc = Document(
            id=doc_id,
            source=source,
            path=image_rel,
            full_path=str(copy_path),
            title=title,
            content=content,
            content_hash=digest,
            indexed_at=datetime.now(UTC),
            created_at=normalize_doc_date(raw_date),
            is_image=True,
            image_path=str(copy_path),
            image_context=image_context,
        )
        session.add(image_doc)
    else:
        image_doc.full_path = str(copy_path)
        image_doc.title = title
        image_doc.content = content
        image_doc.content_hash = digest
        image_doc.indexed_at = datetime.now(UTC)
        # Phase 106 (D4): a content change is a new document version —
        # the date is re-sourced and a manual correction reset.
        image_doc.created_at = normalize_doc_date(raw_date)
        image_doc.created_at_manual = False
        image_doc.is_image = True
        image_doc.image_path = str(copy_path)
        # Phase 125 (task 03): a re-indexed image re-scans its markdown
        # references — the context follows the current source state.
        image_doc.image_context = image_context

    session.flush()  # guarantees doc.id even for brand-new rows

    # Phase 1+2 — the shared pipeline on the description (one
    # transaction per image; a per-image embed failure rolls back THIS
    # image only — the PDF doc and the other images are already
    # committed — and the pass moves on).
    try:
        chunks_n = await _replace_content_chunks(
            session, doc=image_doc, llm=llm, content=content, rel=image_rel
        )
        session.commit()
    except EmbeddingError as e:
        session.rollback()
        # The description succeeded (the copy was written), but the
        # embed gave up: delete the copy so a failure never leaves an
        # orphan (the phase-122 no-orphan rule).
        _delete_image_copy(str(copy_path))
        summary.errors += 1
        logger.error(
            "import: PDF embedded image embed failed source=%s path=%s "
            "page=%d index=%d — %s",
            source, image_rel, img.page, img.index, e,
        )
        return

    if verb == "added":
        summary.added += 1
    else:
        summary.updated += 1
    summary.changed_paths.append((source, image_rel))
    summary.chunks += chunks_n
    logger.info(
        "import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
    )
    # Phase 122 (task 03) shape on the description — image-aware: the
    # description IS the summary (stored verbatim, no ``lite`` call), so
    # ``doc.summary`` == ``doc.content`` and the ``is_summary`` position
    # −1 chunk mirrors it. With a markdown reference (phase 125) the
    # summary is the ENRICHED content — the same value the standalone
    # image's path stores.
    await _store_summary(
        session, doc=image_doc, source=source, rel=image_rel,
        content=content, llm=llm, summary=summary,
    )


def _source_root_for(full_path: Path, rel: str) -> Path:
    """The source root of one walked file (phase 125, task 03).

    The walk's invariant is ``full_path == root / rel`` (``rel`` is the
    source-relative POSIX path, ``full_path`` the absolute path), so the
    root is *full_path*'s parent walked UP by the number of DIRECTORY
    components of *rel* (a top-level file has none — its own directory
    IS the root).
    """
    root = full_path.parent
    for _ in PurePosixPath(rel).parts[:-1]:
        root = root.parent
    return root


def _resolve_ref_path(ref_path: str, md_rel: str) -> str | None:
    """One ``![alt](path)`` reference path → the image's source-relative
    POSIX path, or ``None`` when the reference names no in-source file
    (phase 125 — the single resolution rule shared by the task-03
    context scan, the task-04 markdown bookkeeping, and the task-04
    post-index enrichment pass).

    A leading ``/`` = the source root (a root-absolute doc-style
    reference); otherwise the path is resolved against the markdown
    file's directory (``md_rel`` is the markdown's source-relative
    POSIX path — its ``dirname``, so a top-level markdown joins against
    the root). The result is normalized; a normalized path that still
    escapes (``..``-prefixed — e.g. ``../../other/x.png``) or names no
    file (``.``) matches no image document in this source, ever.
    """
    if ref_path.startswith("/"):
        resolved = ref_path.lstrip("/")
    else:
        resolved = posixpath.join(posixpath.dirname(md_rel), ref_path)
    normalized = posixpath.normpath(resolved)
    if normalized in ("", ".") or normalized.startswith(".."):
        return None
    return normalized


def _ref_points_at(ref: ImageRef, md_rel: str, image_rel: str) -> bool:
    """Does *ref* point at the image at *image_rel* (phase 125, task 03)?

    The reference path is resolved against the markdown file's directory
    (``md_rel`` is the markdown's source-relative POSIX path) — or,
    when it starts with ``/``, against the SOURCE ROOT (a root-absolute
    doc-style reference) — then normalized and compared for equality
    with the image's source-relative path. Handles both the bare
    relative reference (``pic.png`` next to the markdown) and the
    folder-carrying one (``folder/pic.png`` from the root); a reference
    to a different file that shares the filename is NOT a match.
    """
    resolved = _resolve_ref_path(ref.path, md_rel)
    return resolved is not None and resolved == image_rel


#: The page-text cap of one embedded image's document-context block
#: (phase 137, task 02, ASSUMPTION A3) — a module constant, no env var.
#: A single page rarely runs past it; a runaway page (a wall of dense
#: text) is cut exactly at the cap, with the shared
#: :data:`~app.rag.retriever.TRUNCATION_MARKER` on its own line (the
#: summarizer's existing truncation convention).
PDF_IMAGE_CONTEXT_MAX_CHARS = 6000


def _pdf_image_context_block(
    source: str,
    pdf_rel: str,
    page: int,
    page_texts: dict[int, str],
    text_usable: bool = True,
) -> str:
    """The document-context block for one embedded image (phase 137, A2).

    Always non-empty: the origin line is present even for a text-less
    page (the owner's core complaint — a stock-photo description that
    names nothing about the document it came from). The page text (the
    :func:`app.rag.pdf.extract_pdf_page_texts` value for *page* —
    already stripped; a missing key is the text-less-page case) is cut
    exactly at :data:`PDF_IMAGE_CONTEXT_MAX_CHARS` (overflow cut at the
    cap, then :data:`~app.rag.retriever.TRUNCATION_MARKER` on its own
    line — the summarizer's existing truncation convention) and
    OMITTED entirely when the page has no text OR when *text_usable*
    is ``False`` — the caller ran the phase-133 garbage gate
    (:func:`app.rag.pdf.is_usable_pdf_text`) over the PDF's extracted
    text layer and it failed, so the "text" is the pre-126 class of
    garbage and must not be fed to the vision model (ASSUMPTION A7).

    *page* is the 1-indexed page number (the forward path passes
    ``img.page``; the task-03 backfill passes the page parsed from the
    synthetic path — taking a plain ``int``, deliberately NOT a
    ``PdfImage``, is what lets the backfill need no fake image
    object). The header is a stable single line with NO page-total
    (the total is not on the image record, and the task-04 display
    renderer regexes this exact line).
    """
    header = f"Extracted from {source}/{pdf_rel} (page {page}):"
    text = page_texts.get(page, "").strip()
    if not text or not text_usable:
        return header
    if len(text) > PDF_IMAGE_CONTEXT_MAX_CHARS:
        text = text[:PDF_IMAGE_CONTEXT_MAX_CHARS] + "\n" + TRUNCATION_MARKER
    return f"{header}\n{text}"


def _combine_image_contexts(doc_block: str, md_blocks: str | None) -> str:
    """Document block first, markdown reference block(s) second (A2).

    The single combiner of the two context block kinds — provenance
    (WHERE the image came from) before purpose (WHAT the markdown says
    it illustrates). Used by BOTH the forward path
    (:func:`_index_one_embedded_image`) and the task-03 backfill, so
    the two can never drift. ``md_blocks`` ``None``/empty → the doc
    block alone; the doc block is ALWAYS present for embedded images
    (it is the provenance — never empty). Markdown-only values (no doc
    block) are produced by the standalone path only and never pass
    through this helper.
    """
    if not md_blocks:
        return doc_block
    return f"{doc_block}\n\n{md_blocks}"


def _pdf_image_context_inputs(full_path: Path) -> tuple[dict[int, str], bool]:
    """The phase-137 document-context inputs for one PDF on disk — the
    task-02 probe, shared by the forward path
    (:func:`_index_pdf_embedded_images`) and the task-03 backfill
    (:func:`_backfill_image_contexts`) so the two can never drift:

    * the per-page text map (task 01's
      :func:`app.rag.pdf.extract_pdf_page_texts` — fail-soft: an
      unreadable file reads as an empty map with one warning);
    * the phase-133 garbage gate's verdict over the layer (ASSUMPTION
      A7 — :func:`app.rag.pdf.is_usable_pdf_text` over the whole
      extracted text; ``False`` → the blocks are origin-line-only, the
      "page text" is the pre-126 class of garbage).
    """
    page_texts = extract_pdf_page_texts(full_path)
    text_usable = is_usable_pdf_text("\n\n".join(page_texts.values()))
    return page_texts, text_usable


def _markdown_image_context(
    root: Path,
    llm: Embedder,
    *,
    source: str,
    image_rel: str,
) -> str | None:
    """The combined markdown reference context for one image (phase 125,
    task 03, LOCKED A1/A2) — or ``None`` when no markdown file in the
    source references it (the standalone-image case: the whole phase-125
    machinery is a no-op, byte-identical to pre-phase).

    Scans the source's importable MARKDOWN files (``.md``/``.markdown``
    under the source's import-extension set — hidden/excluded-dir rules
    and the extension filter apply exactly as to the walk) and keeps
    every :func:`~app.rag.md_images.extract_image_refs` reference that
    POINTS AT this image (:func:`_md_ref_block` resolves each reference
    via :func:`_resolve_ref_path` — ``pic.png`` from ``guide.md`` →
    ``pic.png``; ``../pic.png`` from ``sub/guide.md`` → ``pic.png``;
    ``folder/pic.png`` from ``guide.md`` → ``folder/pic.png``), one
    block per referencing markdown file in the walk's sorted order.
    The result is both the *context* argument of the vision call (the
    model learns WHAT the image is for) and the stored
    ``image_context`` (the retrieval display's context line, task 05).
    """
    blocks: list[str] = []
    for md_path in iter_importable_files(root, llm.settings.import_extension_set):
        if md_path.suffix.lower() not in _MARKDOWN_EXTENSIONS:
            continue
        try:
            content = md_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
        except OSError:
            continue  # unreadable source file — no context from it
        block = _md_ref_block(source, md_path.relative_to(root).as_posix(), content, image_rel)
        if block is not None:
            blocks.append(block)
    return "\n\n".join(blocks) if blocks else None


def _md_ref_block(source: str, md_rel: str, content: str, image_rel: str) -> str | None:
    """One ``Referenced in <source>/<md>:`` block for one markdown's
    content (phase 125) — the shared builder of the task-03 disk scan
    (:func:`_markdown_image_context`) and the task-04 pass
    (:func:`_pending_image_context`).

    Keeps every image reference in *content* that POINTS AT *image_rel*
    (the resolution rule: :func:`_resolve_ref_path` — a reference to a
    different file that merely shares the filename never matches,
    LOCKED A1), in line order, rendered as::

        Referenced in <source>/<markdown_path>:
        - Alt text: "<alt_text>"
          Context: <context_paragraphs>

    (the ``Context:`` line is omitted for a reference with no
    surrounding paragraphs). ``None`` when *content* has no reference
    to this image.
    """
    refs = [ref for ref in extract_image_refs(content) if _ref_points_at(ref, md_rel, image_rel)]
    if not refs:
        return None
    lines = [f"Referenced in {source}/{md_rel}:"]
    for ref in refs:
        lines.append(f'- Alt text: "{ref.alt_text}"')
        if ref.context:
            lines.append(f"  Context: {ref.context}")
    return "\n".join(lines)


#: Header line of a markdown doc's pending-image record (phase 125,
#: task 04, LOCKED A1) — the stored ``image_context`` of a markdown
#: file that references images with no document yet in the source.
#: :func:`_parse_missing_image_refs` is its parse target (the record is
#: the post-index enrichment pass's reference index; the header is
#: ignored by the parser — the bullets are the data).
IMAGE_CONTEXT_HEADER = "IMAGE_CONTEXT:"


def _format_missing_image_refs(refs: Sequence[ImageRef], source: str) -> str:
    """One markdown's missing-image references → the ``IMAGE_CONTEXT:``
    record stored on the markdown doc's ``image_context`` (phase 125,
    task 04, LOCKED A1) — the durable record that "this markdown
    referenced these images, which have no document yet in this
    source":

    .. code-block:: none

        IMAGE_CONTEXT:
        - [image.png] (line 42): "Alt text here" — 2 paragraphs of context
        - [diagram.jpg] (line 100): "" — 1 paragraph of context

    One bullet per reference, in reference order: the RESOLVED
    source-relative image path in the brackets (the caller resolves via
    :func:`_resolve_ref_path` — the post-index pass matches it against
    the image doc's ``path``, so the resolution must agree with the
    task-03 scan's), the 1-based line number, the alt text between the
    quotes (``""`` for ``![](path)``), and the count of the reference's
    context paragraphs (the task-02 window — the paragraphs themselves
    live on the markdown doc's ``content``; the pass re-extracts them:
    the record is the index, the content is the source). The shape is
    exactly what :func:`_parse_missing_image_refs` parses back (the
    "task 04b" parse target). *source* stays on the signature so the
    call site reads like the record's home (the record rides a doc row
    whose source is already known — display context, not data).

    Empty *refs* → ``""`` (the caller stores ``None`` for "no record"
    — an empty record would be a non-NULL empty string, and the pass's
    ``IS NOT NULL`` gate must see "no record" as NULL, not ``""``).
    """
    if not refs:
        return ""
    lines = [IMAGE_CONTEXT_HEADER]
    for ref in refs:
        n = len(ref.context.split("\n\n")) if ref.context else 0
        unit = "paragraph" if n == 1 else "paragraphs"
        lines.append(
            f'- [{ref.path}] (line {ref.line_number}): "{ref.alt_text}"'
            f" — {n} {unit} of context"
        )
    return "\n".join(lines)


#: One bullet of the ``IMAGE_CONTEXT:`` record — the exact output shape
#: of :func:`_format_missing_image_refs`: the resolved image path in
#: the brackets, the 1-based line number, the alt text between the
#: quotes, the context-paragraph count.
_MISSING_REF_LINE_RE = re.compile(
    r'^- \[(?P<path>[^\]]+)\] \(line (?P<line>\d+)\): "(?P<alt>.*)" — '
    r"(?P<count>\d+) paragraph(?:s)? of context$"
)


def _parse_missing_image_refs(text: str) -> list[tuple[str, int, str]]:
    """Parse a markdown doc's ``image_context`` record back into
    ``(image path, line number, alt text)`` triples (phase 125, task 04
    — the enrichment pass's reference index).

    Only bullets in :func:`_format_missing_image_refs`' exact shape
    parse (the greedy alt field still pins an alt that itself contains
    ``"``, as long as it does not contain the ``" — N paragraph…"``
    tail): the header line, an owner-edited value, or a future shape
    yields no references and the pass simply finds no match (fail-soft
    — a malformed record must never raise into the sync).
    """
    refs: list[tuple[str, int, str]] = []
    for line in text.splitlines():
        m = _MISSING_REF_LINE_RE.match(line)
        if m is not None:
            refs.append((m.group("path"), int(m.group("line")), m.group("alt")))
    return refs


def _missing_image_context(
    session: Session, *, source: str, md_rel: str, content: str
) -> str | None:
    """The markdown doc's ``image_context`` for one (re)indexed markdown
    file (phase 125, task 04, LOCKED A1): the ``IMAGE_CONTEXT:`` record
    of every ``![alt](path)`` reference whose image has NO ``is_image``
    document yet in this source, or ``None`` when every in-source
    reference names an existing image doc (the task-03 path owns those
    — enrichment at image-index time — and the field stays NULL) or the
    file has no references at all.

    Each reference is resolved exactly as the task-03 scan
    (:func:`_resolve_ref_path`); a reference that escapes the source
    (e.g. ``../../other/x.png``) names no in-source image document, ever,
    and is not recorded (the field is the pending-enrichment index, not
    a general reference dump). A re-indexed markdown recomputes the
    field from its CURRENT content — the image now exists (or the
    reference was dropped) → the field drops back to ``None``.
    """
    refs = extract_image_refs(content)
    if not refs:
        return None
    existing = set(
        session.scalars(
            select(Document.path).where(
                Document.source == source, Document.is_image.is_(True)
            )
        ).all()
    )
    missing: list[ImageRef] = []
    for ref in refs:
        resolved = _resolve_ref_path(ref.path, md_rel)
        if resolved is None or resolved in existing:
            continue
        missing.append(
            ImageRef(
                path=resolved,
                alt_text=ref.alt_text,
                context=ref.context,
                line_number=ref.line_number,
            )
        )
    if not missing:
        return None
    return _format_missing_image_refs(missing, source)


def _pending_image_context(
    source: str, image_rel: str, md_docs: Sequence[Document]
) -> str | None:
    """The task-04 pass's context for one pending image doc: the SAME
    combined string the task-03 scan builds (``Referenced in …`` blocks
    joined with a blank line) — but sourced from the markdown DOC ROWS
    instead of the on-disk files: a markdown doc contributes only when
    its stored ``image_context`` record lists the image's source-relative
    path (the durable "this markdown referenced this image" record,
    work 1), and the reference (alt text + surrounding paragraphs) is
    re-extracted from the doc's stored ``content`` (the full text is on
    the row — the record is the index, the content is the source).
    ``None`` when no markdown doc references the image.
    """
    blocks: list[str] = []
    for md in md_docs:
        listed = _parse_missing_image_refs(md.image_context or "")
        if image_rel not in {path for path, _, _ in listed}:
            continue
        block = _md_ref_block(source, md.path, md.content, image_rel)
        if block is not None:
            blocks.append(block)
    return "\n\n".join(blocks) if blocks else None


async def _enrich_pending_images(
    session: Session, *, source: str, llm: Embedder, summary: ImportSummary
) -> None:
    """The phase-125 task-04 post-index enrichment pass — once per
    source, after ALL of its files are indexed (:func:`import_sources`).

    Closes the markdown-FIRST case: a markdown file indexed while its
    referenced image had no document yet stores the reference on its
    ``image_context`` (work 1 — the ``IMAGE_CONTEXT:`` record). When the
    image is later indexed but the task-03 disk scan could not see the
    reference (the markdown file no longer on disk — a deleted / moved
    source file whose doc row survives a ``prune=False`` run — or any
    other disk-vs-DB drift), the image doc is still ``image_context IS
    NULL`` and this pass enriches it from the markdown doc's STORED rows
    (see :func:`_pending_image_context`).

    The enrichment is the task-03 mechanics on the existing doc: a fresh
    context-aware description (``describe_image`` with the context — the
    description remains the primary content, LOCKED A3), ``content`` =
    ``<context>\\n\\n<description>`` (and so the ``summary`` — the
    phase-122 ``summary = content`` convention), the context stored on
    ``image_context``, and the re-chunked + re-embedded pipeline (the
    re-describe reads the image's PERSISTENT copy — the source file may
    be the one that is gone).

    Idempotent by contract (ASSUMPTION 4): an image doc with
    ``image_context`` SET is never touched (already enriched — the
    task-03 path or an earlier pass), and a markdown doc's
    ``image_context`` is NOT cleared on success (it stays the retrieval
    display's record, task 05). Fail-soft: a missing persistent copy, a
    failed/empty re-description, or an embed error keeps the image doc
    as-is (warned / counted) and the next sync retries it; one bad image
    never blocks the others. A success counts ``updated`` + ``chunks``
    (the doc's content really changed — the caller's KB-change gate and
    the sources-version bump both fire on ``added + updated``, which is
    correct here) and a dedicated ``image-context-enriched`` log line
    (PLAN §9).
    """
    pending = session.scalars(
        select(Document).where(
            Document.source == source,
            Document.is_image.is_(True),
            Document.image_context.is_(None),
        )
    ).all()
    if not pending:
        return
    md_docs = session.scalars(
        select(Document).where(
            Document.source == source,
            Document.is_image.is_(False),
            Document.image_context.is_not(None),
        )
    ).all()
    if not md_docs:
        return
    for image_doc in pending:
        context = _pending_image_context(source, image_doc.path, md_docs)
        if context is None:
            continue  # no markdown doc records a reference to this image
        if not image_doc.image_path or not Path(image_doc.image_path).is_file():
            # No persistent bytes to re-describe — keep the doc as-is
            # (the next sync retries; a copy gone for good leaves an
            # unenrichable doc, and the warning is the signal).
            summary.images_failed += 1
            logger.warning(
                "import: image-context enrichment skipped (no persistent copy) "
                "source=%s path=%s",
                source, image_doc.path,
            )
            continue
        copy_path = Path(image_doc.image_path)
        description = await _describe_or_skip(
            llm,
            data=copy_path.read_bytes(),
            source=source,
            rel=image_doc.path,
            full_path=copy_path,
            context=context,
        )
        if description is None:
            summary.images_failed += 1
            logger.warning(
                "import: image re-description failed source=%s path=%s",
                source, image_doc.path,
            )
            continue
        try:
            image_doc.image_context = context
            image_doc.content = f"{context}\n\n{description}"
            chunks_n = await _replace_content_chunks(
                session, doc=image_doc, llm=llm,
                content=image_doc.content, rel=image_doc.path,
            )
            session.commit()
        except EmbeddingError as e:
            session.rollback()
            summary.errors += 1
            logger.error(
                "import: image-context enrichment failed source=%s path=%s — %s",
                source, image_doc.path, e,
            )
            continue
        summary.updated += 1
        summary.changed_paths.append((source, image_doc.path))
        summary.chunks += chunks_n
        logger.info(
            "import: image-context-enriched source=%s path=%s chunks=%d",
            source, image_doc.path, chunks_n,
        )
        await _store_summary(
            session, doc=image_doc, source=source, rel=image_doc.path,
            content=image_doc.content, llm=llm, summary=summary,
        )


#: The phase-126 embedded-image synthetic ``Document.path`` —
#: ``<pdf path>#page=<N>#img=<I>.<ext>`` (the upsert key built by
#: :func:`_index_one_embedded_image`: the parent PDF's source-relative
#: path, the 1-indexed page, the 0-based index on the page, the
#: mime-derived dotted extension). The task-03 backfill gates on this
#: exact shape: a non-matching ``is_image`` row (a standalone image's
#: plain path) is NEVER touched by the backfill. The ``pdf`` suffix is
#: case-insensitive (scoped flag) because the walk matches extensions
#: case-insensitively — an on-disk ``report.PDF`` produces a synthetic
#: path the backfill must still recognize.
_EMBEDDED_IMAGE_PATH_RE = re.compile(
    r"^(?P<pdf>.+(?i:pdf))#page=(?P<page>\d+)#img=(?P<img>\d+)(?P<ext>\.\w+)$"
)


async def _backfill_image_contexts(
    session: Session, *, source: str, root: Path, llm: Embedder, summary: ImportSummary
) -> None:
    """The phase-137 (task 03) per-sync document-context backfill — once
    per source, immediately AFTER the phase-125 enrichment pass
    (:func:`_enrich_pending_images`) in :func:`import_sources`.

    Closes the one-time gap the forward path (task 02) cannot reach:
    the PDF-embedded image docs a PRE-phase-137 sync left behind
    without the document block — ``image_context`` NULL (no context at
    all) or markdown-only (the phase-125 ``Referenced in …`` value —
    the document block missing) — are re-described WITH the document
    context, so the embedder indexes the provenance (the owner's
    HNB-guide stock-photo complaint: descriptions that name nothing
    about the document the image came from).

    Candidates — this source's ``is_image`` docs where ALL hold:

    1. the ``path`` parses as the phase-126 embedded-image synthetic
       path (:data:`_EMBEDDED_IMAGE_PATH_RE` — ``pdf_rel``, page,
       index, extension); a non-match (a standalone image's plain
       path) is NEVER touched;
    2. the stored ``image_context`` is NULL or does NOT start with
       ``Extracted from `` — the idempotency gate: a success writes a
       context starting with the header, so the doc never qualifies
       again (the following sync burns zero vision calls for it);
    3. the ``image_path`` names a file that exists on disk (the
       ``image_dir`` copy — the bytes source; missing → debug log,
       skip, the doc stays as-is).

    Per candidate (fail-soft, ONE transaction per image — the
    :func:`_enrich_pending_images` error handling, mirrored): the page
    text is re-extracted from the parent PDF WHEN IT IS STILL IN THE
    SOURCE (:func:`_pdf_image_context_inputs` ONCE per unique
    ``pdf_rel`` — a local cache, one parent may own many images; a
    parent gone for good reads as an empty map + usable → the origin
    line only, which still needs no disk), the document block
    (:func:`_pdf_image_context_block`) is combined with the stored
    markdown blocks VERBATIM when present (:func:`_combine_image_contexts`
    — document FIRST; the phase-125 pass owns the markdown context and
    ran earlier this sync, so re-scanning the disk for it is NOT this
    pass's job), and the fresh description goes through the SAME seam
    (:func:`_describe_or_skip` — the mime comes from the copy's
    extension, exactly like the forward path). A ``None`` description
    leaves the row UNTOUCHED (nothing was written) —
    ``images_failed`` + a warning, and the gate still matches, so the
    next sync retries it. A success replaces the row's
    ``content``/``summary``/``image_context`` + chunks
    (:func:`_replace_content_chunks` in the image's own transaction —
    an embed failure rolls back THIS image only, the copy stays — then
    the image-convention :func:`_store_summary`, the description IS the
    summary, verbatim) and counts the log-only
    ``summary.image_context_backfilled`` (the sync ``detail`` shape is
    untouched — the stats log-only convention, phase 133) with one loud
    line (PLAN §9).

    Deliberately UNCHANGED on the row: ``content_hash`` (the byte
    digest — the image identity is untouched), ``created_at`` /
    ``created_at_manual`` (the SOURCE did not change — the phase-106
    date rules do not apply), ``title``, and ``image_path``.

    Driven by committed doc rows (like the enrichment pass), so it is
    safe under ``limit`` (it deletes nothing); the common case after
    the one-time backfill (no candidates) returns with zero vision
    calls and zero disk reads beyond the candidate query.
    """
    image_docs = session.scalars(
        select(Document).where(
            Document.source == source, Document.is_image.is_(True)
        )
    ).all()
    if not image_docs:
        return
    # The parent PDF's context inputs, cached per unique ``pdf_rel``
    # (one parent may own many images): a parent still in the source
    # gets the task-02 probe (the per-page map + the phase-133 gate
    # verdict — the same helper the forward path uses, A7); a parent
    # gone for good reads as an empty map + usable — the origin line
    # only (the origin is still valuable and needs no disk).
    context_cache: dict[str, tuple[dict[int, str], bool]] = {}

    def _context_for(pdf_rel: str) -> tuple[dict[int, str], bool]:
        if pdf_rel not in context_cache:
            parent = root / pdf_rel
            if parent.is_file():
                context_cache[pdf_rel] = _pdf_image_context_inputs(parent)
            else:
                context_cache[pdf_rel] = ({}, True)
                logger.debug(
                    "import: backfill parent PDF missing source=%s path=%s "
                    "(origin line only)",
                    source, pdf_rel,
                )
        return context_cache[pdf_rel]

    for doc in image_docs:
        parsed = _EMBEDDED_IMAGE_PATH_RE.match(doc.path)
        if parsed is None:
            continue  # standalone / non-embedded — NEVER touched
        if doc.image_context is not None and doc.image_context.startswith(
            "Extracted from "
        ):
            continue  # already backfilled (the idempotency gate)
        if not doc.image_path or not Path(doc.image_path).is_file():
            # No persistent bytes to re-describe — keep the doc as-is
            # (the next sync retries; a copy gone for good leaves an
            # unbackfillable doc, and the debug line is the signal).
            logger.debug(
                "import: backfill skipped (no persistent copy) source=%s path=%s",
                source, doc.path,
            )
            continue
        pdf_rel = parsed.group("pdf")
        page = int(parsed.group("page"))
        page_texts, text_usable = _context_for(pdf_rel)
        # The stored markdown blocks (the phase-125 value) are kept
        # VERBATIM under the document block — document FIRST. A value
        # that is neither the markdown shape nor NULL contributes
        # nothing (the doc block alone).
        existing = doc.image_context
        md_blocks = (
            existing
            if existing is not None and existing.startswith("Referenced in ")
            else None
        )
        combined = _combine_image_contexts(
            _pdf_image_context_block(
                source, pdf_rel, page, page_texts, text_usable=text_usable
            ),
            md_blocks,
        )
        copy_path = Path(doc.image_path)
        try:
            data = copy_path.read_bytes()
        except OSError as e:
            logger.debug(
                "import: backfill skipped (copy unreadable) source=%s path=%s — %s",
                source, doc.path, e,
            )
            continue
        description = await _describe_or_skip(
            llm, data=data, source=source, rel=doc.path,
            full_path=copy_path, context=combined,
        )
        if description is None:
            # Fail-soft: nothing was written — the row is untouched,
            # the gate still matches, the next sync retries it.
            summary.images_failed += 1
            logger.warning(
                "import: image context backfill re-description failed "
                "source=%s path=%s",
                source, doc.path,
            )
            continue
        try:
            # The SOURCE did not change — the phase-106 date rules do
            # not apply: only the index-time and the context-carrying
            # fields move (``content_hash``/``title``/``image_path`` /
            # the creation date stay deliberately untouched).
            doc.image_context = combined
            doc.content = f"{combined}\n\n{description}"
            doc.indexed_at = datetime.now(UTC)
            chunks_n = await _replace_content_chunks(
                session, doc=doc, llm=llm, content=doc.content, rel=doc.path
            )
            session.commit()
        except EmbeddingError as e:
            session.rollback()
            # The description succeeded, but the embed gave up: the row
            # is untouched (rolled back), the copy stays, the next sync
            # retries it.
            summary.errors += 1
            logger.error(
                "import: image context backfill failed source=%s path=%s — %s",
                source, doc.path, e,
            )
            continue
        summary.image_context_backfilled += 1
        logger.info(
            "import: backfilled image context source=%s path=%s chunks=%d",
            source, doc.path, chunks_n,
        )
        await _store_summary(
            session, doc=doc, source=source, rel=doc.path,
            content=doc.content, llm=llm, summary=summary,
        )


async def _describe_or_skip(
    llm: Embedder,
    *,
    data: bytes,
    source: str,
    rel: str,
    full_path: Path,
    context: str | None = None,
) -> str | None:
    """Phase 122 — the SINGLE seam for the image description (task 03).

    Returns the vision description that becomes the image document's
    ``content`` (and ``summary`` — the ONLY embedded text of the doc),
    or ``None`` when the description failed or came back empty — the
    caller then SKIPS the doc entirely (no row, no copy) and counts
    ``summary.images_failed`` (LOCKED A3, fail-soft: the sync continues,
    the warning + counter are the signal).

    The seam is one line by design (the importer's tests patch exactly
    this function): :func:`app.rag.summarizer.describe_image` — ONE
    CHAT-model (vision) call (LOCKED A3) with the bytes as a data URL
    whose mime comes from :data:`app.rag.summarizer.IMAGE_MIMES`
    (dotted extension; an unlisted ``BOR_IMAGE_EXTENSIONS`` token takes
    the generic fallback — a rejection there fails soft like any other
    description error). ``source``/``rel`` stay on the signature so the
    caller (and the patch) reads like the document being described;
    the failure's doc identity is logged by the caller's warning.

    ``context`` (phase 125, task 03; generalized in phase 137, task
    02) is the optional document/reference context built by the
    importer — the markdown reference blocks
    (:func:`_markdown_image_context`) for a standalone image, the
    document block (:func:`_pdf_image_context_block`) + markdown blocks
    (:func:`_combine_image_contexts`) for a PDF-embedded image —
    forwarded verbatim to :func:`app.rag.summarizer.describe_image`
    — appended AFTER the locked describe prompt so the vision model can
    write a context-aware description (LOCKED A3: the description
    remains the primary content). ``None`` (a reference-free standalone
    image) keeps the message byte-identical to the pre-phase shape.
    """
    mime = IMAGE_MIMES.get(full_path.suffix.lower(), IMAGE_FALLBACK_MIME)
    return await describe_image(llm, data=data, mime=mime, context=context)


def _delete_image_copy(image_path: str | None) -> None:
    """Best-effort removal of a stale image copy (phase 122).

    A missing path is a no-op (already gone — e.g. the owner cleaned
    the image dir); an unreadable one is logged, never raised — copy
    cleanup must not break the sync (the doc row's fate is decided by
    the upsert/prune logic, not by filesystem hygiene).
    """
    if not image_path:
        return
    try:
        Path(image_path).unlink(missing_ok=True)
    except OSError as e:
        logger.warning("import: could not delete image copy %s — %s", image_path, e)


def _delete_page_copies(doc: Document, page_dir: Path) -> None:
    """Best-effort removal of one document's stale PDF page copies
    (phase 148, task 01 — the :func:`_delete_image_copy` convention):
    the ``<doc-id>-p{N}.png`` files for N in 1..``doc.page_count`` —
    the row knows its own extent (a NULL/zero count is a no-op, which
    covers fresh docs, non-qualifying docs, and every fail-soft miss).

    A missing file is a silent no-op (already gone — e.g. the owner
    cleaned the pages dir); an unreadable one is logged, never raised —
    copy cleanup must not break the sync (the doc row's fate is decided
    by the upsert/prune logic, not by filesystem hygiene).
    """
    if not doc.page_count:
        return
    for n in range(1, doc.page_count + 1):
        path = page_dir / f"{doc.id}-p{n}.png"
        try:
            path.unlink(missing_ok=True)
        except OSError as e:
            logger.warning("import: could not delete PDF page copy %s — %s", path, e)


async def _index_image_file(
    session: Session,
    *,
    source: str,
    rel: str,
    full_path: Path,
    llm: Embedder,
    summary: ImportSummary,
    file_digest: str,
    raw_date: datetime,
    on_chunk: Callable[[int, int], None] | None = None,
) -> None:
    """The phase-122 image branch of :func:`_index_file` — a standalone
    image is indexed from its BYTES, never its text:

    * the sha256 digest is over the raw bytes (the digest rule is
      content identity — the same bytes are the same document);
    * the PERSISTENT copy lands in ``settings.image_dir`` as
      ``<doc-id>.<ext>`` (the dir is created on demand; the copy is
      written only AFTER a successful description, so a failure never
      leaves an orphan; a changed image deletes the stale copy before
      replacing it);
    * ``content`` is the vision description — the ONLY embedded text of
      the document (the embedding model never sees pixels) — and the
      normal chunk pipeline then embeds it, with the phase-30 summary
      path running on it (the ``is_summary`` position −1 chunk mirrors
      ``Document.summary``).

    ``raw_date`` follows the text path exactly — the CALLER
    (:func:`_index_file`) computes the phase-106 D2 fallback (no
    source date in the map → the file's mtime) before the fast-path
    check, because the unchanged path refreshes the stored date from
    the same source; the D1 manual-date lock and the D4 refresh apply
    unmodified through the shared :func:`_unchanged_doc`. ``file_digest``
    is the caller's raw-bytes sha256 (phase 147 — the fast-path
    signal and this branch's stamp value).

    Fail-soft (LOCKED A3): a failed/empty description SKIPS the doc
    entirely (no row, no copy) — ``summary.images_failed`` + a warning,
    the sync continues.

    Phase 125 (task 03, LOCKED A1/A2/A3): a NEW/CHANGED image first
    scans the same source's markdown files for references to it (see
    :func:`_markdown_image_context`). When one or more exist, the
    combined context string goes to the vision call (``context`` — the
    model writes a context-aware description; the description remains
    the primary content) and is stored on the doc's ``image_context``;
    the doc's ``content`` becomes ``<context>\n\n<description>`` (and so
    does ``summary`` — the phase-122 ``summary = content`` convention —
    as do the embedded chunks). No reference → ``image_context`` stays
    NULL and ``content`` is the bare description (byte-identical to
    pre-phase). The UNCHANGED path never re-scans (the stored context
    is part of the row's committed content; a changed source state
    re-indexes the file through this path).

    ``on_chunk`` (phase 129, task 02) is the in-flight file's
    chunk-position hook: it is forwarded to the description's content
    embed (:func:`_replace_content_chunks`). A description (enriched or
    not) chunks to ONE chunk, so the hook reports ``1/1`` — acceptable,
    it IS the file's only chunk. ``None`` keeps the branch
    byte-identical.
    """
    settings = llm.settings
    # Phase 147: the fast path (:func:`_index_file`) already returned
    # for an unchanged file — everything below (``read_bytes``, the
    # vision describe, the copy) runs only on the first index or a
    # byte change.
    data = full_path.read_bytes()
    digest = hashlib.sha256(data).hexdigest()
    doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))

    if doc is not None and doc.content_hash == digest:
        # Phase 147: the shared unchanged path — a pre-phase NULL-hash
        # row, or a bytes-changed/content-same image (the byte digest
        # still matches). The helper stamps the new file hash (the
        # next sync takes the fast path); for an image the summary is
        # the stored description (``doc.content``), so the image-aware
        # ``_store_summary`` backfill re-stores it verbatim.
        await _unchanged_doc(
            session, doc, source=source, rel=rel, llm=llm, summary=summary,
            raw_date=raw_date, file_digest=file_digest, full_path=full_path,
        )
        return

    verb = "updated" if doc is not None else "added"
    # Phase 125 (task 03, LOCKED A1/A2): BEFORE the description — the
    # markdown reference context for this image (the alt text +
    # surrounding paragraphs of the ``![alt](path)`` references that
    # name it in the same source), or ``None`` (standalone image — the
    # whole phase-125 enrichment below is a no-op, byte-identical to
    # pre-phase). It goes to the vision model (a context-aware
    # description, LOCKED A3) AND is stored on the doc (``image_context``
    # — the retrieval display's context line, task 05).
    image_context = _markdown_image_context(
        _source_root_for(full_path, rel), llm, source=source, image_rel=rel
    )
    # The description is the doc's content (task 03: and its summary) —
    # it is generated BEFORE anything is written, so a failure skips the
    # doc with no row and no copy (the copy is only made after a
    # successful description — a failure never leaves an orphan).
    content = await _describe_or_skip(
        llm, data=data, source=source, rel=rel, full_path=full_path,
        context=image_context,
    )
    if content is None:
        # LOCKED A3 fail-soft: an undescribed image is unsearchable
        # noise — skip the doc entirely (no row, no copy).
        summary.images_failed += 1
        logger.warning("import: image description failed source=%s path=%s", source, rel)
        return
    if image_context is not None:
        # Phase 125 (task 03): the enriched content — the reference
        # context FIRST (for retrieval), then the description. The
        # ``summary`` field mirrors this (the phase-122 image
        # convention: ``summary = content``), and the chunk pipeline
        # below embeds it.
        content = f"{image_context}\n\n{content}"

    # The persistent copy: uploads are replaced on every upload, git
    # checkouts are re-cloned, local dirs are user-edited — the served
    # bytes must outlive the source file. Named by the doc id: a new
    # doc's id is the uuid4 chosen here (row and copy agree); a changed
    # doc keeps its id (the copy path is stable).
    doc_id = doc.id if doc is not None else uuid.uuid4()
    image_dir = Path(settings.image_dir).expanduser()
    image_dir.mkdir(parents=True, exist_ok=True)
    if doc is not None:
        # A CHANGED image (hash differs): the stale copy is deleted
        # before replacement.
        _delete_image_copy(doc.image_path)
    copy_path = image_dir / f"{doc_id}{full_path.suffix.lower()}"
    copy_path.write_bytes(data)

    # The non-markdown title rule (the image's content is prose, but the
    # doc IS the image — the file stem is the title).
    title = full_path.stem
    if doc is None:
        doc = Document(
            id=doc_id,
            source=source,
            path=rel,
            full_path=str(full_path),
            title=title,
            content=content,
            content_hash=digest,
            # Phase 147: the raw-bytes stamp (the next sync's fast
            # path).
            file_hash=file_digest,
            indexed_at=datetime.now(UTC),
            created_at=normalize_doc_date(raw_date),
            is_image=True,
            image_path=str(copy_path),
            # Phase 125 (task 03, LOCKED A1): the referencing markdown's
            # alt text + surrounding paragraphs (``None`` for a
            # standalone image — the pre-phase row shape).
            image_context=image_context,
        )
        session.add(doc)
    else:
        doc.full_path = str(full_path)
        doc.title = title
        doc.content = content
        doc.content_hash = digest
        # Phase 147: re-stamp (the file's bytes changed).
        doc.file_hash = file_digest
        doc.indexed_at = datetime.now(UTC)
        # Phase 106 (D4): a content change is a new document version —
        # the date is re-sourced and a previous manual correction is
        # reset (it referred to the old content).
        doc.created_at = normalize_doc_date(raw_date)
        doc.created_at_manual = False
        doc.is_image = True
        doc.image_path = str(copy_path)
        # Phase 125 (task 03): a re-indexed image re-scans its markdown
        # references — the context follows the current source state
        # (a deleted referencing markdown drops it back to ``None``).
        doc.image_context = image_context

    session.flush()  # guarantees doc.id even for brand-new rows

    # Phase 1+2 — the same pipeline on the description (chunk, replace
    # the chunk rows (embeddings NULL), embed, commit — one transaction
    # per file; the endpoint token-cap retry lives in the shared helper,
    # and a description is short, so it never fires in practice).
    # Phase 129 (task 02): the in-flight file's chunk-position hook
    # rides this embed (one description chunk → ``1/1``).
    chunks_n = await _replace_content_chunks(
        session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
    )
    session.commit()

    if verb == "added":
        summary.added += 1
    else:
        summary.updated += 1
    summary.changed_paths.append((source, rel))
    summary.chunks += chunks_n
    logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)

    # Phase 30 shape on the description — image-aware (task 03, LOCKED
    # A3): the description IS the summary (stored verbatim, no
    # ``lite`` call), so ``doc.summary`` == ``doc.content`` and the
    # ``is_summary`` position −1 chunk mirrors it.
    await _store_summary(
        session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
    )


#: The synthetic path marker of a PDF-embedded image doc (phase 126,
#: LOCKED A3 — ``<pdf path>#page=<N>#img=<index>.<ext>``):
#: :data:`_EMBEDDED_IMAGE_RE` is the prune's parent rule below.
_EMBEDDED_IMAGE_RE = re.compile(r"#page=\d+#img=\d+")


def _prune(
    session: Session,
    source_names: set[str],
    seen: set[tuple[str, str]],
    images: bool = False,
    summary: ImportSummary | None = None,
    page_dir: Path | None = None,
) -> int:
    """Delete documents of *source_names* whose file is no longer in *seen*.

    *images* (phase 122 prune guard, LOCKED, derived from A3/A4): while
    the image toggle is OFF (``images`` False), every ``is_image`` doc is
    SKIPPED — the image is invisible to an images-off walk, not a deleted
    file, so pruning it would silently destroy image documents on the
    first images-off sync. Toggle ON → normal semantics: a deleted image
    file prunes its doc, and the pruned image's ``image_dir`` copy is
    deleted with it.

    Phase 126 (LOCKED A3) — the PDF-embedded image's parent rule: an
    embedded image doc's synthetic path (``#page=<N>#img=<index>``)
    NEVER appears in the walk's *seen* — the image is derived from its
    parent PDF, whose on-disk file is the walked file. Pruning it on the
    plain rule would destroy it in the very sync that (re)indexes it,
    so it is skipped while its parent PDF's rel is still in *seen* (the
    image is invisible to the walk, not deleted). A pruned/absent parent
    PDF prunes its embedded image docs with it (row + ``image_dir`` copy
    — no orphans when the PDF leaves the source). The rule holds in BOTH
    toggle states (toggle off, the phase-122 guard already skips every
    ``is_image`` doc; this one is the toggle-on half of the same
    "invisible, not deleted" contract).

    *summary* (phase 127, task 01): when given, every document that is
    ACTUALLY deleted (i.e. not skipped by either guard above) records
    its ``(doc.source, doc.path)`` on ``summary.changed_paths`` —
    appended BEFORE the ``session.delete`` — so the sync's incremental
    folder-summary regeneration (phase 127) sees the pruned docs' folders
    too. Callers passing nothing keep the pre-phase count-only contract.

    *page_dir* (phase 148, task 01): when given (the sync's
    ``Path(settings.pdf_page_dir).expanduser()``), every pruned doc's
    PDF page copies (1..``doc.page_count`` — a NULL count is a no-op,
    covering non-qualifying docs) are deleted beside the row and the
    image copy — no orphans when a page-qualifying PDF (its parent or
    a standalone doc) leaves the source. Callers passing nothing keep
    the pre-phase copy-cleanup contract.
    """
    if not source_names:
        return 0
    pruned = 0
    docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()
    for doc in docs:
        if (doc.source, doc.path) not in seen:
            if doc.is_image and not images:
                # Prune guard: invisible to the walk, not deleted.
                continue
            if doc.is_image and _EMBEDDED_IMAGE_RE.search(doc.path):
                # Phase 126 (A3): the parent PDF's rel is the path minus
                # its ``#page=…#img=…`` fragment — the walked file.
                parent_rel = doc.path.split("#", 1)[0]
                if (doc.source, parent_rel) in seen:
                    # Parent PDF still in the source — the image is
                    # invisible to the walk, not deleted.
                    continue
            _delete_image_copy(doc.image_path)
            # Phase 148 (task 01): the page-qualifying doc's page
            # copies go with the row (a pruned parent PDF's copies too
            # — the same site as its embedded-image copies' deletion).
            if page_dir is not None:
                _delete_page_copies(doc, page_dir)
            if summary is not None:
                summary.changed_paths.append((doc.source, doc.path))
            session.delete(doc)
            pruned += 1
            logger.info("import: pruned source=%s path=%s", doc.source, doc.path)
    if pruned:
        session.commit()
    return pruned