phase: 126_pdf_support
All gates verified green — no code changes were needed on this pass. Final report: ## Phase 126 — PDF support: final verification pass **Verified (all tasks 01–05 already complete in working tree):** - `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token - `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk **Test / lint / coverage results:** - `uv run pytest` → 2997 passed - `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** 1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests) 2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test) 3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites) 4. PDF renders as extracted text in viewer — ✅ (E2E) 5. Test/coverage/lint gates — ✅ (above) 6. Commit + phase move — harness handles (per protocol, no `git` on this pass) **Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree. **Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
@@ -85,6 +85,30 @@ summary (stored verbatim, no ``lite`` call, no pointer line), so the
|
||||
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
|
||||
equals ``Document.content``.
|
||||
|
||||
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
|
||||
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
|
||||
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
|
||||
AFTER the image branch, BEFORE any text read — a PDF is never
|
||||
``read_text``-ed; the binary is owned by the phase-126 parser). The
|
||||
document's ``content`` is the extracted text (every text-bearing page
|
||||
prefixed with the ``---PAGE-N---`` marker — A2, one document per PDF),
|
||||
the digest is sha256 over that text (the digest rule is unchanged —
|
||||
content identity), the title is the PDF's embedded metadata title or
|
||||
the file stem, and the page-aware chunker (task 03) chunks it with a
|
||||
``Page N:`` anchor. With the phase-122 ``images`` toggle ON, the
|
||||
embedded images join the image pipeline (A3, :func:
|
||||
`_index_pdf_embedded_images`): each is described through the SAME
|
||||
``_describe_or_skip`` seam (the bytes as a data URL), copied to
|
||||
``settings.image_dir`` (the copy is written only AFTER a successful
|
||||
description — a failure never leaves an orphan), and indexed as a
|
||||
standalone ``is_image`` document under the synthetic path
|
||||
``<pdf path>#page=<N>#img=<index>.<ext>`` (the description IS the
|
||||
content and summary — the phase-122 convention). Toggle OFF → the
|
||||
embedded images are skipped (debug-logged) and the prune guard keeps
|
||||
any existing image docs; an unsupported image mime (outside
|
||||
``BOR_IMAGE_EXTENSIONS``) is skipped with a warning. A PDF's viewer
|
||||
presentation is its extracted text (A4 — nothing binary is served).
|
||||
|
||||
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
|
||||
description, the same source's importable markdown files are scanned for
|
||||
```` references that name the image (the task-02
|
||||
@@ -135,6 +159,7 @@ from datetime import UTC, datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -145,6 +170,7 @@ from app.rag.chunker import chunk_document, extract_title
|
||||
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
|
||||
from app.rag.llm import EmbeddingError, LLMError
|
||||
from app.rag.md_images import ImageRef, extract_image_refs
|
||||
from app.rag.pdf import PdfImage, extract_pdf_images, extract_pdf_text
|
||||
from app.rag.summarizer import (
|
||||
IMAGE_FALLBACK_MIME,
|
||||
IMAGE_MIMES,
|
||||
@@ -652,6 +678,14 @@ async def _index_file(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||||
summary=summary, raw_date=raw_date,
|
||||
)
|
||||
# Phase 126: PDF files take the PDF index path — AFTER the image
|
||||
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
|
||||
# owns it and ``read_text`` is never called for one).
|
||||
if full_path.suffix.lower() == ".pdf":
|
||||
return await _index_pdf_file(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||||
summary=summary, raw_date=raw_date,
|
||||
)
|
||||
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||||
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
@@ -855,6 +889,461 @@ async def _store_summary(
|
||||
logger.error("import: summary failed source=%s path=%s — %s", source, rel, e)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PDF indexing (phase 126, LOCKED A1/A2/A3/A4).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The ``Document.title`` column width (``app.models``) — the metadata
|
||||
#: title is capped to it (an over-long PDF title must not blow up the
|
||||
#: upsert).
|
||||
_PDF_TITLE_MAX_CHARS = 500
|
||||
|
||||
#: MIME → extension for the phase-122 image family (phase 126, task 04)
|
||||
#: — the reverse of the ``app.rag.summarizer.IMAGE_MIMES`` values for the
|
||||
#: formats PyMuPDF reports. An unlisted mime takes ``.bin`` (never a
|
||||
#: member of ``BOR_IMAGE_EXTENSIONS`` — the image is skipped with a
|
||||
#: warning, the honest fail-soft outcome).
|
||||
_MIME_TO_EXT: dict[str, str] = {
|
||||
"image/png": ".png",
|
||||
"image/jpeg": ".jpg",
|
||||
"image/gif": ".gif",
|
||||
"image/webp": ".webp",
|
||||
"image/bmp": ".bmp",
|
||||
"image/tiff": ".tiff",
|
||||
}
|
||||
|
||||
|
||||
def _extract_pdf_title(full_path: Path, fallback: str) -> str:
|
||||
"""The title of one PDF document (phase 126, task 04).
|
||||
|
||||
The embedded metadata title (``fitz``'s ``doc.metadata["title"]``)
|
||||
— trimmed and capped at the :data:`_PDF_TITLE_MAX_CHARS` column
|
||||
width. An unreadable PDF, an absent metadata dict, or an
|
||||
empty/whitespace-only title falls back to *fallback* (the file stem
|
||||
— the non-markdown title rule; the metadata title is a convenience,
|
||||
never a requirement).
|
||||
"""
|
||||
try:
|
||||
doc = fitz.open(str(full_path))
|
||||
except Exception:
|
||||
return fallback
|
||||
with doc:
|
||||
meta = doc.metadata
|
||||
title = str(meta.get("title") or "").strip() if meta else ""
|
||||
if not title:
|
||||
return fallback
|
||||
return title[:_PDF_TITLE_MAX_CHARS]
|
||||
|
||||
|
||||
async def _index_pdf_file(
|
||||
session: Session,
|
||||
*,
|
||||
source: str,
|
||||
rel: str,
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
) -> None:
|
||||
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
|
||||
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
|
||||
for the document, then the embedded images (A3):
|
||||
|
||||
* ``content`` is the parser's text — every text-bearing page
|
||||
prefixed with the ``---PAGE-N---`` marker (A2 — one document per
|
||||
PDF, the markers are the page anchors the page-aware chunker
|
||||
(task 03) splits on); an unopenable/empty PDF yields ``""``
|
||||
(still a document — it may carry embedded images — with no
|
||||
content chunks; a warning is the signal).
|
||||
* the sha256 digest is over the extracted text (the digest rule is
|
||||
content identity — the same text is the same document version;
|
||||
the empty case digests the empty byte string).
|
||||
* the title is the metadata title or the file stem
|
||||
(:func:`_extract_pdf_title`); the date follows the text path
|
||||
exactly (the phase-106 D2 mtime fallback — PDFs carry no
|
||||
filesystem-visible creation date — plus the D1 manual-date lock
|
||||
and the D4 refresh on the unchanged path).
|
||||
* chunk + embed runs through the shared
|
||||
:func:`_replace_content_chunks` helper (the endpoint token-cap
|
||||
retry lives there), and the phase-30/118 (A2) ``lite`` summary
|
||||
path runs on the text — PDFs are no more excluded than any other
|
||||
non-markdown format.
|
||||
* the unchanged path (same text digest) skips everything — no
|
||||
re-extract, no re-embed, no image pass — and runs the same
|
||||
backfill/manual-date/date-refresh dance as the text path.
|
||||
* THEN the embedded images: :func:`_index_pdf_embedded_images`
|
||||
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
|
||||
|
||||
``raw_date`` follows the text path: the caller's git last-commit
|
||||
date or ``None`` (the mtime fallback, read before the unchanged
|
||||
early-return — the unchanged path refreshes from the same source).
|
||||
"""
|
||||
# Extract the text (fail-soft: an unopenable PDF yields "" — the
|
||||
# document may still carry embedded images worth indexing).
|
||||
content = extract_pdf_text(full_path)
|
||||
if not content:
|
||||
logger.warning(
|
||||
"import: PDF has no extractable text source=%s path=%s", source, rel
|
||||
)
|
||||
# Content identity: the digest is over the extracted text (the
|
||||
# empty case is the empty byte string's digest).
|
||||
digest = (
|
||||
hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
if content
|
||||
else hashlib.sha256(b"").hexdigest()
|
||||
)
|
||||
title = _extract_pdf_title(full_path, fallback=full_path.stem)
|
||||
if raw_date is None:
|
||||
# D2 fallback (same as the text path): no source date in the
|
||||
# map → the file's mtime (one stat). Read before the unchanged
|
||||
# early-return — the unchanged path refreshes from it.
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
# Unchanged PDF (text digest) — the text path's unchanged branch,
|
||||
# unmodified in shape: no re-extract, no re-embed, no image pass.
|
||||
summary.unchanged += 1
|
||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||
# Phase 118 (A2): an unchanged doc whose summary is still NULL
|
||||
# gets the same best-effort summary pass (see the text path).
|
||||
if doc.summary is None:
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=content,
|
||||
llm=llm, summary=summary, backfill=True,
|
||||
)
|
||||
if doc.created_at_manual:
|
||||
# D1/D4: the owner's correction survives the sync — no write
|
||||
# at all (the text path's manual-date early-return).
|
||||
return
|
||||
# D4: the date refreshes on every sync and may go OLDER (no
|
||||
# monotonic guard); a date-only refresh counts unchanged.
|
||||
target = normalize_doc_date(raw_date)
|
||||
if target != doc.created_at:
|
||||
doc.created_at = target
|
||||
session.commit()
|
||||
summary.dates_updated += 1
|
||||
logger.info(
|
||||
"import: date-refreshed source=%s path=%s date=%s",
|
||||
source, rel, doc.created_at.isoformat(),
|
||||
)
|
||||
return
|
||||
|
||||
verb = "updated" if doc is not None else "added"
|
||||
if doc is None:
|
||||
doc = Document(
|
||||
source=source,
|
||||
path=rel,
|
||||
full_path=str(full_path),
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash=digest,
|
||||
indexed_at=datetime.now(UTC),
|
||||
created_at=normalize_doc_date(raw_date),
|
||||
)
|
||||
session.add(doc)
|
||||
else:
|
||||
doc.full_path = str(full_path)
|
||||
doc.title = title
|
||||
doc.content = content
|
||||
doc.content_hash = digest
|
||||
doc.indexed_at = datetime.now(UTC)
|
||||
# Phase 106 (D4): a content change is a new document version —
|
||||
# the date is re-sourced and a manual correction reset.
|
||||
doc.created_at = normalize_doc_date(raw_date)
|
||||
doc.created_at_manual = False
|
||||
|
||||
session.flush() # guarantees doc.id even for brand-new rows
|
||||
|
||||
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
|
||||
# aware chunker dispatches on the ``.pdf`` suffix; empty content
|
||||
# clears the chunk rows and returns 0), committed atomically with
|
||||
# the document.
|
||||
chunks_n = await _replace_content_chunks(
|
||||
session, doc=doc, llm=llm, content=content, rel=rel
|
||||
)
|
||||
session.commit()
|
||||
|
||||
if verb == "added":
|
||||
summary.added += 1
|
||||
else:
|
||||
summary.updated += 1
|
||||
summary.chunks += chunks_n
|
||||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||||
|
||||
# Phase 30; phase 118 (A2): the ``lite`` summary of the extracted
|
||||
# text (best-effort — a failure leaves the PDF indexed without a
|
||||
# summary, counted in ``summary_errors``).
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
|
||||
)
|
||||
|
||||
# Phase 126 (A3): the embedded images join the phase-122 pipeline
|
||||
# (gated by the ``images`` toggle — skipped, debug-logged, while it
|
||||
# is off). Runs after the document commit, so a per-image failure
|
||||
# never loses the PDF's text index.
|
||||
await _index_pdf_embedded_images(
|
||||
session, source=source, rel=rel, full_path=full_path,
|
||||
llm=llm, summary=summary, raw_date=raw_date,
|
||||
)
|
||||
|
||||
|
||||
async def _index_pdf_embedded_images(
|
||||
session: Session,
|
||||
*,
|
||||
source: str,
|
||||
rel: str,
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime,
|
||||
) -> None:
|
||||
"""The phase-126 (A3) embedded-image pass — one call per (re)indexed
|
||||
PDF, after its document commit.
|
||||
|
||||
Each embedded image (page order, 0-based index on the page — the
|
||||
parser's order) is indexed as a STANDALONE image document through
|
||||
the phase-122 mechanics, inline (no temp file, no
|
||||
``_index_image_file`` refactor):
|
||||
|
||||
* gated by the phase-122 ``images`` toggle (the single knob, LOCKED
|
||||
A3): while it is OFF the whole pass is skipped (one debug line —
|
||||
the extraction itself is not worth running) and the prune guard
|
||||
keeps any existing image docs;
|
||||
* an image whose mime maps to no ``BOR_IMAGE_EXTENSIONS`` member
|
||||
(:data:`_MIME_TO_EXT`) is skipped with a warning (the unsupported-
|
||||
extension rule, same as a standalone image's);
|
||||
* the document's synthetic ``path`` is
|
||||
``<pdf path>#page=<N>#img=<index>.<ext>`` — unique per image
|
||||
occurrence, stable across re-indexes (the upsert key); the title
|
||||
is ``Embedded image from <pdf stem> page <N>``;
|
||||
* the digest is sha256 over the image BYTES (the digest rule is
|
||||
content identity); an image doc whose digest matches is skipped
|
||||
(no re-describe, no re-embed — unchanged);
|
||||
* the description comes through the SAME single seam
|
||||
(:func:`_describe_or_skip` — the bytes as a data URL whose mime
|
||||
comes from the copy path's extension); a failed/empty description
|
||||
SKIPS the image entirely (no row, no copy) — ``images_failed`` +
|
||||
a warning, the sync continues (LOCKED A3 fail-soft). The
|
||||
``image_dir`` copy (``<doc-id>.<ext>`` — a new image's id is the
|
||||
uuid4 chosen here; a changed image keeps its id and its stale
|
||||
copy is deleted before replacement) is written only AFTER a
|
||||
successful description (a failure never leaves an orphan);
|
||||
* phase 125 parity (the standalone image's task-03 scan): a same-
|
||||
source markdown file that references the image by its synthetic
|
||||
path (````) enriches
|
||||
the description — the combined context goes to the vision call
|
||||
and is stored on the doc's ``image_context``, and ``content``
|
||||
becomes ``<context>\\n\\n<description>`` (as does ``summary`` —
|
||||
the image convention); no reference keeps the bare shape
|
||||
(byte-identical to the un-enriched case). The markdown-FIRST case
|
||||
(a markdown indexed before the PDF recorded its reference on its
|
||||
own row) is closed by the shared post-index pass
|
||||
(:func:`_enrich_pending_images`), exactly as for a standalone
|
||||
image;
|
||||
* the description IS the content and the summary (the phase-122
|
||||
image convention — :func:`_store_summary` stores it verbatim),
|
||||
chunked + embedded through :func:`_replace_content_chunks` (one
|
||||
transaction per image); a per-image embed failure rolls back that
|
||||
image only (``errors`` + the error line) and the pass moves on —
|
||||
the PDF's text index and the other images are already committed.
|
||||
* a new/updated image doc counts ``added``/``updated`` (it really
|
||||
changed the KB — the sources-version bump fires on it, like a
|
||||
standalone image's).
|
||||
|
||||
``raw_date`` is the PDF's resolved source date (the pass runs only
|
||||
for (re)indexed PDFs, so it is always concrete) — the image docs
|
||||
inherit the PDF's date (they are part of it).
|
||||
"""
|
||||
settings = llm.settings
|
||||
if not settings.images:
|
||||
# Phase-122 toggle off: embedded images are skipped (debug-
|
||||
# logged) — and the prune guard keeps any existing image docs
|
||||
# (they are invisible to the walk, not deleted files).
|
||||
logger.debug(
|
||||
"import: PDF embedded images skipped (images toggle off) "
|
||||
"source=%s path=%s",
|
||||
source, rel,
|
||||
)
|
||||
return
|
||||
images = extract_pdf_images(full_path)
|
||||
for img in images:
|
||||
await _index_one_embedded_image(
|
||||
session, img, source=source, rel=rel, full_path=full_path,
|
||||
llm=llm, summary=summary, raw_date=raw_date,
|
||||
)
|
||||
|
||||
|
||||
async def _index_one_embedded_image(
|
||||
session: Session,
|
||||
img: PdfImage,
|
||||
*,
|
||||
source: str,
|
||||
rel: str,
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime,
|
||||
) -> None:
|
||||
"""One embedded image of one PDF → the phase-122 image pipeline,
|
||||
with the phase-125 task-03 markdown context scan (parity with the
|
||||
standalone image's :func:`_index_image_file` — see
|
||||
:func:`_index_pdf_embedded_images` for the full contract)."""
|
||||
settings = llm.settings
|
||||
ext = _MIME_TO_EXT.get(img.mime, ".bin")
|
||||
if ext not in settings.image_extension_set:
|
||||
logger.warning(
|
||||
"import: PDF embedded image skipped (unsupported extension) "
|
||||
"source=%s path=%s page=%d index=%d mime=%s",
|
||||
source, rel, img.page, img.index, img.mime,
|
||||
)
|
||||
return
|
||||
image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
|
||||
digest = hashlib.sha256(img.data).hexdigest()
|
||||
image_doc = session.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == image_rel)
|
||||
)
|
||||
if image_doc is not None and image_doc.content_hash == digest:
|
||||
# Unchanged embedded image (byte digest) — no re-describe, no
|
||||
# re-embed (the standalone image's unchanged rule).
|
||||
logger.debug(
|
||||
"import: PDF embedded image unchanged source=%s path=%s",
|
||||
source, image_rel,
|
||||
)
|
||||
return
|
||||
verb = "updated" if image_doc is not None else "added"
|
||||
# Named by the doc id (the phase-122 copy rule): a new image's id is
|
||||
# the uuid4 chosen here (row and copy agree); a changed image keeps
|
||||
# its id (the copy path is stable, the stale copy is deleted below).
|
||||
doc_id = image_doc.id if image_doc is not None else uuid.uuid4()
|
||||
image_dir = Path(settings.image_dir).expanduser()
|
||||
copy_path = image_dir / f"{doc_id}{ext}"
|
||||
# Phase 125 (task 03, LOCKED A1/A2) — the embedded image's parity
|
||||
# with the standalone image's path (phase 126): BEFORE the
|
||||
# description, the markdown reference context for THIS image — a
|
||||
# ```` reference in a
|
||||
# same-source markdown file (the synthetic path is the doc's
|
||||
# ``path``, and the reference resolves to it through the shared
|
||||
# :func:`_resolve_ref_path` rule) — or ``None`` (no reference —
|
||||
# the description call and the row stay byte-identical to the
|
||||
# un-enriched shape). It goes to the vision model (a context-aware
|
||||
# description, LOCKED A3) AND is stored on the doc (``image_context``
|
||||
# — the retrieval display's context line, task 05). A re-indexed
|
||||
# image re-scans: the context follows the current source state
|
||||
# (a deleted referencing markdown drops it back to ``None``).
|
||||
image_context = _markdown_image_context(
|
||||
_source_root_for(full_path, rel), llm, source=source, image_rel=image_rel
|
||||
)
|
||||
# The description through the single seam (the mime resolves from
|
||||
# the copy path's extension — the extension was derived from the
|
||||
# mime, so the round-trip is lossless for the default family). The
|
||||
# description is generated BEFORE anything is written, so a failure
|
||||
# skips the image with no row and no copy.
|
||||
description = await _describe_or_skip(
|
||||
llm, data=img.data, source=source, rel=image_rel, full_path=copy_path,
|
||||
context=image_context,
|
||||
)
|
||||
if description is None:
|
||||
# LOCKED A3 fail-soft: an undescribed image is unsearchable
|
||||
# noise — skip it (no row, no copy).
|
||||
summary.images_failed += 1
|
||||
logger.warning(
|
||||
"import: PDF embedded image description failed source=%s path=%s "
|
||||
"page=%d index=%d",
|
||||
source, image_rel, img.page, img.index,
|
||||
)
|
||||
return
|
||||
if image_doc is not None:
|
||||
# A CHANGED image: the stale copy is deleted before replacement.
|
||||
_delete_image_copy(image_doc.image_path)
|
||||
image_dir.mkdir(parents=True, exist_ok=True)
|
||||
copy_path.write_bytes(img.data)
|
||||
|
||||
# Phase 125 (task 03): the enriched content — the reference context
|
||||
# FIRST (for retrieval), then the description (the standalone image's
|
||||
# rule; ``summary = content`` below keeps the image convention).
|
||||
content = (
|
||||
f"{image_context}\n\n{description}" if image_context is not None else description
|
||||
)
|
||||
|
||||
title = f"Embedded image from {full_path.stem} page {img.page}"
|
||||
if image_doc is None:
|
||||
image_doc = Document(
|
||||
id=doc_id,
|
||||
source=source,
|
||||
path=image_rel,
|
||||
full_path=str(copy_path),
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash=digest,
|
||||
indexed_at=datetime.now(UTC),
|
||||
created_at=normalize_doc_date(raw_date),
|
||||
is_image=True,
|
||||
image_path=str(copy_path),
|
||||
image_context=image_context,
|
||||
)
|
||||
session.add(image_doc)
|
||||
else:
|
||||
image_doc.full_path = str(copy_path)
|
||||
image_doc.title = title
|
||||
image_doc.content = content
|
||||
image_doc.content_hash = digest
|
||||
image_doc.indexed_at = datetime.now(UTC)
|
||||
# Phase 106 (D4): a content change is a new document version —
|
||||
# the date is re-sourced and a manual correction reset.
|
||||
image_doc.created_at = normalize_doc_date(raw_date)
|
||||
image_doc.created_at_manual = False
|
||||
image_doc.is_image = True
|
||||
image_doc.image_path = str(copy_path)
|
||||
# Phase 125 (task 03): a re-indexed image re-scans its markdown
|
||||
# references — the context follows the current source state.
|
||||
image_doc.image_context = image_context
|
||||
|
||||
session.flush() # guarantees doc.id even for brand-new rows
|
||||
|
||||
# Phase 1+2 — the shared pipeline on the description (one
|
||||
# transaction per image; a per-image embed failure rolls back THIS
|
||||
# image only — the PDF doc and the other images are already
|
||||
# committed — and the pass moves on).
|
||||
try:
|
||||
chunks_n = await _replace_content_chunks(
|
||||
session, doc=image_doc, llm=llm, content=content, rel=image_rel
|
||||
)
|
||||
session.commit()
|
||||
except EmbeddingError as e:
|
||||
session.rollback()
|
||||
# The description succeeded (the copy was written), but the
|
||||
# embed gave up: delete the copy so a failure never leaves an
|
||||
# orphan (the phase-122 no-orphan rule).
|
||||
_delete_image_copy(str(copy_path))
|
||||
summary.errors += 1
|
||||
logger.error(
|
||||
"import: PDF embedded image embed failed source=%s path=%s "
|
||||
"page=%d index=%d — %s",
|
||||
source, image_rel, img.page, img.index, e,
|
||||
)
|
||||
return
|
||||
|
||||
if verb == "added":
|
||||
summary.added += 1
|
||||
else:
|
||||
summary.updated += 1
|
||||
summary.chunks += chunks_n
|
||||
logger.info(
|
||||
"import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
|
||||
)
|
||||
# Phase 122 (task 03) shape on the description — image-aware: the
|
||||
# description IS the summary (stored verbatim, no ``lite`` call), so
|
||||
# ``doc.summary`` == ``doc.content`` and the ``is_summary`` position
|
||||
# −1 chunk mirrors it. With a markdown reference (phase 125) the
|
||||
# summary is the ENRICHED content — the same value the standalone
|
||||
# image's path stores.
|
||||
await _store_summary(
|
||||
session, doc=image_doc, source=source, rel=image_rel,
|
||||
content=content, llm=llm, summary=summary,
|
||||
)
|
||||
|
||||
|
||||
def _source_root_for(full_path: Path, rel: str) -> Path:
|
||||
"""The source root of one walked file (phase 125, task 03).
|
||||
|
||||
@@ -1517,6 +2006,12 @@ async def _index_image_file(
|
||||
)
|
||||
|
||||
|
||||
#: The synthetic path marker of a PDF-embedded image doc (phase 126,
|
||||
#: LOCKED A3 — ``<pdf path>#page=<N>#img=<index>.<ext>``):
|
||||
#: :data:`_EMBEDDED_IMAGE_RE` is the prune's parent rule below.
|
||||
_EMBEDDED_IMAGE_RE = re.compile(r"#page=\d+#img=\d+")
|
||||
|
||||
|
||||
def _prune(
|
||||
session: Session,
|
||||
source_names: set[str],
|
||||
@@ -1532,6 +2027,19 @@ def _prune(
|
||||
first images-off sync. Toggle ON → normal semantics: a deleted image
|
||||
file prunes its doc, and the pruned image's ``image_dir`` copy is
|
||||
deleted with it.
|
||||
|
||||
Phase 126 (LOCKED A3) — the PDF-embedded image's parent rule: an
|
||||
embedded image doc's synthetic path (``#page=<N>#img=<index>``)
|
||||
NEVER appears in the walk's *seen* — the image is derived from its
|
||||
parent PDF, whose on-disk file is the walked file. Pruning it on the
|
||||
plain rule would destroy it in the very sync that (re)indexes it,
|
||||
so it is skipped while its parent PDF's rel is still in *seen* (the
|
||||
image is invisible to the walk, not deleted). A pruned/absent parent
|
||||
PDF prunes its embedded image docs with it (row + ``image_dir`` copy
|
||||
— no orphans when the PDF leaves the source). The rule holds in BOTH
|
||||
toggle states (toggle off, the phase-122 guard already skips every
|
||||
``is_image`` doc; this one is the toggle-on half of the same
|
||||
"invisible, not deleted" contract).
|
||||
"""
|
||||
if not source_names:
|
||||
return 0
|
||||
@@ -1542,6 +2050,14 @@ def _prune(
|
||||
if doc.is_image and not images:
|
||||
# Prune guard: invisible to the walk, not deleted.
|
||||
continue
|
||||
if doc.is_image and _EMBEDDED_IMAGE_RE.search(doc.path):
|
||||
# Phase 126 (A3): the parent PDF's rel is the path minus
|
||||
# its ``#page=…#img=…`` fragment — the walked file.
|
||||
parent_rel = doc.path.split("#", 1)[0]
|
||||
if (doc.source, parent_rel) in seen:
|
||||
# Parent PDF still in the source — the image is
|
||||
# invisible to the walk, not deleted.
|
||||
continue
|
||||
_delete_image_copy(doc.image_path)
|
||||
session.delete(doc)
|
||||
pruned += 1
|
||||
|
||||
Reference in New Issue
Block a user