ducoterra
2d0f608246
All green. Final report:
**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)
- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
2184 lines
100 KiB
Python
2184 lines
100 KiB
Python
"""Knowledge-base importer (PLAN §5 / §9 / §11).
|
||
|
||
Walks the in-scope files (the A9 family by default — the original seven
|
||
plus the quadlet family and ``j2`` — ``BOR_IMPORT_EXTENSIONS``, which may
|
||
name any well-formed extension; case-insensitive), diffs by sha256
|
||
against ``documents.content_hash`` and, for every new or changed file, runs
|
||
the two-phase upsert:
|
||
|
||
1. upsert the document row and replace its chunk rows (embeddings NULL)
|
||
2. embed the new chunks in batches and attach the vectors
|
||
3. commit — one transaction per file, so a failed embedding leaves the
|
||
database untouched and the file is simply retried on the next run
|
||
4. every file (phase 30; phase 118, A2: markdown included — the
|
||
non-markdown-only scope is retired): generate a ``lite``-model summary
|
||
and, best-effort, store it on ``documents.summary`` plus one extra
|
||
embedded chunk (``is_summary``, position −1). The document row and its
|
||
content chunks are already committed at this point, so a summary
|
||
failure only means the file is indexed without a summary (logged and
|
||
counted in ``summary_errors``) — it is never lost.
|
||
|
||
Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
|
||
component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
|
||
or hidden files) is skipped, plus the well-known exclusion list — UNLESS
|
||
the source's phase-105 hidden-folders flag admits dot-prefixed paths; the
|
||
exclusion list always applies. A token may also name extensionless files
|
||
by their exact lowercased full filename (``Dockerfile`` under the
|
||
``dockerfile`` token — phase 102).
|
||
|
||
``prune=True`` deletes documents (of the imported sources only) whose files
|
||
no longer exist **or no longer match the format filter** — this is how
|
||
previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
|
||
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
|
||
summary line with per-format counts (PLAN §9). Phase 122 prune guard
|
||
(LOCKED, derived from A3/A4): while the ``images`` toggle is OFF, an
|
||
``is_image`` doc is INVISIBLE to the walk, not a deleted file — prune
|
||
skips it (turning the toggle off and syncing must never destroy image
|
||
documents); a toggle-ON run prunes a deleted image file normally and
|
||
deletes its ``image_dir`` copy with the row.
|
||
|
||
Document dates (phase 106, D2/D4): every import sources
|
||
``documents.created_at`` from the file's source — the per-file git
|
||
last-commit date when a ``doc_dates_by_root`` entry names the file,
|
||
else the file's mtime — normalized by
|
||
:func:`app.rag.doc_dates.normalize_doc_date` (undetermined or future →
|
||
today, D3) on every add and update. On the unchanged path the stored
|
||
date is REFRESHED from the same source (it may go OLDER — no monotonic
|
||
guard) and counted in ``summary.dates_updated`` — unless the row
|
||
carries the owner's manual correction (``created_at_manual``, D1), which
|
||
the sync never touches.
|
||
|
||
NULL-summary backfill (phase 118, A2): an UNCHANGED file (same
|
||
``content_hash``) whose ``documents.summary`` is still NULL — a
|
||
pre-phase-30 row, or an earlier fail-soft miss — gets the same
|
||
best-effort summary pass on every sync until it sticks. A success counts
|
||
``summary_backfilled`` (never ``summaries``) and touches nothing else: no
|
||
content re-embed, no added/updated/pruned count — so no
|
||
``sources_meta`` bump, no KB-overview/folder-summary regeneration. The
|
||
backfill runs BEFORE the ``created_at_manual`` early-return (the manual
|
||
flag protects the DATE only, D1) and the strict ``is None`` check leaves
|
||
owner-set summaries (even empty strings, phase 57) alone.
|
||
|
||
Standalone images (phase 122, LOCKED A3): with the ``images`` toggle
|
||
(``BOR_IMAGES``) ON, the walk also admits the image extension set
|
||
(``BOR_IMAGE_EXTENSIONS`` — a SEPARATE set from ``import_extension_set``;
|
||
images are never user-added via ``BOR_IMPORT_EXTENSIONS``, the toggle is
|
||
the single knob). Such a file takes the binary index path
|
||
(:func:`_index_image_file`): the sha256 digest is over the raw BYTES
|
||
(content identity — the digest rule is unchanged), the bytes are copied
|
||
to the persistent home ``settings.image_dir/<doc-id>.<ext>`` (dir created
|
||
on demand; the copy is written ONLY after a successful description, so a
|
||
failure never leaves an orphan; a changed image deletes the stale copy
|
||
first; a pruned image doc deletes its copy), the row carries
|
||
``is_image=True`` + ``image_path``, and ``content`` is the vision
|
||
description — the ONLY embedded text of the document (the embedding model
|
||
never sees pixels; ``read_text`` is never called for an image). The
|
||
description comes through the single seam :func:`_describe_or_skip`
|
||
(task 03: :func:`app.rag.summarizer.describe_image` — ONE CHAT-model
|
||
(vision) call with the image bytes as a base64 data URL; the ``lite``
|
||
summary model is NOT assumed vision-capable, LOCKED A3); a failed/empty
|
||
description SKIPS the doc entirely (no row, no copy) — counted in
|
||
``images_failed`` + a warning, the sync continues (fail-soft). The normal
|
||
chunk pipeline then embeds ``content`` and the phase-30 summary path runs
|
||
on it — image-aware (task 03): for an image doc the description IS the
|
||
summary (stored verbatim, no ``lite`` call, no pointer line), so the
|
||
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
|
||
equals ``Document.content``.
|
||
|
||
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
|
||
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
|
||
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
|
||
AFTER the image branch, BEFORE any text read — a PDF is never
|
||
``read_text``-ed; the binary is owned by the phase-126 parser). The
|
||
document's ``content`` is the extracted text (every text-bearing page
|
||
prefixed with the ``---PAGE-N---`` marker — A2, one document per PDF),
|
||
the digest is sha256 over that text (the digest rule is unchanged —
|
||
content identity), the title is the PDF's embedded metadata title or
|
||
the file stem, and the page-aware chunker (task 03) chunks it with a
|
||
``Page N:`` anchor. With the phase-122 ``images`` toggle ON, the
|
||
embedded images join the image pipeline (A3, :func:
|
||
`_index_pdf_embedded_images`): each is described through the SAME
|
||
``_describe_or_skip`` seam (the bytes as a data URL), copied to
|
||
``settings.image_dir`` (the copy is written only AFTER a successful
|
||
description — a failure never leaves an orphan), and indexed as a
|
||
standalone ``is_image`` document under the synthetic path
|
||
``<pdf path>#page=<N>#img=<index>.<ext>`` (the description IS the
|
||
content and summary — the phase-122 convention). Toggle OFF → the
|
||
embedded images are skipped (debug-logged) and the prune guard keeps
|
||
any existing image docs; an unsupported image mime (outside
|
||
``BOR_IMAGE_EXTENSIONS``) is skipped with a warning. A PDF's viewer
|
||
presentation is its extracted text (A4 — nothing binary is served).
|
||
|
||
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
|
||
description, the same source's importable markdown files are scanned for
|
||
```` references that name the image (the task-02
|
||
:func:`app.rag.md_images.extract_image_refs`); when found, the combined
|
||
context string (``Referenced in <source>/<md>:`` + alt text + surrounding
|
||
paragraphs, see :func:`_markdown_image_context`) is (a) passed to the
|
||
vision call as ``context`` (appended AFTER the locked describe prompt —
|
||
the model learns what the image is for) and (b) stored on the doc's
|
||
``image_context`` (the retrieval display's context line, task 05), and
|
||
the stored ``content`` is ``<context>\n\n<description>`` (context first
|
||
for retrieval; ``summary`` mirrors it). A standalone image (no markdown
|
||
reference) gets ``image_context = NULL`` and the bare description —
|
||
byte-identical to pre-phase-125.
|
||
|
||
Markdown-FIRST image context (phase 125, task 04, LOCKED A1): a
|
||
markdown file (re)indexed while a reference's image has NO document in
|
||
the source yet records those references on ITS OWN ``image_context``
|
||
(the ``IMAGE_CONTEXT:`` record — :func:`_format_missing_image_refs`;
|
||
references are resolved exactly as the task-03 scan, and a reference
|
||
escaping the source is not recorded). A reference that DOES name an
|
||
existing image doc records nothing (the task-03 path owns that case —
|
||
enrichment at image-index time — and the markdown doc's field stays
|
||
NULL). After ALL of a source's files are indexed, the post-index
|
||
enrichment pass (:func:`_enrich_pending_images`) enriches the still-
|
||
``image_context IS NULL`` image docs that a markdown doc's record
|
||
references — the case where the markdown was indexed first and its
|
||
on-disk file is gone by the time the image is indexed (the task-03
|
||
disk scan cannot see it) — with the task-03 mechanics (re-describe
|
||
with context, ``content = <context>\n\n<description>``, the context
|
||
stored, re-chunk + re-embed). The pass is idempotent (an image doc
|
||
with ``image_context`` set is never touched; a markdown doc's record
|
||
is NOT cleared on success — it is the retrieval display's record
|
||
also, task 05).
|
||
|
||
``import_sources`` accepts an optional per-file ``progress`` callback
|
||
(phase 64, task 01) reporting the file being processed right now, and
|
||
an optional ``on_chunk_progress`` callback (phase 129, task 02)
|
||
reporting the in-flight file's embedding-chunk position at batch
|
||
granularity (see the function docstring).
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import hashlib
|
||
import logging
|
||
import posixpath
|
||
import re
|
||
import uuid
|
||
from collections.abc import Callable, Sequence
|
||
from dataclasses import dataclass, field
|
||
from datetime import UTC, datetime
|
||
from pathlib import Path, PurePosixPath
|
||
from typing import Any, Protocol
|
||
|
||
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
|
||
from sqlalchemy import select
|
||
from sqlalchemy.orm import Session
|
||
|
||
from app.config import Settings
|
||
from app.db import SessionLocal
|
||
from app.models import Chunk, Document
|
||
from app.rag.chunker import chunk_document, extract_title
|
||
from app.rag.doc_dates import file_mtime_datetime, normalize_doc_date
|
||
from app.rag.llm import EmbeddingError, LLMError
|
||
from app.rag.md_images import ImageRef, extract_image_refs
|
||
from app.rag.pdf import PdfImage, extract_pdf_images, extract_pdf_text
|
||
from app.rag.summarizer import (
|
||
IMAGE_FALLBACK_MIME,
|
||
IMAGE_MIMES,
|
||
describe_image,
|
||
generate_summary,
|
||
)
|
||
|
||
logger = logging.getLogger("app.importer")
|
||
|
||
#: Non-content directories never imported (PLAN anchor A9).
|
||
EXCLUDED_DIRS: frozenset[str] = frozenset(
|
||
{".venv", "node_modules", ".git", "__pycache__", ".pytest_cache", "dist", "build"}
|
||
)
|
||
|
||
#: Markdown extensions that can carry ```` image references
|
||
#: (phase 125, task 03) — the two markdown tokens of the A9 family. The
|
||
#: scan (see :func:`_markdown_image_context`) only reads these, and only
|
||
#: among the source's importable files.
|
||
_MARKDOWN_EXTENSIONS: frozenset[str] = frozenset({".md", ".markdown"})
|
||
|
||
|
||
class Embedder(Protocol):
|
||
"""Everything the importer needs from the LLM client (duck-typed for tests)."""
|
||
|
||
settings: Settings
|
||
embed_batches: int
|
||
|
||
async def embed(
|
||
self,
|
||
texts: list[str],
|
||
*,
|
||
# Phase 129 (task 01): the per-batch position hook — ``None``
|
||
# (every pre-phase caller) is a zero-cost guarded no-op.
|
||
on_progress: Callable[[int, int], None] | None = None,
|
||
) -> list[list[float]]: ...
|
||
|
||
async def chat(self, messages: list[dict[str, Any]], model: str | None = None) -> str: ...
|
||
# ^ the one-shot completion the summarizer uses — the ``lite`` model
|
||
# for text summaries (phase 30, task 01) and the CHAT (vision)
|
||
# model for the phase-122 image description (multimodal content:
|
||
# a string or a list of OpenAI-compatible parts); :class:`app.rag.
|
||
# llm.LLMClient` satisfies it.
|
||
|
||
|
||
@dataclass
|
||
class ImportSummary:
|
||
"""Counts for one import run (also printed by the CLI)."""
|
||
|
||
files: int = 0
|
||
added: int = 0
|
||
updated: int = 0
|
||
unchanged: int = 0
|
||
pruned: int = 0
|
||
errors: int = 0
|
||
chunks: int = 0
|
||
embed_batches: int = 0
|
||
#: Files whose lite summary was generated + indexed (phase 30; phase
|
||
#: 118, A2: every A9 format, markdown included). One ``is_summary``
|
||
#: chunk per success.
|
||
summaries: int = 0
|
||
#: Files whose summary generation failed (best-effort — the document
|
||
#: is still indexed, without a summary).
|
||
summary_errors: int = 0
|
||
#: Unchanged docs whose NULL summary was backfilled (phase 118, A2) —
|
||
#: one ``is_summary`` chunk per success; the content is untouched, so
|
||
#: a backfill NEVER counts added/updated/pruned (no
|
||
#: ``sources_meta`` bump, no overview/folder-summary regeneration).
|
||
summary_backfilled: int = 0
|
||
#: Files whose ``created_at`` was refreshed on the UNCHANGED path —
|
||
#: content untouched, date re-sourced (phase 106, D4: the date may
|
||
#: go OLDER; a date-only refresh NEVER counts added/updated/pruned,
|
||
#: so no ``sources_meta`` bump, no overview/folder-summary
|
||
#: regeneration).
|
||
dates_updated: int = 0
|
||
#: Image docs (phase 122, LOCKED A3) whose vision description failed
|
||
#: or came back empty — the doc is SKIPPED entirely (no row, no
|
||
#: ``image_dir`` copy): an undescribed image is unsearchable noise.
|
||
#: Fail-soft: the sync continues, this counter + the warning line
|
||
#: are the signal.
|
||
images_failed: int = 0
|
||
#: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
|
||
#: tuples in encounter order (added/updated in walk order, pruned in
|
||
#: _prune order). Drives the sync's incremental folder-summary
|
||
#: regeneration (phase 127): len(changed_paths) == added + updated +
|
||
#: pruned. Unchanged / backfilled / date-only refreshes NEVER record.
|
||
changed_paths: list[tuple[str, str]] = field(default_factory=list)
|
||
#: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
|
||
formats: dict[str, int] = field(default_factory=dict)
|
||
|
||
def format_counts(self) -> str:
|
||
"""``md:203,yaml:267,py:14`` — highest count first (PLAN §9)."""
|
||
if not self.formats:
|
||
return "none"
|
||
ordered = sorted(self.formats.items(), key=lambda kv: (-kv[1], kv[0]))
|
||
return ",".join(f"{ext}:{count}" for ext, count in ordered)
|
||
|
||
def log(self) -> None:
|
||
logger.info(
|
||
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
|
||
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
|
||
"summary_backfilled=%d dates_updated=%d images_failed=%d formats=%s",
|
||
self.files,
|
||
self.added,
|
||
self.updated,
|
||
self.unchanged,
|
||
self.pruned,
|
||
self.errors,
|
||
self.chunks,
|
||
self.embed_batches,
|
||
self.summaries,
|
||
self.summary_errors,
|
||
self.summary_backfilled,
|
||
self.dates_updated,
|
||
self.images_failed,
|
||
self.format_counts(),
|
||
)
|
||
|
||
|
||
def normalize_ignore_path(entry: str) -> str:
|
||
"""One ignore-path entry → canonical form (phase 89, A1).
|
||
|
||
Trim surrounding whitespace, then strip ALL leading/trailing
|
||
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
|
||
all become ``"my/files"``. ``""`` / ``"//"``,
|
||
``" "`` normalize to ``""`` (callers drop empties).
|
||
"""
|
||
return entry.strip().strip("/")
|
||
|
||
|
||
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
|
||
"""Phase 89, A1 — the pure prefix rule, nothing else.
|
||
|
||
``rel`` is the source-relative POSIX path WITHOUT a leading
|
||
slash (the ``documents.path`` string). Match = ``rel`` STARTS
|
||
WITH a normalized entry: raw string prefix — deliberately NO
|
||
component-boundary check (``"my/files"`` also matches
|
||
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
|
||
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
|
||
"""
|
||
return any(rel.startswith(p) for p in prefixes)
|
||
|
||
|
||
def _ignore_for_root(
|
||
root: Path, ignore_by_root: dict[str, list[str]] | None
|
||
) -> tuple[str, ...]:
|
||
"""The normalized, non-empty prefix tuple for one root (phase 89).
|
||
|
||
Keyed by ``str(root)`` — the root string exactly as the caller
|
||
passed it in ``sources`` (unambiguous when two rows share a
|
||
source *name* but different dirs). Callers may pass RAW box
|
||
lines: the importer normalizes + drops empties here, the single
|
||
choke point — stored lists (already normalized) normalize to
|
||
themselves.
|
||
"""
|
||
raw = (ignore_by_root or {}).get(str(root)) or []
|
||
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
|
||
|
||
|
||
def _include_hidden_for_root(
|
||
root: Path, include_hidden_by_root: dict[str, bool] | None
|
||
) -> bool:
|
||
"""The per-root hidden-folders flag (phase 105, A1).
|
||
|
||
Keyed by ``str(root)`` — the root string exactly as the caller
|
||
passed it in ``sources`` (the ``_ignore_for_root`` convention,
|
||
phase 89): ``True`` only for roots the caller lists as True;
|
||
unlisted/``None`` roots are ``False`` — every existing caller
|
||
behaves byte-identically (A4).
|
||
"""
|
||
return bool((include_hidden_by_root or {}).get(str(root), False))
|
||
|
||
|
||
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
|
||
"""The bare lowercased token *path* imports under, or ``None``.
|
||
|
||
1. Non-empty lowercased dotted suffix in *extensions* (the A9 rule —
|
||
``kubernetes.md`` → ``md``).
|
||
2. No suffix: the lowercased FULL filename equals a bare token of
|
||
*extensions* (``Dockerfile`` → ``dockerfile``) — the phase-102
|
||
extensionless rule. Exact name only: ``mydockerfile`` never
|
||
matches the ``dockerfile`` token.
|
||
"""
|
||
suffix = path.suffix.lower()
|
||
if suffix and suffix in extensions:
|
||
return suffix.lstrip(".")
|
||
if not suffix and path.name.lower() in {e.lstrip(".") for e in extensions}:
|
||
return path.name.lower()
|
||
return None
|
||
|
||
|
||
def iter_importable_files(
|
||
root: Path,
|
||
extensions: frozenset[str],
|
||
excluded: frozenset[str] = EXCLUDED_DIRS,
|
||
ignore: tuple[str, ...] = (),
|
||
include_hidden: bool = False,
|
||
image_extensions: frozenset[str] = frozenset(),
|
||
) -> list[Path]:
|
||
"""All importable files under *root* (sorted), per the A9 scope rules.
|
||
|
||
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
|
||
Skips: when ``include_hidden`` is False (the default), any path with a
|
||
dot-prefixed component (hidden dirs/files — vendored caches like
|
||
``.esphome/.espressif/**``); when True, dot-prefixed components are
|
||
ADMITTED (files inside hidden folders, and hidden files) and only
|
||
*excluded* is consulted (A1 — caches/VCS internals are never content).
|
||
The well-known non-content directories in *excluded* are skipped in
|
||
BOTH states. *ignore* (phase 89, A1) is a
|
||
tuple of ALREADY-normalized, non-empty source-relative path prefixes
|
||
(the importer's ``_ignore_for_root`` is the normalization choke point
|
||
— raw box lines never reach this function): a file is skipped when its
|
||
source-relative POSIX path starts with any entry; the default ``()``
|
||
keeps every existing caller byte-identical. The *ignore* tuple
|
||
composes additively in both states.
|
||
|
||
*image_extensions* (phase 122) is the lowercased dotted
|
||
image-extension set admitted IN ADDITION to *extensions* — passed by
|
||
:func:`import_sources` only while the ``images`` toggle is on (it
|
||
reads ``llm.settings``; the image set is never merged into
|
||
*extensions*). The empty default admits nothing: every existing caller
|
||
(and the toggle-off walk) stays byte-identical to pre-phase.
|
||
"""
|
||
if not root.is_dir():
|
||
return []
|
||
files: list[Path] = []
|
||
for path in sorted(root.rglob("*")):
|
||
if not path.is_file():
|
||
continue
|
||
rel = path.relative_to(root)
|
||
if any(
|
||
(not include_hidden and part.startswith(".")) or part in excluded
|
||
for part in rel.parts
|
||
):
|
||
continue
|
||
if ignore and is_ignored(rel.as_posix(), ignore):
|
||
continue
|
||
matched = match_extension(path, extensions)
|
||
if matched is None and image_extensions:
|
||
# Phase 122: the image set is admitted IN ADDITION to the
|
||
# import set (toggle on only — the caller passes it in).
|
||
matched = match_extension(path, image_extensions)
|
||
if matched is None:
|
||
continue
|
||
files.append(path)
|
||
return files
|
||
|
||
|
||
async def import_sources(
|
||
sources: list[Path],
|
||
llm: Embedder,
|
||
*,
|
||
prune: bool = False,
|
||
limit: int | None = None,
|
||
session: Session | None = None,
|
||
progress: Callable[[str, str, int, int], None] | None = None,
|
||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||
ignore_by_root: dict[str, list[str]] | None = None,
|
||
include_hidden_by_root: dict[str, bool] | None = None,
|
||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||
) -> ImportSummary:
|
||
"""Import every A9-format file under *sources* (see module docstring).
|
||
|
||
``session`` may be supplied (tests); a private one is opened and closed
|
||
otherwise. ``limit`` caps the number of files processed (debug only) and
|
||
disables pruning, since an incomplete walk must not drive deletions.
|
||
|
||
``progress`` (phase 64, task 01) is an optional per-file hook called
|
||
once per importable file, immediately before that file's
|
||
``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
|
||
same POSIX *rel* the document rows use, ``done`` = the 1-based index of
|
||
the current file **across all sources**, and ``total`` = the combined
|
||
pre-walk count of importable files across all *sources* roots. The
|
||
pre-walk (same extension/exclusion rules, directory stats only, no file
|
||
reads) happens **only when *progress* is provided**: callers passing
|
||
nothing pay no extra walk and behave exactly as before. Under
|
||
``limit``, the hook still fires per processed file only — ``done`` never
|
||
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||
incomplete walk must not misreport the denominator).
|
||
|
||
``on_chunk_progress`` (phase 129, task 02) is an optional
|
||
chunk-position hook for the file currently being indexed: while that
|
||
file's content chunks embed, it is called — through
|
||
:meth:`app.rag.llm.LLMClient.embed`'s ``on_progress`` — before EACH
|
||
embed batch with ``(first_chunk_1based, total_chunks)``: the 1-based
|
||
position of the first chunk of the in-flight batch and the file's
|
||
total content-chunk count (batch granularity, the phase-129 A6
|
||
assumption — the transport embeds token-budgeted batches per HTTP
|
||
call). It carries the in-flight FILE's chunk position only, never
|
||
the file identity — the per-file ``progress`` hook owns that: the
|
||
caller pairs them (a ``progress`` call names the file, and every
|
||
chunk position reported until the next ``progress`` call belongs to
|
||
it; the position climbs through the file's batches and restarts at
|
||
1 for the next file). Same "only when provided" rule as
|
||
``progress``: ``None`` (every existing caller — the CLI, the chat
|
||
path) is a zero-cost guarded no-op, and the run is byte-identical
|
||
to a hook-less run. Only the content-chunk embeds report (the
|
||
shared :func:`_replace_content_chunks` — text, image, and PDF
|
||
branches; the image branch's one description chunk reports
|
||
``1/1``, which IS its only chunk): the document's summary-chunk
|
||
embed (:func:`_store_summary`) NEVER receives the hook, so the
|
||
counter counts content chunks only (no ``1/1`` noise after the
|
||
real progress ends), and the post-index enrichment pass / the
|
||
PDF's embedded-image docs (separate documents, not the walked
|
||
file) do not report either.
|
||
|
||
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
|
||
string exactly as passed in *sources* — to that source's RAW ignore-path
|
||
lines (the importer normalizes them via ``_ignore_for_root``, the single
|
||
choke point): matching files are never walked, so they are never
|
||
embedded and never summarized, and the progress pre-walk uses the same
|
||
per-root tuple as the processing loop, so ``total`` never counts them.
|
||
A file that newly matches a pattern simply never enters ``seen``, so the
|
||
next ``prune=True`` run deletes its row automatically (A2 — the A9
|
||
junk-precedent). ``None`` (the default) changes nothing: the map is read
|
||
per root, unlisted roots get an empty tuple, and every existing caller
|
||
behaves byte-identically.
|
||
|
||
``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the
|
||
stored flag: ``True`` admits dot-prefixed components for that root
|
||
(``EXCLUDED_DIRS`` and the extension filter still apply; the ignore
|
||
tuple composes additively). Unlisted/``None`` roots are ``False`` —
|
||
byte-identical to pre-phase-105. A file that was indexed with the
|
||
flag ON and is walked again with it OFF simply never enters
|
||
``seen``, so the next ``prune=True`` run deletes its row
|
||
automatically (A2 — the A9/phase-89 precedent).
|
||
|
||
``doc_dates_by_root`` (phase 106, D2/D4) maps ``str(root)`` — the
|
||
root path string exactly as passed in *sources* — to that source's
|
||
RAW per-file source dates: source-relative POSIX path → the git
|
||
last-commit datetime (task 03's ``file_commit_dates``). ONLY git
|
||
roots are listed — unlisted roots (local dirs, unpacked uploads)
|
||
take the mtime fallback, and a path missing from its root's map
|
||
does too. The map entry beats the file's mtime when present. The
|
||
progress pre-walk is untouched (dates change no file count).
|
||
``None`` (the default) changes nothing for existing callers: the
|
||
mtime fallback applies to every file — which IS the behavior
|
||
change, D4: an unchanged file now refreshes its stored date from
|
||
its source on every run (the backfill-correction case).
|
||
|
||
Images (phase 122): when ``llm.settings.images`` is on, BOTH walks
|
||
(the progress pre-walk and the processing loop — same rules, so
|
||
``total`` counts images) also admit ``llm.settings.image_extension_set``
|
||
files, each indexed through the binary image path (see the module
|
||
docstring). ``prune=True`` with the toggle ON prunes a deleted image
|
||
file normally (row + ``image_dir`` copy); with the toggle OFF the
|
||
prune skips ``is_image`` docs (the prune guard — the image is
|
||
invisible to the walk, not a deleted file).
|
||
|
||
Markdown-FIRST image context (phase 125, task 04): after ALL of a
|
||
source's files are indexed, the post-index enrichment pass
|
||
(:func:`_enrich_pending_images`) runs for that source — it enriches
|
||
the still-unenriched image docs (``image_context IS NULL``) that a
|
||
markdown doc's ``IMAGE_CONTEXT:`` record references (the case where
|
||
the markdown was indexed first and its on-disk file is gone by the
|
||
time the image is indexed — the task-03 disk scan cannot see it).
|
||
The pass is driven by committed doc rows, so it is safe under
|
||
``limit`` too (it deletes nothing), and a per-image failure
|
||
(missing copy, failed re-description, embed error) keeps that image
|
||
as-is and moves on.
|
||
"""
|
||
if limit is not None and limit <= 0:
|
||
raise ValueError("limit must be >= 1")
|
||
summary = ImportSummary()
|
||
# Phase 122: the image set is admitted by the walks ONLY while the
|
||
# toggle is on — the empty set admits nothing, so the toggle-off run
|
||
# (walk, counts, prune) stays byte-identical to pre-phase.
|
||
image_exts = llm.settings.image_extension_set if llm.settings.images else frozenset()
|
||
owns_session = session is None
|
||
if session is None:
|
||
session = SessionLocal()
|
||
seen: set[tuple[str, str]] = set()
|
||
source_names: set[str] = set()
|
||
# phase 64 (task 01): the hook's combined denominator, walked with the
|
||
# exact same rules as the processing loop below (directory stats only,
|
||
# no file reads). Skipped entirely for ``progress=None`` callers — no
|
||
# extra pass, byte-identical behaviour and cost.
|
||
total = 0
|
||
if progress is not None:
|
||
for root in sources:
|
||
total += len(
|
||
iter_importable_files(
|
||
root,
|
||
llm.settings.import_extension_set,
|
||
ignore=_ignore_for_root(root, ignore_by_root),
|
||
include_hidden=_include_hidden_for_root(
|
||
root, include_hidden_by_root
|
||
),
|
||
image_extensions=image_exts,
|
||
)
|
||
)
|
||
try:
|
||
for root in sources:
|
||
if not root.is_dir():
|
||
logger.warning("import: source dir not found, skipping: %s", root)
|
||
continue
|
||
if limit is not None and summary.files >= limit:
|
||
break
|
||
source = root.name
|
||
source_names.add(source)
|
||
ignore = _ignore_for_root(root, ignore_by_root)
|
||
include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
|
||
# Phase 106 (D2): the root's raw source dates (git last-commit
|
||
# for git roots, keyed by the same str(root) convention); {}
|
||
# for unlisted roots — every file then takes the mtime fallback.
|
||
dates_map = (doc_dates_by_root or {}).get(str(root), {})
|
||
for path in iter_importable_files(
|
||
root,
|
||
llm.settings.import_extension_set,
|
||
ignore=ignore,
|
||
include_hidden=include_hidden,
|
||
image_extensions=image_exts,
|
||
):
|
||
if limit is not None and summary.files >= limit:
|
||
break
|
||
rel = path.relative_to(root).as_posix()
|
||
seen.add((source, rel))
|
||
summary.files += 1
|
||
# Phase 102: the matched bare token (``dockerfile`` for an
|
||
# extensionless ``Dockerfile``), never ``unknown`` — the
|
||
# file is in scope, so the walk matched it. Phase 122: an
|
||
# image file matches the image set, not the import set.
|
||
ext = (
|
||
match_extension(path, llm.settings.import_extension_set)
|
||
or match_extension(path, image_exts)
|
||
or "unknown"
|
||
)
|
||
summary.formats[ext] = summary.formats.get(ext, 0) + 1
|
||
if progress is not None:
|
||
# phase 64: report the file *before* indexing it — a
|
||
# file that then errors or turns out unchanged was
|
||
# already the "current file". No try/except around the
|
||
# call: the hooks in this repo only assign fields.
|
||
progress(source, rel, summary.files, total)
|
||
try:
|
||
await _index_file(
|
||
session, source=source, rel=rel, full_path=path, llm=llm,
|
||
summary=summary, raw_date=dates_map.get(rel),
|
||
on_chunk=on_chunk_progress,
|
||
)
|
||
except EmbeddingError as e:
|
||
# A pathological file (e.g. content the embedding endpoint
|
||
# refuses) must not abort the whole KB: roll back its
|
||
# uncommitted rows, log loudly, and keep going. The next
|
||
# run retries it.
|
||
session.rollback()
|
||
summary.errors += 1
|
||
logger.error("import: error source=%s path=%s — %s", source, rel, e)
|
||
# Phase 125 (task 04): the post-index enrichment pass — the
|
||
# markdown-FIRST case (a markdown indexed while its
|
||
# referenced image had no document yet recorded the
|
||
# reference on its doc row; the image was just indexed, but
|
||
# the task-03 disk scan could not see the reference — the
|
||
# markdown file is no longer on disk). Idempotent +
|
||
# fail-soft (see the function).
|
||
await _enrich_pending_images(session, source=source, llm=llm, summary=summary)
|
||
if prune:
|
||
if limit is not None:
|
||
logger.warning("import: --prune ignored because --limit was given")
|
||
else:
|
||
summary.pruned = _prune(
|
||
session, source_names, seen, images=llm.settings.images,
|
||
summary=summary,
|
||
)
|
||
summary.embed_batches = llm.embed_batches
|
||
summary.log()
|
||
return summary
|
||
finally:
|
||
if owns_session:
|
||
session.close()
|
||
|
||
|
||
async def _replace_content_chunks(
|
||
session: Session,
|
||
*,
|
||
doc: Document,
|
||
llm: Embedder,
|
||
content: str,
|
||
rel: str,
|
||
on_chunk: Callable[[int, int], None] | None = None,
|
||
) -> int:
|
||
"""Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace
|
||
the doc's content chunk rows (embeddings NULL), and embed the new
|
||
chunks in batches — returning the chunk count. The caller owns the
|
||
transaction (it commits, or rolls back on the :class:`EmbeddingError`
|
||
that propagates when the token-cap retry gives up).
|
||
|
||
The endpoint token-cap retry (halve this file's chunk target, floor
|
||
400 — URL-dense paragraphs tokenize at ~1.1 chars/token; the global
|
||
policy stays intact for the rest of the KB) is the house rule, shared
|
||
verbatim by the text path (:func:`_index_file`), the image path
|
||
(task 03), and the task-04 enrichment pass — one copy, three call
|
||
sites.
|
||
|
||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||
chunk-position hook, forwarded as ``on_progress`` to the content
|
||
embed: it fires before each top-level embed batch with
|
||
``(first_chunk_1based, total_chunks)``. The token-cap re-chunk retry
|
||
(the ``while True`` above that halves *target*) re-embeds the NEW
|
||
(smaller-target) chunk list, and the hook reports against the NEW
|
||
total automatically — no special case: the announced total is
|
||
always the chunk list actually in flight. The empty-content
|
||
``doc.chunks = []`` break never embeds, so the hook never fires for
|
||
it. ``None`` (the default; every pre-phase call site — the CLI, the
|
||
phase-125 enrichment pass, and the PDF's embedded-image docs, which
|
||
index separate documents rather than the walked file) keeps the call
|
||
byte-identical.
|
||
"""
|
||
settings = llm.settings
|
||
target = max(400, settings.chunk_target_chars)
|
||
chunks_text: list[str] = []
|
||
while True:
|
||
chunks_text = chunk_document(content, rel, target, settings.chunk_overlap_chars)
|
||
doc.chunks = [
|
||
Chunk(document_id=doc.id, position=i, content=c) for i, c in enumerate(chunks_text)
|
||
]
|
||
session.flush() # delete-orphan cascade drops the previous rows
|
||
if not doc.chunks:
|
||
break
|
||
try:
|
||
# Phase 129 (task 02): the in-flight file's chunk-position
|
||
# hook (``on_progress`` — the task-01 contract; ``None`` is a
|
||
# zero-cost no-op).
|
||
vectors = await llm.embed(
|
||
[c.content for c in doc.chunks], on_progress=on_chunk
|
||
)
|
||
for row, vec in zip(doc.chunks, vectors, strict=True):
|
||
row.embedding = vec
|
||
break
|
||
except EmbeddingError as e:
|
||
if "token cap" not in str(e) or target <= 400:
|
||
raise
|
||
logger.info(
|
||
"import: re-chunking at %d chars after endpoint token cap: %s",
|
||
target // 2,
|
||
rel,
|
||
)
|
||
target //= 2
|
||
return len(chunks_text)
|
||
|
||
|
||
async def _index_file(
|
||
session: Session,
|
||
*,
|
||
source: str,
|
||
rel: str,
|
||
full_path: Path,
|
||
llm: Embedder,
|
||
summary: ImportSummary,
|
||
raw_date: datetime | None = None,
|
||
on_chunk: Callable[[int, int], None] | None = None,
|
||
) -> None:
|
||
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
|
||
|
||
``raw_date`` (phase 106, D2) is the file's RAW source date — the
|
||
git last-commit datetime from the caller's ``doc_dates_by_root``
|
||
map, or ``None`` (every non-git case): the file's mtime is read
|
||
here, once, and becomes the source date (the D2 fallback).
|
||
|
||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||
chunk-position hook (see :func:`import_sources`) — passed through to
|
||
ALL THREE branches' content embeds: the text path's
|
||
:func:`_replace_content_chunks` directly, the image branch via
|
||
:func:`_index_image_file` (its one description chunk reports
|
||
``1/1`` — it IS the file's only chunk), and the PDF branch via
|
||
:func:`_index_pdf_file` (its content embed is the primary target).
|
||
``None`` (the default) keeps every branch byte-identical.
|
||
|
||
Image files (phase 122, toggle on) delegate to
|
||
:func:`_index_image_file` — the binary path (bytes digest,
|
||
persistent copy, ``content`` = the vision description) — BEFORE
|
||
any text read: ``read_text`` is never called for an image.
|
||
"""
|
||
settings = llm.settings
|
||
# Phase 122 (task 02): the image branch FIRST. Only reachable while
|
||
# the ``images`` toggle is on — the walk never admits image files
|
||
# while it is off, and with it off this check is a no-op (the text
|
||
# path below stays byte-identical to pre-phase).
|
||
if settings.images:
|
||
image_set = settings.image_extension_set
|
||
if match_extension(full_path, image_set) is not None:
|
||
return await _index_image_file(
|
||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
|
||
)
|
||
# Phase 126: PDF files take the PDF index path — AFTER the image
|
||
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
|
||
# owns it and ``read_text`` is never called for one).
|
||
if full_path.suffix.lower() == ".pdf":
|
||
return await _index_pdf_file(
|
||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
|
||
)
|
||
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||
if raw_date is None:
|
||
# D2 fallback: no source date in the map → the file's mtime
|
||
# (one stat). Read before the unchanged early-return — the
|
||
# unchanged path refreshes the stored date from the same source.
|
||
raw_date = file_mtime_datetime(full_path)
|
||
if doc is not None and doc.content_hash == digest:
|
||
summary.unchanged += 1
|
||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||
# Phase 118 (A2): an unchanged doc whose summary is still NULL
|
||
# (a pre-phase-30 row, or an earlier fail-soft miss) gets a
|
||
# summary-only backfill — one ``is_summary`` chunk, no content
|
||
# re-embed, and NEVER an added/updated/pruned count (so no
|
||
# ``sources_meta`` bump, no overview/folder-summary
|
||
# regeneration). Strict ``is None``: an empty-string summary is
|
||
# owner-set (phase 57) and is never overwritten. BEFORE the
|
||
# manual-date early-return: ``created_at_manual`` protects the
|
||
# DATE only (phase 106, D1), not the summary.
|
||
if doc.summary is None:
|
||
await _store_summary(
|
||
session, doc=doc, source=source, rel=rel, content=content,
|
||
llm=llm, summary=summary, backfill=True,
|
||
)
|
||
if doc.created_at_manual:
|
||
# D1/D4: the owner's correction survives the sync — no
|
||
# write at all (the phase-97 ``manually_edited`` precedent).
|
||
return
|
||
# D4: the date refreshes on every sync, including unchanged
|
||
# files, and may go OLDER (no monotonic guard). A date-only
|
||
# refresh is still counted ``unchanged`` — never added/updated/
|
||
# pruned, so no ``sources_meta`` bump and no regeneration.
|
||
target = normalize_doc_date(raw_date)
|
||
if target != doc.created_at:
|
||
doc.created_at = target
|
||
session.commit()
|
||
summary.dates_updated += 1
|
||
logger.info(
|
||
"import: date-refreshed source=%s path=%s date=%s",
|
||
source, rel,
|
||
doc.created_at.isoformat(),
|
||
)
|
||
return
|
||
|
||
verb = "updated" if doc is not None else "added"
|
||
# A ``#`` line is a real heading in markdown but a comment in every
|
||
# other format — titles for those come from the file stem.
|
||
if full_path.suffix.lower() in (".md", ".markdown"):
|
||
title = extract_title(content, fallback=full_path.stem)
|
||
else:
|
||
title = full_path.stem
|
||
if doc is None:
|
||
doc = Document(
|
||
source=source,
|
||
path=rel,
|
||
full_path=str(full_path),
|
||
title=title,
|
||
content=content,
|
||
content_hash=digest,
|
||
indexed_at=datetime.now(UTC),
|
||
# Phase 106 (D2/D3): the sourced creation date, normalized
|
||
# (undetermined or future → today). ``created_at_manual``
|
||
# stays the column default (False) — only the date-edit API
|
||
# (task 05) sets it.
|
||
created_at=normalize_doc_date(raw_date),
|
||
)
|
||
session.add(doc)
|
||
else:
|
||
doc.full_path = str(full_path)
|
||
doc.title = title
|
||
doc.content = content
|
||
doc.content_hash = digest
|
||
doc.indexed_at = datetime.now(UTC)
|
||
# Phase 106 (D4): a content change is a new document version —
|
||
# the date is re-sourced and a previous manual correction is
|
||
# reset (it referred to the old content).
|
||
doc.created_at = normalize_doc_date(raw_date)
|
||
doc.created_at_manual = False
|
||
|
||
session.flush() # guarantees doc.id even for brand-new rows
|
||
|
||
# Phase 125 (task 04, LOCKED A1): a markdown file's image-reference
|
||
# bookkeeping — references whose image has NO document yet in this
|
||
# source are recorded on the doc's ``image_context`` (the
|
||
# ``IMAGE_CONTEXT:`` record) so the post-index enrichment pass picks
|
||
# them up when the image is later indexed; references that DO name an
|
||
# existing image doc record nothing (the task-03 path owns that case
|
||
# — the field stays NULL). The record rides the file's own
|
||
# transaction (committed by the ``session.commit()`` below).
|
||
if full_path.suffix.lower() in _MARKDOWN_EXTENSIONS:
|
||
doc.image_context = _missing_image_context(
|
||
session, source=source, md_rel=rel, content=content
|
||
)
|
||
|
||
# Phase 1+2 — chunk, replace the chunk rows (embeddings NULL), embed,
|
||
# and commit the whole file atomically (the endpoint token-cap retry
|
||
# lives in the shared helper — the global policy stays intact for the
|
||
# rest of the KB).
|
||
chunks_n = await _replace_content_chunks(
|
||
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
|
||
)
|
||
session.commit()
|
||
|
||
if verb == "added":
|
||
summary.added += 1
|
||
else:
|
||
summary.updated += 1
|
||
summary.changed_paths.append((source, rel))
|
||
summary.chunks += chunks_n
|
||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||
|
||
# Phase 30; phase 118 (A2, 2026-09-15): EVERY new/changed document
|
||
# gets a ``lite``-model summary — markdown included. Phase 30's
|
||
# "markdown is already natural language" exclusion is retired: the
|
||
# summary is the retrieval seed context (the phase-118 suggestion
|
||
# blocks), not a formatting convenience.
|
||
await _store_summary(
|
||
session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
|
||
)
|
||
|
||
|
||
async def _store_summary(
|
||
session: Session,
|
||
*,
|
||
doc: Document,
|
||
source: str,
|
||
rel: str,
|
||
content: str,
|
||
llm: Embedder,
|
||
summary: ImportSummary,
|
||
backfill: bool = False,
|
||
) -> None:
|
||
"""Best-effort ``lite`` summary for one already-committed document.
|
||
|
||
Generates the summary (task 03), stores it on ``documents.summary``
|
||
and indexes it as one extra embedded chunk (``is_summary``,
|
||
position −1) that hybrid search can hit instead of badly-formatted
|
||
raw text. Replacement is guaranteed: any pre-existing ``is_summary``
|
||
chunk of this document is deleted first, so at most one summary chunk
|
||
exists per document at a time.
|
||
|
||
Best-effort by contract: the doc row + content chunks are committed
|
||
by the caller before this runs, so an :class:`LLMError` /
|
||
:class:`EmbeddingError` only rolls back the summary rows — the file
|
||
stays indexed, without a summary, and the failure is counted in
|
||
``summary_errors`` (PLAN phase 30).
|
||
|
||
``backfill`` (phase 118, A2): the unchanged-doc NULL-summary path —
|
||
a success counts ``summary_backfilled`` instead of ``summaries``
|
||
(the doc content is untouched, so the import's KB-change signal must
|
||
not move); the rest of the mechanics are identical.
|
||
|
||
Image docs (phase 122, task 03, LOCKED A3): for an ``is_image`` doc
|
||
the vision description — ``content`` (which equals ``doc.content``
|
||
on the backfill path) — IS the summary: no ``lite`` call, no
|
||
pointer line (the summary mirrors the description verbatim, so
|
||
``doc.summary`` == ``doc.content``). The phase-30 chunk mechanics
|
||
(one ``is_summary`` position −1 chunk, replacement, best-effort
|
||
rollback) are unchanged; the only remaining failure class is the
|
||
summary chunk's embed (the ``doc`` row + content chunks survive —
|
||
fail-soft, same as the text path).
|
||
"""
|
||
try:
|
||
if doc.is_image:
|
||
# Phase 122 (task 03): the description IS the summary —
|
||
# stored verbatim (no ``lite`` call, no pointer line).
|
||
text = content
|
||
else:
|
||
text = await generate_summary(llm, source=source, path=rel, content=content)
|
||
# Replacement: at most one summary chunk per document at a time.
|
||
# Removing from the collection is what the ``delete-orphan``
|
||
# cascade turns into a row delete on flush — and it keeps the
|
||
# in-memory collection consistent (this session runs with
|
||
# ``expire_on_commit=False``).
|
||
for old in [c for c in doc.chunks if c.is_summary]:
|
||
doc.chunks.remove(old)
|
||
chunk = Chunk(document_id=doc.id, position=-1, content=text, is_summary=True)
|
||
vector = (await llm.embed([text]))[0]
|
||
chunk.embedding = vector
|
||
doc.summary = text
|
||
# Append through the relationship (the ``all`` cascade persists the
|
||
# row) so the collection — live in this session because of
|
||
# ``expire_on_commit=False`` — reflects the committed state.
|
||
doc.chunks.append(chunk)
|
||
session.commit()
|
||
if backfill:
|
||
# Phase 118 (A2): the backfill counts itself apart from fresh
|
||
# imports — the doc content is unchanged, so ``summaries``
|
||
# (a KB-change signal) must not move.
|
||
summary.summary_backfilled += 1
|
||
logger.info(
|
||
"import: summary-backfill source=%s path=%s chars=%d",
|
||
source, rel, len(text),
|
||
)
|
||
else:
|
||
summary.summaries += 1
|
||
logger.info("import: summary source=%s path=%s chars=%d", source, rel, len(text))
|
||
except (LLMError, EmbeddingError) as e:
|
||
session.rollback()
|
||
summary.summary_errors += 1
|
||
logger.error("import: summary failed source=%s path=%s — %s", source, rel, e)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# PDF indexing (phase 126, LOCKED A1/A2/A3/A4).
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: The ``Document.title`` column width (``app.models``) — the metadata
|
||
#: title is capped to it (an over-long PDF title must not blow up the
|
||
#: upsert).
|
||
_PDF_TITLE_MAX_CHARS = 500
|
||
|
||
#: MIME → extension for the phase-122 image family (phase 126, task 04)
|
||
#: — the reverse of the ``app.rag.summarizer.IMAGE_MIMES`` values for the
|
||
#: formats PyMuPDF reports. An unlisted mime takes ``.bin`` (never a
|
||
#: member of ``BOR_IMAGE_EXTENSIONS`` — the image is skipped with a
|
||
#: warning, the honest fail-soft outcome).
|
||
_MIME_TO_EXT: dict[str, str] = {
|
||
"image/png": ".png",
|
||
"image/jpeg": ".jpg",
|
||
"image/gif": ".gif",
|
||
"image/webp": ".webp",
|
||
"image/bmp": ".bmp",
|
||
"image/tiff": ".tiff",
|
||
}
|
||
|
||
|
||
def _extract_pdf_title(full_path: Path, fallback: str) -> str:
|
||
"""The title of one PDF document (phase 126, task 04).
|
||
|
||
The embedded metadata title (``fitz``'s ``doc.metadata["title"]``)
|
||
— trimmed and capped at the :data:`_PDF_TITLE_MAX_CHARS` column
|
||
width. An unreadable PDF, an absent metadata dict, or an
|
||
empty/whitespace-only title falls back to *fallback* (the file stem
|
||
— the non-markdown title rule; the metadata title is a convenience,
|
||
never a requirement).
|
||
"""
|
||
try:
|
||
doc = fitz.open(str(full_path))
|
||
except Exception:
|
||
return fallback
|
||
with doc:
|
||
meta = doc.metadata
|
||
title = str(meta.get("title") or "").strip() if meta else ""
|
||
if not title:
|
||
return fallback
|
||
return title[:_PDF_TITLE_MAX_CHARS]
|
||
|
||
|
||
async def _index_pdf_file(
|
||
session: Session,
|
||
*,
|
||
source: str,
|
||
rel: str,
|
||
full_path: Path,
|
||
llm: Embedder,
|
||
summary: ImportSummary,
|
||
raw_date: datetime | None = None,
|
||
on_chunk: Callable[[int, int], None] | None = None,
|
||
) -> None:
|
||
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
|
||
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
|
||
for the document, then the embedded images (A3):
|
||
|
||
* ``content`` is the parser's text — every text-bearing page
|
||
prefixed with the ``---PAGE-N---`` marker (A2 — one document per
|
||
PDF, the markers are the page anchors the page-aware chunker
|
||
(task 03) splits on); an unopenable/empty PDF yields ``""``
|
||
(still a document — it may carry embedded images — with no
|
||
content chunks; a warning is the signal).
|
||
* the sha256 digest is over the extracted text (the digest rule is
|
||
content identity — the same text is the same document version;
|
||
the empty case digests the empty byte string).
|
||
* the title is the metadata title or the file stem
|
||
(:func:`_extract_pdf_title`); the date follows the text path
|
||
exactly (the phase-106 D2 mtime fallback — PDFs carry no
|
||
filesystem-visible creation date — plus the D1 manual-date lock
|
||
and the D4 refresh on the unchanged path).
|
||
* chunk + embed runs through the shared
|
||
:func:`_replace_content_chunks` helper (the endpoint token-cap
|
||
retry lives there), and the phase-30/118 (A2) ``lite`` summary
|
||
path runs on the text — PDFs are no more excluded than any other
|
||
non-markdown format.
|
||
* the unchanged path (same text digest) skips everything — no
|
||
re-extract, no re-embed, no image pass — and runs the same
|
||
backfill/manual-date/date-refresh dance as the text path.
|
||
* THEN the embedded images: :func:`_index_pdf_embedded_images`
|
||
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
|
||
|
||
``raw_date`` follows the text path: the caller's git last-commit
|
||
date or ``None`` (the mtime fallback, read before the unchanged
|
||
early-return — the unchanged path refreshes from the same source).
|
||
|
||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||
chunk-position hook: it is forwarded to the PDF's CONTENT embed
|
||
(the :func:`_replace_content_chunks` call — the primary target),
|
||
but NOT to the embedded-image pass (those image docs are separate
|
||
documents — their ``1/1`` positions would be noise after the
|
||
file's real progress ends, the A6 exclusion). ``None`` keeps the
|
||
branch byte-identical.
|
||
"""
|
||
# Extract the text (fail-soft: an unopenable PDF yields "" — the
|
||
# document may still carry embedded images worth indexing).
|
||
content = extract_pdf_text(full_path)
|
||
if not content:
|
||
logger.warning(
|
||
"import: PDF has no extractable text source=%s path=%s", source, rel
|
||
)
|
||
# Content identity: the digest is over the extracted text (the
|
||
# empty case is the empty byte string's digest).
|
||
digest = (
|
||
hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||
if content
|
||
else hashlib.sha256(b"").hexdigest()
|
||
)
|
||
title = _extract_pdf_title(full_path, fallback=full_path.stem)
|
||
if raw_date is None:
|
||
# D2 fallback (same as the text path): no source date in the
|
||
# map → the file's mtime (one stat). Read before the unchanged
|
||
# early-return — the unchanged path refreshes from it.
|
||
raw_date = file_mtime_datetime(full_path)
|
||
|
||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||
if doc is not None and doc.content_hash == digest:
|
||
# Unchanged PDF (text digest) — the text path's unchanged branch,
|
||
# unmodified in shape: no re-extract, no re-embed, no image pass.
|
||
summary.unchanged += 1
|
||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||
# Phase 118 (A2): an unchanged doc whose summary is still NULL
|
||
# gets the same best-effort summary pass (see the text path).
|
||
if doc.summary is None:
|
||
await _store_summary(
|
||
session, doc=doc, source=source, rel=rel, content=content,
|
||
llm=llm, summary=summary, backfill=True,
|
||
)
|
||
if doc.created_at_manual:
|
||
# D1/D4: the owner's correction survives the sync — no write
|
||
# at all (the text path's manual-date early-return).
|
||
return
|
||
# D4: the date refreshes on every sync and may go OLDER (no
|
||
# monotonic guard); a date-only refresh counts unchanged.
|
||
target = normalize_doc_date(raw_date)
|
||
if target != doc.created_at:
|
||
doc.created_at = target
|
||
session.commit()
|
||
summary.dates_updated += 1
|
||
logger.info(
|
||
"import: date-refreshed source=%s path=%s date=%s",
|
||
source, rel, doc.created_at.isoformat(),
|
||
)
|
||
return
|
||
|
||
verb = "updated" if doc is not None else "added"
|
||
if doc is None:
|
||
doc = Document(
|
||
source=source,
|
||
path=rel,
|
||
full_path=str(full_path),
|
||
title=title,
|
||
content=content,
|
||
content_hash=digest,
|
||
indexed_at=datetime.now(UTC),
|
||
created_at=normalize_doc_date(raw_date),
|
||
)
|
||
session.add(doc)
|
||
else:
|
||
doc.full_path = str(full_path)
|
||
doc.title = title
|
||
doc.content = content
|
||
doc.content_hash = digest
|
||
doc.indexed_at = datetime.now(UTC)
|
||
# Phase 106 (D4): a content change is a new document version —
|
||
# the date is re-sourced and a manual correction reset.
|
||
doc.created_at = normalize_doc_date(raw_date)
|
||
doc.created_at_manual = False
|
||
|
||
session.flush() # guarantees doc.id even for brand-new rows
|
||
|
||
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
|
||
# aware chunker dispatches on the ``.pdf`` suffix; empty content
|
||
# clears the chunk rows and returns 0), committed atomically with
|
||
# the document. Phase 129 (task 02): the in-flight file's
|
||
# chunk-position hook rides the content embed (the embedded-image
|
||
# pass does not receive it — separate documents, the A6 exclusion).
|
||
chunks_n = await _replace_content_chunks(
|
||
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
|
||
)
|
||
session.commit()
|
||
|
||
if verb == "added":
|
||
summary.added += 1
|
||
else:
|
||
summary.updated += 1
|
||
summary.changed_paths.append((source, rel))
|
||
summary.chunks += chunks_n
|
||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||
|
||
# Phase 30; phase 118 (A2): the ``lite`` summary of the extracted
|
||
# text (best-effort — a failure leaves the PDF indexed without a
|
||
# summary, counted in ``summary_errors``).
|
||
await _store_summary(
|
||
session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
|
||
)
|
||
|
||
# Phase 126 (A3): the embedded images join the phase-122 pipeline
|
||
# (gated by the ``images`` toggle — skipped, debug-logged, while it
|
||
# is off). Runs after the document commit, so a per-image failure
|
||
# never loses the PDF's text index.
|
||
await _index_pdf_embedded_images(
|
||
session, source=source, rel=rel, full_path=full_path,
|
||
llm=llm, summary=summary, raw_date=raw_date,
|
||
)
|
||
|
||
|
||
async def _index_pdf_embedded_images(
|
||
session: Session,
|
||
*,
|
||
source: str,
|
||
rel: str,
|
||
full_path: Path,
|
||
llm: Embedder,
|
||
summary: ImportSummary,
|
||
raw_date: datetime,
|
||
) -> None:
|
||
"""The phase-126 (A3) embedded-image pass — one call per (re)indexed
|
||
PDF, after its document commit.
|
||
|
||
Each embedded image (page order, 0-based index on the page — the
|
||
parser's order) is indexed as a STANDALONE image document through
|
||
the phase-122 mechanics, inline (no temp file, no
|
||
``_index_image_file`` refactor):
|
||
|
||
* gated by the phase-122 ``images`` toggle (the single knob, LOCKED
|
||
A3): while it is OFF the whole pass is skipped (one debug line —
|
||
the extraction itself is not worth running) and the prune guard
|
||
keeps any existing image docs;
|
||
* an image whose mime maps to no ``BOR_IMAGE_EXTENSIONS`` member
|
||
(:data:`_MIME_TO_EXT`) is skipped with a warning (the unsupported-
|
||
extension rule, same as a standalone image's);
|
||
* the document's synthetic ``path`` is
|
||
``<pdf path>#page=<N>#img=<index>.<ext>`` — unique per image
|
||
occurrence, stable across re-indexes (the upsert key); the title
|
||
is ``Embedded image from <pdf stem> page <N>``;
|
||
* the digest is sha256 over the image BYTES (the digest rule is
|
||
content identity); an image doc whose digest matches is skipped
|
||
(no re-describe, no re-embed — unchanged);
|
||
* the description comes through the SAME single seam
|
||
(:func:`_describe_or_skip` — the bytes as a data URL whose mime
|
||
comes from the copy path's extension); a failed/empty description
|
||
SKIPS the image entirely (no row, no copy) — ``images_failed`` +
|
||
a warning, the sync continues (LOCKED A3 fail-soft). The
|
||
``image_dir`` copy (``<doc-id>.<ext>`` — a new image's id is the
|
||
uuid4 chosen here; a changed image keeps its id and its stale
|
||
copy is deleted before replacement) is written only AFTER a
|
||
successful description (a failure never leaves an orphan);
|
||
* phase 125 parity (the standalone image's task-03 scan): a same-
|
||
source markdown file that references the image by its synthetic
|
||
path (````) enriches
|
||
the description — the combined context goes to the vision call
|
||
and is stored on the doc's ``image_context``, and ``content``
|
||
becomes ``<context>\\n\\n<description>`` (as does ``summary`` —
|
||
the image convention); no reference keeps the bare shape
|
||
(byte-identical to the un-enriched case). The markdown-FIRST case
|
||
(a markdown indexed before the PDF recorded its reference on its
|
||
own row) is closed by the shared post-index pass
|
||
(:func:`_enrich_pending_images`), exactly as for a standalone
|
||
image;
|
||
* the description IS the content and the summary (the phase-122
|
||
image convention — :func:`_store_summary` stores it verbatim),
|
||
chunked + embedded through :func:`_replace_content_chunks` (one
|
||
transaction per image); a per-image embed failure rolls back that
|
||
image only (``errors`` + the error line) and the pass moves on —
|
||
the PDF's text index and the other images are already committed.
|
||
* a new/updated image doc counts ``added``/``updated`` (it really
|
||
changed the KB — the sources-version bump fires on it, like a
|
||
standalone image's).
|
||
|
||
``raw_date`` is the PDF's resolved source date (the pass runs only
|
||
for (re)indexed PDFs, so it is always concrete) — the image docs
|
||
inherit the PDF's date (they are part of it).
|
||
"""
|
||
settings = llm.settings
|
||
if not settings.images:
|
||
# Phase-122 toggle off: embedded images are skipped (debug-
|
||
# logged) — and the prune guard keeps any existing image docs
|
||
# (they are invisible to the walk, not deleted files).
|
||
logger.debug(
|
||
"import: PDF embedded images skipped (images toggle off) "
|
||
"source=%s path=%s",
|
||
source, rel,
|
||
)
|
||
return
|
||
images = extract_pdf_images(full_path)
|
||
for img in images:
|
||
await _index_one_embedded_image(
|
||
session, img, source=source, rel=rel, full_path=full_path,
|
||
llm=llm, summary=summary, raw_date=raw_date,
|
||
)
|
||
|
||
|
||
async def _index_one_embedded_image(
|
||
session: Session,
|
||
img: PdfImage,
|
||
*,
|
||
source: str,
|
||
rel: str,
|
||
full_path: Path,
|
||
llm: Embedder,
|
||
summary: ImportSummary,
|
||
raw_date: datetime,
|
||
) -> None:
|
||
"""One embedded image of one PDF → the phase-122 image pipeline,
|
||
with the phase-125 task-03 markdown context scan (parity with the
|
||
standalone image's :func:`_index_image_file` — see
|
||
:func:`_index_pdf_embedded_images` for the full contract)."""
|
||
settings = llm.settings
|
||
ext = _MIME_TO_EXT.get(img.mime, ".bin")
|
||
if ext not in settings.image_extension_set:
|
||
logger.warning(
|
||
"import: PDF embedded image skipped (unsupported extension) "
|
||
"source=%s path=%s page=%d index=%d mime=%s",
|
||
source, rel, img.page, img.index, img.mime,
|
||
)
|
||
return
|
||
image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
|
||
digest = hashlib.sha256(img.data).hexdigest()
|
||
image_doc = session.scalar(
|
||
select(Document).where(Document.source == source, Document.path == image_rel)
|
||
)
|
||
if image_doc is not None and image_doc.content_hash == digest:
|
||
# Unchanged embedded image (byte digest) — no re-describe, no
|
||
# re-embed (the standalone image's unchanged rule).
|
||
logger.debug(
|
||
"import: PDF embedded image unchanged source=%s path=%s",
|
||
source, image_rel,
|
||
)
|
||
return
|
||
verb = "updated" if image_doc is not None else "added"
|
||
# Named by the doc id (the phase-122 copy rule): a new image's id is
|
||
# the uuid4 chosen here (row and copy agree); a changed image keeps
|
||
# its id (the copy path is stable, the stale copy is deleted below).
|
||
doc_id = image_doc.id if image_doc is not None else uuid.uuid4()
|
||
image_dir = Path(settings.image_dir).expanduser()
|
||
copy_path = image_dir / f"{doc_id}{ext}"
|
||
# Phase 125 (task 03, LOCKED A1/A2) — the embedded image's parity
|
||
# with the standalone image's path (phase 126): BEFORE the
|
||
# description, the markdown reference context for THIS image — a
|
||
# ```` reference in a
|
||
# same-source markdown file (the synthetic path is the doc's
|
||
# ``path``, and the reference resolves to it through the shared
|
||
# :func:`_resolve_ref_path` rule) — or ``None`` (no reference —
|
||
# the description call and the row stay byte-identical to the
|
||
# un-enriched shape). It goes to the vision model (a context-aware
|
||
# description, LOCKED A3) AND is stored on the doc (``image_context``
|
||
# — the retrieval display's context line, task 05). A re-indexed
|
||
# image re-scans: the context follows the current source state
|
||
# (a deleted referencing markdown drops it back to ``None``).
|
||
image_context = _markdown_image_context(
|
||
_source_root_for(full_path, rel), llm, source=source, image_rel=image_rel
|
||
)
|
||
# The description through the single seam (the mime resolves from
|
||
# the copy path's extension — the extension was derived from the
|
||
# mime, so the round-trip is lossless for the default family). The
|
||
# description is generated BEFORE anything is written, so a failure
|
||
# skips the image with no row and no copy.
|
||
description = await _describe_or_skip(
|
||
llm, data=img.data, source=source, rel=image_rel, full_path=copy_path,
|
||
context=image_context,
|
||
)
|
||
if description is None:
|
||
# LOCKED A3 fail-soft: an undescribed image is unsearchable
|
||
# noise — skip it (no row, no copy).
|
||
summary.images_failed += 1
|
||
logger.warning(
|
||
"import: PDF embedded image description failed source=%s path=%s "
|
||
"page=%d index=%d",
|
||
source, image_rel, img.page, img.index,
|
||
)
|
||
return
|
||
if image_doc is not None:
|
||
# A CHANGED image: the stale copy is deleted before replacement.
|
||
_delete_image_copy(image_doc.image_path)
|
||
image_dir.mkdir(parents=True, exist_ok=True)
|
||
copy_path.write_bytes(img.data)
|
||
|
||
# Phase 125 (task 03): the enriched content — the reference context
|
||
# FIRST (for retrieval), then the description (the standalone image's
|
||
# rule; ``summary = content`` below keeps the image convention).
|
||
content = (
|
||
f"{image_context}\n\n{description}" if image_context is not None else description
|
||
)
|
||
|
||
title = f"Embedded image from {full_path.stem} page {img.page}"
|
||
if image_doc is None:
|
||
image_doc = Document(
|
||
id=doc_id,
|
||
source=source,
|
||
path=image_rel,
|
||
full_path=str(copy_path),
|
||
title=title,
|
||
content=content,
|
||
content_hash=digest,
|
||
indexed_at=datetime.now(UTC),
|
||
created_at=normalize_doc_date(raw_date),
|
||
is_image=True,
|
||
image_path=str(copy_path),
|
||
image_context=image_context,
|
||
)
|
||
session.add(image_doc)
|
||
else:
|
||
image_doc.full_path = str(copy_path)
|
||
image_doc.title = title
|
||
image_doc.content = content
|
||
image_doc.content_hash = digest
|
||
image_doc.indexed_at = datetime.now(UTC)
|
||
# Phase 106 (D4): a content change is a new document version —
|
||
# the date is re-sourced and a manual correction reset.
|
||
image_doc.created_at = normalize_doc_date(raw_date)
|
||
image_doc.created_at_manual = False
|
||
image_doc.is_image = True
|
||
image_doc.image_path = str(copy_path)
|
||
# Phase 125 (task 03): a re-indexed image re-scans its markdown
|
||
# references — the context follows the current source state.
|
||
image_doc.image_context = image_context
|
||
|
||
session.flush() # guarantees doc.id even for brand-new rows
|
||
|
||
# Phase 1+2 — the shared pipeline on the description (one
|
||
# transaction per image; a per-image embed failure rolls back THIS
|
||
# image only — the PDF doc and the other images are already
|
||
# committed — and the pass moves on).
|
||
try:
|
||
chunks_n = await _replace_content_chunks(
|
||
session, doc=image_doc, llm=llm, content=content, rel=image_rel
|
||
)
|
||
session.commit()
|
||
except EmbeddingError as e:
|
||
session.rollback()
|
||
# The description succeeded (the copy was written), but the
|
||
# embed gave up: delete the copy so a failure never leaves an
|
||
# orphan (the phase-122 no-orphan rule).
|
||
_delete_image_copy(str(copy_path))
|
||
summary.errors += 1
|
||
logger.error(
|
||
"import: PDF embedded image embed failed source=%s path=%s "
|
||
"page=%d index=%d — %s",
|
||
source, image_rel, img.page, img.index, e,
|
||
)
|
||
return
|
||
|
||
if verb == "added":
|
||
summary.added += 1
|
||
else:
|
||
summary.updated += 1
|
||
summary.changed_paths.append((source, image_rel))
|
||
summary.chunks += chunks_n
|
||
logger.info(
|
||
"import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
|
||
)
|
||
# Phase 122 (task 03) shape on the description — image-aware: the
|
||
# description IS the summary (stored verbatim, no ``lite`` call), so
|
||
# ``doc.summary`` == ``doc.content`` and the ``is_summary`` position
|
||
# −1 chunk mirrors it. With a markdown reference (phase 125) the
|
||
# summary is the ENRICHED content — the same value the standalone
|
||
# image's path stores.
|
||
await _store_summary(
|
||
session, doc=image_doc, source=source, rel=image_rel,
|
||
content=content, llm=llm, summary=summary,
|
||
)
|
||
|
||
|
||
def _source_root_for(full_path: Path, rel: str) -> Path:
|
||
"""The source root of one walked file (phase 125, task 03).
|
||
|
||
The walk's invariant is ``full_path == root / rel`` (``rel`` is the
|
||
source-relative POSIX path, ``full_path`` the absolute path), so the
|
||
root is *full_path*'s parent walked UP by the number of DIRECTORY
|
||
components of *rel* (a top-level file has none — its own directory
|
||
IS the root).
|
||
"""
|
||
root = full_path.parent
|
||
for _ in PurePosixPath(rel).parts[:-1]:
|
||
root = root.parent
|
||
return root
|
||
|
||
|
||
def _resolve_ref_path(ref_path: str, md_rel: str) -> str | None:
|
||
"""One ```` reference path → the image's source-relative
|
||
POSIX path, or ``None`` when the reference names no in-source file
|
||
(phase 125 — the single resolution rule shared by the task-03
|
||
context scan, the task-04 markdown bookkeeping, and the task-04
|
||
post-index enrichment pass).
|
||
|
||
A leading ``/`` = the source root (a root-absolute doc-style
|
||
reference); otherwise the path is resolved against the markdown
|
||
file's directory (``md_rel`` is the markdown's source-relative
|
||
POSIX path — its ``dirname``, so a top-level markdown joins against
|
||
the root). The result is normalized; a normalized path that still
|
||
escapes (``..``-prefixed — e.g. ``../../other/x.png``) or names no
|
||
file (``.``) matches no image document in this source, ever.
|
||
"""
|
||
if ref_path.startswith("/"):
|
||
resolved = ref_path.lstrip("/")
|
||
else:
|
||
resolved = posixpath.join(posixpath.dirname(md_rel), ref_path)
|
||
normalized = posixpath.normpath(resolved)
|
||
if normalized in ("", ".") or normalized.startswith(".."):
|
||
return None
|
||
return normalized
|
||
|
||
|
||
def _ref_points_at(ref: ImageRef, md_rel: str, image_rel: str) -> bool:
|
||
"""Does *ref* point at the image at *image_rel* (phase 125, task 03)?
|
||
|
||
The reference path is resolved against the markdown file's directory
|
||
(``md_rel`` is the markdown's source-relative POSIX path) — or,
|
||
when it starts with ``/``, against the SOURCE ROOT (a root-absolute
|
||
doc-style reference) — then normalized and compared for equality
|
||
with the image's source-relative path. Handles both the bare
|
||
relative reference (``pic.png`` next to the markdown) and the
|
||
folder-carrying one (``folder/pic.png`` from the root); a reference
|
||
to a different file that shares the filename is NOT a match.
|
||
"""
|
||
resolved = _resolve_ref_path(ref.path, md_rel)
|
||
return resolved is not None and resolved == image_rel
|
||
|
||
|
||
def _markdown_image_context(
|
||
root: Path,
|
||
llm: Embedder,
|
||
*,
|
||
source: str,
|
||
image_rel: str,
|
||
) -> str | None:
|
||
"""The combined markdown reference context for one image (phase 125,
|
||
task 03, LOCKED A1/A2) — or ``None`` when no markdown file in the
|
||
source references it (the standalone-image case: the whole phase-125
|
||
machinery is a no-op, byte-identical to pre-phase).
|
||
|
||
Scans the source's importable MARKDOWN files (``.md``/``.markdown``
|
||
under the source's import-extension set — hidden/excluded-dir rules
|
||
and the extension filter apply exactly as to the walk) and keeps
|
||
every :func:`~app.rag.md_images.extract_image_refs` reference that
|
||
POINTS AT this image (:func:`_md_ref_block` resolves each reference
|
||
via :func:`_resolve_ref_path` — ``pic.png`` from ``guide.md`` →
|
||
``pic.png``; ``../pic.png`` from ``sub/guide.md`` → ``pic.png``;
|
||
``folder/pic.png`` from ``guide.md`` → ``folder/pic.png``), one
|
||
block per referencing markdown file in the walk's sorted order.
|
||
The result is both the *context* argument of the vision call (the
|
||
model learns WHAT the image is for) and the stored
|
||
``image_context`` (the retrieval display's context line, task 05).
|
||
"""
|
||
blocks: list[str] = []
|
||
for md_path in iter_importable_files(root, llm.settings.import_extension_set):
|
||
if md_path.suffix.lower() not in _MARKDOWN_EXTENSIONS:
|
||
continue
|
||
try:
|
||
content = md_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||
except OSError:
|
||
continue # unreadable source file — no context from it
|
||
block = _md_ref_block(source, md_path.relative_to(root).as_posix(), content, image_rel)
|
||
if block is not None:
|
||
blocks.append(block)
|
||
return "\n\n".join(blocks) if blocks else None
|
||
|
||
|
||
def _md_ref_block(source: str, md_rel: str, content: str, image_rel: str) -> str | None:
|
||
"""One ``Referenced in <source>/<md>:`` block for one markdown's
|
||
content (phase 125) — the shared builder of the task-03 disk scan
|
||
(:func:`_markdown_image_context`) and the task-04 pass
|
||
(:func:`_pending_image_context`).
|
||
|
||
Keeps every image reference in *content* that POINTS AT *image_rel*
|
||
(the resolution rule: :func:`_resolve_ref_path` — a reference to a
|
||
different file that merely shares the filename never matches,
|
||
LOCKED A1), in line order, rendered as::
|
||
|
||
Referenced in <source>/<markdown_path>:
|
||
- Alt text: "<alt_text>"
|
||
Context: <context_paragraphs>
|
||
|
||
(the ``Context:`` line is omitted for a reference with no
|
||
surrounding paragraphs). ``None`` when *content* has no reference
|
||
to this image.
|
||
"""
|
||
refs = [ref for ref in extract_image_refs(content) if _ref_points_at(ref, md_rel, image_rel)]
|
||
if not refs:
|
||
return None
|
||
lines = [f"Referenced in {source}/{md_rel}:"]
|
||
for ref in refs:
|
||
lines.append(f'- Alt text: "{ref.alt_text}"')
|
||
if ref.context:
|
||
lines.append(f" Context: {ref.context}")
|
||
return "\n".join(lines)
|
||
|
||
|
||
#: Header line of a markdown doc's pending-image record (phase 125,
|
||
#: task 04, LOCKED A1) — the stored ``image_context`` of a markdown
|
||
#: file that references images with no document yet in the source.
|
||
#: :func:`_parse_missing_image_refs` is its parse target (the record is
|
||
#: the post-index enrichment pass's reference index; the header is
|
||
#: ignored by the parser — the bullets are the data).
|
||
IMAGE_CONTEXT_HEADER = "IMAGE_CONTEXT:"
|
||
|
||
|
||
def _format_missing_image_refs(refs: Sequence[ImageRef], source: str) -> str:
|
||
"""One markdown's missing-image references → the ``IMAGE_CONTEXT:``
|
||
record stored on the markdown doc's ``image_context`` (phase 125,
|
||
task 04, LOCKED A1) — the durable record that "this markdown
|
||
referenced these images, which have no document yet in this
|
||
source":
|
||
|
||
.. code-block:: none
|
||
|
||
IMAGE_CONTEXT:
|
||
- [image.png] (line 42): "Alt text here" — 2 paragraphs of context
|
||
- [diagram.jpg] (line 100): "" — 1 paragraph of context
|
||
|
||
One bullet per reference, in reference order: the RESOLVED
|
||
source-relative image path in the brackets (the caller resolves via
|
||
:func:`_resolve_ref_path` — the post-index pass matches it against
|
||
the image doc's ``path``, so the resolution must agree with the
|
||
task-03 scan's), the 1-based line number, the alt text between the
|
||
quotes (``""`` for ````), and the count of the reference's
|
||
context paragraphs (the task-02 window — the paragraphs themselves
|
||
live on the markdown doc's ``content``; the pass re-extracts them:
|
||
the record is the index, the content is the source). The shape is
|
||
exactly what :func:`_parse_missing_image_refs` parses back (the
|
||
"task 04b" parse target). *source* stays on the signature so the
|
||
call site reads like the record's home (the record rides a doc row
|
||
whose source is already known — display context, not data).
|
||
|
||
Empty *refs* → ``""`` (the caller stores ``None`` for "no record"
|
||
— an empty record would be a non-NULL empty string, and the pass's
|
||
``IS NOT NULL`` gate must see "no record" as NULL, not ``""``).
|
||
"""
|
||
if not refs:
|
||
return ""
|
||
lines = [IMAGE_CONTEXT_HEADER]
|
||
for ref in refs:
|
||
n = len(ref.context.split("\n\n")) if ref.context else 0
|
||
unit = "paragraph" if n == 1 else "paragraphs"
|
||
lines.append(
|
||
f'- [{ref.path}] (line {ref.line_number}): "{ref.alt_text}"'
|
||
f" — {n} {unit} of context"
|
||
)
|
||
return "\n".join(lines)
|
||
|
||
|
||
#: One bullet of the ``IMAGE_CONTEXT:`` record — the exact output shape
|
||
#: of :func:`_format_missing_image_refs`: the resolved image path in
|
||
#: the brackets, the 1-based line number, the alt text between the
|
||
#: quotes, the context-paragraph count.
|
||
_MISSING_REF_LINE_RE = re.compile(
|
||
r'^- \[(?P<path>[^\]]+)\] \(line (?P<line>\d+)\): "(?P<alt>.*)" — '
|
||
r"(?P<count>\d+) paragraph(?:s)? of context$"
|
||
)
|
||
|
||
|
||
def _parse_missing_image_refs(text: str) -> list[tuple[str, int, str]]:
|
||
"""Parse a markdown doc's ``image_context`` record back into
|
||
``(image path, line number, alt text)`` triples (phase 125, task 04
|
||
— the enrichment pass's reference index).
|
||
|
||
Only bullets in :func:`_format_missing_image_refs`' exact shape
|
||
parse (the greedy alt field still pins an alt that itself contains
|
||
``"``, as long as it does not contain the ``" — N paragraph…"``
|
||
tail): the header line, an owner-edited value, or a future shape
|
||
yields no references and the pass simply finds no match (fail-soft
|
||
— a malformed record must never raise into the sync).
|
||
"""
|
||
refs: list[tuple[str, int, str]] = []
|
||
for line in text.splitlines():
|
||
m = _MISSING_REF_LINE_RE.match(line)
|
||
if m is not None:
|
||
refs.append((m.group("path"), int(m.group("line")), m.group("alt")))
|
||
return refs
|
||
|
||
|
||
def _missing_image_context(
|
||
session: Session, *, source: str, md_rel: str, content: str
|
||
) -> str | None:
|
||
"""The markdown doc's ``image_context`` for one (re)indexed markdown
|
||
file (phase 125, task 04, LOCKED A1): the ``IMAGE_CONTEXT:`` record
|
||
of every ```` reference whose image has NO ``is_image``
|
||
document yet in this source, or ``None`` when every in-source
|
||
reference names an existing image doc (the task-03 path owns those
|
||
— enrichment at image-index time — and the field stays NULL) or the
|
||
file has no references at all.
|
||
|
||
Each reference is resolved exactly as the task-03 scan
|
||
(:func:`_resolve_ref_path`); a reference that escapes the source
|
||
(e.g. ``../../other/x.png``) names no in-source image document, ever,
|
||
and is not recorded (the field is the pending-enrichment index, not
|
||
a general reference dump). A re-indexed markdown recomputes the
|
||
field from its CURRENT content — the image now exists (or the
|
||
reference was dropped) → the field drops back to ``None``.
|
||
"""
|
||
refs = extract_image_refs(content)
|
||
if not refs:
|
||
return None
|
||
existing = set(
|
||
session.scalars(
|
||
select(Document.path).where(
|
||
Document.source == source, Document.is_image.is_(True)
|
||
)
|
||
).all()
|
||
)
|
||
missing: list[ImageRef] = []
|
||
for ref in refs:
|
||
resolved = _resolve_ref_path(ref.path, md_rel)
|
||
if resolved is None or resolved in existing:
|
||
continue
|
||
missing.append(
|
||
ImageRef(
|
||
path=resolved,
|
||
alt_text=ref.alt_text,
|
||
context=ref.context,
|
||
line_number=ref.line_number,
|
||
)
|
||
)
|
||
if not missing:
|
||
return None
|
||
return _format_missing_image_refs(missing, source)
|
||
|
||
|
||
def _pending_image_context(
|
||
source: str, image_rel: str, md_docs: Sequence[Document]
|
||
) -> str | None:
|
||
"""The task-04 pass's context for one pending image doc: the SAME
|
||
combined string the task-03 scan builds (``Referenced in …`` blocks
|
||
joined with a blank line) — but sourced from the markdown DOC ROWS
|
||
instead of the on-disk files: a markdown doc contributes only when
|
||
its stored ``image_context`` record lists the image's source-relative
|
||
path (the durable "this markdown referenced this image" record,
|
||
work 1), and the reference (alt text + surrounding paragraphs) is
|
||
re-extracted from the doc's stored ``content`` (the full text is on
|
||
the row — the record is the index, the content is the source).
|
||
``None`` when no markdown doc references the image.
|
||
"""
|
||
blocks: list[str] = []
|
||
for md in md_docs:
|
||
listed = _parse_missing_image_refs(md.image_context or "")
|
||
if image_rel not in {path for path, _, _ in listed}:
|
||
continue
|
||
block = _md_ref_block(source, md.path, md.content, image_rel)
|
||
if block is not None:
|
||
blocks.append(block)
|
||
return "\n\n".join(blocks) if blocks else None
|
||
|
||
|
||
async def _enrich_pending_images(
|
||
session: Session, *, source: str, llm: Embedder, summary: ImportSummary
|
||
) -> None:
|
||
"""The phase-125 task-04 post-index enrichment pass — once per
|
||
source, after ALL of its files are indexed (:func:`import_sources`).
|
||
|
||
Closes the markdown-FIRST case: a markdown file indexed while its
|
||
referenced image had no document yet stores the reference on its
|
||
``image_context`` (work 1 — the ``IMAGE_CONTEXT:`` record). When the
|
||
image is later indexed but the task-03 disk scan could not see the
|
||
reference (the markdown file no longer on disk — a deleted / moved
|
||
source file whose doc row survives a ``prune=False`` run — or any
|
||
other disk-vs-DB drift), the image doc is still ``image_context IS
|
||
NULL`` and this pass enriches it from the markdown doc's STORED rows
|
||
(see :func:`_pending_image_context`).
|
||
|
||
The enrichment is the task-03 mechanics on the existing doc: a fresh
|
||
context-aware description (``describe_image`` with the context — the
|
||
description remains the primary content, LOCKED A3), ``content`` =
|
||
``<context>\\n\\n<description>`` (and so the ``summary`` — the
|
||
phase-122 ``summary = content`` convention), the context stored on
|
||
``image_context``, and the re-chunked + re-embedded pipeline (the
|
||
re-describe reads the image's PERSISTENT copy — the source file may
|
||
be the one that is gone).
|
||
|
||
Idempotent by contract (ASSUMPTION 4): an image doc with
|
||
``image_context`` SET is never touched (already enriched — the
|
||
task-03 path or an earlier pass), and a markdown doc's
|
||
``image_context`` is NOT cleared on success (it stays the retrieval
|
||
display's record, task 05). Fail-soft: a missing persistent copy, a
|
||
failed/empty re-description, or an embed error keeps the image doc
|
||
as-is (warned / counted) and the next sync retries it; one bad image
|
||
never blocks the others. A success counts ``updated`` + ``chunks``
|
||
(the doc's content really changed — the caller's KB-change gate and
|
||
the sources-version bump both fire on ``added + updated``, which is
|
||
correct here) and a dedicated ``image-context-enriched`` log line
|
||
(PLAN §9).
|
||
"""
|
||
pending = session.scalars(
|
||
select(Document).where(
|
||
Document.source == source,
|
||
Document.is_image.is_(True),
|
||
Document.image_context.is_(None),
|
||
)
|
||
).all()
|
||
if not pending:
|
||
return
|
||
md_docs = session.scalars(
|
||
select(Document).where(
|
||
Document.source == source,
|
||
Document.is_image.is_(False),
|
||
Document.image_context.is_not(None),
|
||
)
|
||
).all()
|
||
if not md_docs:
|
||
return
|
||
for image_doc in pending:
|
||
context = _pending_image_context(source, image_doc.path, md_docs)
|
||
if context is None:
|
||
continue # no markdown doc records a reference to this image
|
||
if not image_doc.image_path or not Path(image_doc.image_path).is_file():
|
||
# No persistent bytes to re-describe — keep the doc as-is
|
||
# (the next sync retries; a copy gone for good leaves an
|
||
# unenrichable doc, and the warning is the signal).
|
||
summary.images_failed += 1
|
||
logger.warning(
|
||
"import: image-context enrichment skipped (no persistent copy) "
|
||
"source=%s path=%s",
|
||
source, image_doc.path,
|
||
)
|
||
continue
|
||
copy_path = Path(image_doc.image_path)
|
||
description = await _describe_or_skip(
|
||
llm,
|
||
data=copy_path.read_bytes(),
|
||
source=source,
|
||
rel=image_doc.path,
|
||
full_path=copy_path,
|
||
context=context,
|
||
)
|
||
if description is None:
|
||
summary.images_failed += 1
|
||
logger.warning(
|
||
"import: image re-description failed source=%s path=%s",
|
||
source, image_doc.path,
|
||
)
|
||
continue
|
||
try:
|
||
image_doc.image_context = context
|
||
image_doc.content = f"{context}\n\n{description}"
|
||
chunks_n = await _replace_content_chunks(
|
||
session, doc=image_doc, llm=llm,
|
||
content=image_doc.content, rel=image_doc.path,
|
||
)
|
||
session.commit()
|
||
except EmbeddingError as e:
|
||
session.rollback()
|
||
summary.errors += 1
|
||
logger.error(
|
||
"import: image-context enrichment failed source=%s path=%s — %s",
|
||
source, image_doc.path, e,
|
||
)
|
||
continue
|
||
summary.updated += 1
|
||
summary.changed_paths.append((source, image_doc.path))
|
||
summary.chunks += chunks_n
|
||
logger.info(
|
||
"import: image-context-enriched source=%s path=%s chunks=%d",
|
||
source, image_doc.path, chunks_n,
|
||
)
|
||
await _store_summary(
|
||
session, doc=image_doc, source=source, rel=image_doc.path,
|
||
content=image_doc.content, llm=llm, summary=summary,
|
||
)
|
||
|
||
|
||
async def _describe_or_skip(
|
||
llm: Embedder,
|
||
*,
|
||
data: bytes,
|
||
source: str,
|
||
rel: str,
|
||
full_path: Path,
|
||
context: str | None = None,
|
||
) -> str | None:
|
||
"""Phase 122 — the SINGLE seam for the image description (task 03).
|
||
|
||
Returns the vision description that becomes the image document's
|
||
``content`` (and ``summary`` — the ONLY embedded text of the doc),
|
||
or ``None`` when the description failed or came back empty — the
|
||
caller then SKIPS the doc entirely (no row, no copy) and counts
|
||
``summary.images_failed`` (LOCKED A3, fail-soft: the sync continues,
|
||
the warning + counter are the signal).
|
||
|
||
The seam is one line by design (the importer's tests patch exactly
|
||
this function): :func:`app.rag.summarizer.describe_image` — ONE
|
||
CHAT-model (vision) call (LOCKED A3) with the bytes as a data URL
|
||
whose mime comes from :data:`app.rag.summarizer.IMAGE_MIMES`
|
||
(dotted extension; an unlisted ``BOR_IMAGE_EXTENSIONS`` token takes
|
||
the generic fallback — a rejection there fails soft like any other
|
||
description error). ``source``/``rel`` stay on the signature so the
|
||
caller (and the patch) reads like the document being described;
|
||
the failure's doc identity is logged by the caller's warning.
|
||
|
||
``context`` (phase 125, task 03) is the optional markdown reference
|
||
context (:func:`_markdown_image_context`), forwarded verbatim to
|
||
:func:`app.rag.summarizer.describe_image` — appended AFTER the
|
||
locked describe prompt so the vision model can write a
|
||
context-aware description (LOCKED A3: the description remains the
|
||
primary content). ``None`` (standalone image) keeps the message
|
||
byte-identical to pre-phase-125.
|
||
"""
|
||
mime = IMAGE_MIMES.get(full_path.suffix.lower(), IMAGE_FALLBACK_MIME)
|
||
return await describe_image(llm, data=data, mime=mime, context=context)
|
||
|
||
|
||
def _delete_image_copy(image_path: str | None) -> None:
|
||
"""Best-effort removal of a stale image copy (phase 122).
|
||
|
||
A missing path is a no-op (already gone — e.g. the owner cleaned
|
||
the image dir); an unreadable one is logged, never raised — copy
|
||
cleanup must not break the sync (the doc row's fate is decided by
|
||
the upsert/prune logic, not by filesystem hygiene).
|
||
"""
|
||
if not image_path:
|
||
return
|
||
try:
|
||
Path(image_path).unlink(missing_ok=True)
|
||
except OSError as e:
|
||
logger.warning("import: could not delete image copy %s — %s", image_path, e)
|
||
|
||
|
||
async def _index_image_file(
|
||
session: Session,
|
||
*,
|
||
source: str,
|
||
rel: str,
|
||
full_path: Path,
|
||
llm: Embedder,
|
||
summary: ImportSummary,
|
||
raw_date: datetime | None = None,
|
||
on_chunk: Callable[[int, int], None] | None = None,
|
||
) -> None:
|
||
"""The phase-122 image branch of :func:`_index_file` — a standalone
|
||
image is indexed from its BYTES, never its text:
|
||
|
||
* the sha256 digest is over the raw bytes (the digest rule is
|
||
content identity — the same bytes are the same document);
|
||
* the PERSISTENT copy lands in ``settings.image_dir`` as
|
||
``<doc-id>.<ext>`` (the dir is created on demand; the copy is
|
||
written only AFTER a successful description, so a failure never
|
||
leaves an orphan; a changed image deletes the stale copy before
|
||
replacing it);
|
||
* ``content`` is the vision description — the ONLY embedded text of
|
||
the document (the embedding model never sees pixels) — and the
|
||
normal chunk pipeline then embeds it, with the phase-30 summary
|
||
path running on it (the ``is_summary`` position −1 chunk mirrors
|
||
``Document.summary``).
|
||
|
||
``raw_date`` follows the text path exactly (the phase-106 D2
|
||
fallback: no source date in the map → the file's mtime, read before
|
||
the unchanged early-return because the unchanged path refreshes the
|
||
stored date from the same source; the D1 manual-date lock and the
|
||
D4 refresh apply unmodified).
|
||
|
||
Fail-soft (LOCKED A3): a failed/empty description SKIPS the doc
|
||
entirely (no row, no copy) — ``summary.images_failed`` + a warning,
|
||
the sync continues.
|
||
|
||
Phase 125 (task 03, LOCKED A1/A2/A3): a NEW/CHANGED image first
|
||
scans the same source's markdown files for references to it (see
|
||
:func:`_markdown_image_context`). When one or more exist, the
|
||
combined context string goes to the vision call (``context`` — the
|
||
model writes a context-aware description; the description remains
|
||
the primary content) and is stored on the doc's ``image_context``;
|
||
the doc's ``content`` becomes ``<context>\n\n<description>`` (and so
|
||
does ``summary`` — the phase-122 ``summary = content`` convention —
|
||
as do the embedded chunks). No reference → ``image_context`` stays
|
||
NULL and ``content`` is the bare description (byte-identical to
|
||
pre-phase). The UNCHANGED path never re-scans (the stored context
|
||
is part of the row's committed content; a changed source state
|
||
re-indexes the file through this path).
|
||
|
||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||
chunk-position hook: it is forwarded to the description's content
|
||
embed (:func:`_replace_content_chunks`). A description (enriched or
|
||
not) chunks to ONE chunk, so the hook reports ``1/1`` — acceptable,
|
||
it IS the file's only chunk. ``None`` keeps the branch
|
||
byte-identical.
|
||
"""
|
||
settings = llm.settings
|
||
data = full_path.read_bytes()
|
||
digest = hashlib.sha256(data).hexdigest()
|
||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||
if raw_date is None:
|
||
# D2 fallback (same as the text path): no source date in the map
|
||
# → the file's mtime (one stat).
|
||
raw_date = file_mtime_datetime(full_path)
|
||
|
||
if doc is not None and doc.content_hash == digest:
|
||
# Unchanged image (byte digest) — the text path's unchanged
|
||
# branch, unmodified in shape.
|
||
summary.unchanged += 1
|
||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||
# Phase 118 (A2) backfill, image flavour: an unchanged image doc
|
||
# whose summary is still NULL (an earlier fail-soft summary miss
|
||
# — for an image, the one remaining failure class: the summary
|
||
# chunk's embed) gets the same best-effort summary pass. For an
|
||
# image the summary IS the stored description (``doc.content``),
|
||
# so the image-aware ``_store_summary`` (task 03) re-stores it
|
||
# verbatim with one ``is_summary`` chunk; a failure (the
|
||
# embed) keeps the doc as-is (no row mutation) — fail-soft,
|
||
# same as the text path.
|
||
if doc.summary is None:
|
||
await _store_summary(
|
||
session, doc=doc, source=source, rel=rel, content=doc.content,
|
||
llm=llm, summary=summary, backfill=True,
|
||
)
|
||
if doc.created_at_manual:
|
||
# D1/D4: the owner's correction survives the sync — no write
|
||
# at all (the text path's manual-date early-return).
|
||
return
|
||
# D4: the date refreshes on every sync, including unchanged
|
||
# files, and may go OLDER (no monotonic guard).
|
||
target = normalize_doc_date(raw_date)
|
||
if target != doc.created_at:
|
||
doc.created_at = target
|
||
session.commit()
|
||
summary.dates_updated += 1
|
||
logger.info(
|
||
"import: date-refreshed source=%s path=%s date=%s",
|
||
source, rel, doc.created_at.isoformat(),
|
||
)
|
||
return
|
||
|
||
verb = "updated" if doc is not None else "added"
|
||
# Phase 125 (task 03, LOCKED A1/A2): BEFORE the description — the
|
||
# markdown reference context for this image (the alt text +
|
||
# surrounding paragraphs of the ```` references that
|
||
# name it in the same source), or ``None`` (standalone image — the
|
||
# whole phase-125 enrichment below is a no-op, byte-identical to
|
||
# pre-phase). It goes to the vision model (a context-aware
|
||
# description, LOCKED A3) AND is stored on the doc (``image_context``
|
||
# — the retrieval display's context line, task 05).
|
||
image_context = _markdown_image_context(
|
||
_source_root_for(full_path, rel), llm, source=source, image_rel=rel
|
||
)
|
||
# The description is the doc's content (task 03: and its summary) —
|
||
# it is generated BEFORE anything is written, so a failure skips the
|
||
# doc with no row and no copy (the copy is only made after a
|
||
# successful description — a failure never leaves an orphan).
|
||
content = await _describe_or_skip(
|
||
llm, data=data, source=source, rel=rel, full_path=full_path,
|
||
context=image_context,
|
||
)
|
||
if content is None:
|
||
# LOCKED A3 fail-soft: an undescribed image is unsearchable
|
||
# noise — skip the doc entirely (no row, no copy).
|
||
summary.images_failed += 1
|
||
logger.warning("import: image description failed source=%s path=%s", source, rel)
|
||
return
|
||
if image_context is not None:
|
||
# Phase 125 (task 03): the enriched content — the reference
|
||
# context FIRST (for retrieval), then the description. The
|
||
# ``summary`` field mirrors this (the phase-122 image
|
||
# convention: ``summary = content``), and the chunk pipeline
|
||
# below embeds it.
|
||
content = f"{image_context}\n\n{content}"
|
||
|
||
# The persistent copy: uploads are replaced on every upload, git
|
||
# checkouts are re-cloned, local dirs are user-edited — the served
|
||
# bytes must outlive the source file. Named by the doc id: a new
|
||
# doc's id is the uuid4 chosen here (row and copy agree); a changed
|
||
# doc keeps its id (the copy path is stable).
|
||
doc_id = doc.id if doc is not None else uuid.uuid4()
|
||
image_dir = Path(settings.image_dir).expanduser()
|
||
image_dir.mkdir(parents=True, exist_ok=True)
|
||
if doc is not None:
|
||
# A CHANGED image (hash differs): the stale copy is deleted
|
||
# before replacement.
|
||
_delete_image_copy(doc.image_path)
|
||
copy_path = image_dir / f"{doc_id}{full_path.suffix.lower()}"
|
||
copy_path.write_bytes(data)
|
||
|
||
# The non-markdown title rule (the image's content is prose, but the
|
||
# doc IS the image — the file stem is the title).
|
||
title = full_path.stem
|
||
if doc is None:
|
||
doc = Document(
|
||
id=doc_id,
|
||
source=source,
|
||
path=rel,
|
||
full_path=str(full_path),
|
||
title=title,
|
||
content=content,
|
||
content_hash=digest,
|
||
indexed_at=datetime.now(UTC),
|
||
created_at=normalize_doc_date(raw_date),
|
||
is_image=True,
|
||
image_path=str(copy_path),
|
||
# Phase 125 (task 03, LOCKED A1): the referencing markdown's
|
||
# alt text + surrounding paragraphs (``None`` for a
|
||
# standalone image — the pre-phase row shape).
|
||
image_context=image_context,
|
||
)
|
||
session.add(doc)
|
||
else:
|
||
doc.full_path = str(full_path)
|
||
doc.title = title
|
||
doc.content = content
|
||
doc.content_hash = digest
|
||
doc.indexed_at = datetime.now(UTC)
|
||
# Phase 106 (D4): a content change is a new document version —
|
||
# the date is re-sourced and a previous manual correction is
|
||
# reset (it referred to the old content).
|
||
doc.created_at = normalize_doc_date(raw_date)
|
||
doc.created_at_manual = False
|
||
doc.is_image = True
|
||
doc.image_path = str(copy_path)
|
||
# Phase 125 (task 03): a re-indexed image re-scans its markdown
|
||
# references — the context follows the current source state
|
||
# (a deleted referencing markdown drops it back to ``None``).
|
||
doc.image_context = image_context
|
||
|
||
session.flush() # guarantees doc.id even for brand-new rows
|
||
|
||
# Phase 1+2 — the same pipeline on the description (chunk, replace
|
||
# the chunk rows (embeddings NULL), embed, commit — one transaction
|
||
# per file; the endpoint token-cap retry lives in the shared helper,
|
||
# and a description is short, so it never fires in practice).
|
||
# Phase 129 (task 02): the in-flight file's chunk-position hook
|
||
# rides this embed (one description chunk → ``1/1``).
|
||
chunks_n = await _replace_content_chunks(
|
||
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
|
||
)
|
||
session.commit()
|
||
|
||
if verb == "added":
|
||
summary.added += 1
|
||
else:
|
||
summary.updated += 1
|
||
summary.changed_paths.append((source, rel))
|
||
summary.chunks += chunks_n
|
||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||
|
||
# Phase 30 shape on the description — image-aware (task 03, LOCKED
|
||
# A3): the description IS the summary (stored verbatim, no
|
||
# ``lite`` call), so ``doc.summary`` == ``doc.content`` and the
|
||
# ``is_summary`` position −1 chunk mirrors it.
|
||
await _store_summary(
|
||
session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
|
||
)
|
||
|
||
|
||
#: The synthetic path marker of a PDF-embedded image doc (phase 126,
|
||
#: LOCKED A3 — ``<pdf path>#page=<N>#img=<index>.<ext>``):
|
||
#: :data:`_EMBEDDED_IMAGE_RE` is the prune's parent rule below.
|
||
_EMBEDDED_IMAGE_RE = re.compile(r"#page=\d+#img=\d+")
|
||
|
||
|
||
def _prune(
|
||
session: Session,
|
||
source_names: set[str],
|
||
seen: set[tuple[str, str]],
|
||
images: bool = False,
|
||
summary: ImportSummary | None = None,
|
||
) -> int:
|
||
"""Delete documents of *source_names* whose file is no longer in *seen*.
|
||
|
||
*images* (phase 122 prune guard, LOCKED, derived from A3/A4): while
|
||
the image toggle is OFF (``images`` False), every ``is_image`` doc is
|
||
SKIPPED — the image is invisible to an images-off walk, not a deleted
|
||
file, so pruning it would silently destroy image documents on the
|
||
first images-off sync. Toggle ON → normal semantics: a deleted image
|
||
file prunes its doc, and the pruned image's ``image_dir`` copy is
|
||
deleted with it.
|
||
|
||
Phase 126 (LOCKED A3) — the PDF-embedded image's parent rule: an
|
||
embedded image doc's synthetic path (``#page=<N>#img=<index>``)
|
||
NEVER appears in the walk's *seen* — the image is derived from its
|
||
parent PDF, whose on-disk file is the walked file. Pruning it on the
|
||
plain rule would destroy it in the very sync that (re)indexes it,
|
||
so it is skipped while its parent PDF's rel is still in *seen* (the
|
||
image is invisible to the walk, not deleted). A pruned/absent parent
|
||
PDF prunes its embedded image docs with it (row + ``image_dir`` copy
|
||
— no orphans when the PDF leaves the source). The rule holds in BOTH
|
||
toggle states (toggle off, the phase-122 guard already skips every
|
||
``is_image`` doc; this one is the toggle-on half of the same
|
||
"invisible, not deleted" contract).
|
||
|
||
*summary* (phase 127, task 01): when given, every document that is
|
||
ACTUALLY deleted (i.e. not skipped by either guard above) records
|
||
its ``(doc.source, doc.path)`` on ``summary.changed_paths`` —
|
||
appended BEFORE the ``session.delete`` — so the sync's incremental
|
||
folder-summary regeneration (phase 127) sees the pruned docs' folders
|
||
too. Callers passing nothing keep the pre-phase count-only contract.
|
||
"""
|
||
if not source_names:
|
||
return 0
|
||
pruned = 0
|
||
docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()
|
||
for doc in docs:
|
||
if (doc.source, doc.path) not in seen:
|
||
if doc.is_image and not images:
|
||
# Prune guard: invisible to the walk, not deleted.
|
||
continue
|
||
if doc.is_image and _EMBEDDED_IMAGE_RE.search(doc.path):
|
||
# Phase 126 (A3): the parent PDF's rel is the path minus
|
||
# its ``#page=…#img=…`` fragment — the walked file.
|
||
parent_rel = doc.path.split("#", 1)[0]
|
||
if (doc.source, parent_rel) in seen:
|
||
# Parent PDF still in the source — the image is
|
||
# invisible to the walk, not deleted.
|
||
continue
|
||
_delete_image_copy(doc.image_path)
|
||
if summary is not None:
|
||
summary.changed_paths.append((doc.source, doc.path))
|
||
session.delete(doc)
|
||
pruned += 1
|
||
logger.info("import: pruned source=%s path=%s", doc.source, doc.path)
|
||
if pruned:
|
||
session.commit()
|
||
return pruned
|