phase: 147_sync_changed_only
All defaults match. Verification complete — everything is green with no defects found. **Phase 147 final verification pass — report** - All 3 task files found in `complete/147_sync_changed_only/` → verification pass only; no code changes needed. - Verified: migration 0025 (`documents.file_hash` nullable String(64), no index) + model field; fast path at top of `_index_file` (streamed 1 MiB sha256 → row lookup → `_unchanged_doc`); stamping on all three branches' create/update + shared helper; pre-phase NULL-hash fallthrough; retriever/`content_hash` untouched. - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3438 passed, `app/` coverage **99%** (>90% ✓); `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` → 3 passed in isolation ✓; `uv run ruff check . && uv run pyright` → all checks passed, 0 errors ✓; prior-phase `tests/e2e/test_sync_button.py` → 3 passed (no regression) ✓. - **Completion criteria:** all 7 pass — (1) column + round-trip ✓ (2) unchanged text/image/garbage-PDF/pictorial-PDF cost zero LLM/extraction/vision, `summary.unchanged` counted, rows byte-still ✓ (3) changed file re-indexes + re-stamps; bytes-changed/content-same stamps only ✓ (4) backfill/D1/D4 + log wordings pinned verbatim ✓ (5) `content_hash`/retriever byte-identical ✓ (6) all gates green ✓ (7) no prior-phase behavior change ✓. - **Documentation:** full review of README.md (1,031 lines) and AGENTS.md against the code: every `.env.example`/code `BOR_*` var documented, all config defaults match `app/config.py`, quick-start/QA commands and referenced test files exist, `file://` URL + troubleshooting claims accurate; the phase's README "Updating the documents" edit is accurate. **Verified consistent — no changes needed.** - **Notable:** no live-infrastructure changes (dev Postgres container used for the standard test gates only). Working tree left uncommitted per harness protocol. - **Next pending phase:** `148_pdf_pages_in_sources`.
This commit is contained in:
@@ -110,6 +110,21 @@ class Document(Base):
|
||||
title: Mapped[str] = mapped_column(String(500))
|
||||
content: Mapped[str] = mapped_column(Text) # full markdown — the RAG context
|
||||
content_hash: Mapped[str] = mapped_column(String(64), index=True) # sha256 for change detection
|
||||
#: The sha256 of the RAW SOURCE FILE's bytes at index time (phase
|
||||
#: 147) — the sync's change signal, compared at the top of the
|
||||
#: importer's ``_index_file`` BEFORE the image/text/PDF dispatch
|
||||
#: (an unchanged file costs one streamed hash + one lookup — no
|
||||
#: extraction, no vision, no embed). NULL for pre-phase rows and
|
||||
#: until the first post-upgrade sync stamps it (no migration
|
||||
#: backfill is possible — a migration cannot see the source files
|
||||
#: on disk). DISTINCT from ``content_hash`` — the FINAL indexed
|
||||
#: content's digest (a pictorial PDF's includes vision text;
|
||||
#: ``content_hash`` is exposed via ``app/rag/retriever.py`` and is
|
||||
#: the dedupe/caching identity — untouched). No index: the fast
|
||||
#: path compares it in Python after the unique ``(source, path)``
|
||||
#: lookup — internal bookkeeping, never a query surface (unlike
|
||||
#: ``content_hash``).
|
||||
file_hash: Mapped[str | None] = mapped_column(String(64), default=None)
|
||||
indexed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
#: The document's CREATION date (phase 106, D1/D2/D3) — sourced at
|
||||
#: sync time (git last-commit date for git sources, file mtime for
|
||||
|
||||
+300
-153
@@ -59,6 +59,32 @@ backfill runs BEFORE the ``created_at_manual`` early-return (the manual
|
||||
flag protects the DATE only, D1) and the strict ``is None`` check leaves
|
||||
owner-set summaries (even empty strings, phase 57) alone.
|
||||
|
||||
Unchanged fast path (phase 147, task 02): the sync's single change
|
||||
signal is the sha256 of the RAW source file's bytes
|
||||
(:func:`_file_hash` — streamed in 1 MiB chunks, a large PDF is never
|
||||
loaded fully), computed ONCE at the top of :func:`_index_file` BEFORE
|
||||
the image/text/PDF dispatch and compared against
|
||||
``documents.file_hash`` (migration 0025 — nullable, no index, internal
|
||||
bookkeeping; DISTINCT from ``content_hash``, the FINAL indexed
|
||||
content's digest, which stays the retriever's dedupe/caching identity).
|
||||
A match returns through :func:`_unchanged_doc` — the ONE shared
|
||||
unchanged path (the three branches' near-identical unchanged blocks,
|
||||
one code site): the ``unchanged`` count + log line, the phase-118 (A2)
|
||||
summary-NULL backfill from the STORED ``doc.content`` (never a
|
||||
re-read/re-extract/re-describe — the digest-equality invariant), the
|
||||
phase-106 (D1) manual-date early-return, and the (D4) date refresh
|
||||
(may go older, counts ``dates_updated``) — with ZERO file read beyond
|
||||
the streamed hash, ZERO PDF extraction, ZERO vision call, and ZERO
|
||||
embed. Every create/update STAMPS ``file_hash`` (all three branches),
|
||||
and the shared helper stamps too (a pre-phase NULL row, or a
|
||||
bytes-changed/content-same file whose content digest still matches),
|
||||
so the NEXT sync is always fast. Pre-phase rows (``file_hash`` NULL)
|
||||
fall through to the branches' content-digest logic exactly as
|
||||
pre-phase on the first post-upgrade sync (one full pass), then stamp
|
||||
and go fast. The previous re-burn costs (a garbage PDF's fallback
|
||||
vision re-issued each sync, a non-deterministic vision model flipping
|
||||
a pictorial PDF's content digest) are gone: the file bytes decide.
|
||||
|
||||
Standalone images (phase 122, LOCKED A3): with the ``images`` toggle
|
||||
(``BOR_IMAGES``) ON, the walk also admits the image extension set
|
||||
(``BOR_IMAGE_EXTENSIONS`` — a SEPARATE set from ``import_extension_set``;
|
||||
@@ -123,11 +149,11 @@ call, the SAME ``---PAGE-N---`` separators, the loud
|
||||
class of PDF whose text layer is garbage. A CLEAN text layer NEVER
|
||||
burns a fallback vision call (the gate is the only gate — the
|
||||
phase-126 happy path is byte-identical), and the digest runs over the
|
||||
FINAL content exactly once — for a stable file + stable vision model
|
||||
the digest is byte-stable, so an unchanged re-sync re-imports the row
|
||||
as unchanged (no re-chunk/re-embed/re-summary, no KB change; the
|
||||
vision calls are re-issued each sync — the no-cache cost, documented
|
||||
in :func:`_fallback_page_screenshots`).
|
||||
FINAL content exactly once — and an UNCHANGED re-sync is caught by the
|
||||
phase-147 fast path BEFORE this pipeline runs at all (the raw file
|
||||
bytes are the change signal: no re-extract, no re-render, no
|
||||
re-issued vision call, no re-chunk/re-embed/re-summary, no KB change —
|
||||
see :func:`_index_file`).
|
||||
|
||||
Pictorial sparse pages (phase 140, task 03, LOCKED A1–A4): AFTER the
|
||||
phase-133 gate, a PDF whose text layer is USABLE gets the sparse-page
|
||||
@@ -557,6 +583,33 @@ def iter_importable_files(
|
||||
return files
|
||||
|
||||
|
||||
#: The streamed-hash read size (phase 147, task 02) — 1 MiB: a
|
||||
#: multi-hundred-MB PDF is hashed in a bounded stream, never loaded
|
||||
#: fully into memory.
|
||||
_HASH_CHUNK_SIZE = 1024 * 1024
|
||||
|
||||
|
||||
def _file_hash(path: Path) -> str:
|
||||
"""Phase 147 (task 02): the sha256 of *path*'s RAW bytes, streamed
|
||||
in 1 MiB chunks (the :data:`_HASH_CHUNK_SIZE` loop — a large file
|
||||
is never loaded fully into memory; the ``hashlib`` incremental
|
||||
pattern).
|
||||
|
||||
The sync's SINGLE change signal — computed ONCE at the top of
|
||||
:func:`_index_file`, before the image/text/PDF dispatch, and
|
||||
compared against the stamped ``documents.file_hash`` (the fast
|
||||
path). Raw bytes on purpose (the owner's "hash the document
|
||||
contents"): every class hashes the same way — images, text, PDFs
|
||||
alike — and a bytes-identical file is an unchanged file, whatever
|
||||
its extraction/vision pipeline would produce.
|
||||
"""
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
while chunk := f.read(_HASH_CHUNK_SIZE):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
async def import_sources(
|
||||
sources: list[Path],
|
||||
llm: Embedder,
|
||||
@@ -876,6 +929,100 @@ async def _replace_content_chunks(
|
||||
return len(chunks_text)
|
||||
|
||||
|
||||
async def _unchanged_doc(
|
||||
session: Session,
|
||||
doc: Document,
|
||||
*,
|
||||
source: str,
|
||||
rel: str,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime,
|
||||
file_digest: str,
|
||||
) -> None:
|
||||
"""Phase 147 (task 02): the ONE shared unchanged path — the three
|
||||
branches' near-identical unchanged blocks, one code site, reached
|
||||
from two places:
|
||||
|
||||
* the FAST PATH at the top of :func:`_index_file` (a
|
||||
``doc.file_hash == file_digest`` match — the raw bytes are what
|
||||
was indexed last time): zero file read beyond the streamed
|
||||
hash, zero extraction, zero vision, zero embed;
|
||||
* each branch's CONTENT-digest hit (a pre-phase NULL-hash row, or
|
||||
a bytes-changed/content-same file whose final content digest
|
||||
still matches): the branch's own expensive work already ran,
|
||||
but the row is skipped exactly as pre-phase.
|
||||
|
||||
The preserved semantics, in order (the per-branch blocks' counts,
|
||||
log wordings, and commits — verbatim):
|
||||
|
||||
1. ``summary.unchanged`` + the ``import: unchanged`` log line;
|
||||
2. the STAMP: ``doc.file_hash = file_digest`` (a pre-phase NULL
|
||||
row — or a bytes-changed/content-same row — learns its hash
|
||||
here, so the NEXT sync takes the fast path; a no-op on the
|
||||
fast path, where the DB value already equals the digest);
|
||||
3. the phase-118 (A2) backfill when ``doc.summary is None`` —
|
||||
from the STORED ``doc.content``: EXACTLY what the file's bytes
|
||||
produced is already stored (text: the file text; image: the
|
||||
vision description; PDF: the final constructed content — the
|
||||
digest-equality invariant), so the backfill never re-reads the
|
||||
file, never re-extracts, never re-describes (strict
|
||||
``is None`` — an owner-set empty string, phase 57, is never
|
||||
touched);
|
||||
4. the phase-106 (D1) early-return — ``created_at_manual``
|
||||
protects the DATE: no date write, the manual correction
|
||||
survives (the phase-97 ``manually_edited`` precedent); the
|
||||
stamp, if dirty, is the only write, and it commits so the next
|
||||
sync is fast even for a manual-date row;
|
||||
5. the phase-106 (D4) date refresh — ``normalize_doc_date`` of
|
||||
the raw date, committed + counted in ``dates_updated`` + the
|
||||
``import: date-refreshed`` line when it changes (it may go
|
||||
OLDER — no monotonic guard);
|
||||
6. the stamp's final commit — when the row is still dirty after
|
||||
the date block (the stamp set ``file_hash`` and the date did
|
||||
not change): quiet bookkeeping — no count, no log (the
|
||||
``unchanged`` line already fired; a date-change run already
|
||||
committed in step 5).
|
||||
"""
|
||||
summary.unchanged += 1
|
||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||
# 2 — the stamp (a no-op on the fast path; a real write on the
|
||||
# branch paths — the next sync is then fast either way).
|
||||
doc.file_hash = file_digest
|
||||
# 3 — the phase-118 (A2) summary-NULL backfill, from the stored
|
||||
# content (see the docstring — never a re-read/re-extract).
|
||||
if doc.summary is None:
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=doc.content,
|
||||
llm=llm, summary=summary, backfill=True,
|
||||
)
|
||||
# 4 — the phase-106 (D1) manual-date lock: no date write; the
|
||||
# stamp, if dirty, is the only write (committed — see step 2's
|
||||
# "next sync takes the fast path").
|
||||
if doc.created_at_manual:
|
||||
if session.is_modified(doc):
|
||||
session.commit()
|
||||
return
|
||||
# 5 — the phase-106 (D4) date refresh: the stored date follows the
|
||||
# source on every sync and may go OLDER (no monotonic guard); a
|
||||
# date-only refresh counts ``unchanged`` — never added/updated/
|
||||
# pruned, so no ``sources_meta`` bump and no regeneration.
|
||||
target = normalize_doc_date(raw_date)
|
||||
if target != doc.created_at:
|
||||
doc.created_at = target
|
||||
session.commit()
|
||||
summary.dates_updated += 1
|
||||
logger.info(
|
||||
"import: date-refreshed source=%s path=%s date=%s",
|
||||
source, rel,
|
||||
doc.created_at.isoformat(),
|
||||
)
|
||||
elif session.is_modified(doc):
|
||||
# 6 — the stamp's commit (the date did not move): quiet
|
||||
# bookkeeping — the ``unchanged`` line already fired.
|
||||
session.commit()
|
||||
|
||||
|
||||
async def _index_file(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -889,6 +1036,23 @@ async def _index_file(
|
||||
) -> None:
|
||||
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
|
||||
|
||||
Phase 147 (task 02): the UNCHANGED FAST PATH. BEFORE the
|
||||
image/text/PDF dispatch, the file's RAW source bytes are hashed
|
||||
(:func:`_file_hash` — streamed), the ``(source, path)`` row is
|
||||
looked up, and the D2 mtime fallback for *raw_date* runs (once,
|
||||
here — the branches receive the already-computed date) — a
|
||||
``doc.file_hash == file_digest`` match returns through the shared
|
||||
unchanged path (:func:`_unchanged_doc`) with ZERO file read beyond
|
||||
the streamed hash, ZERO PDF extraction, ZERO vision call, and
|
||||
ZERO embed (the preserved bookkeeping: the phase-118 summary-NULL
|
||||
backfill from the STORED ``doc.content``, the phase-106 D1
|
||||
manual-date lock, the D4 date refresh). A pre-phase row
|
||||
(``file_hash`` NULL) or a bytes-changed file falls through to the
|
||||
branches below, which keep their own lookups + content-digest
|
||||
checks (the content digest then decides changed vs. the same
|
||||
shared path) and STAMP ``file_hash`` on every create/update — so
|
||||
the next sync is always fast.
|
||||
|
||||
``raw_date`` (phase 106, D2) is the file's RAW source date — the
|
||||
git last-commit datetime from the caller's ``doc_dates_by_root``
|
||||
map, or ``None`` (every non-git case): the file's mtime is read
|
||||
@@ -909,68 +1073,66 @@ async def _index_file(
|
||||
any text read: ``read_text`` is never called for an image.
|
||||
"""
|
||||
settings = llm.settings
|
||||
# Phase 147 (task 02): the unchanged fast path — the RAW source-
|
||||
# file-bytes digest is the single change signal, computed ONCE
|
||||
# here, before the image/text/PDF dispatch, and compared against
|
||||
# the stamped ``documents.file_hash``. A match returns through the
|
||||
# shared unchanged path with zero further work — no
|
||||
# ``read_text``/``read_bytes``, no ``extract_pdf_text``, no
|
||||
# fallback/pictorial vision, no re-chunk/re-embed/re-summary.
|
||||
file_digest = _file_hash(full_path)
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
if raw_date is None:
|
||||
# D2 fallback: no source date in the map → the file's mtime
|
||||
# (one stat). Read before the unchanged early-return — the
|
||||
# fast path (and the branches' unchanged path) refresh the
|
||||
# stored date from the same source.
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
if doc is not None and doc.file_hash == file_digest:
|
||||
# The fast path: the raw bytes are what was indexed last time
|
||||
# — the bookkeeping (backfill / D1 / D4) runs in the shared
|
||||
# helper, nothing else does.
|
||||
await _unchanged_doc(
|
||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||
raw_date=raw_date, file_digest=file_digest,
|
||||
)
|
||||
return
|
||||
# Phase 122 (task 02): the image branch FIRST. Only reachable while
|
||||
# the ``images`` toggle is on — the walk never admits image files
|
||||
# while it is off, and with it off this check is a no-op (the text
|
||||
# path below stays byte-identical to pre-phase).
|
||||
# path below stays byte-identical to pre-phase). Phase 147: this
|
||||
# point is reached only for a pre-phase NULL-hash row or a
|
||||
# bytes-changed file — the branch's own digest then decides.
|
||||
if settings.images:
|
||||
image_set = settings.image_extension_set
|
||||
if match_extension(full_path, image_set) is not None:
|
||||
return await _index_image_file(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||||
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
|
||||
summary=summary, raw_date=raw_date, file_digest=file_digest,
|
||||
on_chunk=on_chunk,
|
||||
)
|
||||
# Phase 126: PDF files take the PDF index path — AFTER the image
|
||||
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
|
||||
# owns it and ``read_text`` is never called for one).
|
||||
# owns it and ``read_text`` is never called for one). Phase 147:
|
||||
# reached only for a pre-phase NULL-hash row or a bytes-changed
|
||||
# file — the branch's own digest then decides.
|
||||
if full_path.suffix.lower() == ".pdf":
|
||||
return await _index_pdf_file(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||||
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
|
||||
summary=summary, raw_date=raw_date, file_digest=file_digest,
|
||||
on_chunk=on_chunk,
|
||||
)
|
||||
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||||
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
if raw_date is None:
|
||||
# D2 fallback: no source date in the map → the file's mtime
|
||||
# (one stat). Read before the unchanged early-return — the
|
||||
# unchanged path refreshes the stored date from the same source.
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
summary.unchanged += 1
|
||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||
# Phase 118 (A2): an unchanged doc whose summary is still NULL
|
||||
# (a pre-phase-30 row, or an earlier fail-soft miss) gets a
|
||||
# summary-only backfill — one ``is_summary`` chunk, no content
|
||||
# re-embed, and NEVER an added/updated/pruned count (so no
|
||||
# ``sources_meta`` bump, no overview/folder-summary
|
||||
# regeneration). Strict ``is None``: an empty-string summary is
|
||||
# owner-set (phase 57) and is never overwritten. BEFORE the
|
||||
# manual-date early-return: ``created_at_manual`` protects the
|
||||
# DATE only (phase 106, D1), not the summary.
|
||||
if doc.summary is None:
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=content,
|
||||
llm=llm, summary=summary, backfill=True,
|
||||
)
|
||||
if doc.created_at_manual:
|
||||
# D1/D4: the owner's correction survives the sync — no
|
||||
# write at all (the phase-97 ``manually_edited`` precedent).
|
||||
return
|
||||
# D4: the date refreshes on every sync, including unchanged
|
||||
# files, and may go OLDER (no monotonic guard). A date-only
|
||||
# refresh is still counted ``unchanged`` — never added/updated/
|
||||
# pruned, so no ``sources_meta`` bump and no regeneration.
|
||||
target = normalize_doc_date(raw_date)
|
||||
if target != doc.created_at:
|
||||
doc.created_at = target
|
||||
session.commit()
|
||||
summary.dates_updated += 1
|
||||
logger.info(
|
||||
"import: date-refreshed source=%s path=%s date=%s",
|
||||
source, rel,
|
||||
doc.created_at.isoformat(),
|
||||
)
|
||||
# Phase 147: the shared unchanged path — the pre-phase
|
||||
# content-digest hit (a pre-phase NULL-hash row, or a
|
||||
# bytes-changed/content-same file). The helper stamps the new
|
||||
# file hash, so the NEXT sync takes the fast path above.
|
||||
await _unchanged_doc(
|
||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||
raw_date=raw_date, file_digest=file_digest,
|
||||
)
|
||||
return
|
||||
|
||||
verb = "updated" if doc is not None else "added"
|
||||
@@ -988,6 +1150,9 @@ async def _index_file(
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash=digest,
|
||||
# Phase 147: the raw-bytes stamp (the next sync's fast
|
||||
# path).
|
||||
file_hash=file_digest,
|
||||
indexed_at=datetime.now(UTC),
|
||||
# Phase 106 (D2/D3): the sourced creation date, normalized
|
||||
# (undetermined or future → today). ``created_at_manual``
|
||||
@@ -1001,6 +1166,8 @@ async def _index_file(
|
||||
doc.title = title
|
||||
doc.content = content
|
||||
doc.content_hash = digest
|
||||
# Phase 147: re-stamp (the file's bytes changed).
|
||||
doc.file_hash = file_digest
|
||||
doc.indexed_at = datetime.now(UTC)
|
||||
# Phase 106 (D4): a content change is a new document version —
|
||||
# the date is re-sourced and a previous manual correction is
|
||||
@@ -1186,7 +1353,8 @@ async def _index_pdf_file(
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
file_digest: str,
|
||||
raw_date: datetime,
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> None:
|
||||
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
|
||||
@@ -1221,12 +1389,13 @@ async def _index_pdf_file(
|
||||
vision call).
|
||||
* the sha256 digest is over the FINAL content (the digest rule is
|
||||
content identity — the same text is the same document version;
|
||||
the fallback's constructed page descriptions digest exactly the
|
||||
same way, so an unchanged fallback doc re-syncs as unchanged —
|
||||
no re-chunk/re-embed/re-summary, no KB change — while the vision
|
||||
model's replies stay stable; a non-deterministic vision model
|
||||
re-describes the row each sync, a cost, not a bug; the empty
|
||||
case digests the empty byte string).
|
||||
the empty case digests the empty byte string). Phase 147: the
|
||||
UNCHANGED detection itself is the caller's RAW-BYTES file hash
|
||||
(the fast path — :func:`_index_file`): an unchanged PDF never
|
||||
re-extracts, never re-renders, and never re-issues a vision
|
||||
call (the previous "non-deterministic vision model re-describes
|
||||
the row each sync" cost is gone — a re-describe now happens
|
||||
only when the file's BYTES change).
|
||||
* the title is the metadata title or the file stem
|
||||
(:func:`_extract_pdf_title`); the date follows the text path
|
||||
exactly (the phase-106 D2 mtime fallback — PDFs carry no
|
||||
@@ -1237,15 +1406,23 @@ async def _index_pdf_file(
|
||||
retry lives there), and the phase-30/118 (A2) ``lite`` summary
|
||||
path runs on the text — PDFs are no more excluded than any other
|
||||
non-markdown format.
|
||||
* the unchanged path (same text digest) skips everything — no
|
||||
re-extract, no re-embed, no image pass — and runs the same
|
||||
backfill/manual-date/date-refresh dance as the text path.
|
||||
* the UNCHANGED path (phase 147): the fast path above catches an
|
||||
unchanged file before any of this runs; a pre-phase NULL-hash
|
||||
row, or a bytes-changed/content-same PDF, that reaches the
|
||||
content-digest hit below runs the SAME shared path
|
||||
(:func:`_unchanged_doc`) — no re-extract, no re-embed, no image
|
||||
pass, the backfill/manual-date/date-refresh semantics in one
|
||||
code site — and the helper stamps the new file hash (the next
|
||||
sync takes the fast path).
|
||||
* THEN the embedded images: :func:`_index_pdf_embedded_images`
|
||||
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
|
||||
|
||||
``raw_date`` follows the text path: the caller's git last-commit
|
||||
date or ``None`` (the mtime fallback, read before the unchanged
|
||||
early-return — the unchanged path refreshes from the same source).
|
||||
``raw_date`` is the caller's git last-commit date, or the file's
|
||||
mtime — :func:`_index_file` computes the D2 fallback (and reads
|
||||
it before the fast-path check) — and the unchanged path
|
||||
refreshes from the same source. ``file_digest`` is the caller's
|
||||
raw-bytes sha256 (phase 147 — the fast-path signal and this
|
||||
branch's stamp value).
|
||||
|
||||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||||
chunk-position hook: it is forwarded to the PDF's CONTENT embed
|
||||
@@ -1294,40 +1471,17 @@ async def _index_pdf_file(
|
||||
else hashlib.sha256(b"").hexdigest()
|
||||
)
|
||||
title = _extract_pdf_title(full_path, fallback=full_path.stem)
|
||||
if raw_date is None:
|
||||
# D2 fallback (same as the text path): no source date in the
|
||||
# map → the file's mtime (one stat). Read before the unchanged
|
||||
# early-return — the unchanged path refreshes from it.
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
# Unchanged PDF (text digest) — the text path's unchanged branch,
|
||||
# unmodified in shape: no re-extract, no re-embed, no image pass.
|
||||
summary.unchanged += 1
|
||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||
# Phase 118 (A2): an unchanged doc whose summary is still NULL
|
||||
# gets the same best-effort summary pass (see the text path).
|
||||
if doc.summary is None:
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=content,
|
||||
llm=llm, summary=summary, backfill=True,
|
||||
)
|
||||
if doc.created_at_manual:
|
||||
# D1/D4: the owner's correction survives the sync — no write
|
||||
# at all (the text path's manual-date early-return).
|
||||
return
|
||||
# D4: the date refreshes on every sync and may go OLDER (no
|
||||
# monotonic guard); a date-only refresh counts unchanged.
|
||||
target = normalize_doc_date(raw_date)
|
||||
if target != doc.created_at:
|
||||
doc.created_at = target
|
||||
session.commit()
|
||||
summary.dates_updated += 1
|
||||
logger.info(
|
||||
"import: date-refreshed source=%s path=%s date=%s",
|
||||
source, rel, doc.created_at.isoformat(),
|
||||
)
|
||||
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
||||
# row, or a bytes-changed/content-same PDF (the content digest
|
||||
# still matches). No re-extract, no re-embed, no image pass;
|
||||
# the helper stamps the new file hash (the next sync takes the
|
||||
# fast path).
|
||||
await _unchanged_doc(
|
||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||
raw_date=raw_date, file_digest=file_digest,
|
||||
)
|
||||
return
|
||||
|
||||
verb = "updated" if doc is not None else "added"
|
||||
@@ -1339,6 +1493,9 @@ async def _index_pdf_file(
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash=digest,
|
||||
# Phase 147: the raw-bytes stamp (the next sync's fast
|
||||
# path).
|
||||
file_hash=file_digest,
|
||||
indexed_at=datetime.now(UTC),
|
||||
created_at=normalize_doc_date(raw_date),
|
||||
)
|
||||
@@ -1348,6 +1505,8 @@ async def _index_pdf_file(
|
||||
doc.title = title
|
||||
doc.content = content
|
||||
doc.content_hash = digest
|
||||
# Phase 147: re-stamp (the file's bytes changed).
|
||||
doc.file_hash = file_digest
|
||||
doc.indexed_at = datetime.now(UTC)
|
||||
# Phase 106 (D4): a content change is a new document version —
|
||||
# the date is re-sourced and a manual correction reset.
|
||||
@@ -1428,18 +1587,18 @@ async def _fallback_page_screenshots(
|
||||
or every page a placeholder) returns ``""``: the existing
|
||||
empty-content path (the doc lands as a zero-text doc, the
|
||||
phase-126 no-extractable-text behavior).
|
||||
* the digest the caller computes is over the constructed content —
|
||||
for a stable file + stable vision model the content (and thus
|
||||
the digest) is byte-stable across re-syncs, so the row
|
||||
re-imports as UNCHANGED: no re-chunk, no re-embed, no
|
||||
re-summary, no KB change. The fallback itself RE-RUNS on every
|
||||
sync of a garbage PDF (the gate is a cheap pure predicate and
|
||||
the render needs no LLM; the vision calls are re-issued before
|
||||
the unchanged check can skip them — a documented cost of the
|
||||
no-cache design, bounded by the page count of the garbage
|
||||
PDFs). If the production vision model's reply is
|
||||
non-deterministic, the row is re-described each sync (updated —
|
||||
a cost, not a bug; the doc is still correct).
|
||||
* the digest the caller computes is over the constructed content
|
||||
— and phase 147's fast path makes a fallback re-run moot for an
|
||||
unchanged file: the raw BYTES are the change signal, so an
|
||||
unchanged garbage PDF is caught at the top of :func:`_index_file`
|
||||
BEFORE the gate runs at all (no re-extract, no re-render, no
|
||||
re-issued vision call — the owner's "really long sync times for
|
||||
PDFs that are mostly pictoral" complaint, fixed). The fallback's
|
||||
vision cost now applies to a garbage PDF's FIRST index — and to
|
||||
a re-index only when the file's bytes change; a
|
||||
non-deterministic production vision model no longer flips an
|
||||
unchanged row's content digest (the row stays put until the
|
||||
bytes move; the doc is still correct either way).
|
||||
|
||||
*session* is part of the signature for symmetry with the other
|
||||
per-file index helpers (the fallback itself needs no DB access —
|
||||
@@ -1520,10 +1679,12 @@ async def _pictorial_pdf_pass(
|
||||
The assembly with NO descriptions is byte-identical to
|
||||
:func:`extract_pdf_text`'s output (the unit pin).
|
||||
* the caller's digest runs over this FINAL content (the
|
||||
phase-126/133 convention — A4): a stable file + a stable vision
|
||||
reply re-syncs as ``unchanged``; a non-deterministic re-describe
|
||||
re-updates the row — a cost, not a bug (the phase-133 docstring
|
||||
documents it, A21 endorses it).
|
||||
phase-126/133 convention — A4) — and phase 147's fast path
|
||||
catches an unchanged file BEFORE the pass runs at all (the raw
|
||||
bytes are the change signal): no re-render, no re-issued vision
|
||||
call, and a non-deterministic vision model no longer re-updates
|
||||
an unchanged row — the pass burns vision only on a file's first
|
||||
index and on byte changes.
|
||||
* stats (log-only — the sync ``detail`` API shape is untouched,
|
||||
the phase-133 convention): ``summary.pdfs_pictorial`` (+1 when
|
||||
≥1 page was described) and ``summary.pdf_pictorial_pages``
|
||||
@@ -2647,7 +2808,8 @@ async def _index_image_file(
|
||||
full_path: Path,
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
file_digest: str,
|
||||
raw_date: datetime,
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> None:
|
||||
"""The phase-122 image branch of :func:`_index_file` — a standalone
|
||||
@@ -2666,11 +2828,14 @@ async def _index_image_file(
|
||||
path running on it (the ``is_summary`` position −1 chunk mirrors
|
||||
``Document.summary``).
|
||||
|
||||
``raw_date`` follows the text path exactly (the phase-106 D2
|
||||
fallback: no source date in the map → the file's mtime, read before
|
||||
the unchanged early-return because the unchanged path refreshes the
|
||||
stored date from the same source; the D1 manual-date lock and the
|
||||
D4 refresh apply unmodified).
|
||||
``raw_date`` follows the text path exactly — the CALLER
|
||||
(:func:`_index_file`) computes the phase-106 D2 fallback (no
|
||||
source date in the map → the file's mtime) before the fast-path
|
||||
check, because the unchanged path refreshes the stored date from
|
||||
the same source; the D1 manual-date lock and the D4 refresh apply
|
||||
unmodified through the shared :func:`_unchanged_doc`. ``file_digest``
|
||||
is the caller's raw-bytes sha256 (phase 147 — the fast-path
|
||||
signal and this branch's stamp value).
|
||||
|
||||
Fail-soft (LOCKED A3): a failed/empty description SKIPS the doc
|
||||
entirely (no row, no copy) — ``summary.images_failed`` + a warning,
|
||||
@@ -2698,48 +2863,25 @@ async def _index_image_file(
|
||||
byte-identical.
|
||||
"""
|
||||
settings = llm.settings
|
||||
# Phase 147: the fast path (:func:`_index_file`) already returned
|
||||
# for an unchanged file — everything below (``read_bytes``, the
|
||||
# vision describe, the copy) runs only on the first index or a
|
||||
# byte change.
|
||||
data = full_path.read_bytes()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
if raw_date is None:
|
||||
# D2 fallback (same as the text path): no source date in the map
|
||||
# → the file's mtime (one stat).
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
# Unchanged image (byte digest) — the text path's unchanged
|
||||
# branch, unmodified in shape.
|
||||
summary.unchanged += 1
|
||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||
# Phase 118 (A2) backfill, image flavour: an unchanged image doc
|
||||
# whose summary is still NULL (an earlier fail-soft summary miss
|
||||
# — for an image, the one remaining failure class: the summary
|
||||
# chunk's embed) gets the same best-effort summary pass. For an
|
||||
# image the summary IS the stored description (``doc.content``),
|
||||
# so the image-aware ``_store_summary`` (task 03) re-stores it
|
||||
# verbatim with one ``is_summary`` chunk; a failure (the
|
||||
# embed) keeps the doc as-is (no row mutation) — fail-soft,
|
||||
# same as the text path.
|
||||
if doc.summary is None:
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=doc.content,
|
||||
llm=llm, summary=summary, backfill=True,
|
||||
)
|
||||
if doc.created_at_manual:
|
||||
# D1/D4: the owner's correction survives the sync — no write
|
||||
# at all (the text path's manual-date early-return).
|
||||
return
|
||||
# D4: the date refreshes on every sync, including unchanged
|
||||
# files, and may go OLDER (no monotonic guard).
|
||||
target = normalize_doc_date(raw_date)
|
||||
if target != doc.created_at:
|
||||
doc.created_at = target
|
||||
session.commit()
|
||||
summary.dates_updated += 1
|
||||
logger.info(
|
||||
"import: date-refreshed source=%s path=%s date=%s",
|
||||
source, rel, doc.created_at.isoformat(),
|
||||
)
|
||||
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
||||
# row, or a bytes-changed/content-same image (the byte digest
|
||||
# still matches). The helper stamps the new file hash (the
|
||||
# next sync takes the fast path); for an image the summary is
|
||||
# the stored description (``doc.content``), so the image-aware
|
||||
# ``_store_summary`` backfill re-stores it verbatim.
|
||||
await _unchanged_doc(
|
||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||
raw_date=raw_date, file_digest=file_digest,
|
||||
)
|
||||
return
|
||||
|
||||
verb = "updated" if doc is not None else "added"
|
||||
@@ -2803,6 +2945,9 @@ async def _index_image_file(
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash=digest,
|
||||
# Phase 147: the raw-bytes stamp (the next sync's fast
|
||||
# path).
|
||||
file_hash=file_digest,
|
||||
indexed_at=datetime.now(UTC),
|
||||
created_at=normalize_doc_date(raw_date),
|
||||
is_image=True,
|
||||
@@ -2818,6 +2963,8 @@ async def _index_image_file(
|
||||
doc.title = title
|
||||
doc.content = content
|
||||
doc.content_hash = digest
|
||||
# Phase 147: re-stamp (the file's bytes changed).
|
||||
doc.file_hash = file_digest
|
||||
doc.indexed_at = datetime.now(UTC)
|
||||
# Phase 106 (D4): a content change is a new document version —
|
||||
# the date is re-sourced and a previous manual correction is
|
||||
|
||||
Reference in New Issue
Block a user