phase: 185_sync_defer_content_fast_path
All gates green. Report: **Implemented / verified (phase 185)** - `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`. - `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget). - **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`). - **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order. **Tests / lint (all 100% green)** - `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes). **Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run). **Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists). **Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified). **Next pending phase**: `186_git_log_streaming_dates`.
This commit is contained in:
+55
-11
@@ -84,6 +84,14 @@ pre-phase on the first post-upgrade sync (one full pass), then stamp
|
||||
and go fast. The previous re-burn costs (a garbage PDF's fallback
|
||||
vision re-issued each sync, a non-deterministic vision model flipping
|
||||
a pictorial PDF's content digest) are gone: the file bytes decide.
|
||||
Phase 185 (task 02): the fast path's ``(source, path)`` row comes from the
|
||||
shared :func:`_doc_row_stmt`, which defers ``Document.content`` — the ONE
|
||||
column the fast path never reads and the fat one of the row (252.6 MB of
|
||||
markdown over 17,407 rows on the incident box) — so a full walk's memory
|
||||
scales with row METADATA, not with the corpus. Exactly one column is
|
||||
deferred (no ``load_only`` allowlist, no ``raiseload``/``noload``): the
|
||||
phase-118 summary-NULL backfill genuinely reads ``doc.content`` and pays
|
||||
ONE lazy SELECT for that one row, by design.
|
||||
|
||||
Standalone images (phase 122, LOCKED A3): with the ``images`` toggle
|
||||
(``BOR_IMAGES``) ON, the walk also admits the image extension set
|
||||
@@ -345,8 +353,8 @@ from pathlib import Path, PurePosixPath
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, load_only
|
||||
from sqlalchemy import Select, select
|
||||
from sqlalchemy.orm import Session, defer, load_only
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
@@ -705,6 +713,37 @@ def _file_hash(path: Path) -> str:
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
|
||||
"""The importer's single Document-row statement (phase 185).
|
||||
|
||||
Used by the FOUR loads that share the phase-147 unchanged fast path —
|
||||
:func:`_index_file`, :func:`_index_pdf_file`,
|
||||
:func:`_index_one_embedded_image` and :func:`_index_image_file` — so
|
||||
the deferral below cannot drift between them.
|
||||
|
||||
``defer(Document.content)`` — the unchanged fast path (phase 147) reads
|
||||
file_hash/is_image/page_count/summary/dates, never the stored markdown,
|
||||
and ``app/db.py`` builds sessions with ``expire_on_commit=False``, so
|
||||
hydrating ``content`` for EVERY inspected file is what took the
|
||||
2026-10-15 run to ~250 MB on a 17,407-document KB (avg 15,217
|
||||
chars/row): the walk inspected ~27k files and re-indexed 217.
|
||||
|
||||
ONE column is deferred on purpose (LOCKED A1): a ``load_only``
|
||||
allowlist would silently defer every column a later phase adds, and the
|
||||
deferred read would then surface as an unexplained N+1. No
|
||||
``raiseload``/``noload`` either (LOCKED A2): the phase-118 summary-NULL
|
||||
backfill (``_unchanged_doc`` step 3) genuinely reads ``doc.content`` and
|
||||
must keep working — it costs one extra SELECT for that one row, a
|
||||
per-row, one-time, already-rare cost, accepted instead of a rule that
|
||||
would crash the branch where it should query.
|
||||
"""
|
||||
return (
|
||||
select(Document)
|
||||
.options(defer(Document.content))
|
||||
.where(Document.source == source, Document.path == rel)
|
||||
)
|
||||
|
||||
|
||||
async def import_sources(
|
||||
sources: list[Path],
|
||||
llm: Embedder,
|
||||
@@ -1194,7 +1233,10 @@ async def _unchanged_doc(
|
||||
# branch paths — the next sync is then fast either way).
|
||||
doc.file_hash = file_digest
|
||||
# 3 — the phase-118 (A2) summary-NULL backfill, from the stored
|
||||
# content (see the docstring — never a re-read/re-extract).
|
||||
# content (see the docstring — never a re-read/re-extract). Phase 185:
|
||||
# this is the ONE fast-path read that pulls the deferred ``content``
|
||||
# back — one SELECT, one time, by design (LOCKED A2: no raiseload, the
|
||||
# branch must query rather than crash).
|
||||
if doc.summary is None:
|
||||
await _store_summary(
|
||||
session, doc=doc, source=source, rel=rel, content=doc.content,
|
||||
@@ -1457,8 +1499,12 @@ async def _index_file(
|
||||
Phase 147 (task 02): the UNCHANGED FAST PATH. BEFORE the
|
||||
image/text/PDF dispatch, the file's RAW source bytes are hashed
|
||||
(:func:`_file_hash` — streamed), the ``(source, path)`` row is
|
||||
looked up, and the D2 mtime fallback for *raw_date* runs (once,
|
||||
here — the branches receive the already-computed date) — a
|
||||
looked up through the shared :func:`_doc_row_stmt` (phase 185: with
|
||||
``Document.content`` DEFERRED — the fast path reads row METADATA only,
|
||||
so a full walk no longer hydrates the KB's markdown; the phase-118
|
||||
summary-NULL backfill is the one read that pulls the text back, one
|
||||
SELECT for that row), and the D2 mtime fallback for *raw_date* runs
|
||||
(once, here — the branches receive the already-computed date) — a
|
||||
``doc.file_hash == file_digest`` match returns through the shared
|
||||
unchanged path (:func:`_unchanged_doc`) with ZERO file read beyond
|
||||
the streamed hash, ZERO PDF extraction, ZERO vision call, and
|
||||
@@ -1508,7 +1554,7 @@ async def _index_file(
|
||||
# ``read_text``/``read_bytes``, no ``extract_pdf_text``, no
|
||||
# fallback/pictorial vision, no re-chunk/re-embed/re-summary.
|
||||
file_digest = _file_hash(full_path)
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
doc = session.scalar(_doc_row_stmt(source, rel))
|
||||
if raw_date is None:
|
||||
# D2 fallback: no source date in the map → the file's mtime
|
||||
# (one stat). Read before the unchanged early-return — the
|
||||
@@ -1969,7 +2015,7 @@ async def _index_pdf_file(
|
||||
else hashlib.sha256(b"").hexdigest()
|
||||
)
|
||||
title = _extract_pdf_title(full_path, fallback=full_path.stem)
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
doc = session.scalar(_doc_row_stmt(source, rel))
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
||||
# row, or a bytes-changed/content-same PDF (the content digest
|
||||
@@ -2728,9 +2774,7 @@ async def _index_one_embedded_image(
|
||||
return
|
||||
image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
|
||||
digest = hashlib.sha256(img.data).hexdigest()
|
||||
image_doc = session.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == image_rel)
|
||||
)
|
||||
image_doc = session.scalar(_doc_row_stmt(source, image_rel))
|
||||
if image_doc is not None and image_doc.content_hash == digest:
|
||||
# Unchanged embedded image (byte digest) — no re-describe, no
|
||||
# re-embed (the standalone image's unchanged rule).
|
||||
@@ -3730,7 +3774,7 @@ async def _index_image_file(
|
||||
# byte change.
|
||||
data = full_path.read_bytes()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
doc = session.scalar(_doc_row_stmt(source, rel))
|
||||
|
||||
if doc is not None and doc.is_image and doc.content_hash == digest:
|
||||
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
||||
|
||||
Reference in New Issue
Block a user