phase: 185_sync_defer_content_fast_path
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s

All gates green. Report:

**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.

**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).

**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).

**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).

**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).

**Next pending phase**: `186_git_log_streaming_dates`.
This commit is contained in:
2026-10-16 04:54:01 -04:00
parent 065304b52f
commit 632fb88e43
35 changed files with 2862 additions and 138 deletions
+55 -11
View File
@@ -84,6 +84,14 @@ pre-phase on the first post-upgrade sync (one full pass), then stamp
and go fast. The previous re-burn costs (a garbage PDF's fallback
vision re-issued each sync, a non-deterministic vision model flipping
a pictorial PDF's content digest) are gone: the file bytes decide.
Phase 185 (task 02): the fast path's ``(source, path)`` row comes from the
shared :func:`_doc_row_stmt`, which defers ``Document.content`` — the ONE
column the fast path never reads and the fat one of the row (252.6 MB of
markdown over 17,407 rows on the incident box) — so a full walk's memory
scales with row METADATA, not with the corpus. Exactly one column is
deferred (no ``load_only`` allowlist, no ``raiseload``/``noload``): the
phase-118 summary-NULL backfill genuinely reads ``doc.content`` and pays
ONE lazy SELECT for that one row, by design.
Standalone images (phase 122, LOCKED A3): with the ``images`` toggle
(``BOR_IMAGES``) ON, the walk also admits the image extension set
@@ -345,8 +353,8 @@ from pathlib import Path, PurePosixPath
from typing import Any, Protocol
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
from sqlalchemy import select
from sqlalchemy.orm import Session, load_only
from sqlalchemy import Select, select
from sqlalchemy.orm import Session, defer, load_only
from app.config import Settings
from app.db import SessionLocal
@@ -705,6 +713,37 @@ def _file_hash(path: Path) -> str:
return h.hexdigest()
def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
"""The importer's single Document-row statement (phase 185).
Used by the FOUR loads that share the phase-147 unchanged fast path —
:func:`_index_file`, :func:`_index_pdf_file`,
:func:`_index_one_embedded_image` and :func:`_index_image_file` — so
the deferral below cannot drift between them.
``defer(Document.content)`` — the unchanged fast path (phase 147) reads
file_hash/is_image/page_count/summary/dates, never the stored markdown,
and ``app/db.py`` builds sessions with ``expire_on_commit=False``, so
hydrating ``content`` for EVERY inspected file is what took the
2026-10-15 run to ~250 MB on a 17,407-document KB (avg 15,217
chars/row): the walk inspected ~27k files and re-indexed 217.
ONE column is deferred on purpose (LOCKED A1): a ``load_only``
allowlist would silently defer every column a later phase adds, and the
deferred read would then surface as an unexplained N+1. No
``raiseload``/``noload`` either (LOCKED A2): the phase-118 summary-NULL
backfill (``_unchanged_doc`` step 3) genuinely reads ``doc.content`` and
must keep working — it costs one extra SELECT for that one row, a
per-row, one-time, already-rare cost, accepted instead of a rule that
would crash the branch where it should query.
"""
return (
select(Document)
.options(defer(Document.content))
.where(Document.source == source, Document.path == rel)
)
async def import_sources(
sources: list[Path],
llm: Embedder,
@@ -1194,7 +1233,10 @@ async def _unchanged_doc(
# branch paths — the next sync is then fast either way).
doc.file_hash = file_digest
# 3 — the phase-118 (A2) summary-NULL backfill, from the stored
# content (see the docstring — never a re-read/re-extract).
# content (see the docstring — never a re-read/re-extract). Phase 185:
# this is the ONE fast-path read that pulls the deferred ``content``
# back — one SELECT, one time, by design (LOCKED A2: no raiseload, the
# branch must query rather than crash).
if doc.summary is None:
await _store_summary(
session, doc=doc, source=source, rel=rel, content=doc.content,
@@ -1457,8 +1499,12 @@ async def _index_file(
Phase 147 (task 02): the UNCHANGED FAST PATH. BEFORE the
image/text/PDF dispatch, the file's RAW source bytes are hashed
(:func:`_file_hash` — streamed), the ``(source, path)`` row is
looked up, and the D2 mtime fallback for *raw_date* runs (once,
here — the branches receive the already-computed date) — a
looked up through the shared :func:`_doc_row_stmt` (phase 185: with
``Document.content`` DEFERRED — the fast path reads row METADATA only,
so a full walk no longer hydrates the KB's markdown; the phase-118
summary-NULL backfill is the one read that pulls the text back, one
SELECT for that row), and the D2 mtime fallback for *raw_date* runs
(once, here — the branches receive the already-computed date) — a
``doc.file_hash == file_digest`` match returns through the shared
unchanged path (:func:`_unchanged_doc`) with ZERO file read beyond
the streamed hash, ZERO PDF extraction, ZERO vision call, and
@@ -1508,7 +1554,7 @@ async def _index_file(
# ``read_text``/``read_bytes``, no ``extract_pdf_text``, no
# fallback/pictorial vision, no re-chunk/re-embed/re-summary.
file_digest = _file_hash(full_path)
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
doc = session.scalar(_doc_row_stmt(source, rel))
if raw_date is None:
# D2 fallback: no source date in the map → the file's mtime
# (one stat). Read before the unchanged early-return — the
@@ -1969,7 +2015,7 @@ async def _index_pdf_file(
else hashlib.sha256(b"").hexdigest()
)
title = _extract_pdf_title(full_path, fallback=full_path.stem)
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
doc = session.scalar(_doc_row_stmt(source, rel))
if doc is not None and doc.content_hash == digest:
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
# row, or a bytes-changed/content-same PDF (the content digest
@@ -2728,9 +2774,7 @@ async def _index_one_embedded_image(
return
image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
digest = hashlib.sha256(img.data).hexdigest()
image_doc = session.scalar(
select(Document).where(Document.source == source, Document.path == image_rel)
)
image_doc = session.scalar(_doc_row_stmt(source, image_rel))
if image_doc is not None and image_doc.content_hash == digest:
# Unchanged embedded image (byte digest) — no re-describe, no
# re-embed (the standalone image's unchanged rule).
@@ -3730,7 +3774,7 @@ async def _index_image_file(
# byte change.
data = full_path.read_bytes()
digest = hashlib.sha256(data).hexdigest()
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
doc = session.scalar(_doc_row_stmt(source, rel))
if doc is not None and doc.is_image and doc.content_hash == digest:
# Phase 147: the shared unchanged path — a pre-phase NULL-hash