phase: 185_sync_defer_content_fast_path
All gates green. Report: **Implemented / verified (phase 185)** - `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`. - `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget). - **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`). - **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order. **Tests / lint (all 100% green)** - `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes). **Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run). **Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists). **Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified). **Next pending phase**: `186_git_log_streaming_dates`.
This commit is contained in:
@@ -429,7 +429,12 @@ idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
|
||||
streamed) and checked against the last index's stamp *before* any
|
||||
extraction or vision work — an unchanged file costs one hash + one row
|
||||
lookup (no re-extract, no vision, no re-embed), while a changed file
|
||||
re-indexes fully. The one exception is a PDF row that still claims a
|
||||
re-indexes fully. That row lookup reads **metadata only** (phase 185:
|
||||
`Document.content` is deferred), so a full-resync walk's cost is
|
||||
metadata-bound instead of corpus-bound; the rare unchanged file whose
|
||||
`summary` is still NULL re-reads its stored content — one query, one time,
|
||||
by design — because that backfill genuinely needs the text. The one
|
||||
exception is a PDF row that still claims a
|
||||
page strip: it is re-gated from its current bytes (no vision, no LLM —
|
||||
see "The strip repairs itself" below). A refresh after a normal editing
|
||||
session takes seconds — pictorial PDFs included (their expensive
|
||||
@@ -683,6 +688,32 @@ only) runs the whole source refresh in one click:
|
||||
`0`-cadence run of the same fixture is byte-identical to the `3`-cadence
|
||||
one (per-path chunk counts, `content`, `summary`, both hashes, the counts
|
||||
line).
|
||||
- **A full walk reads row metadata, not the corpus** (phase 185). The four
|
||||
row lookups the unchanged fast path shares (`_index_file` and the PDF /
|
||||
PDF-embedded-image / standalone-image branches) go through ONE statement,
|
||||
`_doc_row_stmt`, which defers the single column the fast path never
|
||||
reads — `Document.content`. The 2026-10-15 sync inspected ~27k files and
|
||||
re-indexed 217, yet hydrated all 17,407 rows whole (avg 15,217 chars =
|
||||
**252.6 MB of markdown**, held for the whole run by `expire_on_commit=False`
|
||||
and phase 184's stamp); now an inspected row carries file_hash, summary,
|
||||
dates, page count — and no markdown. Exactly ONE column is deferred, and
|
||||
there is deliberately **no `raiseload`/`noload`**: the phase-118
|
||||
summary-NULL backfill genuinely reads the stored content, so that one row
|
||||
pays one extra `SELECT documents.content … WHERE id = …` (per-row,
|
||||
one-time, already rare) instead of crashing. Counts, log words, the D1
|
||||
manual-date lock, the phase-163 strip re-gate and the backfill result are
|
||||
byte-identical — pinned by
|
||||
`tests/integration/test_import_fast_path_columns.py` (the captured
|
||||
fast-path `SELECT` names no `documents.content`, one `SELECT` per
|
||||
unchanged file with a summary, exactly one lazy follow-up for the
|
||||
backfill), `tests/unit/test_importer_defer_content.py` (`content` is
|
||||
the only deferred column, all four loads share the helper, no
|
||||
`raiseload`/`noload` anywhere in the importer) and, through the UI,
|
||||
`tests/e2e/test_fast_path_deferred_content.py` (a hand-nulled summary
|
||||
backfills to its byte-identical text from the Sources-page re-sync with
|
||||
exactly one `is_summary` chunk, an unchanged doc's viewer still shows
|
||||
the full stored text, and a rewritten file re-indexes to the new text
|
||||
with the next sync all-`unchanged` again).
|
||||
- **Cancel**: while a run is in flight, a **Cancel** button appears
|
||||
next to the sync button; clicking it posts `POST /api/sync/cancel`
|
||||
(admin only) and registers a cooperative cancel (`202`) — the run
|
||||
|
||||
Reference in New Issue
Block a user