phase: 185_sync_defer_content_fast_path
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s

All gates green. Report:

**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.

**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).

**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).

**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).

**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).

**Next pending phase**: `186_git_log_streaming_dates`.
This commit is contained in:
2026-10-16 04:54:01 -04:00
parent 065304b52f
commit 632fb88e43
35 changed files with 2862 additions and 138 deletions
+32 -1
View File
@@ -429,7 +429,12 @@ idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
streamed) and checked against the last index's stamp *before* any
extraction or vision work — an unchanged file costs one hash + one row
lookup (no re-extract, no vision, no re-embed), while a changed file
re-indexes fully. The one exception is a PDF row that still claims a
re-indexes fully. That row lookup reads **metadata only** (phase 185:
`Document.content` is deferred), so a full-resync walk's cost is
metadata-bound instead of corpus-bound; the rare unchanged file whose
`summary` is still NULL re-reads its stored content — one query, one time,
by design — because that backfill genuinely needs the text. The one
exception is a PDF row that still claims a
page strip: it is re-gated from its current bytes (no vision, no LLM —
see "The strip repairs itself" below). A refresh after a normal editing
session takes seconds — pictorial PDFs included (their expensive
@@ -683,6 +688,32 @@ only) runs the whole source refresh in one click:
`0`-cadence run of the same fixture is byte-identical to the `3`-cadence
one (per-path chunk counts, `content`, `summary`, both hashes, the counts
line).
- **A full walk reads row metadata, not the corpus** (phase 185). The four
row lookups the unchanged fast path shares (`_index_file` and the PDF /
PDF-embedded-image / standalone-image branches) go through ONE statement,
`_doc_row_stmt`, which defers the single column the fast path never
reads — `Document.content`. The 2026-10-15 sync inspected ~27k files and
re-indexed 217, yet hydrated all 17,407 rows whole (avg 15,217 chars =
**252.6 MB of markdown**, held for the whole run by `expire_on_commit=False`
and phase 184's stamp); now an inspected row carries file_hash, summary,
dates, page count — and no markdown. Exactly ONE column is deferred, and
there is deliberately **no `raiseload`/`noload`**: the phase-118
summary-NULL backfill genuinely reads the stored content, so that one row
pays one extra `SELECT documents.content … WHERE id = …` (per-row,
one-time, already rare) instead of crashing. Counts, log words, the D1
manual-date lock, the phase-163 strip re-gate and the backfill result are
byte-identical — pinned by
`tests/integration/test_import_fast_path_columns.py` (the captured
fast-path `SELECT` names no `documents.content`, one `SELECT` per
unchanged file with a summary, exactly one lazy follow-up for the
backfill), `tests/unit/test_importer_defer_content.py` (`content` is
the only deferred column, all four loads share the helper, no
`raiseload`/`noload` anywhere in the importer) and, through the UI,
`tests/e2e/test_fast_path_deferred_content.py` (a hand-nulled summary
backfills to its byte-identical text from the Sources-page re-sync with
exactly one `is_summary` chunk, an unchanged doc's viewer still shows
the full stored text, and a rewritten file re-indexes to the new text
with the next sync all-`unchanged` again).
- **Cancel**: while a run is in flight, a **Cancel** button appears
next to the sync button; clicking it posts `POST /api/sync/cancel`
(admin only) and registers a cooperative cancel (`202`) — the run