brain-of-reese/.agents/phases/todo/147_sync_changed_only/00_phase.md

Phase 147 — sync only what changed: the source file's byte hash is checked BEFORE extraction and vision

Source: TODO.md L4 — "Local sources, like PDFs, are synced every time even if they haven't changed. This results in really long sync times for PDFs that are mostly pictoral. Only sync documents if they've changed since the last sync (hash the document contents to check)." Story: n/a (TODO-derived phase, Protocol B append). Context: app/rag/importer.py — import_sources (L560: the per-source walk; the progress/on_chunk_progress hooks fire per file before indexing — unchanged by this phase), _index_file (L879: the single upsert entry — dispatches image → _index_image_file (L2642), .pdf → _index_pdf_file (L1181), text fallback), and the three per-branch unchanged blocks that today run only AFTER the expensive work: text (L928: digest over the read text — cheap, the read itself is the cost), image (L2660: digest over read_bytes() — the vision description runs only on a digest MISS, already fine), PDF (L~1303: digest over the FINAL constructed content — reached only after extract_pdf_text (L162 of app/rag/pdf.py) ALWAYS runs and, for garbage layers, _fallback_page_screenshots (every page rendered + vision-described) or, for sparse layers with pdf_pictorial on, _pictorial_pdf_pass (every candidate page rendered + vision-described) — so an UNCHANGED pictorial PDF re-extracts and re-burns vision on EVERY sync; a non-deterministic vision model additionally flips the final-content digest and forces a full re-chunk/re-embed/re-summary each run — the owner's "synced every time even if they haven't changed"). Document.content_hash (app/models.py L112) is the FINAL-CONTENT digest (exposed via app/rag/retriever.py L203/426/555/611 — the dedupe/caching identity) and stays untouched; the new file_hash is the RAW SOURCE-FILE-BYTES digest, compared first, used for change detection only. alembic/versions/ — the latest migration is 0024_git_source_branch.py. The unchanged-block semantics that must survive the fast path (all three branches, identical shape): summary.unchanged += 1 + log; the phase-118 (A2) summary-NULL backfill; the phase-106 (D1) created_at_manual early-return; the phase-106 (D4) date refresh (normalize_doc_date of the raw_date — mtime fallback — may go OLDER, counts summary.dates_updated, commits). Locked (owner delegation 2026-09-30, chat — "pick the best choices"): change detection = sha256 of the RAW FILE BYTES (the owner's "hash the document contents"), computed ONCE at the top of _index_file — before the image/text/PDF dispatch — and compared against a new documents.file_hash column (migration 0025, nullable). A match takes a shared unchanged fast path: NO file read (beyond the streamed hash), NO PDF extraction, NO vision call, NO embed, NO summary call — only the preserved bookkeeping (summary-NULL backfill from the STORED doc.content, manual-date early-return, date refresh). Pre-phase rows (file_hash NULL) fall through to the branches' existing logic exactly as today and get their hash STAMPED on that first post-upgrade sync (one full pass, then fast forever). content_hash and everything built on it (retriever exposure, chunk identity) are untouched.

Objective

An unchanged source file costs one streamed sha256 and a row lookup per sync — zero extraction, zero vision, zero embedding, zero summary LLM — across all three document classes (text, image, pictorial PDF), while every preserved unchanged-path behavior (summary backfill, manual-date lock, date refresh, counts) is byte-for-byte maintained.

Dependencies

• 146_search_tool_scope (todo) — pipeline order only; no shared code (this phase touches the importer + one migration; 146 touches the tool-frame wire + the chat renderer).

Design (shared by all tasks — the executor reads this, not the chat)

• The column (task 01): alembic/versions/0025_documents_file_hash.py — documents.file_hash String(64), nullable, no index (it is compared in Python after the unique (source, path) lookup — unlike content_hash, it is never a query surface and never exposed to the retriever); downgrade drops it. Document.file_hash: Mapped[str | None] + docstring (the raw source-file-bytes sha256 — the sync change signal; DISTINCT from content_hash — the final indexed content's digest, which for a pictorial PDF includes vision text and is the retriever's identity).
• The fast path (task 02):
  • _file_hash(path: Path) -> str — sha256 over the raw bytes, streamed in 1 MiB chunks (a multi-hundred-MB PDF never loads fully; the hashlib incremental pattern).
  • _index_file top (before the image/PDF/text dispatch): file_digest = _file_hash(full_path); the row lookup (the (source, path) scalar select the branches already do — moved up); if doc is not None and doc.file_hash == file_digest: → _unchanged_doc(...) → return. The raw_date mtime fallback (the D2 convention: read before the unchanged early-return — the branches already do this; the fast path needs it for the date refresh) is computed in _index_file before the check when raw_date is None.
  • _unchanged_doc(session, doc, *, source, rel, llm, summary, raw_date) — the ONE shared implementation of the three near-identical unchanged blocks: summary.unchanged += 1 + the import: unchanged log line (unchanged wording); the phase-118 (A2) backfill when doc.summary is None — content=doc.content (EXACTLY what the bytes produced — for text the stored content IS the file text; for an image the stored description; for a PDF the stored final content — so the backfill never re-reads/re-extracts/re-describes); the D1 created_at_manual early-return; the D4 date refresh (target = normalize_doc_date(raw_date); on change: commit + summary.dates_updated + the import: date-refreshed log line — unchanged wording). The three branches' existing unchanged blocks are REPLACED by the shared helper (the per-branch callsite passes the same arguments; behavior is identical, code is one site).
  • Hash stamping: every branch writes doc.file_hash = file_digest on create AND update (all three — text, image, PDF — file_digest passed down from _index_file). The fast path makes the stamp redundant for the common case, but a bytes-changed/content-same edge (e.g. a PDF metadata edit with identical extracted text) lands in a branch whose content digest still matches — the branch's unchanged flow (now the shared helper) ALSO stamps doc.file_hash = file_digest and commits when it is dirty (the date-refresh commit may already have run — stamp BEFORE that block's final decision: set the attribute, and commit once at the end of the helper if dirty) so the NEXT sync takes the fast path. Pre-phase rows (file_hash NULL) take exactly this path on the first post-upgrade sync (full pipeline or the shared helper — whichever the branch's content digest decides) and are stamped either way.
  • The progress/on_chunk_progress hooks are untouched (the progress call fires per file before _index_file as today — an unchanged file was already "the current file" pre-phase; the fast path just makes it cheap). summary.files counts the walked file either way. Pruning, limit, the per-source post-index passes (_enrich_pending_images, _backfill_image_contexts — row-driven) are untouched.
• The E2E (task 03): the file:// fixture-repo module-app idiom (the test_sync_button.py recipe — real clone, mock LLM): run 1 (N added) → capture every document row's indexed_at/content_hash → run 2 → the settle line reads 0 added · 0 updated · N unchanged and every row's indexed_at/content_hash is unchanged (the re-index would bump indexed_at) → commit a new file to the fixture repo → run 3 → only the new file is added.

Tasks

1. 01_file_hash_column.md — migration 0025 + the model field.
2. 02_unchanged_fast_path.md — the streamed hash, the top-of-_index_file check, the shared unchanged helper, the stamping, the branch rewiring, the unit matrix.
3. 03_e2e_unchanged_second_run.md — the dedicated E2E: the idempotent second sync is provably free of re-indexing.

Testing & Quality

• Unit: the fast-path matrix (text / image / garbage-PDF / pictorial-PDF unchanged → zero LLM calls; changed → full pipeline + stamp; NULL hash → pre-phase behavior + stamp; D1/D4 pins on the fast path; the streamed hash on a >1 MiB file).
• Integration: the migration upgrade/downgrade round-trip (the house migration-test idiom).
• E2E: one dedicated Playwright file, run in isolation, mock-only (real git clone).
• Coverage: >90% on new/modified code (app/).

Completion Criteria

• documents.file_hash exists (nullable String(64), no index) via migration 0025; the upgrade/downgrade round-trips on the test DB
• an unchanged text file, image, garbage-layer PDF, and pictorial PDF each cost ZERO llm calls (embed + chat) and zero PDF extraction/vision on the second sync — summary.unchanged counts them; indexed_at/content_hash untouched
• a changed file (any byte) re-indexes fully and stamps the new hash; a bytes-changed/content-same file stamps the hash without a content change
• the preserved unchanged semantics are pinned: summary-NULL backfill, created_at_manual early-return, the date refresh (may go older, counts dates_updated, commits), the log-line wordings
• content_hash and the retriever's exposure of it are byte-identical to pre-phase
• uv run pytest --cov=app --cov-report=term-missing green, app/ coverage >90%; uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov green in isolation; uv run ruff check . && uv run pyright clean
• no behavior change in completed phases (prune semantics, limit/progress contracts, the per-source post-index passes, the sync status/result wire)