phase: 147_sync_changed_only

All defaults match. Verification complete — everything is green with no defects found.

**Phase 147 final verification pass — report**

- All 3 task files found in `complete/147_sync_changed_only/` → verification pass only; no code changes needed.
- Verified: migration 0025 (`documents.file_hash` nullable String(64), no index) + model field; fast path at top of `_index_file` (streamed 1 MiB sha256 → row lookup → `_unchanged_doc`); stamping on all three branches' create/update + shared helper; pre-phase NULL-hash fallthrough; retriever/`content_hash` untouched.
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3438 passed, `app/` coverage **99%** (>90% ✓); `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` → 3 passed in isolation ✓; `uv run ruff check . && uv run pyright` → all checks passed, 0 errors ✓; prior-phase `tests/e2e/test_sync_button.py` → 3 passed (no regression) ✓.
- **Completion criteria:** all 7 pass — (1) column + round-trip ✓ (2) unchanged text/image/garbage-PDF/pictorial-PDF cost zero LLM/extraction/vision, `summary.unchanged` counted, rows byte-still ✓ (3) changed file re-indexes + re-stamps; bytes-changed/content-same stamps only ✓ (4) backfill/D1/D4 + log wordings pinned verbatim ✓ (5) `content_hash`/retriever byte-identical ✓ (6) all gates green ✓ (7) no prior-phase behavior change ✓.
- **Documentation:** full review of README.md (1,031 lines) and AGENTS.md against the code: every `.env.example`/code `BOR_*` var documented, all config defaults match `app/config.py`, quick-start/QA commands and referenced test files exist, `file://` URL + troubleshooting claims accurate; the phase's README "Updating the documents" edit is accurate. **Verified consistent — no changes needed.**
- **Notable:** no live-infrastructure changes (dev Postgres container used for the standard test gates only). Working tree left uncommitted per harness protocol.
- **Next pending phase:** `148_pdf_pages_in_sources`.

.agents/phases/complete/147_sync_changed_only/00_phase.md

@@ -0,0 +1,42 @@
+# Phase 147 — sync only what changed: the source file's byte hash is checked BEFORE extraction and vision
+
+**Source:** `TODO.md` L4 — "Local sources, like PDFs, are synced every time even if they haven't changed. This results in really long sync times for PDFs that are mostly pictoral. Only sync documents if they've changed since the last sync (hash the document contents to check)."
+**Story:** n/a (TODO-derived phase, Protocol B append).
+**Context:** `app/rag/importer.py` — `import_sources` (L560: the per-source walk; the `progress`/`on_chunk_progress` hooks fire per file before indexing — unchanged by this phase), `_index_file` (L879: the single upsert entry — dispatches image → `_index_image_file` (L2642), `.pdf` → `_index_pdf_file` (L1181), text fallback), and the three per-branch unchanged blocks that today run only AFTER the expensive work: text (L~928: digest over the read text — cheap, the read itself is the cost), image (L~2660: digest over `read_bytes()` — the vision description runs only on a digest MISS, already fine), PDF (L~1303: digest over the FINAL constructed content — reached only after `extract_pdf_text` (L162 of `app/rag/pdf.py`) ALWAYS runs and, for garbage layers, `_fallback_page_screenshots` (every page rendered + vision-described) or, for sparse layers with `pdf_pictorial` on, `_pictorial_pdf_pass` (every candidate page rendered + vision-described) — so an UNCHANGED pictorial PDF re-extracts and re-burns vision on EVERY sync; a non-deterministic vision model additionally flips the final-content digest and forces a full re-chunk/re-embed/re-summary each run — the owner's "synced every time even if they haven't changed"). `Document.content_hash` (app/models.py L112) is the FINAL-CONTENT digest (exposed via `app/rag/retriever.py` L203/426/555/611 — the dedupe/caching identity) and stays untouched; the new `file_hash` is the RAW SOURCE-FILE-BYTES digest, compared first, used for change detection only. `alembic/versions/` — the latest migration is `0024_git_source_branch.py`. The unchanged-block semantics that must survive the fast path (all three branches, identical shape): `summary.unchanged += 1` + log; the phase-118 (A2) summary-NULL backfill; the phase-106 (D1) `created_at_manual` early-return; the phase-106 (D4) date refresh (`normalize_doc_date` of the raw_date — mtime fallback — may go OLDER, counts `summary.dates_updated`, commits).
+**Locked (owner delegation 2026-09-30, chat — "pick the best choices"):** change detection = sha256 of the RAW FILE BYTES (the owner's "hash the document contents"), computed ONCE at the top of `_index_file` — before the image/text/PDF dispatch — and compared against a new `documents.file_hash` column (migration 0025, nullable). A match takes a shared unchanged fast path: NO file read (beyond the streamed hash), NO PDF extraction, NO vision call, NO embed, NO summary call — only the preserved bookkeeping (summary-NULL backfill from the STORED `doc.content`, manual-date early-return, date refresh). Pre-phase rows (`file_hash` NULL) fall through to the branches' existing logic exactly as today and get their hash STAMPED on that first post-upgrade sync (one full pass, then fast forever). `content_hash` and everything built on it (retriever exposure, chunk identity) are untouched.
+
+## Objective
+An unchanged source file costs one streamed sha256 and a row lookup per sync — zero extraction, zero vision, zero embedding, zero summary LLM — across all three document classes (text, image, pictorial PDF), while every preserved unchanged-path behavior (summary backfill, manual-date lock, date refresh, counts) is byte-for-byte maintained.
+
+## Dependencies
+- `146_search_tool_scope` (todo) — pipeline order only; no shared code (this phase touches the importer + one migration; 146 touches the tool-frame wire + the chat renderer).
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+- **The column (task 01):** `alembic/versions/0025_documents_file_hash.py` — `documents.file_hash` `String(64)`, **nullable, no index** (it is compared in Python after the unique `(source, path)` lookup — unlike `content_hash`, it is never a query surface and never exposed to the retriever); downgrade drops it. `Document.file_hash: Mapped[str | None]` + docstring (the raw source-file-bytes sha256 — the sync change signal; DISTINCT from `content_hash` — the final indexed content's digest, which for a pictorial PDF includes vision text and is the retriever's identity).
+- **The fast path (task 02):**
+  - `_file_hash(path: Path) -> str` — sha256 over the raw bytes, streamed in 1 MiB chunks (a multi-hundred-MB PDF never loads fully; the `hashlib` incremental pattern).
+  - `_index_file` top (before the image/PDF/text dispatch): `file_digest = _file_hash(full_path)`; the row lookup (the `(source, path)` scalar select the branches already do — moved up); `if doc is not None and doc.file_hash == file_digest:` → `_unchanged_doc(...)` → **return**. The `raw_date` mtime fallback (the D2 convention: read before the unchanged early-return — the branches already do this; the fast path needs it for the date refresh) is computed in `_index_file` before the check when `raw_date is None`.
+  - `_unchanged_doc(session, doc, *, source, rel, llm, summary, raw_date)` — the ONE shared implementation of the three near-identical unchanged blocks: `summary.unchanged += 1` + the `import: unchanged` log line (unchanged wording); the phase-118 (A2) backfill when `doc.summary is None` — `content=doc.content` (EXACTLY what the bytes produced — for text the stored content IS the file text; for an image the stored description; for a PDF the stored final content — so the backfill never re-reads/re-extracts/re-describes); the D1 `created_at_manual` early-return; the D4 date refresh (`target = normalize_doc_date(raw_date)`; on change: commit + `summary.dates_updated` + the `import: date-refreshed` log line — unchanged wording). The three branches' existing unchanged blocks are REPLACED by the shared helper (the per-branch callsite passes the same arguments; behavior is identical, code is one site).
+  - **Hash stamping:** every branch writes `doc.file_hash = file_digest` on create AND update (all three — text, image, PDF — `file_digest` passed down from `_index_file`). The fast path makes the stamp redundant for the common case, but a bytes-changed/content-same edge (e.g. a PDF metadata edit with identical extracted text) lands in a branch whose content digest still matches — the branch's unchanged flow (now the shared helper) ALSO stamps `doc.file_hash = file_digest` and commits when it is dirty (the date-refresh commit may already have run — stamp BEFORE that block's final decision: set the attribute, and commit once at the end of the helper if dirty) so the NEXT sync takes the fast path. Pre-phase rows (`file_hash` NULL) take exactly this path on the first post-upgrade sync (full pipeline or the shared helper — whichever the branch's content digest decides) and are stamped either way.
+  - The `progress`/`on_chunk_progress` hooks are untouched (the `progress` call fires per file before `_index_file` as today — an unchanged file was already "the current file" pre-phase; the fast path just makes it cheap). `summary.files` counts the walked file either way. Pruning, `limit`, the per-source post-index passes (`_enrich_pending_images`, `_backfill_image_contexts` — row-driven) are untouched.
+- **The E2E (task 03):** the `file://` fixture-repo module-app idiom (the `test_sync_button.py` recipe — real clone, mock LLM): run 1 (N added) → capture every document row's `indexed_at`/`content_hash` → run 2 → the settle line reads `0 added · 0 updated · N unchanged` and every row's `indexed_at`/`content_hash` is unchanged (the re-index would bump `indexed_at`) → commit a new file to the fixture repo → run 3 → only the new file is `added`.
+
+## Tasks
+1. `01_file_hash_column.md` — migration 0025 + the model field.
+2. `02_unchanged_fast_path.md` — the streamed hash, the top-of-`_index_file` check, the shared unchanged helper, the stamping, the branch rewiring, the unit matrix.
+3. `03_e2e_unchanged_second_run.md` — the dedicated E2E: the idempotent second sync is provably free of re-indexing.
+
+## Testing & Quality
+- Unit: the fast-path matrix (text / image / garbage-PDF / pictorial-PDF unchanged → zero LLM calls; changed → full pipeline + stamp; NULL hash → pre-phase behavior + stamp; D1/D4 pins on the fast path; the streamed hash on a >1 MiB file).
+- Integration: the migration upgrade/downgrade round-trip (the house migration-test idiom).
+- E2E: one dedicated Playwright file, run in isolation, mock-only (real git clone).
+- Coverage: **>90%** on new/modified code (`app/`).
+
+## Completion Criteria
+- [ ] `documents.file_hash` exists (nullable String(64), no index) via migration 0025; the upgrade/downgrade round-trips on the test DB
+- [ ] an unchanged text file, image, garbage-layer PDF, and pictorial PDF each cost ZERO llm calls (embed + chat) and zero PDF extraction/vision on the second sync — `summary.unchanged` counts them; `indexed_at`/`content_hash` untouched
+- [ ] a changed file (any byte) re-indexes fully and stamps the new hash; a bytes-changed/content-same file stamps the hash without a content change
+- [ ] the preserved unchanged semantics are pinned: summary-NULL backfill, `created_at_manual` early-return, the date refresh (may go older, counts `dates_updated`, commits), the log-line wordings
+- [ ] `content_hash` and the retriever's exposure of it are byte-identical to pre-phase
+- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` green in isolation; `uv run ruff check . && uv run pyright` clean
+- [ ] no behavior change in completed phases (prune semantics, `limit`/`progress` contracts, the per-source post-index passes, the sync status/result wire)