brain-of-reese/.agents/phases/complete/194_sync_pending_count/00_phase.md

Phase 194 — "N remaining" counts the files that will actually change

Source: owner report 2026-10-19 — "I just synced the prod brain of reese and it said '1216 remaining' when it only had like 10 files to sync" (the prod KB holds ~10,000 importable files; that run changed ~10). Story: n/a (owner directive — phase 193's confirmed assumption A1 proved wrong in production) Depends on: 193_sync_remaining_count (complete) — builds on the phase-64/98/128/129/152/179/184/185/187 sync lineage.

Evidence (diagnosed from the code, read this not the chat)

1. The phase-193 label counts the WALK, not the WORK. frontend/assets/sources.js fmtSyncLabel renders N remaining with N = files_total − files_done (the arithmetic is fine). But app/rag/importer.py's pre-walk (L949–962) sets total to the COMBINED COUNT OF EVERY IMPORTABLE FILE across all sources (the 10,000), and the per-file hook (L1034–1039) reports done = summary.files (the 1-based position across ALL files walked). The fast path (phase 147/185) skips re-embedding for the unchanged 9,990 but still COUNTS them. Prod: "1216 remaining" = "8,784 files left to walk", while the actual work was ~10 files. Phase 193's E2E suite passed because its fixture KB was a FRESH import where walk == work — the incremental case the owner just hit was never pinned.
2. The change signal already exists per file. _file_hash (L704–724 — the sha256 of the RAW bytes, streamed in 1 MiB chunks) compared against documents.file_hash (nullable String(64), no index — looked up through the unique (source, path) constraint, app/models.py L119/L138–142) is EXACTLY the fast path's change predicate: stored NULL (no row, or a pre-phase row) or != digest ⇔ the file will be re-indexed (added/updated). A pre-scan reusing that predicate predicts the change set with the SAME semantics the processing loop then applies — pre-phase NULL rows are predicted changed AND actually re-indexed (stamped) by the loop, so the prediction never lies about the stored-hash contract.
3. The pre-scan must be OFF the loop. A 10,000-file hash+SELECT span run inline would park the single uvicorn loop for seconds — the exact phase-179 defect class pinned by tests/integration/test_sync_git_freeze.py (the 3 s git stub froze the 2 s status poll) and tests/e2e/test_sync_git_freeze.py. The git span moved to asyncio.to_thread (phase 179); the pre-scan does the same (the worker runs to completion on cancel; the run stops at the next phase-134 boundary — the CancelledError clause in _run_sync, L178–190 contract).
4. Only the UI sync pays anything. import_sources has exactly two callers: app/api/sync.py (the UI sync — the ONLY one that passes progress) and scripts/import_docs.py L407 (the CLI — no progress). Uploads have no file-level progress at all (phase 90 A2 — app/api/git_sources.py L59; their label renders bare, the test_archive_upload_sources.py pins). Gating the pre-scan on progress is not None (the phase-64 skip rule) keeps every non-UI caller byte-identical.
5. The count is a prediction, and the drift edges are bounded. The pre-scan runs seconds before the processing loop reaches a given file. A predicted file that turns out unchanged (reverted mid-run) still advances the count (it was predicted — total counted it) so the count reaches 0; a file changed AFTER the scan is indexed for real but does not advance the count, and the phase-193 total − done > 0 guard hides the excess (never negative, never a stale "0 remaining"). The terminal counts_line ("N added · M updated …") stays the exact surface — untouched.

Design (shared by all tasks — the executor reads this, not the chat)

• D1 — the pre-scan (task 01). import_sources gains ONE optional callback on_scan_progress: Callable[[int, int], None] | None = None (the 1-based scan position + the importable total — the "only when provided", "hooks only assign fields", no-try/except contract of its three siblings). While progress is not None AND llm.settings.sync_pending_count is truthy (read ONCE before the source loop, the phase-184 read-once discipline — 0 = rollback to the phase-193 walk-position semantics, negative value refuses startup loudly, the house kill-switch pattern):
  • the EXISTING stats-only pre-walk is REPLACED (not augmented — no third walk) by ONE asyncio.to_thread span that, per root with the EXACT iter_importable_files rules of the processing loop (same ignore / include_hidden / image_extension_set / images-toggle gating): (a) takes the per-root file list and its length (the scan total — the same number the old pre-walk computed), then (b) per file: _file_hash(path) (the existing streamed helper) + ONE scalar SELECT documents.file_hash WHERE source = :s AND path = :p (the only column — NO entity load, NO defer/load_only row, no session retention, one light metadata read per file; phase 184/185 compliant: it is not a whole-entity select(Document), it defers nothing because it loads no row, and the processing loop's _doc_row_stmt fast-path row load stays the only ROW load) → pending = {(source, rel)} for every file whose stored hash is NULL or != digest;
  • on_scan_progress(position, total) fires per file (the scan position for the UI's "Scanning… (n/total)" label — called from the worker thread; field assignments are GIL-atomic, the phase-179 off-loop contract), and files_total becomes len(pending) while the per-file progress call's done becomes the 1-based POSITION IN THE PENDING SET (it advances for every predicted file, whether or not it turns out changed at processing time — the count and its denominator stay on the SAME set so the arithmetic never runs away);
  • summary.files (the import: summary … files= log count — every file the loop processed, append-only log discipline) is UNTOUCHED; the sync: done/cancelled/failed … files_done=… files_total=… lines keep their EXACT shape, now carrying the pending counts (the module docstring names the semantic shift);
  • the processing loop's FRESH per-file _file_hash is KEPT (the double hash is deliberate: the pre-scan's digest is the prediction, the loop's is the truth — a file edited between the two is still detected and re-indexed; the second read is page-cached — the cost the owner accepted, A3);
  • on_scan_progress is called only when provided; progress=None callers (the CLI) skip the pre-scan entirely — byte-identical behavior (the phase-64 skip rule, extended);
  • the degenerate empty KB: the scan total is 0, no callback fires, the run's phase stays "scan" until the first progress call (which there is none) or the next transition/terminal — the UI renders bare "Scanning…" (the total === 0 guard → no count part) and the terminal clears the phase; documented, pinned in the task-05 arm;
  • docstrings: the module progress paragraph (the done/total semantic shift), the new on_scan_progress paragraph, the to_thread/cancel/cost paragraphs, the sync_pending_count gating clause.
• D2 — the kill switch (task 01). app/config.py gains sync_pending_count: int = 1 (BOR_SYNC_PENDING_COUNT): 1 (default) = the pre-scan pending-count semantics; 0 = the phase-193 WALK-position semantics (the stats pre-walk + done = summary.files — the full rollback the owner gets if the cost ever bites); a negative value refuses startup with the field named (the agent_max_rounds house pattern). .env.example + README settings documentation. The switch is read ONCE per run (the phase-184 discipline — a mid-run settings save cannot change the semantics between two files of the same walk).
• D3 — the phase machine + status (task 02). SyncStatus.phase gains "scan" (additive — the machine: the model-check + clone/pull prelude reports null, then "scan" (set immediately before import_sources, REPLACING today's = "import"), "import" (flipped by the FIRST progress call while phase == "scan" — the flip point is the existing per-file hook, so no extra callback), "overview", "summaries" — the no-gap summaries skip still keeps the run in "import"). The flip + the pre-import_sources set are phase transitions, so each gets the phase-187 FORCED _sample_rss() (the per-file _scan_hook calls the UNFORCED throttled sample, the progress-hook convention — one /proc read per 5 s, never one per file). SyncStatus gains scan_done / scan_total (int, default 0; reset with the run; KEPT at every terminal — the phase-64 keep-final-values convention, beside the final file counts). The status payload gains the same two keys ADDITIVELY (placed after files_total; every pre-existing key keeps its name, type, position, and meaning — the phase-187 additive-only discipline). current_file stays null during the scan (the scan names no file). Docstrings: the module phase-machine paragraph (L188–200 region), the SyncStatus class paragraph (the phase-64/98/187 clauses gain the scan clause), the GET /api/sync/status docstring field paragraph.
• D4 — the UI scan label (task 03). fmtSyncPhaseLabel (the phase-aware wrapper — frontend/assets/sources.js) gains ONE explicit "scan" branch in its decision tree: main text Scanning… (no file — current_file is null in the phase, the branch must not read it), count part (n/total) rendered in the SAME count slot the import phase's N remaining uses (n = scan_done, total = scan_total; the total === 0 guard → no count part, the phase-64 contract); the caller passes the scan pair through the existing call sites that already forward the summaries/chunk/image parts (the phase value selects which pair the wrapper consumes — the summaries-branch pattern). The null prelude + "import" fallback clause now covers ONLY those two. fmtSyncLabel's N remaining arithmetic is BYTE-IDENTICAL (the phase-193 pins stay unedited — the server-side semantic change flows through the SAME builder unchanged). The A4 untruncated placement (button title + #sync-progress) is untouched — Scanning… (n/total) rides it automatically; the phase-187 · peak N MB trailing clause still lands on the #sync-progress counts line while the scan label is visible. The docstring's decision-tree paragraph (L157–167 region) is updated. The walk position surviving as (n/total) is HONEST for a scan — it is labeled "Scanning", never "Syncing" (the owner's objection, TODO L6, was the sync count wearing the total's shape).
• D5 — the pinned suites move (task 04). See the task for the definitive list (the count-frame pins retarget to the pending-set semantics where the fixture KB is NOT fresh; the fresh-fixture pins (chunk progress, label mobile, upload) stay unedited; the phase-184/185/179 integration pins get the pre-scan's statements distinguished or are verified unaffected).

ASSUMPTIONS (owner-confirmed 2026-10-19, questionnaire)

• A1: N remaining counts down the pre-scan's PREDICTED change set (files_total = len(pending), files_done = the 1-based position in it); the count advances on the prediction set (a predicted-then-unchanged file still advances it — the count reaches 0; a post-scan change is indexed but uncounted — the phase-193 guard hides the excess). First-time imports (empty DB) count the full walk exactly as phase 193 did; a no-change run shows no count part.
• A2: the pre-scan is a NEW VISIBLE "scan" phase — Scanning… (n/total) before the import phase; the scan's walk position is the only surviving n/m shape, and it is labeled a scan.
• A3: the cost is accepted: one extra full read + sha256 + one scalar metadata SELECT per importable file per run (the second read is page-cached; the span is off-loop via to_thread, the phase-179 rule). The rollback knob BOR_SYNC_PENDING_COUNT=0 restores the phase-193 walk-position semantics; the CLI and the upload job pay nothing.

Context

• app/rag/importer.py — import_sources (L756+, the pre-walk L949–962, the per-file progress call L1034–1039), _file_hash (L704–724), _doc_row_stmt (L725+), the module docstring's progress/chunk/image hook paragraphs.
• app/api/sync.py — the module docstring (the phase-179/134/98 paragraphs, L100–200 region), SyncStatus (L308+), GET /api/sync/status (L495+, the payload dict L548+), _run_sync (L651+, the reset block, the _hook/_scan_hook/_chunk_hook/_image_hook closures L815–860 region, the _status.phase = "import" set L880, the terminal log lines L1112/L1133).
• app/config.py — the BOR_SYNC_* settings block + validators (the agent_max_rounds kill-switch pattern); .env.example.
• app/models.py — Document (L117+, the unique (source, path), the file_hash column contract L128–142).
• frontend/assets/sources.js — fmtSyncPhaseLabel (L157–167 docstring, the decision tree), fmtSyncLabel (L1043–1071 — UNTOUCHED), the status call sites (L1121–1127, L1449–1458, L1621–1652), setSyncLabel/A4 (UNTOUCHED).
• scripts/import_docs.py L407 (the CLI caller — progress absent — the byte-identity pin), app/api/git_sources.py L59 (the upload's no-file-progress contract — UNTOUCHED).
• Pinned suites: tests/e2e/test_sync_remaining_count.py (phase 193's own — the count-frame pins + the status key-set pin), tests/e2e/test_sync_chunk_progress.py, test_sync_label_mobile.py, test_sync_upload_progress.py, test_archive_upload_sources.py (the upload's "no N remaining" pins), tests/integration/test_import_fast_path_columns.py (phase 185 — the per-file SELECT pins), test_import_session_retention.py (phase 184), test_sync_git_freeze.py (phase 179 — the loop-freeze pin), test_sync_session_reset.py, test_fast_path_deferred_content.py, test_sync_summary_visibility.py, test_sync_image_progress.py, test_sync_progress_box.py, test_sync_status_freshness.py, test_sync_peak_memory_visible.py, test_smoke.py.

Tasks

1. 01_prescan_importer.md — the importer's pre-scan (the D1 spec) + the BOR_SYNC_PENDING_COUNT kill switch (the D2 spec) + the unit/integration pins.
2. 02_sync_phase_status.md — the "scan" phase + scan_done/scan_total status fields + the additive payload + the docstrings (the D3 spec) + the integration pins.
3. 03_ui_scan_label.md — the fmtSyncPhaseLabel scan branch (the D4 spec) + the JS marker pins.
4. 04_retarget_pinned_suites.md — the pinned-suite retarget (the D5 list); the full suite green.
5. 05_e2e_and_gates.md — the phase's own E2E suite (the pending-count-down through the real UI), the full gates, the docs pass, the atomic commit.

Testing & Quality

• Unit (marker pins): the pre-scan gate (the progress is not None + sync_pending_count read-once), the scalar SELECT's column shape (no entity load), the pending predicate (NULL / mismatch / match), the done-as-pending-position contract, the kill-switch value handling (1 / 0 / negative).
• Integration: the pending set on fresh / unchanged / changed / pre-phase-NULL fixtures; the off-loop pin (the delayed pre-scan span keeps the loop responsive — the phase-179 pattern); the cancel-during-prescan terminal; the phase-184/185/179 pins retargeted or verified; the phase machine sequence (null → scan → import → … → terminal null) + the kept scan counts + the forced/throttled RSS samples.
• E2E (regression, task 04): the retargeted pins green; the fresh-fixture + upload pins green UNEDITED.
• E2E (phase, AGENTS rule 4, task 05): tests/e2e/test_sync_pending_count.py, run in isolation — the first-sync arm, the INCREMENTAL arm (the owner's prod case: K of N changed, the count holds at K remaining while the walk moves over unchanged files, hidden at the final predicted file, terminal K updated), the no-change arm (no count part), the kill-switch arm (0 → the phase-193 walk semantics), the empty-KB arm, the peak-clause-on-scan-line pin.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).
• Lint/types: uv run ruff check . && uv run pyright.

Documentation & Consistency

• Final pass reviews README.md and AGENTS.md in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
• README.md: the sync section's running-label clause gains the scan phase + the pending-count semantics (the phase-193 N remaining example becomes the predicted-change-set one) and the BOR_SYNC_PENDING_COUNT setting (the .env.example entry is part of task 01); AGENTS.md — verify whether any clause names the running count part (phase 193 verified it does not — if still absent, do NOT add; the BOR_SYNC_PENDING_COUNT setting row joins the settings documentation wherever the README/AGENTS settings table lives).

Completion Criteria

• N remaining counts down the pre-scan's predicted change set (files_total = len(pending), files_done = the pending position) on a progress caller; a first-time import counts the full walk (A1's degenerate parity); a no-change run shows no count part — the E2E arms a/b/c.
• the scan phase is visible (Scanning… (n/total), the position climbs, the total === 0 guard) and flips to import on the first progress call; scan_done/scan_total reset with the run and survive the terminals — the E2E arms a/e + the integration phase-machine pin.
• BOR_SYNC_PENDING_COUNT=0 restores the phase-193 walk-position semantics byte-for-byte (the E2E kill-switch arm + the marker pins); a negative value refuses startup with the field named.
• the pre-scan span never parks the loop (the off-loop pin); a cancel during it lands the cancelled terminal (the phase-134 contract); the CLI and the upload job are byte-identical (the unedited pins).
• the chunk / image / summaries parts, the fmtSyncLabel arithmetic, the terminal counts_line, summary.files and every pre-existing log line are byte-identical (the unedited pins green); the status payload is additive-only (the key-set pin).
• test suite green, coverage >90%; uv run ruff check . && uv run pyright clean; README + AGENTS consistent (the final pass).
• one atomic Conventional Commits commit with --no-gpg-sign.