phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
2026-10-20 06:36:21 -04:00
parent ae080ac7f3
commit a45f744604
55 changed files with 7610 additions and 179 deletions
+124 -1
View File
@@ -375,7 +375,7 @@
"That source no longer exists — reload the page." cannot be painted over
by "unsaved changes" in the tick that wrote it (the defect the task-06
story suite walked into); only the next edit (`clearLabelError`) or a
collapse clears a note, never the settle path),
collapse clears a note, never the settle path,
and the Tags token input's chips reuse the phase-89 tag rule
(`.source-label-tag` sits in the `.git-source-hidden-count` rule group — no
bespoke tag-input CSS). Pinned by
@@ -426,6 +426,129 @@
also marker-pin the one-read-per-surface contract, as
`tests/unit/test_chat_gate.py` does on the chat call site).
18. **The sync's running count is the WORK, and the pre-scan that predicts
it is a scalar read off the loop (phase 194):** `import_sources` predicts
the run's change set BEFORE the walk (`_prescan_pending`) so the
`progress` hook's pair is the position in the PENDING set (`total` = its
size, `done` = the 1-based position in it, held — not advanced — by a file
the scan did not predict), which is what makes the Sources page's
`N remaining` mean "files left to re-index" instead of "files left to
walk" (the owner's "1216 remaining" on a ~10,000-file KB with ~10 changes,
2026-10-19). The prediction uses the fast path's OWN predicate —
`_file_hash` (the streamed 1 MiB-chunk sha256) against
`_pending_hash_stmt`, a SCALAR `SELECT documents.file_hash` — so a stored
NULL (no row, or a pre-phase-147 row) is predicted changed AND really
re-indexed and stamped by the loop: the prediction never lies about the
stored-hash contract. Keep it that way: **never** turn that read into a
`select(Document)` / `defer` / `load_only` row (phase 184/185 — a 10,000-file
scan would retain 10,000 rows and re-create the memory incident; the scan
therefore also reads through a session of its OWN, since a `Session` is not
thread-safe and the span's worker runs to completion after a cancel has
moved the run on), never add a THIRD count-only walk (the pre-scan REPLACES
the phase-64 stats pre-walk, which survives as the `0` branch), never move
the scan onto the event loop (phase 179: ONE `asyncio.to_thread` span — a
10,000-file hash+SELECT pass inline parks the single uvicorn loop and
starves the 2 s `GET /api/sync/status` poll), and never drop the loop's own
fresh `_file_hash` (the scan's digest is the prediction, the loop's is the
truth). The scan runs ONLY for a caller that passes `progress` AND while
`llm.settings.sync_pending_count` is truthy — read ONCE before the source
loop (phase 184's read-once discipline) — so the CLI
(`scripts/import_docs.py`) and the upload job stay byte-identical and pay
nothing; `BOR_SYNC_PENDING_COUNT=0` is the exact phase-193 walk-position
rollback and a negative value refuses startup naming the field (the
`agent_max_rounds` pattern). `summary.files` (the `import: summary … files=`
log count), `seen`, the prune, the date refresh, the terminal `counts_line`
and every pre-existing log line are UNCHANGED — the scan builds `pending`,
never `seen` — and an empty KB keeps the phase-64 shape (scan total 0, no
callback, `total` 0 → no count part). Pinned by
`tests/unit/test_sync_pending_count.py` (the gate both ways, the read-once +
`to_thread` markers, the exactly-`documents.file_hash` projection, the four
predicate states + the ignore-rule parity, the unreadable-file arm, the
`done`-as-pending-position, the switch values) and
`tests/integration/test_sync_pending_count.py` (the parity across the fresh
/ unchanged / incremental arms, `len(session.identity_map) == 0` at the
first `progress` call, the heartbeat + thread-identity off-loop pin, and the
cancel-at-the-await arm), and through the real Sources page by
`tests/e2e/test_sync_pending_count.py` — TWO live module apps over one
delay-injecting mock model (the phase-193/107 fixture shape): the first
sync (pending == walk, so the phase-193 frames stand), the K-of-N owner
case with the count HOLDING at `K remaining` while `current_file` names the
UNCHANGED files (the unchanged files in that run cost the phase-118
summary backfill — a real, model-bound, loop-free span — which is what
makes the hold observable in a browser at all), a SECOND app booted with
`BOR_SYNC_PENDING_COUNT=0` rendering the walk numbers for the same fixture
(`7 remaining`, not `1`), the no-change run answering `files_total == 0` on
every tick, the empty source never leaving `"scan"`, and the 1,000-file
pre-scan whose position is caught climbing THROUGH the running server (the
phase-179 contract) before the import answers the predicted `2`.
The STATUS half of the same design (phase 194, task 02) is the machine
the page reads: `SyncStatus.phase` is `null` (the model-check +
clone/pull prelude) → **`"scan"`** (set immediately before
`import_sources`, REPLACING the pre-phase `= "import"` — what runs first
there is the pre-scan) → `"import"` (flipped by the FIRST `progress` call
INSIDE the existing `_hook` closure — the importer gains no extra
callback, so do not add one) → `"overview"` → `"summaries"`, every
terminal clearing it. `SyncStatus.scan_done` / `scan_total` (landed by ONE
`_scan_hook` passed as `on_scan_progress`) are the scan's own position —
reset with the run, `0/0` on the `0` rollback and for a scan that predicted
nothing, and KEPT at every terminal beside the final file counts (the
phase-64 keep-final-values convention); `current_file` stays `null` through
the phase (the scan names no file). The `GET /api/sync/status` payload gains
`scan_done` / `scan_total` ADDITIVELY after `files_total` — every
pre-existing key keeps its name, type and position, and the additive chain
(`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`) is the pin that keeps it
that way. Sampling follows phase 187 exactly: the `"scan"` set and the
`scan`→`import` flip are TRANSITIONS and `_sample_rss(force=True)`, while
`_scan_hook` samples UNFORCED, so a 10,000-file scan costs one `/proc` read
per `_RSS_SAMPLE_INTERVAL_S` and not one per file (LOCKED A4). The degenerate
empty KB stays `"scan"` (no `progress` call exists to flip it) and the
terminal clears it. Pinned by `tests/integration/test_sync_scan_phase.py`
(the real-run scan frames + both pairs kept at the terminal — divergent on
the incremental arm, 3 walked vs 1 predicted; the choreographed flip frame
carrying the predicted denominator; the additive key set idle → terminal;
the forced/throttled sample sequence `null → scan → import → overview →
summaries → null` with EXACTLY one read inside the scan phase; the empty-KB
arm on both sides; the cancel-during-scan terminal with the scan counts kept
and nothing indexed; the `0` switch reporting `0/0` scan and the WALK file
count) plus the retargeted additive pins in `tests/unit/test_sync_button.py`
and `tests/integration/test_sync_api.py` (a canned import that fires no
per-file hook stays in `"scan"` — that is why the 409-concurrency pin waits
on `"scan"` now).
The UI half of the same design (phase 194, task 03) is ONE branch:
`fmtSyncPhaseLabel` renders `phase === "scan"` as main `Scanning…` (NEVER a
file — `current_file` is null in the phase and the branch must not read it)
plus the count part `(scan_done/scan_total)` in the SAME count slot the import
phase's `N remaining` uses, so `setSyncLabel`'s hidden-toggle and the A4
untruncated placement (button title + `#sync-progress`, the sole live region)
carry `Scanning… (n/total)` with no new element, no new live region and no
new writer (the rendered frames — the scan's climb and the import's held
`K remaining` — are pinned on the exact payloads those guards read, served
to the real `sources.js` by `tests/e2e/test_sync_pending_count.py`, because
the 2 s poll cannot be aimed at a frame; the phase-187 peak clause stays on
the TERMINAL line of that same scan-labelled run and appears on NO running
label). Keep it that way: the branch is the FIRST arm (the
`null`/`import` fall-through therefore covers ONLY those two), it carries no
chunk/image part (the scan reads no file), and `scan_total === 0` renders the
BARE label (the phase-64 no-count-part contract — the empty KB and the
`BOR_SYNC_PENDING_COUNT=0` rollback). The scan's walk position surviving as an
`(n/total)` is honest ONLY because the word in front of it is "Scanning": the
owner's objection (TODO L6) was the SYNC count wearing the total's shape, so
never move a `(n/m)` position onto the import label and never rename this
copy to "Syncing". The pair reaches the wrapper through the SAME
whole-status pass-through as the summaries / chunk / image pairs at the two
sync call sites (the poll tick + the load-time re-attach) — do not re-list
the fields there — and `fmtSyncLabel`'s `N remaining` arithmetic, the terminal
`counts_line` and the phase-187 `fmtSyncPeakClause` stay byte-identical (the
clause stays TERMINAL-only — it rides the settle's counts line exactly as
before, never the running scan label, and it still needs no new element).
Pinned by `tests/unit/test_sync_scan_label_ui.py` (the branch order, the exact
`(n/total)` shape + its `total === 0` guard, the whole-status pass-through, the
untouched builder set, and a node run of the REAL label path: `Scanning…
(3/9)` identical on main / count / title / box / aria, the bare `Scanning…`
at `0/0`, every other phase's label byte-identical).
## Quick reference
```bash
podman compose up -d db # start Postgres 17 + pgvector