phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -375,7 +375,7 @@
|
||||
"That source no longer exists — reload the page." cannot be painted over
|
||||
by "unsaved changes" in the tick that wrote it (the defect the task-06
|
||||
story suite walked into); only the next edit (`clearLabelError`) or a
|
||||
collapse clears a note, never the settle path),
|
||||
collapse clears a note, never the settle path,
|
||||
and the Tags token input's chips reuse the phase-89 tag rule
|
||||
(`.source-label-tag` sits in the `.git-source-hidden-count` rule group — no
|
||||
bespoke tag-input CSS). Pinned by
|
||||
@@ -426,6 +426,129 @@
|
||||
also marker-pin the one-read-per-surface contract, as
|
||||
`tests/unit/test_chat_gate.py` does on the chat call site).
|
||||
|
||||
18. **The sync's running count is the WORK, and the pre-scan that predicts
|
||||
it is a scalar read off the loop (phase 194):** `import_sources` predicts
|
||||
the run's change set BEFORE the walk (`_prescan_pending`) so the
|
||||
`progress` hook's pair is the position in the PENDING set (`total` = its
|
||||
size, `done` = the 1-based position in it, held — not advanced — by a file
|
||||
the scan did not predict), which is what makes the Sources page's
|
||||
`N remaining` mean "files left to re-index" instead of "files left to
|
||||
walk" (the owner's "1216 remaining" on a ~10,000-file KB with ~10 changes,
|
||||
2026-10-19). The prediction uses the fast path's OWN predicate —
|
||||
`_file_hash` (the streamed 1 MiB-chunk sha256) against
|
||||
`_pending_hash_stmt`, a SCALAR `SELECT documents.file_hash` — so a stored
|
||||
NULL (no row, or a pre-phase-147 row) is predicted changed AND really
|
||||
re-indexed and stamped by the loop: the prediction never lies about the
|
||||
stored-hash contract. Keep it that way: **never** turn that read into a
|
||||
`select(Document)` / `defer` / `load_only` row (phase 184/185 — a 10,000-file
|
||||
scan would retain 10,000 rows and re-create the memory incident; the scan
|
||||
therefore also reads through a session of its OWN, since a `Session` is not
|
||||
thread-safe and the span's worker runs to completion after a cancel has
|
||||
moved the run on), never add a THIRD count-only walk (the pre-scan REPLACES
|
||||
the phase-64 stats pre-walk, which survives as the `0` branch), never move
|
||||
the scan onto the event loop (phase 179: ONE `asyncio.to_thread` span — a
|
||||
10,000-file hash+SELECT pass inline parks the single uvicorn loop and
|
||||
starves the 2 s `GET /api/sync/status` poll), and never drop the loop's own
|
||||
fresh `_file_hash` (the scan's digest is the prediction, the loop's is the
|
||||
truth). The scan runs ONLY for a caller that passes `progress` AND while
|
||||
`llm.settings.sync_pending_count` is truthy — read ONCE before the source
|
||||
loop (phase 184's read-once discipline) — so the CLI
|
||||
(`scripts/import_docs.py`) and the upload job stay byte-identical and pay
|
||||
nothing; `BOR_SYNC_PENDING_COUNT=0` is the exact phase-193 walk-position
|
||||
rollback and a negative value refuses startup naming the field (the
|
||||
`agent_max_rounds` pattern). `summary.files` (the `import: summary … files=`
|
||||
log count), `seen`, the prune, the date refresh, the terminal `counts_line`
|
||||
and every pre-existing log line are UNCHANGED — the scan builds `pending`,
|
||||
never `seen` — and an empty KB keeps the phase-64 shape (scan total 0, no
|
||||
callback, `total` 0 → no count part). Pinned by
|
||||
`tests/unit/test_sync_pending_count.py` (the gate both ways, the read-once +
|
||||
`to_thread` markers, the exactly-`documents.file_hash` projection, the four
|
||||
predicate states + the ignore-rule parity, the unreadable-file arm, the
|
||||
`done`-as-pending-position, the switch values) and
|
||||
`tests/integration/test_sync_pending_count.py` (the parity across the fresh
|
||||
/ unchanged / incremental arms, `len(session.identity_map) == 0` at the
|
||||
first `progress` call, the heartbeat + thread-identity off-loop pin, and the
|
||||
cancel-at-the-await arm), and through the real Sources page by
|
||||
`tests/e2e/test_sync_pending_count.py` — TWO live module apps over one
|
||||
delay-injecting mock model (the phase-193/107 fixture shape): the first
|
||||
sync (pending == walk, so the phase-193 frames stand), the K-of-N owner
|
||||
case with the count HOLDING at `K remaining` while `current_file` names the
|
||||
UNCHANGED files (the unchanged files in that run cost the phase-118
|
||||
summary backfill — a real, model-bound, loop-free span — which is what
|
||||
makes the hold observable in a browser at all), a SECOND app booted with
|
||||
`BOR_SYNC_PENDING_COUNT=0` rendering the walk numbers for the same fixture
|
||||
(`7 remaining`, not `1`), the no-change run answering `files_total == 0` on
|
||||
every tick, the empty source never leaving `"scan"`, and the 1,000-file
|
||||
pre-scan whose position is caught climbing THROUGH the running server (the
|
||||
phase-179 contract) before the import answers the predicted `2`.
|
||||
|
||||
The STATUS half of the same design (phase 194, task 02) is the machine
|
||||
the page reads: `SyncStatus.phase` is `null` (the model-check +
|
||||
clone/pull prelude) → **`"scan"`** (set immediately before
|
||||
`import_sources`, REPLACING the pre-phase `= "import"` — what runs first
|
||||
there is the pre-scan) → `"import"` (flipped by the FIRST `progress` call
|
||||
INSIDE the existing `_hook` closure — the importer gains no extra
|
||||
callback, so do not add one) → `"overview"` → `"summaries"`, every
|
||||
terminal clearing it. `SyncStatus.scan_done` / `scan_total` (landed by ONE
|
||||
`_scan_hook` passed as `on_scan_progress`) are the scan's own position —
|
||||
reset with the run, `0/0` on the `0` rollback and for a scan that predicted
|
||||
nothing, and KEPT at every terminal beside the final file counts (the
|
||||
phase-64 keep-final-values convention); `current_file` stays `null` through
|
||||
the phase (the scan names no file). The `GET /api/sync/status` payload gains
|
||||
`scan_done` / `scan_total` ADDITIVELY after `files_total` — every
|
||||
pre-existing key keeps its name, type and position, and the additive chain
|
||||
(`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`) is the pin that keeps it
|
||||
that way. Sampling follows phase 187 exactly: the `"scan"` set and the
|
||||
`scan`→`import` flip are TRANSITIONS and `_sample_rss(force=True)`, while
|
||||
`_scan_hook` samples UNFORCED, so a 10,000-file scan costs one `/proc` read
|
||||
per `_RSS_SAMPLE_INTERVAL_S` and not one per file (LOCKED A4). The degenerate
|
||||
empty KB stays `"scan"` (no `progress` call exists to flip it) and the
|
||||
terminal clears it. Pinned by `tests/integration/test_sync_scan_phase.py`
|
||||
(the real-run scan frames + both pairs kept at the terminal — divergent on
|
||||
the incremental arm, 3 walked vs 1 predicted; the choreographed flip frame
|
||||
carrying the predicted denominator; the additive key set idle → terminal;
|
||||
the forced/throttled sample sequence `null → scan → import → overview →
|
||||
summaries → null` with EXACTLY one read inside the scan phase; the empty-KB
|
||||
arm on both sides; the cancel-during-scan terminal with the scan counts kept
|
||||
and nothing indexed; the `0` switch reporting `0/0` scan and the WALK file
|
||||
count) plus the retargeted additive pins in `tests/unit/test_sync_button.py`
|
||||
and `tests/integration/test_sync_api.py` (a canned import that fires no
|
||||
per-file hook stays in `"scan"` — that is why the 409-concurrency pin waits
|
||||
on `"scan"` now).
|
||||
|
||||
The UI half of the same design (phase 194, task 03) is ONE branch:
|
||||
`fmtSyncPhaseLabel` renders `phase === "scan"` as main `Scanning…` (NEVER a
|
||||
file — `current_file` is null in the phase and the branch must not read it)
|
||||
plus the count part `(scan_done/scan_total)` in the SAME count slot the import
|
||||
phase's `N remaining` uses, so `setSyncLabel`'s hidden-toggle and the A4
|
||||
untruncated placement (button title + `#sync-progress`, the sole live region)
|
||||
carry `Scanning… (n/total)` with no new element, no new live region and no
|
||||
new writer (the rendered frames — the scan's climb and the import's held
|
||||
`K remaining` — are pinned on the exact payloads those guards read, served
|
||||
to the real `sources.js` by `tests/e2e/test_sync_pending_count.py`, because
|
||||
the 2 s poll cannot be aimed at a frame; the phase-187 peak clause stays on
|
||||
the TERMINAL line of that same scan-labelled run and appears on NO running
|
||||
label). Keep it that way: the branch is the FIRST arm (the
|
||||
`null`/`import` fall-through therefore covers ONLY those two), it carries no
|
||||
chunk/image part (the scan reads no file), and `scan_total === 0` renders the
|
||||
BARE label (the phase-64 no-count-part contract — the empty KB and the
|
||||
`BOR_SYNC_PENDING_COUNT=0` rollback). The scan's walk position surviving as an
|
||||
`(n/total)` is honest ONLY because the word in front of it is "Scanning": the
|
||||
owner's objection (TODO L6) was the SYNC count wearing the total's shape, so
|
||||
never move a `(n/m)` position onto the import label and never rename this
|
||||
copy to "Syncing". The pair reaches the wrapper through the SAME
|
||||
whole-status pass-through as the summaries / chunk / image pairs at the two
|
||||
sync call sites (the poll tick + the load-time re-attach) — do not re-list
|
||||
the fields there — and `fmtSyncLabel`'s `N remaining` arithmetic, the terminal
|
||||
`counts_line` and the phase-187 `fmtSyncPeakClause` stay byte-identical (the
|
||||
clause stays TERMINAL-only — it rides the settle's counts line exactly as
|
||||
before, never the running scan label, and it still needs no new element).
|
||||
Pinned by `tests/unit/test_sync_scan_label_ui.py` (the branch order, the exact
|
||||
`(n/total)` shape + its `total === 0` guard, the whole-status pass-through, the
|
||||
untouched builder set, and a node run of the REAL label path: `Scanning…
|
||||
(3/9)` identical on main / count / title / box / aria, the bare `Scanning…`
|
||||
at `0/0`, every other phase's label byte-identical).
|
||||
|
||||
## Quick reference
|
||||
```bash
|
||||
podman compose up -d db # start Postgres 17 + pgvector
|
||||
|
||||
Reference in New Issue
Block a user