phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
+103
-23
@@ -159,7 +159,8 @@
|
||||
* "Writing KB overview…", "summaries" → "Summarizing folders…
|
||||
* <source/folder> (n/m)" (the folder part omitted while
|
||||
* current_summary is null — the phase's first poll), anything else
|
||||
* (the null prelude, "import") → today's "Syncing… <file> (n/m)"
|
||||
* (since phase 194 ONLY the null prelude and "import") → today's
|
||||
* "Syncing… <file> (n/m)"
|
||||
* byte-identical (the count part's TEXT is `N remaining` — the distance
|
||||
* to the end — since phase 193). The UPLOAD job (its status has no phase) keeps the
|
||||
* bare label. The untruncated label rides the button title (now set
|
||||
@@ -475,6 +476,31 @@
|
||||
* announces today's wording plus the clause, and the number is the
|
||||
* PROCESS high-water mark app/core/memory.py reports (never the host's
|
||||
* cgroup figure — that one counts page cache).
|
||||
*
|
||||
* Phase 194 (task 03) — the running label counts the WORK, and the
|
||||
* pre-scan says so. The owner's prod report (2026-10-19): "1216
|
||||
* remaining" on a ~10,000-file KB with ~10 files to sync — the
|
||||
* phase-193 count-down was arithmetically right but read the WALK: the
|
||||
* server now pre-scans (one streamed sha256 + one scalar
|
||||
* documents.file_hash read per importable file, off the event loop) and
|
||||
* reports files_done / files_total as the position in the PREDICTED
|
||||
* change set, so `N remaining` is the files left to RE-INDEX
|
||||
* (fmtSyncLabel's arithmetic is byte-identical — only its INPUTS
|
||||
* moved). The scan itself is a visible phase (A2):
|
||||
* fmtSyncPhaseLabel gains ONE "scan" branch rendering
|
||||
* "Scanning… (n/total)" — main "Scanning…" (never a file:
|
||||
* current_file is null while the scan runs) + the count part "(n/total)"
|
||||
* over the status's own scan_done / scan_total, in the SAME
|
||||
* count slot (so setSyncLabel's hidden-toggle and the A4 untruncated
|
||||
* placement — button title + #sync-progress — carry it with no new
|
||||
* element), the phase-64 `total === 0` guard rendering the bare
|
||||
* "Scanning…" for an empty KB, and no chunk/image part (the scan reads
|
||||
* no file). The scan's walk position surviving as (n/total) is honest
|
||||
* because the word in front of it is "Scanning", never "Syncing". The
|
||||
* pair reaches the wrapper the way the summaries, chunk and image pairs
|
||||
* already do — the WHOLE status object at the two sync call sites, the
|
||||
* `phase` value selecting which pair renders. The peak clause, the
|
||||
* terminal counts line and the upload's bare label are untouched.
|
||||
*/
|
||||
|
||||
import { fetchIsAdmin } from "./header.js";
|
||||
@@ -748,16 +774,25 @@ export async function mount(root) {
|
||||
* status endpoints — GET /api/sync/status +
|
||||
* GET /api/git-sources/upload/status — and applies this decision
|
||||
* tree, in order (startSyncPolling):
|
||||
* 1. sync running → the phase-aware label (Phase 98 task 02, D2)
|
||||
* — the null-phase prelude + "import" →
|
||||
* "Syncing… <file> N remaining" — bare "Syncing…"
|
||||
* until the import's first file (clone/pull,
|
||||
* A4, byte-identical); "overview" → "Writing KB
|
||||
* overview…"; "summaries" → "Summarizing
|
||||
* folders… <source/folder> (n/m)" (the folder
|
||||
* part omitted while current_summary is null —
|
||||
* the phase's first poll) — the post-import span
|
||||
* where the file count sits still;
|
||||
* 1. sync running → the phase-aware label (Phase 98 task 02, D2;
|
||||
* Phase 194 task 03 names the scan) —
|
||||
* "scan" → "Scanning… (n/total)" (the pre-scan's
|
||||
* own walk position over scan_done /
|
||||
* scan_total, no file — current_file is null in
|
||||
* that phase); the null-phase prelude +
|
||||
* "import" → "Syncing… <file> N remaining" — bare
|
||||
* "Syncing…" until the import's first file
|
||||
* (clone/pull, A4, byte-identical; since phase
|
||||
* 194 that N is the files left to RE-INDEX, the
|
||||
* pre-scan's predicted change set); "overview" →
|
||||
* "Writing KB overview…"; "summaries" →
|
||||
* "Summarizing folders… <source/folder> (n/m)" (the
|
||||
* folder part omitted while current_summary is
|
||||
* null — the phase's first poll) — the
|
||||
* post-import span where the file count sits
|
||||
* still; the WHOLE status object carries every
|
||||
* pair the label reads (scan, file, chunk, image,
|
||||
* summary) and `phase` picks which one renders;
|
||||
* 2. upload running → BARE "Importing…" — the background upload
|
||||
* RUN (phase 90: unpack + register only, no
|
||||
* scan — its status never carries a file or
|
||||
@@ -1073,20 +1108,39 @@ export async function mount(root) {
|
||||
}
|
||||
|
||||
/* Phase 98 (task 02, D2) + Phase 128 (task 01, the PARTS rework)
|
||||
* + Phase 152 (task 03): the SYNC job's phase-aware running label —
|
||||
* + Phase 152 (task 03) + Phase 194 (task 03, D4/A2): the SYNC job's
|
||||
* phase-aware running label —
|
||||
* the post-import span where the file count sits still (the owner's
|
||||
* "the number pauses for a really long time"): "overview" →
|
||||
* "Writing KB overview…" (no count part); "summaries" → main
|
||||
* "the number pauses for a really long time").
|
||||
* "overview" → "Writing KB overview…" (no count part); "summaries" → main
|
||||
* "Summarizing folders… <source/folder>" + count "(n/m)" — the
|
||||
* folder part is omitted while current_summary is null (the phase's
|
||||
* first poll) and the COUNT IS ALWAYS EMITTED in this phase (the
|
||||
* hook always carries a total) and the CHUNK and IMAGE parts stay
|
||||
* empty (the summaries phase's own (n/m) is its position — the
|
||||
* status's chunk + image fields are cleared at this transition);
|
||||
* anything else (the null prelude, "import") → the phase-64
|
||||
* "scan" (phase 194) → main "Scanning…" — NEVER a file, because
|
||||
* `current_file` is null while the importer pre-scans and this
|
||||
* branch never reads it — + count "(n/total)" over the status's OWN
|
||||
* scan pair (scan_done / scan_total, the pre-scan's walk
|
||||
* position), in the SAME count slot the import phase's
|
||||
* `N remaining` uses (so the hidden-toggle + the A4 untruncated
|
||||
* placement — button title + #sync-progress — carry "Scanning…
|
||||
* (n/total)" automatically), with the phase-64 `scan_total === 0`
|
||||
* guard (an empty KB, or a scan that has not reported its
|
||||
* denominator yet → the bare "Scanning…", no count part) and NO
|
||||
* chunk/image part (the scan reads no file: it embeds no chunk and
|
||||
* describes no image). The walk position surviving as "(n/total)"
|
||||
* is HONEST for a scan: the owner's objection (TODO L6) was the
|
||||
* SYNC count wearing the total's shape — "1216 remaining" on a
|
||||
* ~10,000-file KB — and this label says "Scanning", never
|
||||
* "Syncing"; anything else — ONLY the two states left now that
|
||||
* "scan" names itself (the null prelude and "import") → the phase-64
|
||||
* fmtSyncLabel parts (same builder, same fields, same order —
|
||||
* since phase 193 its count part reads `N remaining`, N =
|
||||
* total - done, hidden at the final file; the summaries (n/m)
|
||||
* since phase 193 its count part reads `N remaining` with
|
||||
* N = total - done, hidden at the final file; since phase 194 that
|
||||
* pair is the position in the pre-scan's PREDICTED change set, so
|
||||
* what counts down is the WORK and not the walk; the summaries (n/m)
|
||||
* folder position and the chunk/image parts below stay as they
|
||||
* are) + the phase-129 CHUNK part: "· chunk n/m" ONLY while
|
||||
* chunks_total > 0 (a file's embed batch is in flight — n the
|
||||
@@ -1105,6 +1159,25 @@ export async function mount(root) {
|
||||
* title + #sync-progress like every running label (A4 —
|
||||
* setSyncLabel). */
|
||||
function fmtSyncPhaseLabel(status) {
|
||||
if (status.phase === "scan") {
|
||||
// Phase 194 (task 03, D4/A2): the pre-scan is its OWN visible
|
||||
// phase, so it gets its OWN branch. The pair here is the scan's
|
||||
// walk position — the honest home for an (n/total), because the
|
||||
// word in front of it is "Scanning"; the sync label's count part
|
||||
// stays the phase-193 count-down (fmtSyncLabel, untouched). The
|
||||
// runner reports no file at all in this phase and this branch
|
||||
// never reads one; the scan names no file, so no chunk part and
|
||||
// no image part.
|
||||
return {
|
||||
main: "Scanning…",
|
||||
count:
|
||||
status.scan_total > 0
|
||||
? `(${status.scan_done}/${status.scan_total})`
|
||||
: "", // 0/0: the empty KB + the scan's own first frame
|
||||
chunk: "", // the scan embeds nothing — no chunk position
|
||||
image: "", // the scan describes nothing — no image position
|
||||
};
|
||||
}
|
||||
if (status.phase === "overview") {
|
||||
return { main: "Writing KB overview…", count: "", chunk: "", image: "" };
|
||||
}
|
||||
@@ -1117,8 +1190,10 @@ export async function mount(root) {
|
||||
image: "", // the image position belongs to the import phase only
|
||||
};
|
||||
}
|
||||
// null phase (the prelude) or "import" — the phase-64 label
|
||||
// shape (since phase 193 its count part reads `N remaining`) +
|
||||
// null phase (the prelude) or "import" — and since phase 194 ONLY
|
||||
// those two ("scan" has its own branch above) — the phase-64 label
|
||||
// shape (since phase 193 its count part reads `N remaining`, and
|
||||
// since phase 194 that pair is the pending position) +
|
||||
// the phase-129
|
||||
// chunk part (hidden before the in-flight file's first batch) +
|
||||
// the phase-152 image part (hidden before the in-flight PDF's
|
||||
@@ -1442,8 +1517,10 @@ export async function mount(root) {
|
||||
}
|
||||
// 1. sync running: the phase-aware label (phase 98 task 02 — the
|
||||
// null prelude + "import" keep the byte-identical file label,
|
||||
// "overview" / "summaries" name themselves; the WHOLE status
|
||||
// goes in for the phase fields).
|
||||
// "scan" / "overview" / "summaries" name themselves; the WHOLE status
|
||||
// goes in for the phase fields — the scan pair (scan_done /
|
||||
// scan_total, phase 194) rides it exactly like the summaries,
|
||||
// chunk and image pairs do, so no field is re-listed here).
|
||||
if (syncStatus.state === "running") {
|
||||
enterSyncRunningState(
|
||||
"sync", syncStatus.current_file, syncStatus.files_done, syncStatus.files_total,
|
||||
@@ -1604,8 +1681,11 @@ export async function mount(root) {
|
||||
* the current catalog). Phase 98 (task 02): a RUNNING sync re-enters
|
||||
* with the WHOLE status — the label is phase-aware, so a mid-
|
||||
* summaries reload re-enters with the summaries label (the
|
||||
* never-stale contract); a RUNNING upload re-attach stays on the
|
||||
* bare label (its status has no phase — D2). */
|
||||
* never-stale contract) and, since phase 194, a reload during the
|
||||
* pre-scan re-enters with "Scanning… (n/total)" — the scan pair rides
|
||||
* the SAME whole-status pass-through as every other pair; a RUNNING
|
||||
* upload re-attach stays on the bare label (its status has no phase —
|
||||
* D2). */
|
||||
async function initSyncButton() {
|
||||
if (!syncBtn) return;
|
||||
if (!(await fetchIsAdmin())) return;
|
||||
|
||||
Reference in New Issue
Block a user