phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
2026-10-20 06:36:21 -04:00
parent ae080ac7f3
commit a45f744604
55 changed files with 7610 additions and 179 deletions
+62 -11
View File
@@ -93,6 +93,18 @@ on all three terminals (success, cancelled, failed) plus the importer's
``import: summary …``. The fields are APPENDED (LOCKED A2), so every
pre-phase prefix below is byte-identical and the pins here assert BOTH
halves of that contract.
Phase 194 (task 02): the status gains the pre-scan. The payload is
ADDITIVE — ``scan_done`` / ``scan_total`` after ``files_total``, the
additive-chain pins below updated to the new key set (the idle shape and
both ``_PRE_187_STATUS_KEYS`` assertions) — and ``files_done`` /
``files_total`` now carry the PREDICTED-change-set position on a progress
caller. One existing pin retargets with it:
``test_double_trigger_while_running_returns_409`` waits on ``"scan"``
instead of ``"import"``, because its canned import fires NO per-file hook
and the scan→import flip lives in that hook — a run that reports no file
stays in the pre-import phase (the full sequence with a live hook is
pinned in ``tests/integration/test_sync_scan_phase.py``).
"""
from __future__ import annotations
@@ -375,6 +387,11 @@ class FakeImportSources:
# image fields stay 0/0, like the chunk fields of a no-batch
# import).
self.image_hooks: list[object] = []
# Phase 194 (task 02): the PRE-SCAN position hook the runner
# passes (a live closure while wired, None if the wiring
# regresses) — the canned fake never fires it, so the status
# scan fields stay 0/0 like a run that scanned nothing.
self.scan_hooks: list[object] = []
async def __call__(
self,
@@ -387,6 +404,7 @@ class FakeImportSources:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -397,6 +415,7 @@ class FakeImportSources:
self.progress_hooks.append(progress)
self.chunk_hooks.append(on_chunk_progress)
self.image_hooks.append(on_image_progress)
self.scan_hooks.append(on_scan_progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
self.doc_dates_maps.append(doc_dates_by_root or {})
@@ -572,6 +591,10 @@ def test_admin_sync_success_reports_full_detail(
# (the dataclass defaults).
"images_done": 0,
"images_total": 0,
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
# (the dataclass defaults).
"scan_done": 0,
"scan_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -703,17 +726,27 @@ def test_double_trigger_while_running_returns_409(
assert r.status_code == 409
assert r.json() == {"detail": "a sync is already running"}
# Phase 179 (task 02): WAIT for the import phase — the run's
# progress is concurrent now that the git ops left the loop, so the
# next request no longer inherits a run that already ran past them.
body = _wait_phase(sync_client, "import")
# Phase 179 (task 02): WAIT for the import to have started — the
# run's progress is concurrent now that the git ops left the loop,
# so the next request no longer inherits a run that already ran past
# them. Phase 194 (task 02): the phase the wait targets is now
# ``"scan"`` — the canned fake import reports NO file, and the
# scan→import flip lives in the runner's per-file progress hook, so
# a hook-less run legitimately stays in the pre-import phase (the
# sequence with a real hook is pinned in test_sync_scan_phase.py and
# in the phase-98 gated test below).
body = _wait_phase(sync_client, "scan")
assert body["state"] == "running"
assert body["started_at"] is not None
assert body["finished_at"] is None
assert body["error"] is None
# Wiring pin (phase 194, task 02): the runner passed its live
# scan-position closure, and a fake that fires nothing keeps the
# position at 0/0.
assert len(fake_import.scan_hooks) == 1 and fake_import.scan_hooks[0] is not None
assert body["scan_done"] == 0 and body["scan_total"] == 0
# Phase 98 (task 01): the in-flight run (parked in the fake
# import's delay) reports the import phase — the summary span has
# not started, so its counters are 0/0.
# import's delay) reports no summary span — its counters are 0/0.
assert body["current_summary"] is None
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
@@ -769,6 +802,10 @@ class _GatedImport:
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
# Phase 194 (task 02): the PRE-SCAN position hook the runner
# passes (recorded for the wiring pin — the gated fake never
# fires it, so the status scan fields stay 0/0).
self.scan_hook: Callable[[int, int], None] | None = None
async def __call__(
self,
@@ -781,10 +818,12 @@ class _GatedImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
self.scan_hook = on_scan_progress
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
@@ -1937,6 +1976,7 @@ def test_import_error_is_reported_with_credentials_masked(
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -2698,6 +2738,12 @@ _PRE_187_STATUS_KEYS = frozenset(
#: Phase 187 (task 03): the two additive memory keys.
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the
#: next link in the same chain (the payload is exactly the pre-187 set
#: plus the memory pair plus the scan pair, and a future addition joins
#: this chain rather than editing the sets above).
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
#: The pair is read from two DIFFERENT kernel counters and the high-water
#: refreshes LAZILY, so a raw ``rss_peak_mb()`` really can sit under a raw
#: ``rss_mb()`` (measured: ``rss_mb=164`` against ``rss_peak_mb=162``) —
@@ -2744,13 +2790,17 @@ def test_memory_status_keys_are_the_only_addition_and_null_before_a_run(
sync_client: TestClient,
) -> None:
"""The additive pin on the wire: the payload's key set is EXACTLY the
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb``, and a run that
never happened answers ``null`` / ``null`` — not 0, not a plausible
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb`` plus the
phase-194 ``scan_done`` / ``scan_total`` pair, and a run that never
happened answers ``null`` / ``null`` — not 0, not a plausible
number. That null is also what makes the Sources page render the
counts line with no peak clause after an app restart."""
_login(sync_client)
body = sync_client.get("/api/sync/status").json()
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, set(body)
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, set(body)
# Phase 194 (task 02): the scan pair joins ADDITIVELY — idle is 0/0
# (never null, never a plausible position).
assert body["scan_done"] == 0 and body["scan_total"] == 0
assert body["rss_mb"] is None
assert body["rss_peak_mb"] is None
# The pre-existing shapes, unchanged: the state word, the two ISO
@@ -2811,8 +2861,9 @@ def test_success_terminal_status_keeps_the_final_memory_pair(
assert done[0].endswith(
f"rss_mb={fmt_mb(current)} rss_peak_mb={fmt_mb(peak)}"
), done[0]
# The pre-existing payload is untouched by all of this.
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS
# The pre-existing payload is untouched by all of this (phase 194's
# scan pair is the only addition, and it is additive).
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS
assert again["files_done"] == 0 or isinstance(again["files_done"], int)