phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -93,6 +93,18 @@ on all three terminals (success, cancelled, failed) plus the importer's
|
||||
``import: summary …``. The fields are APPENDED (LOCKED A2), so every
|
||||
pre-phase prefix below is byte-identical and the pins here assert BOTH
|
||||
halves of that contract.
|
||||
|
||||
Phase 194 (task 02): the status gains the pre-scan. The payload is
|
||||
ADDITIVE — ``scan_done`` / ``scan_total`` after ``files_total``, the
|
||||
additive-chain pins below updated to the new key set (the idle shape and
|
||||
both ``_PRE_187_STATUS_KEYS`` assertions) — and ``files_done`` /
|
||||
``files_total`` now carry the PREDICTED-change-set position on a progress
|
||||
caller. One existing pin retargets with it:
|
||||
``test_double_trigger_while_running_returns_409`` waits on ``"scan"``
|
||||
instead of ``"import"``, because its canned import fires NO per-file hook
|
||||
and the scan→import flip lives in that hook — a run that reports no file
|
||||
stays in the pre-import phase (the full sequence with a live hook is
|
||||
pinned in ``tests/integration/test_sync_scan_phase.py``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -375,6 +387,11 @@ class FakeImportSources:
|
||||
# image fields stay 0/0, like the chunk fields of a no-batch
|
||||
# import).
|
||||
self.image_hooks: list[object] = []
|
||||
# Phase 194 (task 02): the PRE-SCAN position hook the runner
|
||||
# passes (a live closure while wired, None if the wiring
|
||||
# regresses) — the canned fake never fires it, so the status
|
||||
# scan fields stay 0/0 like a run that scanned nothing.
|
||||
self.scan_hooks: list[object] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -387,6 +404,7 @@ class FakeImportSources:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -397,6 +415,7 @@ class FakeImportSources:
|
||||
self.progress_hooks.append(progress)
|
||||
self.chunk_hooks.append(on_chunk_progress)
|
||||
self.image_hooks.append(on_image_progress)
|
||||
self.scan_hooks.append(on_scan_progress)
|
||||
self.ignore_maps.append(ignore_by_root or {})
|
||||
self.include_hidden_maps.append(include_hidden_by_root or {})
|
||||
self.doc_dates_maps.append(doc_dates_by_root or {})
|
||||
@@ -572,6 +591,10 @@ def test_admin_sync_success_reports_full_detail(
|
||||
# (the dataclass defaults).
|
||||
"images_done": 0,
|
||||
"images_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
|
||||
# (the dataclass defaults).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
@@ -703,17 +726,27 @@ def test_double_trigger_while_running_returns_409(
|
||||
assert r.status_code == 409
|
||||
assert r.json() == {"detail": "a sync is already running"}
|
||||
|
||||
# Phase 179 (task 02): WAIT for the import phase — the run's
|
||||
# progress is concurrent now that the git ops left the loop, so the
|
||||
# next request no longer inherits a run that already ran past them.
|
||||
body = _wait_phase(sync_client, "import")
|
||||
# Phase 179 (task 02): WAIT for the import to have started — the
|
||||
# run's progress is concurrent now that the git ops left the loop,
|
||||
# so the next request no longer inherits a run that already ran past
|
||||
# them. Phase 194 (task 02): the phase the wait targets is now
|
||||
# ``"scan"`` — the canned fake import reports NO file, and the
|
||||
# scan→import flip lives in the runner's per-file progress hook, so
|
||||
# a hook-less run legitimately stays in the pre-import phase (the
|
||||
# sequence with a real hook is pinned in test_sync_scan_phase.py and
|
||||
# in the phase-98 gated test below).
|
||||
body = _wait_phase(sync_client, "scan")
|
||||
assert body["state"] == "running"
|
||||
assert body["started_at"] is not None
|
||||
assert body["finished_at"] is None
|
||||
assert body["error"] is None
|
||||
# Wiring pin (phase 194, task 02): the runner passed its live
|
||||
# scan-position closure, and a fake that fires nothing keeps the
|
||||
# position at 0/0.
|
||||
assert len(fake_import.scan_hooks) == 1 and fake_import.scan_hooks[0] is not None
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
# Phase 98 (task 01): the in-flight run (parked in the fake
|
||||
# import's delay) reports the import phase — the summary span has
|
||||
# not started, so its counters are 0/0.
|
||||
# import's delay) reports no summary span — its counters are 0/0.
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
@@ -769,6 +802,10 @@ class _GatedImport:
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 194 (task 02): the PRE-SCAN position hook the runner
|
||||
# passes (recorded for the wiring pin — the gated fake never
|
||||
# fires it, so the status scan fields stay 0/0).
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -781,10 +818,12 @@ class _GatedImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
self.scan_hook = on_scan_progress
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
@@ -1937,6 +1976,7 @@ def test_import_error_is_reported_with_credentials_masked(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -2698,6 +2738,12 @@ _PRE_187_STATUS_KEYS = frozenset(
|
||||
#: Phase 187 (task 03): the two additive memory keys.
|
||||
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the
|
||||
#: next link in the same chain (the payload is exactly the pre-187 set
|
||||
#: plus the memory pair plus the scan pair, and a future addition joins
|
||||
#: this chain rather than editing the sets above).
|
||||
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
#: The pair is read from two DIFFERENT kernel counters and the high-water
|
||||
#: refreshes LAZILY, so a raw ``rss_peak_mb()`` really can sit under a raw
|
||||
#: ``rss_mb()`` (measured: ``rss_mb=164`` against ``rss_peak_mb=162``) —
|
||||
@@ -2744,13 +2790,17 @@ def test_memory_status_keys_are_the_only_addition_and_null_before_a_run(
|
||||
sync_client: TestClient,
|
||||
) -> None:
|
||||
"""The additive pin on the wire: the payload's key set is EXACTLY the
|
||||
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb``, and a run that
|
||||
never happened answers ``null`` / ``null`` — not 0, not a plausible
|
||||
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb`` plus the
|
||||
phase-194 ``scan_done`` / ``scan_total`` pair, and a run that never
|
||||
happened answers ``null`` / ``null`` — not 0, not a plausible
|
||||
number. That null is also what makes the Sources page render the
|
||||
counts line with no peak clause after an app restart."""
|
||||
_login(sync_client)
|
||||
body = sync_client.get("/api/sync/status").json()
|
||||
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, set(body)
|
||||
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, set(body)
|
||||
# Phase 194 (task 02): the scan pair joins ADDITIVELY — idle is 0/0
|
||||
# (never null, never a plausible position).
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
assert body["rss_mb"] is None
|
||||
assert body["rss_peak_mb"] is None
|
||||
# The pre-existing shapes, unchanged: the state word, the two ISO
|
||||
@@ -2811,8 +2861,9 @@ def test_success_terminal_status_keeps_the_final_memory_pair(
|
||||
assert done[0].endswith(
|
||||
f"rss_mb={fmt_mb(current)} rss_peak_mb={fmt_mb(peak)}"
|
||||
), done[0]
|
||||
# The pre-existing payload is untouched by all of this.
|
||||
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS
|
||||
# The pre-existing payload is untouched by all of this (phase 194's
|
||||
# scan pair is the only addition, and it is additive).
|
||||
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS
|
||||
assert again["files_done"] == 0 or isinstance(again["files_done"], int)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user