phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
2026-10-20 06:36:21 -04:00
parent ae080ac7f3
commit a45f744604
55 changed files with 7610 additions and 179 deletions
+62 -20
View File
@@ -1419,6 +1419,10 @@ class _GatedImport:
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
# Phase 194 (task 02): the pre-scan position hook the runner
# passes (recorded for the wiring pin — this fake fires nothing,
# so the status scan fields stay 0/0).
self.scan_hook: Callable[[int, int], None] | None = None
async def __call__(
self,
@@ -1431,11 +1435,13 @@ class _GatedImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
self.prune_flags.append(prune)
self.scan_hook = on_scan_progress
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
@@ -1597,6 +1603,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
"current_file": None,
"files_done": 0,
"files_total": 0,
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
# (the dataclass defaults; a run that scanned nothing keeps them
# 0/0 rather than null).
"scan_done": 0,
"scan_total": 0,
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
# (the dataclass defaults, no special-casing).
"current_chunk": 0,
@@ -2131,6 +2142,7 @@ class _TwoFileImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -2804,6 +2816,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -3170,7 +3183,8 @@ def test_sources_js_section_header_documents_the_cancel_contract() -> None:
#: The status payload's key set BEFORE phase 187 — the additive pin's
#: left-hand side: the payload is exactly this set plus
#: :data:`_MEMORY_KEYS`, nothing else, forever.
#: :data:`_MEMORY_KEYS` plus :data:`_SCAN_KEYS`, nothing else, forever
#: (a future field joins the chain, it never edits these three).
_PRE_187_STATUS_KEYS = frozenset(
{
"state",
@@ -3195,17 +3209,23 @@ _PRE_187_STATUS_KEYS = frozenset(
#: Phase 187 (task 03): the two additive memory keys.
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
#: link of the same additive chain.
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
def test_status_key_set_is_the_old_set_plus_the_memory_pair(
fresh_sync_status: None,
) -> None:
"""The key-set pin (D4: "Additive only — every pre-existing key
keeps its name, type, and meaning"): the payload is EXACTLY the
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb}`` — a fifth sub-position
or a renamed field fails here even when the value assertions pass."""
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb} ∪ {scan_done,
scan_total}`` — a fifth sub-position or a renamed field fails here
even when the value assertions pass."""
keys = set(sync_api.sync_status())
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, keys
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, keys
assert _MEMORY_KEYS.isdisjoint(_PRE_187_STATUS_KEYS)
assert _SCAN_KEYS.isdisjoint(_PRE_187_STATUS_KEYS | _MEMORY_KEYS)
class _FakeClock:
@@ -3377,9 +3397,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
caplog: pytest.LogCaptureFixture,
) -> None:
"""The sampling POINTS, walked through the real runner: run start →
``import`` → ``overview`` → ``summaries`` → the success terminal, one
forced read each, and the per-file progress hook's throttled call
adds NOTHING inside the window the run start just armed (LOCKED A4).
``scan`` → ``import`` (the phase-194 flip) → ``overview`` →
``summaries`` → the success terminal, one forced read each, and the
per-file progress hook's throttled call adds NOTHING inside the window
the previous transition just armed (LOCKED A4 — and the same rule the
pre-scan's own per-file hook follows, pinned through the app in
``tests/integration/test_sync_scan_phase.py``).
The last pair is the number the incident was about: the terminal
KEEPS it (the phase-64 keep-final-values convention) and the
@@ -3388,10 +3411,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
log an operator could be argued with."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
# run start, import, overview, summaries, done (the hook is throttled).
# run start, scan, import (the flip), overview, summaries, done — the
# two phase-194 transitions join the list; the per-file hooks stay
# throttled and add no read.
memory = _StubMemory(
monkeypatch,
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 4_321)],
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 24), (15, 4_321)],
)
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
@@ -3412,14 +3437,23 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
assert (s["rss_mb"], s["rss_peak_mb"]) == (10, 20)
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the transition forced the second read, and
# the hook's own call (this fake fires one file) is inside
# the window — still 2 reads, not 3 (A4 in numbers).
# Import phase: the "scan" set forced the second read and the
# first progress call's FLIP forced the third, while the
# hook's own unforced call (this fake fires one file) is
# inside the window — 3 reads, not 4 (A4 in numbers).
s = sync_api.sync_status()
assert s["phase"] == "import"
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
assert (s["rss_mb"], s["rss_peak_mb"]) == (11, 21)
assert memory.reads == 2, "the per-file hook never bypasses the guard"
assert (s["rss_mb"], s["rss_peak_mb"]) == (12, 22)
assert memory.reads == 3, (
"the scan/import transitions must each force one read and "
f"the per-file hook must never bypass the guard: {memory.reads}"
)
# The scan closure reached the runner wired (the fake fires
# nothing, so the position stays 0/0 — the sequence itself is
# pinned through the app in test_sync_scan_phase.py).
assert fake_import.scan_hook is not None
assert s["scan_done"] == 0 and s["scan_total"] == 0
fake_import.release.set()
finally:
clone.release.set()
@@ -3431,7 +3465,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
# and it survives the run (the page shows it after the settle).
s = sync_api.sync_status()
assert s["state"] == "success"
assert (s["rss_mb"], s["rss_peak_mb"]) == (14, 4_321)
assert (s["rss_mb"], s["rss_peak_mb"]) == (15, 4_321)
assert s["files_done"] == 1 and s["files_total"] == 3
done = [
@@ -3440,7 +3474,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
if r.getMessage().startswith("sync: done detail=")
]
assert len(done) == 1, done
assert done[0].endswith("rss_mb=14 rss_peak_mb=4321"), done[0]
assert done[0].endswith("rss_mb=15 rss_peak_mb=4321"), done[0]
def test_per_file_hook_samples_once_the_window_closes(
@@ -3452,10 +3486,15 @@ def test_per_file_hook_samples_once_the_window_closes(
fires once per file, and once the 5 s window has closed the next
file's call re-reads — so a long walk reports the memory climbing
(PLAN §7.4's never-stale contract) at ~600 reads per 27k files
instead of 27k."""
instead of 27k (phase 194 extended the same rule to the pre-scan's
per-file hook)."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
memory = _StubMemory(monkeypatch, [(100, 200), (110, 210), (140, 260)])
# run start, the "scan" set, the scan->import flip, then the read the
# test forces after moving the clock past the window.
memory = _StubMemory(
monkeypatch, [(100, 200), (110, 210), (120, 220), (140, 260)]
)
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
ImportSummary(files=1, added=1),
@@ -3474,8 +3513,11 @@ def test_per_file_hook_samples_once_the_window_closes(
assert fake_import.started.wait(5.0), "the run never reached the import"
s = sync_api.sync_status()
assert s["phase"] == "import"
assert (s["rss_mb"], s["rss_peak_mb"]) == (110, 210) # the transition
assert memory.reads == 2, "the hook's call inside the window read nothing"
assert (s["rss_mb"], s["rss_peak_mb"]) == (120, 220) # the flip
assert memory.reads == 3, (
"start + scan + flip force three reads and the hook's own call "
f"inside the window adds none: {memory.reads}"
)
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S + 1.0
sync_api._sample_rss() # the NEXT file's hook call
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (140, 260)