phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -1419,6 +1419,10 @@ class _GatedImport:
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 194 (task 02): the pre-scan position hook the runner
|
||||
# passes (recorded for the wiring pin — this fake fires nothing,
|
||||
# so the status scan fields stay 0/0).
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -1431,11 +1435,13 @@ class _GatedImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
self.prune_flags.append(prune)
|
||||
self.scan_hook = on_scan_progress
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
@@ -1597,6 +1603,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
|
||||
# (the dataclass defaults; a run that scanned nothing keeps them
|
||||
# 0/0 rather than null).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"current_chunk": 0,
|
||||
@@ -2131,6 +2142,7 @@ class _TwoFileImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -2804,6 +2816,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -3170,7 +3183,8 @@ def test_sources_js_section_header_documents_the_cancel_contract() -> None:
|
||||
|
||||
#: The status payload's key set BEFORE phase 187 — the additive pin's
|
||||
#: left-hand side: the payload is exactly this set plus
|
||||
#: :data:`_MEMORY_KEYS`, nothing else, forever.
|
||||
#: :data:`_MEMORY_KEYS` plus :data:`_SCAN_KEYS`, nothing else, forever
|
||||
#: (a future field joins the chain, it never edits these three).
|
||||
_PRE_187_STATUS_KEYS = frozenset(
|
||||
{
|
||||
"state",
|
||||
@@ -3195,17 +3209,23 @@ _PRE_187_STATUS_KEYS = frozenset(
|
||||
#: Phase 187 (task 03): the two additive memory keys.
|
||||
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
|
||||
#: link of the same additive chain.
|
||||
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
|
||||
def test_status_key_set_is_the_old_set_plus_the_memory_pair(
|
||||
fresh_sync_status: None,
|
||||
) -> None:
|
||||
"""The key-set pin (D4: "Additive only — every pre-existing key
|
||||
keeps its name, type, and meaning"): the payload is EXACTLY the
|
||||
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb}`` — a fifth sub-position
|
||||
or a renamed field fails here even when the value assertions pass."""
|
||||
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb} ∪ {scan_done,
|
||||
scan_total}`` — a fifth sub-position or a renamed field fails here
|
||||
even when the value assertions pass."""
|
||||
keys = set(sync_api.sync_status())
|
||||
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, keys
|
||||
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, keys
|
||||
assert _MEMORY_KEYS.isdisjoint(_PRE_187_STATUS_KEYS)
|
||||
assert _SCAN_KEYS.isdisjoint(_PRE_187_STATUS_KEYS | _MEMORY_KEYS)
|
||||
|
||||
|
||||
class _FakeClock:
|
||||
@@ -3377,9 +3397,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The sampling POINTS, walked through the real runner: run start →
|
||||
``import`` → ``overview`` → ``summaries`` → the success terminal, one
|
||||
forced read each, and the per-file progress hook's throttled call
|
||||
adds NOTHING inside the window the run start just armed (LOCKED A4).
|
||||
``scan`` → ``import`` (the phase-194 flip) → ``overview`` →
|
||||
``summaries`` → the success terminal, one forced read each, and the
|
||||
per-file progress hook's throttled call adds NOTHING inside the window
|
||||
the previous transition just armed (LOCKED A4 — and the same rule the
|
||||
pre-scan's own per-file hook follows, pinned through the app in
|
||||
``tests/integration/test_sync_scan_phase.py``).
|
||||
|
||||
The last pair is the number the incident was about: the terminal
|
||||
KEEPS it (the phase-64 keep-final-values convention) and the
|
||||
@@ -3388,10 +3411,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
log an operator could be argued with."""
|
||||
clock = _FakeClock(10_000.0)
|
||||
monkeypatch.setattr(sync_api, "time", clock)
|
||||
# run start, import, overview, summaries, done (the hook is throttled).
|
||||
# run start, scan, import (the flip), overview, summaries, done — the
|
||||
# two phase-194 transitions join the list; the per-file hooks stay
|
||||
# throttled and add no read.
|
||||
memory = _StubMemory(
|
||||
monkeypatch,
|
||||
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 4_321)],
|
||||
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 24), (15, 4_321)],
|
||||
)
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _GatedImport(
|
||||
@@ -3412,14 +3437,23 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (10, 20)
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the transition forced the second read, and
|
||||
# the hook's own call (this fake fires one file) is inside
|
||||
# the window — still 2 reads, not 3 (A4 in numbers).
|
||||
# Import phase: the "scan" set forced the second read and the
|
||||
# first progress call's FLIP forced the third, while the
|
||||
# hook's own unforced call (this fake fires one file) is
|
||||
# inside the window — 3 reads, not 4 (A4 in numbers).
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (11, 21)
|
||||
assert memory.reads == 2, "the per-file hook never bypasses the guard"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (12, 22)
|
||||
assert memory.reads == 3, (
|
||||
"the scan/import transitions must each force one read and "
|
||||
f"the per-file hook must never bypass the guard: {memory.reads}"
|
||||
)
|
||||
# The scan closure reached the runner wired (the fake fires
|
||||
# nothing, so the position stays 0/0 — the sequence itself is
|
||||
# pinned through the app in test_sync_scan_phase.py).
|
||||
assert fake_import.scan_hook is not None
|
||||
assert s["scan_done"] == 0 and s["scan_total"] == 0
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -3431,7 +3465,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
# and it survives the run (the page shows it after the settle).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (14, 4_321)
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (15, 4_321)
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
done = [
|
||||
@@ -3440,7 +3474,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
if r.getMessage().startswith("sync: done detail=")
|
||||
]
|
||||
assert len(done) == 1, done
|
||||
assert done[0].endswith("rss_mb=14 rss_peak_mb=4321"), done[0]
|
||||
assert done[0].endswith("rss_mb=15 rss_peak_mb=4321"), done[0]
|
||||
|
||||
|
||||
def test_per_file_hook_samples_once_the_window_closes(
|
||||
@@ -3452,10 +3486,15 @@ def test_per_file_hook_samples_once_the_window_closes(
|
||||
fires once per file, and once the 5 s window has closed the next
|
||||
file's call re-reads — so a long walk reports the memory climbing
|
||||
(PLAN §7.4's never-stale contract) at ~600 reads per 27k files
|
||||
instead of 27k."""
|
||||
instead of 27k (phase 194 extended the same rule to the pre-scan's
|
||||
per-file hook)."""
|
||||
clock = _FakeClock(10_000.0)
|
||||
monkeypatch.setattr(sync_api, "time", clock)
|
||||
memory = _StubMemory(monkeypatch, [(100, 200), (110, 210), (140, 260)])
|
||||
# run start, the "scan" set, the scan->import flip, then the read the
|
||||
# test forces after moving the clock past the window.
|
||||
memory = _StubMemory(
|
||||
monkeypatch, [(100, 200), (110, 210), (120, 220), (140, 260)]
|
||||
)
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=1, added=1),
|
||||
@@ -3474,8 +3513,11 @@ def test_per_file_hook_samples_once_the_window_closes(
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (110, 210) # the transition
|
||||
assert memory.reads == 2, "the hook's call inside the window read nothing"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (120, 220) # the flip
|
||||
assert memory.reads == 3, (
|
||||
"start + scan + flip force three reads and the hook's own call "
|
||||
f"inside the window adds none: {memory.reads}"
|
||||
)
|
||||
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S + 1.0
|
||||
sync_api._sample_rss() # the NEXT file's hook call
|
||||
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (140, 260)
|
||||
|
||||
Reference in New Issue
Block a user