phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -1027,3 +1027,33 @@ def test_sync_prune_batch_rejects_negative(
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_prune_batch"):
|
||||
_settings()
|
||||
|
||||
|
||||
# --- Sync pending count (phase 194) ---
|
||||
|
||||
|
||||
def test_sync_pending_count_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 194 (D2): the pre-scan ships ON (``1``) — the owner's "1216
|
||||
remaining" report means the walk-position count was wrong by default, so
|
||||
the fix is the default and the rollback is the opt-in. ``0`` is accepted
|
||||
and means the exact phase-193 behaviour: count every importable file of
|
||||
the walk and never hash a file before the loop reached it."""
|
||||
monkeypatch.delenv("BOR_SYNC_PENDING_COUNT", raising=False)
|
||||
assert _settings().sync_pending_count == 1
|
||||
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "0")
|
||||
assert _settings().sync_pending_count == 0
|
||||
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "1")
|
||||
assert _settings().sync_pending_count == 1
|
||||
|
||||
|
||||
def test_sync_pending_count_rejects_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A negative switch is a typo — no branch of the importer can mean it,
|
||||
so startup refuses naming the field (the ``agent_max_rounds`` pattern,
|
||||
phase 194). The value is a boolean knob, so ONLY ``0`` is the rollback."""
|
||||
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_pending_count"):
|
||||
_settings()
|
||||
|
||||
@@ -1146,8 +1146,16 @@ def test_no_progress_means_no_prewalk(
|
||||
) -> None:
|
||||
"""``progress=None`` callers pay no extra pass: the real walker is hit
|
||||
exactly once per source root (one pass — as before phase 64), proven
|
||||
with a counting sentinel; with the hook it is hit twice (pre-walk for
|
||||
``total`` + the processing pass)."""
|
||||
with a counting sentinel; with the hook it is hit twice (the phase-194
|
||||
pre-scan for the pending set — the phase-64 stats pre-walk when
|
||||
``BOR_SYNC_PENDING_COUNT=0`` — plus the processing pass).
|
||||
|
||||
Phase 194 (task 01) moved the hook's pair, and this fixture is the
|
||||
second run over an ALREADY-INDEXED file: nothing is pending, so the
|
||||
pair is ``(0, 0)`` — the denominator is the predicted change set and an
|
||||
unchanged KB predicts nothing (the UI then renders no count part). The
|
||||
walk COUNT is what this test owns and it is untouched: one pass without
|
||||
the hook, two with it."""
|
||||
root = tmp_path / "nowalk"
|
||||
root.mkdir()
|
||||
(root / "a.md").write_text("# A\n\na\n")
|
||||
@@ -1186,8 +1194,12 @@ def test_no_progress_means_no_prewalk(
|
||||
)
|
||||
)
|
||||
assert summary2.files == 1
|
||||
assert walk_calls == 2 # pre-walk (total) + processing pass
|
||||
assert events == [(root.name, "a.md", 1, 1)]
|
||||
assert walk_calls == 2 # the count pass (pending set / pre-walk) + walk
|
||||
# phase 194: a.md is unchanged, so the predicted change set is empty
|
||||
# — ``total`` 0 and the position never advances. With the kill switch
|
||||
# at ``0`` this same run reports the phase-193 walk position (1 of 1);
|
||||
# pinned in tests/unit/test_sync_pending_count.py.
|
||||
assert events == [(root.name, "a.md", 0, 0)]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -1195,7 +1207,17 @@ def test_no_progress_means_no_prewalk(
|
||||
def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> None:
|
||||
"""The hook fires *before* ``_index_file``: a file whose embedding
|
||||
fails (and one that re-imports as unchanged) is still reported as the
|
||||
current file — the sequence covers every importable file."""
|
||||
current file — the sequence covers every importable file.
|
||||
|
||||
Phase 194 (task 01) retargeted the pair to the PENDING position (the
|
||||
owner's "1216 remaining" fix), which reads exactly right here. The
|
||||
first run is fresh: both files are predicted, ``1 of 2`` then ``2 of 2``
|
||||
— identical to the pre-phase numbers (A1's first-import parity). The
|
||||
second run is the incremental case: ``bad.md`` rolled back with its
|
||||
embed error so its row is still missing (predicted again, the ONLY
|
||||
work), ``good.md`` is unchanged so it holds the position instead of
|
||||
advancing it — ``1 of 1``, ``1 of 1``: every file is still reported as
|
||||
the current file, and the count is the work."""
|
||||
root = tmp_path / "progress-mixed"
|
||||
root.mkdir()
|
||||
(root / "bad.md").write_text("# Bad\n\npoison content the endpoint refuses\n")
|
||||
@@ -1214,7 +1236,8 @@ def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> N
|
||||
assert events == expected
|
||||
assert first.errors == 1 and first.added == 1
|
||||
# Re-run: good.md is now unchanged, bad.md is retried and fails
|
||||
# again — both still count in the sequence.
|
||||
# again — both still report as the current file, but only bad.md is
|
||||
# pending, so the denominator is 1 and good.md holds the position.
|
||||
events.clear()
|
||||
second = asyncio.run(
|
||||
import_sources(
|
||||
@@ -1222,7 +1245,7 @@ def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> N
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
)
|
||||
)
|
||||
assert events == expected
|
||||
assert events == [(root.name, "bad.md", 1, 1), (root.name, "good.md", 1, 1)]
|
||||
assert second.errors == 1 and second.unchanged == 1
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -253,6 +253,29 @@ def test_sync_session_bound_field_types() -> None:
|
||||
assert Settings.model_fields["sync_prune_batch"].default == 500
|
||||
|
||||
|
||||
def test_sync_pending_count_row_is_a_plain_live_overridable_row() -> None:
|
||||
"""Phase 194 (task 01, D2): the pre-scan switch sits in ``Import &
|
||||
sources`` beside the other two sync-run knobs with the ``Sync pending
|
||||
count`` label and is plain — not ``secret`` (an int), not
|
||||
``restart_required`` (the importer reads ``llm.settings`` once per run, so
|
||||
a saved override applies to the NEXT sync), overridable (the overlay is
|
||||
generic over ``int`` fields). ``1`` is the shipped default — the pending
|
||||
count is the FIX the owner asked for, so it ships on and the phase-193
|
||||
walk position is the ``0`` rollback; a negative value fails startup, not
|
||||
the panel."""
|
||||
meta = CATALOG["sync_pending_count"]
|
||||
assert meta.group == "Import & sources"
|
||||
assert meta.label == "Sync pending count"
|
||||
assert meta.secret is False
|
||||
assert meta.restart_required is False
|
||||
assert meta.overridable is True
|
||||
assert "sync_pending_count" not in SECRETS
|
||||
assert "sync_pending_count" not in RESTART_REQUIRED
|
||||
assert "sync_pending_count" not in NON_OVERRIDABLE
|
||||
assert Settings.model_fields["sync_pending_count"].annotation is int
|
||||
assert Settings.model_fields["sync_pending_count"].default == 1
|
||||
|
||||
|
||||
def test_source_name_hit_bonus_row_is_a_plain_live_overridable_row() -> None:
|
||||
"""Phase 191 (task 04, D3): the second selection-bonus row sits in
|
||||
``Retrieval & honesty`` (right beside ``name_hit_bonus`` — the two
|
||||
|
||||
@@ -1419,6 +1419,10 @@ class _GatedImport:
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 194 (task 02): the pre-scan position hook the runner
|
||||
# passes (recorded for the wiring pin — this fake fires nothing,
|
||||
# so the status scan fields stay 0/0).
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -1431,11 +1435,13 @@ class _GatedImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
self.prune_flags.append(prune)
|
||||
self.scan_hook = on_scan_progress
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
@@ -1597,6 +1603,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
|
||||
# (the dataclass defaults; a run that scanned nothing keeps them
|
||||
# 0/0 rather than null).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"current_chunk": 0,
|
||||
@@ -2131,6 +2142,7 @@ class _TwoFileImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -2804,6 +2816,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -3170,7 +3183,8 @@ def test_sources_js_section_header_documents_the_cancel_contract() -> None:
|
||||
|
||||
#: The status payload's key set BEFORE phase 187 — the additive pin's
|
||||
#: left-hand side: the payload is exactly this set plus
|
||||
#: :data:`_MEMORY_KEYS`, nothing else, forever.
|
||||
#: :data:`_MEMORY_KEYS` plus :data:`_SCAN_KEYS`, nothing else, forever
|
||||
#: (a future field joins the chain, it never edits these three).
|
||||
_PRE_187_STATUS_KEYS = frozenset(
|
||||
{
|
||||
"state",
|
||||
@@ -3195,17 +3209,23 @@ _PRE_187_STATUS_KEYS = frozenset(
|
||||
#: Phase 187 (task 03): the two additive memory keys.
|
||||
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
|
||||
#: link of the same additive chain.
|
||||
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
|
||||
def test_status_key_set_is_the_old_set_plus_the_memory_pair(
|
||||
fresh_sync_status: None,
|
||||
) -> None:
|
||||
"""The key-set pin (D4: "Additive only — every pre-existing key
|
||||
keeps its name, type, and meaning"): the payload is EXACTLY the
|
||||
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb}`` — a fifth sub-position
|
||||
or a renamed field fails here even when the value assertions pass."""
|
||||
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb} ∪ {scan_done,
|
||||
scan_total}`` — a fifth sub-position or a renamed field fails here
|
||||
even when the value assertions pass."""
|
||||
keys = set(sync_api.sync_status())
|
||||
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, keys
|
||||
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, keys
|
||||
assert _MEMORY_KEYS.isdisjoint(_PRE_187_STATUS_KEYS)
|
||||
assert _SCAN_KEYS.isdisjoint(_PRE_187_STATUS_KEYS | _MEMORY_KEYS)
|
||||
|
||||
|
||||
class _FakeClock:
|
||||
@@ -3377,9 +3397,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The sampling POINTS, walked through the real runner: run start →
|
||||
``import`` → ``overview`` → ``summaries`` → the success terminal, one
|
||||
forced read each, and the per-file progress hook's throttled call
|
||||
adds NOTHING inside the window the run start just armed (LOCKED A4).
|
||||
``scan`` → ``import`` (the phase-194 flip) → ``overview`` →
|
||||
``summaries`` → the success terminal, one forced read each, and the
|
||||
per-file progress hook's throttled call adds NOTHING inside the window
|
||||
the previous transition just armed (LOCKED A4 — and the same rule the
|
||||
pre-scan's own per-file hook follows, pinned through the app in
|
||||
``tests/integration/test_sync_scan_phase.py``).
|
||||
|
||||
The last pair is the number the incident was about: the terminal
|
||||
KEEPS it (the phase-64 keep-final-values convention) and the
|
||||
@@ -3388,10 +3411,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
log an operator could be argued with."""
|
||||
clock = _FakeClock(10_000.0)
|
||||
monkeypatch.setattr(sync_api, "time", clock)
|
||||
# run start, import, overview, summaries, done (the hook is throttled).
|
||||
# run start, scan, import (the flip), overview, summaries, done — the
|
||||
# two phase-194 transitions join the list; the per-file hooks stay
|
||||
# throttled and add no read.
|
||||
memory = _StubMemory(
|
||||
monkeypatch,
|
||||
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 4_321)],
|
||||
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 24), (15, 4_321)],
|
||||
)
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _GatedImport(
|
||||
@@ -3412,14 +3437,23 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (10, 20)
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the transition forced the second read, and
|
||||
# the hook's own call (this fake fires one file) is inside
|
||||
# the window — still 2 reads, not 3 (A4 in numbers).
|
||||
# Import phase: the "scan" set forced the second read and the
|
||||
# first progress call's FLIP forced the third, while the
|
||||
# hook's own unforced call (this fake fires one file) is
|
||||
# inside the window — 3 reads, not 4 (A4 in numbers).
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (11, 21)
|
||||
assert memory.reads == 2, "the per-file hook never bypasses the guard"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (12, 22)
|
||||
assert memory.reads == 3, (
|
||||
"the scan/import transitions must each force one read and "
|
||||
f"the per-file hook must never bypass the guard: {memory.reads}"
|
||||
)
|
||||
# The scan closure reached the runner wired (the fake fires
|
||||
# nothing, so the position stays 0/0 — the sequence itself is
|
||||
# pinned through the app in test_sync_scan_phase.py).
|
||||
assert fake_import.scan_hook is not None
|
||||
assert s["scan_done"] == 0 and s["scan_total"] == 0
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -3431,7 +3465,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
# and it survives the run (the page shows it after the settle).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (14, 4_321)
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (15, 4_321)
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
done = [
|
||||
@@ -3440,7 +3474,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
if r.getMessage().startswith("sync: done detail=")
|
||||
]
|
||||
assert len(done) == 1, done
|
||||
assert done[0].endswith("rss_mb=14 rss_peak_mb=4321"), done[0]
|
||||
assert done[0].endswith("rss_mb=15 rss_peak_mb=4321"), done[0]
|
||||
|
||||
|
||||
def test_per_file_hook_samples_once_the_window_closes(
|
||||
@@ -3452,10 +3486,15 @@ def test_per_file_hook_samples_once_the_window_closes(
|
||||
fires once per file, and once the 5 s window has closed the next
|
||||
file's call re-reads — so a long walk reports the memory climbing
|
||||
(PLAN §7.4's never-stale contract) at ~600 reads per 27k files
|
||||
instead of 27k."""
|
||||
instead of 27k (phase 194 extended the same rule to the pre-scan's
|
||||
per-file hook)."""
|
||||
clock = _FakeClock(10_000.0)
|
||||
monkeypatch.setattr(sync_api, "time", clock)
|
||||
memory = _StubMemory(monkeypatch, [(100, 200), (110, 210), (140, 260)])
|
||||
# run start, the "scan" set, the scan->import flip, then the read the
|
||||
# test forces after moving the clock past the window.
|
||||
memory = _StubMemory(
|
||||
monkeypatch, [(100, 200), (110, 210), (120, 220), (140, 260)]
|
||||
)
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=1, added=1),
|
||||
@@ -3474,8 +3513,11 @@ def test_per_file_hook_samples_once_the_window_closes(
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (110, 210) # the transition
|
||||
assert memory.reads == 2, "the hook's call inside the window read nothing"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (120, 220) # the flip
|
||||
assert memory.reads == 3, (
|
||||
"start + scan + flip force three reads and the hook's own call "
|
||||
f"inside the window adds none: {memory.reads}"
|
||||
)
|
||||
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S + 1.0
|
||||
sync_api._sample_rss() # the NEXT file's hook call
|
||||
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (140, 260)
|
||||
|
||||
@@ -0,0 +1,700 @@
|
||||
"""Phase 194, task 01 — the importer's pre-scan + the pending-count switch.
|
||||
|
||||
The owner report that opened the phase (2026-10-19): *"I just synced the prod
|
||||
brain of reese and it said '1216 remaining' when it only had like 10 files to
|
||||
sync."* The arithmetic of the phase-193 label was fine; its INPUTS were not —
|
||||
``total`` was every importable file (prod: ~10,000) and ``done`` was the
|
||||
position in that walk, so the label counted the WALK instead of the WORK.
|
||||
Task 01 moves the pair to the PREDICTED change set: ``import_sources``
|
||||
pre-scans (one streamed sha256 + one scalar ``documents.file_hash`` read per
|
||||
importable file, off the event loop) and reports ``done``/``total`` as the
|
||||
position in the pending set, with ``BOR_SYNC_PENDING_COUNT`` as the kill
|
||||
switch (``0`` = the exact phase-193 walk position, negative = refuses
|
||||
startup).
|
||||
|
||||
Pinned here (the behaviour under a real run — the pending-set parity, the
|
||||
zero-hydration pin, the off-loop span and the cancel — lives in
|
||||
``tests/integration/test_sync_pending_count.py``):
|
||||
|
||||
* the GATE: no ``progress`` caller means no scan at all (the callback never
|
||||
fires and the scalar statement is never built — the phase-64 skip rule,
|
||||
extended), and the switch at ``0`` on a progress caller runs the phase-64
|
||||
stats pre-walk instead (a source marker pin: that old code IS the ``0``
|
||||
branch, and a behaviour pin on the pair it reports);
|
||||
* the switch is read ONCE per run (the phase-184 read-once discipline), on
|
||||
the worker-thread side of ``asyncio.to_thread`` (the phase-179 rule);
|
||||
* the scalar statement's projection is EXACTLY ``documents.file_hash`` — one
|
||||
scalar column, no entity, no ``defer``/``load_only`` row (the phase-185
|
||||
column-list style); :func:`_doc_row_stmt` stays the importer's only ROW
|
||||
load;
|
||||
* the pending PREDICTATE: fresh (no row), changed (digest mismatch) and a
|
||||
pre-phase ``NULL`` row are pending, a hash match is not, a file the ignore
|
||||
rule hides is never even walked, and a file that cannot be read is
|
||||
predicted changed rather than fatal;
|
||||
* ``on_scan_progress``: once per file, the 1-based position against the
|
||||
combined importable total, and NOTHING for an empty KB (the phase-64
|
||||
``total === 0`` contract);
|
||||
* ``done`` as the pending position: a 3-file KB with 2 predicted reports
|
||||
``1``, ``1``, ``2`` against ``total = 2`` — the file that is not predicted
|
||||
holds the position while the walk moves past it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import asyncio
|
||||
import inspect
|
||||
import re
|
||||
import threading
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.importer import (
|
||||
_pending_hash_stmt,
|
||||
_prescan_pending,
|
||||
import_sources,
|
||||
)
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _code_only(fn: Any) -> str:
|
||||
"""A function's source WITHOUT its docstring.
|
||||
|
||||
The marker pins below count occurrences of names in the code; the
|
||||
docstrings argue about those same names by name (the house convention), so
|
||||
counting the raw source would pin prose. The docstring is located through
|
||||
the AST and sliced out by its own end line.
|
||||
"""
|
||||
src = inspect.getsource(fn)
|
||||
lines = src.splitlines()
|
||||
node = ast.parse(src).body[0]
|
||||
# `import_sources` is an ASYNC def, so it parses to AsyncFunctionDef —
|
||||
# narrowing to FunctionDef alone would silently skip the docstring strip
|
||||
# and pin prose.
|
||||
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
return src
|
||||
first = node.body[0]
|
||||
if (
|
||||
isinstance(first, ast.Expr)
|
||||
and isinstance(first.value, ast.Constant)
|
||||
and isinstance(first.value.value, str)
|
||||
):
|
||||
return "\n".join(lines[first.end_lineno or 1 :])
|
||||
return src
|
||||
|
||||
|
||||
def _settings(**overrides: Any) -> Settings:
|
||||
"""Fresh settings, never the developer's ``.env`` (the house pattern)."""
|
||||
return Settings(_env_file=None, **overrides) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _root(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
|
||||
"""One source root with *files* (relative path → content) under it."""
|
||||
root = tmp_path / name
|
||||
for rel, body in files.items():
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _prescan(
|
||||
*sources: Path,
|
||||
session: Session,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
on_scan_progress: Any = None,
|
||||
image_extensions: frozenset[str] = frozenset(),
|
||||
) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
""":func:`_prescan_pending` with the shipped import extensions and an
|
||||
explicit session, so the predicate is testable without a full run."""
|
||||
return _prescan_pending(
|
||||
list(sources),
|
||||
_settings().import_extension_set,
|
||||
image_extensions,
|
||||
ignore_by_root,
|
||||
include_hidden_by_root,
|
||||
on_scan_progress,
|
||||
session,
|
||||
)
|
||||
|
||||
|
||||
def _seed_row(
|
||||
db: Session, source: str, rel: str, *, file_hash: str | None
|
||||
) -> None:
|
||||
"""One ``documents`` row with a KNOWN ``file_hash`` (the pre-scan reads
|
||||
that one column, so this is the whole fixture: raw SQL, no importer, no
|
||||
chunks)."""
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO documents (id, source, path, full_path, title, content, "
|
||||
"content_hash, file_hash, indexed_at, created_at, created_at_manual, "
|
||||
"summary, is_image) VALUES (:id, :source, :path, :full_path, :title, "
|
||||
":content, :content_hash, :file_hash, now(), now(), false, :summary, "
|
||||
"false)"
|
||||
),
|
||||
{
|
||||
"id": uuid.uuid4(),
|
||||
"source": source,
|
||||
"path": rel,
|
||||
"full_path": f"/tmp/{source}/{rel}",
|
||||
"title": rel,
|
||||
"content": "stored markdown",
|
||||
"content_hash": "content-digest",
|
||||
"file_hash": file_hash,
|
||||
"summary": "stored summary",
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_probe_rows(db: Session) -> Any:
|
||||
"""The probe's own source-name prefix, removed around every test (the
|
||||
tables are global state — the ``test_importer.py`` hygiene pattern)."""
|
||||
_drop(db)
|
||||
yield
|
||||
_drop(db)
|
||||
|
||||
|
||||
def _drop(db: Session) -> None:
|
||||
db.execute(
|
||||
text("DELETE FROM documents WHERE source LIKE 'phase194-%'"),
|
||||
)
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The gate: who pays for the scan
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_progress_caller_runs_no_scan_at_all(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE skip rule (phase 64), extended: ``progress=None`` → no pre-scan.
|
||||
|
||||
The CLI (``scripts/import_docs.py``) and the upload job pass no
|
||||
``progress``, so they must not pay one extra read + hash + SELECT per
|
||||
importable file. A recording stand-in for the span proves the gate
|
||||
itself: the CLI-shaped call imports the file and never reaches the
|
||||
scan, and an ``on_scan_progress`` hook handed to a ``progress=None``
|
||||
caller fires never (the scan exists only to serve the running counts).
|
||||
"""
|
||||
calls: list[tuple[Any, ...]] = []
|
||||
scan_events: list[tuple[int, int]] = []
|
||||
|
||||
def _spy(*args: Any, **kwargs: Any) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
calls.append(args)
|
||||
return 0, {}
|
||||
|
||||
monkeypatch.setattr(importer, "_prescan_pending", _spy)
|
||||
root = _root(tmp_path, "phase194-gate-cli", {"a.md": "# A\n\na\n"})
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
|
||||
assert summary.files == 1 and summary.added == 1
|
||||
assert calls == [], "a progress=None caller ran the pre-scan"
|
||||
|
||||
# The scan hook alone does not open the gate either.
|
||||
summary2 = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
FakeEmbedder(),
|
||||
session=db,
|
||||
on_scan_progress=lambda done, total: scan_events.append((done, total)),
|
||||
)
|
||||
)
|
||||
assert summary2.files == 1
|
||||
assert calls == []
|
||||
assert scan_events == []
|
||||
finally:
|
||||
db.execute(
|
||||
text("DELETE FROM documents WHERE source = :s"), {"s": root.name}
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_switch_zero_runs_the_phase_193_stats_prewalk(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The rollback branch, from both sides: the marker (the old stats
|
||||
pre-walk code IS the ``0`` branch) and the reported pair.
|
||||
|
||||
With the switch at ``0`` a progress caller never reaches the scan span,
|
||||
and the hook reports the phase-193 walk position byte-for-byte — every
|
||||
importable file in ``total``, the 1-based index across all sources in
|
||||
``done`` — on a run where nothing is pending. That last clause is the
|
||||
point of the arm: the SAME unchanged fixture reports ``(0, 0)`` with the
|
||||
switch on and ``(1, 1)``, ``(2, 2)`` with it off.
|
||||
"""
|
||||
calls: list[tuple[Any, ...]] = []
|
||||
|
||||
def _spy(*args: Any, **kwargs: Any) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
calls.append(args)
|
||||
return 0, {}
|
||||
|
||||
monkeypatch.setattr(importer, "_prescan_pending", _spy)
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-gate-zero",
|
||||
{"a.md": "# A\n\nalpha\n", "b.md": "# B\n\nbravo\n"},
|
||||
)
|
||||
llm = FakeEmbedder()
|
||||
llm.settings = _settings(sync_pending_count=0)
|
||||
events: list[tuple[str, str, int, int]] = []
|
||||
try:
|
||||
# Seed the KB (a walk-position run of its own), so nothing pending.
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
assert calls == []
|
||||
|
||||
events.clear()
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
llm,
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
)
|
||||
)
|
||||
assert calls == [], "the switch was 0 but the pre-scan ran"
|
||||
assert (summary.files, summary.unchanged) == (2, 2)
|
||||
assert events == [
|
||||
(root.name, "a.md", 1, 2),
|
||||
(root.name, "b.md", 2, 2),
|
||||
], "the 0 branch must report the phase-193 walk position exactly"
|
||||
finally:
|
||||
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
|
||||
db.commit()
|
||||
|
||||
# The marker: the stats-only pre-walk still EXISTS in the importer, and
|
||||
# it lives in the ``0`` branch (a third walk would mean the replacement
|
||||
# was not a replacement).
|
||||
src = _code_only(importer.import_sources)
|
||||
assert "BOR_SYNC_PENDING_COUNT=0" in src
|
||||
assert src.count("iter_importable_files(") == 2, (
|
||||
"the importer walks the sources for the count more than once besides "
|
||||
"the pre-scan — the pre-scan REPLACES the stats pre-walk (phase 194 "
|
||||
"D1: no third walk)"
|
||||
)
|
||||
|
||||
|
||||
def test_switch_is_read_once_before_the_loop_and_off_the_loop() -> None:
|
||||
"""The phase-184 read-once discipline + the phase-179 off-loop rule.
|
||||
|
||||
``llm.settings.sync_pending_count`` is read exactly ONCE, before the
|
||||
source loop, so a settings overlay saved mid-run cannot flip the meaning
|
||||
of ``done``/``total`` between two files of the same walk; and the span
|
||||
is entered through ``asyncio.to_thread``, so a 10,000-file hash+SELECT
|
||||
pass never parks the single uvicorn loop.
|
||||
"""
|
||||
src = _code_only(importer.import_sources)
|
||||
|
||||
assert src.count("llm.settings.sync_pending_count") == 1, (
|
||||
"sync_pending_count is read more than once per run — a mid-run "
|
||||
"settings save would change the pair's meaning between two files "
|
||||
"(phase 184's read-once discipline)"
|
||||
)
|
||||
read_at = src.index("llm.settings.sync_pending_count")
|
||||
loop_at = src.index("for root in sources:")
|
||||
assert read_at < loop_at, "the switch must be read BEFORE the source loop"
|
||||
|
||||
assert "await asyncio.to_thread(\n _prescan_pending" in src, (
|
||||
"the pre-scan is no longer entered through asyncio.to_thread — a long "
|
||||
"scan would park the event loop (the phase-179 rule, "
|
||||
"test_sync_git_freeze.py)"
|
||||
)
|
||||
# The scan itself is a plain sync function (it runs in the worker).
|
||||
assert not inspect.iscoroutinefunction(importer._prescan_pending)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The scalar statement's shape (the phase-185 column-list style)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_scalar_statement_projects_exactly_file_hash() -> None:
|
||||
"""The pre-scan reads ONE column, so it loads no entity and keeps nothing.
|
||||
|
||||
The compiled projection names ``documents.file_hash`` and no other
|
||||
``documents`` column; the statement resolves to exactly one scalar
|
||||
column (``column_descriptions``), so ``session.scalar`` returns a string
|
||||
and no ORM object exists to land in an identity map. ``_doc_row_stmt``
|
||||
stays the importer's only ``(source, path)`` ROW load (phase 185).
|
||||
"""
|
||||
compiled = str(
|
||||
_pending_hash_stmt("the-source", "notes/a.md").compile(
|
||||
dialect=postgresql.dialect()
|
||||
)
|
||||
)
|
||||
projection = re.split(r"\bFROM\b", compiled, maxsplit=1)[0]
|
||||
columns = set(re.findall(r"documents\.([a-z_]+)", projection))
|
||||
|
||||
assert columns == {"file_hash"}, (
|
||||
f"the pre-scan statement projects {sorted(columns)} — it must project "
|
||||
"documents.file_hash and nothing else (one light metadata read per "
|
||||
"file, no entity hydration)"
|
||||
)
|
||||
# The predicate is the unique (source, path) constraint, like the fast path.
|
||||
predicate = re.split(r"\bFROM\b", compiled, maxsplit=1)[1]
|
||||
assert "documents.source" in predicate and "documents.path" in predicate
|
||||
|
||||
stmt = _pending_hash_stmt("s", "p")
|
||||
assert len(stmt.column_descriptions) == 1
|
||||
assert stmt.column_descriptions[0]["type"] == Document.file_hash.type
|
||||
assert "Document" not in {
|
||||
str(desc.get("entity")) for desc in stmt.column_descriptions
|
||||
}, "the pre-scan statement selects an ENTITY — it must select the column"
|
||||
|
||||
# The span is the only consumer, and the fast path's row load is intact.
|
||||
span_src = inspect.getsource(importer._prescan_pending)
|
||||
assert "_pending_hash_stmt(" in span_src
|
||||
assert "select(Document)" not in span_src
|
||||
assert "defer(" not in span_src and "load_only(" not in span_src, (
|
||||
"the pre-scan picked up a loader option — there is no ROW to defer: "
|
||||
"it selects one column (phase 185 keeps _doc_row_stmt as the only "
|
||||
"row load)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The pending predicate
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pending_predicate_fresh_unchanged_changed_and_null_row(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The four stored-hash states, and the position the set assigns.
|
||||
|
||||
* no row at all (a first sync) → pending;
|
||||
* a row whose ``file_hash`` equals the file's digest → NOT pending;
|
||||
* a row whose ``file_hash`` differs → pending;
|
||||
* a pre-phase-147 row (``file_hash`` NULL) → pending — the same call the
|
||||
fast path makes, so the prediction and the loop agree and the loop
|
||||
stamps the hash on the way through.
|
||||
|
||||
The positions are the 1-based walk order of the PREDICTED files only,
|
||||
and the scan total is every importable file (3 here, 3 predicted).
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-predicate",
|
||||
{
|
||||
"fresh.md": "# Fresh\n\nnever indexed\n",
|
||||
"same.md": "# Same\n\nindexed as-is\n",
|
||||
"moved.md": "# Moved\n\nindexed differently now\n",
|
||||
},
|
||||
)
|
||||
from app.rag.importer import _file_hash
|
||||
|
||||
_seed_row(db, root.name, "same.md", file_hash=_file_hash(root / "same.md"))
|
||||
_seed_row(db, root.name, "moved.md", file_hash="0" * 64)
|
||||
db.expunge_all()
|
||||
|
||||
scan_total, pending = _prescan(root, session=db)
|
||||
|
||||
assert scan_total == 3
|
||||
assert pending == {
|
||||
(root.name, "fresh.md"): 1,
|
||||
(root.name, "moved.md"): 2,
|
||||
}, (
|
||||
"the pending set is wrong — no row and a NULL row are pending, a "
|
||||
"digest match is not, a mismatch is (the fast path's own predicate)"
|
||||
)
|
||||
# The scan reads metadata only: nothing was hydrated into the session.
|
||||
assert len(db.identity_map) == 0
|
||||
|
||||
|
||||
def test_pending_predicate_ignores_what_the_walk_ignores(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""A file the ignore rule hides is never walked, so it is never pending.
|
||||
|
||||
The scan uses the EXACT per-root ``ignore`` / ``include_hidden`` rules of
|
||||
the processing loop: a hidden dot-directory is out by default and in when
|
||||
the root's flag admits it, and an ignored prefix is out even with the flag
|
||||
on. A predicted file the loop can never see would be a count that never
|
||||
reaches zero, so this is the parity edge, not a nicety.
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-ignore",
|
||||
{
|
||||
"keep.md": "# Keep\n\nbody\n",
|
||||
"skipme/hidden-by-rule.md": "# Skip\n\nbody\n",
|
||||
".dot/secret.md": "# Dot\n\nbody\n",
|
||||
},
|
||||
)
|
||||
ignore = {str(root): ["skipme"]}
|
||||
|
||||
scan_total, pending = _prescan(root, session=db, ignore_by_root=ignore)
|
||||
assert scan_total == 1
|
||||
assert list(pending) == [(root.name, "keep.md")]
|
||||
|
||||
# With the hidden-folders flag ON the dot-dir file joins the walk (and
|
||||
# the scan total with it) — still no pending entry for the ignored one.
|
||||
scan_total, pending = _prescan(
|
||||
root,
|
||||
session=db,
|
||||
ignore_by_root=ignore,
|
||||
include_hidden_by_root={str(root): True},
|
||||
)
|
||||
assert scan_total == 2
|
||||
assert sorted(pending) == [
|
||||
(root.name, ".dot/secret.md"),
|
||||
(root.name, "keep.md"),
|
||||
]
|
||||
|
||||
|
||||
def test_unreadable_file_is_predicted_changed_and_the_scan_survives(
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""A file the digest cannot reach is a PREDICTION of change, not a failure.
|
||||
|
||||
The window is the phase's own: the scan walks the tree, then hashes it
|
||||
seconds later, and a live source can lose a file in between. Failing the
|
||||
whole sync for that would be the worse bug — the pre-scan only informs a
|
||||
label — so the file lands in the pending set (the loop's own walk decides
|
||||
whether it still exists at all) and one warning names it.
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-unreadable",
|
||||
{"a.md": "# A\n\na\n", "b.md": "# B\n\nb\n"},
|
||||
)
|
||||
real_hash = importer._file_hash
|
||||
|
||||
def flaky(path: Path) -> str:
|
||||
if path.name == "b.md":
|
||||
raise OSError(2, "No such file or directory", str(path))
|
||||
return real_hash(path)
|
||||
|
||||
monkeypatch.setattr(importer, "_file_hash", flaky)
|
||||
with caplog.at_level("WARNING", logger="app.importer"):
|
||||
scan_total, pending = _prescan(root, session=db)
|
||||
|
||||
assert scan_total == 2
|
||||
assert sorted(pending) == [
|
||||
(root.name, "a.md"),
|
||||
(root.name, "b.md"),
|
||||
]
|
||||
lines = [r.getMessage() for r in caplog.records if r.name == "app.importer"]
|
||||
assert any(
|
||||
line.startswith("import: pre-scan unreadable")
|
||||
and f"source={root.name}" in line
|
||||
and "path=b.md" in line
|
||||
for line in lines
|
||||
), f"the skipped file was not named in a warning: {lines}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# on_scan_progress + the empty KB
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_scan_progress_reports_every_file_across_roots(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The scan pair is ``(1-based position, combined importable total)``.
|
||||
|
||||
The denominator is the number the phase-64 stats pre-walk used to
|
||||
compute (every importable file, across every root, in *sources* order),
|
||||
because the label is about the SCAN — how far the prediction has got —
|
||||
and the events cover every file, changed or not.
|
||||
"""
|
||||
root_a = _root(
|
||||
tmp_path, "phase194-scan-a", {"a1.md": "# A1\n\na\n", "a2.md": "# A2\n\nb\n"}
|
||||
)
|
||||
root_b = _root(tmp_path, "phase194-scan-b", {"sub/b1.md": "# B1\n\nc\n"})
|
||||
events: list[tuple[int, int]] = []
|
||||
|
||||
scan_total, pending = _prescan(
|
||||
root_a, root_b, session=db, on_scan_progress=lambda p, t: events.append((p, t))
|
||||
)
|
||||
|
||||
assert scan_total == 3
|
||||
assert events == [(1, 3), (2, 3), (3, 3)]
|
||||
assert len(pending) == 3
|
||||
|
||||
|
||||
def test_scan_progress_reports_nothing_for_an_empty_kb(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The degenerate KB: scan total 0, no callback, empty pending set.
|
||||
|
||||
The phase-64 ``total === 0`` contract, extended to the scan: with nothing
|
||||
importable there is no position to report, so the UI's scan label renders
|
||||
bare (``Scanning…``, no count part) instead of ``0/0``.
|
||||
"""
|
||||
empty = _root(tmp_path, "phase194-empty", {})
|
||||
missing = tmp_path / "phase194-not-a-dir"
|
||||
events: list[tuple[int, int]] = []
|
||||
|
||||
scan_total, pending = _prescan(
|
||||
empty, missing, session=db, on_scan_progress=lambda p, t: events.append((p, t))
|
||||
)
|
||||
|
||||
assert scan_total == 0
|
||||
assert pending == {}
|
||||
assert events == []
|
||||
|
||||
|
||||
def test_scan_hook_is_called_from_the_worker_thread(
|
||||
tmp_path: Path, db: Session
|
||||
) -> None:
|
||||
"""The documented threading contract of ``on_scan_progress``.
|
||||
|
||||
The span runs in an ``asyncio.to_thread`` worker, so the hook fires on
|
||||
THAT thread — which is only safe because the house hook rule is "assign a
|
||||
field", a single GIL-atomic store. The pin records the thread identity
|
||||
from inside the callback and from the awaiting coroutine.
|
||||
"""
|
||||
root = _root(tmp_path, "phase194-thread", {"a.md": "# A\n\na\n"})
|
||||
seen: dict[str, int] = {}
|
||||
|
||||
def hook(position: int, total: int) -> None: # noqa: ARG001
|
||||
seen["scan"] = threading.get_ident()
|
||||
|
||||
async def main() -> None:
|
||||
seen["loop"] = threading.get_ident()
|
||||
await asyncio.to_thread(
|
||||
importer._prescan_pending,
|
||||
[root],
|
||||
_settings().import_extension_set,
|
||||
frozenset(),
|
||||
None,
|
||||
None,
|
||||
hook,
|
||||
)
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
assert seen["scan"] != seen["loop"], (
|
||||
"the pre-scan hook ran on the event-loop thread — the span must live "
|
||||
"in a worker (phase 179)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# done = the position in the pending set
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_done_is_the_pending_position_through_import_sources(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""3 importable files, 2 of them pending: the pair is ``1/2``, ``1/2``,
|
||||
``2/2``.
|
||||
|
||||
``stable.md`` is indexed and unchanged, so the scan does not predict it
|
||||
and the walk moving past it does NOT advance the count — the label holds
|
||||
its position (this is the shape the owner's prod case takes: the count
|
||||
stops climbing across a run of unchanged files). The two changed files
|
||||
advance it in walk order, and the denominator is the pending set's size,
|
||||
never the 3-file walk.
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-position",
|
||||
{
|
||||
"a.md": "# A\n\nalpha\n",
|
||||
"stable.md": "# Stable\n\nunchanged\n",
|
||||
"z.md": "# Z\n\nzulu\n",
|
||||
},
|
||||
)
|
||||
llm = FakeEmbedder()
|
||||
events: list[tuple[str, str, int, int]] = []
|
||||
try:
|
||||
# Index everything, then change two files.
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
(root / "a.md").write_text("# A\n\nalpha edited\n", encoding="utf-8")
|
||||
(root / "z.md").write_text("# Z\n\nzulu edited\n", encoding="utf-8")
|
||||
|
||||
events.clear()
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
llm,
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
)
|
||||
)
|
||||
|
||||
assert (summary.files, summary.updated, summary.unchanged) == (3, 2, 1)
|
||||
assert events == [
|
||||
(root.name, "a.md", 1, 2),
|
||||
(root.name, "stable.md", 1, 2), # not predicted: the position holds
|
||||
(root.name, "z.md", 2, 2),
|
||||
]
|
||||
# The parity the UI depends on: the denominator IS the work.
|
||||
assert summary.updated + summary.added == events[-1][3]
|
||||
finally:
|
||||
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_empty_kb_reports_no_count_part_through_import_sources(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""An empty source with the hook on: no file, no pair, ``files_total`` 0.
|
||||
|
||||
The import is a no-op and the hook never fires, so the status keeps the
|
||||
zero denominator the UI's ``total === 0`` guard hides the count part for
|
||||
(the phase-64 contract, arriving through the pre-scan path rather than
|
||||
the stats pre-walk).
|
||||
"""
|
||||
root = _root(tmp_path, "phase194-empty-run", {})
|
||||
events: list[tuple[str, str, int, int]] = []
|
||||
scan_events: list[tuple[int, int]] = []
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
FakeEmbedder(),
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
on_scan_progress=lambda p, t: scan_events.append((p, t)),
|
||||
)
|
||||
)
|
||||
|
||||
assert (summary.files, summary.added, summary.errors) == (0, 0, 0)
|
||||
assert events == []
|
||||
assert scan_events == []
|
||||
|
||||
|
||||
def test_cli_caller_is_untouched_by_the_pending_semantics(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The CLI-shaped run (no hook at all) on an unchanged KB is the pre-phase
|
||||
run: one file processed, nothing counted, one walk.
|
||||
|
||||
The byte-identity of the non-UI caller is the phase's cheapest promise —
|
||||
``scripts/import_docs.py`` passes no ``progress``, so it neither pays the
|
||||
scan nor sees its numbers.
|
||||
"""
|
||||
root = _root(tmp_path, "phase194-cli", {"a.md": "# A\n\na\n"})
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (first.files, first.added) == (1, 1)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.files, second.added, second.unchanged) == (1, 0, 1)
|
||||
finally:
|
||||
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
|
||||
db.commit()
|
||||
@@ -0,0 +1,544 @@
|
||||
"""Phase 194, task 03 — the Sources page renders the scan as a scan.
|
||||
|
||||
The owner report the whole phase exists for (2026-10-19): *"1216
|
||||
remaining"* on a ~10,000-file KB that had ~10 files to sync. Task 01 moved
|
||||
the server's ``done``/``total`` to the pre-scan's predicted change set and
|
||||
task 02 exposed the scan as a phase (``phase: "scan"`` + ``scan_done`` /
|
||||
``scan_total``). This module pins the UI half (D4, owner-confirmed A2):
|
||||
``fmtSyncPhaseLabel`` gains ONE explicit ``"scan"`` branch that says
|
||||
``Scanning…`` and puts the scan's own ``(n/total)`` in the SAME count slot
|
||||
the import phase's ``N remaining`` uses — so the walk position surviving as
|
||||
an ``n/m`` is honest, because the word in front of it is "Scanning" and
|
||||
never "Syncing" (TODO L6's objection was the sync count wearing the
|
||||
total's shape).
|
||||
|
||||
Pinned here (source markers, plus one node run of the REAL label path):
|
||||
|
||||
* the branch — ``status.phase === "scan"`` is in the decision tree BEFORE
|
||||
``overview`` / ``summaries`` / the fall-through, its main text is exactly
|
||||
``Scanning…``, and it never reads ``current_file`` (null in the phase),
|
||||
the file pair, the chunk pair, the image pair or the summaries pair;
|
||||
* the shape — the count part is ``(${status.scan_done}/${status.scan_total})``
|
||||
landing on the ``count`` slot (``setSyncLabel``'s hidden-toggle + the A4
|
||||
untruncated button-title / ``#sync-progress`` placement carry it with no
|
||||
new element), with the phase-64 ``total === 0`` guard (the empty KB and
|
||||
the scan's own first frame render the bare ``Scanning…``) and no
|
||||
chunk/image part (the scan reads no file);
|
||||
* the pass-through — both sync status call sites (the poll tick, the
|
||||
load-time re-attach) forward the WHOLE status object, which is what
|
||||
carries the scan pair to the wrapper (the summaries / chunk / image
|
||||
pattern: ``phase`` selects which pair renders, no field is re-listed);
|
||||
* the UNTOUCHED set — ``fmtSyncLabel``'s ``N remaining`` expression
|
||||
verbatim (the phase-193 pin), the chunk / image / summaries part
|
||||
builders, ``setSyncLabel``'s composed full string, ``fmtSyncResult`` and
|
||||
``fmtSyncPeakClause`` (the phase-187 trailing clause lands on the scan
|
||||
label's line unchanged, and stays a terminal-only clause);
|
||||
* the docstrings — the decision-tree paragraph names the new branch,
|
||||
narrows the fallback to the null prelude + ``"import"``, and keeps the
|
||||
phase-193 ``N remaining`` clause.
|
||||
|
||||
The owner-visible arms (the count-down over a real incremental sync, the
|
||||
scan label in a real browser) live in
|
||||
``tests/e2e/test_sync_pending_count.py`` (task 05); the stale pinned suites
|
||||
retarget in task 04.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
#: The node behavior check needs node (the test-document-viewer pattern).
|
||||
HAVE_NODE = shutil.which("node") is not None
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SOURCES_JS = ROOT / "frontend" / "assets" / "sources.js"
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return SOURCES_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _fn(js: str, name: str) -> str:
|
||||
"""The source of the first ``function <name>`` (brace balanced — the
|
||||
house pin pattern from tests/unit/test_frontend_sync_upload.py: the
|
||||
functions live inside ``mount(root)``, so the closing brace is
|
||||
indented)."""
|
||||
fn = js.find(f"function {name}")
|
||||
assert fn != -1, f"{name} must be defined in sources.js"
|
||||
open_idx = js.find("{", fn)
|
||||
depth = 0
|
||||
for i in range(open_idx, len(js)):
|
||||
c = js[i]
|
||||
if c == "{":
|
||||
depth += 1
|
||||
elif c == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return js[fn : i + 1]
|
||||
raise AssertionError(f"unbalanced braces in {name}")
|
||||
|
||||
|
||||
def _doc_above(js: str, name: str) -> str:
|
||||
"""The block comment immediately above ``function <name>``."""
|
||||
i = js.find(f"function {name}")
|
||||
assert i != -1, f"{name} must be defined in sources.js"
|
||||
start = js.rfind("/*", 0, i)
|
||||
assert start != -1, f"{name} must carry a docstring"
|
||||
end = js.find("*/", start)
|
||||
assert start < end < i, f"{name}'s docstring must sit above the function"
|
||||
return js[start:end]
|
||||
|
||||
|
||||
def _scan_slice(js: str) -> str:
|
||||
"""The ``"scan"`` branch of ``fmtSyncPhaseLabel`` — from its ``if`` to
|
||||
the ``overview`` branch that follows it (every shape pin below reads
|
||||
ONLY this slice, so a shape that leaked into another branch is caught
|
||||
and a branch that moved is caught)."""
|
||||
body = _fn(js, "fmtSyncPhaseLabel")
|
||||
start = body.find('status.phase === "scan"')
|
||||
assert start != -1, 'the "scan" branch is missing from fmtSyncPhaseLabel'
|
||||
end = body.find('status.phase === "overview"', start)
|
||||
assert end != -1, "the overview branch must follow the scan branch"
|
||||
return body[start:end]
|
||||
|
||||
|
||||
# ---------- the branch ----------
|
||||
|
||||
|
||||
def test_scan_branch_is_the_first_arm_of_the_decision_tree() -> None:
|
||||
"""The wrapper gains ONE explicit ``"scan"`` branch (D4): the phase
|
||||
comparison sits BEFORE ``overview``, ``summaries`` and the
|
||||
``fmtSyncLabel`` fall-through (the machine order is ``null`` →
|
||||
``"scan"`` → ``"import"`` → ``"overview"`` → ``"summaries"``, and
|
||||
keeping the arms in that order is what leaves every pre-existing
|
||||
branch slice byte-identical), and the main text is exactly
|
||||
``Scanning…`` with the ellipsis the other running labels use."""
|
||||
js = _js()
|
||||
body = _fn(js, "fmtSyncPhaseLabel")
|
||||
i_scan = body.find('status.phase === "scan"')
|
||||
i_ov = body.find('status.phase === "overview"')
|
||||
i_sum = body.find('status.phase === "summaries"')
|
||||
i_fall = body.find("fmtSyncLabel(", i_sum)
|
||||
assert -1 < i_scan < i_ov < i_sum < i_fall, (
|
||||
"the scan branch must be the first arm, ahead of overview / "
|
||||
"summaries / the null-prelude + import fall-through"
|
||||
)
|
||||
assert 'main: "Scanning\u2026"' in _scan_slice(js), (
|
||||
"the main text is the bare Scanning label — no file"
|
||||
)
|
||||
|
||||
|
||||
def test_scan_branch_reads_the_scan_pair_and_nothing_else() -> None:
|
||||
"""The branch consumes the scan pair the way it already consumes the
|
||||
summaries pair (``phase`` selects which pair renders). It NEVER reads
|
||||
``current_file`` — the runner keeps it null through the phase, because
|
||||
the scan names no file — nor the file pair, the chunk pair, the image
|
||||
pair or the summaries pair (a scan embeds no chunk and describes no
|
||||
image, so those fields carry nothing meaningful here)."""
|
||||
scan = _scan_slice(_js())
|
||||
assert "status.scan_done" in scan and "status.scan_total" in scan, (
|
||||
"the count part is the status's own scan pair"
|
||||
)
|
||||
for field in (
|
||||
"current_file",
|
||||
"files_done",
|
||||
"files_total",
|
||||
"chunks_total",
|
||||
"current_chunk",
|
||||
"images_total",
|
||||
"images_done",
|
||||
"summaries_total",
|
||||
"summaries_done",
|
||||
"current_summary",
|
||||
):
|
||||
assert field not in scan, (
|
||||
f'the "scan" branch must not read {field} — that field is the '
|
||||
"import/overview/summaries phase's own position"
|
||||
)
|
||||
|
||||
|
||||
def test_scan_branch_emits_no_chunk_and_no_image_part() -> None:
|
||||
"""The scan names no file, so its parts are the main text + the count
|
||||
only: ``chunk: ""`` and ``image: ""`` (the writer hides both spans and
|
||||
keeps them out of the composed full string — the overview branch's
|
||||
shape, phase 98/152)."""
|
||||
scan = _scan_slice(_js())
|
||||
assert 'chunk: ""' in scan, "the scan label carries no chunk part"
|
||||
assert 'image: ""' in scan, "the scan label carries no image part"
|
||||
|
||||
|
||||
# ---------- the (n/total) count part ----------
|
||||
|
||||
|
||||
def test_scan_count_part_is_the_pair_in_the_count_slot() -> None:
|
||||
"""The count part is the exact ``(${status.scan_done}/${status.scan_total})``
|
||||
template — a plain ``(n/total)``, no thousands separator, no suffix —
|
||||
and it lands on the ``count`` key, i.e. the SAME slot the import
|
||||
phase's ``N remaining`` uses (``setSyncLabel``'s
|
||||
``.sync-label-count`` span + the composed full string), so the count
|
||||
stays pinned on a phone and the A4 untruncated label (button title +
|
||||
``#sync-progress``) carries ``Scanning… (n/total)`` with no new
|
||||
element and no new live region."""
|
||||
scan = _scan_slice(_js())
|
||||
assert "`(${status.scan_done}/${status.scan_total})`" in scan, (
|
||||
"the exact (n/total) shape over the scan pair"
|
||||
)
|
||||
i_count = scan.find("count:")
|
||||
i_template = scan.find("`(${status.scan_done}/${status.scan_total})`")
|
||||
assert -1 < i_count < i_template, "the pair renders INTO the count slot"
|
||||
assert "remaining" not in scan, (
|
||||
"the scan never says remaining — that part belongs to fmtSyncLabel"
|
||||
)
|
||||
for helper in ("toLocaleString", ".toFixed"):
|
||||
assert helper not in scan, "plain integers, the house label style"
|
||||
|
||||
|
||||
def test_scan_count_part_hides_while_the_total_is_zero() -> None:
|
||||
"""The phase-64 ``total === 0`` contract, extended to the scan: while
|
||||
the scan has not reported its denominator (and for the degenerate
|
||||
empty KB, where the importer fires no scan callback at all) the count
|
||||
part is the empty string — the bare ``Scanning…`` renders, never
|
||||
``(0/0)``. The guard precedes the template and the false arm is the
|
||||
empty string."""
|
||||
scan = _scan_slice(_js())
|
||||
i_guard = scan.find("status.scan_total > 0")
|
||||
i_template = scan.find("`(${status.scan_done}/${status.scan_total})`")
|
||||
i_else = scan.find(': ""', i_template)
|
||||
assert -1 < i_guard < i_template < i_else, (
|
||||
"scan_total > 0 ? (n/total) : \"\" — the phase-64 no-count-part guard"
|
||||
)
|
||||
|
||||
|
||||
# ---------- the caller pass-through ----------
|
||||
|
||||
|
||||
def test_poll_tick_forwards_the_whole_status_so_the_scan_pair_rides() -> None:
|
||||
"""The running-sync branch of the poll tick passes the WHOLE status
|
||||
object as ``enterSyncRunningState``'s fifth argument — that object IS
|
||||
the pass-through for the scan pair, exactly like the summaries /
|
||||
chunk / image pairs (the call site names it). A status with
|
||||
``phase: "scan"`` therefore reaches the wrapper with its
|
||||
``scan_done`` / ``scan_total`` intact."""
|
||||
js = _js()
|
||||
start = js.find("function startSyncPolling")
|
||||
assert start != -1
|
||||
tick = js[start : js.find("async function startSync", start)]
|
||||
i_run = tick.find('syncStatus.state === "running"')
|
||||
assert i_run != -1
|
||||
# the call-site comment sits directly above the branch — it is part of
|
||||
# the pass-through's documentation, so the slice starts there
|
||||
branch = tick[tick.rfind("// 1. sync running", 0, i_run) : tick.find(
|
||||
'uploadStatus && uploadStatus.state === "running"', i_run
|
||||
)]
|
||||
assert (
|
||||
'"sync", syncStatus.current_file, syncStatus.files_done, syncStatus.files_total'
|
||||
in branch
|
||||
), "the pre-phase positional args are unchanged"
|
||||
i_args = branch.find("syncStatus.files_total")
|
||||
i_pass = branch.find("syncStatus\n", i_args)
|
||||
assert i_pass != -1, "the WHOLE syncStatus object rides in as the 5th arg"
|
||||
assert "scan_done" in branch and "scan_total" in branch, (
|
||||
"the call-site comment names the scan pair riding the status (phase 194)"
|
||||
)
|
||||
|
||||
|
||||
def test_load_time_reattach_forwards_the_whole_status_too() -> None:
|
||||
"""The load-time re-attach is the other sync call site: a reload
|
||||
during the pre-scan re-enters running state with the WHOLE status, so
|
||||
the re-attached label reads ``Scanning… (n/total)`` (the never-stale
|
||||
contract — phase 98's mid-summaries reload, now also mid-scan)."""
|
||||
js = _js()
|
||||
body = _fn(js, "initSyncButton")
|
||||
assert (
|
||||
'"sync", status.current_file, status.files_done, status.files_total' in body
|
||||
), "the pre-phase positional args are unchanged"
|
||||
i_args = body.find("status.files_total")
|
||||
i_pass = body.find("status\n", i_args)
|
||||
assert i_pass != -1, "the WHOLE status object rides in as the 5th arg"
|
||||
doc = _doc_above(js, "initSyncButton")
|
||||
assert "phase 194" in doc and "Scanning" in doc, (
|
||||
"the re-attach docstring names the scan label it can re-enter (194)"
|
||||
)
|
||||
|
||||
|
||||
def test_running_entry_hands_the_status_to_the_phase_wrapper() -> None:
|
||||
"""``enterSyncRunningState`` keeps the phase-aware split untouched:
|
||||
sync + status → ``fmtSyncPhaseLabel(status)`` (the wrapper that now
|
||||
owns the scan branch), everything else → the bare ``fmtSyncLabel``
|
||||
parts, and the parts still go to the ONE writer with the running
|
||||
flag (so the scan label lands on the button title + the box)."""
|
||||
entry = _fn(_js(), "enterSyncRunningState")
|
||||
assert "fmtSyncPhaseLabel(status)" in entry
|
||||
assert 'kind === "sync" && status' in entry
|
||||
assert "setSyncLabel(main, count, chunk, true, image)" in entry
|
||||
|
||||
|
||||
# ---------- the UNTOUCHED set ----------
|
||||
|
||||
|
||||
def test_fmt_sync_label_n_remaining_arithmetic_is_byte_identical() -> None:
|
||||
"""The phase-193 count-down builder is UNTOUCHED by this task — the
|
||||
server-side semantic change (the pair becomes the pending position)
|
||||
flows through the SAME expression. The exact line, both guards and the
|
||||
plain-integer shape, plus the absence of anything scan-shaped in this
|
||||
builder."""
|
||||
bare = _fn(_js(), "fmtSyncLabel")
|
||||
assert (
|
||||
"const count = total > 0 && total - done > 0"
|
||||
" ? `${total - done} remaining` : \"\";"
|
||||
) in bare, "the phase-193 count expression, byte for byte"
|
||||
assert 'const prefix = kind === "upload" ? "Importing" : "Syncing\u2026";' in bare
|
||||
assert "return { main, count }" in bare
|
||||
for word in ("scan", "chunk", "image", "summaries", "phase"):
|
||||
assert word not in bare, f"fmtSyncLabel stays free of {word!r}"
|
||||
|
||||
|
||||
def test_other_phase_labels_and_the_writers_are_untouched() -> None:
|
||||
"""The regression pins for everything the scan branch could have
|
||||
disturbed: the overview copy, the summaries folder + ``(n/m)``
|
||||
position, the import fall-through's chunk + image parts, the single
|
||||
writer's signature + composed full string, the terminal counts line
|
||||
(no running count part, no scan word) and the phase-187 peak clause
|
||||
(still a terminal-only trailing clause, null renders nothing)."""
|
||||
js = _js()
|
||||
phase = _fn(js, "fmtSyncPhaseLabel")
|
||||
assert (
|
||||
'return { main: "Writing KB overview\u2026", count: "", chunk: "", image: "" };'
|
||||
in phase
|
||||
)
|
||||
assert "`Summarizing folders\u2026${folder}`" in phase
|
||||
assert (
|
||||
"count: `(${status.summaries_done}/${status.summaries_total})`" in phase
|
||||
), "the summaries folder position keeps its (n/m)"
|
||||
assert (
|
||||
'const { main, count } = fmtSyncLabel(\n'
|
||||
' "sync", status.current_file, status.files_done, status.files_total\n'
|
||||
" );" in phase
|
||||
), "the import fall-through still calls the bare builder unchanged"
|
||||
assert "`\u00b7 chunk ${status.current_chunk}/${status.chunks_total}`" in phase
|
||||
assert "`\u00b7 image ${status.images_done}/${status.images_total}`" in phase
|
||||
assert "return { main, count, chunk, image };" in phase
|
||||
|
||||
writer = _fn(js, "setSyncLabel")
|
||||
assert (
|
||||
'function setSyncLabel(mainText, countText = "", chunkText = "",'
|
||||
' running = false, imageText = "")' in writer
|
||||
)
|
||||
assert "syncLabelCount.textContent = countText" in writer
|
||||
assert "syncLabelCount.hidden = false" in writer
|
||||
assert (
|
||||
"const full =\n"
|
||||
" mainText +\n"
|
||||
" (countText ? ` ${countText}` : \"\") +\n"
|
||||
" (chunkText ? ` ${chunkText}` : \"\") +\n"
|
||||
" (imageText ? ` ${imageText}` : \"\");" in writer
|
||||
), "the A4 untruncated label is still main + count + chunk + image"
|
||||
assert "if (running) setSyncProgress(full);" in writer
|
||||
assert "syncBtn.title = full" in writer
|
||||
|
||||
result = _fn(js, "fmtSyncResult")
|
||||
for word in ("remaining", "Scanning", "scan"):
|
||||
assert word not in result, f"the terminal counts line stays {word!r}-free"
|
||||
|
||||
peak = _fn(js, "fmtSyncPeakClause")
|
||||
assert "return ` \u00b7 peak ${peak} MB`;" in peak
|
||||
assert "status.rss_peak_mb ?? status.rss_mb" in peak
|
||||
assert "Scanning" not in peak, "the clause is terminal-only, unchanged"
|
||||
|
||||
|
||||
# ---------- the docstrings ----------
|
||||
|
||||
|
||||
def test_phase_label_docstring_names_the_branch_and_narrows_the_fallback() -> None:
|
||||
"""The decision-tree paragraph above the wrapper names the new arm and
|
||||
narrows the fall-through to the two states that remain, keeps the
|
||||
phase-193 ``N remaining`` clause, states the ``total === 0`` guard,
|
||||
and says WHY an ``(n/total)`` is honest here (TODO L6: the objection
|
||||
was the sync count wearing the total's shape — this label says
|
||||
Scanning, never Syncing)."""
|
||||
doc = _doc_above(_js(), "fmtSyncPhaseLabel")
|
||||
assert '"scan"' in doc and "Scanning\u2026" in doc, "the new branch is named"
|
||||
assert "scan_done" in doc and "scan_total" in doc, "the pair is named"
|
||||
assert "scan_total === 0" in doc, "the no-count-part guard is documented"
|
||||
assert "ONLY the two" in doc and '"import"' in doc, "the fallback is narrowed"
|
||||
assert "`N remaining`" in doc and "N = total - done" in doc, (
|
||||
"the phase-193 count-down clause survives the rewrite"
|
||||
)
|
||||
assert "never" in doc and '"Syncing"' in doc, (
|
||||
"the honesty clause: Scanning, never Syncing (TODO L6)"
|
||||
)
|
||||
assert "A4" in doc, "the untruncated placement is still documented"
|
||||
|
||||
|
||||
def test_module_and_section_headers_document_the_scan_label() -> None:
|
||||
"""The module header carries the phase-194 paragraph (the count is the
|
||||
WORK, the scan says so), and the sync section's decision tree names
|
||||
the ``"scan"`` arm with its ``(n/total)`` — the header the wrapper's
|
||||
docstring points at."""
|
||||
js = _js()
|
||||
header = js[: js.find("import { fetchIsAdmin }")]
|
||||
assert "Phase 194 (task 03)" in header, "the module header names the phase"
|
||||
assert "Scanning\u2026 (n/total)" in header, "the module header names the label"
|
||||
|
||||
marker = js.find("Sync sources button (Sources page only)")
|
||||
assert marker != -1
|
||||
section = js[marker : js.find("const syncBtn", marker)]
|
||||
assert "Phase 194 task 03" in section, "the section header names the phase"
|
||||
assert "Scanning\u2026 (n/total)" in section, "the decision tree names the label"
|
||||
assert "scan_done" in section, "the pair the WHOLE status carries is named"
|
||||
|
||||
|
||||
# ---------- the REAL label path under node ----------
|
||||
|
||||
#: The stub element refs (the test-document-viewer pattern; the same
|
||||
#: shapes tests/unit/test_frontend_sync_upload.py drives). The label
|
||||
#: writer, the box writer and the cancel control are the REAL functions.
|
||||
_NODE_HEAD = """
|
||||
const syncLabelMain = { textContent: "" };
|
||||
const syncLabelCount = { textContent: "", hidden: true };
|
||||
const syncLabelChunk = { textContent: "", hidden: true };
|
||||
const syncProgress = { hidden: true };
|
||||
const syncProgressText = { textContent: "" };
|
||||
const syncBtn = {
|
||||
disabled: false,
|
||||
title: "",
|
||||
attrs: {},
|
||||
setAttribute(k, v) { this.attrs[k] = v; },
|
||||
removeAttribute(k) { delete this.attrs[k]; },
|
||||
classes: new Set(),
|
||||
classList: {
|
||||
add(c) { syncBtn.classes.add(c); },
|
||||
remove(c) { syncBtn.classes.delete(c); },
|
||||
has(c) { return syncBtn.classes.has(c); },
|
||||
},
|
||||
};
|
||||
const syncIcon = {
|
||||
classes: new Set(),
|
||||
classList: {
|
||||
add(c) { syncIcon.classes.add(c); },
|
||||
remove(c) { syncIcon.classes.delete(c); },
|
||||
has(c) { return syncIcon.classes.has(c); },
|
||||
},
|
||||
};
|
||||
const syncCancelBtn = {
|
||||
hidden: true,
|
||||
disabled: true,
|
||||
attrs: {},
|
||||
setAttribute(k, v) { this.attrs[k] = v; },
|
||||
removeAttribute(k) { delete this.attrs[k]; },
|
||||
};
|
||||
"""
|
||||
|
||||
_NODE_TAIL = """
|
||||
const show = (tag) => {
|
||||
console.log(JSON.stringify([
|
||||
tag,
|
||||
syncLabelMain.textContent,
|
||||
syncLabelCount.textContent,
|
||||
syncLabelCount.hidden,
|
||||
syncBtn.title,
|
||||
syncProgressText.textContent,
|
||||
syncBtn.attrs['aria-label'] || '',
|
||||
]));
|
||||
};
|
||||
// 1. the scan, denominator known + the position climbing (the mid-scan poll):
|
||||
enterSyncRunningState("sync", null, 0, 0, {
|
||||
phase: "scan", current_file: null, files_done: 0, files_total: 0,
|
||||
scan_done: 3, scan_total: 9, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("scan-3-9");
|
||||
// 2. the scan's degenerate arm: an empty KB / the scan's own first frame
|
||||
// (0/0) — the phase-64 guard, the bare label with NO count part:
|
||||
enterSyncRunningState("sync", null, 0, 0, {
|
||||
phase: "scan", current_file: null, files_done: 0, files_total: 0,
|
||||
scan_done: 0, scan_total: 0, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("scan-0-0");
|
||||
// 3. the prelude (null phase) is untouched — bare "Syncing\u2026":
|
||||
enterSyncRunningState("sync", null, 0, 0, {
|
||||
phase: null, current_file: null, files_done: 0, files_total: 0,
|
||||
scan_done: 4, scan_total: 9, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("prelude");
|
||||
// 4. the import phase is untouched — the phase-193 count-down + the chunk
|
||||
// part, and the scan pair present on the payload but NOT rendered:
|
||||
enterSyncRunningState("sync", "src/a/b.md", 2, 9, {
|
||||
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 9,
|
||||
scan_done: 9, scan_total: 9, current_chunk: 14, chunks_total: 38,
|
||||
});
|
||||
show("import");
|
||||
// 5. the overview + summaries labels are untouched:
|
||||
enterSyncRunningState("sync", null, 9, 9, {
|
||||
phase: "overview", current_file: null, files_done: 9, files_total: 9,
|
||||
scan_done: 9, scan_total: 9, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("overview");
|
||||
enterSyncRunningState("sync", null, 9, 9, {
|
||||
phase: "summaries", current_file: null, files_done: 9, files_total: 9,
|
||||
scan_done: 9, scan_total: 9, current_summary: "src/alpha",
|
||||
summaries_done: 2, summaries_total: 3, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("summaries");
|
||||
// 6. the 202/409 click (no status at all) stays bare:
|
||||
enterSyncRunningState("sync", null, 0, 0);
|
||||
show("click");
|
||||
"""
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_scan_label_renders_through_the_real_label_path_under_node() -> None:
|
||||
"""Behavior, not just markers: the REAL extracted functions run
|
||||
against stub refs. ``phase: "scan"`` renders main ``Scanning…`` with
|
||||
the count ``(3/9)`` visible, and the SAME string lands on the button
|
||||
title, the box (the sole aria-live region) and the button's
|
||||
accessible name (A4) — while every other phase renders exactly what it
|
||||
rendered before this task."""
|
||||
js = _js()
|
||||
fns = "\n".join(
|
||||
_fn(js, name)
|
||||
for name in (
|
||||
"fmtSyncLabel",
|
||||
"fmtSyncPhaseLabel",
|
||||
"setSyncLabel",
|
||||
"setSyncProgress",
|
||||
"showSyncCancelBtn",
|
||||
"hideSyncCancelBtn",
|
||||
"enterSyncRunningState",
|
||||
)
|
||||
)
|
||||
out = subprocess.run(
|
||||
["node", "-e", _NODE_HEAD + fns + _NODE_TAIL],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
)
|
||||
assert out.returncode == 0, out.stderr
|
||||
rows = {row[0]: row for row in (json.loads(line) for line in out.stdout.splitlines())}
|
||||
|
||||
main, count, hidden, title, box, aria = rows["scan-3-9"][1:]
|
||||
assert main == "Scanning\u2026", rows["scan-3-9"]
|
||||
assert count == "(3/9)" and hidden is False, rows["scan-3-9"]
|
||||
assert title == "Scanning\u2026 (3/9)" == box == aria, (
|
||||
f"A4: the untruncated scan label rides title + box + aria: {rows['scan-3-9']}"
|
||||
)
|
||||
|
||||
main, count, hidden, title, box, aria = rows["scan-0-0"][1:]
|
||||
assert main == "Scanning\u2026" and count == "" and hidden is True, rows["scan-0-0"]
|
||||
assert title == "Scanning\u2026" == box == aria, (
|
||||
f"0/0 renders NO count part (the empty-KB guard): {rows['scan-0-0']}"
|
||||
)
|
||||
|
||||
# every other state is byte-identical to its pre-phase label
|
||||
assert rows["prelude"][1] == "Syncing\u2026" and rows["prelude"][2] == ""
|
||||
assert rows["prelude"][4] == "Syncing\u2026", "a scan pair never renders in the prelude"
|
||||
assert rows["import"][1] == "Syncing\u2026 src/a/b.md"
|
||||
assert rows["import"][2] == "7 remaining", rows["import"]
|
||||
assert rows["import"][4] == "Syncing\u2026 src/a/b.md 7 remaining \u00b7 chunk 14/38"
|
||||
assert rows["overview"][1] == "Writing KB overview\u2026" and rows["overview"][2] == ""
|
||||
assert rows["summaries"][1] == "Summarizing folders\u2026 src/alpha"
|
||||
assert rows["summaries"][2] == "(2/3)", "the summaries position keeps its (n/m)"
|
||||
assert rows["click"][1] == "Syncing\u2026" and rows["click"][2] == ""
|
||||
Reference in New Issue
Block a user