phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
2026-10-20 06:36:21 -04:00
parent ae080ac7f3
commit a45f744604
55 changed files with 7610 additions and 179 deletions
+6 -1
View File
@@ -67,7 +67,12 @@ tiny chunk). The sorted walk processes ``big.md`` FIRST, so the label's
phase-193 count part reads ``1 remaining`` during the big span and is
HIDDEN during the small one (the final file: N = total - done = 0 — the
endpoint's ``files_done``/``files_total`` stay the 1-based 1/2 then 2/2
position; only the LABEL counts down) — and the small file's single
position; only the LABEL counts down). Phase 194 moves where that pair
comes FROM — the pre-scan's PREDICTED change set instead of the walk —
and changes nothing here, because this fixture is a FRESH import: both
files are pending, so the pending position IS the walk position (see
``tests/e2e/test_sync_remaining_count.py`` for the arm where they
diverge) — and the small file's single
chunk is the proof that the
chunk position RESTARTS per file (the big file ends at its last
batch's first index, the small file reports ``(1, 1)``).
+11 -3
View File
@@ -231,12 +231,19 @@ PRE_187_STATUS_KEYS = frozenset(
)
MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
#: link of the same additive chain (the payload is exactly the pre-187 set
#: plus the memory pair plus the scan pair).
SCAN_KEYS = frozenset({"scan_done", "scan_total"})
#: The pre-phase integer count slots (their types are part of the
#: unchanged-shape pin in test 2 — never a float, never null).
_COUNT_KEYS = frozenset(
{
"files_done",
"files_total",
"scan_done",
"scan_total",
"current_chunk",
"chunks_total",
"images_done",
@@ -643,8 +650,9 @@ def test_status_payload_reports_rss(page: Page, app_url: str, db_ready: None) ->
running = _wait_sync_state(page, app_url, {"running"})
assert running["state"] == "running", running
# The additive contract on the wire: exactly the pre-phase keys plus
# the memory pair — nothing renamed, nothing dropped.
assert set(running) == PRE_187_STATUS_KEYS | MEMORY_KEYS, set(running)
# the memory pair plus the phase-194 scan pair — nothing renamed,
# nothing dropped.
assert set(running) == PRE_187_STATUS_KEYS | MEMORY_KEYS | SCAN_KEYS, set(running)
for key, value in running.items():
if key in MEMORY_KEYS:
# The documented type: int | null (null is "no reading").
@@ -674,7 +682,7 @@ def test_status_payload_reports_rss(page: Page, app_url: str, db_ready: None) ->
# --- after the terminal ---------------------------------------------
terminal = _wait_sync_state(page, app_url, {"success", "failed", "cancelled"})
assert terminal["state"] == "success", terminal
assert set(terminal) == PRE_187_STATUS_KEYS | MEMORY_KEYS, set(terminal)
assert set(terminal) == PRE_187_STATUS_KEYS | MEMORY_KEYS | SCAN_KEYS, set(terminal)
current, peak = terminal["rss_mb"], terminal["rss_peak_mb"]
if MEMORY_KNOWABLE:
assert peak is not None and isinstance(peak, int), terminal
File diff suppressed because it is too large Load Diff
+4
View File
@@ -27,6 +27,10 @@ distance to the end — ``N remaining`` with ``N = total - done`` — so the
long file (the FIRST of the three) renders ``2 remaining`` while the
endpoint's ``files_done``/``files_total`` stay the 1-based 1/3 position.
The ``· chunk`` part and the terminal counts line are byte-identical.
Phase 194 (task 04, D5) leaves this suite UNEDITED for the same reason:
its fixture is a fresh import, so the pair's new source (the pre-scan's
predicted change set) and the old one (the walk) are the same three
files, and every ``2 remaining`` frame lands where phase 193 pinned it.
Run in isolation (DB must be up: ``podman compose up -d db``; git on PATH —
a documented environment prerequisite, phase 28):
+501 -56
View File
@@ -11,20 +11,46 @@ position *in* the whole; the owner reads the distance *to the end*.
Phase 193 changes exactly ONE label part: ``fmtSyncLabel``'s count is
now ``N remaining`` with ``N = total − done``.
**Server untouched.** No endpoint, no payload field, no poll changed:
``GET /api/sync/status`` still answers with the phase-64 key set
(``current_file`` / ``files_done`` / ``files_total`` — the 1-based
position of the in-flight file over the pre-walk count), and the suite
pins exactly that (test 1's cross-layer invariant): the payload still
carries the ``(done, total)`` position while the label speaks the
subtraction of the very same pair. The change is one string built by
``frontend/assets/sources.js``.
**Phase 194 (task 04, D5) retargets what the pair COUNTS — the label
builder stays byte-identical.** Phase 193 itself touched no server
code; phase 194 did, and this is the suite whose pins move with it.
``GET /api/sync/status`` no longer answers with the bare phase-64 key
set: it is that set plus the phase-187 memory pair plus the phase-194
scan pair (``scan_done`` / ``scan_total`` — the PRE-SCAN's own walk
position), and ``phase`` gained the ``"scan"`` stage between the null
prelude and ``"import"`` (the machine the page renders as
``Scanning… (n/total)``). More importantly the ``(done, total)`` pair
the label subtracts is no longer the WALK position: on a progress
caller (the UI sync is the only one) it is the position in the
importer's pre-scan PREDICTED change set — ``files_total`` the number
of files that will actually be re-indexed, ``files_done`` the 1-based
position in that set. ``N remaining`` therefore counts the WORK, which
is the fix for the owner's prod report of 2026-10-19 ("1216 remaining"
on a ~10,000-file KB with ~10 files to sync). The phase-193 wording —
the 1-based position across EVERY importable file of the stats
pre-walk — is exactly what ``BOR_SYNC_PENDING_COUNT=0`` restores, and
nothing below asserts it except as that rollback's history.
**A1 (owner-confirmed):** ``N remaining`` shows while ``N ≥ 1`` and is
HIDDEN at the final file (``N == 0``) — the main part names that file;
``total === 0`` (the pre-walk, the phase-64 contract) also renders no
count part. **A2 (owner-confirmed):** the upload job shares the ONE
builder, so it shares the shape (``Importing <file> N remaining``).
What survives the semantic shift UNEDITED is a property of this
suite's own fixture: every arm starts from a TRUNCATED KB, so the
prediction and the walk coincide (all ``N_FILES`` files are pending),
``files_total`` stays ``N_FILES`` and the ``1 … N−1`` count frames land
in the same slots phase 193 pinned — test 1's cross-layer invariant is
untouched arithmetic. The case where the two DIVERGE — a re-sync — is
test 4, and the phase's own suite
(``tests/e2e/test_sync_pending_count.py``, task 05) carries the K-of-N
incremental arm.
**A1 (owner-confirmed, phase 193):** ``N remaining`` shows while
``N ≥ 1`` and is HIDDEN at the final file (``N == 0``) — the main part
names that file; ``total === 0`` renders no count part either — and
since phase 194 there are TWO things that make it ``0``: the scan has
not reported a denominator yet (the null prelude / the ``0`` rollback,
test 2) and a run whose predicted change set is EMPTY (test 4).
**A2 (owner-confirmed):** the upload job shares the ONE builder, so it
shares the shape (``Importing <file> N remaining``) — and it gains
NOTHING from phase 194: the upload job runs no pre-scan and its
endpoint carries no scan fields (phase 90 A2), so test 3 is UNEDITED.
Fixture: a real git repo (the phase-28/32/107 ``file://`` transport-true
convention — ``BOR_GIT_SOURCES=file://<fixture repo>``, so the env
@@ -59,22 +85,44 @@ Arms (the phase's three, in test order):
label carries a count part at all while the main part still names
that file; no ``Syncing…`` label anywhere carries the pre-phase
``(n/m)`` tail; the ``(n/m)`` shape survives ONLY in the summaries
phase (A1's exception) and the overview label still paints; the
terminal result line is byte-identical to the phase-64 counts line
(the ``counts_line`` helper); and the endpoint's own tick series
still reports the untouched ``(done, total)`` position whose
subtraction equals every rendered ``N``.
phase (A1's exception) and, since phase 194, in the scan phase's own
``Scanning… (n/total)`` (never on a ``Syncing…`` label, and no scan
label ever follows an import one); the overview label still paints;
the terminal result line is byte-identical to the phase-64 counts
line (the ``counts_line`` helper); and the endpoint's own tick
series still reports the ``(done, total)`` pair whose subtraction
equals every rendered ``N``. Phase 194 adds the machine's own
evidence to that same tick series
(``_assert_phase_machine_tick_series``): every body is exactly the
pre-phase key set plus the memory pair plus the scan pair, the null
prelude reports the scan pair as ``0/0``, a tick caught INSIDE the
scan names no file and carries the full scan denominator, no scan
tick follows a tick that already counted files, and the terminal
KEEPS both pairs beside each other (the phase-64 keep-final-values
convention) with ``scan_total == files_total == N_FILES`` on this
fresh KB — the parity that lets the phase-193 frames stand. The scan
window itself is five tiny files (single-digit milliseconds), so a
100 ms tick is not guaranteed to land inside it: the deterministic
``Scanning… (n/total)`` frames are test 2's, and the choreographed
server-side sequence lives in
``tests/integration/test_sync_scan_phase.py``.
2. ``test_prewalk_and_final_file_labels_have_no_count_part`` — **(c)
the pre-walk edge**, pinned on the exact payload the guard reads:
the page's own 2 s poll fed ``phase: null`` / ``files_total: 0``
renders the bare ``Syncing…`` (no count part, the count span hidden,
the title equal), the first file renders ``N − 1 remaining``, a
the zero-total edges**, pinned on the exact payload the guards read:
the page's own 2 s poll fed a ``phase: "scan"`` body renders
``Scanning… (n/total)`` (count span visible, title equal, never a
file and never ``N remaining``), the same stage with
``scan_total: 0`` renders the bare ``Scanning…`` (the empty-KB
guard), a ``phase: null`` / ``files_total: 0`` prelude renders the
bare ``Syncing…`` (no count part, the count span hidden, the title
equal), the first file renders ``N − 1 remaining``, a
mid-run tick renders ``1 remaining`` *beside* ``· chunk 2/5``, the
final-file tick (``done == total``) renders the file with the count
part GONE and never brings it back, and the success terminal settles
to the untouched counts line. The served bodies use the status
endpoint's unchanged key set — the client's own decision points
(``total > 0``, ``total − done > 0``) are what is under test.
to the untouched counts line. The served bodies carry the status
endpoint's key set — the phase-64 fields plus the memory and scan
pairs, the stubs pinned against the same constant test 1 reads —
because the client's own decision points (``total > 0``,
``total − done > 0``, ``scan_total > 0``) are what is under test.
3. ``test_upload_label_shares_the_count_down_builder`` — **(b) the
upload arm (A2)**: the upload branch of the same two-job tree shares
the builder, so a running upload status that carries counts renders
@@ -88,12 +136,30 @@ Arms (the phase's three, in test order):
upload label can ever regress to ``(n/m)``. The upload branch never
receives a status object (its endpoint has no phase / chunk / image
fields — phase 129 / 152), so the chunk part coexists in the SYNC
arms only (test 1 and 2); that is asserted here too.
arms only (test 1 and 2); that is asserted here too. UNEDITED by
phase 194 — the upload job runs no pre-scan and its endpoint gains
no scan keys.
4. ``test_resync_of_an_unchanged_repo_reports_zero_work`` — **the
phase-194 re-sync arm**, the run phase 193's fixture never had (the
owner's prod case in its mildest form): sync this same repo TWICE
and the second run predicts NOTHING, so its ``(done, total)`` pair
is ``0/0`` on every tick — the A1 no-count-part exception driven by
an EMPTY pending set instead of by a missing denominator — while the
scan pair reports the whole KB walked (``scan_done == scan_total ==
N_FILES``, kept at the terminal). That divergence (5 scanned, 0 to
re-index) is the phase in one line, and it is exactly the shape that
used to render "``N`` remaining" for files nobody would re-index.
The terminal counts line stays the byte-identical
``0 added · N unchanged``. A no-change run makes no LLM call, so its
import window is milliseconds and it renders no file label: the wire
pair (``0/0``) plus test 2's ``total === 0`` guard pin is the honest
form of "no count part" here.
Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c):
1 → (a) + the terminal-line pin + the untouched-payload pin,
2 → (c) + the A1 hidden-at-final-file edge (+ (a)'s arithmetic on exact
inputs), 3 → (b).
Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c
+ task 04's D5 retarget): 1 → (a) + the terminal-line pin + the
additive-payload + phase-machine tick-series pins,
2 → (c) + the A1 hidden-at-final-file edge + the ``Scanning…`` frames
(+ (a)'s arithmetic on exact inputs), 3 → (b), 4 → the D5 re-sync arm.
"""
from __future__ import annotations
@@ -148,10 +214,11 @@ SOURCES_URL = "/sources.html"
#: its own (the module docstring's determinism note).
SLOW_DELAY_S = "1.5"
#: The module's wall-time budget (app boot + one full 5-file sync + the
#: two served-status arms). A breach is a speed canary, not flake
#: tolerance (the phase-64 convention).
WALL_BUDGET_S = 300.0
#: The module's wall-time budget (app boot + two full 5-file syncs — the
#: phase-194 re-sync arm runs the fixture's sync twice — plus the two
#: served-status arms). A breach is a speed canary, not flake tolerance
#: (the phase-64 convention).
WALL_BUDGET_S = 480.0
#: The sync's subject: a git repo whose DIRECTORY NAME is the source
#: name (the ``repo_name`` basename rule, phase 28). Deliberately NOT
@@ -196,11 +263,52 @@ CHUNK_OVERLAP = int(Settings.model_fields["chunk_overlap_chars"].default)
IMPORT_LABEL_RE = re.compile(
r"^Syncing… \S+(?: [0-9]+ remaining)?(?: · chunk \d+/\d+)?$"
)
#: The scan-phase label grammar phase 194 (task 03, D4) adds: the bare
#: ``Scanning…`` (no denominator yet — the empty KB and the scan's own
#: first frame) or it plus the ``(n/total)`` count part. NEVER a file
#: (``current_file`` is null through the phase) and never ``N
#: remaining`` — the walk position is honest behind the word "Scanning",
#: never behind "Syncing" (TODO L6).
SCAN_LABEL_RE = re.compile(r"^Scanning…(?: \(\d+/\d+\))?$")
#: The pre-phase shape phase 193 removed from this part — no ``Syncing…``
#: label may ever carry a ``(n/m)`` tail again.
#: label may ever carry a ``(n/m)`` tail again. Phase 194 leaves that
#: true: the ``(n/m)`` shape is legal ONLY on a ``Scanning…`` or a
#: ``Summarizing folders…`` label (and no label starting ``Syncing…``
#: matches either).
OLD_FRACTION_RE = re.compile(r"\(\d+/\d+\)")
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
#: The FULL ``GET /api/sync/status`` key set this suite's stubs mirror and
#: every real tick is pinned against — the phase-64/98/129/152 fields,
#: the phase-187 memory pair and the phase-194 SCAN pair (``scan_done`` /
#: ``scan_total``). The additive chain, pinned as ONE constant so a future
#: key has to be added here (and to the stub below) instead of drifting
#: away from the client's view of the payload.
EXPECTED_STATUS_KEYS = frozenset(
{
"state",
"started_at",
"finished_at",
"current_file",
"files_done",
"files_total",
"scan_done",
"scan_total",
"phase",
"current_summary",
"summaries_done",
"summaries_total",
"current_chunk",
"chunks_total",
"images_done",
"images_total",
"detail",
"error",
"rss_mb",
"rss_peak_mb",
}
)
SETTLE_TIMEOUT_MS = 180_000
@@ -527,17 +635,55 @@ class _TickRecorder:
"""The deterministic layer: a daemon thread tight-polling (~100 ms)
the SAME status endpoint the UI's 2 s poll reads, with its OWN admin
session (the Playwright sync API is not thread-safe, so the thread
never touches the page). It starts nothing — it only watches."""
never touches the page). It starts nothing — it only watches.
def __init__(self, base_url: str, path: str) -> None:
``wait_for_running`` is the phase-194 re-sync arm's need: a recorder
started while a PREVIOUS run's terminal is still on the endpoint must
not treat that body as its own. A body belongs to THIS run once the
recorder has seen it go ``running``, or — the belt-and-braces half,
for a run faster than the 100 ms cadence — once its own
``started_at`` post-dates the recorder's start (the
``test_fast_path_deferred_content.py`` idiom). Without the flag every
body counts, which is what the module's first arm wants (there is no
previous run to confuse it with).
"""
def __init__(self, base_url: str, path: str, *, wait_for_running: bool = False) -> None:
self._url = f"{base_url}{path}"
self._login_url = f"{base_url}/api/login"
self._wait_for_running = wait_for_running
self._saw_running = not wait_for_running
self._started_after = 0.0
self._ticks: list[dict[str, Any]] = []
self._terminal: dict[str, Any] | None = None
self._stop = threading.Event()
self._thread: threading.Thread | None = None
@staticmethod
def _epoch(body: dict[str, Any]) -> float | None:
"""The body's ``started_at`` as epoch seconds (``None`` when no
run ever started, or the stamp is unreadable)."""
raw = body.get("started_at")
if not raw:
return None
try:
return datetime.fromisoformat(str(raw)).timestamp()
except ValueError: # pragma: no cover — defensive, an odd stamp
return None
def _is_ours(self, body: dict[str, Any]) -> bool:
"""Does this body belong to the run this recorder was started for?"""
if body["state"] == "running":
self._saw_running = True
if self._saw_running:
return True
started = self._epoch(body)
return started is not None and started >= self._started_after - 1.0
def start(self) -> None:
if self._wait_for_running:
self._started_after = time.time()
def run() -> None:
with httpx.Client(timeout=5.0) as client:
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
@@ -546,6 +692,10 @@ class _TickRecorder:
r = client.get(self._url)
if r.status_code == 200:
body = r.json()
if not self._is_ours(body):
# A previous run's terminal — not ours.
self._stop.wait(0.02)
continue
self._ticks.append(body)
if body["state"] in ("success", "failed", "cancelled"):
self._terminal = body
@@ -575,6 +725,13 @@ class _TickRecorder:
def running_ticks(self) -> list[dict[str, Any]]:
return [t for t in self._ticks if t["state"] == "running"]
@property
def ticks(self) -> list[dict[str, Any]]:
"""EVERY body this recorder saw, the terminal included (the
phase-194 phase-machine pin reads the whole series, not just the
running frames)."""
return list(self._ticks)
def _serve_statuses(page: Page, stages: dict[str, Any]) -> None:
"""Route the two job-status endpoints to the CURRENT stage body (the
@@ -605,10 +762,12 @@ def _serve_statuses(page: Page, stages: dict[str, Any]) -> None:
def _sync_status(**overrides: Any) -> dict[str, Any]:
"""One ``GET /api/sync/status`` body with the endpoint's UNCHANGED
key set (the phase-64/98/129/152/187 fields the client reads) — the
served stubs are faithful to the payload shape; only the values are
the test's."""
"""One ``GET /api/sync/status`` body with the endpoint's CURRENT key
set (the phase-64/98/129/152/187 fields the client reads plus the
phase-194 scan pair — the additive chain ``EXPECTED_STATUS_KEYS``
pins) — the served stubs are faithful to the payload shape; only the
values are the test's. The equality against that constant is asserted
below so a stub can never drift from the real endpoint unnoticed."""
body: dict[str, Any] = {
"state": "idle",
"started_at": None,
@@ -616,6 +775,11 @@ def _sync_status(**overrides: Any) -> dict[str, Any]:
"current_file": None,
"files_done": 0,
"files_total": 0,
# Phase 194 (task 02): the pre-scan's position, additive after
# files_total and 0/0 until the scan reports (the degenerate and
# BOR_SYNC_PENDING_COUNT=0 edges both render no count part).
"scan_done": 0,
"scan_total": 0,
"phase": None,
"current_summary": None,
"summaries_done": 0,
@@ -629,6 +793,7 @@ def _sync_status(**overrides: Any) -> dict[str, Any]:
"rss_mb": None,
"rss_peak_mb": None,
}
assert set(body) == EXPECTED_STATUS_KEYS, set(body) ^ EXPECTED_STATUS_KEYS
body.update(overrides)
return body
@@ -648,6 +813,85 @@ def _upload_status(**overrides: Any) -> dict[str, Any]:
return body
def _assert_phase_machine_tick_series(
ticks: list[dict[str, Any]],
terminal: dict[str, Any],
*,
scan_total: int,
files_total: int,
prelude_zero: bool = True,
) -> None:
"""The phase-194 machine, read off the status endpoint the UI polls.
Every claim here is deterministic on a real run, which is why none of
them depends on a tick happening to land inside the scan (a five-file
scan is milliseconds — see the module docstring):
* the payload is ADDITIVE — exactly :data:`EXPECTED_STATUS_KEYS` on
every body, idle/running/terminal alike;
* the null prelude reports the scan pair as ``0/0`` (the run has not
reached ``import_sources`` yet — the scan is what sets it);
* a tick caught inside ``phase == "scan"`` names NO file, carries the
full ``scan_total`` denominator and a position that only climbs;
* the machine is ONE-DIRECTIONAL — no scan tick follows a tick that
already counted files (``files_total > 0`` means the first
``progress`` call flipped it, and the phase never flips back);
* the terminal CLEARS ``phase`` and KEEPS both pairs side by side
(the phase-64 keep-final-values convention);
* ``files_total`` is never larger than the scan's denominator on a
run whose KB did not change mid-run — the count and its denominator
stay on the SAME predicted set.
"""
for tick in ticks:
assert set(tick) == EXPECTED_STATUS_KEYS, (
f"the status payload drifted from the additive key set: "
f"{sorted(set(tick) ^ EXPECTED_STATUS_KEYS)}"
)
if terminal not in ticks:
assert set(terminal) == EXPECTED_STATUS_KEYS, terminal
if prelude_zero:
for tick in ticks:
if tick["state"] == "running" and tick["phase"] is None:
# The prelude (the model-check + clone/pull span): the
# scan has not started, so it has reported nothing. The
# success terminal sets state BEFORE clearing phase, so
# no terminal body can reach this arm.
assert (tick["scan_done"], tick["scan_total"]) == (0, 0), (
f"the null prelude already carried scan counts: {tick}"
)
scan_ticks = [t for t in ticks if t["phase"] == "scan"]
for tick in scan_ticks:
assert tick["current_file"] is None, (
f"the scan named a file — it counts, it does not read: {tick}"
)
assert tick["scan_total"] == scan_total, tick
assert 0 < tick["scan_done"] <= scan_total, tick
positions = [t["scan_done"] for t in scan_ticks]
assert positions == sorted(positions), (
f"the scan position went backwards across ticks: {positions}"
)
counted = [i for i, t in enumerate(ticks) if t["files_total"] > 0]
if scan_ticks and counted:
last_scan = max(i for i, t in enumerate(ticks) if t["phase"] == "scan")
assert last_scan < counted[0], (
"a scan frame followed a frame that already counted files — "
"the phase machine ran backwards"
)
assert terminal["phase"] is None, terminal
assert terminal["scan_total"] == scan_total, terminal
assert terminal["scan_done"] == scan_total, (
f"the terminal did not keep the completed scan: {terminal}"
)
assert terminal["files_total"] == files_total, terminal
assert terminal["files_total"] <= terminal["scan_total"], (
"the predicted change set is bigger than the scan that predicted "
f"it: {terminal}"
)
def _admin_sources_page(page: Page, app_url: str, *, settled: bool = True) -> None:
"""Real form login landing on the RAG page (the button's home).
@@ -796,9 +1040,11 @@ def test_running_label_counts_down_across_a_real_sync(
assert terminal["detail"]["added"] == N_FILES, terminal
# --- the cross-layer invariant: the label is the payload's subtraction
# The endpoint is untouched: it still answers with the 1-based
# position, and each rendered N equals total − done of the file it
# named.
# Neither half of this changed in phase 194: the SUBTRACTION is the
# phase-193 builder, and on this truncated-KB fixture the pair it
# subtracts (the pre-scan's pending position, phase 194) coincides
# with the walk position, so every rendered N still equals
# total − done of the file the label named.
position: dict[str, tuple[int, int]] = {}
for tick in recorder.running_ticks:
if tick["current_file"]:
@@ -821,27 +1067,62 @@ def test_running_label_counts_down_across_a_real_sync(
else:
assert shown is None, f"{path}: done==total but a part rendered: {frame}"
# --- phase 194 (task 04, D5): the machine, off the SAME tick series --
# The additive key set, the 0/0 prelude, the file-less scan ticks,
# the one-directional sequence and the terminal that keeps BOTH
# pairs (here equal, because a fresh KB's prediction IS its walk).
_assert_phase_machine_tick_series(
recorder.ticks, terminal, scan_total=N_FILES, files_total=N_FILES
)
# --- phase 194: the scan label, WHEREVER the page caught the stage ---
# Whether the UI's 2 s poll lands inside a five-file scan is a coin
# flip (test 2 pins the stage deterministically through a served
# status); what is never a coin flip is the SHAPE of the frames it
# does catch and where they may sit in the sequence.
scan_frames = [f for f in frames if f["text"].startswith("Scanning…")]
for frame in scan_frames:
assert SCAN_LABEL_RE.fullmatch(frame["text"]), frame
assert " remaining" not in frame["text"], frame
assert SOURCE not in frame["text"], f"the scan named a file: {frame}"
# A4: the untruncated label rides the title AND the announcer.
assert frame["title"] == frame["text"], frame
assert frame["aria"] == frame["text"], frame
assert frame["countHidden"] is not ("(" in frame["text"]), frame
first_import = next(
(i for i, t in enumerate(texts) if t.startswith("Syncing… ")), None
)
if first_import is not None:
assert all(
i < first_import for i, t in enumerate(texts) if t.startswith("Scanning…")
), f"a scan label rendered after the import phase began: {texts[:14]}"
# ---------------------------------------------------------------------------
# 2. (c) The pre-walk edge + the A1 hidden edge, on the exact payloads the
# guards read
# 2. (c) The zero-total edges + the A1 hidden edge, on the exact payloads
# the guards read
# ---------------------------------------------------------------------------
def test_prewalk_and_final_file_labels_have_no_count_part(
page: Page, app_url: str, db_ready: None
) -> None:
"""The two edges of A1, pinned on the payloads the guards actually
read (the served bodies use the status endpoint's unchanged key set —
what is under test is the client's ONE builder): ``total === 0``
renders no count part, and neither does ``done === total``."""
"""The zero-total edges of A1 plus the phase-194 scan stage, pinned on
the payloads the guards actually read (the served bodies carry the
status endpoint's key set — what is under test is the client's ONE
builder): ``scan_total === 0`` renders no count part, ``total === 0``
renders none either, and neither does ``done === total``."""
page.set_default_timeout(SETTLE_TIMEOUT_MS)
stages: dict[str, Any] = {
"sync": _sync_status(state="running"), # the pre-walk: no file, 0/0
# The stage phase 194 inserts where the pre-walk used to stand:
# the pre-scan mid-flight, denominator known, position climbing.
"sync": _sync_status(
state="running", phase="scan", scan_done=2, scan_total=N_FILES
),
"upload": None,
}
_serve_statuses(page, stages)
# settled=False: the boot fetch is the pre-walk stub — the re-attach
# settled=False: the boot fetch is the scan stub — the re-attach
# itself is part of what this arm pins.
_admin_sources_page(page, app_url, settled=False)
_install_recorder(page)
@@ -849,9 +1130,36 @@ def test_prewalk_and_final_file_labels_have_no_count_part(
box = page.locator("#sync-progress-text")
btn = page.locator("#sync-btn")
# (c) The pre-walk — phase null, current_file null, files_total 0:
# the bare prefix, no count part, the count span hidden, the title
# (phase 194, D4) The scan stage speaks its OWN label: "Scanning…"
# plus the (n/total) part in the SAME count slot the import phase's
# "N remaining" uses (one writer, one hidden-toggle, one A4
# placement) — never a file, never a chunk part, and the full label
# on the button title.
expect(box).to_have_text(f"Scanning… (2/{N_FILES})", timeout=SETTLE_TIMEOUT_MS)
# The PART's own toggle, not Playwright visibility: while a run is in
# flight the button is the spinner and CSS hides the label spans, so
# the honest pin is the writer's ``hidden`` flag plus the span's text
# (the in-browser recorder in test 1 reads the same property).
expect(page.locator(".sync-label-count")).to_have_text(f"(2/{N_FILES})")
expect(page.locator(".sync-label-count")).to_have_js_property("hidden", False)
expect(btn).to_have_attribute("title", f"Scanning… (2/{N_FILES})")
expect(btn).to_be_disabled()
# …and the same stage with NO denominator yet — the empty KB, and the
# scan's own first frame: the phase-64 ``total === 0`` guard renders
# the bare phase name, never "(0/0)".
stages["sync"] = _sync_status(state="running", phase="scan")
expect(box).to_have_text("Scanning…", timeout=SETTLE_TIMEOUT_MS)
expect(page.locator(".sync-label-count")).to_have_text("")
expect(page.locator(".sync-label-count")).to_have_js_property("hidden", True)
expect(btn).to_have_attribute("title", "Scanning…")
# (c) The null prelude — phase null, current_file null, files_total 0
# (the pre-phase pre-walk, and exactly what the
# BOR_SYNC_PENDING_COUNT=0 rollback reports for the whole run): the
# bare prefix, no count part, the count span hidden, the title
# identical (A4).
stages["sync"] = _sync_status(state="running")
expect(box).to_have_text("Syncing…", timeout=SETTLE_TIMEOUT_MS)
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(btn).to_have_attribute("title", "Syncing…")
@@ -913,19 +1221,41 @@ def test_prewalk_and_final_file_labels_have_no_count_part(
finished_at=_now_iso(),
files_done=N_FILES,
files_total=N_FILES,
# The scan pair KEPT at the terminal beside the file counts (the
# phase-64 keep-final-values convention) — the counts line below
# ignores them, which is the point of pinning them here.
scan_done=N_FILES,
scan_total=N_FILES,
detail={"added": N_FILES, "updated": 0, "unchanged": 0, "pruned": 0},
)
expect(box).to_have_text(counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
expect(btn).to_be_enabled()
# Nothing rendered in this run ever spoke the pre-phase shape or a
# "0 remaining".
# Nothing rendered in this run ever spoke the pre-phase shape on a
# "Syncing…" label or a "0 remaining".
texts = _texts(_frames(page))
assert not any(OLD_FRACTION_RE.search(t) for t in texts if t.startswith("Syncing…")), texts
assert not any("0 remaining" in t for t in texts), texts
assert "Syncing…" in texts, texts
# The scan stage rendered, in its own shape, and ONLY before the
# import phase started: the (n/m) is the scan's position (and the
# summaries phase's), never the sync's.
scan_texts = [t for t in texts if t.startswith("Scanning…")]
assert f"Scanning… (2/{N_FILES})" in scan_texts, texts
assert "Scanning…" in scan_texts, texts
for label in scan_texts:
assert SCAN_LABEL_RE.fullmatch(label), label
assert " remaining" not in label, label
first_import = next(
(i for i, t in enumerate(texts) if t.startswith("Syncing… ")), None
)
assert first_import is not None, texts
assert all(
i < first_import for i, t in enumerate(texts) if t.startswith("Scanning…")
), texts
# ---------------------------------------------------------------------------
# 3. (b) A2 — the upload job shares the builder and the shape
@@ -1021,3 +1351,118 @@ def test_upload_label_shares_the_count_down_builder(
# has no phase / chunk / image fields — phase 129 / 152), so the
# chunk part belongs to the sync arms only.
assert " · chunk " not in label, label
# ---------------------------------------------------------------------------
# 4. (phase 194, task 04 / D5) The re-sync: the prediction, not the walk
# ---------------------------------------------------------------------------
def test_resync_of_an_unchanged_repo_reports_zero_work(
page: Page, app_url: str, db_ready: None, remaining_repo: Path
) -> None:
"""The run phase 193's fixture never had: the SAME repo synced twice.
Run 1 fills the truncated KB, so its predicted change set IS the
whole walk (``files_total == scan_total == N_FILES`` — the parity
test 1 rides). Run 2 finds every byte where it left them, so the
pre-scan predicts NOTHING: the ``progress`` pair is ``0/0`` on every
tick and the client's ``total === 0`` guard (test 2) renders no count
part — while ``scan_done`` / ``scan_total`` report the whole KB walked
and are KEPT at the terminal beside the zeroed file pair. That
divergence (5 scanned, 0 to re-index) is phase 194 in one line, and
it is exactly the shape that used to render "``N`` remaining" for
files nobody was going to re-index (the owner's prod report).
A no-change run makes ZERO LLM call (the phase-147 fast path), so its
import window is milliseconds and the 2 s UI poll is not expected to
render a file label for it: the pin is the WIRE pair plus the
ABSENCE of a count part in whatever the page did render, not a
rendered per-file label (test 1 covers the rendered side).
"""
page.set_default_timeout(SETTLE_TIMEOUT_MS)
_admin_sources_page(page, app_url)
_install_recorder(page)
btn = page.locator("#sync-btn")
box = page.locator("#sync-progress-text")
# --- run 1: the fresh import, the parity test 1 rides ---------------
expect(btn).to_be_enabled()
# wait_for_running: test 1's TERMINAL is still what the endpoint
# answers with here (the run state lives in the app, not the page),
# and without the flag this recorder would hand that body back as its
# own before this test had synced anything at all.
rec1 = _TickRecorder(app_url, "/api/sync/status", wait_for_running=True)
rec1.start()
btn.click()
terminal1 = rec1.stop()
assert terminal1["state"] == "success", terminal1
expect(box).to_have_text(
counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS
)
assert terminal1["files_total"] == N_FILES, terminal1
assert terminal1["scan_total"] == N_FILES, terminal1
frame_mark = page.evaluate("() => window.__borFrames.length")
# --- run 2: the same bytes, the owner's incremental case ------------
expect(btn).to_be_enabled(timeout=SETTLE_TIMEOUT_MS)
rec2 = _TickRecorder(app_url, "/api/sync/status", wait_for_running=True)
rec2.start()
btn.click()
terminal2 = rec2.stop()
assert terminal2["state"] == "success", terminal2
# The owner-visible terminal line stays the byte-identical phase-64
# counts line — the exact surface the prediction never replaces.
expect(box).to_have_text(
counts_line(f"0 added · {N_FILES} unchanged"), timeout=SETTLE_TIMEOUT_MS
)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL)
expect(btn).to_be_enabled()
detail = terminal2["detail"]
assert (detail["added"], detail["updated"], detail["unchanged"]) == (
0,
0,
N_FILES,
), detail
# THE PHASE IN ONE LINE: the scan walked the KB, the work was nothing.
assert terminal2["scan_total"] == N_FILES, terminal2
assert terminal2["scan_done"] == N_FILES, terminal2
assert terminal2["files_total"] == 0, (
f"a no-change run reported a non-zero denominator: {terminal2}"
)
assert terminal2["files_done"] == 0, terminal2
for tick in rec2.ticks:
assert (tick["files_done"], tick["files_total"]) == (0, 0), (
f"the empty pending set leaked a walk position: {tick}"
)
assert tick["scan_total"] in (0, N_FILES), tick
# `prelude_zero=False`: the runner resets ``state`` to ``running``
# one synchronous statement before it resets the scan pair, and the
# status endpoint answers from a threadpool — a poll landing in that
# window would legitimately see the PREVIOUS run's kept 5/5. The
# claim itself is pinned race-free in test 1 and test 2; repeating it
# here would buy nothing but a 1-in-100k flake.
_assert_phase_machine_tick_series(
rec2.ticks, terminal2, scan_total=N_FILES, files_total=0, prelude_zero=False
)
# Whatever run 2 rendered carried NO count part: no "N remaining"
# (the empty pending set → the total === 0 guard) and no scan (n/m)
# once the import phase had begun.
run2_texts = _texts(_frames(page)[frame_mark:])
assert run2_texts, (
f"nothing was rendered after the run-1 settle mark ({frame_mark}) — "
"the click's own label is missing from the recorder"
)
assert not any(" remaining" in t for t in run2_texts), run2_texts
for label in run2_texts:
if label.startswith("Scanning…"):
assert SCAN_LABEL_RE.fullmatch(label), label
if label.startswith("Syncing…"):
assert not OLD_FRACTION_RE.search(label), label
scan_idx = [i for i, t in enumerate(run2_texts) if t.startswith("Scanning…")]
imp_idx = [i for i, t in enumerate(run2_texts) if t.startswith("Syncing… ")]
if scan_idx and imp_idx:
assert max(scan_idx) < imp_idx[0], run2_texts
@@ -101,6 +101,20 @@ turn the backfill into a crash instead of a query.
``_prune``'s light row (phase 184), the identity-map cadence (phase 184),
``app/db.py:30``, ``iter_importable_files`` (phase 185 LOCKED A3) and the
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
**Why phase 194 left these pins UNEDITED (the D5 sweep).** The phase-194
pre-scan is gated on the caller passing ``progress`` — and every
``import_sources`` call in this file passes NO ``progress`` (the statement
capture is the point, so the run needs no hook). The pre-scan therefore
never runs here: the captured fast-path ROW statement above stays the only
statement over ``documents`` per inspected file, the "exactly one
``SELECT`` per unchanged file" pin and the "one PK follow-up for the
summary-NULL backfill" pin are byte-identical, and the pre-scan's own
scalar statement (``SELECT documents.file_hash``, ``_pending_hash_stmt``)
appears nowhere in the capture — it is pinned in
``tests/unit/test_sync_pending_count.py`` and
``tests/integration/test_sync_pending_count.py`` instead, where a
``progress`` caller does exist.
"""
from __future__ import annotations
@@ -135,6 +135,18 @@ lazy SELECT per row and fail that test, not get papered over.
**What this file does NOT touch:** the walk's column set (phase 185),
``scripts/git_sync.py`` (phase 186), ``app/db.py:30``, and the
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
**Why phase 194 left the bound UNCHANGED (the D5 sweep).** Several tests
here call ``import_sources`` WITH a ``progress`` hook — so the phase-194
pre-scan really runs on those walks. It cannot move a single number in
this file: ``_prescan_pending`` reads the SCALAR projection
``_pending_hash_stmt`` (one column, no entity), through a session of its
OWN inside the ``to_thread`` span, so nothing lands in the RUN's
``identity_map`` — the staircase, the sawtooth, the ``MAP_BOUND`` ceiling
and the byte-identical ``ImportSummary`` / database comparisons are all
exactly what phase 184 measured. The hydration pin the claim rests on
(``len(session.identity_map) == 0`` at the first ``progress`` call) lives
in ``tests/integration/test_sync_pending_count.py``.
"""
from __future__ import annotations
+62 -11
View File
@@ -93,6 +93,18 @@ on all three terminals (success, cancelled, failed) plus the importer's
``import: summary …``. The fields are APPENDED (LOCKED A2), so every
pre-phase prefix below is byte-identical and the pins here assert BOTH
halves of that contract.
Phase 194 (task 02): the status gains the pre-scan. The payload is
ADDITIVE — ``scan_done`` / ``scan_total`` after ``files_total``, the
additive-chain pins below updated to the new key set (the idle shape and
both ``_PRE_187_STATUS_KEYS`` assertions) — and ``files_done`` /
``files_total`` now carry the PREDICTED-change-set position on a progress
caller. One existing pin retargets with it:
``test_double_trigger_while_running_returns_409`` waits on ``"scan"``
instead of ``"import"``, because its canned import fires NO per-file hook
and the scan→import flip lives in that hook — a run that reports no file
stays in the pre-import phase (the full sequence with a live hook is
pinned in ``tests/integration/test_sync_scan_phase.py``).
"""
from __future__ import annotations
@@ -375,6 +387,11 @@ class FakeImportSources:
# image fields stay 0/0, like the chunk fields of a no-batch
# import).
self.image_hooks: list[object] = []
# Phase 194 (task 02): the PRE-SCAN position hook the runner
# passes (a live closure while wired, None if the wiring
# regresses) — the canned fake never fires it, so the status
# scan fields stay 0/0 like a run that scanned nothing.
self.scan_hooks: list[object] = []
async def __call__(
self,
@@ -387,6 +404,7 @@ class FakeImportSources:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -397,6 +415,7 @@ class FakeImportSources:
self.progress_hooks.append(progress)
self.chunk_hooks.append(on_chunk_progress)
self.image_hooks.append(on_image_progress)
self.scan_hooks.append(on_scan_progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
self.doc_dates_maps.append(doc_dates_by_root or {})
@@ -572,6 +591,10 @@ def test_admin_sync_success_reports_full_detail(
# (the dataclass defaults).
"images_done": 0,
"images_total": 0,
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
# (the dataclass defaults).
"scan_done": 0,
"scan_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -703,17 +726,27 @@ def test_double_trigger_while_running_returns_409(
assert r.status_code == 409
assert r.json() == {"detail": "a sync is already running"}
# Phase 179 (task 02): WAIT for the import phase — the run's
# progress is concurrent now that the git ops left the loop, so the
# next request no longer inherits a run that already ran past them.
body = _wait_phase(sync_client, "import")
# Phase 179 (task 02): WAIT for the import to have started — the
# run's progress is concurrent now that the git ops left the loop,
# so the next request no longer inherits a run that already ran past
# them. Phase 194 (task 02): the phase the wait targets is now
# ``"scan"`` — the canned fake import reports NO file, and the
# scan→import flip lives in the runner's per-file progress hook, so
# a hook-less run legitimately stays in the pre-import phase (the
# sequence with a real hook is pinned in test_sync_scan_phase.py and
# in the phase-98 gated test below).
body = _wait_phase(sync_client, "scan")
assert body["state"] == "running"
assert body["started_at"] is not None
assert body["finished_at"] is None
assert body["error"] is None
# Wiring pin (phase 194, task 02): the runner passed its live
# scan-position closure, and a fake that fires nothing keeps the
# position at 0/0.
assert len(fake_import.scan_hooks) == 1 and fake_import.scan_hooks[0] is not None
assert body["scan_done"] == 0 and body["scan_total"] == 0
# Phase 98 (task 01): the in-flight run (parked in the fake
# import's delay) reports the import phase — the summary span has
# not started, so its counters are 0/0.
# import's delay) reports no summary span — its counters are 0/0.
assert body["current_summary"] is None
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
@@ -769,6 +802,10 @@ class _GatedImport:
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
# Phase 194 (task 02): the PRE-SCAN position hook the runner
# passes (recorded for the wiring pin — the gated fake never
# fires it, so the status scan fields stay 0/0).
self.scan_hook: Callable[[int, int], None] | None = None
async def __call__(
self,
@@ -781,10 +818,12 @@ class _GatedImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
self.scan_hook = on_scan_progress
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
@@ -1937,6 +1976,7 @@ def test_import_error_is_reported_with_credentials_masked(
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -2698,6 +2738,12 @@ _PRE_187_STATUS_KEYS = frozenset(
#: Phase 187 (task 03): the two additive memory keys.
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the
#: next link in the same chain (the payload is exactly the pre-187 set
#: plus the memory pair plus the scan pair, and a future addition joins
#: this chain rather than editing the sets above).
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
#: The pair is read from two DIFFERENT kernel counters and the high-water
#: refreshes LAZILY, so a raw ``rss_peak_mb()`` really can sit under a raw
#: ``rss_mb()`` (measured: ``rss_mb=164`` against ``rss_peak_mb=162``) —
@@ -2744,13 +2790,17 @@ def test_memory_status_keys_are_the_only_addition_and_null_before_a_run(
sync_client: TestClient,
) -> None:
"""The additive pin on the wire: the payload's key set is EXACTLY the
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb``, and a run that
never happened answers ``null`` / ``null`` — not 0, not a plausible
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb`` plus the
phase-194 ``scan_done`` / ``scan_total`` pair, and a run that never
happened answers ``null`` / ``null`` — not 0, not a plausible
number. That null is also what makes the Sources page render the
counts line with no peak clause after an app restart."""
_login(sync_client)
body = sync_client.get("/api/sync/status").json()
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, set(body)
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, set(body)
# Phase 194 (task 02): the scan pair joins ADDITIVELY — idle is 0/0
# (never null, never a plausible position).
assert body["scan_done"] == 0 and body["scan_total"] == 0
assert body["rss_mb"] is None
assert body["rss_peak_mb"] is None
# The pre-existing shapes, unchanged: the state word, the two ISO
@@ -2811,8 +2861,9 @@ def test_success_terminal_status_keeps_the_final_memory_pair(
assert done[0].endswith(
f"rss_mb={fmt_mb(current)} rss_peak_mb={fmt_mb(peak)}"
), done[0]
# The pre-existing payload is untouched by all of this.
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS
# The pre-existing payload is untouched by all of this (phase 194's
# scan pair is the only addition, and it is additive).
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS
assert again["files_done"] == 0 or isinstance(again["files_done"], int)
+14
View File
@@ -58,6 +58,19 @@ clone's result. Python threads are not interruptible: the git worker
runs its ``subprocess`` to completion, but the EVENT LOOP is free the
whole time.
**Why phase 194 leaves these git-span pins alone (the D5 sweep).** The
runner's import here is a stub (``fake_import`` below, which accepts the
phase-194 ``on_scan_progress`` argument and fires nothing), so no real
pre-scan runs inside this file's measurement window — the clone window
this suite pins stays the only off-loop span it measures. The pre-scan's
OWN loop-responsiveness pin (the same phase-179 pattern: a blocking
``_file_hash`` over a multi-file fixture while a heartbeat coroutine keeps
being scheduled, plus the worker-thread identity) lives in
``tests/integration/test_sync_pending_count.py`` — beside the cancel at
that span's await (the importer-side twin of this file's
cancel-during-clone pin) and its app-side twin in
``tests/integration/test_sync_scan_phase.py``.
**TestClient mechanics (why the trigger still rides a side thread).**
The shared app loop here is anyio's ``BlockingPortal`` thread. Firing
``POST /api/sync`` from the test thread would make the mid-clone probe
@@ -326,6 +339,7 @@ class _FakeImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
on_image_progress: Callable[[int, int], None] | None = None,
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
@@ -0,0 +1,571 @@
"""Integration: phase 194 — the pre-scan predicts the change set, off the loop.
The owner report that opened the phase (2026-10-19, verbatim): *"I just synced
the prod brain of reese and it said '1216 remaining' when it only had like 10
files to sync."* Prod holds ~10,000 importable files and that run changed ~10.
The phase-193 ``N remaining`` label was arithmetically fine — it was fed the
WALK (every importable file) instead of the WORK. The fix lives in
``app/rag/importer.py``: before the walk, ``_prescan_pending`` hashes every
importable file once and reads ``documents.file_hash`` once per file — the
exact predicate the phase-147 unchanged fast path applies — and the per-file
``progress`` hook reports the position in THAT set.
What this file measures with a real ``import_sources`` run:
* **the parity that makes the number honest** — on a stable knowledge base
the predicted set's size equals the run's terminal ``added + updated``: the
fresh arm (everything pending, exactly what a first import indexes), the
unchanged arm (nothing pending, so no count part at all), and the
incremental arm (one changed file of four — the owner's prod case, where
the walk is 4 and the work is 1);
* **zero entity hydration** — the scan's statement projects exactly
``documents.file_hash`` (no other column, no ORM row), so a 10,000-file
scan retains nothing: the run's session is still empty at the first
``progress`` call, before the processing loop has loaded any row (phase
184/185);
* **the span is OFF the loop** — with a stubbed hash that blocks, a heartbeat
coroutine keeps ticking through the scan and the hash runs on a different
thread than the one that awaited it (the phase-179 pattern of
``test_sync_git_freeze.py``: the git clone used to park the single uvicorn
loop and freeze the ``GET /api/sync/status`` poll the UI runs every 2 s,
and a 10,000-file hash+SELECT span inline is the same defect class);
* **the cancel contract** — cancelling the run at the scan's await lands the
cancellation immediately, while the worker itself runs to completion
(threads are not interruptible, phase 134) and nothing gets indexed.
The CLI and the upload job pass no ``progress``, so they run no scan at all;
their byte-identity is pinned in ``tests/unit/test_sync_pending_count.py``.
"""
from __future__ import annotations
import asyncio
import re
import threading
import time
from collections.abc import Callable, Iterator
from pathlib import Path
from typing import Any, NamedTuple
import pytest
from sqlalchemy import event, select, text
from sqlalchemy.orm import Session
import app.rag.importer as importer
from app.db import engine
from app.models import Document
from app.rag.importer import ImportSummary, import_sources
from tests.fakes import FakeEmbedder
#: The probe's own source-name prefix — nothing else in the KB uses it.
PREFIX = "phase194-it-"
#: A blocking hash of 60 ms per file: four files is a ~250 ms scan — long
#: enough that a heartbeat polling every 5 ms cannot tick during it if the
#: span ever runs on the loop thread, short enough that the suite stays fast.
HASH_BLOCK_S = 0.06
#: The heartbeat tick count that proves the loop was free across ONE scanned
#: file (60 ms / 5 ms ≈ 12 ticks; a blocked loop ticks 0). The ceiling-side
#: assertion is the same number generous: this is loop-freeness, not timing.
MIN_TICKS_PER_FILE = 4
class ProgressCall(NamedTuple):
"""One ``progress(source, rel, done, total)`` report."""
source: str
rel: str
done: int
total: int
class Run(NamedTuple):
"""One real ``import_sources`` pass and everything it reported."""
summary: ImportSummary
progress: list[ProgressCall]
scan: list[tuple[int, int]]
def _llm(**setting_overrides: Any) -> FakeEmbedder:
"""The deterministic in-process fake on the SHIPPED defaults (the
pending-count switch is therefore ON — the phase's default)."""
from app.config import Settings
llm = FakeEmbedder()
llm.settings = Settings(_env_file=None, **setting_overrides) # pyright: ignore[reportCallIssue]
return llm
def _root(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
"""One source root (the ``file://``-style temp dir of the house
import fixtures) with *files* under it."""
assert name.startswith(PREFIX), f"the probe source must be {PREFIX}-prefixed"
root = tmp_path / name
for rel, body in files.items():
target = root / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
return root
def _run(
session: Session,
*roots: Path,
llm: FakeEmbedder | None = None,
**kwargs: Any,
) -> Run:
"""One real import over *roots*, collecting both position hooks."""
progress: list[ProgressCall] = []
scan: list[tuple[int, int]] = []
summary = asyncio.run(
import_sources(
list(roots),
llm or _llm(),
session=session,
progress=lambda s, r, d, t: progress.append(ProgressCall(s, r, d, t)),
on_scan_progress=lambda p, t: scan.append((p, t)),
**kwargs,
)
)
return Run(summary, list(progress), list(scan))
@pytest.fixture()
def statement_capture() -> Iterator[list[str]]:
"""Every statement the engine executes for the duration of ONE test.
The ``before_cursor_execute`` technique of phase 183/185
(``tests/integration/test_import_fast_path_columns.py``): what a SELECT
actually projects is invisible in the ORM's Python API and plain in the
statement it compiles, so a capture is the only way to pin "the pre-scan
reads one column". Removed in teardown, so no other suite sees it.
"""
captured: list[str] = []
def _listener(
conn: object, # noqa: ARG001
cursor: object, # noqa: ARG001
statement: str,
parameters: object, # noqa: ARG001
context: object, # noqa: ARG001
executemany: bool, # noqa: ARG001
) -> None:
captured.append(statement)
event.listen(engine, "before_cursor_execute", _listener)
try:
yield captured
finally:
event.remove(engine, "before_cursor_execute", _listener)
@pytest.fixture(autouse=True)
def _clean_probe_documents(db: Session) -> Iterator[None]:
"""``documents``/``chunks`` are global state — the probe's rows are
removed before and after every test (the ``test_sync_api.py`` hygiene
pattern, narrowed to this module's own source names)."""
_drop(db)
yield
_drop(db)
def _drop(db: Session) -> None:
db.execute(text("DELETE FROM chunks WHERE document_id IN (SELECT id FROM "
"documents WHERE source LIKE :p)"), {"p": f"{PREFIX}%"})
db.execute(text("DELETE FROM documents WHERE source LIKE :p"), {"p": f"{PREFIX}%"})
db.commit()
db.expunge_all()
# --------------------------------------------------------------------------
# The parity: the predicted set IS the work
# --------------------------------------------------------------------------
_FOUR_FILES = {
"a.md": "# A\n\nalpha content\n",
"b.md": "# B\n\nbravo content\n",
"c.md": "# C\n\ncharlie content\n",
"notes/d.md": "# D\n\ndelta content\n",
}
def test_fresh_kb_pending_set_equals_the_added_count(
db: Session, tmp_path: Path
) -> None:
"""Arm 1 — the first import: walk == work, so the numbers are the
phase-193 ones (A1's degenerate parity, the reason the phase-193 E2E
suite never saw the defect).
Every file has no row, so every file is pending: the hook reports
``1/4 … 4/4`` and the terminal says ``added=4`` — the predicted size
equals ``added + updated`` exactly."""
root = _root(tmp_path, f"{PREFIX}fresh", _FOUR_FILES)
run = _run(db, root)
assert (run.summary.files, run.summary.added, run.summary.errors) == (4, 4, 0)
assert [(c.done, c.total) for c in run.progress] == [(1, 4), (2, 4), (3, 4), (4, 4)]
assert run.scan == [(1, 4), (2, 4), (3, 4), (4, 4)]
assert run.progress[-1].total == run.summary.added + run.summary.updated
def test_unchanged_kb_predicts_nothing_and_terminal_says_no_work(
db: Session, tmp_path: Path
) -> None:
"""Arm 2 — the no-change sync: nothing pending, so no count part.
The scan walks all four files and predicts none of them; every ``progress``
call therefore carries ``(0, 0)``, which the phase-64 ``total === 0``
guard renders as a bare label with no ``N remaining`` — the honest shape
for a run with no work, where the pre-phase label said ``4 remaining``
while walking. ``summary.files`` (the log's walked-file count) is
untouched: the run still processed 4 files and still logged 4 unchanged.
"""
root = _root(tmp_path, f"{PREFIX}unchanged", _FOUR_FILES)
seed = _run(db, root)
assert seed.summary.added == 4
run = _run(db, root)
assert (run.summary.files, run.summary.added, run.summary.updated) == (4, 0, 0)
assert run.summary.unchanged == 4
assert [(c.done, c.total) for c in run.progress] == [(0, 0)] * 4
# The scan still walked (and reported) every file — the DENOMINATOR of the
# scan is the importable count, not the prediction.
assert run.scan == [(1, 4), (2, 4), (3, 4), (4, 4)]
assert run.summary.added + run.summary.updated == 0
def test_incremental_kb_counts_the_changed_files_only(
db: Session, tmp_path: Path
) -> None:
"""Arm 3 — the owner's prod case at 1-of-4 scale.
One changed file out of four: the walk is still 4 (``summary.files``, the
log line, and every ``progress`` call naming a file), but the count is the
WORK — ``total`` 1, the position climbing only when the walk reaches the
changed file, and holding at 0 while it moves over the unchanged ones.
The parity pin: the prediction's size equals the terminal's
``added + updated``.
"""
root = _root(tmp_path, f"{PREFIX}incremental", _FOUR_FILES)
seed = _run(db, root)
assert seed.summary.added == 4
(root / "notes" / "d.md").write_text(
"# D\n\ndelta content EDITED on disk\n", encoding="utf-8"
)
run = _run(db, root)
assert (run.summary.files, run.summary.updated, run.summary.unchanged) == (
4,
1,
3,
)
assert [(c.rel, c.done, c.total) for c in run.progress] == [
("a.md", 0, 1),
("b.md", 0, 1),
("c.md", 0, 1),
("notes/d.md", 1, 1),
]
assert run.progress[-1].total == run.summary.added + run.summary.updated
def test_two_changed_files_count_down_to_zero(
db: Session, tmp_path: Path
) -> None:
"""The count the owner asked for, at the scale the label is read at.
Two changed files among four: the denominator is 2 from the first report,
the position advances only at the predicted files (the unchanged walk
between them holds it), and ``total - done`` reaches 0 exactly when the
last predicted file starts — where phase 193's ``N ≥ 1`` guard hides the
count part. The terminal reports the same two files as updated."""
root = _root(tmp_path, f"{PREFIX}two", _FOUR_FILES)
_run(db, root)
(root / "a.md").write_text("# A\n\nalpha edited\n", encoding="utf-8")
(root / "notes" / "d.md").write_text(
"# D\n\ndelta edited\n", encoding="utf-8"
)
run = _run(db, root)
assert (run.summary.updated, run.summary.unchanged) == (2, 2)
assert [(c.rel, c.done, c.total) for c in run.progress] == [
("a.md", 1, 2),
("b.md", 1, 2), # unchanged: the walk moves, the count holds
("c.md", 1, 2),
("notes/d.md", 2, 2),
]
remaining = [c.total - c.done for c in run.progress]
assert remaining == [1, 1, 1, 0], (
f"the label's remaining count went {remaining} — it must hold across "
"the unchanged files and reach 0 at the last predicted one (where "
"phase 193's guard hides the part)"
)
# --------------------------------------------------------------------------
# Zero entity hydration
# --------------------------------------------------------------------------
def test_prescan_reads_one_column_and_hydrates_nothing(
db: Session, tmp_path: Path, statement_capture: list[str]
) -> None:
"""The scan's statement projects EXACTLY ``documents.file_hash`` and the
run's session is still empty when the walk begins.
Three pins on the same claim, because the failure mode is the 2026-10-15
incident (a full walk hydrating 252 MB of markdown):
* every ``documents`` ``(source, path)`` SELECT issued before the first
``progress`` call — i.e. by the scan — names ``file_hash`` and NOTHING
else, no ``content`` above all (the phase-185 column-list technique);
* exactly one such read per importable file;
* ``len(session.identity_map) == 0`` at the first ``progress`` call: the
scan put no ``Document`` anywhere, so the run's retention stays bounded
by the phase-184 cadence instead of growing with the KB.
"""
root = _root(tmp_path, f"{PREFIX}hydrate", _FOUR_FILES)
_run(db, root)
db.expunge_all()
statement_capture.clear()
map_at_first_progress: list[int] = []
def progress(source: str, rel: str, done: int, total: int) -> None: # noqa: ARG001
map_at_first_progress.append(len(db.identity_map))
summary = asyncio.run(
import_sources(
[root], _llm(), session=db, progress=progress, on_scan_progress=None
)
)
assert summary.unchanged == 4
# The scan reads precede the loop's row loads in capture order, and the
# first progress call is the boundary between them. Classified on the
# PROJECTION (phase 185's technique): the scan's projection is the single
# column token, the fast path's is every column but ``content``.
def predicate(stmt: str) -> str:
"""Everything after the statement's ``FROM`` (case-insensitive —
SQLAlchemy compiles the keyword upper-case)."""
return re.split(r"\bfrom\s+documents\b", stmt, maxsplit=1, flags=re.I)[-1]
doc_reads = [
stmt
for stmt in statement_capture
if stmt.lstrip().lower().startswith("select")
and re.search(r"\bfrom\s+documents\b", stmt, re.I)
and "documents.source" in predicate(stmt)
and "documents.path" in predicate(stmt)
]
def projection(stmt: str) -> set[str]:
head = re.split(r"\bfrom\b", stmt, maxsplit=1, flags=re.I)[0]
return set(re.findall(r"documents\.([a-z_]+)", head))
scan_reads = [s for s in doc_reads if projection(s) == {"file_hash"}]
row_loads = [s for s in doc_reads if projection(s) != {"file_hash"}]
assert len(scan_reads) == 4, (
f"the unchanged run issued {len(scan_reads)} single-column file_hash "
"reads — the scan reads each importable file's hash exactly once"
)
assert len(row_loads) == 4, (
f"the loop issued {len(row_loads)} fast-path row loads — the scan must "
"not replace _doc_row_stmt, which stays the only ROW load (phase 185)"
)
assert all(
projection(s) == {"file_hash"} for s in scan_reads
), "a scan read projects more than file_hash — it would hydrate a row"
assert all(
"content" not in projection(s) for s in row_loads
), "the fast-path row load stopped deferring content (phase 185)"
assert map_at_first_progress and map_at_first_progress[0] == 0, (
"the run's session already held entities when the loop reached its "
"first file — the pre-scan leaked a row into the identity map"
)
# --------------------------------------------------------------------------
# The span is off the event loop (the phase-179 pattern)
# --------------------------------------------------------------------------
class _SlowHash:
"""The timing fixture: a PLAIN blocking ``_file_hash`` (like the real one
— the point under test is WHERE it runs), counting its calls.
The thread identity is NOT taken from here: ``_file_hash`` is called both
by the scan (worker thread) and by the processing loop (loop thread), so
the attribution comes from ``on_scan_progress``, which fires on the same
worker thread as the scan's own hashes."""
def __init__(self, real: Callable[[Path], str], block: float = HASH_BLOCK_S) -> None:
self.real = real
self.block = block
self.calls = 0
def __call__(self, path: Path) -> str:
self.calls += 1
time.sleep(self.block)
return self.real(path)
def test_prescan_span_does_not_park_the_event_loop(
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE phase-179 contract, re-pinned for the pre-scan.
A 4-file scan with a 60 ms blocking hash is ~250 ms of work that used to
be inline. The heartbeat coroutine polls every 5 ms; the ticks between two
consecutive scan reports prove the LOOP kept running while the worker
blocked (a regression that runs the scan inline parks the loop thread and
the heartbeat cannot tick — the task-01 git measurement was 0 ticks over
a 3 s clone). The thread identities are asserted too, so the pin fails on
a machine fast enough to hide the latency.
"""
root = _root(tmp_path, f"{PREFIX}freeze", _FOUR_FILES)
stub = _SlowHash(importer._file_hash)
monkeypatch.setattr(importer, "_file_hash", stub)
ticks = 0
scan_ticks: list[int] = []
scan_positions: list[tuple[int, int]] = []
scan_thread: list[int] = []
def scan_hook(position: int, total: int) -> None:
# Runs on the span's worker thread — the same thread that just hashed.
scan_positions.append((position, total))
scan_ticks.append(ticks)
scan_thread.append(threading.get_ident())
# A progress caller is what opens the gate (the CLI shape, ``progress=None``,
# would skip the scan entirely) — this is the UI sync's shape.
async def main() -> int:
nonlocal ticks
loop_ident = threading.get_ident()
async def heartbeat() -> None:
nonlocal ticks
while True:
await asyncio.sleep(0.005)
ticks += 1
hb = asyncio.create_task(heartbeat())
try:
await import_sources(
[root],
_llm(),
session=db,
progress=lambda s, r, d, t: None,
on_scan_progress=scan_hook,
)
finally:
hb.cancel()
return loop_ident
loop_ident = asyncio.run(main())
assert scan_positions == [(1, 4), (2, 4), (3, 4), (4, 4)]
assert scan_thread and loop_ident is not None
assert all(ident != loop_ident for ident in scan_thread), (
"the pre-scan ran on the event-loop thread — a long scan parks the "
"single uvicorn loop again (phase 179's defect class)"
)
deltas = [
b - a for a, b in zip([0, *scan_ticks], scan_ticks, strict=False)
]
assert min(deltas) >= MIN_TICKS_PER_FILE, (
f"the heartbeat ticked {deltas} times across consecutive scanned files "
f"(expected >= {MIN_TICKS_PER_FILE} each) — the loop was starved while "
"the scan blocked, so the span is not off-loop"
)
# --------------------------------------------------------------------------
# The cancel (phase 134): the run stops, the worker finishes
# --------------------------------------------------------------------------
def test_cancel_during_the_prescan_stops_the_run_while_the_worker_finishes(
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The phase-134 cooperative cancel, measured at the scan's await.
Cancelling the run ~80 ms into a ~250 ms scan raises ``CancelledError``
out of ``import_sources`` PROMPTLY (well before the scan's own remaining
work), NOTHING is indexed, and the worker thread runs to completion —
threads are not interruptible, and this is exactly why the span reads
through a session of its own instead of the run's (which the cancel
closes on its way out). The pre-phase-194 pipeline had no await here at
all, so this is the new boundary the UI's Stop button must survive.
"""
root = _root(tmp_path, f"{PREFIX}cancel", _FOUR_FILES)
stub = _SlowHash(importer._file_hash, block=0.12)
monkeypatch.setattr(importer, "_file_hash", stub)
real_prescan = importer._prescan_pending
worker_finished = threading.Event()
worker_results: list[Any] = []
def spy(*args: Any, **kwargs: Any) -> Any:
try:
result = real_prescan(*args, **kwargs)
worker_results.append(result)
return result
finally:
worker_finished.set()
monkeypatch.setattr(importer, "_prescan_pending", spy)
async def main() -> float:
# The hook runs on the WORKER thread, so it only appends to a list —
# the house hook contract (a GIL-atomic store). Touching an
# ``asyncio.Event`` from here would be a cross-thread loop call.
seen: list[tuple[int, int]] = []
def scan_hook(position: int, total: int) -> None:
seen.append((position, total))
task = asyncio.create_task(
import_sources(
[root],
_llm(),
session=db,
progress=lambda s, r, d, t: None,
on_scan_progress=scan_hook,
)
)
# Wait (from the loop) until the scan has reported its first file.
deadline = time.monotonic() + 10.0
while not seen and time.monotonic() < deadline:
await asyncio.sleep(0.005)
assert seen, "the pre-scan never reported a file"
await asyncio.sleep(0.08)
t0 = time.monotonic()
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
return time.monotonic() - t0
cancel_latency = asyncio.run(main())
# The cancel is prompt: the run did NOT wait for the remaining scan work.
assert cancel_latency < 0.5, (
f"the cancelled run took {cancel_latency:.3f}s to unblock — it is "
"still waiting for the blocking scan worker"
)
# The worker is not interruptible: it finishes its own session's work.
assert worker_finished.wait(10.0), "the pre-scan worker never completed"
assert stub.calls == 4, (
f"the cancelled run's scan worker hashed {stub.calls} of 4 files — "
"the contract is that the worker runs to completion"
)
# And the pipeline that would have consumed the prediction never ran.
assert db.scalar(
select(select(Document).where(Document.source == root.name).exists())
) is False, "the cancelled run indexed documents — the import never started"
+927
View File
@@ -0,0 +1,927 @@
"""Integration: phase 194 (task 02) — the ``"scan"`` phase on the status surface.
The owner report that opened the phase (2026-10-19): *"it said '1216
remaining' when it only had like 10 files to sync"*. Task 01 moved the
count onto the pre-scan's PREDICTED change set; this file pins the
status half (design D3, owner-confirmed A2 — the pre-scan is a VISIBLE
stage, not an invisible pause):
* the phase machine is ``null`` → ``"scan"`` (set immediately before
``import_sources``, where the scan actually runs) → ``"import"``
(flipped by the FIRST per-file ``progress`` call) → ``"overview"`` →
``"summaries"`` → the terminal clears ``phase``;
* ``scan_done`` / ``scan_total`` carry the scan's own position through
that stage, reset with the run, and are KEPT at every terminal beside
the final file counts (the phase-64 keep-final-values convention);
* the payload is ADDITIVE — the pre-phase key set plus exactly those two
keys, byte-identical in name and type, stable from idle through the
terminal;
* the phase-187 sampling discipline holds on the new stage: the ``"scan"``
set and the ``scan``→``import`` flip are TRANSITIONS and force the
``/proc`` read, while the scan's own per-file hook is THROTTLED — one
read per interval, never one per file (LOCKED A4);
* the two degenerate/terminal edges: an empty knowledge base sits in
``"scan"`` and never reaches ``"import"`` (no ``progress`` call exists
to flip it) and a cancel DURING the scan lands the ``cancelled``
terminal with the scan counts kept and nothing indexed (the phase-134
contract, the app-side twin of the importer-side pin in
``tests/integration/test_sync_pending_count.py``).
**Why one arm is choreographed and one is real.** An in-process import
never awaits real I/O (the embedder is a fake), so the loop cannot answer a
status poll from inside it — a real run's ``"import"`` phase is therefore
NOT reliably observable, and the real arm pins what the terminal keeps
(both pairs, divergent on the incremental run). The flip itself gets a
deterministic pin from the choreographed double (the ``test_sync_api``
gated-import idiom), which drives the runner's OWN closures and parks at
every frame the test reads. The scan, by contrast, runs in a
``to_thread`` worker, so the loop is FREE through it — that is exactly why
its frames are observable, and why the blocking-hash fixture slows ONLY the
scan window (a blanket stub would park the loop through the import too).
House patterns (``test_sync_api.py`` / ``test_sync_git_freeze.py``): the
``sync_client`` fixture is a CONTEXT-MANAGED ``TestClient`` — one app event
loop across requests, so the background run survives between the POST and
the polls; the module ``_status`` / ``_task`` / ``_cancel_event`` /
``_rss_sampled_at`` and the ``git_sources`` / ``sources_meta`` /
``folder_summaries`` / ``documents`` global state are reset around every
test; the probe is stubbed so nothing reaches the network, and the import
runs for real over a temp ``kind=local`` source with the deterministic
``FakeEmbedder``. The scan is made observable with a PLAIN blocking
``_file_hash`` stub scoped to the scan window (the phase-179 timing-fixture
idiom): the point under test is WHICH phase the run is in WHEN the page
polls, and a scan of instant hashes is over before the first poll.
"""
from __future__ import annotations
import asyncio
import logging
import threading
import time
from collections.abc import Callable, Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy.orm import Session
import app.rag.importer as importer
from app.api import sync as sync_api
from app.config import Settings
from app.db import SessionLocal, db_available
from app.main import app as fastapi_app
from app.models import Document, GitSource
from app.rag import git_sources as git_sources_resolver
from app.rag.importer import ImportSummary
from app.rag.llm import LLMClient
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
#: The probe's own source-name prefix — nothing else in the KB uses it,
#: so the module's DELETE-by-source cleanup can never touch another
#: suite's rows.
PREFIX = "phase194-t02-"
#: One blocking ``_file_hash`` per file. Three files is a ~360 ms scan:
#: long enough that a 10 ms poll catches several distinct positions inside
#: the phase (the frames are what the UI's 2 s poll would render), short
#: enough that the suite stays fast.
HASH_BLOCK_S = 0.12
# --- fixtures (the house sync hygiene) --------------------------------------
@pytest.fixture(autouse=True)
def _fresh_sync_state() -> Iterator[None]:
"""The module-level status object + task + cancel event + sampling
guard are process-global — reset before AND after every test so no
terminal state, stale cancel or stale throttle window leaks into the
next one (``test_sync_api`` / ``test_sync_git_freeze``)."""
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
sync_api._cancel_event = asyncio.Event()
sync_api._rss_sampled_at = float("-inf")
yield
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
sync_api._cancel_event = asyncio.Event()
sync_api._rss_sampled_at = float("-inf")
@pytest.fixture(autouse=True)
def clean_git_sources(db: Session) -> Iterator[None]:
"""The runner resolves through the real ``git_sources`` table —
truncated around every test (the rows this file seeds are
``kind=local``)."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def _reset_sources_version() -> Iterator[None]:
"""The phase-53 bump is global mutable state — pin it to the
migration-0010 seed (0) around every test (own session: the runner
bumps through its own short-lived ``SessionLocal``)."""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
session = SessionLocal()
try:
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
session.commit()
yield
finally:
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
session.commit()
session.close()
@pytest.fixture(autouse=True)
def _clean_folder_summaries(db: Session) -> Iterator[None]:
"""``folder_summaries`` is global state the real generator writes on a
changed KB — truncated around every test."""
db.execute(text("TRUNCATE folder_summaries"))
db.commit()
yield
db.execute(text("TRUNCATE folder_summaries"))
db.commit()
@pytest.fixture(autouse=True)
def _clean_probe_documents(db: Session) -> Iterator[None]:
"""The real imports of this module write ``documents``/``chunks`` —
removed before and after every test, narrowed to this module's own
source names (the ``test_sync_pending_count`` hygiene)."""
_drop(db)
yield
_drop(db)
def _drop(db: Session) -> None:
db.execute(
text("DELETE FROM chunks WHERE document_id IN "
"(SELECT id FROM documents WHERE source LIKE :p)"),
{"p": f"{PREFIX}%"},
)
db.execute(text("DELETE FROM documents WHERE source LIKE :p"), {"p": f"{PREFIX}%"})
db.commit()
db.expunge_all()
@pytest.fixture()
def sync_client() -> Iterator[TestClient]:
"""Context-managed ``TestClient`` — one app event loop across requests
(the background run must survive between the POST and the polls)."""
with TestClient(fastapi_app) as client:
yield client
# --- helpers ----------------------------------------------------------------
def _settings(sources_dir: str) -> Settings:
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""Empty env fallback (the dev ``.env`` never leaks in) — this file's
runs resolve their one local row from the table."""
monkeypatch.setattr(
git_sources_resolver,
"get_settings",
lambda: Settings(_env_file=None, git_sources=""), # pyright: ignore[reportCallIssue]
)
def _seed_local(db: Session, path: Path) -> None:
"""A ``kind=local`` row exactly as the phase-38 API stores it."""
db.add(GitSource(url=str(path), kind="local", path=str(path)))
db.commit()
def _local_source(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
"""One source root (always created — the empty-KB arm needs a real
empty directory, not a missing one), with *files* under it."""
assert name.startswith(PREFIX), f"the probe source must be {PREFIX}-prefixed"
root = tmp_path / name
root.mkdir(parents=True, exist_ok=True)
for rel, body in files.items():
target = root / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
return root
def _login(client: TestClient) -> None:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
def _poll(client: TestClient, want: str, timeout: float = 20.0) -> dict:
"""Poll until ``state == want``, failing loudly on any other state."""
deadline = time.monotonic() + timeout
body: dict = {}
while time.monotonic() < deadline:
body = client.get("/api/sync/status").json()
if body["state"] == want:
return body
assert body["state"] == "running", (
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
)
time.sleep(0.02)
raise AssertionError(f"sync did not reach {want!r} within {timeout}s: {body}")
def _watch_scan(client: TestClient, timeout: float = 20.0) -> tuple[list[dict], dict | None]:
"""Poll through the run's SCAN window; stop at the first frame that is
no longer a scan.
Returns ``(frames, terminal)`` — ``terminal`` is non-``None`` when the
run finished inside the window. Both stops are honest: the scan runs
OFF the loop (so its frames are served, and this list is exactly what
the label was built from), while the import that follows is one
straight run of the importer that can finish before the next poll.
The flip frame therefore gets pinned deterministically by the
choreographed test below; what the real run pins here is the scan's
own frames plus the values the terminal keeps."""
frames: list[dict] = []
terminal: dict | None = None
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
body = client.get("/api/sync/status").json()
if body["state"] != "running":
terminal = body
break
frames.append(body)
if body["phase"] != "scan":
break
time.sleep(0.01)
else:
raise AssertionError(f"the run never left the scan phase: {frames[-1]}")
return frames, terminal
class _FakeScanImport:
"""A parked ``import_sources`` double for the app-side edges.
It records the two position closures the runner passes (the wiring
pin) and fires NOTHING — which is exactly what a run with no
importable file does (no scan report, so no position, and no
``progress`` call to flip the phase; the importer-side proof lives in
the empty-KB arm below and in ``test_sync_pending_count.py``). The
park lets the test read the status inside the ``"scan"`` phase without
racing the pipeline; the caller always releases it in ``finally``.
"""
def __init__(self, summary: ImportSummary | None = None) -> None:
self.summary = summary if summary is not None else ImportSummary()
self.started = threading.Event()
self.release = threading.Event()
self.progress_hook: Callable[[str, str, int, int], None] | None = None
self.scan_hook: Callable[[int, int], None] | None = None
async def __call__(
self,
sources: list[Path],
llm: LLMClient,
*,
prune: bool = False,
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
on_image_progress: Callable[[int, int], None] | None = None,
on_scan_progress: Callable[[int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
) -> ImportSummary:
self.progress_hook = progress
self.scan_hook = on_scan_progress
self.started.set()
await asyncio.to_thread(self.release.wait)
return self.summary
def _install_counting_memory(monkeypatch: pytest.MonkeyPatch) -> list[dict[str, Any]]:
"""Replace the two memory helpers on ``app.api.sync`` with recorders.
``_sample_rss`` calls them ONLY when the throttle lets it, so one
entry per entry below is one real read — and each entry carries the
phase (and the scan/file position) at that instant, which is the
difference between "a transition forced a read" and "a per-file hook
read once per file" (LOCKED A4)."""
reads: list[dict[str, Any]] = []
def _record() -> None:
reads.append(
{
"phase": sync_api._status.phase,
"scan_done": sync_api._status.scan_done,
"files_done": sync_api._status.files_done,
"current_file": sync_api._status.current_file,
}
)
def fake_rss_mb() -> int:
_record()
return 100
def fake_rss_peak_mb() -> int:
return 120
monkeypatch.setattr(sync_api, "rss_mb", fake_rss_mb)
monkeypatch.setattr(sync_api, "rss_peak_mb", fake_rss_peak_mb)
return reads
def _slow_hash(
monkeypatch: pytest.MonkeyPatch, block: float = HASH_BLOCK_S
) -> dict[str, bool]:
"""Make ONLY the pre-scan window slow, and return its flag.
``_file_hash`` is called by the scan AND by the processing loop, so a
blanket blocking stub would also park the event loop through the whole
import (an in-process import never awaits real I/O, so the loop cannot
answer a status poll inside it) — the frames this file needs would
then be unservable rather than absent. The flag is raised around
:func:`app.rag.importer._prescan_pending` only, so the scan's window is
the fixture's window: the loop stays free exactly while the run is in
the phase under test (the phase-179 off-loop claim, observed from the
HTTP side), and the import after it runs at full speed.
The flag is a plain ``dict`` entry written/read from two threads: the
house hook contract (a GIL-atomic store) applies here for the same
reason it applies to the runner's closures.
"""
real_hash = importer._file_hash
real_prescan = importer._prescan_pending
state = {"in_scan": False}
def slow(path: Path) -> str:
if state["in_scan"]:
time.sleep(block)
return real_hash(path)
def spy_prescan(*args: Any, **kwargs: Any) -> Any:
state["in_scan"] = True
try:
return real_prescan(*args, **kwargs)
finally:
state["in_scan"] = False
monkeypatch.setattr(importer, "_file_hash", slow)
monkeypatch.setattr(importer, "_prescan_pending", spy_prescan)
return state
class _ChoreographedImport:
"""An ``import_sources`` double that replays the importer's hook script
through the runner's OWN closures, parking at every point the test
reads the status.
The choreography (all four stops are ``threading.Event`` pairs, so no
test here races the pipeline):
1. fire ``scan_pairs[:1]`` through ``on_scan_progress`` → ``first``
(the test reads the scan's FIRST position);
2. on ``go_on``: fire the rest → ``scanned`` (the test reads the last
position, still inside the phase);
3. on ``go_import``: fire ``progress_call`` through ``progress`` —
THE FLIP → ``flipped`` (the test reads the ``"import"`` frame);
4. on ``release``: return the canned summary.
"""
def __init__(
self,
summary: ImportSummary,
scan_pairs: list[tuple[int, int]],
progress_call: tuple[str, str, int, int],
) -> None:
self.summary = summary
self.scan_pairs = list(scan_pairs)
self.progress_call = progress_call
self.first = threading.Event()
self.go_on = threading.Event()
self.scanned = threading.Event()
self.go_import = threading.Event()
self.flipped = threading.Event()
self.release = threading.Event()
self.scan_hook: Callable[[int, int], None] | None = None
self.progress_hook: Callable[[str, str, int, int], None] | None = None
async def __call__(
self,
sources: list[Path],
llm: LLMClient,
*,
prune: bool = False,
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
on_image_progress: Callable[[int, int], None] | None = None,
on_scan_progress: Callable[[int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
) -> ImportSummary:
# The wiring pin: both closures arrive live (None here would mean
# the runner stopped passing them).
self.scan_hook = on_scan_progress
self.progress_hook = progress
assert on_scan_progress is not None and progress is not None
for pair in self.scan_pairs[:1]:
on_scan_progress(*pair)
self.first.set()
await asyncio.to_thread(self.go_on.wait)
for pair in self.scan_pairs[1:]:
on_scan_progress(*pair)
self.scanned.set()
await asyncio.to_thread(self.go_import.wait)
progress(*self.progress_call)
self.flipped.set()
await asyncio.to_thread(self.release.wait)
return self.summary
def _stub_probe(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_check_models(llm: LLMClient) -> None:
return None
monkeypatch.setattr(sync_api, "check_models", fake_check_models)
def _install_local_run(
monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path, source: Path
) -> None:
"""Point one app run at *source*: local row, tmp sources root, stubbed
probe, the deterministic in-process embedder (the real import, no
network)."""
_seed_local(db, source)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api, "get_settings", lambda: _settings(sources_dir=str(tmp_path / "bor"))
)
_stub_probe(monkeypatch)
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder())
_THREE_FILES = {
"a.md": "# A\n\nalpha content\n",
"b.md": "# B\n\nbravo content\n",
"notes/c.md": "# C\n\ncharlie content\n",
}
def _log_lines(caplog: pytest.LogCaptureFixture, prefix: str) -> list[str]:
return [r.getMessage() for r in caplog.records if r.getMessage().startswith(prefix)]
# --- the phase-machine sequence --------------------------------------------
def test_real_run_reports_the_scan_frames_and_keeps_both_pairs(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""D3 through the REAL pipeline, twice (one changed file included).
Run 1 (a first import): the prelude reports ``phase: null`` with
``scan_*`` 0/0, then the run reports ``"scan"`` with ``scan_total`` at
the importable count and ``scan_done`` climbing through AT LEAST two
distinct positions (the frames the ``Scanning… (n/total)`` label
needs) with ``current_file`` null throughout — the scan names no file.
The terminal clears ``phase`` and KEEPS the final scan pair beside the
file pair, and on a first import both pairs are 3/3 (A1's degenerate
parity: walk == work, the only shape the phase-193 suite ever saw).
Run 2 (one file edited — the owner's prod shape at 1-of-3): the scan
denominator is STILL the importable 3, while the kept file pair is
``1/1`` and the terminal reports exactly one updated document. That
divergence — the scan counted the walk, the count counted the work —
IS the phase, and it survives on the terminal where no poll can miss
it."""
source = _local_source(tmp_path, f"{PREFIX}machine", _THREE_FILES)
_install_local_run(monkeypatch, db, tmp_path, source)
_slow_hash(monkeypatch)
_login(sync_client)
idle = sync_client.get("/api/sync/status").json()
assert idle["phase"] is None
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
frames, _late = _watch_scan(sync_client)
scan_frames = [f for f in frames if f["phase"] == "scan"]
assert scan_frames, f"the run never reported the scan phase: {frames}"
assert all(f["current_file"] is None for f in scan_frames), (
"the scan named a file — current_file stays null until the first "
"progress call flips the phase to import"
)
# ``0`` is the transition frame — the phase is set before the scan
# reports its first file, and the UI's total===0 guard renders it bare.
totals = {f["scan_total"] for f in scan_frames}
assert totals <= {0, 3} and max(totals) == 3, totals
positions = sorted({f["scan_done"] for f in scan_frames if f["scan_total"] == 3})
assert len(positions) >= 2, (
f"only scan position(s) {positions} were observable — the label needs "
"the scan's own n/total to move (the fixture scan blocks on purpose)"
)
assert positions[0] >= 1 and positions[-1] <= 3
body = _poll(sync_client, "success")
assert body["detail"]["added"] == 3
assert body["phase"] is None and body["current_file"] is None
# The keep-final-values convention, beside the file counts.
assert body["scan_total"] == 3 and body["scan_done"] == 3
assert body["files_total"] == 3 and body["files_done"] == 3
# Run 2 — one changed file of three (the owner's prod case, scaled).
(source / "b.md").write_text("# B\n\nbravo content EDITED\n", encoding="utf-8")
assert sync_client.post("/api/sync").status_code == 202
frames2, _late2 = _watch_scan(sync_client)
scan2 = [f for f in frames2 if f["phase"] == "scan"]
totals2 = {f["scan_total"] for f in scan2 if f["scan_total"]}
assert scan2 and totals2 == {3}, (
"the scan denominator is the importable count, not the prediction"
)
assert sorted({f["scan_done"] for f in scan2 if f["scan_total"] == 3})[-1] >= 2
body2 = _poll(sync_client, "success")
assert (body2["detail"]["added"], body2["detail"]["updated"]) == (0, 1)
assert body2["phase"] is None
# The divergence the phase exists for: the kept scan pair is the WALK
# it did (3 files hashed), the kept file pair is the WORK (1 file).
assert (body2["scan_done"], body2["scan_total"]) == (3, 3)
assert (body2["files_done"], body2["files_total"]) == (1, 1)
def test_scan_flips_to_import_on_the_first_progress_call(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""The sequence, frame by frame, with the flip pinned.
The in-process import runs straight through without awaiting real I/O,
so a poll cannot reliably land inside the ``"import"`` phase (that is
why the real arm above pins the terminal's kept pairs). This arm drives
the runner's OWN closures through a choreographed double — the
``test_sync_api`` gated-import idiom — and reads the status at each
stop: prelude ``null``/0/0 → ``"scan"`` at position 1 of 3 → still
``"scan"`` at position 3 of 3 → ``"import"`` with the PREDICTED
``files_total`` (1, the count that reaches ``N remaining``) and the
scan pair still showing 3/3 → the terminal, which clears ``phase`` and
keeps both pairs."""
source = _local_source(tmp_path, f"{PREFIX}flip", _THREE_FILES)
_install_local_run(monkeypatch, db, tmp_path, source)
gated = _ChoreographedImport(
ImportSummary(files=3, updated=1, unchanged=2),
scan_pairs=[(1, 3), (2, 3), (3, 3)],
progress_call=(source.name, "b.md", 1, 1),
)
monkeypatch.setattr(sync_api, "import_sources", gated)
_login(sync_client)
idle = sync_client.get("/api/sync/status").json()
assert idle["state"] == "idle"
assert idle["phase"] is None
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
try:
assert gated.first.wait(10.0), "the run never reached the scan"
s1 = sync_client.get("/api/sync/status").json()
assert s1["state"] == "running"
assert s1["phase"] == "scan"
assert (s1["scan_done"], s1["scan_total"]) == (1, 3)
assert s1["current_file"] is None
# Still the pre-import phase: no file count yet.
assert (s1["files_done"], s1["files_total"]) == (0, 0)
gated.go_on.set()
assert gated.scanned.wait(10.0), "the scan never reported its last file"
s2 = sync_client.get("/api/sync/status").json()
assert s2["phase"] == "scan"
assert (s2["scan_done"], s2["scan_total"]) == (3, 3)
gated.go_import.set()
assert gated.flipped.wait(10.0), "the progress hook never fired"
s3 = sync_client.get("/api/sync/status").json()
assert s3["phase"] == "import", (
"the first progress call must flip scan -> import: "
f"{ {k: s3[k] for k in ('phase', 'current_file', 'files_done', 'files_total')} }"
)
assert s3["current_file"] == f"{source.name}/b.md"
# The flip already carries the PREDICTED denominator — the number
# the label turns into "N remaining".
assert (s3["files_done"], s3["files_total"]) == (1, 1)
# … and the scan pair stays where the scan left it.
assert (s3["scan_done"], s3["scan_total"]) == (3, 3)
finally:
gated.go_on.set()
gated.go_import.set()
gated.release.set()
body = _poll(sync_client, "success")
assert body["phase"] is None and body["current_file"] is None
assert (body["scan_done"], body["scan_total"]) == (3, 3)
assert (body["files_done"], body["files_total"]) == (1, 1)
# --- the additive payload ----------------------------------------------------
def test_scan_keys_are_additive_and_stable_from_idle_to_terminal(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""The additive-only contract on the wire (the phase-187 discipline).
The payload's key set is EXACTLY the pre-phase-194 set plus
``scan_done`` / ``scan_total``; it does not change between idle,
mid-run and the terminal (no key appears or disappears with the
phase); and both new keys are plain ``int`` — 0 when there is nothing
to report, never ``null`` and never a plausible position."""
expected_pre = {
"state",
"started_at",
"finished_at",
"detail",
"error",
"current_file",
"files_done",
"files_total",
"current_chunk",
"chunks_total",
"images_done",
"images_total",
"phase",
"current_summary",
"summaries_done",
"summaries_total",
"rss_mb",
"rss_peak_mb",
}
source = _local_source(tmp_path, f"{PREFIX}keys", _THREE_FILES)
_install_local_run(monkeypatch, db, tmp_path, source)
_stub_probe(monkeypatch)
_login(sync_client)
idle = sync_client.get("/api/sync/status").json()
assert set(idle) == expected_pre | {"scan_done", "scan_total"}, set(idle)
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
mid = sync_client.get("/api/sync/status").json()
assert set(mid) == expected_pre | {"scan_done", "scan_total"}
assert mid["state"] == "running"
body = _poll(sync_client, "success")
assert set(body) == expected_pre | {"scan_done", "scan_total"}
for key in ("scan_done", "scan_total"):
assert isinstance(body[key], int) and not isinstance(body[key], bool), body
# Every pre-existing key keeps its type too (the additive pin's other
# half — the file pair is still ints, the phase still a string|null).
assert isinstance(body["files_done"], int) and isinstance(body["files_total"], int)
assert body["phase"] is None
# --- the sampling discipline ------------------------------------------------
def test_scan_and_import_transitions_force_the_sample_and_the_scan_hook_does_not(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 187's rule, extended to the new stage (LOCKED A4).
Every entry below is one REAL ``/proc`` read (the two helpers are only
called when the throttle lets ``_sample_rss`` through), tagged with
the phase it happened in. Collapsing consecutive same-phase reads, the
sequence is one read per transition: run start (``phase`` still null)
→ ``"scan"`` → ``"import"`` (the flip) → ``"overview"`` →
``"summaries"`` → the terminal (null again). The scan transition's
read carries ``scan_done == 0`` — it is the transition, not a file —
and there is EXACTLY one read inside the whole scan phase, so a
per-file scan read (one ``/proc`` read per file, the regression this
pins) fails here even though the fixture scanned three files."""
source = _local_source(tmp_path, f"{PREFIX}sampling", _THREE_FILES)
_install_local_run(monkeypatch, db, tmp_path, source)
_slow_hash(monkeypatch)
reads = _install_counting_memory(monkeypatch)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert reads, "the run sampled no memory at all"
phases = [r["phase"] for r in reads]
transitions = [p for i, p in enumerate(phases) if i == 0 or p != phases[i - 1]]
assert transitions == [None, "scan", "import", "overview", "summaries", None], reads
scan_reads = [r for r in reads if r["phase"] == "scan"]
assert len(scan_reads) == 1, (
f"the scan phase cost {len(scan_reads)} /proc reads for 3 files — the "
"per-file scan hook must sample UNFORCED (one read per interval)"
)
assert scan_reads[0]["scan_done"] == 0, (
"the scan-phase read is not the transition — the forced sample must "
"land when the phase is SET, before the first file is scanned"
)
flip_reads = [r for r in reads if r["phase"] == "import"]
assert flip_reads and flip_reads[0]["current_file"] is not None, (
"no read at the scan->import flip — the transition is a forced "
f"sampling point: {reads}"
)
# The payload reports what the sampler wrote, not a stale pair.
assert (body["rss_mb"], body["rss_peak_mb"]) == (100, 120), body
# --- the degenerate empty knowledge base ------------------------------------
def test_empty_kb_stays_in_scan_and_never_reaches_import(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""The D1 degenerate clause, pinned on BOTH sides.
Importer side (real): an empty source dir makes the pre-scan report
NOTHING — no ``on_scan_progress`` call, no ``progress`` call, a zero
denominator. App side (the double that fires nothing, i.e. exactly
what that run reports): the status sits in ``"scan"`` — the flip
lives in the per-file hook and no hook ever fires — with ``scan_*``
0/0 and ``current_file`` null, and the terminal clears ``phase``
while keeping the 0/0 pair (the UI's ``total === 0`` guard then
renders the bare label with no count part)."""
# Importer side: the real scan of an empty root reports nothing.
empty = _local_source(tmp_path, f"{PREFIX}empty", {})
llm = FakeEmbedder()
scan_calls: list[tuple[int, int]] = []
progress_calls: list[tuple[str, str, int, int]] = []
summary = asyncio.run(
importer.import_sources(
[empty],
llm,
progress=lambda s, r, d, t: progress_calls.append((s, r, d, t)),
on_scan_progress=lambda p, t: scan_calls.append((p, t)),
)
)
assert (scan_calls, progress_calls) == ([], [])
assert summary.files == 0
# App side: a run that reports nothing is a run that stays "scan".
_install_local_run(monkeypatch, db, tmp_path, empty)
fake_import = _FakeScanImport()
monkeypatch.setattr(sync_api, "import_sources", fake_import)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
try:
assert fake_import.started.wait(10.0), "the run never reached the import"
# Wiring: the runner passed BOTH live closures (so the empty
# phase above is the scan reporting nothing, not a missing hook).
assert fake_import.scan_hook is not None
assert fake_import.progress_hook is not None
body = sync_client.get("/api/sync/status").json()
assert body["state"] == "running"
assert body["phase"] == "scan"
assert body["current_file"] is None
assert body["scan_done"] == 0 and body["scan_total"] == 0
assert body["files_done"] == 0 and body["files_total"] == 0
finally:
fake_import.release.set()
terminal = _poll(sync_client, "success")
assert terminal["phase"] is None
assert terminal["scan_done"] == 0 and terminal["scan_total"] == 0
assert terminal["files_done"] == 0 and terminal["files_total"] == 0
# --- the cancel during the pre-scan -----------------------------------------
def test_cancel_during_the_scan_lands_cancelled_with_the_scan_counts_kept(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""The phase-134 contract at the pipeline's NEW await (the app-side
twin of the importer-side cancel pin).
The scan is blocking on purpose, so the cancel provably lands inside
it: ``POST /api/sync/cancel`` answers, the ``cancelled`` terminal
arrives well before the abandoned scan's remaining work could finish,
``phase`` clears, the position the scan had reached is KEPT beside the
(still zero) file pair, ``error`` stays null and ``detail`` empty (a
cancel is intent, not failure), and NOTHING was indexed — the walk
never started. The lifecycle line keeps its exact pre-phase prefix
(LOCKED A2's append-only rule)."""
source = _local_source(tmp_path, f"{PREFIX}cancel", _THREE_FILES)
_install_local_run(monkeypatch, db, tmp_path, source)
_slow_hash(monkeypatch, block=0.4)
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
# Wait for proof the run is INSIDE the scan: the phase, and a
# position that moved.
deadline = time.monotonic() + 20.0
live: dict = {}
while time.monotonic() < deadline:
live = sync_client.get("/api/sync/status").json()
if live["phase"] == "scan" and live["scan_done"] >= 1:
break
assert live["state"] == "running", live
time.sleep(0.01)
assert live.get("phase") == "scan" and live["scan_done"] >= 1, (
f"the cancel never landed inside the scan: {live}"
)
position_at_cancel = live["scan_done"]
t0 = time.monotonic()
r = sync_client.post("/api/sync/cancel")
assert r.status_code == 202, f"cancel not accepted: {r.status_code} {r.text}"
assert time.monotonic() - t0 < 2.0, "the cancel request itself stalled"
body = _poll(sync_client, "cancelled", timeout=3.0)
assert body["error"] is None
assert body["detail"] == {}
assert body["phase"] is None
assert body["current_file"] is None
# The scan's final position is kept (the keep-final-values
# convention), and it is the position the cancel found — the run did
# not finish the scan it was cancelled in.
assert body["scan_total"] == 3
assert body["scan_done"] >= position_at_cancel
assert body["files_done"] == 0 and body["files_total"] == 0
# Nothing was indexed: the walk never started.
assert db.scalar(
select(select(Document).where(Document.source == source.name).exists())
) is False
# The lifecycle line keeps its pre-phase-194 prefix exactly.
cancelled = _log_lines(caplog, "sync: cancelled by user state=cancelled files_done=")
assert len(cancelled) == 1, cancelled
# --- the kill switch's status shape ----------------------------------------
def test_kill_switch_off_reports_no_scan_position_but_the_same_machine(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""``BOR_SYNC_PENDING_COUNT=0`` on the STATUS side.
The importer takes the phase-193 stats pre-walk, which reports NOTHING,
so the scan pair stays 0/0 for the whole run (the page renders no
count from it) while ``files_done`` / ``files_total`` carry the WALK
position again — 3 walked, not 1 predicted. The phase machine itself is
UNCHANGED by the switch (the pre-``import_sources`` set is still
``"scan"`` and the first file still flips it: that flip is pinned in
the choreographed test above, and the switch's importer-side halves are
pinned in ``tests/unit/test_sync_pending_count.py`` and
``tests/integration/test_sync_pending_count.py``). The pins below read
the TERMINAL, which is where both pairs survive deterministically."""
source = _local_source(tmp_path, f"{PREFIX}off", _THREE_FILES)
_install_local_run(monkeypatch, db, tmp_path, source)
# The switch lives on the settings object the import reads (llm).
real_embedder = FakeEmbedder()
real_embedder.settings = Settings(_env_file=None, sync_pending_count=0) # pyright: ignore[reportCallIssue]
monkeypatch.setattr(sync_api, "LLMClient", lambda: real_embedder)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["detail"]["added"] == 3
# No scan, so no position — kept at 0/0, the value the UI renders as
# "no count part" rather than a stale or invented one.
assert body["scan_done"] == 0 and body["scan_total"] == 0
# The WALK count is back (phase-193 semantics, byte-for-byte).
assert (body["files_done"], body["files_total"]) == (3, 3)
assert body["phase"] is None
+30
View File
@@ -1027,3 +1027,33 @@ def test_sync_prune_batch_rejects_negative(
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "-1")
with pytest.raises(ValidationError, match="sync_prune_batch"):
_settings()
# --- Sync pending count (phase 194) ---
def test_sync_pending_count_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 194 (D2): the pre-scan ships ON (``1``) — the owner's "1216
remaining" report means the walk-position count was wrong by default, so
the fix is the default and the rollback is the opt-in. ``0`` is accepted
and means the exact phase-193 behaviour: count every importable file of
the walk and never hash a file before the loop reached it."""
monkeypatch.delenv("BOR_SYNC_PENDING_COUNT", raising=False)
assert _settings().sync_pending_count == 1
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "0")
assert _settings().sync_pending_count == 0
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "1")
assert _settings().sync_pending_count == 1
def test_sync_pending_count_rejects_negative(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A negative switch is a typo — no branch of the importer can mean it,
so startup refuses naming the field (the ``agent_max_rounds`` pattern,
phase 194). The value is a boolean knob, so ONLY ``0`` is the rollback."""
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "-1")
with pytest.raises(ValidationError, match="sync_pending_count"):
_settings()
+30 -7
View File
@@ -1146,8 +1146,16 @@ def test_no_progress_means_no_prewalk(
) -> None:
"""``progress=None`` callers pay no extra pass: the real walker is hit
exactly once per source root (one pass — as before phase 64), proven
with a counting sentinel; with the hook it is hit twice (pre-walk for
``total`` + the processing pass)."""
with a counting sentinel; with the hook it is hit twice (the phase-194
pre-scan for the pending set — the phase-64 stats pre-walk when
``BOR_SYNC_PENDING_COUNT=0`` — plus the processing pass).
Phase 194 (task 01) moved the hook's pair, and this fixture is the
second run over an ALREADY-INDEXED file: nothing is pending, so the
pair is ``(0, 0)`` — the denominator is the predicted change set and an
unchanged KB predicts nothing (the UI then renders no count part). The
walk COUNT is what this test owns and it is untouched: one pass without
the hook, two with it."""
root = tmp_path / "nowalk"
root.mkdir()
(root / "a.md").write_text("# A\n\na\n")
@@ -1186,8 +1194,12 @@ def test_no_progress_means_no_prewalk(
)
)
assert summary2.files == 1
assert walk_calls == 2 # pre-walk (total) + processing pass
assert events == [(root.name, "a.md", 1, 1)]
assert walk_calls == 2 # the count pass (pending set / pre-walk) + walk
# phase 194: a.md is unchanged, so the predicted change set is empty
# — ``total`` 0 and the position never advances. With the kill switch
# at ``0`` this same run reports the phase-193 walk position (1 of 1);
# pinned in tests/unit/test_sync_pending_count.py.
assert events == [(root.name, "a.md", 0, 0)]
finally:
_cleanup_source(db, root.name)
@@ -1195,7 +1207,17 @@ def test_no_progress_means_no_prewalk(
def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> None:
"""The hook fires *before* ``_index_file``: a file whose embedding
fails (and one that re-imports as unchanged) is still reported as the
current file — the sequence covers every importable file."""
current file — the sequence covers every importable file.
Phase 194 (task 01) retargeted the pair to the PENDING position (the
owner's "1216 remaining" fix), which reads exactly right here. The
first run is fresh: both files are predicted, ``1 of 2`` then ``2 of 2``
— identical to the pre-phase numbers (A1's first-import parity). The
second run is the incremental case: ``bad.md`` rolled back with its
embed error so its row is still missing (predicted again, the ONLY
work), ``good.md`` is unchanged so it holds the position instead of
advancing it — ``1 of 1``, ``1 of 1``: every file is still reported as
the current file, and the count is the work."""
root = tmp_path / "progress-mixed"
root.mkdir()
(root / "bad.md").write_text("# Bad\n\npoison content the endpoint refuses\n")
@@ -1214,7 +1236,8 @@ def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> N
assert events == expected
assert first.errors == 1 and first.added == 1
# Re-run: good.md is now unchanged, bad.md is retried and fails
# again — both still count in the sequence.
# again — both still report as the current file, but only bad.md is
# pending, so the denominator is 1 and good.md holds the position.
events.clear()
second = asyncio.run(
import_sources(
@@ -1222,7 +1245,7 @@ def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> N
progress=lambda s, r, d, t: events.append((s, r, d, t)),
)
)
assert events == expected
assert events == [(root.name, "bad.md", 1, 1), (root.name, "good.md", 1, 1)]
assert second.errors == 1 and second.unchanged == 1
finally:
_cleanup_source(db, root.name)
+23
View File
@@ -253,6 +253,29 @@ def test_sync_session_bound_field_types() -> None:
assert Settings.model_fields["sync_prune_batch"].default == 500
def test_sync_pending_count_row_is_a_plain_live_overridable_row() -> None:
"""Phase 194 (task 01, D2): the pre-scan switch sits in ``Import &
sources`` beside the other two sync-run knobs with the ``Sync pending
count`` label and is plain — not ``secret`` (an int), not
``restart_required`` (the importer reads ``llm.settings`` once per run, so
a saved override applies to the NEXT sync), overridable (the overlay is
generic over ``int`` fields). ``1`` is the shipped default — the pending
count is the FIX the owner asked for, so it ships on and the phase-193
walk position is the ``0`` rollback; a negative value fails startup, not
the panel."""
meta = CATALOG["sync_pending_count"]
assert meta.group == "Import & sources"
assert meta.label == "Sync pending count"
assert meta.secret is False
assert meta.restart_required is False
assert meta.overridable is True
assert "sync_pending_count" not in SECRETS
assert "sync_pending_count" not in RESTART_REQUIRED
assert "sync_pending_count" not in NON_OVERRIDABLE
assert Settings.model_fields["sync_pending_count"].annotation is int
assert Settings.model_fields["sync_pending_count"].default == 1
def test_source_name_hit_bonus_row_is_a_plain_live_overridable_row() -> None:
"""Phase 191 (task 04, D3): the second selection-bonus row sits in
``Retrieval & honesty`` (right beside ``name_hit_bonus`` — the two
+62 -20
View File
@@ -1419,6 +1419,10 @@ class _GatedImport:
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
# Phase 194 (task 02): the pre-scan position hook the runner
# passes (recorded for the wiring pin — this fake fires nothing,
# so the status scan fields stay 0/0).
self.scan_hook: Callable[[int, int], None] | None = None
async def __call__(
self,
@@ -1431,11 +1435,13 @@ class _GatedImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
self.prune_flags.append(prune)
self.scan_hook = on_scan_progress
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
@@ -1597,6 +1603,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
"current_file": None,
"files_done": 0,
"files_total": 0,
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
# (the dataclass defaults; a run that scanned nothing keeps them
# 0/0 rather than null).
"scan_done": 0,
"scan_total": 0,
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
# (the dataclass defaults, no special-casing).
"current_chunk": 0,
@@ -2131,6 +2142,7 @@ class _TwoFileImport:
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -2804,6 +2816,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -3170,7 +3183,8 @@ def test_sources_js_section_header_documents_the_cancel_contract() -> None:
#: The status payload's key set BEFORE phase 187 — the additive pin's
#: left-hand side: the payload is exactly this set plus
#: :data:`_MEMORY_KEYS`, nothing else, forever.
#: :data:`_MEMORY_KEYS` plus :data:`_SCAN_KEYS`, nothing else, forever
#: (a future field joins the chain, it never edits these three).
_PRE_187_STATUS_KEYS = frozenset(
{
"state",
@@ -3195,17 +3209,23 @@ _PRE_187_STATUS_KEYS = frozenset(
#: Phase 187 (task 03): the two additive memory keys.
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
#: link of the same additive chain.
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
def test_status_key_set_is_the_old_set_plus_the_memory_pair(
fresh_sync_status: None,
) -> None:
"""The key-set pin (D4: "Additive only — every pre-existing key
keeps its name, type, and meaning"): the payload is EXACTLY the
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb}`` — a fifth sub-position
or a renamed field fails here even when the value assertions pass."""
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb} ∪ {scan_done,
scan_total}`` — a fifth sub-position or a renamed field fails here
even when the value assertions pass."""
keys = set(sync_api.sync_status())
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, keys
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, keys
assert _MEMORY_KEYS.isdisjoint(_PRE_187_STATUS_KEYS)
assert _SCAN_KEYS.isdisjoint(_PRE_187_STATUS_KEYS | _MEMORY_KEYS)
class _FakeClock:
@@ -3377,9 +3397,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
caplog: pytest.LogCaptureFixture,
) -> None:
"""The sampling POINTS, walked through the real runner: run start →
``import`` → ``overview`` → ``summaries`` → the success terminal, one
forced read each, and the per-file progress hook's throttled call
adds NOTHING inside the window the run start just armed (LOCKED A4).
``scan`` → ``import`` (the phase-194 flip) → ``overview`` →
``summaries`` → the success terminal, one forced read each, and the
per-file progress hook's throttled call adds NOTHING inside the window
the previous transition just armed (LOCKED A4 — and the same rule the
pre-scan's own per-file hook follows, pinned through the app in
``tests/integration/test_sync_scan_phase.py``).
The last pair is the number the incident was about: the terminal
KEEPS it (the phase-64 keep-final-values convention) and the
@@ -3388,10 +3411,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
log an operator could be argued with."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
# run start, import, overview, summaries, done (the hook is throttled).
# run start, scan, import (the flip), overview, summaries, done — the
# two phase-194 transitions join the list; the per-file hooks stay
# throttled and add no read.
memory = _StubMemory(
monkeypatch,
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 4_321)],
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 24), (15, 4_321)],
)
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
@@ -3412,14 +3437,23 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
assert (s["rss_mb"], s["rss_peak_mb"]) == (10, 20)
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the transition forced the second read, and
# the hook's own call (this fake fires one file) is inside
# the window — still 2 reads, not 3 (A4 in numbers).
# Import phase: the "scan" set forced the second read and the
# first progress call's FLIP forced the third, while the
# hook's own unforced call (this fake fires one file) is
# inside the window — 3 reads, not 4 (A4 in numbers).
s = sync_api.sync_status()
assert s["phase"] == "import"
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
assert (s["rss_mb"], s["rss_peak_mb"]) == (11, 21)
assert memory.reads == 2, "the per-file hook never bypasses the guard"
assert (s["rss_mb"], s["rss_peak_mb"]) == (12, 22)
assert memory.reads == 3, (
"the scan/import transitions must each force one read and "
f"the per-file hook must never bypass the guard: {memory.reads}"
)
# The scan closure reached the runner wired (the fake fires
# nothing, so the position stays 0/0 — the sequence itself is
# pinned through the app in test_sync_scan_phase.py).
assert fake_import.scan_hook is not None
assert s["scan_done"] == 0 and s["scan_total"] == 0
fake_import.release.set()
finally:
clone.release.set()
@@ -3431,7 +3465,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
# and it survives the run (the page shows it after the settle).
s = sync_api.sync_status()
assert s["state"] == "success"
assert (s["rss_mb"], s["rss_peak_mb"]) == (14, 4_321)
assert (s["rss_mb"], s["rss_peak_mb"]) == (15, 4_321)
assert s["files_done"] == 1 and s["files_total"] == 3
done = [
@@ -3440,7 +3474,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
if r.getMessage().startswith("sync: done detail=")
]
assert len(done) == 1, done
assert done[0].endswith("rss_mb=14 rss_peak_mb=4321"), done[0]
assert done[0].endswith("rss_mb=15 rss_peak_mb=4321"), done[0]
def test_per_file_hook_samples_once_the_window_closes(
@@ -3452,10 +3486,15 @@ def test_per_file_hook_samples_once_the_window_closes(
fires once per file, and once the 5 s window has closed the next
file's call re-reads — so a long walk reports the memory climbing
(PLAN §7.4's never-stale contract) at ~600 reads per 27k files
instead of 27k."""
instead of 27k (phase 194 extended the same rule to the pre-scan's
per-file hook)."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
memory = _StubMemory(monkeypatch, [(100, 200), (110, 210), (140, 260)])
# run start, the "scan" set, the scan->import flip, then the read the
# test forces after moving the clock past the window.
memory = _StubMemory(
monkeypatch, [(100, 200), (110, 210), (120, 220), (140, 260)]
)
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
ImportSummary(files=1, added=1),
@@ -3474,8 +3513,11 @@ def test_per_file_hook_samples_once_the_window_closes(
assert fake_import.started.wait(5.0), "the run never reached the import"
s = sync_api.sync_status()
assert s["phase"] == "import"
assert (s["rss_mb"], s["rss_peak_mb"]) == (110, 210) # the transition
assert memory.reads == 2, "the hook's call inside the window read nothing"
assert (s["rss_mb"], s["rss_peak_mb"]) == (120, 220) # the flip
assert memory.reads == 3, (
"start + scan + flip force three reads and the hook's own call "
f"inside the window adds none: {memory.reads}"
)
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S + 1.0
sync_api._sample_rss() # the NEXT file's hook call
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (140, 260)
+700
View File
@@ -0,0 +1,700 @@
"""Phase 194, task 01 — the importer's pre-scan + the pending-count switch.
The owner report that opened the phase (2026-10-19): *"I just synced the prod
brain of reese and it said '1216 remaining' when it only had like 10 files to
sync."* The arithmetic of the phase-193 label was fine; its INPUTS were not —
``total`` was every importable file (prod: ~10,000) and ``done`` was the
position in that walk, so the label counted the WALK instead of the WORK.
Task 01 moves the pair to the PREDICTED change set: ``import_sources``
pre-scans (one streamed sha256 + one scalar ``documents.file_hash`` read per
importable file, off the event loop) and reports ``done``/``total`` as the
position in the pending set, with ``BOR_SYNC_PENDING_COUNT`` as the kill
switch (``0`` = the exact phase-193 walk position, negative = refuses
startup).
Pinned here (the behaviour under a real run — the pending-set parity, the
zero-hydration pin, the off-loop span and the cancel — lives in
``tests/integration/test_sync_pending_count.py``):
* the GATE: no ``progress`` caller means no scan at all (the callback never
fires and the scalar statement is never built — the phase-64 skip rule,
extended), and the switch at ``0`` on a progress caller runs the phase-64
stats pre-walk instead (a source marker pin: that old code IS the ``0``
branch, and a behaviour pin on the pair it reports);
* the switch is read ONCE per run (the phase-184 read-once discipline), on
the worker-thread side of ``asyncio.to_thread`` (the phase-179 rule);
* the scalar statement's projection is EXACTLY ``documents.file_hash`` — one
scalar column, no entity, no ``defer``/``load_only`` row (the phase-185
column-list style); :func:`_doc_row_stmt` stays the importer's only ROW
load;
* the pending PREDICTATE: fresh (no row), changed (digest mismatch) and a
pre-phase ``NULL`` row are pending, a hash match is not, a file the ignore
rule hides is never even walked, and a file that cannot be read is
predicted changed rather than fatal;
* ``on_scan_progress``: once per file, the 1-based position against the
combined importable total, and NOTHING for an empty KB (the phase-64
``total === 0`` contract);
* ``done`` as the pending position: a 3-file KB with 2 predicted reports
``1``, ``1``, ``2`` against ``total = 2`` — the file that is not predicted
holds the position while the walk moves past it.
"""
from __future__ import annotations
import ast
import asyncio
import inspect
import re
import threading
import uuid
from pathlib import Path
from typing import Any
import pytest
from sqlalchemy import text
from sqlalchemy.dialects import postgresql
from sqlalchemy.orm import Session
import app.rag.importer as importer
from app.config import Settings
from app.models import Document
from app.rag.importer import (
_pending_hash_stmt,
_prescan_pending,
import_sources,
)
from tests.fakes import FakeEmbedder
# --------------------------------------------------------------------------
# Helpers
# --------------------------------------------------------------------------
def _code_only(fn: Any) -> str:
"""A function's source WITHOUT its docstring.
The marker pins below count occurrences of names in the code; the
docstrings argue about those same names by name (the house convention), so
counting the raw source would pin prose. The docstring is located through
the AST and sliced out by its own end line.
"""
src = inspect.getsource(fn)
lines = src.splitlines()
node = ast.parse(src).body[0]
# `import_sources` is an ASYNC def, so it parses to AsyncFunctionDef —
# narrowing to FunctionDef alone would silently skip the docstring strip
# and pin prose.
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
return src
first = node.body[0]
if (
isinstance(first, ast.Expr)
and isinstance(first.value, ast.Constant)
and isinstance(first.value.value, str)
):
return "\n".join(lines[first.end_lineno or 1 :])
return src
def _settings(**overrides: Any) -> Settings:
"""Fresh settings, never the developer's ``.env`` (the house pattern)."""
return Settings(_env_file=None, **overrides) # pyright: ignore[reportCallIssue]
def _root(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
"""One source root with *files* (relative path → content) under it."""
root = tmp_path / name
for rel, body in files.items():
target = root / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
return root
def _prescan(
*sources: Path,
session: Session,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
on_scan_progress: Any = None,
image_extensions: frozenset[str] = frozenset(),
) -> tuple[int, dict[tuple[str, str], int]]:
""":func:`_prescan_pending` with the shipped import extensions and an
explicit session, so the predicate is testable without a full run."""
return _prescan_pending(
list(sources),
_settings().import_extension_set,
image_extensions,
ignore_by_root,
include_hidden_by_root,
on_scan_progress,
session,
)
def _seed_row(
db: Session, source: str, rel: str, *, file_hash: str | None
) -> None:
"""One ``documents`` row with a KNOWN ``file_hash`` (the pre-scan reads
that one column, so this is the whole fixture: raw SQL, no importer, no
chunks)."""
db.execute(
text(
"INSERT INTO documents (id, source, path, full_path, title, content, "
"content_hash, file_hash, indexed_at, created_at, created_at_manual, "
"summary, is_image) VALUES (:id, :source, :path, :full_path, :title, "
":content, :content_hash, :file_hash, now(), now(), false, :summary, "
"false)"
),
{
"id": uuid.uuid4(),
"source": source,
"path": rel,
"full_path": f"/tmp/{source}/{rel}",
"title": rel,
"content": "stored markdown",
"content_hash": "content-digest",
"file_hash": file_hash,
"summary": "stored summary",
},
)
db.commit()
@pytest.fixture(autouse=True)
def _clean_probe_rows(db: Session) -> Any:
"""The probe's own source-name prefix, removed around every test (the
tables are global state — the ``test_importer.py`` hygiene pattern)."""
_drop(db)
yield
_drop(db)
def _drop(db: Session) -> None:
db.execute(
text("DELETE FROM documents WHERE source LIKE 'phase194-%'"),
)
db.commit()
db.expunge_all()
# --------------------------------------------------------------------------
# The gate: who pays for the scan
# --------------------------------------------------------------------------
def test_no_progress_caller_runs_no_scan_at_all(
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE skip rule (phase 64), extended: ``progress=None`` → no pre-scan.
The CLI (``scripts/import_docs.py``) and the upload job pass no
``progress``, so they must not pay one extra read + hash + SELECT per
importable file. A recording stand-in for the span proves the gate
itself: the CLI-shaped call imports the file and never reaches the
scan, and an ``on_scan_progress`` hook handed to a ``progress=None``
caller fires never (the scan exists only to serve the running counts).
"""
calls: list[tuple[Any, ...]] = []
scan_events: list[tuple[int, int]] = []
def _spy(*args: Any, **kwargs: Any) -> tuple[int, dict[tuple[str, str], int]]:
calls.append(args)
return 0, {}
monkeypatch.setattr(importer, "_prescan_pending", _spy)
root = _root(tmp_path, "phase194-gate-cli", {"a.md": "# A\n\na\n"})
try:
summary = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
assert summary.files == 1 and summary.added == 1
assert calls == [], "a progress=None caller ran the pre-scan"
# The scan hook alone does not open the gate either.
summary2 = asyncio.run(
import_sources(
[root],
FakeEmbedder(),
session=db,
on_scan_progress=lambda done, total: scan_events.append((done, total)),
)
)
assert summary2.files == 1
assert calls == []
assert scan_events == []
finally:
db.execute(
text("DELETE FROM documents WHERE source = :s"), {"s": root.name}
)
db.commit()
def test_switch_zero_runs_the_phase_193_stats_prewalk(
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The rollback branch, from both sides: the marker (the old stats
pre-walk code IS the ``0`` branch) and the reported pair.
With the switch at ``0`` a progress caller never reaches the scan span,
and the hook reports the phase-193 walk position byte-for-byte — every
importable file in ``total``, the 1-based index across all sources in
``done`` — on a run where nothing is pending. That last clause is the
point of the arm: the SAME unchanged fixture reports ``(0, 0)`` with the
switch on and ``(1, 1)``, ``(2, 2)`` with it off.
"""
calls: list[tuple[Any, ...]] = []
def _spy(*args: Any, **kwargs: Any) -> tuple[int, dict[tuple[str, str], int]]:
calls.append(args)
return 0, {}
monkeypatch.setattr(importer, "_prescan_pending", _spy)
root = _root(
tmp_path,
"phase194-gate-zero",
{"a.md": "# A\n\nalpha\n", "b.md": "# B\n\nbravo\n"},
)
llm = FakeEmbedder()
llm.settings = _settings(sync_pending_count=0)
events: list[tuple[str, str, int, int]] = []
try:
# Seed the KB (a walk-position run of its own), so nothing pending.
asyncio.run(import_sources([root], llm, session=db))
assert calls == []
events.clear()
summary = asyncio.run(
import_sources(
[root],
llm,
session=db,
progress=lambda s, r, d, t: events.append((s, r, d, t)),
)
)
assert calls == [], "the switch was 0 but the pre-scan ran"
assert (summary.files, summary.unchanged) == (2, 2)
assert events == [
(root.name, "a.md", 1, 2),
(root.name, "b.md", 2, 2),
], "the 0 branch must report the phase-193 walk position exactly"
finally:
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
db.commit()
# The marker: the stats-only pre-walk still EXISTS in the importer, and
# it lives in the ``0`` branch (a third walk would mean the replacement
# was not a replacement).
src = _code_only(importer.import_sources)
assert "BOR_SYNC_PENDING_COUNT=0" in src
assert src.count("iter_importable_files(") == 2, (
"the importer walks the sources for the count more than once besides "
"the pre-scan — the pre-scan REPLACES the stats pre-walk (phase 194 "
"D1: no third walk)"
)
def test_switch_is_read_once_before_the_loop_and_off_the_loop() -> None:
"""The phase-184 read-once discipline + the phase-179 off-loop rule.
``llm.settings.sync_pending_count`` is read exactly ONCE, before the
source loop, so a settings overlay saved mid-run cannot flip the meaning
of ``done``/``total`` between two files of the same walk; and the span
is entered through ``asyncio.to_thread``, so a 10,000-file hash+SELECT
pass never parks the single uvicorn loop.
"""
src = _code_only(importer.import_sources)
assert src.count("llm.settings.sync_pending_count") == 1, (
"sync_pending_count is read more than once per run — a mid-run "
"settings save would change the pair's meaning between two files "
"(phase 184's read-once discipline)"
)
read_at = src.index("llm.settings.sync_pending_count")
loop_at = src.index("for root in sources:")
assert read_at < loop_at, "the switch must be read BEFORE the source loop"
assert "await asyncio.to_thread(\n _prescan_pending" in src, (
"the pre-scan is no longer entered through asyncio.to_thread — a long "
"scan would park the event loop (the phase-179 rule, "
"test_sync_git_freeze.py)"
)
# The scan itself is a plain sync function (it runs in the worker).
assert not inspect.iscoroutinefunction(importer._prescan_pending)
# --------------------------------------------------------------------------
# The scalar statement's shape (the phase-185 column-list style)
# --------------------------------------------------------------------------
def test_scalar_statement_projects_exactly_file_hash() -> None:
"""The pre-scan reads ONE column, so it loads no entity and keeps nothing.
The compiled projection names ``documents.file_hash`` and no other
``documents`` column; the statement resolves to exactly one scalar
column (``column_descriptions``), so ``session.scalar`` returns a string
and no ORM object exists to land in an identity map. ``_doc_row_stmt``
stays the importer's only ``(source, path)`` ROW load (phase 185).
"""
compiled = str(
_pending_hash_stmt("the-source", "notes/a.md").compile(
dialect=postgresql.dialect()
)
)
projection = re.split(r"\bFROM\b", compiled, maxsplit=1)[0]
columns = set(re.findall(r"documents\.([a-z_]+)", projection))
assert columns == {"file_hash"}, (
f"the pre-scan statement projects {sorted(columns)} — it must project "
"documents.file_hash and nothing else (one light metadata read per "
"file, no entity hydration)"
)
# The predicate is the unique (source, path) constraint, like the fast path.
predicate = re.split(r"\bFROM\b", compiled, maxsplit=1)[1]
assert "documents.source" in predicate and "documents.path" in predicate
stmt = _pending_hash_stmt("s", "p")
assert len(stmt.column_descriptions) == 1
assert stmt.column_descriptions[0]["type"] == Document.file_hash.type
assert "Document" not in {
str(desc.get("entity")) for desc in stmt.column_descriptions
}, "the pre-scan statement selects an ENTITY — it must select the column"
# The span is the only consumer, and the fast path's row load is intact.
span_src = inspect.getsource(importer._prescan_pending)
assert "_pending_hash_stmt(" in span_src
assert "select(Document)" not in span_src
assert "defer(" not in span_src and "load_only(" not in span_src, (
"the pre-scan picked up a loader option — there is no ROW to defer: "
"it selects one column (phase 185 keeps _doc_row_stmt as the only "
"row load)"
)
# --------------------------------------------------------------------------
# The pending predicate
# --------------------------------------------------------------------------
def test_pending_predicate_fresh_unchanged_changed_and_null_row(
db: Session, tmp_path: Path
) -> None:
"""The four stored-hash states, and the position the set assigns.
* no row at all (a first sync) → pending;
* a row whose ``file_hash`` equals the file's digest → NOT pending;
* a row whose ``file_hash`` differs → pending;
* a pre-phase-147 row (``file_hash`` NULL) → pending — the same call the
fast path makes, so the prediction and the loop agree and the loop
stamps the hash on the way through.
The positions are the 1-based walk order of the PREDICTED files only,
and the scan total is every importable file (3 here, 3 predicted).
"""
root = _root(
tmp_path,
"phase194-predicate",
{
"fresh.md": "# Fresh\n\nnever indexed\n",
"same.md": "# Same\n\nindexed as-is\n",
"moved.md": "# Moved\n\nindexed differently now\n",
},
)
from app.rag.importer import _file_hash
_seed_row(db, root.name, "same.md", file_hash=_file_hash(root / "same.md"))
_seed_row(db, root.name, "moved.md", file_hash="0" * 64)
db.expunge_all()
scan_total, pending = _prescan(root, session=db)
assert scan_total == 3
assert pending == {
(root.name, "fresh.md"): 1,
(root.name, "moved.md"): 2,
}, (
"the pending set is wrong — no row and a NULL row are pending, a "
"digest match is not, a mismatch is (the fast path's own predicate)"
)
# The scan reads metadata only: nothing was hydrated into the session.
assert len(db.identity_map) == 0
def test_pending_predicate_ignores_what_the_walk_ignores(
db: Session, tmp_path: Path
) -> None:
"""A file the ignore rule hides is never walked, so it is never pending.
The scan uses the EXACT per-root ``ignore`` / ``include_hidden`` rules of
the processing loop: a hidden dot-directory is out by default and in when
the root's flag admits it, and an ignored prefix is out even with the flag
on. A predicted file the loop can never see would be a count that never
reaches zero, so this is the parity edge, not a nicety.
"""
root = _root(
tmp_path,
"phase194-ignore",
{
"keep.md": "# Keep\n\nbody\n",
"skipme/hidden-by-rule.md": "# Skip\n\nbody\n",
".dot/secret.md": "# Dot\n\nbody\n",
},
)
ignore = {str(root): ["skipme"]}
scan_total, pending = _prescan(root, session=db, ignore_by_root=ignore)
assert scan_total == 1
assert list(pending) == [(root.name, "keep.md")]
# With the hidden-folders flag ON the dot-dir file joins the walk (and
# the scan total with it) — still no pending entry for the ignored one.
scan_total, pending = _prescan(
root,
session=db,
ignore_by_root=ignore,
include_hidden_by_root={str(root): True},
)
assert scan_total == 2
assert sorted(pending) == [
(root.name, ".dot/secret.md"),
(root.name, "keep.md"),
]
def test_unreadable_file_is_predicted_changed_and_the_scan_survives(
db: Session,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""A file the digest cannot reach is a PREDICTION of change, not a failure.
The window is the phase's own: the scan walks the tree, then hashes it
seconds later, and a live source can lose a file in between. Failing the
whole sync for that would be the worse bug — the pre-scan only informs a
label — so the file lands in the pending set (the loop's own walk decides
whether it still exists at all) and one warning names it.
"""
root = _root(
tmp_path,
"phase194-unreadable",
{"a.md": "# A\n\na\n", "b.md": "# B\n\nb\n"},
)
real_hash = importer._file_hash
def flaky(path: Path) -> str:
if path.name == "b.md":
raise OSError(2, "No such file or directory", str(path))
return real_hash(path)
monkeypatch.setattr(importer, "_file_hash", flaky)
with caplog.at_level("WARNING", logger="app.importer"):
scan_total, pending = _prescan(root, session=db)
assert scan_total == 2
assert sorted(pending) == [
(root.name, "a.md"),
(root.name, "b.md"),
]
lines = [r.getMessage() for r in caplog.records if r.name == "app.importer"]
assert any(
line.startswith("import: pre-scan unreadable")
and f"source={root.name}" in line
and "path=b.md" in line
for line in lines
), f"the skipped file was not named in a warning: {lines}"
# --------------------------------------------------------------------------
# on_scan_progress + the empty KB
# --------------------------------------------------------------------------
def test_scan_progress_reports_every_file_across_roots(
db: Session, tmp_path: Path
) -> None:
"""The scan pair is ``(1-based position, combined importable total)``.
The denominator is the number the phase-64 stats pre-walk used to
compute (every importable file, across every root, in *sources* order),
because the label is about the SCAN — how far the prediction has got —
and the events cover every file, changed or not.
"""
root_a = _root(
tmp_path, "phase194-scan-a", {"a1.md": "# A1\n\na\n", "a2.md": "# A2\n\nb\n"}
)
root_b = _root(tmp_path, "phase194-scan-b", {"sub/b1.md": "# B1\n\nc\n"})
events: list[tuple[int, int]] = []
scan_total, pending = _prescan(
root_a, root_b, session=db, on_scan_progress=lambda p, t: events.append((p, t))
)
assert scan_total == 3
assert events == [(1, 3), (2, 3), (3, 3)]
assert len(pending) == 3
def test_scan_progress_reports_nothing_for_an_empty_kb(
db: Session, tmp_path: Path
) -> None:
"""The degenerate KB: scan total 0, no callback, empty pending set.
The phase-64 ``total === 0`` contract, extended to the scan: with nothing
importable there is no position to report, so the UI's scan label renders
bare (``Scanning…``, no count part) instead of ``0/0``.
"""
empty = _root(tmp_path, "phase194-empty", {})
missing = tmp_path / "phase194-not-a-dir"
events: list[tuple[int, int]] = []
scan_total, pending = _prescan(
empty, missing, session=db, on_scan_progress=lambda p, t: events.append((p, t))
)
assert scan_total == 0
assert pending == {}
assert events == []
def test_scan_hook_is_called_from_the_worker_thread(
tmp_path: Path, db: Session
) -> None:
"""The documented threading contract of ``on_scan_progress``.
The span runs in an ``asyncio.to_thread`` worker, so the hook fires on
THAT thread — which is only safe because the house hook rule is "assign a
field", a single GIL-atomic store. The pin records the thread identity
from inside the callback and from the awaiting coroutine.
"""
root = _root(tmp_path, "phase194-thread", {"a.md": "# A\n\na\n"})
seen: dict[str, int] = {}
def hook(position: int, total: int) -> None: # noqa: ARG001
seen["scan"] = threading.get_ident()
async def main() -> None:
seen["loop"] = threading.get_ident()
await asyncio.to_thread(
importer._prescan_pending,
[root],
_settings().import_extension_set,
frozenset(),
None,
None,
hook,
)
asyncio.run(main())
assert seen["scan"] != seen["loop"], (
"the pre-scan hook ran on the event-loop thread — the span must live "
"in a worker (phase 179)"
)
# --------------------------------------------------------------------------
# done = the position in the pending set
# --------------------------------------------------------------------------
def test_done_is_the_pending_position_through_import_sources(
db: Session, tmp_path: Path
) -> None:
"""3 importable files, 2 of them pending: the pair is ``1/2``, ``1/2``,
``2/2``.
``stable.md`` is indexed and unchanged, so the scan does not predict it
and the walk moving past it does NOT advance the count — the label holds
its position (this is the shape the owner's prod case takes: the count
stops climbing across a run of unchanged files). The two changed files
advance it in walk order, and the denominator is the pending set's size,
never the 3-file walk.
"""
root = _root(
tmp_path,
"phase194-position",
{
"a.md": "# A\n\nalpha\n",
"stable.md": "# Stable\n\nunchanged\n",
"z.md": "# Z\n\nzulu\n",
},
)
llm = FakeEmbedder()
events: list[tuple[str, str, int, int]] = []
try:
# Index everything, then change two files.
asyncio.run(import_sources([root], llm, session=db))
(root / "a.md").write_text("# A\n\nalpha edited\n", encoding="utf-8")
(root / "z.md").write_text("# Z\n\nzulu edited\n", encoding="utf-8")
events.clear()
summary = asyncio.run(
import_sources(
[root],
llm,
session=db,
progress=lambda s, r, d, t: events.append((s, r, d, t)),
)
)
assert (summary.files, summary.updated, summary.unchanged) == (3, 2, 1)
assert events == [
(root.name, "a.md", 1, 2),
(root.name, "stable.md", 1, 2), # not predicted: the position holds
(root.name, "z.md", 2, 2),
]
# The parity the UI depends on: the denominator IS the work.
assert summary.updated + summary.added == events[-1][3]
finally:
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
db.commit()
def test_empty_kb_reports_no_count_part_through_import_sources(
db: Session, tmp_path: Path
) -> None:
"""An empty source with the hook on: no file, no pair, ``files_total`` 0.
The import is a no-op and the hook never fires, so the status keeps the
zero denominator the UI's ``total === 0`` guard hides the count part for
(the phase-64 contract, arriving through the pre-scan path rather than
the stats pre-walk).
"""
root = _root(tmp_path, "phase194-empty-run", {})
events: list[tuple[str, str, int, int]] = []
scan_events: list[tuple[int, int]] = []
summary = asyncio.run(
import_sources(
[root],
FakeEmbedder(),
session=db,
progress=lambda s, r, d, t: events.append((s, r, d, t)),
on_scan_progress=lambda p, t: scan_events.append((p, t)),
)
)
assert (summary.files, summary.added, summary.errors) == (0, 0, 0)
assert events == []
assert scan_events == []
def test_cli_caller_is_untouched_by_the_pending_semantics(
db: Session, tmp_path: Path
) -> None:
"""The CLI-shaped run (no hook at all) on an unchanged KB is the pre-phase
run: one file processed, nothing counted, one walk.
The byte-identity of the non-UI caller is the phase's cheapest promise —
``scripts/import_docs.py`` passes no ``progress``, so it neither pays the
scan nor sees its numbers.
"""
root = _root(tmp_path, "phase194-cli", {"a.md": "# A\n\na\n"})
llm = FakeEmbedder()
try:
first = asyncio.run(import_sources([root], llm, session=db))
assert (first.files, first.added) == (1, 1)
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.files, second.added, second.unchanged) == (1, 0, 1)
finally:
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
db.commit()
+544
View File
@@ -0,0 +1,544 @@
"""Phase 194, task 03 — the Sources page renders the scan as a scan.
The owner report the whole phase exists for (2026-10-19): *"1216
remaining"* on a ~10,000-file KB that had ~10 files to sync. Task 01 moved
the server's ``done``/``total`` to the pre-scan's predicted change set and
task 02 exposed the scan as a phase (``phase: "scan"`` + ``scan_done`` /
``scan_total``). This module pins the UI half (D4, owner-confirmed A2):
``fmtSyncPhaseLabel`` gains ONE explicit ``"scan"`` branch that says
``Scanning…`` and puts the scan's own ``(n/total)`` in the SAME count slot
the import phase's ``N remaining`` uses — so the walk position surviving as
an ``n/m`` is honest, because the word in front of it is "Scanning" and
never "Syncing" (TODO L6's objection was the sync count wearing the
total's shape).
Pinned here (source markers, plus one node run of the REAL label path):
* the branch — ``status.phase === "scan"`` is in the decision tree BEFORE
``overview`` / ``summaries`` / the fall-through, its main text is exactly
``Scanning…``, and it never reads ``current_file`` (null in the phase),
the file pair, the chunk pair, the image pair or the summaries pair;
* the shape — the count part is ``(${status.scan_done}/${status.scan_total})``
landing on the ``count`` slot (``setSyncLabel``'s hidden-toggle + the A4
untruncated button-title / ``#sync-progress`` placement carry it with no
new element), with the phase-64 ``total === 0`` guard (the empty KB and
the scan's own first frame render the bare ``Scanning…``) and no
chunk/image part (the scan reads no file);
* the pass-through — both sync status call sites (the poll tick, the
load-time re-attach) forward the WHOLE status object, which is what
carries the scan pair to the wrapper (the summaries / chunk / image
pattern: ``phase`` selects which pair renders, no field is re-listed);
* the UNTOUCHED set — ``fmtSyncLabel``'s ``N remaining`` expression
verbatim (the phase-193 pin), the chunk / image / summaries part
builders, ``setSyncLabel``'s composed full string, ``fmtSyncResult`` and
``fmtSyncPeakClause`` (the phase-187 trailing clause lands on the scan
label's line unchanged, and stays a terminal-only clause);
* the docstrings — the decision-tree paragraph names the new branch,
narrows the fallback to the null prelude + ``"import"``, and keeps the
phase-193 ``N remaining`` clause.
The owner-visible arms (the count-down over a real incremental sync, the
scan label in a real browser) live in
``tests/e2e/test_sync_pending_count.py`` (task 05); the stale pinned suites
retarget in task 04.
"""
from __future__ import annotations
import json
import shutil
import subprocess
from pathlib import Path
import pytest
#: The node behavior check needs node (the test-document-viewer pattern).
HAVE_NODE = shutil.which("node") is not None
ROOT = Path(__file__).resolve().parents[2]
SOURCES_JS = ROOT / "frontend" / "assets" / "sources.js"
def _js() -> str:
return SOURCES_JS.read_text(encoding="utf-8")
def _fn(js: str, name: str) -> str:
"""The source of the first ``function <name>`` (brace balanced — the
house pin pattern from tests/unit/test_frontend_sync_upload.py: the
functions live inside ``mount(root)``, so the closing brace is
indented)."""
fn = js.find(f"function {name}")
assert fn != -1, f"{name} must be defined in sources.js"
open_idx = js.find("{", fn)
depth = 0
for i in range(open_idx, len(js)):
c = js[i]
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return js[fn : i + 1]
raise AssertionError(f"unbalanced braces in {name}")
def _doc_above(js: str, name: str) -> str:
"""The block comment immediately above ``function <name>``."""
i = js.find(f"function {name}")
assert i != -1, f"{name} must be defined in sources.js"
start = js.rfind("/*", 0, i)
assert start != -1, f"{name} must carry a docstring"
end = js.find("*/", start)
assert start < end < i, f"{name}'s docstring must sit above the function"
return js[start:end]
def _scan_slice(js: str) -> str:
"""The ``"scan"`` branch of ``fmtSyncPhaseLabel`` — from its ``if`` to
the ``overview`` branch that follows it (every shape pin below reads
ONLY this slice, so a shape that leaked into another branch is caught
and a branch that moved is caught)."""
body = _fn(js, "fmtSyncPhaseLabel")
start = body.find('status.phase === "scan"')
assert start != -1, 'the "scan" branch is missing from fmtSyncPhaseLabel'
end = body.find('status.phase === "overview"', start)
assert end != -1, "the overview branch must follow the scan branch"
return body[start:end]
# ---------- the branch ----------
def test_scan_branch_is_the_first_arm_of_the_decision_tree() -> None:
"""The wrapper gains ONE explicit ``"scan"`` branch (D4): the phase
comparison sits BEFORE ``overview``, ``summaries`` and the
``fmtSyncLabel`` fall-through (the machine order is ``null`` →
``"scan"`` → ``"import"`` → ``"overview"`` → ``"summaries"``, and
keeping the arms in that order is what leaves every pre-existing
branch slice byte-identical), and the main text is exactly
``Scanning…`` with the ellipsis the other running labels use."""
js = _js()
body = _fn(js, "fmtSyncPhaseLabel")
i_scan = body.find('status.phase === "scan"')
i_ov = body.find('status.phase === "overview"')
i_sum = body.find('status.phase === "summaries"')
i_fall = body.find("fmtSyncLabel(", i_sum)
assert -1 < i_scan < i_ov < i_sum < i_fall, (
"the scan branch must be the first arm, ahead of overview / "
"summaries / the null-prelude + import fall-through"
)
assert 'main: "Scanning\u2026"' in _scan_slice(js), (
"the main text is the bare Scanning label — no file"
)
def test_scan_branch_reads_the_scan_pair_and_nothing_else() -> None:
"""The branch consumes the scan pair the way it already consumes the
summaries pair (``phase`` selects which pair renders). It NEVER reads
``current_file`` — the runner keeps it null through the phase, because
the scan names no file — nor the file pair, the chunk pair, the image
pair or the summaries pair (a scan embeds no chunk and describes no
image, so those fields carry nothing meaningful here)."""
scan = _scan_slice(_js())
assert "status.scan_done" in scan and "status.scan_total" in scan, (
"the count part is the status's own scan pair"
)
for field in (
"current_file",
"files_done",
"files_total",
"chunks_total",
"current_chunk",
"images_total",
"images_done",
"summaries_total",
"summaries_done",
"current_summary",
):
assert field not in scan, (
f'the "scan" branch must not read {field} — that field is the '
"import/overview/summaries phase's own position"
)
def test_scan_branch_emits_no_chunk_and_no_image_part() -> None:
"""The scan names no file, so its parts are the main text + the count
only: ``chunk: ""`` and ``image: ""`` (the writer hides both spans and
keeps them out of the composed full string — the overview branch's
shape, phase 98/152)."""
scan = _scan_slice(_js())
assert 'chunk: ""' in scan, "the scan label carries no chunk part"
assert 'image: ""' in scan, "the scan label carries no image part"
# ---------- the (n/total) count part ----------
def test_scan_count_part_is_the_pair_in_the_count_slot() -> None:
"""The count part is the exact ``(${status.scan_done}/${status.scan_total})``
template — a plain ``(n/total)``, no thousands separator, no suffix —
and it lands on the ``count`` key, i.e. the SAME slot the import
phase's ``N remaining`` uses (``setSyncLabel``'s
``.sync-label-count`` span + the composed full string), so the count
stays pinned on a phone and the A4 untruncated label (button title +
``#sync-progress``) carries ``Scanning… (n/total)`` with no new
element and no new live region."""
scan = _scan_slice(_js())
assert "`(${status.scan_done}/${status.scan_total})`" in scan, (
"the exact (n/total) shape over the scan pair"
)
i_count = scan.find("count:")
i_template = scan.find("`(${status.scan_done}/${status.scan_total})`")
assert -1 < i_count < i_template, "the pair renders INTO the count slot"
assert "remaining" not in scan, (
"the scan never says remaining — that part belongs to fmtSyncLabel"
)
for helper in ("toLocaleString", ".toFixed"):
assert helper not in scan, "plain integers, the house label style"
def test_scan_count_part_hides_while_the_total_is_zero() -> None:
"""The phase-64 ``total === 0`` contract, extended to the scan: while
the scan has not reported its denominator (and for the degenerate
empty KB, where the importer fires no scan callback at all) the count
part is the empty string — the bare ``Scanning…`` renders, never
``(0/0)``. The guard precedes the template and the false arm is the
empty string."""
scan = _scan_slice(_js())
i_guard = scan.find("status.scan_total > 0")
i_template = scan.find("`(${status.scan_done}/${status.scan_total})`")
i_else = scan.find(': ""', i_template)
assert -1 < i_guard < i_template < i_else, (
"scan_total > 0 ? (n/total) : \"\" — the phase-64 no-count-part guard"
)
# ---------- the caller pass-through ----------
def test_poll_tick_forwards_the_whole_status_so_the_scan_pair_rides() -> None:
"""The running-sync branch of the poll tick passes the WHOLE status
object as ``enterSyncRunningState``'s fifth argument — that object IS
the pass-through for the scan pair, exactly like the summaries /
chunk / image pairs (the call site names it). A status with
``phase: "scan"`` therefore reaches the wrapper with its
``scan_done`` / ``scan_total`` intact."""
js = _js()
start = js.find("function startSyncPolling")
assert start != -1
tick = js[start : js.find("async function startSync", start)]
i_run = tick.find('syncStatus.state === "running"')
assert i_run != -1
# the call-site comment sits directly above the branch — it is part of
# the pass-through's documentation, so the slice starts there
branch = tick[tick.rfind("// 1. sync running", 0, i_run) : tick.find(
'uploadStatus && uploadStatus.state === "running"', i_run
)]
assert (
'"sync", syncStatus.current_file, syncStatus.files_done, syncStatus.files_total'
in branch
), "the pre-phase positional args are unchanged"
i_args = branch.find("syncStatus.files_total")
i_pass = branch.find("syncStatus\n", i_args)
assert i_pass != -1, "the WHOLE syncStatus object rides in as the 5th arg"
assert "scan_done" in branch and "scan_total" in branch, (
"the call-site comment names the scan pair riding the status (phase 194)"
)
def test_load_time_reattach_forwards_the_whole_status_too() -> None:
"""The load-time re-attach is the other sync call site: a reload
during the pre-scan re-enters running state with the WHOLE status, so
the re-attached label reads ``Scanning… (n/total)`` (the never-stale
contract — phase 98's mid-summaries reload, now also mid-scan)."""
js = _js()
body = _fn(js, "initSyncButton")
assert (
'"sync", status.current_file, status.files_done, status.files_total' in body
), "the pre-phase positional args are unchanged"
i_args = body.find("status.files_total")
i_pass = body.find("status\n", i_args)
assert i_pass != -1, "the WHOLE status object rides in as the 5th arg"
doc = _doc_above(js, "initSyncButton")
assert "phase 194" in doc and "Scanning" in doc, (
"the re-attach docstring names the scan label it can re-enter (194)"
)
def test_running_entry_hands_the_status_to_the_phase_wrapper() -> None:
"""``enterSyncRunningState`` keeps the phase-aware split untouched:
sync + status → ``fmtSyncPhaseLabel(status)`` (the wrapper that now
owns the scan branch), everything else → the bare ``fmtSyncLabel``
parts, and the parts still go to the ONE writer with the running
flag (so the scan label lands on the button title + the box)."""
entry = _fn(_js(), "enterSyncRunningState")
assert "fmtSyncPhaseLabel(status)" in entry
assert 'kind === "sync" && status' in entry
assert "setSyncLabel(main, count, chunk, true, image)" in entry
# ---------- the UNTOUCHED set ----------
def test_fmt_sync_label_n_remaining_arithmetic_is_byte_identical() -> None:
"""The phase-193 count-down builder is UNTOUCHED by this task — the
server-side semantic change (the pair becomes the pending position)
flows through the SAME expression. The exact line, both guards and the
plain-integer shape, plus the absence of anything scan-shaped in this
builder."""
bare = _fn(_js(), "fmtSyncLabel")
assert (
"const count = total > 0 && total - done > 0"
" ? `${total - done} remaining` : \"\";"
) in bare, "the phase-193 count expression, byte for byte"
assert 'const prefix = kind === "upload" ? "Importing" : "Syncing\u2026";' in bare
assert "return { main, count }" in bare
for word in ("scan", "chunk", "image", "summaries", "phase"):
assert word not in bare, f"fmtSyncLabel stays free of {word!r}"
def test_other_phase_labels_and_the_writers_are_untouched() -> None:
"""The regression pins for everything the scan branch could have
disturbed: the overview copy, the summaries folder + ``(n/m)``
position, the import fall-through's chunk + image parts, the single
writer's signature + composed full string, the terminal counts line
(no running count part, no scan word) and the phase-187 peak clause
(still a terminal-only trailing clause, null renders nothing)."""
js = _js()
phase = _fn(js, "fmtSyncPhaseLabel")
assert (
'return { main: "Writing KB overview\u2026", count: "", chunk: "", image: "" };'
in phase
)
assert "`Summarizing folders\u2026${folder}`" in phase
assert (
"count: `(${status.summaries_done}/${status.summaries_total})`" in phase
), "the summaries folder position keeps its (n/m)"
assert (
'const { main, count } = fmtSyncLabel(\n'
' "sync", status.current_file, status.files_done, status.files_total\n'
" );" in phase
), "the import fall-through still calls the bare builder unchanged"
assert "`\u00b7 chunk ${status.current_chunk}/${status.chunks_total}`" in phase
assert "`\u00b7 image ${status.images_done}/${status.images_total}`" in phase
assert "return { main, count, chunk, image };" in phase
writer = _fn(js, "setSyncLabel")
assert (
'function setSyncLabel(mainText, countText = "", chunkText = "",'
' running = false, imageText = "")' in writer
)
assert "syncLabelCount.textContent = countText" in writer
assert "syncLabelCount.hidden = false" in writer
assert (
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\") +\n"
" (imageText ? ` ${imageText}` : \"\");" in writer
), "the A4 untruncated label is still main + count + chunk + image"
assert "if (running) setSyncProgress(full);" in writer
assert "syncBtn.title = full" in writer
result = _fn(js, "fmtSyncResult")
for word in ("remaining", "Scanning", "scan"):
assert word not in result, f"the terminal counts line stays {word!r}-free"
peak = _fn(js, "fmtSyncPeakClause")
assert "return ` \u00b7 peak ${peak} MB`;" in peak
assert "status.rss_peak_mb ?? status.rss_mb" in peak
assert "Scanning" not in peak, "the clause is terminal-only, unchanged"
# ---------- the docstrings ----------
def test_phase_label_docstring_names_the_branch_and_narrows_the_fallback() -> None:
"""The decision-tree paragraph above the wrapper names the new arm and
narrows the fall-through to the two states that remain, keeps the
phase-193 ``N remaining`` clause, states the ``total === 0`` guard,
and says WHY an ``(n/total)`` is honest here (TODO L6: the objection
was the sync count wearing the total's shape — this label says
Scanning, never Syncing)."""
doc = _doc_above(_js(), "fmtSyncPhaseLabel")
assert '"scan"' in doc and "Scanning\u2026" in doc, "the new branch is named"
assert "scan_done" in doc and "scan_total" in doc, "the pair is named"
assert "scan_total === 0" in doc, "the no-count-part guard is documented"
assert "ONLY the two" in doc and '"import"' in doc, "the fallback is narrowed"
assert "`N remaining`" in doc and "N = total - done" in doc, (
"the phase-193 count-down clause survives the rewrite"
)
assert "never" in doc and '"Syncing"' in doc, (
"the honesty clause: Scanning, never Syncing (TODO L6)"
)
assert "A4" in doc, "the untruncated placement is still documented"
def test_module_and_section_headers_document_the_scan_label() -> None:
"""The module header carries the phase-194 paragraph (the count is the
WORK, the scan says so), and the sync section's decision tree names
the ``"scan"`` arm with its ``(n/total)`` — the header the wrapper's
docstring points at."""
js = _js()
header = js[: js.find("import { fetchIsAdmin }")]
assert "Phase 194 (task 03)" in header, "the module header names the phase"
assert "Scanning\u2026 (n/total)" in header, "the module header names the label"
marker = js.find("Sync sources button (Sources page only)")
assert marker != -1
section = js[marker : js.find("const syncBtn", marker)]
assert "Phase 194 task 03" in section, "the section header names the phase"
assert "Scanning\u2026 (n/total)" in section, "the decision tree names the label"
assert "scan_done" in section, "the pair the WHOLE status carries is named"
# ---------- the REAL label path under node ----------
#: The stub element refs (the test-document-viewer pattern; the same
#: shapes tests/unit/test_frontend_sync_upload.py drives). The label
#: writer, the box writer and the cancel control are the REAL functions.
_NODE_HEAD = """
const syncLabelMain = { textContent: "" };
const syncLabelCount = { textContent: "", hidden: true };
const syncLabelChunk = { textContent: "", hidden: true };
const syncProgress = { hidden: true };
const syncProgressText = { textContent: "" };
const syncBtn = {
disabled: false,
title: "",
attrs: {},
setAttribute(k, v) { this.attrs[k] = v; },
removeAttribute(k) { delete this.attrs[k]; },
classes: new Set(),
classList: {
add(c) { syncBtn.classes.add(c); },
remove(c) { syncBtn.classes.delete(c); },
has(c) { return syncBtn.classes.has(c); },
},
};
const syncIcon = {
classes: new Set(),
classList: {
add(c) { syncIcon.classes.add(c); },
remove(c) { syncIcon.classes.delete(c); },
has(c) { return syncIcon.classes.has(c); },
},
};
const syncCancelBtn = {
hidden: true,
disabled: true,
attrs: {},
setAttribute(k, v) { this.attrs[k] = v; },
removeAttribute(k) { delete this.attrs[k]; },
};
"""
_NODE_TAIL = """
const show = (tag) => {
console.log(JSON.stringify([
tag,
syncLabelMain.textContent,
syncLabelCount.textContent,
syncLabelCount.hidden,
syncBtn.title,
syncProgressText.textContent,
syncBtn.attrs['aria-label'] || '',
]));
};
// 1. the scan, denominator known + the position climbing (the mid-scan poll):
enterSyncRunningState("sync", null, 0, 0, {
phase: "scan", current_file: null, files_done: 0, files_total: 0,
scan_done: 3, scan_total: 9, current_chunk: 0, chunks_total: 0,
});
show("scan-3-9");
// 2. the scan's degenerate arm: an empty KB / the scan's own first frame
// (0/0) — the phase-64 guard, the bare label with NO count part:
enterSyncRunningState("sync", null, 0, 0, {
phase: "scan", current_file: null, files_done: 0, files_total: 0,
scan_done: 0, scan_total: 0, current_chunk: 0, chunks_total: 0,
});
show("scan-0-0");
// 3. the prelude (null phase) is untouched — bare "Syncing\u2026":
enterSyncRunningState("sync", null, 0, 0, {
phase: null, current_file: null, files_done: 0, files_total: 0,
scan_done: 4, scan_total: 9, current_chunk: 0, chunks_total: 0,
});
show("prelude");
// 4. the import phase is untouched — the phase-193 count-down + the chunk
// part, and the scan pair present on the payload but NOT rendered:
enterSyncRunningState("sync", "src/a/b.md", 2, 9, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 9,
scan_done: 9, scan_total: 9, current_chunk: 14, chunks_total: 38,
});
show("import");
// 5. the overview + summaries labels are untouched:
enterSyncRunningState("sync", null, 9, 9, {
phase: "overview", current_file: null, files_done: 9, files_total: 9,
scan_done: 9, scan_total: 9, current_chunk: 0, chunks_total: 0,
});
show("overview");
enterSyncRunningState("sync", null, 9, 9, {
phase: "summaries", current_file: null, files_done: 9, files_total: 9,
scan_done: 9, scan_total: 9, current_summary: "src/alpha",
summaries_done: 2, summaries_total: 3, current_chunk: 0, chunks_total: 0,
});
show("summaries");
// 6. the 202/409 click (no status at all) stays bare:
enterSyncRunningState("sync", null, 0, 0);
show("click");
"""
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_scan_label_renders_through_the_real_label_path_under_node() -> None:
"""Behavior, not just markers: the REAL extracted functions run
against stub refs. ``phase: "scan"`` renders main ``Scanning…`` with
the count ``(3/9)`` visible, and the SAME string lands on the button
title, the box (the sole aria-live region) and the button's
accessible name (A4) — while every other phase renders exactly what it
rendered before this task."""
js = _js()
fns = "\n".join(
_fn(js, name)
for name in (
"fmtSyncLabel",
"fmtSyncPhaseLabel",
"setSyncLabel",
"setSyncProgress",
"showSyncCancelBtn",
"hideSyncCancelBtn",
"enterSyncRunningState",
)
)
out = subprocess.run(
["node", "-e", _NODE_HEAD + fns + _NODE_TAIL],
capture_output=True,
text=True,
timeout=60,
)
assert out.returncode == 0, out.stderr
rows = {row[0]: row for row in (json.loads(line) for line in out.stdout.splitlines())}
main, count, hidden, title, box, aria = rows["scan-3-9"][1:]
assert main == "Scanning\u2026", rows["scan-3-9"]
assert count == "(3/9)" and hidden is False, rows["scan-3-9"]
assert title == "Scanning\u2026 (3/9)" == box == aria, (
f"A4: the untruncated scan label rides title + box + aria: {rows['scan-3-9']}"
)
main, count, hidden, title, box, aria = rows["scan-0-0"][1:]
assert main == "Scanning\u2026" and count == "" and hidden is True, rows["scan-0-0"]
assert title == "Scanning\u2026" == box == aria, (
f"0/0 renders NO count part (the empty-KB guard): {rows['scan-0-0']}"
)
# every other state is byte-identical to its pre-phase label
assert rows["prelude"][1] == "Syncing\u2026" and rows["prelude"][2] == ""
assert rows["prelude"][4] == "Syncing\u2026", "a scan pair never renders in the prelude"
assert rows["import"][1] == "Syncing\u2026 src/a/b.md"
assert rows["import"][2] == "7 remaining", rows["import"]
assert rows["import"][4] == "Syncing\u2026 src/a/b.md 7 remaining \u00b7 chunk 14/38"
assert rows["overview"][1] == "Writing KB overview\u2026" and rows["overview"][2] == ""
assert rows["summaries"][1] == "Summarizing folders\u2026 src/alpha"
assert rows["summaries"][2] == "(2/3)", "the summaries position keeps its (n/m)"
assert rows["click"][1] == "Syncing\u2026" and rows["click"][2] == ""