phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -11,20 +11,46 @@ position *in* the whole; the owner reads the distance *to the end*.
|
||||
Phase 193 changes exactly ONE label part: ``fmtSyncLabel``'s count is
|
||||
now ``N remaining`` with ``N = total − done``.
|
||||
|
||||
**Server untouched.** No endpoint, no payload field, no poll changed:
|
||||
``GET /api/sync/status`` still answers with the phase-64 key set
|
||||
(``current_file`` / ``files_done`` / ``files_total`` — the 1-based
|
||||
position of the in-flight file over the pre-walk count), and the suite
|
||||
pins exactly that (test 1's cross-layer invariant): the payload still
|
||||
carries the ``(done, total)`` position while the label speaks the
|
||||
subtraction of the very same pair. The change is one string built by
|
||||
``frontend/assets/sources.js``.
|
||||
**Phase 194 (task 04, D5) retargets what the pair COUNTS — the label
|
||||
builder stays byte-identical.** Phase 193 itself touched no server
|
||||
code; phase 194 did, and this is the suite whose pins move with it.
|
||||
``GET /api/sync/status`` no longer answers with the bare phase-64 key
|
||||
set: it is that set plus the phase-187 memory pair plus the phase-194
|
||||
scan pair (``scan_done`` / ``scan_total`` — the PRE-SCAN's own walk
|
||||
position), and ``phase`` gained the ``"scan"`` stage between the null
|
||||
prelude and ``"import"`` (the machine the page renders as
|
||||
``Scanning… (n/total)``). More importantly the ``(done, total)`` pair
|
||||
the label subtracts is no longer the WALK position: on a progress
|
||||
caller (the UI sync is the only one) it is the position in the
|
||||
importer's pre-scan PREDICTED change set — ``files_total`` the number
|
||||
of files that will actually be re-indexed, ``files_done`` the 1-based
|
||||
position in that set. ``N remaining`` therefore counts the WORK, which
|
||||
is the fix for the owner's prod report of 2026-10-19 ("1216 remaining"
|
||||
on a ~10,000-file KB with ~10 files to sync). The phase-193 wording —
|
||||
the 1-based position across EVERY importable file of the stats
|
||||
pre-walk — is exactly what ``BOR_SYNC_PENDING_COUNT=0`` restores, and
|
||||
nothing below asserts it except as that rollback's history.
|
||||
|
||||
**A1 (owner-confirmed):** ``N remaining`` shows while ``N ≥ 1`` and is
|
||||
HIDDEN at the final file (``N == 0``) — the main part names that file;
|
||||
``total === 0`` (the pre-walk, the phase-64 contract) also renders no
|
||||
count part. **A2 (owner-confirmed):** the upload job shares the ONE
|
||||
builder, so it shares the shape (``Importing <file> N remaining``).
|
||||
What survives the semantic shift UNEDITED is a property of this
|
||||
suite's own fixture: every arm starts from a TRUNCATED KB, so the
|
||||
prediction and the walk coincide (all ``N_FILES`` files are pending),
|
||||
``files_total`` stays ``N_FILES`` and the ``1 … N−1`` count frames land
|
||||
in the same slots phase 193 pinned — test 1's cross-layer invariant is
|
||||
untouched arithmetic. The case where the two DIVERGE — a re-sync — is
|
||||
test 4, and the phase's own suite
|
||||
(``tests/e2e/test_sync_pending_count.py``, task 05) carries the K-of-N
|
||||
incremental arm.
|
||||
|
||||
**A1 (owner-confirmed, phase 193):** ``N remaining`` shows while
|
||||
``N ≥ 1`` and is HIDDEN at the final file (``N == 0``) — the main part
|
||||
names that file; ``total === 0`` renders no count part either — and
|
||||
since phase 194 there are TWO things that make it ``0``: the scan has
|
||||
not reported a denominator yet (the null prelude / the ``0`` rollback,
|
||||
test 2) and a run whose predicted change set is EMPTY (test 4).
|
||||
**A2 (owner-confirmed):** the upload job shares the ONE builder, so it
|
||||
shares the shape (``Importing <file> N remaining``) — and it gains
|
||||
NOTHING from phase 194: the upload job runs no pre-scan and its
|
||||
endpoint carries no scan fields (phase 90 A2), so test 3 is UNEDITED.
|
||||
|
||||
Fixture: a real git repo (the phase-28/32/107 ``file://`` transport-true
|
||||
convention — ``BOR_GIT_SOURCES=file://<fixture repo>``, so the env
|
||||
@@ -59,22 +85,44 @@ Arms (the phase's three, in test order):
|
||||
label carries a count part at all while the main part still names
|
||||
that file; no ``Syncing…`` label anywhere carries the pre-phase
|
||||
``(n/m)`` tail; the ``(n/m)`` shape survives ONLY in the summaries
|
||||
phase (A1's exception) and the overview label still paints; the
|
||||
terminal result line is byte-identical to the phase-64 counts line
|
||||
(the ``counts_line`` helper); and the endpoint's own tick series
|
||||
still reports the untouched ``(done, total)`` position whose
|
||||
subtraction equals every rendered ``N``.
|
||||
phase (A1's exception) and, since phase 194, in the scan phase's own
|
||||
``Scanning… (n/total)`` (never on a ``Syncing…`` label, and no scan
|
||||
label ever follows an import one); the overview label still paints;
|
||||
the terminal result line is byte-identical to the phase-64 counts
|
||||
line (the ``counts_line`` helper); and the endpoint's own tick
|
||||
series still reports the ``(done, total)`` pair whose subtraction
|
||||
equals every rendered ``N``. Phase 194 adds the machine's own
|
||||
evidence to that same tick series
|
||||
(``_assert_phase_machine_tick_series``): every body is exactly the
|
||||
pre-phase key set plus the memory pair plus the scan pair, the null
|
||||
prelude reports the scan pair as ``0/0``, a tick caught INSIDE the
|
||||
scan names no file and carries the full scan denominator, no scan
|
||||
tick follows a tick that already counted files, and the terminal
|
||||
KEEPS both pairs beside each other (the phase-64 keep-final-values
|
||||
convention) with ``scan_total == files_total == N_FILES`` on this
|
||||
fresh KB — the parity that lets the phase-193 frames stand. The scan
|
||||
window itself is five tiny files (single-digit milliseconds), so a
|
||||
100 ms tick is not guaranteed to land inside it: the deterministic
|
||||
``Scanning… (n/total)`` frames are test 2's, and the choreographed
|
||||
server-side sequence lives in
|
||||
``tests/integration/test_sync_scan_phase.py``.
|
||||
2. ``test_prewalk_and_final_file_labels_have_no_count_part`` — **(c)
|
||||
the pre-walk edge**, pinned on the exact payload the guard reads:
|
||||
the page's own 2 s poll fed ``phase: null`` / ``files_total: 0``
|
||||
renders the bare ``Syncing…`` (no count part, the count span hidden,
|
||||
the title equal), the first file renders ``N − 1 remaining``, a
|
||||
the zero-total edges**, pinned on the exact payload the guards read:
|
||||
the page's own 2 s poll fed a ``phase: "scan"`` body renders
|
||||
``Scanning… (n/total)`` (count span visible, title equal, never a
|
||||
file and never ``N remaining``), the same stage with
|
||||
``scan_total: 0`` renders the bare ``Scanning…`` (the empty-KB
|
||||
guard), a ``phase: null`` / ``files_total: 0`` prelude renders the
|
||||
bare ``Syncing…`` (no count part, the count span hidden, the title
|
||||
equal), the first file renders ``N − 1 remaining``, a
|
||||
mid-run tick renders ``1 remaining`` *beside* ``· chunk 2/5``, the
|
||||
final-file tick (``done == total``) renders the file with the count
|
||||
part GONE and never brings it back, and the success terminal settles
|
||||
to the untouched counts line. The served bodies use the status
|
||||
endpoint's unchanged key set — the client's own decision points
|
||||
(``total > 0``, ``total − done > 0``) are what is under test.
|
||||
to the untouched counts line. The served bodies carry the status
|
||||
endpoint's key set — the phase-64 fields plus the memory and scan
|
||||
pairs, the stubs pinned against the same constant test 1 reads —
|
||||
because the client's own decision points (``total > 0``,
|
||||
``total − done > 0``, ``scan_total > 0``) are what is under test.
|
||||
3. ``test_upload_label_shares_the_count_down_builder`` — **(b) the
|
||||
upload arm (A2)**: the upload branch of the same two-job tree shares
|
||||
the builder, so a running upload status that carries counts renders
|
||||
@@ -88,12 +136,30 @@ Arms (the phase's three, in test order):
|
||||
upload label can ever regress to ``(n/m)``. The upload branch never
|
||||
receives a status object (its endpoint has no phase / chunk / image
|
||||
fields — phase 129 / 152), so the chunk part coexists in the SYNC
|
||||
arms only (test 1 and 2); that is asserted here too.
|
||||
arms only (test 1 and 2); that is asserted here too. UNEDITED by
|
||||
phase 194 — the upload job runs no pre-scan and its endpoint gains
|
||||
no scan keys.
|
||||
4. ``test_resync_of_an_unchanged_repo_reports_zero_work`` — **the
|
||||
phase-194 re-sync arm**, the run phase 193's fixture never had (the
|
||||
owner's prod case in its mildest form): sync this same repo TWICE
|
||||
and the second run predicts NOTHING, so its ``(done, total)`` pair
|
||||
is ``0/0`` on every tick — the A1 no-count-part exception driven by
|
||||
an EMPTY pending set instead of by a missing denominator — while the
|
||||
scan pair reports the whole KB walked (``scan_done == scan_total ==
|
||||
N_FILES``, kept at the terminal). That divergence (5 scanned, 0 to
|
||||
re-index) is the phase in one line, and it is exactly the shape that
|
||||
used to render "``N`` remaining" for files nobody would re-index.
|
||||
The terminal counts line stays the byte-identical
|
||||
``0 added · N unchanged``. A no-change run makes no LLM call, so its
|
||||
import window is milliseconds and it renders no file label: the wire
|
||||
pair (``0/0``) plus test 2's ``total === 0`` guard pin is the honest
|
||||
form of "no count part" here.
|
||||
|
||||
Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c):
|
||||
1 → (a) + the terminal-line pin + the untouched-payload pin,
|
||||
2 → (c) + the A1 hidden-at-final-file edge (+ (a)'s arithmetic on exact
|
||||
inputs), 3 → (b).
|
||||
Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c
|
||||
+ task 04's D5 retarget): 1 → (a) + the terminal-line pin + the
|
||||
additive-payload + phase-machine tick-series pins,
|
||||
2 → (c) + the A1 hidden-at-final-file edge + the ``Scanning…`` frames
|
||||
(+ (a)'s arithmetic on exact inputs), 3 → (b), 4 → the D5 re-sync arm.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -148,10 +214,11 @@ SOURCES_URL = "/sources.html"
|
||||
#: its own (the module docstring's determinism note).
|
||||
SLOW_DELAY_S = "1.5"
|
||||
|
||||
#: The module's wall-time budget (app boot + one full 5-file sync + the
|
||||
#: two served-status arms). A breach is a speed canary, not flake
|
||||
#: tolerance (the phase-64 convention).
|
||||
WALL_BUDGET_S = 300.0
|
||||
#: The module's wall-time budget (app boot + two full 5-file syncs — the
|
||||
#: phase-194 re-sync arm runs the fixture's sync twice — plus the two
|
||||
#: served-status arms). A breach is a speed canary, not flake tolerance
|
||||
#: (the phase-64 convention).
|
||||
WALL_BUDGET_S = 480.0
|
||||
|
||||
#: The sync's subject: a git repo whose DIRECTORY NAME is the source
|
||||
#: name (the ``repo_name`` basename rule, phase 28). Deliberately NOT
|
||||
@@ -196,11 +263,52 @@ CHUNK_OVERLAP = int(Settings.model_fields["chunk_overlap_chars"].default)
|
||||
IMPORT_LABEL_RE = re.compile(
|
||||
r"^Syncing… \S+(?: [0-9]+ remaining)?(?: · chunk \d+/\d+)?$"
|
||||
)
|
||||
#: The scan-phase label grammar phase 194 (task 03, D4) adds: the bare
|
||||
#: ``Scanning…`` (no denominator yet — the empty KB and the scan's own
|
||||
#: first frame) or it plus the ``(n/total)`` count part. NEVER a file
|
||||
#: (``current_file`` is null through the phase) and never ``N
|
||||
#: remaining`` — the walk position is honest behind the word "Scanning",
|
||||
#: never behind "Syncing" (TODO L6).
|
||||
SCAN_LABEL_RE = re.compile(r"^Scanning…(?: \(\d+/\d+\))?$")
|
||||
#: The pre-phase shape phase 193 removed from this part — no ``Syncing…``
|
||||
#: label may ever carry a ``(n/m)`` tail again.
|
||||
#: label may ever carry a ``(n/m)`` tail again. Phase 194 leaves that
|
||||
#: true: the ``(n/m)`` shape is legal ONLY on a ``Scanning…`` or a
|
||||
#: ``Summarizing folders…`` label (and no label starting ``Syncing…``
|
||||
#: matches either).
|
||||
OLD_FRACTION_RE = re.compile(r"\(\d+/\d+\)")
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
|
||||
#: The FULL ``GET /api/sync/status`` key set this suite's stubs mirror and
|
||||
#: every real tick is pinned against — the phase-64/98/129/152 fields,
|
||||
#: the phase-187 memory pair and the phase-194 SCAN pair (``scan_done`` /
|
||||
#: ``scan_total``). The additive chain, pinned as ONE constant so a future
|
||||
#: key has to be added here (and to the stub below) instead of drifting
|
||||
#: away from the client's view of the payload.
|
||||
EXPECTED_STATUS_KEYS = frozenset(
|
||||
{
|
||||
"state",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"current_file",
|
||||
"files_done",
|
||||
"files_total",
|
||||
"scan_done",
|
||||
"scan_total",
|
||||
"phase",
|
||||
"current_summary",
|
||||
"summaries_done",
|
||||
"summaries_total",
|
||||
"current_chunk",
|
||||
"chunks_total",
|
||||
"images_done",
|
||||
"images_total",
|
||||
"detail",
|
||||
"error",
|
||||
"rss_mb",
|
||||
"rss_peak_mb",
|
||||
}
|
||||
)
|
||||
|
||||
SETTLE_TIMEOUT_MS = 180_000
|
||||
|
||||
|
||||
@@ -527,17 +635,55 @@ class _TickRecorder:
|
||||
"""The deterministic layer: a daemon thread tight-polling (~100 ms)
|
||||
the SAME status endpoint the UI's 2 s poll reads, with its OWN admin
|
||||
session (the Playwright sync API is not thread-safe, so the thread
|
||||
never touches the page). It starts nothing — it only watches."""
|
||||
never touches the page). It starts nothing — it only watches.
|
||||
|
||||
def __init__(self, base_url: str, path: str) -> None:
|
||||
``wait_for_running`` is the phase-194 re-sync arm's need: a recorder
|
||||
started while a PREVIOUS run's terminal is still on the endpoint must
|
||||
not treat that body as its own. A body belongs to THIS run once the
|
||||
recorder has seen it go ``running``, or — the belt-and-braces half,
|
||||
for a run faster than the 100 ms cadence — once its own
|
||||
``started_at`` post-dates the recorder's start (the
|
||||
``test_fast_path_deferred_content.py`` idiom). Without the flag every
|
||||
body counts, which is what the module's first arm wants (there is no
|
||||
previous run to confuse it with).
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: str, path: str, *, wait_for_running: bool = False) -> None:
|
||||
self._url = f"{base_url}{path}"
|
||||
self._login_url = f"{base_url}/api/login"
|
||||
self._wait_for_running = wait_for_running
|
||||
self._saw_running = not wait_for_running
|
||||
self._started_after = 0.0
|
||||
self._ticks: list[dict[str, Any]] = []
|
||||
self._terminal: dict[str, Any] | None = None
|
||||
self._stop = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
|
||||
@staticmethod
|
||||
def _epoch(body: dict[str, Any]) -> float | None:
|
||||
"""The body's ``started_at`` as epoch seconds (``None`` when no
|
||||
run ever started, or the stamp is unreadable)."""
|
||||
raw = body.get("started_at")
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(str(raw)).timestamp()
|
||||
except ValueError: # pragma: no cover — defensive, an odd stamp
|
||||
return None
|
||||
|
||||
def _is_ours(self, body: dict[str, Any]) -> bool:
|
||||
"""Does this body belong to the run this recorder was started for?"""
|
||||
if body["state"] == "running":
|
||||
self._saw_running = True
|
||||
if self._saw_running:
|
||||
return True
|
||||
started = self._epoch(body)
|
||||
return started is not None and started >= self._started_after - 1.0
|
||||
|
||||
def start(self) -> None:
|
||||
if self._wait_for_running:
|
||||
self._started_after = time.time()
|
||||
|
||||
def run() -> None:
|
||||
with httpx.Client(timeout=5.0) as client:
|
||||
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
|
||||
@@ -546,6 +692,10 @@ class _TickRecorder:
|
||||
r = client.get(self._url)
|
||||
if r.status_code == 200:
|
||||
body = r.json()
|
||||
if not self._is_ours(body):
|
||||
# A previous run's terminal — not ours.
|
||||
self._stop.wait(0.02)
|
||||
continue
|
||||
self._ticks.append(body)
|
||||
if body["state"] in ("success", "failed", "cancelled"):
|
||||
self._terminal = body
|
||||
@@ -575,6 +725,13 @@ class _TickRecorder:
|
||||
def running_ticks(self) -> list[dict[str, Any]]:
|
||||
return [t for t in self._ticks if t["state"] == "running"]
|
||||
|
||||
@property
|
||||
def ticks(self) -> list[dict[str, Any]]:
|
||||
"""EVERY body this recorder saw, the terminal included (the
|
||||
phase-194 phase-machine pin reads the whole series, not just the
|
||||
running frames)."""
|
||||
return list(self._ticks)
|
||||
|
||||
|
||||
def _serve_statuses(page: Page, stages: dict[str, Any]) -> None:
|
||||
"""Route the two job-status endpoints to the CURRENT stage body (the
|
||||
@@ -605,10 +762,12 @@ def _serve_statuses(page: Page, stages: dict[str, Any]) -> None:
|
||||
|
||||
|
||||
def _sync_status(**overrides: Any) -> dict[str, Any]:
|
||||
"""One ``GET /api/sync/status`` body with the endpoint's UNCHANGED
|
||||
key set (the phase-64/98/129/152/187 fields the client reads) — the
|
||||
served stubs are faithful to the payload shape; only the values are
|
||||
the test's."""
|
||||
"""One ``GET /api/sync/status`` body with the endpoint's CURRENT key
|
||||
set (the phase-64/98/129/152/187 fields the client reads plus the
|
||||
phase-194 scan pair — the additive chain ``EXPECTED_STATUS_KEYS``
|
||||
pins) — the served stubs are faithful to the payload shape; only the
|
||||
values are the test's. The equality against that constant is asserted
|
||||
below so a stub can never drift from the real endpoint unnoticed."""
|
||||
body: dict[str, Any] = {
|
||||
"state": "idle",
|
||||
"started_at": None,
|
||||
@@ -616,6 +775,11 @@ def _sync_status(**overrides: Any) -> dict[str, Any]:
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan's position, additive after
|
||||
# files_total and 0/0 until the scan reports (the degenerate and
|
||||
# BOR_SYNC_PENDING_COUNT=0 edges both render no count part).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
"summaries_done": 0,
|
||||
@@ -629,6 +793,7 @@ def _sync_status(**overrides: Any) -> dict[str, Any]:
|
||||
"rss_mb": None,
|
||||
"rss_peak_mb": None,
|
||||
}
|
||||
assert set(body) == EXPECTED_STATUS_KEYS, set(body) ^ EXPECTED_STATUS_KEYS
|
||||
body.update(overrides)
|
||||
return body
|
||||
|
||||
@@ -648,6 +813,85 @@ def _upload_status(**overrides: Any) -> dict[str, Any]:
|
||||
return body
|
||||
|
||||
|
||||
def _assert_phase_machine_tick_series(
|
||||
ticks: list[dict[str, Any]],
|
||||
terminal: dict[str, Any],
|
||||
*,
|
||||
scan_total: int,
|
||||
files_total: int,
|
||||
prelude_zero: bool = True,
|
||||
) -> None:
|
||||
"""The phase-194 machine, read off the status endpoint the UI polls.
|
||||
|
||||
Every claim here is deterministic on a real run, which is why none of
|
||||
them depends on a tick happening to land inside the scan (a five-file
|
||||
scan is milliseconds — see the module docstring):
|
||||
|
||||
* the payload is ADDITIVE — exactly :data:`EXPECTED_STATUS_KEYS` on
|
||||
every body, idle/running/terminal alike;
|
||||
* the null prelude reports the scan pair as ``0/0`` (the run has not
|
||||
reached ``import_sources`` yet — the scan is what sets it);
|
||||
* a tick caught inside ``phase == "scan"`` names NO file, carries the
|
||||
full ``scan_total`` denominator and a position that only climbs;
|
||||
* the machine is ONE-DIRECTIONAL — no scan tick follows a tick that
|
||||
already counted files (``files_total > 0`` means the first
|
||||
``progress`` call flipped it, and the phase never flips back);
|
||||
* the terminal CLEARS ``phase`` and KEEPS both pairs side by side
|
||||
(the phase-64 keep-final-values convention);
|
||||
* ``files_total`` is never larger than the scan's denominator on a
|
||||
run whose KB did not change mid-run — the count and its denominator
|
||||
stay on the SAME predicted set.
|
||||
"""
|
||||
for tick in ticks:
|
||||
assert set(tick) == EXPECTED_STATUS_KEYS, (
|
||||
f"the status payload drifted from the additive key set: "
|
||||
f"{sorted(set(tick) ^ EXPECTED_STATUS_KEYS)}"
|
||||
)
|
||||
if terminal not in ticks:
|
||||
assert set(terminal) == EXPECTED_STATUS_KEYS, terminal
|
||||
|
||||
if prelude_zero:
|
||||
for tick in ticks:
|
||||
if tick["state"] == "running" and tick["phase"] is None:
|
||||
# The prelude (the model-check + clone/pull span): the
|
||||
# scan has not started, so it has reported nothing. The
|
||||
# success terminal sets state BEFORE clearing phase, so
|
||||
# no terminal body can reach this arm.
|
||||
assert (tick["scan_done"], tick["scan_total"]) == (0, 0), (
|
||||
f"the null prelude already carried scan counts: {tick}"
|
||||
)
|
||||
|
||||
scan_ticks = [t for t in ticks if t["phase"] == "scan"]
|
||||
for tick in scan_ticks:
|
||||
assert tick["current_file"] is None, (
|
||||
f"the scan named a file — it counts, it does not read: {tick}"
|
||||
)
|
||||
assert tick["scan_total"] == scan_total, tick
|
||||
assert 0 < tick["scan_done"] <= scan_total, tick
|
||||
positions = [t["scan_done"] for t in scan_ticks]
|
||||
assert positions == sorted(positions), (
|
||||
f"the scan position went backwards across ticks: {positions}"
|
||||
)
|
||||
counted = [i for i, t in enumerate(ticks) if t["files_total"] > 0]
|
||||
if scan_ticks and counted:
|
||||
last_scan = max(i for i, t in enumerate(ticks) if t["phase"] == "scan")
|
||||
assert last_scan < counted[0], (
|
||||
"a scan frame followed a frame that already counted files — "
|
||||
"the phase machine ran backwards"
|
||||
)
|
||||
|
||||
assert terminal["phase"] is None, terminal
|
||||
assert terminal["scan_total"] == scan_total, terminal
|
||||
assert terminal["scan_done"] == scan_total, (
|
||||
f"the terminal did not keep the completed scan: {terminal}"
|
||||
)
|
||||
assert terminal["files_total"] == files_total, terminal
|
||||
assert terminal["files_total"] <= terminal["scan_total"], (
|
||||
"the predicted change set is bigger than the scan that predicted "
|
||||
f"it: {terminal}"
|
||||
)
|
||||
|
||||
|
||||
def _admin_sources_page(page: Page, app_url: str, *, settled: bool = True) -> None:
|
||||
"""Real form login landing on the RAG page (the button's home).
|
||||
|
||||
@@ -796,9 +1040,11 @@ def test_running_label_counts_down_across_a_real_sync(
|
||||
assert terminal["detail"]["added"] == N_FILES, terminal
|
||||
|
||||
# --- the cross-layer invariant: the label is the payload's subtraction
|
||||
# The endpoint is untouched: it still answers with the 1-based
|
||||
# position, and each rendered N equals total − done of the file it
|
||||
# named.
|
||||
# Neither half of this changed in phase 194: the SUBTRACTION is the
|
||||
# phase-193 builder, and on this truncated-KB fixture the pair it
|
||||
# subtracts (the pre-scan's pending position, phase 194) coincides
|
||||
# with the walk position, so every rendered N still equals
|
||||
# total − done of the file the label named.
|
||||
position: dict[str, tuple[int, int]] = {}
|
||||
for tick in recorder.running_ticks:
|
||||
if tick["current_file"]:
|
||||
@@ -821,27 +1067,62 @@ def test_running_label_counts_down_across_a_real_sync(
|
||||
else:
|
||||
assert shown is None, f"{path}: done==total but a part rendered: {frame}"
|
||||
|
||||
# --- phase 194 (task 04, D5): the machine, off the SAME tick series --
|
||||
# The additive key set, the 0/0 prelude, the file-less scan ticks,
|
||||
# the one-directional sequence and the terminal that keeps BOTH
|
||||
# pairs (here equal, because a fresh KB's prediction IS its walk).
|
||||
_assert_phase_machine_tick_series(
|
||||
recorder.ticks, terminal, scan_total=N_FILES, files_total=N_FILES
|
||||
)
|
||||
|
||||
# --- phase 194: the scan label, WHEREVER the page caught the stage ---
|
||||
# Whether the UI's 2 s poll lands inside a five-file scan is a coin
|
||||
# flip (test 2 pins the stage deterministically through a served
|
||||
# status); what is never a coin flip is the SHAPE of the frames it
|
||||
# does catch and where they may sit in the sequence.
|
||||
scan_frames = [f for f in frames if f["text"].startswith("Scanning…")]
|
||||
for frame in scan_frames:
|
||||
assert SCAN_LABEL_RE.fullmatch(frame["text"]), frame
|
||||
assert " remaining" not in frame["text"], frame
|
||||
assert SOURCE not in frame["text"], f"the scan named a file: {frame}"
|
||||
# A4: the untruncated label rides the title AND the announcer.
|
||||
assert frame["title"] == frame["text"], frame
|
||||
assert frame["aria"] == frame["text"], frame
|
||||
assert frame["countHidden"] is not ("(" in frame["text"]), frame
|
||||
first_import = next(
|
||||
(i for i, t in enumerate(texts) if t.startswith("Syncing… ")), None
|
||||
)
|
||||
if first_import is not None:
|
||||
assert all(
|
||||
i < first_import for i, t in enumerate(texts) if t.startswith("Scanning…")
|
||||
), f"a scan label rendered after the import phase began: {texts[:14]}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. (c) The pre-walk edge + the A1 hidden edge, on the exact payloads the
|
||||
# guards read
|
||||
# 2. (c) The zero-total edges + the A1 hidden edge, on the exact payloads
|
||||
# the guards read
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prewalk_and_final_file_labels_have_no_count_part(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
"""The two edges of A1, pinned on the payloads the guards actually
|
||||
read (the served bodies use the status endpoint's unchanged key set —
|
||||
what is under test is the client's ONE builder): ``total === 0``
|
||||
renders no count part, and neither does ``done === total``."""
|
||||
"""The zero-total edges of A1 plus the phase-194 scan stage, pinned on
|
||||
the payloads the guards actually read (the served bodies carry the
|
||||
status endpoint's key set — what is under test is the client's ONE
|
||||
builder): ``scan_total === 0`` renders no count part, ``total === 0``
|
||||
renders none either, and neither does ``done === total``."""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
stages: dict[str, Any] = {
|
||||
"sync": _sync_status(state="running"), # the pre-walk: no file, 0/0
|
||||
# The stage phase 194 inserts where the pre-walk used to stand:
|
||||
# the pre-scan mid-flight, denominator known, position climbing.
|
||||
"sync": _sync_status(
|
||||
state="running", phase="scan", scan_done=2, scan_total=N_FILES
|
||||
),
|
||||
"upload": None,
|
||||
}
|
||||
_serve_statuses(page, stages)
|
||||
# settled=False: the boot fetch is the pre-walk stub — the re-attach
|
||||
# settled=False: the boot fetch is the scan stub — the re-attach
|
||||
# itself is part of what this arm pins.
|
||||
_admin_sources_page(page, app_url, settled=False)
|
||||
_install_recorder(page)
|
||||
@@ -849,9 +1130,36 @@ def test_prewalk_and_final_file_labels_have_no_count_part(
|
||||
box = page.locator("#sync-progress-text")
|
||||
btn = page.locator("#sync-btn")
|
||||
|
||||
# (c) The pre-walk — phase null, current_file null, files_total 0:
|
||||
# the bare prefix, no count part, the count span hidden, the title
|
||||
# (phase 194, D4) The scan stage speaks its OWN label: "Scanning…"
|
||||
# plus the (n/total) part in the SAME count slot the import phase's
|
||||
# "N remaining" uses (one writer, one hidden-toggle, one A4
|
||||
# placement) — never a file, never a chunk part, and the full label
|
||||
# on the button title.
|
||||
expect(box).to_have_text(f"Scanning… (2/{N_FILES})", timeout=SETTLE_TIMEOUT_MS)
|
||||
# The PART's own toggle, not Playwright visibility: while a run is in
|
||||
# flight the button is the spinner and CSS hides the label spans, so
|
||||
# the honest pin is the writer's ``hidden`` flag plus the span's text
|
||||
# (the in-browser recorder in test 1 reads the same property).
|
||||
expect(page.locator(".sync-label-count")).to_have_text(f"(2/{N_FILES})")
|
||||
expect(page.locator(".sync-label-count")).to_have_js_property("hidden", False)
|
||||
expect(btn).to_have_attribute("title", f"Scanning… (2/{N_FILES})")
|
||||
expect(btn).to_be_disabled()
|
||||
|
||||
# …and the same stage with NO denominator yet — the empty KB, and the
|
||||
# scan's own first frame: the phase-64 ``total === 0`` guard renders
|
||||
# the bare phase name, never "(0/0)".
|
||||
stages["sync"] = _sync_status(state="running", phase="scan")
|
||||
expect(box).to_have_text("Scanning…", timeout=SETTLE_TIMEOUT_MS)
|
||||
expect(page.locator(".sync-label-count")).to_have_text("")
|
||||
expect(page.locator(".sync-label-count")).to_have_js_property("hidden", True)
|
||||
expect(btn).to_have_attribute("title", "Scanning…")
|
||||
|
||||
# (c) The null prelude — phase null, current_file null, files_total 0
|
||||
# (the pre-phase pre-walk, and exactly what the
|
||||
# BOR_SYNC_PENDING_COUNT=0 rollback reports for the whole run): the
|
||||
# bare prefix, no count part, the count span hidden, the title
|
||||
# identical (A4).
|
||||
stages["sync"] = _sync_status(state="running")
|
||||
expect(box).to_have_text("Syncing…", timeout=SETTLE_TIMEOUT_MS)
|
||||
expect(page.locator(".sync-label-count")).to_be_hidden()
|
||||
expect(btn).to_have_attribute("title", "Syncing…")
|
||||
@@ -913,19 +1221,41 @@ def test_prewalk_and_final_file_labels_have_no_count_part(
|
||||
finished_at=_now_iso(),
|
||||
files_done=N_FILES,
|
||||
files_total=N_FILES,
|
||||
# The scan pair KEPT at the terminal beside the file counts (the
|
||||
# phase-64 keep-final-values convention) — the counts line below
|
||||
# ignores them, which is the point of pinning them here.
|
||||
scan_done=N_FILES,
|
||||
scan_total=N_FILES,
|
||||
detail={"added": N_FILES, "updated": 0, "unchanged": 0, "pruned": 0},
|
||||
)
|
||||
expect(box).to_have_text(counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS)
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
|
||||
expect(btn).to_be_enabled()
|
||||
|
||||
# Nothing rendered in this run ever spoke the pre-phase shape or a
|
||||
# "0 remaining".
|
||||
# Nothing rendered in this run ever spoke the pre-phase shape on a
|
||||
# "Syncing…" label or a "0 remaining".
|
||||
texts = _texts(_frames(page))
|
||||
assert not any(OLD_FRACTION_RE.search(t) for t in texts if t.startswith("Syncing…")), texts
|
||||
assert not any("0 remaining" in t for t in texts), texts
|
||||
assert "Syncing…" in texts, texts
|
||||
|
||||
# The scan stage rendered, in its own shape, and ONLY before the
|
||||
# import phase started: the (n/m) is the scan's position (and the
|
||||
# summaries phase's), never the sync's.
|
||||
scan_texts = [t for t in texts if t.startswith("Scanning…")]
|
||||
assert f"Scanning… (2/{N_FILES})" in scan_texts, texts
|
||||
assert "Scanning…" in scan_texts, texts
|
||||
for label in scan_texts:
|
||||
assert SCAN_LABEL_RE.fullmatch(label), label
|
||||
assert " remaining" not in label, label
|
||||
first_import = next(
|
||||
(i for i, t in enumerate(texts) if t.startswith("Syncing… ")), None
|
||||
)
|
||||
assert first_import is not None, texts
|
||||
assert all(
|
||||
i < first_import for i, t in enumerate(texts) if t.startswith("Scanning…")
|
||||
), texts
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. (b) A2 — the upload job shares the builder and the shape
|
||||
@@ -1021,3 +1351,118 @@ def test_upload_label_shares_the_count_down_builder(
|
||||
# has no phase / chunk / image fields — phase 129 / 152), so the
|
||||
# chunk part belongs to the sync arms only.
|
||||
assert " · chunk " not in label, label
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. (phase 194, task 04 / D5) The re-sync: the prediction, not the walk
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_resync_of_an_unchanged_repo_reports_zero_work(
|
||||
page: Page, app_url: str, db_ready: None, remaining_repo: Path
|
||||
) -> None:
|
||||
"""The run phase 193's fixture never had: the SAME repo synced twice.
|
||||
|
||||
Run 1 fills the truncated KB, so its predicted change set IS the
|
||||
whole walk (``files_total == scan_total == N_FILES`` — the parity
|
||||
test 1 rides). Run 2 finds every byte where it left them, so the
|
||||
pre-scan predicts NOTHING: the ``progress`` pair is ``0/0`` on every
|
||||
tick and the client's ``total === 0`` guard (test 2) renders no count
|
||||
part — while ``scan_done`` / ``scan_total`` report the whole KB walked
|
||||
and are KEPT at the terminal beside the zeroed file pair. That
|
||||
divergence (5 scanned, 0 to re-index) is phase 194 in one line, and
|
||||
it is exactly the shape that used to render "``N`` remaining" for
|
||||
files nobody was going to re-index (the owner's prod report).
|
||||
|
||||
A no-change run makes ZERO LLM call (the phase-147 fast path), so its
|
||||
import window is milliseconds and the 2 s UI poll is not expected to
|
||||
render a file label for it: the pin is the WIRE pair plus the
|
||||
ABSENCE of a count part in whatever the page did render, not a
|
||||
rendered per-file label (test 1 covers the rendered side).
|
||||
"""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
_admin_sources_page(page, app_url)
|
||||
_install_recorder(page)
|
||||
btn = page.locator("#sync-btn")
|
||||
box = page.locator("#sync-progress-text")
|
||||
|
||||
# --- run 1: the fresh import, the parity test 1 rides ---------------
|
||||
expect(btn).to_be_enabled()
|
||||
# wait_for_running: test 1's TERMINAL is still what the endpoint
|
||||
# answers with here (the run state lives in the app, not the page),
|
||||
# and without the flag this recorder would hand that body back as its
|
||||
# own before this test had synced anything at all.
|
||||
rec1 = _TickRecorder(app_url, "/api/sync/status", wait_for_running=True)
|
||||
rec1.start()
|
||||
btn.click()
|
||||
terminal1 = rec1.stop()
|
||||
assert terminal1["state"] == "success", terminal1
|
||||
expect(box).to_have_text(
|
||||
counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS
|
||||
)
|
||||
assert terminal1["files_total"] == N_FILES, terminal1
|
||||
assert terminal1["scan_total"] == N_FILES, terminal1
|
||||
frame_mark = page.evaluate("() => window.__borFrames.length")
|
||||
|
||||
# --- run 2: the same bytes, the owner's incremental case ------------
|
||||
expect(btn).to_be_enabled(timeout=SETTLE_TIMEOUT_MS)
|
||||
rec2 = _TickRecorder(app_url, "/api/sync/status", wait_for_running=True)
|
||||
rec2.start()
|
||||
btn.click()
|
||||
terminal2 = rec2.stop()
|
||||
assert terminal2["state"] == "success", terminal2
|
||||
|
||||
# The owner-visible terminal line stays the byte-identical phase-64
|
||||
# counts line — the exact surface the prediction never replaces.
|
||||
expect(box).to_have_text(
|
||||
counts_line(f"0 added · {N_FILES} unchanged"), timeout=SETTLE_TIMEOUT_MS
|
||||
)
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL)
|
||||
expect(btn).to_be_enabled()
|
||||
detail = terminal2["detail"]
|
||||
assert (detail["added"], detail["updated"], detail["unchanged"]) == (
|
||||
0,
|
||||
0,
|
||||
N_FILES,
|
||||
), detail
|
||||
|
||||
# THE PHASE IN ONE LINE: the scan walked the KB, the work was nothing.
|
||||
assert terminal2["scan_total"] == N_FILES, terminal2
|
||||
assert terminal2["scan_done"] == N_FILES, terminal2
|
||||
assert terminal2["files_total"] == 0, (
|
||||
f"a no-change run reported a non-zero denominator: {terminal2}"
|
||||
)
|
||||
assert terminal2["files_done"] == 0, terminal2
|
||||
for tick in rec2.ticks:
|
||||
assert (tick["files_done"], tick["files_total"]) == (0, 0), (
|
||||
f"the empty pending set leaked a walk position: {tick}"
|
||||
)
|
||||
assert tick["scan_total"] in (0, N_FILES), tick
|
||||
# `prelude_zero=False`: the runner resets ``state`` to ``running``
|
||||
# one synchronous statement before it resets the scan pair, and the
|
||||
# status endpoint answers from a threadpool — a poll landing in that
|
||||
# window would legitimately see the PREVIOUS run's kept 5/5. The
|
||||
# claim itself is pinned race-free in test 1 and test 2; repeating it
|
||||
# here would buy nothing but a 1-in-100k flake.
|
||||
_assert_phase_machine_tick_series(
|
||||
rec2.ticks, terminal2, scan_total=N_FILES, files_total=0, prelude_zero=False
|
||||
)
|
||||
|
||||
# Whatever run 2 rendered carried NO count part: no "N remaining"
|
||||
# (the empty pending set → the total === 0 guard) and no scan (n/m)
|
||||
# once the import phase had begun.
|
||||
run2_texts = _texts(_frames(page)[frame_mark:])
|
||||
assert run2_texts, (
|
||||
f"nothing was rendered after the run-1 settle mark ({frame_mark}) — "
|
||||
"the click's own label is missing from the recorder"
|
||||
)
|
||||
assert not any(" remaining" in t for t in run2_texts), run2_texts
|
||||
for label in run2_texts:
|
||||
if label.startswith("Scanning…"):
|
||||
assert SCAN_LABEL_RE.fullmatch(label), label
|
||||
if label.startswith("Syncing…"):
|
||||
assert not OLD_FRACTION_RE.search(label), label
|
||||
scan_idx = [i for i, t in enumerate(run2_texts) if t.startswith("Scanning…")]
|
||||
imp_idx = [i for i, t in enumerate(run2_texts) if t.startswith("Syncing… ")]
|
||||
if scan_idx and imp_idx:
|
||||
assert max(scan_idx) < imp_idx[0], run2_texts
|
||||
|
||||
Reference in New Issue
Block a user