phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -67,7 +67,12 @@ tiny chunk). The sorted walk processes ``big.md`` FIRST, so the label's
|
||||
phase-193 count part reads ``1 remaining`` during the big span and is
|
||||
HIDDEN during the small one (the final file: N = total - done = 0 — the
|
||||
endpoint's ``files_done``/``files_total`` stay the 1-based 1/2 then 2/2
|
||||
position; only the LABEL counts down) — and the small file's single
|
||||
position; only the LABEL counts down). Phase 194 moves where that pair
|
||||
comes FROM — the pre-scan's PREDICTED change set instead of the walk —
|
||||
and changes nothing here, because this fixture is a FRESH import: both
|
||||
files are pending, so the pending position IS the walk position (see
|
||||
``tests/e2e/test_sync_remaining_count.py`` for the arm where they
|
||||
diverge) — and the small file's single
|
||||
chunk is the proof that the
|
||||
chunk position RESTARTS per file (the big file ends at its last
|
||||
batch's first index, the small file reports ``(1, 1)``).
|
||||
|
||||
@@ -231,12 +231,19 @@ PRE_187_STATUS_KEYS = frozenset(
|
||||
)
|
||||
MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
|
||||
#: link of the same additive chain (the payload is exactly the pre-187 set
|
||||
#: plus the memory pair plus the scan pair).
|
||||
SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
#: The pre-phase integer count slots (their types are part of the
|
||||
#: unchanged-shape pin in test 2 — never a float, never null).
|
||||
_COUNT_KEYS = frozenset(
|
||||
{
|
||||
"files_done",
|
||||
"files_total",
|
||||
"scan_done",
|
||||
"scan_total",
|
||||
"current_chunk",
|
||||
"chunks_total",
|
||||
"images_done",
|
||||
@@ -643,8 +650,9 @@ def test_status_payload_reports_rss(page: Page, app_url: str, db_ready: None) ->
|
||||
running = _wait_sync_state(page, app_url, {"running"})
|
||||
assert running["state"] == "running", running
|
||||
# The additive contract on the wire: exactly the pre-phase keys plus
|
||||
# the memory pair — nothing renamed, nothing dropped.
|
||||
assert set(running) == PRE_187_STATUS_KEYS | MEMORY_KEYS, set(running)
|
||||
# the memory pair plus the phase-194 scan pair — nothing renamed,
|
||||
# nothing dropped.
|
||||
assert set(running) == PRE_187_STATUS_KEYS | MEMORY_KEYS | SCAN_KEYS, set(running)
|
||||
for key, value in running.items():
|
||||
if key in MEMORY_KEYS:
|
||||
# The documented type: int | null (null is "no reading").
|
||||
@@ -674,7 +682,7 @@ def test_status_payload_reports_rss(page: Page, app_url: str, db_ready: None) ->
|
||||
# --- after the terminal ---------------------------------------------
|
||||
terminal = _wait_sync_state(page, app_url, {"success", "failed", "cancelled"})
|
||||
assert terminal["state"] == "success", terminal
|
||||
assert set(terminal) == PRE_187_STATUS_KEYS | MEMORY_KEYS, set(terminal)
|
||||
assert set(terminal) == PRE_187_STATUS_KEYS | MEMORY_KEYS | SCAN_KEYS, set(terminal)
|
||||
current, peak = terminal["rss_mb"], terminal["rss_peak_mb"]
|
||||
if MEMORY_KNOWABLE:
|
||||
assert peak is not None and isinstance(peak, int), terminal
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -27,6 +27,10 @@ distance to the end — ``N remaining`` with ``N = total - done`` — so the
|
||||
long file (the FIRST of the three) renders ``2 remaining`` while the
|
||||
endpoint's ``files_done``/``files_total`` stay the 1-based 1/3 position.
|
||||
The ``· chunk`` part and the terminal counts line are byte-identical.
|
||||
Phase 194 (task 04, D5) leaves this suite UNEDITED for the same reason:
|
||||
its fixture is a fresh import, so the pair's new source (the pre-scan's
|
||||
predicted change set) and the old one (the walk) are the same three
|
||||
files, and every ``2 remaining`` frame lands where phase 193 pinned it.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``; git on PATH —
|
||||
a documented environment prerequisite, phase 28):
|
||||
|
||||
@@ -11,20 +11,46 @@ position *in* the whole; the owner reads the distance *to the end*.
|
||||
Phase 193 changes exactly ONE label part: ``fmtSyncLabel``'s count is
|
||||
now ``N remaining`` with ``N = total − done``.
|
||||
|
||||
**Server untouched.** No endpoint, no payload field, no poll changed:
|
||||
``GET /api/sync/status`` still answers with the phase-64 key set
|
||||
(``current_file`` / ``files_done`` / ``files_total`` — the 1-based
|
||||
position of the in-flight file over the pre-walk count), and the suite
|
||||
pins exactly that (test 1's cross-layer invariant): the payload still
|
||||
carries the ``(done, total)`` position while the label speaks the
|
||||
subtraction of the very same pair. The change is one string built by
|
||||
``frontend/assets/sources.js``.
|
||||
**Phase 194 (task 04, D5) retargets what the pair COUNTS — the label
|
||||
builder stays byte-identical.** Phase 193 itself touched no server
|
||||
code; phase 194 did, and this is the suite whose pins move with it.
|
||||
``GET /api/sync/status`` no longer answers with the bare phase-64 key
|
||||
set: it is that set plus the phase-187 memory pair plus the phase-194
|
||||
scan pair (``scan_done`` / ``scan_total`` — the PRE-SCAN's own walk
|
||||
position), and ``phase`` gained the ``"scan"`` stage between the null
|
||||
prelude and ``"import"`` (the machine the page renders as
|
||||
``Scanning… (n/total)``). More importantly the ``(done, total)`` pair
|
||||
the label subtracts is no longer the WALK position: on a progress
|
||||
caller (the UI sync is the only one) it is the position in the
|
||||
importer's pre-scan PREDICTED change set — ``files_total`` the number
|
||||
of files that will actually be re-indexed, ``files_done`` the 1-based
|
||||
position in that set. ``N remaining`` therefore counts the WORK, which
|
||||
is the fix for the owner's prod report of 2026-10-19 ("1216 remaining"
|
||||
on a ~10,000-file KB with ~10 files to sync). The phase-193 wording —
|
||||
the 1-based position across EVERY importable file of the stats
|
||||
pre-walk — is exactly what ``BOR_SYNC_PENDING_COUNT=0`` restores, and
|
||||
nothing below asserts it except as that rollback's history.
|
||||
|
||||
**A1 (owner-confirmed):** ``N remaining`` shows while ``N ≥ 1`` and is
|
||||
HIDDEN at the final file (``N == 0``) — the main part names that file;
|
||||
``total === 0`` (the pre-walk, the phase-64 contract) also renders no
|
||||
count part. **A2 (owner-confirmed):** the upload job shares the ONE
|
||||
builder, so it shares the shape (``Importing <file> N remaining``).
|
||||
What survives the semantic shift UNEDITED is a property of this
|
||||
suite's own fixture: every arm starts from a TRUNCATED KB, so the
|
||||
prediction and the walk coincide (all ``N_FILES`` files are pending),
|
||||
``files_total`` stays ``N_FILES`` and the ``1 … N−1`` count frames land
|
||||
in the same slots phase 193 pinned — test 1's cross-layer invariant is
|
||||
untouched arithmetic. The case where the two DIVERGE — a re-sync — is
|
||||
test 4, and the phase's own suite
|
||||
(``tests/e2e/test_sync_pending_count.py``, task 05) carries the K-of-N
|
||||
incremental arm.
|
||||
|
||||
**A1 (owner-confirmed, phase 193):** ``N remaining`` shows while
|
||||
``N ≥ 1`` and is HIDDEN at the final file (``N == 0``) — the main part
|
||||
names that file; ``total === 0`` renders no count part either — and
|
||||
since phase 194 there are TWO things that make it ``0``: the scan has
|
||||
not reported a denominator yet (the null prelude / the ``0`` rollback,
|
||||
test 2) and a run whose predicted change set is EMPTY (test 4).
|
||||
**A2 (owner-confirmed):** the upload job shares the ONE builder, so it
|
||||
shares the shape (``Importing <file> N remaining``) — and it gains
|
||||
NOTHING from phase 194: the upload job runs no pre-scan and its
|
||||
endpoint carries no scan fields (phase 90 A2), so test 3 is UNEDITED.
|
||||
|
||||
Fixture: a real git repo (the phase-28/32/107 ``file://`` transport-true
|
||||
convention — ``BOR_GIT_SOURCES=file://<fixture repo>``, so the env
|
||||
@@ -59,22 +85,44 @@ Arms (the phase's three, in test order):
|
||||
label carries a count part at all while the main part still names
|
||||
that file; no ``Syncing…`` label anywhere carries the pre-phase
|
||||
``(n/m)`` tail; the ``(n/m)`` shape survives ONLY in the summaries
|
||||
phase (A1's exception) and the overview label still paints; the
|
||||
terminal result line is byte-identical to the phase-64 counts line
|
||||
(the ``counts_line`` helper); and the endpoint's own tick series
|
||||
still reports the untouched ``(done, total)`` position whose
|
||||
subtraction equals every rendered ``N``.
|
||||
phase (A1's exception) and, since phase 194, in the scan phase's own
|
||||
``Scanning… (n/total)`` (never on a ``Syncing…`` label, and no scan
|
||||
label ever follows an import one); the overview label still paints;
|
||||
the terminal result line is byte-identical to the phase-64 counts
|
||||
line (the ``counts_line`` helper); and the endpoint's own tick
|
||||
series still reports the ``(done, total)`` pair whose subtraction
|
||||
equals every rendered ``N``. Phase 194 adds the machine's own
|
||||
evidence to that same tick series
|
||||
(``_assert_phase_machine_tick_series``): every body is exactly the
|
||||
pre-phase key set plus the memory pair plus the scan pair, the null
|
||||
prelude reports the scan pair as ``0/0``, a tick caught INSIDE the
|
||||
scan names no file and carries the full scan denominator, no scan
|
||||
tick follows a tick that already counted files, and the terminal
|
||||
KEEPS both pairs beside each other (the phase-64 keep-final-values
|
||||
convention) with ``scan_total == files_total == N_FILES`` on this
|
||||
fresh KB — the parity that lets the phase-193 frames stand. The scan
|
||||
window itself is five tiny files (single-digit milliseconds), so a
|
||||
100 ms tick is not guaranteed to land inside it: the deterministic
|
||||
``Scanning… (n/total)`` frames are test 2's, and the choreographed
|
||||
server-side sequence lives in
|
||||
``tests/integration/test_sync_scan_phase.py``.
|
||||
2. ``test_prewalk_and_final_file_labels_have_no_count_part`` — **(c)
|
||||
the pre-walk edge**, pinned on the exact payload the guard reads:
|
||||
the page's own 2 s poll fed ``phase: null`` / ``files_total: 0``
|
||||
renders the bare ``Syncing…`` (no count part, the count span hidden,
|
||||
the title equal), the first file renders ``N − 1 remaining``, a
|
||||
the zero-total edges**, pinned on the exact payload the guards read:
|
||||
the page's own 2 s poll fed a ``phase: "scan"`` body renders
|
||||
``Scanning… (n/total)`` (count span visible, title equal, never a
|
||||
file and never ``N remaining``), the same stage with
|
||||
``scan_total: 0`` renders the bare ``Scanning…`` (the empty-KB
|
||||
guard), a ``phase: null`` / ``files_total: 0`` prelude renders the
|
||||
bare ``Syncing…`` (no count part, the count span hidden, the title
|
||||
equal), the first file renders ``N − 1 remaining``, a
|
||||
mid-run tick renders ``1 remaining`` *beside* ``· chunk 2/5``, the
|
||||
final-file tick (``done == total``) renders the file with the count
|
||||
part GONE and never brings it back, and the success terminal settles
|
||||
to the untouched counts line. The served bodies use the status
|
||||
endpoint's unchanged key set — the client's own decision points
|
||||
(``total > 0``, ``total − done > 0``) are what is under test.
|
||||
to the untouched counts line. The served bodies carry the status
|
||||
endpoint's key set — the phase-64 fields plus the memory and scan
|
||||
pairs, the stubs pinned against the same constant test 1 reads —
|
||||
because the client's own decision points (``total > 0``,
|
||||
``total − done > 0``, ``scan_total > 0``) are what is under test.
|
||||
3. ``test_upload_label_shares_the_count_down_builder`` — **(b) the
|
||||
upload arm (A2)**: the upload branch of the same two-job tree shares
|
||||
the builder, so a running upload status that carries counts renders
|
||||
@@ -88,12 +136,30 @@ Arms (the phase's three, in test order):
|
||||
upload label can ever regress to ``(n/m)``. The upload branch never
|
||||
receives a status object (its endpoint has no phase / chunk / image
|
||||
fields — phase 129 / 152), so the chunk part coexists in the SYNC
|
||||
arms only (test 1 and 2); that is asserted here too.
|
||||
arms only (test 1 and 2); that is asserted here too. UNEDITED by
|
||||
phase 194 — the upload job runs no pre-scan and its endpoint gains
|
||||
no scan keys.
|
||||
4. ``test_resync_of_an_unchanged_repo_reports_zero_work`` — **the
|
||||
phase-194 re-sync arm**, the run phase 193's fixture never had (the
|
||||
owner's prod case in its mildest form): sync this same repo TWICE
|
||||
and the second run predicts NOTHING, so its ``(done, total)`` pair
|
||||
is ``0/0`` on every tick — the A1 no-count-part exception driven by
|
||||
an EMPTY pending set instead of by a missing denominator — while the
|
||||
scan pair reports the whole KB walked (``scan_done == scan_total ==
|
||||
N_FILES``, kept at the terminal). That divergence (5 scanned, 0 to
|
||||
re-index) is the phase in one line, and it is exactly the shape that
|
||||
used to render "``N`` remaining" for files nobody would re-index.
|
||||
The terminal counts line stays the byte-identical
|
||||
``0 added · N unchanged``. A no-change run makes no LLM call, so its
|
||||
import window is milliseconds and it renders no file label: the wire
|
||||
pair (``0/0``) plus test 2's ``total === 0`` guard pin is the honest
|
||||
form of "no count part" here.
|
||||
|
||||
Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c):
|
||||
1 → (a) + the terminal-line pin + the untouched-payload pin,
|
||||
2 → (c) + the A1 hidden-at-final-file edge (+ (a)'s arithmetic on exact
|
||||
inputs), 3 → (b).
|
||||
Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c
|
||||
+ task 04's D5 retarget): 1 → (a) + the terminal-line pin + the
|
||||
additive-payload + phase-machine tick-series pins,
|
||||
2 → (c) + the A1 hidden-at-final-file edge + the ``Scanning…`` frames
|
||||
(+ (a)'s arithmetic on exact inputs), 3 → (b), 4 → the D5 re-sync arm.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -148,10 +214,11 @@ SOURCES_URL = "/sources.html"
|
||||
#: its own (the module docstring's determinism note).
|
||||
SLOW_DELAY_S = "1.5"
|
||||
|
||||
#: The module's wall-time budget (app boot + one full 5-file sync + the
|
||||
#: two served-status arms). A breach is a speed canary, not flake
|
||||
#: tolerance (the phase-64 convention).
|
||||
WALL_BUDGET_S = 300.0
|
||||
#: The module's wall-time budget (app boot + two full 5-file syncs — the
|
||||
#: phase-194 re-sync arm runs the fixture's sync twice — plus the two
|
||||
#: served-status arms). A breach is a speed canary, not flake tolerance
|
||||
#: (the phase-64 convention).
|
||||
WALL_BUDGET_S = 480.0
|
||||
|
||||
#: The sync's subject: a git repo whose DIRECTORY NAME is the source
|
||||
#: name (the ``repo_name`` basename rule, phase 28). Deliberately NOT
|
||||
@@ -196,11 +263,52 @@ CHUNK_OVERLAP = int(Settings.model_fields["chunk_overlap_chars"].default)
|
||||
IMPORT_LABEL_RE = re.compile(
|
||||
r"^Syncing… \S+(?: [0-9]+ remaining)?(?: · chunk \d+/\d+)?$"
|
||||
)
|
||||
#: The scan-phase label grammar phase 194 (task 03, D4) adds: the bare
|
||||
#: ``Scanning…`` (no denominator yet — the empty KB and the scan's own
|
||||
#: first frame) or it plus the ``(n/total)`` count part. NEVER a file
|
||||
#: (``current_file`` is null through the phase) and never ``N
|
||||
#: remaining`` — the walk position is honest behind the word "Scanning",
|
||||
#: never behind "Syncing" (TODO L6).
|
||||
SCAN_LABEL_RE = re.compile(r"^Scanning…(?: \(\d+/\d+\))?$")
|
||||
#: The pre-phase shape phase 193 removed from this part — no ``Syncing…``
|
||||
#: label may ever carry a ``(n/m)`` tail again.
|
||||
#: label may ever carry a ``(n/m)`` tail again. Phase 194 leaves that
|
||||
#: true: the ``(n/m)`` shape is legal ONLY on a ``Scanning…`` or a
|
||||
#: ``Summarizing folders…`` label (and no label starting ``Syncing…``
|
||||
#: matches either).
|
||||
OLD_FRACTION_RE = re.compile(r"\(\d+/\d+\)")
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
|
||||
#: The FULL ``GET /api/sync/status`` key set this suite's stubs mirror and
|
||||
#: every real tick is pinned against — the phase-64/98/129/152 fields,
|
||||
#: the phase-187 memory pair and the phase-194 SCAN pair (``scan_done`` /
|
||||
#: ``scan_total``). The additive chain, pinned as ONE constant so a future
|
||||
#: key has to be added here (and to the stub below) instead of drifting
|
||||
#: away from the client's view of the payload.
|
||||
EXPECTED_STATUS_KEYS = frozenset(
|
||||
{
|
||||
"state",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"current_file",
|
||||
"files_done",
|
||||
"files_total",
|
||||
"scan_done",
|
||||
"scan_total",
|
||||
"phase",
|
||||
"current_summary",
|
||||
"summaries_done",
|
||||
"summaries_total",
|
||||
"current_chunk",
|
||||
"chunks_total",
|
||||
"images_done",
|
||||
"images_total",
|
||||
"detail",
|
||||
"error",
|
||||
"rss_mb",
|
||||
"rss_peak_mb",
|
||||
}
|
||||
)
|
||||
|
||||
SETTLE_TIMEOUT_MS = 180_000
|
||||
|
||||
|
||||
@@ -527,17 +635,55 @@ class _TickRecorder:
|
||||
"""The deterministic layer: a daemon thread tight-polling (~100 ms)
|
||||
the SAME status endpoint the UI's 2 s poll reads, with its OWN admin
|
||||
session (the Playwright sync API is not thread-safe, so the thread
|
||||
never touches the page). It starts nothing — it only watches."""
|
||||
never touches the page). It starts nothing — it only watches.
|
||||
|
||||
def __init__(self, base_url: str, path: str) -> None:
|
||||
``wait_for_running`` is the phase-194 re-sync arm's need: a recorder
|
||||
started while a PREVIOUS run's terminal is still on the endpoint must
|
||||
not treat that body as its own. A body belongs to THIS run once the
|
||||
recorder has seen it go ``running``, or — the belt-and-braces half,
|
||||
for a run faster than the 100 ms cadence — once its own
|
||||
``started_at`` post-dates the recorder's start (the
|
||||
``test_fast_path_deferred_content.py`` idiom). Without the flag every
|
||||
body counts, which is what the module's first arm wants (there is no
|
||||
previous run to confuse it with).
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: str, path: str, *, wait_for_running: bool = False) -> None:
|
||||
self._url = f"{base_url}{path}"
|
||||
self._login_url = f"{base_url}/api/login"
|
||||
self._wait_for_running = wait_for_running
|
||||
self._saw_running = not wait_for_running
|
||||
self._started_after = 0.0
|
||||
self._ticks: list[dict[str, Any]] = []
|
||||
self._terminal: dict[str, Any] | None = None
|
||||
self._stop = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
|
||||
@staticmethod
|
||||
def _epoch(body: dict[str, Any]) -> float | None:
|
||||
"""The body's ``started_at`` as epoch seconds (``None`` when no
|
||||
run ever started, or the stamp is unreadable)."""
|
||||
raw = body.get("started_at")
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(str(raw)).timestamp()
|
||||
except ValueError: # pragma: no cover — defensive, an odd stamp
|
||||
return None
|
||||
|
||||
def _is_ours(self, body: dict[str, Any]) -> bool:
|
||||
"""Does this body belong to the run this recorder was started for?"""
|
||||
if body["state"] == "running":
|
||||
self._saw_running = True
|
||||
if self._saw_running:
|
||||
return True
|
||||
started = self._epoch(body)
|
||||
return started is not None and started >= self._started_after - 1.0
|
||||
|
||||
def start(self) -> None:
|
||||
if self._wait_for_running:
|
||||
self._started_after = time.time()
|
||||
|
||||
def run() -> None:
|
||||
with httpx.Client(timeout=5.0) as client:
|
||||
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
|
||||
@@ -546,6 +692,10 @@ class _TickRecorder:
|
||||
r = client.get(self._url)
|
||||
if r.status_code == 200:
|
||||
body = r.json()
|
||||
if not self._is_ours(body):
|
||||
# A previous run's terminal — not ours.
|
||||
self._stop.wait(0.02)
|
||||
continue
|
||||
self._ticks.append(body)
|
||||
if body["state"] in ("success", "failed", "cancelled"):
|
||||
self._terminal = body
|
||||
@@ -575,6 +725,13 @@ class _TickRecorder:
|
||||
def running_ticks(self) -> list[dict[str, Any]]:
|
||||
return [t for t in self._ticks if t["state"] == "running"]
|
||||
|
||||
@property
|
||||
def ticks(self) -> list[dict[str, Any]]:
|
||||
"""EVERY body this recorder saw, the terminal included (the
|
||||
phase-194 phase-machine pin reads the whole series, not just the
|
||||
running frames)."""
|
||||
return list(self._ticks)
|
||||
|
||||
|
||||
def _serve_statuses(page: Page, stages: dict[str, Any]) -> None:
|
||||
"""Route the two job-status endpoints to the CURRENT stage body (the
|
||||
@@ -605,10 +762,12 @@ def _serve_statuses(page: Page, stages: dict[str, Any]) -> None:
|
||||
|
||||
|
||||
def _sync_status(**overrides: Any) -> dict[str, Any]:
|
||||
"""One ``GET /api/sync/status`` body with the endpoint's UNCHANGED
|
||||
key set (the phase-64/98/129/152/187 fields the client reads) — the
|
||||
served stubs are faithful to the payload shape; only the values are
|
||||
the test's."""
|
||||
"""One ``GET /api/sync/status`` body with the endpoint's CURRENT key
|
||||
set (the phase-64/98/129/152/187 fields the client reads plus the
|
||||
phase-194 scan pair — the additive chain ``EXPECTED_STATUS_KEYS``
|
||||
pins) — the served stubs are faithful to the payload shape; only the
|
||||
values are the test's. The equality against that constant is asserted
|
||||
below so a stub can never drift from the real endpoint unnoticed."""
|
||||
body: dict[str, Any] = {
|
||||
"state": "idle",
|
||||
"started_at": None,
|
||||
@@ -616,6 +775,11 @@ def _sync_status(**overrides: Any) -> dict[str, Any]:
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan's position, additive after
|
||||
# files_total and 0/0 until the scan reports (the degenerate and
|
||||
# BOR_SYNC_PENDING_COUNT=0 edges both render no count part).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
"summaries_done": 0,
|
||||
@@ -629,6 +793,7 @@ def _sync_status(**overrides: Any) -> dict[str, Any]:
|
||||
"rss_mb": None,
|
||||
"rss_peak_mb": None,
|
||||
}
|
||||
assert set(body) == EXPECTED_STATUS_KEYS, set(body) ^ EXPECTED_STATUS_KEYS
|
||||
body.update(overrides)
|
||||
return body
|
||||
|
||||
@@ -648,6 +813,85 @@ def _upload_status(**overrides: Any) -> dict[str, Any]:
|
||||
return body
|
||||
|
||||
|
||||
def _assert_phase_machine_tick_series(
|
||||
ticks: list[dict[str, Any]],
|
||||
terminal: dict[str, Any],
|
||||
*,
|
||||
scan_total: int,
|
||||
files_total: int,
|
||||
prelude_zero: bool = True,
|
||||
) -> None:
|
||||
"""The phase-194 machine, read off the status endpoint the UI polls.
|
||||
|
||||
Every claim here is deterministic on a real run, which is why none of
|
||||
them depends on a tick happening to land inside the scan (a five-file
|
||||
scan is milliseconds — see the module docstring):
|
||||
|
||||
* the payload is ADDITIVE — exactly :data:`EXPECTED_STATUS_KEYS` on
|
||||
every body, idle/running/terminal alike;
|
||||
* the null prelude reports the scan pair as ``0/0`` (the run has not
|
||||
reached ``import_sources`` yet — the scan is what sets it);
|
||||
* a tick caught inside ``phase == "scan"`` names NO file, carries the
|
||||
full ``scan_total`` denominator and a position that only climbs;
|
||||
* the machine is ONE-DIRECTIONAL — no scan tick follows a tick that
|
||||
already counted files (``files_total > 0`` means the first
|
||||
``progress`` call flipped it, and the phase never flips back);
|
||||
* the terminal CLEARS ``phase`` and KEEPS both pairs side by side
|
||||
(the phase-64 keep-final-values convention);
|
||||
* ``files_total`` is never larger than the scan's denominator on a
|
||||
run whose KB did not change mid-run — the count and its denominator
|
||||
stay on the SAME predicted set.
|
||||
"""
|
||||
for tick in ticks:
|
||||
assert set(tick) == EXPECTED_STATUS_KEYS, (
|
||||
f"the status payload drifted from the additive key set: "
|
||||
f"{sorted(set(tick) ^ EXPECTED_STATUS_KEYS)}"
|
||||
)
|
||||
if terminal not in ticks:
|
||||
assert set(terminal) == EXPECTED_STATUS_KEYS, terminal
|
||||
|
||||
if prelude_zero:
|
||||
for tick in ticks:
|
||||
if tick["state"] == "running" and tick["phase"] is None:
|
||||
# The prelude (the model-check + clone/pull span): the
|
||||
# scan has not started, so it has reported nothing. The
|
||||
# success terminal sets state BEFORE clearing phase, so
|
||||
# no terminal body can reach this arm.
|
||||
assert (tick["scan_done"], tick["scan_total"]) == (0, 0), (
|
||||
f"the null prelude already carried scan counts: {tick}"
|
||||
)
|
||||
|
||||
scan_ticks = [t for t in ticks if t["phase"] == "scan"]
|
||||
for tick in scan_ticks:
|
||||
assert tick["current_file"] is None, (
|
||||
f"the scan named a file — it counts, it does not read: {tick}"
|
||||
)
|
||||
assert tick["scan_total"] == scan_total, tick
|
||||
assert 0 < tick["scan_done"] <= scan_total, tick
|
||||
positions = [t["scan_done"] for t in scan_ticks]
|
||||
assert positions == sorted(positions), (
|
||||
f"the scan position went backwards across ticks: {positions}"
|
||||
)
|
||||
counted = [i for i, t in enumerate(ticks) if t["files_total"] > 0]
|
||||
if scan_ticks and counted:
|
||||
last_scan = max(i for i, t in enumerate(ticks) if t["phase"] == "scan")
|
||||
assert last_scan < counted[0], (
|
||||
"a scan frame followed a frame that already counted files — "
|
||||
"the phase machine ran backwards"
|
||||
)
|
||||
|
||||
assert terminal["phase"] is None, terminal
|
||||
assert terminal["scan_total"] == scan_total, terminal
|
||||
assert terminal["scan_done"] == scan_total, (
|
||||
f"the terminal did not keep the completed scan: {terminal}"
|
||||
)
|
||||
assert terminal["files_total"] == files_total, terminal
|
||||
assert terminal["files_total"] <= terminal["scan_total"], (
|
||||
"the predicted change set is bigger than the scan that predicted "
|
||||
f"it: {terminal}"
|
||||
)
|
||||
|
||||
|
||||
def _admin_sources_page(page: Page, app_url: str, *, settled: bool = True) -> None:
|
||||
"""Real form login landing on the RAG page (the button's home).
|
||||
|
||||
@@ -796,9 +1040,11 @@ def test_running_label_counts_down_across_a_real_sync(
|
||||
assert terminal["detail"]["added"] == N_FILES, terminal
|
||||
|
||||
# --- the cross-layer invariant: the label is the payload's subtraction
|
||||
# The endpoint is untouched: it still answers with the 1-based
|
||||
# position, and each rendered N equals total − done of the file it
|
||||
# named.
|
||||
# Neither half of this changed in phase 194: the SUBTRACTION is the
|
||||
# phase-193 builder, and on this truncated-KB fixture the pair it
|
||||
# subtracts (the pre-scan's pending position, phase 194) coincides
|
||||
# with the walk position, so every rendered N still equals
|
||||
# total − done of the file the label named.
|
||||
position: dict[str, tuple[int, int]] = {}
|
||||
for tick in recorder.running_ticks:
|
||||
if tick["current_file"]:
|
||||
@@ -821,27 +1067,62 @@ def test_running_label_counts_down_across_a_real_sync(
|
||||
else:
|
||||
assert shown is None, f"{path}: done==total but a part rendered: {frame}"
|
||||
|
||||
# --- phase 194 (task 04, D5): the machine, off the SAME tick series --
|
||||
# The additive key set, the 0/0 prelude, the file-less scan ticks,
|
||||
# the one-directional sequence and the terminal that keeps BOTH
|
||||
# pairs (here equal, because a fresh KB's prediction IS its walk).
|
||||
_assert_phase_machine_tick_series(
|
||||
recorder.ticks, terminal, scan_total=N_FILES, files_total=N_FILES
|
||||
)
|
||||
|
||||
# --- phase 194: the scan label, WHEREVER the page caught the stage ---
|
||||
# Whether the UI's 2 s poll lands inside a five-file scan is a coin
|
||||
# flip (test 2 pins the stage deterministically through a served
|
||||
# status); what is never a coin flip is the SHAPE of the frames it
|
||||
# does catch and where they may sit in the sequence.
|
||||
scan_frames = [f for f in frames if f["text"].startswith("Scanning…")]
|
||||
for frame in scan_frames:
|
||||
assert SCAN_LABEL_RE.fullmatch(frame["text"]), frame
|
||||
assert " remaining" not in frame["text"], frame
|
||||
assert SOURCE not in frame["text"], f"the scan named a file: {frame}"
|
||||
# A4: the untruncated label rides the title AND the announcer.
|
||||
assert frame["title"] == frame["text"], frame
|
||||
assert frame["aria"] == frame["text"], frame
|
||||
assert frame["countHidden"] is not ("(" in frame["text"]), frame
|
||||
first_import = next(
|
||||
(i for i, t in enumerate(texts) if t.startswith("Syncing… ")), None
|
||||
)
|
||||
if first_import is not None:
|
||||
assert all(
|
||||
i < first_import for i, t in enumerate(texts) if t.startswith("Scanning…")
|
||||
), f"a scan label rendered after the import phase began: {texts[:14]}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. (c) The pre-walk edge + the A1 hidden edge, on the exact payloads the
|
||||
# guards read
|
||||
# 2. (c) The zero-total edges + the A1 hidden edge, on the exact payloads
|
||||
# the guards read
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prewalk_and_final_file_labels_have_no_count_part(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
"""The two edges of A1, pinned on the payloads the guards actually
|
||||
read (the served bodies use the status endpoint's unchanged key set —
|
||||
what is under test is the client's ONE builder): ``total === 0``
|
||||
renders no count part, and neither does ``done === total``."""
|
||||
"""The zero-total edges of A1 plus the phase-194 scan stage, pinned on
|
||||
the payloads the guards actually read (the served bodies carry the
|
||||
status endpoint's key set — what is under test is the client's ONE
|
||||
builder): ``scan_total === 0`` renders no count part, ``total === 0``
|
||||
renders none either, and neither does ``done === total``."""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
stages: dict[str, Any] = {
|
||||
"sync": _sync_status(state="running"), # the pre-walk: no file, 0/0
|
||||
# The stage phase 194 inserts where the pre-walk used to stand:
|
||||
# the pre-scan mid-flight, denominator known, position climbing.
|
||||
"sync": _sync_status(
|
||||
state="running", phase="scan", scan_done=2, scan_total=N_FILES
|
||||
),
|
||||
"upload": None,
|
||||
}
|
||||
_serve_statuses(page, stages)
|
||||
# settled=False: the boot fetch is the pre-walk stub — the re-attach
|
||||
# settled=False: the boot fetch is the scan stub — the re-attach
|
||||
# itself is part of what this arm pins.
|
||||
_admin_sources_page(page, app_url, settled=False)
|
||||
_install_recorder(page)
|
||||
@@ -849,9 +1130,36 @@ def test_prewalk_and_final_file_labels_have_no_count_part(
|
||||
box = page.locator("#sync-progress-text")
|
||||
btn = page.locator("#sync-btn")
|
||||
|
||||
# (c) The pre-walk — phase null, current_file null, files_total 0:
|
||||
# the bare prefix, no count part, the count span hidden, the title
|
||||
# (phase 194, D4) The scan stage speaks its OWN label: "Scanning…"
|
||||
# plus the (n/total) part in the SAME count slot the import phase's
|
||||
# "N remaining" uses (one writer, one hidden-toggle, one A4
|
||||
# placement) — never a file, never a chunk part, and the full label
|
||||
# on the button title.
|
||||
expect(box).to_have_text(f"Scanning… (2/{N_FILES})", timeout=SETTLE_TIMEOUT_MS)
|
||||
# The PART's own toggle, not Playwright visibility: while a run is in
|
||||
# flight the button is the spinner and CSS hides the label spans, so
|
||||
# the honest pin is the writer's ``hidden`` flag plus the span's text
|
||||
# (the in-browser recorder in test 1 reads the same property).
|
||||
expect(page.locator(".sync-label-count")).to_have_text(f"(2/{N_FILES})")
|
||||
expect(page.locator(".sync-label-count")).to_have_js_property("hidden", False)
|
||||
expect(btn).to_have_attribute("title", f"Scanning… (2/{N_FILES})")
|
||||
expect(btn).to_be_disabled()
|
||||
|
||||
# …and the same stage with NO denominator yet — the empty KB, and the
|
||||
# scan's own first frame: the phase-64 ``total === 0`` guard renders
|
||||
# the bare phase name, never "(0/0)".
|
||||
stages["sync"] = _sync_status(state="running", phase="scan")
|
||||
expect(box).to_have_text("Scanning…", timeout=SETTLE_TIMEOUT_MS)
|
||||
expect(page.locator(".sync-label-count")).to_have_text("")
|
||||
expect(page.locator(".sync-label-count")).to_have_js_property("hidden", True)
|
||||
expect(btn).to_have_attribute("title", "Scanning…")
|
||||
|
||||
# (c) The null prelude — phase null, current_file null, files_total 0
|
||||
# (the pre-phase pre-walk, and exactly what the
|
||||
# BOR_SYNC_PENDING_COUNT=0 rollback reports for the whole run): the
|
||||
# bare prefix, no count part, the count span hidden, the title
|
||||
# identical (A4).
|
||||
stages["sync"] = _sync_status(state="running")
|
||||
expect(box).to_have_text("Syncing…", timeout=SETTLE_TIMEOUT_MS)
|
||||
expect(page.locator(".sync-label-count")).to_be_hidden()
|
||||
expect(btn).to_have_attribute("title", "Syncing…")
|
||||
@@ -913,19 +1221,41 @@ def test_prewalk_and_final_file_labels_have_no_count_part(
|
||||
finished_at=_now_iso(),
|
||||
files_done=N_FILES,
|
||||
files_total=N_FILES,
|
||||
# The scan pair KEPT at the terminal beside the file counts (the
|
||||
# phase-64 keep-final-values convention) — the counts line below
|
||||
# ignores them, which is the point of pinning them here.
|
||||
scan_done=N_FILES,
|
||||
scan_total=N_FILES,
|
||||
detail={"added": N_FILES, "updated": 0, "unchanged": 0, "pruned": 0},
|
||||
)
|
||||
expect(box).to_have_text(counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS)
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
|
||||
expect(btn).to_be_enabled()
|
||||
|
||||
# Nothing rendered in this run ever spoke the pre-phase shape or a
|
||||
# "0 remaining".
|
||||
# Nothing rendered in this run ever spoke the pre-phase shape on a
|
||||
# "Syncing…" label or a "0 remaining".
|
||||
texts = _texts(_frames(page))
|
||||
assert not any(OLD_FRACTION_RE.search(t) for t in texts if t.startswith("Syncing…")), texts
|
||||
assert not any("0 remaining" in t for t in texts), texts
|
||||
assert "Syncing…" in texts, texts
|
||||
|
||||
# The scan stage rendered, in its own shape, and ONLY before the
|
||||
# import phase started: the (n/m) is the scan's position (and the
|
||||
# summaries phase's), never the sync's.
|
||||
scan_texts = [t for t in texts if t.startswith("Scanning…")]
|
||||
assert f"Scanning… (2/{N_FILES})" in scan_texts, texts
|
||||
assert "Scanning…" in scan_texts, texts
|
||||
for label in scan_texts:
|
||||
assert SCAN_LABEL_RE.fullmatch(label), label
|
||||
assert " remaining" not in label, label
|
||||
first_import = next(
|
||||
(i for i, t in enumerate(texts) if t.startswith("Syncing… ")), None
|
||||
)
|
||||
assert first_import is not None, texts
|
||||
assert all(
|
||||
i < first_import for i, t in enumerate(texts) if t.startswith("Scanning…")
|
||||
), texts
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. (b) A2 — the upload job shares the builder and the shape
|
||||
@@ -1021,3 +1351,118 @@ def test_upload_label_shares_the_count_down_builder(
|
||||
# has no phase / chunk / image fields — phase 129 / 152), so the
|
||||
# chunk part belongs to the sync arms only.
|
||||
assert " · chunk " not in label, label
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. (phase 194, task 04 / D5) The re-sync: the prediction, not the walk
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_resync_of_an_unchanged_repo_reports_zero_work(
|
||||
page: Page, app_url: str, db_ready: None, remaining_repo: Path
|
||||
) -> None:
|
||||
"""The run phase 193's fixture never had: the SAME repo synced twice.
|
||||
|
||||
Run 1 fills the truncated KB, so its predicted change set IS the
|
||||
whole walk (``files_total == scan_total == N_FILES`` — the parity
|
||||
test 1 rides). Run 2 finds every byte where it left them, so the
|
||||
pre-scan predicts NOTHING: the ``progress`` pair is ``0/0`` on every
|
||||
tick and the client's ``total === 0`` guard (test 2) renders no count
|
||||
part — while ``scan_done`` / ``scan_total`` report the whole KB walked
|
||||
and are KEPT at the terminal beside the zeroed file pair. That
|
||||
divergence (5 scanned, 0 to re-index) is phase 194 in one line, and
|
||||
it is exactly the shape that used to render "``N`` remaining" for
|
||||
files nobody was going to re-index (the owner's prod report).
|
||||
|
||||
A no-change run makes ZERO LLM call (the phase-147 fast path), so its
|
||||
import window is milliseconds and the 2 s UI poll is not expected to
|
||||
render a file label for it: the pin is the WIRE pair plus the
|
||||
ABSENCE of a count part in whatever the page did render, not a
|
||||
rendered per-file label (test 1 covers the rendered side).
|
||||
"""
|
||||
page.set_default_timeout(SETTLE_TIMEOUT_MS)
|
||||
_admin_sources_page(page, app_url)
|
||||
_install_recorder(page)
|
||||
btn = page.locator("#sync-btn")
|
||||
box = page.locator("#sync-progress-text")
|
||||
|
||||
# --- run 1: the fresh import, the parity test 1 rides ---------------
|
||||
expect(btn).to_be_enabled()
|
||||
# wait_for_running: test 1's TERMINAL is still what the endpoint
|
||||
# answers with here (the run state lives in the app, not the page),
|
||||
# and without the flag this recorder would hand that body back as its
|
||||
# own before this test had synced anything at all.
|
||||
rec1 = _TickRecorder(app_url, "/api/sync/status", wait_for_running=True)
|
||||
rec1.start()
|
||||
btn.click()
|
||||
terminal1 = rec1.stop()
|
||||
assert terminal1["state"] == "success", terminal1
|
||||
expect(box).to_have_text(
|
||||
counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS
|
||||
)
|
||||
assert terminal1["files_total"] == N_FILES, terminal1
|
||||
assert terminal1["scan_total"] == N_FILES, terminal1
|
||||
frame_mark = page.evaluate("() => window.__borFrames.length")
|
||||
|
||||
# --- run 2: the same bytes, the owner's incremental case ------------
|
||||
expect(btn).to_be_enabled(timeout=SETTLE_TIMEOUT_MS)
|
||||
rec2 = _TickRecorder(app_url, "/api/sync/status", wait_for_running=True)
|
||||
rec2.start()
|
||||
btn.click()
|
||||
terminal2 = rec2.stop()
|
||||
assert terminal2["state"] == "success", terminal2
|
||||
|
||||
# The owner-visible terminal line stays the byte-identical phase-64
|
||||
# counts line — the exact surface the prediction never replaces.
|
||||
expect(box).to_have_text(
|
||||
counts_line(f"0 added · {N_FILES} unchanged"), timeout=SETTLE_TIMEOUT_MS
|
||||
)
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL)
|
||||
expect(btn).to_be_enabled()
|
||||
detail = terminal2["detail"]
|
||||
assert (detail["added"], detail["updated"], detail["unchanged"]) == (
|
||||
0,
|
||||
0,
|
||||
N_FILES,
|
||||
), detail
|
||||
|
||||
# THE PHASE IN ONE LINE: the scan walked the KB, the work was nothing.
|
||||
assert terminal2["scan_total"] == N_FILES, terminal2
|
||||
assert terminal2["scan_done"] == N_FILES, terminal2
|
||||
assert terminal2["files_total"] == 0, (
|
||||
f"a no-change run reported a non-zero denominator: {terminal2}"
|
||||
)
|
||||
assert terminal2["files_done"] == 0, terminal2
|
||||
for tick in rec2.ticks:
|
||||
assert (tick["files_done"], tick["files_total"]) == (0, 0), (
|
||||
f"the empty pending set leaked a walk position: {tick}"
|
||||
)
|
||||
assert tick["scan_total"] in (0, N_FILES), tick
|
||||
# `prelude_zero=False`: the runner resets ``state`` to ``running``
|
||||
# one synchronous statement before it resets the scan pair, and the
|
||||
# status endpoint answers from a threadpool — a poll landing in that
|
||||
# window would legitimately see the PREVIOUS run's kept 5/5. The
|
||||
# claim itself is pinned race-free in test 1 and test 2; repeating it
|
||||
# here would buy nothing but a 1-in-100k flake.
|
||||
_assert_phase_machine_tick_series(
|
||||
rec2.ticks, terminal2, scan_total=N_FILES, files_total=0, prelude_zero=False
|
||||
)
|
||||
|
||||
# Whatever run 2 rendered carried NO count part: no "N remaining"
|
||||
# (the empty pending set → the total === 0 guard) and no scan (n/m)
|
||||
# once the import phase had begun.
|
||||
run2_texts = _texts(_frames(page)[frame_mark:])
|
||||
assert run2_texts, (
|
||||
f"nothing was rendered after the run-1 settle mark ({frame_mark}) — "
|
||||
"the click's own label is missing from the recorder"
|
||||
)
|
||||
assert not any(" remaining" in t for t in run2_texts), run2_texts
|
||||
for label in run2_texts:
|
||||
if label.startswith("Scanning…"):
|
||||
assert SCAN_LABEL_RE.fullmatch(label), label
|
||||
if label.startswith("Syncing…"):
|
||||
assert not OLD_FRACTION_RE.search(label), label
|
||||
scan_idx = [i for i, t in enumerate(run2_texts) if t.startswith("Scanning…")]
|
||||
imp_idx = [i for i, t in enumerate(run2_texts) if t.startswith("Syncing… ")]
|
||||
if scan_idx and imp_idx:
|
||||
assert max(scan_idx) < imp_idx[0], run2_texts
|
||||
|
||||
@@ -101,6 +101,20 @@ turn the backfill into a crash instead of a query.
|
||||
``_prune``'s light row (phase 184), the identity-map cadence (phase 184),
|
||||
``app/db.py:30``, ``iter_importable_files`` (phase 185 LOCKED A3) and the
|
||||
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
|
||||
|
||||
**Why phase 194 left these pins UNEDITED (the D5 sweep).** The phase-194
|
||||
pre-scan is gated on the caller passing ``progress`` — and every
|
||||
``import_sources`` call in this file passes NO ``progress`` (the statement
|
||||
capture is the point, so the run needs no hook). The pre-scan therefore
|
||||
never runs here: the captured fast-path ROW statement above stays the only
|
||||
statement over ``documents`` per inspected file, the "exactly one
|
||||
``SELECT`` per unchanged file" pin and the "one PK follow-up for the
|
||||
summary-NULL backfill" pin are byte-identical, and the pre-scan's own
|
||||
scalar statement (``SELECT documents.file_hash``, ``_pending_hash_stmt``)
|
||||
appears nowhere in the capture — it is pinned in
|
||||
``tests/unit/test_sync_pending_count.py`` and
|
||||
``tests/integration/test_sync_pending_count.py`` instead, where a
|
||||
``progress`` caller does exist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -135,6 +135,18 @@ lazy SELECT per row and fail that test, not get papered over.
|
||||
**What this file does NOT touch:** the walk's column set (phase 185),
|
||||
``scripts/git_sync.py`` (phase 186), ``app/db.py:30``, and the
|
||||
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
|
||||
|
||||
**Why phase 194 left the bound UNCHANGED (the D5 sweep).** Several tests
|
||||
here call ``import_sources`` WITH a ``progress`` hook — so the phase-194
|
||||
pre-scan really runs on those walks. It cannot move a single number in
|
||||
this file: ``_prescan_pending`` reads the SCALAR projection
|
||||
``_pending_hash_stmt`` (one column, no entity), through a session of its
|
||||
OWN inside the ``to_thread`` span, so nothing lands in the RUN's
|
||||
``identity_map`` — the staircase, the sawtooth, the ``MAP_BOUND`` ceiling
|
||||
and the byte-identical ``ImportSummary`` / database comparisons are all
|
||||
exactly what phase 184 measured. The hydration pin the claim rests on
|
||||
(``len(session.identity_map) == 0`` at the first ``progress`` call) lives
|
||||
in ``tests/integration/test_sync_pending_count.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -93,6 +93,18 @@ on all three terminals (success, cancelled, failed) plus the importer's
|
||||
``import: summary …``. The fields are APPENDED (LOCKED A2), so every
|
||||
pre-phase prefix below is byte-identical and the pins here assert BOTH
|
||||
halves of that contract.
|
||||
|
||||
Phase 194 (task 02): the status gains the pre-scan. The payload is
|
||||
ADDITIVE — ``scan_done`` / ``scan_total`` after ``files_total``, the
|
||||
additive-chain pins below updated to the new key set (the idle shape and
|
||||
both ``_PRE_187_STATUS_KEYS`` assertions) — and ``files_done`` /
|
||||
``files_total`` now carry the PREDICTED-change-set position on a progress
|
||||
caller. One existing pin retargets with it:
|
||||
``test_double_trigger_while_running_returns_409`` waits on ``"scan"``
|
||||
instead of ``"import"``, because its canned import fires NO per-file hook
|
||||
and the scan→import flip lives in that hook — a run that reports no file
|
||||
stays in the pre-import phase (the full sequence with a live hook is
|
||||
pinned in ``tests/integration/test_sync_scan_phase.py``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -375,6 +387,11 @@ class FakeImportSources:
|
||||
# image fields stay 0/0, like the chunk fields of a no-batch
|
||||
# import).
|
||||
self.image_hooks: list[object] = []
|
||||
# Phase 194 (task 02): the PRE-SCAN position hook the runner
|
||||
# passes (a live closure while wired, None if the wiring
|
||||
# regresses) — the canned fake never fires it, so the status
|
||||
# scan fields stay 0/0 like a run that scanned nothing.
|
||||
self.scan_hooks: list[object] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -387,6 +404,7 @@ class FakeImportSources:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -397,6 +415,7 @@ class FakeImportSources:
|
||||
self.progress_hooks.append(progress)
|
||||
self.chunk_hooks.append(on_chunk_progress)
|
||||
self.image_hooks.append(on_image_progress)
|
||||
self.scan_hooks.append(on_scan_progress)
|
||||
self.ignore_maps.append(ignore_by_root or {})
|
||||
self.include_hidden_maps.append(include_hidden_by_root or {})
|
||||
self.doc_dates_maps.append(doc_dates_by_root or {})
|
||||
@@ -572,6 +591,10 @@ def test_admin_sync_success_reports_full_detail(
|
||||
# (the dataclass defaults).
|
||||
"images_done": 0,
|
||||
"images_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
|
||||
# (the dataclass defaults).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
@@ -703,17 +726,27 @@ def test_double_trigger_while_running_returns_409(
|
||||
assert r.status_code == 409
|
||||
assert r.json() == {"detail": "a sync is already running"}
|
||||
|
||||
# Phase 179 (task 02): WAIT for the import phase — the run's
|
||||
# progress is concurrent now that the git ops left the loop, so the
|
||||
# next request no longer inherits a run that already ran past them.
|
||||
body = _wait_phase(sync_client, "import")
|
||||
# Phase 179 (task 02): WAIT for the import to have started — the
|
||||
# run's progress is concurrent now that the git ops left the loop,
|
||||
# so the next request no longer inherits a run that already ran past
|
||||
# them. Phase 194 (task 02): the phase the wait targets is now
|
||||
# ``"scan"`` — the canned fake import reports NO file, and the
|
||||
# scan→import flip lives in the runner's per-file progress hook, so
|
||||
# a hook-less run legitimately stays in the pre-import phase (the
|
||||
# sequence with a real hook is pinned in test_sync_scan_phase.py and
|
||||
# in the phase-98 gated test below).
|
||||
body = _wait_phase(sync_client, "scan")
|
||||
assert body["state"] == "running"
|
||||
assert body["started_at"] is not None
|
||||
assert body["finished_at"] is None
|
||||
assert body["error"] is None
|
||||
# Wiring pin (phase 194, task 02): the runner passed its live
|
||||
# scan-position closure, and a fake that fires nothing keeps the
|
||||
# position at 0/0.
|
||||
assert len(fake_import.scan_hooks) == 1 and fake_import.scan_hooks[0] is not None
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
# Phase 98 (task 01): the in-flight run (parked in the fake
|
||||
# import's delay) reports the import phase — the summary span has
|
||||
# not started, so its counters are 0/0.
|
||||
# import's delay) reports no summary span — its counters are 0/0.
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
@@ -769,6 +802,10 @@ class _GatedImport:
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 194 (task 02): the PRE-SCAN position hook the runner
|
||||
# passes (recorded for the wiring pin — the gated fake never
|
||||
# fires it, so the status scan fields stay 0/0).
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -781,10 +818,12 @@ class _GatedImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
self.scan_hook = on_scan_progress
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
@@ -1937,6 +1976,7 @@ def test_import_error_is_reported_with_credentials_masked(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -2698,6 +2738,12 @@ _PRE_187_STATUS_KEYS = frozenset(
|
||||
#: Phase 187 (task 03): the two additive memory keys.
|
||||
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the
|
||||
#: next link in the same chain (the payload is exactly the pre-187 set
|
||||
#: plus the memory pair plus the scan pair, and a future addition joins
|
||||
#: this chain rather than editing the sets above).
|
||||
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
#: The pair is read from two DIFFERENT kernel counters and the high-water
|
||||
#: refreshes LAZILY, so a raw ``rss_peak_mb()`` really can sit under a raw
|
||||
#: ``rss_mb()`` (measured: ``rss_mb=164`` against ``rss_peak_mb=162``) —
|
||||
@@ -2744,13 +2790,17 @@ def test_memory_status_keys_are_the_only_addition_and_null_before_a_run(
|
||||
sync_client: TestClient,
|
||||
) -> None:
|
||||
"""The additive pin on the wire: the payload's key set is EXACTLY the
|
||||
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb``, and a run that
|
||||
never happened answers ``null`` / ``null`` — not 0, not a plausible
|
||||
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb`` plus the
|
||||
phase-194 ``scan_done`` / ``scan_total`` pair, and a run that never
|
||||
happened answers ``null`` / ``null`` — not 0, not a plausible
|
||||
number. That null is also what makes the Sources page render the
|
||||
counts line with no peak clause after an app restart."""
|
||||
_login(sync_client)
|
||||
body = sync_client.get("/api/sync/status").json()
|
||||
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, set(body)
|
||||
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, set(body)
|
||||
# Phase 194 (task 02): the scan pair joins ADDITIVELY — idle is 0/0
|
||||
# (never null, never a plausible position).
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
assert body["rss_mb"] is None
|
||||
assert body["rss_peak_mb"] is None
|
||||
# The pre-existing shapes, unchanged: the state word, the two ISO
|
||||
@@ -2811,8 +2861,9 @@ def test_success_terminal_status_keeps_the_final_memory_pair(
|
||||
assert done[0].endswith(
|
||||
f"rss_mb={fmt_mb(current)} rss_peak_mb={fmt_mb(peak)}"
|
||||
), done[0]
|
||||
# The pre-existing payload is untouched by all of this.
|
||||
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS
|
||||
# The pre-existing payload is untouched by all of this (phase 194's
|
||||
# scan pair is the only addition, and it is additive).
|
||||
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS
|
||||
assert again["files_done"] == 0 or isinstance(again["files_done"], int)
|
||||
|
||||
|
||||
|
||||
@@ -58,6 +58,19 @@ clone's result. Python threads are not interruptible: the git worker
|
||||
runs its ``subprocess`` to completion, but the EVENT LOOP is free the
|
||||
whole time.
|
||||
|
||||
**Why phase 194 leaves these git-span pins alone (the D5 sweep).** The
|
||||
runner's import here is a stub (``fake_import`` below, which accepts the
|
||||
phase-194 ``on_scan_progress`` argument and fires nothing), so no real
|
||||
pre-scan runs inside this file's measurement window — the clone window
|
||||
this suite pins stays the only off-loop span it measures. The pre-scan's
|
||||
OWN loop-responsiveness pin (the same phase-179 pattern: a blocking
|
||||
``_file_hash`` over a multi-file fixture while a heartbeat coroutine keeps
|
||||
being scheduled, plus the worker-thread identity) lives in
|
||||
``tests/integration/test_sync_pending_count.py`` — beside the cancel at
|
||||
that span's await (the importer-side twin of this file's
|
||||
cancel-during-clone pin) and its app-side twin in
|
||||
``tests/integration/test_sync_scan_phase.py``.
|
||||
|
||||
**TestClient mechanics (why the trigger still rides a side thread).**
|
||||
The shared app loop here is anyio's ``BlockingPortal`` thread. Firing
|
||||
``POST /api/sync`` from the test thread would make the mid-clone probe
|
||||
@@ -326,6 +339,7 @@ class _FakeImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
|
||||
@@ -0,0 +1,571 @@
|
||||
"""Integration: phase 194 — the pre-scan predicts the change set, off the loop.
|
||||
|
||||
The owner report that opened the phase (2026-10-19, verbatim): *"I just synced
|
||||
the prod brain of reese and it said '1216 remaining' when it only had like 10
|
||||
files to sync."* Prod holds ~10,000 importable files and that run changed ~10.
|
||||
The phase-193 ``N remaining`` label was arithmetically fine — it was fed the
|
||||
WALK (every importable file) instead of the WORK. The fix lives in
|
||||
``app/rag/importer.py``: before the walk, ``_prescan_pending`` hashes every
|
||||
importable file once and reads ``documents.file_hash`` once per file — the
|
||||
exact predicate the phase-147 unchanged fast path applies — and the per-file
|
||||
``progress`` hook reports the position in THAT set.
|
||||
|
||||
What this file measures with a real ``import_sources`` run:
|
||||
|
||||
* **the parity that makes the number honest** — on a stable knowledge base
|
||||
the predicted set's size equals the run's terminal ``added + updated``: the
|
||||
fresh arm (everything pending, exactly what a first import indexes), the
|
||||
unchanged arm (nothing pending, so no count part at all), and the
|
||||
incremental arm (one changed file of four — the owner's prod case, where
|
||||
the walk is 4 and the work is 1);
|
||||
* **zero entity hydration** — the scan's statement projects exactly
|
||||
``documents.file_hash`` (no other column, no ORM row), so a 10,000-file
|
||||
scan retains nothing: the run's session is still empty at the first
|
||||
``progress`` call, before the processing loop has loaded any row (phase
|
||||
184/185);
|
||||
* **the span is OFF the loop** — with a stubbed hash that blocks, a heartbeat
|
||||
coroutine keeps ticking through the scan and the hash runs on a different
|
||||
thread than the one that awaited it (the phase-179 pattern of
|
||||
``test_sync_git_freeze.py``: the git clone used to park the single uvicorn
|
||||
loop and freeze the ``GET /api/sync/status`` poll the UI runs every 2 s,
|
||||
and a 10,000-file hash+SELECT span inline is the same defect class);
|
||||
* **the cancel contract** — cancelling the run at the scan's await lands the
|
||||
cancellation immediately, while the worker itself runs to completion
|
||||
(threads are not interruptible, phase 134) and nothing gets indexed.
|
||||
|
||||
The CLI and the upload job pass no ``progress``, so they run no scan at all;
|
||||
their byte-identity is pinned in ``tests/unit/test_sync_pending_count.py``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import event, select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.db import engine
|
||||
from app.models import Document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The probe's own source-name prefix — nothing else in the KB uses it.
|
||||
PREFIX = "phase194-it-"
|
||||
|
||||
#: A blocking hash of 60 ms per file: four files is a ~250 ms scan — long
|
||||
#: enough that a heartbeat polling every 5 ms cannot tick during it if the
|
||||
#: span ever runs on the loop thread, short enough that the suite stays fast.
|
||||
HASH_BLOCK_S = 0.06
|
||||
|
||||
#: The heartbeat tick count that proves the loop was free across ONE scanned
|
||||
#: file (60 ms / 5 ms ≈ 12 ticks; a blocked loop ticks 0). The ceiling-side
|
||||
#: assertion is the same number generous: this is loop-freeness, not timing.
|
||||
MIN_TICKS_PER_FILE = 4
|
||||
|
||||
|
||||
class ProgressCall(NamedTuple):
|
||||
"""One ``progress(source, rel, done, total)`` report."""
|
||||
|
||||
source: str
|
||||
rel: str
|
||||
done: int
|
||||
total: int
|
||||
|
||||
|
||||
class Run(NamedTuple):
|
||||
"""One real ``import_sources`` pass and everything it reported."""
|
||||
|
||||
summary: ImportSummary
|
||||
progress: list[ProgressCall]
|
||||
scan: list[tuple[int, int]]
|
||||
|
||||
|
||||
def _llm(**setting_overrides: Any) -> FakeEmbedder:
|
||||
"""The deterministic in-process fake on the SHIPPED defaults (the
|
||||
pending-count switch is therefore ON — the phase's default)."""
|
||||
from app.config import Settings
|
||||
|
||||
llm = FakeEmbedder()
|
||||
llm.settings = Settings(_env_file=None, **setting_overrides) # pyright: ignore[reportCallIssue]
|
||||
return llm
|
||||
|
||||
|
||||
def _root(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
|
||||
"""One source root (the ``file://``-style temp dir of the house
|
||||
import fixtures) with *files* under it."""
|
||||
assert name.startswith(PREFIX), f"the probe source must be {PREFIX}-prefixed"
|
||||
root = tmp_path / name
|
||||
for rel, body in files.items():
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _run(
|
||||
session: Session,
|
||||
*roots: Path,
|
||||
llm: FakeEmbedder | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Run:
|
||||
"""One real import over *roots*, collecting both position hooks."""
|
||||
progress: list[ProgressCall] = []
|
||||
scan: list[tuple[int, int]] = []
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
list(roots),
|
||||
llm or _llm(),
|
||||
session=session,
|
||||
progress=lambda s, r, d, t: progress.append(ProgressCall(s, r, d, t)),
|
||||
on_scan_progress=lambda p, t: scan.append((p, t)),
|
||||
**kwargs,
|
||||
)
|
||||
)
|
||||
return Run(summary, list(progress), list(scan))
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def statement_capture() -> Iterator[list[str]]:
|
||||
"""Every statement the engine executes for the duration of ONE test.
|
||||
|
||||
The ``before_cursor_execute`` technique of phase 183/185
|
||||
(``tests/integration/test_import_fast_path_columns.py``): what a SELECT
|
||||
actually projects is invisible in the ORM's Python API and plain in the
|
||||
statement it compiles, so a capture is the only way to pin "the pre-scan
|
||||
reads one column". Removed in teardown, so no other suite sees it.
|
||||
"""
|
||||
captured: list[str] = []
|
||||
|
||||
def _listener(
|
||||
conn: object, # noqa: ARG001
|
||||
cursor: object, # noqa: ARG001
|
||||
statement: str,
|
||||
parameters: object, # noqa: ARG001
|
||||
context: object, # noqa: ARG001
|
||||
executemany: bool, # noqa: ARG001
|
||||
) -> None:
|
||||
captured.append(statement)
|
||||
|
||||
event.listen(engine, "before_cursor_execute", _listener)
|
||||
try:
|
||||
yield captured
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", _listener)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_probe_documents(db: Session) -> Iterator[None]:
|
||||
"""``documents``/``chunks`` are global state — the probe's rows are
|
||||
removed before and after every test (the ``test_sync_api.py`` hygiene
|
||||
pattern, narrowed to this module's own source names)."""
|
||||
_drop(db)
|
||||
yield
|
||||
_drop(db)
|
||||
|
||||
|
||||
def _drop(db: Session) -> None:
|
||||
db.execute(text("DELETE FROM chunks WHERE document_id IN (SELECT id FROM "
|
||||
"documents WHERE source LIKE :p)"), {"p": f"{PREFIX}%"})
|
||||
db.execute(text("DELETE FROM documents WHERE source LIKE :p"), {"p": f"{PREFIX}%"})
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The parity: the predicted set IS the work
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
_FOUR_FILES = {
|
||||
"a.md": "# A\n\nalpha content\n",
|
||||
"b.md": "# B\n\nbravo content\n",
|
||||
"c.md": "# C\n\ncharlie content\n",
|
||||
"notes/d.md": "# D\n\ndelta content\n",
|
||||
}
|
||||
|
||||
|
||||
def test_fresh_kb_pending_set_equals_the_added_count(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Arm 1 — the first import: walk == work, so the numbers are the
|
||||
phase-193 ones (A1's degenerate parity, the reason the phase-193 E2E
|
||||
suite never saw the defect).
|
||||
|
||||
Every file has no row, so every file is pending: the hook reports
|
||||
``1/4 … 4/4`` and the terminal says ``added=4`` — the predicted size
|
||||
equals ``added + updated`` exactly."""
|
||||
root = _root(tmp_path, f"{PREFIX}fresh", _FOUR_FILES)
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.files, run.summary.added, run.summary.errors) == (4, 4, 0)
|
||||
assert [(c.done, c.total) for c in run.progress] == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert run.scan == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert run.progress[-1].total == run.summary.added + run.summary.updated
|
||||
|
||||
|
||||
def test_unchanged_kb_predicts_nothing_and_terminal_says_no_work(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Arm 2 — the no-change sync: nothing pending, so no count part.
|
||||
|
||||
The scan walks all four files and predicts none of them; every ``progress``
|
||||
call therefore carries ``(0, 0)``, which the phase-64 ``total === 0``
|
||||
guard renders as a bare label with no ``N remaining`` — the honest shape
|
||||
for a run with no work, where the pre-phase label said ``4 remaining``
|
||||
while walking. ``summary.files`` (the log's walked-file count) is
|
||||
untouched: the run still processed 4 files and still logged 4 unchanged.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}unchanged", _FOUR_FILES)
|
||||
seed = _run(db, root)
|
||||
assert seed.summary.added == 4
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.files, run.summary.added, run.summary.updated) == (4, 0, 0)
|
||||
assert run.summary.unchanged == 4
|
||||
assert [(c.done, c.total) for c in run.progress] == [(0, 0)] * 4
|
||||
# The scan still walked (and reported) every file — the DENOMINATOR of the
|
||||
# scan is the importable count, not the prediction.
|
||||
assert run.scan == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert run.summary.added + run.summary.updated == 0
|
||||
|
||||
|
||||
def test_incremental_kb_counts_the_changed_files_only(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Arm 3 — the owner's prod case at 1-of-4 scale.
|
||||
|
||||
One changed file out of four: the walk is still 4 (``summary.files``, the
|
||||
log line, and every ``progress`` call naming a file), but the count is the
|
||||
WORK — ``total`` 1, the position climbing only when the walk reaches the
|
||||
changed file, and holding at 0 while it moves over the unchanged ones.
|
||||
The parity pin: the prediction's size equals the terminal's
|
||||
``added + updated``.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}incremental", _FOUR_FILES)
|
||||
seed = _run(db, root)
|
||||
assert seed.summary.added == 4
|
||||
|
||||
(root / "notes" / "d.md").write_text(
|
||||
"# D\n\ndelta content EDITED on disk\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.files, run.summary.updated, run.summary.unchanged) == (
|
||||
4,
|
||||
1,
|
||||
3,
|
||||
)
|
||||
assert [(c.rel, c.done, c.total) for c in run.progress] == [
|
||||
("a.md", 0, 1),
|
||||
("b.md", 0, 1),
|
||||
("c.md", 0, 1),
|
||||
("notes/d.md", 1, 1),
|
||||
]
|
||||
assert run.progress[-1].total == run.summary.added + run.summary.updated
|
||||
|
||||
|
||||
def test_two_changed_files_count_down_to_zero(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The count the owner asked for, at the scale the label is read at.
|
||||
|
||||
Two changed files among four: the denominator is 2 from the first report,
|
||||
the position advances only at the predicted files (the unchanged walk
|
||||
between them holds it), and ``total - done`` reaches 0 exactly when the
|
||||
last predicted file starts — where phase 193's ``N ≥ 1`` guard hides the
|
||||
count part. The terminal reports the same two files as updated."""
|
||||
root = _root(tmp_path, f"{PREFIX}two", _FOUR_FILES)
|
||||
_run(db, root)
|
||||
(root / "a.md").write_text("# A\n\nalpha edited\n", encoding="utf-8")
|
||||
(root / "notes" / "d.md").write_text(
|
||||
"# D\n\ndelta edited\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.updated, run.summary.unchanged) == (2, 2)
|
||||
assert [(c.rel, c.done, c.total) for c in run.progress] == [
|
||||
("a.md", 1, 2),
|
||||
("b.md", 1, 2), # unchanged: the walk moves, the count holds
|
||||
("c.md", 1, 2),
|
||||
("notes/d.md", 2, 2),
|
||||
]
|
||||
remaining = [c.total - c.done for c in run.progress]
|
||||
assert remaining == [1, 1, 1, 0], (
|
||||
f"the label's remaining count went {remaining} — it must hold across "
|
||||
"the unchanged files and reach 0 at the last predicted one (where "
|
||||
"phase 193's guard hides the part)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Zero entity hydration
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prescan_reads_one_column_and_hydrates_nothing(
|
||||
db: Session, tmp_path: Path, statement_capture: list[str]
|
||||
) -> None:
|
||||
"""The scan's statement projects EXACTLY ``documents.file_hash`` and the
|
||||
run's session is still empty when the walk begins.
|
||||
|
||||
Three pins on the same claim, because the failure mode is the 2026-10-15
|
||||
incident (a full walk hydrating 252 MB of markdown):
|
||||
|
||||
* every ``documents`` ``(source, path)`` SELECT issued before the first
|
||||
``progress`` call — i.e. by the scan — names ``file_hash`` and NOTHING
|
||||
else, no ``content`` above all (the phase-185 column-list technique);
|
||||
* exactly one such read per importable file;
|
||||
* ``len(session.identity_map) == 0`` at the first ``progress`` call: the
|
||||
scan put no ``Document`` anywhere, so the run's retention stays bounded
|
||||
by the phase-184 cadence instead of growing with the KB.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}hydrate", _FOUR_FILES)
|
||||
_run(db, root)
|
||||
db.expunge_all()
|
||||
statement_capture.clear()
|
||||
|
||||
map_at_first_progress: list[int] = []
|
||||
|
||||
def progress(source: str, rel: str, done: int, total: int) -> None: # noqa: ARG001
|
||||
map_at_first_progress.append(len(db.identity_map))
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], _llm(), session=db, progress=progress, on_scan_progress=None
|
||||
)
|
||||
)
|
||||
assert summary.unchanged == 4
|
||||
|
||||
# The scan reads precede the loop's row loads in capture order, and the
|
||||
# first progress call is the boundary between them. Classified on the
|
||||
# PROJECTION (phase 185's technique): the scan's projection is the single
|
||||
# column token, the fast path's is every column but ``content``.
|
||||
def predicate(stmt: str) -> str:
|
||||
"""Everything after the statement's ``FROM`` (case-insensitive —
|
||||
SQLAlchemy compiles the keyword upper-case)."""
|
||||
return re.split(r"\bfrom\s+documents\b", stmt, maxsplit=1, flags=re.I)[-1]
|
||||
|
||||
doc_reads = [
|
||||
stmt
|
||||
for stmt in statement_capture
|
||||
if stmt.lstrip().lower().startswith("select")
|
||||
and re.search(r"\bfrom\s+documents\b", stmt, re.I)
|
||||
and "documents.source" in predicate(stmt)
|
||||
and "documents.path" in predicate(stmt)
|
||||
]
|
||||
|
||||
def projection(stmt: str) -> set[str]:
|
||||
head = re.split(r"\bfrom\b", stmt, maxsplit=1, flags=re.I)[0]
|
||||
return set(re.findall(r"documents\.([a-z_]+)", head))
|
||||
|
||||
scan_reads = [s for s in doc_reads if projection(s) == {"file_hash"}]
|
||||
row_loads = [s for s in doc_reads if projection(s) != {"file_hash"}]
|
||||
assert len(scan_reads) == 4, (
|
||||
f"the unchanged run issued {len(scan_reads)} single-column file_hash "
|
||||
"reads — the scan reads each importable file's hash exactly once"
|
||||
)
|
||||
assert len(row_loads) == 4, (
|
||||
f"the loop issued {len(row_loads)} fast-path row loads — the scan must "
|
||||
"not replace _doc_row_stmt, which stays the only ROW load (phase 185)"
|
||||
)
|
||||
assert all(
|
||||
projection(s) == {"file_hash"} for s in scan_reads
|
||||
), "a scan read projects more than file_hash — it would hydrate a row"
|
||||
assert all(
|
||||
"content" not in projection(s) for s in row_loads
|
||||
), "the fast-path row load stopped deferring content (phase 185)"
|
||||
assert map_at_first_progress and map_at_first_progress[0] == 0, (
|
||||
"the run's session already held entities when the loop reached its "
|
||||
"first file — the pre-scan leaked a row into the identity map"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The span is off the event loop (the phase-179 pattern)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _SlowHash:
|
||||
"""The timing fixture: a PLAIN blocking ``_file_hash`` (like the real one
|
||||
— the point under test is WHERE it runs), counting its calls.
|
||||
|
||||
The thread identity is NOT taken from here: ``_file_hash`` is called both
|
||||
by the scan (worker thread) and by the processing loop (loop thread), so
|
||||
the attribution comes from ``on_scan_progress``, which fires on the same
|
||||
worker thread as the scan's own hashes."""
|
||||
|
||||
def __init__(self, real: Callable[[Path], str], block: float = HASH_BLOCK_S) -> None:
|
||||
self.real = real
|
||||
self.block = block
|
||||
self.calls = 0
|
||||
|
||||
def __call__(self, path: Path) -> str:
|
||||
self.calls += 1
|
||||
time.sleep(self.block)
|
||||
return self.real(path)
|
||||
|
||||
|
||||
def test_prescan_span_does_not_park_the_event_loop(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE phase-179 contract, re-pinned for the pre-scan.
|
||||
|
||||
A 4-file scan with a 60 ms blocking hash is ~250 ms of work that used to
|
||||
be inline. The heartbeat coroutine polls every 5 ms; the ticks between two
|
||||
consecutive scan reports prove the LOOP kept running while the worker
|
||||
blocked (a regression that runs the scan inline parks the loop thread and
|
||||
the heartbeat cannot tick — the task-01 git measurement was 0 ticks over
|
||||
a 3 s clone). The thread identities are asserted too, so the pin fails on
|
||||
a machine fast enough to hide the latency.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}freeze", _FOUR_FILES)
|
||||
stub = _SlowHash(importer._file_hash)
|
||||
monkeypatch.setattr(importer, "_file_hash", stub)
|
||||
|
||||
ticks = 0
|
||||
scan_ticks: list[int] = []
|
||||
scan_positions: list[tuple[int, int]] = []
|
||||
scan_thread: list[int] = []
|
||||
|
||||
def scan_hook(position: int, total: int) -> None:
|
||||
# Runs on the span's worker thread — the same thread that just hashed.
|
||||
scan_positions.append((position, total))
|
||||
scan_ticks.append(ticks)
|
||||
scan_thread.append(threading.get_ident())
|
||||
|
||||
# A progress caller is what opens the gate (the CLI shape, ``progress=None``,
|
||||
# would skip the scan entirely) — this is the UI sync's shape.
|
||||
async def main() -> int:
|
||||
nonlocal ticks
|
||||
loop_ident = threading.get_ident()
|
||||
|
||||
async def heartbeat() -> None:
|
||||
nonlocal ticks
|
||||
while True:
|
||||
await asyncio.sleep(0.005)
|
||||
ticks += 1
|
||||
|
||||
hb = asyncio.create_task(heartbeat())
|
||||
try:
|
||||
await import_sources(
|
||||
[root],
|
||||
_llm(),
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: None,
|
||||
on_scan_progress=scan_hook,
|
||||
)
|
||||
finally:
|
||||
hb.cancel()
|
||||
return loop_ident
|
||||
|
||||
loop_ident = asyncio.run(main())
|
||||
|
||||
assert scan_positions == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert scan_thread and loop_ident is not None
|
||||
assert all(ident != loop_ident for ident in scan_thread), (
|
||||
"the pre-scan ran on the event-loop thread — a long scan parks the "
|
||||
"single uvicorn loop again (phase 179's defect class)"
|
||||
)
|
||||
deltas = [
|
||||
b - a for a, b in zip([0, *scan_ticks], scan_ticks, strict=False)
|
||||
]
|
||||
assert min(deltas) >= MIN_TICKS_PER_FILE, (
|
||||
f"the heartbeat ticked {deltas} times across consecutive scanned files "
|
||||
f"(expected >= {MIN_TICKS_PER_FILE} each) — the loop was starved while "
|
||||
"the scan blocked, so the span is not off-loop"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The cancel (phase 134): the run stops, the worker finishes
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cancel_during_the_prescan_stops_the_run_while_the_worker_finishes(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The phase-134 cooperative cancel, measured at the scan's await.
|
||||
|
||||
Cancelling the run ~80 ms into a ~250 ms scan raises ``CancelledError``
|
||||
out of ``import_sources`` PROMPTLY (well before the scan's own remaining
|
||||
work), NOTHING is indexed, and the worker thread runs to completion —
|
||||
threads are not interruptible, and this is exactly why the span reads
|
||||
through a session of its own instead of the run's (which the cancel
|
||||
closes on its way out). The pre-phase-194 pipeline had no await here at
|
||||
all, so this is the new boundary the UI's Stop button must survive.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}cancel", _FOUR_FILES)
|
||||
stub = _SlowHash(importer._file_hash, block=0.12)
|
||||
monkeypatch.setattr(importer, "_file_hash", stub)
|
||||
|
||||
real_prescan = importer._prescan_pending
|
||||
worker_finished = threading.Event()
|
||||
worker_results: list[Any] = []
|
||||
|
||||
def spy(*args: Any, **kwargs: Any) -> Any:
|
||||
try:
|
||||
result = real_prescan(*args, **kwargs)
|
||||
worker_results.append(result)
|
||||
return result
|
||||
finally:
|
||||
worker_finished.set()
|
||||
|
||||
monkeypatch.setattr(importer, "_prescan_pending", spy)
|
||||
|
||||
async def main() -> float:
|
||||
# The hook runs on the WORKER thread, so it only appends to a list —
|
||||
# the house hook contract (a GIL-atomic store). Touching an
|
||||
# ``asyncio.Event`` from here would be a cross-thread loop call.
|
||||
seen: list[tuple[int, int]] = []
|
||||
|
||||
def scan_hook(position: int, total: int) -> None:
|
||||
seen.append((position, total))
|
||||
|
||||
task = asyncio.create_task(
|
||||
import_sources(
|
||||
[root],
|
||||
_llm(),
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: None,
|
||||
on_scan_progress=scan_hook,
|
||||
)
|
||||
)
|
||||
# Wait (from the loop) until the scan has reported its first file.
|
||||
deadline = time.monotonic() + 10.0
|
||||
while not seen and time.monotonic() < deadline:
|
||||
await asyncio.sleep(0.005)
|
||||
assert seen, "the pre-scan never reported a file"
|
||||
await asyncio.sleep(0.08)
|
||||
t0 = time.monotonic()
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
return time.monotonic() - t0
|
||||
|
||||
cancel_latency = asyncio.run(main())
|
||||
|
||||
# The cancel is prompt: the run did NOT wait for the remaining scan work.
|
||||
assert cancel_latency < 0.5, (
|
||||
f"the cancelled run took {cancel_latency:.3f}s to unblock — it is "
|
||||
"still waiting for the blocking scan worker"
|
||||
)
|
||||
# The worker is not interruptible: it finishes its own session's work.
|
||||
assert worker_finished.wait(10.0), "the pre-scan worker never completed"
|
||||
assert stub.calls == 4, (
|
||||
f"the cancelled run's scan worker hashed {stub.calls} of 4 files — "
|
||||
"the contract is that the worker runs to completion"
|
||||
)
|
||||
# And the pipeline that would have consumed the prediction never ran.
|
||||
assert db.scalar(
|
||||
select(select(Document).where(Document.source == root.name).exists())
|
||||
) is False, "the cancelled run indexed documents — the import never started"
|
||||
@@ -0,0 +1,927 @@
|
||||
"""Integration: phase 194 (task 02) — the ``"scan"`` phase on the status surface.
|
||||
|
||||
The owner report that opened the phase (2026-10-19): *"it said '1216
|
||||
remaining' when it only had like 10 files to sync"*. Task 01 moved the
|
||||
count onto the pre-scan's PREDICTED change set; this file pins the
|
||||
status half (design D3, owner-confirmed A2 — the pre-scan is a VISIBLE
|
||||
stage, not an invisible pause):
|
||||
|
||||
* the phase machine is ``null`` → ``"scan"`` (set immediately before
|
||||
``import_sources``, where the scan actually runs) → ``"import"``
|
||||
(flipped by the FIRST per-file ``progress`` call) → ``"overview"`` →
|
||||
``"summaries"`` → the terminal clears ``phase``;
|
||||
* ``scan_done`` / ``scan_total`` carry the scan's own position through
|
||||
that stage, reset with the run, and are KEPT at every terminal beside
|
||||
the final file counts (the phase-64 keep-final-values convention);
|
||||
* the payload is ADDITIVE — the pre-phase key set plus exactly those two
|
||||
keys, byte-identical in name and type, stable from idle through the
|
||||
terminal;
|
||||
* the phase-187 sampling discipline holds on the new stage: the ``"scan"``
|
||||
set and the ``scan``→``import`` flip are TRANSITIONS and force the
|
||||
``/proc`` read, while the scan's own per-file hook is THROTTLED — one
|
||||
read per interval, never one per file (LOCKED A4);
|
||||
* the two degenerate/terminal edges: an empty knowledge base sits in
|
||||
``"scan"`` and never reaches ``"import"`` (no ``progress`` call exists
|
||||
to flip it) and a cancel DURING the scan lands the ``cancelled``
|
||||
terminal with the scan counts kept and nothing indexed (the phase-134
|
||||
contract, the app-side twin of the importer-side pin in
|
||||
``tests/integration/test_sync_pending_count.py``).
|
||||
|
||||
**Why one arm is choreographed and one is real.** An in-process import
|
||||
never awaits real I/O (the embedder is a fake), so the loop cannot answer a
|
||||
status poll from inside it — a real run's ``"import"`` phase is therefore
|
||||
NOT reliably observable, and the real arm pins what the terminal keeps
|
||||
(both pairs, divergent on the incremental run). The flip itself gets a
|
||||
deterministic pin from the choreographed double (the ``test_sync_api``
|
||||
gated-import idiom), which drives the runner's OWN closures and parks at
|
||||
every frame the test reads. The scan, by contrast, runs in a
|
||||
``to_thread`` worker, so the loop is FREE through it — that is exactly why
|
||||
its frames are observable, and why the blocking-hash fixture slows ONLY the
|
||||
scan window (a blanket stub would park the loop through the import too).
|
||||
|
||||
House patterns (``test_sync_api.py`` / ``test_sync_git_freeze.py``): the
|
||||
``sync_client`` fixture is a CONTEXT-MANAGED ``TestClient`` — one app event
|
||||
loop across requests, so the background run survives between the POST and
|
||||
the polls; the module ``_status`` / ``_task`` / ``_cancel_event`` /
|
||||
``_rss_sampled_at`` and the ``git_sources`` / ``sources_meta`` /
|
||||
``folder_summaries`` / ``documents`` global state are reset around every
|
||||
test; the probe is stubbed so nothing reaches the network, and the import
|
||||
runs for real over a temp ``kind=local`` source with the deterministic
|
||||
``FakeEmbedder``. The scan is made observable with a PLAIN blocking
|
||||
``_file_hash`` stub scoped to the scan window (the phase-179 timing-fixture
|
||||
idiom): the point under test is WHICH phase the run is in WHEN the page
|
||||
polls, and a scan of instant hashes is over before the first poll.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.api import sync as sync_api
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document, GitSource
|
||||
from app.rag import git_sources as git_sources_resolver
|
||||
from app.rag.importer import ImportSummary
|
||||
from app.rag.llm import LLMClient
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The probe's own source-name prefix — nothing else in the KB uses it,
|
||||
#: so the module's DELETE-by-source cleanup can never touch another
|
||||
#: suite's rows.
|
||||
PREFIX = "phase194-t02-"
|
||||
|
||||
#: One blocking ``_file_hash`` per file. Three files is a ~360 ms scan:
|
||||
#: long enough that a 10 ms poll catches several distinct positions inside
|
||||
#: the phase (the frames are what the UI's 2 s poll would render), short
|
||||
#: enough that the suite stays fast.
|
||||
HASH_BLOCK_S = 0.12
|
||||
|
||||
|
||||
# --- fixtures (the house sync hygiene) --------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_sync_state() -> Iterator[None]:
|
||||
"""The module-level status object + task + cancel event + sampling
|
||||
guard are process-global — reset before AND after every test so no
|
||||
terminal state, stale cancel or stale throttle window leaks into the
|
||||
next one (``test_sync_api`` / ``test_sync_git_freeze``)."""
|
||||
sync_api._status = sync_api.SyncStatus()
|
||||
sync_api._task = None
|
||||
sync_api._cancel_event = asyncio.Event()
|
||||
sync_api._rss_sampled_at = float("-inf")
|
||||
yield
|
||||
sync_api._status = sync_api.SyncStatus()
|
||||
sync_api._task = None
|
||||
sync_api._cancel_event = asyncio.Event()
|
||||
sync_api._rss_sampled_at = float("-inf")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_git_sources(db: Session) -> Iterator[None]:
|
||||
"""The runner resolves through the real ``git_sources`` table —
|
||||
truncated around every test (the rows this file seeds are
|
||||
``kind=local``)."""
|
||||
db.execute(text("TRUNCATE git_sources"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE git_sources"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_sources_version() -> Iterator[None]:
|
||||
"""The phase-53 bump is global mutable state — pin it to the
|
||||
migration-0010 seed (0) around every test (own session: the runner
|
||||
bumps through its own short-lived ``SessionLocal``)."""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
session = SessionLocal()
|
||||
try:
|
||||
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
||||
session.commit()
|
||||
yield
|
||||
finally:
|
||||
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
||||
session.commit()
|
||||
session.close()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_folder_summaries(db: Session) -> Iterator[None]:
|
||||
"""``folder_summaries`` is global state the real generator writes on a
|
||||
changed KB — truncated around every test."""
|
||||
db.execute(text("TRUNCATE folder_summaries"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE folder_summaries"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_probe_documents(db: Session) -> Iterator[None]:
|
||||
"""The real imports of this module write ``documents``/``chunks`` —
|
||||
removed before and after every test, narrowed to this module's own
|
||||
source names (the ``test_sync_pending_count`` hygiene)."""
|
||||
_drop(db)
|
||||
yield
|
||||
_drop(db)
|
||||
|
||||
|
||||
def _drop(db: Session) -> None:
|
||||
db.execute(
|
||||
text("DELETE FROM chunks WHERE document_id IN "
|
||||
"(SELECT id FROM documents WHERE source LIKE :p)"),
|
||||
{"p": f"{PREFIX}%"},
|
||||
)
|
||||
db.execute(text("DELETE FROM documents WHERE source LIKE :p"), {"p": f"{PREFIX}%"})
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def sync_client() -> Iterator[TestClient]:
|
||||
"""Context-managed ``TestClient`` — one app event loop across requests
|
||||
(the background run must survive between the POST and the polls)."""
|
||||
with TestClient(fastapi_app) as client:
|
||||
yield client
|
||||
|
||||
|
||||
# --- helpers ----------------------------------------------------------------
|
||||
|
||||
|
||||
def _settings(sources_dir: str) -> Settings:
|
||||
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Empty env fallback (the dev ``.env`` never leaks in) — this file's
|
||||
runs resolve their one local row from the table."""
|
||||
monkeypatch.setattr(
|
||||
git_sources_resolver,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, git_sources=""), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
|
||||
|
||||
def _seed_local(db: Session, path: Path) -> None:
|
||||
"""A ``kind=local`` row exactly as the phase-38 API stores it."""
|
||||
db.add(GitSource(url=str(path), kind="local", path=str(path)))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _local_source(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
|
||||
"""One source root (always created — the empty-KB arm needs a real
|
||||
empty directory, not a missing one), with *files* under it."""
|
||||
assert name.startswith(PREFIX), f"the probe source must be {PREFIX}-prefixed"
|
||||
root = tmp_path / name
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
for rel, body in files.items():
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _login(client: TestClient) -> None:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
||||
|
||||
|
||||
def _poll(client: TestClient, want: str, timeout: float = 20.0) -> dict:
|
||||
"""Poll until ``state == want``, failing loudly on any other state."""
|
||||
deadline = time.monotonic() + timeout
|
||||
body: dict = {}
|
||||
while time.monotonic() < deadline:
|
||||
body = client.get("/api/sync/status").json()
|
||||
if body["state"] == want:
|
||||
return body
|
||||
assert body["state"] == "running", (
|
||||
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
|
||||
)
|
||||
time.sleep(0.02)
|
||||
raise AssertionError(f"sync did not reach {want!r} within {timeout}s: {body}")
|
||||
|
||||
|
||||
def _watch_scan(client: TestClient, timeout: float = 20.0) -> tuple[list[dict], dict | None]:
|
||||
"""Poll through the run's SCAN window; stop at the first frame that is
|
||||
no longer a scan.
|
||||
|
||||
Returns ``(frames, terminal)`` — ``terminal`` is non-``None`` when the
|
||||
run finished inside the window. Both stops are honest: the scan runs
|
||||
OFF the loop (so its frames are served, and this list is exactly what
|
||||
the label was built from), while the import that follows is one
|
||||
straight run of the importer that can finish before the next poll.
|
||||
The flip frame therefore gets pinned deterministically by the
|
||||
choreographed test below; what the real run pins here is the scan's
|
||||
own frames plus the values the terminal keeps."""
|
||||
frames: list[dict] = []
|
||||
terminal: dict | None = None
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
body = client.get("/api/sync/status").json()
|
||||
if body["state"] != "running":
|
||||
terminal = body
|
||||
break
|
||||
frames.append(body)
|
||||
if body["phase"] != "scan":
|
||||
break
|
||||
time.sleep(0.01)
|
||||
else:
|
||||
raise AssertionError(f"the run never left the scan phase: {frames[-1]}")
|
||||
return frames, terminal
|
||||
|
||||
|
||||
class _FakeScanImport:
|
||||
"""A parked ``import_sources`` double for the app-side edges.
|
||||
|
||||
It records the two position closures the runner passes (the wiring
|
||||
pin) and fires NOTHING — which is exactly what a run with no
|
||||
importable file does (no scan report, so no position, and no
|
||||
``progress`` call to flip the phase; the importer-side proof lives in
|
||||
the empty-KB arm below and in ``test_sync_pending_count.py``). The
|
||||
park lets the test read the status inside the ``"scan"`` phase without
|
||||
racing the pipeline; the caller always releases it in ``finally``.
|
||||
"""
|
||||
|
||||
def __init__(self, summary: ImportSummary | None = None) -> None:
|
||||
self.summary = summary if summary is not None else ImportSummary()
|
||||
self.started = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.progress_hook: Callable[[str, str, int, int], None] | None = None
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: LLMClient,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
) -> ImportSummary:
|
||||
self.progress_hook = progress
|
||||
self.scan_hook = on_scan_progress
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
def _install_counting_memory(monkeypatch: pytest.MonkeyPatch) -> list[dict[str, Any]]:
|
||||
"""Replace the two memory helpers on ``app.api.sync`` with recorders.
|
||||
|
||||
``_sample_rss`` calls them ONLY when the throttle lets it, so one
|
||||
entry per entry below is one real read — and each entry carries the
|
||||
phase (and the scan/file position) at that instant, which is the
|
||||
difference between "a transition forced a read" and "a per-file hook
|
||||
read once per file" (LOCKED A4)."""
|
||||
reads: list[dict[str, Any]] = []
|
||||
|
||||
def _record() -> None:
|
||||
reads.append(
|
||||
{
|
||||
"phase": sync_api._status.phase,
|
||||
"scan_done": sync_api._status.scan_done,
|
||||
"files_done": sync_api._status.files_done,
|
||||
"current_file": sync_api._status.current_file,
|
||||
}
|
||||
)
|
||||
|
||||
def fake_rss_mb() -> int:
|
||||
_record()
|
||||
return 100
|
||||
|
||||
def fake_rss_peak_mb() -> int:
|
||||
return 120
|
||||
|
||||
monkeypatch.setattr(sync_api, "rss_mb", fake_rss_mb)
|
||||
monkeypatch.setattr(sync_api, "rss_peak_mb", fake_rss_peak_mb)
|
||||
return reads
|
||||
|
||||
|
||||
def _slow_hash(
|
||||
monkeypatch: pytest.MonkeyPatch, block: float = HASH_BLOCK_S
|
||||
) -> dict[str, bool]:
|
||||
"""Make ONLY the pre-scan window slow, and return its flag.
|
||||
|
||||
``_file_hash`` is called by the scan AND by the processing loop, so a
|
||||
blanket blocking stub would also park the event loop through the whole
|
||||
import (an in-process import never awaits real I/O, so the loop cannot
|
||||
answer a status poll inside it) — the frames this file needs would
|
||||
then be unservable rather than absent. The flag is raised around
|
||||
:func:`app.rag.importer._prescan_pending` only, so the scan's window is
|
||||
the fixture's window: the loop stays free exactly while the run is in
|
||||
the phase under test (the phase-179 off-loop claim, observed from the
|
||||
HTTP side), and the import after it runs at full speed.
|
||||
|
||||
The flag is a plain ``dict`` entry written/read from two threads: the
|
||||
house hook contract (a GIL-atomic store) applies here for the same
|
||||
reason it applies to the runner's closures.
|
||||
"""
|
||||
real_hash = importer._file_hash
|
||||
real_prescan = importer._prescan_pending
|
||||
state = {"in_scan": False}
|
||||
|
||||
def slow(path: Path) -> str:
|
||||
if state["in_scan"]:
|
||||
time.sleep(block)
|
||||
return real_hash(path)
|
||||
|
||||
def spy_prescan(*args: Any, **kwargs: Any) -> Any:
|
||||
state["in_scan"] = True
|
||||
try:
|
||||
return real_prescan(*args, **kwargs)
|
||||
finally:
|
||||
state["in_scan"] = False
|
||||
|
||||
monkeypatch.setattr(importer, "_file_hash", slow)
|
||||
monkeypatch.setattr(importer, "_prescan_pending", spy_prescan)
|
||||
return state
|
||||
|
||||
|
||||
class _ChoreographedImport:
|
||||
"""An ``import_sources`` double that replays the importer's hook script
|
||||
through the runner's OWN closures, parking at every point the test
|
||||
reads the status.
|
||||
|
||||
The choreography (all four stops are ``threading.Event`` pairs, so no
|
||||
test here races the pipeline):
|
||||
|
||||
1. fire ``scan_pairs[:1]`` through ``on_scan_progress`` → ``first``
|
||||
(the test reads the scan's FIRST position);
|
||||
2. on ``go_on``: fire the rest → ``scanned`` (the test reads the last
|
||||
position, still inside the phase);
|
||||
3. on ``go_import``: fire ``progress_call`` through ``progress`` —
|
||||
THE FLIP → ``flipped`` (the test reads the ``"import"`` frame);
|
||||
4. on ``release``: return the canned summary.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
summary: ImportSummary,
|
||||
scan_pairs: list[tuple[int, int]],
|
||||
progress_call: tuple[str, str, int, int],
|
||||
) -> None:
|
||||
self.summary = summary
|
||||
self.scan_pairs = list(scan_pairs)
|
||||
self.progress_call = progress_call
|
||||
self.first = threading.Event()
|
||||
self.go_on = threading.Event()
|
||||
self.scanned = threading.Event()
|
||||
self.go_import = threading.Event()
|
||||
self.flipped = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
self.progress_hook: Callable[[str, str, int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: LLMClient,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
) -> ImportSummary:
|
||||
# The wiring pin: both closures arrive live (None here would mean
|
||||
# the runner stopped passing them).
|
||||
self.scan_hook = on_scan_progress
|
||||
self.progress_hook = progress
|
||||
assert on_scan_progress is not None and progress is not None
|
||||
for pair in self.scan_pairs[:1]:
|
||||
on_scan_progress(*pair)
|
||||
self.first.set()
|
||||
await asyncio.to_thread(self.go_on.wait)
|
||||
for pair in self.scan_pairs[1:]:
|
||||
on_scan_progress(*pair)
|
||||
self.scanned.set()
|
||||
await asyncio.to_thread(self.go_import.wait)
|
||||
progress(*self.progress_call)
|
||||
self.flipped.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
def _stub_probe(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
async def fake_check_models(llm: LLMClient) -> None:
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(sync_api, "check_models", fake_check_models)
|
||||
|
||||
|
||||
def _install_local_run(
|
||||
monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path, source: Path
|
||||
) -> None:
|
||||
"""Point one app run at *source*: local row, tmp sources root, stubbed
|
||||
probe, the deterministic in-process embedder (the real import, no
|
||||
network)."""
|
||||
_seed_local(db, source)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api, "get_settings", lambda: _settings(sources_dir=str(tmp_path / "bor"))
|
||||
)
|
||||
_stub_probe(monkeypatch)
|
||||
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder())
|
||||
|
||||
|
||||
_THREE_FILES = {
|
||||
"a.md": "# A\n\nalpha content\n",
|
||||
"b.md": "# B\n\nbravo content\n",
|
||||
"notes/c.md": "# C\n\ncharlie content\n",
|
||||
}
|
||||
|
||||
|
||||
def _log_lines(caplog: pytest.LogCaptureFixture, prefix: str) -> list[str]:
|
||||
return [r.getMessage() for r in caplog.records if r.getMessage().startswith(prefix)]
|
||||
|
||||
|
||||
# --- the phase-machine sequence --------------------------------------------
|
||||
|
||||
|
||||
def test_real_run_reports_the_scan_frames_and_keeps_both_pairs(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""D3 through the REAL pipeline, twice (one changed file included).
|
||||
|
||||
Run 1 (a first import): the prelude reports ``phase: null`` with
|
||||
``scan_*`` 0/0, then the run reports ``"scan"`` with ``scan_total`` at
|
||||
the importable count and ``scan_done`` climbing through AT LEAST two
|
||||
distinct positions (the frames the ``Scanning… (n/total)`` label
|
||||
needs) with ``current_file`` null throughout — the scan names no file.
|
||||
The terminal clears ``phase`` and KEEPS the final scan pair beside the
|
||||
file pair, and on a first import both pairs are 3/3 (A1's degenerate
|
||||
parity: walk == work, the only shape the phase-193 suite ever saw).
|
||||
|
||||
Run 2 (one file edited — the owner's prod shape at 1-of-3): the scan
|
||||
denominator is STILL the importable 3, while the kept file pair is
|
||||
``1/1`` and the terminal reports exactly one updated document. That
|
||||
divergence — the scan counted the walk, the count counted the work —
|
||||
IS the phase, and it survives on the terminal where no poll can miss
|
||||
it."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}machine", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_slow_hash(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
idle = sync_client.get("/api/sync/status").json()
|
||||
assert idle["phase"] is None
|
||||
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
frames, _late = _watch_scan(sync_client)
|
||||
scan_frames = [f for f in frames if f["phase"] == "scan"]
|
||||
assert scan_frames, f"the run never reported the scan phase: {frames}"
|
||||
assert all(f["current_file"] is None for f in scan_frames), (
|
||||
"the scan named a file — current_file stays null until the first "
|
||||
"progress call flips the phase to import"
|
||||
)
|
||||
# ``0`` is the transition frame — the phase is set before the scan
|
||||
# reports its first file, and the UI's total===0 guard renders it bare.
|
||||
totals = {f["scan_total"] for f in scan_frames}
|
||||
assert totals <= {0, 3} and max(totals) == 3, totals
|
||||
positions = sorted({f["scan_done"] for f in scan_frames if f["scan_total"] == 3})
|
||||
assert len(positions) >= 2, (
|
||||
f"only scan position(s) {positions} were observable — the label needs "
|
||||
"the scan's own n/total to move (the fixture scan blocks on purpose)"
|
||||
)
|
||||
assert positions[0] >= 1 and positions[-1] <= 3
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
assert body["detail"]["added"] == 3
|
||||
assert body["phase"] is None and body["current_file"] is None
|
||||
# The keep-final-values convention, beside the file counts.
|
||||
assert body["scan_total"] == 3 and body["scan_done"] == 3
|
||||
assert body["files_total"] == 3 and body["files_done"] == 3
|
||||
|
||||
# Run 2 — one changed file of three (the owner's prod case, scaled).
|
||||
(source / "b.md").write_text("# B\n\nbravo content EDITED\n", encoding="utf-8")
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
frames2, _late2 = _watch_scan(sync_client)
|
||||
scan2 = [f for f in frames2 if f["phase"] == "scan"]
|
||||
totals2 = {f["scan_total"] for f in scan2 if f["scan_total"]}
|
||||
assert scan2 and totals2 == {3}, (
|
||||
"the scan denominator is the importable count, not the prediction"
|
||||
)
|
||||
assert sorted({f["scan_done"] for f in scan2 if f["scan_total"] == 3})[-1] >= 2
|
||||
|
||||
body2 = _poll(sync_client, "success")
|
||||
assert (body2["detail"]["added"], body2["detail"]["updated"]) == (0, 1)
|
||||
assert body2["phase"] is None
|
||||
# The divergence the phase exists for: the kept scan pair is the WALK
|
||||
# it did (3 files hashed), the kept file pair is the WORK (1 file).
|
||||
assert (body2["scan_done"], body2["scan_total"]) == (3, 3)
|
||||
assert (body2["files_done"], body2["files_total"]) == (1, 1)
|
||||
|
||||
|
||||
def test_scan_flips_to_import_on_the_first_progress_call(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The sequence, frame by frame, with the flip pinned.
|
||||
|
||||
The in-process import runs straight through without awaiting real I/O,
|
||||
so a poll cannot reliably land inside the ``"import"`` phase (that is
|
||||
why the real arm above pins the terminal's kept pairs). This arm drives
|
||||
the runner's OWN closures through a choreographed double — the
|
||||
``test_sync_api`` gated-import idiom — and reads the status at each
|
||||
stop: prelude ``null``/0/0 → ``"scan"`` at position 1 of 3 → still
|
||||
``"scan"`` at position 3 of 3 → ``"import"`` with the PREDICTED
|
||||
``files_total`` (1, the count that reaches ``N remaining``) and the
|
||||
scan pair still showing 3/3 → the terminal, which clears ``phase`` and
|
||||
keeps both pairs."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}flip", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
gated = _ChoreographedImport(
|
||||
ImportSummary(files=3, updated=1, unchanged=2),
|
||||
scan_pairs=[(1, 3), (2, 3), (3, 3)],
|
||||
progress_call=(source.name, "b.md", 1, 1),
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "import_sources", gated)
|
||||
|
||||
_login(sync_client)
|
||||
idle = sync_client.get("/api/sync/status").json()
|
||||
assert idle["state"] == "idle"
|
||||
assert idle["phase"] is None
|
||||
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert gated.first.wait(10.0), "the run never reached the scan"
|
||||
s1 = sync_client.get("/api/sync/status").json()
|
||||
assert s1["state"] == "running"
|
||||
assert s1["phase"] == "scan"
|
||||
assert (s1["scan_done"], s1["scan_total"]) == (1, 3)
|
||||
assert s1["current_file"] is None
|
||||
# Still the pre-import phase: no file count yet.
|
||||
assert (s1["files_done"], s1["files_total"]) == (0, 0)
|
||||
|
||||
gated.go_on.set()
|
||||
assert gated.scanned.wait(10.0), "the scan never reported its last file"
|
||||
s2 = sync_client.get("/api/sync/status").json()
|
||||
assert s2["phase"] == "scan"
|
||||
assert (s2["scan_done"], s2["scan_total"]) == (3, 3)
|
||||
|
||||
gated.go_import.set()
|
||||
assert gated.flipped.wait(10.0), "the progress hook never fired"
|
||||
s3 = sync_client.get("/api/sync/status").json()
|
||||
assert s3["phase"] == "import", (
|
||||
"the first progress call must flip scan -> import: "
|
||||
f"{ {k: s3[k] for k in ('phase', 'current_file', 'files_done', 'files_total')} }"
|
||||
)
|
||||
assert s3["current_file"] == f"{source.name}/b.md"
|
||||
# The flip already carries the PREDICTED denominator — the number
|
||||
# the label turns into "N remaining".
|
||||
assert (s3["files_done"], s3["files_total"]) == (1, 1)
|
||||
# … and the scan pair stays where the scan left it.
|
||||
assert (s3["scan_done"], s3["scan_total"]) == (3, 3)
|
||||
finally:
|
||||
gated.go_on.set()
|
||||
gated.go_import.set()
|
||||
gated.release.set()
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
assert body["phase"] is None and body["current_file"] is None
|
||||
assert (body["scan_done"], body["scan_total"]) == (3, 3)
|
||||
assert (body["files_done"], body["files_total"]) == (1, 1)
|
||||
|
||||
|
||||
# --- the additive payload ----------------------------------------------------
|
||||
|
||||
|
||||
def test_scan_keys_are_additive_and_stable_from_idle_to_terminal(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The additive-only contract on the wire (the phase-187 discipline).
|
||||
|
||||
The payload's key set is EXACTLY the pre-phase-194 set plus
|
||||
``scan_done`` / ``scan_total``; it does not change between idle,
|
||||
mid-run and the terminal (no key appears or disappears with the
|
||||
phase); and both new keys are plain ``int`` — 0 when there is nothing
|
||||
to report, never ``null`` and never a plausible position."""
|
||||
expected_pre = {
|
||||
"state",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"detail",
|
||||
"error",
|
||||
"current_file",
|
||||
"files_done",
|
||||
"files_total",
|
||||
"current_chunk",
|
||||
"chunks_total",
|
||||
"images_done",
|
||||
"images_total",
|
||||
"phase",
|
||||
"current_summary",
|
||||
"summaries_done",
|
||||
"summaries_total",
|
||||
"rss_mb",
|
||||
"rss_peak_mb",
|
||||
}
|
||||
source = _local_source(tmp_path, f"{PREFIX}keys", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_stub_probe(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
idle = sync_client.get("/api/sync/status").json()
|
||||
assert set(idle) == expected_pre | {"scan_done", "scan_total"}, set(idle)
|
||||
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
mid = sync_client.get("/api/sync/status").json()
|
||||
assert set(mid) == expected_pre | {"scan_done", "scan_total"}
|
||||
assert mid["state"] == "running"
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
assert set(body) == expected_pre | {"scan_done", "scan_total"}
|
||||
for key in ("scan_done", "scan_total"):
|
||||
assert isinstance(body[key], int) and not isinstance(body[key], bool), body
|
||||
# Every pre-existing key keeps its type too (the additive pin's other
|
||||
# half — the file pair is still ints, the phase still a string|null).
|
||||
assert isinstance(body["files_done"], int) and isinstance(body["files_total"], int)
|
||||
assert body["phase"] is None
|
||||
|
||||
|
||||
# --- the sampling discipline ------------------------------------------------
|
||||
|
||||
|
||||
def test_scan_and_import_transitions_force_the_sample_and_the_scan_hook_does_not(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 187's rule, extended to the new stage (LOCKED A4).
|
||||
|
||||
Every entry below is one REAL ``/proc`` read (the two helpers are only
|
||||
called when the throttle lets ``_sample_rss`` through), tagged with
|
||||
the phase it happened in. Collapsing consecutive same-phase reads, the
|
||||
sequence is one read per transition: run start (``phase`` still null)
|
||||
→ ``"scan"`` → ``"import"`` (the flip) → ``"overview"`` →
|
||||
``"summaries"`` → the terminal (null again). The scan transition's
|
||||
read carries ``scan_done == 0`` — it is the transition, not a file —
|
||||
and there is EXACTLY one read inside the whole scan phase, so a
|
||||
per-file scan read (one ``/proc`` read per file, the regression this
|
||||
pins) fails here even though the fixture scanned three files."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}sampling", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_slow_hash(monkeypatch)
|
||||
reads = _install_counting_memory(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
assert reads, "the run sampled no memory at all"
|
||||
phases = [r["phase"] for r in reads]
|
||||
transitions = [p for i, p in enumerate(phases) if i == 0 or p != phases[i - 1]]
|
||||
assert transitions == [None, "scan", "import", "overview", "summaries", None], reads
|
||||
|
||||
scan_reads = [r for r in reads if r["phase"] == "scan"]
|
||||
assert len(scan_reads) == 1, (
|
||||
f"the scan phase cost {len(scan_reads)} /proc reads for 3 files — the "
|
||||
"per-file scan hook must sample UNFORCED (one read per interval)"
|
||||
)
|
||||
assert scan_reads[0]["scan_done"] == 0, (
|
||||
"the scan-phase read is not the transition — the forced sample must "
|
||||
"land when the phase is SET, before the first file is scanned"
|
||||
)
|
||||
flip_reads = [r for r in reads if r["phase"] == "import"]
|
||||
assert flip_reads and flip_reads[0]["current_file"] is not None, (
|
||||
"no read at the scan->import flip — the transition is a forced "
|
||||
f"sampling point: {reads}"
|
||||
)
|
||||
# The payload reports what the sampler wrote, not a stale pair.
|
||||
assert (body["rss_mb"], body["rss_peak_mb"]) == (100, 120), body
|
||||
|
||||
|
||||
# --- the degenerate empty knowledge base ------------------------------------
|
||||
|
||||
|
||||
def test_empty_kb_stays_in_scan_and_never_reaches_import(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The D1 degenerate clause, pinned on BOTH sides.
|
||||
|
||||
Importer side (real): an empty source dir makes the pre-scan report
|
||||
NOTHING — no ``on_scan_progress`` call, no ``progress`` call, a zero
|
||||
denominator. App side (the double that fires nothing, i.e. exactly
|
||||
what that run reports): the status sits in ``"scan"`` — the flip
|
||||
lives in the per-file hook and no hook ever fires — with ``scan_*``
|
||||
0/0 and ``current_file`` null, and the terminal clears ``phase``
|
||||
while keeping the 0/0 pair (the UI's ``total === 0`` guard then
|
||||
renders the bare label with no count part)."""
|
||||
# Importer side: the real scan of an empty root reports nothing.
|
||||
empty = _local_source(tmp_path, f"{PREFIX}empty", {})
|
||||
llm = FakeEmbedder()
|
||||
scan_calls: list[tuple[int, int]] = []
|
||||
progress_calls: list[tuple[str, str, int, int]] = []
|
||||
summary = asyncio.run(
|
||||
importer.import_sources(
|
||||
[empty],
|
||||
llm,
|
||||
progress=lambda s, r, d, t: progress_calls.append((s, r, d, t)),
|
||||
on_scan_progress=lambda p, t: scan_calls.append((p, t)),
|
||||
)
|
||||
)
|
||||
assert (scan_calls, progress_calls) == ([], [])
|
||||
assert summary.files == 0
|
||||
|
||||
# App side: a run that reports nothing is a run that stays "scan".
|
||||
_install_local_run(monkeypatch, db, tmp_path, empty)
|
||||
fake_import = _FakeScanImport()
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert fake_import.started.wait(10.0), "the run never reached the import"
|
||||
# Wiring: the runner passed BOTH live closures (so the empty
|
||||
# phase above is the scan reporting nothing, not a missing hook).
|
||||
assert fake_import.scan_hook is not None
|
||||
assert fake_import.progress_hook is not None
|
||||
body = sync_client.get("/api/sync/status").json()
|
||||
assert body["state"] == "running"
|
||||
assert body["phase"] == "scan"
|
||||
assert body["current_file"] is None
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
assert body["files_done"] == 0 and body["files_total"] == 0
|
||||
finally:
|
||||
fake_import.release.set()
|
||||
|
||||
terminal = _poll(sync_client, "success")
|
||||
assert terminal["phase"] is None
|
||||
assert terminal["scan_done"] == 0 and terminal["scan_total"] == 0
|
||||
assert terminal["files_done"] == 0 and terminal["files_total"] == 0
|
||||
|
||||
|
||||
# --- the cancel during the pre-scan -----------------------------------------
|
||||
|
||||
|
||||
def test_cancel_during_the_scan_lands_cancelled_with_the_scan_counts_kept(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The phase-134 contract at the pipeline's NEW await (the app-side
|
||||
twin of the importer-side cancel pin).
|
||||
|
||||
The scan is blocking on purpose, so the cancel provably lands inside
|
||||
it: ``POST /api/sync/cancel`` answers, the ``cancelled`` terminal
|
||||
arrives well before the abandoned scan's remaining work could finish,
|
||||
``phase`` clears, the position the scan had reached is KEPT beside the
|
||||
(still zero) file pair, ``error`` stays null and ``detail`` empty (a
|
||||
cancel is intent, not failure), and NOTHING was indexed — the walk
|
||||
never started. The lifecycle line keeps its exact pre-phase prefix
|
||||
(LOCKED A2's append-only rule)."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}cancel", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_slow_hash(monkeypatch, block=0.4)
|
||||
|
||||
_login(sync_client)
|
||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
# Wait for proof the run is INSIDE the scan: the phase, and a
|
||||
# position that moved.
|
||||
deadline = time.monotonic() + 20.0
|
||||
live: dict = {}
|
||||
while time.monotonic() < deadline:
|
||||
live = sync_client.get("/api/sync/status").json()
|
||||
if live["phase"] == "scan" and live["scan_done"] >= 1:
|
||||
break
|
||||
assert live["state"] == "running", live
|
||||
time.sleep(0.01)
|
||||
assert live.get("phase") == "scan" and live["scan_done"] >= 1, (
|
||||
f"the cancel never landed inside the scan: {live}"
|
||||
)
|
||||
position_at_cancel = live["scan_done"]
|
||||
|
||||
t0 = time.monotonic()
|
||||
r = sync_client.post("/api/sync/cancel")
|
||||
assert r.status_code == 202, f"cancel not accepted: {r.status_code} {r.text}"
|
||||
assert time.monotonic() - t0 < 2.0, "the cancel request itself stalled"
|
||||
|
||||
body = _poll(sync_client, "cancelled", timeout=3.0)
|
||||
|
||||
assert body["error"] is None
|
||||
assert body["detail"] == {}
|
||||
assert body["phase"] is None
|
||||
assert body["current_file"] is None
|
||||
# The scan's final position is kept (the keep-final-values
|
||||
# convention), and it is the position the cancel found — the run did
|
||||
# not finish the scan it was cancelled in.
|
||||
assert body["scan_total"] == 3
|
||||
assert body["scan_done"] >= position_at_cancel
|
||||
assert body["files_done"] == 0 and body["files_total"] == 0
|
||||
|
||||
# Nothing was indexed: the walk never started.
|
||||
assert db.scalar(
|
||||
select(select(Document).where(Document.source == source.name).exists())
|
||||
) is False
|
||||
|
||||
# The lifecycle line keeps its pre-phase-194 prefix exactly.
|
||||
cancelled = _log_lines(caplog, "sync: cancelled by user state=cancelled files_done=")
|
||||
assert len(cancelled) == 1, cancelled
|
||||
|
||||
|
||||
# --- the kill switch's status shape ----------------------------------------
|
||||
|
||||
|
||||
def test_kill_switch_off_reports_no_scan_position_but_the_same_machine(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""``BOR_SYNC_PENDING_COUNT=0`` on the STATUS side.
|
||||
|
||||
The importer takes the phase-193 stats pre-walk, which reports NOTHING,
|
||||
so the scan pair stays 0/0 for the whole run (the page renders no
|
||||
count from it) while ``files_done`` / ``files_total`` carry the WALK
|
||||
position again — 3 walked, not 1 predicted. The phase machine itself is
|
||||
UNCHANGED by the switch (the pre-``import_sources`` set is still
|
||||
``"scan"`` and the first file still flips it: that flip is pinned in
|
||||
the choreographed test above, and the switch's importer-side halves are
|
||||
pinned in ``tests/unit/test_sync_pending_count.py`` and
|
||||
``tests/integration/test_sync_pending_count.py``). The pins below read
|
||||
the TERMINAL, which is where both pairs survive deterministically."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}off", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
# The switch lives on the settings object the import reads (llm).
|
||||
real_embedder = FakeEmbedder()
|
||||
real_embedder.settings = Settings(_env_file=None, sync_pending_count=0) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(sync_api, "LLMClient", lambda: real_embedder)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
assert body["detail"]["added"] == 3
|
||||
# No scan, so no position — kept at 0/0, the value the UI renders as
|
||||
# "no count part" rather than a stale or invented one.
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
# The WALK count is back (phase-193 semantics, byte-for-byte).
|
||||
assert (body["files_done"], body["files_total"]) == (3, 3)
|
||||
assert body["phase"] is None
|
||||
@@ -1027,3 +1027,33 @@ def test_sync_prune_batch_rejects_negative(
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_prune_batch"):
|
||||
_settings()
|
||||
|
||||
|
||||
# --- Sync pending count (phase 194) ---
|
||||
|
||||
|
||||
def test_sync_pending_count_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 194 (D2): the pre-scan ships ON (``1``) — the owner's "1216
|
||||
remaining" report means the walk-position count was wrong by default, so
|
||||
the fix is the default and the rollback is the opt-in. ``0`` is accepted
|
||||
and means the exact phase-193 behaviour: count every importable file of
|
||||
the walk and never hash a file before the loop reached it."""
|
||||
monkeypatch.delenv("BOR_SYNC_PENDING_COUNT", raising=False)
|
||||
assert _settings().sync_pending_count == 1
|
||||
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "0")
|
||||
assert _settings().sync_pending_count == 0
|
||||
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "1")
|
||||
assert _settings().sync_pending_count == 1
|
||||
|
||||
|
||||
def test_sync_pending_count_rejects_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A negative switch is a typo — no branch of the importer can mean it,
|
||||
so startup refuses naming the field (the ``agent_max_rounds`` pattern,
|
||||
phase 194). The value is a boolean knob, so ONLY ``0`` is the rollback."""
|
||||
monkeypatch.setenv("BOR_SYNC_PENDING_COUNT", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_pending_count"):
|
||||
_settings()
|
||||
|
||||
@@ -1146,8 +1146,16 @@ def test_no_progress_means_no_prewalk(
|
||||
) -> None:
|
||||
"""``progress=None`` callers pay no extra pass: the real walker is hit
|
||||
exactly once per source root (one pass — as before phase 64), proven
|
||||
with a counting sentinel; with the hook it is hit twice (pre-walk for
|
||||
``total`` + the processing pass)."""
|
||||
with a counting sentinel; with the hook it is hit twice (the phase-194
|
||||
pre-scan for the pending set — the phase-64 stats pre-walk when
|
||||
``BOR_SYNC_PENDING_COUNT=0`` — plus the processing pass).
|
||||
|
||||
Phase 194 (task 01) moved the hook's pair, and this fixture is the
|
||||
second run over an ALREADY-INDEXED file: nothing is pending, so the
|
||||
pair is ``(0, 0)`` — the denominator is the predicted change set and an
|
||||
unchanged KB predicts nothing (the UI then renders no count part). The
|
||||
walk COUNT is what this test owns and it is untouched: one pass without
|
||||
the hook, two with it."""
|
||||
root = tmp_path / "nowalk"
|
||||
root.mkdir()
|
||||
(root / "a.md").write_text("# A\n\na\n")
|
||||
@@ -1186,8 +1194,12 @@ def test_no_progress_means_no_prewalk(
|
||||
)
|
||||
)
|
||||
assert summary2.files == 1
|
||||
assert walk_calls == 2 # pre-walk (total) + processing pass
|
||||
assert events == [(root.name, "a.md", 1, 1)]
|
||||
assert walk_calls == 2 # the count pass (pending set / pre-walk) + walk
|
||||
# phase 194: a.md is unchanged, so the predicted change set is empty
|
||||
# — ``total`` 0 and the position never advances. With the kill switch
|
||||
# at ``0`` this same run reports the phase-193 walk position (1 of 1);
|
||||
# pinned in tests/unit/test_sync_pending_count.py.
|
||||
assert events == [(root.name, "a.md", 0, 0)]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -1195,7 +1207,17 @@ def test_no_progress_means_no_prewalk(
|
||||
def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> None:
|
||||
"""The hook fires *before* ``_index_file``: a file whose embedding
|
||||
fails (and one that re-imports as unchanged) is still reported as the
|
||||
current file — the sequence covers every importable file."""
|
||||
current file — the sequence covers every importable file.
|
||||
|
||||
Phase 194 (task 01) retargeted the pair to the PENDING position (the
|
||||
owner's "1216 remaining" fix), which reads exactly right here. The
|
||||
first run is fresh: both files are predicted, ``1 of 2`` then ``2 of 2``
|
||||
— identical to the pre-phase numbers (A1's first-import parity). The
|
||||
second run is the incremental case: ``bad.md`` rolled back with its
|
||||
embed error so its row is still missing (predicted again, the ONLY
|
||||
work), ``good.md`` is unchanged so it holds the position instead of
|
||||
advancing it — ``1 of 1``, ``1 of 1``: every file is still reported as
|
||||
the current file, and the count is the work."""
|
||||
root = tmp_path / "progress-mixed"
|
||||
root.mkdir()
|
||||
(root / "bad.md").write_text("# Bad\n\npoison content the endpoint refuses\n")
|
||||
@@ -1214,7 +1236,8 @@ def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> N
|
||||
assert events == expected
|
||||
assert first.errors == 1 and first.added == 1
|
||||
# Re-run: good.md is now unchanged, bad.md is retried and fails
|
||||
# again — both still count in the sequence.
|
||||
# again — both still report as the current file, but only bad.md is
|
||||
# pending, so the denominator is 1 and good.md holds the position.
|
||||
events.clear()
|
||||
second = asyncio.run(
|
||||
import_sources(
|
||||
@@ -1222,7 +1245,7 @@ def test_progress_hook_counts_unchanged_and_error_files(db, tmp_path: Path) -> N
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
)
|
||||
)
|
||||
assert events == expected
|
||||
assert events == [(root.name, "bad.md", 1, 1), (root.name, "good.md", 1, 1)]
|
||||
assert second.errors == 1 and second.unchanged == 1
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -253,6 +253,29 @@ def test_sync_session_bound_field_types() -> None:
|
||||
assert Settings.model_fields["sync_prune_batch"].default == 500
|
||||
|
||||
|
||||
def test_sync_pending_count_row_is_a_plain_live_overridable_row() -> None:
|
||||
"""Phase 194 (task 01, D2): the pre-scan switch sits in ``Import &
|
||||
sources`` beside the other two sync-run knobs with the ``Sync pending
|
||||
count`` label and is plain — not ``secret`` (an int), not
|
||||
``restart_required`` (the importer reads ``llm.settings`` once per run, so
|
||||
a saved override applies to the NEXT sync), overridable (the overlay is
|
||||
generic over ``int`` fields). ``1`` is the shipped default — the pending
|
||||
count is the FIX the owner asked for, so it ships on and the phase-193
|
||||
walk position is the ``0`` rollback; a negative value fails startup, not
|
||||
the panel."""
|
||||
meta = CATALOG["sync_pending_count"]
|
||||
assert meta.group == "Import & sources"
|
||||
assert meta.label == "Sync pending count"
|
||||
assert meta.secret is False
|
||||
assert meta.restart_required is False
|
||||
assert meta.overridable is True
|
||||
assert "sync_pending_count" not in SECRETS
|
||||
assert "sync_pending_count" not in RESTART_REQUIRED
|
||||
assert "sync_pending_count" not in NON_OVERRIDABLE
|
||||
assert Settings.model_fields["sync_pending_count"].annotation is int
|
||||
assert Settings.model_fields["sync_pending_count"].default == 1
|
||||
|
||||
|
||||
def test_source_name_hit_bonus_row_is_a_plain_live_overridable_row() -> None:
|
||||
"""Phase 191 (task 04, D3): the second selection-bonus row sits in
|
||||
``Retrieval & honesty`` (right beside ``name_hit_bonus`` — the two
|
||||
|
||||
@@ -1419,6 +1419,10 @@ class _GatedImport:
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 194 (task 02): the pre-scan position hook the runner
|
||||
# passes (recorded for the wiring pin — this fake fires nothing,
|
||||
# so the status scan fields stay 0/0).
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -1431,11 +1435,13 @@ class _GatedImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
self.prune_flags.append(prune)
|
||||
self.scan_hook = on_scan_progress
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
@@ -1597,6 +1603,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
|
||||
# (the dataclass defaults; a run that scanned nothing keeps them
|
||||
# 0/0 rather than null).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"current_chunk": 0,
|
||||
@@ -2131,6 +2142,7 @@ class _TwoFileImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -2804,6 +2816,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -3170,7 +3183,8 @@ def test_sources_js_section_header_documents_the_cancel_contract() -> None:
|
||||
|
||||
#: The status payload's key set BEFORE phase 187 — the additive pin's
|
||||
#: left-hand side: the payload is exactly this set plus
|
||||
#: :data:`_MEMORY_KEYS`, nothing else, forever.
|
||||
#: :data:`_MEMORY_KEYS` plus :data:`_SCAN_KEYS`, nothing else, forever
|
||||
#: (a future field joins the chain, it never edits these three).
|
||||
_PRE_187_STATUS_KEYS = frozenset(
|
||||
{
|
||||
"state",
|
||||
@@ -3195,17 +3209,23 @@ _PRE_187_STATUS_KEYS = frozenset(
|
||||
#: Phase 187 (task 03): the two additive memory keys.
|
||||
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the next
|
||||
#: link of the same additive chain.
|
||||
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
|
||||
def test_status_key_set_is_the_old_set_plus_the_memory_pair(
|
||||
fresh_sync_status: None,
|
||||
) -> None:
|
||||
"""The key-set pin (D4: "Additive only — every pre-existing key
|
||||
keeps its name, type, and meaning"): the payload is EXACTLY the
|
||||
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb}`` — a fifth sub-position
|
||||
or a renamed field fails here even when the value assertions pass."""
|
||||
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb} ∪ {scan_done,
|
||||
scan_total}`` — a fifth sub-position or a renamed field fails here
|
||||
even when the value assertions pass."""
|
||||
keys = set(sync_api.sync_status())
|
||||
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, keys
|
||||
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, keys
|
||||
assert _MEMORY_KEYS.isdisjoint(_PRE_187_STATUS_KEYS)
|
||||
assert _SCAN_KEYS.isdisjoint(_PRE_187_STATUS_KEYS | _MEMORY_KEYS)
|
||||
|
||||
|
||||
class _FakeClock:
|
||||
@@ -3377,9 +3397,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The sampling POINTS, walked through the real runner: run start →
|
||||
``import`` → ``overview`` → ``summaries`` → the success terminal, one
|
||||
forced read each, and the per-file progress hook's throttled call
|
||||
adds NOTHING inside the window the run start just armed (LOCKED A4).
|
||||
``scan`` → ``import`` (the phase-194 flip) → ``overview`` →
|
||||
``summaries`` → the success terminal, one forced read each, and the
|
||||
per-file progress hook's throttled call adds NOTHING inside the window
|
||||
the previous transition just armed (LOCKED A4 — and the same rule the
|
||||
pre-scan's own per-file hook follows, pinned through the app in
|
||||
``tests/integration/test_sync_scan_phase.py``).
|
||||
|
||||
The last pair is the number the incident was about: the terminal
|
||||
KEEPS it (the phase-64 keep-final-values convention) and the
|
||||
@@ -3388,10 +3411,12 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
log an operator could be argued with."""
|
||||
clock = _FakeClock(10_000.0)
|
||||
monkeypatch.setattr(sync_api, "time", clock)
|
||||
# run start, import, overview, summaries, done (the hook is throttled).
|
||||
# run start, scan, import (the flip), overview, summaries, done — the
|
||||
# two phase-194 transitions join the list; the per-file hooks stay
|
||||
# throttled and add no read.
|
||||
memory = _StubMemory(
|
||||
monkeypatch,
|
||||
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 4_321)],
|
||||
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 24), (15, 4_321)],
|
||||
)
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _GatedImport(
|
||||
@@ -3412,14 +3437,23 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (10, 20)
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the transition forced the second read, and
|
||||
# the hook's own call (this fake fires one file) is inside
|
||||
# the window — still 2 reads, not 3 (A4 in numbers).
|
||||
# Import phase: the "scan" set forced the second read and the
|
||||
# first progress call's FLIP forced the third, while the
|
||||
# hook's own unforced call (this fake fires one file) is
|
||||
# inside the window — 3 reads, not 4 (A4 in numbers).
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (11, 21)
|
||||
assert memory.reads == 2, "the per-file hook never bypasses the guard"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (12, 22)
|
||||
assert memory.reads == 3, (
|
||||
"the scan/import transitions must each force one read and "
|
||||
f"the per-file hook must never bypass the guard: {memory.reads}"
|
||||
)
|
||||
# The scan closure reached the runner wired (the fake fires
|
||||
# nothing, so the position stays 0/0 — the sequence itself is
|
||||
# pinned through the app in test_sync_scan_phase.py).
|
||||
assert fake_import.scan_hook is not None
|
||||
assert s["scan_done"] == 0 and s["scan_total"] == 0
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -3431,7 +3465,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
# and it survives the run (the page shows it after the settle).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (14, 4_321)
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (15, 4_321)
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
done = [
|
||||
@@ -3440,7 +3474,7 @@ def test_run_samples_at_start_phase_transitions_and_terminal(
|
||||
if r.getMessage().startswith("sync: done detail=")
|
||||
]
|
||||
assert len(done) == 1, done
|
||||
assert done[0].endswith("rss_mb=14 rss_peak_mb=4321"), done[0]
|
||||
assert done[0].endswith("rss_mb=15 rss_peak_mb=4321"), done[0]
|
||||
|
||||
|
||||
def test_per_file_hook_samples_once_the_window_closes(
|
||||
@@ -3452,10 +3486,15 @@ def test_per_file_hook_samples_once_the_window_closes(
|
||||
fires once per file, and once the 5 s window has closed the next
|
||||
file's call re-reads — so a long walk reports the memory climbing
|
||||
(PLAN §7.4's never-stale contract) at ~600 reads per 27k files
|
||||
instead of 27k."""
|
||||
instead of 27k (phase 194 extended the same rule to the pre-scan's
|
||||
per-file hook)."""
|
||||
clock = _FakeClock(10_000.0)
|
||||
monkeypatch.setattr(sync_api, "time", clock)
|
||||
memory = _StubMemory(monkeypatch, [(100, 200), (110, 210), (140, 260)])
|
||||
# run start, the "scan" set, the scan->import flip, then the read the
|
||||
# test forces after moving the clock past the window.
|
||||
memory = _StubMemory(
|
||||
monkeypatch, [(100, 200), (110, 210), (120, 220), (140, 260)]
|
||||
)
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=1, added=1),
|
||||
@@ -3474,8 +3513,11 @@ def test_per_file_hook_samples_once_the_window_closes(
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (110, 210) # the transition
|
||||
assert memory.reads == 2, "the hook's call inside the window read nothing"
|
||||
assert (s["rss_mb"], s["rss_peak_mb"]) == (120, 220) # the flip
|
||||
assert memory.reads == 3, (
|
||||
"start + scan + flip force three reads and the hook's own call "
|
||||
f"inside the window adds none: {memory.reads}"
|
||||
)
|
||||
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S + 1.0
|
||||
sync_api._sample_rss() # the NEXT file's hook call
|
||||
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (140, 260)
|
||||
|
||||
@@ -0,0 +1,700 @@
|
||||
"""Phase 194, task 01 — the importer's pre-scan + the pending-count switch.
|
||||
|
||||
The owner report that opened the phase (2026-10-19): *"I just synced the prod
|
||||
brain of reese and it said '1216 remaining' when it only had like 10 files to
|
||||
sync."* The arithmetic of the phase-193 label was fine; its INPUTS were not —
|
||||
``total`` was every importable file (prod: ~10,000) and ``done`` was the
|
||||
position in that walk, so the label counted the WALK instead of the WORK.
|
||||
Task 01 moves the pair to the PREDICTED change set: ``import_sources``
|
||||
pre-scans (one streamed sha256 + one scalar ``documents.file_hash`` read per
|
||||
importable file, off the event loop) and reports ``done``/``total`` as the
|
||||
position in the pending set, with ``BOR_SYNC_PENDING_COUNT`` as the kill
|
||||
switch (``0`` = the exact phase-193 walk position, negative = refuses
|
||||
startup).
|
||||
|
||||
Pinned here (the behaviour under a real run — the pending-set parity, the
|
||||
zero-hydration pin, the off-loop span and the cancel — lives in
|
||||
``tests/integration/test_sync_pending_count.py``):
|
||||
|
||||
* the GATE: no ``progress`` caller means no scan at all (the callback never
|
||||
fires and the scalar statement is never built — the phase-64 skip rule,
|
||||
extended), and the switch at ``0`` on a progress caller runs the phase-64
|
||||
stats pre-walk instead (a source marker pin: that old code IS the ``0``
|
||||
branch, and a behaviour pin on the pair it reports);
|
||||
* the switch is read ONCE per run (the phase-184 read-once discipline), on
|
||||
the worker-thread side of ``asyncio.to_thread`` (the phase-179 rule);
|
||||
* the scalar statement's projection is EXACTLY ``documents.file_hash`` — one
|
||||
scalar column, no entity, no ``defer``/``load_only`` row (the phase-185
|
||||
column-list style); :func:`_doc_row_stmt` stays the importer's only ROW
|
||||
load;
|
||||
* the pending PREDICTATE: fresh (no row), changed (digest mismatch) and a
|
||||
pre-phase ``NULL`` row are pending, a hash match is not, a file the ignore
|
||||
rule hides is never even walked, and a file that cannot be read is
|
||||
predicted changed rather than fatal;
|
||||
* ``on_scan_progress``: once per file, the 1-based position against the
|
||||
combined importable total, and NOTHING for an empty KB (the phase-64
|
||||
``total === 0`` contract);
|
||||
* ``done`` as the pending position: a 3-file KB with 2 predicted reports
|
||||
``1``, ``1``, ``2`` against ``total = 2`` — the file that is not predicted
|
||||
holds the position while the walk moves past it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import asyncio
|
||||
import inspect
|
||||
import re
|
||||
import threading
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.importer import (
|
||||
_pending_hash_stmt,
|
||||
_prescan_pending,
|
||||
import_sources,
|
||||
)
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _code_only(fn: Any) -> str:
|
||||
"""A function's source WITHOUT its docstring.
|
||||
|
||||
The marker pins below count occurrences of names in the code; the
|
||||
docstrings argue about those same names by name (the house convention), so
|
||||
counting the raw source would pin prose. The docstring is located through
|
||||
the AST and sliced out by its own end line.
|
||||
"""
|
||||
src = inspect.getsource(fn)
|
||||
lines = src.splitlines()
|
||||
node = ast.parse(src).body[0]
|
||||
# `import_sources` is an ASYNC def, so it parses to AsyncFunctionDef —
|
||||
# narrowing to FunctionDef alone would silently skip the docstring strip
|
||||
# and pin prose.
|
||||
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
return src
|
||||
first = node.body[0]
|
||||
if (
|
||||
isinstance(first, ast.Expr)
|
||||
and isinstance(first.value, ast.Constant)
|
||||
and isinstance(first.value.value, str)
|
||||
):
|
||||
return "\n".join(lines[first.end_lineno or 1 :])
|
||||
return src
|
||||
|
||||
|
||||
def _settings(**overrides: Any) -> Settings:
|
||||
"""Fresh settings, never the developer's ``.env`` (the house pattern)."""
|
||||
return Settings(_env_file=None, **overrides) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _root(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
|
||||
"""One source root with *files* (relative path → content) under it."""
|
||||
root = tmp_path / name
|
||||
for rel, body in files.items():
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _prescan(
|
||||
*sources: Path,
|
||||
session: Session,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
on_scan_progress: Any = None,
|
||||
image_extensions: frozenset[str] = frozenset(),
|
||||
) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
""":func:`_prescan_pending` with the shipped import extensions and an
|
||||
explicit session, so the predicate is testable without a full run."""
|
||||
return _prescan_pending(
|
||||
list(sources),
|
||||
_settings().import_extension_set,
|
||||
image_extensions,
|
||||
ignore_by_root,
|
||||
include_hidden_by_root,
|
||||
on_scan_progress,
|
||||
session,
|
||||
)
|
||||
|
||||
|
||||
def _seed_row(
|
||||
db: Session, source: str, rel: str, *, file_hash: str | None
|
||||
) -> None:
|
||||
"""One ``documents`` row with a KNOWN ``file_hash`` (the pre-scan reads
|
||||
that one column, so this is the whole fixture: raw SQL, no importer, no
|
||||
chunks)."""
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO documents (id, source, path, full_path, title, content, "
|
||||
"content_hash, file_hash, indexed_at, created_at, created_at_manual, "
|
||||
"summary, is_image) VALUES (:id, :source, :path, :full_path, :title, "
|
||||
":content, :content_hash, :file_hash, now(), now(), false, :summary, "
|
||||
"false)"
|
||||
),
|
||||
{
|
||||
"id": uuid.uuid4(),
|
||||
"source": source,
|
||||
"path": rel,
|
||||
"full_path": f"/tmp/{source}/{rel}",
|
||||
"title": rel,
|
||||
"content": "stored markdown",
|
||||
"content_hash": "content-digest",
|
||||
"file_hash": file_hash,
|
||||
"summary": "stored summary",
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_probe_rows(db: Session) -> Any:
|
||||
"""The probe's own source-name prefix, removed around every test (the
|
||||
tables are global state — the ``test_importer.py`` hygiene pattern)."""
|
||||
_drop(db)
|
||||
yield
|
||||
_drop(db)
|
||||
|
||||
|
||||
def _drop(db: Session) -> None:
|
||||
db.execute(
|
||||
text("DELETE FROM documents WHERE source LIKE 'phase194-%'"),
|
||||
)
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The gate: who pays for the scan
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_progress_caller_runs_no_scan_at_all(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE skip rule (phase 64), extended: ``progress=None`` → no pre-scan.
|
||||
|
||||
The CLI (``scripts/import_docs.py``) and the upload job pass no
|
||||
``progress``, so they must not pay one extra read + hash + SELECT per
|
||||
importable file. A recording stand-in for the span proves the gate
|
||||
itself: the CLI-shaped call imports the file and never reaches the
|
||||
scan, and an ``on_scan_progress`` hook handed to a ``progress=None``
|
||||
caller fires never (the scan exists only to serve the running counts).
|
||||
"""
|
||||
calls: list[tuple[Any, ...]] = []
|
||||
scan_events: list[tuple[int, int]] = []
|
||||
|
||||
def _spy(*args: Any, **kwargs: Any) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
calls.append(args)
|
||||
return 0, {}
|
||||
|
||||
monkeypatch.setattr(importer, "_prescan_pending", _spy)
|
||||
root = _root(tmp_path, "phase194-gate-cli", {"a.md": "# A\n\na\n"})
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
|
||||
assert summary.files == 1 and summary.added == 1
|
||||
assert calls == [], "a progress=None caller ran the pre-scan"
|
||||
|
||||
# The scan hook alone does not open the gate either.
|
||||
summary2 = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
FakeEmbedder(),
|
||||
session=db,
|
||||
on_scan_progress=lambda done, total: scan_events.append((done, total)),
|
||||
)
|
||||
)
|
||||
assert summary2.files == 1
|
||||
assert calls == []
|
||||
assert scan_events == []
|
||||
finally:
|
||||
db.execute(
|
||||
text("DELETE FROM documents WHERE source = :s"), {"s": root.name}
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_switch_zero_runs_the_phase_193_stats_prewalk(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The rollback branch, from both sides: the marker (the old stats
|
||||
pre-walk code IS the ``0`` branch) and the reported pair.
|
||||
|
||||
With the switch at ``0`` a progress caller never reaches the scan span,
|
||||
and the hook reports the phase-193 walk position byte-for-byte — every
|
||||
importable file in ``total``, the 1-based index across all sources in
|
||||
``done`` — on a run where nothing is pending. That last clause is the
|
||||
point of the arm: the SAME unchanged fixture reports ``(0, 0)`` with the
|
||||
switch on and ``(1, 1)``, ``(2, 2)`` with it off.
|
||||
"""
|
||||
calls: list[tuple[Any, ...]] = []
|
||||
|
||||
def _spy(*args: Any, **kwargs: Any) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
calls.append(args)
|
||||
return 0, {}
|
||||
|
||||
monkeypatch.setattr(importer, "_prescan_pending", _spy)
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-gate-zero",
|
||||
{"a.md": "# A\n\nalpha\n", "b.md": "# B\n\nbravo\n"},
|
||||
)
|
||||
llm = FakeEmbedder()
|
||||
llm.settings = _settings(sync_pending_count=0)
|
||||
events: list[tuple[str, str, int, int]] = []
|
||||
try:
|
||||
# Seed the KB (a walk-position run of its own), so nothing pending.
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
assert calls == []
|
||||
|
||||
events.clear()
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
llm,
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
)
|
||||
)
|
||||
assert calls == [], "the switch was 0 but the pre-scan ran"
|
||||
assert (summary.files, summary.unchanged) == (2, 2)
|
||||
assert events == [
|
||||
(root.name, "a.md", 1, 2),
|
||||
(root.name, "b.md", 2, 2),
|
||||
], "the 0 branch must report the phase-193 walk position exactly"
|
||||
finally:
|
||||
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
|
||||
db.commit()
|
||||
|
||||
# The marker: the stats-only pre-walk still EXISTS in the importer, and
|
||||
# it lives in the ``0`` branch (a third walk would mean the replacement
|
||||
# was not a replacement).
|
||||
src = _code_only(importer.import_sources)
|
||||
assert "BOR_SYNC_PENDING_COUNT=0" in src
|
||||
assert src.count("iter_importable_files(") == 2, (
|
||||
"the importer walks the sources for the count more than once besides "
|
||||
"the pre-scan — the pre-scan REPLACES the stats pre-walk (phase 194 "
|
||||
"D1: no third walk)"
|
||||
)
|
||||
|
||||
|
||||
def test_switch_is_read_once_before_the_loop_and_off_the_loop() -> None:
|
||||
"""The phase-184 read-once discipline + the phase-179 off-loop rule.
|
||||
|
||||
``llm.settings.sync_pending_count`` is read exactly ONCE, before the
|
||||
source loop, so a settings overlay saved mid-run cannot flip the meaning
|
||||
of ``done``/``total`` between two files of the same walk; and the span
|
||||
is entered through ``asyncio.to_thread``, so a 10,000-file hash+SELECT
|
||||
pass never parks the single uvicorn loop.
|
||||
"""
|
||||
src = _code_only(importer.import_sources)
|
||||
|
||||
assert src.count("llm.settings.sync_pending_count") == 1, (
|
||||
"sync_pending_count is read more than once per run — a mid-run "
|
||||
"settings save would change the pair's meaning between two files "
|
||||
"(phase 184's read-once discipline)"
|
||||
)
|
||||
read_at = src.index("llm.settings.sync_pending_count")
|
||||
loop_at = src.index("for root in sources:")
|
||||
assert read_at < loop_at, "the switch must be read BEFORE the source loop"
|
||||
|
||||
assert "await asyncio.to_thread(\n _prescan_pending" in src, (
|
||||
"the pre-scan is no longer entered through asyncio.to_thread — a long "
|
||||
"scan would park the event loop (the phase-179 rule, "
|
||||
"test_sync_git_freeze.py)"
|
||||
)
|
||||
# The scan itself is a plain sync function (it runs in the worker).
|
||||
assert not inspect.iscoroutinefunction(importer._prescan_pending)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The scalar statement's shape (the phase-185 column-list style)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_scalar_statement_projects_exactly_file_hash() -> None:
|
||||
"""The pre-scan reads ONE column, so it loads no entity and keeps nothing.
|
||||
|
||||
The compiled projection names ``documents.file_hash`` and no other
|
||||
``documents`` column; the statement resolves to exactly one scalar
|
||||
column (``column_descriptions``), so ``session.scalar`` returns a string
|
||||
and no ORM object exists to land in an identity map. ``_doc_row_stmt``
|
||||
stays the importer's only ``(source, path)`` ROW load (phase 185).
|
||||
"""
|
||||
compiled = str(
|
||||
_pending_hash_stmt("the-source", "notes/a.md").compile(
|
||||
dialect=postgresql.dialect()
|
||||
)
|
||||
)
|
||||
projection = re.split(r"\bFROM\b", compiled, maxsplit=1)[0]
|
||||
columns = set(re.findall(r"documents\.([a-z_]+)", projection))
|
||||
|
||||
assert columns == {"file_hash"}, (
|
||||
f"the pre-scan statement projects {sorted(columns)} — it must project "
|
||||
"documents.file_hash and nothing else (one light metadata read per "
|
||||
"file, no entity hydration)"
|
||||
)
|
||||
# The predicate is the unique (source, path) constraint, like the fast path.
|
||||
predicate = re.split(r"\bFROM\b", compiled, maxsplit=1)[1]
|
||||
assert "documents.source" in predicate and "documents.path" in predicate
|
||||
|
||||
stmt = _pending_hash_stmt("s", "p")
|
||||
assert len(stmt.column_descriptions) == 1
|
||||
assert stmt.column_descriptions[0]["type"] == Document.file_hash.type
|
||||
assert "Document" not in {
|
||||
str(desc.get("entity")) for desc in stmt.column_descriptions
|
||||
}, "the pre-scan statement selects an ENTITY — it must select the column"
|
||||
|
||||
# The span is the only consumer, and the fast path's row load is intact.
|
||||
span_src = inspect.getsource(importer._prescan_pending)
|
||||
assert "_pending_hash_stmt(" in span_src
|
||||
assert "select(Document)" not in span_src
|
||||
assert "defer(" not in span_src and "load_only(" not in span_src, (
|
||||
"the pre-scan picked up a loader option — there is no ROW to defer: "
|
||||
"it selects one column (phase 185 keeps _doc_row_stmt as the only "
|
||||
"row load)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The pending predicate
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pending_predicate_fresh_unchanged_changed_and_null_row(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The four stored-hash states, and the position the set assigns.
|
||||
|
||||
* no row at all (a first sync) → pending;
|
||||
* a row whose ``file_hash`` equals the file's digest → NOT pending;
|
||||
* a row whose ``file_hash`` differs → pending;
|
||||
* a pre-phase-147 row (``file_hash`` NULL) → pending — the same call the
|
||||
fast path makes, so the prediction and the loop agree and the loop
|
||||
stamps the hash on the way through.
|
||||
|
||||
The positions are the 1-based walk order of the PREDICTED files only,
|
||||
and the scan total is every importable file (3 here, 3 predicted).
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-predicate",
|
||||
{
|
||||
"fresh.md": "# Fresh\n\nnever indexed\n",
|
||||
"same.md": "# Same\n\nindexed as-is\n",
|
||||
"moved.md": "# Moved\n\nindexed differently now\n",
|
||||
},
|
||||
)
|
||||
from app.rag.importer import _file_hash
|
||||
|
||||
_seed_row(db, root.name, "same.md", file_hash=_file_hash(root / "same.md"))
|
||||
_seed_row(db, root.name, "moved.md", file_hash="0" * 64)
|
||||
db.expunge_all()
|
||||
|
||||
scan_total, pending = _prescan(root, session=db)
|
||||
|
||||
assert scan_total == 3
|
||||
assert pending == {
|
||||
(root.name, "fresh.md"): 1,
|
||||
(root.name, "moved.md"): 2,
|
||||
}, (
|
||||
"the pending set is wrong — no row and a NULL row are pending, a "
|
||||
"digest match is not, a mismatch is (the fast path's own predicate)"
|
||||
)
|
||||
# The scan reads metadata only: nothing was hydrated into the session.
|
||||
assert len(db.identity_map) == 0
|
||||
|
||||
|
||||
def test_pending_predicate_ignores_what_the_walk_ignores(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""A file the ignore rule hides is never walked, so it is never pending.
|
||||
|
||||
The scan uses the EXACT per-root ``ignore`` / ``include_hidden`` rules of
|
||||
the processing loop: a hidden dot-directory is out by default and in when
|
||||
the root's flag admits it, and an ignored prefix is out even with the flag
|
||||
on. A predicted file the loop can never see would be a count that never
|
||||
reaches zero, so this is the parity edge, not a nicety.
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-ignore",
|
||||
{
|
||||
"keep.md": "# Keep\n\nbody\n",
|
||||
"skipme/hidden-by-rule.md": "# Skip\n\nbody\n",
|
||||
".dot/secret.md": "# Dot\n\nbody\n",
|
||||
},
|
||||
)
|
||||
ignore = {str(root): ["skipme"]}
|
||||
|
||||
scan_total, pending = _prescan(root, session=db, ignore_by_root=ignore)
|
||||
assert scan_total == 1
|
||||
assert list(pending) == [(root.name, "keep.md")]
|
||||
|
||||
# With the hidden-folders flag ON the dot-dir file joins the walk (and
|
||||
# the scan total with it) — still no pending entry for the ignored one.
|
||||
scan_total, pending = _prescan(
|
||||
root,
|
||||
session=db,
|
||||
ignore_by_root=ignore,
|
||||
include_hidden_by_root={str(root): True},
|
||||
)
|
||||
assert scan_total == 2
|
||||
assert sorted(pending) == [
|
||||
(root.name, ".dot/secret.md"),
|
||||
(root.name, "keep.md"),
|
||||
]
|
||||
|
||||
|
||||
def test_unreadable_file_is_predicted_changed_and_the_scan_survives(
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""A file the digest cannot reach is a PREDICTION of change, not a failure.
|
||||
|
||||
The window is the phase's own: the scan walks the tree, then hashes it
|
||||
seconds later, and a live source can lose a file in between. Failing the
|
||||
whole sync for that would be the worse bug — the pre-scan only informs a
|
||||
label — so the file lands in the pending set (the loop's own walk decides
|
||||
whether it still exists at all) and one warning names it.
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-unreadable",
|
||||
{"a.md": "# A\n\na\n", "b.md": "# B\n\nb\n"},
|
||||
)
|
||||
real_hash = importer._file_hash
|
||||
|
||||
def flaky(path: Path) -> str:
|
||||
if path.name == "b.md":
|
||||
raise OSError(2, "No such file or directory", str(path))
|
||||
return real_hash(path)
|
||||
|
||||
monkeypatch.setattr(importer, "_file_hash", flaky)
|
||||
with caplog.at_level("WARNING", logger="app.importer"):
|
||||
scan_total, pending = _prescan(root, session=db)
|
||||
|
||||
assert scan_total == 2
|
||||
assert sorted(pending) == [
|
||||
(root.name, "a.md"),
|
||||
(root.name, "b.md"),
|
||||
]
|
||||
lines = [r.getMessage() for r in caplog.records if r.name == "app.importer"]
|
||||
assert any(
|
||||
line.startswith("import: pre-scan unreadable")
|
||||
and f"source={root.name}" in line
|
||||
and "path=b.md" in line
|
||||
for line in lines
|
||||
), f"the skipped file was not named in a warning: {lines}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# on_scan_progress + the empty KB
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_scan_progress_reports_every_file_across_roots(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The scan pair is ``(1-based position, combined importable total)``.
|
||||
|
||||
The denominator is the number the phase-64 stats pre-walk used to
|
||||
compute (every importable file, across every root, in *sources* order),
|
||||
because the label is about the SCAN — how far the prediction has got —
|
||||
and the events cover every file, changed or not.
|
||||
"""
|
||||
root_a = _root(
|
||||
tmp_path, "phase194-scan-a", {"a1.md": "# A1\n\na\n", "a2.md": "# A2\n\nb\n"}
|
||||
)
|
||||
root_b = _root(tmp_path, "phase194-scan-b", {"sub/b1.md": "# B1\n\nc\n"})
|
||||
events: list[tuple[int, int]] = []
|
||||
|
||||
scan_total, pending = _prescan(
|
||||
root_a, root_b, session=db, on_scan_progress=lambda p, t: events.append((p, t))
|
||||
)
|
||||
|
||||
assert scan_total == 3
|
||||
assert events == [(1, 3), (2, 3), (3, 3)]
|
||||
assert len(pending) == 3
|
||||
|
||||
|
||||
def test_scan_progress_reports_nothing_for_an_empty_kb(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The degenerate KB: scan total 0, no callback, empty pending set.
|
||||
|
||||
The phase-64 ``total === 0`` contract, extended to the scan: with nothing
|
||||
importable there is no position to report, so the UI's scan label renders
|
||||
bare (``Scanning…``, no count part) instead of ``0/0``.
|
||||
"""
|
||||
empty = _root(tmp_path, "phase194-empty", {})
|
||||
missing = tmp_path / "phase194-not-a-dir"
|
||||
events: list[tuple[int, int]] = []
|
||||
|
||||
scan_total, pending = _prescan(
|
||||
empty, missing, session=db, on_scan_progress=lambda p, t: events.append((p, t))
|
||||
)
|
||||
|
||||
assert scan_total == 0
|
||||
assert pending == {}
|
||||
assert events == []
|
||||
|
||||
|
||||
def test_scan_hook_is_called_from_the_worker_thread(
|
||||
tmp_path: Path, db: Session
|
||||
) -> None:
|
||||
"""The documented threading contract of ``on_scan_progress``.
|
||||
|
||||
The span runs in an ``asyncio.to_thread`` worker, so the hook fires on
|
||||
THAT thread — which is only safe because the house hook rule is "assign a
|
||||
field", a single GIL-atomic store. The pin records the thread identity
|
||||
from inside the callback and from the awaiting coroutine.
|
||||
"""
|
||||
root = _root(tmp_path, "phase194-thread", {"a.md": "# A\n\na\n"})
|
||||
seen: dict[str, int] = {}
|
||||
|
||||
def hook(position: int, total: int) -> None: # noqa: ARG001
|
||||
seen["scan"] = threading.get_ident()
|
||||
|
||||
async def main() -> None:
|
||||
seen["loop"] = threading.get_ident()
|
||||
await asyncio.to_thread(
|
||||
importer._prescan_pending,
|
||||
[root],
|
||||
_settings().import_extension_set,
|
||||
frozenset(),
|
||||
None,
|
||||
None,
|
||||
hook,
|
||||
)
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
assert seen["scan"] != seen["loop"], (
|
||||
"the pre-scan hook ran on the event-loop thread — the span must live "
|
||||
"in a worker (phase 179)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# done = the position in the pending set
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_done_is_the_pending_position_through_import_sources(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""3 importable files, 2 of them pending: the pair is ``1/2``, ``1/2``,
|
||||
``2/2``.
|
||||
|
||||
``stable.md`` is indexed and unchanged, so the scan does not predict it
|
||||
and the walk moving past it does NOT advance the count — the label holds
|
||||
its position (this is the shape the owner's prod case takes: the count
|
||||
stops climbing across a run of unchanged files). The two changed files
|
||||
advance it in walk order, and the denominator is the pending set's size,
|
||||
never the 3-file walk.
|
||||
"""
|
||||
root = _root(
|
||||
tmp_path,
|
||||
"phase194-position",
|
||||
{
|
||||
"a.md": "# A\n\nalpha\n",
|
||||
"stable.md": "# Stable\n\nunchanged\n",
|
||||
"z.md": "# Z\n\nzulu\n",
|
||||
},
|
||||
)
|
||||
llm = FakeEmbedder()
|
||||
events: list[tuple[str, str, int, int]] = []
|
||||
try:
|
||||
# Index everything, then change two files.
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
(root / "a.md").write_text("# A\n\nalpha edited\n", encoding="utf-8")
|
||||
(root / "z.md").write_text("# Z\n\nzulu edited\n", encoding="utf-8")
|
||||
|
||||
events.clear()
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
llm,
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
)
|
||||
)
|
||||
|
||||
assert (summary.files, summary.updated, summary.unchanged) == (3, 2, 1)
|
||||
assert events == [
|
||||
(root.name, "a.md", 1, 2),
|
||||
(root.name, "stable.md", 1, 2), # not predicted: the position holds
|
||||
(root.name, "z.md", 2, 2),
|
||||
]
|
||||
# The parity the UI depends on: the denominator IS the work.
|
||||
assert summary.updated + summary.added == events[-1][3]
|
||||
finally:
|
||||
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_empty_kb_reports_no_count_part_through_import_sources(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""An empty source with the hook on: no file, no pair, ``files_total`` 0.
|
||||
|
||||
The import is a no-op and the hook never fires, so the status keeps the
|
||||
zero denominator the UI's ``total === 0`` guard hides the count part for
|
||||
(the phase-64 contract, arriving through the pre-scan path rather than
|
||||
the stats pre-walk).
|
||||
"""
|
||||
root = _root(tmp_path, "phase194-empty-run", {})
|
||||
events: list[tuple[str, str, int, int]] = []
|
||||
scan_events: list[tuple[int, int]] = []
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
FakeEmbedder(),
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: events.append((s, r, d, t)),
|
||||
on_scan_progress=lambda p, t: scan_events.append((p, t)),
|
||||
)
|
||||
)
|
||||
|
||||
assert (summary.files, summary.added, summary.errors) == (0, 0, 0)
|
||||
assert events == []
|
||||
assert scan_events == []
|
||||
|
||||
|
||||
def test_cli_caller_is_untouched_by_the_pending_semantics(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The CLI-shaped run (no hook at all) on an unchanged KB is the pre-phase
|
||||
run: one file processed, nothing counted, one walk.
|
||||
|
||||
The byte-identity of the non-UI caller is the phase's cheapest promise —
|
||||
``scripts/import_docs.py`` passes no ``progress``, so it neither pays the
|
||||
scan nor sees its numbers.
|
||||
"""
|
||||
root = _root(tmp_path, "phase194-cli", {"a.md": "# A\n\na\n"})
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (first.files, first.added) == (1, 1)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.files, second.added, second.unchanged) == (1, 0, 1)
|
||||
finally:
|
||||
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": root.name})
|
||||
db.commit()
|
||||
@@ -0,0 +1,544 @@
|
||||
"""Phase 194, task 03 — the Sources page renders the scan as a scan.
|
||||
|
||||
The owner report the whole phase exists for (2026-10-19): *"1216
|
||||
remaining"* on a ~10,000-file KB that had ~10 files to sync. Task 01 moved
|
||||
the server's ``done``/``total`` to the pre-scan's predicted change set and
|
||||
task 02 exposed the scan as a phase (``phase: "scan"`` + ``scan_done`` /
|
||||
``scan_total``). This module pins the UI half (D4, owner-confirmed A2):
|
||||
``fmtSyncPhaseLabel`` gains ONE explicit ``"scan"`` branch that says
|
||||
``Scanning…`` and puts the scan's own ``(n/total)`` in the SAME count slot
|
||||
the import phase's ``N remaining`` uses — so the walk position surviving as
|
||||
an ``n/m`` is honest, because the word in front of it is "Scanning" and
|
||||
never "Syncing" (TODO L6's objection was the sync count wearing the
|
||||
total's shape).
|
||||
|
||||
Pinned here (source markers, plus one node run of the REAL label path):
|
||||
|
||||
* the branch — ``status.phase === "scan"`` is in the decision tree BEFORE
|
||||
``overview`` / ``summaries`` / the fall-through, its main text is exactly
|
||||
``Scanning…``, and it never reads ``current_file`` (null in the phase),
|
||||
the file pair, the chunk pair, the image pair or the summaries pair;
|
||||
* the shape — the count part is ``(${status.scan_done}/${status.scan_total})``
|
||||
landing on the ``count`` slot (``setSyncLabel``'s hidden-toggle + the A4
|
||||
untruncated button-title / ``#sync-progress`` placement carry it with no
|
||||
new element), with the phase-64 ``total === 0`` guard (the empty KB and
|
||||
the scan's own first frame render the bare ``Scanning…``) and no
|
||||
chunk/image part (the scan reads no file);
|
||||
* the pass-through — both sync status call sites (the poll tick, the
|
||||
load-time re-attach) forward the WHOLE status object, which is what
|
||||
carries the scan pair to the wrapper (the summaries / chunk / image
|
||||
pattern: ``phase`` selects which pair renders, no field is re-listed);
|
||||
* the UNTOUCHED set — ``fmtSyncLabel``'s ``N remaining`` expression
|
||||
verbatim (the phase-193 pin), the chunk / image / summaries part
|
||||
builders, ``setSyncLabel``'s composed full string, ``fmtSyncResult`` and
|
||||
``fmtSyncPeakClause`` (the phase-187 trailing clause lands on the scan
|
||||
label's line unchanged, and stays a terminal-only clause);
|
||||
* the docstrings — the decision-tree paragraph names the new branch,
|
||||
narrows the fallback to the null prelude + ``"import"``, and keeps the
|
||||
phase-193 ``N remaining`` clause.
|
||||
|
||||
The owner-visible arms (the count-down over a real incremental sync, the
|
||||
scan label in a real browser) live in
|
||||
``tests/e2e/test_sync_pending_count.py`` (task 05); the stale pinned suites
|
||||
retarget in task 04.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
#: The node behavior check needs node (the test-document-viewer pattern).
|
||||
HAVE_NODE = shutil.which("node") is not None
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SOURCES_JS = ROOT / "frontend" / "assets" / "sources.js"
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return SOURCES_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _fn(js: str, name: str) -> str:
|
||||
"""The source of the first ``function <name>`` (brace balanced — the
|
||||
house pin pattern from tests/unit/test_frontend_sync_upload.py: the
|
||||
functions live inside ``mount(root)``, so the closing brace is
|
||||
indented)."""
|
||||
fn = js.find(f"function {name}")
|
||||
assert fn != -1, f"{name} must be defined in sources.js"
|
||||
open_idx = js.find("{", fn)
|
||||
depth = 0
|
||||
for i in range(open_idx, len(js)):
|
||||
c = js[i]
|
||||
if c == "{":
|
||||
depth += 1
|
||||
elif c == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return js[fn : i + 1]
|
||||
raise AssertionError(f"unbalanced braces in {name}")
|
||||
|
||||
|
||||
def _doc_above(js: str, name: str) -> str:
|
||||
"""The block comment immediately above ``function <name>``."""
|
||||
i = js.find(f"function {name}")
|
||||
assert i != -1, f"{name} must be defined in sources.js"
|
||||
start = js.rfind("/*", 0, i)
|
||||
assert start != -1, f"{name} must carry a docstring"
|
||||
end = js.find("*/", start)
|
||||
assert start < end < i, f"{name}'s docstring must sit above the function"
|
||||
return js[start:end]
|
||||
|
||||
|
||||
def _scan_slice(js: str) -> str:
|
||||
"""The ``"scan"`` branch of ``fmtSyncPhaseLabel`` — from its ``if`` to
|
||||
the ``overview`` branch that follows it (every shape pin below reads
|
||||
ONLY this slice, so a shape that leaked into another branch is caught
|
||||
and a branch that moved is caught)."""
|
||||
body = _fn(js, "fmtSyncPhaseLabel")
|
||||
start = body.find('status.phase === "scan"')
|
||||
assert start != -1, 'the "scan" branch is missing from fmtSyncPhaseLabel'
|
||||
end = body.find('status.phase === "overview"', start)
|
||||
assert end != -1, "the overview branch must follow the scan branch"
|
||||
return body[start:end]
|
||||
|
||||
|
||||
# ---------- the branch ----------
|
||||
|
||||
|
||||
def test_scan_branch_is_the_first_arm_of_the_decision_tree() -> None:
|
||||
"""The wrapper gains ONE explicit ``"scan"`` branch (D4): the phase
|
||||
comparison sits BEFORE ``overview``, ``summaries`` and the
|
||||
``fmtSyncLabel`` fall-through (the machine order is ``null`` →
|
||||
``"scan"`` → ``"import"`` → ``"overview"`` → ``"summaries"``, and
|
||||
keeping the arms in that order is what leaves every pre-existing
|
||||
branch slice byte-identical), and the main text is exactly
|
||||
``Scanning…`` with the ellipsis the other running labels use."""
|
||||
js = _js()
|
||||
body = _fn(js, "fmtSyncPhaseLabel")
|
||||
i_scan = body.find('status.phase === "scan"')
|
||||
i_ov = body.find('status.phase === "overview"')
|
||||
i_sum = body.find('status.phase === "summaries"')
|
||||
i_fall = body.find("fmtSyncLabel(", i_sum)
|
||||
assert -1 < i_scan < i_ov < i_sum < i_fall, (
|
||||
"the scan branch must be the first arm, ahead of overview / "
|
||||
"summaries / the null-prelude + import fall-through"
|
||||
)
|
||||
assert 'main: "Scanning\u2026"' in _scan_slice(js), (
|
||||
"the main text is the bare Scanning label — no file"
|
||||
)
|
||||
|
||||
|
||||
def test_scan_branch_reads_the_scan_pair_and_nothing_else() -> None:
|
||||
"""The branch consumes the scan pair the way it already consumes the
|
||||
summaries pair (``phase`` selects which pair renders). It NEVER reads
|
||||
``current_file`` — the runner keeps it null through the phase, because
|
||||
the scan names no file — nor the file pair, the chunk pair, the image
|
||||
pair or the summaries pair (a scan embeds no chunk and describes no
|
||||
image, so those fields carry nothing meaningful here)."""
|
||||
scan = _scan_slice(_js())
|
||||
assert "status.scan_done" in scan and "status.scan_total" in scan, (
|
||||
"the count part is the status's own scan pair"
|
||||
)
|
||||
for field in (
|
||||
"current_file",
|
||||
"files_done",
|
||||
"files_total",
|
||||
"chunks_total",
|
||||
"current_chunk",
|
||||
"images_total",
|
||||
"images_done",
|
||||
"summaries_total",
|
||||
"summaries_done",
|
||||
"current_summary",
|
||||
):
|
||||
assert field not in scan, (
|
||||
f'the "scan" branch must not read {field} — that field is the '
|
||||
"import/overview/summaries phase's own position"
|
||||
)
|
||||
|
||||
|
||||
def test_scan_branch_emits_no_chunk_and_no_image_part() -> None:
|
||||
"""The scan names no file, so its parts are the main text + the count
|
||||
only: ``chunk: ""`` and ``image: ""`` (the writer hides both spans and
|
||||
keeps them out of the composed full string — the overview branch's
|
||||
shape, phase 98/152)."""
|
||||
scan = _scan_slice(_js())
|
||||
assert 'chunk: ""' in scan, "the scan label carries no chunk part"
|
||||
assert 'image: ""' in scan, "the scan label carries no image part"
|
||||
|
||||
|
||||
# ---------- the (n/total) count part ----------
|
||||
|
||||
|
||||
def test_scan_count_part_is_the_pair_in_the_count_slot() -> None:
|
||||
"""The count part is the exact ``(${status.scan_done}/${status.scan_total})``
|
||||
template — a plain ``(n/total)``, no thousands separator, no suffix —
|
||||
and it lands on the ``count`` key, i.e. the SAME slot the import
|
||||
phase's ``N remaining`` uses (``setSyncLabel``'s
|
||||
``.sync-label-count`` span + the composed full string), so the count
|
||||
stays pinned on a phone and the A4 untruncated label (button title +
|
||||
``#sync-progress``) carries ``Scanning… (n/total)`` with no new
|
||||
element and no new live region."""
|
||||
scan = _scan_slice(_js())
|
||||
assert "`(${status.scan_done}/${status.scan_total})`" in scan, (
|
||||
"the exact (n/total) shape over the scan pair"
|
||||
)
|
||||
i_count = scan.find("count:")
|
||||
i_template = scan.find("`(${status.scan_done}/${status.scan_total})`")
|
||||
assert -1 < i_count < i_template, "the pair renders INTO the count slot"
|
||||
assert "remaining" not in scan, (
|
||||
"the scan never says remaining — that part belongs to fmtSyncLabel"
|
||||
)
|
||||
for helper in ("toLocaleString", ".toFixed"):
|
||||
assert helper not in scan, "plain integers, the house label style"
|
||||
|
||||
|
||||
def test_scan_count_part_hides_while_the_total_is_zero() -> None:
|
||||
"""The phase-64 ``total === 0`` contract, extended to the scan: while
|
||||
the scan has not reported its denominator (and for the degenerate
|
||||
empty KB, where the importer fires no scan callback at all) the count
|
||||
part is the empty string — the bare ``Scanning…`` renders, never
|
||||
``(0/0)``. The guard precedes the template and the false arm is the
|
||||
empty string."""
|
||||
scan = _scan_slice(_js())
|
||||
i_guard = scan.find("status.scan_total > 0")
|
||||
i_template = scan.find("`(${status.scan_done}/${status.scan_total})`")
|
||||
i_else = scan.find(': ""', i_template)
|
||||
assert -1 < i_guard < i_template < i_else, (
|
||||
"scan_total > 0 ? (n/total) : \"\" — the phase-64 no-count-part guard"
|
||||
)
|
||||
|
||||
|
||||
# ---------- the caller pass-through ----------
|
||||
|
||||
|
||||
def test_poll_tick_forwards_the_whole_status_so_the_scan_pair_rides() -> None:
|
||||
"""The running-sync branch of the poll tick passes the WHOLE status
|
||||
object as ``enterSyncRunningState``'s fifth argument — that object IS
|
||||
the pass-through for the scan pair, exactly like the summaries /
|
||||
chunk / image pairs (the call site names it). A status with
|
||||
``phase: "scan"`` therefore reaches the wrapper with its
|
||||
``scan_done`` / ``scan_total`` intact."""
|
||||
js = _js()
|
||||
start = js.find("function startSyncPolling")
|
||||
assert start != -1
|
||||
tick = js[start : js.find("async function startSync", start)]
|
||||
i_run = tick.find('syncStatus.state === "running"')
|
||||
assert i_run != -1
|
||||
# the call-site comment sits directly above the branch — it is part of
|
||||
# the pass-through's documentation, so the slice starts there
|
||||
branch = tick[tick.rfind("// 1. sync running", 0, i_run) : tick.find(
|
||||
'uploadStatus && uploadStatus.state === "running"', i_run
|
||||
)]
|
||||
assert (
|
||||
'"sync", syncStatus.current_file, syncStatus.files_done, syncStatus.files_total'
|
||||
in branch
|
||||
), "the pre-phase positional args are unchanged"
|
||||
i_args = branch.find("syncStatus.files_total")
|
||||
i_pass = branch.find("syncStatus\n", i_args)
|
||||
assert i_pass != -1, "the WHOLE syncStatus object rides in as the 5th arg"
|
||||
assert "scan_done" in branch and "scan_total" in branch, (
|
||||
"the call-site comment names the scan pair riding the status (phase 194)"
|
||||
)
|
||||
|
||||
|
||||
def test_load_time_reattach_forwards_the_whole_status_too() -> None:
|
||||
"""The load-time re-attach is the other sync call site: a reload
|
||||
during the pre-scan re-enters running state with the WHOLE status, so
|
||||
the re-attached label reads ``Scanning… (n/total)`` (the never-stale
|
||||
contract — phase 98's mid-summaries reload, now also mid-scan)."""
|
||||
js = _js()
|
||||
body = _fn(js, "initSyncButton")
|
||||
assert (
|
||||
'"sync", status.current_file, status.files_done, status.files_total' in body
|
||||
), "the pre-phase positional args are unchanged"
|
||||
i_args = body.find("status.files_total")
|
||||
i_pass = body.find("status\n", i_args)
|
||||
assert i_pass != -1, "the WHOLE status object rides in as the 5th arg"
|
||||
doc = _doc_above(js, "initSyncButton")
|
||||
assert "phase 194" in doc and "Scanning" in doc, (
|
||||
"the re-attach docstring names the scan label it can re-enter (194)"
|
||||
)
|
||||
|
||||
|
||||
def test_running_entry_hands_the_status_to_the_phase_wrapper() -> None:
|
||||
"""``enterSyncRunningState`` keeps the phase-aware split untouched:
|
||||
sync + status → ``fmtSyncPhaseLabel(status)`` (the wrapper that now
|
||||
owns the scan branch), everything else → the bare ``fmtSyncLabel``
|
||||
parts, and the parts still go to the ONE writer with the running
|
||||
flag (so the scan label lands on the button title + the box)."""
|
||||
entry = _fn(_js(), "enterSyncRunningState")
|
||||
assert "fmtSyncPhaseLabel(status)" in entry
|
||||
assert 'kind === "sync" && status' in entry
|
||||
assert "setSyncLabel(main, count, chunk, true, image)" in entry
|
||||
|
||||
|
||||
# ---------- the UNTOUCHED set ----------
|
||||
|
||||
|
||||
def test_fmt_sync_label_n_remaining_arithmetic_is_byte_identical() -> None:
|
||||
"""The phase-193 count-down builder is UNTOUCHED by this task — the
|
||||
server-side semantic change (the pair becomes the pending position)
|
||||
flows through the SAME expression. The exact line, both guards and the
|
||||
plain-integer shape, plus the absence of anything scan-shaped in this
|
||||
builder."""
|
||||
bare = _fn(_js(), "fmtSyncLabel")
|
||||
assert (
|
||||
"const count = total > 0 && total - done > 0"
|
||||
" ? `${total - done} remaining` : \"\";"
|
||||
) in bare, "the phase-193 count expression, byte for byte"
|
||||
assert 'const prefix = kind === "upload" ? "Importing" : "Syncing\u2026";' in bare
|
||||
assert "return { main, count }" in bare
|
||||
for word in ("scan", "chunk", "image", "summaries", "phase"):
|
||||
assert word not in bare, f"fmtSyncLabel stays free of {word!r}"
|
||||
|
||||
|
||||
def test_other_phase_labels_and_the_writers_are_untouched() -> None:
|
||||
"""The regression pins for everything the scan branch could have
|
||||
disturbed: the overview copy, the summaries folder + ``(n/m)``
|
||||
position, the import fall-through's chunk + image parts, the single
|
||||
writer's signature + composed full string, the terminal counts line
|
||||
(no running count part, no scan word) and the phase-187 peak clause
|
||||
(still a terminal-only trailing clause, null renders nothing)."""
|
||||
js = _js()
|
||||
phase = _fn(js, "fmtSyncPhaseLabel")
|
||||
assert (
|
||||
'return { main: "Writing KB overview\u2026", count: "", chunk: "", image: "" };'
|
||||
in phase
|
||||
)
|
||||
assert "`Summarizing folders\u2026${folder}`" in phase
|
||||
assert (
|
||||
"count: `(${status.summaries_done}/${status.summaries_total})`" in phase
|
||||
), "the summaries folder position keeps its (n/m)"
|
||||
assert (
|
||||
'const { main, count } = fmtSyncLabel(\n'
|
||||
' "sync", status.current_file, status.files_done, status.files_total\n'
|
||||
" );" in phase
|
||||
), "the import fall-through still calls the bare builder unchanged"
|
||||
assert "`\u00b7 chunk ${status.current_chunk}/${status.chunks_total}`" in phase
|
||||
assert "`\u00b7 image ${status.images_done}/${status.images_total}`" in phase
|
||||
assert "return { main, count, chunk, image };" in phase
|
||||
|
||||
writer = _fn(js, "setSyncLabel")
|
||||
assert (
|
||||
'function setSyncLabel(mainText, countText = "", chunkText = "",'
|
||||
' running = false, imageText = "")' in writer
|
||||
)
|
||||
assert "syncLabelCount.textContent = countText" in writer
|
||||
assert "syncLabelCount.hidden = false" in writer
|
||||
assert (
|
||||
"const full =\n"
|
||||
" mainText +\n"
|
||||
" (countText ? ` ${countText}` : \"\") +\n"
|
||||
" (chunkText ? ` ${chunkText}` : \"\") +\n"
|
||||
" (imageText ? ` ${imageText}` : \"\");" in writer
|
||||
), "the A4 untruncated label is still main + count + chunk + image"
|
||||
assert "if (running) setSyncProgress(full);" in writer
|
||||
assert "syncBtn.title = full" in writer
|
||||
|
||||
result = _fn(js, "fmtSyncResult")
|
||||
for word in ("remaining", "Scanning", "scan"):
|
||||
assert word not in result, f"the terminal counts line stays {word!r}-free"
|
||||
|
||||
peak = _fn(js, "fmtSyncPeakClause")
|
||||
assert "return ` \u00b7 peak ${peak} MB`;" in peak
|
||||
assert "status.rss_peak_mb ?? status.rss_mb" in peak
|
||||
assert "Scanning" not in peak, "the clause is terminal-only, unchanged"
|
||||
|
||||
|
||||
# ---------- the docstrings ----------
|
||||
|
||||
|
||||
def test_phase_label_docstring_names_the_branch_and_narrows_the_fallback() -> None:
|
||||
"""The decision-tree paragraph above the wrapper names the new arm and
|
||||
narrows the fall-through to the two states that remain, keeps the
|
||||
phase-193 ``N remaining`` clause, states the ``total === 0`` guard,
|
||||
and says WHY an ``(n/total)`` is honest here (TODO L6: the objection
|
||||
was the sync count wearing the total's shape — this label says
|
||||
Scanning, never Syncing)."""
|
||||
doc = _doc_above(_js(), "fmtSyncPhaseLabel")
|
||||
assert '"scan"' in doc and "Scanning\u2026" in doc, "the new branch is named"
|
||||
assert "scan_done" in doc and "scan_total" in doc, "the pair is named"
|
||||
assert "scan_total === 0" in doc, "the no-count-part guard is documented"
|
||||
assert "ONLY the two" in doc and '"import"' in doc, "the fallback is narrowed"
|
||||
assert "`N remaining`" in doc and "N = total - done" in doc, (
|
||||
"the phase-193 count-down clause survives the rewrite"
|
||||
)
|
||||
assert "never" in doc and '"Syncing"' in doc, (
|
||||
"the honesty clause: Scanning, never Syncing (TODO L6)"
|
||||
)
|
||||
assert "A4" in doc, "the untruncated placement is still documented"
|
||||
|
||||
|
||||
def test_module_and_section_headers_document_the_scan_label() -> None:
|
||||
"""The module header carries the phase-194 paragraph (the count is the
|
||||
WORK, the scan says so), and the sync section's decision tree names
|
||||
the ``"scan"`` arm with its ``(n/total)`` — the header the wrapper's
|
||||
docstring points at."""
|
||||
js = _js()
|
||||
header = js[: js.find("import { fetchIsAdmin }")]
|
||||
assert "Phase 194 (task 03)" in header, "the module header names the phase"
|
||||
assert "Scanning\u2026 (n/total)" in header, "the module header names the label"
|
||||
|
||||
marker = js.find("Sync sources button (Sources page only)")
|
||||
assert marker != -1
|
||||
section = js[marker : js.find("const syncBtn", marker)]
|
||||
assert "Phase 194 task 03" in section, "the section header names the phase"
|
||||
assert "Scanning\u2026 (n/total)" in section, "the decision tree names the label"
|
||||
assert "scan_done" in section, "the pair the WHOLE status carries is named"
|
||||
|
||||
|
||||
# ---------- the REAL label path under node ----------
|
||||
|
||||
#: The stub element refs (the test-document-viewer pattern; the same
|
||||
#: shapes tests/unit/test_frontend_sync_upload.py drives). The label
|
||||
#: writer, the box writer and the cancel control are the REAL functions.
|
||||
_NODE_HEAD = """
|
||||
const syncLabelMain = { textContent: "" };
|
||||
const syncLabelCount = { textContent: "", hidden: true };
|
||||
const syncLabelChunk = { textContent: "", hidden: true };
|
||||
const syncProgress = { hidden: true };
|
||||
const syncProgressText = { textContent: "" };
|
||||
const syncBtn = {
|
||||
disabled: false,
|
||||
title: "",
|
||||
attrs: {},
|
||||
setAttribute(k, v) { this.attrs[k] = v; },
|
||||
removeAttribute(k) { delete this.attrs[k]; },
|
||||
classes: new Set(),
|
||||
classList: {
|
||||
add(c) { syncBtn.classes.add(c); },
|
||||
remove(c) { syncBtn.classes.delete(c); },
|
||||
has(c) { return syncBtn.classes.has(c); },
|
||||
},
|
||||
};
|
||||
const syncIcon = {
|
||||
classes: new Set(),
|
||||
classList: {
|
||||
add(c) { syncIcon.classes.add(c); },
|
||||
remove(c) { syncIcon.classes.delete(c); },
|
||||
has(c) { return syncIcon.classes.has(c); },
|
||||
},
|
||||
};
|
||||
const syncCancelBtn = {
|
||||
hidden: true,
|
||||
disabled: true,
|
||||
attrs: {},
|
||||
setAttribute(k, v) { this.attrs[k] = v; },
|
||||
removeAttribute(k) { delete this.attrs[k]; },
|
||||
};
|
||||
"""
|
||||
|
||||
_NODE_TAIL = """
|
||||
const show = (tag) => {
|
||||
console.log(JSON.stringify([
|
||||
tag,
|
||||
syncLabelMain.textContent,
|
||||
syncLabelCount.textContent,
|
||||
syncLabelCount.hidden,
|
||||
syncBtn.title,
|
||||
syncProgressText.textContent,
|
||||
syncBtn.attrs['aria-label'] || '',
|
||||
]));
|
||||
};
|
||||
// 1. the scan, denominator known + the position climbing (the mid-scan poll):
|
||||
enterSyncRunningState("sync", null, 0, 0, {
|
||||
phase: "scan", current_file: null, files_done: 0, files_total: 0,
|
||||
scan_done: 3, scan_total: 9, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("scan-3-9");
|
||||
// 2. the scan's degenerate arm: an empty KB / the scan's own first frame
|
||||
// (0/0) — the phase-64 guard, the bare label with NO count part:
|
||||
enterSyncRunningState("sync", null, 0, 0, {
|
||||
phase: "scan", current_file: null, files_done: 0, files_total: 0,
|
||||
scan_done: 0, scan_total: 0, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("scan-0-0");
|
||||
// 3. the prelude (null phase) is untouched — bare "Syncing\u2026":
|
||||
enterSyncRunningState("sync", null, 0, 0, {
|
||||
phase: null, current_file: null, files_done: 0, files_total: 0,
|
||||
scan_done: 4, scan_total: 9, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("prelude");
|
||||
// 4. the import phase is untouched — the phase-193 count-down + the chunk
|
||||
// part, and the scan pair present on the payload but NOT rendered:
|
||||
enterSyncRunningState("sync", "src/a/b.md", 2, 9, {
|
||||
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 9,
|
||||
scan_done: 9, scan_total: 9, current_chunk: 14, chunks_total: 38,
|
||||
});
|
||||
show("import");
|
||||
// 5. the overview + summaries labels are untouched:
|
||||
enterSyncRunningState("sync", null, 9, 9, {
|
||||
phase: "overview", current_file: null, files_done: 9, files_total: 9,
|
||||
scan_done: 9, scan_total: 9, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("overview");
|
||||
enterSyncRunningState("sync", null, 9, 9, {
|
||||
phase: "summaries", current_file: null, files_done: 9, files_total: 9,
|
||||
scan_done: 9, scan_total: 9, current_summary: "src/alpha",
|
||||
summaries_done: 2, summaries_total: 3, current_chunk: 0, chunks_total: 0,
|
||||
});
|
||||
show("summaries");
|
||||
// 6. the 202/409 click (no status at all) stays bare:
|
||||
enterSyncRunningState("sync", null, 0, 0);
|
||||
show("click");
|
||||
"""
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_scan_label_renders_through_the_real_label_path_under_node() -> None:
|
||||
"""Behavior, not just markers: the REAL extracted functions run
|
||||
against stub refs. ``phase: "scan"`` renders main ``Scanning…`` with
|
||||
the count ``(3/9)`` visible, and the SAME string lands on the button
|
||||
title, the box (the sole aria-live region) and the button's
|
||||
accessible name (A4) — while every other phase renders exactly what it
|
||||
rendered before this task."""
|
||||
js = _js()
|
||||
fns = "\n".join(
|
||||
_fn(js, name)
|
||||
for name in (
|
||||
"fmtSyncLabel",
|
||||
"fmtSyncPhaseLabel",
|
||||
"setSyncLabel",
|
||||
"setSyncProgress",
|
||||
"showSyncCancelBtn",
|
||||
"hideSyncCancelBtn",
|
||||
"enterSyncRunningState",
|
||||
)
|
||||
)
|
||||
out = subprocess.run(
|
||||
["node", "-e", _NODE_HEAD + fns + _NODE_TAIL],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
)
|
||||
assert out.returncode == 0, out.stderr
|
||||
rows = {row[0]: row for row in (json.loads(line) for line in out.stdout.splitlines())}
|
||||
|
||||
main, count, hidden, title, box, aria = rows["scan-3-9"][1:]
|
||||
assert main == "Scanning\u2026", rows["scan-3-9"]
|
||||
assert count == "(3/9)" and hidden is False, rows["scan-3-9"]
|
||||
assert title == "Scanning\u2026 (3/9)" == box == aria, (
|
||||
f"A4: the untruncated scan label rides title + box + aria: {rows['scan-3-9']}"
|
||||
)
|
||||
|
||||
main, count, hidden, title, box, aria = rows["scan-0-0"][1:]
|
||||
assert main == "Scanning\u2026" and count == "" and hidden is True, rows["scan-0-0"]
|
||||
assert title == "Scanning\u2026" == box == aria, (
|
||||
f"0/0 renders NO count part (the empty-KB guard): {rows['scan-0-0']}"
|
||||
)
|
||||
|
||||
# every other state is byte-identical to its pre-phase label
|
||||
assert rows["prelude"][1] == "Syncing\u2026" and rows["prelude"][2] == ""
|
||||
assert rows["prelude"][4] == "Syncing\u2026", "a scan pair never renders in the prelude"
|
||||
assert rows["import"][1] == "Syncing\u2026 src/a/b.md"
|
||||
assert rows["import"][2] == "7 remaining", rows["import"]
|
||||
assert rows["import"][4] == "Syncing\u2026 src/a/b.md 7 remaining \u00b7 chunk 14/38"
|
||||
assert rows["overview"][1] == "Writing KB overview\u2026" and rows["overview"][2] == ""
|
||||
assert rows["summaries"][1] == "Summarizing folders\u2026 src/alpha"
|
||||
assert rows["summaries"][2] == "(2/3)", "the summaries position keeps its (n/m)"
|
||||
assert rows["click"][1] == "Syncing\u2026" and rows["click"][2] == ""
|
||||
Reference in New Issue
Block a user