phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
@@ -101,6 +101,20 @@ turn the backfill into a crash instead of a query.
|
||||
``_prune``'s light row (phase 184), the identity-map cadence (phase 184),
|
||||
``app/db.py:30``, ``iter_importable_files`` (phase 185 LOCKED A3) and the
|
||||
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
|
||||
|
||||
**Why phase 194 left these pins UNEDITED (the D5 sweep).** The phase-194
|
||||
pre-scan is gated on the caller passing ``progress`` — and every
|
||||
``import_sources`` call in this file passes NO ``progress`` (the statement
|
||||
capture is the point, so the run needs no hook). The pre-scan therefore
|
||||
never runs here: the captured fast-path ROW statement above stays the only
|
||||
statement over ``documents`` per inspected file, the "exactly one
|
||||
``SELECT`` per unchanged file" pin and the "one PK follow-up for the
|
||||
summary-NULL backfill" pin are byte-identical, and the pre-scan's own
|
||||
scalar statement (``SELECT documents.file_hash``, ``_pending_hash_stmt``)
|
||||
appears nowhere in the capture — it is pinned in
|
||||
``tests/unit/test_sync_pending_count.py`` and
|
||||
``tests/integration/test_sync_pending_count.py`` instead, where a
|
||||
``progress`` caller does exist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -135,6 +135,18 @@ lazy SELECT per row and fail that test, not get papered over.
|
||||
**What this file does NOT touch:** the walk's column set (phase 185),
|
||||
``scripts/git_sync.py`` (phase 186), ``app/db.py:30``, and the
|
||||
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
|
||||
|
||||
**Why phase 194 left the bound UNCHANGED (the D5 sweep).** Several tests
|
||||
here call ``import_sources`` WITH a ``progress`` hook — so the phase-194
|
||||
pre-scan really runs on those walks. It cannot move a single number in
|
||||
this file: ``_prescan_pending`` reads the SCALAR projection
|
||||
``_pending_hash_stmt`` (one column, no entity), through a session of its
|
||||
OWN inside the ``to_thread`` span, so nothing lands in the RUN's
|
||||
``identity_map`` — the staircase, the sawtooth, the ``MAP_BOUND`` ceiling
|
||||
and the byte-identical ``ImportSummary`` / database comparisons are all
|
||||
exactly what phase 184 measured. The hydration pin the claim rests on
|
||||
(``len(session.identity_map) == 0`` at the first ``progress`` call) lives
|
||||
in ``tests/integration/test_sync_pending_count.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -93,6 +93,18 @@ on all three terminals (success, cancelled, failed) plus the importer's
|
||||
``import: summary …``. The fields are APPENDED (LOCKED A2), so every
|
||||
pre-phase prefix below is byte-identical and the pins here assert BOTH
|
||||
halves of that contract.
|
||||
|
||||
Phase 194 (task 02): the status gains the pre-scan. The payload is
|
||||
ADDITIVE — ``scan_done`` / ``scan_total`` after ``files_total``, the
|
||||
additive-chain pins below updated to the new key set (the idle shape and
|
||||
both ``_PRE_187_STATUS_KEYS`` assertions) — and ``files_done`` /
|
||||
``files_total`` now carry the PREDICTED-change-set position on a progress
|
||||
caller. One existing pin retargets with it:
|
||||
``test_double_trigger_while_running_returns_409`` waits on ``"scan"``
|
||||
instead of ``"import"``, because its canned import fires NO per-file hook
|
||||
and the scan→import flip lives in that hook — a run that reports no file
|
||||
stays in the pre-import phase (the full sequence with a live hook is
|
||||
pinned in ``tests/integration/test_sync_scan_phase.py``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -375,6 +387,11 @@ class FakeImportSources:
|
||||
# image fields stay 0/0, like the chunk fields of a no-batch
|
||||
# import).
|
||||
self.image_hooks: list[object] = []
|
||||
# Phase 194 (task 02): the PRE-SCAN position hook the runner
|
||||
# passes (a live closure while wired, None if the wiring
|
||||
# regresses) — the canned fake never fires it, so the status
|
||||
# scan fields stay 0/0 like a run that scanned nothing.
|
||||
self.scan_hooks: list[object] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -387,6 +404,7 @@ class FakeImportSources:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -397,6 +415,7 @@ class FakeImportSources:
|
||||
self.progress_hooks.append(progress)
|
||||
self.chunk_hooks.append(on_chunk_progress)
|
||||
self.image_hooks.append(on_image_progress)
|
||||
self.scan_hooks.append(on_scan_progress)
|
||||
self.ignore_maps.append(ignore_by_root or {})
|
||||
self.include_hidden_maps.append(include_hidden_by_root or {})
|
||||
self.doc_dates_maps.append(doc_dates_by_root or {})
|
||||
@@ -572,6 +591,10 @@ def test_admin_sync_success_reports_full_detail(
|
||||
# (the dataclass defaults).
|
||||
"images_done": 0,
|
||||
"images_total": 0,
|
||||
# Phase 194 (task 02): the pre-scan position keys — 0/0 idle
|
||||
# (the dataclass defaults).
|
||||
"scan_done": 0,
|
||||
"scan_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
@@ -703,17 +726,27 @@ def test_double_trigger_while_running_returns_409(
|
||||
assert r.status_code == 409
|
||||
assert r.json() == {"detail": "a sync is already running"}
|
||||
|
||||
# Phase 179 (task 02): WAIT for the import phase — the run's
|
||||
# progress is concurrent now that the git ops left the loop, so the
|
||||
# next request no longer inherits a run that already ran past them.
|
||||
body = _wait_phase(sync_client, "import")
|
||||
# Phase 179 (task 02): WAIT for the import to have started — the
|
||||
# run's progress is concurrent now that the git ops left the loop,
|
||||
# so the next request no longer inherits a run that already ran past
|
||||
# them. Phase 194 (task 02): the phase the wait targets is now
|
||||
# ``"scan"`` — the canned fake import reports NO file, and the
|
||||
# scan→import flip lives in the runner's per-file progress hook, so
|
||||
# a hook-less run legitimately stays in the pre-import phase (the
|
||||
# sequence with a real hook is pinned in test_sync_scan_phase.py and
|
||||
# in the phase-98 gated test below).
|
||||
body = _wait_phase(sync_client, "scan")
|
||||
assert body["state"] == "running"
|
||||
assert body["started_at"] is not None
|
||||
assert body["finished_at"] is None
|
||||
assert body["error"] is None
|
||||
# Wiring pin (phase 194, task 02): the runner passed its live
|
||||
# scan-position closure, and a fake that fires nothing keeps the
|
||||
# position at 0/0.
|
||||
assert len(fake_import.scan_hooks) == 1 and fake_import.scan_hooks[0] is not None
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
# Phase 98 (task 01): the in-flight run (parked in the fake
|
||||
# import's delay) reports the import phase — the summary span has
|
||||
# not started, so its counters are 0/0.
|
||||
# import's delay) reports no summary span — its counters are 0/0.
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
@@ -769,6 +802,10 @@ class _GatedImport:
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 194 (task 02): the PRE-SCAN position hook the runner
|
||||
# passes (recorded for the wiring pin — the gated fake never
|
||||
# fires it, so the status scan fields stay 0/0).
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -781,10 +818,12 @@ class _GatedImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
self.scan_hook = on_scan_progress
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
@@ -1937,6 +1976,7 @@ def test_import_error_is_reported_with_credentials_masked(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -2698,6 +2738,12 @@ _PRE_187_STATUS_KEYS = frozenset(
|
||||
#: Phase 187 (task 03): the two additive memory keys.
|
||||
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
|
||||
|
||||
#: Phase 194 (task 02): the two additive PRE-SCAN position keys — the
|
||||
#: next link in the same chain (the payload is exactly the pre-187 set
|
||||
#: plus the memory pair plus the scan pair, and a future addition joins
|
||||
#: this chain rather than editing the sets above).
|
||||
_SCAN_KEYS = frozenset({"scan_done", "scan_total"})
|
||||
|
||||
#: The pair is read from two DIFFERENT kernel counters and the high-water
|
||||
#: refreshes LAZILY, so a raw ``rss_peak_mb()`` really can sit under a raw
|
||||
#: ``rss_mb()`` (measured: ``rss_mb=164`` against ``rss_peak_mb=162``) —
|
||||
@@ -2744,13 +2790,17 @@ def test_memory_status_keys_are_the_only_addition_and_null_before_a_run(
|
||||
sync_client: TestClient,
|
||||
) -> None:
|
||||
"""The additive pin on the wire: the payload's key set is EXACTLY the
|
||||
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb``, and a run that
|
||||
never happened answers ``null`` / ``null`` — not 0, not a plausible
|
||||
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb`` plus the
|
||||
phase-194 ``scan_done`` / ``scan_total`` pair, and a run that never
|
||||
happened answers ``null`` / ``null`` — not 0, not a plausible
|
||||
number. That null is also what makes the Sources page render the
|
||||
counts line with no peak clause after an app restart."""
|
||||
_login(sync_client)
|
||||
body = sync_client.get("/api/sync/status").json()
|
||||
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, set(body)
|
||||
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS, set(body)
|
||||
# Phase 194 (task 02): the scan pair joins ADDITIVELY — idle is 0/0
|
||||
# (never null, never a plausible position).
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
assert body["rss_mb"] is None
|
||||
assert body["rss_peak_mb"] is None
|
||||
# The pre-existing shapes, unchanged: the state word, the two ISO
|
||||
@@ -2811,8 +2861,9 @@ def test_success_terminal_status_keeps_the_final_memory_pair(
|
||||
assert done[0].endswith(
|
||||
f"rss_mb={fmt_mb(current)} rss_peak_mb={fmt_mb(peak)}"
|
||||
), done[0]
|
||||
# The pre-existing payload is untouched by all of this.
|
||||
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS
|
||||
# The pre-existing payload is untouched by all of this (phase 194's
|
||||
# scan pair is the only addition, and it is additive).
|
||||
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS
|
||||
assert again["files_done"] == 0 or isinstance(again["files_done"], int)
|
||||
|
||||
|
||||
|
||||
@@ -58,6 +58,19 @@ clone's result. Python threads are not interruptible: the git worker
|
||||
runs its ``subprocess`` to completion, but the EVENT LOOP is free the
|
||||
whole time.
|
||||
|
||||
**Why phase 194 leaves these git-span pins alone (the D5 sweep).** The
|
||||
runner's import here is a stub (``fake_import`` below, which accepts the
|
||||
phase-194 ``on_scan_progress`` argument and fires nothing), so no real
|
||||
pre-scan runs inside this file's measurement window — the clone window
|
||||
this suite pins stays the only off-loop span it measures. The pre-scan's
|
||||
OWN loop-responsiveness pin (the same phase-179 pattern: a blocking
|
||||
``_file_hash`` over a multi-file fixture while a heartbeat coroutine keeps
|
||||
being scheduled, plus the worker-thread identity) lives in
|
||||
``tests/integration/test_sync_pending_count.py`` — beside the cancel at
|
||||
that span's await (the importer-side twin of this file's
|
||||
cancel-during-clone pin) and its app-side twin in
|
||||
``tests/integration/test_sync_scan_phase.py``.
|
||||
|
||||
**TestClient mechanics (why the trigger still rides a side thread).**
|
||||
The shared app loop here is anyio's ``BlockingPortal`` thread. Firing
|
||||
``POST /api/sync`` from the test thread would make the mid-clone probe
|
||||
@@ -326,6 +339,7 @@ class _FakeImport:
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None, # phase 194
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
|
||||
@@ -0,0 +1,571 @@
|
||||
"""Integration: phase 194 — the pre-scan predicts the change set, off the loop.
|
||||
|
||||
The owner report that opened the phase (2026-10-19, verbatim): *"I just synced
|
||||
the prod brain of reese and it said '1216 remaining' when it only had like 10
|
||||
files to sync."* Prod holds ~10,000 importable files and that run changed ~10.
|
||||
The phase-193 ``N remaining`` label was arithmetically fine — it was fed the
|
||||
WALK (every importable file) instead of the WORK. The fix lives in
|
||||
``app/rag/importer.py``: before the walk, ``_prescan_pending`` hashes every
|
||||
importable file once and reads ``documents.file_hash`` once per file — the
|
||||
exact predicate the phase-147 unchanged fast path applies — and the per-file
|
||||
``progress`` hook reports the position in THAT set.
|
||||
|
||||
What this file measures with a real ``import_sources`` run:
|
||||
|
||||
* **the parity that makes the number honest** — on a stable knowledge base
|
||||
the predicted set's size equals the run's terminal ``added + updated``: the
|
||||
fresh arm (everything pending, exactly what a first import indexes), the
|
||||
unchanged arm (nothing pending, so no count part at all), and the
|
||||
incremental arm (one changed file of four — the owner's prod case, where
|
||||
the walk is 4 and the work is 1);
|
||||
* **zero entity hydration** — the scan's statement projects exactly
|
||||
``documents.file_hash`` (no other column, no ORM row), so a 10,000-file
|
||||
scan retains nothing: the run's session is still empty at the first
|
||||
``progress`` call, before the processing loop has loaded any row (phase
|
||||
184/185);
|
||||
* **the span is OFF the loop** — with a stubbed hash that blocks, a heartbeat
|
||||
coroutine keeps ticking through the scan and the hash runs on a different
|
||||
thread than the one that awaited it (the phase-179 pattern of
|
||||
``test_sync_git_freeze.py``: the git clone used to park the single uvicorn
|
||||
loop and freeze the ``GET /api/sync/status`` poll the UI runs every 2 s,
|
||||
and a 10,000-file hash+SELECT span inline is the same defect class);
|
||||
* **the cancel contract** — cancelling the run at the scan's await lands the
|
||||
cancellation immediately, while the worker itself runs to completion
|
||||
(threads are not interruptible, phase 134) and nothing gets indexed.
|
||||
|
||||
The CLI and the upload job pass no ``progress``, so they run no scan at all;
|
||||
their byte-identity is pinned in ``tests/unit/test_sync_pending_count.py``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import event, select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.db import engine
|
||||
from app.models import Document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The probe's own source-name prefix — nothing else in the KB uses it.
|
||||
PREFIX = "phase194-it-"
|
||||
|
||||
#: A blocking hash of 60 ms per file: four files is a ~250 ms scan — long
|
||||
#: enough that a heartbeat polling every 5 ms cannot tick during it if the
|
||||
#: span ever runs on the loop thread, short enough that the suite stays fast.
|
||||
HASH_BLOCK_S = 0.06
|
||||
|
||||
#: The heartbeat tick count that proves the loop was free across ONE scanned
|
||||
#: file (60 ms / 5 ms ≈ 12 ticks; a blocked loop ticks 0). The ceiling-side
|
||||
#: assertion is the same number generous: this is loop-freeness, not timing.
|
||||
MIN_TICKS_PER_FILE = 4
|
||||
|
||||
|
||||
class ProgressCall(NamedTuple):
|
||||
"""One ``progress(source, rel, done, total)`` report."""
|
||||
|
||||
source: str
|
||||
rel: str
|
||||
done: int
|
||||
total: int
|
||||
|
||||
|
||||
class Run(NamedTuple):
|
||||
"""One real ``import_sources`` pass and everything it reported."""
|
||||
|
||||
summary: ImportSummary
|
||||
progress: list[ProgressCall]
|
||||
scan: list[tuple[int, int]]
|
||||
|
||||
|
||||
def _llm(**setting_overrides: Any) -> FakeEmbedder:
|
||||
"""The deterministic in-process fake on the SHIPPED defaults (the
|
||||
pending-count switch is therefore ON — the phase's default)."""
|
||||
from app.config import Settings
|
||||
|
||||
llm = FakeEmbedder()
|
||||
llm.settings = Settings(_env_file=None, **setting_overrides) # pyright: ignore[reportCallIssue]
|
||||
return llm
|
||||
|
||||
|
||||
def _root(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
|
||||
"""One source root (the ``file://``-style temp dir of the house
|
||||
import fixtures) with *files* under it."""
|
||||
assert name.startswith(PREFIX), f"the probe source must be {PREFIX}-prefixed"
|
||||
root = tmp_path / name
|
||||
for rel, body in files.items():
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _run(
|
||||
session: Session,
|
||||
*roots: Path,
|
||||
llm: FakeEmbedder | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Run:
|
||||
"""One real import over *roots*, collecting both position hooks."""
|
||||
progress: list[ProgressCall] = []
|
||||
scan: list[tuple[int, int]] = []
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
list(roots),
|
||||
llm or _llm(),
|
||||
session=session,
|
||||
progress=lambda s, r, d, t: progress.append(ProgressCall(s, r, d, t)),
|
||||
on_scan_progress=lambda p, t: scan.append((p, t)),
|
||||
**kwargs,
|
||||
)
|
||||
)
|
||||
return Run(summary, list(progress), list(scan))
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def statement_capture() -> Iterator[list[str]]:
|
||||
"""Every statement the engine executes for the duration of ONE test.
|
||||
|
||||
The ``before_cursor_execute`` technique of phase 183/185
|
||||
(``tests/integration/test_import_fast_path_columns.py``): what a SELECT
|
||||
actually projects is invisible in the ORM's Python API and plain in the
|
||||
statement it compiles, so a capture is the only way to pin "the pre-scan
|
||||
reads one column". Removed in teardown, so no other suite sees it.
|
||||
"""
|
||||
captured: list[str] = []
|
||||
|
||||
def _listener(
|
||||
conn: object, # noqa: ARG001
|
||||
cursor: object, # noqa: ARG001
|
||||
statement: str,
|
||||
parameters: object, # noqa: ARG001
|
||||
context: object, # noqa: ARG001
|
||||
executemany: bool, # noqa: ARG001
|
||||
) -> None:
|
||||
captured.append(statement)
|
||||
|
||||
event.listen(engine, "before_cursor_execute", _listener)
|
||||
try:
|
||||
yield captured
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", _listener)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_probe_documents(db: Session) -> Iterator[None]:
|
||||
"""``documents``/``chunks`` are global state — the probe's rows are
|
||||
removed before and after every test (the ``test_sync_api.py`` hygiene
|
||||
pattern, narrowed to this module's own source names)."""
|
||||
_drop(db)
|
||||
yield
|
||||
_drop(db)
|
||||
|
||||
|
||||
def _drop(db: Session) -> None:
|
||||
db.execute(text("DELETE FROM chunks WHERE document_id IN (SELECT id FROM "
|
||||
"documents WHERE source LIKE :p)"), {"p": f"{PREFIX}%"})
|
||||
db.execute(text("DELETE FROM documents WHERE source LIKE :p"), {"p": f"{PREFIX}%"})
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The parity: the predicted set IS the work
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
_FOUR_FILES = {
|
||||
"a.md": "# A\n\nalpha content\n",
|
||||
"b.md": "# B\n\nbravo content\n",
|
||||
"c.md": "# C\n\ncharlie content\n",
|
||||
"notes/d.md": "# D\n\ndelta content\n",
|
||||
}
|
||||
|
||||
|
||||
def test_fresh_kb_pending_set_equals_the_added_count(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Arm 1 — the first import: walk == work, so the numbers are the
|
||||
phase-193 ones (A1's degenerate parity, the reason the phase-193 E2E
|
||||
suite never saw the defect).
|
||||
|
||||
Every file has no row, so every file is pending: the hook reports
|
||||
``1/4 … 4/4`` and the terminal says ``added=4`` — the predicted size
|
||||
equals ``added + updated`` exactly."""
|
||||
root = _root(tmp_path, f"{PREFIX}fresh", _FOUR_FILES)
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.files, run.summary.added, run.summary.errors) == (4, 4, 0)
|
||||
assert [(c.done, c.total) for c in run.progress] == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert run.scan == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert run.progress[-1].total == run.summary.added + run.summary.updated
|
||||
|
||||
|
||||
def test_unchanged_kb_predicts_nothing_and_terminal_says_no_work(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Arm 2 — the no-change sync: nothing pending, so no count part.
|
||||
|
||||
The scan walks all four files and predicts none of them; every ``progress``
|
||||
call therefore carries ``(0, 0)``, which the phase-64 ``total === 0``
|
||||
guard renders as a bare label with no ``N remaining`` — the honest shape
|
||||
for a run with no work, where the pre-phase label said ``4 remaining``
|
||||
while walking. ``summary.files`` (the log's walked-file count) is
|
||||
untouched: the run still processed 4 files and still logged 4 unchanged.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}unchanged", _FOUR_FILES)
|
||||
seed = _run(db, root)
|
||||
assert seed.summary.added == 4
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.files, run.summary.added, run.summary.updated) == (4, 0, 0)
|
||||
assert run.summary.unchanged == 4
|
||||
assert [(c.done, c.total) for c in run.progress] == [(0, 0)] * 4
|
||||
# The scan still walked (and reported) every file — the DENOMINATOR of the
|
||||
# scan is the importable count, not the prediction.
|
||||
assert run.scan == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert run.summary.added + run.summary.updated == 0
|
||||
|
||||
|
||||
def test_incremental_kb_counts_the_changed_files_only(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Arm 3 — the owner's prod case at 1-of-4 scale.
|
||||
|
||||
One changed file out of four: the walk is still 4 (``summary.files``, the
|
||||
log line, and every ``progress`` call naming a file), but the count is the
|
||||
WORK — ``total`` 1, the position climbing only when the walk reaches the
|
||||
changed file, and holding at 0 while it moves over the unchanged ones.
|
||||
The parity pin: the prediction's size equals the terminal's
|
||||
``added + updated``.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}incremental", _FOUR_FILES)
|
||||
seed = _run(db, root)
|
||||
assert seed.summary.added == 4
|
||||
|
||||
(root / "notes" / "d.md").write_text(
|
||||
"# D\n\ndelta content EDITED on disk\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.files, run.summary.updated, run.summary.unchanged) == (
|
||||
4,
|
||||
1,
|
||||
3,
|
||||
)
|
||||
assert [(c.rel, c.done, c.total) for c in run.progress] == [
|
||||
("a.md", 0, 1),
|
||||
("b.md", 0, 1),
|
||||
("c.md", 0, 1),
|
||||
("notes/d.md", 1, 1),
|
||||
]
|
||||
assert run.progress[-1].total == run.summary.added + run.summary.updated
|
||||
|
||||
|
||||
def test_two_changed_files_count_down_to_zero(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""The count the owner asked for, at the scale the label is read at.
|
||||
|
||||
Two changed files among four: the denominator is 2 from the first report,
|
||||
the position advances only at the predicted files (the unchanged walk
|
||||
between them holds it), and ``total - done`` reaches 0 exactly when the
|
||||
last predicted file starts — where phase 193's ``N ≥ 1`` guard hides the
|
||||
count part. The terminal reports the same two files as updated."""
|
||||
root = _root(tmp_path, f"{PREFIX}two", _FOUR_FILES)
|
||||
_run(db, root)
|
||||
(root / "a.md").write_text("# A\n\nalpha edited\n", encoding="utf-8")
|
||||
(root / "notes" / "d.md").write_text(
|
||||
"# D\n\ndelta edited\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
run = _run(db, root)
|
||||
|
||||
assert (run.summary.updated, run.summary.unchanged) == (2, 2)
|
||||
assert [(c.rel, c.done, c.total) for c in run.progress] == [
|
||||
("a.md", 1, 2),
|
||||
("b.md", 1, 2), # unchanged: the walk moves, the count holds
|
||||
("c.md", 1, 2),
|
||||
("notes/d.md", 2, 2),
|
||||
]
|
||||
remaining = [c.total - c.done for c in run.progress]
|
||||
assert remaining == [1, 1, 1, 0], (
|
||||
f"the label's remaining count went {remaining} — it must hold across "
|
||||
"the unchanged files and reach 0 at the last predicted one (where "
|
||||
"phase 193's guard hides the part)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Zero entity hydration
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prescan_reads_one_column_and_hydrates_nothing(
|
||||
db: Session, tmp_path: Path, statement_capture: list[str]
|
||||
) -> None:
|
||||
"""The scan's statement projects EXACTLY ``documents.file_hash`` and the
|
||||
run's session is still empty when the walk begins.
|
||||
|
||||
Three pins on the same claim, because the failure mode is the 2026-10-15
|
||||
incident (a full walk hydrating 252 MB of markdown):
|
||||
|
||||
* every ``documents`` ``(source, path)`` SELECT issued before the first
|
||||
``progress`` call — i.e. by the scan — names ``file_hash`` and NOTHING
|
||||
else, no ``content`` above all (the phase-185 column-list technique);
|
||||
* exactly one such read per importable file;
|
||||
* ``len(session.identity_map) == 0`` at the first ``progress`` call: the
|
||||
scan put no ``Document`` anywhere, so the run's retention stays bounded
|
||||
by the phase-184 cadence instead of growing with the KB.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}hydrate", _FOUR_FILES)
|
||||
_run(db, root)
|
||||
db.expunge_all()
|
||||
statement_capture.clear()
|
||||
|
||||
map_at_first_progress: list[int] = []
|
||||
|
||||
def progress(source: str, rel: str, done: int, total: int) -> None: # noqa: ARG001
|
||||
map_at_first_progress.append(len(db.identity_map))
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], _llm(), session=db, progress=progress, on_scan_progress=None
|
||||
)
|
||||
)
|
||||
assert summary.unchanged == 4
|
||||
|
||||
# The scan reads precede the loop's row loads in capture order, and the
|
||||
# first progress call is the boundary between them. Classified on the
|
||||
# PROJECTION (phase 185's technique): the scan's projection is the single
|
||||
# column token, the fast path's is every column but ``content``.
|
||||
def predicate(stmt: str) -> str:
|
||||
"""Everything after the statement's ``FROM`` (case-insensitive —
|
||||
SQLAlchemy compiles the keyword upper-case)."""
|
||||
return re.split(r"\bfrom\s+documents\b", stmt, maxsplit=1, flags=re.I)[-1]
|
||||
|
||||
doc_reads = [
|
||||
stmt
|
||||
for stmt in statement_capture
|
||||
if stmt.lstrip().lower().startswith("select")
|
||||
and re.search(r"\bfrom\s+documents\b", stmt, re.I)
|
||||
and "documents.source" in predicate(stmt)
|
||||
and "documents.path" in predicate(stmt)
|
||||
]
|
||||
|
||||
def projection(stmt: str) -> set[str]:
|
||||
head = re.split(r"\bfrom\b", stmt, maxsplit=1, flags=re.I)[0]
|
||||
return set(re.findall(r"documents\.([a-z_]+)", head))
|
||||
|
||||
scan_reads = [s for s in doc_reads if projection(s) == {"file_hash"}]
|
||||
row_loads = [s for s in doc_reads if projection(s) != {"file_hash"}]
|
||||
assert len(scan_reads) == 4, (
|
||||
f"the unchanged run issued {len(scan_reads)} single-column file_hash "
|
||||
"reads — the scan reads each importable file's hash exactly once"
|
||||
)
|
||||
assert len(row_loads) == 4, (
|
||||
f"the loop issued {len(row_loads)} fast-path row loads — the scan must "
|
||||
"not replace _doc_row_stmt, which stays the only ROW load (phase 185)"
|
||||
)
|
||||
assert all(
|
||||
projection(s) == {"file_hash"} for s in scan_reads
|
||||
), "a scan read projects more than file_hash — it would hydrate a row"
|
||||
assert all(
|
||||
"content" not in projection(s) for s in row_loads
|
||||
), "the fast-path row load stopped deferring content (phase 185)"
|
||||
assert map_at_first_progress and map_at_first_progress[0] == 0, (
|
||||
"the run's session already held entities when the loop reached its "
|
||||
"first file — the pre-scan leaked a row into the identity map"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The span is off the event loop (the phase-179 pattern)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _SlowHash:
|
||||
"""The timing fixture: a PLAIN blocking ``_file_hash`` (like the real one
|
||||
— the point under test is WHERE it runs), counting its calls.
|
||||
|
||||
The thread identity is NOT taken from here: ``_file_hash`` is called both
|
||||
by the scan (worker thread) and by the processing loop (loop thread), so
|
||||
the attribution comes from ``on_scan_progress``, which fires on the same
|
||||
worker thread as the scan's own hashes."""
|
||||
|
||||
def __init__(self, real: Callable[[Path], str], block: float = HASH_BLOCK_S) -> None:
|
||||
self.real = real
|
||||
self.block = block
|
||||
self.calls = 0
|
||||
|
||||
def __call__(self, path: Path) -> str:
|
||||
self.calls += 1
|
||||
time.sleep(self.block)
|
||||
return self.real(path)
|
||||
|
||||
|
||||
def test_prescan_span_does_not_park_the_event_loop(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE phase-179 contract, re-pinned for the pre-scan.
|
||||
|
||||
A 4-file scan with a 60 ms blocking hash is ~250 ms of work that used to
|
||||
be inline. The heartbeat coroutine polls every 5 ms; the ticks between two
|
||||
consecutive scan reports prove the LOOP kept running while the worker
|
||||
blocked (a regression that runs the scan inline parks the loop thread and
|
||||
the heartbeat cannot tick — the task-01 git measurement was 0 ticks over
|
||||
a 3 s clone). The thread identities are asserted too, so the pin fails on
|
||||
a machine fast enough to hide the latency.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}freeze", _FOUR_FILES)
|
||||
stub = _SlowHash(importer._file_hash)
|
||||
monkeypatch.setattr(importer, "_file_hash", stub)
|
||||
|
||||
ticks = 0
|
||||
scan_ticks: list[int] = []
|
||||
scan_positions: list[tuple[int, int]] = []
|
||||
scan_thread: list[int] = []
|
||||
|
||||
def scan_hook(position: int, total: int) -> None:
|
||||
# Runs on the span's worker thread — the same thread that just hashed.
|
||||
scan_positions.append((position, total))
|
||||
scan_ticks.append(ticks)
|
||||
scan_thread.append(threading.get_ident())
|
||||
|
||||
# A progress caller is what opens the gate (the CLI shape, ``progress=None``,
|
||||
# would skip the scan entirely) — this is the UI sync's shape.
|
||||
async def main() -> int:
|
||||
nonlocal ticks
|
||||
loop_ident = threading.get_ident()
|
||||
|
||||
async def heartbeat() -> None:
|
||||
nonlocal ticks
|
||||
while True:
|
||||
await asyncio.sleep(0.005)
|
||||
ticks += 1
|
||||
|
||||
hb = asyncio.create_task(heartbeat())
|
||||
try:
|
||||
await import_sources(
|
||||
[root],
|
||||
_llm(),
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: None,
|
||||
on_scan_progress=scan_hook,
|
||||
)
|
||||
finally:
|
||||
hb.cancel()
|
||||
return loop_ident
|
||||
|
||||
loop_ident = asyncio.run(main())
|
||||
|
||||
assert scan_positions == [(1, 4), (2, 4), (3, 4), (4, 4)]
|
||||
assert scan_thread and loop_ident is not None
|
||||
assert all(ident != loop_ident for ident in scan_thread), (
|
||||
"the pre-scan ran on the event-loop thread — a long scan parks the "
|
||||
"single uvicorn loop again (phase 179's defect class)"
|
||||
)
|
||||
deltas = [
|
||||
b - a for a, b in zip([0, *scan_ticks], scan_ticks, strict=False)
|
||||
]
|
||||
assert min(deltas) >= MIN_TICKS_PER_FILE, (
|
||||
f"the heartbeat ticked {deltas} times across consecutive scanned files "
|
||||
f"(expected >= {MIN_TICKS_PER_FILE} each) — the loop was starved while "
|
||||
"the scan blocked, so the span is not off-loop"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The cancel (phase 134): the run stops, the worker finishes
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cancel_during_the_prescan_stops_the_run_while_the_worker_finishes(
|
||||
db: Session, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The phase-134 cooperative cancel, measured at the scan's await.
|
||||
|
||||
Cancelling the run ~80 ms into a ~250 ms scan raises ``CancelledError``
|
||||
out of ``import_sources`` PROMPTLY (well before the scan's own remaining
|
||||
work), NOTHING is indexed, and the worker thread runs to completion —
|
||||
threads are not interruptible, and this is exactly why the span reads
|
||||
through a session of its own instead of the run's (which the cancel
|
||||
closes on its way out). The pre-phase-194 pipeline had no await here at
|
||||
all, so this is the new boundary the UI's Stop button must survive.
|
||||
"""
|
||||
root = _root(tmp_path, f"{PREFIX}cancel", _FOUR_FILES)
|
||||
stub = _SlowHash(importer._file_hash, block=0.12)
|
||||
monkeypatch.setattr(importer, "_file_hash", stub)
|
||||
|
||||
real_prescan = importer._prescan_pending
|
||||
worker_finished = threading.Event()
|
||||
worker_results: list[Any] = []
|
||||
|
||||
def spy(*args: Any, **kwargs: Any) -> Any:
|
||||
try:
|
||||
result = real_prescan(*args, **kwargs)
|
||||
worker_results.append(result)
|
||||
return result
|
||||
finally:
|
||||
worker_finished.set()
|
||||
|
||||
monkeypatch.setattr(importer, "_prescan_pending", spy)
|
||||
|
||||
async def main() -> float:
|
||||
# The hook runs on the WORKER thread, so it only appends to a list —
|
||||
# the house hook contract (a GIL-atomic store). Touching an
|
||||
# ``asyncio.Event`` from here would be a cross-thread loop call.
|
||||
seen: list[tuple[int, int]] = []
|
||||
|
||||
def scan_hook(position: int, total: int) -> None:
|
||||
seen.append((position, total))
|
||||
|
||||
task = asyncio.create_task(
|
||||
import_sources(
|
||||
[root],
|
||||
_llm(),
|
||||
session=db,
|
||||
progress=lambda s, r, d, t: None,
|
||||
on_scan_progress=scan_hook,
|
||||
)
|
||||
)
|
||||
# Wait (from the loop) until the scan has reported its first file.
|
||||
deadline = time.monotonic() + 10.0
|
||||
while not seen and time.monotonic() < deadline:
|
||||
await asyncio.sleep(0.005)
|
||||
assert seen, "the pre-scan never reported a file"
|
||||
await asyncio.sleep(0.08)
|
||||
t0 = time.monotonic()
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
return time.monotonic() - t0
|
||||
|
||||
cancel_latency = asyncio.run(main())
|
||||
|
||||
# The cancel is prompt: the run did NOT wait for the remaining scan work.
|
||||
assert cancel_latency < 0.5, (
|
||||
f"the cancelled run took {cancel_latency:.3f}s to unblock — it is "
|
||||
"still waiting for the blocking scan worker"
|
||||
)
|
||||
# The worker is not interruptible: it finishes its own session's work.
|
||||
assert worker_finished.wait(10.0), "the pre-scan worker never completed"
|
||||
assert stub.calls == 4, (
|
||||
f"the cancelled run's scan worker hashed {stub.calls} of 4 files — "
|
||||
"the contract is that the worker runs to completion"
|
||||
)
|
||||
# And the pipeline that would have consumed the prediction never ran.
|
||||
assert db.scalar(
|
||||
select(select(Document).where(Document.source == root.name).exists())
|
||||
) is False, "the cancelled run indexed documents — the import never started"
|
||||
@@ -0,0 +1,927 @@
|
||||
"""Integration: phase 194 (task 02) — the ``"scan"`` phase on the status surface.
|
||||
|
||||
The owner report that opened the phase (2026-10-19): *"it said '1216
|
||||
remaining' when it only had like 10 files to sync"*. Task 01 moved the
|
||||
count onto the pre-scan's PREDICTED change set; this file pins the
|
||||
status half (design D3, owner-confirmed A2 — the pre-scan is a VISIBLE
|
||||
stage, not an invisible pause):
|
||||
|
||||
* the phase machine is ``null`` → ``"scan"`` (set immediately before
|
||||
``import_sources``, where the scan actually runs) → ``"import"``
|
||||
(flipped by the FIRST per-file ``progress`` call) → ``"overview"`` →
|
||||
``"summaries"`` → the terminal clears ``phase``;
|
||||
* ``scan_done`` / ``scan_total`` carry the scan's own position through
|
||||
that stage, reset with the run, and are KEPT at every terminal beside
|
||||
the final file counts (the phase-64 keep-final-values convention);
|
||||
* the payload is ADDITIVE — the pre-phase key set plus exactly those two
|
||||
keys, byte-identical in name and type, stable from idle through the
|
||||
terminal;
|
||||
* the phase-187 sampling discipline holds on the new stage: the ``"scan"``
|
||||
set and the ``scan``→``import`` flip are TRANSITIONS and force the
|
||||
``/proc`` read, while the scan's own per-file hook is THROTTLED — one
|
||||
read per interval, never one per file (LOCKED A4);
|
||||
* the two degenerate/terminal edges: an empty knowledge base sits in
|
||||
``"scan"`` and never reaches ``"import"`` (no ``progress`` call exists
|
||||
to flip it) and a cancel DURING the scan lands the ``cancelled``
|
||||
terminal with the scan counts kept and nothing indexed (the phase-134
|
||||
contract, the app-side twin of the importer-side pin in
|
||||
``tests/integration/test_sync_pending_count.py``).
|
||||
|
||||
**Why one arm is choreographed and one is real.** An in-process import
|
||||
never awaits real I/O (the embedder is a fake), so the loop cannot answer a
|
||||
status poll from inside it — a real run's ``"import"`` phase is therefore
|
||||
NOT reliably observable, and the real arm pins what the terminal keeps
|
||||
(both pairs, divergent on the incremental run). The flip itself gets a
|
||||
deterministic pin from the choreographed double (the ``test_sync_api``
|
||||
gated-import idiom), which drives the runner's OWN closures and parks at
|
||||
every frame the test reads. The scan, by contrast, runs in a
|
||||
``to_thread`` worker, so the loop is FREE through it — that is exactly why
|
||||
its frames are observable, and why the blocking-hash fixture slows ONLY the
|
||||
scan window (a blanket stub would park the loop through the import too).
|
||||
|
||||
House patterns (``test_sync_api.py`` / ``test_sync_git_freeze.py``): the
|
||||
``sync_client`` fixture is a CONTEXT-MANAGED ``TestClient`` — one app event
|
||||
loop across requests, so the background run survives between the POST and
|
||||
the polls; the module ``_status`` / ``_task`` / ``_cancel_event`` /
|
||||
``_rss_sampled_at`` and the ``git_sources`` / ``sources_meta`` /
|
||||
``folder_summaries`` / ``documents`` global state are reset around every
|
||||
test; the probe is stubbed so nothing reaches the network, and the import
|
||||
runs for real over a temp ``kind=local`` source with the deterministic
|
||||
``FakeEmbedder``. The scan is made observable with a PLAIN blocking
|
||||
``_file_hash`` stub scoped to the scan window (the phase-179 timing-fixture
|
||||
idiom): the point under test is WHICH phase the run is in WHEN the page
|
||||
polls, and a scan of instant hashes is over before the first poll.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.api import sync as sync_api
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document, GitSource
|
||||
from app.rag import git_sources as git_sources_resolver
|
||||
from app.rag.importer import ImportSummary
|
||||
from app.rag.llm import LLMClient
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The probe's own source-name prefix — nothing else in the KB uses it,
|
||||
#: so the module's DELETE-by-source cleanup can never touch another
|
||||
#: suite's rows.
|
||||
PREFIX = "phase194-t02-"
|
||||
|
||||
#: One blocking ``_file_hash`` per file. Three files is a ~360 ms scan:
|
||||
#: long enough that a 10 ms poll catches several distinct positions inside
|
||||
#: the phase (the frames are what the UI's 2 s poll would render), short
|
||||
#: enough that the suite stays fast.
|
||||
HASH_BLOCK_S = 0.12
|
||||
|
||||
|
||||
# --- fixtures (the house sync hygiene) --------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_sync_state() -> Iterator[None]:
|
||||
"""The module-level status object + task + cancel event + sampling
|
||||
guard are process-global — reset before AND after every test so no
|
||||
terminal state, stale cancel or stale throttle window leaks into the
|
||||
next one (``test_sync_api`` / ``test_sync_git_freeze``)."""
|
||||
sync_api._status = sync_api.SyncStatus()
|
||||
sync_api._task = None
|
||||
sync_api._cancel_event = asyncio.Event()
|
||||
sync_api._rss_sampled_at = float("-inf")
|
||||
yield
|
||||
sync_api._status = sync_api.SyncStatus()
|
||||
sync_api._task = None
|
||||
sync_api._cancel_event = asyncio.Event()
|
||||
sync_api._rss_sampled_at = float("-inf")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_git_sources(db: Session) -> Iterator[None]:
|
||||
"""The runner resolves through the real ``git_sources`` table —
|
||||
truncated around every test (the rows this file seeds are
|
||||
``kind=local``)."""
|
||||
db.execute(text("TRUNCATE git_sources"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE git_sources"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_sources_version() -> Iterator[None]:
|
||||
"""The phase-53 bump is global mutable state — pin it to the
|
||||
migration-0010 seed (0) around every test (own session: the runner
|
||||
bumps through its own short-lived ``SessionLocal``)."""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
session = SessionLocal()
|
||||
try:
|
||||
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
||||
session.commit()
|
||||
yield
|
||||
finally:
|
||||
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
||||
session.commit()
|
||||
session.close()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_folder_summaries(db: Session) -> Iterator[None]:
|
||||
"""``folder_summaries`` is global state the real generator writes on a
|
||||
changed KB — truncated around every test."""
|
||||
db.execute(text("TRUNCATE folder_summaries"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE folder_summaries"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_probe_documents(db: Session) -> Iterator[None]:
|
||||
"""The real imports of this module write ``documents``/``chunks`` —
|
||||
removed before and after every test, narrowed to this module's own
|
||||
source names (the ``test_sync_pending_count`` hygiene)."""
|
||||
_drop(db)
|
||||
yield
|
||||
_drop(db)
|
||||
|
||||
|
||||
def _drop(db: Session) -> None:
|
||||
db.execute(
|
||||
text("DELETE FROM chunks WHERE document_id IN "
|
||||
"(SELECT id FROM documents WHERE source LIKE :p)"),
|
||||
{"p": f"{PREFIX}%"},
|
||||
)
|
||||
db.execute(text("DELETE FROM documents WHERE source LIKE :p"), {"p": f"{PREFIX}%"})
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def sync_client() -> Iterator[TestClient]:
|
||||
"""Context-managed ``TestClient`` — one app event loop across requests
|
||||
(the background run must survive between the POST and the polls)."""
|
||||
with TestClient(fastapi_app) as client:
|
||||
yield client
|
||||
|
||||
|
||||
# --- helpers ----------------------------------------------------------------
|
||||
|
||||
|
||||
def _settings(sources_dir: str) -> Settings:
|
||||
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Empty env fallback (the dev ``.env`` never leaks in) — this file's
|
||||
runs resolve their one local row from the table."""
|
||||
monkeypatch.setattr(
|
||||
git_sources_resolver,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, git_sources=""), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
|
||||
|
||||
def _seed_local(db: Session, path: Path) -> None:
|
||||
"""A ``kind=local`` row exactly as the phase-38 API stores it."""
|
||||
db.add(GitSource(url=str(path), kind="local", path=str(path)))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _local_source(tmp_path: Path, name: str, files: dict[str, str]) -> Path:
|
||||
"""One source root (always created — the empty-KB arm needs a real
|
||||
empty directory, not a missing one), with *files* under it."""
|
||||
assert name.startswith(PREFIX), f"the probe source must be {PREFIX}-prefixed"
|
||||
root = tmp_path / name
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
for rel, body in files.items():
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _login(client: TestClient) -> None:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
||||
|
||||
|
||||
def _poll(client: TestClient, want: str, timeout: float = 20.0) -> dict:
|
||||
"""Poll until ``state == want``, failing loudly on any other state."""
|
||||
deadline = time.monotonic() + timeout
|
||||
body: dict = {}
|
||||
while time.monotonic() < deadline:
|
||||
body = client.get("/api/sync/status").json()
|
||||
if body["state"] == want:
|
||||
return body
|
||||
assert body["state"] == "running", (
|
||||
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
|
||||
)
|
||||
time.sleep(0.02)
|
||||
raise AssertionError(f"sync did not reach {want!r} within {timeout}s: {body}")
|
||||
|
||||
|
||||
def _watch_scan(client: TestClient, timeout: float = 20.0) -> tuple[list[dict], dict | None]:
|
||||
"""Poll through the run's SCAN window; stop at the first frame that is
|
||||
no longer a scan.
|
||||
|
||||
Returns ``(frames, terminal)`` — ``terminal`` is non-``None`` when the
|
||||
run finished inside the window. Both stops are honest: the scan runs
|
||||
OFF the loop (so its frames are served, and this list is exactly what
|
||||
the label was built from), while the import that follows is one
|
||||
straight run of the importer that can finish before the next poll.
|
||||
The flip frame therefore gets pinned deterministically by the
|
||||
choreographed test below; what the real run pins here is the scan's
|
||||
own frames plus the values the terminal keeps."""
|
||||
frames: list[dict] = []
|
||||
terminal: dict | None = None
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
body = client.get("/api/sync/status").json()
|
||||
if body["state"] != "running":
|
||||
terminal = body
|
||||
break
|
||||
frames.append(body)
|
||||
if body["phase"] != "scan":
|
||||
break
|
||||
time.sleep(0.01)
|
||||
else:
|
||||
raise AssertionError(f"the run never left the scan phase: {frames[-1]}")
|
||||
return frames, terminal
|
||||
|
||||
|
||||
class _FakeScanImport:
|
||||
"""A parked ``import_sources`` double for the app-side edges.
|
||||
|
||||
It records the two position closures the runner passes (the wiring
|
||||
pin) and fires NOTHING — which is exactly what a run with no
|
||||
importable file does (no scan report, so no position, and no
|
||||
``progress`` call to flip the phase; the importer-side proof lives in
|
||||
the empty-KB arm below and in ``test_sync_pending_count.py``). The
|
||||
park lets the test read the status inside the ``"scan"`` phase without
|
||||
racing the pipeline; the caller always releases it in ``finally``.
|
||||
"""
|
||||
|
||||
def __init__(self, summary: ImportSummary | None = None) -> None:
|
||||
self.summary = summary if summary is not None else ImportSummary()
|
||||
self.started = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.progress_hook: Callable[[str, str, int, int], None] | None = None
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: LLMClient,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
) -> ImportSummary:
|
||||
self.progress_hook = progress
|
||||
self.scan_hook = on_scan_progress
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
def _install_counting_memory(monkeypatch: pytest.MonkeyPatch) -> list[dict[str, Any]]:
|
||||
"""Replace the two memory helpers on ``app.api.sync`` with recorders.
|
||||
|
||||
``_sample_rss`` calls them ONLY when the throttle lets it, so one
|
||||
entry per entry below is one real read — and each entry carries the
|
||||
phase (and the scan/file position) at that instant, which is the
|
||||
difference between "a transition forced a read" and "a per-file hook
|
||||
read once per file" (LOCKED A4)."""
|
||||
reads: list[dict[str, Any]] = []
|
||||
|
||||
def _record() -> None:
|
||||
reads.append(
|
||||
{
|
||||
"phase": sync_api._status.phase,
|
||||
"scan_done": sync_api._status.scan_done,
|
||||
"files_done": sync_api._status.files_done,
|
||||
"current_file": sync_api._status.current_file,
|
||||
}
|
||||
)
|
||||
|
||||
def fake_rss_mb() -> int:
|
||||
_record()
|
||||
return 100
|
||||
|
||||
def fake_rss_peak_mb() -> int:
|
||||
return 120
|
||||
|
||||
monkeypatch.setattr(sync_api, "rss_mb", fake_rss_mb)
|
||||
monkeypatch.setattr(sync_api, "rss_peak_mb", fake_rss_peak_mb)
|
||||
return reads
|
||||
|
||||
|
||||
def _slow_hash(
|
||||
monkeypatch: pytest.MonkeyPatch, block: float = HASH_BLOCK_S
|
||||
) -> dict[str, bool]:
|
||||
"""Make ONLY the pre-scan window slow, and return its flag.
|
||||
|
||||
``_file_hash`` is called by the scan AND by the processing loop, so a
|
||||
blanket blocking stub would also park the event loop through the whole
|
||||
import (an in-process import never awaits real I/O, so the loop cannot
|
||||
answer a status poll inside it) — the frames this file needs would
|
||||
then be unservable rather than absent. The flag is raised around
|
||||
:func:`app.rag.importer._prescan_pending` only, so the scan's window is
|
||||
the fixture's window: the loop stays free exactly while the run is in
|
||||
the phase under test (the phase-179 off-loop claim, observed from the
|
||||
HTTP side), and the import after it runs at full speed.
|
||||
|
||||
The flag is a plain ``dict`` entry written/read from two threads: the
|
||||
house hook contract (a GIL-atomic store) applies here for the same
|
||||
reason it applies to the runner's closures.
|
||||
"""
|
||||
real_hash = importer._file_hash
|
||||
real_prescan = importer._prescan_pending
|
||||
state = {"in_scan": False}
|
||||
|
||||
def slow(path: Path) -> str:
|
||||
if state["in_scan"]:
|
||||
time.sleep(block)
|
||||
return real_hash(path)
|
||||
|
||||
def spy_prescan(*args: Any, **kwargs: Any) -> Any:
|
||||
state["in_scan"] = True
|
||||
try:
|
||||
return real_prescan(*args, **kwargs)
|
||||
finally:
|
||||
state["in_scan"] = False
|
||||
|
||||
monkeypatch.setattr(importer, "_file_hash", slow)
|
||||
monkeypatch.setattr(importer, "_prescan_pending", spy_prescan)
|
||||
return state
|
||||
|
||||
|
||||
class _ChoreographedImport:
|
||||
"""An ``import_sources`` double that replays the importer's hook script
|
||||
through the runner's OWN closures, parking at every point the test
|
||||
reads the status.
|
||||
|
||||
The choreography (all four stops are ``threading.Event`` pairs, so no
|
||||
test here races the pipeline):
|
||||
|
||||
1. fire ``scan_pairs[:1]`` through ``on_scan_progress`` → ``first``
|
||||
(the test reads the scan's FIRST position);
|
||||
2. on ``go_on``: fire the rest → ``scanned`` (the test reads the last
|
||||
position, still inside the phase);
|
||||
3. on ``go_import``: fire ``progress_call`` through ``progress`` —
|
||||
THE FLIP → ``flipped`` (the test reads the ``"import"`` frame);
|
||||
4. on ``release``: return the canned summary.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
summary: ImportSummary,
|
||||
scan_pairs: list[tuple[int, int]],
|
||||
progress_call: tuple[str, str, int, int],
|
||||
) -> None:
|
||||
self.summary = summary
|
||||
self.scan_pairs = list(scan_pairs)
|
||||
self.progress_call = progress_call
|
||||
self.first = threading.Event()
|
||||
self.go_on = threading.Event()
|
||||
self.scanned = threading.Event()
|
||||
self.go_import = threading.Event()
|
||||
self.flipped = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.scan_hook: Callable[[int, int], None] | None = None
|
||||
self.progress_hook: Callable[[str, str, int, int], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: LLMClient,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
) -> ImportSummary:
|
||||
# The wiring pin: both closures arrive live (None here would mean
|
||||
# the runner stopped passing them).
|
||||
self.scan_hook = on_scan_progress
|
||||
self.progress_hook = progress
|
||||
assert on_scan_progress is not None and progress is not None
|
||||
for pair in self.scan_pairs[:1]:
|
||||
on_scan_progress(*pair)
|
||||
self.first.set()
|
||||
await asyncio.to_thread(self.go_on.wait)
|
||||
for pair in self.scan_pairs[1:]:
|
||||
on_scan_progress(*pair)
|
||||
self.scanned.set()
|
||||
await asyncio.to_thread(self.go_import.wait)
|
||||
progress(*self.progress_call)
|
||||
self.flipped.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
def _stub_probe(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
async def fake_check_models(llm: LLMClient) -> None:
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(sync_api, "check_models", fake_check_models)
|
||||
|
||||
|
||||
def _install_local_run(
|
||||
monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path, source: Path
|
||||
) -> None:
|
||||
"""Point one app run at *source*: local row, tmp sources root, stubbed
|
||||
probe, the deterministic in-process embedder (the real import, no
|
||||
network)."""
|
||||
_seed_local(db, source)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api, "get_settings", lambda: _settings(sources_dir=str(tmp_path / "bor"))
|
||||
)
|
||||
_stub_probe(monkeypatch)
|
||||
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder())
|
||||
|
||||
|
||||
_THREE_FILES = {
|
||||
"a.md": "# A\n\nalpha content\n",
|
||||
"b.md": "# B\n\nbravo content\n",
|
||||
"notes/c.md": "# C\n\ncharlie content\n",
|
||||
}
|
||||
|
||||
|
||||
def _log_lines(caplog: pytest.LogCaptureFixture, prefix: str) -> list[str]:
|
||||
return [r.getMessage() for r in caplog.records if r.getMessage().startswith(prefix)]
|
||||
|
||||
|
||||
# --- the phase-machine sequence --------------------------------------------
|
||||
|
||||
|
||||
def test_real_run_reports_the_scan_frames_and_keeps_both_pairs(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""D3 through the REAL pipeline, twice (one changed file included).
|
||||
|
||||
Run 1 (a first import): the prelude reports ``phase: null`` with
|
||||
``scan_*`` 0/0, then the run reports ``"scan"`` with ``scan_total`` at
|
||||
the importable count and ``scan_done`` climbing through AT LEAST two
|
||||
distinct positions (the frames the ``Scanning… (n/total)`` label
|
||||
needs) with ``current_file`` null throughout — the scan names no file.
|
||||
The terminal clears ``phase`` and KEEPS the final scan pair beside the
|
||||
file pair, and on a first import both pairs are 3/3 (A1's degenerate
|
||||
parity: walk == work, the only shape the phase-193 suite ever saw).
|
||||
|
||||
Run 2 (one file edited — the owner's prod shape at 1-of-3): the scan
|
||||
denominator is STILL the importable 3, while the kept file pair is
|
||||
``1/1`` and the terminal reports exactly one updated document. That
|
||||
divergence — the scan counted the walk, the count counted the work —
|
||||
IS the phase, and it survives on the terminal where no poll can miss
|
||||
it."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}machine", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_slow_hash(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
idle = sync_client.get("/api/sync/status").json()
|
||||
assert idle["phase"] is None
|
||||
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
frames, _late = _watch_scan(sync_client)
|
||||
scan_frames = [f for f in frames if f["phase"] == "scan"]
|
||||
assert scan_frames, f"the run never reported the scan phase: {frames}"
|
||||
assert all(f["current_file"] is None for f in scan_frames), (
|
||||
"the scan named a file — current_file stays null until the first "
|
||||
"progress call flips the phase to import"
|
||||
)
|
||||
# ``0`` is the transition frame — the phase is set before the scan
|
||||
# reports its first file, and the UI's total===0 guard renders it bare.
|
||||
totals = {f["scan_total"] for f in scan_frames}
|
||||
assert totals <= {0, 3} and max(totals) == 3, totals
|
||||
positions = sorted({f["scan_done"] for f in scan_frames if f["scan_total"] == 3})
|
||||
assert len(positions) >= 2, (
|
||||
f"only scan position(s) {positions} were observable — the label needs "
|
||||
"the scan's own n/total to move (the fixture scan blocks on purpose)"
|
||||
)
|
||||
assert positions[0] >= 1 and positions[-1] <= 3
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
assert body["detail"]["added"] == 3
|
||||
assert body["phase"] is None and body["current_file"] is None
|
||||
# The keep-final-values convention, beside the file counts.
|
||||
assert body["scan_total"] == 3 and body["scan_done"] == 3
|
||||
assert body["files_total"] == 3 and body["files_done"] == 3
|
||||
|
||||
# Run 2 — one changed file of three (the owner's prod case, scaled).
|
||||
(source / "b.md").write_text("# B\n\nbravo content EDITED\n", encoding="utf-8")
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
frames2, _late2 = _watch_scan(sync_client)
|
||||
scan2 = [f for f in frames2 if f["phase"] == "scan"]
|
||||
totals2 = {f["scan_total"] for f in scan2 if f["scan_total"]}
|
||||
assert scan2 and totals2 == {3}, (
|
||||
"the scan denominator is the importable count, not the prediction"
|
||||
)
|
||||
assert sorted({f["scan_done"] for f in scan2 if f["scan_total"] == 3})[-1] >= 2
|
||||
|
||||
body2 = _poll(sync_client, "success")
|
||||
assert (body2["detail"]["added"], body2["detail"]["updated"]) == (0, 1)
|
||||
assert body2["phase"] is None
|
||||
# The divergence the phase exists for: the kept scan pair is the WALK
|
||||
# it did (3 files hashed), the kept file pair is the WORK (1 file).
|
||||
assert (body2["scan_done"], body2["scan_total"]) == (3, 3)
|
||||
assert (body2["files_done"], body2["files_total"]) == (1, 1)
|
||||
|
||||
|
||||
def test_scan_flips_to_import_on_the_first_progress_call(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The sequence, frame by frame, with the flip pinned.
|
||||
|
||||
The in-process import runs straight through without awaiting real I/O,
|
||||
so a poll cannot reliably land inside the ``"import"`` phase (that is
|
||||
why the real arm above pins the terminal's kept pairs). This arm drives
|
||||
the runner's OWN closures through a choreographed double — the
|
||||
``test_sync_api`` gated-import idiom — and reads the status at each
|
||||
stop: prelude ``null``/0/0 → ``"scan"`` at position 1 of 3 → still
|
||||
``"scan"`` at position 3 of 3 → ``"import"`` with the PREDICTED
|
||||
``files_total`` (1, the count that reaches ``N remaining``) and the
|
||||
scan pair still showing 3/3 → the terminal, which clears ``phase`` and
|
||||
keeps both pairs."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}flip", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
gated = _ChoreographedImport(
|
||||
ImportSummary(files=3, updated=1, unchanged=2),
|
||||
scan_pairs=[(1, 3), (2, 3), (3, 3)],
|
||||
progress_call=(source.name, "b.md", 1, 1),
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "import_sources", gated)
|
||||
|
||||
_login(sync_client)
|
||||
idle = sync_client.get("/api/sync/status").json()
|
||||
assert idle["state"] == "idle"
|
||||
assert idle["phase"] is None
|
||||
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert gated.first.wait(10.0), "the run never reached the scan"
|
||||
s1 = sync_client.get("/api/sync/status").json()
|
||||
assert s1["state"] == "running"
|
||||
assert s1["phase"] == "scan"
|
||||
assert (s1["scan_done"], s1["scan_total"]) == (1, 3)
|
||||
assert s1["current_file"] is None
|
||||
# Still the pre-import phase: no file count yet.
|
||||
assert (s1["files_done"], s1["files_total"]) == (0, 0)
|
||||
|
||||
gated.go_on.set()
|
||||
assert gated.scanned.wait(10.0), "the scan never reported its last file"
|
||||
s2 = sync_client.get("/api/sync/status").json()
|
||||
assert s2["phase"] == "scan"
|
||||
assert (s2["scan_done"], s2["scan_total"]) == (3, 3)
|
||||
|
||||
gated.go_import.set()
|
||||
assert gated.flipped.wait(10.0), "the progress hook never fired"
|
||||
s3 = sync_client.get("/api/sync/status").json()
|
||||
assert s3["phase"] == "import", (
|
||||
"the first progress call must flip scan -> import: "
|
||||
f"{ {k: s3[k] for k in ('phase', 'current_file', 'files_done', 'files_total')} }"
|
||||
)
|
||||
assert s3["current_file"] == f"{source.name}/b.md"
|
||||
# The flip already carries the PREDICTED denominator — the number
|
||||
# the label turns into "N remaining".
|
||||
assert (s3["files_done"], s3["files_total"]) == (1, 1)
|
||||
# … and the scan pair stays where the scan left it.
|
||||
assert (s3["scan_done"], s3["scan_total"]) == (3, 3)
|
||||
finally:
|
||||
gated.go_on.set()
|
||||
gated.go_import.set()
|
||||
gated.release.set()
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
assert body["phase"] is None and body["current_file"] is None
|
||||
assert (body["scan_done"], body["scan_total"]) == (3, 3)
|
||||
assert (body["files_done"], body["files_total"]) == (1, 1)
|
||||
|
||||
|
||||
# --- the additive payload ----------------------------------------------------
|
||||
|
||||
|
||||
def test_scan_keys_are_additive_and_stable_from_idle_to_terminal(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The additive-only contract on the wire (the phase-187 discipline).
|
||||
|
||||
The payload's key set is EXACTLY the pre-phase-194 set plus
|
||||
``scan_done`` / ``scan_total``; it does not change between idle,
|
||||
mid-run and the terminal (no key appears or disappears with the
|
||||
phase); and both new keys are plain ``int`` — 0 when there is nothing
|
||||
to report, never ``null`` and never a plausible position."""
|
||||
expected_pre = {
|
||||
"state",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"detail",
|
||||
"error",
|
||||
"current_file",
|
||||
"files_done",
|
||||
"files_total",
|
||||
"current_chunk",
|
||||
"chunks_total",
|
||||
"images_done",
|
||||
"images_total",
|
||||
"phase",
|
||||
"current_summary",
|
||||
"summaries_done",
|
||||
"summaries_total",
|
||||
"rss_mb",
|
||||
"rss_peak_mb",
|
||||
}
|
||||
source = _local_source(tmp_path, f"{PREFIX}keys", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_stub_probe(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
idle = sync_client.get("/api/sync/status").json()
|
||||
assert set(idle) == expected_pre | {"scan_done", "scan_total"}, set(idle)
|
||||
assert idle["scan_done"] == 0 and idle["scan_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
mid = sync_client.get("/api/sync/status").json()
|
||||
assert set(mid) == expected_pre | {"scan_done", "scan_total"}
|
||||
assert mid["state"] == "running"
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
assert set(body) == expected_pre | {"scan_done", "scan_total"}
|
||||
for key in ("scan_done", "scan_total"):
|
||||
assert isinstance(body[key], int) and not isinstance(body[key], bool), body
|
||||
# Every pre-existing key keeps its type too (the additive pin's other
|
||||
# half — the file pair is still ints, the phase still a string|null).
|
||||
assert isinstance(body["files_done"], int) and isinstance(body["files_total"], int)
|
||||
assert body["phase"] is None
|
||||
|
||||
|
||||
# --- the sampling discipline ------------------------------------------------
|
||||
|
||||
|
||||
def test_scan_and_import_transitions_force_the_sample_and_the_scan_hook_does_not(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 187's rule, extended to the new stage (LOCKED A4).
|
||||
|
||||
Every entry below is one REAL ``/proc`` read (the two helpers are only
|
||||
called when the throttle lets ``_sample_rss`` through), tagged with
|
||||
the phase it happened in. Collapsing consecutive same-phase reads, the
|
||||
sequence is one read per transition: run start (``phase`` still null)
|
||||
→ ``"scan"`` → ``"import"`` (the flip) → ``"overview"`` →
|
||||
``"summaries"`` → the terminal (null again). The scan transition's
|
||||
read carries ``scan_done == 0`` — it is the transition, not a file —
|
||||
and there is EXACTLY one read inside the whole scan phase, so a
|
||||
per-file scan read (one ``/proc`` read per file, the regression this
|
||||
pins) fails here even though the fixture scanned three files."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}sampling", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_slow_hash(monkeypatch)
|
||||
reads = _install_counting_memory(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
assert reads, "the run sampled no memory at all"
|
||||
phases = [r["phase"] for r in reads]
|
||||
transitions = [p for i, p in enumerate(phases) if i == 0 or p != phases[i - 1]]
|
||||
assert transitions == [None, "scan", "import", "overview", "summaries", None], reads
|
||||
|
||||
scan_reads = [r for r in reads if r["phase"] == "scan"]
|
||||
assert len(scan_reads) == 1, (
|
||||
f"the scan phase cost {len(scan_reads)} /proc reads for 3 files — the "
|
||||
"per-file scan hook must sample UNFORCED (one read per interval)"
|
||||
)
|
||||
assert scan_reads[0]["scan_done"] == 0, (
|
||||
"the scan-phase read is not the transition — the forced sample must "
|
||||
"land when the phase is SET, before the first file is scanned"
|
||||
)
|
||||
flip_reads = [r for r in reads if r["phase"] == "import"]
|
||||
assert flip_reads and flip_reads[0]["current_file"] is not None, (
|
||||
"no read at the scan->import flip — the transition is a forced "
|
||||
f"sampling point: {reads}"
|
||||
)
|
||||
# The payload reports what the sampler wrote, not a stale pair.
|
||||
assert (body["rss_mb"], body["rss_peak_mb"]) == (100, 120), body
|
||||
|
||||
|
||||
# --- the degenerate empty knowledge base ------------------------------------
|
||||
|
||||
|
||||
def test_empty_kb_stays_in_scan_and_never_reaches_import(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""The D1 degenerate clause, pinned on BOTH sides.
|
||||
|
||||
Importer side (real): an empty source dir makes the pre-scan report
|
||||
NOTHING — no ``on_scan_progress`` call, no ``progress`` call, a zero
|
||||
denominator. App side (the double that fires nothing, i.e. exactly
|
||||
what that run reports): the status sits in ``"scan"`` — the flip
|
||||
lives in the per-file hook and no hook ever fires — with ``scan_*``
|
||||
0/0 and ``current_file`` null, and the terminal clears ``phase``
|
||||
while keeping the 0/0 pair (the UI's ``total === 0`` guard then
|
||||
renders the bare label with no count part)."""
|
||||
# Importer side: the real scan of an empty root reports nothing.
|
||||
empty = _local_source(tmp_path, f"{PREFIX}empty", {})
|
||||
llm = FakeEmbedder()
|
||||
scan_calls: list[tuple[int, int]] = []
|
||||
progress_calls: list[tuple[str, str, int, int]] = []
|
||||
summary = asyncio.run(
|
||||
importer.import_sources(
|
||||
[empty],
|
||||
llm,
|
||||
progress=lambda s, r, d, t: progress_calls.append((s, r, d, t)),
|
||||
on_scan_progress=lambda p, t: scan_calls.append((p, t)),
|
||||
)
|
||||
)
|
||||
assert (scan_calls, progress_calls) == ([], [])
|
||||
assert summary.files == 0
|
||||
|
||||
# App side: a run that reports nothing is a run that stays "scan".
|
||||
_install_local_run(monkeypatch, db, tmp_path, empty)
|
||||
fake_import = _FakeScanImport()
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert fake_import.started.wait(10.0), "the run never reached the import"
|
||||
# Wiring: the runner passed BOTH live closures (so the empty
|
||||
# phase above is the scan reporting nothing, not a missing hook).
|
||||
assert fake_import.scan_hook is not None
|
||||
assert fake_import.progress_hook is not None
|
||||
body = sync_client.get("/api/sync/status").json()
|
||||
assert body["state"] == "running"
|
||||
assert body["phase"] == "scan"
|
||||
assert body["current_file"] is None
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
assert body["files_done"] == 0 and body["files_total"] == 0
|
||||
finally:
|
||||
fake_import.release.set()
|
||||
|
||||
terminal = _poll(sync_client, "success")
|
||||
assert terminal["phase"] is None
|
||||
assert terminal["scan_done"] == 0 and terminal["scan_total"] == 0
|
||||
assert terminal["files_done"] == 0 and terminal["files_total"] == 0
|
||||
|
||||
|
||||
# --- the cancel during the pre-scan -----------------------------------------
|
||||
|
||||
|
||||
def test_cancel_during_the_scan_lands_cancelled_with_the_scan_counts_kept(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The phase-134 contract at the pipeline's NEW await (the app-side
|
||||
twin of the importer-side cancel pin).
|
||||
|
||||
The scan is blocking on purpose, so the cancel provably lands inside
|
||||
it: ``POST /api/sync/cancel`` answers, the ``cancelled`` terminal
|
||||
arrives well before the abandoned scan's remaining work could finish,
|
||||
``phase`` clears, the position the scan had reached is KEPT beside the
|
||||
(still zero) file pair, ``error`` stays null and ``detail`` empty (a
|
||||
cancel is intent, not failure), and NOTHING was indexed — the walk
|
||||
never started. The lifecycle line keeps its exact pre-phase prefix
|
||||
(LOCKED A2's append-only rule)."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}cancel", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
_slow_hash(monkeypatch, block=0.4)
|
||||
|
||||
_login(sync_client)
|
||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
# Wait for proof the run is INSIDE the scan: the phase, and a
|
||||
# position that moved.
|
||||
deadline = time.monotonic() + 20.0
|
||||
live: dict = {}
|
||||
while time.monotonic() < deadline:
|
||||
live = sync_client.get("/api/sync/status").json()
|
||||
if live["phase"] == "scan" and live["scan_done"] >= 1:
|
||||
break
|
||||
assert live["state"] == "running", live
|
||||
time.sleep(0.01)
|
||||
assert live.get("phase") == "scan" and live["scan_done"] >= 1, (
|
||||
f"the cancel never landed inside the scan: {live}"
|
||||
)
|
||||
position_at_cancel = live["scan_done"]
|
||||
|
||||
t0 = time.monotonic()
|
||||
r = sync_client.post("/api/sync/cancel")
|
||||
assert r.status_code == 202, f"cancel not accepted: {r.status_code} {r.text}"
|
||||
assert time.monotonic() - t0 < 2.0, "the cancel request itself stalled"
|
||||
|
||||
body = _poll(sync_client, "cancelled", timeout=3.0)
|
||||
|
||||
assert body["error"] is None
|
||||
assert body["detail"] == {}
|
||||
assert body["phase"] is None
|
||||
assert body["current_file"] is None
|
||||
# The scan's final position is kept (the keep-final-values
|
||||
# convention), and it is the position the cancel found — the run did
|
||||
# not finish the scan it was cancelled in.
|
||||
assert body["scan_total"] == 3
|
||||
assert body["scan_done"] >= position_at_cancel
|
||||
assert body["files_done"] == 0 and body["files_total"] == 0
|
||||
|
||||
# Nothing was indexed: the walk never started.
|
||||
assert db.scalar(
|
||||
select(select(Document).where(Document.source == source.name).exists())
|
||||
) is False
|
||||
|
||||
# The lifecycle line keeps its pre-phase-194 prefix exactly.
|
||||
cancelled = _log_lines(caplog, "sync: cancelled by user state=cancelled files_done=")
|
||||
assert len(cancelled) == 1, cancelled
|
||||
|
||||
|
||||
# --- the kill switch's status shape ----------------------------------------
|
||||
|
||||
|
||||
def test_kill_switch_off_reports_no_scan_position_but_the_same_machine(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""``BOR_SYNC_PENDING_COUNT=0`` on the STATUS side.
|
||||
|
||||
The importer takes the phase-193 stats pre-walk, which reports NOTHING,
|
||||
so the scan pair stays 0/0 for the whole run (the page renders no
|
||||
count from it) while ``files_done`` / ``files_total`` carry the WALK
|
||||
position again — 3 walked, not 1 predicted. The phase machine itself is
|
||||
UNCHANGED by the switch (the pre-``import_sources`` set is still
|
||||
``"scan"`` and the first file still flips it: that flip is pinned in
|
||||
the choreographed test above, and the switch's importer-side halves are
|
||||
pinned in ``tests/unit/test_sync_pending_count.py`` and
|
||||
``tests/integration/test_sync_pending_count.py``). The pins below read
|
||||
the TERMINAL, which is where both pairs survive deterministically."""
|
||||
source = _local_source(tmp_path, f"{PREFIX}off", _THREE_FILES)
|
||||
_install_local_run(monkeypatch, db, tmp_path, source)
|
||||
# The switch lives on the settings object the import reads (llm).
|
||||
real_embedder = FakeEmbedder()
|
||||
real_embedder.settings = Settings(_env_file=None, sync_pending_count=0) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(sync_api, "LLMClient", lambda: real_embedder)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
assert body["detail"]["added"] == 3
|
||||
# No scan, so no position — kept at 0/0, the value the UI renders as
|
||||
# "no count part" rather than a stale or invented one.
|
||||
assert body["scan_done"] == 0 and body["scan_total"] == 0
|
||||
# The WALK count is back (phase-193 semantics, byte-for-byte).
|
||||
assert (body["files_done"], body["files_total"]) == (3, 3)
|
||||
assert body["phase"] is None
|
||||
Reference in New Issue
Block a user