"""Phase 194 task 05 — the phase's own E2E suite: the count counts the WORK. Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov **The owner report (2026-10-19, TODO L6).** A sync on a roughly 10,000-file knowledge base with a handful of changes counted down ``10000 → 9999 → … → 1216 remaining``. There was almost nothing left to DO at 1216; the sentence still read like there was, because the number was a WALK POSITION wearing the shape of a work queue (``app/rag/importer.py``'s per-file counter → ``app/api/sync.py``'s ``files_done`` / ``files_total`` → ``frontend/assets/sources.js``'s ``fmtSyncLabel`` ``N remaining`` part). Phase 194 moved the count onto the WORK and gave the walk its own phase: * ``app/rag/importer.py`` predicts the run's change set BEFORE the walk (``_prescan_pending``: the streamed 1 MiB-chunk ``_file_hash`` against ``_pending_hash_stmt``, a SCALAR ``SELECT documents.file_hash`` on a session of its own, off the event loop in ONE ``asyncio.to_thread`` span), and the ``progress`` hook reports the position in THAT set; * ``app/api/sync.py`` reports ``phase: "scan"`` for the pre-scan with ``scan_done`` / ``scan_total`` beside it, and flips to ``"import"`` on the FIRST ``progress`` call inside the one existing ``_hook`` closure; * ``frontend/assets/sources.js``'s ``fmtSyncPhaseLabel`` renders the scan as ``Scanning… (n/total)`` in the SAME count slot the import's ``N remaining`` uses — one writer, one branch, no new element. Fixture (the house convention, the ``test_sync_remaining_count.py`` shape) ------------------------------------------------------------------------- A real git repo over the ``file://`` transport (``BOR_GIT_SOURCES``, so the env fallback IS the effective source under the autouse truncate) with ``N_FILES = 8`` markdown documents in walk order (``sorted(rel)``, the importer's rule) — the FIRST and the LAST multi-chunk so the phase-129 ``· chunk n/m`` part coexists with the count part and then stands alone — and a mock upstream that delays EVERY call ``SLOW_DELAY_S`` (``tests/e2e/slow_llm.py``) so one single-chunk file's import span (3 calls ≈ 3.6 s) outlives the page's 2 s status poll. A SECOND module app, same repo shape, boots with ``BOR_SYNC_PENDING_COUNT=0`` (the phase-184 two-module-app convention) and is arm d's rollback machine. The KB state a re-sync needs is established with the importer ITSELF, in-process, with the ``FakeEmbedder`` (the same code path, the same stamped ``file_hash``, the same git commit dates) — a real slow-proxy first import of a 1,000-file KB would turn the module into ten minutes for no extra evidence: every run UNDER test is a real uvicorn sync driven from the real Sources page over HTTP. Determinism — what is pinned live and what is pinned on the exact payload ------------------------------------------------------------------------ The scan reads no file and embeds nothing, so it is FAST: eight tiny files pre-scan in single-digit milliseconds and the 2 s poll cannot be made to land inside that window. This suite does not pretend otherwise and does not pad the fixture to fake it: * the **live** scan pins are the ones a real run answers DETERMINISTICALLY — every tick caught in ``phase == "scan"`` names NO file, carries the full denominator, and no scan tick ever follows a tick that already counted files; the terminal KEEPS the completed pair; and the **``test_prod_shaped_run``** arm (a seeded 1,000-file KB, 2 files changed — the owner's report at scale) makes the scan window long enough that its position is caught climbing across several distinct ``scan_done`` values THROUGH the running server while the loop keeps answering (the phase-179 contract — the scan is off the loop precisely so this poll is served); * the ``Scanning… (n/total)`` **rendered** frames — and arm b's unchanged-file HOLD — are pinned on the exact payloads the client's own guards read, served through ``page.route`` to the real ``sources.js`` + the real 2 s poll (``test_sync_remaining_count.py`` test 2 and the phase-129/152 suites' pattern). The pair itself is pinned against the real importer in ``tests/integration/test_sync_pending_count.py`` (its hook sequence IS ``[(changed,1,2), (unchanged,1,2), (unchanged,1,2), (changed,2,2)]``) and through the live status machine in ``tests/integration/test_sync_scan_phase.py``; here it is pinned against the guards that turn it into a sentence. Arms (the task file's a–e + the two standing contracts), in test order ---------------------------------------------------------------------- 1. ``test_first_sync_counts_the_predicted_set_and_the_machine_completes`` — **(a) the first sync, fresh KB:** pending == walk, so the label counts the phase-193 arithmetic exactly — ``7 remaining`` down to ``1``, the part HIDDEN at the final file while the main part still names it, the chunk part coexisting — and the wire's null prelude reports ``0/0``, every tick is the additive key set, every import tick reports the whole 8-file denominator, the terminal keeps ``scan_done == scan_total == 8`` beside ``files_done == 8`` and settles to ``8 added``. 2. ``test_incremental_sync_counts_only_the_changed_files`` — **(b) the owner's production case, the phase's heart:** the KB already holds all 8 documents, TWO files changed. Every running tick answers the PREDICTED denominator (``files_total == 2``, never ``8``) and the pair HOLDS across the six unchanged files (the exact expected pair per walked file, so the pin does not depend on which frames the 2 s poll happened to sample); rendered, the part reads ``1 remaining`` across the first predicted file and is GONE across the final predicted file while the main part names it; no label ever shows a walk-derived number; the terminal is ``2 updated · 6 unchanged`` with ``2/2`` files and the completed ``8/8`` scan kept beside it. 3. ``test_scan_frames_and_the_hold_render_from_the_exact_payloads`` — the same two machines rendered by the REAL client on the frames the live run produces but the 2 s poll cannot be aimed at: two CONSECUTIVE unchanged-file frames at the same ``2 remaining`` then the first predicted file at ``1 remaining`` then the final predicted file with the part gone; ``Scanning… (3/8)`` → ``(8/8)`` with no file and never ``N remaining``; ``scan_total: 0`` → the BARE ``Scanning…``; the null prelude → the bare ``Syncing…``; every frame A4-complete (button ``title`` + the sole ``#sync-progress`` live region byte-equal to the text) and no ``Syncing…`` label ever wearing an ``(n/m)`` tail. 4. ``test_kill_switch_zero_restores_the_walk_position_count`` — **(d):** the second app, arm b's fixture. No ``scan`` tick exists at all, ``scan_done`` / ``scan_total`` are ``0/0`` for the whole run including the terminal, and the count part counts the WALK: ``files_total == 8`` and the rendered figure naming files that will never be re-indexed (arm b answers ``1 remaining`` for the same file at the same moment in the same repository). 5. ``test_no_change_sync_reports_no_denominator`` — **(c):** the re-sync of a repository nothing touched. The scan completes (``8/8`` kept), the predicted set is EMPTY so every tick answers ``files_total == 0`` and no label EVER carries a count part (the phase-64 ``total === 0`` guard driven by zero work rather than by a missing denominator), and the terminal is the byte-identical ``0 added · 8 unchanged``. 6. ``test_empty_source_stays_in_scan_with_no_count_part`` — **(e) the D1 degenerate:** a source with ZERO importable files. Nothing is walked, so no ``progress`` call exists to flip the phase: the machine never answers ``import``, both pairs stay ``0/0`` through the run, no label ever carries a count part, and the terminal clears the phase and settles to ``0 added · 0 unchanged``. 7. ``test_prod_shaped_run_scan_climbs_then_counts_two`` — the report AT SCALE: a seeded 1,000-file KB with 2 files changed, over the real page. The scan's own position climbs across ≥2 distinct ``scan_done`` values while ``current_file`` stays null, the import then answers the PREDICTED denominator (``files_total == 2`` against ``scan_total == 1000`` — the divergence the pre-phase shape could not express), and the terminal keeps BOTH pairs beside each other. 8. ``test_upload_run_carries_no_scan_fields_and_no_count_part`` — the phase-90 (A2) convention with the scan pair ADDED to the not-there list: a real background upload's live status answers the bare phase-64 key set (no ``phase``, no chunk/image/summary fields, no ``scan_*``, no memory pair) with ``null``/``0``/``0``, and the shared label renders that job bare — no file, no count part (the upload branch never receives a status object, so phase 194 gives it nothing). The phase-187 peak clause, across these same runs (AGENTS rule 10 — the clause is TERMINAL text; task 05's "visible while the scan label is showing" is read against that rule, so what is pinned is the clause's presence at the terminal of a scan-labelled run and its ABSENCE on every running label): test 1 pins the real run's settle — the counts line ends in ``· peak MB`` equal to the terminal payload's ``rss_peak_mb`` (one reading, two surfaces), as TEXT on the line that already exists, with the box holding its ONE child span and its ``role=status`` / ``aria-live=polite`` intact; test 3 pins the served corollary — a running frame that CARRIES ``rss_mb`` / ``rss_peak_mb`` renders NO clause, and the terminal of that same served sequence renders ``0 added · 2 updated · 6 unchanged · peak 1234 MB`` (a served payload answers the clause independently of this host, so the shape is pinned even where ``MEMORY_KNOWABLE`` is False). Test → task mapping (the Playwright Mapping Rule): 1 → arm a + the real-run peak clause, 2 → arm b, 3 → arms a/b's rendered frames + the A4 placement + the peak-clause absence-while-running, 4 → arm d, 5 → arm c, 6 → arm e, 7 → arm a's scan-climb at prod shape (arm b's denominator at scale), 8 → the upload pin. """ from __future__ import annotations import asyncio import io import json import os import re import subprocess import sys import tarfile import threading import time from collections.abc import Iterator from datetime import UTC, datetime from pathlib import Path from typing import Any import httpx import pytest from playwright.sync_api import Page, expect from sqlalchemy import text from app.config import Settings from app.core.memory import rss_mb, rss_peak_mb from app.db import SessionLocal from app.rag.chunker import chunk_document from app.rag.importer import import_sources from e2e.auth_helpers import login from e2e.conftest import ( ADMIN_PASSWORD, MOCK_PORT, SESSION_SECRET, USE_REAL_LLM, _wait_http, ) from e2e.sync_helpers import counts_line from scripts.git_sync import file_commit_dates from tests.fakes import FakeEmbedder REPO = Path(__file__).resolve().parents[2] #: This module's own ports — the sync suites' ladder continues past the #: phase-193 suite's 8211/8914 (the one-suite-at-a-time rule makes a #: collision harmless, but these are free and env-overridable). APP_PORT = int(os.environ.get("E2E_APP_PORT_PENDING", "8212")) APP_URL = f"http://127.0.0.1:{APP_PORT}" ZERO_PORT = int(os.environ.get("E2E_APP_PORT_PENDING_ZERO", "8213")) ZERO_URL = f"http://127.0.0.1:{ZERO_PORT}" SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_PENDING", "8915")) SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}" SOURCES_URL = "/sources.html" #: Per-LLM-request delay on the proxy. A one-chunk file costs THREE calls #: (content embed batch + the summary chat + the summary-chunk embed) ≈ 3.6 s #: against the page's 2 s poll, so every file gets a rendered label of its #: own — and an UNCHANGED file whose summary is NULL costs TWO (the phase-118 #: backfill), which is what makes an unchanged file's HELD frame observable #: at all (arm b). SLOW_DELAY_S = "1.2" #: The module's wall-time budget (two app boots + three real syncs of the #: 8-file repo + the 1,000-file prod-shaped run + the served-status arms). A #: breach is a speed canary, not flake tolerance (the phase-64 convention). WALL_BUDGET_S = 600.0 #: The sync's subject: a git repo whose DIRECTORY NAME is the source name #: (the ``repo_name`` basename rule). Deliberately NOT named after anything #: under test — a path containing "remaining" would make every "does this #: label carry the count part?" check match the path itself. SOURCE = "pending" N_FILES = 8 #: The walk order is ``sorted(rel)`` (the importer's rule), so each file's #: 1-based WALK position is its rank here — the map the arms assert against. #: The FIRST and the LAST file are multi-chunk (the phase-129 convention: the #: chunk part climbs on the first file and stands alone, without a count #: part, on the last). REL_ORDER = [ "notes/a-multi-first.md", "notes/b-second.md", "notes/c-third.md", "notes/d-fourth.md", "notes/e-fifth.md", "notes/f-sixth.md", "notes/g-seventh.md", "notes/z-multi-last.md", ] FIRST_REL = REL_ORDER[0] LAST_REL = REL_ORDER[-1] #: Arm b's change set: walk positions 5 and 7 of 8. The LAST predicted file #: is therefore neither the last walked file nor the first — the count part #: holds at ``2 remaining`` across the unchanged files before it, reads #: ``1 remaining`` across the first predicted one and disappears across the #: final predicted one while the walk still has a file to go (the exact shape #: the pre-phase count read "``N`` remaining" through). CHANGED_RELS = ("notes/e-fifth.md", "notes/g-seventh.md") #: Arm b's UNCHANGED files whose stored summary the seed clears — walk #: positions 1 and 2, both BEFORE the first predicted file. Walking one still #: costs the phase-118 summary backfill (a ``lite`` call + its embed ≈ 2.4 s #: at this delay), which is what makes an unchanged file's HELD frame land in #: the browser instead of living only in the integration suite. HELD_RELS = ("notes/a-multi-first.md", "notes/b-second.md", "notes/d-fourth.md") #: Arm d (the ``BOR_SYNC_PENDING_COUNT=0`` app) changes positions 5 and 8 of #: the same fixture: under the rollback semantics the pair is the WALK, so the #: file arm b renders as "``1`` remaining" appears here as "``3`` remaining" #: and the last walked file is the only one that hides the part. ZERO_CHANGED_RELS = ("notes/e-fifth.md", "notes/z-multi-last.md") #: The count-bearing values the label may ever show (A1): ``1 … N−1``. MAX_COUNT = N_FILES - 1 #: The prod-shaped arm's KB (the owner's report at scale): 1,000 unchanged #: documents walked, TWO of them changed. A ``kind="local"`` source row, so #: the walk reads the seeded directory itself (no clone, no commit dates — #: the importer's mtime fallback) and the run's ONLY model work is the two #: changed files. WIDE_SOURCE = "pendingwide" WIDE_FILES = 1_000 WIDE_CHANGED_RELS = ("notes/w0003.md", "notes/w0008.md") #: The archive the upload arm uploads (the phase-98 fixture convention). UPLOAD_NAME = "uploadkb" UPLOAD_FILES: dict[str, str] = { f"docs/{i:02d}.md": f"# Upload doc {i:02d}\n\nDeterministic upload body {i:02d}.\n" for i in range(3) } #: The chunker settings the apps under test are pinned to (the leak-guard #: convention): the expected ``chunks_total`` is computed with the SAME #: function and arguments the importer uses, so a local ``.env`` cannot move #: the number this suite asserts. CHUNK_TARGET = int(Settings.model_fields["chunk_target_chars"].default) CHUNK_OVERLAP = int(Settings.model_fields["chunk_overlap_chars"].default) #: The import-phase label grammar (phase 193) — the main part (the bare #: ``Syncing…`` of the null prelude, which names no file, or the prefix plus #: the ``source/path``), the OPTIONAL count part, the OPTIONAL phase-129 chunk #: part. The arms that pin a NAMED file re-anchor it on their own paths. IMPORT_LABEL_RE = re.compile( r"^Syncing…(?: \S+(?: [0-9]+ remaining)?)?(?: · chunk \d+/\d+)?$" ) #: The scan-phase label grammar phase 194 (task 03, D4) adds: the bare #: ``Scanning…`` (no denominator yet) or it plus the ``(n/total)`` count #: part. NEVER a file (``current_file`` is null through the phase) and never #: ``N remaining`` — the walk position is honest behind the word "Scanning", #: never behind "Syncing" (TODO L6). SCAN_LABEL_RE = re.compile(r"^Scanning…(?: \(\d+/\d+\))?$") #: What phase 194 leaves forbidden: no ``Syncing…`` label may wear an #: ``(n/m)`` tail again (the shape is legal only on ``Scanning…`` and #: ``Summarizing folders…``). OLD_FRACTION_RE = re.compile(r"\(\d+/\d+\)") SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$") #: The FULL ``GET /api/sync/status`` key set — the additive chain's right-hand #: side as of phase 194 (phase 64/98/129/152 fields + the phase-187 memory #: pair + the phase-194 scan pair). ONE constant, mirrored by the served stubs #: below and pinned on every live tick, so a future key has to be added here #: (and to the stub) instead of drifting away from the client's view. EXPECTED_STATUS_KEYS = frozenset( { "state", "started_at", "finished_at", "current_file", "files_done", "files_total", "scan_done", "scan_total", "phase", "current_summary", "summaries_done", "summaries_total", "current_chunk", "chunks_total", "images_done", "images_total", "detail", "error", "rss_mb", "rss_peak_mb", } ) #: The UPLOAD status key set (phase 90 A2): the bare sync subset, and the #: whole point of the arm — no ``phase``, no chunk / image / summary fields, #: no memory pair and (new in phase 194) NO ``scan_done`` / ``scan_total``. UPLOAD_STATUS_KEYS = frozenset( { "state", "started_at", "finished_at", "current_file", "files_done", "files_total", "detail", "error", } ) #: The phase-187 clause, and the shape it must NEVER take on a running label. PEAK_CLAUSE_RE = re.compile(r" · peak (\d+) MB$") #: Does THIS platform answer the memory helpers? The app under test is a #: subprocess of this process's own OS, so this answer is the app's answer #: (the phase-187 suite's reasoning). Where it is False the real-run clause #: pin steps aside and the ABSENCE contract is pinned instead — the served #: arm keeps the clause's SHAPE pinned either way, because a served payload #: answers independently of this host. MEMORY_KNOWABLE = rss_mb() is not None and rss_peak_mb() is not None SETTLE_TIMEOUT_MS = 240_000 def _now_iso() -> str: """A "just finished" stamp for the served terminals — the phase-154 same-day rule renders it as the plain ``Synced HH:MM``.""" return datetime.now(UTC).isoformat() # --------------------------------------------------------------------------- # The fixture KB # --------------------------------------------------------------------------- def _tiny_md(title: str) -> str: """A one-chunk document (its import span = 3 slowed LLM calls).""" return f"# {title}\n\nDeterministic one-chunk body for the {title}.\n" def _multi_md(tag: str, sections: int = 3) -> str: """A multi-chunk document: an H1 preamble (one chunk) + *sections* ``##`` sections of ~1500 chars each (one chunk apiece at the code-default target, one embed batch apiece at the 700-token request budget) — no randomness, so the chunk count is stable run to run (the phase-129 fixture idiom).""" parts = [f"# {tag} multi-chunk fixture document\n"] parts.append( f"Preamble paragraph of the {tag} pending-count fixture. This paragraph " "sits under the H1 and chunks on its own.\n" ) for k in range(1, sections + 1): parts.append(f"\n## Section {k}\n") base = ( f"Section {k} paragraph of the {tag} pending-count fixture carries " "deterministic filler for the chunker: alpha bravo charlie delta " "echo foxtrot golf hotel india juliet kilo lima mike november oscar " "papa quebec romeo sierra tango uniform victor whiskey xray yankee " "zulu number " ) parts.append((base * 4)[:1500] + "\n\n") return "".join(parts) def _fixture_files() -> dict[str, str]: files = {rel: _tiny_md(Path(rel).stem) for rel in REL_ORDER} files[FIRST_REL] = _multi_md("first") files[LAST_REL] = _multi_md("last") return files def _expected_content_chunks(rel: str) -> int: """The file's TRUE content-chunk count — the SAME function and arguments the importer's chunk replacement uses (phase 129: ``chunks_total`` is a chunk count, never a batch count).""" return len(chunk_document(_fixture_files()[rel], rel, max(400, CHUNK_TARGET), CHUNK_OVERLAP)) def _wide_files() -> dict[str, str]: """The prod-shaped fixture: ``WIDE_FILES`` one-chunk documents under ``notes/`` (the walk order is the zero-padded stem, so the changed pair lands at WALK positions 3 and 8).""" return { f"notes/w{i:04d}.md": _tiny_md(f"wide document {i:04d}") for i in range(1, WIDE_FILES + 1) } def _git(cwd: Path, *args: str) -> None: proc = subprocess.run(["git", *args], cwd=cwd, capture_output=True, text=True) if proc.returncode != 0: raise AssertionError(f"git {' '.join(args)} failed: {proc.stderr.strip()}") def _commit(cwd: Path, msg: str) -> None: _git(cwd, "add", "-A") _git( cwd, "-c", "user.email=e@x", "-c", "user.name=t", "-c", "commit.gpgsign=false", "commit", "-qm", msg, ) def _write_repo(root: Path, files: dict[str, str], msg: str) -> Path: root.mkdir(parents=True, exist_ok=True) for rel, content in files.items(): target = root / rel target.parent.mkdir(parents=True, exist_ok=True) target.write_text(content, encoding="utf-8") _git(root, "init", "-q") _commit(root, msg) assert (root / ".git").is_dir() return root @pytest.fixture(scope="module") def pending_repo(tmp_path_factory: pytest.TempPathFactory) -> Path: """The real repo the sync clones over ``file://`` — 8 markdown docs in the walk order ``REL_ORDER``.""" root = tmp_path_factory.mktemp("bor_pending") / SOURCE _write_repo(root, _fixture_files(), "pending-count fixture") assert len(list(root.rglob("*.md"))) == N_FILES return root @pytest.fixture(scope="module") def zero_repo(tmp_path_factory: pytest.TempPathFactory) -> Path: """The kill-switch app's OWN repo (same shape, different source name): arms b and d change files in their own repo, so neither arm's committed state depends on the other's order.""" root = tmp_path_factory.mktemp("bor_pending_zero") / "pendingzero" _write_repo(root, _fixture_files(), "pending-count rollback fixture") assert len(list(root.rglob("*.md"))) == N_FILES return root @pytest.fixture(scope="module") def wide_kb(tmp_path_factory: pytest.TempPathFactory) -> Path: """The prod-shaped KB: 1,000 tiny docs in a plain DIRECTORY, registered per test as a ``kind=local`` row. The walk reads the seeded directory itself (no clone, no commit dates — the importer's mtime fallback), which is the cheap, honest way to give the pre-scan real work: the alternative is a git repo whose 1,000-file clone costs more than the scan under test.""" root = tmp_path_factory.mktemp("bor_pending_wide") / WIDE_SOURCE root.mkdir(parents=True, exist_ok=True) for rel, content in _wide_files().items(): target = root / rel target.parent.mkdir(parents=True, exist_ok=True) target.write_text(content, encoding="utf-8") assert len(list(root.rglob("*.md"))) == WIDE_FILES return root @pytest.fixture(scope="module") def empty_kb_dir(tmp_path_factory: pytest.TempPathFactory) -> Path: """Arm e's source: a directory with ZERO importable files — one visible file whose extension is not in the A9 family (so "nothing importable" is a property of the CONTENT, not of an empty directory) and one hidden ``.md`` the walk skips. Registered per test as a ``kind=local`` row (the phase-49 contract: a stored row IS the effective source, which is what lets this arm exist on the module's own app instead of a third boot).""" root = tmp_path_factory.mktemp("bor_pending_empty") / "pendingempty" root.mkdir(parents=True, exist_ok=True) (root / "README.gitkeep").write_text("nothing here is importable\n", encoding="utf-8") (root / "notes").mkdir() # A REAL dotfile (the walk's hidden rule), not merely a file about # hiding — the arm's premise is a source with NOTHING importable. (root / "notes" / ".hidden.md").write_text( "# hidden\n\nSkipped: dotfile.\n", encoding="utf-8" ) return root @pytest.fixture(scope="module") def upload_archive(tmp_path_factory: pytest.TempPathFactory) -> Path: root = tmp_path_factory.mktemp("bor_pending_archive") path = root / f"{UPLOAD_NAME}.tar.gz" with tarfile.open(path, "w:gz") as tf: for rel, content in UPLOAD_FILES.items(): data = content.encode("utf-8") info = tarfile.TarInfo(rel) info.size = len(data) info.mtime = 0 tf.addfile(info, io.BytesIO(data)) return path # --------------------------------------------------------------------------- # The apps # --------------------------------------------------------------------------- def _app_env( base_url: str, source: str, checkout_dir: Path, upload_dir: Path, *, pending_count: str | None = None, ) -> dict[str, str]: """The per-module app env (the conftest pattern): the slow proxy is the LLM, ``BOR_GIT_SOURCES`` is a suite-owned ``file://`` repo (the env fallback IS the effective source — the autouse truncate keeps the ``git_sources`` table empty), and the chunker knobs are pinned to the code defaults so the expected ``chunks_total`` stays predictable. ``pending_count`` is the phase's ONE switch: ``None`` leaves ``BOR_SYNC_PENDING_COUNT`` at the default (``1`` — count the predicted change set), ``"0"`` is the exact phase-193 rollback arm d drives. """ env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url env["BOR_RELEVANCE_THRESHOLD"] = "0.30" env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15" env["BOR_LLM_RETRY_DELAY"] = "0" env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET env["BOR_GIT_SOURCES"] = f"file://{source}" env["BOR_SOURCES_DIR"] = str(checkout_dir) env["BOR_UPLOAD_DIR"] = str(upload_dir) env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET) env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP) env["BOR_DOCS_REPO"] = "" env["BOR_SUGGESTIONS"] = json.dumps(["Ask something neutral"]) if pending_count is not None: env["BOR_SYNC_PENDING_COUNT"] = pending_count return env def _spawn_uvicorn(app: str, port: int, env: dict[str, str], probe: str) -> Iterator[None]: """One uvicorn subprocess, terminated on teardown (the conftest body, factored so this module can own TWO apps without duplicating the lifecycle). The caller yields its OWN base URL — that is what the suite's ``app_url`` hands to the page.""" proc = subprocess.Popen( [sys.executable, "-m", "uvicorn", app, "--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"], cwd=REPO, env=env, ) try: _wait_http(probe) yield finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() def _spawn_slow_proxy(port: int, delay_s: str, probe: str) -> Iterator[int]: """The delay-injecting proxy (``tests/e2e/slow_llm.py``) in front of the mock LLM — the suite's timing fixture: it delays EVERY LLM call, embeds included, so each file's import span outlives the UI's 2 s poll.""" env = dict(os.environ) env.pop("DEBUGPY", None) env["SLOW_LLM_DELAY_S"] = delay_s env["E2E_MOCK_PORT"] = str(MOCK_PORT) for _ in _spawn_uvicorn( "tests.e2e.slow_llm:app", port, env, f"{probe}/v1/models" ): yield port @pytest.fixture(scope="module") def slow_llm(mock_llm: int) -> Iterator[int]: yield from _spawn_slow_proxy(SLOW_PORT, SLOW_DELAY_S, SLOW_URL) @pytest.fixture(scope="module") def app_server( mock_llm: int, slow_llm: int, pending_repo: Path, tmp_path_factory: pytest.TempPathFactory, ) -> Iterator[str]: """The app under test — the DEFAULT ``BOR_SYNC_PENDING_COUNT`` (1).""" env = _app_env( f"{SLOW_URL}/v1", str(pending_repo), tmp_path_factory.mktemp("bor_pending_checkouts"), tmp_path_factory.mktemp("bor_pending_uploads") / "uploads", ) for _ in _spawn_uvicorn("app.main:app", APP_PORT, env, f"{APP_URL}/api/health"): yield APP_URL @pytest.fixture(scope="module") def zero_server( mock_llm: int, slow_llm: int, zero_repo: Path, tmp_path_factory: pytest.TempPathFactory, ) -> Iterator[str]: """The SECOND module app — the same fixture shape booted with ``BOR_SYNC_PENDING_COUNT=0`` (the phase-184 two-module-app convention: one setting, two live servers, the owner-visible rollback proven instead of asserted about).""" env = _app_env( f"{SLOW_URL}/v1", str(zero_repo), tmp_path_factory.mktemp("bor_pending_zero_checkouts"), tmp_path_factory.mktemp("bor_pending_zero_uploads") / "uploads", pending_count="0", ) for _ in _spawn_uvicorn("app.main:app", ZERO_PORT, env, f"{ZERO_URL}/api/health"): yield ZERO_URL @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server def _truncate_all() -> None: """Fresh registry + KB per test (the E2E isolation pattern): the suites share one Postgres, and a leftover document would move both the ``added`` counts the terminal line pins AND the pre-scan's prediction.""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, steering_notes, " "kb_overview, git_sources, folder_summaries" ) ) db.commit() def _wait_sync_settled(base_url: str, timeout_s: float = 300.0) -> None: """No run may leak across a test boundary (the run state lives in the app's memory and a running run keeps importing into the NEXT test's truncated KB). Own admin session; usually a no-op.""" with httpx.Client(base_url=base_url, timeout=5.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: r = client.get("/api/sync/status") assert r.status_code == 200, r.text body = r.json() if body["state"] == "failed": print(f"sync pending-count: FAILED terminal at a boundary: {body.get('error')}") if body["state"] != "running": return time.sleep(0.2) raise AssertionError(f"a sync was still running at a test boundary: {base_url}") def _settle(base_url: str) -> None: """Settle an app that MAY not be up (the zero app boots lazily with its fixture, and the autouse cleaner must not force it into every test).""" try: httpx.get(f"{base_url}/api/health", timeout=2.0).raise_for_status() except Exception: # noqa: BLE001 — not booted (yet): nothing to settle return _wait_sync_settled(base_url) @pytest.fixture(autouse=True) def _clean(app_url: str, db_ready: None) -> Iterator[None]: _settle(app_url) _settle(ZERO_URL) _truncate_all() yield _truncate_all() @pytest.fixture(scope="module", autouse=True) def _wall_time_guard() -> Iterator[None]: start = time.monotonic() yield elapsed = time.monotonic() - start assert elapsed < WALL_BUDGET_S, ( f"the module took {elapsed:.1f}s — over the {WALL_BUDGET_S:.0f}s wall budget" ) # --------------------------------------------------------------------------- # The KB state a re-sync starts from: the importer itself, in-process # --------------------------------------------------------------------------- def _seed_kb( root: Path, *, null_summary_rels: tuple[str, ...] = (), git: bool = True, ) -> None: """Index *root* with the importer ITSELF (in-process, ``FakeEmbedder``). The same code path a real sync runs, so the rows carry the real ``file_hash`` stamps the phase-147 fast path compares against — which is what makes the NEXT (real, in-app, slow-proxy) sync predict exactly the files this test changed and nothing else. The git arms pass the checkout's own commit dates, so the run under test finds every date already correct and ``unchanged`` means unchanged. ``null_summary_rels`` then clears those rows' ``summary`` — the phase-118 (A2) backfill case: the file stays UNCHANGED (its digest matches, so the pre-scan predicts nothing for it), but walking it still costs the ``lite`` summary call plus its embed. That is the ONE honest way to make an unchanged file occupy real, model-bound, loop-free time, and therefore the way arm b's held frame becomes observable in the browser instead of theoretical. ``git=False`` skips the commit-date map — the seed of a ``kind=local`` source, whose run uses the importer's mtime fallback on BOTH sides (matching, so the arm under test reports no date churn either). """ dates = {str(root): file_commit_dates(root)} if git else None outcome: dict[str, BaseException] = {} async def _go() -> None: # The Playwright sync API owns an event loop in this process, so the # seed gets its OWN loop on its OWN thread. The importer is the same # async code either way and the session is a plain sync one — this is # the ``tests/integration/test_sync_pending_count.py`` call, one # thread over. with SessionLocal() as inner: await import_sources( [root], FakeEmbedder(), session=inner, doc_dates_by_root=dates, ) def _worker() -> None: try: asyncio.run(_go()) except BaseException as exc: # noqa: BLE001 — re-raised on the caller outcome["error"] = exc seed_thread = threading.Thread(target=_worker) seed_thread.start() seed_thread.join() if "error" in outcome: raise AssertionError( f"the KB seed import over {root} failed: {outcome['error']!r}" ) from outcome["error"] if null_summary_rels: with SessionLocal() as db: db.execute( text( "UPDATE documents SET summary = NULL " "WHERE source = :source AND path = ANY(:rels)" ), {"source": root.name, "rels": list(null_summary_rels)}, ) db.execute( text( "DELETE FROM chunks WHERE is_summary AND document_id IN " "(SELECT id FROM documents WHERE source = :source " "AND path = ANY(:rels))" ), {"source": root.name, "rels": list(null_summary_rels)}, ) db.commit() def _change_files(root: Path, rels: tuple[str, ...], marker: str) -> None: """Touch *rels* on disk (and commit them, so the ``file://`` clone/pull the app performs carries the change).""" for rel in rels: path = root / rel path.write_text( path.read_text(encoding="utf-8") + f"\n{marker} edit of {rel}\n", encoding="utf-8", ) _commit(root, f"{marker}: {', '.join(rels)}") # --------------------------------------------------------------------------- # The in-browser recorder: every value the ONE label writer lands # --------------------------------------------------------------------------- #: Read in ONE JS task, so a frame can never straddle two writes: the box #: text (the phase-150 sole sync text surface + sole ``aria-live`` region), #: the button ``title`` and ``aria-label`` (the phase-64 A4 untruncated #: placement), and the ``.sync-label-count`` span's own visibility. _INSTALL_RECORDER = """ () => { if (window.__borFrames) return window.__borFrames.length; window.__borFrames = []; const box = document.querySelector('#sync-progress'); const textEl = document.querySelector('#sync-progress-text'); const btn = document.querySelector('#sync-btn'); const countEl = document.querySelector('.sync-label-count'); const rec = () => { const frame = { text: textEl.textContent, title: btn.getAttribute('title') || '', aria: btn.getAttribute('aria-label') || '', count: countEl ? countEl.textContent : '', countHidden: countEl ? countEl.hidden : null, hidden: box.hidden, }; const last = window.__borFrames[window.__borFrames.length - 1]; if (last && last.text === frame.text && last.title === frame.title && last.aria === frame.aria && last.countHidden === frame.countHidden) { return; } window.__borFrames.push(frame); }; const mo = new MutationObserver(rec); mo.observe(textEl, { childList: true, characterData: true, subtree: true }); mo.observe(btn, { attributes: true, attributeFilter: ['title', 'aria-label'] }); if (countEl) mo.observe(countEl, { attributes: true, attributeFilter: ['hidden'] }); rec(); return window.__borFrames.length; } """ def _install_recorder(page: Page) -> None: page.evaluate(_INSTALL_RECORDER) def _frames(page: Page) -> list[dict[str, Any]]: frames: list[dict[str, Any]] = page.evaluate("() => window.__borFrames") assert frames, "the in-browser recorder captured no label frames" return frames def _texts(frames: list[dict[str, Any]]) -> list[str]: """The recorded label sequence, empty frames dropped (the box's idle-hide is not a label).""" return [f["text"] for f in frames if f["text"]] def _count_of(label: str) -> int | None: """The rendered ``N remaining`` value, or ``None`` when the label carries no count part (A1's hidden state).""" m = re.search(r" (\d+) remaining(?: · |$)", label) return int(m.group(1)) if m else None def _file_of(label: str) -> str | None: """The document an import label names, RELATIVE to its source (the fixture's own ``rel`` — the source-name segment is dropped, since the label's ``source/path`` identity is what the walk orders by).""" m = re.match(r"^Syncing… (\S+?)(?: \d+ remaining| · chunk |\s*$)", label) if not m: return None full = m.group(1) return full.split("/", 1)[1] if "/" in full else full #: The running-label prefixes (the ONLY states ``setSyncLabel`` gives a full #: untruncated title/announcer with — the terminals are the counts line + the #: ``Synced HH:MM`` button label, whose title is the idle one by contract). _RUNNING_PREFIXES = ( "Syncing…", "Scanning…", "Importing", "Writing KB overview…", "Summarizing folders…", ) def _assert_a4(frames: list[dict[str, Any]]) -> None: """A4 (phase 64, unchanged by 194): every RUNNING label — the scan's ``Scanning… (n/total)`` included — rides the button ``title``, the button ``aria-label`` and the announcer byte-equal to the text, and the count span is un-hidden exactly when the text carries a count part.""" for frame in frames: if not frame["text"].startswith(_RUNNING_PREFIXES): continue assert frame["title"] == frame["text"], frame assert frame["aria"] == frame["text"], frame assert frame["countHidden"] is not ( " remaining" in frame["text"] or "(" in frame["text"] ), frame def _pending_positions( walk_order: tuple[str, ...], changed: tuple[str, ...] ) -> dict[str, tuple[int, int]]: """The pair the status must carry while each walked file is in flight — the arithmetic the pre-scan produces, computed HERE from the fixture (``total`` = the size of the predicted set, ``done`` = the 1-based rank of the current file INSIDE it, held for every file outside it).""" pending = [rel for rel in walk_order if rel in set(changed)] out: dict[str, tuple[int, int]] = {} seen = 0 for rel in walk_order: if rel in pending: seen += 1 out[rel] = (seen, len(pending)) return out def _walk_positions(walk_order: tuple[str, ...]) -> dict[str, tuple[int, int]]: """The same map under ``BOR_SYNC_PENDING_COUNT=0``: every walked file advances the count, and the denominator is the whole walk (phase 193).""" return { rel: (i + 1, len(walk_order)) for i, rel in enumerate(walk_order) } class _TickRecorder: """The deterministic layer: a daemon thread tight-polling the SAME status endpoint the UI's 2 s poll reads, with its OWN admin session (the Playwright sync API is not thread-safe, so the thread never touches the page). It starts nothing — it only watches. A body belongs to THIS run once the recorder has seen it go ``running``, or — for a run faster than the cadence — once its own ``started_at`` post-dates the recorder's start. ``interval`` is the cadence: the default 100 ms resolves the phase machine on an 8-file run, and the prod-shaped arm tightens it to resolve the pre-scan's position (the scan runs OFF the loop, so this poll is answered THROUGH it — the phase-179 contract). """ def __init__( self, base_url: str, path: str, *, wait_for_running: bool = True, interval_s: float = 0.1 ) -> None: self._url = f"{base_url}{path}" self._login_url = f"{base_url}/api/login" self._wait_for_running = wait_for_running self._interval_s = interval_s self._saw_running = not wait_for_running self._started_after = 0.0 self._ticks: list[dict[str, Any]] = [] self._terminal: dict[str, Any] | None = None self._stop = threading.Event() self._thread: threading.Thread | None = None @staticmethod def _epoch(body: dict[str, Any]) -> float | None: raw = body.get("started_at") if not raw: return None try: return datetime.fromisoformat(str(raw)).timestamp() except ValueError: # pragma: no cover — defensive, an odd stamp return None def _is_ours(self, body: dict[str, Any]) -> bool: if body["state"] == "running": self._saw_running = True if self._saw_running: return True started = self._epoch(body) return started is not None and started >= self._started_after - 1.0 def start(self) -> None: if self._wait_for_running: self._started_after = time.time() def run() -> None: with httpx.Client(timeout=10.0) as client: client.post(self._login_url, json={"password": ADMIN_PASSWORD}) while not self._stop.is_set(): try: r = client.get(self._url) if r.status_code == 200: body = r.json() if not self._is_ours(body): self._stop.wait(self._interval_s) continue self._ticks.append(body) if body["state"] in ("success", "failed", "cancelled"): self._terminal = body return except Exception: # noqa: BLE001 — a blocked loop: retry pass self._stop.wait(self._interval_s) self._thread = threading.Thread(target=run, daemon=True) self._thread.start() def stop(self, timeout_s: float = 300.0) -> dict[str, Any]: deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: if self._terminal is not None: break time.sleep(0.05) self._stop.set() if self._thread is not None: self._thread.join(timeout=5) assert self._terminal is not None, ( f"the recorder saw no terminal state (last: {self._ticks[-3:]})" ) return self._terminal @property def ticks(self) -> list[dict[str, Any]]: return list(self._ticks) @property def running_ticks(self) -> list[dict[str, Any]]: return [t for t in self._ticks if t["state"] == "running"] def _serve_statuses(page: Page, stages: dict[str, Any]) -> None: """Route the two job-status endpoints to the CURRENT stage body (the house ``page.route`` pattern): ``stages`` is mutated by the test between the UI waits, so each 2 s poll of the REAL page renders exactly the payload the test put in front of it — the client's own decision points, pinned without a timing race.""" def handle(route: Any) -> None: path = route.request.url if "/api/sync/status" in path: body = stages.get("sync") elif "/api/git-sources/upload/status" in path: body = stages.get("upload") else: # pragma: no cover — only the two patterns are routed route.fallback() return if body is None: route.fallback() return route.fulfill(status=200, content_type="application/json", body=json.dumps(body)) page.route("**/api/sync/status", handle) page.route("**/api/git-sources/upload/status", handle) def _sync_status(**overrides: Any) -> dict[str, Any]: """One ``GET /api/sync/status`` body with the endpoint's CURRENT key set (``EXPECTED_STATUS_KEYS``) — the served stubs stay faithful to the payload shape; only the values are the test's.""" body: dict[str, Any] = { "state": "idle", "started_at": None, "finished_at": None, "current_file": None, "files_done": 0, "files_total": 0, "scan_done": 0, "scan_total": 0, "phase": None, "current_summary": None, "summaries_done": 0, "summaries_total": 0, "current_chunk": 0, "chunks_total": 0, "images_done": 0, "images_total": 0, "detail": {}, "error": None, "rss_mb": None, "rss_peak_mb": None, } assert set(body) == EXPECTED_STATUS_KEYS, set(body) ^ EXPECTED_STATUS_KEYS body.update(overrides) return body def _upload_status(**overrides: Any) -> dict[str, Any]: body: dict[str, Any] = { "state": "idle", "started_at": None, "finished_at": None, "current_file": None, "files_done": 0, "files_total": 0, "detail": {}, "error": None, } assert set(body) == UPLOAD_STATUS_KEYS, set(body) ^ UPLOAD_STATUS_KEYS body.update(overrides) return body def _wait_result_line(page: Page, expected: str) -> str: """Wait for the PAGE (not the recorder) to settle on its terminal counts line, and hand it back for the byte-shape assertions. The tight-polling recorder sees the server's terminal one to two ticks BEFORE the 2 s UI poll does — reading the box right after ``recorder.stop()`` would race the page's own settle (the phase-64 'never stale' contract is what this waits for, so the wait is the pin).""" loc = page.locator("#sync-progress-text") expect(loc).to_have_text(counts_line(expected), timeout=SETTLE_TIMEOUT_MS) return loc.inner_text() def _admin_sources_page(page: Page, base_url: str, *, settled: bool = True) -> None: """Real form login landing on the Sources page (the button's home). ``settled=True`` also waits for the boot re-attach to finish in a NON-running state; the served-status arms pass ``settled=False`` because their boot fetch IS the state under test.""" login(page, base_url) expect(page).to_have_url(base_url + SOURCES_URL, timeout=30_000) expect(page.locator("#sync-btn")).to_be_visible(timeout=30_000) if settled: expect(page.locator("#sync-label")).to_have_text( re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$"), timeout=30_000 ) def _assert_machine( ticks: list[dict[str, Any]], terminal: dict[str, Any], *, scan_total: int, files_total: int, scan_pair_survives_terminal: bool = True, ) -> None: """The phase-194 machine, read off the endpoint the UI polls — every claim deterministic on a real run (none depends on a tick landing inside the scan window): * the payload is ADDITIVE — exactly :data:`EXPECTED_STATUS_KEYS` on every body, running and terminal alike; * the null prelude reports the scan pair as ``0/0`` (the run has not reached ``import_sources`` yet); * a tick caught in ``phase == "scan"`` names NO file, carries the scan's own denominator and a position that only climbs, and reports no file-work denominator yet (``files_total == 0`` — the flip happens on the first ``progress`` call); * ONE-DIRECTIONAL: no scan tick follows a tick that already counted; * ``files_total`` never exceeds the scan's denominator (the count and its denominator stay on the SAME predicted set); * the terminal CLEARS ``phase`` and KEEPS both pairs side by side (the phase-64 keep-final-values convention) — on the ``0`` rollback the kept scan pair is ``0/0``, because that run scans nothing.""" for tick in ticks: assert set(tick) == EXPECTED_STATUS_KEYS, ( f"the status payload drifted from the additive key set: " f"{sorted(set(tick) ^ EXPECTED_STATUS_KEYS)}" ) assert set(terminal) == EXPECTED_STATUS_KEYS, terminal for tick in ticks: if tick["state"] == "running" and tick["phase"] is None: assert (tick["scan_done"], tick["scan_total"]) == (0, 0), ( f"the null prelude already carried scan counts: {tick}" ) scan_ticks = [t for t in ticks if t["phase"] == "scan"] for tick in scan_ticks: assert tick["current_file"] is None, ( f"the scan named a file — it counts, it does not read: {tick}" ) if tick["scan_total"] == 0: # The scan's OWN first frame (or the degenerate empty walk): the # phase is set the moment ``import_sources`` is entered and the # pair lands with the first position, so a tick here legitimately # carries 0/0 — exactly the payload the UI's ``scan_total === 0`` # guard renders as the bare "Scanning…". assert tick["scan_done"] == 0, tick continue assert tick["scan_total"] == scan_total, tick assert 0 < tick["scan_done"] <= scan_total, tick positions = [t["scan_done"] for t in scan_ticks] assert positions == sorted(positions), f"the scan position went backwards: {positions}" counted = [i for i, t in enumerate(ticks) if t["files_total"] > 0] if scan_ticks and counted: last_scan = max(i for i, t in enumerate(ticks) if t["phase"] == "scan") assert last_scan < counted[0], ( "a scan frame followed a frame that already counted files — the " f"machine ran backwards: ticks={[(t['phase'], t['files_total']) for t in ticks]}" ) assert terminal["phase"] is None, terminal if scan_pair_survives_terminal: assert terminal["scan_total"] == scan_total, terminal assert terminal["scan_done"] == scan_total, ( f"the terminal did not keep the completed scan: {terminal}" ) assert terminal["files_total"] == files_total, terminal if scan_total > 0: # The count and its denominator stay on the SAME predicted set — the # work can never exceed what the scan walked. Meaningless on the # ``BOR_SYNC_PENDING_COUNT=0`` rollback, which scans nothing and whose # denominator IS the walk. assert terminal["files_total"] <= terminal["scan_total"], terminal # --------------------------------------------------------------------------- # 1. (arm a) The first sync: pending == walk, and the machine completes # --------------------------------------------------------------------------- def test_first_sync_counts_the_predicted_set_and_the_machine_completes( page: Page, app_url: str, db_ready: None ) -> None: """A FRESH knowledge base: the predicted change set IS the walk, so the phase-193 arithmetic survives UNEDITED (the reason the phase-193 fixture never saw the defect) and the machine completes its full sequence.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) _admin_sources_page(page, app_url) _install_recorder(page) recorder = _TickRecorder(app_url, "/api/sync/status") recorder.start() btn = page.locator("#sync-btn") btn.click() expect(btn).to_be_disabled() expect(page.locator("#sync-progress")).to_have_text("Syncing…") # The FINAL file: the main part names it and the count part is GONE (A1). # ONE live wait per arm on the state that ENDS the import phase (every # window here is ≫ the 2 s poll, so it cannot be missed); the rest of the # sequence is read off the in-browser recorder, which captured every value # the ONE writer landed — a per-label live wait would race its own poll. expect(page.locator("#sync-progress-text")).to_have_text( re.compile( rf"^Syncing… {re.escape(SOURCE)}/{re.escape(LAST_REL)}" r"(?! \d+ remaining)(?: · chunk \d+/\d+)?$" ), timeout=SETTLE_TIMEOUT_MS, ) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS) terminal = recorder.stop() assert terminal["state"] == "success", terminal frames = _frames(page) texts = _texts(frames) walk_map = _walk_positions(tuple(REL_ORDER)) # --- the rendered import frames ------------------------------------ import_frames = [f for f in frames if f["text"].startswith("Syncing… ")] assert len(import_frames) >= 4, f"too few rendered import labels: {texts[:8]}" for frame in import_frames: assert IMPORT_LABEL_RE.match(frame["text"]), frame assert not OLD_FRACTION_RE.search(frame["text"]), frame rel = _file_of(frame["text"]) expected = walk_map.get(rel or "") assert expected is not None, f"the label named an unknown file: {frame}" done, total = expected # The cross-layer invariant, per FILE (not per sample): the rendered # N is exactly what this file's pair subtracts to. assert _count_of(frame["text"]) == (total - done or None), (frame, expected) # --- the count-down -------------------------------------------------- observed = [ n for n in (_count_of(t) for t in texts if t.startswith("Syncing… ")) if n is not None ] assert len(observed) >= 3, f"the label never visibly counted down: {observed}" assert observed == sorted(observed, reverse=True), f"not a count-DOWN: {observed}" assert observed[0] == MAX_COUNT, f"the walk did not start at N − 1: {observed}" assert observed[-1] == 1, f"the count part never reached the last file: {observed}" assert 0 not in observed, f"a count-bearing label showed 0 remaining: {observed}" # The FIRST file's frame: the pending-count part at N − 1 AND the # phase-129 chunk part, coexisting in ONE label. first_frames = [ t for t in texts if t.startswith(f"Syncing… {SOURCE}/{FIRST_REL} {MAX_COUNT} remaining") ] assert first_frames, f"the first file never rendered {MAX_COUNT} remaining: {texts[:8]}" # --- the chunk part coexists (phase 129 unbroken) -------------------- both = [t for t in texts if " remaining" in t and " · chunk " in t] assert both, f"the count part and the chunk part never coexisted: {texts[:8]}" for label in both: m = re.search(r" \d+ remaining · chunk (\d+)/(\d+)$", label) assert m is not None, label assert int(m.group(2)) == _expected_content_chunks(str(_file_of(label))), label # --- the scan branch, if the 2 s poll landed inside it --------------- scan_frames = [f for f in frames if f["text"].startswith("Scanning…")] import_at = [i for i, f in enumerate(frames) if f["text"].startswith("Syncing… ")] for frame in scan_frames: assert SCAN_LABEL_RE.fullmatch(frame["text"]), frame assert " remaining" not in frame["text"], frame assert "/" not in frame["text"], frame if scan_frames and import_at: assert max(i for i, f in enumerate(frames) if f["text"].startswith("Scanning…")) < ( import_at[0] ), f"a scan label followed an import label: {texts}" _assert_a4(frames) # --- the machine, off the endpoint the page polls -------------------- _assert_machine(recorder.ticks, terminal, scan_total=N_FILES, files_total=N_FILES) for tick in recorder.running_ticks: if tick["phase"] == "import" and tick["current_file"]: rel = str(tick["current_file"]).removeprefix(f"{SOURCE}/") assert (tick["files_done"], tick["files_total"]) == walk_map[rel], tick assert tick["files_total"] == N_FILES, tick # --- the terminal ----------------------------------------------------- # The counts line is the byte-identical phase-64 one (the pair below it # changed meaning; this did not) + the phase-187 clause, TERMINAL-only. assert terminal["detail"]["added"] == N_FILES, terminal assert terminal["detail"]["updated"] == 0, terminal box = page.locator("#sync-progress") line = _wait_result_line(page, f"{N_FILES} added") expect(box).to_be_visible() assert box.locator("> *").count() == 1 expect(box).to_have_attribute("role", "status") expect(box).to_have_attribute("aria-live", "polite") if MEMORY_KNOWABLE: assert terminal["rss_peak_mb"] is not None, terminal m = PEAK_CLAUSE_RE.search(line) assert m is not None, line assert int(m.group(1)) == terminal["rss_peak_mb"], (line, terminal) else: assert not PEAK_CLAUSE_RE.search(line), line # --------------------------------------------------------------------------- # 2. (arm b) The owner's production case: K of N changed # --------------------------------------------------------------------------- def test_incremental_sync_counts_only_the_changed_files( page: Page, app_url: str, db_ready: None, pending_repo: Path ) -> None: """THE phase's heart, in the browser. The KB already holds all 8 documents and TWO files changed: every tick of the running sync answers the PREDICTED denominator, so the label reads ``2 remaining`` — and holds there while ``current_file`` walks over the unchanged ones — instead of the walk-derived ``7``/``6``/``5`` the owner saw as "1216 remaining" on a ~10,000-file KB. The unchanged files in this run are not idle: their stored summaries were cleared by the seed, so each costs the phase-118 backfill (a ``lite`` call + its embed) — real model time, real frames, ZERO pending-work. That is what makes the HOLD visible here rather than only in ``tests/integration/test_sync_pending_count.py``'s hook sequence. """ page.set_default_timeout(SETTLE_TIMEOUT_MS) _seed_kb(pending_repo, null_summary_rels=HELD_RELS) _change_files(pending_repo, CHANGED_RELS, "arm b") expected = _pending_positions(tuple(REL_ORDER), CHANGED_RELS) _admin_sources_page(page, app_url) _install_recorder(page) recorder = _TickRecorder(app_url, "/api/sync/status") recorder.start() btn = page.locator("#sync-btn") btn.click() expect(page.locator("#sync-progress")).to_have_text("Syncing…") # The FINAL predicted file: the part is GONE while the walk still has # files left to go (the phase-193 A1 edge, now on the PREDICTED set). # One live wait; the held frames and the first predicted file are read off # the recorder below (a per-label live wait races the page's own 2 s poll). expect(page.locator("#sync-progress-text")).to_have_text( re.compile( rf"^Syncing… {re.escape(SOURCE)}/{re.escape(CHANGED_RELS[1])}" r"(?! \d+ remaining)(?: · chunk \d+/\d+)?$" ), timeout=SETTLE_TIMEOUT_MS, ) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS) terminal = recorder.stop() assert terminal["state"] == "success", terminal frames = _frames(page) texts = _texts(frames) # --- the wire: the denominator is the WORK, never the walk ----------- for tick in recorder.running_ticks: assert tick["files_total"] in (0, len(CHANGED_RELS)), ( f"a tick answered a walk-derived denominator: {tick}" ) if tick["current_file"]: rel = str(tick["current_file"]).removeprefix(f"{SOURCE}/") assert (tick["files_done"], tick["files_total"]) == expected[rel], tick # The HOLD, on the frames the server actually answered: at least TWO # distinct unchanged files, each named while the pair sat at (0, 2) — the # count did NOT advance on them. held_ticks = { str(t["current_file"]).removeprefix(f"{SOURCE}/") for t in recorder.running_ticks if t["current_file"] and (t["files_done"], t["files_total"]) == (0, len(CHANGED_RELS)) } assert len(held_ticks) >= 2, f"the count never held over unchanged files: {held_ticks}" assert held_ticks <= set(HELD_RELS), held_ticks # --- the rendered hold + the hidden final part ----------------------- held_labels = [ f["text"] for f in frames if f["text"].startswith("Syncing… ") and _count_of(f["text"]) == len(CHANGED_RELS) and _file_of(f["text"]) in HELD_RELS ] assert len(held_labels) >= 2, f"fewer than two held frames rendered: {texts[:12]}" assert len({_file_of(t) for t in held_labels}) >= 2, held_labels final_prefix = f"Syncing… {SOURCE}/{CHANGED_RELS[1]}" final_frames = [f for f in frames if f["text"].startswith(final_prefix)] assert final_frames, texts for frame in final_frames: assert _count_of(frame["text"]) is None, frame first_prefix = f"Syncing… {SOURCE}/{CHANGED_RELS[0]}" first_frames = [f for f in frames if f["text"].startswith(first_prefix)] assert first_frames and all(_count_of(f["text"]) == 1 for f in first_frames), first_frames # THE owner-visible regression pin: nothing in this run may show a # walk-derived count (7/6/5/4/3 remaining are the pre-phase numbers for # this exact fixture — see the kill-switch arm). for label in texts: n = _count_of(label) assert n is None or n <= len(CHANGED_RELS), ( f"the label counted the WALK, not the work: {label}" ) _assert_a4(frames) # --- the machine + the terminal -------------------------------------- _assert_machine( recorder.ticks, terminal, scan_total=N_FILES, files_total=len(CHANGED_RELS) ) assert terminal["detail"]["updated"] == len(CHANGED_RELS), terminal assert terminal["detail"]["unchanged"] == N_FILES - len(CHANGED_RELS), terminal assert terminal["scan_done"] == N_FILES, terminal assert terminal["files_done"] == len(CHANGED_RELS), terminal _wait_result_line( page, f"0 added · {len(CHANGED_RELS)} updated · {N_FILES - len(CHANGED_RELS)} unchanged", ) # --------------------------------------------------------------------------- # 3. The frames the live run produces, rendered by the REAL client # --------------------------------------------------------------------------- def test_scan_frames_and_the_hold_render_from_the_exact_payloads( page: Page, app_url: str, db_ready: None ) -> None: """Arms a and b, rendered. The scan window on an 8-file KB is milliseconds and the hold frames depend on which unchanged files happen to cost model time, so neither can be AIMED at from the browser. This arm feeds the exact payloads the real run answers — including the real HOLD sequence the importer's hook is pinned to in ``tests/integration/test_sync_pending_count.py`` (``[(changed,1,2), (unchanged,1,2), (unchanged,1,2), (changed,2,2)]``) — to the real ``sources.js`` through the page's own 2 s poll, and pins what the owner would read: the scan's own ``(n/total)``, the count holding at ``2 remaining`` while the named file changes, the part shrinking to ``1 remaining``, the part GONE on the final predicted file, and the phase-187 clause on the TERMINAL line only.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) src = SOURCE stages: dict[str, Any] = {"sync": None, "upload": None} _serve_statuses(page, stages) def show(**overrides: Any) -> None: stages["sync"] = _sync_status(state="running", started_at=_now_iso(), **overrides) # The scan: its OWN position, no file, no ``N remaining``. show(phase="scan", scan_done=3, scan_total=N_FILES) box = page.locator("#sync-progress-text") btn = page.locator("#sync-btn") _admin_sources_page(page, app_url, settled=False) _install_recorder(page) expect(box).to_have_text(f"Scanning… (3/{N_FILES})", timeout=SETTLE_TIMEOUT_MS) expect(btn).to_have_attribute("title", f"Scanning… (3/{N_FILES})") expect(btn).to_have_attribute("aria-label", f"Scanning… (3/{N_FILES})") # The PART's own toggle, not Playwright visibility: while a run is in # flight the button is the spinner and CSS hides the label spans, so the # honest pin is the ONE writer's ``hidden`` flag plus the span's text # (the in-browser recorder reads the same property). expect(page.locator(".sync-label-count")).to_have_text(f"(3/{N_FILES})") expect(page.locator(".sync-label-count")).to_have_js_property("hidden", False) expect(btn).to_be_disabled() # The scan climbs: the SAME label slot, the SAME branch, a new position. show(phase="scan", scan_done=8, scan_total=N_FILES) expect(box).to_have_text(f"Scanning… ({N_FILES}/{N_FILES})", timeout=SETTLE_TIMEOUT_MS) # …and the empty-KB / first-frame guard: 0/0 renders the BARE label. show(phase="scan", scan_done=0, scan_total=0) expect(box).to_have_text("Scanning…", timeout=SETTLE_TIMEOUT_MS) expect(page.locator(".sync-label-count")).to_have_text("") expect(page.locator(".sync-label-count")).to_have_js_property("hidden", True) expect(btn).to_have_attribute("title", "Scanning…") # The null prelude (the model check + clone/pull span): bare ``Syncing…``. show(phase=None, files_done=0, files_total=0) expect(box).to_have_text("Syncing…", timeout=SETTLE_TIMEOUT_MS) # THE HOLD: three unchanged files, one pair. The count does not move. _install_recorder(page) for rel in HELD_RELS: show(phase="import", current_file=f"{src}/{rel}", files_done=0, files_total=2) expect(box).to_have_text( f"Syncing… {src}/{rel} 2 remaining", timeout=SETTLE_TIMEOUT_MS ) # The first predicted file — with the phase-129 chunk part alongside. show( phase="import", current_file=f"{src}/{CHANGED_RELS[0]}", files_done=1, files_total=2, current_chunk=2, chunks_total=3, ) expect(box).to_have_text( f"Syncing… {src}/{CHANGED_RELS[0]} 1 remaining · chunk 2/3", timeout=SETTLE_TIMEOUT_MS, ) # The final predicted file: the part is GONE, the file stays. show(phase="import", current_file=f"{src}/{CHANGED_RELS[1]}", files_done=2, files_total=2) expect(box).to_have_text(f"Syncing… {src}/{CHANGED_RELS[1]}", timeout=SETTLE_TIMEOUT_MS) expect(page.locator(".sync-label-count")).to_have_text("") expect(page.locator(".sync-label-count")).to_have_js_property("hidden", True) # The terminal of the same run: the byte-identical counts line, and the # ONLY place the phase-187 clause appears (the running frames above # carried rss values and rendered NONE — AGENTS rule 10, terminal-only). stages["sync"] = _sync_status( state="success", started_at=_now_iso(), finished_at=_now_iso(), current_file=f"{src}/{CHANGED_RELS[1]}", files_done=2, files_total=2, scan_done=N_FILES, scan_total=N_FILES, detail={"added": 0, "updated": 2, "unchanged": N_FILES - 2}, rss_mb=512, rss_peak_mb=1234, ) expect(box).to_have_text("0 added · 2 updated · 6 unchanged · peak 1234 MB") expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL) expect(btn).to_be_enabled() texts = _texts(_frames(page)) scan_labels = [t for t in texts if t.startswith("Scanning…")] assert scan_labels, texts for label in scan_labels: assert SCAN_LABEL_RE.fullmatch(label), label syncing = [t for t in texts if t.startswith("Syncing…")] assert syncing, texts for label in syncing: assert IMPORT_LABEL_RE.match(label), label # The pre-phase shape stays dead on this part (the scan's own # ``(n/total)`` lives only behind the word "Scanning"). assert not OLD_FRACTION_RE.search(label), label held = [t for t in syncing if _count_of(t) == 2] assert len(held) >= 2, held assert len({_file_of(t) for t in held}) >= 2, held # The clause NEVER appears on a running label — scan or import — and the # terminal's is text on the line that already exists (one child span). for label in scan_labels + syncing: assert "peak" not in label, label assert page.locator("#sync-progress > *").count() == 1 # --------------------------------------------------------------------------- # 4. (arm d) BOR_SYNC_PENDING_COUNT=0 — the phase-193 rollback, live # --------------------------------------------------------------------------- def test_kill_switch_zero_restores_the_walk_position_count( page: Page, zero_server: str, db_ready: None, zero_repo: Path ) -> None: """The second module app, same fixture, ``BOR_SYNC_PENDING_COUNT=0``: the exact phase-193 machine and the exact phase-193 numbers. The contrast IS the phase. For the file at walk position 5 of 8 this app renders ``3 remaining`` (``8 − 5`` — files that will never be re-indexed counted as work) while the default app, on the same repository shape and the same moment of the same walk, renders ``1 remaining``. And no tick of this run ever reports ``phase: "scan"``: the pre-scan does not run, so the scan pair is ``0/0`` for its whole life, terminal included.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) _seed_kb(zero_repo, null_summary_rels=HELD_RELS) _change_files(zero_repo, ZERO_CHANGED_RELS, "arm d") walk_map = _walk_positions(tuple(REL_ORDER)) _admin_sources_page(page, zero_server) _install_recorder(page) recorder = _TickRecorder(zero_server, "/api/sync/status") recorder.start() page.locator("#sync-btn").click() expect(page.locator("#sync-progress")).to_have_text("Syncing…") # The LAST walked file (changed here) is the only one that hides the part # under the walk semantics. expect(page.locator("#sync-progress-text")).to_have_text( re.compile( rf"^Syncing… pendingzero/{re.escape(LAST_REL)}" r"(?! \d+ remaining)(?: · chunk \d+/\d+)?$" ), timeout=SETTLE_TIMEOUT_MS, ) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS) terminal = recorder.stop() assert terminal["state"] == "success", terminal # No scan phase exists on this app, at all, ever. assert [t for t in recorder.ticks if t["phase"] == "scan"] == [], recorder.ticks for tick in recorder.ticks: assert (tick["scan_done"], tick["scan_total"]) == (0, 0), tick assert terminal["phase"] is None, terminal assert (terminal["scan_done"], terminal["scan_total"]) == (0, 0), terminal # The pair is the WALK: every named tick carries the 1-based walk position # over all 8 importable files, and the denominator is the whole KB. named = [t for t in recorder.running_ticks if t["current_file"]] assert named, "the run never reported a file at all" for tick in named: rel = str(tick["current_file"]).removeprefix("pendingzero/") assert (tick["files_done"], tick["files_total"]) == walk_map[rel], tick assert tick["files_total"] == N_FILES, tick assert terminal["files_done"] == N_FILES, terminal assert terminal["files_total"] == N_FILES, terminal assert terminal["detail"]["updated"] == len(ZERO_CHANGED_RELS), terminal _assert_machine(recorder.ticks, terminal, scan_total=0, files_total=N_FILES) # The rollback's OWN defect, rendered: a walk-derived count (the files in # front of the first changed one still counted). Arm b pins the number # this fixture produces with the switch on: ``2 remaining``. frames = _frames(page) texts = _texts(frames) walk_counts = [ n for t in texts if t.startswith("Syncing… ") and _file_of(t) in HELD_RELS and (n := _count_of(t)) is not None ] assert walk_counts, f"no unchanged file rendered a walk-derived count: {texts[:12]}" assert max(walk_counts) > len(ZERO_CHANGED_RELS), ( f"the rollback app counted the WORK, not the walk: {walk_counts}" ) for frame in frames: if frame["text"].startswith("Scanning…"): raise AssertionError(f"the rollback app rendered a scan label: {frame}") _assert_a4(frames) _wait_result_line( page, f"0 added · {len(ZERO_CHANGED_RELS)} updated " f"· {N_FILES - len(ZERO_CHANGED_RELS)} unchanged", ) # --------------------------------------------------------------------------- # 5. (arm c) The no-change re-sync: an empty predicted set, no count part # --------------------------------------------------------------------------- def test_no_change_sync_reports_no_denominator( page: Page, app_url: str, db_ready: None, pending_repo: Path ) -> None: """The re-sync of a repository nothing touched: the scan walks all 8 files and predicts NONE of them, so every tick answers ``files_total == 0`` — the phase-64 ``total === 0`` guard driven by zero WORK rather than by a missing denominator — and no label EVER carries a count part. The pre-phase label on this exact run read ``8 remaining``, ``7 remaining``… while nothing was going to be re-indexed. A no-change run makes no import LLM call, so its import window is milliseconds and it renders no per-file label: the wire pair (``0/0`` on every tick) plus test 3's ``total === 0`` guard render the owner-visible half deterministically.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) _seed_kb(pending_repo) _admin_sources_page(page, app_url) _install_recorder(page) recorder = _TickRecorder(app_url, "/api/sync/status") recorder.start() btn = page.locator("#sync-btn") btn.click() # The click's OWN state first (phase 64): the app's in-memory status still # carries the PREVIOUS test's terminal, so a settle wait started straight # after the click can pass on that stale "Synced HH:MM" instead of on this # run. The spinner + the bare live label are THIS run's first frame. expect(btn).to_be_disabled() expect(page.locator("#sync-progress")).to_have_text("Syncing…") expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS) terminal = recorder.stop() assert terminal["state"] == "success", terminal for tick in recorder.running_ticks: assert (tick["files_done"], tick["files_total"]) == (0, 0), ( f"a no-change run reported work to do: {tick}" ) _assert_machine(recorder.ticks, terminal, scan_total=N_FILES, files_total=0) # The scan DID walk the whole KB — the divergence (8 scanned, 0 to # re-index) kept at the terminal, beside the untouched counts line. assert terminal["scan_done"] == N_FILES, terminal assert terminal["scan_total"] == N_FILES, terminal assert terminal["detail"]["unchanged"] == N_FILES, terminal texts = _texts(_frames(page)) for label in texts: assert _count_of(label) is None, ( f"a no-change run rendered a count part: {label}" ) if label.startswith("Syncing…"): assert not OLD_FRACTION_RE.search(label), label _wait_result_line(page, f"0 added · {N_FILES} unchanged") # --------------------------------------------------------------------------- # 6. (arm e) The empty source: the degenerate D1 pin # --------------------------------------------------------------------------- def _register_local_source(base_url: str, path: Path) -> None: """Store one ``kind=local`` source row through the admin API (the phase-49 contract the Sources page's form used to wrap): a stored row IS the effective source, which is what lets the degenerate and prod-shaped arms run on the module's own app instead of a third boot.""" with httpx.Client(base_url=base_url, timeout=30.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text r = client.post("/api/git-sources", json={"kind": "local", "path": str(path)}) assert r.status_code == 201, f"{r.status_code}: {r.text}" def test_empty_source_stays_in_scan_with_no_count_part( page: Page, app_url: str, db_ready: None, empty_kb_dir: Path ) -> None: """A source with ZERO importable files: the walk yields nothing, so no ``progress`` call exists to flip the phase — the machine goes ``null`` → ``"scan"`` → terminal and NEVER says ``"import"``, both pairs stay ``0/0``, and the page renders no count part anywhere (the same ``scan_total === 0`` guard test 3 pins on the exact payload). The scan window of an empty walk is a handful of milliseconds, so the live pins are the deterministic ones (no import tick exists to catch, and the terminal keeps the zeroed pair); the server-side degenerate is ``tests/integration/test_sync_scan_phase.py``'s empty-KB arm.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) _register_local_source(app_url, empty_kb_dir) _admin_sources_page(page, app_url) _install_recorder(page) recorder = _TickRecorder(app_url, "/api/sync/status", interval_s=0.02) recorder.start() btn = page.locator("#sync-btn") btn.click() # The click's OWN state first (phase 64): the app's in-memory status still # carries the PREVIOUS test's terminal, so a settle wait started straight # after the click can pass on that stale "Synced HH:MM" instead of on this # run. The spinner + the bare live label are THIS run's first frame. expect(btn).to_be_disabled() expect(page.locator("#sync-progress")).to_have_text("Syncing…") expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS) terminal = recorder.stop() assert terminal["state"] == "success", terminal assert [t for t in recorder.ticks if t["phase"] == "import"] == [], ( f"an empty walk flipped to import: {recorder.ticks}" ) for tick in recorder.ticks: assert (tick["scan_done"], tick["scan_total"]) == (0, 0), tick assert (tick["files_done"], tick["files_total"]) == (0, 0), tick _assert_machine(recorder.ticks, terminal, scan_total=0, files_total=0) assert terminal["detail"] == {} or not any( terminal["detail"].get(k) for k in ("added", "updated", "pruned", "errors") ), terminal texts = _texts(_frames(page)) for label in texts: assert _count_of(label) is None, f"the empty run rendered a count part: {label}" if label.startswith("Scanning…"): assert label == "Scanning…", label _wait_result_line(page, "0 added · 0 unchanged") # --------------------------------------------------------------------------- # 7. The owner's report at scale: a 1,000-file scan, two files of work # --------------------------------------------------------------------------- def test_prod_shaped_run_scan_climbs_then_counts_two( page: Page, app_url: str, db_ready: None, wide_kb: Path ) -> None: """The report itself, in a browser. A 1,000-file KB, TWO files changed: the pre-scan's OWN position climbs across several distinct values while the server keeps answering (the scan runs off the loop — the phase-179 contract, which is exactly why this poll is served THROUGH it), the import then reports the PREDICTED denominator (``files_total == 2`` against ``scan_total == 1000``: the divergence the pre-phase shape could not express), and the terminal keeps BOTH pairs side by side. This is also the arm that makes the scan's ``(n/total)`` visible for real: eight tiny files pre-scan in single-digit milliseconds, a thousand take long enough for the position to be observed moving. The ``N remaining`` part never exceeds the work — the owner's ``1216 remaining`` is the bug.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) _seed_kb(wide_kb, git=False) for rel in WIDE_CHANGED_RELS: path = wide_kb / rel path.write_text( path.read_text(encoding="utf-8") + f"\narm f edit of {rel}\n", encoding="utf-8" ) _register_local_source(app_url, wide_kb) _admin_sources_page(page, app_url) _install_recorder(page) recorder = _TickRecorder(app_url, "/api/sync/status", interval_s=0.02) recorder.start() btn = page.locator("#sync-btn") btn.click() # The click's OWN state first (phase 64): the app's in-memory status still # carries the PREVIOUS test's terminal, so a settle wait started straight # after the click can pass on that stale "Synced HH:MM" instead of on this # run. The spinner + the bare live label are THIS run's first frame. expect(btn).to_be_disabled() expect(page.locator("#sync-progress")).to_have_text("Syncing…") expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS) terminal = recorder.stop() assert terminal["state"] == "success", terminal scan_ticks = [t for t in recorder.ticks if t["phase"] == "scan"] positions = sorted({t["scan_done"] for t in scan_ticks}) assert len(positions) >= 2, ( "the scan's position was never observably climbing — the pre-scan is " f"the phase's own progress surface: {positions} / {WIDE_FILES}" ) assert positions[-1] > positions[0], positions for tick in scan_ticks: # 0/0 is legal here (the scan's own first frame — the phase is set # before the first position is reported); every REPORTED denominator is # the whole importable walk. assert tick["scan_total"] in (0, WIDE_FILES), tick assert tick["current_file"] is None, tick assert (tick["files_done"], tick["files_total"]) == (0, 0), tick # The scan is answered THROUGH the running server: the poll kept getting # 200s while the pre-scan hashed and compared a thousand files. assert len(scan_ticks) >= 2, f"the poll starved during the scan: {len(scan_ticks)}" # The import: the WORK is the denominator, and it is never the walk. import_ticks = [t for t in recorder.running_ticks if t["phase"] == "import"] assert import_ticks, "the run never flipped to import" for tick in import_ticks: assert tick["files_total"] == len(WIDE_CHANGED_RELS), tick assert [ t for t in recorder.ticks if t["files_total"] > len(WIDE_CHANGED_RELS) ] == [], ( "the running count reported a WALK denominator on a 1,000-file KB — " "the owner's exact report" ) _assert_machine( recorder.ticks, terminal, scan_total=WIDE_FILES, files_total=len(WIDE_CHANGED_RELS) ) assert terminal["detail"]["updated"] == len(WIDE_CHANGED_RELS), terminal assert terminal["detail"]["unchanged"] == WIDE_FILES - len(WIDE_CHANGED_RELS), terminal # The owner-visible half: no label ever says a walk-sized number remains, # and a scan label — if the 2 s poll landed in that window — is the honest # ``Scanning… (n/1000)`` and nothing else. texts = _texts(_frames(page)) for label in texts: n = _count_of(label) assert n is None or n <= len(WIDE_CHANGED_RELS), ( f"a label counted the walk instead of the work: {label}" ) if label.startswith("Scanning…"): assert SCAN_LABEL_RE.fullmatch(label), label assert str(WIDE_FILES) in label, label _wait_result_line( page, f"0 added · {len(WIDE_CHANGED_RELS)} updated " f"· {WIDE_FILES - len(WIDE_CHANGED_RELS)} unchanged", ) # --------------------------------------------------------------------------- # 8. The upload surface: phase 90's "no sync fields", plus the scan pair # --------------------------------------------------------------------------- def test_upload_run_carries_no_scan_fields_and_no_count_part( page: Page, app_url: str, db_ready: None, upload_archive: Path ) -> None: """Phase 194 gives the upload job NOTHING — and this is the pin that it did not leak. The REAL background upload's live status answers the bare phase-64 key set: no ``phase``, no chunk / image / summary fields, no memory pair and no ``scan_done`` / ``scan_total``, with ``null``/``0``/``0`` for the whole run (unpack + register, phase 90 A1/A2). Rendered through the ONE shared builder, that payload is a BARE ``Importing`` — no file and no count part.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) with httpx.Client(base_url=app_url, timeout=60.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text with upload_archive.open("rb") as fh: r = client.post( "/api/git-sources/upload", files={"file": (f"{UPLOAD_NAME}.tar.gz", fh, "application/gzip")}, ) assert r.status_code == 202, r.text seen: list[dict[str, Any]] = [] body: dict[str, Any] = {} for _ in range(200): body = client.get("/api/git-sources/upload/status").json() seen.append(body) if body["state"] in ("success", "failed"): break time.sleep(0.05) assert body["state"] == "success", body for tick in seen: assert set(tick) == UPLOAD_STATUS_KEYS, ( f"the upload status grew sync fields: {sorted(set(tick) ^ UPLOAD_STATUS_KEYS)}" ) assert tick["current_file"] is None, tick assert tick["files_done"] == 0 and tick["files_total"] == 0, tick # The shared label, on the page's own poll: the upload's REAL payload is # the bare prefix — no file, no count part (the part needs a denominator # the upload job never reports). stages: dict[str, Any] = { "sync": _sync_status(state="idle"), "upload": _upload_status(state="running", started_at=_now_iso()), } _serve_statuses(page, stages) _admin_sources_page(page, app_url, settled=False) _install_recorder(page) box = page.locator("#sync-progress-text") btn = page.locator("#sync-btn") expect(box).to_have_text("Importing", timeout=SETTLE_TIMEOUT_MS) expect(btn).to_have_attribute("title", "Importing") expect(page.locator(".sync-label-count")).to_have_text("") expect(page.locator(".sync-label-count")).to_have_js_property("hidden", True) expect(btn).to_be_disabled() stages["upload"] = _upload_status( state="success", detail={"message": "uploaded"}, finished_at=_now_iso() ) expect(page.locator("#sync-label")).to_have_text("Sync sources", timeout=SETTLE_TIMEOUT_MS) expect(page.locator("#sync-progress")).to_be_hidden(timeout=SETTLE_TIMEOUT_MS) texts = _texts(_frames(page)) importing = [t for t in texts if t.startswith("Importing")] assert importing, texts for label in importing: assert _count_of(label) is None, label assert not OLD_FRACTION_RE.search(label), label assert "Scanning" not in label and "(" not in label, label