"""Phase 193 story E2E (Playwright): the sync label counts DOWN. Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_sync_remaining_count.py -v --no-cov **Source (``TODO.md:7``):** "The sync status should show how many files are left to sync, not how many files out of the total." The pre-phase running label said ``Syncing… notes/a.md (1234/27000)`` — the owner's position *in* the whole; the owner reads the distance *to the end*. Phase 193 changes exactly ONE label part: ``fmtSyncLabel``'s count is now ``N remaining`` with ``N = total − done``. **Server untouched.** No endpoint, no payload field, no poll changed: ``GET /api/sync/status`` still answers with the phase-64 key set (``current_file`` / ``files_done`` / ``files_total`` — the 1-based position of the in-flight file over the pre-walk count), and the suite pins exactly that (test 1's cross-layer invariant): the payload still carries the ``(done, total)`` position while the label speaks the subtraction of the very same pair. The change is one string built by ``frontend/assets/sources.js``. **A1 (owner-confirmed):** ``N remaining`` shows while ``N ≥ 1`` and is HIDDEN at the final file (``N == 0``) — the main part names that file; ``total === 0`` (the pre-walk, the phase-64 contract) also renders no count part. **A2 (owner-confirmed):** the upload job shares the ONE builder, so it shares the shape (``Importing N remaining``). Fixture: a real git repo (the phase-28/32/107 ``file://`` transport-true convention — ``BOR_GIT_SOURCES=file://``, so the env fallback IS the effective source under the autouse truncate) holding ``N_FILES = 5`` markdown docs — the first AND the last multi-chunk (the phase-129 chunk-part climb rides a real embed span, and the last one is the window whose label must carry NO count part), the three in between one chunk each. ``tests/e2e/slow_llm.py`` delays every LLM call ``SLOW_DELAY_S``, so a file's import span (content-embed batches + the summary chat + the summary-chunk embed) is ≫ the UI's 2 s status poll: every file gets its own rendered label, deterministically, instead of racing it. What the owner sees is captured, not sampled: a MutationObserver installed before the click records EVERY value the single label writer lands — the ``#sync-progress-text`` box (the phase-150 sole sync text surface + sole ``aria-live`` region), the button ``title`` and ``aria-label`` (the phase-64 A4 untruncated placement) and the ``.sync-label-count`` span's visibility — so the assertions read the whole rendered sequence, not the few instants a Python poll happens to hit. Arms (the phase's three, in test order): 1. ``test_running_label_counts_down_across_a_real_sync`` — **(a) the count-down**: every rendered import label matches ``^Syncing… [1-N] remaining(?: · chunk n/m)?$``; the sequence of rendered ``N``s is strictly DECREASING (each new value smaller), never ``0``, never ``≥ total``; the ``· chunk n/m`` part coexists with it (its ``m`` equals the in-flight file's real ``chunk_document`` count — phase 129 unbroken); at the FINAL file no label carries a count part at all while the main part still names that file; no ``Syncing…`` label anywhere carries the pre-phase ``(n/m)`` tail; the ``(n/m)`` shape survives ONLY in the summaries phase (A1's exception) and the overview label still paints; the terminal result line is byte-identical to the phase-64 counts line (the ``counts_line`` helper); and the endpoint's own tick series still reports the untouched ``(done, total)`` position whose subtraction equals every rendered ``N``. 2. ``test_prewalk_and_final_file_labels_have_no_count_part`` — **(c) the pre-walk edge**, pinned on the exact payload the guard reads: the page's own 2 s poll fed ``phase: null`` / ``files_total: 0`` renders the bare ``Syncing…`` (no count part, the count span hidden, the title equal), the first file renders ``N − 1 remaining``, a mid-run tick renders ``1 remaining`` *beside* ``· chunk 2/5``, the final-file tick (``done == total``) renders the file with the count part GONE and never brings it back, and the success terminal settles to the untouched counts line. The served bodies use the status endpoint's unchanged key set — the client's own decision points (``total > 0``, ``total − done > 0``) are what is under test. 3. ``test_upload_label_shares_the_count_down_builder`` — **(b) the upload arm (A2)**: the upload branch of the same two-job tree shares the builder, so a running upload status that carries counts renders ``Importing N remaining`` and hides the part at the upload's final file, settling to the upload's own unchanged idle surface. The REAL background upload run is asserted first, against the live endpoint: since phase 90 it is unpack + register only, so it answers ``null``/``0``/``0`` for the whole run — which is exactly why the shared *shape* is pinned through a served status (the house ``page.route`` pattern) instead of a real upload, and why no real upload label can ever regress to ``(n/m)``. The upload branch never receives a status object (its endpoint has no phase / chunk / image fields — phase 129 / 152), so the chunk part coexists in the SYNC arms only (test 1 and 2); that is asserted here too. Test → phase-task mapping (Playwright Mapping Rule, task 03 arms a/b/c): 1 → (a) + the terminal-line pin + the untouched-payload pin, 2 → (c) + the A1 hidden-at-final-file edge (+ (a)'s arithmetic on exact inputs), 3 → (b). """ from __future__ import annotations import io import json import os import re import subprocess import sys import tarfile import threading import time from collections.abc import Iterator from datetime import UTC, datetime from pathlib import Path from typing import Any import httpx import pytest from playwright.sync_api import Page, expect from sqlalchemy import text from app.config import Settings from app.db import SessionLocal from app.rag.chunker import chunk_document from e2e.auth_helpers import login from e2e.conftest import ( ADMIN_PASSWORD, MOCK_PORT, SESSION_SECRET, USE_REAL_LLM, _wait_http, ) from e2e.sync_helpers import counts_line REPO = Path(__file__).resolve().parents[2] #: This module's own port (the conftest session app owns 8123 in a #: combined run — the phase-79 convention). Env-overridable. APP_PORT = int(os.environ.get("E2E_APP_PORT_REMAINING", "8211")) APP_URL = f"http://127.0.0.1:{APP_PORT}" SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_REMAINING", "8914")) SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}" SOURCES_URL = "/sources.html" #: Per-LLM-request delay on the proxy (the phase-64/129 house value). #: The run's PRELUDE (the two model probes) is ≈ 3 s and therefore #: outlives the page's first 2 s tick (the bare-label contract the click #: leaves up), and a one-chunk file costs 3 calls ≈ 4.5 s while a #: multi-chunk file ≈ 10 s — every file gets several rendered labels of #: its own (the module docstring's determinism note). SLOW_DELAY_S = "1.5" #: The module's wall-time budget (app boot + one full 5-file sync + the #: two served-status arms). A breach is a speed canary, not flake #: tolerance (the phase-64 convention). WALL_BUDGET_S = 300.0 #: The sync's subject: a git repo whose DIRECTORY NAME is the source #: name (the ``repo_name`` basename rule, phase 28). Deliberately NOT #: named after the part under test — a source path containing the word #: "remaining" would make every "does this label carry the count part?" #: check match the path itself. SOURCE = "countdown" N_FILES = 5 #: The walk order is the sorted relative path, so the names below pin #: the ORDER the owner watches: the first file (multi-chunk → the chunk #: part climbs while the count reads ``4 remaining``), three single-chunk #: files (3 → 2 → 1 remaining), the LAST file (multi-chunk → its own #: chunk part, NO count part — the A1 hidden edge). FIRST_REL = "notes/a-multi-first.md" LAST_REL = "notes/z-multi-last.md" REL_ORDER = [FIRST_REL, "notes/b-second.md", "notes/c-third.md", "notes/d-fourth.md", LAST_REL] #: The count-bearing values the label may ever show (A1): ``1 … N−1``. MAX_COUNT = N_FILES - 1 #: The archive the upload arm uploads (the phase-98 fixture convention — #: the same kind of content as the sync fixture, packaged). UPLOAD_NAME = "uploadkb" UPLOAD_FILES: dict[str, str] = { f"docs/{i:02d}.md": f"# Upload doc {i:02d}\n\nDeterministic upload body {i:02d}.\n" for i in range(3) } #: The chunker settings the app under test is pinned to (the leak-guard #: convention): the expected ``chunks_total`` is computed with the SAME #: function and arguments the importer uses, so a local ``.env`` cannot #: move the number the suite asserts. CHUNK_TARGET = int(Settings.model_fields["chunk_target_chars"].default) CHUNK_OVERLAP = int(Settings.model_fields["chunk_overlap_chars"].default) #: The import-phase label grammar since phase 193 — the main part (the #: prefix + the ``source/relative/path``), the OPTIONAL count part, the #: OPTIONAL phase-129 chunk part. ```` is kept loose on purpose #: where a suite-wide grammar is the point; the per-file assertions #: below re-anchor it on this suite's own paths. IMPORT_LABEL_RE = re.compile( r"^Syncing… \S+(?: [0-9]+ remaining)?(?: · chunk \d+/\d+)?$" ) #: The pre-phase shape phase 193 removed from this part — no ``Syncing…`` #: label may ever carry a ``(n/m)`` tail again. OLD_FRACTION_RE = re.compile(r"\(\d+/\d+\)") SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$") SETTLE_TIMEOUT_MS = 180_000 def _now_iso() -> str: """A "just finished" stamp for the served terminals — the phase-154 same-day rule renders it as the plain ``Synced HH:MM`` (computed per call, so a module that runs across midnight stays honest).""" return datetime.now(UTC).isoformat() # --------------------------------------------------------------------------- # The fixture KB # --------------------------------------------------------------------------- def _tiny_md(title: str) -> str: """A one-chunk document (its import span = 3 slowed LLM calls).""" return f"# {title}\n\nDeterministic one-chunk body for the {title}.\n" def _multi_md(tag: str, sections: int = 4) -> str: """A multi-chunk document: an H1 preamble (one chunk) + *sections* ``##`` sections of ~1500 chars each (one chunk apiece at the code-default 2000-char target, and one embed batch apiece at the 700-token request budget) — no randomness, so the chunk count is stable run to run (the phase-129 fixture idiom).""" parts = [f"# {tag} multi-chunk fixture document\n"] parts.append( f"Preamble paragraph of the {tag} remaining-count fixture. This " "paragraph sits under the H1 and chunks on its own.\n" ) for k in range(1, sections + 1): parts.append(f"\n## Section {k}\n") base = ( f"Section {k} paragraph of the {tag} remaining-count fixture " "carries deterministic filler for the chunker: alpha bravo " "charlie delta echo foxtrot golf hotel india juliet kilo lima " "mike november oscar papa quebec romeo sierra tango uniform " "victor whiskey xray yankee zulu number " ) parts.append((base * 4)[:1500] + "\n\n") return "".join(parts) def _fixture_files() -> dict[str, str]: return { FIRST_REL: _multi_md("first"), REL_ORDER[1]: _tiny_md("second"), REL_ORDER[2]: _tiny_md("third"), REL_ORDER[3]: _tiny_md("fourth"), LAST_REL: _multi_md("last"), } def _expected_content_chunks(rel: str) -> int: """The file's TRUE content-chunk count — the SAME function and arguments the importer's chunk replacement uses (the phase-129 convention: the endpoint's ``chunks_total`` is the chunk count, not a batch count).""" content = _fixture_files()[rel] return len(chunk_document(content, rel, max(400, CHUNK_TARGET), CHUNK_OVERLAP)) def _git(cwd: Path, *args: str) -> None: """Run git in *cwd*; a non-zero exit fails the fixture loudly.""" proc = subprocess.run(["git", *args], cwd=cwd, capture_output=True, text=True) if proc.returncode != 0: raise AssertionError(f"git {' '.join(args)} failed: {proc.stderr.strip()}") @pytest.fixture(scope="module") def remaining_repo(tmp_path_factory: pytest.TempPathFactory) -> Path: """The real git repo the sync clones over ``file://`` (5 markdown docs — see ``REL_ORDER``). ``tmp_path_factory`` because the module-scoped app needs it for the module's lifetime.""" root = tmp_path_factory.mktemp("bor_remaining") repo = root / SOURCE for rel, content in _fixture_files().items(): target = repo / rel target.parent.mkdir(parents=True, exist_ok=True) target.write_text(content, encoding="utf-8") _git(repo, "init", "-q") _git(repo, "add", "-A") _git( repo, "-c", "user.email=e@x", "-c", "user.name=t", "-c", "commit.gpgsign=false", "commit", "-qm", "remaining-count fixture", ) assert (repo / ".git").is_dir() assert len(list(repo.rglob("*.md"))) == N_FILES return repo @pytest.fixture(scope="module") def upload_archive(tmp_path_factory: pytest.TempPathFactory) -> Path: """The upload arm's archive (the ``test_sync_upload_progress.py`` convention: the same shape of content as the sync fixture, packed as a deterministic ``.tar.gz``).""" root = tmp_path_factory.mktemp("bor_remaining_archive") path = root / f"{UPLOAD_NAME}.tar.gz" with tarfile.open(path, "w:gz") as tf: for rel, content in UPLOAD_FILES.items(): data = content.encode("utf-8") info = tarfile.TarInfo(rel) info.size = len(data) info.mtime = 0 tf.addfile(info, io.BytesIO(data)) return path def _app_env( base_url: str, repo: Path, upload_dir: Path, tmp_path_factory: pytest.TempPathFactory ) -> dict[str, str]: """The per-module app env (the conftest pattern): the slow proxy is the LLM, ``BOR_GIT_SOURCES`` is this suite's own ``file://`` repo (the env fallback IS the effective source — the autouse truncate keeps the ``git_sources`` table empty), and the chunker knobs are pinned to the code defaults so the expected ``chunks_total`` stays predictable (process env outranks an operator's gitignored ``.env``).""" env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = ( "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url ) env["BOR_RELEVANCE_THRESHOLD"] = "0.30" env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15" env["BOR_LLM_RETRY_DELAY"] = "0" env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET env["BOR_GIT_SOURCES"] = f"file://{repo}" env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts")) env["BOR_UPLOAD_DIR"] = str(upload_dir) env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET) env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP) env["BOR_DOCS_REPO"] = "" env["BOR_SUGGESTIONS"] = json.dumps(["Ask something neutral"]) return env @pytest.fixture(scope="module") def slow_llm(mock_llm: int) -> Iterator[int]: """The delay-injecting reverse proxy in front of the mock LLM (``tests/e2e/slow_llm.py``) — the suite's timing fixture: it delays EVERY LLM call, embeds included, so each file's import span outlives the UI's 2 s poll (see ``SLOW_DELAY_S``).""" env = dict(os.environ) env.pop("DEBUGPY", None) env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S env["E2E_MOCK_PORT"] = str(MOCK_PORT) proc = subprocess.Popen( [sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app", "--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"], cwd=REPO, env=env, ) try: _wait_http(f"{SLOW_URL}/v1/models") yield SLOW_PORT finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_server( mock_llm: int, slow_llm: int, remaining_repo: Path, tmp_path_factory: pytest.TempPathFactory, ) -> Iterator[str]: """The app under test — the slow one, on its own port.""" env = _app_env( f"{SLOW_URL}/v1", remaining_repo, tmp_path_factory.mktemp("bor_remaining_uploads") / "uploads", tmp_path_factory, ) proc = subprocess.Popen( [sys.executable, "-m", "uvicorn", "app.main:app", "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], cwd=REPO, env=env, ) try: _wait_http(f"{APP_URL}/api/health") yield APP_URL finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server def _truncate_all() -> None: """Fresh registry + KB per test (the E2E isolation pattern): the suites share one Postgres, and a leftover document would move the sync's ``added`` counts the terminal line pins.""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, steering_notes, " "kb_overview, git_sources, folder_summaries" ) ) db.commit() def _wait_sync_settled(base_url: str, timeout_s: float = 240.0) -> None: """No run may leak across a test boundary (the run state lives in the app's memory and a running run keeps importing into the NEXT test's truncated KB). Own admin session; usually a no-op.""" with httpx.Client(base_url=base_url, timeout=5.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: r = client.get("/api/sync/status") assert r.status_code == 200, r.text body = r.json() if body["state"] == "failed": print(f"sync remaining-count: FAILED terminal at a boundary: {body.get('error')}") if body["state"] != "running": return time.sleep(0.2) raise AssertionError(f"a sync was still running at a test boundary: {base_url}") @pytest.fixture(autouse=True) def _clean(app_url: str, db_ready: None) -> Iterator[None]: _wait_sync_settled(app_url) _truncate_all() yield _truncate_all() @pytest.fixture(scope="module", autouse=True) def _wall_time_guard() -> Iterator[None]: start = time.monotonic() yield elapsed = time.monotonic() - start assert elapsed < WALL_BUDGET_S, ( f"the module took {elapsed:.1f}s — over the {WALL_BUDGET_S:.0f}s wall budget" ) # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- #: The in-browser recorder: every value the ONE label writer lands, with #: the surfaces phase 64's A4 pins together (the box text, the button #: title, the button aria-label) plus the count span's own visibility — #: read in one JS task, so a frame can never straddle two writes. _INSTALL_RECORDER = """ () => { if (window.__borFrames) return window.__borFrames.length; window.__borFrames = []; const box = document.querySelector('#sync-progress'); const textEl = document.querySelector('#sync-progress-text'); const btn = document.querySelector('#sync-btn'); const countEl = document.querySelector('.sync-label-count'); const rec = () => { const frame = { text: textEl.textContent, title: btn.getAttribute('title') || '', aria: btn.getAttribute('aria-label') || '', count: countEl ? countEl.textContent : '', countHidden: countEl ? countEl.hidden : null, hidden: box.hidden, }; const last = window.__borFrames[window.__borFrames.length - 1]; if (last && last.text === frame.text && last.title === frame.title && last.aria === frame.aria && last.countHidden === frame.countHidden) { return; } window.__borFrames.push(frame); }; const mo = new MutationObserver(rec); mo.observe(textEl, { childList: true, characterData: true, subtree: true }); mo.observe(btn, { attributes: true, attributeFilter: ['title', 'aria-label'] }); if (countEl) mo.observe(countEl, { attributes: true, attributeFilter: ['hidden'] }); rec(); return window.__borFrames.length; } """ def _install_recorder(page: Page) -> None: page.evaluate(_INSTALL_RECORDER) def _frames(page: Page) -> list[dict[str, Any]]: frames: list[dict[str, Any]] = page.evaluate("() => window.__borFrames") assert frames, "the in-browser recorder captured no label frames" return frames def _texts(frames: list[dict[str, Any]]) -> list[str]: """The recorded label sequence, empty frames dropped (the box's idle-hide is not a label).""" return [f["text"] for f in frames if f["text"]] class _TickRecorder: """The deterministic layer: a daemon thread tight-polling (~100 ms) the SAME status endpoint the UI's 2 s poll reads, with its OWN admin session (the Playwright sync API is not thread-safe, so the thread never touches the page). It starts nothing — it only watches.""" def __init__(self, base_url: str, path: str) -> None: self._url = f"{base_url}{path}" self._login_url = f"{base_url}/api/login" self._ticks: list[dict[str, Any]] = [] self._terminal: dict[str, Any] | None = None self._stop = threading.Event() self._thread: threading.Thread | None = None def start(self) -> None: def run() -> None: with httpx.Client(timeout=5.0) as client: client.post(self._login_url, json={"password": ADMIN_PASSWORD}) while not self._stop.is_set(): try: r = client.get(self._url) if r.status_code == 200: body = r.json() self._ticks.append(body) if body["state"] in ("success", "failed", "cancelled"): self._terminal = body return except Exception: # noqa: BLE001 — blip: retry next tick pass self._stop.wait(0.1) self._thread = threading.Thread(target=run, daemon=True) self._thread.start() def stop(self, timeout_s: float = 240.0) -> dict[str, Any]: deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: if self._terminal is not None: break time.sleep(0.05) self._stop.set() if self._thread is not None: self._thread.join(timeout=5) assert self._terminal is not None, ( f"the recorder saw no terminal state (last: {self._ticks[-3:]})" ) return self._terminal @property def running_ticks(self) -> list[dict[str, Any]]: return [t for t in self._ticks if t["state"] == "running"] def _serve_statuses(page: Page, stages: dict[str, Any]) -> None: """Route the two job-status endpoints to the CURRENT stage body (the house ``page.route`` pattern). ``stages`` is mutated by the test between the UI waits, so each 2 s poll of the REAL page renders exactly the payload the test put in front of it — the client's own decision points, pinned without a timing race. ``stages`` keys: ``sync`` / ``upload`` → the body dicts. A ``None`` value falls through to the real endpoint.""" def handle(route: Any) -> None: path = route.request.url if "/api/sync/status" in path: body = stages.get("sync") elif "/api/git-sources/upload/status" in path: body = stages.get("upload") else: # pragma: no cover — only the two patterns are routed route.fallback() return if body is None: route.fallback() return route.fulfill(status=200, content_type="application/json", body=json.dumps(body)) page.route("**/api/sync/status", handle) page.route("**/api/git-sources/upload/status", handle) def _sync_status(**overrides: Any) -> dict[str, Any]: """One ``GET /api/sync/status`` body with the endpoint's UNCHANGED key set (the phase-64/98/129/152/187 fields the client reads) — the served stubs are faithful to the payload shape; only the values are the test's.""" body: dict[str, Any] = { "state": "idle", "started_at": None, "finished_at": None, "current_file": None, "files_done": 0, "files_total": 0, "phase": None, "current_summary": None, "summaries_done": 0, "summaries_total": 0, "current_chunk": 0, "chunks_total": 0, "images_done": 0, "images_total": 0, "detail": {}, "error": None, "rss_mb": None, "rss_peak_mb": None, } body.update(overrides) return body def _upload_status(**overrides: Any) -> dict[str, Any]: body: dict[str, Any] = { "state": "idle", "started_at": None, "finished_at": None, "current_file": None, "files_done": 0, "files_total": 0, "detail": {}, "error": None, } body.update(overrides) return body def _admin_sources_page(page: Page, app_url: str, *, settled: bool = True) -> None: """Real form login landing on the RAG page (the button's home). ``settled=True`` also waits for the boot re-attach to finish in a NON-running state (the phase-64 pattern — no later re-render can clobber the state under test). The served-status arms pass ``settled=False``: their boot fetch IS the state under test.""" login(page, app_url) expect(page).to_have_url(app_url + SOURCES_URL, timeout=30_000) expect(page.locator("#sync-btn")).to_be_visible(timeout=30_000) if settled: expect(page.locator("#sync-label")).to_have_text( re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$"), timeout=30_000 ) # --------------------------------------------------------------------------- # 1. (a) The count-down across a REAL sync of a REAL git source # --------------------------------------------------------------------------- def test_running_label_counts_down_across_a_real_sync( page: Page, app_url: str, db_ready: None, remaining_repo: Path ) -> None: """The owner-visible contract: the label counts DOWN and goes quiet at the final file — over a real ``file://`` git sync with the real (untouched) status endpoint.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) _admin_sources_page(page, app_url) _install_recorder(page) recorder = _TickRecorder(app_url, "/api/sync/status") recorder.start() btn = page.locator("#sync-btn") btn.click() # The click's own state (the phase-64 contract the pre-walk keeps): # the button becomes the spinner and the box reveals the BARE label. expect(btn).to_be_disabled() expect(btn).to_have_attribute("aria-busy", "true") expect(page.locator("#sync-progress")).to_have_text("Syncing…") # The first file's window: the count part AND the chunk part (the # first file is the multi-chunk one, so its embed batches climb # while the count reads N − 1 remaining — the coexistence pin). expect(page.locator("#sync-progress-text")).to_have_text( re.compile( rf"^Syncing… {re.escape(SOURCE)}/{re.escape(FIRST_REL)}" rf" {MAX_COUNT} remaining · chunk \d+/\d+$" ), timeout=SETTLE_TIMEOUT_MS, ) # The final file's window: the main part names it and NO count part # rides along (A1's hidden edge; the chunk part may — the last file # is multi-chunk too). expect(page.locator("#sync-progress-text")).to_have_text( re.compile( rf"^Syncing… {re.escape(SOURCE)}/{re.escape(LAST_REL)}" r"(?!.* remaining)(?: · chunk \d+/\d+)?$" ), timeout=SETTLE_TIMEOUT_MS, ) terminal = recorder.stop() assert terminal["state"] == "success", terminal frames = _frames(page) texts = _texts(frames) # --- the rendered sequence's shape --------------------------------- import_frames = [f for f in frames if f["text"].startswith("Syncing… ")] assert import_frames, f"no running import label was ever rendered: {texts[:8]}" for frame in import_frames: assert IMPORT_LABEL_RE.match(frame["text"]), frame # The pre-phase position shape is gone from this part for good. assert not OLD_FRACTION_RE.search(frame["text"]), frame # A4: the untruncated label rides the title AND the announcer. assert frame["title"] == frame["text"], frame assert frame["aria"] == frame["text"], frame # The count span mirrors the part: visible iff the text carries it. assert frame["countHidden"] is not (" remaining" in frame["text"]), frame if " remaining" in frame["text"]: shown = re.search(r" (\d+) remaining", frame["text"]) assert shown is not None, frame assert 1 <= int(shown.group(1)) <= MAX_COUNT, ( f"a count-bearing label showed {shown.group(1)} " f"(allowed 1…{MAX_COUNT}): {frame}" ) # --- the count-down itself ----------------------------------------- observed: list[int] = [] for label in texts: m = re.search(r" (\d+) remaining(?: · |$)", label) if m: value = int(m.group(1)) if value not in observed: observed.append(value) assert len(observed) >= 3, ( f"the label never visibly counted down (distinct N values: {observed})" ) assert observed == sorted(observed, reverse=True), ( f"the count part did not count DOWN: {observed}" ) assert observed[0] == MAX_COUNT, f"the walk did not start at N − 1: {observed}" assert observed[-1] == 1, f"the count part never reached the last file: {observed}" assert 0 not in observed, "a '0 remaining' label rendered (A1's hidden edge)" # --- the chunk part coexists (phase 129, unbroken) ------------------ coexist = [ t for t in texts if re.search(r" \d+ remaining · chunk \d+/\d+$", t) ] assert coexist, f"the count and chunk parts never shared a label: {texts[:12]}" chunk_totals = { int(m.group(1)) for t in texts for m in [re.search(r" · chunk \d+/(\d+)$", t)] if m } assert _expected_content_chunks(FIRST_REL) in chunk_totals, ( f"the chunk part never reported the first file's real chunk count " f"({_expected_content_chunks(FIRST_REL)}): {sorted(chunk_totals)}" ) # --- the A1 edge, sequence-wide ------------------------------------ first_last = next( (i for i, t in enumerate(texts) if t.startswith(f"Syncing… {SOURCE}/{LAST_REL}")), None, ) assert first_last is not None, f"the final file was never named: {texts[:12]}" after = texts[first_last:] assert not any(" remaining" in t for t in after), ( f"the count part came back after the final file: {after[:8]}" ) # --- the untouched phases and the untouched terminal line ----------- assert any(t == "Writing KB overview…" for t in texts), texts[:20] summaries = [t for t in texts if t.startswith("Summarizing folders…")] assert summaries and all(OLD_FRACTION_RE.search(t) for t in summaries), ( f"the summaries phase's folder position changed shape: {summaries}" ) box = page.locator("#sync-progress-text") expect(box).to_have_text(counts_line(f"{N_FILES} added")) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL) expect(btn).to_be_enabled() assert terminal["detail"]["added"] == N_FILES, terminal # --- the cross-layer invariant: the label is the payload's subtraction # The endpoint is untouched: it still answers with the 1-based # position, and each rendered N equals total − done of the file it # named. position: dict[str, tuple[int, int]] = {} for tick in recorder.running_ticks: if tick["current_file"]: position[tick["current_file"]] = (tick["files_done"], tick["files_total"]) assert {f"{SOURCE}/{rel}" for rel in REL_ORDER} <= set(position), ( f"the status never reported every file: {sorted(position)}" ) for frame in import_frames: named = frame["text"].split(" · ")[0] path = named[len("Syncing… ") :].split(" ")[0] done, total = position[path] assert total == N_FILES, (path, done, total) expected = total - done shown = re.search(r" (\d+) remaining", frame["text"]) if expected > 0: assert shown is not None, ( f"{path}: the payload said {expected} left, the label said none: {frame}" ) assert int(shown.group(1)) == expected, (path, done, total, frame) else: assert shown is None, f"{path}: done==total but a part rendered: {frame}" # --------------------------------------------------------------------------- # 2. (c) The pre-walk edge + the A1 hidden edge, on the exact payloads the # guards read # --------------------------------------------------------------------------- def test_prewalk_and_final_file_labels_have_no_count_part( page: Page, app_url: str, db_ready: None ) -> None: """The two edges of A1, pinned on the payloads the guards actually read (the served bodies use the status endpoint's unchanged key set — what is under test is the client's ONE builder): ``total === 0`` renders no count part, and neither does ``done === total``.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) stages: dict[str, Any] = { "sync": _sync_status(state="running"), # the pre-walk: no file, 0/0 "upload": None, } _serve_statuses(page, stages) # settled=False: the boot fetch is the pre-walk stub — the re-attach # itself is part of what this arm pins. _admin_sources_page(page, app_url, settled=False) _install_recorder(page) box = page.locator("#sync-progress-text") btn = page.locator("#sync-btn") # (c) The pre-walk — phase null, current_file null, files_total 0: # the bare prefix, no count part, the count span hidden, the title # identical (A4). expect(box).to_have_text("Syncing…", timeout=SETTLE_TIMEOUT_MS) expect(page.locator(".sync-label-count")).to_be_hidden() expect(btn).to_have_attribute("title", "Syncing…") expect(btn).to_be_disabled() # The walk knows the total: the first file counts down from N − 1. stages["sync"] = _sync_status( state="running", phase="import", current_file=f"{SOURCE}/{FIRST_REL}", files_done=1, files_total=N_FILES, ) expect(box).to_have_text( f"Syncing… {SOURCE}/{FIRST_REL} {MAX_COUNT} remaining", timeout=SETTLE_TIMEOUT_MS, ) # The last count-bearing position, with the phase-129 chunk part # beside it (the two parts are independent and coexist). stages["sync"] = _sync_status( state="running", phase="import", current_file=f"{SOURCE}/{REL_ORDER[3]}", files_done=N_FILES - 1, files_total=N_FILES, current_chunk=2, chunks_total=5, ) expect(box).to_have_text( f"Syncing… {SOURCE}/{REL_ORDER[3]} 1 remaining · chunk 2/5", timeout=SETTLE_TIMEOUT_MS, ) # (a/A1) The final file: done == total → the part is ABSENT while the # main part still names the file (and the chunk part may stay). stages["sync"] = _sync_status( state="running", phase="import", current_file=f"{SOURCE}/{LAST_REL}", files_done=N_FILES, files_total=N_FILES, current_chunk=1, chunks_total=4, ) expect(box).to_have_text( f"Syncing… {SOURCE}/{LAST_REL} · chunk 1/4", timeout=SETTLE_TIMEOUT_MS, ) expect(page.locator(".sync-label-count")).to_be_hidden() expect(btn).to_have_attribute("title", f"Syncing… {SOURCE}/{LAST_REL} · chunk 1/4") # The success terminal: the untouched phase-64 counts line. The # stamp is NOW (the phase-154 rule renders a same-day completion as # the plain "Synced HH:MM"). stages["sync"] = _sync_status( state="success", started_at=_now_iso(), finished_at=_now_iso(), files_done=N_FILES, files_total=N_FILES, detail={"added": N_FILES, "updated": 0, "unchanged": 0, "pruned": 0}, ) expect(box).to_have_text(counts_line(f"{N_FILES} added"), timeout=SETTLE_TIMEOUT_MS) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000) expect(btn).to_be_enabled() # Nothing rendered in this run ever spoke the pre-phase shape or a # "0 remaining". texts = _texts(_frames(page)) assert not any(OLD_FRACTION_RE.search(t) for t in texts if t.startswith("Syncing…")), texts assert not any("0 remaining" in t for t in texts), texts assert "Syncing…" in texts, texts # --------------------------------------------------------------------------- # 3. (b) A2 — the upload job shares the builder and the shape # --------------------------------------------------------------------------- def test_upload_label_shares_the_count_down_builder( page: Page, app_url: str, db_ready: None, upload_archive: Path ) -> None: """A2: the upload's running label is the SAME builder's output, so it counts down in the SAME shape.""" page.set_default_timeout(SETTLE_TIMEOUT_MS) # The real upload run, against the live endpoint first: since phase # 90 it is unpack + register only, so its status carries # null/0/0 for the WHOLE run — the reason the shared shape is pinned # through a served status below (a real upload can never show a # count at all, and can therefore never regress to "(n/m)"). with httpx.Client(base_url=app_url, timeout=60.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text with upload_archive.open("rb") as fh: r = client.post( "/api/git-sources/upload", files={"file": (f"{UPLOAD_NAME}.tar.gz", fh, "application/gzip")}, ) assert r.status_code == 202, r.text body: dict[str, Any] = {} for _ in range(200): body = client.get("/api/git-sources/upload/status").json() if body["state"] in ("success", "failed"): break time.sleep(0.05) assert body["state"] == "success", body assert body["current_file"] is None, body assert body["files_done"] == 0 and body["files_total"] == 0, body # The shared shape, on the page's own poll: a running upload status # that DOES carry counts (the key set the endpoint already answers # with) renders the count-down part through the one builder. stages: dict[str, Any] = { "sync": _sync_status(state="idle"), "upload": _upload_status( state="running", current_file=f"{UPLOAD_NAME}/docs/01.md", files_done=1, files_total=3, ), } _serve_statuses(page, stages) # settled=False: the boot re-attach of the in-flight UPLOAD run is # the entry point this arm pins. _admin_sources_page(page, app_url, settled=False) _install_recorder(page) box = page.locator("#sync-progress-text") btn = page.locator("#sync-btn") expect(box).to_have_text( f"Importing {UPLOAD_NAME}/docs/01.md 2 remaining", timeout=SETTLE_TIMEOUT_MS ) expect(btn).to_have_attribute("title", f"Importing {UPLOAD_NAME}/docs/01.md 2 remaining") expect(btn).to_be_disabled() # The upload's final file: the same A1 edge through the same builder. stages["upload"] = _upload_status( state="running", current_file=f"{UPLOAD_NAME}/docs/02.md", files_done=3, files_total=3, ) expect(box).to_have_text(f"Importing {UPLOAD_NAME}/docs/02.md", timeout=SETTLE_TIMEOUT_MS) expect(page.locator(".sync-label-count")).to_be_hidden() # The upload settle (branch 6): the button returns to "Sync sources" # and the box hides — the upload's own unchanged surface. stages["upload"] = _upload_status( state="success", detail={"message": "uploaded"}, finished_at=_now_iso() ) expect(page.locator("#sync-label")).to_have_text("Sync sources", timeout=SETTLE_TIMEOUT_MS) expect(page.locator("#sync-progress")).to_be_hidden(timeout=SETTLE_TIMEOUT_MS) texts = _texts(_frames(page)) importing = [t for t in texts if t.startswith("Importing")] assert importing, texts for label in importing: assert re.fullmatch( rf"Importing {re.escape(UPLOAD_NAME)}/docs/\d{{2}}\.md" r"(?: [12] remaining)?", label, ), label assert not OLD_FRACTION_RE.search(label), label # The upload branch never receives a status object (its endpoint # has no phase / chunk / image fields — phase 129 / 152), so the # chunk part belongs to the sync arms only. assert " · chunk " not in label, label