"""Phase 152 task 05 E2E (Playwright, mock-only): during a PDF sync whose images take time to describe, the live label shows the image position MOVING — the owner's "stuck for several minutes" (TODO.md:3: the label sat on ``Syncing… … (2540/2540) · chunk 125/126`` through the PDF's vision spans) is impossible on the fixture. The dedicated story suite for ``152_sync_progress_image_pass`` (A16 — one Playwright file per phase, run in isolation): the ``on_image_progress hook (phase 152, tasks 02/03) fires immediately before each vision describe call with ``(position, total)``, the runner's ``_image_hook`` closure lands it on ``SyncStatus.images_done/images_total``, and ``fmtSyncPhaseLabel`` renders ``· image n/m`` — the phase-149 box (``#sync-progress``) is the surface that shows it, the button ``title``/``aria-label`` carry the UNTRUNCATED composed string (A4). Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the deterministic mock plus the phase-64 ``tests/e2e/slow_llm.py`` delay proxy in front of it (``SLOW_DELAY_S = 1.5 s``) that stretches EVERY LLM call (vision describes included) so the fixture's vision span outlives the UI's 2 s status poll and each position outlives the recorder's ~300 ms page-sampling cadence by ~10× — the observed positions are dense samples of a real climb, not races. Timing fixture (the phase-129/98 rationale, re-sized for vision spans): the fixture's ONE-PDF run is ≈ 27 LLM calls ≈ 40 s: 2 model probes + 10 in-flight embed batches (the ~4 KB-per-page text packs into 20 content chunks at the app's code-default, pinned sizing — the token budget groups them into 10 requests of 2) + 4 image pipelines (one vision describe + 2 content-embed batches each — the image doc's context + description content packs into 3 chunks) + 1 PDF summary + 1 overview + 1 folder summary. The image span itself is 4 × (1.5 s describe + 2 × 1.5 s embed) = 18 s: each of the 4 positions holds for 4.5 s — that outlives the UI's 2 s poll (at least one tick lands in every 4.5 s window — every position is rendered, not a race) and the test's ~300 ms page sampling sees every rendered value. KB fixture (the ``test_local_directory_sources.py`` / phase-94 API registration pattern — a host temp dir, the app runs on the same host; no git anywhere): ``imgpass/`` with EXACTLY ONE file, a PyMuPDF-built ``imgpass.pdf`` (LOCKED A1) — 10 text pages (each ~4 KB of deterministic filler, well over the 100-char bar — the phase-140 pictorial gate never fires, and the layer is clean — the phase-133 fallback never fires: the EMBEDDED-IMAGE pass is the file's one and only vision span) + 4 GENUINELY DISTINCT non-blank images on pages 1–4 (different sizes + seeds → different stored bytes — the phase-151 dedupe keeps all four; colorful noise, sample mean ~127 — well under the 235 near-white floor, so the blank gate keeps all four: the span's total is the post-dedupe/post-gate count, the work actually about to happen). The expected chunk count is computed IN THE TEST by calling ``extract_pdf_text`` (the SAME parser the importer uses) and ``chunk_document`` (the SAME function + arguments — the importer's ``max(400, target)`` floor + the overlap setting — as ``_replace_content_chunks``), read back from the PDF on disk — the ground truth never drifts from the app under test. The sizing knobs are pinned to the code defaults in the app-under-test env (the leak-guard pattern — an operator's gitignored ``.env`` tuning ``BOR_CHUNK_TARGET_CHARS`` must not change the suite's ground truth). The suite runs ONE module app (it binds its own port — the conftest session app is never started in an isolated run) over the shared E2E Postgres: the SLOW app (``BOR_LLM_BASE_URL`` = the slow proxy in front of the mock), ``BOR_IMAGES=true`` + the ``pdf`` import token pinned EXPLICITLY (an operator's local gitignored ``.env`` cannot leak the scope in either direction) + ``BOR_PDF_PICTORIAL`` pinned off (the house E2E convention — this suite's span is the embedded pass; the fixture's ≥100-char pages make the pictorial pass a no-op even if the knob leaked on). Autouse cleanup (the phase-96 pattern): before each test wait for no running sync on the module app (the run state lives in its memory), then truncate the shared registry + KB tables (documents/chunks/ query_log/steering_notes/kb_overview/git_sources/sources_meta/ folder_summaries) — each test seeds its own source row and runs its own sync; after each test the same truncate. Test → observable mapping (Playwright Mapping Rule): 1. ``test_label_advances_through_the_image_span`` (the anti-freeze proof — the UI's own 2 s poll, densely sampled at ~300 ms): after clicking **Sync sources**, the test collects the DISTINCT ``#sync-progress-text`` values seen while the run is in flight (first-seen order) — and asserts: (a) at least THREE distinct labels containing ``· image `` are observed (the freeze the owner reported is a single repeated label — "≥3 distinct positions" IS the bug's negation), the positions START at 1, are NON-DECREASING in observation order, never EXCEED the total (4 — the post-gate count), and REACH it; (b) the CHUNK part appeared EARLIER in the run (the first ``· chunk n/20`` label precedes the first ``· image `` label — the two positions are distinct) and every image-span label carries the chunk part WITH IT (the last chunk position persists through the file's image span — "Syncing… imgpass/imgpass.pdf · chunk 19/20 · image 3/4" — no ``N remaining`` part, phase 193 hides it on this final file), its total == the file's TRUE chunk count; (c) the button ``title`` + ``aria-label`` carry the SAME full string as the box on every running sample (A4 — asserted per sample, atomically); (d) at settle the box shows the counts line (``5 added`` — the PDF + its 4 image docs), the label reads the terminal ``Synced HH:MM``, the count + chunk parts are hidden + empty, and the image part is GONE (no stale position in a terminal state — the endpoint's terminal carries 0/0 + the cleared chunk position while the file counts keep their final values). 2. ``test_mid_run_reload_re_attaches_with_image_position`` (the never-stale contract): the API starts the sync, the page's own poll renders the image span (the FIRST observed position is ``1/4`` — the whole span is ahead), the test reloads the page mid-run, and the load-time re-attach (``initSyncButton`` — a running sync re-enters with the WHOLE status, phase 98) renders the CURRENT image position with the A4 title/aria — then the run settles to the counts line. Phase 193 (task 02) moves the running file-count part to the distance to the end — ``N remaining``, ``N = files_total - files_done`` — and this fixture makes that edge VISIBLE rather than pinned away: its ONE file is also its FINAL file (``N == 0``), so the import span renders NO count part at all (the main part names the last file). The ``· chunk`` / ``· image`` positions and the terminal counts line are byte-identical. """ from __future__ import annotations import json import os import re import subprocess import sys import time from collections.abc import Iterator from pathlib import Path from typing import Any import httpx import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (LOCKED A1) import pytest from playwright.sync_api import Page, expect from sqlalchemy import text from app.config import Settings as _Settings from app.db import SessionLocal from app.rag.chunker import chunk_document from app.rag.llm import LLMClient from app.rag.pdf import extract_pdf_text, is_usable_pdf_text from e2e.auth_helpers import login from e2e.conftest import ( ADMIN_PASSWORD, MOCK_PORT, SESSION_SECRET, USE_REAL_LLM, _wait_http, ) from e2e.sync_helpers import counts_line REPO = Path(__file__).resolve().parents[2] # Phase 79 (task 04, full inventory): the conftest session app owns its # port in a combined run — this module's app binds its own port # instead (a same-port second uvicorn dies on bind and would drive the # wrong server). Env-overridable. APP_PORT = int(os.environ.get("E2E_APP_PORT_IMGPROG", "8171")) APP_URL = f"http://127.0.0.1:{APP_PORT}" #: The slow-LLM proxy's port (the conftest's mock LLM stays on #: MOCK_PORT — the proxy fronts it). SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_IMGPROG", "8909")) SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}" #: Per-LLM-request delay on the proxy — sized so each in-flight VISION #: PIPELINE's position (the describe + the content embed) outlives the #: UI's 2 s poll (see the module docstring's timing fixture paragraph). SLOW_DELAY_S = "1.5" # -------------------------------------------------------------------------- # Fixture constants (deterministic, token-controlled) # -------------------------------------------------------------------------- #: The local source — the temp directory's basename (``kind=local`` → #: the directory's basename is the source name, phase 38). SOURCE = "imgpass" #: The single fixture file (the source is one file, so phase 193 hides #: the count part for the whole import span — N = total - done = 0 on #: the only file there is). PDF_NAME = "imgpass.pdf" N_FILES = 1 #: The fixture PDF's shape — 10 text pages (each ~4 KB of filler — #: 20 content chunks at the code-default sizing, 10 embed batches) + #: 4 distinct non-blank images on pages 1–4 (the span's total — #: post-dedupe, post-blank-gate: the work actually about to happen). N_PAGES = 10 N_IMAGES = 4 #: The settled counts line (phase 150 — the box is the sole sync text #: surface): the PDF + its 4 embedded-image docs, a fresh KB (nothing #: updated/unchanged/pruned). SETTLE_TEXT = f"{1 + N_IMAGES} added" #: The three sizing knobs, pinned to the CODE DEFAULTS in both the #: app-under-test env (the leak guards below) and the test process's #: expected-value computation — an operator's gitignored ``.env`` #: (which may legitimately tune ``BOR_CHUNK_TARGET_CHARS``) cannot #: change the suite's ground truth. Derived from the class fields, so #: they never drift from app/config.py (the conftest leak-guard #: pattern). CHUNK_TARGET = _Settings.model_fields["chunk_target_chars"].default CHUNK_OVERLAP = _Settings.model_fields["chunk_overlap_chars"].default EMBED_BATCH_SIZE = _Settings.model_fields["embed_batch_size"].default #: The test process's mirror of the app-under-test sizing settings #: (explicit kwargs beat any ``.env`` for the fields the expected #: values depend on). _EXPECTED_SETTINGS = _Settings( chunk_target_chars=CHUNK_TARGET, chunk_overlap_chars=CHUNK_OVERLAP, embed_batch_size=EMBED_BATCH_SIZE, ) #: "Synced HH:MM" — the local-time last-result label (sources.js's #: fmtSyncTime), any hour/minute (the test_sync_button.py pattern). SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$") #: The settled boot label — idle ("Sync sources") or the RE-ATTACHED #: terminal from the previous test's sync (the module app is shared, #: its in-memory sync state survives the test boundary, and the #: phase-32 boot re-attach renders the last result). IDLE_OR_SETTLED = re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$") #: The chunk part (the phase-129 unit-pinned shape): "· chunk n/m" — #: ``m`` is asserted against the expected chunk count by the test. CHUNK_PART_RE = re.compile(r"· chunk (\d+)/(\d+)") #: The image part (the phase-152 unit-pinned shape): "· image n/m". IMAGE_PART_RE = re.compile(r"· image (\d+)/(\d+)") #: Generous settle budgets: a changed sync against the slowed LLM is #: ≈ 40 s; the UI's 2 s poll settles at most one tick after the #: terminal state lands. SETTLE_TIMEOUT_MS = 120_000 # --------------------------------------------------------------------------- # The fixture PDF (built with PyMuPDF — LOCKED A1) # --------------------------------------------------------------------------- def _noise_png(w: int, h: int, seed: int) -> bytes: """A deterministic colorful-noise PNG (the "real image" stand-in). Different ``(w, h, seed)`` → different stored bytes (the phase-151 byte-dedupe keeps all four); the formula's sample mean is ~127 — well under the phase-151 blank gate's 235 near-white floor, and the samples scatter far outside the ±17 band — every image survives the gate with room to spare (see the module docstring's KB fixture paragraph).""" pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, w, h)) for y in range(h): for x in range(w): pix.set_pixel( x, y, ((x * 7 + y * 11 + seed * 13) % 256, (x * 3 + y * 17 + seed * 29) % 256, (x * 5 + y * 23 + seed * 41) % 256), ) return pix.tobytes("png") #: The four fixture images' shapes (distinct sizes — the distinct- #: bytes guarantee even if two seeds collided) — one per page 1–4. _IMAGE_SHAPES = [(24, 18), (32, 24), (40, 20), (28, 28)] def _page_text(k: int) -> str: """One page's deterministic ~4 KB filler (no randomness — the expected chunk count is stable run to run).""" base = ( f"Page {k} notes for the imgpass fixture document carry " f"deterministic filler number {k} of the ten-page pdf, written " f"out to size so the chunker has real content to work with " f"across the page. " ) return base * 11 def _build_pdf(pdf_path: Path) -> None: """The fixture PDF: ``N_PAGES`` text pages (``insert_textbox`` — the return value is checked — an overfull rect would silently drop text and skew the ground truth) + the 4 images on pages 1–4 (distinct ``insert_image`` streams — distinct xrefs, distinct bytes). The metadata title is the stem (no title — the stem fallback, phase 126 A2).""" doc = fitz.open() for k in range(1, N_PAGES + 1): page = doc.new_page() rc = page.insert_textbox( fitz.Rect(48, 90, 548, 812), _page_text(k), fontsize=10 ) assert rc >= 0, f"the page {k} text did not fit the rect (rc={rc})" if k <= N_IMAGES: w, h = _IMAGE_SHAPES[k - 1] page.insert_image( fitz.Rect(72, 820 - h * 2, 72 + w * 2, 860), stream=_noise_png(w, h, k), ) doc.save(pdf_path) doc.close() def _expected_chunks(pdf_path: Path) -> list[str]: """The PDF's expected content chunks — the SAME function and arguments the importer's ``_replace_content_chunks`` uses (``chunk_document(content, rel, max(400, target), overlap)`` — the importer's token-cap floor rule; ``rel`` = the source-relative path) over the SAME parser the importer reads (``extract_pdf_text`` — the ``---PAGE-N---`` page markers): the endpoint's ``chunks_total`` must equal ``len(chunk_document(...))`` — the file's TRUE chunk count, not a batch count.""" content = extract_pdf_text(pdf_path) return chunk_document( content, PDF_NAME, max(400, CHUNK_TARGET), CHUNK_OVERLAP ) def _expected_batch_count(chunks: list[str]) -> int: """The expected embed-batch count — the SAME ``LLMClient._batch_texts`` partition the app's embed loop walks (the phase-129 unit pattern — call it directly for the boundaries). The task's sizing bar: ≥ 2 batches (the chunk part's climb is a real sequence, not a single point).""" return len(LLMClient(_EXPECTED_SETTINGS)._batch_texts(chunks)) # --------------------------------------------------------------------------- # The in-browser snapshot functions (Playwright Mapping Rule — the # rendered DOM is the observable). # # ``_JS_SAMPLE`` — test 1's dense sampler: the box text + the button # title/aria read in the SAME in-browser evaluation (a 2 s poll tick # updates the parts + the title together in one JS task, so the atomic # check can never straddle one — the phase-128 pattern); ``running`` # marks the in-flight labels (the A4 title/aria contract applies to # them). # # ``_JS_IMAGE_SNAP`` — test 2's wait: null until the box carries the # ``· image n/`` part, then the texts (read atomically with the # title/aria). # --------------------------------------------------------------------------- _JS_SAMPLE = """ () => { const boxTextEl = document.querySelector('#sync-progress-text'); const btn = document.querySelector('#sync-btn'); if (!boxTextEl || !btn) return null; const box = boxTextEl.textContent; return { box: box, title: btn.getAttribute('title') || '', aria: btn.getAttribute('aria-label') || '', running: box.startsWith('Syncing…'), }; } """ _JS_IMAGE_SNAP = """ (n) => { const boxTextEl = document.querySelector('#sync-progress-text'); const btn = document.querySelector('#sync-btn'); if (!boxTextEl || !btn) return null; const box = boxTextEl.textContent; const m = box.match(/^Syncing… .*· image (\\d+)\\/(\\d+)$/); if (!m || Number(m[2]) !== n) return null; return { box: box, posN: Number(m[1]), title: btn.getAttribute('title') || '', aria: btn.getAttribute('aria-label') || '', }; } """ # --------------------------------------------------------------------------- # Fixtures # --------------------------------------------------------------------------- @pytest.fixture(scope="module") def imgpass_dir(tmp_path_factory: pytest.TempPathFactory) -> Path: """The one-source temp tree (see the module docstring): the app server runs on the same host, so the paths are visible to it. The directory NAME is the source name (``kind=local``, phase 38). EXACTLY ONE file: the PyMuPDF-built ``imgpass.pdf`` (10 text pages + 4 distinct non-blank images — see :func:`_build_pdf`).""" root = tmp_path_factory.mktemp("bor_imgprog") src = root / SOURCE src.mkdir() _build_pdf(src / PDF_NAME) assert len(list(src.rglob("*"))) == N_FILES return src def _app_env(base_url: str, tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]: """The per-module app env (the conftest pattern, cf. ``test_oneshot_llm_retry.py`` / ``test_kb_tree.py``): the ``base_url`` picks the LLM (the slow proxy in front of the mock), ``BOR_GIT_SOURCES`` forced empty (the sync source is this suite's own DB-registered local row), the sizing knobs pinned to the code defaults (the leak guards — see the module constants), the phase-126 knobs pinned EXPLICITLY (``BOR_IMPORT_EXTENSIONS`` with the ``pdf`` token — without it the walk is blind to the fixture — and ``BOR_IMAGES=true`` — the embedded pass is the span under test), the E2E ``BOR_PDF_PICTORIAL`` off-pin (the house convention — the suite's span is the embedded pass; the fixture's ≥100-char pages keep the pictorial pass a no-op even if the knob leaked on), and the other leak-guarded code defaults.""" env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = ( "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url ) # Mock-calibrated threshold (conftest pattern): this suite never # asks the chat model anything — the gate is never on a path. env["BOR_RELEVANCE_THRESHOLD"] = "0.30" # Phase 67: instant retry waits + the code-default budget (the # conftest leak-guard pattern). env["BOR_LLM_RETRY_DELAY"] = "0" env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default) env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) # Phase 16: admin auth must be set or create_app() refuses to boot. env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET # The repo's .env file carries the owner's BOR_GIT_SOURCES (the app # reads it from cwd) — override it with an EMPTY value (the env var # beats the .env file): the registry must hold EXACTLY the local # directory this suite registers. env["BOR_GIT_SOURCES"] = "" # Leak guards (conftest pattern): an operator's local (gitignored) # .env cannot leak corpus-specific settings into the app under # test — including the chunking / batching knobs this suite's # ground truth is computed from (see the module constants). env["BOR_DOCS_REPO"] = "" env["BOR_SUGGESTIONS"] = json.dumps( _Settings.model_fields["suggestions"].default ) env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET) env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP) env["BOR_EMBED_BATCH_SIZE"] = str(EMBED_BATCH_SIZE) # Phase 126/122: the import scope WITH the ``pdf`` token + the # embedded-image toggle ON (the span under test) + the E2E # pictorial off-pin + the suite-private image home (the app under # test must not write copies into the owner's real ~/bor-sources). env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf" env["BOR_IMAGES"] = "true" env["BOR_PDF_PICTORIAL"] = "0" env["BOR_IMAGE_DIR"] = str(tmp_path_factory.mktemp("bor_imgprog_images")) env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts")) return env @pytest.fixture(scope="module") def slow_llm(mock_llm: int) -> Iterator[int]: """The delay-injecting reverse proxy in front of the mock LLM (``tests/e2e/slow_llm.py``) — the timing fixture for the whole suite: it delays EVERY LLM call, vision describes included, so each in-flight image pipeline's position outlives the UI's 2 s poll (see ``SLOW_DELAY_S`` + the module docstring's timing fixture paragraph).""" env = dict(os.environ) env.pop("DEBUGPY", None) env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S env["E2E_MOCK_PORT"] = str(MOCK_PORT) proc = subprocess.Popen( [sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app", "--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"], cwd=REPO, env=env, ) try: _wait_http(f"{SLOW_URL}/v1/models") yield SLOW_PORT finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_server( mock_llm: int, slow_llm: int, imgpass_dir: Path, tmp_path_factory: pytest.TempPathFactory, ) -> Iterator[str]: """The SLOW app under test (the whole suite): ``BOR_LLM_BASE_URL`` is the slow proxy in front of the mock (the timing fixture — see the module docstring). ``imgpass_dir`` is a dependency only for the fixture ordering (the temp tree exists before the first test). Skips the module (clear instructions) when Postgres is down — the sync needs the KB tables.""" env = _app_env(f"{SLOW_URL}/v1", tmp_path_factory) proc = subprocess.Popen( [sys.executable, "-m", "uvicorn", "app.main:app", "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], cwd=REPO, env=env, ) try: _wait_http(f"{APP_URL}/api/health") body = httpx.get(f"{APP_URL}/api/health", timeout=5).json() if body["db"] != "up": pytest.skip("Postgres not reachable — run `podman compose up -d db` first") yield APP_URL finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server def _truncate_all() -> None: """Fresh registry + KB (the E2E isolation pattern): the E2E suites share one Postgres, so a leftover git_sources row or document would pollute the file counts + the counts line the syncs and the assertions pin exactly. ``sources_meta`` (the KB generation counter) resets with the KB — the sync re-creates the row.""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, steering_notes, " "kb_overview, git_sources, sources_meta, folder_summaries" ) ) db.commit() # without the commit the TRUNCATE rolls back (the house pattern) def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None: """No background sync may leak across tests (the run state lives in the app's memory, and a still-running run would keep importing into the NEXT test's truncated KB): wait for the app's sync status to be non-running BEFORE the truncate. Own admin session (the endpoint is admin-only) — usually a no-op: the tests settle only after their run's terminal state.""" with httpx.Client(base_url=base_url, timeout=5.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: r = client.get("/api/sync/status") assert r.status_code == 200, r.text if r.json()["state"] != "running": return time.sleep(0.2) raise AssertionError(f"a sync was still running at a test boundary: {base_url}") @pytest.fixture(autouse=True) def _clean(app_url: str) -> Iterator[None]: """Per-test KB isolation (the phase-96 cleanup pattern): the shared E2E Postgres is truncated before AND after every test — each test seeds its own source row and runs its own sync, so the KB rows + folder_summaries rows of the temp source are exactly this test's own. The pre-wait keeps a leaked run from the previous test (the module app is shared) from racing the truncate.""" _wait_sync_settled(app_url) _truncate_all() yield _truncate_all() # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- def _register_source(base_url: str, path: str) -> None: """Register the temp directory through the authenticated API (the ``test_local_directory_sources.py`` / phase-94 pattern — a plain directory is an API/DB-only operation).""" with httpx.Client(base_url=base_url, timeout=30.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text r = client.post( "/api/git-sources", json={"kind": "local", "path": path} ) assert r.status_code == 201, r.text def _wait_terminal(app_url: str, timeout_s: float = 180.0) -> dict[str, Any]: """Poll the status endpoint (own admin session) until the UI-/API-triggered run reaches a terminal state — the source of truth for the settle (the UI's 2 s poll renders it at most one tick later). A FAILED terminal fails the test with the terminal body (the ``detail`` / error text) — never an opaque label timeout.""" with httpx.Client(base_url=app_url, timeout=30.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text deadline = time.monotonic() + timeout_s body: dict[str, Any] = {} while time.monotonic() < deadline: r = client.get("/api/sync/status") assert r.status_code == 200, r.text body = r.json() if body["state"] in ("success", "failed"): assert body["state"] == "success", body return body time.sleep(0.2) raise AssertionError(f"the run never reached a terminal state: {body}") def _start_sync_click(page: Page, app_url: str, sync_dir: Path) -> None: """Register the source, land the admin on the sources view, wait for the settled boot label (the phase-64 pattern — also proves the phase-32 boot re-attach completed, so no later re-render can clobber the running state), then click **Sync sources** and pin the click's immediate 202 state (the bare "Syncing…" — the count AND the chunk parts hidden: no file yet, no batch in flight — and the image part is absent too: no PDF's vision span has started, ``images_total == 0``).""" _register_source(app_url, str(sync_dir)) login(page, app_url) # lands on /sources.html (the button's home) btn = page.locator("#sync-btn") expect(btn).to_be_visible(timeout=30_000) expect(page.locator("#sync-label")).to_have_text(IDLE_OR_SETTLED) expect(btn).to_be_enabled() btn.click() # The click's immediate state (the 202 moment — the bare prefix, # the phase-64 contract): disabled, aria-busy, spinning icon, and # the label's MAIN part alone ("Syncing…" — the 202 click carries # no status, so the pinned parts hide, phase 128/129 writers). # Phase 149 (task 01): the running detail rides the box — the 202's # bare label is in #sync-progress (the button is the spinner). expect(btn).to_be_disabled() expect(btn).to_have_attribute("aria-busy", "true") expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b")) expect(page.locator("#sync-progress")).to_have_text("Syncing…") expect(page.locator(".sync-label-count")).to_be_hidden() expect(page.locator(".sync-label-chunk")).to_be_hidden() expect(page.locator("#sync-error-banner")).to_be_hidden() # --------------------------------------------------------------------------- # 1. The anti-freeze proof — the label advances through the PDF's # image span (≥3 distinct positions, monotonic, chunk part earlier, # the A4 title/aria, the settled clearing) # --------------------------------------------------------------------------- def test_label_advances_through_the_image_span( page: Page, app_url: str, imgpass_dir: Path ) -> None: """A fresh admin page on ``/sources.html`` clicks **Sync sources**; the test then densely samples (≈ every 300 ms) the rendered ``#sync-progress-text`` — NOT the API (the rendered DOM is the observable; the UI's own 2 s poll is the writer) — collecting the DISTINCT values in first-seen order until the settled counts line, and asserts the phase-152 contract end to end (see the module docstring's test mapping): the image part climbs (≥3 distinct positions, starting at 1, non-decreasing, never above the total, reaching it), the chunk part appeared earlier (and persists through the image span with the file's TRUE chunk total), the A4 title/aria ride the full string on every running sample, and the settle clears the image position (the counts line, the terminal label, the cleared endpoint position — no stale position in a terminal state).""" page.set_default_timeout(30_000) # The ground truth — computed from the PDF ON DISK with the SAME # parser + function + arguments the importer uses (see the helper # docstrings): the TRUE chunk count + the batch count (the task's # sizing bar — ≥2 embed batches, so the chunk part's climb is a # real sequence). pdf_path = imgpass_dir / PDF_NAME chunks = _expected_chunks(pdf_path) n_chunks = len(chunks) assert n_chunks >= EMBED_BATCH_SIZE + 1, n_chunks # 2+ embed batches assert _expected_batch_count(chunks) >= 2 # the sizing bar # The happy path's premise: a clean text layer (the phase-133 # fallback never fires) — the embedded-image pass is the file's # one and only vision span (the ≥100-char pages keep the phase-140 # gate candidate-free even under the app's pinned-off knob). assert is_usable_pdf_text(extract_pdf_text(pdf_path)) _start_sync_click(page, app_url, imgpass_dir) # The observation loop (the task's "poll the page, not the API"): # dense samples of the rendered box text (≈ 300 ms — the evaluate # round-trip + the sleep) in first-seen order, each sample read # atomically with the button's title/aria (A4 checked per running # sample), until the settled counts line lands (or the generous # deadline — a failed run then fails LOUDLY in _wait_terminal # with the terminal body, never an opaque timeout). seen: list[str] = [] last: str | None = None deadline = time.monotonic() + 150 while time.monotonic() < deadline: snap = page.evaluate(_JS_SAMPLE) assert snap is not None if snap["running"]: # A4 — while a run is in flight, the title + the # aria-label carry the SAME untruncated full string as the # box (the "· image" part included). assert snap["title"] == snap["box"], snap assert snap["aria"] == snap["box"], snap if snap["box"] != last: seen.append(snap["box"]) last = snap["box"] if counts_line(SETTLE_TEXT).fullmatch(snap["box"]): break time.sleep(0.2) # --- the settle (the UI's 2 s poll renders the terminal at most one # tick after it lands; the endpoint is the source of truth) --- assert counts_line(SETTLE_TEXT).fullmatch(str(last)), ( f"the run never settled to the counts line: {seen[-5:]}" ) terminal = _wait_terminal(app_url) # The counts line's ground truth: the PDF + its 4 embedded-image # docs — one file, 5 added. assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal # The keep-final-file-counts split (the phase-64/129 convention, # restated for the phase-152 sub-position): the FILE counts keep # their final values, the sub-file chunk + image positions clear # with the file (0/0 in the terminal — no stale position), the # phase clears. assert terminal["files_done"] == N_FILES and terminal["files_total"] == N_FILES, terminal assert terminal["current_file"] is None, terminal assert terminal["current_chunk"] == 0 and terminal["chunks_total"] == 0, terminal assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal assert terminal["phase"] is None, terminal # --- the run was real (the prelude + the import span were # sampled) --- assert any(t == "Syncing…" for t in seen), f"no bare prelude label: {seen[:6]}" import_labels = [t for t in seen if t.startswith("Syncing… ")] assert any(f"Syncing… {SOURCE}/{PDF_NAME}" in t for t in import_labels), ( f"no live-file label: {import_labels[:6]}" ) # Phase 193 (A1): the count part counts DOWN and hides at the FINAL # file — this fixture's one file IS its final file (N = 0), so no # running label carries it (the pre-phase "(1/1)" is gone). assert all(" remaining" not in t for t in import_labels), ( f"the final file must render no count part: {import_labels[:6]}" ) # --- the chunk part appeared EARLIER in the run (the two # positions are distinct) --- chunk_idx = next( (i for i, t in enumerate(seen) if CHUNK_PART_RE.search(t)), None ) assert chunk_idx is not None, f"no chunk part observed: {seen}" first_chunk = CHUNK_PART_RE.search(seen[chunk_idx]) assert first_chunk is not None, seen[chunk_idx] # The chunk part's total is the file's TRUE chunk count (not a # batch count — the phase-129 A6 assumption, the E2E lens). assert int(first_chunk.group(2)) == n_chunks, (first_chunk.group(0), n_chunks) # --- the image part: ≥3 distinct positions (the anti-freeze # proof — the owner's freeze is a single repeated label), the # positions START at 1, are NON-DECREASING in observation order, # never EXCEED the total (the post-dedupe/post-gate count — the # work actually about to happen), and REACH it --- def _image_part(t: str) -> re.Match[str]: m = IMAGE_PART_RE.search(t) assert m is not None, t # the list comprehension's filter guarantees it return m image_items = [ (i, _image_part(t)) for i, t in enumerate(seen) if IMAGE_PART_RE.search(t) ] assert len(image_items) >= 3, ( f"fewer than 3 distinct image positions observed: " f"{[m.group(0) for _, m in image_items]}" ) first_image_idx = image_items[0][0] assert chunk_idx < first_image_idx, (chunk_idx, first_image_idx) positions = [(int(m.group(1)), int(m.group(2))) for _, m in image_items] assert positions[0][0] == 1, f"the climb does not start at 1: {positions}" assert all(m == N_IMAGES for _, m in positions), positions assert all(n <= N_IMAGES for n, _ in positions), positions ns = [n for n, _ in positions] assert ns == sorted(ns), f"the climb is backwards somewhere: {positions}" assert N_IMAGES in ns, f"the climb never reached the total: {positions}" # The two positions COEXIST (distinct and both may show — # "Syncing… imgpass/imgpass.pdf · chunk 19/20 · image 3/4", with no # `N remaining` part on this single-file run — phase 193): every image-span label carries the # chunk part with it (the last chunk position persists through the # file's image span — the per-file reset + the phase transitions # clear it, not the end of the chunk span). for i, _ in image_items: cm = CHUNK_PART_RE.search(seen[i]) assert cm is not None, f"an image-span label lost the chunk part: {seen[i]}" assert int(cm.group(2)) == n_chunks, (cm.group(0), n_chunks) # --- the settle: the counts line, the terminal label, the parts # cleared + empty, and the IMAGE part GONE (no stale position in a # terminal state — the endpoint pin above is the server-side # half) --- expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000) btn = page.locator("#sync-btn") expect(btn).to_be_enabled() expect(btn).not_to_have_attribute("aria-busy") expect(page.locator(".sync-label-count")).to_be_hidden() expect(page.locator(".sync-label-count")).to_have_text("") expect(page.locator(".sync-label-chunk")).to_be_hidden() expect(page.locator(".sync-label-chunk")).to_have_text("") # The counts line rides the box (phase 150 — the sole sync text # surface): the image part is not in it, not in the title, not in # the aria (the settle restores the ship aria value). box = page.locator("#sync-progress-text").inner_text() assert counts_line(SETTLE_TEXT).fullmatch(box), box assert "· image" not in box and "· image" not in (btn.get_attribute("title") or "") assert btn.get_attribute("aria-label") == "Sync sources", btn.get_attribute("aria-label") expect(page.locator("#sync-error-banner")).to_be_hidden() # --------------------------------------------------------------------------- # 2. The never-stale contract — a mid-run page reload re-attaches with # the current image position visible # --------------------------------------------------------------------------- def test_mid_run_reload_re_attaches_with_image_position( page: Page, app_url: str, imgpass_dir: Path ) -> None: """The admin clicks **Sync sources** (the run is started the way the owner starts it — the click is what starts the page's 2 s poll; the reload is the contract under test), the page's own 2 s poll renders the image span (the FIRST observed position is ``1/4`` — the whole span is ahead, so the reload + the load-time re-attach happen with the span in flight), the test reloads the page mid-run, and the load-time re-attach (``initSyncButton`` — a running sync re-enters with the WHOLE status, phase 98's never-stale contract) renders the CURRENT ``· image n/4`` with the A4 title/aria; the run then settles to the counts line (the image part gone — no stale position).""" page.set_default_timeout(30_000) _start_sync_click(page, app_url, imgpass_dir) # The image span, rendered by the page's own 2 s poll (the span # starts after the prelude + the file's embed batches — ≈ 21 s # under the slow proxy; each position holds 4.5 s — the describe # + the image doc's 2 embed batches). snap = page.wait_for_function( _JS_IMAGE_SNAP, arg=N_IMAGES, timeout=60_000 ).json_value() # The FIRST observed position is 1 — the whole span is ahead (the # reload's re-attach is guaranteed to catch the span in flight: # ≈ 18 s remain, the local reload + the load-time status fetch # land well inside it). assert snap["posN"] == 1, snap # A4 — the pre-reload running state is untruncated on the title + # the aria (the "· image" part included). assert snap["title"] == snap["box"] and "· image" in snap["title"], snap assert snap["aria"] == snap["box"], snap # The mid-run reload (the session cookie survives — the re-attach # is the admin's, phase 98). page.reload() # The load-time re-attach: a running sync re-enters with the WHOLE # status — the CURRENT image position is visible (the never-stale # contract), the A4 title/aria ride it. snap2 = page.wait_for_function( _JS_IMAGE_SNAP, arg=N_IMAGES, timeout=30_000 ).json_value() assert snap2["title"] == snap2["box"] and "· image" in snap2["title"], snap2 assert snap2["aria"] == snap2["box"], snap2 # Then the settle: the counts line (the image part gone — the # terminal clears the position, test 1 pins the endpoint half), # the terminal label, the parts cleared + empty. expect(page.locator("#sync-progress-text")).to_have_text( counts_line(SETTLE_TEXT), timeout=SETTLE_TIMEOUT_MS ) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000) btn = page.locator("#sync-btn") expect(btn).to_be_enabled() expect(btn).not_to_have_attribute("aria-busy") expect(page.locator(".sync-label-count")).to_be_hidden() expect(page.locator(".sync-label-chunk")).to_be_hidden() assert "· image" not in (btn.get_attribute("title") or "") expect(page.locator("#sync-error-banner")).to_be_hidden() terminal = _wait_terminal(app_url) assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal