brain-of-reese/tests/e2e/test_sync_image_progress.py

"""Phase 152 task 05 E2E (Playwright, mock-only): during a PDF sync whose
images take time to describe, the live label shows the image position
MOVING — the owner's "stuck for several minutes" (TODO.md:3: the label
sat on ``Syncing… … (2540/2540) · chunk 125/126`` through the PDF's
vision spans) is impossible on the fixture.

The dedicated story suite for ``152_sync_progress_image_pass`` (A16 —
one Playwright file per phase, run in isolation): the ``on_image_progress
hook (phase 152, tasks 02/03) fires immediately before each vision
describe call with ``(position, total)``, the runner's ``_image_hook``
closure lands it on ``SyncStatus.images_done/images_total``, and
``fmtSyncPhaseLabel`` renders ``· image n/m`` — the phase-149 box
(``#sync-progress``) is the surface that shows it, the button
``title``/``aria-label`` carry the UNTRUNCATED composed string (A4).

Run in isolation (DB must be up: ``podman compose up -d db``):

    uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov

MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic mock plus the phase-64 ``tests/e2e/slow_llm.py`` delay
proxy in front of it (``SLOW_DELAY_S = 1.5 s``) that stretches EVERY
LLM call (vision describes included) so the fixture's vision span
outlives the UI's 2 s status poll and each position outlives the
recorder's ~300 ms page-sampling cadence by ~10× — the observed
positions are dense samples of a real climb, not races.

Timing fixture (the phase-129/98 rationale, re-sized for vision spans):
the fixture's ONE-PDF run is ≈ 27 LLM calls ≈ 40 s: 2 model probes +
10 in-flight embed batches (the ~4 KB-per-page text packs into 20
content chunks at the app's code-default, pinned sizing — the token
budget groups them into 10 requests of 2) + 4 image pipelines (one
vision describe + 2 content-embed batches each — the image doc's
context + description content packs into 3 chunks) + 1 PDF summary +
1 overview + 1 folder summary. The image span itself is 4 × (1.5 s
describe + 2 × 1.5 s embed) = 18 s: each of the 4 positions holds
for 4.5 s — that outlives the UI's 2 s poll (at least one tick lands
in every 4.5 s window — every position is rendered, not a race) and
the test's ~300 ms page sampling sees every rendered value.

KB fixture (the ``test_local_directory_sources.py`` / phase-94 API
registration pattern — a host temp dir, the app runs on the same host;
no git anywhere): ``imgpass/`` with EXACTLY ONE file, a PyMuPDF-built
``imgpass.pdf`` (LOCKED A1) — 10 text pages (each ~4 KB of
deterministic filler, well over the 100-char bar — the phase-140
pictorial gate never fires, and the layer is clean — the phase-133
fallback never fires: the EMBEDDED-IMAGE pass is the file's one and
only vision span) + 4 GENUINELY DISTINCT non-blank images on pages
1–4 (different sizes + seeds → different stored bytes — the phase-151
dedupe keeps all four; colorful noise, sample mean ~127 — well under
the 235 near-white floor, so the blank gate keeps all four: the
span's total is the post-dedupe/post-gate count, the work actually
about to happen). The expected chunk count is computed IN THE TEST by
calling ``extract_pdf_text`` (the SAME parser the importer uses) and
``chunk_document`` (the SAME function + arguments — the importer's
``max(400, target)`` floor + the overlap setting — as
``_replace_content_chunks``), read back from the PDF on disk — the
ground truth never drifts from the app under test. The sizing knobs
are pinned to the code defaults in the app-under-test env (the
leak-guard pattern — an operator's gitignored ``.env`` tuning
``BOR_CHUNK_TARGET_CHARS`` must not change the suite's ground truth).

The suite runs ONE module app (it binds its own port — the conftest
session app is never started in an isolated run) over the shared E2E
Postgres: the SLOW app (``BOR_LLM_BASE_URL`` = the slow proxy in front
of the mock), ``BOR_IMAGES=true`` + the ``pdf`` import token pinned
EXPLICITLY (an operator's local gitignored ``.env`` cannot leak the
scope in either direction) + ``BOR_PDF_PICTORIAL`` pinned off (the
house E2E convention — this suite's span is the embedded pass; the
fixture's ≥100-char pages make the pictorial pass a no-op even if the
knob leaked on).

Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on the module app (the run state lives in its memory),
then truncate the shared registry + KB tables (documents/chunks/
query_log/steering_notes/kb_overview/git_sources/sources_meta/
folder_summaries) — each test seeds its own source row and runs its
own sync; after each test the same truncate.

Test → observable mapping (Playwright Mapping Rule):
1. ``test_label_advances_through_the_image_span`` (the anti-freeze
   proof — the UI's own 2 s poll, densely sampled at ~300 ms): after
   clicking **Sync sources**, the test collects the DISTINCT
   ``#sync-progress-text`` values seen while the run is in flight
   (first-seen order) — and asserts: (a) at least THREE distinct
   labels containing ``· image `` are observed (the freeze the owner
   reported is a single repeated label — "≥3 distinct positions" IS
   the bug's negation), the positions START at 1, are
   NON-DECREASING in observation order, never EXCEED the total (4 —
   the post-gate count), and REACH it; (b) the CHUNK part appeared
   EARLIER in the run (the first ``· chunk n/20`` label precedes the
   first ``· image `` label — the two positions are distinct) and
   every image-span label carries the chunk part WITH IT (the last
   chunk position persists through the file's image span —
   "… (1/1) · chunk 19/20 · image 3/4"), its total == the file's
   TRUE chunk count; (c) the button ``title`` + ``aria-label`` carry
   the SAME full string as the box on every running sample (A4 —
   asserted per sample, atomically); (d) at settle the box shows the
   counts line (``5 added`` — the PDF + its 4 image docs), the label
   reads the terminal ``Synced HH:MM``, the count + chunk parts are
   hidden + empty, and the image part is GONE (no stale position in a
   terminal state — the endpoint's terminal carries 0/0 + the cleared
   chunk position while the file counts keep their final values).
2. ``test_mid_run_reload_re_attaches_with_image_position`` (the
   never-stale contract): the API starts the sync, the page's own
   poll renders the image span (the FIRST observed position is
   ``1/4`` — the whole span is ahead), the test reloads the page
   mid-run, and the load-time re-attach (``initSyncButton`` — a
   running sync re-enters with the WHOLE status, phase 98) renders
   the CURRENT image position with the A4 title/aria — then the run
   settles to the counts line.
"""
from __future__ import annotations

import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any

import httpx
import pymupdf as fitz  # PyMuPDF — the fixture-PDF builder (LOCKED A1)
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text

from app.config import Settings as _Settings
from app.db import SessionLocal
from app.rag.chunker import chunk_document
from app.rag.llm import LLMClient
from app.rag.pdf import extract_pdf_text, is_usable_pdf_text
from e2e.auth_helpers import login
from e2e.conftest import (
    ADMIN_PASSWORD,
    MOCK_PORT,
    SESSION_SECRET,
    USE_REAL_LLM,
    _wait_http,
)

REPO = Path(__file__).resolve().parents[2]

# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module's app binds its own port
# instead (a same-port second uvicorn dies on bind and would drive the
# wrong server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_IMGPROG", "8171"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"

#: The slow-LLM proxy's port (the conftest's mock LLM stays on
#: MOCK_PORT — the proxy fronts it).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_IMGPROG", "8909"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"

#: Per-LLM-request delay on the proxy — sized so each in-flight VISION
#: PIPELINE's position (the describe + the content embed) outlives the
#: UI's 2 s poll (see the module docstring's timing fixture paragraph).
SLOW_DELAY_S = "1.5"

# --------------------------------------------------------------------------
# Fixture constants (deterministic, token-controlled)
# --------------------------------------------------------------------------

#: The local source — the temp directory's basename (``kind=local`` →
#: the directory's basename is the source name, phase 38).
SOURCE = "imgpass"
#: The single fixture file (the source is one file — the count part
#: reads "(1/1)" throughout the import phase).
PDF_NAME = "imgpass.pdf"
N_FILES = 1

#: The fixture PDF's shape — 10 text pages (each ~4 KB of filler —
#: 20 content chunks at the code-default sizing, 10 embed batches) +
#: 4 distinct non-blank images on pages 1–4 (the span's total —
#: post-dedupe, post-blank-gate: the work actually about to happen).
N_PAGES = 10
N_IMAGES = 4

#: The settled counts line (phase 150 — the box is the sole sync text
#: surface): the PDF + its 4 embedded-image docs, a fresh KB (nothing
#: updated/unchanged/pruned).
SETTLE_TEXT = f"{1 + N_IMAGES} added"

#: The three sizing knobs, pinned to the CODE DEFAULTS in both the
#: app-under-test env (the leak guards below) and the test process's
#: expected-value computation — an operator's gitignored ``.env``
#: (which may legitimately tune ``BOR_CHUNK_TARGET_CHARS``) cannot
#: change the suite's ground truth. Derived from the class fields, so
#: they never drift from app/config.py (the conftest leak-guard
#: pattern).
CHUNK_TARGET = _Settings.model_fields["chunk_target_chars"].default
CHUNK_OVERLAP = _Settings.model_fields["chunk_overlap_chars"].default
EMBED_BATCH_SIZE = _Settings.model_fields["embed_batch_size"].default

#: The test process's mirror of the app-under-test sizing settings
#: (explicit kwargs beat any ``.env`` for the fields the expected
#: values depend on).
_EXPECTED_SETTINGS = _Settings(
    chunk_target_chars=CHUNK_TARGET,
    chunk_overlap_chars=CHUNK_OVERLAP,
    embed_batch_size=EMBED_BATCH_SIZE,
)

#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: The settled boot label — idle ("Sync sources") or the RE-ATTACHED
#: terminal from the previous test's sync (the module app is shared,
#: its in-memory sync state survives the test boundary, and the
#: phase-32 boot re-attach renders the last result).
IDLE_OR_SETTLED = re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$")
#: The chunk part (the phase-129 unit-pinned shape): "· chunk n/m" —
#: ``m`` is asserted against the expected chunk count by the test.
CHUNK_PART_RE = re.compile(r"· chunk (\d+)/(\d+)")
#: The image part (the phase-152 unit-pinned shape): "· image n/m".
IMAGE_PART_RE = re.compile(r"· image (\d+)/(\d+)")

#: Generous settle budgets: a changed sync against the slowed LLM is
#: ≈ 40 s; the UI's 2 s poll settles at most one tick after the
#: terminal state lands.
SETTLE_TIMEOUT_MS = 120_000


# ---------------------------------------------------------------------------
# The fixture PDF (built with PyMuPDF — LOCKED A1)
# ---------------------------------------------------------------------------


def _noise_png(w: int, h: int, seed: int) -> bytes:
    """A deterministic colorful-noise PNG (the "real image" stand-in).
    Different ``(w, h, seed)`` → different stored bytes (the phase-151
    byte-dedupe keeps all four); the formula's sample mean is ~127 —
    well under the phase-151 blank gate's 235 near-white floor, and
    the samples scatter far outside the ±17 band — every image
    survives the gate with room to spare (see the module docstring's
    KB fixture paragraph)."""
    pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, w, h))
    for y in range(h):
        for x in range(w):
            pix.set_pixel(
                x,
                y,
                ((x * 7 + y * 11 + seed * 13) % 256,
                 (x * 3 + y * 17 + seed * 29) % 256,
                 (x * 5 + y * 23 + seed * 41) % 256),
            )
    return pix.tobytes("png")


#: The four fixture images' shapes (distinct sizes — the distinct-
#: bytes guarantee even if two seeds collided) — one per page 1–4.
_IMAGE_SHAPES = [(24, 18), (32, 24), (40, 20), (28, 28)]


def _page_text(k: int) -> str:
    """One page's deterministic ~4 KB filler (no randomness — the
    expected chunk count is stable run to run)."""
    base = (
        f"Page {k} notes for the imgpass fixture document carry "
        f"deterministic filler number {k} of the ten-page pdf, written "
        f"out to size so the chunker has real content to work with "
        f"across the page. "
    )
    return base * 11


def _build_pdf(pdf_path: Path) -> None:
    """The fixture PDF: ``N_PAGES`` text pages (``insert_textbox`` —
    the return value is checked — an overfull rect would silently
    drop text and skew the ground truth) + the 4 images on pages 1–4
    (distinct ``insert_image`` streams — distinct xrefs, distinct
    bytes). The metadata title is the stem (no title — the stem
    fallback, phase 126 A2)."""
    doc = fitz.open()
    for k in range(1, N_PAGES + 1):
        page = doc.new_page()
        rc = page.insert_textbox(
            fitz.Rect(48, 90, 548, 812), _page_text(k), fontsize=10
        )
        assert rc >= 0, f"the page {k} text did not fit the rect (rc={rc})"
        if k <= N_IMAGES:
            w, h = _IMAGE_SHAPES[k - 1]
            page.insert_image(
                fitz.Rect(72, 820 - h * 2, 72 + w * 2, 860),
                stream=_noise_png(w, h, k),
            )
    doc.save(pdf_path)
    doc.close()


def _expected_chunks(pdf_path: Path) -> list[str]:
    """The PDF's expected content chunks — the SAME function and
    arguments the importer's ``_replace_content_chunks`` uses
    (``chunk_document(content, rel, max(400, target), overlap)`` — the
    importer's token-cap floor rule; ``rel`` = the source-relative
    path) over the SAME parser the importer reads
    (``extract_pdf_text`` — the ``---PAGE-N---`` page markers): the
    endpoint's ``chunks_total`` must equal
    ``len(chunk_document(...))`` — the file's TRUE chunk count, not a
    batch count."""
    content = extract_pdf_text(pdf_path)
    return chunk_document(
        content, PDF_NAME, max(400, CHUNK_TARGET), CHUNK_OVERLAP
    )


def _expected_batch_count(chunks: list[str]) -> int:
    """The expected embed-batch count — the SAME
    ``LLMClient._batch_texts`` partition the app's embed loop walks
    (the phase-129 unit pattern — call it directly for the boundaries).
    The task's sizing bar: ≥ 2 batches (the chunk part's climb is a
    real sequence, not a single point)."""
    return len(LLMClient(_EXPECTED_SETTINGS)._batch_texts(chunks))


# ---------------------------------------------------------------------------
# The in-browser snapshot functions (Playwright Mapping Rule — the
# rendered DOM is the observable).
#
# ``_JS_SAMPLE`` — test 1's dense sampler: the box text + the button
# title/aria read in the SAME in-browser evaluation (a 2 s poll tick
# updates the parts + the title together in one JS task, so the atomic
# check can never straddle one — the phase-128 pattern); ``running``
# marks the in-flight labels (the A4 title/aria contract applies to
# them).
#
# ``_JS_IMAGE_SNAP`` — test 2's wait: null until the box carries the
# ``· image n/<arg>`` part, then the texts (read atomically with the
# title/aria).
# ---------------------------------------------------------------------------

_JS_SAMPLE = """
() => {
    const boxTextEl = document.querySelector('#sync-progress-text');
    const btn = document.querySelector('#sync-btn');
    if (!boxTextEl || !btn) return null;
    const box = boxTextEl.textContent;
    return {
        box: box,
        title: btn.getAttribute('title') || '',
        aria: btn.getAttribute('aria-label') || '',
        running: box.startsWith('Syncing…'),
    };
}
"""

_JS_IMAGE_SNAP = """
(n) => {
    const boxTextEl = document.querySelector('#sync-progress-text');
    const btn = document.querySelector('#sync-btn');
    if (!boxTextEl || !btn) return null;
    const box = boxTextEl.textContent;
    const m = box.match(/^Syncing… .*· image (\\d+)\\/(\\d+)$/);
    if (!m || Number(m[2]) !== n) return null;
    return {
        box: box,
        posN: Number(m[1]),
        title: btn.getAttribute('title') || '',
        aria: btn.getAttribute('aria-label') || '',
    };
}
"""


# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------


@pytest.fixture(scope="module")
def imgpass_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
    """The one-source temp tree (see the module docstring): the app
    server runs on the same host, so the paths are visible to it.
    The directory NAME is the source name (``kind=local``, phase 38).
    EXACTLY ONE file: the PyMuPDF-built ``imgpass.pdf`` (10 text
    pages + 4 distinct non-blank images — see :func:`_build_pdf`)."""
    root = tmp_path_factory.mktemp("bor_imgprog")
    src = root / SOURCE
    src.mkdir()
    _build_pdf(src / PDF_NAME)
    assert len(list(src.rglob("*"))) == N_FILES
    return src


def _app_env(base_url: str, tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]:
    """The per-module app env (the conftest pattern, cf.
    ``test_oneshot_llm_retry.py`` / ``test_kb_tree.py``): the
    ``base_url`` picks the LLM (the slow proxy in front of the mock),
    ``BOR_GIT_SOURCES`` forced empty (the sync source is this suite's
    own DB-registered local row), the sizing knobs pinned to the code
    defaults (the leak guards — see the module constants), the
    phase-126 knobs pinned EXPLICITLY (``BOR_IMPORT_EXTENSIONS`` with
    the ``pdf`` token — without it the walk is blind to the fixture —
    and ``BOR_IMAGES=true`` — the embedded pass is the span under
    test), the E2E ``BOR_PDF_PICTORIAL`` off-pin (the house
    convention — the suite's span is the embedded pass; the fixture's
    ≥100-char pages keep the pictorial pass a no-op even if the knob
    leaked on), and the other leak-guarded code defaults."""
    env = dict(os.environ)
    env.pop("DEBUGPY", None)
    env["BOR_ENVIRONMENT"] = "e2e"
    env["BOR_STATIC_DIR"] = str(REPO / "frontend")
    env["BOR_LLM_BASE_URL"] = (
        "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
    )
    # Mock-calibrated threshold (conftest pattern): this suite never
    # asks the chat model anything — the gate is never on a path.
    env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
    # Phase 67: instant retry waits + the code-default budget (the
    # conftest leak-guard pattern).
    env["BOR_LLM_RETRY_DELAY"] = "0"
    env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
    env.setdefault(
        "BOR_DATABASE_URL",
        "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
    )
    # Phase 16: admin auth must be set or create_app() refuses to boot.
    env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
    env["BOR_SESSION_SECRET"] = SESSION_SECRET
    # The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
    # reads it from cwd) — override it with an EMPTY value (the env var
    # beats the .env file): the registry must hold EXACTLY the local
    # directory this suite registers.
    env["BOR_GIT_SOURCES"] = ""
    # Leak guards (conftest pattern): an operator's local (gitignored)
    # .env cannot leak corpus-specific settings into the app under
    # test — including the chunking / batching knobs this suite's
    # ground truth is computed from (see the module constants).
    env["BOR_DOCS_REPO"] = ""
    env["BOR_SUGGESTIONS"] = json.dumps(
        _Settings.model_fields["suggestions"].default
    )
    env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
    env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
    env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET)
    env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP)
    env["BOR_EMBED_BATCH_SIZE"] = str(EMBED_BATCH_SIZE)
    # Phase 126/122: the import scope WITH the ``pdf`` token + the
    # embedded-image toggle ON (the span under test) + the E2E
    # pictorial off-pin + the suite-private image home (the app under
    # test must not write copies into the owner's real ~/bor-sources).
    env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
    env["BOR_IMAGES"] = "true"
    env["BOR_PDF_PICTORIAL"] = "0"
    env["BOR_IMAGE_DIR"] = str(tmp_path_factory.mktemp("bor_imgprog_images"))
    env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
    return env


@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
    """The delay-injecting reverse proxy in front of the mock LLM
    (``tests/e2e/slow_llm.py``) — the timing fixture for the whole
    suite: it delays EVERY LLM call, vision describes included, so
    each in-flight image pipeline's position outlives the UI's 2 s
    poll (see ``SLOW_DELAY_S`` + the module docstring's timing
    fixture paragraph)."""
    env = dict(os.environ)
    env.pop("DEBUGPY", None)
    env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
    env["E2E_MOCK_PORT"] = str(MOCK_PORT)
    proc = subprocess.Popen(
        [sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
         "--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
        cwd=REPO,
        env=env,
    )
    try:
        _wait_http(f"{SLOW_URL}/v1/models")
        yield SLOW_PORT
    finally:
        proc.terminate()
        try:
            proc.wait(timeout=10)
        except subprocess.TimeoutExpired:
            proc.kill()


@pytest.fixture(scope="module")
def app_server(
    mock_llm: int,
    slow_llm: int,
    imgpass_dir: Path,
    tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
    """The SLOW app under test (the whole suite):
    ``BOR_LLM_BASE_URL`` is the slow proxy in front of the mock (the
    timing fixture — see the module docstring). ``imgpass_dir`` is a
    dependency only for the fixture ordering (the temp tree exists
    before the first test). Skips the module (clear instructions)
    when Postgres is down — the sync needs the KB tables."""
    env = _app_env(f"{SLOW_URL}/v1", tmp_path_factory)
    proc = subprocess.Popen(
        [sys.executable, "-m", "uvicorn", "app.main:app",
         "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
        cwd=REPO,
        env=env,
    )
    try:
        _wait_http(f"{APP_URL}/api/health")
        body = httpx.get(f"{APP_URL}/api/health", timeout=5).json()
        if body["db"] != "up":
            pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
        yield APP_URL
    finally:
        proc.terminate()
        try:
            proc.wait(timeout=10)
        except subprocess.TimeoutExpired:
            proc.kill()


@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
    return app_server


def _truncate_all() -> None:
    """Fresh registry + KB (the E2E isolation pattern): the E2E suites
    share one Postgres, so a leftover git_sources row or document would
    pollute the file counts + the counts line the syncs and the
    assertions pin exactly. ``sources_meta`` (the KB generation counter)
    resets with the KB — the sync re-creates the row."""
    with SessionLocal() as db:
        db.execute(
            text(
                "TRUNCATE chunks, documents, query_log, steering_notes, "
                "kb_overview, git_sources, sources_meta, folder_summaries"
            )
        )
        db.commit()  # without the commit the TRUNCATE rolls back (the house pattern)


def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
    """No background sync may leak across tests (the run state lives in
    the app's memory, and a still-running run would keep importing into
    the NEXT test's truncated KB): wait for the app's sync status to
    be non-running BEFORE the truncate. Own admin session (the
    endpoint is admin-only) — usually a no-op: the tests settle only
    after their run's terminal state."""
    with httpx.Client(base_url=base_url, timeout=5.0) as client:
        r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
        assert r.status_code == 204, r.text
        deadline = time.monotonic() + timeout_s
        while time.monotonic() < deadline:
            r = client.get("/api/sync/status")
            assert r.status_code == 200, r.text
            if r.json()["state"] != "running":
                return
            time.sleep(0.2)
    raise AssertionError(f"a sync was still running at a test boundary: {base_url}")


@pytest.fixture(autouse=True)
def _clean(app_url: str) -> Iterator[None]:
    """Per-test KB isolation (the phase-96 cleanup pattern): the
    shared E2E Postgres is truncated before AND after every test —
    each test seeds its own source row and runs its own sync, so the
    KB rows + folder_summaries rows of the temp source are exactly
    this test's own. The pre-wait keeps a leaked run from the previous
    test (the module app is shared) from racing the truncate."""
    _wait_sync_settled(app_url)
    _truncate_all()
    yield
    _truncate_all()


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------


def _register_source(base_url: str, path: str) -> None:
    """Register the temp directory through the authenticated API (the
    ``test_local_directory_sources.py`` / phase-94 pattern — a plain
    directory is an API/DB-only operation)."""
    with httpx.Client(base_url=base_url, timeout=30.0) as client:
        r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
        assert r.status_code == 204, r.text
        r = client.post(
            "/api/git-sources", json={"kind": "local", "path": path}
        )
        assert r.status_code == 201, r.text


def _wait_terminal(app_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
    """Poll the status endpoint (own admin session) until the
    UI-/API-triggered run reaches a terminal state — the source of
    truth for the settle (the UI's 2 s poll renders it at most one
    tick later). A FAILED terminal fails the test with the terminal
    body (the ``detail`` / error text) — never an opaque label
    timeout."""
    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
        assert r.status_code == 204, r.text
        deadline = time.monotonic() + timeout_s
        body: dict[str, Any] = {}
        while time.monotonic() < deadline:
            r = client.get("/api/sync/status")
            assert r.status_code == 200, r.text
            body = r.json()
            if body["state"] in ("success", "failed"):
                assert body["state"] == "success", body
                return body
            time.sleep(0.2)
    raise AssertionError(f"the run never reached a terminal state: {body}")


def _start_sync_click(page: Page, app_url: str, sync_dir: Path) -> None:
    """Register the source, land the admin on the sources view, wait
    for the settled boot label (the phase-64 pattern — also proves the
    phase-32 boot re-attach completed, so no later re-render can
    clobber the running state), then click **Sync sources** and pin
    the click's immediate 202 state (the bare "Syncing…" — the count
    AND the chunk parts hidden: no file yet, no batch in flight — and
    the image part is absent too: no PDF's vision span has started,
    ``images_total == 0``)."""
    _register_source(app_url, str(sync_dir))
    login(page, app_url)  # lands on /sources.html (the button's home)

    btn = page.locator("#sync-btn")
    expect(btn).to_be_visible(timeout=30_000)
    expect(page.locator("#sync-label")).to_have_text(IDLE_OR_SETTLED)
    expect(btn).to_be_enabled()

    btn.click()

    # The click's immediate state (the 202 moment — the bare prefix,
    # the phase-64 contract): disabled, aria-busy, spinning icon, and
    # the label's MAIN part alone ("Syncing…" — the 202 click carries
    # no status, so the pinned parts hide, phase 128/129 writers).
    # Phase 149 (task 01): the running detail rides the box — the 202's
    # bare label is in #sync-progress (the button is the spinner).
    expect(btn).to_be_disabled()
    expect(btn).to_have_attribute("aria-busy", "true")
    expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
    expect(page.locator("#sync-progress")).to_have_text("Syncing…")
    expect(page.locator(".sync-label-count")).to_be_hidden()
    expect(page.locator(".sync-label-chunk")).to_be_hidden()
    expect(page.locator("#sync-error-banner")).to_be_hidden()


# ---------------------------------------------------------------------------
# 1. The anti-freeze proof — the label advances through the PDF's
#    image span (≥3 distinct positions, monotonic, chunk part earlier,
#    the A4 title/aria, the settled clearing)
# ---------------------------------------------------------------------------


def test_label_advances_through_the_image_span(
    page: Page, app_url: str, imgpass_dir: Path
) -> None:
    """A fresh admin page on ``/sources.html`` clicks **Sync sources**;
    the test then densely samples (≈ every 300 ms) the rendered
    ``#sync-progress-text`` — NOT the API (the rendered DOM is the
    observable; the UI's own 2 s poll is the writer) — collecting the
    DISTINCT values in first-seen order until the settled counts line,
    and asserts the phase-152 contract end to end (see the module
    docstring's test mapping): the image part climbs (≥3 distinct
    positions, starting at 1, non-decreasing, never above the total,
    reaching it), the chunk part appeared earlier (and persists
    through the image span with the file's TRUE chunk total), the
    A4 title/aria ride the full string on every running sample, and
    the settle clears the image position (the counts line, the
    terminal label, the cleared endpoint position — no stale
    position in a terminal state)."""
    page.set_default_timeout(30_000)

    # The ground truth — computed from the PDF ON DISK with the SAME
    # parser + function + arguments the importer uses (see the helper
    # docstrings): the TRUE chunk count + the batch count (the task's
    # sizing bar — ≥2 embed batches, so the chunk part's climb is a
    # real sequence).
    pdf_path = imgpass_dir / PDF_NAME
    chunks = _expected_chunks(pdf_path)
    n_chunks = len(chunks)
    assert n_chunks >= EMBED_BATCH_SIZE + 1, n_chunks  # 2+ embed batches
    assert _expected_batch_count(chunks) >= 2  # the sizing bar
    # The happy path's premise: a clean text layer (the phase-133
    # fallback never fires) — the embedded-image pass is the file's
    # one and only vision span (the ≥100-char pages keep the phase-140
    # gate candidate-free even under the app's pinned-off knob).
    assert is_usable_pdf_text(extract_pdf_text(pdf_path))

    _start_sync_click(page, app_url, imgpass_dir)

    # The observation loop (the task's "poll the page, not the API"):
    # dense samples of the rendered box text (≈ 300 ms — the evaluate
    # round-trip + the sleep) in first-seen order, each sample read
    # atomically with the button's title/aria (A4 checked per running
    # sample), until the settled counts line lands (or the generous
    # deadline — a failed run then fails LOUDLY in _wait_terminal
    # with the terminal body, never an opaque timeout).
    seen: list[str] = []
    last: str | None = None
    deadline = time.monotonic() + 150
    while time.monotonic() < deadline:
        snap = page.evaluate(_JS_SAMPLE)
        assert snap is not None
        if snap["running"]:
            # A4 — while a run is in flight, the title + the
            # aria-label carry the SAME untruncated full string as the
            # box (the "· image" part included).
            assert snap["title"] == snap["box"], snap
            assert snap["aria"] == snap["box"], snap
        if snap["box"] != last:
            seen.append(snap["box"])
            last = snap["box"]
        if snap["box"] == SETTLE_TEXT:
            break
        time.sleep(0.2)

    # --- the settle (the UI's 2 s poll renders the terminal at most one
    # tick after it lands; the endpoint is the source of truth) ---
    assert last == SETTLE_TEXT, f"the run never settled to the counts line: {seen[-5:]}"
    terminal = _wait_terminal(app_url)
    # The counts line's ground truth: the PDF + its 4 embedded-image
    # docs — one file, 5 added.
    assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal
    # The keep-final-file-counts split (the phase-64/129 convention,
    # restated for the phase-152 sub-position): the FILE counts keep
    # their final values, the sub-file chunk + image positions clear
    # with the file (0/0 in the terminal — no stale position), the
    # phase clears.
    assert terminal["files_done"] == N_FILES and terminal["files_total"] == N_FILES, terminal
    assert terminal["current_file"] is None, terminal
    assert terminal["current_chunk"] == 0 and terminal["chunks_total"] == 0, terminal
    assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal
    assert terminal["phase"] is None, terminal

    # --- the run was real (the prelude + the import span were
    # sampled) ---
    assert any(t == "Syncing…" for t in seen), f"no bare prelude label: {seen[:6]}"
    import_labels = [t for t in seen if t.startswith("Syncing… ")]
    assert any(f"({N_FILES}/{N_FILES})" in t for t in import_labels), (
        f"no file count part: {import_labels[:6]}"
    )

    # --- the chunk part appeared EARLIER in the run (the two
    # positions are distinct) ---
    chunk_idx = next(
        (i for i, t in enumerate(seen) if CHUNK_PART_RE.search(t)), None
    )
    assert chunk_idx is not None, f"no chunk part observed: {seen}"
    first_chunk = CHUNK_PART_RE.search(seen[chunk_idx])
    assert first_chunk is not None, seen[chunk_idx]
    # The chunk part's total is the file's TRUE chunk count (not a
    # batch count — the phase-129 A6 assumption, the E2E lens).
    assert int(first_chunk.group(2)) == n_chunks, (first_chunk.group(0), n_chunks)

    # --- the image part: ≥3 distinct positions (the anti-freeze
    # proof — the owner's freeze is a single repeated label), the
    # positions START at 1, are NON-DECREASING in observation order,
    # never EXCEED the total (the post-dedupe/post-gate count — the
    # work actually about to happen), and REACH it ---
    def _image_part(t: str) -> re.Match[str]:
        m = IMAGE_PART_RE.search(t)
        assert m is not None, t  # the list comprehension's filter guarantees it
        return m

    image_items = [
        (i, _image_part(t))
        for i, t in enumerate(seen)
        if IMAGE_PART_RE.search(t)
    ]
    assert len(image_items) >= 3, (
        f"fewer than 3 distinct image positions observed: "
        f"{[m.group(0) for _, m in image_items]}"
    )
    first_image_idx = image_items[0][0]
    assert chunk_idx < first_image_idx, (chunk_idx, first_image_idx)
    positions = [(int(m.group(1)), int(m.group(2))) for _, m in image_items]
    assert positions[0][0] == 1, f"the climb does not start at 1: {positions}"
    assert all(m == N_IMAGES for _, m in positions), positions
    assert all(n <= N_IMAGES for n, _ in positions), positions
    ns = [n for n, _ in positions]
    assert ns == sorted(ns), f"the climb is backwards somewhere: {positions}"
    assert N_IMAGES in ns, f"the climb never reached the total: {positions}"
    # The two positions COEXIST (distinct and both may show — "… (1/1)
    # · chunk 19/20 · image 3/4"): every image-span label carries the
    # chunk part with it (the last chunk position persists through the
    # file's image span — the per-file reset + the phase transitions
    # clear it, not the end of the chunk span).
    for i, _ in image_items:
        cm = CHUNK_PART_RE.search(seen[i])
        assert cm is not None, f"an image-span label lost the chunk part: {seen[i]}"
        assert int(cm.group(2)) == n_chunks, (cm.group(0), n_chunks)

    # --- the settle: the counts line, the terminal label, the parts
    # cleared + empty, and the IMAGE part GONE (no stale position in a
    # terminal state — the endpoint pin above is the server-side
    # half) ---
    expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
    btn = page.locator("#sync-btn")
    expect(btn).to_be_enabled()
    expect(btn).not_to_have_attribute("aria-busy")
    expect(page.locator(".sync-label-count")).to_be_hidden()
    expect(page.locator(".sync-label-count")).to_have_text("")
    expect(page.locator(".sync-label-chunk")).to_be_hidden()
    expect(page.locator(".sync-label-chunk")).to_have_text("")
    # The counts line rides the box (phase 150 — the sole sync text
    # surface): the image part is not in it, not in the title, not in
    # the aria (the settle restores the ship aria value).
    box = page.locator("#sync-progress-text").inner_text()
    assert box == SETTLE_TEXT, box
    assert "· image" not in box and "· image" not in (btn.get_attribute("title") or "")
    assert btn.get_attribute("aria-label") == "Sync sources", btn.get_attribute("aria-label")
    expect(page.locator("#sync-error-banner")).to_be_hidden()


# ---------------------------------------------------------------------------
# 2. The never-stale contract — a mid-run page reload re-attaches with
#    the current image position visible
# ---------------------------------------------------------------------------


def test_mid_run_reload_re_attaches_with_image_position(
    page: Page, app_url: str, imgpass_dir: Path
) -> None:
    """The admin clicks **Sync sources** (the run is started the way
    the owner starts it — the click is what starts the page's 2 s
    poll; the reload is the contract under test), the page's own 2 s
    poll renders the image span (the FIRST observed position is
    ``1/4`` — the whole span is ahead, so the reload + the load-time
    re-attach happen with the span in flight), the test reloads the
    page mid-run, and the load-time re-attach (``initSyncButton`` —
    a running sync re-enters with the WHOLE status, phase 98's
    never-stale contract) renders the CURRENT ``· image n/4`` with
    the A4 title/aria; the run then settles to the counts line (the
    image part gone — no stale position)."""
    page.set_default_timeout(30_000)
    _start_sync_click(page, app_url, imgpass_dir)

    # The image span, rendered by the page's own 2 s poll (the span
    # starts after the prelude + the file's embed batches — ≈ 21 s
    # under the slow proxy; each position holds 4.5 s — the describe
    # + the image doc's 2 embed batches).
    snap = page.wait_for_function(
        _JS_IMAGE_SNAP, arg=N_IMAGES, timeout=60_000
    ).json_value()
    # The FIRST observed position is 1 — the whole span is ahead (the
    # reload's re-attach is guaranteed to catch the span in flight:
    # ≈ 18 s remain, the local reload + the load-time status fetch
    # land well inside it).
    assert snap["posN"] == 1, snap
    # A4 — the pre-reload running state is untruncated on the title +
    # the aria (the "· image" part included).
    assert snap["title"] == snap["box"] and "· image" in snap["title"], snap
    assert snap["aria"] == snap["box"], snap

    # The mid-run reload (the session cookie survives — the re-attach
    # is the admin's, phase 98).
    page.reload()

    # The load-time re-attach: a running sync re-enters with the WHOLE
    # status — the CURRENT image position is visible (the never-stale
    # contract), the A4 title/aria ride it.
    snap2 = page.wait_for_function(
        _JS_IMAGE_SNAP, arg=N_IMAGES, timeout=30_000
    ).json_value()
    assert snap2["title"] == snap2["box"] and "· image" in snap2["title"], snap2
    assert snap2["aria"] == snap2["box"], snap2

    # Then the settle: the counts line (the image part gone — the
    # terminal clears the position, test 1 pins the endpoint half),
    # the terminal label, the parts cleared + empty.
    expect(page.locator("#sync-progress-text")).to_have_text(
        SETTLE_TEXT, timeout=SETTLE_TIMEOUT_MS
    )
    expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
    btn = page.locator("#sync-btn")
    expect(btn).to_be_enabled()
    expect(btn).not_to_have_attribute("aria-busy")
    expect(page.locator(".sync-label-count")).to_be_hidden()
    expect(page.locator(".sync-label-chunk")).to_be_hidden()
    assert "· image" not in (btn.get_attribute("title") or "")
    expect(page.locator("#sync-error-banner")).to_be_hidden()

    terminal = _wait_terminal(app_url)
    assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal
    assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal