"""Phase 184 E2E (Playwright, mock-only): the BOUNDED sync still syncs all. Source: owner incident 2026-10-15 — the same run whose prune phase 183 fixed. An import run holds ONE session and ``app/db.py`` builds it with ``expire_on_commit=False``, so a commit released NOTHING: the walk kept every ``Document`` it touched resident (each unchanged row holds its full ~15 KB ``content``, and ``_unchanged_doc``'s ``file_hash`` stamp puts the row in the session's own strong ``dirty`` set, out of reach of the cycle collector), and ``_prune`` read every document of every pruned source WHOLE — ``content`` included — holding all of its deletes in ONE transaction until the end. The run's memory therefore grew with the SIZE OF THE KNOWLEDGE BASE instead of with the work the run did. The fix is two operator knobs, both ``BOR_``-prefixed settings the importer reads once per RUN: * ``BOR_SYNC_SESSION_RESET_FILES`` (default ``200``) — every Nth file of the walk, at the one point where no object from a finished file is reachable, the session calls ``expunge_all()``; it also runs once per source, before the two post-passes. ``0`` = NEVER release = the exact pre-phase-184 run. * ``BOR_SYNC_PRUNE_BATCH`` (default ``500``) — the prune commits every Nth delete instead of once at the end of the whole import. ``0`` = the exact pre-phase-184 single commit. A non-zero batch deliberately trades the all-or-nothing prune for a PREFIX (LOCKED A2: ``seen`` is recomputed from disk, so the next sync converges). Both knobs change the run's MEMORY PROFILE and nothing else — and that "nothing else" is precisely what a browser can see. This module pins it: the same counts line, every document exactly once, an all-``unchanged`` re-sync, a multi-batch prune that behaves, and a Sources page that never stalls, never errors and reports the same totals as the database. The retention itself (the identity-map bound, the prune's six-column ``SELECT``, the commit counts) is deterministic and pinned in ``tests/integration/test_import_session_retention.py``; RSS is asserted nowhere in this phase (phase 183 LOCKED A4, inherited). Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is skipped — every number here (the per-path chunk counts, the byte-identical cross-check) is a property of the deterministic mock embedder/summarizer; against the real endpoint the comparison would be a measurement, not a pin. TWO module apps on dedicated ports (the ``test_sync_summary_visibility.py`` pattern — the conftest session app owns 8123 in a combined run, and a same-port second uvicorn would drive the wrong server): * the CADENCE app — ``BOR_SYNC_SESSION_RESET_FILES=3`` and ``BOR_SYNC_PRUNE_BATCH=2``, so a 10-file walk releases the session THREE times mid-run (before files 4, 7 and 10) plus once at the source boundary, and a 5-document prune commits THREE times (at 2, at 4, and the final one). Tests 1–3 drive it; * the ZERO app — both knobs ``0``, i.e. the exact pre-phase-184 run. Test 4 drives it over the SAME fixture directory, which is what makes the two runs comparable at all. Both boot against the mock LLM with ``BOR_GIT_SOURCES`` forced empty (the owner's local gitignored ``.env`` must never leak a real source into a run), their own ``BOR_SOURCES_DIR`` / ``BOR_PDF_PAGE_DIR`` under ``tmp_path_factory`` (nothing here clones or renders a PDF — the dirs exist so no run writes into the operator's), and the mock-calibrated gate (the conftest convention). The fixture KB is ONE plain (non-git) directory named :data:`SOURCE_NAME` (``kind='local'`` rows index under ``Path(path).name``, so the source label is stable whatever temp root pytest picked) holding TEN ``.md`` files across TWO folders — ``a/1..5.md`` and ``b/1..5.md`` — with ``a/1.md`` carrying the sentinel phrase. Isolation: the shared E2E Postgres is truncated BEFORE **and** AFTER every test (``chunks, documents, query_log, kb_overview, git_sources, sources_meta, folder_summaries``), after waiting for the app to hold no running sync (the run state lives in the app's memory, and a leaked run would keep importing into the next test's truncated KB). Each test registers its own source row. Test → observable mapping (Playwright Mapping Rule): 1. ``test_reset_cadence_produces_the_identical_run`` — the cadence ON is a complete, healthy sync: the counts line reads ``10 added``, the terminal reports ``added=10`` with ``errors=0``, every one of the ten paths exists EXACTLY ONCE in the KB (the test's own SQL), the ls/tree rows of both folders render all ten files, the stat cards equal ``select(count())`` over ``documents`` (10) and ``chunks`` (> 10), the sentinel's phrase is in the stored content, and THROUGH THE RUN no ``/api/`` request answered 4xx/5xx and none failed — no error banner, no modal. 2. ``test_resync_is_unchanged_with_the_cadence_on`` — the second sync of the same bytes settles ``0 added · 10 unchanged`` with ``pruned=0`` and ``success``, and the chunk total and per-path chunk counts are byte-still: the cadence loses no rows (a lost row would come back as ``added``) and re-embeds nothing. 3. ``test_multi_batch_prune_from_the_ui`` — five of the ten files go and the re-sync settles ``0 added · 5 unchanged · 5 pruned`` (5 deletes at batch ``2`` = THREE commits inside the UI's own run); the five documents are gone from the KB AND from the ls/tree rows of BOTH folders, the survivors' chunk total is unchanged, and ZERO orphan chunk rows exist (the test's own SQL). 4. ``test_cadence_zero_matches_cadence_three`` — the byte-identical cross-check the phase lives on: the SAME fixture walked with ``BOR_SYNC_SESSION_RESET_FILES=0`` lands the same paths (each exactly once), the same per-path chunk counts, the same ``content`` / ``summary`` / ``content_hash`` / ``file_hash`` on every document, the same chunk contents in position order and the same counts line as the cadence-``3`` run. The reference is test 1's recorded run when the module runs in order; alone, the test walks the cadence-``3`` app itself first — and either way it asserts its own 10-documents / >10-chunks shape BEFORE comparing, so an empty baseline can never make it pass vacuously. """ from __future__ import annotations import os import re import subprocess import sys import time from collections.abc import Iterator from datetime import datetime from pathlib import Path from typing import Any import httpx import pytest from playwright.sync_api import Page, expect from sqlalchemy import func, select, text from app.db import SessionLocal from app.models import Chunk, Document from e2e.auth_helpers import login from e2e.conftest import ( ADMIN_PASSWORD, MOCK_PORT, SESSION_SECRET, USE_REAL_LLM, _wait_http, ) REPO = Path(__file__).resolve().parents[2] #: The CADENCE app (tests 1–3). Env-overridable, the house port pattern: the #: conftest session app owns 8123 in a combined run. CADENCE_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET", "8200")) #: The ZERO app (test 4's reference leg): both knobs ``0`` — the exact #: pre-phase-184 run (LOCKED A3's no-redeploy rollback). ZERO_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET_ZERO", "8201")) SOURCES_URL = "/sources.html" #: The fixture directory's basename — the ``source`` the importer records for #: a ``kind='local'`` row (``Path(path).name``), so it is the same string in #: the DB whatever temp root this run got. SOURCE_NAME = "sync-session-reset" #: The fixture layout: two folders × five importable files. After the prune, #: ``a`` keeps three survivors and ``b`` two — so BOTH folders still render #: rows, and a prune that stopped after its first batch would show up. ALL_PATHS = [*[f"a/{n}.md" for n in range(1, 6)], *[f"b/{n}.md" for n in range(1, 6)]] KEEP_PATHS = ["a/1.md", "a/2.md", "a/3.md", "b/1.md", "b/2.md"] GONE_PATHS = ["a/4.md", "a/5.md", "b/3.md", "b/4.md", "b/5.md"] N_FILES = len(ALL_PATHS) #: The sentinel phrase, carried by :data:`SENTINEL_PATH` ONLY. Finding it in #: the stored ``content`` proves the walk really indexed the fixture rather #: than leaving a KB of merely the right SIZE. SENTINEL = "RESE-SYNC-RESET-SENTINEL-4f27" SENTINEL_PATH = "a/1.md" #: The cadences the two apps boot with — module constants so the narrative and #: the app env cannot drift apart. 10 files / every 3rd = releases before #: files 4, 7 and 10 (THREE mid-run resets, then the source boundary); a #: 5-document prune at batch 2 = commits at 2, at 4 and the final one. RESET_FILES = 3 PRUNE_BATCH = 2 #: "Synced HH:MM" — the local-time last-result label (sources.js's #: fmtSyncTime), any hour/minute (the test_sync_button.py pattern). SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$") #: Settle budgets: a ten-file mock sync (walk + embeds + overview + three #: folder summaries) lands in seconds and the UI settles a terminal within one #: 2 s poll tick, so this is an order of magnitude of head-room with no #: client-side hard timeout (the test_prune_cascade_cleanup budget). SETTLE_TIMEOUT_MS = 90_000 #: Test 1 records its numbers here for test 4's cross-check (the #: ``test_sync_changed_only.py`` "state from the earlier test" idiom). Test 4 #: runs its own cadence leg when this is empty, so it stays honest alone. _CADENCE_RUN: dict[str, Any] = {} # --------------------------------------------------------------------------- # The fixture source (ONE directory BOTH apps walk) # --------------------------------------------------------------------------- def _doc_body(path: str) -> str: """One small markdown file per fixture path, byte-stable across runs. The bodies differ from each other (so a per-path chunk count means something) but never carry a timestamp or the absolute path, and only :data:`SENTINEL_PATH` carries the sentinel — that is what lets test 4 compare two runs of two different processes line for line. """ if path == SENTINEL_PATH: return ( "# Session reset sentinel\n" "\n" "The bounded sync keeps the sentinel phrase in the knowledge base.\n" f"Marker: {SENTINEL}\n" ) folder, name = path.rsplit("/", 1) stem = name.removesuffix(".md") return ( f"# Session reset note {folder} {stem}\n" "\n" f"One unrelated note from the {folder} folder of the {SOURCE_NAME} " "fixture source, indexed so the cadence has something to walk.\n" ) def _seed_source(root: Path) -> Path: """(Re-)write all ten fixture files under ``root / SOURCE_NAME``. Idempotent on purpose: test 3 deletes five files from the tree and test 4 re-seeds the SAME directory, so the reference leg walks byte-identical content through the other app. """ base = root / SOURCE_NAME for rel in ALL_PATHS: target = base / rel target.parent.mkdir(parents=True, exist_ok=True) target.write_text(_doc_body(rel), encoding="utf-8") assert not (base / ".git").exists() # a plain directory — no git anywhere return base @pytest.fixture(scope="module") def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path: """The one fixture tree BOTH module apps walk (the app servers run on the same host, so the path is visible to them).""" return _seed_source(tmp_path_factory.mktemp("bor_sync_session_reset")) # --------------------------------------------------------------------------- # The two module apps # --------------------------------------------------------------------------- def _app_env(scratch: Path, reset_files: int, prune_batch: int) -> dict[str, str]: """The per-module app env (the conftest pattern): the mock LLM, the mock-calibrated gate, no git fallback, dedicated scratch dirs — and THIS app's two phase-184 cadences, pinned explicitly so an operator's local (gitignored) ``.env`` cannot retune them.""" env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = ( "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else f"http://127.0.0.1:{MOCK_PORT}/v1" ) # Mock-calibrated gate (the conftest pattern): the token-overlap mock # embeddings score far below the production 0.62, and BOTH floors must # stay <= the threshold or the app refuses to boot (app/config.py). env["BOR_RELEVANCE_THRESHOLD"] = "0.30" env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15" env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) # Phase 16: admin auth must be set or create_app() refuses to boot. env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET # No env-fallback rows: the registry each test writes through the # authenticated API is the only source list, so every sync's counts are # exactly this test's doing. env["BOR_GIT_SOURCES"] = "" env["BOR_SOURCES_DIR"] = str(scratch / "checkouts") env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages") # THE SUBJECT OF THE PHASE. env["BOR_SYNC_SESSION_RESET_FILES"] = str(reset_files) env["BOR_SYNC_PRUNE_BATCH"] = str(prune_batch) return env def _serve(env: dict[str, str], port: int) -> Iterator[str]: """Boot one uvicorn app under test on *port* and yield its URL.""" proc = subprocess.Popen( [ sys.executable, "-m", "uvicorn", "app.main:app", "--host", "127.0.0.1", "--port", str(port), "--log-level", "warning", ], cwd=REPO, env=env, ) url = f"http://127.0.0.1:{port}" try: _wait_http(f"{url}/api/health") yield url finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_server( mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory ) -> Iterator[str]: """The CADENCE app (tests 1–3): ``reset=3``, ``prune_batch=2``. ``mock_llm`` and ``source_dir`` are dependencies for the ORDERING only — the mock answers before the app boots, and the fixture tree exists before the first test registers it. """ if USE_REAL_LLM: pytest.skip( "mock-only suite — the byte-identical cross-check is a property " "of the deterministic mock (E2E_REAL_LLM unsupported)" ) assert mock_llm and source_dir scratch = tmp_path_factory.mktemp("bor_sync_session_reset_cadence") yield from _serve(_app_env(scratch, RESET_FILES, PRUNE_BATCH), CADENCE_PORT) @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server @pytest.fixture(scope="module") def zero_app_server( mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory ) -> Iterator[str]: """The ZERO app (test 4's reference leg): both knobs ``0``, the exact pre-phase-184 run the bounded one must be byte-identical to.""" if USE_REAL_LLM: pytest.skip( "mock-only suite — the byte-identical cross-check is a property " "of the deterministic mock (E2E_REAL_LLM unsupported)" ) assert mock_llm and source_dir scratch = tmp_path_factory.mktemp("bor_sync_session_reset_zero") yield from _serve(_app_env(scratch, 0, 0), ZERO_PORT) @pytest.fixture(scope="module") def zero_app_url(zero_app_server: str) -> str: return zero_app_server # --------------------------------------------------------------------------- # Isolation # --------------------------------------------------------------------------- def _truncate_all() -> None: """Fresh registry + KB (the E2E isolation pattern). ``sources_meta`` (the KB generation counter) and ``folder_summaries`` reset with the KB: the sync's overview/bump is an upsert, and a leftover folder summary would ride the very ``changed_paths`` feed the prune writes.""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, kb_overview, " "git_sources, sources_meta, folder_summaries" ) ) db.commit() # without the commit the TRUNCATE rolls back def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None: """No run may leak across a truncate: wait for a non-running status with an own admin session (the endpoint is admin-only). Usually a no-op — the tests settle before they end.""" deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: with httpx.Client(base_url=base_url, timeout=5.0) as client: client.post("/api/login", json={"password": ADMIN_PASSWORD}) r = client.get("/api/sync/status") assert r.status_code == 200, r.text if r.json()["state"] != "running": return time.sleep(0.2) raise AssertionError(f"a sync was still running at a test boundary: {base_url}") @pytest.fixture(autouse=True) def _clean(app_url: str, db_ready: None) -> Iterator[None]: """Per-test KB isolation: the shared E2E Postgres is truncated before AND after every test, the app holding no running sync at both boundaries. The ZERO app is deliberately NOT waited on: it is booting lazily for test 4 only, and that test's own run is settled inside the test body, so the post-truncate can never race it.""" _wait_sync_settled(app_url) _truncate_all() yield _wait_sync_settled(app_url) _truncate_all() # --------------------------------------------------------------------------- # Helpers: the UI side # --------------------------------------------------------------------------- def _watch_network(page: Page) -> list[str]: """Collect every ``/api/`` 4xx/5xx and every failed ``/api/`` request the page sees — the owner's "the Sources page errored / stalled". Returns the (growing) list the caller asserts empty at the end of the run: the page's own 2 s status poll lands in this list, so a run that made the API answer 500 is caught here rather than at some later assertion. """ problems: list[str] = [] def _on_response(response: Any) -> None: if "/api/" in response.url and response.status >= 400: problems.append(f"HTTP {response.status} {response.url}") def _on_failed(request: Any) -> None: if "/api/" in request.url: problems.append(f"FAILED {request.url} ({request.failure})") page.on("response", _on_response) page.on("requestfailed", _on_failed) return problems def _add_local_source(page: Page, base_url: str, path: Path) -> None: """Register the fixture directory through the authenticated API (the phase-49 shape — the page's own form for this is gone; the admin cookie rides ``page.request``).""" r = page.request.post( f"{base_url}/api/git-sources", data={"kind": "local", "path": str(path)}, ) assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}" def _started_at(body: dict[str, Any]) -> float: """The status body's ``started_at`` as a wall-clock epoch (the server stamps it in UTC and this process runs on the same host).""" started = body.get("started_at") if not started: return 0.0 try: return datetime.fromisoformat(str(started)).timestamp() except ValueError: return 0.0 def _sync_from_sources_page(page: Page, base_url: str) -> dict[str, Any]: """Click the Sources page **Sync sources** button and wait for THIS run's terminal state (the module app is shared, so a terminal counts only after a ``running`` tick or when its ``started_at`` post-dates the click). Settles the UI on success and proves the owner saw no error surface: the button re-enabled, ``Synced HH:MM``, no error banner, no modal.""" btn = page.locator("#sync-btn") expect(btn).to_be_visible(timeout=30_000) expect(btn).to_be_enabled() clicked_at = time.time() btn.click() expect(btn).to_be_disabled() expect(page.locator("#sync-label")).to_have_text("Syncing…") deadline = time.monotonic() + SETTLE_TIMEOUT_MS / 1000 saw_running = False body: dict[str, Any] = {} while time.monotonic() < deadline: r = page.request.get(f"{base_url}/api/sync/status") assert r.status == 200, r.text body = r.json() state = body["state"] if state == "running": saw_running = True elif state in ("success", "failed", "cancelled") and ( saw_running or _started_at(body) >= clicked_at - 1.0 ): break time.sleep(0.2) else: raise AssertionError(f"sync never reached a terminal state: {body}") assert body["state"] == "success", f"sync failed: {body}" assert body["error"] is None, body expect(btn).to_be_enabled(timeout=15_000) expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000) expect(page.locator("#sync-error-banner")).to_be_hidden() assert page.locator(".sync-modal-backdrop").count() == 0 return body def _expect_counts_line(page: Page, expected: str) -> None: """The counts line — the owner-visible sync report on the single sync text surface (phase 150; sources.js fmtSyncResult).""" expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000) def _counts_line(page: Page) -> str: """The counts line as text — the string test 4 compares across the apps.""" return str(page.locator("#sync-progress-text").inner_text()) def _open_level(page: Page, *names: str) -> None: """Open ONE level at a time from the catalogue top — each name the EXACT link text of the source or folder row at the current level (the phase-97 drill idiom, the ``test_kb_tree_lazy_load.py`` settle). The top is reached through the breadcrumb's first segment when the page is drilled (the phase-99 drill state rides ``history.state``, so a nav re-click would KEEP the level — the crumb is the control that actually leaves it), and is a no-op when the page already sits there (the crumb hidden). Every click after that awaits the level it produced: the phase-167 node fetch is over once the breadcrumb names the level and no in-flight ``.kb-level-loading`` row remains.""" if not page.locator("#kb-crumb").is_hidden(): page.click('#kb-crumb a.kb-crumb-link:text-is("Knowledge base")') expect(page.locator("#kb-crumb")).to_be_hidden(timeout=30_000) expect(page.locator("#folders-tbody tr")).to_have_count(1, timeout=30_000) for name in names: page.click(f'#folders-tbody a.folder-link:text-is("{name}")') expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text( name, timeout=30_000 ) expect(page.locator(".kb-level-loading")).to_have_count(0) # --------------------------------------------------------------------------- # Helpers: the DB side # --------------------------------------------------------------------------- def _paths_with_counts() -> dict[str, int]: """``path`` → number of ``documents`` rows for that path — THE exactly-once probe: a reset that dropped work and re-added it shows up here.""" with SessionLocal() as db: rows = db.execute( select(Document.path, func.count(Document.id)) .where(Document.source == SOURCE_NAME) .group_by(Document.path) ).all() return {str(path): int(count) for path, count in rows} def _chunk_count_by_path(paths: list[str] | None = None) -> dict[str, int]: """``path`` → chunk rows per document (an absent path is not listed).""" query = ( select(Document.path, func.count(Chunk.id)) .outerjoin(Chunk, Chunk.document_id == Document.id) .where(Document.source == SOURCE_NAME) .group_by(Document.path) ) if paths is not None: query = query.where(Document.path.in_(paths)) with SessionLocal() as db: rows = db.execute(query).all() return {str(path): int(count) for path, count in rows} def _total_chunks() -> int: with SessionLocal() as db: return int(db.scalar(select(func.count()).select_from(Chunk)) or 0) def _doc_ids(paths: list[str]) -> list[str]: """The indexed document ids of ``(SOURCE_NAME, path)``, sorted. Test 3 needs them BEFORE the prune — the orphan query runs on ids that no longer exist afterwards.""" with SessionLocal() as db: return sorted( str(doc_id) for (doc_id,) in db.execute( select(Document.id).where( Document.source == SOURCE_NAME, Document.path.in_(paths) ) ).all() ) def _orphan_chunks() -> int: """Chunk rows whose document is gone — phase 183's contract, asserted here too so a BATCHED prune cannot leave any behind.""" with SessionLocal() as db: return int( db.scalar( select(func.count()) .select_from(Chunk) .where(Chunk.document_id.not_in(select(Document.id))) ) or 0 ) def _kb_snapshot() -> dict[str, tuple[str, str | None, str, str, list[str]]]: """The byte-level shape of the fixture source: ``path`` → ``(content, summary, content_hash, file_hash, chunk contents in position order)`` — every field the two runs must agree on. Timestamps and ``full_path`` are deliberately absent: they belong to the run and to the temp root, not to the imported bytes. """ with SessionLocal() as db: docs = db.scalars( select(Document) .where(Document.source == SOURCE_NAME) .order_by(Document.path) ).all() snapshot: dict[str, tuple[str, str | None, str, str, list[str]]] = {} for doc in docs: assert doc.content_hash and doc.file_hash, doc.path contents = db.scalars( select(Chunk.content) .where(Chunk.document_id == doc.id) .order_by(Chunk.position) ).all() snapshot[str(doc.path)] = ( doc.content, doc.summary, str(doc.content_hash), str(doc.file_hash), [str(c) for c in contents], ) return snapshot def _sentinel_in_kb() -> bool: """Does the stored content of the sentinel document carry the sentinel? (The walk really indexed the fixture; the KB is not merely the right size.)""" with SessionLocal() as db: content = db.scalar( select(Document.content).where( Document.source == SOURCE_NAME, Document.path == SENTINEL_PATH ) ) return bool(content and SENTINEL in content) def _file_rows(page: Page) -> list[str]: """The current level's file rows, by path — the ls/tree rows the owner reads (a pruned document must leave this list).""" links = page.locator("#docs-tbody tr a.doc-link") return [str(links.nth(i).inner_text()) for i in range(links.count())] def _run_fresh_sync(page: Page, base_url: str, source: Path) -> dict[str, Any]: """Register the fixture and run ONE fresh-KB sync through the Sources page, returning the shape the cross-check compares.""" _add_local_source(page, base_url, source) body = _sync_from_sources_page(page, base_url) return { "detail": dict(body["detail"]), "line": _counts_line(page), "paths": _paths_with_counts(), "chunks_by_path": _chunk_count_by_path(), "total_chunks": _total_chunks(), "snapshot": _kb_snapshot(), } # --------------------------------------------------------------------------- # 1. The cadence ON syncs everything, cleanly, and the page agrees # --------------------------------------------------------------------------- def test_reset_cadence_produces_the_identical_run( page: Page, app_url: str, source_dir: Path, db_ready: None ) -> None: """``BOR_SYNC_SESSION_RESET_FILES=3`` over ten files: three mid-run releases — and a sync whose every owner-visible number is the number the unbounded run produces. The counts line reads ``10 added``, the terminal reports ``added=10`` with ``errors=0``, each of the ten paths exists EXACTLY ONCE (a release that discarded unflushed work would show up as a missing or a duplicated path), the ls/tree rows of both folders render all ten files, the stat cards equal the database's own ``select(count())`` over ``documents`` and ``chunks``, the sentinel's phrase is in the stored content — and not one ``/api/`` request answered 4xx/5xx or failed through the whole run.""" page.set_default_timeout(30_000) _seed_source(source_dir.parent) # idempotent — the tree is byte-stable login(page, app_url, next=SOURCES_URL) problems = _watch_network(page) leg = _run_fresh_sync(page, app_url, source_dir) # --- the owner's settle line + the terminal detail ------------------- assert leg["line"] == f"{N_FILES} added", leg["line"] detail = leg["detail"] assert detail["added"] == N_FILES, detail assert detail["updated"] == 0 and detail["unchanged"] == 0, detail assert detail["pruned"] == 0 and detail["errors"] == 0, detail assert detail["files"] == N_FILES, detail # --- every document landed EXACTLY once, fully embedded -------------- assert leg["paths"] == dict.fromkeys(ALL_PATHS, 1), leg["paths"] assert all(count > 0 for count in leg["chunks_by_path"].values()), ( f"a document has no chunks — the release lost its embed work: " f"{leg['chunks_by_path']}" ) assert leg["total_chunks"] > N_FILES, leg["total_chunks"] assert _sentinel_in_kb(), "the sentinel document never reached the KB" # --- the page agrees with the database (the phase-167 totals) -------- expect(page.locator("#stat-docs")).to_have_text(str(N_FILES), timeout=15_000) expect(page.locator("#stat-chunks")).to_have_text( str(leg["total_chunks"]), timeout=15_000 ) # --- the ls/tree rows render the whole fixture ----------------------- _open_level(page, SOURCE_NAME, "a") assert _file_rows(page) == [f"a/{n}.md" for n in range(1, 6)] _open_level(page, SOURCE_NAME, "b") assert _file_rows(page) == [f"b/{n}.md" for n in range(1, 6)] # --- the run never errored on the wire or in the UI ------------------ assert problems == [], f"the run put an error on the wire: {problems}" # Record for test 4's cross-check (the phase's byte-identical contract). _CADENCE_RUN.clear() _CADENCE_RUN.update(leg) # --------------------------------------------------------------------------- # 2. The unchanged re-sync with the cadence ON: nothing moves # --------------------------------------------------------------------------- def test_resync_is_unchanged_with_the_cadence_on( page: Page, app_url: str, source_dir: Path, db_ready: None ) -> None: """The run a production sync actually takes — every file unchanged — with the release firing three times through it: the phase-147 fast path stays the fast path. The second sync settles ``0 added · 10 unchanged`` with ``pruned=0``, the terminal is ``success``, and the database's chunk total and per-path chunk counts are byte-still: the cadence neither loses rows (which the next sync would report as ``added``) nor silently re-embeds an unchanged document.""" page.set_default_timeout(30_000) _seed_source(source_dir.parent) login(page, app_url, next=SOURCES_URL) problems = _watch_network(page) first = _run_fresh_sync(page, app_url, source_dir) assert first["detail"]["added"] == N_FILES, first["detail"] _expect_counts_line(page, f"{N_FILES} added") chunks_after_first = _chunk_count_by_path() paths_after_first = _paths_with_counts() total_after_first = _total_chunks() assert total_after_first == sum(chunks_after_first.values()) # --- the unchanged re-sync ------------------------------------------- second = _sync_from_sources_page(page, app_url) detail = second["detail"] assert detail["unchanged"] == N_FILES, detail assert detail["added"] == 0 and detail["updated"] == 0, detail assert detail["pruned"] == 0 and detail["errors"] == 0, detail _expect_counts_line(page, f"0 added · {N_FILES} unchanged") # No silent re-embed: the same chunk total, per path, after the run. assert _chunk_count_by_path() == chunks_after_first, ( "an unchanged file changed chunk count — the cadence re-embedded a " "document the fast path had left alone" ) assert _total_chunks() == total_after_first assert _paths_with_counts() == paths_after_first assert _sentinel_in_kb() assert problems == [], f"the re-sync put an error on the wire: {problems}" # --------------------------------------------------------------------------- # 3. The prune through the UI at BOR_SYNC_PRUNE_BATCH=2 # --------------------------------------------------------------------------- def test_multi_batch_prune_from_the_ui( page: Page, app_url: str, source_dir: Path, db_ready: None ) -> None: """Five documents go, and the prune that takes them is a MULTI-BATCH run: at ``BOR_SYNC_PRUNE_BATCH=2`` the five deletes commit at 2, at 4 and once more at the end — THREE commits inside the UI's own run — while the owner still sees exactly one prune report. The counts line settles ``0 added · 5 unchanged · 5 pruned``; the five documents are gone from the KB AND from the ls/tree rows of BOTH folders (three survivors in ``a``, two in ``b``); the survivors' chunk total is unchanged; and ZERO orphan chunk rows exist. A batched commit changes WHEN a delete becomes durable, never what the run removes.""" page.set_default_timeout(30_000) _seed_source(source_dir.parent) login(page, app_url, next=SOURCES_URL) problems = _watch_network(page) first = _run_fresh_sync(page, app_url, source_dir) assert first["detail"]["added"] == N_FILES, first["detail"] # The subject of the prune, captured BEFORE it happens: the doomed rows' # ids (the orphan query needs ids that no longer exist by the time it # runs) and every survivor's chunk count. doomed_ids = _doc_ids(GONE_PATHS) keep_counts = _chunk_count_by_path(KEEP_PATHS) assert len(doomed_ids) == len(GONE_PATHS), doomed_ids assert set(keep_counts) == set(KEEP_PATHS), keep_counts assert all(count > 0 for count in keep_counts.values()), keep_counts # --- the upstream deletion (this test owns the fixture tree) --------- for rel in GONE_PATHS: (source_dir / rel).unlink() assert not any((source_dir / rel).exists() for rel in GONE_PATHS) # --- the prune -------------------------------------------------------- body = _sync_from_sources_page(page, app_url) detail = body["detail"] assert detail["pruned"] == len(GONE_PATHS), detail assert detail["added"] == 0 and detail["updated"] == 0, detail assert detail["unchanged"] == len(KEEP_PATHS), detail assert detail["errors"] == 0, detail _expect_counts_line( page, f"0 added · {len(KEEP_PATHS)} unchanged · {len(GONE_PATHS)} pruned", ) # The catalogue rows went WITH the documents — in both folders, which is # where a prune that stopped after one batch would leave a stale row. _open_level(page, SOURCE_NAME, "a") assert _file_rows(page) == ["a/1.md", "a/2.md", "a/3.md"] _open_level(page, SOURCE_NAME, "b") assert _file_rows(page) == ["b/1.md", "b/2.md"] # The survivors are untouched and nothing was orphaned. assert _chunk_count_by_path() == keep_counts, ( "the prune took a survivor's chunks with it, or left a doomed row" ) assert _total_chunks() == sum(keep_counts.values()) assert _orphan_chunks() == 0, ( "a batched prune left orphan chunk rows — the deletes committed " "without the FK cascade doing the child work" ) assert _sentinel_in_kb() assert problems == [], f"the prune put an error on the wire: {problems}" # --------------------------------------------------------------------------- # 4. cadence 0 == cadence 3 (the byte-identical cross-check) # --------------------------------------------------------------------------- def test_cadence_zero_matches_cadence_three( page: Page, app_url: str, zero_app_url: str, source_dir: Path, db_ready: None, ) -> None: """THE PIN the phase lives on: the cadences change the run's memory profile and NOTHING the owner or the database can see. The same fixture tree, walked fresh by the ZERO app (``BOR_SYNC_SESSION_RESET_FILES=0``, ``BOR_SYNC_PRUNE_BATCH=0`` — the pre-phase-184 run) against the CADENCE app's numbers (``3`` / ``2``): the same paths (each exactly once), the same chunk count per path, the same ``content`` / ``summary`` / ``content_hash`` / ``file_hash`` on every document, the same chunk contents in position order, and the same counts line. The reference is test 1's recorded run when the module runs in order; run ALONE, the test walks the cadence-``3`` app itself first. Either way it asserts the reference's own 10-documents / >10-chunks shape BEFORE comparing, so an empty baseline can never make it pass vacuously.""" page.set_default_timeout(30_000) # Re-seed: test 3 deleted five files from the fixture tree. _seed_source(source_dir.parent) on_disk = sorted(p.relative_to(source_dir).as_posix() for p in source_dir.rglob("*.md")) assert on_disk == sorted(ALL_PATHS), on_disk # --- the reference leg: cadence 3 (test 1's numbers, or a fresh run) -- if _CADENCE_RUN: reference = dict(_CADENCE_RUN) else: login(page, app_url, next=SOURCES_URL) reference = _run_fresh_sync(page, app_url, source_dir) _truncate_all() assert reference["paths"] == dict.fromkeys(ALL_PATHS, 1), reference["paths"] assert reference["total_chunks"] > N_FILES, reference["total_chunks"] # --- the subject leg: cadence 0, the exact pre-phase run -------------- login(page, zero_app_url, next=SOURCES_URL) problems = _watch_network(page) zero = _run_fresh_sync(page, zero_app_url, source_dir) assert zero["detail"]["added"] == N_FILES, zero["detail"] assert zero["detail"]["errors"] == 0, zero["detail"] assert zero["detail"]["pruned"] == 0, zero["detail"] # --- every number agrees --------------------------------------------- assert zero["line"] == reference["line"], ( f"the owner-visible counts line moved with the cadence: " f"{zero['line']!r} != {reference['line']!r}" ) assert zero["paths"] == reference["paths"], ( "the set of paths (or a path's row count) moved with the cadence" ) assert zero["chunks_by_path"] == reference["chunks_by_path"], ( "the per-path chunk counts moved with the cadence: " f"{zero['chunks_by_path']} != {reference['chunks_by_path']}" ) assert zero["total_chunks"] == reference["total_chunks"] assert zero["snapshot"].keys() == reference["snapshot"].keys() for path, row in zero["snapshot"].items(): assert row == reference["snapshot"][path], ( f"{path} is not byte-identical across the cadences" ) assert _orphan_chunks() == 0 assert problems == [], f"the reference run put an error on the wire: {problems}"