phase: 184_sync_bounded_session
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure. **Verified / implemented (phase 184 final pass)** - All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries. - `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched. - `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact. **Tests / lint / coverage (all 100% green)** - `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%). - `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**. - `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**. - Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed. - `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings. **Completion criteria** 1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green 2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅ 3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅ 4. Every existing importer/sync/removal test green with unchanged expectations → ✅ 5. Full suite green, coverage >90%, ruff + pyright clean → ✅ 6. Phase E2E green in isolation → ✅ 7. README + AGENTS accurate/consistent → ✅ 8. Atomic commit — left to the harness (I ran no git write commands). **Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written). **Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched. **Next pending phase:** `185_sync_defer_content_fast_path`.
This commit is contained in:
@@ -0,0 +1,919 @@
|
||||
"""Phase 184 E2E (Playwright, mock-only): the BOUNDED sync still syncs all.
|
||||
|
||||
Source: owner incident 2026-10-15 — the same run whose prune phase 183 fixed.
|
||||
An import run holds ONE session and ``app/db.py`` builds it with
|
||||
``expire_on_commit=False``, so a commit released NOTHING: the walk kept every
|
||||
``Document`` it touched resident (each unchanged row holds its full ~15 KB
|
||||
``content``, and ``_unchanged_doc``'s ``file_hash`` stamp puts the row in the
|
||||
session's own strong ``dirty`` set, out of reach of the cycle collector), and
|
||||
``_prune`` read every document of every pruned source WHOLE — ``content``
|
||||
included — holding all of its deletes in ONE transaction until the end. The
|
||||
run's memory therefore grew with the SIZE OF THE KNOWLEDGE BASE instead of
|
||||
with the work the run did.
|
||||
|
||||
The fix is two operator knobs, both ``BOR_``-prefixed settings the importer
|
||||
reads once per RUN:
|
||||
|
||||
* ``BOR_SYNC_SESSION_RESET_FILES`` (default ``200``) — every Nth file of the
|
||||
walk, at the one point where no object from a finished file is reachable,
|
||||
the session calls ``expunge_all()``; it also runs once per source, before
|
||||
the two post-passes. ``0`` = NEVER release = the exact pre-phase-184 run.
|
||||
* ``BOR_SYNC_PRUNE_BATCH`` (default ``500``) — the prune commits every Nth
|
||||
delete instead of once at the end of the whole import. ``0`` = the exact
|
||||
pre-phase-184 single commit. A non-zero batch deliberately trades the
|
||||
all-or-nothing prune for a PREFIX (LOCKED A2: ``seen`` is recomputed from
|
||||
disk, so the next sync converges).
|
||||
|
||||
Both knobs change the run's MEMORY PROFILE and nothing else — and that
|
||||
"nothing else" is precisely what a browser can see. This module pins it: the
|
||||
same counts line, every document exactly once, an all-``unchanged`` re-sync,
|
||||
a multi-batch prune that behaves, and a Sources page that never stalls, never
|
||||
errors and reports the same totals as the database. The retention itself (the
|
||||
identity-map bound, the prune's six-column ``SELECT``, the commit counts) is
|
||||
deterministic and pinned in
|
||||
``tests/integration/test_import_session_retention.py``; RSS is asserted
|
||||
nowhere in this phase (phase 183 LOCKED A4, inherited).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is skipped — every number here (the
|
||||
per-path chunk counts, the byte-identical cross-check) is a property of the
|
||||
deterministic mock embedder/summarizer; against the real endpoint the
|
||||
comparison would be a measurement, not a pin.
|
||||
|
||||
TWO module apps on dedicated ports (the ``test_sync_summary_visibility.py``
|
||||
pattern — the conftest session app owns 8123 in a combined run, and a
|
||||
same-port second uvicorn would drive the wrong server):
|
||||
|
||||
* the CADENCE app — ``BOR_SYNC_SESSION_RESET_FILES=3`` and
|
||||
``BOR_SYNC_PRUNE_BATCH=2``, so a 10-file walk releases the session THREE
|
||||
times mid-run (before files 4, 7 and 10) plus once at the source boundary,
|
||||
and a 5-document prune commits THREE times (at 2, at 4, and the final one).
|
||||
Tests 1–3 drive it;
|
||||
* the ZERO app — both knobs ``0``, i.e. the exact pre-phase-184 run. Test 4
|
||||
drives it over the SAME fixture directory, which is what makes the two runs
|
||||
comparable at all.
|
||||
|
||||
Both boot against the mock LLM with ``BOR_GIT_SOURCES`` forced empty (the
|
||||
owner's local gitignored ``.env`` must never leak a real source into a run),
|
||||
their own ``BOR_SOURCES_DIR`` / ``BOR_PDF_PAGE_DIR`` under
|
||||
``tmp_path_factory`` (nothing here clones or renders a PDF — the dirs exist so
|
||||
no run writes into the operator's), and the mock-calibrated gate (the conftest
|
||||
convention). The fixture KB is ONE plain (non-git) directory named
|
||||
:data:`SOURCE_NAME` (``kind='local'`` rows index under ``Path(path).name``, so
|
||||
the source label is stable whatever temp root pytest picked) holding TEN
|
||||
``.md`` files across TWO folders — ``a/1..5.md`` and ``b/1..5.md`` — with
|
||||
``a/1.md`` carrying the sentinel phrase.
|
||||
|
||||
Isolation: the shared E2E Postgres is truncated BEFORE **and** AFTER every
|
||||
test (``chunks, documents, query_log, kb_overview, git_sources, sources_meta,
|
||||
folder_summaries``), after waiting for the app to hold no running sync (the
|
||||
run state lives in the app's memory, and a leaked run would keep importing
|
||||
into the next test's truncated KB). Each test registers its own source row.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_reset_cadence_produces_the_identical_run`` — the cadence ON is a
|
||||
complete, healthy sync: the counts line reads ``10 added``, the terminal
|
||||
reports ``added=10`` with ``errors=0``, every one of the ten paths exists
|
||||
EXACTLY ONCE in the KB (the test's own SQL), the ls/tree rows of both
|
||||
folders render all ten files, the stat cards equal ``select(count())`` over
|
||||
``documents`` (10) and ``chunks`` (> 10), the sentinel's phrase is in the
|
||||
stored content, and THROUGH THE RUN no ``/api/`` request answered 4xx/5xx
|
||||
and none failed — no error banner, no modal.
|
||||
2. ``test_resync_is_unchanged_with_the_cadence_on`` — the second sync of the
|
||||
same bytes settles ``0 added · 10 unchanged`` with ``pruned=0`` and
|
||||
``success``, and the chunk total and per-path chunk counts are byte-still:
|
||||
the cadence loses no rows (a lost row would come back as ``added``) and
|
||||
re-embeds nothing.
|
||||
3. ``test_multi_batch_prune_from_the_ui`` — five of the ten files go and the
|
||||
re-sync settles ``0 added · 5 unchanged · 5 pruned`` (5 deletes at batch
|
||||
``2`` = THREE commits inside the UI's own run); the five documents are gone
|
||||
from the KB AND from the ls/tree rows of BOTH folders, the survivors' chunk
|
||||
total is unchanged, and ZERO orphan chunk rows exist (the test's own SQL).
|
||||
4. ``test_cadence_zero_matches_cadence_three`` — the byte-identical
|
||||
cross-check the phase lives on: the SAME fixture walked with
|
||||
``BOR_SYNC_SESSION_RESET_FILES=0`` lands the same paths (each exactly
|
||||
once), the same per-path chunk counts, the same ``content`` / ``summary`` /
|
||||
``content_hash`` / ``file_hash`` on every document, the same chunk contents
|
||||
in position order and the same counts line as the cadence-``3`` run. The
|
||||
reference is test 1's recorded run when the module runs in order; alone,
|
||||
the test walks the cadence-``3`` app itself first — and either way it
|
||||
asserts its own 10-documents / >10-chunks shape BEFORE comparing, so an
|
||||
empty baseline can never make it pass vacuously.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import func, select, text
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
MOCK_PORT,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The CADENCE app (tests 1–3). Env-overridable, the house port pattern: the
|
||||
#: conftest session app owns 8123 in a combined run.
|
||||
CADENCE_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET", "8200"))
|
||||
|
||||
#: The ZERO app (test 4's reference leg): both knobs ``0`` — the exact
|
||||
#: pre-phase-184 run (LOCKED A3's no-redeploy rollback).
|
||||
ZERO_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET_ZERO", "8201"))
|
||||
|
||||
SOURCES_URL = "/sources.html"
|
||||
|
||||
#: The fixture directory's basename — the ``source`` the importer records for
|
||||
#: a ``kind='local'`` row (``Path(path).name``), so it is the same string in
|
||||
#: the DB whatever temp root this run got.
|
||||
SOURCE_NAME = "sync-session-reset"
|
||||
|
||||
#: The fixture layout: two folders × five importable files. After the prune,
|
||||
#: ``a`` keeps three survivors and ``b`` two — so BOTH folders still render
|
||||
#: rows, and a prune that stopped after its first batch would show up.
|
||||
ALL_PATHS = [*[f"a/{n}.md" for n in range(1, 6)], *[f"b/{n}.md" for n in range(1, 6)]]
|
||||
KEEP_PATHS = ["a/1.md", "a/2.md", "a/3.md", "b/1.md", "b/2.md"]
|
||||
GONE_PATHS = ["a/4.md", "a/5.md", "b/3.md", "b/4.md", "b/5.md"]
|
||||
N_FILES = len(ALL_PATHS)
|
||||
|
||||
#: The sentinel phrase, carried by :data:`SENTINEL_PATH` ONLY. Finding it in
|
||||
#: the stored ``content`` proves the walk really indexed the fixture rather
|
||||
#: than leaving a KB of merely the right SIZE.
|
||||
SENTINEL = "RESE-SYNC-RESET-SENTINEL-4f27"
|
||||
SENTINEL_PATH = "a/1.md"
|
||||
|
||||
#: The cadences the two apps boot with — module constants so the narrative and
|
||||
#: the app env cannot drift apart. 10 files / every 3rd = releases before
|
||||
#: files 4, 7 and 10 (THREE mid-run resets, then the source boundary); a
|
||||
#: 5-document prune at batch 2 = commits at 2, at 4 and the final one.
|
||||
RESET_FILES = 3
|
||||
PRUNE_BATCH = 2
|
||||
|
||||
#: "Synced HH:MM" — the local-time last-result label (sources.js's
|
||||
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
|
||||
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
|
||||
|
||||
#: Settle budgets: a ten-file mock sync (walk + embeds + overview + three
|
||||
#: folder summaries) lands in seconds and the UI settles a terminal within one
|
||||
#: 2 s poll tick, so this is an order of magnitude of head-room with no
|
||||
#: client-side hard timeout (the test_prune_cascade_cleanup budget).
|
||||
SETTLE_TIMEOUT_MS = 90_000
|
||||
|
||||
#: Test 1 records its numbers here for test 4's cross-check (the
|
||||
#: ``test_sync_changed_only.py`` "state from the earlier test" idiom). Test 4
|
||||
#: runs its own cadence leg when this is empty, so it stays honest alone.
|
||||
_CADENCE_RUN: dict[str, Any] = {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The fixture source (ONE directory BOTH apps walk)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _doc_body(path: str) -> str:
|
||||
"""One small markdown file per fixture path, byte-stable across runs.
|
||||
|
||||
The bodies differ from each other (so a per-path chunk count means
|
||||
something) but never carry a timestamp or the absolute path, and only
|
||||
:data:`SENTINEL_PATH` carries the sentinel — that is what lets test 4
|
||||
compare two runs of two different processes line for line.
|
||||
"""
|
||||
if path == SENTINEL_PATH:
|
||||
return (
|
||||
"# Session reset sentinel\n"
|
||||
"\n"
|
||||
"The bounded sync keeps the sentinel phrase in the knowledge base.\n"
|
||||
f"Marker: {SENTINEL}\n"
|
||||
)
|
||||
folder, name = path.rsplit("/", 1)
|
||||
stem = name.removesuffix(".md")
|
||||
return (
|
||||
f"# Session reset note {folder} {stem}\n"
|
||||
"\n"
|
||||
f"One unrelated note from the {folder} folder of the {SOURCE_NAME} "
|
||||
"fixture source, indexed so the cadence has something to walk.\n"
|
||||
)
|
||||
|
||||
|
||||
def _seed_source(root: Path) -> Path:
|
||||
"""(Re-)write all ten fixture files under ``root / SOURCE_NAME``.
|
||||
|
||||
Idempotent on purpose: test 3 deletes five files from the tree and test 4
|
||||
re-seeds the SAME directory, so the reference leg walks byte-identical
|
||||
content through the other app.
|
||||
"""
|
||||
base = root / SOURCE_NAME
|
||||
for rel in ALL_PATHS:
|
||||
target = base / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(_doc_body(rel), encoding="utf-8")
|
||||
assert not (base / ".git").exists() # a plain directory — no git anywhere
|
||||
return base
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one fixture tree BOTH module apps walk (the app servers run on the
|
||||
same host, so the path is visible to them)."""
|
||||
return _seed_source(tmp_path_factory.mktemp("bor_sync_session_reset"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The two module apps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(scratch: Path, reset_files: int, prune_batch: int) -> dict[str, str]:
|
||||
"""The per-module app env (the conftest pattern): the mock LLM, the
|
||||
mock-calibrated gate, no git fallback, dedicated scratch dirs — and THIS
|
||||
app's two phase-184 cadences, pinned explicitly so an operator's local
|
||||
(gitignored) ``.env`` cannot retune them."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{MOCK_PORT}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (the conftest pattern): the token-overlap mock
|
||||
# embeddings score far below the production 0.62, and BOTH floors must
|
||||
# stay <= the threshold or the app refuses to boot (app/config.py).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# No env-fallback rows: the registry each test writes through the
|
||||
# authenticated API is the only source list, so every sync's counts are
|
||||
# exactly this test's doing.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages")
|
||||
# THE SUBJECT OF THE PHASE.
|
||||
env["BOR_SYNC_SESSION_RESET_FILES"] = str(reset_files)
|
||||
env["BOR_SYNC_PRUNE_BATCH"] = str(prune_batch)
|
||||
return env
|
||||
|
||||
|
||||
def _serve(env: dict[str, str], port: int) -> Iterator[str]:
|
||||
"""Boot one uvicorn app under test on *port* and yield its URL."""
|
||||
proc = subprocess.Popen(
|
||||
[
|
||||
sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning",
|
||||
],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
try:
|
||||
_wait_http(f"{url}/api/health")
|
||||
yield url
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The CADENCE app (tests 1–3): ``reset=3``, ``prune_batch=2``.
|
||||
|
||||
``mock_llm`` and ``source_dir`` are dependencies for the ORDERING only —
|
||||
the mock answers before the app boots, and the fixture tree exists before
|
||||
the first test registers it.
|
||||
"""
|
||||
if USE_REAL_LLM:
|
||||
pytest.skip(
|
||||
"mock-only suite — the byte-identical cross-check is a property "
|
||||
"of the deterministic mock (E2E_REAL_LLM unsupported)"
|
||||
)
|
||||
assert mock_llm and source_dir
|
||||
scratch = tmp_path_factory.mktemp("bor_sync_session_reset_cadence")
|
||||
yield from _serve(_app_env(scratch, RESET_FILES, PRUNE_BATCH), CADENCE_PORT)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zero_app_server(
|
||||
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The ZERO app (test 4's reference leg): both knobs ``0``, the exact
|
||||
pre-phase-184 run the bounded one must be byte-identical to."""
|
||||
if USE_REAL_LLM:
|
||||
pytest.skip(
|
||||
"mock-only suite — the byte-identical cross-check is a property "
|
||||
"of the deterministic mock (E2E_REAL_LLM unsupported)"
|
||||
)
|
||||
assert mock_llm and source_dir
|
||||
scratch = tmp_path_factory.mktemp("bor_sync_session_reset_zero")
|
||||
yield from _serve(_app_env(scratch, 0, 0), ZERO_PORT)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zero_app_url(zero_app_server: str) -> str:
|
||||
return zero_app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Isolation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern). ``sources_meta`` (the
|
||||
KB generation counter) and ``folder_summaries`` reset with the KB: the
|
||||
sync's overview/bump is an upsert, and a leftover folder summary would
|
||||
ride the very ``changed_paths`` feed the prune writes."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
|
||||
"""No run may leak across a truncate: wait for a non-running status with
|
||||
an own admin session (the endpoint is admin-only). Usually a no-op — the
|
||||
tests settle before they end."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
||||
"""Per-test KB isolation: the shared E2E Postgres is truncated before AND
|
||||
after every test, the app holding no running sync at both boundaries.
|
||||
|
||||
The ZERO app is deliberately NOT waited on: it is booting lazily for test 4
|
||||
only, and that test's own run is settled inside the test body, so the
|
||||
post-truncate can never race it."""
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
yield
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers: the UI side
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _watch_network(page: Page) -> list[str]:
|
||||
"""Collect every ``/api/`` 4xx/5xx and every failed ``/api/`` request the
|
||||
page sees — the owner's "the Sources page errored / stalled".
|
||||
|
||||
Returns the (growing) list the caller asserts empty at the end of the
|
||||
run: the page's own 2 s status poll lands in this list, so a run that made
|
||||
the API answer 500 is caught here rather than at some later assertion.
|
||||
"""
|
||||
problems: list[str] = []
|
||||
|
||||
def _on_response(response: Any) -> None:
|
||||
if "/api/" in response.url and response.status >= 400:
|
||||
problems.append(f"HTTP {response.status} {response.url}")
|
||||
|
||||
def _on_failed(request: Any) -> None:
|
||||
if "/api/" in request.url:
|
||||
problems.append(f"FAILED {request.url} ({request.failure})")
|
||||
|
||||
page.on("response", _on_response)
|
||||
page.on("requestfailed", _on_failed)
|
||||
return problems
|
||||
|
||||
|
||||
def _add_local_source(page: Page, base_url: str, path: Path) -> None:
|
||||
"""Register the fixture directory through the authenticated API (the
|
||||
phase-49 shape — the page's own form for this is gone; the admin cookie
|
||||
rides ``page.request``)."""
|
||||
r = page.request.post(
|
||||
f"{base_url}/api/git-sources",
|
||||
data={"kind": "local", "path": str(path)},
|
||||
)
|
||||
assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}"
|
||||
|
||||
|
||||
def _started_at(body: dict[str, Any]) -> float:
|
||||
"""The status body's ``started_at`` as a wall-clock epoch (the server
|
||||
stamps it in UTC and this process runs on the same host)."""
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return 0.0
|
||||
try:
|
||||
return datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _sync_from_sources_page(page: Page, base_url: str) -> dict[str, Any]:
|
||||
"""Click the Sources page **Sync sources** button and wait for THIS run's
|
||||
terminal state (the module app is shared, so a terminal counts only after
|
||||
a ``running`` tick or when its ``started_at`` post-dates the click).
|
||||
|
||||
Settles the UI on success and proves the owner saw no error surface: the
|
||||
button re-enabled, ``Synced HH:MM``, no error banner, no modal."""
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(btn).to_be_enabled()
|
||||
clicked_at = time.time()
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
|
||||
deadline = time.monotonic() + SETTLE_TIMEOUT_MS / 1000
|
||||
saw_running = False
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = page.request.get(f"{base_url}/api/sync/status")
|
||||
assert r.status == 200, r.text
|
||||
body = r.json()
|
||||
state = body["state"]
|
||||
if state == "running":
|
||||
saw_running = True
|
||||
elif state in ("success", "failed", "cancelled") and (
|
||||
saw_running or _started_at(body) >= clicked_at - 1.0
|
||||
):
|
||||
break
|
||||
time.sleep(0.2)
|
||||
else:
|
||||
raise AssertionError(f"sync never reached a terminal state: {body}")
|
||||
|
||||
assert body["state"] == "success", f"sync failed: {body}"
|
||||
assert body["error"] is None, body
|
||||
expect(btn).to_be_enabled(timeout=15_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
|
||||
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
||||
assert page.locator(".sync-modal-backdrop").count() == 0
|
||||
return body
|
||||
|
||||
|
||||
def _expect_counts_line(page: Page, expected: str) -> None:
|
||||
"""The counts line — the owner-visible sync report on the single sync text
|
||||
surface (phase 150; sources.js fmtSyncResult)."""
|
||||
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
|
||||
|
||||
|
||||
def _counts_line(page: Page) -> str:
|
||||
"""The counts line as text — the string test 4 compares across the apps."""
|
||||
return str(page.locator("#sync-progress-text").inner_text())
|
||||
|
||||
|
||||
def _open_level(page: Page, *names: str) -> None:
|
||||
"""Open ONE level at a time from the catalogue top — each name the EXACT
|
||||
link text of the source or folder row at the current level (the phase-97
|
||||
drill idiom, the ``test_kb_tree_lazy_load.py`` settle).
|
||||
|
||||
The top is reached through the breadcrumb's first segment when the page is
|
||||
drilled (the phase-99 drill state rides ``history.state``, so a nav
|
||||
re-click would KEEP the level — the crumb is the control that actually
|
||||
leaves it), and is a no-op when the page already sits there (the crumb
|
||||
hidden). Every click after that awaits the level it produced: the phase-167
|
||||
node fetch is over once the breadcrumb names the level and no in-flight
|
||||
``.kb-level-loading`` row remains."""
|
||||
if not page.locator("#kb-crumb").is_hidden():
|
||||
page.click('#kb-crumb a.kb-crumb-link:text-is("Knowledge base")')
|
||||
expect(page.locator("#kb-crumb")).to_be_hidden(timeout=30_000)
|
||||
expect(page.locator("#folders-tbody tr")).to_have_count(1, timeout=30_000)
|
||||
for name in names:
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text(
|
||||
name, timeout=30_000
|
||||
)
|
||||
expect(page.locator(".kb-level-loading")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers: the DB side
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _paths_with_counts() -> dict[str, int]:
|
||||
"""``path`` → number of ``documents`` rows for that path — THE exactly-once
|
||||
probe: a reset that dropped work and re-added it shows up here."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(Document.path, func.count(Document.id))
|
||||
.where(Document.source == SOURCE_NAME)
|
||||
.group_by(Document.path)
|
||||
).all()
|
||||
return {str(path): int(count) for path, count in rows}
|
||||
|
||||
|
||||
def _chunk_count_by_path(paths: list[str] | None = None) -> dict[str, int]:
|
||||
"""``path`` → chunk rows per document (an absent path is not listed)."""
|
||||
query = (
|
||||
select(Document.path, func.count(Chunk.id))
|
||||
.outerjoin(Chunk, Chunk.document_id == Document.id)
|
||||
.where(Document.source == SOURCE_NAME)
|
||||
.group_by(Document.path)
|
||||
)
|
||||
if paths is not None:
|
||||
query = query.where(Document.path.in_(paths))
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(query).all()
|
||||
return {str(path): int(count) for path, count in rows}
|
||||
|
||||
|
||||
def _total_chunks() -> int:
|
||||
with SessionLocal() as db:
|
||||
return int(db.scalar(select(func.count()).select_from(Chunk)) or 0)
|
||||
|
||||
|
||||
def _doc_ids(paths: list[str]) -> list[str]:
|
||||
"""The indexed document ids of ``(SOURCE_NAME, path)``, sorted. Test 3
|
||||
needs them BEFORE the prune — the orphan query runs on ids that no longer
|
||||
exist afterwards."""
|
||||
with SessionLocal() as db:
|
||||
return sorted(
|
||||
str(doc_id)
|
||||
for (doc_id,) in db.execute(
|
||||
select(Document.id).where(
|
||||
Document.source == SOURCE_NAME, Document.path.in_(paths)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
|
||||
|
||||
def _orphan_chunks() -> int:
|
||||
"""Chunk rows whose document is gone — phase 183's contract, asserted here
|
||||
too so a BATCHED prune cannot leave any behind."""
|
||||
with SessionLocal() as db:
|
||||
return int(
|
||||
db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.where(Chunk.document_id.not_in(select(Document.id)))
|
||||
)
|
||||
or 0
|
||||
)
|
||||
|
||||
|
||||
def _kb_snapshot() -> dict[str, tuple[str, str | None, str, str, list[str]]]:
|
||||
"""The byte-level shape of the fixture source:
|
||||
|
||||
``path`` → ``(content, summary, content_hash, file_hash, chunk contents in
|
||||
position order)`` — every field the two runs must agree on. Timestamps and
|
||||
``full_path`` are deliberately absent: they belong to the run and to the
|
||||
temp root, not to the imported bytes.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
docs = db.scalars(
|
||||
select(Document)
|
||||
.where(Document.source == SOURCE_NAME)
|
||||
.order_by(Document.path)
|
||||
).all()
|
||||
snapshot: dict[str, tuple[str, str | None, str, str, list[str]]] = {}
|
||||
for doc in docs:
|
||||
assert doc.content_hash and doc.file_hash, doc.path
|
||||
contents = db.scalars(
|
||||
select(Chunk.content)
|
||||
.where(Chunk.document_id == doc.id)
|
||||
.order_by(Chunk.position)
|
||||
).all()
|
||||
snapshot[str(doc.path)] = (
|
||||
doc.content,
|
||||
doc.summary,
|
||||
str(doc.content_hash),
|
||||
str(doc.file_hash),
|
||||
[str(c) for c in contents],
|
||||
)
|
||||
return snapshot
|
||||
|
||||
|
||||
def _sentinel_in_kb() -> bool:
|
||||
"""Does the stored content of the sentinel document carry the sentinel?
|
||||
(The walk really indexed the fixture; the KB is not merely the right size.)"""
|
||||
with SessionLocal() as db:
|
||||
content = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == SOURCE_NAME, Document.path == SENTINEL_PATH
|
||||
)
|
||||
)
|
||||
return bool(content and SENTINEL in content)
|
||||
|
||||
|
||||
def _file_rows(page: Page) -> list[str]:
|
||||
"""The current level's file rows, by path — the ls/tree rows the owner
|
||||
reads (a pruned document must leave this list)."""
|
||||
links = page.locator("#docs-tbody tr a.doc-link")
|
||||
return [str(links.nth(i).inner_text()) for i in range(links.count())]
|
||||
|
||||
|
||||
def _run_fresh_sync(page: Page, base_url: str, source: Path) -> dict[str, Any]:
|
||||
"""Register the fixture and run ONE fresh-KB sync through the Sources
|
||||
page, returning the shape the cross-check compares."""
|
||||
_add_local_source(page, base_url, source)
|
||||
body = _sync_from_sources_page(page, base_url)
|
||||
return {
|
||||
"detail": dict(body["detail"]),
|
||||
"line": _counts_line(page),
|
||||
"paths": _paths_with_counts(),
|
||||
"chunks_by_path": _chunk_count_by_path(),
|
||||
"total_chunks": _total_chunks(),
|
||||
"snapshot": _kb_snapshot(),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The cadence ON syncs everything, cleanly, and the page agrees
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_reset_cadence_produces_the_identical_run(
|
||||
page: Page, app_url: str, source_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""``BOR_SYNC_SESSION_RESET_FILES=3`` over ten files: three mid-run
|
||||
releases — and a sync whose every owner-visible number is the number the
|
||||
unbounded run produces.
|
||||
|
||||
The counts line reads ``10 added``, the terminal reports ``added=10`` with
|
||||
``errors=0``, each of the ten paths exists EXACTLY ONCE (a release that
|
||||
discarded unflushed work would show up as a missing or a duplicated path),
|
||||
the ls/tree rows of both folders render all ten files, the stat cards
|
||||
equal the database's own ``select(count())`` over ``documents`` and
|
||||
``chunks``, the sentinel's phrase is in the stored content — and not one
|
||||
``/api/`` request answered 4xx/5xx or failed through the whole run."""
|
||||
page.set_default_timeout(30_000)
|
||||
_seed_source(source_dir.parent) # idempotent — the tree is byte-stable
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
|
||||
leg = _run_fresh_sync(page, app_url, source_dir)
|
||||
|
||||
# --- the owner's settle line + the terminal detail -------------------
|
||||
assert leg["line"] == f"{N_FILES} added", leg["line"]
|
||||
detail = leg["detail"]
|
||||
assert detail["added"] == N_FILES, detail
|
||||
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
|
||||
assert detail["pruned"] == 0 and detail["errors"] == 0, detail
|
||||
assert detail["files"] == N_FILES, detail
|
||||
|
||||
# --- every document landed EXACTLY once, fully embedded --------------
|
||||
assert leg["paths"] == dict.fromkeys(ALL_PATHS, 1), leg["paths"]
|
||||
assert all(count > 0 for count in leg["chunks_by_path"].values()), (
|
||||
f"a document has no chunks — the release lost its embed work: "
|
||||
f"{leg['chunks_by_path']}"
|
||||
)
|
||||
assert leg["total_chunks"] > N_FILES, leg["total_chunks"]
|
||||
assert _sentinel_in_kb(), "the sentinel document never reached the KB"
|
||||
|
||||
# --- the page agrees with the database (the phase-167 totals) --------
|
||||
expect(page.locator("#stat-docs")).to_have_text(str(N_FILES), timeout=15_000)
|
||||
expect(page.locator("#stat-chunks")).to_have_text(
|
||||
str(leg["total_chunks"]), timeout=15_000
|
||||
)
|
||||
|
||||
# --- the ls/tree rows render the whole fixture -----------------------
|
||||
_open_level(page, SOURCE_NAME, "a")
|
||||
assert _file_rows(page) == [f"a/{n}.md" for n in range(1, 6)]
|
||||
_open_level(page, SOURCE_NAME, "b")
|
||||
assert _file_rows(page) == [f"b/{n}.md" for n in range(1, 6)]
|
||||
|
||||
# --- the run never errored on the wire or in the UI ------------------
|
||||
assert problems == [], f"the run put an error on the wire: {problems}"
|
||||
|
||||
# Record for test 4's cross-check (the phase's byte-identical contract).
|
||||
_CADENCE_RUN.clear()
|
||||
_CADENCE_RUN.update(leg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The unchanged re-sync with the cadence ON: nothing moves
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_resync_is_unchanged_with_the_cadence_on(
|
||||
page: Page, app_url: str, source_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""The run a production sync actually takes — every file unchanged — with
|
||||
the release firing three times through it: the phase-147 fast path stays
|
||||
the fast path.
|
||||
|
||||
The second sync settles ``0 added · 10 unchanged`` with ``pruned=0``, the
|
||||
terminal is ``success``, and the database's chunk total and per-path chunk
|
||||
counts are byte-still: the cadence neither loses rows (which the next sync
|
||||
would report as ``added``) nor silently re-embeds an unchanged document."""
|
||||
page.set_default_timeout(30_000)
|
||||
_seed_source(source_dir.parent)
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
|
||||
first = _run_fresh_sync(page, app_url, source_dir)
|
||||
assert first["detail"]["added"] == N_FILES, first["detail"]
|
||||
_expect_counts_line(page, f"{N_FILES} added")
|
||||
chunks_after_first = _chunk_count_by_path()
|
||||
paths_after_first = _paths_with_counts()
|
||||
total_after_first = _total_chunks()
|
||||
assert total_after_first == sum(chunks_after_first.values())
|
||||
|
||||
# --- the unchanged re-sync -------------------------------------------
|
||||
second = _sync_from_sources_page(page, app_url)
|
||||
detail = second["detail"]
|
||||
assert detail["unchanged"] == N_FILES, detail
|
||||
assert detail["added"] == 0 and detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0 and detail["errors"] == 0, detail
|
||||
_expect_counts_line(page, f"0 added · {N_FILES} unchanged")
|
||||
|
||||
# No silent re-embed: the same chunk total, per path, after the run.
|
||||
assert _chunk_count_by_path() == chunks_after_first, (
|
||||
"an unchanged file changed chunk count — the cadence re-embedded a "
|
||||
"document the fast path had left alone"
|
||||
)
|
||||
assert _total_chunks() == total_after_first
|
||||
assert _paths_with_counts() == paths_after_first
|
||||
assert _sentinel_in_kb()
|
||||
assert problems == [], f"the re-sync put an error on the wire: {problems}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The prune through the UI at BOR_SYNC_PRUNE_BATCH=2
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_multi_batch_prune_from_the_ui(
|
||||
page: Page, app_url: str, source_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Five documents go, and the prune that takes them is a MULTI-BATCH run:
|
||||
at ``BOR_SYNC_PRUNE_BATCH=2`` the five deletes commit at 2, at 4 and once
|
||||
more at the end — THREE commits inside the UI's own run — while the owner
|
||||
still sees exactly one prune report.
|
||||
|
||||
The counts line settles ``0 added · 5 unchanged · 5 pruned``; the five
|
||||
documents are gone from the KB AND from the ls/tree rows of BOTH folders
|
||||
(three survivors in ``a``, two in ``b``); the survivors' chunk total is
|
||||
unchanged; and ZERO orphan chunk rows exist. A batched commit changes WHEN
|
||||
a delete becomes durable, never what the run removes."""
|
||||
page.set_default_timeout(30_000)
|
||||
_seed_source(source_dir.parent)
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
|
||||
first = _run_fresh_sync(page, app_url, source_dir)
|
||||
assert first["detail"]["added"] == N_FILES, first["detail"]
|
||||
|
||||
# The subject of the prune, captured BEFORE it happens: the doomed rows'
|
||||
# ids (the orphan query needs ids that no longer exist by the time it
|
||||
# runs) and every survivor's chunk count.
|
||||
doomed_ids = _doc_ids(GONE_PATHS)
|
||||
keep_counts = _chunk_count_by_path(KEEP_PATHS)
|
||||
assert len(doomed_ids) == len(GONE_PATHS), doomed_ids
|
||||
assert set(keep_counts) == set(KEEP_PATHS), keep_counts
|
||||
assert all(count > 0 for count in keep_counts.values()), keep_counts
|
||||
|
||||
# --- the upstream deletion (this test owns the fixture tree) ---------
|
||||
for rel in GONE_PATHS:
|
||||
(source_dir / rel).unlink()
|
||||
assert not any((source_dir / rel).exists() for rel in GONE_PATHS)
|
||||
|
||||
# --- the prune --------------------------------------------------------
|
||||
body = _sync_from_sources_page(page, app_url)
|
||||
detail = body["detail"]
|
||||
assert detail["pruned"] == len(GONE_PATHS), detail
|
||||
assert detail["added"] == 0 and detail["updated"] == 0, detail
|
||||
assert detail["unchanged"] == len(KEEP_PATHS), detail
|
||||
assert detail["errors"] == 0, detail
|
||||
_expect_counts_line(
|
||||
page,
|
||||
f"0 added · {len(KEEP_PATHS)} unchanged · {len(GONE_PATHS)} pruned",
|
||||
)
|
||||
|
||||
# The catalogue rows went WITH the documents — in both folders, which is
|
||||
# where a prune that stopped after one batch would leave a stale row.
|
||||
_open_level(page, SOURCE_NAME, "a")
|
||||
assert _file_rows(page) == ["a/1.md", "a/2.md", "a/3.md"]
|
||||
_open_level(page, SOURCE_NAME, "b")
|
||||
assert _file_rows(page) == ["b/1.md", "b/2.md"]
|
||||
|
||||
# The survivors are untouched and nothing was orphaned.
|
||||
assert _chunk_count_by_path() == keep_counts, (
|
||||
"the prune took a survivor's chunks with it, or left a doomed row"
|
||||
)
|
||||
assert _total_chunks() == sum(keep_counts.values())
|
||||
assert _orphan_chunks() == 0, (
|
||||
"a batched prune left orphan chunk rows — the deletes committed "
|
||||
"without the FK cascade doing the child work"
|
||||
)
|
||||
assert _sentinel_in_kb()
|
||||
assert problems == [], f"the prune put an error on the wire: {problems}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. cadence 0 == cadence 3 (the byte-identical cross-check)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cadence_zero_matches_cadence_three(
|
||||
page: Page,
|
||||
app_url: str,
|
||||
zero_app_url: str,
|
||||
source_dir: Path,
|
||||
db_ready: None,
|
||||
) -> None:
|
||||
"""THE PIN the phase lives on: the cadences change the run's memory
|
||||
profile and NOTHING the owner or the database can see.
|
||||
|
||||
The same fixture tree, walked fresh by the ZERO app
|
||||
(``BOR_SYNC_SESSION_RESET_FILES=0``, ``BOR_SYNC_PRUNE_BATCH=0`` — the
|
||||
pre-phase-184 run) against the CADENCE app's numbers (``3`` / ``2``): the
|
||||
same paths (each exactly once), the same chunk count per path, the same
|
||||
``content`` / ``summary`` / ``content_hash`` / ``file_hash`` on every
|
||||
document, the same chunk contents in position order, and the same counts
|
||||
line.
|
||||
|
||||
The reference is test 1's recorded run when the module runs in order; run
|
||||
ALONE, the test walks the cadence-``3`` app itself first. Either way it
|
||||
asserts the reference's own 10-documents / >10-chunks shape BEFORE
|
||||
comparing, so an empty baseline can never make it pass vacuously."""
|
||||
page.set_default_timeout(30_000)
|
||||
|
||||
# Re-seed: test 3 deleted five files from the fixture tree.
|
||||
_seed_source(source_dir.parent)
|
||||
on_disk = sorted(p.relative_to(source_dir).as_posix() for p in source_dir.rglob("*.md"))
|
||||
assert on_disk == sorted(ALL_PATHS), on_disk
|
||||
|
||||
# --- the reference leg: cadence 3 (test 1's numbers, or a fresh run) --
|
||||
if _CADENCE_RUN:
|
||||
reference = dict(_CADENCE_RUN)
|
||||
else:
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
reference = _run_fresh_sync(page, app_url, source_dir)
|
||||
_truncate_all()
|
||||
assert reference["paths"] == dict.fromkeys(ALL_PATHS, 1), reference["paths"]
|
||||
assert reference["total_chunks"] > N_FILES, reference["total_chunks"]
|
||||
|
||||
# --- the subject leg: cadence 0, the exact pre-phase run --------------
|
||||
login(page, zero_app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
zero = _run_fresh_sync(page, zero_app_url, source_dir)
|
||||
|
||||
assert zero["detail"]["added"] == N_FILES, zero["detail"]
|
||||
assert zero["detail"]["errors"] == 0, zero["detail"]
|
||||
assert zero["detail"]["pruned"] == 0, zero["detail"]
|
||||
|
||||
# --- every number agrees ---------------------------------------------
|
||||
assert zero["line"] == reference["line"], (
|
||||
f"the owner-visible counts line moved with the cadence: "
|
||||
f"{zero['line']!r} != {reference['line']!r}"
|
||||
)
|
||||
assert zero["paths"] == reference["paths"], (
|
||||
"the set of paths (or a path's row count) moved with the cadence"
|
||||
)
|
||||
assert zero["chunks_by_path"] == reference["chunks_by_path"], (
|
||||
"the per-path chunk counts moved with the cadence: "
|
||||
f"{zero['chunks_by_path']} != {reference['chunks_by_path']}"
|
||||
)
|
||||
assert zero["total_chunks"] == reference["total_chunks"]
|
||||
assert zero["snapshot"].keys() == reference["snapshot"].keys()
|
||||
for path, row in zero["snapshot"].items():
|
||||
assert row == reference["snapshot"][path], (
|
||||
f"{path} is not byte-identical across the cadences"
|
||||
)
|
||||
assert _orphan_chunks() == 0
|
||||
assert problems == [], f"the reference run put an error on the wire: {problems}"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1287,12 +1287,16 @@ def test_backfill_never_touches_standalone_and_copyless_rows(db, tmp_path) -> No
|
||||
assert s.image_context_backfilled == 0
|
||||
assert s.images_failed == 0 and s.errors == 0
|
||||
assert llm.text_parts == [] # zero describe calls, both runs
|
||||
db.expire(standalone)
|
||||
db.expire(missing_copy)
|
||||
assert standalone.image_context is None
|
||||
assert standalone.content == DESCRIPTION
|
||||
assert missing_copy.image_context is None
|
||||
assert missing_copy.content == DESCRIPTION
|
||||
# Phase 184: an import run is now ALLOWED to release the session it
|
||||
# walks (the identity-map cadence expunges every object at the source
|
||||
# boundary), so the two seeded rows are re-read by key instead of
|
||||
# expiring the objects the run detached. Same assertion, one level
|
||||
# closer to the database — nothing about the run's output changed.
|
||||
for row in (standalone, missing_copy):
|
||||
reloaded = db.get(Document, row.id)
|
||||
assert reloaded is not None
|
||||
assert reloaded.image_context is None
|
||||
assert reloaded.content == DESCRIPTION
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
@@ -906,3 +906,58 @@ def test_ui_customization_env_overrides(monkeypatch: pytest.MonkeyPatch) -> None
|
||||
assert s.footer_text == "Powered by my own models"
|
||||
monkeypatch.setenv("BOR_INPUT_PLACEHOLDER", "")
|
||||
assert _settings().input_placeholder == "" # empty stands
|
||||
|
||||
|
||||
# --- Sync session bounds (phase 184) ---
|
||||
|
||||
|
||||
def test_sync_session_reset_files_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 184 (LOCKED A3): the import walk releases its session every
|
||||
``BOR_SYNC_SESSION_RESET_FILES`` files (default 200). ``0`` is accepted
|
||||
and means NEVER release — the exact pre-phase-184 behaviour, the
|
||||
operator's no-redeploy rollback."""
|
||||
monkeypatch.delenv("BOR_SYNC_SESSION_RESET_FILES", raising=False)
|
||||
assert _settings().sync_session_reset_files == 200
|
||||
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "50")
|
||||
assert _settings().sync_session_reset_files == 50
|
||||
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "0")
|
||||
assert _settings().sync_session_reset_files == 0
|
||||
|
||||
|
||||
def test_sync_session_reset_files_rejects_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A negative cadence is a typo — a ``%`` against it never fires, so
|
||||
startup refuses naming the field (the ``agent_max_rounds`` pattern,
|
||||
phase 184)."""
|
||||
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_session_reset_files"):
|
||||
_settings()
|
||||
|
||||
|
||||
def test_sync_prune_batch_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 184 (LOCKED A3): the prune commits every
|
||||
``BOR_SYNC_PRUNE_BATCH`` deleted documents (default 500). ``0`` is
|
||||
accepted and means ONE commit at the end of the whole prune — the
|
||||
exact pre-phase-184 behaviour."""
|
||||
monkeypatch.delenv("BOR_SYNC_PRUNE_BATCH", raising=False)
|
||||
assert _settings().sync_prune_batch == 500
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "25")
|
||||
assert _settings().sync_prune_batch == 25
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "0")
|
||||
assert _settings().sync_prune_batch == 0
|
||||
|
||||
|
||||
def test_sync_prune_batch_rejects_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A negative batch size is a typo — the batch comparison would never
|
||||
fire, so startup refuses naming the field (the ``agent_max_rounds``
|
||||
pattern, phase 184)."""
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_prune_batch"):
|
||||
_settings()
|
||||
|
||||
@@ -188,6 +188,41 @@ def test_verbatim_display_field_types() -> None:
|
||||
assert Settings.model_fields["show_text_max_blocks"].default == 5
|
||||
|
||||
|
||||
def test_sync_session_bound_rows_are_plain_live_overridable() -> None:
|
||||
"""Phase 184 (task 01, LOCKED A3/D2): the two sync-run knobs sit in
|
||||
``Import & sources`` (beside ``upload_max_mb`` — they tune the sync, not
|
||||
the walk's content) with the ``Sync session reset files`` / ``Sync prune
|
||||
batch`` labels and are plain — not ``secret`` (two ints), not
|
||||
``restart_required`` (the importer reads ``llm.settings`` per run, so a
|
||||
saved override applies to the NEXT sync), overridable (the overlay is
|
||||
generic over ``int`` fields). Both accept ``0`` = the exact pre-phase
|
||||
behaviour (the no-redeploy rollback); a negative value fails startup,
|
||||
not the panel."""
|
||||
for key, label in (
|
||||
("sync_session_reset_files", "Sync session reset files"),
|
||||
("sync_prune_batch", "Sync prune batch"),
|
||||
):
|
||||
meta = CATALOG[key]
|
||||
assert meta.group == "Import & sources"
|
||||
assert meta.label == label
|
||||
assert meta.secret is False
|
||||
assert meta.restart_required is False
|
||||
assert meta.overridable is True
|
||||
assert key not in SECRETS
|
||||
assert key not in RESTART_REQUIRED
|
||||
assert key not in NON_OVERRIDABLE
|
||||
|
||||
|
||||
def test_sync_session_bound_field_types() -> None:
|
||||
"""Phase 184 (task 01, D2): both knobs are ``int``s with the LOCKED A3
|
||||
defaults (``200`` files / ``500`` documents) — the type the panel renders
|
||||
its input for and the overlay coerces from the stored raw string."""
|
||||
assert Settings.model_fields["sync_session_reset_files"].annotation is int
|
||||
assert Settings.model_fields["sync_prune_batch"].annotation is int
|
||||
assert Settings.model_fields["sync_session_reset_files"].default == 200
|
||||
assert Settings.model_fields["sync_prune_batch"].default == 500
|
||||
|
||||
|
||||
def test_labels_nonempty_and_unique() -> None:
|
||||
labels = [meta.label for meta in CATALOG.values()]
|
||||
assert all(label.strip() for label in labels), "empty label in the catalogue"
|
||||
|
||||
Reference in New Issue
Block a user