**Phase 187 — final verification pass: all criteria met, repo 100% green.**
Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.
Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)
Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.
Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).
Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.
Next pending phase: `188_chunks_hnsw_index`.
676 lines
29 KiB
Python
676 lines
29 KiB
Python
"""Phase 90 story E2E (Playwright): upload → (no scan) → edit ignores →
|
|
Sync sources.
|
|
|
|
Source: ``TODO.md`` L3 — "Uploading a source archive should not trigger a
|
|
scan — that should be left to the sync button on the RAG page. Right now
|
|
the sync starts right away which doesn't give the user time to edit the
|
|
ignore list."
|
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
|
|
|
uv run pytest tests/e2e/test_upload_no_scan.py -v --no-cov
|
|
|
|
The story gate for the whole **deferred-scan loop** end-to-end against the
|
|
real pipeline: an upload indexes **nothing** (phase 90 A1 — unpack +
|
|
register only, the button reads "Upload"), the owner edits the new
|
|
source's ignore list in the phase-89 per-row editor while the scan is
|
|
still owed, and the RAG page's **Sync sources** button performs the scan
|
|
and honors the edited ignores (phase 90 A4 — the sync's existing
|
|
``kind='local'`` + prune + ``ignore_paths`` path is the proof, no sync
|
|
change in this phase).
|
|
|
|
**Timing (the task's note):** the upload leg needs NO ``slow_llm`` proxy
|
|
— the upload makes **zero LLM calls** now (phase 90 removed the model
|
|
check + import), so its background run settles in milliseconds and the
|
|
2 s status poll settles one tick after the 202. The sync leg is short
|
|
(2 files + 1 ignored → a handful of mock-LLM requests), so instead of
|
|
racing a timer the suite polls the RAG page's settled sync UI
|
|
(``#sync-label`` "Synced HH:MM" + the ``#sync-progress`` box's counts
|
|
line — phase 150: the box is the single sync text surface) with
|
|
generous timeouts — the live "Syncing… <file> (n/m)" label is the
|
|
``test_sync_upload_progress.py`` suite's subject, not this one's.
|
|
|
|
The archive is **built in-test** with Python's ``tarfile``:
|
|
``e2e-upload-no-scan.tar.gz`` (source name ``e2e-upload-no-scan``) holds
|
|
``alpha.md`` + ``beta.md`` + ``notes/skipme.md`` with markdown sentinels
|
|
(``ALPHA-…`` / ``BETA-…`` / ``SKIPME-…``); the v2 archive (test 3, same
|
|
basename) modifies ``beta.md``, adds ``gamma.md`` (``GAMMA-…``), and
|
|
drops ``alpha.md`` — the in-place-replace subject. ``notes`` is the
|
|
phase-89 ignore entry: the pure prefix rule
|
|
(``is_ignored("notes/skipme.md", ("notes",))``) excludes the skipme file
|
|
from the sync's walk — and from its ``files_total`` (the pre-walk uses
|
|
the same ignore tuple), so a 3-file upload imports exactly 2 documents.
|
|
|
|
Per-module app env (the conftest pattern, module-scoped — mirroring
|
|
``test_archive_upload_sources.py``'s local helpers, NOT importing that
|
|
module): ``BOR_UPLOAD_DIR`` points at a scratch dir the host-side
|
|
assertions inspect (the app runs on the same machine), ``BOR_GIT_SOURCES``
|
|
is forced empty (the dev ``.env``'s fallback URL must never render as an
|
|
env row or become a second sync source), ``BOR_LLM_BASE_URL`` is the mock
|
|
LLM (the sync leg's model check + embeds + overview), and the autouse
|
|
``_clean`` truncates the shared Postgres (plus waits for no running
|
|
background job — this suite's sync must never leak into the next test).
|
|
|
|
Contract under test:
|
|
|
|
* **test 1 — upload does not scan**: the "Upload" button (phase 90 A3)
|
|
→ the archive → the 202 "Successfully uploaded — <source>" toast →
|
|
the settled result line "Uploaded <source> (<stamp>) — press Sync
|
|
sources to import it." (the phase-154 completion stamp) with the
|
|
terminal no-count status payload
|
|
(``{"message": "uploaded"}``, null/0/0 progress — phase 90 A2); the
|
|
source row is present (Local badge + the phase-89 "Ignore paths"
|
|
control) and its folder exists on the host under ``BOR_UPLOAD_DIR``
|
|
with all three files — but **zero documents are indexed**:
|
|
``GET /api/docs`` is empty and the RAG catalog (``/sources.html``)
|
|
settles on the source's 0-document row (phase 97: ``#sources-empty``
|
|
is the zero-SOURCES state only — a registered 0-document source
|
|
renders its ``0`` count instead);
|
|
* **test 2 — ignore list, then Sync scans**: fresh state → upload →
|
|
success line → the row's phase-89 "Ignore paths" editor: type
|
|
``notes``, Save → the row shows the "1 ignored" count tag (the A5
|
|
round-trip through ``GET``) and the stored row carries
|
|
``ignore_paths == ["notes"]``; navigate to the RAG page → click
|
|
**Sync sources** (``#sync-btn``) → the sync settles ("Synced HH:MM",
|
|
"2 added", counts on the status endpoint) and the catalog lists
|
|
``alpha.md`` + ``beta.md`` for the source and **not**
|
|
``notes/skipme.md`` — the edit made BEFORE the sync is honored;
|
|
* **test 3 — re-upload replaces without a scan**: fresh state → upload
|
|
v1, success → upload v2 (same basename: the in-place identity) →
|
|
success again with exactly ONE source row (phase-49 contract — one
|
|
folder, one row) whose on-disk contents are ONLY v2's files — and
|
|
still **zero** documents indexed from it.
|
|
|
|
Test → story mapping (Playwright Mapping Rule):
|
|
1. ``test_upload_does_not_scan``
|
|
2. ``test_ignore_list_then_sync_scans``
|
|
3. ``test_reupload_replaces_without_scan``
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import io
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tarfile
|
|
import time
|
|
from collections.abc import Iterator
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import httpx
|
|
import pytest
|
|
from playwright.sync_api import Page, expect
|
|
from sqlalchemy import text
|
|
|
|
from app.db import SessionLocal
|
|
from e2e.auth_helpers import login
|
|
from e2e.conftest import (
|
|
ADMIN_PASSWORD,
|
|
SESSION_SECRET,
|
|
USE_REAL_LLM,
|
|
_wait_http,
|
|
)
|
|
from e2e.sync_helpers import counts_line
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
|
|
# Phase 79 (task 04, full inventory): in a combined session run the
|
|
# conftest session app already owns its port — a second uvicorn on it
|
|
# dies on bind and this suite would silently drive the wrong server.
|
|
# The module app binds its own port instead (env-overridable).
|
|
APP_PORT = int(os.environ.get("E2E_APP_PORT_UPLOADSCAN", "8135"))
|
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
|
|
|
GIT_SOURCES_URL = "/git-sources.html"
|
|
SOURCES_URL = "/sources.html"
|
|
|
|
#: The archive basename (both versions) — the source/folder name is the
|
|
#: filename minus the archive suffix (the phase-49 naming rule).
|
|
SOURCE_NAME = "e2e-upload-no-scan"
|
|
|
|
|
|
def _ready_for_sync_line(name: str) -> re.Pattern[str]:
|
|
"""Phase 154 (task 02): the ready-for-sync line carrying the
|
|
completion stamp — "Uploaded <name> (<stamp>) — press Sync sources
|
|
to import it." (the fresh 202 settle, the page-local name). In
|
|
E2E the stamp is ALWAYS the same-day ``HH:MM`` form — the run
|
|
settles within seconds (the date-qualified forms are unit-pinned
|
|
on a fixed clock — E2E never depends on the wall clock)."""
|
|
return re.compile(
|
|
rf"^Uploaded {re.escape(name)} \(\d{{1,2}}:\d{{2}}\)"
|
|
r" — press Sync sources to import it\.$"
|
|
)
|
|
|
|
#: v1: two sentinel docs + the notes/ file the phase-89 ignore entry
|
|
#: (``notes``) excludes from the sync. v2 (same basename): beta CHANGED,
|
|
#: alpha DROPPED, gamma ADDED — the in-place-replace subject (the
|
|
#: on-disk folder swap).
|
|
ALPHA_SENTINEL = "ALPHA-NO-SCAN-7f31"
|
|
BETA_SENTINEL_V1 = "BETA-NO-SCAN-v1-2c90"
|
|
BETA_SENTINEL_V2 = "BETA-NO-SCAN-v2-8b42"
|
|
SKIPME_SENTINEL = "SKIPME-NO-SCAN-5e44"
|
|
GAMMA_SENTINEL = "GAMMA-NO-SCAN-9d16"
|
|
|
|
V1_FILES: dict[str, str] = {
|
|
"alpha.md": (
|
|
"# Alpha note\n"
|
|
"\n"
|
|
"First version of the alpha note — v2 drops it.\n"
|
|
f"\nMarker: {ALPHA_SENTINEL}\n"
|
|
),
|
|
"beta.md": (
|
|
"# Beta note\n"
|
|
"\n"
|
|
"First version of the beta note — it changes in v2.\n"
|
|
f"\nMarker: {BETA_SENTINEL_V1}\n"
|
|
),
|
|
"notes/skipme.md": (
|
|
"# Skip me\n"
|
|
"\n"
|
|
"Lives under the notes/ dir the owner ignores before the sync.\n"
|
|
f"\nMarker: {SKIPME_SENTINEL}\n"
|
|
),
|
|
}
|
|
V2_FILES: dict[str, str] = {
|
|
"beta.md": (
|
|
"# Beta note\n"
|
|
"\n"
|
|
"Second version of the beta note — modified in place.\n"
|
|
f"\nMarker: {BETA_SENTINEL_V2}\n"
|
|
),
|
|
"gamma.md": (
|
|
"# Gamma note\n"
|
|
"\n"
|
|
"Brand new in v2 — the add subject of the re-upload.\n"
|
|
f"\nMarker: {GAMMA_SENTINEL}\n"
|
|
),
|
|
}
|
|
|
|
#: The phase-89 ignore entry typed into the editor — the pure prefix
|
|
#: rule excludes ``notes/skipme.md`` (and every other notes/ file).
|
|
IGNORE_ENTRY = "notes"
|
|
|
|
#: "Synced HH:MM" — the local-time last-result label (sources.js's
|
|
#: fmtSyncTime), any hour/minute (test_sync_button.py's pattern).
|
|
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
|
|
|
|
#: Generous settle budgets: the upload run settles in milliseconds
|
|
#: (phase 90) and the UI's 2 s poll lands one tick after; the sync leg
|
|
#: is short (2 files against the fast mock LLM) and may settle between
|
|
#: the 2 s poll ticks — the settled-state assertions retry until then.
|
|
UPLOAD_TIMEOUT_MS = 30_000
|
|
SYNC_TIMEOUT_MS = 45_000
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _build_targz(path: Path, files: dict[str, str]) -> Path:
|
|
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
|
|
with tarfile.open(path, "w:gz") as tf:
|
|
for rel, content in files.items():
|
|
data = content.encode("utf-8")
|
|
info = tarfile.TarInfo(rel)
|
|
info.size = len(data)
|
|
info.mtime = 0
|
|
tf.addfile(info, io.BytesIO(data))
|
|
return path
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
|
|
host-side assertions inspect (the app server runs on the same
|
|
machine). The app creates it on the first upload."""
|
|
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def tarball_v1(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""v1 — in its OWN subdirectory so v2 can reuse the same basename
|
|
(``e2e-upload-no-scan.tar.gz``): the in-place-replace identity IS
|
|
the filename, and ``set_input_files`` sends the path's basename."""
|
|
root = tmp_path_factory.mktemp("bor_archive_v1")
|
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V1_FILES)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""v2 — same basename as v1 (a different parent dir)."""
|
|
root = tmp_path_factory.mktemp("bor_archive_v2")
|
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_server(
|
|
mock_llm: int,
|
|
upload_dir: Path,
|
|
tmp_path_factory: pytest.TempPathFactory,
|
|
) -> Iterator[str]:
|
|
"""The real app under test — per-module env: uploads unpack into a
|
|
scratch dir, the env git list is forced empty (this suite's own
|
|
upload row is the only sync source), and the LLM base URL is the
|
|
mock (the upload makes zero LLM calls — phase 90; the mock serves
|
|
the SYNC leg's model check + embeds + overview). No ``slow_llm``
|
|
proxy: the sync leg is 2 files and the suite polls the settled sync
|
|
UI with generous timeouts instead of racing a live label."""
|
|
env = dict(os.environ)
|
|
env.pop("DEBUGPY", None)
|
|
env["BOR_ENVIRONMENT"] = "e2e"
|
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
|
env["BOR_LLM_BASE_URL"] = (
|
|
"https://aipi.reeseapps.com/v1"
|
|
if USE_REAL_LLM
|
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
|
)
|
|
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
|
# ever sent in this suite, but the app boots with the same env shape.
|
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
|
env.setdefault(
|
|
"BOR_DATABASE_URL",
|
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
|
)
|
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
|
env["BOR_GIT_SOURCES"] = ""
|
|
# Phase 49: unpack uploads into the suite's scratch dir (host-
|
|
# inspectable) and keep the (unused) git checkouts out of the dev
|
|
# location.
|
|
env["BOR_UPLOAD_DIR"] = str(upload_dir)
|
|
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
|
proc = subprocess.Popen(
|
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
|
cwd=REPO,
|
|
env=env,
|
|
)
|
|
try:
|
|
_wait_http(f"{APP_URL}/api/health")
|
|
yield APP_URL
|
|
finally:
|
|
proc.terminate()
|
|
try:
|
|
proc.wait(timeout=10)
|
|
except subprocess.TimeoutExpired:
|
|
proc.kill()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_url(app_server: str) -> str:
|
|
return app_server
|
|
|
|
|
|
def _truncate_all() -> None:
|
|
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
|
row-count and doc-list assertions must be this test's own doing.
|
|
The E2E suites share one Postgres, and a leftover git_sources row
|
|
or document would corrupt them (a leftover row would be a SECOND
|
|
sync source, skewing the sync's counts)."""
|
|
with SessionLocal() as db:
|
|
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
|
|
db.commit()
|
|
|
|
|
|
def _wait_no_running_jobs(app_url: str) -> None:
|
|
"""No background job may leak across tests (the run states live in
|
|
the app's memory, and a still-running sync would keep importing
|
|
into the NEXT test's truncated KB): wait for both status endpoints
|
|
to be non-running BEFORE the truncate. Own admin session (the
|
|
endpoints are admin-only) — usually a no-op: test 2 settles only
|
|
after its sync's terminal state, and the upload run is
|
|
sub-second."""
|
|
with httpx.Client(timeout=5.0) as client:
|
|
client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
|
|
for path in ("/api/sync/status", "/api/git-sources/upload/status"):
|
|
body: dict[str, Any] = {}
|
|
deadline = time.monotonic() + 90
|
|
while time.monotonic() < deadline:
|
|
r = client.get(f"{app_url}{path}")
|
|
if r.status_code == 200:
|
|
body = r.json()
|
|
if body["state"] != "running":
|
|
break
|
|
time.sleep(0.2)
|
|
assert body["state"] != "running", (
|
|
f"a background job was still running at test boundary: {path} {body}"
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
|
_wait_no_running_jobs(app_url)
|
|
_truncate_all()
|
|
yield
|
|
_truncate_all()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
|
"""Real form login landing on the git sources page (admin settled:
|
|
Sign out visible, the manager revealed by the page module)."""
|
|
login(page, app_url, next=GIT_SOURCES_URL)
|
|
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
|
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
|
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
|
expect(page.locator("#git-sources-content")).to_be_visible()
|
|
|
|
|
|
def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
|
|
"""``GET /api/docs`` as the signed-in page → sorted (source, path)
|
|
pairs (the admin cookie rides the browser context)."""
|
|
r = page.request.get(f"{app_url}/api/docs")
|
|
assert r.status == 200, r.text
|
|
return sorted((d["source"], d["path"]) for d in r.json()["documents"])
|
|
|
|
|
|
def _upload_via_page(page: Page, archive: Path) -> str:
|
|
"""Pick the archive, submit the form, and wait for the result line
|
|
(the phase-64 202 path: toast at the 202, then the button's status
|
|
polling renders the line from the run's ``success``) — returns its
|
|
text. Every upload in this suite succeeds (the failure path is
|
|
another suite's subject), so any non-result outcome here is a test
|
|
error."""
|
|
page.set_input_files("#archive-upload-file", str(archive))
|
|
page.click("#archive-upload-btn")
|
|
result = page.locator("#archive-upload-result")
|
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
|
text = result.text_content()
|
|
assert text is not None
|
|
return text
|
|
|
|
|
|
def _folder_files(folder: Path) -> set[str]:
|
|
"""The unpacked folder's file set, source-relative POSIX paths."""
|
|
return {
|
|
p.relative_to(folder).as_posix()
|
|
for p in folder.rglob("*")
|
|
if p.is_file()
|
|
}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Upload → 202 + toast → ready-for-sync line, the row + folder on
|
|
# disk — and ZERO documents indexed (the scan is the Sync button's
|
|
# job, phase 90 A1)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_upload_does_not_scan(
|
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
|
) -> None:
|
|
"""One real upload through the page: the button reads **Upload**
|
|
(phase 90 A3); the 202 fires the "Successfully uploaded — <source>"
|
|
toast; the settled result line points at the next step — "Uploaded
|
|
<source> (<stamp>) — press Sync sources to import it." (A3 — the
|
|
phase-154 completion stamp) — with the
|
|
terminal no-count status payload (``{"message": "uploaded"}``,
|
|
null/0/0 progress — A2). The source row is present (Local badge +
|
|
the phase-89 "Ignore paths" control) and its folder exists on the
|
|
host under ``BOR_UPLOAD_DIR`` with all three files — but
|
|
**zero documents are indexed**: ``GET /api/docs`` is empty and the
|
|
RAG catalog settles on its empty state (phase 90 A1 — the scan is
|
|
the RAG page's Sync button's job)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
|
|
|
# The button reads exactly "Upload" (phase 90 A3 — the
|
|
# scan-suffixed label is gone).
|
|
btn = page.locator("#archive-upload-btn")
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
|
|
page.set_input_files("#archive-upload-file", str(tarball_v1))
|
|
btn.click()
|
|
|
|
# The 202 moment (phase-64 A2): a single .toast node, visible,
|
|
# role=status, naming the safe source name…
|
|
toast = page.locator(".toast")
|
|
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
|
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
|
assert toast.get_attribute("role") == "status"
|
|
expect(toast).to_have_text(f"Successfully uploaded — {SOURCE_NAME}")
|
|
|
|
# …and the background run (unpack + register only) settles at the
|
|
# next 2 s poll tick: the ready-for-sync result line (A3), the
|
|
# never-stale restore (button + input)…
|
|
result = page.locator("#archive-upload-result")
|
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
|
expect(result).to_have_text(_ready_for_sync_line(SOURCE_NAME))
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
expect(page.locator("#archive-upload-file")).to_have_value("")
|
|
|
|
# …and the terminal status is the no-count payload with null/0/0
|
|
# progress (the phase-64 key set, phase 90 A2).
|
|
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
|
assert r.status == 200, r.text
|
|
status = r.json()
|
|
assert status["state"] == "success", status
|
|
assert status["detail"] == {"message": "uploaded"}, status
|
|
assert status["current_file"] is None
|
|
assert status["files_done"] == 0 and status["files_total"] == 0
|
|
|
|
# The row landed — Local badge, the unpacked path in the mono cell,
|
|
# and the phase-89 "Ignore paths" control (the editor the
|
|
# deferral exists for — test 2 opens it).
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
|
|
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
|
|
expect(row).to_have_count(1)
|
|
expect(row.locator("span.git-source-kind")).to_have_text("Local")
|
|
expect(row.locator("td.git-source-url-cell code")).to_have_text(
|
|
str(upload_dir / SOURCE_NAME)
|
|
)
|
|
expect(row.locator("button.git-source-ignore")).to_have_count(1)
|
|
expect(row.locator("button.git-source-ignore")).to_have_text("Ignore paths")
|
|
|
|
# Phase 90 A1: the upload indexes NOTHING — the KB is empty…
|
|
assert _docs(page, app_url) == []
|
|
# …and the folder exists on the host with ALL THREE files unpacked
|
|
# (the notes/ file included — the ignore list, not the upload,
|
|
# decides what the sync walks).
|
|
folder = upload_dir / SOURCE_NAME
|
|
assert folder.is_dir(), f"the unpacked folder is missing: {folder}"
|
|
assert _folder_files(folder) == {"alpha.md", "beta.md", "notes/skipme.md"}, (
|
|
f"unexpected unpacked files: {_folder_files(folder)}"
|
|
)
|
|
# …and so is the RAG catalog: the registered source renders its
|
|
# 0-document row (phase 97: the "zero documents" signal moved from
|
|
# #sources-empty — the zero-SOURCES state only — to the source
|
|
# row's own `0` count, which is ALSO the deterministic load-settled
|
|
# signal; a count-0 check alone would race the boot loadTree
|
|
# fetch).
|
|
page.goto(app_url + SOURCES_URL)
|
|
row = page.locator("#folders-tbody tr", has_text=SOURCE_NAME)
|
|
expect(row).to_have_count(1, timeout=30_000)
|
|
expect(row.locator("td:nth-child(2)")).to_have_text("0")
|
|
expect(page.locator("#sources-empty")).to_be_hidden()
|
|
expect(page.locator("#docs-tbody tr")).to_have_count(0)
|
|
expect(page.locator("#stat-docs")).to_have_text("0")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. The phase-89 ignore list is edited BEFORE the scan — and the RAG
|
|
# page's Sync sources button honors it (phase 90 A4)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_ignore_list_then_sync_scans(
|
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
|
) -> None:
|
|
"""The whole deferred-scan loop: upload (nothing indexed) → open
|
|
the row's phase-89 "Ignore paths" editor → type ``notes`` → Save
|
|
(the row shows the "1 ignored" count tag; the stored row carries
|
|
``ignore_paths == ["notes"]`` — the A5 round-trip) → navigate to
|
|
the RAG page → click **Sync sources** → the sync settles
|
|
("Synced HH:MM" label, "2 added" result, the status endpoint's
|
|
counts: added 2 / pruned 0 / 2 of 2 files — the ignored file is
|
|
excluded from the walk AND the denominator) → the catalog lists
|
|
``alpha.md`` + ``beta.md`` for the source and **not**
|
|
``notes/skipme.md``: the edit made before the sync is honored
|
|
(phase 90 A4 — the sync's existing kind='local' + prune +
|
|
ignore_paths path, unchanged in this phase)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
|
|
# Upload: the ready-for-sync line (the phase-154 completion
|
|
# stamp), and still nothing indexed.
|
|
assert _ready_for_sync_line(SOURCE_NAME).fullmatch(
|
|
_upload_via_page(page, tarball_v1)
|
|
)
|
|
assert _docs(page, app_url) == []
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
|
|
# The phase-89 per-row editor: open it from the row's button…
|
|
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
|
|
row.locator("button.git-source-ignore").click()
|
|
dialog = page.locator("#ignore-editor-dialog")
|
|
expect(dialog).to_be_visible(timeout=15_000)
|
|
expect(page.locator("#ignore-editor-source")).to_have_text(
|
|
str(upload_dir / SOURCE_NAME)
|
|
)
|
|
# …type the ignore entry (one path per line; the prefix rule
|
|
# excludes everything under notes/), and save.
|
|
page.fill("#ignore-editor-textarea", IGNORE_ENTRY)
|
|
page.click("#ignore-editor-save")
|
|
expect(dialog).to_be_hidden(timeout=30_000)
|
|
# The A5 round-trip: the row re-loads and shows the "1 ignored"
|
|
# count tag…
|
|
expect(row.locator(".git-source-ignore-count")).to_have_text(
|
|
"1 ignored", timeout=30_000
|
|
)
|
|
# …and the stored row carries the normalized list the sync will
|
|
# read.
|
|
r = page.request.get(f"{app_url}/api/git-sources")
|
|
assert r.status == 200, r.text
|
|
body = r.json()
|
|
assert [(s["kind"], s["path"], s["ignore_paths"]) for s in body["sources"]] == [
|
|
("local", str(upload_dir / SOURCE_NAME), [IGNORE_ENTRY])
|
|
]
|
|
|
|
# The RAG page — the catalog is still empty before the scan (the
|
|
# source row reads 0 — phase 97's load-settled zero signal)…
|
|
page.goto(app_url + SOURCES_URL)
|
|
expect(page.locator("#sync-btn")).to_be_visible(timeout=30_000)
|
|
row = page.locator("#folders-tbody tr", has_text=SOURCE_NAME)
|
|
expect(row).to_have_count(1, timeout=30_000)
|
|
expect(row.locator("td:nth-child(2)")).to_have_text("0")
|
|
expect(page.locator("#sync-label")).to_have_text("Sync sources")
|
|
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
|
|
|
# Click Sync sources (the button the deferral exists for). The
|
|
# short sync (2 files against the fast mock LLM) may settle between
|
|
# the 2 s poll ticks — the settled-state assertions below retry
|
|
# with generous timeouts instead of racing a live label.
|
|
page.click("#sync-btn")
|
|
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
|
expect(page.locator("#sync-label")).to_have_text(
|
|
SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS
|
|
)
|
|
expect(page.locator("#sync-progress-text")).to_have_text(
|
|
counts_line("2 added"), timeout=SYNC_TIMEOUT_MS
|
|
)
|
|
expect(page.locator("#sync-btn")).to_be_enabled()
|
|
expect(page.locator("#sync-btn")).not_to_have_attribute("aria-busy")
|
|
|
|
# The status endpoint agrees: 2 added, nothing pruned, and the
|
|
# ignored file is absent from the walk's denominator (2 of 2 —
|
|
# the pre-walk uses the row's ignore list).
|
|
r = page.request.get(f"{app_url}/api/sync/status")
|
|
assert r.status == 200, r.text
|
|
status = r.json()
|
|
assert status["state"] == "success", status
|
|
assert status["detail"]["added"] == 2, status
|
|
assert status["detail"]["pruned"] == 0, status
|
|
assert status["current_file"] is None
|
|
assert status["files_done"] == 2 and status["files_total"] == 2
|
|
|
|
# The catalog: exactly the two non-ignored docs, for the source —
|
|
# and the ignored one is NOT there. Phase 97: files are seen per
|
|
# source — drill into the source row first (the two docs sit at
|
|
# its root; the ignored notes/ folder is not even listed — the
|
|
# existence rule runs over INDEXED paths).
|
|
page.click(f'#folders-tbody a.folder-link:text-is("{SOURCE_NAME}")')
|
|
expect(page.locator("#folders-wrap")).to_be_hidden() # no subfolders
|
|
expect(page.locator("#docs-tbody tr")).to_have_count(2, timeout=30_000)
|
|
expect(page.locator("#docs-tbody tr", has_text=SOURCE_NAME)).to_have_count(2)
|
|
expect(page.locator("#docs-tbody tr", has_text="alpha.md")).to_have_count(1)
|
|
expect(page.locator("#docs-tbody tr", has_text="beta.md")).to_have_count(1)
|
|
expect(page.locator("#docs-tbody tr", has_text="notes/skipme.md")).to_have_count(0)
|
|
assert _docs(page, app_url) == [
|
|
(SOURCE_NAME, "alpha.md"),
|
|
(SOURCE_NAME, "beta.md"),
|
|
], f"the ignore list was not honored: {_docs(page, app_url)}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Re-upload, same filename → in-place replace (no duplicate row,
|
|
# folder swap on disk) — and still nothing indexed
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_reupload_replaces_without_scan(
|
|
page: Page,
|
|
app_url: str,
|
|
db_ready: None,
|
|
tarball_v1: Path,
|
|
tarball_v2: Path,
|
|
upload_dir: Path,
|
|
) -> None:
|
|
"""v1 then v2 under the SAME filename (``e2e-upload-no-scan.tar.gz``
|
|
— v2 modifies ``beta.md``, adds ``gamma.md``, drops ``alpha.md``):
|
|
both unpack + register only (phase 90). After the re-upload the
|
|
list has exactly ONE ``e2e-upload-no-scan`` row (the in-place
|
|
identity, phase-49 contract — no duplicate), the registry agrees
|
|
(one kind=local row at the unpacked path), the on-disk folder holds
|
|
ONLY v2's files (the atomic swap) — and the KB is empty
|
|
THROUGHOUT (zero documents indexed by either upload: the scan is
|
|
the Sync button's job, never the upload's)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
|
|
# Baseline: v1 through the page (202 → the ready-for-sync line,
|
|
# one row, no index).
|
|
assert _ready_for_sync_line(SOURCE_NAME).fullmatch(
|
|
_upload_via_page(page, tarball_v1)
|
|
)
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
folder = upload_dir / SOURCE_NAME
|
|
assert _folder_files(folder) == set(V1_FILES)
|
|
assert _docs(page, app_url) == []
|
|
|
|
# Re-upload v2 — SAME basename, different parent dir (the file
|
|
# input's selection is replaced wholesale).
|
|
assert _ready_for_sync_line(SOURCE_NAME).fullmatch(
|
|
_upload_via_page(page, tarball_v2)
|
|
)
|
|
|
|
# No duplicate: exactly ONE row for that source (and one row
|
|
# total)…
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
|
# …and the registry agrees: one kind=local row, the unpacked path.
|
|
r = page.request.get(f"{app_url}/api/git-sources")
|
|
assert r.status == 200, r.text
|
|
body = r.json()
|
|
assert [(s["kind"], s["path"]) for s in body["sources"]] == [
|
|
("local", str(upload_dir / SOURCE_NAME))
|
|
]
|
|
|
|
# The on-disk folder holds ONLY v2's files (the swap replaced the
|
|
# whole folder in place — no stale v1 file survived)…
|
|
assert _folder_files(folder) == set(V2_FILES), (
|
|
f"unexpected unpacked files: {_folder_files(folder)}"
|
|
)
|
|
# …and the KB is STILL empty (the upload never scans, phase 90 A1
|
|
# — the Sync button is what will index v2's files).
|
|
assert _docs(page, app_url) == []
|