All verification is complete and green. Final report:
**Phase 90 (upload_no_scan) — final verification pass: all criteria verified, no fixes needed**
- Verified (not re-implemented): all 3 tasks already complete; working tree carries the phase's uncommitted changes
- Upload pipeline: `_run_upload` stops after row upsert; success = `{"message": "uploaded"}`, null/0/0 progress; `UploadOut` removed from `app/schemas.py`; gates/unpack/swap/failed states intact
- Copy: button "Upload", bare "Processing…", result line "Uploaded \<name\> — press Sync sources to import it."; hint + caption re-pointed at Sync; no "Upload &" remnants in `frontend/`
- Tests: `pytest tests/unit/test_git_sources.py tests/integration/test_git_sources_upload.py tests/unit/test_frontend_sync_upload.py` → 56 passed; E2E phase suite `tests/e2e/test_upload_no_scan.py` → 3 passed (zero docs after upload; ignore edit honored by Sync — 2 added, `notes/skipme.md` excluded; re-upload in-place, still zero docs); affected suites `test_archive_upload_sources.py` + `test_sync_upload_progress.py` + `test_source_removal_cleanup.py` → 15 passed
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, 99% coverage** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors
**Completion criteria:**
1. Upload settles "ready for sync", zero docs, row + ignore editor — ✅ (E2E test 1)
2. Button "Upload", no "Upload &" copy — ✅ (`rg` empty in frontend/, app/, tests/)
3. RAG-page "Sync sources" imports upload honoring edited ignores — ✅ (isolated E2E test 2)
4. Suite green, >90% coverage, lint clean — ✅ (99%, ruff+pyright clean)
5. Atomic commit — ✅ left to harness per executor rules (no git add/commit run; tree staged-ready)
**Deviations:** `regenerate_overview` still appears once in `git_sources.py` — inside `delete_git_source` (phase-69 locked total-removal behavior, not the upload leg; upload leg is scan-free). `.agents/PLAN.md` does not exist in this repo; locked decisions A1–A4 were enforced from the phase files.
**Next pending phase:** `91_admin_theme_tab` (in `todo/`).
643 lines
28 KiB
Python
643 lines
28 KiB
Python
"""Phase 90 story E2E (Playwright): upload → (no scan) → edit ignores →
|
|
Sync sources.
|
|
|
|
Source: ``TODO.md`` L3 — "Uploading a source archive should not trigger a
|
|
scan — that should be left to the sync button on the RAG page. Right now
|
|
the sync starts right away which doesn't give the user time to edit the
|
|
ignore list."
|
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
|
|
|
uv run pytest tests/e2e/test_upload_no_scan.py -v --no-cov
|
|
|
|
The story gate for the whole **deferred-scan loop** end-to-end against the
|
|
real pipeline: an upload indexes **nothing** (phase 90 A1 — unpack +
|
|
register only, the button reads "Upload"), the owner edits the new
|
|
source's ignore list in the phase-89 per-row editor while the scan is
|
|
still owed, and the RAG page's **Sync sources** button performs the scan
|
|
and honors the edited ignores (phase 90 A4 — the sync's existing
|
|
``kind='local'`` + prune + ``ignore_paths`` path is the proof, no sync
|
|
change in this phase).
|
|
|
|
**Timing (the task's note):** the upload leg needs NO ``slow_llm`` proxy
|
|
— the upload makes **zero LLM calls** now (phase 90 removed the model
|
|
check + import), so its background run settles in milliseconds and the
|
|
2 s status poll settles one tick after the 202. The sync leg is short
|
|
(2 files + 1 ignored → a handful of mock-LLM requests), so instead of
|
|
racing a timer the suite polls the RAG page's settled sync UI
|
|
(``#sync-label`` "Synced HH:MM" + ``#sync-result`` counts) with generous
|
|
timeouts — the live "Syncing… <file> (n/m)" label is the
|
|
``test_sync_upload_progress.py`` suite's subject, not this one's.
|
|
|
|
The archive is **built in-test** with Python's ``tarfile``:
|
|
``e2e-upload-no-scan.tar.gz`` (source name ``e2e-upload-no-scan``) holds
|
|
``alpha.md`` + ``beta.md`` + ``notes/skipme.md`` with markdown sentinels
|
|
(``ALPHA-…`` / ``BETA-…`` / ``SKIPME-…``); the v2 archive (test 3, same
|
|
basename) modifies ``beta.md``, adds ``gamma.md`` (``GAMMA-…``), and
|
|
drops ``alpha.md`` — the in-place-replace subject. ``notes`` is the
|
|
phase-89 ignore entry: the pure prefix rule
|
|
(``is_ignored("notes/skipme.md", ("notes",))``) excludes the skipme file
|
|
from the sync's walk — and from its ``files_total`` (the pre-walk uses
|
|
the same ignore tuple), so a 3-file upload imports exactly 2 documents.
|
|
|
|
Per-module app env (the conftest pattern, module-scoped — mirroring
|
|
``test_archive_upload_sources.py``'s local helpers, NOT importing that
|
|
module): ``BOR_UPLOAD_DIR`` points at a scratch dir the host-side
|
|
assertions inspect (the app runs on the same machine), ``BOR_GIT_SOURCES``
|
|
is forced empty (the dev ``.env``'s fallback URL must never render as an
|
|
env row or become a second sync source), ``BOR_LLM_BASE_URL`` is the mock
|
|
LLM (the sync leg's model check + embeds + overview), and the autouse
|
|
``_clean`` truncates the shared Postgres (plus waits for no running
|
|
background job — this suite's sync must never leak into the next test).
|
|
|
|
Contract under test:
|
|
|
|
* **test 1 — upload does not scan**: the "Upload" button (phase 90 A3)
|
|
→ the archive → the 202 "Successfully uploaded — <source>" toast →
|
|
the settled result line "Uploaded <source> — press Sync sources to
|
|
import it." with the terminal no-count status payload
|
|
(``{"message": "uploaded"}``, null/0/0 progress — phase 90 A2); the
|
|
source row is present (Local badge + the phase-89 "Ignore paths"
|
|
control) and its folder exists on the host under ``BOR_UPLOAD_DIR``
|
|
with all three files — but **zero documents are indexed**:
|
|
``GET /api/docs`` is empty and the RAG catalog (``/sources.html``)
|
|
settles on its empty state;
|
|
* **test 2 — ignore list, then Sync scans**: fresh state → upload →
|
|
success line → the row's phase-89 "Ignore paths" editor: type
|
|
``notes``, Save → the row shows the "1 ignored" count tag (the A5
|
|
round-trip through ``GET``) and the stored row carries
|
|
``ignore_paths == ["notes"]``; navigate to the RAG page → click
|
|
**Sync sources** (``#sync-btn``) → the sync settles ("Synced HH:MM",
|
|
"2 added", counts on the status endpoint) and the catalog lists
|
|
``alpha.md`` + ``beta.md`` for the source and **not**
|
|
``notes/skipme.md`` — the edit made BEFORE the sync is honored;
|
|
* **test 3 — re-upload replaces without a scan**: fresh state → upload
|
|
v1, success → upload v2 (same basename: the in-place identity) →
|
|
success again with exactly ONE source row (phase-49 contract — one
|
|
folder, one row) whose on-disk contents are ONLY v2's files — and
|
|
still **zero** documents indexed from it.
|
|
|
|
Test → story mapping (Playwright Mapping Rule):
|
|
1. ``test_upload_does_not_scan``
|
|
2. ``test_ignore_list_then_sync_scans``
|
|
3. ``test_reupload_replaces_without_scan``
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import io
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tarfile
|
|
import time
|
|
from collections.abc import Iterator
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import httpx
|
|
import pytest
|
|
from playwright.sync_api import Page, expect
|
|
from sqlalchemy import text
|
|
|
|
from app.db import SessionLocal
|
|
from e2e.auth_helpers import login
|
|
from e2e.conftest import (
|
|
ADMIN_PASSWORD,
|
|
SESSION_SECRET,
|
|
USE_REAL_LLM,
|
|
_wait_http,
|
|
)
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
|
|
# Phase 79 (task 04, full inventory): in a combined session run the
|
|
# conftest session app already owns its port — a second uvicorn on it
|
|
# dies on bind and this suite would silently drive the wrong server.
|
|
# The module app binds its own port instead (env-overridable).
|
|
APP_PORT = int(os.environ.get("E2E_APP_PORT_UPLOADSCAN", "8135"))
|
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
|
|
|
GIT_SOURCES_URL = "/git-sources.html"
|
|
SOURCES_URL = "/sources.html"
|
|
|
|
#: The archive basename (both versions) — the source/folder name is the
|
|
#: filename minus the archive suffix (the phase-49 naming rule).
|
|
SOURCE_NAME = "e2e-upload-no-scan"
|
|
|
|
#: v1: two sentinel docs + the notes/ file the phase-89 ignore entry
|
|
#: (``notes``) excludes from the sync. v2 (same basename): beta CHANGED,
|
|
#: alpha DROPPED, gamma ADDED — the in-place-replace subject (the
|
|
#: on-disk folder swap).
|
|
ALPHA_SENTINEL = "ALPHA-NO-SCAN-7f31"
|
|
BETA_SENTINEL_V1 = "BETA-NO-SCAN-v1-2c90"
|
|
BETA_SENTINEL_V2 = "BETA-NO-SCAN-v2-8b42"
|
|
SKIPME_SENTINEL = "SKIPME-NO-SCAN-5e44"
|
|
GAMMA_SENTINEL = "GAMMA-NO-SCAN-9d16"
|
|
|
|
V1_FILES: dict[str, str] = {
|
|
"alpha.md": (
|
|
"# Alpha note\n"
|
|
"\n"
|
|
"First version of the alpha note — v2 drops it.\n"
|
|
f"\nMarker: {ALPHA_SENTINEL}\n"
|
|
),
|
|
"beta.md": (
|
|
"# Beta note\n"
|
|
"\n"
|
|
"First version of the beta note — it changes in v2.\n"
|
|
f"\nMarker: {BETA_SENTINEL_V1}\n"
|
|
),
|
|
"notes/skipme.md": (
|
|
"# Skip me\n"
|
|
"\n"
|
|
"Lives under the notes/ dir the owner ignores before the sync.\n"
|
|
f"\nMarker: {SKIPME_SENTINEL}\n"
|
|
),
|
|
}
|
|
V2_FILES: dict[str, str] = {
|
|
"beta.md": (
|
|
"# Beta note\n"
|
|
"\n"
|
|
"Second version of the beta note — modified in place.\n"
|
|
f"\nMarker: {BETA_SENTINEL_V2}\n"
|
|
),
|
|
"gamma.md": (
|
|
"# Gamma note\n"
|
|
"\n"
|
|
"Brand new in v2 — the add subject of the re-upload.\n"
|
|
f"\nMarker: {GAMMA_SENTINEL}\n"
|
|
),
|
|
}
|
|
|
|
#: The phase-89 ignore entry typed into the editor — the pure prefix
|
|
#: rule excludes ``notes/skipme.md`` (and every other notes/ file).
|
|
IGNORE_ENTRY = "notes"
|
|
|
|
#: "Synced HH:MM" — the local-time last-result label (sources.js's
|
|
#: fmtSyncTime), any hour/minute (test_sync_button.py's pattern).
|
|
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
|
|
|
|
#: Generous settle budgets: the upload run settles in milliseconds
|
|
#: (phase 90) and the UI's 2 s poll lands one tick after; the sync leg
|
|
#: is short (2 files against the fast mock LLM) and may settle between
|
|
#: the 2 s poll ticks — the settled-state assertions retry until then.
|
|
UPLOAD_TIMEOUT_MS = 30_000
|
|
SYNC_TIMEOUT_MS = 45_000
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _build_targz(path: Path, files: dict[str, str]) -> Path:
|
|
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
|
|
with tarfile.open(path, "w:gz") as tf:
|
|
for rel, content in files.items():
|
|
data = content.encode("utf-8")
|
|
info = tarfile.TarInfo(rel)
|
|
info.size = len(data)
|
|
info.mtime = 0
|
|
tf.addfile(info, io.BytesIO(data))
|
|
return path
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
|
|
host-side assertions inspect (the app server runs on the same
|
|
machine). The app creates it on the first upload."""
|
|
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def tarball_v1(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""v1 — in its OWN subdirectory so v2 can reuse the same basename
|
|
(``e2e-upload-no-scan.tar.gz``): the in-place-replace identity IS
|
|
the filename, and ``set_input_files`` sends the path's basename."""
|
|
root = tmp_path_factory.mktemp("bor_archive_v1")
|
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V1_FILES)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""v2 — same basename as v1 (a different parent dir)."""
|
|
root = tmp_path_factory.mktemp("bor_archive_v2")
|
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_server(
|
|
mock_llm: int,
|
|
upload_dir: Path,
|
|
tmp_path_factory: pytest.TempPathFactory,
|
|
) -> Iterator[str]:
|
|
"""The real app under test — per-module env: uploads unpack into a
|
|
scratch dir, the env git list is forced empty (this suite's own
|
|
upload row is the only sync source), and the LLM base URL is the
|
|
mock (the upload makes zero LLM calls — phase 90; the mock serves
|
|
the SYNC leg's model check + embeds + overview). No ``slow_llm``
|
|
proxy: the sync leg is 2 files and the suite polls the settled sync
|
|
UI with generous timeouts instead of racing a live label."""
|
|
env = dict(os.environ)
|
|
env.pop("DEBUGPY", None)
|
|
env["BOR_ENVIRONMENT"] = "e2e"
|
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
|
env["BOR_LLM_BASE_URL"] = (
|
|
"https://aipi.reeseapps.com/v1"
|
|
if USE_REAL_LLM
|
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
|
)
|
|
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
|
# ever sent in this suite, but the app boots with the same env shape.
|
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
|
env.setdefault(
|
|
"BOR_DATABASE_URL",
|
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
|
)
|
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
|
env["BOR_GIT_SOURCES"] = ""
|
|
# Phase 49: unpack uploads into the suite's scratch dir (host-
|
|
# inspectable) and keep the (unused) git checkouts out of the dev
|
|
# location.
|
|
env["BOR_UPLOAD_DIR"] = str(upload_dir)
|
|
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
|
proc = subprocess.Popen(
|
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
|
cwd=REPO,
|
|
env=env,
|
|
)
|
|
try:
|
|
_wait_http(f"{APP_URL}/api/health")
|
|
yield APP_URL
|
|
finally:
|
|
proc.terminate()
|
|
try:
|
|
proc.wait(timeout=10)
|
|
except subprocess.TimeoutExpired:
|
|
proc.kill()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_url(app_server: str) -> str:
|
|
return app_server
|
|
|
|
|
|
def _truncate_all() -> None:
|
|
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
|
row-count and doc-list assertions must be this test's own doing.
|
|
The E2E suites share one Postgres, and a leftover git_sources row
|
|
or document would corrupt them (a leftover row would be a SECOND
|
|
sync source, skewing the sync's counts)."""
|
|
with SessionLocal() as db:
|
|
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
|
|
db.commit()
|
|
|
|
|
|
def _wait_no_running_jobs(app_url: str) -> None:
|
|
"""No background job may leak across tests (the run states live in
|
|
the app's memory, and a still-running sync would keep importing
|
|
into the NEXT test's truncated KB): wait for both status endpoints
|
|
to be non-running BEFORE the truncate. Own admin session (the
|
|
endpoints are admin-only) — usually a no-op: test 2 settles only
|
|
after its sync's terminal state, and the upload run is
|
|
sub-second."""
|
|
with httpx.Client(timeout=5.0) as client:
|
|
client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
|
|
for path in ("/api/sync/status", "/api/git-sources/upload/status"):
|
|
body: dict[str, Any] = {}
|
|
deadline = time.monotonic() + 90
|
|
while time.monotonic() < deadline:
|
|
r = client.get(f"{app_url}{path}")
|
|
if r.status_code == 200:
|
|
body = r.json()
|
|
if body["state"] != "running":
|
|
break
|
|
time.sleep(0.2)
|
|
assert body["state"] != "running", (
|
|
f"a background job was still running at test boundary: {path} {body}"
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
|
_wait_no_running_jobs(app_url)
|
|
_truncate_all()
|
|
yield
|
|
_truncate_all()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
|
"""Real form login landing on the git sources page (admin settled:
|
|
Sign out visible, the manager revealed by the page module)."""
|
|
login(page, app_url, next=GIT_SOURCES_URL)
|
|
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
|
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
|
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
|
expect(page.locator("#git-sources-content")).to_be_visible()
|
|
|
|
|
|
def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
|
|
"""``GET /api/docs`` as the signed-in page → sorted (source, path)
|
|
pairs (the admin cookie rides the browser context)."""
|
|
r = page.request.get(f"{app_url}/api/docs")
|
|
assert r.status == 200, r.text
|
|
return sorted((d["source"], d["path"]) for d in r.json()["documents"])
|
|
|
|
|
|
def _upload_via_page(page: Page, archive: Path) -> str:
|
|
"""Pick the archive, submit the form, and wait for the result line
|
|
(the phase-64 202 path: toast at the 202, then the button's status
|
|
polling renders the line from the run's ``success``) — returns its
|
|
text. Every upload in this suite succeeds (the failure path is
|
|
another suite's subject), so any non-result outcome here is a test
|
|
error."""
|
|
page.set_input_files("#archive-upload-file", str(archive))
|
|
page.click("#archive-upload-btn")
|
|
result = page.locator("#archive-upload-result")
|
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
|
text = result.text_content()
|
|
assert text is not None
|
|
return text
|
|
|
|
|
|
def _folder_files(folder: Path) -> set[str]:
|
|
"""The unpacked folder's file set, source-relative POSIX paths."""
|
|
return {
|
|
p.relative_to(folder).as_posix()
|
|
for p in folder.rglob("*")
|
|
if p.is_file()
|
|
}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Upload → 202 + toast → ready-for-sync line, the row + folder on
|
|
# disk — and ZERO documents indexed (the scan is the Sync button's
|
|
# job, phase 90 A1)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_upload_does_not_scan(
|
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
|
) -> None:
|
|
"""One real upload through the page: the button reads **Upload**
|
|
(phase 90 A3); the 202 fires the "Successfully uploaded — <source>"
|
|
toast; the settled result line points at the next step — "Uploaded
|
|
<source> — press Sync sources to import it." (A3) — with the
|
|
terminal no-count status payload (``{"message": "uploaded"}``,
|
|
null/0/0 progress — A2). The source row is present (Local badge +
|
|
the phase-89 "Ignore paths" control) and its folder exists on the
|
|
host under ``BOR_UPLOAD_DIR`` with all three files — but
|
|
**zero documents are indexed**: ``GET /api/docs`` is empty and the
|
|
RAG catalog settles on its empty state (phase 90 A1 — the scan is
|
|
the RAG page's Sync button's job)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
|
|
|
# The button reads exactly "Upload" (phase 90 A3 — the
|
|
# scan-suffixed label is gone).
|
|
btn = page.locator("#archive-upload-btn")
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
|
|
page.set_input_files("#archive-upload-file", str(tarball_v1))
|
|
btn.click()
|
|
|
|
# The 202 moment (phase-64 A2): a single .toast node, visible,
|
|
# role=status, naming the safe source name…
|
|
toast = page.locator(".toast")
|
|
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
|
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
|
assert toast.get_attribute("role") == "status"
|
|
expect(toast).to_have_text(f"Successfully uploaded — {SOURCE_NAME}")
|
|
|
|
# …and the background run (unpack + register only) settles at the
|
|
# next 2 s poll tick: the ready-for-sync result line (A3), the
|
|
# never-stale restore (button + input)…
|
|
result = page.locator("#archive-upload-result")
|
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
|
expect(result).to_have_text(
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
expect(page.locator("#archive-upload-file")).to_have_value("")
|
|
|
|
# …and the terminal status is the no-count payload with null/0/0
|
|
# progress (the phase-64 key set, phase 90 A2).
|
|
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
|
assert r.status == 200, r.text
|
|
status = r.json()
|
|
assert status["state"] == "success", status
|
|
assert status["detail"] == {"message": "uploaded"}, status
|
|
assert status["current_file"] is None
|
|
assert status["files_done"] == 0 and status["files_total"] == 0
|
|
|
|
# The row landed — Local badge, the unpacked path in the mono cell,
|
|
# and the phase-89 "Ignore paths" control (the editor the
|
|
# deferral exists for — test 2 opens it).
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
|
|
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
|
|
expect(row).to_have_count(1)
|
|
expect(row.locator("span.git-source-kind")).to_have_text("Local")
|
|
expect(row.locator("td.git-source-url-cell code")).to_have_text(
|
|
str(upload_dir / SOURCE_NAME)
|
|
)
|
|
expect(row.locator("button.git-source-ignore")).to_have_count(1)
|
|
expect(row.locator("button.git-source-ignore")).to_have_text("Ignore paths")
|
|
|
|
# Phase 90 A1: the upload indexes NOTHING — the KB is empty…
|
|
assert _docs(page, app_url) == []
|
|
# …and the folder exists on the host with ALL THREE files unpacked
|
|
# (the notes/ file included — the ignore list, not the upload,
|
|
# decides what the sync walks).
|
|
folder = upload_dir / SOURCE_NAME
|
|
assert folder.is_dir(), f"the unpacked folder is missing: {folder}"
|
|
assert _folder_files(folder) == {"alpha.md", "beta.md", "notes/skipme.md"}, (
|
|
f"unexpected unpacked files: {_folder_files(folder)}"
|
|
)
|
|
# …and so is the RAG catalog: it settles on its empty state (the
|
|
# visible #sources-empty is the deterministic "the load finished
|
|
# with zero documents" signal — a count-0 check alone would race
|
|
# the boot loadDocs fetch).
|
|
page.goto(app_url + SOURCES_URL)
|
|
expect(page.locator("#sources-empty")).to_be_visible(timeout=30_000)
|
|
expect(page.locator("#docs-tbody tr")).to_have_count(0)
|
|
expect(page.locator("#stat-docs")).to_have_text("0")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. The phase-89 ignore list is edited BEFORE the scan — and the RAG
|
|
# page's Sync sources button honors it (phase 90 A4)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_ignore_list_then_sync_scans(
|
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
|
) -> None:
|
|
"""The whole deferred-scan loop: upload (nothing indexed) → open
|
|
the row's phase-89 "Ignore paths" editor → type ``notes`` → Save
|
|
(the row shows the "1 ignored" count tag; the stored row carries
|
|
``ignore_paths == ["notes"]`` — the A5 round-trip) → navigate to
|
|
the RAG page → click **Sync sources** → the sync settles
|
|
("Synced HH:MM" label, "2 added" result, the status endpoint's
|
|
counts: added 2 / pruned 0 / 2 of 2 files — the ignored file is
|
|
excluded from the walk AND the denominator) → the catalog lists
|
|
``alpha.md`` + ``beta.md`` for the source and **not**
|
|
``notes/skipme.md``: the edit made before the sync is honored
|
|
(phase 90 A4 — the sync's existing kind='local' + prune +
|
|
ignore_paths path, unchanged in this phase)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
|
|
# Upload: the ready-for-sync line, and still nothing indexed.
|
|
assert _upload_via_page(page, tarball_v1) == (
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
assert _docs(page, app_url) == []
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
|
|
# The phase-89 per-row editor: open it from the row's button…
|
|
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
|
|
row.locator("button.git-source-ignore").click()
|
|
dialog = page.locator("#ignore-editor-dialog")
|
|
expect(dialog).to_be_visible(timeout=15_000)
|
|
expect(page.locator("#ignore-editor-source")).to_have_text(
|
|
str(upload_dir / SOURCE_NAME)
|
|
)
|
|
# …type the ignore entry (one path per line; the prefix rule
|
|
# excludes everything under notes/), and save.
|
|
page.fill("#ignore-editor-textarea", IGNORE_ENTRY)
|
|
page.click("#ignore-editor-save")
|
|
expect(dialog).to_be_hidden(timeout=30_000)
|
|
# The A5 round-trip: the row re-loads and shows the "1 ignored"
|
|
# count tag…
|
|
expect(row.locator(".git-source-ignore-count")).to_have_text(
|
|
"1 ignored", timeout=30_000
|
|
)
|
|
# …and the stored row carries the normalized list the sync will
|
|
# read.
|
|
r = page.request.get(f"{app_url}/api/git-sources")
|
|
assert r.status == 200, r.text
|
|
body = r.json()
|
|
assert [(s["kind"], s["path"], s["ignore_paths"]) for s in body["sources"]] == [
|
|
("local", str(upload_dir / SOURCE_NAME), [IGNORE_ENTRY])
|
|
]
|
|
|
|
# The RAG page — the catalog is still empty before the scan…
|
|
page.goto(app_url + SOURCES_URL)
|
|
expect(page.locator("#sync-btn")).to_be_visible(timeout=30_000)
|
|
expect(page.locator("#sources-empty")).to_be_visible(timeout=30_000)
|
|
expect(page.locator("#sync-label")).to_have_text("Sync sources")
|
|
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
|
|
|
# Click Sync sources (the button the deferral exists for). The
|
|
# short sync (2 files against the fast mock LLM) may settle between
|
|
# the 2 s poll ticks — the settled-state assertions below retry
|
|
# with generous timeouts instead of racing a live label.
|
|
page.click("#sync-btn")
|
|
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
|
expect(page.locator("#sync-label")).to_have_text(
|
|
SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS
|
|
)
|
|
expect(page.locator("#sync-result")).to_have_text("2 added", timeout=SYNC_TIMEOUT_MS)
|
|
expect(page.locator("#sync-btn")).to_be_enabled()
|
|
expect(page.locator("#sync-btn")).not_to_have_attribute("aria-busy")
|
|
|
|
# The status endpoint agrees: 2 added, nothing pruned, and the
|
|
# ignored file is absent from the walk's denominator (2 of 2 —
|
|
# the pre-walk uses the row's ignore list).
|
|
r = page.request.get(f"{app_url}/api/sync/status")
|
|
assert r.status == 200, r.text
|
|
status = r.json()
|
|
assert status["state"] == "success", status
|
|
assert status["detail"]["added"] == 2, status
|
|
assert status["detail"]["pruned"] == 0, status
|
|
assert status["current_file"] is None
|
|
assert status["files_done"] == 2 and status["files_total"] == 2
|
|
|
|
# The catalog: exactly the two non-ignored docs, for the source —
|
|
# and the ignored one is NOT there.
|
|
expect(page.locator("#docs-tbody tr")).to_have_count(2, timeout=30_000)
|
|
expect(page.locator("#docs-tbody tr", has_text=SOURCE_NAME)).to_have_count(2)
|
|
expect(page.locator("#docs-tbody tr", has_text="alpha.md")).to_have_count(1)
|
|
expect(page.locator("#docs-tbody tr", has_text="beta.md")).to_have_count(1)
|
|
expect(page.locator("#docs-tbody tr", has_text="notes/skipme.md")).to_have_count(0)
|
|
assert _docs(page, app_url) == [
|
|
(SOURCE_NAME, "alpha.md"),
|
|
(SOURCE_NAME, "beta.md"),
|
|
], f"the ignore list was not honored: {_docs(page, app_url)}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Re-upload, same filename → in-place replace (no duplicate row,
|
|
# folder swap on disk) — and still nothing indexed
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_reupload_replaces_without_scan(
|
|
page: Page,
|
|
app_url: str,
|
|
db_ready: None,
|
|
tarball_v1: Path,
|
|
tarball_v2: Path,
|
|
upload_dir: Path,
|
|
) -> None:
|
|
"""v1 then v2 under the SAME filename (``e2e-upload-no-scan.tar.gz``
|
|
— v2 modifies ``beta.md``, adds ``gamma.md``, drops ``alpha.md``):
|
|
both unpack + register only (phase 90). After the re-upload the
|
|
list has exactly ONE ``e2e-upload-no-scan`` row (the in-place
|
|
identity, phase-49 contract — no duplicate), the registry agrees
|
|
(one kind=local row at the unpacked path), the on-disk folder holds
|
|
ONLY v2's files (the atomic swap) — and the KB is empty
|
|
THROUGHOUT (zero documents indexed by either upload: the scan is
|
|
the Sync button's job, never the upload's)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
|
|
# Baseline: v1 through the page (202 → the ready-for-sync line,
|
|
# one row, no index).
|
|
assert _upload_via_page(page, tarball_v1) == (
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
folder = upload_dir / SOURCE_NAME
|
|
assert _folder_files(folder) == set(V1_FILES)
|
|
assert _docs(page, app_url) == []
|
|
|
|
# Re-upload v2 — SAME basename, different parent dir (the file
|
|
# input's selection is replaced wholesale).
|
|
assert _upload_via_page(page, tarball_v2) == (
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
|
|
# No duplicate: exactly ONE row for that source (and one row
|
|
# total)…
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
|
# …and the registry agrees: one kind=local row, the unpacked path.
|
|
r = page.request.get(f"{app_url}/api/git-sources")
|
|
assert r.status == 200, r.text
|
|
body = r.json()
|
|
assert [(s["kind"], s["path"]) for s in body["sources"]] == [
|
|
("local", str(upload_dir / SOURCE_NAME))
|
|
]
|
|
|
|
# The on-disk folder holds ONLY v2's files (the swap replaced the
|
|
# whole folder in place — no stale v1 file survived)…
|
|
assert _folder_files(folder) == set(V2_FILES), (
|
|
f"unexpected unpacked files: {_folder_files(folder)}"
|
|
)
|
|
# …and the KB is STILL empty (the upload never scans, phase 90 A1
|
|
# — the Sync button is what will index v2's files).
|
|
assert _docs(page, app_url) == []
|