All verification is complete and green. Final report:
**Phase 90 (upload_no_scan) — final verification pass: all criteria verified, no fixes needed**
- Verified (not re-implemented): all 3 tasks already complete; working tree carries the phase's uncommitted changes
- Upload pipeline: `_run_upload` stops after row upsert; success = `{"message": "uploaded"}`, null/0/0 progress; `UploadOut` removed from `app/schemas.py`; gates/unpack/swap/failed states intact
- Copy: button "Upload", bare "Processing…", result line "Uploaded \<name\> — press Sync sources to import it."; hint + caption re-pointed at Sync; no "Upload &" remnants in `frontend/`
- Tests: `pytest tests/unit/test_git_sources.py tests/integration/test_git_sources_upload.py tests/unit/test_frontend_sync_upload.py` → 56 passed; E2E phase suite `tests/e2e/test_upload_no_scan.py` → 3 passed (zero docs after upload; ignore edit honored by Sync — 2 added, `notes/skipme.md` excluded; re-upload in-place, still zero docs); affected suites `test_archive_upload_sources.py` + `test_sync_upload_progress.py` + `test_source_removal_cleanup.py` → 15 passed
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, 99% coverage** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors
**Completion criteria:**
1. Upload settles "ready for sync", zero docs, row + ignore editor — ✅ (E2E test 1)
2. Button "Upload", no "Upload &" copy — ✅ (`rg` empty in frontend/, app/, tests/)
3. RAG-page "Sync sources" imports upload honoring edited ignores — ✅ (isolated E2E test 2)
4. Suite green, >90% coverage, lint clean — ✅ (99%, ruff+pyright clean)
5. Atomic commit — ✅ left to harness per executor rules (no git add/commit run; tree staged-ready)
**Deviations:** `regenerate_overview` still appears once in `git_sources.py` — inside `delete_git_source` (phase-69 locked total-removal behavior, not the upload leg; upload leg is scan-free). `.agents/PLAN.md` does not exist in this repo; locked decisions A1–A4 were enforced from the phase files.
**Next pending phase:** `91_admin_theme_tab` (in `todo/`).
670 lines
29 KiB
Python
670 lines
29 KiB
Python
"""Phase 49 story E2E (Playwright): archive upload sources.
|
|
|
|
Story: ``.agents/user_stories/archive-upload-sources.md``
|
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
|
|
|
uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov
|
|
|
|
The story gate for the **archive upload** form on the admin Sources page
|
|
(``/git-sources.html``, phase 49 — the phase-38 "Add a local directory"
|
|
form is gone, replaced by this form): an uploaded
|
|
``.tar``/``.tar.gz``/``.tgz``/``.zip`` is safely unpacked under
|
|
``BOR_UPLOAD_DIR/<name>/`` (name = filename minus the archive suffix)
|
|
and the ``git_sources`` row is upserted (``kind='local'``, no
|
|
duplicates) — and **nothing else** (phase 90, owner-locked A1: no
|
|
model check, no import, no overview refresh). The POST answers **202
|
|
the moment the archive is safely on disk** (the phase-64 A1 contract —
|
|
the "Successfully uploaded — <source>" toast fires then and the user
|
|
may navigate away) while the unpack → swap → row upsert run continues
|
|
server-side behind ``GET /api/git-sources/upload/status`` (the
|
|
phase-32 ``SyncStatus`` pattern; the phase-64 key set with
|
|
``current_file``/``files_done``/``files_total`` null/0/0 for the whole
|
|
run — phase 90 A2). The settled result line points at the next step:
|
|
"Uploaded <name> — press Sync sources to import it." (phase 90 A3) —
|
|
the scan is the RAG page's **Sync sources** button's job, so this
|
|
suite asserts **zero indexed documents** after every upload.
|
|
|
|
**Timing (phase 90):** the background run is unpack + register only —
|
|
no LLM call at all — so it settles in well under the UI's 2 s status
|
|
poll and the phase-64 slow-LLM proxy is GONE from this suite. The
|
|
"Uploading… → Processing… → restored" lifecycle is made observable the
|
|
old way (the POST is held in the browser via ``page.route``) plus a
|
|
held FIRST status GET, so the bare "Processing…" in-run label (no
|
|
file, no "(n/m)") is asserted across the whole background run.
|
|
|
|
The archives are **built in-test** with Python's ``tarfile`` over
|
|
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
|
|
``BETA-…`` / ``GAMMA-…``) and are always named
|
|
``e2e-upload.tar.gz`` — so the source name is ``e2e-upload`` and
|
|
re-uploading under the same filename exercises the in-place replace
|
|
(one folder, one row). ``v1`` holds ``alpha.md`` + ``beta.md``;
|
|
``v2`` (same basename) modifies ``alpha``, drops ``beta``, adds
|
|
``gamma`` — the in-place-replace subject (the on-disk folder swap).
|
|
|
|
Per-module app env (the conftest pattern, module-scoped — as in
|
|
``test_git_sources_admin.py`` / ``test_local_directory_sources.py``):
|
|
``BOR_UPLOAD_DIR`` points at a scratch dir the suite can inspect from
|
|
the host (the app runs on the same machine), ``BOR_GIT_SOURCES`` is
|
|
forced empty so the dev ``.env``'s fallback URL never renders as an
|
|
env row on the (initially empty) table, and ``BOR_LLM_BASE_URL`` is
|
|
the mock LLM (no LLM call happens in this suite at all — phase 90
|
|
removed the upload's only LLM leg; the mock keeps the env shape
|
|
honest).
|
|
|
|
Contract under test:
|
|
|
|
* the **swap**: the phase-38 local form is gone (count 0); the upload
|
|
form is in its place with the labeled file input (accept = the four
|
|
archive extensions), the "Upload" button (phase 90 A3 — was "Upload
|
|
& scan"), and the hint explaining unpack + register only (in-place
|
|
replace, the "Sync sources" next step, the ignore-paths edit) with
|
|
no "unpack and scan" claim;
|
|
* **upload → 202 + toast → unpack-only processing → ready-for-sync**
|
|
(§7.4 never-stale, phase-64 A1/A2, phase-90 A2/A3): the button shows
|
|
"Uploading…" while the POST is in flight (held via ``page.route``);
|
|
at the 202 the "Successfully uploaded — <source>" toast fires; the
|
|
button then carries the BARE "Processing…" label for the whole
|
|
background run (no file, no "(n/m)" — the first status GET is held
|
|
so the in-run window outlives the 2 s poll) until the status
|
|
``success`` lands: the result line reads "Uploaded e2e-upload —
|
|
press Sync sources to import it.", the button restores ("Upload"),
|
|
the file input clears, the list gains exactly one row for
|
|
``e2e-upload`` with the **Local** badge and its "Ignore paths"
|
|
control (the phase-89 editor the deferral exists for), the terminal
|
|
status carries the no-count ``{"message": "uploaded"}`` detail with
|
|
null/0/0 progress — and **zero documents are indexed**:
|
|
``GET /api/docs`` is empty and the RAG catalog (``/sources.html``)
|
|
shows no rows;
|
|
* **re-upload, same filename** → in-place replace, still no index:
|
|
the folder on disk holds only the new archive's files (the atomic
|
|
swap), the list still has exactly ONE ``e2e-upload`` row (no
|
|
duplicate), the result line points at Sync again after each run,
|
|
and the KB stays empty;
|
|
* **bad file** → inline 422 (role=alert) naming the accepted formats
|
|
(UNCHANGED — the name/format/cap gates are inline, pre-202, exactly
|
|
as before), button restored ("Upload"), the file selection kept, the
|
|
list unchanged, and a subsequent good upload still works (the form
|
|
is not wedged) — and still indexes nothing;
|
|
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
|
|
manager (and thus the upload form) stays hidden, and
|
|
``POST /api/git-sources/upload`` is 403 — as is the phase-64
|
|
``GET /api/git-sources/upload/status`` (same wall).
|
|
|
|
Test → story mapping (Playwright Mapping Rule):
|
|
1. ``test_form_swapped``
|
|
2. ``test_upload_registers_without_indexing``
|
|
3. ``test_reupload_replaces_in_place``
|
|
4. ``test_bad_file_inline_error``
|
|
5. ``test_anonymous_gate``
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import io
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tarfile
|
|
import time
|
|
from collections.abc import Iterator
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
from playwright.sync_api import Page, expect
|
|
from sqlalchemy import text
|
|
|
|
from app.db import SessionLocal
|
|
from e2e.auth_helpers import login
|
|
from e2e.conftest import (
|
|
ADMIN_PASSWORD,
|
|
SESSION_SECRET,
|
|
USE_REAL_LLM,
|
|
_wait_http,
|
|
)
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
|
|
# Phase 79 (task 04, full inventory): in a combined session run the
|
|
# conftest session app already owns its port — a second uvicorn on it
|
|
# dies on bind and this suite would silently drive the wrong server.
|
|
# The module app binds its own port instead (env-overridable).
|
|
APP_PORT = int(os.environ.get("E2E_APP_PORT_ARCHIVE", "8124"))
|
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
|
|
|
GIT_SOURCES_URL = "/git-sources.html"
|
|
SOURCES_URL = "/sources.html"
|
|
|
|
#: The archive basename (both versions) — the source/folder name is the
|
|
#: filename minus the archive suffix (the phase's locked naming rule).
|
|
SOURCE_NAME = "e2e-upload"
|
|
|
|
#: v1: two sentinel docs. v2 (same filename): alpha CHANGED, beta DROPPED,
|
|
#: gamma ADDED — the in-place-replace subject (the on-disk folder swap).
|
|
ALPHA_SENTINEL_V1 = "ALPHA-TOKEN-v1-7f31"
|
|
ALPHA_SENTINEL_V2 = "ALPHA-TOKEN-v2-8b42"
|
|
BETA_SENTINEL_V1 = "BETA-TOKEN-v1-2c90"
|
|
GAMMA_SENTINEL_V2 = "GAMMA-TOKEN-v2-5e44"
|
|
|
|
V1_FILES: dict[str, str] = {
|
|
"alpha.md": (
|
|
"# Alpha note\n"
|
|
"\n"
|
|
"First version of the alpha note — it changes in v2.\n"
|
|
f"\nMarker: {ALPHA_SENTINEL_V1}\n"
|
|
),
|
|
"beta.md": (
|
|
"# Beta note\n"
|
|
"\n"
|
|
"Only present in v1 — v2 drops it.\n"
|
|
f"\nMarker: {BETA_SENTINEL_V1}\n"
|
|
),
|
|
}
|
|
V2_FILES: dict[str, str] = {
|
|
"alpha.md": (
|
|
"# Alpha note\n"
|
|
"\n"
|
|
"Second version of the alpha note — modified in place.\n"
|
|
f"\nMarker: {ALPHA_SENTINEL_V2}\n"
|
|
),
|
|
"gamma.md": (
|
|
"# Gamma note\n"
|
|
"\n"
|
|
"Brand new in v2 — the add subject of the re-upload.\n"
|
|
f"\nMarker: {GAMMA_SENTINEL_V2}\n"
|
|
),
|
|
}
|
|
|
|
#: Generous settle budget: the unpack-only run settles in milliseconds
|
|
#: (phase 90), the 4.5 s status hold dominates, and the UI's 2 s poll
|
|
#: settles one tick after the release.
|
|
UPLOAD_TIMEOUT_MS = 30_000
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixtures
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _build_targz(path: Path, files: dict[str, str]) -> Path:
|
|
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
|
|
with tarfile.open(path, "w:gz") as tf:
|
|
for rel, content in files.items():
|
|
data = content.encode("utf-8")
|
|
info = tarfile.TarInfo(rel)
|
|
info.size = len(data)
|
|
info.mtime = 0
|
|
tf.addfile(info, io.BytesIO(data))
|
|
return path
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
|
|
host-side assertions inspect (the app server runs on the same
|
|
machine). The app creates it on the first upload."""
|
|
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def tarball_v1(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""v1 — in its OWN subdirectory so v2 can reuse the same basename
|
|
(``e2e-upload.tar.gz``): the in-place-replace identity IS the
|
|
filename, and ``set_input_files`` sends the path's basename."""
|
|
root = tmp_path_factory.mktemp("bor_archive_v1")
|
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V1_FILES)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""v2 — same basename as v1 (a different parent dir)."""
|
|
root = tmp_path_factory.mktemp("bor_archive_v2")
|
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_server(
|
|
mock_llm: int,
|
|
upload_dir: Path,
|
|
tmp_path_factory: pytest.TempPathFactory,
|
|
) -> Iterator[str]:
|
|
"""The real app under test — per-module env: the LLM base URL is
|
|
the mock (phase 90 removed the upload's only LLM leg — no LLM call
|
|
happens in this suite at all; the mock keeps the env shape
|
|
honest), uploads unpack into a scratch dir, and the env git list
|
|
is forced empty (the dev ``.env``'s ``BOR_GIT_SOURCES`` must not
|
|
render as env rows on the initially empty table). No sync is
|
|
triggered here — the upload is unpack + register only, and the
|
|
no-index assertions are the point."""
|
|
env = dict(os.environ)
|
|
env.pop("DEBUGPY", None)
|
|
env["BOR_ENVIRONMENT"] = "e2e"
|
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
|
env["BOR_LLM_BASE_URL"] = (
|
|
"https://aipi.reeseapps.com/v1"
|
|
if USE_REAL_LLM
|
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
|
)
|
|
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
|
# ever sent in this suite, but the app boots with the same env shape.
|
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
|
env.setdefault(
|
|
"BOR_DATABASE_URL",
|
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
|
)
|
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
|
env["BOR_GIT_SOURCES"] = ""
|
|
# Phase 49: unpack uploads into the suite's scratch dir (host-
|
|
# inspectable) and keep the (unused) git checkouts out of the dev
|
|
# location.
|
|
env["BOR_UPLOAD_DIR"] = str(upload_dir)
|
|
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
|
proc = subprocess.Popen(
|
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
|
cwd=REPO,
|
|
env=env,
|
|
)
|
|
try:
|
|
_wait_http(f"{APP_URL}/api/health")
|
|
yield APP_URL
|
|
finally:
|
|
proc.terminate()
|
|
try:
|
|
proc.wait(timeout=10)
|
|
except subprocess.TimeoutExpired:
|
|
proc.kill()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_url(app_server: str) -> str:
|
|
return app_server
|
|
|
|
|
|
def _truncate_all() -> None:
|
|
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
|
row-count and doc-list assertions must be this test's own doing.
|
|
The E2E suites share one Postgres, and a leftover git_sources row
|
|
or document would corrupt them."""
|
|
with SessionLocal() as db:
|
|
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
|
|
db.commit()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean(db_ready: None) -> Iterator[None]:
|
|
_truncate_all()
|
|
yield
|
|
_truncate_all()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
|
"""Real form login landing on the git sources page (admin settled:
|
|
Sign out visible, the manager revealed by the page module)."""
|
|
login(page, app_url, next=GIT_SOURCES_URL)
|
|
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
|
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
|
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
|
expect(page.locator("#git-sources-content")).to_be_visible()
|
|
|
|
|
|
def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
|
|
"""``GET /api/docs`` as the signed-in page → sorted (source, path)
|
|
pairs (the admin cookie rides the browser context)."""
|
|
r = page.request.get(f"{app_url}/api/docs")
|
|
assert r.status == 200, r.text
|
|
return sorted((d["source"], d["path"]) for d in r.json()["documents"])
|
|
|
|
|
|
def _upload_via_page(page: Page, archive: Path) -> str:
|
|
"""Pick the archive, submit the form, and wait for the result line
|
|
(the phase-64 202 path: toast at the 202, then the button's status
|
|
polling renders the line from the run's ``success``) — returns its
|
|
text. The failing path is asserted explicitly by the bad-file test,
|
|
so any non-result outcome here is a test error."""
|
|
page.set_input_files("#archive-upload-file", str(archive))
|
|
page.click("#archive-upload-btn")
|
|
result = page.locator("#archive-upload-result")
|
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
|
text = result.text_content()
|
|
assert text is not None
|
|
return text
|
|
|
|
|
|
def _hold_upload_request(page: Page, hold_s: float) -> None:
|
|
"""Intercept the upload POST and hold the REQUEST in the browser for
|
|
``hold_s`` seconds before letting it reach the server. While it is
|
|
held, the page's fetch is guaranteed pending — so the §7.4 in-flight
|
|
state (disabled button, "Uploading…" label) is observable
|
|
deterministically."""
|
|
|
|
def handle(route: Any) -> None:
|
|
time.sleep(hold_s)
|
|
route.continue_()
|
|
|
|
page.route("**/api/git-sources/upload", handle)
|
|
|
|
|
|
def _hold_first_status_fetch(page: Page, hold_s: float) -> None:
|
|
"""Intercept the upload-status GETs and hold ONLY THE FIRST one for
|
|
``hold_s`` seconds (later fetches pass straight through). Install
|
|
AFTER the page's boot re-attach fetch, before the submit. The
|
|
poll's first tick fires 2 s after the 202; holding its fetch keeps
|
|
the button in the in-run state long enough to assert the bare
|
|
"Processing…" label (no file, no "(n/m)") across the whole
|
|
background run — phase 90's run settles in milliseconds, so without
|
|
the hold the in-run window is only the 2 s pre-tick gap."""
|
|
state = {"held": False}
|
|
|
|
def handle(route: Any) -> None:
|
|
if not state["held"]:
|
|
state["held"] = True
|
|
time.sleep(hold_s)
|
|
route.continue_()
|
|
|
|
page.route("**/api/git-sources/upload/status", handle)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. The swap: local form out, upload form in
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_form_swapped(page: Page, app_url: str, db_ready: None) -> None:
|
|
"""The phase-38 "Add a local directory" form is GONE and the archive
|
|
upload form stands in its place: visible file input (accept = the
|
|
four archive extensions), the "Upload" button (phase 90 A3 — the
|
|
scan-suffixed label is gone), and a hint that explains the unpack
|
|
+ register semantics (in-place replace, the "Sync sources" next
|
|
step, the ignore-paths edit) with no claim that an upload scans."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
|
|
# The phase-38 local form is gone (phase 49 replaced it)…
|
|
expect(page.locator("#local-source-form")).to_have_count(0)
|
|
expect(page.locator("#local-source-path")).to_have_count(0)
|
|
expect(page.locator("#local-source-add")).to_have_count(0)
|
|
|
|
# …and the upload form is in its place, visible with its parts.
|
|
expect(page.locator("#archive-upload-form")).to_be_visible()
|
|
file_input = page.locator("#archive-upload-file")
|
|
expect(file_input).to_be_visible()
|
|
accept = file_input.get_attribute("accept") or ""
|
|
for ext in (".tar", ".tar.gz", ".tgz", ".zip"):
|
|
assert ext in accept, f"accept={accept!r} is missing {ext!r}"
|
|
btn = page.locator("#archive-upload-btn")
|
|
expect(btn).to_be_visible()
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
# The error/result lines ship (hidden) with the right roles.
|
|
assert page.locator("#archive-upload-error").get_attribute("role") == "alert"
|
|
result = page.locator("#archive-upload-result")
|
|
assert result.get_attribute("role") == "status"
|
|
expect(result).to_be_hidden()
|
|
|
|
# The hint explains unpack + register ONLY (phase 90): in-place
|
|
# replace, the "Sync sources" next step, the ignore-paths edit —
|
|
# and no "unpack and scan" claim.
|
|
hint = page.locator("#git-sources-hint")
|
|
expect(hint).to_be_visible()
|
|
expect(hint).to_contain_text("unpack and register")
|
|
expect(hint).to_contain_text("in place")
|
|
expect(hint).to_contain_text("Sync sources")
|
|
expect(hint).to_contain_text("ignore paths")
|
|
assert "unpack and scan" not in (hint.text_content() or "")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. Upload → 202 + toast → unpack-only processing → ready-for-sync line,
|
|
# the Local row (+ Ignore paths control), ZERO documents indexed
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_upload_registers_without_indexing(
|
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
|
) -> None:
|
|
"""One real upload through the page: while the POST is in flight the
|
|
button is disabled and reads "Uploading…"; at the 202 the
|
|
"Successfully uploaded — <source>" toast fires (A2); the button
|
|
then carries the BARE "Processing…" label for the whole background
|
|
run (no file, no "(n/m)" — phase 90 A2, proven across a held first
|
|
status GET); when the status ``success`` lands the result line
|
|
reads "Uploaded <source> — press Sync sources to import it."
|
|
(A3), the button restores ("Upload"), the file input clears, the
|
|
list gains exactly ONE row for ``e2e-upload`` (Local badge, the
|
|
"Ignore paths" control), the terminal status carries the no-count
|
|
``{"message": "uploaded"}`` detail with null/0/0 progress — and
|
|
**zero documents are indexed**: ``/api/docs`` is empty and the RAG
|
|
catalog shows no rows (the scan is the Sync button's job, phase 90
|
|
A1)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
|
|
|
btn = page.locator("#archive-upload-btn")
|
|
result = page.locator("#archive-upload-result")
|
|
|
|
# Hold the upload request in the browser: the in-flight state below
|
|
# cannot race the receive while it is held. Hold the FIRST status
|
|
# GET too (installed now — after the boot re-attach fetch — so only
|
|
# the poll's ticks hit it): the bare in-run label gets a window
|
|
# wider than the 2 s pre-tick gap.
|
|
_hold_upload_request(page, hold_s=0.8)
|
|
_hold_first_status_fetch(page, hold_s=4.5)
|
|
page.set_input_files("#archive-upload-file", str(tarball_v1))
|
|
btn.click()
|
|
|
|
# In flight (§7.4): disabled + relabeled, no result yet.
|
|
expect(btn).to_be_disabled()
|
|
expect(btn).to_have_text("Uploading…")
|
|
expect(result).to_be_hidden()
|
|
|
|
# The request goes out; the server stores the archive and answers
|
|
# 202 the moment it is safely on disk (A1) → the toast fires NOW
|
|
# (A2) — and the button hands over to the background run as the
|
|
# BARE "Processing…" (phase 90 A2: the unpack has no file-level
|
|
# progress — no file, no counts, no title).
|
|
toast = page.locator(".toast")
|
|
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
|
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
|
assert toast.get_attribute("role") == "status"
|
|
expect(toast).to_have_text(f"Successfully uploaded — {SOURCE_NAME}")
|
|
expect(result).to_be_hidden()
|
|
expect(btn).to_be_disabled()
|
|
expect(btn).to_have_text("Processing…", timeout=5_000)
|
|
|
|
# The run is in flight (or just settled) server-side — and the
|
|
# label STAYS bare across the whole background run: the first
|
|
# status GET is held, so the tick that should settle the button is
|
|
# in flight — the label carries no file and no "(n/m)".
|
|
time.sleep(2.5) # just past the poll's first tick (the fetch is held)
|
|
expect(btn).to_have_text("Processing…")
|
|
|
|
# …release: the status success lands → the ready-for-sync line +
|
|
# the never-stale restore (input cleared).
|
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
|
expect(result).to_have_text(
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
expect(page.locator("#archive-upload-file")).to_have_value("")
|
|
|
|
# The terminal status: the no-count payload with null/0/0 progress
|
|
# (the phase-64 key set, phase 90 A2).
|
|
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
|
assert r.status == 200, r.text
|
|
status = r.json()
|
|
assert status["state"] == "success", status
|
|
assert status["detail"] == {"message": "uploaded"}, status
|
|
assert status["current_file"] is None
|
|
assert status["files_done"] == 0 and status["files_total"] == 0
|
|
|
|
# The list gained exactly one row — for the source, with the Local
|
|
# badge, the full unpacked path in the mono cell, and its
|
|
# "Ignore paths" control (the phase-89 editor the deferral exists
|
|
# for).
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
|
|
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
|
|
expect(row).to_have_count(1)
|
|
expect(row.locator("span.git-source-kind")).to_have_text("Local")
|
|
expect(row.locator("td.git-source-url-cell code")).to_have_text(
|
|
str(upload_dir / SOURCE_NAME)
|
|
)
|
|
expect(row.locator("button.git-source-ignore")).to_have_count(1)
|
|
expect(row.locator("button.git-source-ignore")).to_have_text("Ignore paths")
|
|
|
|
# Phase 90 A1: the upload indexes NOTHING — the KB is empty…
|
|
assert _docs(page, app_url) == []
|
|
# …and so is the RAG catalog (the scan is the Sync button's job).
|
|
page.goto(app_url + SOURCES_URL)
|
|
expect(page.locator("#docs-tbody tr")).to_have_count(0)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Re-upload, same filename → in-place replace (no duplicate row,
|
|
# folder swap on disk, still nothing indexed)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_reupload_replaces_in_place(
|
|
page: Page,
|
|
app_url: str,
|
|
db_ready: None,
|
|
tarball_v1: Path,
|
|
tarball_v2: Path,
|
|
upload_dir: Path,
|
|
) -> None:
|
|
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``) —
|
|
both unpack + register only (phase 90): the list still has exactly
|
|
ONE ``e2e-upload`` row after the re-upload (no duplicate — the
|
|
in-place identity), the on-disk folder holds only the new
|
|
archive's files (the atomic swap), the result line points at
|
|
"Sync sources" after each run, and the KB stays EMPTY throughout
|
|
(the scan is the Sync button's job)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
|
|
# Baseline: v1 through the page (202 → the ready-for-sync line,
|
|
# one row, no index).
|
|
assert _upload_via_page(page, tarball_v1) == (
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
folder = upload_dir / SOURCE_NAME
|
|
assert {p.name for p in folder.iterdir()} == set(V1_FILES)
|
|
assert _docs(page, app_url) == []
|
|
|
|
# Re-upload v2 — SAME basename, different parent dir (the file
|
|
# input's selection is replaced wholesale).
|
|
assert _upload_via_page(page, tarball_v2) == (
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
|
|
# No duplicate: exactly ONE row for that source (and one row total).
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
|
# The registry agrees: one kind=local row, the unpacked path.
|
|
r = page.request.get(f"{app_url}/api/git-sources")
|
|
assert r.status == 200, r.text
|
|
body = r.json()
|
|
assert [(s["kind"], s["path"]) for s in body["sources"]] == [
|
|
("local", str(upload_dir / SOURCE_NAME))
|
|
]
|
|
|
|
# The on-disk folder holds ONLY v2's files (the swap replaced the
|
|
# whole folder in place — no stale v1 file survived)…
|
|
assert {p.name for p in folder.iterdir()} == set(V2_FILES)
|
|
# …and the KB is STILL empty (the upload never scans, phase 90 A1
|
|
# — the Sync button is what will index v2's files).
|
|
assert _docs(page, app_url) == []
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. Bad file → inline 422; the form is not wedged
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_bad_file_inline_error(
|
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, tmp_path: Path
|
|
) -> None:
|
|
"""A ``.txt`` through the file input: the role=alert line shows the
|
|
422 detail naming the accepted formats, the button restores, the
|
|
file selection is KEPT (the fix is one re-pick), the list is
|
|
unchanged — and a subsequent good upload still works (the form is
|
|
not wedged), indexing nothing (phase 90 A1)."""
|
|
page.set_default_timeout(30_000)
|
|
_admin_git_sources_page(page, app_url)
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
|
|
|
bad = tmp_path / "notes.txt"
|
|
bad.write_text("I am not an archive.\n", encoding="utf-8")
|
|
|
|
error = page.locator("#archive-upload-error")
|
|
btn = page.locator("#archive-upload-btn")
|
|
page.set_input_files("#archive-upload-file", str(bad))
|
|
btn.click()
|
|
|
|
# The 422 detail inline (role=alert), naming the accepted formats.
|
|
expect(error).to_be_visible(timeout=30_000)
|
|
assert error.get_attribute("role") == "alert"
|
|
expect(error).to_contain_text("only .tar, .tar.gz, .tgz or .zip archives are accepted")
|
|
|
|
# Never stale + the selection kept + no result line + list unchanged.
|
|
expect(btn).to_be_enabled()
|
|
expect(btn).to_have_text("Upload")
|
|
# The selection is kept (the fix is one re-pick) — Chromium reports
|
|
# a fake path (``…/notes.txt``), so assert on the basename.
|
|
bad_value = page.locator("#archive-upload-file").input_value()
|
|
assert bad_value.endswith("notes.txt"), bad_value
|
|
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
|
|
|
# The form is not wedged: a good upload right after still works —
|
|
# and indexes nothing (phase 90 A1).
|
|
assert _upload_via_page(page, tarball_v1) == (
|
|
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
|
)
|
|
expect(error).to_be_hidden()
|
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
|
assert _docs(page, app_url) == []
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 5. Anonymous: the gate, the hidden manager, the 403
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_anonymous_gate(page: Page, app_url: str, db_ready: None) -> None:
|
|
"""Anonymous on the page: the sign-in gate shows, the manager (and
|
|
thus the upload form) stays hidden, and the upload route 403s
|
|
(``require_admin`` — A10)."""
|
|
page.set_default_timeout(30_000)
|
|
|
|
page.goto(app_url + GIT_SOURCES_URL)
|
|
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
|
|
expect(page.locator("#sign-out-btn")).to_be_hidden()
|
|
|
|
gate = page.locator("#git-sources-gate")
|
|
expect(gate).to_be_visible()
|
|
expect(gate).to_contain_text("Sign in to manage the git sources")
|
|
|
|
# The manager is hidden — so is the upload form inside it.
|
|
expect(page.locator("#git-sources-content")).to_be_hidden()
|
|
expect(page.locator("#archive-upload-form")).to_be_hidden()
|
|
|
|
# The upload route 403s anonymous callers (require_admin runs before
|
|
# the multipart body is parsed — the body is a stand-in, the
|
|
# test_local_directory_sources.py pattern for this route)…
|
|
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
|
|
assert r.status == 403
|
|
# …and so does the phase-64 upload STATUS endpoint (same wall).
|
|
assert page.request.get(f"{app_url}/api/git-sources/upload/status").status == 403
|