phase: 90_upload_no_scan
All verification is complete and green. Final report:
**Phase 90 (upload_no_scan) — final verification pass: all criteria verified, no fixes needed**
- Verified (not re-implemented): all 3 tasks already complete; working tree carries the phase's uncommitted changes
- Upload pipeline: `_run_upload` stops after row upsert; success = `{"message": "uploaded"}`, null/0/0 progress; `UploadOut` removed from `app/schemas.py`; gates/unpack/swap/failed states intact
- Copy: button "Upload", bare "Processing…", result line "Uploaded \<name\> — press Sync sources to import it."; hint + caption re-pointed at Sync; no "Upload &" remnants in `frontend/`
- Tests: `pytest tests/unit/test_git_sources.py tests/integration/test_git_sources_upload.py tests/unit/test_frontend_sync_upload.py` → 56 passed; E2E phase suite `tests/e2e/test_upload_no_scan.py` → 3 passed (zero docs after upload; ignore edit honored by Sync — 2 added, `notes/skipme.md` excluded; re-upload in-place, still zero docs); affected suites `test_archive_upload_sources.py` + `test_sync_upload_progress.py` + `test_source_removal_cleanup.py` → 15 passed
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, 99% coverage** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors
**Completion criteria:**
1. Upload settles "ready for sync", zero docs, row + ignore editor — ✅ (E2E test 1)
2. Button "Upload", no "Upload &" copy — ✅ (`rg` empty in frontend/, app/, tests/)
3. RAG-page "Sync sources" imports upload honoring edited ignores — ✅ (isolated E2E test 2)
4. Suite green, >90% coverage, lint clean — ✅ (99%, ruff+pyright clean)
5. Atomic commit — ✅ left to harness per executor rules (no git add/commit run; tree staged-ready)
**Deviations:** `regenerate_overview` still appears once in `git_sources.py` — inside `delete_git_source` (phase-69 locked total-removal behavior, not the upload leg; upload leg is scan-free). `.agents/PLAN.md` does not exist in this repo; locked decisions A1–A4 were enforced from the phase files.
**Next pending phase:** `91_admin_theme_tab` (in `todo/`).
This commit is contained in:
@@ -7,76 +7,84 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
The story gate for the **archive upload** form on the admin Sources page
|
||||
(``/git-sources.html``, phase 49 — the phase-38 "Add a local directory"
|
||||
form is gone, replaced by this form): an uploaded ``.tar``/``.tar.gz``/
|
||||
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
|
||||
(name = filename minus the archive suffix), the ``git_sources`` row is
|
||||
upserted (``kind='local'``, no duplicates), and the source is **scanned
|
||||
in a background task** (phase 64, task 03 — owner-locked A1: the POST
|
||||
answers **202 the moment the archive is safely on disk** — the
|
||||
"Successfully uploaded — <source>" toast fires then and the user may
|
||||
navigate away — while unpack → swap → row upsert → model check →
|
||||
single-source ``import_sources`` with ``prune=True`` + the change-gated
|
||||
overview refresh run server-side behind
|
||||
``GET /api/git-sources/upload/status``, the phase-32 ``SyncStatus``
|
||||
pattern with the phase-64 ``current_file``/counts). The result line and
|
||||
the row land from the status ``success`` (same ``UploadOut`` counts,
|
||||
uncompressed in shape) — the real pipeline, against the deterministic
|
||||
mock LLM (no real models, no network beyond the app itself).
|
||||
form is gone, replaced by this form): an uploaded
|
||||
``.tar``/``.tar.gz``/``.tgz``/``.zip`` is safely unpacked under
|
||||
``BOR_UPLOAD_DIR/<name>/`` (name = filename minus the archive suffix)
|
||||
and the ``git_sources`` row is upserted (``kind='local'``, no
|
||||
duplicates) — and **nothing else** (phase 90, owner-locked A1: no
|
||||
model check, no import, no overview refresh). The POST answers **202
|
||||
the moment the archive is safely on disk** (the phase-64 A1 contract —
|
||||
the "Successfully uploaded — <source>" toast fires then and the user
|
||||
may navigate away) while the unpack → swap → row upsert run continues
|
||||
server-side behind ``GET /api/git-sources/upload/status`` (the
|
||||
phase-32 ``SyncStatus`` pattern; the phase-64 key set with
|
||||
``current_file``/``files_done``/``files_total`` null/0/0 for the whole
|
||||
run — phase 90 A2). The settled result line points at the next step:
|
||||
"Uploaded <name> — press Sync sources to import it." (phase 90 A3) —
|
||||
the scan is the RAG page's **Sync sources** button's job, so this
|
||||
suite asserts **zero indexed documents** after every upload.
|
||||
|
||||
**Timing fixture (phase 64):** the mock LLM answers instantly, so this
|
||||
module's app boots behind ``tests/e2e/slow_llm.py`` — a delay-injecting
|
||||
reverse proxy in front of it (``SLOW_DELAY_S`` per request). A 2-file
|
||||
scan is 5 LLM requests ≈ 5 × 0.6 s ≈ 3 s: long enough to outlive the
|
||||
UI's 2 s status poll, so the button's literal
|
||||
"Uploading… → Processing… → restored" lifecycle is observable
|
||||
(the "Processing…" tick even carries the live file, A4) instead of
|
||||
racing the mock.
|
||||
**Timing (phase 90):** the background run is unpack + register only —
|
||||
no LLM call at all — so it settles in well under the UI's 2 s status
|
||||
poll and the phase-64 slow-LLM proxy is GONE from this suite. The
|
||||
"Uploading… → Processing… → restored" lifecycle is made observable the
|
||||
old way (the POST is held in the browser via ``page.route``) plus a
|
||||
held FIRST status GET, so the bare "Processing…" in-run label (no
|
||||
file, no "(n/m)") is asserted across the whole background run.
|
||||
|
||||
The archives are **built in-test** with Python's ``tarfile`` over
|
||||
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
|
||||
``BETA-…`` / ``GAMMA-…``) and are always named
|
||||
``e2e-upload.tar.gz`` — so the source name is ``e2e-upload`` and
|
||||
re-uploading under the same filename exercises the in-place replace
|
||||
(one folder, one row, dropped files pruned from the KB). ``v1`` holds
|
||||
``alpha.md`` + ``beta.md``; ``v2`` (same basename) modifies ``alpha``,
|
||||
drops ``beta``, adds ``gamma``.
|
||||
(one folder, one row). ``v1`` holds ``alpha.md`` + ``beta.md``;
|
||||
``v2`` (same basename) modifies ``alpha``, drops ``beta``, adds
|
||||
``gamma`` — the in-place-replace subject (the on-disk folder swap).
|
||||
|
||||
Per-module app env (the conftest pattern, module-scoped — as in
|
||||
``test_git_sources_admin.py`` / ``test_local_directory_sources.py``):
|
||||
``BOR_UPLOAD_DIR`` points at a scratch dir the suite can inspect from
|
||||
the host (the app runs on the same machine), and
|
||||
``BOR_GIT_SOURCES`` is forced empty so the dev ``.env``'s fallback URL
|
||||
never renders as an env row on the (initially empty) table.
|
||||
the host (the app runs on the same machine), ``BOR_GIT_SOURCES`` is
|
||||
forced empty so the dev ``.env``'s fallback URL never renders as an
|
||||
env row on the (initially empty) table, and ``BOR_LLM_BASE_URL`` is
|
||||
the mock LLM (no LLM call happens in this suite at all — phase 90
|
||||
removed the upload's only LLM leg; the mock keeps the env shape
|
||||
honest).
|
||||
|
||||
Contract under test:
|
||||
|
||||
* the **swap** (task 03): the phase-38 local form is gone (count 0);
|
||||
the upload form is in its place with the labeled file input (accept
|
||||
= the four archive extensions), the "Upload & scan" button, and the
|
||||
hint explains unpack/scan + in-place replace;
|
||||
* **upload → 202 + toast → background scan → list** (§7.4 never-stale,
|
||||
phase-64 A1/A2): the button shows "Uploading…" while the POST is in
|
||||
flight (the request is held in the browser via ``page.route`` so the
|
||||
in-flight state is deterministic); at the 202 the "Successfully
|
||||
uploaded — <source>" toast fires (``.toast.is-visible``,
|
||||
``role="status"``) WHILE the scan is still running, and the button
|
||||
hands over to the scan — "Processing…" (the live-file tick carries
|
||||
the current file, A4) — then restores when the status ``success``
|
||||
lands: the result line shows the added count; the list gains exactly
|
||||
one row for ``e2e-upload`` with the **Local** badge; ``GET
|
||||
/api/docs`` lists both sentinel files under source ``e2e-upload``;
|
||||
the RAG catalog (``/sources.html``) shows them;
|
||||
* **re-upload, same filename** → in-place replace: the result line
|
||||
shows the prune, the SECOND RUN'S STATUS ``detail`` carries the
|
||||
prune/refresh counts, the list still has exactly ONE ``e2e-upload``
|
||||
row (no duplicate), the KB shows the changed ``alpha`` + the new
|
||||
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
|
||||
only the new archive's files;
|
||||
* the **swap**: the phase-38 local form is gone (count 0); the upload
|
||||
form is in its place with the labeled file input (accept = the four
|
||||
archive extensions), the "Upload" button (phase 90 A3 — was "Upload
|
||||
& scan"), and the hint explaining unpack + register only (in-place
|
||||
replace, the "Sync sources" next step, the ignore-paths edit) with
|
||||
no "unpack and scan" claim;
|
||||
* **upload → 202 + toast → unpack-only processing → ready-for-sync**
|
||||
(§7.4 never-stale, phase-64 A1/A2, phase-90 A2/A3): the button shows
|
||||
"Uploading…" while the POST is in flight (held via ``page.route``);
|
||||
at the 202 the "Successfully uploaded — <source>" toast fires; the
|
||||
button then carries the BARE "Processing…" label for the whole
|
||||
background run (no file, no "(n/m)" — the first status GET is held
|
||||
so the in-run window outlives the 2 s poll) until the status
|
||||
``success`` lands: the result line reads "Uploaded e2e-upload —
|
||||
press Sync sources to import it.", the button restores ("Upload"),
|
||||
the file input clears, the list gains exactly one row for
|
||||
``e2e-upload`` with the **Local** badge and its "Ignore paths"
|
||||
control (the phase-89 editor the deferral exists for), the terminal
|
||||
status carries the no-count ``{"message": "uploaded"}`` detail with
|
||||
null/0/0 progress — and **zero documents are indexed**:
|
||||
``GET /api/docs`` is empty and the RAG catalog (``/sources.html``)
|
||||
shows no rows;
|
||||
* **re-upload, same filename** → in-place replace, still no index:
|
||||
the folder on disk holds only the new archive's files (the atomic
|
||||
swap), the list still has exactly ONE ``e2e-upload`` row (no
|
||||
duplicate), the result line points at Sync again after each run,
|
||||
and the KB stays empty;
|
||||
* **bad file** → inline 422 (role=alert) naming the accepted formats
|
||||
(UNCHANGED — the name/format/cap gates are inline, pre-202, exactly
|
||||
as before), button restored, the file selection kept, the list
|
||||
unchanged, and a subsequent good upload still works (the form is not
|
||||
wedged);
|
||||
as before), button restored ("Upload"), the file selection kept, the
|
||||
list unchanged, and a subsequent good upload still works (the form
|
||||
is not wedged) — and still indexes nothing;
|
||||
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
|
||||
manager (and thus the upload form) stays hidden, and
|
||||
``POST /api/git-sources/upload`` is 403 — as is the phase-64
|
||||
@@ -84,7 +92,7 @@ Contract under test:
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_form_swapped``
|
||||
2. ``test_upload_scans_and_lists``
|
||||
2. ``test_upload_registers_without_indexing``
|
||||
3. ``test_reupload_replaces_in_place``
|
||||
4. ``test_bad_file_inline_error``
|
||||
5. ``test_anonymous_gate``
|
||||
@@ -110,7 +118,6 @@ from app.db import SessionLocal
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
MOCK_PORT,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
@@ -125,17 +132,6 @@ REPO = Path(__file__).resolve().parents[2]
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_ARCHIVE", "8124"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
#: The slow-LLM proxy's port (the conftest's mock LLM stays on MOCK_PORT).
|
||||
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT", "8902"))
|
||||
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
|
||||
|
||||
#: Per-LLM-request delay on the proxy — a 2-file scan is 5 LLM requests
|
||||
#: (the check_models embed + chat probe, one embed per file, the
|
||||
#: change-gated overview chat) ≈ 5 × 0.6 s ≈ 3 s: the scan outlives the
|
||||
#: UI's 2 s status poll, so the button's "Uploading… → Processing… →
|
||||
#: restored" lifecycle (with the live-file tick, A4) is observable.
|
||||
SLOW_DELAY_S = "0.6"
|
||||
|
||||
GIT_SOURCES_URL = "/git-sources.html"
|
||||
SOURCES_URL = "/sources.html"
|
||||
|
||||
@@ -144,7 +140,7 @@ SOURCES_URL = "/sources.html"
|
||||
SOURCE_NAME = "e2e-upload"
|
||||
|
||||
#: v1: two sentinel docs. v2 (same filename): alpha CHANGED, beta DROPPED,
|
||||
#: gamma ADDED — the in-place-replace subject.
|
||||
#: gamma ADDED — the in-place-replace subject (the on-disk folder swap).
|
||||
ALPHA_SENTINEL_V1 = "ALPHA-TOKEN-v1-7f31"
|
||||
ALPHA_SENTINEL_V2 = "ALPHA-TOKEN-v2-8b42"
|
||||
BETA_SENTINEL_V1 = "BETA-TOKEN-v1-2c90"
|
||||
@@ -160,7 +156,7 @@ V1_FILES: dict[str, str] = {
|
||||
"beta.md": (
|
||||
"# Beta note\n"
|
||||
"\n"
|
||||
"Only present in v1 — v2 drops it (the prune subject).\n"
|
||||
"Only present in v1 — v2 drops it.\n"
|
||||
f"\nMarker: {BETA_SENTINEL_V1}\n"
|
||||
),
|
||||
}
|
||||
@@ -179,10 +175,10 @@ V2_FILES: dict[str, str] = {
|
||||
),
|
||||
}
|
||||
|
||||
#: The scan runs the full pipeline against the mock LLM (models probe +
|
||||
#: embed batch + per-doc summaries + the change-gated overview) —
|
||||
#: generous, like the sync suites; no client-side hard timeout.
|
||||
UPLOAD_TIMEOUT_MS = 90_000
|
||||
#: Generous settle budget: the unpack-only run settles in milliseconds
|
||||
#: (phase 90), the 4.5 s status hold dominates, and the UI's 2 s poll
|
||||
#: settles one tick after the release.
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -226,47 +222,20 @@ def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def slow_llm(mock_llm: int) -> Iterator[int]:
|
||||
"""The delay-injecting reverse proxy in front of the mock LLM
|
||||
(tests/e2e/slow_llm.py) — this suite's timing fixture: the phase-64
|
||||
button lifecycle ("Uploading… → Processing… → restored") needs the
|
||||
2-file scan to outlive the UI's 2 s status poll (see
|
||||
``SLOW_DELAY_S``)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
|
||||
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
|
||||
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{SLOW_URL}/v1/models")
|
||||
yield SLOW_PORT
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int,
|
||||
slow_llm: int,
|
||||
upload_dir: Path,
|
||||
tmp_path_factory: pytest.TempPathFactory,
|
||||
) -> Iterator[str]:
|
||||
"""The real app under test — per-module env: the LLM base URL is the
|
||||
SLOW PROXY in front of the mock (the timing fixture), uploads unpack
|
||||
into a scratch dir and the env git list is forced empty (the dev
|
||||
``.env``'s ``BOR_GIT_SOURCES`` must not render as env rows on the
|
||||
initially empty table). No sync is triggered here — the upload's own
|
||||
scan is the pipeline under test."""
|
||||
"""The real app under test — per-module env: the LLM base URL is
|
||||
the mock (phase 90 removed the upload's only LLM leg — no LLM call
|
||||
happens in this suite at all; the mock keeps the env shape
|
||||
honest), uploads unpack into a scratch dir, and the env git list
|
||||
is forced empty (the dev ``.env``'s ``BOR_GIT_SOURCES`` must not
|
||||
render as env rows on the initially empty table). No sync is
|
||||
triggered here — the upload is unpack + register only, and the
|
||||
no-index assertions are the point."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
@@ -274,7 +243,7 @@ def app_server(
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"{SLOW_URL}/v1"
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
||||
# ever sent in this suite, but the app boots with the same env shape.
|
||||
@@ -316,11 +285,9 @@ def app_url(app_server: str) -> str:
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
||||
upload's counts and every ``/api/docs`` assertion must be this
|
||||
test's own doing. The E2E suites share one Postgres, and a leftover
|
||||
git_sources row or document would corrupt the row-count and doc-list
|
||||
assertions (and a leftover document under the same source name would
|
||||
survive the re-upload's single-source prune)."""
|
||||
row-count and doc-list assertions must be this test's own doing.
|
||||
The E2E suites share one Postgres, and a leftover git_sources row
|
||||
or document would corrupt them."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
|
||||
db.commit()
|
||||
@@ -371,30 +338,12 @@ def _upload_via_page(page: Page, archive: Path) -> str:
|
||||
return text
|
||||
|
||||
|
||||
def _wait_upload_running(page: Page, app_url: str, timeout_s: float = 15.0) -> dict[str, Any]:
|
||||
"""Poll (cookie-authenticated) the upload status endpoint until the
|
||||
run is ``running`` — the phase-64 single source of truth for the
|
||||
background scan (A1)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
||||
assert r.status == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] == "running":
|
||||
return body
|
||||
if body["state"] in ("success", "failed"):
|
||||
raise AssertionError(f"the scan settled too fast to observe: {body}")
|
||||
time.sleep(0.1)
|
||||
raise AssertionError(f"the scan never entered running: {body}")
|
||||
|
||||
|
||||
def _hold_upload_request(page: Page, hold_s: float) -> None:
|
||||
"""Intercept the upload POST and hold the REQUEST in the browser for
|
||||
``hold_s`` seconds before letting it reach the server. While it is
|
||||
held, the page's fetch is guaranteed pending — so the §7.4 in-flight
|
||||
state (disabled button, "Uploading…" label) is observable
|
||||
deterministically instead of racing the mock LLM's fast scan."""
|
||||
deterministically."""
|
||||
|
||||
def handle(route: Any) -> None:
|
||||
time.sleep(hold_s)
|
||||
@@ -403,6 +352,26 @@ def _hold_upload_request(page: Page, hold_s: float) -> None:
|
||||
page.route("**/api/git-sources/upload", handle)
|
||||
|
||||
|
||||
def _hold_first_status_fetch(page: Page, hold_s: float) -> None:
|
||||
"""Intercept the upload-status GETs and hold ONLY THE FIRST one for
|
||||
``hold_s`` seconds (later fetches pass straight through). Install
|
||||
AFTER the page's boot re-attach fetch, before the submit. The
|
||||
poll's first tick fires 2 s after the 202; holding its fetch keeps
|
||||
the button in the in-run state long enough to assert the bare
|
||||
"Processing…" label (no file, no "(n/m)") across the whole
|
||||
background run — phase 90's run settles in milliseconds, so without
|
||||
the hold the in-run window is only the 2 s pre-tick gap."""
|
||||
state = {"held": False}
|
||||
|
||||
def handle(route: Any) -> None:
|
||||
if not state["held"]:
|
||||
state["held"] = True
|
||||
time.sleep(hold_s)
|
||||
route.continue_()
|
||||
|
||||
page.route("**/api/git-sources/upload/status", handle)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The swap: local form out, upload form in
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -411,8 +380,10 @@ def _hold_upload_request(page: Page, hold_s: float) -> None:
|
||||
def test_form_swapped(page: Page, app_url: str, db_ready: None) -> None:
|
||||
"""The phase-38 "Add a local directory" form is GONE and the archive
|
||||
upload form stands in its place: visible file input (accept = the
|
||||
four archive extensions), the "Upload & scan" button, and a hint
|
||||
that explains the unpack/scan + in-place-replace semantics."""
|
||||
four archive extensions), the "Upload" button (phase 90 A3 — the
|
||||
scan-suffixed label is gone), and a hint that explains the unpack
|
||||
+ register semantics (in-place replace, the "Sync sources" next
|
||||
step, the ignore-paths edit) with no claim that an upload scans."""
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
|
||||
@@ -431,40 +402,48 @@ def test_form_swapped(page: Page, app_url: str, db_ready: None) -> None:
|
||||
btn = page.locator("#archive-upload-btn")
|
||||
expect(btn).to_be_visible()
|
||||
expect(btn).to_be_enabled()
|
||||
expect(btn).to_have_text("Upload & scan")
|
||||
expect(btn).to_have_text("Upload")
|
||||
# The error/result lines ship (hidden) with the right roles.
|
||||
assert page.locator("#archive-upload-error").get_attribute("role") == "alert"
|
||||
result = page.locator("#archive-upload-result")
|
||||
assert result.get_attribute("role") == "status"
|
||||
expect(result).to_be_hidden()
|
||||
|
||||
# The hint explains unpack/scan + in-place replace (task 03).
|
||||
# The hint explains unpack + register ONLY (phase 90): in-place
|
||||
# replace, the "Sync sources" next step, the ignore-paths edit —
|
||||
# and no "unpack and scan" claim.
|
||||
hint = page.locator("#git-sources-hint")
|
||||
expect(hint).to_be_visible()
|
||||
expect(hint).to_contain_text("unpack")
|
||||
expect(hint).to_contain_text("scan")
|
||||
expect(hint).to_contain_text("unpack and register")
|
||||
expect(hint).to_contain_text("in place")
|
||||
expect(hint).to_contain_text("Sync sources")
|
||||
expect(hint).to_contain_text("ignore paths")
|
||||
assert "unpack and scan" not in (hint.text_content() or "")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Upload → scan → list (the §7.4 in-flight state, the counts, the
|
||||
# Local row, the KB, the RAG catalog)
|
||||
# 2. Upload → 202 + toast → unpack-only processing → ready-for-sync line,
|
||||
# the Local row (+ Ignore paths control), ZERO documents indexed
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_upload_scans_and_lists(
|
||||
def test_upload_registers_without_indexing(
|
||||
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
||||
) -> None:
|
||||
"""One real upload through the page: while the POST is in flight the
|
||||
button is disabled and reads "Uploading…"; at the 202 the
|
||||
"Successfully uploaded — <source>" toast fires (A2) WHILE the
|
||||
background scan is still running and the button hands over to it —
|
||||
"Processing…" (the 2 s poll tick carries the live file, A4); when
|
||||
the status ``success`` lands the button restores, the result line
|
||||
shows the added count (2), the file input clears, the list gains
|
||||
exactly ONE row for ``e2e-upload`` with the Local badge,
|
||||
``/api/docs`` lists both sentinel files under the source, and the
|
||||
RAG catalog shows them where the admin expects them."""
|
||||
"Successfully uploaded — <source>" toast fires (A2); the button
|
||||
then carries the BARE "Processing…" label for the whole background
|
||||
run (no file, no "(n/m)" — phase 90 A2, proven across a held first
|
||||
status GET); when the status ``success`` lands the result line
|
||||
reads "Uploaded <source> — press Sync sources to import it."
|
||||
(A3), the button restores ("Upload"), the file input clears, the
|
||||
list gains exactly ONE row for ``e2e-upload`` (Local badge, the
|
||||
"Ignore paths" control), the terminal status carries the no-count
|
||||
``{"message": "uploaded"}`` detail with null/0/0 progress — and
|
||||
**zero documents are indexed**: ``/api/docs`` is empty and the RAG
|
||||
catalog shows no rows (the scan is the Sync button's job, phase 90
|
||||
A1)."""
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
||||
@@ -473,8 +452,12 @@ def test_upload_scans_and_lists(
|
||||
result = page.locator("#archive-upload-result")
|
||||
|
||||
# Hold the upload request in the browser: the in-flight state below
|
||||
# cannot race the receive while it is held.
|
||||
# cannot race the receive while it is held. Hold the FIRST status
|
||||
# GET too (installed now — after the boot re-attach fetch — so only
|
||||
# the poll's ticks hit it): the bare in-run label gets a window
|
||||
# wider than the 2 s pre-tick gap.
|
||||
_hold_upload_request(page, hold_s=0.8)
|
||||
_hold_first_status_fetch(page, hold_s=4.5)
|
||||
page.set_input_files("#archive-upload-file", str(tarball_v1))
|
||||
btn.click()
|
||||
|
||||
@@ -485,9 +468,9 @@ def test_upload_scans_and_lists(
|
||||
|
||||
# The request goes out; the server stores the archive and answers
|
||||
# 202 the moment it is safely on disk (A1) → the toast fires NOW
|
||||
# (A2) — while the scan is still running — and the button hands
|
||||
# over to the scan (bare "Processing…" — A4: no file yet during the
|
||||
# unpack phase).
|
||||
# (A2) — and the button hands over to the background run as the
|
||||
# BARE "Processing…" (phase 90 A2: the unpack has no file-level
|
||||
# progress — no file, no counts, no title).
|
||||
toast = page.locator(".toast")
|
||||
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
||||
@@ -497,27 +480,37 @@ def test_upload_scans_and_lists(
|
||||
expect(btn).to_be_disabled()
|
||||
expect(btn).to_have_text("Processing…", timeout=5_000)
|
||||
|
||||
# The scan is running server-side (the status endpoint is the
|
||||
# single source of truth, A1) — the run the UI's poll tracks.
|
||||
_wait_upload_running(page, app_url)
|
||||
# The run is in flight (or just settled) server-side — and the
|
||||
# label STAYS bare across the whole background run: the first
|
||||
# status GET is held, so the tick that should settle the button is
|
||||
# in flight — the label carries no file and no "(n/m)".
|
||||
time.sleep(2.5) # just past the poll's first tick (the fetch is held)
|
||||
expect(btn).to_have_text("Processing…")
|
||||
|
||||
# …and the button's 2 s poll tick renders the live file label
|
||||
# ("Processing… <file> (n/m)", A4).
|
||||
expect(btn).to_have_text(
|
||||
re.compile(rf"Processing… {re.escape(SOURCE_NAME)}/.+\.md"),
|
||||
timeout=UPLOAD_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
# The status success lands → the result line (the same UploadOut
|
||||
# counts) + the never-stale restore (input cleared).
|
||||
# …release: the status success lands → the ready-for-sync line +
|
||||
# the never-stale restore (input cleared).
|
||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(result).to_have_text("2 added")
|
||||
expect(result).to_have_text(
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
||||
)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(btn).to_have_text("Upload & scan")
|
||||
expect(btn).to_have_text("Upload")
|
||||
expect(page.locator("#archive-upload-file")).to_have_value("")
|
||||
|
||||
# The terminal status: the no-count payload with null/0/0 progress
|
||||
# (the phase-64 key set, phase 90 A2).
|
||||
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
||||
assert r.status == 200, r.text
|
||||
status = r.json()
|
||||
assert status["state"] == "success", status
|
||||
assert status["detail"] == {"message": "uploaded"}, status
|
||||
assert status["current_file"] is None
|
||||
assert status["files_done"] == 0 and status["files_total"] == 0
|
||||
|
||||
# The list gained exactly one row — for the source, with the Local
|
||||
# badge and the full unpacked path in the mono cell.
|
||||
# badge, the full unpacked path in the mono cell, and its
|
||||
# "Ignore paths" control (the phase-89 editor the deferral exists
|
||||
# for).
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
|
||||
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
|
||||
expect(row).to_have_count(1)
|
||||
@@ -525,21 +518,19 @@ def test_upload_scans_and_lists(
|
||||
expect(row.locator("td.git-source-url-cell code")).to_have_text(
|
||||
str(upload_dir / SOURCE_NAME)
|
||||
)
|
||||
expect(row.locator("button.git-source-ignore")).to_have_count(1)
|
||||
expect(row.locator("button.git-source-ignore")).to_have_text("Ignore paths")
|
||||
|
||||
# The KB: both sentinel files, under the source name e2e-upload.
|
||||
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "beta.md")]
|
||||
|
||||
# The RAG catalog (admin sees it): both docs, under the source.
|
||||
# Phase 90 A1: the upload indexes NOTHING — the KB is empty…
|
||||
assert _docs(page, app_url) == []
|
||||
# …and so is the RAG catalog (the scan is the Sync button's job).
|
||||
page.goto(app_url + SOURCES_URL)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
||||
expect(page.locator("#docs-tbody tr", has_text="alpha.md")).to_have_count(1)
|
||||
expect(page.locator("#docs-tbody tr", has_text="beta.md")).to_have_count(1)
|
||||
expect(page.locator("#docs-tbody tr", has_text=SOURCE_NAME)).to_have_count(2)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Re-upload, same filename → in-place replace (no duplicate row,
|
||||
# dropped file pruned, changed/new file indexed)
|
||||
# folder swap on disk, still nothing indexed)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -551,39 +542,31 @@ def test_reupload_replaces_in_place(
|
||||
tarball_v2: Path,
|
||||
upload_dir: Path,
|
||||
) -> None:
|
||||
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
|
||||
result line shows the prune, the SECOND RUN'S STATUS ``detail``
|
||||
carries the prune/refresh counts (phase 64 — the line is rendered
|
||||
from the status success), the list still has exactly ONE
|
||||
``e2e-upload`` row (the row count for that source is invariant — no
|
||||
duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
|
||||
and NOT the dropped ``beta``, and the on-disk folder holds only the
|
||||
new archive's files."""
|
||||
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``) —
|
||||
both unpack + register only (phase 90): the list still has exactly
|
||||
ONE ``e2e-upload`` row after the re-upload (no duplicate — the
|
||||
in-place identity), the on-disk folder holds only the new
|
||||
archive's files (the atomic swap), the result line points at
|
||||
"Sync sources" after each run, and the KB stays EMPTY throughout
|
||||
(the scan is the Sync button's job)."""
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
|
||||
# Baseline: v1 through the page (202 → "2 added", one row).
|
||||
assert _upload_via_page(page, tarball_v1) == "2 added"
|
||||
# Baseline: v1 through the page (202 → the ready-for-sync line,
|
||||
# one row, no index).
|
||||
assert _upload_via_page(page, tarball_v1) == (
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
||||
)
|
||||
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
||||
folder = upload_dir / SOURCE_NAME
|
||||
assert {p.name for p in folder.iterdir()} == set(V1_FILES)
|
||||
assert _docs(page, app_url) == []
|
||||
|
||||
# Re-upload v2 — SAME basename, different parent dir (the file
|
||||
# input's selection is replaced wholesale).
|
||||
assert _upload_via_page(page, tarball_v2) is not None
|
||||
result = page.locator("#archive-upload-result")
|
||||
expect(result).to_have_text(re.compile(r"\d+ pruned"))
|
||||
|
||||
# The SECOND RUN's status ``detail`` shows the prune/refresh counts
|
||||
# (phase 64: the result line is rendered from this success).
|
||||
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
||||
assert r.status == 200, r.text
|
||||
status = r.json()
|
||||
assert status["state"] == "success", status
|
||||
detail = status["detail"]
|
||||
assert detail["source"] == SOURCE_NAME
|
||||
assert detail["files"] == 2
|
||||
assert detail["added"] == 1 # gamma — new in v2
|
||||
assert detail["updated"] == 1 # alpha — changed in v2
|
||||
assert detail["pruned"] == 1 # beta — dropped in v2
|
||||
assert _upload_via_page(page, tarball_v2) == (
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
||||
)
|
||||
|
||||
# No duplicate: exactly ONE row for that source (and one row total).
|
||||
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
||||
@@ -596,21 +579,12 @@ def test_reupload_replaces_in_place(
|
||||
("local", str(upload_dir / SOURCE_NAME))
|
||||
]
|
||||
|
||||
# The KB: gamma + the CHANGED alpha, NOT the dropped beta.
|
||||
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "gamma.md")]
|
||||
# …and the indexed alpha is the v2 one (in-place replace, proven in
|
||||
# the KB, not just the filesystem).
|
||||
content = page.request.get(
|
||||
f"{app_url}/api/documents/content?source={SOURCE_NAME}&path=alpha.md"
|
||||
)
|
||||
assert content.status == 200, content.text
|
||||
assert ALPHA_SENTINEL_V2 in content.json()["content"]
|
||||
assert ALPHA_SENTINEL_V1 not in content.json()["content"]
|
||||
|
||||
# The on-disk folder holds ONLY v2's files (the swap replaced the
|
||||
# whole folder — no stale v1 file survived).
|
||||
folder = upload_dir / SOURCE_NAME
|
||||
# whole folder in place — no stale v1 file survived)…
|
||||
assert {p.name for p in folder.iterdir()} == set(V2_FILES)
|
||||
# …and the KB is STILL empty (the upload never scans, phase 90 A1
|
||||
# — the Sync button is what will index v2's files).
|
||||
assert _docs(page, app_url) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -625,7 +599,7 @@ def test_bad_file_inline_error(
|
||||
422 detail naming the accepted formats, the button restores, the
|
||||
file selection is KEPT (the fix is one re-pick), the list is
|
||||
unchanged — and a subsequent good upload still works (the form is
|
||||
not wedged)."""
|
||||
not wedged), indexing nothing (phase 90 A1)."""
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
||||
@@ -645,7 +619,7 @@ def test_bad_file_inline_error(
|
||||
|
||||
# Never stale + the selection kept + no result line + list unchanged.
|
||||
expect(btn).to_be_enabled()
|
||||
expect(btn).to_have_text("Upload & scan")
|
||||
expect(btn).to_have_text("Upload")
|
||||
# The selection is kept (the fix is one re-pick) — Chromium reports
|
||||
# a fake path (``…/notes.txt``), so assert on the basename.
|
||||
bad_value = page.locator("#archive-upload-file").input_value()
|
||||
@@ -653,11 +627,14 @@ def test_bad_file_inline_error(
|
||||
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
||||
|
||||
# The form is not wedged: a good upload right after still works.
|
||||
assert _upload_via_page(page, tarball_v1) == "2 added"
|
||||
# The form is not wedged: a good upload right after still works —
|
||||
# and indexes nothing (phase 90 A1).
|
||||
assert _upload_via_page(page, tarball_v1) == (
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it."
|
||||
)
|
||||
expect(error).to_be_hidden()
|
||||
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
||||
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "beta.md")]
|
||||
assert _docs(page, app_url) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user