Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): real-time file progress for sync and upload — background upload with success toast

Browse Source
This commit is contained in:
ducoterra
2026-09-01 23:51:43 -04:00
parent cddc84c7db
commit 4677d86f49
103 changed files with 5914 additions and 456 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/test_archive_upload_sources.py
+169 -39
View File
@@ -11,10 +11,26 @@ form is gone, replaced by this form): an uploaded ``.tar``/``.tar.gz``/
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
(name = filename minus the archive suffix), the ``git_sources`` row is
upserted (``kind='local'``, no duplicates), and the source is **scanned
synchronously in the request** (single-source ``import_sources`` with
``prune=True`` + the change-gated overview refresh) — the real pipeline,
against the deterministic mock LLM (no real models, no network beyond
the app itself).
in a background task** (phase 64, task 03 — owner-locked A1: the POST
answers **202 the moment the archive is safely on disk** — the
"Successfully uploaded — <source>" toast fires then and the user may
navigate away — while unpack → swap → row upsert → model check →
single-source ``import_sources`` with ``prune=True`` + the change-gated
overview refresh run server-side behind
``GET /api/git-sources/upload/status``, the phase-32 ``SyncStatus``
pattern with the phase-64 ``current_file``/counts). The result line and
the row land from the status ``success`` (same ``UploadOut`` counts,
uncompressed in shape) — the real pipeline, against the deterministic
mock LLM (no real models, no network beyond the app itself).
**Timing fixture (phase 64):** the mock LLM answers instantly, so this
module's app boots behind ``tests/e2e/slow_llm.py`` — a delay-injecting
reverse proxy in front of it (``SLOW_DELAY_S`` per request). A 2-file
scan is 5 LLM requests ≈ 5 × 0.6 s ≈ 3 s: long enough to outlive the
UI's 2 s status poll, so the button's literal
"Uploading… → Processing… → restored" lifecycle is observable
(the "Processing…" tick even carries the live file, A4) instead of
racing the mock.
The archives are **built in-test** with Python's ``tarfile`` over
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
@@ -38,24 +54,33 @@ Contract under test:
the upload form is in its place with the labeled file input (accept
= the four archive extensions), the "Upload & scan" button, and the
hint explains unpack/scan + in-place replace;
* **upload → scan → list** (§7.4 never-stale): the button shows
"Uploading…" while the POST is in flight (the request is held in the
browser via ``page.route`` so the in-flight state is deterministic),
then restores; the result line shows the added count; the list gains
exactly one row for ``e2e-upload`` with the **Local** badge;
``GET /api/docs`` lists both sentinel files under source
``e2e-upload``; the RAG catalog (``/sources.html``) shows them;
* **upload → 202 + toast → background scan → list** (§7.4 never-stale,
phase-64 A1/A2): the button shows "Uploading…" while the POST is in
flight (the request is held in the browser via ``page.route`` so the
in-flight state is deterministic); at the 202 the "Successfully
uploaded — <source>" toast fires (``.toast.is-visible``,
``role="status"``) WHILE the scan is still running, and the button
hands over to the scan — "Processing…" (the live-file tick carries
the current file, A4) — then restores when the status ``success``
lands: the result line shows the added count; the list gains exactly
one row for ``e2e-upload`` with the **Local** badge; ``GET
/api/docs`` lists both sentinel files under source ``e2e-upload``;
the RAG catalog (``/sources.html``) shows them;
* **re-upload, same filename** → in-place replace: the result line
shows the prune, the list still has exactly ONE ``e2e-upload`` row
(no duplicate), the KB shows the changed ``alpha`` + the new
shows the prune, the SECOND RUN'S STATUS ``detail`` carries the
prune/refresh counts, the list still has exactly ONE ``e2e-upload``
row (no duplicate), the KB shows the changed ``alpha`` + the new
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
only the new archive's files;
* **bad file** → inline 422 (role=alert) naming the accepted formats,
button restored, the file selection kept, the list unchanged, and a
subsequent good upload still works (the form is not wedged);
* **bad file** → inline 422 (role=alert) naming the accepted formats
(UNCHANGED — the name/format/cap gates are inline, pre-202, exactly
as before), button restored, the file selection kept, the list
unchanged, and a subsequent good upload still works (the form is not
wedged);
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
manager (and thus the upload form) stays hidden, and
``POST /api/git-sources/upload`` is 403.
``POST /api/git-sources/upload`` is 403 — as is the phase-64
``GET /api/git-sources/upload/status`` (same wall).
Test → story mapping (Playwright Mapping Rule):
1. ``test_form_swapped``
@@ -86,6 +111,7 @@ from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
@@ -94,6 +120,17 @@ from e2e.conftest import (
REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The slow-LLM proxy's port (the conftest's mock LLM stays on MOCK_PORT).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT", "8902"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay on the proxy — a 2-file scan is 5 LLM requests
#: (the check_models embed + chat probe, one embed per file, the
#: change-gated overview chat) ≈ 5 × 0.6 s ≈ 3 s: the scan outlives the
#: UI's 2 s status poll, so the button's "Uploading… → Processing… →
#: restored" lifecycle (with the live-file tick, A4) is observable.
SLOW_DELAY_S = "0.6"
GIT_SOURCES_URL = "/git-sources.html"
SOURCES_URL = "/sources.html"
@@ -184,17 +221,47 @@ def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(tests/e2e/slow_llm.py) — this suite's timing fixture: the phase-64
button lifecycle ("Uploading… → Processing… → restored") needs the
2-file scan to outlive the UI's 2 s status poll (see
``SLOW_DELAY_S``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
slow_llm: int,
upload_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The real app under test — per-module env: uploads unpack into a
scratch dir and the env git list is forced empty (the dev ``.env``'s
``BOR_GIT_SOURCES`` must not render as env rows on the initially
empty table). No sync is triggered here — the upload's own scan is
the pipeline under test."""
"""The real app under test — per-module env: the LLM base URL is the
SLOW PROXY in front of the mock (the timing fixture), uploads unpack
into a scratch dir and the env git list is forced empty (the dev
``.env``'s ``BOR_GIT_SOURCES`` must not render as env rows on the
initially empty table). No sync is triggered here — the upload's own
scan is the pipeline under test."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
@@ -202,7 +269,7 @@ def app_server(
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
else f"{SLOW_URL}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env shape.
@@ -286,9 +353,10 @@ def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
def _upload_via_page(page: Page, archive: Path) -> str:
"""Pick the archive, submit the form, and wait for the result line
(the 200 path) — returns its text. The failing path is asserted
explicitly by the bad-file test, so any non-result outcome here is
a test error."""
(the phase-64 202 path: toast at the 202, then the button's status
polling renders the line from the run's ``success``) — returns its
text. The failing path is asserted explicitly by the bad-file test,
so any non-result outcome here is a test error."""
page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result")
@@ -298,6 +366,24 @@ def _upload_via_page(page: Page, archive: Path) -> str:
return text
def _wait_upload_running(page: Page, app_url: str, timeout_s: float = 15.0) -> dict[str, Any]:
"""Poll (cookie-authenticated) the upload status endpoint until the
run is ``running`` — the phase-64 single source of truth for the
background scan (A1)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
assert r.status == 200, r.text
body = r.json()
if body["state"] == "running":
return body
if body["state"] in ("success", "failed"):
raise AssertionError(f"the scan settled too fast to observe: {body}")
time.sleep(0.1)
raise AssertionError(f"the scan never entered running: {body}")
def _hold_upload_request(page: Page, hold_s: float) -> None:
"""Intercept the upload POST and hold the REQUEST in the browser for
``hold_s`` seconds before letting it reach the server. While it is
@@ -365,11 +451,15 @@ def test_upload_scans_and_lists(
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
) -> None:
"""One real upload through the page: while the POST is in flight the
button is disabled and reads "Uploading…"; on the 200 it restores,
the result line shows the added count (2), the file input clears,
the list gains exactly ONE row for ``e2e-upload`` with the Local
badge, ``/api/docs`` lists both sentinel files under the source, and
the RAG catalog shows them where the admin expects them."""
button is disabled and reads "Uploading…"; at the 202 the
"Successfully uploaded — <source>" toast fires (A2) WHILE the
background scan is still running and the button hands over to it —
"Processing…" (the 2 s poll tick carries the live file, A4); when
the status ``success`` lands the button restores, the result line
shows the added count (2), the file input clears, the list gains
exactly ONE row for ``e2e-upload`` with the Local badge,
``/api/docs`` lists both sentinel files under the source, and the
RAG catalog shows them where the admin expects them."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
@@ -378,7 +468,7 @@ def test_upload_scans_and_lists(
result = page.locator("#archive-upload-result")
# Hold the upload request in the browser: the in-flight state below
# cannot race the (fast) mock-LLM scan while it is held.
# cannot race the receive while it is held.
_hold_upload_request(page, hold_s=0.8)
page.set_input_files("#archive-upload-file", str(tarball_v1))
btn.click()
@@ -388,12 +478,35 @@ def test_upload_scans_and_lists(
expect(btn).to_have_text("Uploading…")
expect(result).to_be_hidden()
# The request goes out, the server unpacks + scans (mock LLM) and
# answers 200 → the result line shows the added count.
# The request goes out; the server stores the archive and answers
# 202 the moment it is safely on disk (A1) → the toast fires NOW
# (A2) — while the scan is still running — and the button hands
# over to the scan (bare "Processing…" — A4: no file yet during the
# unpack phase).
toast = page.locator(".toast")
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
assert toast.get_attribute("role") == "status"
expect(toast).to_have_text(f"Successfully uploaded — {SOURCE_NAME}")
expect(result).to_be_hidden()
expect(btn).to_be_disabled()
expect(btn).to_have_text("Processing…", timeout=5_000)
# The scan is running server-side (the status endpoint is the
# single source of truth, A1) — the run the UI's poll tracks.
_wait_upload_running(page, app_url)
# …and the button's 2 s poll tick renders the live file label
# ("Processing… <file> (n/m)", A4).
expect(btn).to_have_text(
re.compile(rf"Processing… {re.escape(SOURCE_NAME)}/.+\.md"),
timeout=UPLOAD_TIMEOUT_MS,
)
# The status success lands → the result line (the same UploadOut
# counts) + the never-stale restore (input cleared).
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text("2 added")
# Never stale: the button restored on success and the input cleared.
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
expect(page.locator("#archive-upload-file")).to_have_value("")
@@ -434,7 +547,9 @@ def test_reupload_replaces_in_place(
upload_dir: Path,
) -> None:
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
result line shows the prune, the list still has exactly ONE
result line shows the prune, the SECOND RUN'S STATUS ``detail``
carries the prune/refresh counts (phase 64 — the line is rendered
from the status success), the list still has exactly ONE
``e2e-upload`` row (the row count for that source is invariant — no
duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
and NOT the dropped ``beta``, and the on-disk folder holds only the
@@ -442,7 +557,7 @@ def test_reupload_replaces_in_place(
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Baseline: v1 through the page (200 → "2 added", one row).
# Baseline: v1 through the page (202 → "2 added", one row).
assert _upload_via_page(page, tarball_v1) == "2 added"
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
@@ -452,6 +567,19 @@ def test_reupload_replaces_in_place(
result = page.locator("#archive-upload-result")
expect(result).to_have_text(re.compile(r"\d+ pruned"))
# The SECOND RUN's status ``detail`` shows the prune/refresh counts
# (phase 64: the result line is rendered from this success).
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
assert r.status == 200, r.text
status = r.json()
assert status["state"] == "success", status
detail = status["detail"]
assert detail["source"] == SOURCE_NAME
assert detail["files"] == 2
assert detail["added"] == 1 # gamma — new in v2
assert detail["updated"] == 1 # alpha — changed in v2
assert detail["pruned"] == 1 # beta — dropped in v2
# No duplicate: exactly ONE row for that source (and one row total).
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
@@ -552,6 +680,8 @@ def test_anonymous_gate(page: Page, app_url: str, db_ready: None) -> None:
# The upload route 403s anonymous callers (require_admin runs before
# the multipart body is parsed — the body is a stand-in, the
# test_local_directory_sources.py pattern for this route).
# test_local_directory_sources.py pattern for this route)…
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
assert r.status == 403
# …and so does the phase-64 upload STATUS endpoint (same wall).
assert page.request.get(f"{app_url}/api/git-sources/upload/status").status == 403