chore(agent): phase roadmap from TODO.md, 1 phase (64_sync_upload_progress)
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# Phase 64 — Real-Time Progress for Sync + Upload, Background Upload
|
||||
|
||||
**Source:** `TODO.md` L3 — "Need better indication of sync and upload progress. Both should show current file being processed in real time, not just 'syncing' or 'uploading'." (+ the navigate-away / toast half of the same item)
|
||||
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-01)
|
||||
**Context:** `app/api/sync.py` (phase 32 — background sync + the 2 s `GET /api/sync/status` polling pattern, the template for everything here), `app/api/git_sources.py` (phase 49 — the upload route whose scan runs **synchronously in the request**: the browser `fetch` blocks until unpack + import finish, so navigating away mid-upload aborts it — the exact defect this phase removes), `app/rag/importer.py` (`import_sources` loops file-by-file with **no** progress hook — the single place both flows can be instrumented), `frontend/assets/sources.js` (the sync button's §7.4 never-stale lifecycle + load-time re-attach), `frontend/assets/git-sources.js` (the upload form's "Uploading…" lifecycle), `frontend/assets/styles.css` (`.toast` — the phase-55 share-success toast, reused verbatim for the new "successfully uploaded" toast). E2E conventions: `tests/e2e/test_archive_upload_sources.py` (archive fixture builder, mock LLM, `BOR_UPLOAD_DIR` scratch) and `tests/e2e/test_sync_button.py`.
|
||||
|
||||
## Objective
|
||||
Both long-running KB jobs report **which file is being processed right now** (not just "Syncing…"/"Uploading…"), and the archive upload is fully **backgrounded**: `POST /api/git-sources/upload` answers 202 the moment the archive is on disk (a "successfully uploaded" toast fires — the user may navigate away), the unpack/scan continues server-side with a status endpoint, and the sync button on the RAG page (`/sources.html`) animates with the upload's current file while that scan runs.
|
||||
|
||||
## Dependencies
|
||||
- `63_unambiguous_document_listing` (complete) — the immediately preceding phase (the todo queue was empty at authoring; this phase builds on no unfinished work).
|
||||
|
||||
## Tasks
|
||||
1. `01_importer_progress_hook.md` — optional per-file progress callback on `import_sources` (source, rel path, done/total).
|
||||
2. `02_sync_status_current_file.md` — `SyncStatus` + `GET /api/sync/status` carry `current_file` (+ counts), wired through the hook.
|
||||
3. `03_upload_background_202.md` — `POST /upload` → 202 + background task + `GET /api/git-sources/upload/status` (phase-32 pattern, incl. `current_file`).
|
||||
4. `04_sync_button_live_file.md` — RAG-page sync button: live file label for sync runs AND for in-flight upload scans, catalog refresh + settle on upload completion, load-time re-attach.
|
||||
5. `05_upload_toast_progress.md` — Sources-page upload UI: "successfully uploaded" toast on 202, live "Processing… <file>" label via status polling, 409/load-time re-attach, failure banner.
|
||||
6. `06_e2e_sync_upload_progress.md` — the story Playwright suite + `test_archive_upload_sources.py` adaptation + regressions + commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the progress-hook contract in `tests/unit/test_importer.py`; status shapes in `tests/unit/test_sync_button.py`; the 202/background/upload-status contract in `tests/unit/test_archive_upload.py` (adapted from the synchronous expectations).
|
||||
- Frontend source pins (house pattern): a new `tests/unit/test_frontend_sync_upload.py` — label builders, the polling decision trees, toast-on-202, re-attach paths.
|
||||
- Coverage: **>90%** on `app/` (`validate.sh` gate).
|
||||
- E2E (mandatory, A16): `tests/e2e/test_sync_upload_progress.py`, run in isolation; `test_archive_upload_sources.py` updated to the 202 + toast + polling flow; regressions `test_sync_button.py`, `test_git_sources_admin.py`, `test_sync_model_down.py` in isolation.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/sync/status` and `GET /api/git-sources/upload/status` both report `current_file` / `files_done` / `files_total` while their job runs (null/0 idle).
|
||||
- [ ] `POST /api/git-sources/upload` returns **202** with the safe source name once the archive is fully received; unpack + scan continue in a background task; one upload at a time (409 in flight).
|
||||
- [ ] The "Successfully uploaded — <file>" toast appears on the Sources page at 202 (before the scan finishes); navigating to `/sources.html` mid-scan shows the sync button animating with the upload's current file; on completion the button settles and the catalog shows the new documents.
|
||||
- [ ] The sync button's label shows the current file during sync runs ("Syncing… <file>") and during upload scans ("Importing <file>"); the upload area shows "Processing… <file>" during the scan.
|
||||
- [ ] `uv run pytest` green; `app/` coverage >90%.
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_upload_progress.py -v --no-cov` green in isolation (DB up); `test_archive_upload_sources.py`, `test_sync_button.py`, `test_git_sources_admin.py`, `test_sync_model_down.py` green in isolation.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation):**
|
||||
- **A1 — the upload goes fully background.** 202 means "archive safely on disk"; unpack → swap → row upsert → model check → import → overview run in a background task; `GET /api/git-sources/upload/status` is the single source of truth (phase-32 `SyncStatus` pattern). One upload at a time stays (409). This is the only way "navigate away before it finishes" works.
|
||||
- **A2 — toast timing/scope.** The "successfully uploaded" toast fires on the Sources page **at 202** (file received), auto-dismisses (~5 s, phase-55 `.toast`); failures are announced by the existing `#archive-upload-error` banner, never a toast.
|
||||
- **A3 — RAG-page sync button during an upload scan.** It enters its running animation showing the upload's current file ("Importing <file>"); on upload completion it settles to "Sync sources" and the catalog refreshes (`loadDocs()`). The upload's count line itself stays on the Sources (git-sources) page — the sync-result line never renders upload counts.
|
||||
- **A4 — progress granularity.** Per importable file: labels show `source/relative/path` (truncated with ellipsis; the full path lives in the button `title` + the aria-live result line). The unpack phase (before any file is indexed) shows no file yet — just "Processing…"/"Syncing…".
|
||||
- **A5 — post-202 failures.** Unpack / zero-entry / swap / model / import failures land in the upload status as `failed` with a sanitized error (the `_sanitize_error` credential mask). Pre-swap failures leave the KB, folders, and rows untouched; post-swap failures keep the folder + row so a re-upload or the next sync retries idempotently (the existing phase-49 step-7 semantics, now in the status state instead of an HTTP error).
|
||||
- **A9/A10/A16/A17 honoured** — the admin-only API surface stays stateless apart from the in-memory run state (phase 32 precedent); one story E2E suite; one atomic commit.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(sources): real-time file progress for sync and upload — background upload with success toast"
|
||||
```
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 01 — The importer progress hook
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time, not just 'syncing' or 'uploading'."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`import_sources` gains an optional per-file progress callback so both long-running jobs (admin sync, upload scan — tasks 02/03) can report the file being processed right now. Existing callers pass nothing and see zero behavior or performance change.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py`:
|
||||
- Add an optional keyword argument to `import_sources`: `progress: Callable[[str, str, int, int], None] | None = None` — signature `(source, rel_posix_path, done, total)`.
|
||||
- **Semantics:** called once per importable file, immediately before `await _index_file(...)`, with `done` = the 1-based index of the current file **across all sources** and `total` = the total number of importable files across all `sources` roots. `rel` is the same POSIX path the doc rows use (`path.relative_to(root).as_posix()`).
|
||||
- `total` is computed **only when `progress` is provided**: pre-walk every root with the existing `iter_importable_files` (same extension/exclusion rules — directory stats only, no file reads). When `progress is None`, no pre-walk happens: existing callers (`scripts/import_docs.py`, `app/api/sync.py` until task 02, `app/api/git_sources.py` until task 03) are byte-identical in behavior and cost.
|
||||
- The `limit` debug path is unchanged: the callback still fires per processed file; `done` never exceeds the limit, `total` stays the full pre-walk count (an incomplete walk must not misreport the denominator).
|
||||
- No special handling for a raising callback — the hooks in this repo (tasks 02/03) only assign dataclass fields. Keep the loop clean; no try/except around the call.
|
||||
- Module docstring: one line noting the optional progress hook (phase 64).
|
||||
2. `tests/unit/test_importer.py` — unit tests:
|
||||
- Multi-root, multi-file: the callback receives the exact `(source, rel, done, total)` sequence (both roots interleaved in `sources` order, `total` = combined count).
|
||||
- `progress=None`: no callback, and a pre-walk sentinel (e.g., monkeypatch `iter_importable_files` with a call counter) proves the walk happens exactly as many times as before this change (no extra pass).
|
||||
- Skipped/unchanged/error files still count in the sequence (the callback fires before `_index_file`, so an `EmbeddingError` file was already reported as current).
|
||||
- `limit=`: callback fires only for processed files; `total` is still the full count.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (the hook is small; the `None` path and the pre-walk path both get dedicated tests).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `import_sources(..., progress=fn)` reports every importable file in order with correct `done`/`total`; `progress=None` callers are unchanged (no extra walk).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (sync/upload still pass no hook yet).
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 02 — Sync status reports the current file
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time, not just 'syncing' or 'uploading'."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`GET /api/sync/status` carries the file the admin sync is processing right now (`current_file` + `files_done`/`files_total`), so the RAG-page button (task 04) can render a live label instead of a bare "Syncing…".
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py`:
|
||||
- `SyncStatus` dataclass: add `current_file: str | None = None`, `files_done: int = 0`, `files_total: int = 0` (the four-state machine is untouched).
|
||||
- `GET /api/sync/status` response: add the three keys — `"current_file": str | null`, `"files_done": int`, `"files_total": int` (idle: `null`/`0`/`0`). The existing keys (`state`, `started_at`, `finished_at`, `detail`, `error`) are unchanged, so the current UI and every existing consumer keep working.
|
||||
- `_run_sync`: on start, explicitly reset the three fields (alongside the existing resets). Pass the task-01 hook to `import_sources(sources, llm, prune=True, progress=_hook)` where `_hook(source, rel, done, total)` assigns `_status.current_file = f"{source}/{rel}"`, `_status.files_done = done`, `_status.files_total = total`. The module already has its single `_status` instance — the closure captures it exactly like the existing state assignments.
|
||||
- Terminal states: on `success` and on `failed` set `_status.current_file = None` (keep the final `files_done`/`files_total` — the run's last position is useful context and costs nothing). The clone/pull phase before the import reports no file yet (`current_file` stays `None`) — per ASSUMPTION A4 the label then shows just "Syncing…".
|
||||
- Module docstring: the status paragraph gains one line on the progress fields (phase 64).
|
||||
2. `tests/unit/test_sync_button.py` — unit tests (extend the existing sync-API unit coverage, house fixture for the in-memory run):
|
||||
- Idle status shape: the three new keys present with `null`/`0`/`0`.
|
||||
- Mid-run: drive `_run_sync` with the existing mocked-import seam (the suite already stubs the pipeline) + a `progress`-shaped call injected through the real hook closure — `GET /api/sync/status` reports the assigned `current_file`/`files_done`/`files_total` while `state == "running"`.
|
||||
- Terminal: after success and after failure, `current_file` is `null` (counts retain the final values).
|
||||
- Backward shape: every pre-existing key in the response is unchanged (pin the full response dict on the idle state).
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A4 — per-file granularity; `source/relative/path` form; no file shown during the clone/pull phase (bare "Syncing…").
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (the `_run_sync` terminal branches already have coverage — extend, don't duplicate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/sync/status` returns `current_file`/`files_done`/`files_total` (null/0 idle), updated per file while a sync runs, `current_file` null in terminal states.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (the UI still shows "Syncing…" until task 04).
|
||||
@@ -0,0 +1,48 @@
|
||||
# Task 03 — The upload becomes background (202 + status endpoint)
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Also the user can navigate away from the upload way before 'uploading' finishes." (+ "once the zipfile/tarball is on the server there should be a little 'successfully uploaded' notification toast")
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`POST /api/git-sources/upload` returns **202** the moment the archive is fully received (safe on disk), and unpack → swap → row upsert → model check → import → overview run in a background task behind a new `GET /api/git-sources/upload/status` (the phase-32 `SyncStatus` pattern, including task-02-style `current_file`/counts). Navigating away mid-scan no longer aborts anything.
|
||||
|
||||
## Work
|
||||
1. `app/schemas.py` — add `UploadAccepted(BaseModel)`: `detail: str = "upload received"`, `name: str` (the safe source name). `UploadOut` is **kept** — it becomes the shape of the status `detail` on success.
|
||||
2. `app/api/git_sources.py`:
|
||||
- **`UploadStatus` dataclass** (module level, mirrors `app.api.sync.SyncStatus`): `state: Literal["idle", "running", "success", "failed"]`, `started_at`/`finished_at`, `current_file: str | None`, `files_done: int`, `files_total: int`, `detail: dict`, `error: str | None`; one instance `_upload_status`. Keep the module-level `_upload_in_progress` **bool flag** (not a task-done check) with its existing rationale — the flag is checked and set with **no await in between, BEFORE streaming**, because the handler now awaits (the 1 MiB-chunk stream) before the background task exists: a task-done check alone would let a concurrent POST slip through during the receive and start a second run. The flag is cleared in `_run_upload`'s `finally` (at the end of the background run) and on the exception path where the task was never created (wrap stream + `create_task` in try/except, clear + re-raise).
|
||||
- **`POST /upload` → `status_code=202`, `response_model=UploadAccepted`**; drop the `db: Session = Depends(get_db)` dependency from the signature (the handler no longer touches the DB — the upsert moves to the background). Inline (request) work, unchanged semantics: (1) name/format gate (422s as today, including `archive_source_name`'s messages), (2) 409 `an upload is already in progress` while the flag is held, (3) stream the upload in `_STREAM_CHUNK` chunks into the dotfile temp with the `upload_max_mb` cap (413 naming the cap, temp deleted). Then `asyncio.create_task(_run_upload(...))` and return `UploadAccepted(name=<safe name>)` — **the file is on disk; 202 is the "successfully uploaded" moment the UI toasts on (ASSUMPTION A2).**
|
||||
- **`_run_upload(name, filename, total_bytes, upload_root, temp_upload, temp_unpack)`** (module function, the phase-32 `_run_sync` shape):
|
||||
1. `_upload_status` → `running` (started_at, finished_at `None`, `current_file` `None`, counts 0, detail `{}`, error `None`).
|
||||
2. `unpack_archive(temp_upload, temp_unpack, max_bytes)` — `ArchiveUploadError` → `failed` (sanitized via the imported `_sanitize_error`), temps deleted; the compressed temp is unlinked after unpack (phase-49 locked decision: only unpacked content is kept).
|
||||
3. Zero entries → `failed` "the archive contains no files".
|
||||
4. `swap_in(temp_unpack, final_dir)` — `ArchiveUploadError` → `failed` sanitized (a failure here leaves the previous folder/row/KB untouched).
|
||||
5. Row upsert **in a short-lived `SessionLocal()`** (open/close around it, the `effective_sources`/`bump_sources_version` pattern from `app/api/sync.py` — never the request session, whose lock discipline is what the old inline `db.close()` comment guarded): by `path` (expanded `final_dir`), `kind="local"`, `url` = same path; an existing row is left as-is (`added_at` preserved); a concurrent-insert `IntegrityError` → `failed` "a local source with this path already exists: <path>" (the folder stays — the row exists, the next sync sees it).
|
||||
6. `check_models(LLMClient())` — `ModelUnavailableError` → `failed` with the sanitized message (the phase-49 503 becomes a status state; the folder/row are committed, so the next sync/re-upload retries idempotently — ASSUMPTION A5).
|
||||
7. `import_sources([final_dir], llm, prune=True, progress=_hook)` — `_hook(source, rel, done, total)` assigns `_upload_status.current_file = f"{source}/{rel}"` + the counts (task-01 hook).
|
||||
8. Change-gated `regenerate_overview(llm)` (added + updated > 0 — unchanged).
|
||||
9. The per-upload INFO log line (PLAN §9 / AGENTS.md rule 10) **moves here**, same fields as today (`upload: name=… file=… bytes_in=… files=… added=… updated=… unchanged=… pruned=… errors=… overview=… total_ms=…` — `total_ms` now the background run's duration).
|
||||
10. `success`: finished_at, `detail` = the `UploadOut` fields as a dict (`source=name`, `files`, `added`, `updated`, `unchanged`, `pruned`, `errors`, `chunks`, `overview`).
|
||||
- `CancelledError` is deliberately **not** caught (app shutdown cancels the task — the `_run_sync` rule). The `finally` cleans both temps (defensive, as today) and clears `_upload_in_progress`.
|
||||
- **`GET /upload/status`** (router dependency already admin-only): response `{"state", "started_at", "finished_at", "current_file", "files_done", "files_total", "detail", "error"}` — identical key set to `GET /api/sync/status` (ISO-8601 or null, same as there).
|
||||
- Docstrings: module docstring's upload paragraph → phase-64 contract (202 + background + status endpoint; the inline gate list stays accurate — steps 1–3 are inline, 4–10 are background); route docstring rewritten to match.
|
||||
3. `tests/unit/test_archive_upload.py` — adapt the suite to the 202 contract (keep every scenario, change the observation point from the HTTP response to the status endpoint, polling until terminal):
|
||||
- 202 + `UploadAccepted` body (`detail` + safe `name`); the temp upload file exists on disk at that point.
|
||||
- Inline gates unchanged: non-archive extension 422 (accepted set named), unsafe name 422, over-cap 413 (cap named, temp deleted), 409 while a run is in flight (flag still held).
|
||||
- Success: status `success`, `detail` carries the `UploadOut` fields with correct counts, the `kind="local"` row exists (and a re-upload under the same name preserves it — the existing in-place-replace scenario, now observed via the second run's status), the folder is in place, no dotfile temps left in `upload_root`.
|
||||
- Corrupt/traversal archive → status `failed`, sanitized error, KB + rows untouched, temps deleted.
|
||||
- Zero-entry archive → status `failed` "the archive contains no files".
|
||||
- Model down (stub `check_models` raising `ModelUnavailableError`) → status `failed` sanitized; folder + row exist (idempotent-retry precondition, ASSUMPTION A5).
|
||||
- Mid-run: `current_file`/`files_done`/`files_total` reported while `running` (same hook-injection seam as task 02), `current_file` null in terminal states.
|
||||
- Re-upload while a run is in flight → 409 (the flag, not task-done, is the gate — pin a second POST during the receive window in the existing flag test, or extend it).
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A1 — 202 = "archive safely on disk"; the scan runs server-side afterwards; one upload at a time. A5 — post-202 failures are status states (`failed` + sanitized error), never HTTP errors; pre-swap failures leave KB/folders/rows untouched, post-swap failures keep folder + row for an idempotent retry.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: as above (the adapted suite is this task's gate); full suite green.
|
||||
- Coverage: **>90%** on `app/` — every background branch (unpack fail, zero-entry, swap fail, IntegrityError, model down, success, cancel-cleanup) gets a dedicated test; the `_run_upload` failure branches are the new code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `POST /api/git-sources/upload` answers 202 after the receive; `GET /api/git-sources/upload/status` mirrors the sync status shape (incl. `current_file`) and is admin-only (403 anonymous).
|
||||
- [ ] A scan keeps running (and completes) after the client disconnects — unit-level: the background task is created before the response and its outcome lands in `_upload_status`.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] The `test_archive_upload_sources.py` E2E suite will be adapted in task 06 (it is expected to fail until then — noted, not fixed here).
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 04 — RAG-page sync button: live file, upload-scan awareness
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time" + "Need to trigger the syncing button animation if the user clicks on sources after clicking upload"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The sync button on `/sources.html` shows the current file while a **sync** runs ("Syncing… <file>") and, while an **upload scan** is in flight, enters the same running animation showing the upload's current file ("Importing <file>"); when the upload scan finishes the button settles and the catalog refreshes.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/sources.js`:
|
||||
- **Label builder** `fmtSyncLabel(kind, currentFile, done, total)` — `kind` is `"sync" | "upload"`: `"sync"` → `Syncing…` + (file ? ` ${file}` : "") ; `"upload"` → `Importing` + (file ? ` ${file}` : "") ; both append ` (done/total)` only when `total > 0` (A4 — no file yet during clone/pull or unpack → bare prefix). The full untruncated path is what the status endpoints report; the label shows it, truncated by CSS.
|
||||
- **`enterSyncRunningState(kind, currentFile, done, total)`** (the existing no-arg version gains parameters): same mechanics as today (disabled, `aria-busy`, icon `is-spinning`, no `is-error`) plus: `syncBtn.title = currentFile` (removed when null — full path on hover) and `syncResult.textContent = fmtSyncLabel(...)` **without** the truncation, so the existing `role="status"` `#sync-result` announcer reads the full live path to screen readers.
|
||||
- **Unified polling** — `startSyncPolling`'s tick now fetches **both** `GET /api/sync/status` and `GET /api/git-sources/upload/status` (both admin-only; the 403 branch on the sync fetch already hides the button — a 403 on the upload fetch is treated as "no upload", a network blip retries next tick). Decision tree, in order, per tick:
|
||||
1. sync `running` → `enterSyncRunningState("sync", sync.current_file, sync.files_done, sync.files_total)`; reschedule.
|
||||
2. else upload `running` → `enterSyncRunningState("upload", upload.current_file, upload.files_done, upload.files_total)`; reschedule. ← *the "clicking on sources after clicking upload" contract (A3).*
|
||||
3. else sync `success` → `applySyncSuccess` (unchanged); stop.
|
||||
4. else sync `failed` → `applySyncFailure` (unchanged); stop.
|
||||
5. else upload `success` → `settleSyncButton("Sync sources")`; `syncResult.textContent = ""` (A3 — the sync-result line never renders upload counts; they live on the Sources page); `hideSyncError()`; `emitSyncStatus({ state: "idle" })`; `loadDocs()` (the KB changed — the new documents must appear); stop.
|
||||
6. else upload `failed` → `settleSyncButton("Sync sources")`; `syncResult.textContent = ""`; `hideSyncError()`; `emitSyncStatus({ state: "idle" })`; stop (the failure UI is the Sources page's error banner — A3).
|
||||
7. else both idle → `applySyncIdle`; stop.
|
||||
- **`startSync` click handler**: the 202/409 branch calls `enterSyncRunningState("sync", null, 0, 0)` (the existing `emitSyncStatus({ state: "running" })` dedup via `lastSyncState` stays).
|
||||
- **`initSyncButton` load-time re-attach**: unchanged for sync states; when the sync state is `idle`, additionally fetch the upload status — if it is `running`, `enterSyncRunningState("upload", …)` + `emitSyncStatus({ state: "running" })` + `startSyncPolling()` (a terminal upload is a no-op: the boot-time `loadDocs()` already shows the current catalog).
|
||||
- Header comment: the sync-button block documents the phase-64 contract (live file label, the two-job decision tree, the A3 settle behavior).
|
||||
2. `frontend/assets/styles.css` — `.sync-label`: truncate long paths — `display: inline-block; max-width: min(16rem, 40vw); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; vertical-align: bottom;` (the full path stays in the button `title` + `#sync-result`).
|
||||
3. `frontend/sources.html` — no structural change; the sync-button comment block updated to the phase-64 contract.
|
||||
4. Frontend source pins — new `tests/unit/test_frontend_sync_upload.py` (the existing `tests/unit/test_frontend_feedback.py` pin style), covering: the `fmtSyncLabel` contract (both kinds, file present/absent, counts only when `total > 0`); `enterSyncRunningState` writing the full path into `title` + `#sync-result`; the tick decision tree (running-upload → "Importing" label; upload success → settle + `loadDocs` + no sync-result line; upload failed → settle, no error banner; both idle → `applySyncIdle`); the re-attach branch (sync idle + upload running → running state + polling starts).
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A3 — during an upload scan the sync button animates with "Importing <file>"; on upload completion it settles to "Sync sources", clears the result line, and refreshes the catalog; the sync-result line never renders upload counts, and an upload failure is not surfaced on the RAG page. A4 — per-file granularity, `source/relative/path`, ellipsis truncation, bare prefix before the first file.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: source pins as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only task).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] During a sync run the button label shows `Syncing… <file> (n/m)`; during an in-flight upload scan it shows `Importing <file> (n/m)` with the spinning icon.
|
||||
- [ ] On upload completion the button settles, the catalog refreshes, and no upload counts appear in `#sync-result`; on page load with a running upload the button re-attaches to the running state.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (plain sync run renders as before plus the file in the label).
|
||||
@@ -0,0 +1,39 @@
|
||||
# Task 05 — Upload UI: success toast on receive, live "Processing…" label
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "once the zipfile/tarball is on the server there should be a little 'successfully uploaded' notification toast so the user knows they can navigate away"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
On `/git-sources.html` the upload flow follows the new 202 contract: the moment the archive is received the page shows a **"Successfully uploaded — <file>" toast** (safe to navigate away), the button switches to a live **"Processing… <file> (n/m)"** label driven by `GET /api/git-sources/upload/status` polling, and a page reload mid-scan re-attaches instead of dead-ending.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/git-sources.js`:
|
||||
- **`showUploadToast(message)`** — page-local, the phase-55 share-toast pattern from `frontend/assets/app.js` (lazy-created single node, class `toast`, `.is-visible` toggles the entry transition, `role="status" aria-live="polite"`, ~5 s auto-dismiss, a new toast replaces a pending one — clear the prior timer, never stack). Uses the existing `.toast` CSS as-is.
|
||||
- **Submit handler** (the existing one, reworked):
|
||||
- The no-file guard, the error/result clearing, and the "Uploading…" transfer label stay (the transfer is now short — 202 arrives when the receive finishes).
|
||||
- **202** (parse the `UploadAccepted` body for `name`; body parse failure degrades to the picked file's name): `showUploadToast(\`Successfully uploaded — ${file.name}\`)`; `uploadFileInput.value = ""`; enter the **processing state** (`uploadBtn.disabled = true`, `uploadBtn.textContent = "Processing…"`, `uploadBtn.title = ""`) and `startUploadPolling()`.
|
||||
- **409** (`an upload is already in progress`): **no error banner** — enter the processing state + `startUploadPolling()` (re-attach; never stale). The old "server detail inline" branch does NOT apply to 409 anymore.
|
||||
- **other non-ok** (422 name/format, 413 cap, 5xx): the existing `apiDetail` error banner, file selection KEPT (the existing re-pick convention), button restored in `finally`.
|
||||
- **network failure** (`catch`): the existing "Could not reach the server" banner, button restored.
|
||||
- `finally` restores the button **only when no polling is active** (while `startUploadPolling` owns the button, it stays disabled/Processing — the §7.4 never-stale rule).
|
||||
- **`startUploadPolling()`** — 2 s cadence (the `SYNC_POLL_MS` house value, local `const UPLOAD_POLL_MS = 2000`), single timer, one at a time (guard against double-start): each tick fetches `GET /api/git-sources/upload/status`:
|
||||
- `running` → `uploadBtn.textContent = "Processing…" + (current_file ? \` ${current_file}\` : "") + (files_total > 0 ? \` (${files_done}/${files_total})\` : "")`; `uploadBtn.title = current_file || ""` (full path on hover); reschedule. (A4 — unpack phase shows bare "Processing…".)
|
||||
- `success` → stop; `uploadResult` line = `fmtUploadResult(detail)` (the existing helper reads exactly these keys); `announce(\`Archive uploaded: ${detail.source}.\`)`; `uploadFileInput.value = ""`; restore the button (enabled, "Upload & scan", title removed); `loadSources()` (the row lands / refreshes). **No toast here** — it already fired at 202.
|
||||
- `failed` → stop; `uploadError` banner = `status.error` (sanitized server-side); restore the button; `loadSources()` (post-swap failures keep the row — the list state may have changed; the file selection is kept for a one-click re-upload).
|
||||
- `idle` → stop; restore the button (defensive — a started run never returns to idle).
|
||||
- **Boot re-attach** — in the admin branch where `loadSources()` runs at boot, fetch the upload status once: `running` → processing state + `startUploadPolling()`; `success` → render the last result line only (no announce, no toast); `failed` → the error banner; `idle` → nothing.
|
||||
- Header comment: the upload block rewritten to the phase-64 contract (202 + toast + polling + re-attach; the phase-49 synchronous paragraph marked superseded).
|
||||
2. `frontend/git-sources.html` — no structural change (the toast node is JS-created, phase-55 pattern); the phase-49 form comment updated to the phase-64 contract.
|
||||
3. Frontend source pins — extend `tests/unit/test_frontend_sync_upload.py`: the toast contract (fires on 202 with `Successfully uploaded — <name>`, `role="status"`, auto-dismiss timer, single-node reuse); the processing-label builder (file present/absent, counts only when `total > 0`, full path in `title`); 409 → processing state + polling (NOT the error banner); the polling decision tree (success → result line + `loadSources`, no toast; failed → error banner + `loadSources`; idle → restore); the `finally` never-restoring-while-polling guard; the boot re-attach branches.
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A2 — the toast fires at 202 on this page, auto-dismisses, and is success-only (failures use the existing `#archive-upload-error` banner). A4 — bare "Processing…" during the unpack phase; `source/relative/path` granularity afterwards.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: source pins as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only task).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Uploading an archive shows the "Successfully uploaded — <file>" toast as soon as the 202 arrives — before the scan finishes — and the button then tracks the scan with a live file label.
|
||||
- [ ] Reloading `/git-sources.html` mid-scan resumes the Processing state (no error, no second upload); a finished/failed run re-renders its result/banner.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (the 422/413 error paths, the re-upload flow, and the source list are untouched apart from the new async flow).
|
||||
@@ -0,0 +1,40 @@
|
||||
# Task 06 — E2E suite, regressions, commit
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — the whole item (the executable proof of every locked decision).
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The story's Playwright suite proves the full contract end-to-end (toast on receive → navigate away → sync button animating with the upload's current file → catalog refreshed; live file labels on both jobs; reload re-attach), `test_archive_upload_sources.py` is brought onto the 202 contract, the regression suites stay green in isolation, and the phase is committed.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_sync_upload_progress.py` — new suite (house conventions: `tests/e2e/conftest.py` + `mock_llm.py`, the admin sign-in and fixture style of `tests/e2e/test_git_sources_admin.py`, the archive builder + `BOR_UPLOAD_DIR` scratch of `tests/e2e/test_archive_upload_sources.py`, the local-source sync fixture style of `tests/e2e/test_sync_button.py`):
|
||||
- **Fixture note (timing):** the mock LLM indexes fast — the in-progress state is real but brief. Build the upload archive from **20+ small `.md` files** so the scan outlasts the 2 s poll, and assert the live-file label at TWO layers: the deterministic one is the status endpoint (`page.request.get("/api/git-sources/upload/status")` / `"/api/sync/status"` — `state == "running"` with non-null `current_file` observed at some tick); the UI one polls the button label for the `Importing ` / `Syncing…` / `Processing…` prefix plus a file path (generous timeout).
|
||||
- `test_upload_toast_then_navigate_away` — on `/git-sources.html`: pick the multi-file archive, submit → the `Successfully uploaded — <archive>` toast (`.toast.is-visible`, `role="status"`) appears **while the scan is still running**; immediately navigate to `/sources.html` → the sync button is present, animating (icon `is-spinning`, `aria-busy`) with the `Importing` label; wait for the settle → button idle ("Sync sources"), no error UI, and the catalog table shows the uploaded documents (the phase-63 listing is untouched).
|
||||
- `test_upload_progress_shows_current_file` — during the scan: the status endpoint reports a non-null `current_file` (`source/relative/path` shape) at some running tick, and the upload button label shows `Processing…` with a file path (UI layer) before the result line (`fmtUploadResult` counts) lands; the toast fired earlier in the run (not after the result).
|
||||
- `test_sync_live_file_label` — a multi-file local source configured (the `test_sync_button.py` fixture style): on `/sources.html` click **Sync sources** → the label shows `Syncing…` with a file path while running (endpoint layer: `/api/sync/status` `current_file` non-null; UI layer: label poll), then the success settle with the counts result line (the pre-phase-64 sync UX is preserved, plus the file).
|
||||
- `test_upload_reattach_after_reload` — start the upload, and **during the scan** reload `/git-sources.html` → the button is in the Processing state (disabled) with **no** error banner and no second upload (the status endpoint's single run is still the one from before the reload); it then settles with the result line and the list shows exactly one row for the archive (in-place identity preserved).
|
||||
2. `tests/e2e/test_archive_upload_sources.py` — adapt to the 202 contract (keep every scenario, move observation points):
|
||||
- upload → the toast appears at 202, the button goes `Uploading…` → `Processing…` → restore; the result line + list now come from the status `success` (counts unchanged in shape).
|
||||
- the inline 422/413 gates are unchanged (asserted exactly as today — name, format, cap, file kept).
|
||||
- a bad-archive scenario (if present) now lands as the **error banner** via status `failed` instead of a 422 response; re-upload after it still works.
|
||||
- the re-upload-in-place-replace scenario: the second run's status `detail` shows the prune/refresh counts; the list still has exactly ONE row for the archive name.
|
||||
- anonymous: the gate/form stay hidden, `POST /api/git-sources/upload` 403, and `GET /api/git-sources/upload/status` 403 (new endpoint, same wall).
|
||||
3. Run in isolation (DB up, the AGENTS.md rule-9 command): `test_sync_upload_progress.py` (new), `test_archive_upload_sources.py` (updated), `test_sync_button.py`, `test_git_sources_admin.py`, `test_sync_model_down.py`.
|
||||
4. Full gate: `uv run pytest --cov=app --cov-report=term-missing` (suite green, `app/` >90%), `uv run ruff check . && uv run pyright`.
|
||||
5. Commit + hand-off:
|
||||
```bash
|
||||
git add -A .agent/ app/ frontend/ tests/
|
||||
git commit --no-gpg-sign -m "feat(sources): real-time file progress for sync and upload — background upload with success toast"
|
||||
```
|
||||
then move `.agent/phases/todo/64_sync_upload_progress/` → `.agent/phases/complete/64_sync_upload_progress/` (the pipeline's `validate.sh` gate is the move's precondition).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new suite + the five isolated runs above are this phase's A16 gate.
|
||||
- Coverage: **>90%** on `app/` (no new app code in this task — the gate guards against drift from tasks 01–03).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_upload_progress.py -v --no-cov` green in isolation (DB up).
|
||||
- [ ] `test_archive_upload_sources.py`, `test_sync_button.py`, `test_git_sources_admin.py`, `test_sync_model_down.py` green in isolation.
|
||||
- [ ] `uv run pytest --cov=app` green with `app/` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` Conventional Commits commit; phase dir in `.agent/phases/complete/`.
|
||||
- [ ] `TODO.md` contains only `# TODO` (cleared by the phase pipeline's conversion step — items now live here).
|
||||
Reference in New Issue
Block a user