brain-of-reese/.agents/phases/complete/64_sync_upload_progress/00_phase.md

Phase 64 — Real-Time Progress for Sync + Upload, Background Upload

Source: TODO.md L3 — "Need better indication of sync and upload progress. Both should show current file being processed in real time, not just 'syncing' or 'uploading'." (+ the navigate-away / toast half of the same item) Story: n/a (TODO-derived — owner roadmap confirmation 2026-09-01) Context: app/api/sync.py (phase 32 — background sync + the 2 s GET /api/sync/status polling pattern, the template for everything here), app/api/git_sources.py (phase 49 — the upload route whose scan runs synchronously in the request: the browser fetch blocks until unpack + import finish, so navigating away mid-upload aborts it — the exact defect this phase removes), app/rag/importer.py (import_sources loops file-by-file with no progress hook — the single place both flows can be instrumented), frontend/assets/sources.js (the sync button's §7.4 never-stale lifecycle + load-time re-attach), frontend/assets/git-sources.js (the upload form's "Uploading…" lifecycle), frontend/assets/styles.css (.toast — the phase-55 share-success toast, reused verbatim for the new "successfully uploaded" toast). E2E conventions: tests/e2e/test_archive_upload_sources.py (archive fixture builder, mock LLM, BOR_UPLOAD_DIR scratch) and tests/e2e/test_sync_button.py.

Objective

Both long-running KB jobs report which file is being processed right now (not just "Syncing…"/"Uploading…"), and the archive upload is fully backgrounded: POST /api/git-sources/upload answers 202 the moment the archive is on disk (a "successfully uploaded" toast fires — the user may navigate away), the unpack/scan continues server-side with a status endpoint, and the sync button on the RAG page (/sources.html) animates with the upload's current file while that scan runs.

Dependencies

• 63_unambiguous_document_listing (complete) — the immediately preceding phase (the todo queue was empty at authoring; this phase builds on no unfinished work).

Tasks

1. 01_importer_progress_hook.md — optional per-file progress callback on import_sources (source, rel path, done/total).
2. 02_sync_status_current_file.md — SyncStatus + GET /api/sync/status carry current_file (+ counts), wired through the hook.
3. 03_upload_background_202.md — POST /upload → 202 + background task + GET /api/git-sources/upload/status (phase-32 pattern, incl. current_file).
4. 04_sync_button_live_file.md — RAG-page sync button: live file label for sync runs AND for in-flight upload scans, catalog refresh + settle on upload completion, load-time re-attach.
5. 05_upload_toast_progress.md — Sources-page upload UI: "successfully uploaded" toast on 202, live "Processing… " label via status polling, 409/load-time re-attach, failure banner.
6. 06_e2e_sync_upload_progress.md — the story Playwright suite + test_archive_upload_sources.py adaptation + regressions + commit.

Testing & Quality

• Unit: the progress-hook contract in tests/unit/test_importer.py; status shapes in tests/unit/test_sync_button.py; the 202/background/upload-status contract in tests/unit/test_archive_upload.py (adapted from the synchronous expectations).
• Frontend source pins (house pattern): a new tests/unit/test_frontend_sync_upload.py — label builders, the polling decision trees, toast-on-202, re-attach paths.
• Coverage: >90% on app/ (validate.sh gate).
• E2E (mandatory, A16): tests/e2e/test_sync_upload_progress.py, run in isolation; test_archive_upload_sources.py updated to the 202 + toast + polling flow; regressions test_sync_button.py, test_git_sources_admin.py, test_sync_model_down.py in isolation.

Completion Criteria

• GET /api/sync/status and GET /api/git-sources/upload/status both report current_file / files_done / files_total while their job runs (null/0 idle).
• POST /api/git-sources/upload returns 202 with the safe source name once the archive is fully received; unpack + scan continue in a background task; one upload at a time (409 in flight).
• The "Successfully uploaded — " toast appears on the Sources page at 202 (before the scan finishes); navigating to /sources.html mid-scan shows the sync button animating with the upload's current file; on completion the button settles and the catalog shows the new documents.
• The sync button's label shows the current file during sync runs ("Syncing… ") and during upload scans ("Importing "); the upload area shows "Processing… " during the scan.
• uv run pytest green; app/ coverage >90%.
• uv run pytest tests/e2e/test_sync_upload_progress.py -v --no-cov green in isolation (DB up); test_archive_upload_sources.py, test_sync_button.py, test_git_sources_admin.py, test_sync_model_down.py green in isolation.
• uv run ruff check . && uv run pyright clean.
• One --no-gpg-sign commit; phase dir moved to .agents/phases/complete/.

Locked decisions

• Owner-locked (2026-09-01, roadmap confirmation):
  • A1 — the upload goes fully background. 202 means "archive safely on disk"; unpack → swap → row upsert → model check → import → overview run in a background task; GET /api/git-sources/upload/status is the single source of truth (phase-32 SyncStatus pattern). One upload at a time stays (409). This is the only way "navigate away before it finishes" works.
  • A2 — toast timing/scope. The "successfully uploaded" toast fires on the Sources page at 202 (file received), auto-dismisses (~5 s, phase-55 .toast); failures are announced by the existing #archive-upload-error banner, never a toast.
  • A3 — RAG-page sync button during an upload scan. It enters its running animation showing the upload's current file ("Importing "); on upload completion it settles to "Sync sources" and the catalog refreshes (loadDocs()). The upload's count line itself stays on the Sources (git-sources) page — the sync-result line never renders upload counts.
  • A4 — progress granularity. Per importable file: labels show source/relative/path (truncated with ellipsis; the full path lives in the button title + the aria-live result line). The unpack phase (before any file is indexed) shows no file yet — just "Processing…"/"Syncing…".
  • A5 — post-202 failures. Unpack / zero-entry / swap / model / import failures land in the upload status as failed with a sanitized error (the _sanitize_error credential mask). Pre-swap failures leave the KB, folders, and rows untouched; post-swap failures keep the folder + row so a re-upload or the next sync retries idempotently (the existing phase-49 step-7 semantics, now in the status state instead of an HTTP error).
• A9/A10/A16/A17 honoured — the admin-only API surface stays stateless apart from the in-memory run state (phase 32 precedent); one story E2E suite; one atomic commit.

Commit

git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(sources): real-time file progress for sync and upload — background upload with success toast"