refactor(agents): migrate .agent/ planning tree to .agents/

Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
2026-09-05 10:57:07 -04:00
parent 766702c750
commit dbf2af26c6
1118 changed files with 664 additions and 664 deletions
@@ -0,0 +1,49 @@
# Phase 64 — Real-Time Progress for Sync + Upload, Background Upload
**Source:** `TODO.md` L3 — "Need better indication of sync and upload progress. Both should show current file being processed in real time, not just 'syncing' or 'uploading'." (+ the navigate-away / toast half of the same item)
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-01)
**Context:** `app/api/sync.py` (phase 32 — background sync + the 2 s `GET /api/sync/status` polling pattern, the template for everything here), `app/api/git_sources.py` (phase 49 — the upload route whose scan runs **synchronously in the request**: the browser `fetch` blocks until unpack + import finish, so navigating away mid-upload aborts it — the exact defect this phase removes), `app/rag/importer.py` (`import_sources` loops file-by-file with **no** progress hook — the single place both flows can be instrumented), `frontend/assets/sources.js` (the sync button's §7.4 never-stale lifecycle + load-time re-attach), `frontend/assets/git-sources.js` (the upload form's "Uploading…" lifecycle), `frontend/assets/styles.css` (`.toast` — the phase-55 share-success toast, reused verbatim for the new "successfully uploaded" toast). E2E conventions: `tests/e2e/test_archive_upload_sources.py` (archive fixture builder, mock LLM, `BOR_UPLOAD_DIR` scratch) and `tests/e2e/test_sync_button.py`.
## Objective
Both long-running KB jobs report **which file is being processed right now** (not just "Syncing…"/"Uploading…"), and the archive upload is fully **backgrounded**: `POST /api/git-sources/upload` answers 202 the moment the archive is on disk (a "successfully uploaded" toast fires — the user may navigate away), the unpack/scan continues server-side with a status endpoint, and the sync button on the RAG page (`/sources.html`) animates with the upload's current file while that scan runs.
## Dependencies
- `63_unambiguous_document_listing` (complete) — the immediately preceding phase (the todo queue was empty at authoring; this phase builds on no unfinished work).
## Tasks
1. `01_importer_progress_hook.md` — optional per-file progress callback on `import_sources` (source, rel path, done/total).
2. `02_sync_status_current_file.md` — `SyncStatus` + `GET /api/sync/status` carry `current_file` (+ counts), wired through the hook.
3. `03_upload_background_202.md` — `POST /upload` → 202 + background task + `GET /api/git-sources/upload/status` (phase-32 pattern, incl. `current_file`).
4. `04_sync_button_live_file.md` — RAG-page sync button: live file label for sync runs AND for in-flight upload scans, catalog refresh + settle on upload completion, load-time re-attach.
5. `05_upload_toast_progress.md` — Sources-page upload UI: "successfully uploaded" toast on 202, live "Processing… <file>" label via status polling, 409/load-time re-attach, failure banner.
6. `06_e2e_sync_upload_progress.md` — the story Playwright suite + `test_archive_upload_sources.py` adaptation + regressions + commit.
## Testing & Quality
- Unit: the progress-hook contract in `tests/unit/test_importer.py`; status shapes in `tests/unit/test_sync_button.py`; the 202/background/upload-status contract in `tests/unit/test_archive_upload.py` (adapted from the synchronous expectations).
- Frontend source pins (house pattern): a new `tests/unit/test_frontend_sync_upload.py` — label builders, the polling decision trees, toast-on-202, re-attach paths.
- Coverage: **>90%** on `app/` (`validate.sh` gate).
- E2E (mandatory, A16): `tests/e2e/test_sync_upload_progress.py`, run in isolation; `test_archive_upload_sources.py` updated to the 202 + toast + polling flow; regressions `test_sync_button.py`, `test_git_sources_admin.py`, `test_sync_model_down.py` in isolation.
## Completion Criteria
- [ ] `GET /api/sync/status` and `GET /api/git-sources/upload/status` both report `current_file` / `files_done` / `files_total` while their job runs (null/0 idle).
- [ ] `POST /api/git-sources/upload` returns **202** with the safe source name once the archive is fully received; unpack + scan continue in a background task; one upload at a time (409 in flight).
- [ ] The "Successfully uploaded — <file>" toast appears on the Sources page at 202 (before the scan finishes); navigating to `/sources.html` mid-scan shows the sync button animating with the upload's current file; on completion the button settles and the catalog shows the new documents.
- [ ] The sync button's label shows the current file during sync runs ("Syncing… <file>") and during upload scans ("Importing <file>"); the upload area shows "Processing… <file>" during the scan.
- [ ] `uv run pytest` green; `app/` coverage >90%.
- [ ] `uv run pytest tests/e2e/test_sync_upload_progress.py -v --no-cov` green in isolation (DB up); `test_archive_upload_sources.py`, `test_sync_button.py`, `test_git_sources_admin.py`, `test_sync_model_down.py` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
## Locked decisions
- **Owner-locked (2026-09-01, roadmap confirmation):**
- **A1 — the upload goes fully background.** 202 means "archive safely on disk"; unpack → swap → row upsert → model check → import → overview run in a background task; `GET /api/git-sources/upload/status` is the single source of truth (phase-32 `SyncStatus` pattern). One upload at a time stays (409). This is the only way "navigate away before it finishes" works.
- **A2 — toast timing/scope.** The "successfully uploaded" toast fires on the Sources page **at 202** (file received), auto-dismisses (~5 s, phase-55 `.toast`); failures are announced by the existing `#archive-upload-error` banner, never a toast.
- **A3 — RAG-page sync button during an upload scan.** It enters its running animation showing the upload's current file ("Importing <file>"); on upload completion it settles to "Sync sources" and the catalog refreshes (`loadDocs()`). The upload's count line itself stays on the Sources (git-sources) page — the sync-result line never renders upload counts.
- **A4 — progress granularity.** Per importable file: labels show `source/relative/path` (truncated with ellipsis; the full path lives in the button `title` + the aria-live result line). The unpack phase (before any file is indexed) shows no file yet — just "Processing…"/"Syncing…".
- **A5 — post-202 failures.** Unpack / zero-entry / swap / model / import failures land in the upload status as `failed` with a sanitized error (the `_sanitize_error` credential mask). Pre-swap failures leave the KB, folders, and rows untouched; post-swap failures keep the folder + row so a re-upload or the next sync retries idempotently (the existing phase-49 step-7 semantics, now in the status state instead of an HTTP error).
- **A9/A10/A16/A17 honoured** — the admin-only API surface stays stateless apart from the in-memory run state (phase 32 precedent); one story E2E suite; one atomic commit.
## Commit
```bash
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(sources): real-time file progress for sync and upload — background upload with success toast"
```