Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status

Browse Source
This commit is contained in:
ducoterra
2026-08-25 21:39:38 -04:00
parent 0654b304e1
commit 52136fe307
13 changed files with 1621 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agent/phases/complete/32_admin_sync_button/01_sync_api.md
+37
View File
@@ -0,0 +1,37 @@
# Task 01 — Sync API: in-process runner + status
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "triggers a doc import sync by cloning the relevant repos and then running import doc script"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The backend of the sync button: an admin-only `POST /api/sync` that starts the clone → import → overview pipeline as one in-process background task, and `GET /api/sync/status` for the UI's polling loop.
## Work
1. `app/api/sync.py` (new):
- `@dataclass SyncStatus` — `state: Literal["idle", "running", "success", "failed"] = "idle"`, `started_at: datetime | None`, `finished_at: datetime | None`, `detail: dict[str, Any] = field(default_factory=dict)`, `error: str | None`; module-level `_status` + `_task: asyncio.Task | None`.
- `GET /api/sync/status` (`Depends(require_admin)`) → JSON `{state, started_at, finished_at, detail, error}` (datetimes ISO-8601 or null).
- `POST /api/sync` (`Depends(require_admin)`) — if `_task` is not done → `409 {"detail": "a sync is already running"}`; else `_task = asyncio.create_task(_run_sync())` → `202 {"detail": "sync started"}`.
- `async def _run_sync()`:
1. `_status.state = "running"`, `started_at = now(UTC)`.
2. Resolve repos from `settings.git_source_list` — empty → fail with `"no git sources configured (BOR_GIT_SOURCES)"`.
3. For each URL: `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))` (imported from `scripts.git_sync` / `scripts.import_docs` — no git re-implementation; `GitSyncError` carries git's stderr).
4. `summary = await import_sources(sources, LLMClient(), prune=True)` (prune per phase locked decision).
5. If `summary.added + summary.updated > 0`: `await regenerate_overview(llm)`.
6. `_status.state = "success"`, `finished_at`, `detail = {files, added, updated, unchanged, pruned, errors, chunks, summaries, summary_errors, overview: bool}`; log `sync: done detail=…`.
7. Any `GitSyncError | EmbeddingError | Exception` → `_status.state = "failed"`, `finished_at`, `error = str(e)` (sanitized: no secrets; git's stderr is fine), `logger.exception("sync: failed")`.
2. `app/main.py` — `from app.api.sync import router as sync_router` + `app.include_router(sync_router, prefix="/api")` (next to the other routers).
3. `tests/integration/test_sync_api.py` (new) — sign in via the existing auth test helper (`tests/integration/test_auth_api.py` pattern):
- anonymous: `GET /api/sync/status` → 403; `POST /api/sync` → 403.
- admin: idle state initially; `BOR_GIT_SOURCES` set to one `file://` URL with `clone_or_pull`, `import_sources`, `regenerate_overview` **monkeypatched** in `app.api.sync` (the mock import returns a canned `ImportSummary`; the mock overview returns True) → `POST` → 202; poll status → `success` with the canned detail (all ImportSummary fields + `overview: true`).
- 409: mock runner sleeps briefly (asyncio.sleep) → second `POST` while running → 409.
- failure: mock `clone_or_pull` raises `GitSyncError("git clone failed …")` → status `failed`, `error` names the failure; import is **not** called.
- empty `BOR_GIT_SOURCES` → `POST` 202 → status `failed` with the "no git sources configured" message.
- prune: assert the monkeypatched `import_sources` received `prune=True`.
## Testing & Quality
- Integration: Work step 3 (real Postgres not required for the runner logic beyond none — keep DB-free; if the session needs Postgres for nothing, use the app fixture without DB).
- Coverage: **>90%** on `app/api/sync.py` (all states/branches hit).
## Completion Criteria
- [ ] All integration tests green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] SSE/API routes untouched — `test_chat_api.py` green (no middleware or router precedence change).
.agent/phases/complete/32_admin_sync_button/02_ui_button.md
+40
View File
@@ -0,0 +1,40 @@
# Task 02 — The admin-only Sync button on Sources (§7.4 feedback)
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "a button that only the admin can see that triggers a doc import sync"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The UI: a **"Sync sources"** button in the Sources page header — hidden by default, revealed only for the signed-in admin (the existing `header.js` whoami gate) — with the full "never stale" feedback lifecycle: idle → "Syncing…" (disabled, spinner, 2 s status polling) → last-result label or error banner.
## Work
1. `frontend/sources.html` — in the header actions area (next to `.new-chat-btn`, inside the same `.header-inner` container the phase-19 shared header uses on this page):
- `<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">` — a small refresh-cycle `<svg aria-hidden="true">` icon (spin it via CSS in the running state) + `<span class="sync-label" id="sync-label">Sync sources</span>`.
- `<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>` right after the button (announces last-result / counts to screen readers).
- Update the `.page-sub` copy: it still says "Re-run the import to refresh" — extend it to mention the button (e.g. "…or hit **Sync sources** in the header to clone the repos and re-import.").
2. `frontend/assets/header.js` — add `#sync-btn` to the **admin reveal** that already handles `#nav-sources` / `#nav-tuning` after `fetchIsAdmin()` (one fetch, no extra whoami call); anonymous users never see it (stays `hidden`).
3. `frontend/assets/sources.js` — the sync state machine (new, isolated section):
- On load (admin only — `header.js` exposes the whoami result or a shared `isAdmin` flag; reuse whatever mechanism it already provides for the nav reveals): `GET /api/sync/status` →
- `running` → enter the running state and start polling (the user may have reloaded mid-sync).
- `success` / `failed` → render the last result (below) but keep the button ready for a fresh sync.
- Click → `POST /api/sync` → `202` → running state: button `disabled` + `aria-busy="true"`, icon spinning, label **"Syncing…"**, start polling `GET /api/sync/status` every **2000 ms**.
- Terminal state (stop polling):
- `success` → enabled, icon reset, label **"Synced HH:MM"** (local time of `finished_at`), `#sync-result` = `"{added} added · {updated} updated · {pruned} pruned"` (omit zero terms) — announced via `aria-live`.
- `failed` → enabled, label **"Sync sources"** (retry-ready), and show the page error banner (the existing `role="alert"` pattern used elsewhere in `sources.js`, or the chat error-banner markup style) with the `error` text; `#sync-result` cleared.
- `409` on POST (a run started elsewhere) → just enter running state + polling (adopt the in-flight run); `403` → treat as not-admin (hide the button — defense in depth).
- **No client-side hard timeout** (phase locked decision — the poll is the feedback loop; the server state is authoritative).
4. `frontend/assets/styles.css` — `.sync-btn` styled like `.new-chat-btn`/`.auth-link` (dark tech theme tokens; text contrast ≥4.5:1 — use the dark-ink-on-brand pairing per PLAN §7.2 if the button is filled, else soft-ink on surface), `.sync-btn[disabled]` state, `.sync-btn .sync-icon.is-spinning { animation: spin 1s linear infinite }` with the existing `prefers-reduced-motion` opt-out, `:focus-visible` 3px outline, ≥44 px touch target on mobile.
5. `tests/unit/test_sync_button.py` (new, frontend-assertion style of `test_shared_header.py` / `test_frontend_feedback.py`):
- `sources.html` contains `#sync-btn` with `hidden`, `aria-label="Sync sources"`, and `#sync-result` with `role="status"` + `aria-live="polite"`.
- `header.js` reveals `#sync-btn` in the admin branch (assert the element id appears in the reveal logic, same as `#nav-sources`).
- `sources.js` references `/api/sync` (POST + status GET), the 2000 ms poll, `409` adoption, `403` hide, and the terminal labels (`Syncing…` / `Synced` / failure banner).
- no-CDN integration test stays green (no new external references).
## Testing & Quality
- Unit: Work step 5 (frontend-assertion tests); no-CDN integration test green.
- Coverage: `app/` TOTAL unchanged (frontend-only task); the UI is gated end-to-end by task 03's E2E.
## Completion Criteria
- [ ] Anonymous: `#sync-btn` never leaves `hidden` in the DOM; admin: it is revealed without a page reload round-trip beyond the existing whoami fetch.
- [ ] Full lifecycle works against the dev server (manual check): click → "Syncing…" (disabled) → "Synced HH:MM" + counts, or error banner with retry; reload mid-sync re-enters the running state.
- [ ] UI Structure Check (AGENTS.md rule 5): labeled + focus-visible + contrast ≥4.5:1 + `aria-live` result; reduced-motion respected; no CDN.
- [ ] `uv run pytest` green (including the new frontend-assertion tests); `uv run ruff check . && uv run pyright` clean.