chore(agent): phase roadmap from TODO.md — 4 phases (lite document summaries, KB overview in prompt, admin sync button, cache busting)

.agent/phases/todo/32_admin_sync_button/00_phase.md

@@ -0,0 +1,45 @@
+# Phase 32 — Admin Sync Button (one-click doc import sync)
+
+**Source:** `TODO.md L5 — "Need a button that only the admin can see that triggers a doc import sync by cloning the relevant repos and then running import doc script"`
+**Story:** `.agent/user_stories/admin-sync-button.md`
+**Context:** Phase 28 (`scripts/git_sync.py::clone_or_pull` — shallow clone / `--ff-only` pull; `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; `repo_name` in `scripts/import_docs.py`), phase 31 (`regenerate_overview` — the sync refreshes the KB outline), phase 16 (`require_admin` dependency + the `header.js` `fetchIsAdmin()` reveal gate for admin-only UI like `#nav-sources` / `#nav-tuning`), PLAN §7.4 "never stale" feedback contract (the UI can never sit on a stale button state).
+
+## Objective
+Give the admin a **"Sync sources"** button (Sources page, visible to the admin only) that triggers the full document sync in-process — clone/pull every `BOR_GIT_SOURCES` repo, re-import (with prune) so the KB mirrors the repos, and refresh the KB overview — with live, non-stale UI feedback driven by a polled sync-status endpoint.
+
+## Dependencies
+- `31_kb_overview_prompt` (todo) — `regenerate_overview(llm)` is the sync's final step; `LLMClient.chat` for it.
+- `28_git_based_sources` (complete) — `clone_or_pull` / `GitSyncError` / `BOR_GIT_SOURCES` / `repo_name` (the sync reuses them, does not re-implement git).
+- `16_admin_auth` (complete) — `require_admin` for the new endpoints; the `header.js` whoami gate for the button.
+- `19_shared_header` / `29_tuning_nav_link` (complete) — the Sources page header actions area where the button lives.
+
+## Tasks
+1. `01_sync_api.md` — in-process sync runner: `POST /api/sync` (admin, 409 when running) + `GET /api/sync/status` (admin).
+2. `02_ui_button.md` — the admin-only button on Sources with §7.4 feedback states (polling, last-result, error banner) + frontend unit assertions.
+3. `03_e2e_and_docs.md` — `tests/e2e/test_sync_button.py` (real `file://` git fixture), README, story file, commit.
+
+## Testing & Quality
+- Integration: sync API — anonymous 403s, admin idle/running/success/failed transitions, 409 double-trigger, GitSyncError → `failed` with the repo named (git + import + overview mocked, as `test_import_docs_git.py` does).
+- Unit (frontend-assertion style, cf. `tests/unit/test_shared_header.py`): button markup hidden-by-default + labeled; `header.js` reveal; `sources.js` polling/terminal-state logic.
+- Coverage: **>90%** on `app/` (`app/api/sync.py` fully covered); `app/` TOTAL ≥ pre-change.
+- E2E (mandatory, A16): `tests/e2e/test_sync_button.py` — one story, run **in isolation**; uses a **real** local `file://` git repo fixture (deterministic, no network) with the mock LLM for embeddings.
+- UI Structure Check (AGENTS.md rule 5): labeled button, focus-visible, contrast ≥4.5:1, `aria-live` result region, no CDN.
+
+## Completion Criteria
+- [ ] Anonymous: the button is not revealed (stays `hidden`) and both endpoints return 403.
+- [ ] Admin: clicking "Sync sources" starts the sync (202), the button goes disabled with "Syncing…" while polling `GET /api/sync/status` every 2 s, and on completion shows the last result (`Synced HH:MM` + `N added · M updated`); a failed sync re-enables the button with an error banner (`role="alert"`) naming the failure.
+- [ ] A double trigger while running returns 409 and the UI never starts a second poll loop.
+- [ ] After a successful sync against the `file://` fixture repo, the newly committed fixture doc appears in the Sources table and the `kb_overview` row is fresh (phase-31 trigger).
+- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
+- [ ] `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green in isolation; `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py` stay green.
+- [ ] `uv run ruff check . && uv run pyright` clean.
+- [ ] `.agent/user_stories/admin-sync-button.md` exists; README documents the button (behavior, states, prerequisites).
+- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status`); `.agent/phases/todo/32_admin_sync_button/` moved to `.agent/phases/complete/`.
+
+## Locked decisions
+- **A10 extended (recorded, not a revision)** — two new **admin-only** endpoints (`POST /api/sync`, `GET /api/sync/status`) behind the existing `require_admin`; the public API surface stays stateless, the signed cookie remains the only session state (same pattern as `/api/steering`).
+- **A12 untouched** — the sync runs **in-process** (one `asyncio` background task + a module-level status object in `app/api/sync.py`). The app is a single instance on the homelab; no Valkey/queue. Status is in memory — a restart mid-sync loses the running state (accepted: the next click re-syncs idempotently).
+- **Sync semantics** — the button targets `BOR_GIT_SOURCES` only (manual `--source` dirs have no repo to clone; an unset/empty `BOR_GIT_SOURCES` → the sync fails loudly with "no git sources configured"); the import runs with **`prune=True`** so files deleted upstream leave the index (the button is the canonical "mirror the repos" action — the CLI default of no-prune is unchanged); phase-31's `regenerate_overview` runs after the import when docs changed.
+- **Concurrency** — one sync at a time: `POST /api/sync` while running → `409 {"detail": "a sync is already running"}`; the UI reflects the in-flight run (re-attaches on page load while a sync is running).
+- **§7.4 adaptation (recorded)** — the 120 s client guard applies to LLM turns; a sync can legitimately run for minutes (clone + embed), so the button has **no client-side hard timeout** — the 2 s status poll is the feedback loop and the server state is authoritative. The button is disabled until the run reaches a terminal state, so it can never be stale *or* stuck: a failed run re-enables it, a running run always shows "Syncing…".
+- **A16 / A17 honoured** — one dedicated story E2E suite (real `file://` git fixture — git is a documented environment prerequisite, as in phase 28); one atomic `--no-gpg-sign` commit.

.agent/phases/todo/32_admin_sync_button/01_sync_api.md

@@ -0,0 +1,37 @@
+# Task 01 — Sync API: in-process runner + status
+
+**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "triggers a doc import sync by cloning the relevant repos and then running import doc script"`
+**Story:** `.agent/user_stories/admin-sync-button.md`
+
+## Objective
+The backend of the sync button: an admin-only `POST /api/sync` that starts the clone → import → overview pipeline as one in-process background task, and `GET /api/sync/status` for the UI's polling loop.
+
+## Work
+1. `app/api/sync.py` (new):
+   - `@dataclass SyncStatus` — `state: Literal["idle", "running", "success", "failed"] = "idle"`, `started_at: datetime | None`, `finished_at: datetime | None`, `detail: dict[str, Any] = field(default_factory=dict)`, `error: str | None`; module-level `_status` + `_task: asyncio.Task | None`.
+   - `GET /api/sync/status` (`Depends(require_admin)`) → JSON `{state, started_at, finished_at, detail, error}` (datetimes ISO-8601 or null).
+   - `POST /api/sync` (`Depends(require_admin)`) — if `_task` is not done → `409 {"detail": "a sync is already running"}`; else `_task = asyncio.create_task(_run_sync())` → `202 {"detail": "sync started"}`.
+   - `async def _run_sync()`:
+     1. `_status.state = "running"`, `started_at = now(UTC)`.
+     2. Resolve repos from `settings.git_source_list` — empty → fail with `"no git sources configured (BOR_GIT_SOURCES)"`.
+     3. For each URL: `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))` (imported from `scripts.git_sync` / `scripts.import_docs` — no git re-implementation; `GitSyncError` carries git's stderr).
+     4. `summary = await import_sources(sources, LLMClient(), prune=True)` (prune per phase locked decision).
+     5. If `summary.added + summary.updated > 0`: `await regenerate_overview(llm)`.
+     6. `_status.state = "success"`, `finished_at`, `detail = {files, added, updated, unchanged, pruned, errors, chunks, summaries, summary_errors, overview: bool}`; log `sync: done detail=…`.
+     7. Any `GitSyncError | EmbeddingError | Exception` → `_status.state = "failed"`, `finished_at`, `error = str(e)` (sanitized: no secrets; git's stderr is fine), `logger.exception("sync: failed")`.
+2. `app/main.py` — `from app.api.sync import router as sync_router` + `app.include_router(sync_router, prefix="/api")` (next to the other routers).
+3. `tests/integration/test_sync_api.py` (new) — sign in via the existing auth test helper (`tests/integration/test_auth_api.py` pattern):
+   - anonymous: `GET /api/sync/status` → 403; `POST /api/sync` → 403.
+   - admin: idle state initially; `BOR_GIT_SOURCES` set to one `file://` URL with `clone_or_pull`, `import_sources`, `regenerate_overview` **monkeypatched** in `app.api.sync` (the mock import returns a canned `ImportSummary`; the mock overview returns True) → `POST` → 202; poll status → `success` with the canned detail (all ImportSummary fields + `overview: true`).
+   - 409: mock runner sleeps briefly (asyncio.sleep) → second `POST` while running → 409.
+   - failure: mock `clone_or_pull` raises `GitSyncError("git clone failed …")` → status `failed`, `error` names the failure; import is **not** called.
+   - empty `BOR_GIT_SOURCES` → `POST` 202 → status `failed` with the "no git sources configured" message.
+   - prune: assert the monkeypatched `import_sources` received `prune=True`.
+
+## Testing & Quality
+- Integration: Work step 3 (real Postgres not required for the runner logic beyond none — keep DB-free; if the session needs Postgres for nothing, use the app fixture without DB).
+- Coverage: **>90%** on `app/api/sync.py` (all states/branches hit).
+
+## Completion Criteria
+- [ ] All integration tests green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
+- [ ] SSE/API routes untouched — `test_chat_api.py` green (no middleware or router precedence change).

.agent/phases/todo/32_admin_sync_button/02_ui_button.md

@@ -0,0 +1,40 @@
+# Task 02 — The admin-only Sync button on Sources (§7.4 feedback)
+
+**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "a button that only the admin can see that triggers a doc import sync"`
+**Story:** `.agent/user_stories/admin-sync-button.md`
+
+## Objective
+The UI: a **"Sync sources"** button in the Sources page header — hidden by default, revealed only for the signed-in admin (the existing `header.js` whoami gate) — with the full "never stale" feedback lifecycle: idle → "Syncing…" (disabled, spinner, 2 s status polling) → last-result label or error banner.
+
+## Work
+1. `frontend/sources.html` — in the header actions area (next to `.new-chat-btn`, inside the same `.header-inner` container the phase-19 shared header uses on this page):
+   - `<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">` — a small refresh-cycle `<svg aria-hidden="true">` icon (spin it via CSS in the running state) + `<span class="sync-label" id="sync-label">Sync sources</span>`.
+   - `<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>` right after the button (announces last-result / counts to screen readers).
+   - Update the `.page-sub` copy: it still says "Re-run the import to refresh" — extend it to mention the button (e.g. "…or hit **Sync sources** in the header to clone the repos and re-import.").
+2. `frontend/assets/header.js` — add `#sync-btn` to the **admin reveal** that already handles `#nav-sources` / `#nav-tuning` after `fetchIsAdmin()` (one fetch, no extra whoami call); anonymous users never see it (stays `hidden`).
+3. `frontend/assets/sources.js` — the sync state machine (new, isolated section):
+   - On load (admin only — `header.js` exposes the whoami result or a shared `isAdmin` flag; reuse whatever mechanism it already provides for the nav reveals): `GET /api/sync/status` →
+     - `running` → enter the running state and start polling (the user may have reloaded mid-sync).
+     - `success` / `failed` → render the last result (below) but keep the button ready for a fresh sync.
+   - Click → `POST /api/sync` → `202` → running state: button `disabled` + `aria-busy="true"`, icon spinning, label **"Syncing…"**, start polling `GET /api/sync/status` every **2000 ms**.
+   - Terminal state (stop polling):
+     - `success` → enabled, icon reset, label **"Synced HH:MM"** (local time of `finished_at`), `#sync-result` = `"{added} added · {updated} updated · {pruned} pruned"` (omit zero terms) — announced via `aria-live`.
+     - `failed` → enabled, label **"Sync sources"** (retry-ready), and show the page error banner (the existing `role="alert"` pattern used elsewhere in `sources.js`, or the chat error-banner markup style) with the `error` text; `#sync-result` cleared.
+   - `409` on POST (a run started elsewhere) → just enter running state + polling (adopt the in-flight run); `403` → treat as not-admin (hide the button — defense in depth).
+   - **No client-side hard timeout** (phase locked decision — the poll is the feedback loop; the server state is authoritative).
+4. `frontend/assets/styles.css` — `.sync-btn` styled like `.new-chat-btn`/`.auth-link` (dark tech theme tokens; text contrast ≥4.5:1 — use the dark-ink-on-brand pairing per PLAN §7.2 if the button is filled, else soft-ink on surface), `.sync-btn[disabled]` state, `.sync-btn .sync-icon.is-spinning { animation: spin 1s linear infinite }` with the existing `prefers-reduced-motion` opt-out, `:focus-visible` 3px outline, ≥44 px touch target on mobile.
+5. `tests/unit/test_sync_button.py` (new, frontend-assertion style of `test_shared_header.py` / `test_frontend_feedback.py`):
+   - `sources.html` contains `#sync-btn` with `hidden`, `aria-label="Sync sources"`, and `#sync-result` with `role="status"` + `aria-live="polite"`.
+   - `header.js` reveals `#sync-btn` in the admin branch (assert the element id appears in the reveal logic, same as `#nav-sources`).
+   - `sources.js` references `/api/sync` (POST + status GET), the 2000 ms poll, `409` adoption, `403` hide, and the terminal labels (`Syncing…` / `Synced` / failure banner).
+   - no-CDN integration test stays green (no new external references).
+
+## Testing & Quality
+- Unit: Work step 5 (frontend-assertion tests); no-CDN integration test green.
+- Coverage: `app/` TOTAL unchanged (frontend-only task); the UI is gated end-to-end by task 03's E2E.
+
+## Completion Criteria
+- [ ] Anonymous: `#sync-btn` never leaves `hidden` in the DOM; admin: it is revealed without a page reload round-trip beyond the existing whoami fetch.
+- [ ] Full lifecycle works against the dev server (manual check): click → "Syncing…" (disabled) → "Synced HH:MM" + counts, or error banner with retry; reload mid-sync re-enters the running state.
+- [ ] UI Structure Check (AGENTS.md rule 5): labeled + focus-visible + contrast ≥4.5:1 + `aria-live` result; reduced-motion respected; no CDN.
+- [ ] `uv run pytest` green (including the new frontend-assertion tests); `uv run ruff check . && uv run pyright` clean.

.agent/phases/todo/32_admin_sync_button/03_e2e_and_docs.md

@@ -0,0 +1,34 @@
+# Task 03 — Story E2E (real file:// git fixture) + README + commit
+
+**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — (whole item: admin-only button → clone the relevant repos → run the import script)"`
+**Story:** `.agent/user_stories/admin-sync-button.md`
+
+## Objective
+The story gate: a deterministic E2E that runs the **real** sync path end-to-end (real `git clone` of a local `file://` fixture repo, real import against the mock LLM, real overview regeneration) and verifies both the admin-only visibility and the full button lifecycle, plus README docs, story file, and the phase commit.
+
+## Work
+1. `tests/e2e/test_sync_button.py` (new) — module fixtures:
+   - Build a temp git repo in a `tmp_path` via `subprocess` (`git init -q`, write `notes/sync-fixture.md` containing a unique sentinel `RESE-SYNC-SENTINEL-9b2c`, `git add -A && git -c user.email=e@x -c user.name=t commit -qm one`); set the app's env for this module: `BOR_GIT_SOURCES=file://<repo>`, `BOR_SOURCES_DIR=<tmp_path>/checkouts` (follow the E2E conftest pattern for per-module app env; git is a documented environment prerequisite — phase 28).
+   - Truncate `query_log` (and the KB tables the fixture needs) per the existing E2E isolation pattern so the run starts clean.
+2. Tests (in isolation):
+   - **`test_anonymous_sees_no_button`** — load `/sources.html` logged out: `#sync-btn` is `hidden` (or absent from the revealed DOM); `POST /api/sync` via `page.request` → 403.
+   - **`test_admin_sync_lifecycle`** — sign in (reuse `tests/e2e/auth_helpers.py`):
+     - `#sync-btn` visible with label "Sync sources".
+     - Click → button disabled, label "Syncing…".
+     - Wait (poll with Playwright, generous timeout ~60 s — real git + embed against the mock LLM): label becomes `Synced …`, `#sync-result` shows `1 added` (the fixture doc).
+     - The Sources table (`#docs-tbody`) now contains the fixture path `notes/sync-fixture.md`; the `kb_overview` row is non-empty (phase-31 regeneration ran — DB check in the test).
+     - Re-click → a second run completes with `0 added · 1 unchanged` (idempotent pull + hash-skip).
+   - **`test_double_trigger_409`** (integration-level, may live in `tests/integration/test_sync_api.py` if E2E timing is too flaky — the integration test already covers this; include here only if deterministic): start a sync, immediately `POST /api/sync` again → 409.
+3. `README.md` — new "Sync from the UI" subsection under the import workflow: what the button does (clone/pull → import `--prune`-equivalent → KB overview refresh), the states (Syncing…/Synced/error), the 409 behavior, and prerequisites (`BOR_GIT_SOURCES` set; git available).
+4. `.agent/user_stories/admin-sync-button.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
+5. Commit: `git commit --no-gpg-sign -m "feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status"`; move `.agent/phases/todo/32_admin_sync_button/` → `.agent/phases/complete/`.
+
+## Testing & Quality
+- E2E: `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` + git on PATH; mock LLM — no live aipi).
+- Regression in isolation: `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py`, `test_global_tuning.py`.
+- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
+
+## Completion Criteria
+- [ ] All E2E tests green in isolation, including the idempotent second run and the `kb_overview` freshness check.
+- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
+- [ ] README + story file complete; one `--no-gpg-sign` commit made.