feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status

This commit is contained in:
2026-08-25 21:39:38 -04:00
parent 0654b304e1
commit 52136fe307
13 changed files with 1621 additions and 2 deletions
+92
View File
@@ -0,0 +1,92 @@
# Story: Admin Sync Button (One-Click Doc Import Sync)
**Phase:** `32_admin_sync_button.md` · **E2E:** `tests/e2e/test_sync_button.py`
## Narrative
As **the admin (owner)**, I don't want to SSH in and hand-run the import
script every time my notes repos move. I want **a button that only I can
see** — on the Sources page — that **triggers a doc import sync by
cloning the relevant repos and then running the import script**, with
live feedback so I always know whether it is running, what it changed,
or why it failed.
- **Given** `BOR_GIT_SOURCES` is set (the git repos) and I am signed in
as the admin
- **When** I click **Sync sources** on the Sources page
- **Then** the app clones/pulls each repo, re-imports with prune (the
`--prune` equivalent — the canonical "mirror the repos" action), and
refreshes the KB overview when anything changed — while the button
reports the whole lifecycle (**Syncing…** → **Synced HH:MM** + counts,
or an error banner naming the failure) and never sits stale or stuck.
Anonymous visitors never see the button, and the sync endpoints answer
them with 403.
## Acceptance criteria
1. **Admin-only visibility.** The button ships `hidden` in
`sources.html` (anonymous-safe) and `header.js` reveals it for the
admin on the SAME cached whoami that reveals `#nav-sources` /
`#nav-tuning` (one fetch, no extra whoami call); anonymous users
never see it.
2. **The sync API (A10 extended, A12 in-process).** `POST /api/sync`
(admin only) starts one background task — `clone_or_pull` each
`BOR_GIT_SOURCES` repo (phase 28, reused) → `import_sources(prune=True)`
→ `regenerate_overview` when the KB changed (phase 31) — and returns
`202`. A second trigger while a run is in flight returns
`409 {"detail": "a sync is already running"}` (one sync at a time).
`GET /api/sync/status` (admin only) reports `idle | running |
success | failed` with ISO-8601 timestamps and the run's `detail` /
`error`. Both endpoints answer anonymous callers with 403.
3. **Loud failure.** An unset/empty `BOR_GIT_SOURCES` fails the sync with
"no git sources configured (BOR_GIT_SOURCES)" (manual `--source` dirs
have no repo to clone); a `GitSyncError` fails the run with git's
stderr (the repo named, credentials masked) before any import.
4. **The §7.4 "never stale" lifecycle.** Click → `202` → button disabled
with **Syncing…** (spinning icon, `aria-busy`) + a 2 s poll of the
status endpoint — the ONLY feedback timer; there is no client-side
hard timeout (a sync can legitimately run for minutes; the server
state is authoritative). Success → enabled, **Synced HH:MM** (local
time of `finished_at`) + the last result in `#sync-result`
(`role="status"` / `aria-live="polite"`; "added" always announced,
zero terms omitted, a no-op run reads `0 added · 1 unchanged`).
Failure → enabled, retry-ready **Sync sources** + the `role="alert"`
banner naming the error. A `409` adopts the in-flight run (never a
second poll loop); a reload mid-sync re-attaches to the running run;
a `403` hides the button (defense in depth).
5. **Idempotent.** Re-syncing an unchanged repo is a fast-forward pull +
sha256 hash skip — nothing re-embedded, the overview left alone
(change-gated), the result `0 added · 1 unchanged`.
6. **Quality gates.** Integration (`tests/integration/test_sync_api.py`):
anonymous 403s; idle → running → success/failed transitions with git +
import + overview mocked; 409 double trigger; `GitSyncError` →
`failed` with the repo named and the import never called; `prune=True`
asserted. Unit (`tests/unit/test_sync_button.py`, frontend-assertion):
the ship-hidden markup, the header reveal, the state machine (2 s
poll, 202/409/403 branches, terminal labels, single-poll guard, no
client timeout), the CSS states (spin + reduced-motion opt-out,
disabled, focus-visible, contrast ≥ 4.5:1, 44 px touch floor).
Coverage `app/` > 90 % (`app/api/sync.py` fully covered).
7. **E2E (this story's gate).** `tests/e2e/test_sync_button.py`, run in
isolation: a real local `file://` git fixture repo (deterministic, no
network — git is a documented environment prerequisite, phase 28) with
the mock LLM proves the admin-only visibility, the full lifecycle
against the REAL clone → import → overview path (including the
idempotent second run and the fresh `kb_overview` row), and the 409
double trigger.
## Playwright Mapping Rule
`tests/e2e/test_sync_button.py` — run in isolation (Chromium +
`podman compose up -d db` + git on PATH; mock LLM, no live aipi). The
module overrides the session app fixture with per-module env
(`BOR_GIT_SOURCES=file://<fixture repo>`, its own `BOR_SOURCES_DIR`) and
truncates the KB tables before each test (the E2E isolation pattern):
1. `test_anonymous_sees_no_button` → AC 1 + 2 (the button never leaves
`hidden`; both endpoints 403).
2. `test_admin_sync_lifecycle` → AC 2 + 4 + 5 (button visible for the
admin; click → **Syncing…** (disabled) → **Synced HH:MM** +
`1 added`; the fixture path `notes/sync-fixture.md` in the Sources
table; the `kb_overview` row fresh and non-empty (DB check); the
idempotent second run → `0 added · 1 unchanged`).
3. `test_double_trigger_409` → AC 2 (second trigger while running → 409
with the exact detail; the single in-flight run still completes).
+38
View File
@@ -287,6 +287,44 @@ BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in <dir>/<repo-name>/
**nothing** (no partial junk). Fix the URL/connectivity and re-run — the
other checkouts stay on disk and are pulled as usual.
### Sync from the UI
The **Sync sources** button on the **Sources** page — visible to the
**admin only** (anonymous visitors never see it) — runs the whole
git-source refresh in one click, in-process:
1. **clone/pull** every `BOR_GIT_SOURCES` repo (the same
`clone_or_pull` the CLI uses — shallow clone on first run,
`git pull --ff-only` afterwards);
2. **re-import with prune** — the `--prune` equivalent, so files deleted
upstream leave the index (the button is the canonical "mirror the
repos" action); the sha256 delta still skips unchanged files, so an
unchanged re-sync re-embeds nothing;
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
chat turn injects) — but only when the import actually changed the
knowledge base.
- **Prerequisites:** `BOR_GIT_SOURCES` must be set — an unset/empty list
fails the sync loudly ("no git sources configured"), because the button
targets the git repos only (manual `--source` directories have no repo
to clone) — and `git` must be on the app's `PATH`.
- **States:** clicking starts the run (`202`) and the button goes
disabled with **Syncing…** (spinning icon) while the page polls
`GET /api/sync/status` every 2 s. There is deliberately **no
client-side timeout** — a clone + embed can legitimately take minutes,
so the poll is the feedback loop and the server state is authoritative.
On success the button settles to **Synced HH:MM** with the last result
in a live region (`1 added`, `0 added · 1 unchanged`, …); on failure it
re-enables (retry-ready) and a red error banner names the failure (git's
stderr, with any embedded credentials masked).
- **One sync at a time:** a second trigger while a run is in flight gets
`409` ("a sync is already running"); the UI adopts the in-flight run
instead of starting a second one, and a page reload mid-sync re-attaches
to it the same way.
- **Idempotent:** re-syncing unchanged repos is a no-op — fast-forward
pull, hash skip, and the overview is left alone (its regeneration is
change-gated).
## Checking retrieval quality
Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
+177
View File
@@ -0,0 +1,177 @@
"""Sources sync API — one-click KB mirror (phase 32, task 01).
Admin-only ``POST /api/sync`` + ``GET /api/sync/status`` behind the
existing :func:`app.core.auth.require_admin` (A10 extended, phase 16
pattern — the public API surface stays stateless, the signed cookie
remains the only session state, same as ``/api/steering``).
The button's backend runs the full document sync **in-process** (A12
untouched — no queue, no new services): one ``asyncio`` background task
plus a module-level :class:`SyncStatus` that the UI polls every 2 s
(task 02). One sync at a time — ``POST`` while a run is in flight is
409; the status object is authoritative, so the UI can never sit on a
stale button state (§7.4 adaptation, phase locked decisions).
Pipeline (the canonical "mirror the repos" action — phase locked
decisions):
1. resolve the ``BOR_GIT_SOURCES`` URLs — empty/missing fails loudly
(``no git sources configured``) instead of silently importing the
legacy local directories;
2. :func:`scripts.git_sync.clone_or_pull` each repo into
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
re-implemented; a failing repo aborts before any import);
3. ``import_sources(..., prune=True)`` — prune so files deleted
upstream leave the index (the CLI's no-prune default is unchanged);
4. when the import changed the KB (added + updated > 0),
``regenerate_overview`` refreshes the single ``kb_overview`` row
(phase 31 trigger, best-effort inside).
Status is in memory: a restart mid-sync loses the running state
(accepted — the next click re-syncs idempotently).
"""
from __future__ import annotations
import asyncio
import logging
import re
from dataclasses import dataclass, field
from datetime import UTC, datetime
from pathlib import Path
from typing import Any, Literal
from fastapi import APIRouter, Depends, HTTPException
from app.config import get_settings
from app.core.auth import require_admin
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
from scripts.git_sync import GitSyncError, clone_or_pull
from scripts.import_docs import repo_name
logger = logging.getLogger("app.api.sync")
router = APIRouter(
prefix="/sync",
tags=["sync"],
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
)
#: ``user:pass@`` inside any error text (git stderr, endpoint URLs) —
#: masked so a sync failure can never leak credentials into the UI.
_CREDS_RE = re.compile(r"[A-Za-z0-9._~%*-]+:[A-Za-z0-9._~%*-]+@")
def _sanitize_error(message: str) -> str:
"""Mask credentials embedded in an error string (no secrets in the UI).
Git's stderr is otherwise surfaced verbatim (phase locked decisions) —
it names the failing repo and git's reason, which is what the admin
needs to fix things.
"""
return _CREDS_RE.sub("*****@", message)
@dataclass
class SyncStatus:
"""In-memory state of the (at most one) in-flight sync run.
``state`` is a four-state machine: ``idle`` (never run / reset),
``running``, ``success``, ``failed``. Terminal states carry the run's
``detail`` (success) or ``error`` (failure) so the UI can render the
last result after a page reload (task 02's re-attach behavior).
"""
state: Literal["idle", "running", "success", "failed"] = "idle"
started_at: datetime | None = None
finished_at: datetime | None = None
detail: dict[str, Any] = field(default_factory=dict)
error: str | None = None
_status = SyncStatus()
_task: asyncio.Task[None] | None = None
@router.get("/status")
def sync_status() -> dict[str, Any]:
"""Current sync state (the UI polls this every 2 s — task 02).
``started_at`` / ``finished_at`` are ISO-8601 strings or null.
"""
return {
"state": _status.state,
"started_at": _status.started_at.isoformat() if _status.started_at else None,
"finished_at": _status.finished_at.isoformat() if _status.finished_at else None,
"detail": _status.detail,
"error": _status.error,
}
@router.post("", status_code=202)
async def start_sync() -> dict[str, str]:
"""Start the clone → import → overview sync as a background task.
202 + ``sync started`` kicks off :func:`_run_sync` on the app's event
loop. 409 when a run is already in flight (one sync at a time — the
status endpoint is the single source of truth for the run, and the
UI re-attaches to it rather than starting a second one).
"""
global _task
if _task is not None and not _task.done():
raise HTTPException(status_code=409, detail="a sync is already running")
_task = asyncio.create_task(_run_sync())
return {"detail": "sync started"}
async def _run_sync() -> None:
"""The full sync pipeline, one in-process background task.
Every failure mode (git, embeddings, anything else) lands in the
``failed`` state with a sanitized ``error`` string — a background
task must die in state, never as an unobserved exception.
``CancelledError`` is deliberately *not* caught: app shutdown
cancels the task, and swallowing that would mask a real stop.
"""
_status.state = "running"
_status.started_at = datetime.now(UTC)
_status.finished_at = None
_status.detail = {}
_status.error = None
try:
settings = get_settings()
git_urls = settings.git_source_list
if not git_urls:
# The button targets BOR_GIT_SOURCES only (manual --source
# dirs have no repo to clone) — an empty config fails loudly
# instead of silently importing the legacy directories.
raise GitSyncError("no git sources configured (BOR_GIT_SOURCES)")
logger.info("sync: started repos=%d", len(git_urls))
sources_root = Path(settings.sources_dir).expanduser()
sources = [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
llm = LLMClient()
summary: ImportSummary = await import_sources(sources, llm, prune=True)
overview = False
if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm)
_status.state = "success"
_status.finished_at = datetime.now(UTC)
_status.detail = {
"files": summary.files,
"added": summary.added,
"updated": summary.updated,
"unchanged": summary.unchanged,
"pruned": summary.pruned,
"errors": summary.errors,
"chunks": summary.chunks,
"summaries": summary.summaries,
"summary_errors": summary.summary_errors,
"overview": overview,
}
logger.info("sync: done detail=%s", _status.detail)
except Exception as e: # noqa: BLE001 — a background task dies in state, see above
logger.exception("sync: failed")
_status.state = "failed"
_status.finished_at = datetime.now(UTC)
_status.error = _sanitize_error(str(e))
+2
View File
@@ -25,6 +25,7 @@ from app.api.docs import router as docs_router
from app.api.health import router as health_router
from app.api.steering import router as steering_router
from app.api.suggestions import router as suggestions_router
from app.api.sync import router as sync_router
from app.config import get_settings
from app.core.auth import ensure_admin_configured
from app.core.debugging import configure_debugging
@@ -64,6 +65,7 @@ def create_app() -> FastAPI:
app.include_router(docs_router, prefix="/api")
app.include_router(chat_router, prefix="/api")
app.include_router(steering_router, prefix="/api")
app.include_router(sync_router, prefix="/api")
static_dir = Path(settings.static_dir).resolve()
if static_dir.is_dir():
+8 -1
View File
@@ -12,7 +12,10 @@
* every page that has a nav (chat, sources, tuning, login),
* revealed for admin. The links SHIP hidden in the HTML
* (anonymous-safe default — the phase-16 "absent, not hidden"
* spirit), so no anonymous user ever sees one for a frame;
* spirit), so no anonymous user ever sees one for a frame; and the
* Sources page's "Sync sources" button (#sync-btn, phase 32) —
* the same ship-hidden / reveal-for-admin contract on the SAME
* cached whoami (one fetch, no extra request);
* • the sign-out click binding (POST /api/logout → reload) — moved
* here from app.js so there is exactly one implementation;
* • clearChatStorage() — the phase-14 conversation key, for the
@@ -68,6 +71,10 @@ export async function initSharedHeader() {
// same ship-hidden / reveal-for-admin contract as the Sources link.
const navTuning = document.querySelector("#nav-tuning");
if (navTuning) navTuning.hidden = !admin;
// Phase 32: the Sources page's "Sync sources" button — admin-only,
// revealed on this same cached whoami (anonymous users never see it).
const syncBtn = document.querySelector("#sync-btn");
if (syncBtn) syncBtn.hidden = !admin;
return admin;
}
+231
View File
@@ -146,6 +146,236 @@ function showEmpty() {
if (tableWrap) tableWrap.hidden = true;
}
/* ---------- Phase 32: the admin "Sync sources" button (§7.4) ----------
* The "never stale" lifecycle for a long background job:
*
* idle → click → POST /api/sync
* 202 → "Syncing…" (disabled, aria-busy, spinning icon) + a
* 2 s poll of GET /api/sync/status — the feedback loop;
* 409 adopts the in-flight run the same way (one poll
* loop at a time, never two);
* success → "Synced HH:MM" + last-result counts in #sync-result
* (aria-live — announced to screen readers) + the
* catalog re-fetches live (never a stale table);
* failed → "Sync sources" (retry-ready) + the role="alert"
* banner naming the error.
*
* NO client-side hard timeout (phase locked decision): a sync can
* legitimately run for minutes (clone + embed), so the 2 s poll is the
* feedback loop and the server state is authoritative — the button is
* disabled until the run reaches a terminal state, so it can never sit
* stale OR stuck. On load (admin only) the page re-attaches: a running
* run re-enters the running state (reload mid-sync), a terminal run
* renders its last result. A 403 anywhere hides the button (defense in
* depth — header.js's whoami reveal is the primary gate).
*/
const syncBtn = document.querySelector("#sync-btn");
const syncLabel = document.querySelector("#sync-label");
const syncIcon = syncBtn ? syncBtn.querySelector(".sync-icon") : null;
const syncResult = document.querySelector("#sync-result");
const syncErrorBanner = document.querySelector("#sync-error-banner");
const syncErrorText = document.querySelector("#sync-error-text");
const SYNC_POLL_MS = 2000; // the 2 s status poll (task 02)
let syncPollTimer = null; // at most ONE live poll loop
function stopSyncPolling() {
if (syncPollTimer !== null) {
clearTimeout(syncPollTimer);
syncPollTimer = null;
}
}
function showSyncError(detail) {
if (syncErrorText) syncErrorText.textContent = detail || "The sync failed.";
if (syncErrorBanner) syncErrorBanner.hidden = false;
}
function hideSyncError() {
if (syncErrorText) syncErrorText.textContent = "";
if (syncErrorBanner) syncErrorBanner.hidden = true;
}
/* The local HH:MM of finished_at — 24-hour, locale-independent, so the
* "Synced 14:32" last-result label is deterministic. */
function fmtSyncTime(iso) {
const d = new Date(iso);
if (Number.isNaN(d.getTime())) return "";
const pad = (n) => String(n).padStart(2, "0");
return `${pad(d.getHours())}:${pad(d.getMinutes())}`;
}
/* The last-result line for #sync-result (aria-live). "added" is ALWAYS
* announced (the run's headline term); "updated" / "pruned" only when
* they happened (zero terms omitted); "unchanged" whenever it is
* non-zero — or whenever nothing was added or updated, so a no-op
* re-sync reads "0 added · 1 unchanged" instead of an empty live
* region (the story gate's idempotency check). */
function fmtSyncResult(detail) {
const d = detail || {};
const added = d.added || 0;
const updated = d.updated || 0;
const parts = [`${added} added`];
if (updated > 0) parts.push(`${updated} updated`);
if ((d.unchanged || 0) > 0 || (added === 0 && updated === 0)) {
parts.push(`${d.unchanged || 0} unchanged`);
}
if ((d.pruned || 0) > 0) parts.push(`${d.pruned} pruned`);
return parts.join(" · ");
}
function enterRunningState() {
syncBtn.disabled = true;
syncBtn.setAttribute("aria-busy", "true");
if (syncIcon) syncIcon.classList.add("is-spinning");
syncLabel.textContent = "Syncing…";
if (syncResult) syncResult.textContent = "";
hideSyncError();
}
/* Settle the button back to clickable + un-spun with the given label. */
function settleSyncButton(label) {
syncBtn.disabled = false;
syncBtn.removeAttribute("aria-busy");
if (syncIcon) syncIcon.classList.remove("is-spinning");
syncLabel.textContent = label;
}
function applySyncSuccess(status) {
const time = fmtSyncTime(status.finished_at);
settleSyncButton(time ? `Synced ${time}` : "Synced");
if (syncResult) syncResult.textContent = fmtSyncResult(status.detail);
hideSyncError();
// The KB just changed — refresh the catalog live so the table, stats,
// and empty state never sit stale under the "Synced" label (the sync is
// the page's own action; a reload should not be needed to see it).
loadDocs();
}
function applySyncFailure(status) {
settleSyncButton("Sync sources"); // retry-ready
if (syncResult) syncResult.textContent = "";
showSyncError(status.error);
}
/* A run can only vanish with a server restart mid-sync (status resets
* to idle — the phase-accepted behavior): re-enable retry-ready with no
* banner (there is no error to name; the next click re-syncs).
* Idempotent — also the post-403 cleanup. */
function applySyncIdle() {
settleSyncButton("Sync sources");
if (syncResult) syncResult.textContent = "";
hideSyncError();
}
/* The 2 s poll loop — the ONLY feedback timer (no client-side hard
* timeout, phase locked decision). One tick at a time (re-scheduled
* only while the run is still live, so an in-flight fetch can never
* overlap the next tick), and startSyncPolling refuses to run a second
* loop (a 409 adoption or a reload never doubles the polling). */
function startSyncPolling() {
if (syncPollTimer !== null) return;
const tick = async () => {
let status = null;
let notAdmin = false;
try {
const r = await fetch("/api/sync/status");
if (r.status === 403) notAdmin = true;
else if (r.ok) status = await r.json();
} catch {
/* network blip — the next tick retries (no client timeout to trip) */
}
if (notAdmin) {
// Session lost mid-sync: defense in depth — hide the button.
stopSyncPolling();
syncBtn.hidden = true;
applySyncIdle();
return;
}
if (status && status.state === "success") {
stopSyncPolling();
applySyncSuccess(status);
return;
}
if (status && status.state === "failed") {
stopSyncPolling();
applySyncFailure(status);
return;
}
if (status && status.state === "idle") {
// The run died with a server restart — retry-ready, no banner.
stopSyncPolling();
applySyncIdle();
return;
}
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
};
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
}
/* Click → POST /api/sync. 202 starts the run; 409 adopts the in-flight
* one (started elsewhere — e.g. a second tab); 403 hides the button
* (defense in depth); anything else names the failure in the banner and
* leaves the button retry-ready (the never-stale contract). */
async function startSync() {
let r;
try {
r = await fetch("/api/sync", { method: "POST" });
} catch {
showSyncError("Could not reach the server to start the sync — try again.");
return;
}
if (r.status === 403) {
stopSyncPolling();
syncBtn.hidden = true;
applySyncIdle();
return;
}
if (r.status === 202 || r.status === 409) {
enterRunningState();
startSyncPolling();
return;
}
let detail = "";
try {
detail = (await r.json()).detail || "";
} catch {
/* non-JSON error body */
}
showSyncError(detail || `The server refused to start the sync (${r.status}).`);
}
/* Load-time re-attach (admin only — the IIFE runs this after the
* whoami gate): a running run re-enters the running state (the user may
* have reloaded mid-sync), a terminal run renders its last result, idle
* renders nothing. */
async function initSyncButton() {
if (!syncBtn) return;
let status;
try {
const r = await fetch("/api/sync/status");
if (r.status === 403) {
syncBtn.hidden = true; // defense in depth
return;
}
if (!r.ok) return;
status = await r.json();
} catch {
return; // network blip — the button stays idle and clickable
}
if (status.state === "running") {
enterRunningState();
startSyncPolling();
} else if (status.state === "success") {
applySyncSuccess(status);
} else if (status.state === "failed") {
applySyncFailure(status);
}
/* idle → nothing to render */
}
if (syncBtn) syncBtn.addEventListener("click", startSync);
(async () => {
await initSharedHeader(); // phase 19: Sign in/out + Sources link in the shared bar
if (!(await isAdmin())) {
@@ -158,4 +388,5 @@ function showEmpty() {
}
if (gateEl) gateEl.hidden = true;
loadDocs();
initSyncButton(); // phase 32: re-attach to a running / last sync run
})();
+59
View File
@@ -313,6 +313,48 @@ html::after {
.auth-link:disabled { opacity: 0.6; cursor: wait; }
.auth-link svg { width: 16px; height: 16px; display: none; }
/* Phase 32: the admin-only "Sync sources" pill (Sources header) — the
same ghost pill as New chat / the auth links, so the bar keeps one
visual language. ink-soft on surface ≈6.9:1 (WCAG AA); hover pair
brand-ink/brand-soft ≈6.9:1. The refresh icon is always visible (it
doubles as the running-state spinner); icon-only below 640px like
the other pills (aria-label keeps the accessible name). ≥44px touch
target at every width; :focus-visible via the global rule. */
.sync-btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 0.4rem;
min-height: 44px;
padding: 0.5rem 0.9rem;
border-radius: 999px;
border: 1px solid var(--line);
background: transparent;
color: var(--ink-soft);
font: inherit;
font-weight: 600;
font-size: 0.95rem;
white-space: nowrap;
cursor: pointer;
}
.sync-btn:hover { background: var(--brand-soft); color: var(--brand-ink); }
.sync-btn:disabled { opacity: 0.6; cursor: wait; }
.sync-icon { width: 16px; height: 16px; display: block; flex: 0 0 auto; }
/* Running state: the refresh icon spins (reuses the shared spin
keyframes) — the visible half of "Syncing…" while the 2 s poll waits. */
.sync-btn .sync-icon.is-spinning { animation: spin 1s linear infinite; }
@media (prefers-reduced-motion: reduce) {
.sync-btn .sync-icon.is-spinning { animation: none; }
}
/* The aria-live last-result announcer ("2 added · 1 pruned") — soft ink
on the header surface (≈6.9:1), small mono to match the stat cards. */
.sync-result {
color: var(--ink-soft);
font-family: var(--mono);
font-size: 0.8rem;
white-space: nowrap;
}
/* "Tuning" toggle (phase 15): ghost pill like New chat + a mono count
badge (brand-ink on brand-soft ≈6.9:1). The label is visually-hidden
(not removed) below 640px so the accessible name keeps the word.
@@ -1598,6 +1640,23 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.auth-link { padding: 0.4rem 0.55rem; }
.auth-label { display: none; }
.auth-link svg { display: block; }
/* Phase 32: the sync pill goes icon-only like the other pills (the
aria-label keeps the accessible name); the spinning icon is the
visible running state on a touch screen. */
.sync-btn { padding: 0.4rem 0.55rem; }
.sync-label { display: none; }
/* The last-result counts stay ANNOUNCED (aria-live is untouched) but
go visually hidden — the 58px bar has no room for the text; the
icon carries the visible state. Same clip recipe as .steering-label. */
.sync-result {
position: absolute !important;
width: 1px; height: 1px;
margin: -1px; padding: 0;
overflow: hidden;
clip: rect(0 0 0 0);
white-space: nowrap;
border: 0;
}
.steering-toggle { padding: 0.4rem 0.55rem; }
/* Visually hidden, NOT display:none — the accessible name keeps the
word "Tuning" next to the count badge. */
+27 -1
View File
@@ -37,6 +37,20 @@
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M12 5v14M5 12h14"/></svg>
<span class="new-chat-label">New chat</span>
</button>
<!-- Phase 32: the admin-only "Sync sources" button (TODO.md:5) —
SHIPS hidden (anonymous-safe), header.js reveals it for the
admin on the SAME cached whoami that reveals #nav-sources /
#nav-tuning (one fetch, no extra whoami call). sources.js
drives the §7.4 "never stale" lifecycle: idle → "Syncing…"
(disabled + spinning icon + 2 s GET /api/sync/status poll) →
last result ("Synced HH:MM" + counts in #sync-result) or the
role="alert" error banner. #sync-result is the aria-live
announcer for the last result. -->
<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">
<svg class="sync-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12a9 9 0 1 1-9-9c2.52 0 4.93 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/></svg>
<span class="sync-label" id="sync-label">Sync sources</span>
</button>
<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>
<a href="/login.html?next=/sources.html" class="auth-link" id="sign-in-link" hidden>
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M10 4h8a2 2 0 0 1 2 2v12a2 2 0 0 1-2 2h-8"/><path d="M4 12h11"/><path d="m12 9 3 3-3 3"/></svg>
<span class="auth-label">Sign in</span>
@@ -50,11 +64,23 @@
<main id="main" class="app-main" tabindex="-1">
<div class="container sources-shell">
<!-- Phase 32: the sync failure banner — the chat error-banner
markup style (kb-banner + is-error), role="alert" so a failed
sync is announced. sources.js fills #sync-error-text and
un-hides it on a failed run (the button re-enables,
retry-ready); a new sync hides it again. -->
<div class="kb-banner is-error" id="sync-error-banner" role="alert" hidden>
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 3.6 22.2 20.4H1.8Z"/><path d="M12 9.5v4.6"/><path d="M12 17.4h.01"/></svg>
<span id="sync-error-text"></span>
</div>
<div class="page-head">
<h1>Knowledge base</h1>
<p class="page-sub">
Every <code>*.md</code> file indexed from <code>~/Homelab</code> and
<code>~/Deployments</code>. Re-run the import to refresh.
<code>~/Deployments</code>. Re-run the import to refresh — or hit
<strong>Sync sources</strong> in the header to clone the repos and
re-import.
</p>
</div>
+283
View File
@@ -0,0 +1,283 @@
"""Phase 32 E2E (Playwright): the admin-only "Sync sources" button.
Story: ``.agent/user_stories/admin-sync-button.md``
Run in isolation (DB must be up: ``podman compose up -d db``; git on
PATH — a documented environment prerequisite, phase 28):
uv run pytest tests/e2e/test_sync_button.py -v --no-cov
The story gate runs the **real** sync path end to end — a real
``git clone`` of a local ``file://`` fixture repo (deterministic, no
network), a real import against the mock LLM, a real KB-overview
regeneration — plus the admin-only visibility and the full button
lifecycle (§7.4: "Syncing…" → "Synced HH:MM" + counts, the idempotent
second run, the 409 double trigger).
Per-module app env (the E2E conftest pattern, module-scoped): this
story's app boots with ``BOR_GIT_SOURCES=file://<fixture repo>`` and
its own ``BOR_SOURCES_DIR`` — the session app (no git sources) is
never started in this isolated run, so no port clash.
Test → story mapping (Playwright Mapping Rule):
1. ``test_anonymous_sees_no_button`` → AC 1 + 3 (hidden button, 403s)
2. ``test_admin_sync_lifecycle`` → AC 2 + 4 (full lifecycle + idempotent
second run + the fresh ``kb_overview`` row)
3. ``test_double_trigger_409`` → AC 2 (one sync at a time)
"""
from __future__ import annotations
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from app.models import KbOverview
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The unique sentinel inside the fixture doc (task 03 step 1) — its
#: import into the Sources table proves the REAL clone was indexed.
SENTINEL = "RESE-SYNC-SENTINEL-9b2c"
FIXTURE_DOC = "notes/sync-fixture.md"
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute.
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
#: Real git clone + embed against the mock LLM — generous budget
#: (task 03: the sync can legitimately take a while, no client timeout).
SYNC_TIMEOUT_MS = 60_000
def _git(cwd: Path, *args: str) -> None:
"""Run git in *cwd*; a non-zero exit fails the fixture loudly."""
proc = subprocess.run(["git", *args], cwd=cwd, capture_output=True, text=True)
if proc.returncode != 0:
raise AssertionError(f"git {' '.join(args)} failed: {proc.stderr.strip()}")
@pytest.fixture(scope="module")
def sync_git_repo(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""A real one-commit git repo the sync must clone (task 03 step 1).
``tmp_path`` is function-scoped while the module-scoped app fixture
needs the repo for the module's lifetime, so it is built under
``tmp_path_factory`` (the same pytest-managed temp area, module-safe)
via real ``git`` subprocess calls.
"""
root = tmp_path_factory.mktemp("sync_git")
repo = root / "homelab-notes"
(repo / "notes").mkdir(parents=True)
(repo / "notes" / "sync-fixture.md").write_text(
"# Sync fixture note\n"
"\n"
"One small note that exists only to prove the admin sync button\n"
"end to end: a real git clone, a real import, a real KB overview\n"
"regeneration.\n"
"\n"
f"Marker: {SENTINEL}\n",
encoding="utf-8",
)
_git(repo, "init", "-q")
_git(repo, "add", "-A")
_git(
repo,
"-c", "user.email=e@x", "-c", "user.name=t",
"-c", "commit.gpgsign=false", # the fixture commit never signs
"commit", "-qm", "one",
)
assert (repo / ".git").is_dir()
return repo
@pytest.fixture(scope="module")
def app_server(mock_llm: int, sync_git_repo: Path) -> Iterator[str]:
"""The real app under test — per-module env: the sync's subject is a
real ``file://`` git source with its own checkout dir (the conftest
session app boots without ``BOR_GIT_SOURCES`` and is never started
in this isolated run)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — keeps every story
# suite's deterministic gate behavior; production default stays 0.62.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# Phase 32: the sync button's subject — one real local repo.
env["BOR_GIT_SOURCES"] = f"file://{sync_git_repo}"
env["BOR_SOURCES_DIR"] = str(sync_git_repo.parent / "checkouts")
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_kb() -> None:
"""Fresh KB per test (the E2E isolation pattern): the sync's counts
and the ``kb_overview`` row must be the sync's own doing."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview"))
db.commit()
@pytest.fixture(autouse=True)
def _clean_kb(db_ready: None) -> Iterator[None]:
_truncate_kb()
yield
def _overview_row() -> KbOverview | None:
with SessionLocal() as db:
return db.get(KbOverview, 1)
def _wait_sync_done(page: Page, app_url: str, timeout_s: float = 60.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state — exactly what the UI's 2 s poll loop
observes."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = page.request.get(f"{app_url}/api/sync/status")
assert r.status == 200
body = r.json()
if body["state"] in ("success", "failed"):
return body
time.sleep(0.5)
raise AssertionError(f"sync did not reach a terminal state: {body}")
# --- 1. Anonymous: no button, locked endpoints ----------------------------
def test_anonymous_sees_no_button(page: Page, app_url: str, db_ready: None) -> None:
"""AC 1 + 3 — anonymous: ``#sync-btn`` never leaves ``hidden`` (the
whoami reveal is admin-only, so for a logged-out visitor it is
absent from the *revealed* DOM) and both sync endpoints answer
403 — the admin-only surface (A10 extended, phase 16 pattern)."""
page.goto(f"{app_url}/sources.html")
expect(page.locator("#sync-btn")).to_be_hidden()
# The same whoami gate that hides the button also hides the admin
# nav link and shows the catalog sign-in gate.
expect(page.locator("#nav-sources")).to_be_hidden()
expect(page.locator("#sources-gate")).to_be_visible()
assert page.request.get(f"{app_url}/api/sync/status").status == 403
assert page.request.post(f"{app_url}/api/sync").status == 403
# --- 2. Admin: the full lifecycle against the real sync path --------------
def test_admin_sync_lifecycle(page: Page, app_url: str, db_ready: None) -> None:
"""AC 2 + 4 — click → "Syncing…" (disabled) → "Synced HH:MM" + the
counts, against the REAL pipeline (git clone of the ``file://``
fixture → import with prune → KB overview regeneration, mock LLM).
Then the idempotent second run: fast-forward pull + sha256 hash
skip → "0 added · 1 unchanged"."""
login(page, app_url) # lands on /sources.html (the button's home)
btn = page.locator("#sync-btn")
expect(btn).to_be_visible()
expect(page.locator("#sync-label")).to_have_text("Sync sources")
# --- run 1: clone + import + overview --------------------------------
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled() # never stale — re-enabled at the terminal state
# The fixture doc (one A9-format file) is the only change.
expect(page.locator("#sync-result")).to_have_text("1 added")
# The REAL clone was imported: the fixture path is in the Sources
# table (the sentinel lives inside it).
expect(page.locator("#docs-tbody tr", has_text=FIXTURE_DOC)).to_have_count(1)
# Phase-31 regeneration ran (the import changed the KB): the single
# kb_overview row is fresh and non-empty (DB check — truncated
# before this test, so it is the sync's own doing).
row = _overview_row()
assert row is not None and row.content.strip(), (
"kb_overview must be regenerated by a successful, KB-changing sync"
)
# --- run 2: idempotent pull + hash skip -------------------------------
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
# Nothing re-embedded (sha256 delta) — the no-op run announces the
# unchanged count instead of an empty live region.
expect(page.locator("#sync-result")).to_have_text("0 added · 1 unchanged")
# The doc survived the prune re-import (its file is still in the repo).
expect(page.locator("#docs-tbody tr", has_text=FIXTURE_DOC)).to_have_count(1)
# --- 3. Concurrency: one sync at a time ------------------------------------
def test_double_trigger_409(page: Page, app_url: str, db_ready: None) -> None:
"""AC 2 — a second trigger while a run is in flight gets 409
("a sync is already running"); the UI adopts the in-flight run
instead of starting a second one. The E2E path adds that the REAL
run behind the 409 still completes (the adopted run is the one and
only run)."""
login(page, app_url)
first = page.request.post(f"{app_url}/api/sync")
assert first.status == 202, first.text
second = page.request.post(f"{app_url}/api/sync")
assert second.status == 409, second.text
assert second.json()["detail"] == "a sync is already running"
# The adopted (single) run still completes successfully.
body = _wait_sync_done(page, app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == 1 # the fixture doc, fresh after the truncate
+379
View File
@@ -0,0 +1,379 @@
"""Integration: the admin sources-sync API (phase 32, task 01).
Covers the in-process sync runner end to end over HTTP: anonymous 403s
on both endpoints; admin idle → 202 → ``success`` with the full
ImportSummary detail; 409 on a double trigger while a run is in flight;
``GitSyncError`` → ``failed`` with the failing repo named and **zero**
import attempts; empty ``BOR_GIT_SOURCES`` → ``failed`` loudly; an
embedding failure → ``failed`` with any credentials masked; the import
always runs with ``prune=True``; and the phase-31 overview trigger is
change-gated (no ``lite`` call on an unchanged KB).
The git / import / overview layers are monkeypatched in ``app.api.sync``
(same fake style as ``test_import_docs_git.py``) — no real git, no real
DB, no LLM: the runner's state machine and HTTP surface are under test.
The admin client is used **as a context manager** on purpose: the
background sync task lives on the app's event loop, so the loop must
survive across requests — exactly how the app runs under uvicorn.
(A TestClient without the context manager starts a fresh loop per
request and would cancel the task on request exit.)
"""
from __future__ import annotations
import asyncio
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy.orm import Session
from app.api import sync as sync_api
from app.config import Settings
from app.main import app as fastapi_app
from app.rag.importer import ImportSummary
from app.rag.llm import EmbeddingError, LLMClient
from scripts.git_sync import GitSyncError
from tests.conftest import ADMIN_PASSWORD
@pytest.fixture(autouse=True)
def _fresh_sync_state() -> Iterator[None]:
"""The module-level status object + task are process-global: reset them
around every test (both before — a previous test's terminal state
would leak into the idle assertion — and after)."""
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
yield
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
@pytest.fixture()
def sync_client() -> Iterator[TestClient]:
"""Context-managed TestClient — one app event loop across requests
(the background task must survive between the POST and the polls)."""
with TestClient(fastapi_app) as client:
yield client
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
"""Fresh settings (no .env file); explicit kwargs beat any env leaks."""
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _login(client: TestClient) -> None:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
def _poll(client: TestClient, want: str, timeout: float = 5.0) -> dict:
"""Poll ``GET /api/sync/status`` until ``state == want`` (terminal).
Any state other than ``running`` before the deadline fails loudly —
an unexpected ``failed`` must never be masked by the wait.
"""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
body = client.get("/api/sync/status").json()
if body["state"] == want:
return body
assert body["state"] == "running", (
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
)
time.sleep(0.05)
raise AssertionError(f"sync did not reach {want!r} within {timeout}s")
class FakeImportSources:
"""Records every ``import_sources`` call; returns a canned summary."""
def __init__(self, summary: ImportSummary, delay: float = 0.0) -> None:
self.summary = summary
self.delay = delay
self.sources: list[list[Path]] = []
self.llms: list[LLMClient] = []
self.prune_flags: list[bool] = []
async def __call__(
self,
sources: list[Path],
llm: LLMClient,
*,
prune: bool = False,
limit: int | None = None,
session: Session | None = None,
) -> ImportSummary:
self.sources.append(list(sources))
self.llms.append(llm)
self.prune_flags.append(prune)
if self.delay:
await asyncio.sleep(self.delay)
return self.summary
class FakeOverview:
"""Records every ``regenerate_overview`` call; canned result."""
def __init__(self, ok: bool = True) -> None:
self.ok = ok
self.llms: list[LLMClient] = []
async def __call__(self, llm: LLMClient, session: Session | None = None) -> bool:
self.llms.append(llm)
return self.ok
def _fake_clone() -> tuple[list[tuple[str, Path]], object]:
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
calls: list[tuple[str, Path]] = []
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
dest.mkdir(parents=True, exist_ok=True)
(dest / "notes.md").write_text(f"# {dest.name}\ncontent for the KB\n", encoding="utf-8")
calls.append((url, dest))
return dest
return calls, fake_clone_or_pull
# --- anonymous -------------------------------------------------------------
def test_anonymous_gets_403_on_both_endpoints(client: TestClient) -> None:
r = client.get("/api/sync/status")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
r = client.post("/api/sync")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# --- admin: success --------------------------------------------------------
def test_admin_sync_success_reports_full_detail(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
summary = ImportSummary(
files=5, added=1, updated=2, unchanged=2, pruned=3, errors=0,
chunks=11, embed_batches=4, summaries=1, summary_errors=0,
)
fake_import = FakeImportSources(summary)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
fake_overview = FakeOverview(ok=True)
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
_login(sync_client)
assert sync_client.get("/api/sync/status").json() == {
"state": "idle",
"started_at": None,
"finished_at": None,
"detail": {},
"error": None,
}
r = sync_client.post("/api/sync")
assert r.status_code == 202
assert r.json() == {"detail": "sync started"}
body = _poll(sync_client, "success")
assert body["error"] is None
# ISO-8601 timestamps round-trip; finished after started.
started = datetime.fromisoformat(body["started_at"])
finished = datetime.fromisoformat(body["finished_at"])
assert finished >= started
# Every ImportSummary field + the overview flag, verbatim.
assert body["detail"] == {
"files": 5, "added": 1, "updated": 2, "unchanged": 2, "pruned": 3,
"errors": 0, "chunks": 11, "summaries": 1, "summary_errors": 0,
"overview": True,
}
# Git: the configured repo was cloned into BOR_SOURCES_DIR/<repo-name>/.
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")]
# Import: exactly the checkouts, with prune=True (the button is the
# canonical "mirror the repos" action) and a real LLMClient.
assert fake_import.sources == [[tmp_path / "bor" / "repo"]]
assert fake_import.prune_flags == [True]
assert len(fake_import.llms) == 1
assert isinstance(fake_import.llms[0], LLMClient)
# Overview: refreshed (added + updated > 0) with the same client.
assert fake_overview.llms == [fake_import.llms[0]]
def test_unchanged_kb_skips_overview_refresh(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Phase-31 trigger is change-gated: added + updated == 0 → no ``lite`` call."""
repo_url = f"file://{tmp_path / 'repo.git'}"
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
summary = ImportSummary(files=7, added=0, updated=0, unchanged=7, pruned=0)
fake_import = FakeImportSources(summary)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
fake_overview = FakeOverview(ok=True)
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["detail"]["overview"] is False
assert fake_overview.llms == [] # no wasted model call
assert len(fake_import.llms) == 1 # the import itself ran
# --- admin: concurrency ----------------------------------------------------
def test_double_trigger_while_running_returns_409(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
# The in-flight run takes a while (asyncio.sleep) so the second POST
# lands while it is still running.
fake_import = FakeImportSources(ImportSummary(files=1, added=1), delay=0.5)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
r = sync_client.post("/api/sync") # second trigger while running
assert r.status_code == 409
assert r.json() == {"detail": "a sync is already running"}
body = sync_client.get("/api/sync/status").json()
assert body["state"] == "running"
assert body["started_at"] is not None
assert body["finished_at"] is None
assert body["error"] is None
# The (single) run completes; the import ran exactly once.
_poll(sync_client, "success")
assert len(fake_import.sources) == 1
# --- admin: failures -------------------------------------------------------
def test_git_failure_marks_failed_and_skips_import(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'bad.git'}"
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
)
def failing_clone(url: str, dest: Path | str) -> Path:
raise GitSyncError(
f"git clone --depth 1 {url} failed (exit 128): "
"fatal: repository not found"
)
monkeypatch.setattr(sync_api, "clone_or_pull", failing_clone)
fake_import = FakeImportSources(ImportSummary())
monkeypatch.setattr(sync_api, "import_sources", fake_import)
fake_overview = FakeOverview(ok=True)
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert "bad.git" in body["error"] # the failing repo is named
assert "fatal: repository not found" in body["error"]
assert body["detail"] == {}
assert body["finished_at"] is not None
assert fake_import.sources == [] # no partial import
assert fake_overview.llms == []
# A failed run leaves the system restartable: a new POST is accepted.
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "failed")
def test_no_git_sources_configured_fails_loudly(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
monkeypatch.setattr(
sync_api,
"get_settings",
# Whitespace-only is just as unconfigured as empty.
lambda: _settings(git_sources=" , ", sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
fake_import = FakeImportSources(ImportSummary())
monkeypatch.setattr(sync_api, "import_sources", fake_import)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert body["error"] == "no git sources configured (BOR_GIT_SOURCES)"
assert clone_calls == [] # git is never touched
assert fake_import.sources == []
def test_import_error_is_reported_with_credentials_masked(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
async def failing_import(
sources: list[Path],
llm: LLMClient,
*,
prune: bool = False,
limit: int | None = None,
session: Session | None = None,
) -> ImportSummary:
raise EmbeddingError(
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
"failed: connection refused"
)
monkeypatch.setattr(sync_api, "import_sources", failing_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert "*****@aipi.reeseapps.com" in body["error"] # credentials masked
assert "user:secret" not in body["error"]
assert "connection refused" in body["error"] # the reason survives
+325
View File
@@ -0,0 +1,325 @@
"""Unit: the admin "Sync sources" button contract (phase 32, task 02).
The browser behavior is E2E-covered (tests/e2e/test_sync_button.py,
task 03); here we pin the source-level wiring — the anonymous-safe
ship-hidden button markup, the header.js admin reveal on the SAME
cached whoami (no extra fetch), the sources.js sync state machine
(2 s poll, 202 start / 409 adoption / 403 hide, terminal labels,
the aria-live result, the single-poll-loop guard, no client-side hard
timeout), the §7.4 never-stale CSS (spin + reduced-motion opt-out,
disabled state, 44px floor, contrast pair) — so a silent regression is
caught without a browser.
"""
from __future__ import annotations
import re
from pathlib import Path
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
ASSETS = FRONTEND / "assets"
HEADER_JS = ASSETS / "header.js"
SOURCES_JS = ASSETS / "sources.js"
STYLES_CSS = ASSETS / "styles.css"
SOURCES_HTML = FRONTEND / "sources.html"
def _text(path: Path) -> str:
assert path.is_file(), f"missing frontend file: {path}"
return path.read_text(encoding="utf-8")
def _body(js: str, fn_name: str) -> str:
"""The source of the first top-level `function <fn_name>` in js."""
fn = js.find(f"function {fn_name}")
assert fn != -1, f"{fn_name} must be defined"
return js[fn : js.find("\n}", fn)]
# ---------- sources.html: anonymous-safe ship-hidden markup ----------
def test_sync_button_ships_hidden_and_labeled() -> None:
"""#sync-btn SHIPS with the hidden attribute (anonymous-safe —
header.js reveals it for the admin), is a real <button type=
"button">, and carries aria-label="Sync sources" so the accessible
name stays stable across the label states."""
tag = re.search(r"<button[^>]*id=\"sync-btn\"[^>]*>", _text(SOURCES_HTML))
assert tag, "sources.html must carry the #sync-btn button"
attrs = tag.group(0)
assert 'class="sync-btn"' in attrs
assert 'type="button"' in attrs
assert re.search(r"\bhidden\b", attrs), "#sync-btn must ship hidden"
assert 'aria-label="Sync sources"' in attrs
def test_sync_button_has_icon_and_label_span() -> None:
"""The button body is a refresh-cycle svg (aria-hidden — decorative,
the spin is the visible running state) + the #sync-label span with
the idle text, so the label can be swapped by sources.js."""
text = _text(SOURCES_HTML)
btn = text[text.find('id="sync-btn"') : text.find("</button>", text.find('id="sync-btn"'))]
assert re.search(r'<svg[^>]*class="sync-icon"[^>]*aria-hidden="true"', btn)
assert 'id="sync-label"' in btn
assert re.search(r'<span[^>]*id="sync-label"[^>]*>Sync sources</span>', btn)
def test_sync_result_is_the_aria_live_announcer() -> None:
"""#sync-result sits right after the button and is a polite live
region (role="status" + aria-live="polite") — the last-result /
counts announcement for screen readers."""
text = _text(SOURCES_HTML)
tag = re.search(r'<span[^>]*id="sync-result"[^>]*>', text)
assert tag, "sources.html must carry the #sync-result announcer"
attrs = tag.group(0)
assert 'role="status"' in attrs
assert 'aria-live="polite"' in attrs
assert text.find('id="sync-result"') > text.find("</button>", text.find('id="sync-btn"'))
def test_sync_error_banner_is_a_hidden_alert() -> None:
"""The failure banner uses the chat error-banner markup style
(kb-banner + is-error) and role="alert", shipping hidden —
sources.js un-hides it with the error text on a failed run."""
text = _text(SOURCES_HTML)
tag = re.search(r'<div[^>]*id="sync-error-banner"[^>]*>', text)
assert tag, "sources.html must carry the #sync-error-banner"
attrs = tag.group(0)
assert "kb-banner" in attrs and "is-error" in attrs
assert 'role="alert"' in attrs
assert re.search(r"\bhidden\b", attrs)
assert 'id="sync-error-text"' in text
# The banner lives in the page content, not the 64px header bar.
assert text.find('id="sync-error-banner"') > text.find('<main id="main"')
def test_page_sub_copy_mentions_the_button() -> None:
"""The page-sub copy still points at the import CLI and now names
the header button as the one-click alternative (task 02 step 1)."""
sub = re.search(r'<p class="page-sub">(.*?)</p>', _text(SOURCES_HTML), re.DOTALL)
assert sub, "sources.html must keep the .page-sub copy"
assert "Re-run the import" in sub.group(1)
assert "Sync sources" in sub.group(1)
assert "header" in sub.group(1)
def test_sources_page_stays_cdn_free() -> None:
"""No-CDN rule (PLAN §7.3, A11): the new button markup adds no
external references — same-origin assets only (the integration
test_index_html_served_locally re-checks this on the served page)."""
text = _text(SOURCES_HTML)
assert 'src="https://' not in text
assert 'href="https://' not in text
# ---------- header.js: the admin reveal ----------
def test_header_reveals_sync_btn_on_the_admin_branch() -> None:
"""initSharedHeader reveals #sync-btn in the SAME admin branch as
#nav-sources (querySelector + hidden = !admin) — one cached whoami,
no extra whoami call; anonymous users never leave the hidden
default."""
js = _text(HEADER_JS)
fn = js.find("function initSharedHeader")
assert fn != -1
body = js[fn : js.find("\n}", fn)]
assert 'querySelector("#sync-btn")' in body, "#sync-btn must join the admin reveal"
assert "syncBtn.hidden = !admin" in body
# The reveal must not introduce a second whoami call site.
assert js.count('fetch("/api/whoami")') == 1
# ---------- sources.js: the sync state machine ----------
def test_sources_js_calls_the_sync_api() -> None:
"""The click posts to POST /api/sync and the poll loop GETs
/api/sync/status — both through the same-origin API (A10)."""
js = _text(SOURCES_JS)
assert 'fetch("/api/sync", { method: "POST" })' in js
assert 'fetch("/api/sync/status")' in js
def test_sources_js_polls_every_2000ms() -> None:
"""The feedback loop is a 2000 ms poll of the status endpoint,
re-scheduled one tick at a time (setTimeout, not setInterval — an
in-flight fetch can never overlap the next tick)."""
js = _text(SOURCES_JS)
assert "SYNC_POLL_MS = 2000" in js
assert "setTimeout(tick, SYNC_POLL_MS)" in js
assert "setInterval" not in js
def test_sources_js_adopts_409_and_starts_on_202() -> None:
"""202 (started) and 409 (a run started elsewhere — e.g. a second
tab) both enter the running state and start polling: the UI never
starts a second run, it adopts the in-flight one."""
js = _text(SOURCES_JS)
assert "r.status === 202 || r.status === 409" in js
idx = js.find("r.status === 202 || r.status === 409")
branch = js[idx : idx + 200]
assert "enterRunningState()" in branch
assert "startSyncPolling()" in branch
def test_sources_js_hides_the_button_on_403() -> None:
"""A 403 anywhere (POST or poll) is treated as not-admin: the
button hides — defense in depth behind header.js's whoami reveal."""
js = _text(SOURCES_JS)
for occurrence in re.finditer(r"r\.status === 403", js):
window = js[occurrence.start() : occurrence.start() + 400]
assert "syncBtn.hidden = true" in window, "every 403 branch must hide the button"
assert len(re.findall(r"r\.status === 403", js)) >= 3, (
"POST, the status poll, and the load re-attach must all handle 403"
)
def test_sources_js_running_state_is_never_stale() -> None:
"""Entering the running state disables the button, sets aria-busy,
spins the icon, and swaps the label to 'Syncing…' (the §7.4
feedback while the poll waits)."""
js = _text(SOURCES_JS)
fn = js.find("function enterRunningState")
assert fn != -1
body = js[fn : js.find("\n}", fn)]
assert "syncBtn.disabled = true" in body
assert 'syncBtn.setAttribute("aria-busy", "true")' in body
assert "syncIcon.classList.add(\"is-spinning\")" in body
assert '"Syncing…"' in body
def test_sources_js_terminal_states() -> None:
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
time of finished_at) + the last-result counts ('added' always
announced, zero terms omitted — a no-op re-sync reads '0 added ·
1 unchanged', never an empty live region) + a live catalog refresh
(the KB just changed — never a stale table); failed → enabled +
retry-ready 'Sync sources' label + the role='alert' banner with
the error; the result is cleared on a failure."""
js = _text(SOURCES_JS)
success = _body(js, "applySyncSuccess")
assert '"Synced"' in success and "fmtSyncTime(status.finished_at)" in success
assert "fmtSyncResult(status.detail)" in success
# A successful sync just changed the KB: the catalog re-fetches live
# (table / stats / empty state never sit stale under "Synced").
assert "loadDocs()" in success
failure = _body(js, "applySyncFailure")
assert 'settleSyncButton("Sync sources")' in failure # retry-ready
assert "showSyncError(status.error)" in failure
result = _body(js, "fmtSyncResult")
# "added" is the always-announced headline term; "unchanged" covers
# the no-op case ("0 added · 1 unchanged"); updated/pruned are
# zero-omitted.
assert "added" in result and "unchanged" in result
assert " · " in result
assert "> 0" in result, "zero terms must be omitted"
time = _body(js, "fmtSyncTime")
assert "getHours()" in time and "getMinutes()" in time, "local HH:MM of finished_at"
def test_sources_js_settles_the_button_on_terminal() -> None:
"""settleSyncButton re-enables the control, drops aria-busy, and
un-spins the icon — the button can never sit disabled after a run
reaches a terminal state (failed included: retry-ready)."""
js = _text(SOURCES_JS)
fn = js.find("function settleSyncButton")
assert fn != -1
body = js[fn : js.find("\n}", fn)]
assert "syncBtn.disabled = false" in body
assert 'syncBtn.removeAttribute("aria-busy")' in body
assert "syncIcon.classList.remove(\"is-spinning\")" in body
def test_sources_js_never_starts_a_second_poll_loop() -> None:
"""startSyncPolling is guarded by the module-level timer: a 409
adoption, a reload re-attach, or a stray call can never run two
poll loops at once (phase completion criterion)."""
js = _text(SOURCES_JS)
fn = js.find("function startSyncPolling")
assert fn != -1
head = js[fn : js.find("const tick", fn)]
assert re.search(r"if\s*\(\s*syncPollTimer\s*!==\s*null\s*\)\s*return", head), (
"the single-loop guard must be the first statement"
)
assert "clearTimeout(syncPollTimer)" in _body(js, "stopSyncPolling")
def test_sources_js_has_no_client_side_hard_timeout() -> None:
"""Phase locked decision: a sync can legitimately run for minutes,
so there is NO client-side hard timeout — the 2 s poll is the
feedback loop and the server state is authoritative (the 120 s
LLM-turn guard must not leak into the sync path)."""
js = _text(SOURCES_JS)
assert "TURN_TIMEOUT" not in js
assert "120" not in js[js.find("Phase 32") :], (
"no turn-timeout constant in the sync section"
)
def test_sources_js_reattaches_on_load() -> None:
"""initSyncButton (run from the IIFE on the admin path, after
initSharedHeader) fetches the status once and re-enters the running
state on 'running' (reload mid-sync) or renders the last result on
a terminal state; the click binding wires startSync to the button."""
js = _text(SOURCES_JS)
fn = js.find("function initSyncButton")
assert fn != -1
body = js[fn : js.find("\n}\n", fn)]
assert 'fetch("/api/sync/status")' in body
assert 'status.state === "running"' in body
assert 'status.state === "success"' in body
assert 'status.state === "failed"' in body
assert "syncBtn.addEventListener(\"click\", startSync)" in js
# The IIFE runs it on the admin path only (after the whoami gate).
iife = js[js.find("(async () => {") :]
admin_idx = iife.find("await isAdmin()")
init_idx = iife.find("initSyncButton();")
assert -1 < admin_idx < init_idx, "re-attach must run only for the admin"
# ---------- styles.css: the §7.4 states ----------
def test_sync_button_css_ghost_pill_and_disabled_state() -> None:
""".sync-btn is the same ghost pill as .new-chat-btn (contrast pair
ink-soft on surface ≈6.9:1 ≥ 4.5:1), with a ≥44px touch floor and a
:disabled state (never stale — the busy look is visible)."""
css = _text(STYLES_CSS)
block = re.search(r"\.sync-btn\s*\{([^}]*)\}", css)
assert block, "styles.css must define .sync-btn"
body = block.group(1)
assert "min-height: 44px" in body
assert "color: var(--ink-soft)" in body
assert "border-radius: 999px" in body
disabled = re.search(r"\.sync-btn:disabled\s*\{([^}]*)\}", css)
assert disabled, ".sync-btn:disabled must be styled"
assert "cursor: wait" in disabled.group(1)
def test_sync_icon_spins_and_respects_reduced_motion() -> None:
"""The running state spins the refresh icon on the shared spin
keyframes (1s linear infinite), and prefers-reduced-motion stills
it — the existing opt-out pattern."""
css = _text(STYLES_CSS)
spin = re.search(r"\.sync-btn \.sync-icon\.is-spinning\s*\{([^}]*)\}", css)
assert spin, "the .is-spinning state must be styled"
assert "animation: spin 1s linear infinite" in spin.group(1)
spinner = r"\.sync-btn \.sync-icon\.is-spinning\s*\{([^}]*)\}"
reduced = re.search(r"@media \(prefers-reduced-motion: reduce\)\s*\{\s*" + spinner, css)
assert reduced, "the spin must opt out under prefers-reduced-motion"
assert "animation: none" in reduced.group(1)
assert "@keyframes spin" in css, "the spin keyframes are shared (pre-existing)"
def test_sync_result_is_styled() -> None:
"""#sync-result (the aria-live last-result line) is styled in the
theme tokens — soft ink, small mono, no wrap in the header bar."""
css = _text(STYLES_CSS)
block = re.search(r"\.sync-result\s*\{([^}]*)\}", css)
assert block, "styles.css must define .sync-result"
assert "var(--ink-soft)" in block.group(1)