Files
brain-of-reese/frontend/assets/git-sources.js
T
ducoterra 03d26255c6 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
2026-08-28 15:57:59 -04:00

478 lines
20 KiB
JavaScript

/* Brain of Reese — Git sources admin page (phase 35, task 04;
* archive uploads, phase 49 task 03).
*
* The page module for /git-sources.html: the admin-only manager for the
* stored source list (git-sources table, phase 35 tasks 01/02) — git
* repo URLs (kind "git") and uploaded archives unpacked under
* BOR_UPLOAD_DIR (kind "local", phase 49; the phase-38
* local-directory form is gone — the kind=local API POST is
* unchanged, the page just no longer offers it).
* This module is the single owner of the page's behaviour:
*
* • boot — initSharedHeader() (one cached whoami, shared with the
* header toggling): anonymous → the sign-in gate shows and the
* manager stays hidden (the exact Sources page gate pattern, and
* NO /api/git-sources call is made); admin → gate hidden,
* #git-sources-content revealed, loadSources().
* • loadSources() — GET /api/git-sources → the table rows
* (#git-sources-tbody), the env-fallback note's visibility
* (from_env), and the empty state. Each row leads with its kind
* badge (Git/Local — text + color, never color alone) plus the
* location in a mono <code>: the git URL, or the full local path
* for kind "local" rows (phase 38). Values are ALWAYS rendered
* with textContent — never innerHTML (URLs may embed user:pass@
* credentials; phase 32's masking discipline). Non-2xx or a
* network failure renders the role="alert" load error with a
* retry button — never a stuck page.
* • add — #git-source-form submit → POST /api/git-sources {url}.
* The §7.4 never-stale lifecycle (wireAddForm): the button
* disables + relabels "Adding…" while the request is out,
* re-enables on success AND failure. 201 clears the input,
* reloads the list, and focuses the new row's Remove button
* (a11y); a failure (409 duplicate, 422 validation) shows the
* server detail inline under the form (role="alert", 422
* shape-aware like the tuning forms) and keeps the input — the
* instruction survives. 409/422 details are fixed generic strings
* (credential safety — the URL is never echoed).
* • upload — #archive-upload-form submit (phase 49) → POST
* /api/git-sources/upload with a FormData file (NO manual
* Content-Type — the browser sets the multipart boundary). The
* SAME §7.4 never-stale lifecycle: the button disables +
* relabels "Uploading…" while the request is out and is restored
* on success AND failure. 200 clears the file input, shows the
* sync-style count line ("2 added · 1 pruned" — fmtUploadResult,
* sources.js's fmtSyncResult convention) in the role=status
* result line, announces "Archive uploaded: …" and reloads the
* list (the new/updated row lands with the Local badge; a
* re-upload simply refreshes the row — no duplicate). Non-2xx
* inlines the server detail (422 format/name/traversal, 413
* size, 409 busy — the messages are already user-safe) and KEEPS
* the file selection — the fix is one re-pick, not a re-type.
* • remove — a row's Remove button asks window.confirm first
* (removal prunes the documents only on the NEXT sync — the
* confirm says so). Cancel → nothing; ok → the row button
* disables, DELETE /api/git-sources/{id}, loadSources(). A
* failure shows a per-row role="alert" error and re-enables the
* button. Env-fallback rows (id null — the list comes from
* BOR_GIT_SOURCES, not the table) carry no Remove: nothing is
* stored to remove — they show a "from .env" tag instead.
* • announce(msg) — #git-sources-announcer (role=status,
* aria-live=polite): the screen-reader confirmation for loads,
* adds, and removals.
*
* Scope boundary (phase locked decisions): adding a git repo or
* removing a source does NOT clone, import, or prune — the sync
* service (server-side) performs that; the page's hint box says so.
* The phase-49 upload is the exception: it unpacks and scans the
* single source in place, and its response counts render as the
* result line.
*
* The shared header module loads through this script's own relative
* import ("./header.js") — a hoisted import evaluated before this body
* runs (single-evaluation design: no direct <script> tag; esbuild
* inlines it into the page bundle in the image build).
*/
import { initSharedHeader } from "./header.js";
/* ---------- page elements (git-sources.html, task 04) ---------- */
const gateEl = document.querySelector("#git-sources-gate");
const contentEl = document.querySelector("#git-sources-content");
const formEl = document.querySelector("#git-source-form");
const urlInput = document.querySelector("#git-source-url");
const addBtn = document.querySelector("#git-source-add");
const addError = document.querySelector("#git-source-error");
/* Phase 49: the archive upload form (replaces the phase-38 local
directory form — same card, a file input instead of a path input).
The response counts render in the role=status result line. */
const uploadFormEl = document.querySelector("#archive-upload-form");
const uploadFileInput = document.querySelector("#archive-upload-file");
const uploadBtn = document.querySelector("#archive-upload-btn");
const uploadError = document.querySelector("#archive-upload-error");
const uploadResult = document.querySelector("#archive-upload-result");
const loadErrorEl = document.querySelector("#git-sources-load-error");
const loadErrorText = document.querySelector("#git-sources-load-error-text");
const retryBtn = document.querySelector("#git-sources-retry");
const tableWrap = document.querySelector("#git-sources-table-wrap");
const tbody = document.querySelector("#git-sources-tbody");
const emptyEl = document.querySelector("#git-sources-empty");
const envNote = document.querySelector("#git-sources-env-note");
const announcer = document.querySelector("#git-sources-announcer");
/* Polite live region: the screen-reader confirmation for loads, adds,
and removals (the phase-15 announcer pattern). */
function announce(message) {
if (announcer) announcer.textContent = message;
}
/* Added date — localized (toLocaleString); env-fallback rows carry
added_at null, and a corrupt timestamp must not blank the cell. */
function fmtDate(iso) {
if (!iso) return "—";
try {
return new Date(iso).toLocaleString();
} catch {
return "—";
}
}
/* FastAPI error bodies: a string detail or the validation-error array
(the first entry's msg is the human line). Same extraction as
tuning.js — 422 shape-aware. */
async function apiDetail(r, fallback) {
try {
const data = await r.json();
if (Array.isArray(data.detail) && data.detail[0] && data.detail[0].msg) {
return String(data.detail[0].msg);
}
if (typeof data.detail === "string" && data.detail) return data.detail;
} catch {
/* non-JSON error body */
}
return fallback;
}
/* ---------- load / render ---------- */
/* GET /api/git-sources → the table rows + env note + empty state.
The load error (role="alert" + retry) is the ONLY terminal state a
failed fetch may reach — never a stuck page. */
async function loadSources() {
let r;
try {
r = await fetch("/api/git-sources");
} catch {
showLoadError("Could not reach the server — is the app running?");
return;
}
if (!r.ok) {
showLoadError(await apiDetail(r, `The server could not list the git sources (${r.status}).`));
return;
}
let data;
try {
data = await r.json();
} catch {
showLoadError("The server sent an unreadable list — try again.");
return;
}
hideLoadError();
const sources = Array.isArray(data.sources) ? data.sources : [];
renderSources(sources, data.from_env === true);
announce(`${sources.length} source${sources.length === 1 ? "" : "s"} listed.`);
}
function showLoadError(message) {
if (loadErrorText) loadErrorText.textContent = message;
if (loadErrorEl) loadErrorEl.hidden = false;
// The list state is unknown — hide the table AND the empty state so
// the error is the only claim about the list's contents.
if (tableWrap) tableWrap.hidden = true;
if (emptyEl) emptyEl.hidden = true;
}
function hideLoadError() {
if (loadErrorEl) loadErrorEl.hidden = true;
if (loadErrorText) loadErrorText.textContent = "";
}
function renderSources(sources, fromEnv) {
if (envNote) envNote.hidden = !fromEnv;
if (tbody) {
tbody.replaceChildren();
for (const s of sources) tbody.appendChild(makeRow(s));
}
const hasRows = sources.length > 0;
if (tableWrap) tableWrap.hidden = !hasRows;
if (emptyEl) emptyEl.hidden = hasRows;
}
/* One row: the kind badge (Git/Local — phase 38) followed by the
location in a mono <code> (textContent only — git URLs may contain
credentials, local paths may contain anything), the localized added
date ("—" for env fallback rows), and the per-row Remove button —
or the "from .env" tag for env-fallback rows (id null: nothing is
stored to remove; the env note says where the active list comes
from). The value is the git URL for kind "git" rows and the full
local path for kind "local" rows (the API reports the path in both
`path` and `url`; `path` is the kind-typed field). */
const REMOVE_ICON =
'<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round"><path d="M5 7h14M10 7V5h4v2M8.5 7l.7 12h5.6l.7-12"/></svg>';
function makeRow(s) {
const tr = document.createElement("tr");
if (s.id) tr.dataset.id = s.id;
const isLocal = s.kind === "local";
const value = isLocal ? (s.path ?? s.url) : s.url;
const kindLabel = isLocal ? "local" : "git";
const urlTd = document.createElement("td");
urlTd.className = "git-source-url-cell";
urlTd.title = value; // full URL/path on hover (long values scroll the wrapper)
const badge = document.createElement("span");
badge.className = `git-source-kind is-${isLocal ? "local" : "git"}`;
badge.textContent = isLocal ? "Local" : "Git"; // text + color, never color alone
const code = document.createElement("code");
code.textContent = value; // rendered as text, never as HTML
urlTd.append(badge, code);
tr.appendChild(urlTd);
const addedTd = document.createElement("td");
addedTd.textContent = fmtDate(s.added_at);
tr.appendChild(addedTd);
const actTd = document.createElement("td");
actTd.className = "git-source-actions-cell";
if (s.id) {
const btn = document.createElement("button");
btn.type = "button";
btn.className = "git-source-remove";
btn.setAttribute("aria-label", `Remove ${kindLabel} source: ${value}`);
btn.innerHTML = REMOVE_ICON + "<span>Remove</span>";
const rowError = document.createElement("span");
rowError.className = "git-source-row-error";
rowError.setAttribute("role", "alert");
rowError.hidden = true;
btn.addEventListener("click", () => removeSource(s, btn, rowError, kindLabel));
actTd.append(btn, rowError);
} else {
const tag = document.createElement("span");
tag.className = "git-source-env-tag";
tag.textContent = "from .env";
actTd.appendChild(tag);
}
tr.appendChild(actTd);
return tr;
}
/* ---------- remove (DELETE /api/git-sources/{id}) ----------
* Removal does not prune anything immediately — the next sync does
* (phase scope boundary), so the confirm says exactly that. Cancel →
* nothing; a failed delete → per-row role="alert" error + re-enabled
* button (never a stuck row); success → the list reloads. */
async function removeSource(s, btn, rowError, kindLabel) {
const ok = window.confirm(
`Remove this ${kindLabel} source from the list? Its documents stay indexed until the next sync prunes them.`,
);
if (!ok) return;
btn.disabled = true; // one delete per click
rowError.hidden = true;
try {
const r = await fetch(`/api/git-sources/${encodeURIComponent(s.id)}`, { method: "DELETE" });
if (!r.ok) {
rowError.textContent = await apiDetail(r, `Could not remove the ${kindLabel} source — try again.`);
rowError.hidden = false;
btn.disabled = false;
return;
}
announce("Source removed.");
await loadSources(); // 204: the server confirmed — the list re-renders
} catch {
rowError.textContent = `Could not remove the ${kindLabel} source — is the app reachable?`;
rowError.hidden = false;
btn.disabled = false;
}
}
/* ---------- add (POST /api/git-sources) — the git form ----------
* wireAddForm gives the form the §7.4 never-stale lifecycle: while
* the request is out the button disables + relabels "Adding…" and
* re-enables (idle label restored) on success AND failure. 201 clears
* the input, reloads the list, and focuses the new row's Remove button
* (a11y); a failure (409 duplicate, 422 validation) shows the server
* detail inline under the form (role="alert", 422 shape-aware via
* apiDetail) and keeps the input — the fix is one edit, not a re-type.
* 409/422 details are fixed generic strings (credential safety — the
* URL is never echoed). */
function wireAddForm(opts) {
const { form, input, btn, error } = opts;
if (!form || !input || !btn) return;
form.addEventListener("submit", async (e) => {
e.preventDefault();
// Client-side non-empty check (the input is `required` too — the
// browser's native prompt is the first line, this one the second).
const value = input.value.trim();
if (!value) {
if (error) {
error.textContent = opts.emptyMessage;
error.hidden = false;
}
return;
}
if (error) error.hidden = true; // a new attempt starts clean
btn.disabled = true; // §7.4: one POST per click
btn.textContent = "Adding…";
try {
const r = await fetch("/api/git-sources", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(opts.body(value)),
});
if (r.ok) {
let createdId = null;
try {
createdId = (await r.json()).id ?? null;
} catch {
/* the 201 body is advisory — the reload is the truth */
}
input.value = ""; // 201: the source is stored
announce(opts.addedMessage);
await loadSources(); // the new row lands in the table
focusNewRow(createdId); // a11y: land the caret on the new row
return;
}
// 409 duplicate / 422 validation / anything else: the server
// detail inline, form kept — the input survives so the fix is
// one edit, not a re-type.
if (error) {
error.textContent = await apiDetail(r, opts.failMessage);
error.hidden = false;
}
} catch {
if (error) {
error.textContent = opts.networkMessage;
error.hidden = false;
}
} finally {
btn.disabled = false; // never stale — success OR failure
btn.textContent = opts.idleLabel;
}
});
}
wireAddForm({
form: formEl,
input: urlInput,
btn: addBtn,
error: addError,
body: (url) => ({ url }),
emptyMessage: "Enter a git URL to add.",
failMessage: "Could not add the git source — try again.",
networkMessage: "Could not add the git source — is the app reachable?",
addedMessage: "Git source added.",
idleLabel: "Add source",
});
/* ---------- upload (POST /api/git-sources/upload) — phase 49 -------
* The archive upload form: the file input's selection is posted as
* FormData (the browser sets the multipart boundary — no manual
* Content-Type). §7.4 never-stale: "Uploading…" while in flight,
* restored in the finally block on success AND failure. 200 → the
* input clears, the sync-style counts land in the role=status result
* line, the announcer confirms, and loadSources() re-renders the row
* (Local badge; a re-upload refreshes the existing row — no
* duplicate). Non-2xx → the server detail inline (role=alert; 422
* format/name/traversal, 413 size, 409 busy — user-safe as-is) with
* the file selection KEPT; network failure → the fixed line. */
/* The success line's text — the sync-result shape (sources.js's
fmtSyncResult convention): "N added" always leads, then updated /
unchanged / pruned — zero parts omitted (unchanged is shown
when nothing was added or updated). */
function fmtUploadResult(detail) {
const d = detail || {};
const added = d.added || 0;
const updated = d.updated || 0;
const parts = [`${added} added`];
if (updated > 0) parts.push(`${updated} updated`);
if ((d.unchanged || 0) > 0 || (added === 0 && updated === 0)) {
parts.push(`${d.unchanged || 0} unchanged`);
}
if ((d.pruned || 0) > 0) parts.push(`${d.pruned} pruned`);
return parts.join(" · ");
}
if (uploadFormEl && uploadFileInput && uploadBtn) {
uploadFormEl.addEventListener("submit", async (e) => {
e.preventDefault();
// Client-side no-file check (the input is `required` too — the
// browser's native prompt is the first line, this one the second).
const file = uploadFileInput.files && uploadFileInput.files[0];
if (!file) {
if (uploadError) {
uploadError.textContent = "Choose an archive file to upload.";
uploadError.hidden = false;
}
return;
}
if (uploadError) uploadError.hidden = true;
if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean
uploadBtn.disabled = true; // §7.4: one upload per click
uploadBtn.textContent = "Uploading…";
try {
// Multipart from the form itself (the file input's name is
// "file") — the browser sets the boundary; NO manual
// Content-Type header.
const r = await fetch("/api/git-sources/upload", {
method: "POST",
body: new FormData(uploadFormEl),
});
if (r.ok) {
let data = {};
try {
data = await r.json();
} catch {
/* the body is advisory — the counts line degrades gracefully */
}
uploadFileInput.value = ""; // 200: the archive is unpacked + scanned
if (uploadResult) {
uploadResult.textContent = fmtUploadResult(data);
uploadResult.hidden = false;
}
announce(`Archive uploaded: ${data.source || file.name}.`);
await loadSources(); // the new/updated row lands (Local badge)
return;
}
// 422 (format/name/traversal), 413 (size), 409 (busy): the server
// detail inline, the file selection KEPT — the fix is one
// re-pick, not a re-type.
if (uploadError) {
uploadError.textContent = await apiDetail(r, "Could not upload the archive — try again.");
uploadError.hidden = false;
}
} catch {
if (uploadError) {
uploadError.textContent = "Could not upload the archive — is the app reachable?";
uploadError.hidden = false;
}
} finally {
uploadBtn.disabled = false; // never stale — success OR failure
uploadBtn.textContent = "Upload & scan";
}
});
}
/* After a successful add, focus the new row's Remove button so the
keyboard/screen-reader user lands where the new data is. The 201
body carries the row id (tr[data-id]); without one, the first row
is the fallback (the list is small and ordered). */
function focusNewRow(createdId) {
if (!tbody) return;
const row = createdId
? tbody.querySelector(`tr[data-id="${CSS.escape(createdId)}"]`)
: null;
const target = (row || tbody.querySelector("tr"))?.querySelector(".git-source-remove");
if (target) target.focus();
}
/* ---------- retry + boot ---------- */
if (retryBtn) retryBtn.addEventListener("click", () => loadSources());
(async () => {
// The shared header FIRST (Sign in/out + the admin-only nav links +
// the Sync button — one cached whoami), then the gate: anonymous
// visitors get the gate and NO /api/git-sources call (the Sources
// page gate pattern); the admin gets the manager.
const admin = await initSharedHeader();
if (!admin) {
if (gateEl) gateEl.hidden = false;
if (contentEl) contentEl.hidden = true; // ships hidden — stays hidden
return;
}
if (gateEl) gateEl.hidden = true;
if (contentEl) contentEl.hidden = false;
await loadSources();
})();