feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
+136
-42
@@ -1,10 +1,12 @@
|
||||
/* Brain of Reese — Git sources admin page (phase 35, task 04;
|
||||
* local directories, phase 38 task 04).
|
||||
* archive uploads, phase 49 task 03).
|
||||
*
|
||||
* The page module for /git-sources.html: the admin-only manager for the
|
||||
* stored source list (git-sources table, phase 35 tasks 01/02) — git
|
||||
* repo URLs (kind "git") and existing local directories (kind
|
||||
* "local", phase 38).
|
||||
* repo URLs (kind "git") and uploaded archives unpacked under
|
||||
* BOR_UPLOAD_DIR (kind "local", phase 49; the phase-38
|
||||
* local-directory form is gone — the kind=local API POST is
|
||||
* unchanged, the page just no longer offers it).
|
||||
* This module is the single owner of the page's behaviour:
|
||||
*
|
||||
* • boot — initSharedHeader() (one cached whoami, shared with the
|
||||
@@ -22,17 +24,30 @@
|
||||
* credentials; phase 32's masking discipline). Non-2xx or a
|
||||
* network failure renders the role="alert" load error with a
|
||||
* retry button — never a stuck page.
|
||||
* • add — #git-source-form submit → POST /api/git-sources {url};
|
||||
* #local-source-form submit → POST /api/git-sources
|
||||
* {kind: "local", path} (phase 38). ONE §7.4 never-stale
|
||||
* lifecycle for both (wireAddForm): the button disables +
|
||||
* relabels "Adding…" while the request is out, re-enables on
|
||||
* success AND failure. 201 clears the input, reloads the list,
|
||||
* and focuses the new row's Remove button (a11y); a failure (409
|
||||
* duplicate, 422 validation) shows the server detail inline under
|
||||
* the form (role="alert", 422 shape-aware like the tuning forms)
|
||||
* and keeps the input — the instruction survives. Local 422/409
|
||||
* details name the path (paths are not secrets, unlike URLs).
|
||||
* • add — #git-source-form submit → POST /api/git-sources {url}.
|
||||
* The §7.4 never-stale lifecycle (wireAddForm): the button
|
||||
* disables + relabels "Adding…" while the request is out,
|
||||
* re-enables on success AND failure. 201 clears the input,
|
||||
* reloads the list, and focuses the new row's Remove button
|
||||
* (a11y); a failure (409 duplicate, 422 validation) shows the
|
||||
* server detail inline under the form (role="alert", 422
|
||||
* shape-aware like the tuning forms) and keeps the input — the
|
||||
* instruction survives. 409/422 details are fixed generic strings
|
||||
* (credential safety — the URL is never echoed).
|
||||
* • upload — #archive-upload-form submit (phase 49) → POST
|
||||
* /api/git-sources/upload with a FormData file (NO manual
|
||||
* Content-Type — the browser sets the multipart boundary). The
|
||||
* SAME §7.4 never-stale lifecycle: the button disables +
|
||||
* relabels "Uploading…" while the request is out and is restored
|
||||
* on success AND failure. 200 clears the file input, shows the
|
||||
* sync-style count line ("2 added · 1 pruned" — fmtUploadResult,
|
||||
* sources.js's fmtSyncResult convention) in the role=status
|
||||
* result line, announces "Archive uploaded: …" and reloads the
|
||||
* list (the new/updated row lands with the Local badge; a
|
||||
* re-upload simply refreshes the row — no duplicate). Non-2xx
|
||||
* inlines the server detail (422 format/name/traversal, 413
|
||||
* size, 409 busy — the messages are already user-safe) and KEEPS
|
||||
* the file selection — the fix is one re-pick, not a re-type.
|
||||
* • remove — a row's Remove button asks window.confirm first
|
||||
* (removal prunes the documents only on the NEXT sync — the
|
||||
* confirm says so). Cancel → nothing; ok → the row button
|
||||
@@ -45,9 +60,12 @@
|
||||
* aria-live=polite): the screen-reader confirmation for loads,
|
||||
* adds, and removals.
|
||||
*
|
||||
* Scope boundary (phase locked decisions): adding or removing a repo
|
||||
* does NOT clone, import, or prune — the sync service (server-side)
|
||||
* performs that; the page's hint box says so.
|
||||
* Scope boundary (phase locked decisions): adding a git repo or
|
||||
* removing a source does NOT clone, import, or prune — the sync
|
||||
* service (server-side) performs that; the page's hint box says so.
|
||||
* The phase-49 upload is the exception: it unpacks and scans the
|
||||
* single source in place, and its response counts render as the
|
||||
* result line.
|
||||
*
|
||||
* The shared header module loads through this script's own relative
|
||||
* import ("./header.js") — a hoisted import evaluated before this body
|
||||
@@ -64,12 +82,14 @@ const formEl = document.querySelector("#git-source-form");
|
||||
const urlInput = document.querySelector("#git-source-url");
|
||||
const addBtn = document.querySelector("#git-source-add");
|
||||
const addError = document.querySelector("#git-source-error");
|
||||
/* Phase 38: the second add form — "Local directory" (same element
|
||||
contract as the git form, own ids). */
|
||||
const localFormEl = document.querySelector("#local-source-form");
|
||||
const pathInput = document.querySelector("#local-source-path");
|
||||
const localAddBtn = document.querySelector("#local-source-add");
|
||||
const localAddError = document.querySelector("#local-source-error");
|
||||
/* Phase 49: the archive upload form (replaces the phase-38 local
|
||||
directory form — same card, a file input instead of a path input).
|
||||
The response counts render in the role=status result line. */
|
||||
const uploadFormEl = document.querySelector("#archive-upload-form");
|
||||
const uploadFileInput = document.querySelector("#archive-upload-file");
|
||||
const uploadBtn = document.querySelector("#archive-upload-btn");
|
||||
const uploadError = document.querySelector("#archive-upload-error");
|
||||
const uploadResult = document.querySelector("#archive-upload-result");
|
||||
const loadErrorEl = document.querySelector("#git-sources-load-error");
|
||||
const loadErrorText = document.querySelector("#git-sources-load-error-text");
|
||||
const retryBtn = document.querySelector("#git-sources-retry");
|
||||
@@ -255,18 +275,16 @@ async function removeSource(s, btn, rowError, kindLabel) {
|
||||
}
|
||||
}
|
||||
|
||||
/* ---------- add (POST /api/git-sources) — both forms, one lifecycle
|
||||
* (the local form is phase 38) ----------
|
||||
* The git form posts {url}; the local form posts {kind:"local",path}.
|
||||
* wireAddForm gives both the §7.4 never-stale lifecycle: while the
|
||||
* request is out the button disables + relabels "Adding…" and
|
||||
/* ---------- add (POST /api/git-sources) — the git form ----------
|
||||
* wireAddForm gives the form the §7.4 never-stale lifecycle: while
|
||||
* the request is out the button disables + relabels "Adding…" and
|
||||
* re-enables (idle label restored) on success AND failure. 201 clears
|
||||
* the input, reloads the list, and focuses the new row's Remove button
|
||||
* (a11y); a failure (409 duplicate, 422 validation) shows the server
|
||||
* detail inline under the form (role="alert", 422 shape-aware via
|
||||
* apiDetail) and keeps the input — the fix is one edit, not a re-type.
|
||||
* Git 409/422 details are fixed generic strings (credential safety);
|
||||
* local details name the path (not a secret). */
|
||||
* 409/422 details are fixed generic strings (credential safety — the
|
||||
* URL is never echoed). */
|
||||
function wireAddForm(opts) {
|
||||
const { form, input, btn, error } = opts;
|
||||
if (!form || !input || !btn) return;
|
||||
@@ -336,18 +354,94 @@ wireAddForm({
|
||||
idleLabel: "Add source",
|
||||
});
|
||||
|
||||
wireAddForm({
|
||||
form: localFormEl,
|
||||
input: pathInput,
|
||||
btn: localAddBtn,
|
||||
error: localAddError,
|
||||
body: (path) => ({ kind: "local", path }),
|
||||
emptyMessage: "Enter a directory path to add.",
|
||||
failMessage: "Could not add the local directory — try again.",
|
||||
networkMessage: "Could not add the local directory — is the app reachable?",
|
||||
addedMessage: "Local source added.",
|
||||
idleLabel: "Add directory",
|
||||
});
|
||||
/* ---------- upload (POST /api/git-sources/upload) — phase 49 -------
|
||||
* The archive upload form: the file input's selection is posted as
|
||||
* FormData (the browser sets the multipart boundary — no manual
|
||||
* Content-Type). §7.4 never-stale: "Uploading…" while in flight,
|
||||
* restored in the finally block on success AND failure. 200 → the
|
||||
* input clears, the sync-style counts land in the role=status result
|
||||
* line, the announcer confirms, and loadSources() re-renders the row
|
||||
* (Local badge; a re-upload refreshes the existing row — no
|
||||
* duplicate). Non-2xx → the server detail inline (role=alert; 422
|
||||
* format/name/traversal, 413 size, 409 busy — user-safe as-is) with
|
||||
* the file selection KEPT; network failure → the fixed line. */
|
||||
|
||||
/* The success line's text — the sync-result shape (sources.js's
|
||||
fmtSyncResult convention): "N added" always leads, then updated /
|
||||
unchanged / pruned — zero parts omitted (unchanged is shown
|
||||
when nothing was added or updated). */
|
||||
function fmtUploadResult(detail) {
|
||||
const d = detail || {};
|
||||
const added = d.added || 0;
|
||||
const updated = d.updated || 0;
|
||||
const parts = [`${added} added`];
|
||||
if (updated > 0) parts.push(`${updated} updated`);
|
||||
if ((d.unchanged || 0) > 0 || (added === 0 && updated === 0)) {
|
||||
parts.push(`${d.unchanged || 0} unchanged`);
|
||||
}
|
||||
if ((d.pruned || 0) > 0) parts.push(`${d.pruned} pruned`);
|
||||
return parts.join(" · ");
|
||||
}
|
||||
|
||||
if (uploadFormEl && uploadFileInput && uploadBtn) {
|
||||
uploadFormEl.addEventListener("submit", async (e) => {
|
||||
e.preventDefault();
|
||||
// Client-side no-file check (the input is `required` too — the
|
||||
// browser's native prompt is the first line, this one the second).
|
||||
const file = uploadFileInput.files && uploadFileInput.files[0];
|
||||
if (!file) {
|
||||
if (uploadError) {
|
||||
uploadError.textContent = "Choose an archive file to upload.";
|
||||
uploadError.hidden = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (uploadError) uploadError.hidden = true;
|
||||
if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean
|
||||
uploadBtn.disabled = true; // §7.4: one upload per click
|
||||
uploadBtn.textContent = "Uploading…";
|
||||
try {
|
||||
// Multipart from the form itself (the file input's name is
|
||||
// "file") — the browser sets the boundary; NO manual
|
||||
// Content-Type header.
|
||||
const r = await fetch("/api/git-sources/upload", {
|
||||
method: "POST",
|
||||
body: new FormData(uploadFormEl),
|
||||
});
|
||||
if (r.ok) {
|
||||
let data = {};
|
||||
try {
|
||||
data = await r.json();
|
||||
} catch {
|
||||
/* the body is advisory — the counts line degrades gracefully */
|
||||
}
|
||||
uploadFileInput.value = ""; // 200: the archive is unpacked + scanned
|
||||
if (uploadResult) {
|
||||
uploadResult.textContent = fmtUploadResult(data);
|
||||
uploadResult.hidden = false;
|
||||
}
|
||||
announce(`Archive uploaded: ${data.source || file.name}.`);
|
||||
await loadSources(); // the new/updated row lands (Local badge)
|
||||
return;
|
||||
}
|
||||
// 422 (format/name/traversal), 413 (size), 409 (busy): the server
|
||||
// detail inline, the file selection KEPT — the fix is one
|
||||
// re-pick, not a re-type.
|
||||
if (uploadError) {
|
||||
uploadError.textContent = await apiDetail(r, "Could not upload the archive — try again.");
|
||||
uploadError.hidden = false;
|
||||
}
|
||||
} catch {
|
||||
if (uploadError) {
|
||||
uploadError.textContent = "Could not upload the archive — is the app reachable?";
|
||||
uploadError.hidden = false;
|
||||
}
|
||||
} finally {
|
||||
uploadBtn.disabled = false; // never stale — success OR failure
|
||||
uploadBtn.textContent = "Upload & scan";
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/* After a successful add, focus the new row's Remove button so the
|
||||
keyboard/screen-reader user lands where the new data is. The 201
|
||||
|
||||
Reference in New Issue
Block a user