feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place

Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
2026-08-28 15:57:59 -04:00
parent 872a07cee7
commit 03d26255c6
21 changed files with 3280 additions and 233 deletions
@@ -0,0 +1,75 @@
# Story: Archive upload sources
**Phase:** `49_archive_upload_sources` · **E2E:** `tests/e2e/test_archive_upload_sources.py`
## Narrative
As **the admin (owner)**, the "Add a local directory" form makes me type
server paths — but the directories I want to index often live on *another
machine*. I want to **upload a tarball or zipfile** on the Sources
(`/git-sources.html`) page: the server **unpacks it and scans it** (the
content is indexed, visible in the RAG catalog immediately after the
upload settles). Re-uploading the **same filename** must **replace that
source in place** — same folder, same list row, previous content
overwritten — never a second folder or a duplicate row.
- **Given** I am signed in as admin, on the Sources page, and I have
`homelab.tar.gz` (containing `k3s.md`, `gitlab.md`)
- **When** I upload it
- **Then** it is unpacked to a server folder named after the archive
(`homelab`), imported (added/updated/pruned counts shown), appears in
the source list with the Local badge, and its documents show up in the
RAG catalog (`/sources.html`).
- **Given** I later edit the tarball (drop `gitlab.md`, add `caddy.md`)
and re-upload **`homelab.tar.gz`** (same name)
- **When** the upload settles
- **Then** there is still exactly one `homelab` folder and one list row;
`gitlab.md` is pruned from the index, `caddy.md` is indexed, `k3s.md`
is unchanged.
## Acceptance criteria
1. `POST /api/git-sources/upload` (multipart `file`, admin-only —
anonymous 403 like the rest of the router): accepts
`.tar`, `.tar.gz`, `.tgz`, `.zip` (else 422 naming the accepted set);
derives the source name from the filename minus the archive suffix
(`homelab.tar.gz` → `homelab`); rejects empty/unsafe names (422);
caps compressed upload AND extracted bytes at `BOR_UPLOAD_MAX_MB`
(default 512 — new settings `BOR_UPLOAD_DIR`, default
`~/bor-sources/uploads`, and `BOR_UPLOAD_MAX_MB`); blocks zip-slip /
tar-slip (absolute members, `..`, symlink/hardlink escapes, device
files) with a 422 and **no partial state** — a failed upload never
touches an existing folder, row, or the KB.
2. Successful upload: unpack → atomic swap-in of the folder → the
`git_sources` row is upserted by path (`kind=local`, no new row when
the path exists) → fail-fast model check (phase 41) →
`import_sources([folder], prune=True)` (single source) →
`regenerate_overview` when the KB changed → **200 with the same count
keys as the sync detail** (`files, added, updated, unchanged, pruned,
errors, chunks, overview`); one upload at a time (409 while a run is
in flight — the phase-32 pattern); a per-upload log line (PLAN §9).
3. Re-upload of the same filename replaces the folder's content in place
(temp unpack + rename swap — no missing window) and prunes files that
left the archive; no duplicate folder, no duplicate row.
4. The page: the "Add a local directory" form is **removed**; a labeled
archive upload form (file input `accept=".tar,.tar.gz,.tgz,.zip"`,
inline error `role=alert`, result line `role=status`, never-stale
button per §7.4 showing the counts) takes its place; the hint and
table caption mention upload+scan and in-place replace; existing
Local rows (incl. pre-existing hand-added dirs) still list/remove.
The `POST /api/git-sources` `kind=local` API contract is unchanged
(the capability survives via the API — no regression).
5. The phase-38 story E2E (`test_local_directory_sources.py`) is
rewritten to add local sources via the API (`page.request.post`)
instead of the removed form — its other assertions (Local badge,
sync-import, prune-remove, 422 naming the path) stand.
6. Unit + integration green, `app/` coverage >90%, story E2E green in
isolation, ruff + pyright clean, one `--no-gpg-sign` commit.
## Playwright Mapping Rule
`tests/e2e/test_archive_upload_sources.py` — one story, one file, run in
isolation: the admin uploads a real (test-built) tarball through the
page's file input → counts shown + Local-badged row named after the
archive stem + documents visible via the catalog; re-upload of the same
filename (modified archive) → still one row/folder, dropped file pruned,
new file indexed; a non-archive file gets an inline error; the
local-directory form is absent; anonymous still gets the gate.