feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place

Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.

.agent/phases/complete/49_archive_upload_sources/00_phase.md

@@ -0,0 +1,51 @@
+# Phase 49 — Archive upload sources (tarball/zipfile → unpack → scan)
+
+**Source:** owner request (chat, 2026-08-28) — "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan. Note that reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content." (Design confirmed by the owner in the same conversation.)
+**Story:** `.agent/user_stories/archive-upload-sources.md`
+**Context:** `35_git_sources_admin` (complete) — the `git_sources` table (`id, url, kind, path, added_at`), the admin-only `/api/git-sources` router, and the `/git-sources.html` manager page; `38_local_directory_sources` (complete) — `kind='local'` rows the Sync pipeline and `import_docs` walk directly, plus the page's "Add a local directory" form this phase removes; `32_admin_sync_button` + `41_sync_fail_fast_models` (complete) — the in-process pipeline parts this phase reuses: `check_models` fail-fast, `import_sources(sources, llm, prune=True)` (source name = folder basename, per-file transactions, per-source prune), `regenerate_overview`, and the sync-detail count keys (`files, added, updated, unchanged, pruned, errors, chunks, overview`); `16_admin_auth` (complete) — the `require_admin` router dependency the new route inherits.
+
+## Objective
+Replace the local-directory form on the Sources page with an **archive upload** form: `POST /api/git-sources/upload` accepts `.tar`/`.tar.gz`/`.tgz`/`.zip`, unpacks it **safely** into `BOR_UPLOAD_DIR/<name>/` (name = filename minus the archive suffix), atomically swaps it in when the name already exists, upserts the `git_sources` row (`kind=local`, no duplicates), and **scans it** — single-source `import_sources(prune=True)` + overview refresh — returning the sync-style counts. Re-uploading the same filename overwrites the previous content in place: one folder, one row, dropped files pruned from the KB.
+
+## Dependencies
+- `48_nav_rename_sources` (todo — runs first) — sequential only: it relabels the nav in the same `git-sources.html` this phase edits (keeps the diffs clean).
+- `35_git_sources_admin` (complete) — the table/API/page this phase extends; the `require_admin` router; the `IntegrityError → 409` backstop pattern (`_commit_new`).
+- `38_local_directory_sources` (complete) — the `kind=local` rows uploads register; the local form removed; the phase-38 story E2E rewritten in this phase.
+- `32_admin_sync_button` / `41_sync_fail_fast_models` (complete) — `check_models` + `import_sources` + `regenerate_overview` + the count-key contract the upload response mirrors.
+- `16_admin_auth` (complete) — admin-only surface (A10 revision).
+
+## Tasks
+1. `01_settings_and_unpack_utility.md` — `BOR_UPLOAD_DIR` + `BOR_UPLOAD_MAX_MB` settings, `.env.example`, and the new `app/rag/archive_upload.py` (name derivation, safe tar/zip unpack with traversal/symlink/size guards, atomic swap-in) + unit tests.
+2. `02_upload_api.md` — `python-multipart` dependency + `POST /api/git-sources/upload` (stream-with-cap, one-at-a-time 409, upsert row, fail-fast models, single-source scan, sync-style 200 body, log line) + integration tests incl. re-upload/overwrite and no-partial-state.
+3. `03_admin_page_upload.md` — the page: local form out, upload form in (§7.4 lifecycle, result line, hint/caption) + the phase-38 story E2E rewritten API-driven.
+4. `04_story_e2e_docs_commit.md` — the story E2E (`test_archive_upload_sources.py`), README, regression suites in isolation, the one `--no-gpg-sign` commit, and the phase-dir move.
+
+## Testing & Quality
+- Unit: `tests/unit/test_archive_upload.py` — the name-derivation matrix (suffix stripping incl. double `.tar.gz`; empty/`..`/separator/control-char rejection); safe unpack (valid zip + tar; zip-slip `../`; absolute member; symlink + hardlink escape; device member; extracted-byte cap); `swap_in` (fresh, replace-existing with full content replacement, failure leaves the previous folder intact).
+- Integration: `tests/integration/test_git_sources_upload.py` — anonymous 403 on the new route; 422 (bad extension, unsafe/empty name, traversal archive, corrupt archive); 413 (compressed cap, via the settings-override pattern of `test_git_sources_api.py`); 409 (second upload while the first is in flight); 200 happy path (real temp tarball, counts correct, row `kind=local` under `upload_dir`, docs in the KB); **re-upload same name** (one row, old folder content fully replaced, dropped file pruned, new file indexed); **failed re-upload leaves the previous folder + row + KB untouched**. The existing `test_git_sources_api.py` / `test_sync_api.py` / `test_import_docs_git.py` suites stay green through the change (the `kind=local` POST contract is untouched).
+- Coverage: **>90%** on `app/` — the new module + endpoint fully covered.
+- E2E (mandatory, A16): `tests/e2e/test_archive_upload_sources.py`, run in isolation.
+
+## Completion Criteria
+- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
+- [ ] Uploading `homelab.tar.gz` via the page: unpacked under `BOR_UPLOAD_DIR/homelab/`, scanned (result line shows the counts), one list row (Local badge, name `homelab`), documents visible on `/sources.html`.
+- [ ] Re-uploading `homelab.tar.gz` (modified): still exactly one folder and one row; dropped files pruned from the KB; added/changed files indexed.
+- [ ] Non-archive file → inline 422; oversized → 413; zip-slip/tar-slip archive → 422 with the previous folder/row/KB untouched; second concurrent upload → 409.
+- [ ] `#local-source-form` is gone from the page; the phase-38 story E2E green in isolation, API-driven; anonymous still gets the gate.
+- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
+- [ ] Regression E2E suites green in isolation: `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (when 48 is complete), `test_smoke.py`.
+- [ ] README + `.env.example` document the upload (formats, naming, in-place replace, both new settings); ruff + pyright clean.
+- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
+- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
+
+## Locked decisions (owner permission 2026-08-28 — the confirmed design)
+- **Formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` only (422 naming the accepted set otherwise).
+- **Identity & in-place replace:** the source name is the uploaded filename minus the archive suffix (`homelab.tar.gz` → `homelab`, case-sensitive — Linux FS). The name determines the folder under `BOR_UPLOAD_DIR`; re-uploading the same name unpacks to a temp sibling and **renames it over the existing folder** (no missing window; a failed upload never touches the existing folder, row, or KB). **No second folder, no second row** — the `git_sources` row is upserted by `path` (`kind='local'`, reusing the phase-38 discriminator — **no migration, no new table**; A13 honoured).
+- **New settings:** `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads` — deliberately separate from the git checkouts in `BOR_SOURCES_DIR`) and `BOR_UPLOAD_MAX_MB` (default **512**) capping BOTH the compressed upload and the total extracted bytes (zip-bomb guard).
+- **The scan is synchronous in the upload request** (owner-confirmed): fail-fast `check_models` (phase 41) → `import_sources([folder], llm, prune=True)` (single source) → `regenerate_overview` when the KB changed → **200** with the sync-detail count keys so the page renders the same "N added · N pruned" line. One upload at a time — 409 while a run is in flight (the phase-32 pattern).
+- **Unpack safety:** absolute member paths, `..` traversal, symlink/hardlink targets escaping the unpack folder, and device/FIFO members are rejected (422); extracted bytes are counted against the cap while writing.
+- **Page:** the "Add a local directory" form is **removed**; the `POST /api/git-sources` `kind=local` **API contract is unchanged** (admin can still register a plain directory via the API — no regression; existing Local rows still list/remove, and the Sync button + `import_docs` keep walking them).
+- **No auto-unwrap** of a single top-level folder — files land in the KB exactly as packed (documented in the hint/README).
+- **`python-multipart`** is added to the dependencies — FastAPI's required multipart parser for file uploads (an A2 FastAPI implementation detail, not a new architectural anchor; recorded here per AGENTS.md rule 3).
+- **Boundaries (deliberately out of scope):** page `<title>`/`<h1>` rename (flagged in phase 48); background/202 upload runs (synchronous locked above); deleting the uploaded archive bytes (temp file removed after unpack — only the unpacked content is kept); cross-kind source-name collisions with a git repo of the same folder name (pre-existing importer behavior, unchanged); coordinating an in-flight full Sync with an upload (accepted edge — per-file transactions + per-source-name prune keep the KB consistent).
+- **A10 / A11 / A16 / A17 honoured** — admin-only surface (no new session state), vanilla frontend (no CDN), one story E2E, one atomic `--no-gpg-sign` commit.