feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place

Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.

README.md

@@ -63,7 +63,7 @@ uv run python -m scripts.llm_probe      # sanity: models + 768-dim check
 uv run python -m scripts.import_docs    # import the configured sources (below)
 ```
 
-Two managed source kinds (one page, one registry) plus a manual override:
+Managed source kinds (one page, one registry) plus a manual override:
 
 - **Git sources** — managed on the **admin Git sources page**
   (`/git-sources.html`) and stored in Postgres (see
@@ -73,11 +73,19 @@ Two managed source kinds (one page, one registry) plus a manual override:
   checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable
   in `.env` is the fallback — the moment the page stores a source, the
   variable is ignored.
+- **Archive upload sources** (phase 49) — a `.tar`/`.tar.gz`/`.tgz`/`.zip`
+  uploaded on the *same* admin page (see
+  [Archive upload sources](#archive-upload-sources)). The archive is
+  unpacked under `BOR_UPLOAD_DIR/<name>/` and scanned immediately; it is
+  registered as a `kind=local` row, like a local directory.
 - **Local directory sources** (phase 38) — an existing, non-git directory
   on the server, registered on the *same* admin page (see
   [Local directory sources](#local-directory-sources)). No clone, no
   checkout copy: the directory is walked in place. There is **no env var
-  for local paths** — the DB is the registry.
+  for local paths** — the DB is the registry. Since phase 49 the page's
+  “Add a local directory” form is gone (the archive upload replaced it):
+  a plain directory is registered via `POST /api/git-sources` with
+  `kind=local`; existing Local rows are unchanged.
 - **Manual directories** — `--source <path>` (repeatable) imports local
   directories directly and *always wins* over the stored sources (git and
   local) and the env fallback.
@@ -115,16 +123,24 @@ uv run uvicorn app.main:app --reload
   open to everyone).
 - **Git sources** (`/git-sources.html`) — the admin-managed source
   registry: the git repositories the **Sync sources** button clones and
-  indexes, **and** existing local directories it imports directly
-  (phase 38 — one table with a `kind` discriminator, one page); **admin-only**
-  (the same sign-in gate as Sources). Add or remove sources here — no
-  `.env` editing, no restart. A local directory must be an absolute,
-  existing directory at add-time (a missing/relative path is rejected
-  inline, naming the path; so are duplicates); list rows carry a **Git**
-  or **Local** badge. Adding/removing does not clone or prune on its
-  own: the Sync button performs that (git + local together, one run,
-  prune over the union), and a removed source's documents leave the index
-  on the next sync.
+  indexes, uploaded archives it unpacks and scans, **and** existing local
+  directories it imports directly (one table with a `kind` discriminator,
+  one page); **admin-only** (the same sign-in gate as Sources). Add or
+  remove sources here — no `.env` editing, no restart. The **archive
+  upload form** (phase 49) accepts `.tar`, `.tar.gz`, `.tgz`, `.zip`: the
+  archive is unpacked under `BOR_UPLOAD_DIR/<name>/` (name = filename
+  minus the archive suffix) and scanned immediately — re-uploading the
+  same filename replaces that source **in place** (one folder, one row,
+  dropped files pruned; see
+  [Archive upload sources](#archive-upload-sources)). A local directory
+  must be an absolute, existing directory at add-time (a
+  missing/relative path is rejected, naming the path; so are duplicates)
+  — since phase 49 this is an API-only operation (`POST /api/git-sources`
+  with `kind=local`; the page's form was replaced by the upload form). List
+  rows carry a **Git** or **Local** badge. Adding/removing does not clone
+  or prune on its own: the Sync button performs that (git + local together,
+  one run, prune over the union), and a removed source's documents leave
+  the index on the next sync.
 
 ## Thinking
 
@@ -368,13 +384,17 @@ first-class source too (phase 38). It shares the git sources' **one
 table** (the `git_sources` registry with a `kind` discriminator: `git` |
 `local`, migration 0007), **one admin page**, and **one Sync button**:
 
-- **Add it on the Git sources page** — the “Add a local directory” form
-  next to the git form. Add-time validation fails loud: the path is
-  trimmed, `~` is expanded, and must be an **absolute, existing directory
-  on the server** — anything else (missing, relative, a file) is rejected
-  with the path named inline; a duplicate path is rejected the same way.
-  There is **no env var for local paths** — the DB is the local-source
-  registry (`BOR_GIT_SOURCES` stays a git-only fallback).
+- **Register it via the API (phase 49)** — the phase-38 “Add a local
+  directory” form on the Git sources page was replaced by the archive
+  upload form; adding a plain directory is now an **API-only** operation:
+  `POST /api/git-sources` with `{"kind": "local", "path": …}`. Add-time
+  validation fails loud: the path is trimmed, `~` is expanded, and must be
+  an **absolute, existing directory on the server** — anything else
+  (missing, relative, a file) is 422 with the path named; a duplicate path
+  is 409 the same way. **Existing Local rows are unchanged**: they still
+  list, remove, and sync exactly as before. There is **no env var for
+  local paths** — the DB is the local-source registry
+  (`BOR_GIT_SOURCES` stays a git-only fallback).
 - **Sync walks it directly** — no clone, no checkout copy: each run
   indexes the directory in place (A9 format filter, hidden-dir skip,
   sha256 delta), together with the git checkouts in the **same run**.
@@ -394,6 +414,48 @@ table** (the `git_sources` registry with a `kind` discriminator: `git` |
   `BOR_GIT_SOURCES` is the git-only fallback; no git rows, no local rows,
   and no env URLs fails loudly ("no sources configured (git or local)").
 
+### Archive upload sources
+
+Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a
+source (phase 49, owner permission 2026-08-28). The form on the admin
+Git sources page — and the `POST /api/git-sources/upload` route behind
+it — replaced the phase-38 “Add a local directory” form. An uploaded
+source is registered as a `kind=local` row, so everything local
+directory sources do (Sync, prune, remove) applies to it:
+
+- **Accepted formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` — anything else
+  is 422 naming the accepted set. Unpacking is guarded: absolute member
+  paths, `..` traversal, symlink/hardlink targets escaping the unpack
+  folder, and device/FIFO members are rejected (422), and the total
+  *extracted* bytes count against the size cap (zip-bomb guard). A
+  zero-entry archive is 422; an archive with only non-A9 files is a
+  **valid replacement** (it indexes nothing and prunes the source's
+  previous documents).
+- **Naming rule:** the source name is the **filename minus the archive
+  suffix** (`homelab.tar.gz` → `homelab`, case-sensitive). The name is
+  both the folder under `BOR_UPLOAD_DIR` and the row's identity; files
+  land in the KB exactly as packed (no auto-unwrap of a single top-level
+  folder).
+- **In-place replace:** re-uploading the same filename creates **no
+  second folder and no second row** — the new content is unpacked to a
+  temp sibling and atomically renamed over the existing folder (no
+  missing window; a failed upload never touches the existing folder,
+  row, or KB), the row is upserted by path (`kind='local'`,
+  `added_at` preserved), and the source is re-scanned with `prune=True`
+  — files dropped from the archive leave the index in the same request.
+- **The scan is synchronous in the request:** it fails fast on the
+  models (503 when they are down — the folder/row are already committed,
+  so the next sync or re-upload retries idempotently), then runs the
+  single-source import (embeddings + per-document summaries) and the
+  change-gated KB overview refresh, and answers 200 with the sync-style
+  counts (`added`, `updated`, `unchanged`, `pruned`, …) the page renders
+  as its result line. One upload at a time — a concurrent upload gets
+  409.
+- **Where + how big:** archives unpack under `BOR_UPLOAD_DIR` (default
+  `~/bor-sources/uploads` — deliberately separate from the git checkouts
+  in `BOR_SOURCES_DIR`); `BOR_UPLOAD_MAX_MB` (default 512) caps **both**
+  the compressed upload and the total extracted bytes.
+
 ### Sync from the UI
 
 The **Sync sources** button on the **Sources** page — visible to the
@@ -404,9 +466,11 @@ git-source refresh in one click, in-process:
    admin-managed `git_sources` table; `BOR_GIT_SOURCES` only while that
    list is empty) through the same `clone_or_pull` the CLI uses (shallow
    clone on first run, `git pull --ff-only` afterwards), **and** the
-   local directories registered on the same page, walked directly
-   (re-verified to exist at sync time — a missing directory fails the run
-   loudly, naming the path);
+   local directories registered on the same page — including uploaded
+   archives (their `BOR_UPLOAD_DIR/<name>/` folders are `kind=local`
+   rows, *Archive upload sources*) — walked directly (re-verified to
+   exist at sync time — a missing directory fails the run loudly, naming
+   the path);
 2. **re-import with prune** — the `--prune` equivalent, so files deleted
    upstream leave the index (the button is the canonical "mirror the
    repos" action); the sha256 delta still skips unchanged files, so an
@@ -651,6 +715,8 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
 | `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) |
 | `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
 | `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
+| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | where uploaded source archives are unpacked — one subdirectory per source name (filename minus the archive suffix); separate from the git checkouts (see *Archive upload sources*) |
+| `BOR_UPLOAD_MAX_MB` | `512` | cap (MiB) for uploaded source archives — bounds **both** the compressed upload and the total extracted bytes (zip-bomb guard); must be > 0 |
 | `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
 | `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
 | `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | char budget for the `<knowledge_base>` (KB overview) prompt section |