# Phase 49 — Archive upload sources (tarball/zipfile → unpack → scan)

**Source:** owner request (chat, 2026-08-28) — "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan. Note that reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content." (Design confirmed by the owner in the same conversation.)
**Story:** `.agent/user_stories/archive-upload-sources.md`
**Context:** `35_git_sources_admin` (complete) — the `git_sources` table (`id, url, kind, path, added_at`), the admin-only `/api/git-sources` router, and the `/git-sources.html` manager page; `38_local_directory_sources` (complete) — `kind='local'` rows the Sync pipeline and `import_docs` walk directly, plus the page's "Add a local directory" form this phase removes; `32_admin_sync_button` + `41_sync_fail_fast_models` (complete) — the in-process pipeline parts this phase reuses: `check_models` fail-fast, `import_sources(sources, llm, prune=True)` (source name = folder basename, per-file transactions, per-source prune), `regenerate_overview`, and the sync-detail count keys (`files, added, updated, unchanged, pruned, errors, chunks, overview`); `16_admin_auth` (complete) — the `require_admin` router dependency the new route inherits.

## Objective
Replace the local-directory form on the Sources page with an **archive upload** form: `POST /api/git-sources/upload` accepts `.tar`/`.tar.gz`/`.tgz`/`.zip`, unpacks it **safely** into `BOR_UPLOAD_DIR/<name>/` (name = filename minus the archive suffix), atomically swaps it in when the name already exists, upserts the `git_sources` row (`kind=local`, no duplicates), and **scans it** — single-source `import_sources(prune=True)` + overview refresh — returning the sync-style counts. Re-uploading the same filename overwrites the previous content in place: one folder, one row, dropped files pruned from the KB.

## Dependencies
- `48_nav_rename_sources` (todo — runs first) — sequential only: it relabels the nav in the same `git-sources.html` this phase edits (keeps the diffs clean).
- `35_git_sources_admin` (complete) — the table/API/page this phase extends; the `require_admin` router; the `IntegrityError → 409` backstop pattern (`_commit_new`).
- `38_local_directory_sources` (complete) — the `kind=local` rows uploads register; the local form removed; the phase-38 story E2E rewritten in this phase.
- `32_admin_sync_button` / `41_sync_fail_fast_models` (complete) — `check_models` + `import_sources` + `regenerate_overview` + the count-key contract the upload response mirrors.
- `16_admin_auth` (complete) — admin-only surface (A10 revision).

## Tasks
1. `01_settings_and_unpack_utility.md` — `BOR_UPLOAD_DIR` + `BOR_UPLOAD_MAX_MB` settings, `.env.example`, and the new `app/rag/archive_upload.py` (name derivation, safe tar/zip unpack with traversal/symlink/size guards, atomic swap-in) + unit tests.
2. `02_upload_api.md` — `python-multipart` dependency + `POST /api/git-sources/upload` (stream-with-cap, one-at-a-time 409, upsert row, fail-fast models, single-source scan, sync-style 200 body, log line) + integration tests incl. re-upload/overwrite and no-partial-state.
3. `03_admin_page_upload.md` — the page: local form out, upload form in (§7.4 lifecycle, result line, hint/caption) + the phase-38 story E2E rewritten API-driven.
4. `04_story_e2e_docs_commit.md` — the story E2E (`test_archive_upload_sources.py`), README, regression suites in isolation, the one `--no-gpg-sign` commit, and the phase-dir move.

## Testing & Quality
- Unit: `tests/unit/test_archive_upload.py` — the name-derivation matrix (suffix stripping incl. double `.tar.gz`; empty/`..`/separator/control-char rejection); safe unpack (valid zip + tar; zip-slip `../`; absolute member; symlink + hardlink escape; device member; extracted-byte cap); `swap_in` (fresh, replace-existing with full content replacement, failure leaves the previous folder intact).
- Integration: `tests/integration/test_git_sources_upload.py` — anonymous 403 on the new route; 422 (bad extension, unsafe/empty name, traversal archive, corrupt archive); 413 (compressed cap, via the settings-override pattern of `test_git_sources_api.py`); 409 (second upload while the first is in flight); 200 happy path (real temp tarball, counts correct, row `kind=local` under `upload_dir`, docs in the KB); **re-upload same name** (one row, old folder content fully replaced, dropped file pruned, new file indexed); **failed re-upload leaves the previous folder + row + KB untouched**. The existing `test_git_sources_api.py` / `test_sync_api.py` / `test_import_docs_git.py` suites stay green through the change (the `kind=local` POST contract is untouched).
- Coverage: **>90%** on `app/` — the new module + endpoint fully covered.
- E2E (mandatory, A16): `tests/e2e/test_archive_upload_sources.py`, run in isolation.

## Completion Criteria
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] Uploading `homelab.tar.gz` via the page: unpacked under `BOR_UPLOAD_DIR/homelab/`, scanned (result line shows the counts), one list row (Local badge, name `homelab`), documents visible on `/sources.html`.
- [ ] Re-uploading `homelab.tar.gz` (modified): still exactly one folder and one row; dropped files pruned from the KB; added/changed files indexed.
- [ ] Non-archive file → inline 422; oversized → 413; zip-slip/tar-slip archive → 422 with the previous folder/row/KB untouched; second concurrent upload → 409.
- [ ] `#local-source-form` is gone from the page; the phase-38 story E2E green in isolation, API-driven; anonymous still gets the gate.
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (when 48 is complete), `test_smoke.py`.
- [ ] README + `.env.example` document the upload (formats, naming, in-place replace, both new settings); ruff + pyright clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.

## Locked decisions (owner permission 2026-08-28 — the confirmed design)
- **Formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` only (422 naming the accepted set otherwise).
- **Identity & in-place replace:** the source name is the uploaded filename minus the archive suffix (`homelab.tar.gz` → `homelab`, case-sensitive — Linux FS). The name determines the folder under `BOR_UPLOAD_DIR`; re-uploading the same name unpacks to a temp sibling and **renames it over the existing folder** (no missing window; a failed upload never touches the existing folder, row, or KB). **No second folder, no second row** — the `git_sources` row is upserted by `path` (`kind='local'`, reusing the phase-38 discriminator — **no migration, no new table**; A13 honoured).
- **New settings:** `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads` — deliberately separate from the git checkouts in `BOR_SOURCES_DIR`) and `BOR_UPLOAD_MAX_MB` (default **512**) capping BOTH the compressed upload and the total extracted bytes (zip-bomb guard).
- **The scan is synchronous in the upload request** (owner-confirmed): fail-fast `check_models` (phase 41) → `import_sources([folder], llm, prune=True)` (single source) → `regenerate_overview` when the KB changed → **200** with the sync-detail count keys so the page renders the same "N added · N pruned" line. One upload at a time — 409 while a run is in flight (the phase-32 pattern).
- **Unpack safety:** absolute member paths, `..` traversal, symlink/hardlink targets escaping the unpack folder, and device/FIFO members are rejected (422); extracted bytes are counted against the cap while writing.
- **Page:** the "Add a local directory" form is **removed**; the `POST /api/git-sources` `kind=local` **API contract is unchanged** (admin can still register a plain directory via the API — no regression; existing Local rows still list/remove, and the Sync button + `import_docs` keep walking them).
- **No auto-unwrap** of a single top-level folder — files land in the KB exactly as packed (documented in the hint/README).
- **`python-multipart`** is added to the dependencies — FastAPI's required multipart parser for file uploads (an A2 FastAPI implementation detail, not a new architectural anchor; recorded here per AGENTS.md rule 3).
- **Boundaries (deliberately out of scope):** page `<title>`/`<h1>` rename (flagged in phase 48); background/202 upload runs (synchronous locked above); deleting the uploaded archive bytes (temp file removed after unpack — only the unpacked content is kept); cross-kind source-name collisions with a git repo of the same folder name (pre-existing importer behavior, unchanged); coordinating an in-flight full Sync with an upload (accepted edge — per-file transactions + per-source-name prune keep the KB consistent).
- **A10 / A11 / A16 / A17 honoured** — admin-only surface (no new session state), vanilla frontend (no CDN), one story E2E, one atomic `--no-gpg-sign` commit.