ducoterra
03d26255c6
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block.
76 lines
4.3 KiB
Markdown
76 lines
4.3 KiB
Markdown
# Story: Archive upload sources
|
|
|
|
**Phase:** `49_archive_upload_sources` · **E2E:** `tests/e2e/test_archive_upload_sources.py`
|
|
|
|
## Narrative
|
|
|
|
As **the admin (owner)**, the "Add a local directory" form makes me type
|
|
server paths — but the directories I want to index often live on *another
|
|
machine*. I want to **upload a tarball or zipfile** on the Sources
|
|
(`/git-sources.html`) page: the server **unpacks it and scans it** (the
|
|
content is indexed, visible in the RAG catalog immediately after the
|
|
upload settles). Re-uploading the **same filename** must **replace that
|
|
source in place** — same folder, same list row, previous content
|
|
overwritten — never a second folder or a duplicate row.
|
|
|
|
- **Given** I am signed in as admin, on the Sources page, and I have
|
|
`homelab.tar.gz` (containing `k3s.md`, `gitlab.md`)
|
|
- **When** I upload it
|
|
- **Then** it is unpacked to a server folder named after the archive
|
|
(`homelab`), imported (added/updated/pruned counts shown), appears in
|
|
the source list with the Local badge, and its documents show up in the
|
|
RAG catalog (`/sources.html`).
|
|
- **Given** I later edit the tarball (drop `gitlab.md`, add `caddy.md`)
|
|
and re-upload **`homelab.tar.gz`** (same name)
|
|
- **When** the upload settles
|
|
- **Then** there is still exactly one `homelab` folder and one list row;
|
|
`gitlab.md` is pruned from the index, `caddy.md` is indexed, `k3s.md`
|
|
is unchanged.
|
|
|
|
## Acceptance criteria
|
|
1. `POST /api/git-sources/upload` (multipart `file`, admin-only —
|
|
anonymous 403 like the rest of the router): accepts
|
|
`.tar`, `.tar.gz`, `.tgz`, `.zip` (else 422 naming the accepted set);
|
|
derives the source name from the filename minus the archive suffix
|
|
(`homelab.tar.gz` → `homelab`); rejects empty/unsafe names (422);
|
|
caps compressed upload AND extracted bytes at `BOR_UPLOAD_MAX_MB`
|
|
(default 512 — new settings `BOR_UPLOAD_DIR`, default
|
|
`~/bor-sources/uploads`, and `BOR_UPLOAD_MAX_MB`); blocks zip-slip /
|
|
tar-slip (absolute members, `..`, symlink/hardlink escapes, device
|
|
files) with a 422 and **no partial state** — a failed upload never
|
|
touches an existing folder, row, or the KB.
|
|
2. Successful upload: unpack → atomic swap-in of the folder → the
|
|
`git_sources` row is upserted by path (`kind=local`, no new row when
|
|
the path exists) → fail-fast model check (phase 41) →
|
|
`import_sources([folder], prune=True)` (single source) →
|
|
`regenerate_overview` when the KB changed → **200 with the same count
|
|
keys as the sync detail** (`files, added, updated, unchanged, pruned,
|
|
errors, chunks, overview`); one upload at a time (409 while a run is
|
|
in flight — the phase-32 pattern); a per-upload log line (PLAN §9).
|
|
3. Re-upload of the same filename replaces the folder's content in place
|
|
(temp unpack + rename swap — no missing window) and prunes files that
|
|
left the archive; no duplicate folder, no duplicate row.
|
|
4. The page: the "Add a local directory" form is **removed**; a labeled
|
|
archive upload form (file input `accept=".tar,.tar.gz,.tgz,.zip"`,
|
|
inline error `role=alert`, result line `role=status`, never-stale
|
|
button per §7.4 showing the counts) takes its place; the hint and
|
|
table caption mention upload+scan and in-place replace; existing
|
|
Local rows (incl. pre-existing hand-added dirs) still list/remove.
|
|
The `POST /api/git-sources` `kind=local` API contract is unchanged
|
|
(the capability survives via the API — no regression).
|
|
5. The phase-38 story E2E (`test_local_directory_sources.py`) is
|
|
rewritten to add local sources via the API (`page.request.post`)
|
|
instead of the removed form — its other assertions (Local badge,
|
|
sync-import, prune-remove, 422 naming the path) stand.
|
|
6. Unit + integration green, `app/` coverage >90%, story E2E green in
|
|
isolation, ruff + pyright clean, one `--no-gpg-sign` commit.
|
|
|
|
## Playwright Mapping Rule
|
|
`tests/e2e/test_archive_upload_sources.py` — one story, one file, run in
|
|
isolation: the admin uploads a real (test-built) tarball through the
|
|
page's file input → counts shown + Local-badged row named after the
|
|
archive stem + documents visible via the catalog; re-upload of the same
|
|
filename (modified archive) → still one row/folder, dropped file pruned,
|
|
new file indexed; a non-archive file gets an inline error; the
|
|
local-directory form is absent; anonymous still gets the gate.
|