feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
@@ -66,6 +66,17 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
|
||||
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
|
||||
# BOR_SOURCES_DIR=~/bor-sources
|
||||
|
||||
# Phase 49: uploaded source archives (the Sources page upload form).
|
||||
# An uploaded .tar / .tar.gz / .tgz / .zip is unpacked to
|
||||
# BOR_UPLOAD_DIR/<name>/ where <name> is the filename minus the archive
|
||||
# suffix (homelab.tar.gz -> homelab/). Re-uploading the same name
|
||||
# replaces the folder's content IN PLACE — one folder, one row, no
|
||||
# missing window; a failed upload never touches the existing folder,
|
||||
# row, or KB. Deliberately separate from BOR_SOURCES_DIR (git checkouts).
|
||||
# BOR_UPLOAD_DIR=~/bor-sources/uploads
|
||||
# BOR_UPLOAD_MAX_MB=512 # caps BOTH the compressed upload and the total
|
||||
# extracted bytes (zip-bomb guard); must be > 0
|
||||
|
||||
# --- Admin & sign-in (single-admin password login; BOTH required) ---
|
||||
# The app refuses to start while either is empty (names the missing
|
||||
# variable(s) — README "Admin & sign-in"). Generate the secret with:
|
||||
|
||||
Reference in New Issue
Block a user