Files

T

ducoterra 03d26255c6 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place

Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.

2026-08-28 15:57:59 -04:00

4.3 KiB

Raw Blame History

Story: Archive upload sources

Phase: 49_archive_upload_sources · E2E: tests/e2e/test_archive_upload_sources.py

Narrative

As the admin (owner), the "Add a local directory" form makes me type server paths — but the directories I want to index often live on another machine. I want to upload a tarball or zipfile on the Sources (/git-sources.html) page: the server unpacks it and scans it (the content is indexed, visible in the RAG catalog immediately after the upload settles). Re-uploading the same filename must replace that source in place — same folder, same list row, previous content overwritten — never a second folder or a duplicate row.

Given I am signed in as admin, on the Sources page, and I have homelab.tar.gz (containing k3s.md, gitlab.md)
When I upload it
Then it is unpacked to a server folder named after the archive (homelab), imported (added/updated/pruned counts shown), appears in the source list with the Local badge, and its documents show up in the RAG catalog (/sources.html).
Given I later edit the tarball (drop gitlab.md, add caddy.md) and re-upload homelab.tar.gz (same name)
When the upload settles
Then there is still exactly one homelab folder and one list row; gitlab.md is pruned from the index, caddy.md is indexed, k3s.md is unchanged.

Acceptance criteria

POST /api/git-sources/upload (multipart file, admin-only — anonymous 403 like the rest of the router): accepts .tar, .tar.gz, .tgz, .zip (else 422 naming the accepted set); derives the source name from the filename minus the archive suffix (homelab.tar.gz → homelab); rejects empty/unsafe names (422); caps compressed upload AND extracted bytes at BOR_UPLOAD_MAX_MB (default 512 — new settings BOR_UPLOAD_DIR, default ~/bor-sources/uploads, and BOR_UPLOAD_MAX_MB); blocks zip-slip / tar-slip (absolute members, .., symlink/hardlink escapes, device files) with a 422 and no partial state — a failed upload never touches an existing folder, row, or the KB.
Successful upload: unpack → atomic swap-in of the folder → the git_sources row is upserted by path (kind=local, no new row when the path exists) → fail-fast model check (phase 41) → import_sources([folder], prune=True) (single source) → regenerate_overview when the KB changed → 200 with the same count keys as the sync detail (files, added, updated, unchanged, pruned, errors, chunks, overview); one upload at a time (409 while a run is in flight — the phase-32 pattern); a per-upload log line (PLAN §9).
Re-upload of the same filename replaces the folder's content in place (temp unpack + rename swap — no missing window) and prunes files that left the archive; no duplicate folder, no duplicate row.
The page: the "Add a local directory" form is removed; a labeled archive upload form (file input accept=".tar,.tar.gz,.tgz,.zip", inline error role=alert, result line role=status, never-stale button per §7.4 showing the counts) takes its place; the hint and table caption mention upload+scan and in-place replace; existing Local rows (incl. pre-existing hand-added dirs) still list/remove. The POST /api/git-sources kind=local API contract is unchanged (the capability survives via the API — no regression).
The phase-38 story E2E (test_local_directory_sources.py) is rewritten to add local sources via the API (page.request.post) instead of the removed form — its other assertions (Local badge, sync-import, prune-remove, 422 naming the path) stand.
Unit + integration green, app/ coverage >90%, story E2E green in isolation, ruff + pyright clean, one --no-gpg-sign commit.

Playwright Mapping Rule

tests/e2e/test_archive_upload_sources.py — one story, one file, run in isolation: the admin uploads a real (test-built) tarball through the page's file input → counts shown + Local-badged row named after the archive stem + documents visible via the catalog; re-upload of the same filename (modified archive) → still one row/folder, dropped file pruned, new file indexed; a non-archive file gets an inline error; the local-directory form is absent; anonymous still gets the gate.

4.3 KiB Raw Blame History

Story: Archive upload sources

Narrative

Acceptance criteria

Playwright Mapping Rule

4.3 KiB

Raw Blame History