Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
03d26255c605ae61267e1304eff859caff983dbe
brain-of-reese/.agent/phases/complete/49_archive_upload_sources/00_phase.md
T
ducoterra 03d26255c6 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place 
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
2026-08-28 15:57:59 -04:00

9.1 KiB
Raw Blame History

Phase 49 — Archive upload sources (tarball/zipfile → unpack → scan)

Source: owner request (chat, 2026-08-28) — "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan. Note that reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content." (Design confirmed by the owner in the same conversation.) Story: .agent/user_stories/archive-upload-sources.md Context: 35_git_sources_admin (complete) — the git_sources table (id, url, kind, path, added_at), the admin-only /api/git-sources router, and the /git-sources.html manager page; 38_local_directory_sources (complete) — kind='local' rows the Sync pipeline and import_docs walk directly, plus the page's "Add a local directory" form this phase removes; 32_admin_sync_button + 41_sync_fail_fast_models (complete) — the in-process pipeline parts this phase reuses: check_models fail-fast, import_sources(sources, llm, prune=True) (source name = folder basename, per-file transactions, per-source prune), regenerate_overview, and the sync-detail count keys (files, added, updated, unchanged, pruned, errors, chunks, overview); 16_admin_auth (complete) — the require_admin router dependency the new route inherits.

Objective

Replace the local-directory form on the Sources page with an archive upload form: POST /api/git-sources/upload accepts .tar/.tar.gz/.tgz/.zip, unpacks it safely into BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix), atomically swaps it in when the name already exists, upserts the git_sources row (kind=local, no duplicates), and scans it — single-source import_sources(prune=True) + overview refresh — returning the sync-style counts. Re-uploading the same filename overwrites the previous content in place: one folder, one row, dropped files pruned from the KB.

Dependencies

  • 48_nav_rename_sources (todo — runs first) — sequential only: it relabels the nav in the same git-sources.html this phase edits (keeps the diffs clean).
  • 35_git_sources_admin (complete) — the table/API/page this phase extends; the require_admin router; the IntegrityError → 409 backstop pattern (_commit_new).
  • 38_local_directory_sources (complete) — the kind=local rows uploads register; the local form removed; the phase-38 story E2E rewritten in this phase.
  • 32_admin_sync_button / 41_sync_fail_fast_models (complete) — check_models + import_sources + regenerate_overview + the count-key contract the upload response mirrors.
  • 16_admin_auth (complete) — admin-only surface (A10 revision).

Tasks

  1. 01_settings_and_unpack_utility.md — BOR_UPLOAD_DIR + BOR_UPLOAD_MAX_MB settings, .env.example, and the new app/rag/archive_upload.py (name derivation, safe tar/zip unpack with traversal/symlink/size guards, atomic swap-in) + unit tests.
  2. 02_upload_api.md — python-multipart dependency + POST /api/git-sources/upload (stream-with-cap, one-at-a-time 409, upsert row, fail-fast models, single-source scan, sync-style 200 body, log line) + integration tests incl. re-upload/overwrite and no-partial-state.
  3. 03_admin_page_upload.md — the page: local form out, upload form in (§7.4 lifecycle, result line, hint/caption) + the phase-38 story E2E rewritten API-driven.
  4. 04_story_e2e_docs_commit.md — the story E2E (test_archive_upload_sources.py), README, regression suites in isolation, the one --no-gpg-sign commit, and the phase-dir move.

Testing & Quality

  • Unit: tests/unit/test_archive_upload.py — the name-derivation matrix (suffix stripping incl. double .tar.gz; empty/../separator/control-char rejection); safe unpack (valid zip + tar; zip-slip ../; absolute member; symlink + hardlink escape; device member; extracted-byte cap); swap_in (fresh, replace-existing with full content replacement, failure leaves the previous folder intact).
  • Integration: tests/integration/test_git_sources_upload.py — anonymous 403 on the new route; 422 (bad extension, unsafe/empty name, traversal archive, corrupt archive); 413 (compressed cap, via the settings-override pattern of test_git_sources_api.py); 409 (second upload while the first is in flight); 200 happy path (real temp tarball, counts correct, row kind=local under upload_dir, docs in the KB); re-upload same name (one row, old folder content fully replaced, dropped file pruned, new file indexed); failed re-upload leaves the previous folder + row + KB untouched. The existing test_git_sources_api.py / test_sync_api.py / test_import_docs_git.py suites stay green through the change (the kind=local POST contract is untouched).
  • Coverage: >90% on app/ — the new module + endpoint fully covered.
  • E2E (mandatory, A16): tests/e2e/test_archive_upload_sources.py, run in isolation.

Completion Criteria

  • uv run pytest green; uv run pytest --cov=app --cov-report=term-missing >90%.
  • Uploading homelab.tar.gz via the page: unpacked under BOR_UPLOAD_DIR/homelab/, scanned (result line shows the counts), one list row (Local badge, name homelab), documents visible on /sources.html.
  • Re-uploading homelab.tar.gz (modified): still exactly one folder and one row; dropped files pruned from the KB; added/changed files indexed.
  • Non-archive file → inline 422; oversized → 413; zip-slip/tar-slip archive → 422 with the previous folder/row/KB untouched; second concurrent upload → 409.
  • #local-source-form is gone from the page; the phase-38 story E2E green in isolation, API-driven; anonymous still gets the gate.
  • uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov green in isolation (DB up).
  • Regression E2E suites green in isolation: test_git_sources_admin.py, test_local_directory_sources.py, test_sync_button.py, test_import_documents.py, test_nav_rename_sources.py (when 48 is complete), test_smoke.py.
  • README + .env.example document the upload (formats, naming, in-place replace, both new settings); ruff + pyright clean.
  • UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
  • One --no-gpg-sign commit; phase directory moved to .agent/phases/complete/.

Locked decisions (owner permission 2026-08-28 — the confirmed design)

  • Formats: .tar, .tar.gz, .tgz, .zip only (422 naming the accepted set otherwise).
  • Identity & in-place replace: the source name is the uploaded filename minus the archive suffix (homelab.tar.gz → homelab, case-sensitive — Linux FS). The name determines the folder under BOR_UPLOAD_DIR; re-uploading the same name unpacks to a temp sibling and renames it over the existing folder (no missing window; a failed upload never touches the existing folder, row, or KB). No second folder, no second row — the git_sources row is upserted by path (kind='local', reusing the phase-38 discriminator — no migration, no new table; A13 honoured).
  • New settings: BOR_UPLOAD_DIR (default ~/bor-sources/uploads — deliberately separate from the git checkouts in BOR_SOURCES_DIR) and BOR_UPLOAD_MAX_MB (default 512) capping BOTH the compressed upload and the total extracted bytes (zip-bomb guard).
  • The scan is synchronous in the upload request (owner-confirmed): fail-fast check_models (phase 41) → import_sources([folder], llm, prune=True) (single source) → regenerate_overview when the KB changed → 200 with the sync-detail count keys so the page renders the same "N added · N pruned" line. One upload at a time — 409 while a run is in flight (the phase-32 pattern).
  • Unpack safety: absolute member paths, .. traversal, symlink/hardlink targets escaping the unpack folder, and device/FIFO members are rejected (422); extracted bytes are counted against the cap while writing.
  • Page: the "Add a local directory" form is removed; the POST /api/git-sources kind=local API contract is unchanged (admin can still register a plain directory via the API — no regression; existing Local rows still list/remove, and the Sync button + import_docs keep walking them).
  • No auto-unwrap of a single top-level folder — files land in the KB exactly as packed (documented in the hint/README).
  • python-multipart is added to the dependencies — FastAPI's required multipart parser for file uploads (an A2 FastAPI implementation detail, not a new architectural anchor; recorded here per AGENTS.md rule 3).
  • Boundaries (deliberately out of scope): page <title>/<h1> rename (flagged in phase 48); background/202 upload runs (synchronous locked above); deleting the uploaded archive bytes (temp file removed after unpack — only the unpacked content is kept); cross-kind source-name collisions with a git repo of the same folder name (pre-existing importer behavior, unchanged); coordinating an in-flight full Sync with an upload (accepted edge — per-file transactions + per-source-name prune keep the KB consistent).
  • A10 / A11 / A16 / A17 honoured — admin-only surface (no new session state), vanilla frontend (no CDN), one story E2E, one atomic --no-gpg-sign commit.
Reference in New Issue View Git Blame Copy Permalink