feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
+28
-30
@@ -174,33 +174,29 @@
|
||||
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
|
||||
</form>
|
||||
|
||||
<!-- Phase 38: the second add form — "Local directory": an
|
||||
existing directory on the server (NOT a git repo), walked
|
||||
directly by Sync / import_docs. The SAME never-stale-button
|
||||
+ inline-error pattern as the git form (PLAN §7.4): the
|
||||
button disables + relabels "Adding…" while the POST is out
|
||||
and recovers on success AND failure; on success the input
|
||||
clears and the list re-fetches (the new row lands with the
|
||||
Local badge). A missing/relative path 422s with the path
|
||||
named inline (paths are not secrets, unlike git URLs). -->
|
||||
<form id="local-source-form">
|
||||
<label for="local-source-path">Add a local directory</label>
|
||||
<input
|
||||
id="local-source-path"
|
||||
name="path"
|
||||
type="text"
|
||||
maxlength="2000"
|
||||
autocomplete="off"
|
||||
placeholder="~/Notes"
|
||||
required
|
||||
>
|
||||
<button type="submit" id="local-source-add">Add directory</button>
|
||||
<p class="git-source-error" id="local-source-error" role="alert" hidden></p>
|
||||
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form
|
||||
replaces the phase-38 local-directory form — an uploaded
|
||||
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and
|
||||
scanned immediately; the same filename replaces the source in
|
||||
place (no new folder, no duplicate row). The file control is
|
||||
labeled (visible <label for=…> — WCAG input-label rule); the
|
||||
button runs the §7.4 never-stale lifecycle ("Uploading…"
|
||||
while the POST is out, restored on success AND failure);
|
||||
non-2xx shows the server detail inline (role=alert), 200
|
||||
shows the sync-style counts (role=status). -->
|
||||
<form id="archive-upload-form">
|
||||
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
|
||||
<input id="archive-upload-file" name="file" type="file"
|
||||
accept=".tar,.tar.gz,.tgz,.zip" required>
|
||||
<button type="submit" id="archive-upload-btn">Upload & scan</button>
|
||||
<p class="git-source-error" id="archive-upload-error" role="alert" hidden></p>
|
||||
<p class="git-source-result" id="archive-upload-result" role="status"
|
||||
aria-live="polite" hidden></p>
|
||||
</form>
|
||||
|
||||
<div class="table-wrap" id="git-sources-table-wrap" role="region" aria-label="Sources" tabindex="0">
|
||||
<table class="git-sources-table" id="git-sources-table">
|
||||
<caption class="visually-hidden">Sources the Sync button imports — git repositories it clones and local directories it walks</caption>
|
||||
<caption class="visually-hidden">Sources the Sync button imports — git repositories it clones, local directories it walks, and uploaded archives (unpacked under the upload directory)</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Source</th>
|
||||
@@ -219,14 +215,16 @@
|
||||
|
||||
<!-- Scope boundary (phase locked decision): adding/removing a
|
||||
source does NOT clone or prune — the Sync button performs
|
||||
that. The hint says so (phase 38: git + local together,
|
||||
files removed from a source pruned). -->
|
||||
that; the phase-49 upload is the exception (it unpacks and
|
||||
scans in place, and a same-name re-upload replaces the
|
||||
source in place). -->
|
||||
<p class="git-source-hint" id="git-sources-hint" role="note">
|
||||
Sync clones/pulls the git repos and imports the local
|
||||
directories together (files removed from a source are
|
||||
pruned). Run the sync to clone/pull the git repos and import
|
||||
the local directories — removing a source
|
||||
prunes its documents from the index on the next sync.
|
||||
Uploads unpack and scan immediately — re-uploading the same
|
||||
filename replaces that source in place (no new folder, no
|
||||
duplicate row). The Sync button still imports the git
|
||||
checkouts and local directories together (files removed from
|
||||
a source are pruned) — removing a source prunes its documents
|
||||
from the index on the next sync.
|
||||
</p>
|
||||
</div>
|
||||
<!-- Polite live region: the screen-reader confirmation for list
|
||||
|
||||
Reference in New Issue
Block a user