Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place

Browse Source 
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
ducoterra
2026-08-28 15:57:59 -04:00
parent 872a07cee7
commit 03d26255c6
21 changed files with 3280 additions and 233 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agent/phases/complete/49_archive_upload_sources/00_phase.md
+51
View File
@@ -0,0 +1,51 @@
# Phase 49 — Archive upload sources (tarball/zipfile → unpack → scan)
**Source:** owner request (chat, 2026-08-28) — "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan. Note that reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content." (Design confirmed by the owner in the same conversation.)
**Story:** `.agent/user_stories/archive-upload-sources.md`
**Context:** `35_git_sources_admin` (complete) — the `git_sources` table (`id, url, kind, path, added_at`), the admin-only `/api/git-sources` router, and the `/git-sources.html` manager page; `38_local_directory_sources` (complete) — `kind='local'` rows the Sync pipeline and `import_docs` walk directly, plus the page's "Add a local directory" form this phase removes; `32_admin_sync_button` + `41_sync_fail_fast_models` (complete) — the in-process pipeline parts this phase reuses: `check_models` fail-fast, `import_sources(sources, llm, prune=True)` (source name = folder basename, per-file transactions, per-source prune), `regenerate_overview`, and the sync-detail count keys (`files, added, updated, unchanged, pruned, errors, chunks, overview`); `16_admin_auth` (complete) — the `require_admin` router dependency the new route inherits.
## Objective
Replace the local-directory form on the Sources page with an **archive upload** form: `POST /api/git-sources/upload` accepts `.tar`/`.tar.gz`/`.tgz`/`.zip`, unpacks it **safely** into `BOR_UPLOAD_DIR/<name>/` (name = filename minus the archive suffix), atomically swaps it in when the name already exists, upserts the `git_sources` row (`kind=local`, no duplicates), and **scans it** — single-source `import_sources(prune=True)` + overview refresh — returning the sync-style counts. Re-uploading the same filename overwrites the previous content in place: one folder, one row, dropped files pruned from the KB.
## Dependencies
- `48_nav_rename_sources` (todo — runs first) — sequential only: it relabels the nav in the same `git-sources.html` this phase edits (keeps the diffs clean).
- `35_git_sources_admin` (complete) — the table/API/page this phase extends; the `require_admin` router; the `IntegrityError → 409` backstop pattern (`_commit_new`).
- `38_local_directory_sources` (complete) — the `kind=local` rows uploads register; the local form removed; the phase-38 story E2E rewritten in this phase.
- `32_admin_sync_button` / `41_sync_fail_fast_models` (complete) — `check_models` + `import_sources` + `regenerate_overview` + the count-key contract the upload response mirrors.
- `16_admin_auth` (complete) — admin-only surface (A10 revision).
## Tasks
1. `01_settings_and_unpack_utility.md` — `BOR_UPLOAD_DIR` + `BOR_UPLOAD_MAX_MB` settings, `.env.example`, and the new `app/rag/archive_upload.py` (name derivation, safe tar/zip unpack with traversal/symlink/size guards, atomic swap-in) + unit tests.
2. `02_upload_api.md` — `python-multipart` dependency + `POST /api/git-sources/upload` (stream-with-cap, one-at-a-time 409, upsert row, fail-fast models, single-source scan, sync-style 200 body, log line) + integration tests incl. re-upload/overwrite and no-partial-state.
3. `03_admin_page_upload.md` — the page: local form out, upload form in (§7.4 lifecycle, result line, hint/caption) + the phase-38 story E2E rewritten API-driven.
4. `04_story_e2e_docs_commit.md` — the story E2E (`test_archive_upload_sources.py`), README, regression suites in isolation, the one `--no-gpg-sign` commit, and the phase-dir move.
## Testing & Quality
- Unit: `tests/unit/test_archive_upload.py` — the name-derivation matrix (suffix stripping incl. double `.tar.gz`; empty/`..`/separator/control-char rejection); safe unpack (valid zip + tar; zip-slip `../`; absolute member; symlink + hardlink escape; device member; extracted-byte cap); `swap_in` (fresh, replace-existing with full content replacement, failure leaves the previous folder intact).
- Integration: `tests/integration/test_git_sources_upload.py` — anonymous 403 on the new route; 422 (bad extension, unsafe/empty name, traversal archive, corrupt archive); 413 (compressed cap, via the settings-override pattern of `test_git_sources_api.py`); 409 (second upload while the first is in flight); 200 happy path (real temp tarball, counts correct, row `kind=local` under `upload_dir`, docs in the KB); **re-upload same name** (one row, old folder content fully replaced, dropped file pruned, new file indexed); **failed re-upload leaves the previous folder + row + KB untouched**. The existing `test_git_sources_api.py` / `test_sync_api.py` / `test_import_docs_git.py` suites stay green through the change (the `kind=local` POST contract is untouched).
- Coverage: **>90%** on `app/` — the new module + endpoint fully covered.
- E2E (mandatory, A16): `tests/e2e/test_archive_upload_sources.py`, run in isolation.
## Completion Criteria
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] Uploading `homelab.tar.gz` via the page: unpacked under `BOR_UPLOAD_DIR/homelab/`, scanned (result line shows the counts), one list row (Local badge, name `homelab`), documents visible on `/sources.html`.
- [ ] Re-uploading `homelab.tar.gz` (modified): still exactly one folder and one row; dropped files pruned from the KB; added/changed files indexed.
- [ ] Non-archive file → inline 422; oversized → 413; zip-slip/tar-slip archive → 422 with the previous folder/row/KB untouched; second concurrent upload → 409.
- [ ] `#local-source-form` is gone from the page; the phase-38 story E2E green in isolation, API-driven; anonymous still gets the gate.
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (when 48 is complete), `test_smoke.py`.
- [ ] README + `.env.example` document the upload (formats, naming, in-place replace, both new settings); ruff + pyright clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions (owner permission 2026-08-28 — the confirmed design)
- **Formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` only (422 naming the accepted set otherwise).
- **Identity & in-place replace:** the source name is the uploaded filename minus the archive suffix (`homelab.tar.gz` → `homelab`, case-sensitive — Linux FS). The name determines the folder under `BOR_UPLOAD_DIR`; re-uploading the same name unpacks to a temp sibling and **renames it over the existing folder** (no missing window; a failed upload never touches the existing folder, row, or KB). **No second folder, no second row** — the `git_sources` row is upserted by `path` (`kind='local'`, reusing the phase-38 discriminator — **no migration, no new table**; A13 honoured).
- **New settings:** `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads` — deliberately separate from the git checkouts in `BOR_SOURCES_DIR`) and `BOR_UPLOAD_MAX_MB` (default **512**) capping BOTH the compressed upload and the total extracted bytes (zip-bomb guard).
- **The scan is synchronous in the upload request** (owner-confirmed): fail-fast `check_models` (phase 41) → `import_sources([folder], llm, prune=True)` (single source) → `regenerate_overview` when the KB changed → **200** with the sync-detail count keys so the page renders the same "N added · N pruned" line. One upload at a time — 409 while a run is in flight (the phase-32 pattern).
- **Unpack safety:** absolute member paths, `..` traversal, symlink/hardlink targets escaping the unpack folder, and device/FIFO members are rejected (422); extracted bytes are counted against the cap while writing.
- **Page:** the "Add a local directory" form is **removed**; the `POST /api/git-sources` `kind=local` **API contract is unchanged** (admin can still register a plain directory via the API — no regression; existing Local rows still list/remove, and the Sync button + `import_docs` keep walking them).
- **No auto-unwrap** of a single top-level folder — files land in the KB exactly as packed (documented in the hint/README).
- **`python-multipart`** is added to the dependencies — FastAPI's required multipart parser for file uploads (an A2 FastAPI implementation detail, not a new architectural anchor; recorded here per AGENTS.md rule 3).
- **Boundaries (deliberately out of scope):** page `<title>`/`<h1>` rename (flagged in phase 48); background/202 upload runs (synchronous locked above); deleting the uploaded archive bytes (temp file removed after unpack — only the unpacked content is kept); cross-kind source-name collisions with a git repo of the same folder name (pre-existing importer behavior, unchanged); coordinating an in-flight full Sync with an upload (accepted edge — per-file transactions + per-source-name prune keep the KB consistent).
- **A10 / A11 / A16 / A17 honoured** — admin-only surface (no new session state), vanilla frontend (no CDN), one story E2E, one atomic `--no-gpg-sign` commit.
.agent/phases/complete/49_archive_upload_sources/01_settings_and_unpack_utility.md
+36
View File
@@ -0,0 +1,36 @@
# Task 01 — Settings + safe archive unpack utility
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Add the two upload settings and the reusable, unit-tested unpack machinery in a new module `app/rag/archive_upload.py`: archive-name derivation, safe tar/zip extraction (traversal/symlink/device/size guards), and the atomic swap-in that makes re-uploads replace in place without ever exposing a missing or partial folder.
## Work
1. `app/config.py` — add to `Settings` (both env-overridable, documented like `sources_dir`):
- `upload_dir: str = "~/bor-sources/uploads"` (→ `BOR_UPLOAD_DIR`) — where uploaded archives are unpacked (one subdirectory per source name). Kept **separate** from `sources_dir` (git checkouts).
- `upload_max_mb: int = 512` (→ `BOR_UPLOAD_MAX_MB`) — caps both the compressed upload and the total extracted bytes (zip-bomb guard). Add a validator rejecting `<= 0` (fail loud at startup, the `agent_max_rounds` pattern).
2. `.env.example` — document both settings next to `BOR_SOURCES_DIR` (default, meaning, the cap's dual role).
3. `app/rag/archive_upload.py` (new module) — pure file-system logic, no FastAPI imports (the API layer maps its exceptions to status codes):
- `class ArchiveUploadError(Exception)` — carries a user-safe message (no paths beyond the owner's own upload dir, never secrets).
- `ARCHIVE_SUFFIXES: tuple[str, ...] = (".tar.gz", ".tgz", ".zip", ".tar")` (longest-first — `.tar.gz` must strip before `.tar` would).
- `archive_source_name(filename: str) -> str` — take the basename (defensively strip any `/` or `\` a client could send), strip ONE trailing archive suffix from `ARCHIVE_SUFFIXES`; the result must be non-empty and not `.`/`..`, contain no path separators or control characters, or raise `ArchiveUploadError` (the API maps to 422). `homelab.tar.gz` → `homelab`; `notes.tgz` → `notes`; `a.zip` → `a`; `x.tar` → `x`; bare `tar.gz` → error (empty stem).
- `unpack_archive(archive: Path, target_dir: Path, max_extract_bytes: int) -> None` — extract into `target_dir` (the caller guarantees it does not exist yet and creates it empty):
- **zip** (`zipfile`): per member — reject absolute names and any name with a `..` part; resolve the final path and require it to stay within `target_dir`; reject symlink entries (mode bits) and non-file/non-dir entries; write file members while counting bytes — exceeding `max_extract_bytes` raises `ArchiveUploadError` (name the cap, not the archive content).
- **tar** (`tarfile.open(mode="r:*")` — handles gz/bz2/xz transparently): per member — the same name/containment checks; reject char/block devices and FIFOs; for symlinks/hardlinks, resolve the link target against the member's directory and reject any target that escapes `target_dir`; write regular files with the byte-counted cap.
- Clean up partial state: on any error, remove `target_dir` (shutil.rmtree, ignore missing) so no half-unpacked tree survives.
- `swap_in(new_dir: Path, final_dir: Path) -> None` — make `new_dir` become `final_dir` with **no missing window**: if `final_dir` exists, rename it to a same-filesystem sibling `final_dir.with_name(final_dir.name + ".old-" + uuid4().hex)`, rename `new_dir` → `final_dir`, then delete the `.old-` sibling; if it does not exist, just rename. On a rename failure, best-effort restore (`.old-` back, `new_dir` cleaned) and re-raise as `ArchiveUploadError`.
4. `tests/unit/test_archive_upload.py` (new) — full coverage of the module:
- name derivation: the matrix above incl. `upper.TAR.GZ` (case-sensitive stems preserved — `upper`), `a.tar.gz` double-strip, `..tar.gz` / `..` / `a/b.tar` / `a\tb.zip` / empty-stem rejections.
- unpack: a valid zip (nested dir + file) and a valid tar.gz extract byte-identically; zip-slip (`../evil.txt`), absolute member (`/etc/x`), tar symlink escaping (`ln -s /etc/passwd link`), tar hardlink escaping, a char-device member, and the extracted-cap (e.g. cap=10 bytes, 20-byte file) each raise `ArchiveUploadError` AND leave no partial `target_dir` behind.
- `swap_in`: fresh (final absent), replace (final's previous content fully gone, new content complete — no interleave), and restore-on-failure (monkeypatch `os.rename` to fail on the second rename → previous folder intact, new dir cleaned, `ArchiveUploadError` raised).
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_archive_upload.py -v` green; the module is fully covered (the >90% gate applies to `app/` — this new module must not drag the TOTAL down; aim for ~100% here).
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — no drop vs. baseline (no existing behavior touched).
## Completion Criteria
- [ ] `Settings` exposes `upload_dir` / `upload_max_mb` (defaults as above); `BOR_UPLOAD_MAX_MB=0` or negative fails startup with the validator message.
- [ ] `.env.example` documents both settings.
- [ ] `app/rag/archive_upload.py` exists with `ArchiveUploadError`, `ARCHIVE_SUFFIXES`, `archive_source_name`, `unpack_archive`, `swap_in`; no FastAPI/DB imports in the module.
- [ ] `uv run pytest tests/unit/test_archive_upload.py -v` green; full `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed phases (no existing file edited beyond config + .env.example).
.agent/phases/complete/49_archive_upload_sources/02_upload_api.md
+43
View File
@@ -0,0 +1,43 @@
# Task 02 — `POST /api/git-sources/upload` endpoint
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Ship the admin-only upload endpoint: stream the archive with the size cap, unpack it safely to a temp sibling, swap it in atomically, upsert the `git_sources` row (no duplicates), fail-fast check the models, scan the single source (`prune=True`) with the overview refresh, and answer 200 with the sync-style counts. One upload at a time (409).
## Work
1. `pyproject.toml` — add `python-multipart` to the main `dependencies` list (FastAPI's required multipart parser; the owner-confirmed implementation detail, phase locked decisions), then `uv lock`.
2. `app/api/git_sources.py` — extend the existing admin-only router (the `require_admin` dependency covers the new route automatically):
- `@router.post("/upload", response_model=UploadOut)` (new `UploadOut` schema in `app/schemas.py`: `source: str` + the sync-detail keys `files, added, updated, unchanged, pruned, errors, chunks: int` and `overview: bool` — same names as `_run_sync`'s `detail` so the page reuses its result-line shape).
- Handler (`async def upload_archive(file: UploadFile = File(...)) -> UploadOut`):
1. **Name/format gate** — `archive_source_name(file.filename or "")`: `ArchiveUploadError` → **422** with its message (naming the accepted formats when the extension is the problem).
2. **One at a time** — module-level `asyncio.Lock` (or a `bool` flag, the phase-32 `_task` spirit): already in flight → **409** `"an upload is already in progress"`.
3. **Stream with cap** — read the upload in 1 MiB chunks into `upload_dir / f".{name}.{uuid4().hex}.upload"`; total bytes > `upload_max_mb * 1024 * 1024` → delete the temp file, **413** naming the cap. (Compute `upload_dir` once: `Path(get_settings().upload_dir).expanduser()` — create it with `mkdir(parents=True, exist_ok=True)`.)
4. **Unpack to temp sibling** — `unpack_archive(archive, upload_dir / f".{name}.{uuid4().hex}.unpack", same cap)`; `ArchiveUploadError` → delete both temps, **422** (the message is already user-safe). A **completely empty archive** (zero entries) is **422** `"the archive contains no files"`. An archive with only non-A9 files is a **valid replacement**: the swap happens, the scan indexes nothing, and prune removes that source's docs — that is the intended "replace" semantics, do not reject it.
5. **Swap in** — `swap_in(temp_unpack, upload_dir / name)`; `ArchiveUploadError` → clean temps, **422** (the previous folder/row/KB are untouched — assert this in the tests).
6. **Upsert the row** — `path = str(upload_dir / name)`: a `GitSource` row with `path == path` already exists → leave it (no new row, `added_at` preserved); otherwise create `GitSource(url=path, kind="local", path=path)` (the `url` column is the NOT-NULL location column, phase-38 convention) via the shared `_commit_new` IntegrityError → 409 backstop.
7. **Fail-fast models** (phase 41) — `llm = LLMClient()`; `await check_models(llm)`; failure → **503** with the sanitized model-unavailable message (clean up nothing else — the folder/row are already committed, and the next sync/re-upload retries idempotently).
8. **Scan** — `summary = await import_sources([upload_dir / name], llm, prune=True)`; `overview = await regenerate_overview(llm) if summary.added + summary.updated > 0 else False`.
9. **Log** (PLAN §9, AGENTS.md rule 10) — one INFO line: `upload: name=… file=… bytes_in=… files=… added=… updated=… unchanged=… pruned=… errors=… overview=… total_ms=…`.
10. **Respond 200** with `UploadOut(source=name, …counts…, overview=overview)`.
- Keep every existing route byte-identical (the `kind=local` POST contract is untouched — the page just stops offering it).
3. `tests/integration/test_git_sources_upload.py` (new) — real Postgres + TestClient, following `tests/integration/test_git_sources_api.py`'s conventions (`clean_git_sources`-style TRUNCATE autouse fixture, the monkeypatched `get_settings` pattern to point `upload_dir` at a tmp dir and shrink `upload_max_mb`, the shared `client` / `admin_client` / `db` fixtures). Build real archives in-test with `tarfile`/`zipfile` over tmp fixture files (two `.md` sentinels). Cases:
- anonymous `client` → 403 on `/api/git-sources/upload` (same body as the rest of the router).
- `admin_client` + `data={"file": ("notes.txt", b"…", "text/plain")}` → **422** (accepted formats named); `("…", …)` with a `..`/empty stem → 422.
- oversized (shrink `upload_max_mb` via the settings override, upload a bigger file) → **413**, temp file cleaned up (the upload dir holds no stray `.` files).
- zip-slip archive (a member named `../evil.txt`) and a tar-slip symlink archive → **422**; then a prior good upload's folder + row + docs are **untouched** (the no-partial-state locked decision, asserted explicitly).
- happy path: `homelab.tar.gz` (2 md files) → **200**, `source="homelab"`, `added=2`, a `git_sources` row exists (`kind=local`, `path` under the tmp `upload_dir`), the unpacked folder exists, `GET /api/docs` lists both files under source `homelab`.
- **re-upload, same name, modified archive** (drop one file, add one, change one) → 200; `git_sources` still has exactly **one** row for that path (`added_at` unchanged); the folder contains only the new archive's files; the KB: dropped file **pruned** (pruned ≥ 1), new file added, changed file updated.
- corrupt archive (truncated zip bytes) → 422, previous state intact.
- 409: hold the in-flight flag (test seam: a module-level `upload_in_progress()` helper or the lock object exposed for tests — pick the smallest seam) → second request → **409**.
- the existing `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` suites stay green (run them).
## Testing & Quality
- Integration: all cases above green; the endpoint's every branch (422/413/409/503/200 + both upsert branches + the empty-archive 422) is covered.
- Coverage: **>90%** on `app/` (the new handler + schema fully exercised).
## Completion Criteria
- [ ] `python-multipart` in `pyproject.toml` `dependencies` + lock file updated; `uv sync` clean.
- [ ] `POST /api/git-sources/upload` implements steps 1–10; `git diff app/api/git_sources.py` shows no change to the existing GET/POST/DELETE handlers' behavior.
- [ ] `uv run pytest tests/integration/test_git_sources_upload.py -v` green (DB up); `uv run pytest tests/unit tests/integration -q` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] `uv run ruff check . && uv run pyright` clean.
.agent/phases/complete/49_archive_upload_sources/03_admin_page_upload.md
+58
View File
@@ -0,0 +1,58 @@
# Task 03 — Sources page: local form out, upload form in (+ phase-38 E2E rewrite)
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Rework the manager UI: remove the "Add a local directory" form and replace it with the labeled archive upload form (never-stale button, inline error, result line), update the hint/caption, and — because the form it drives is gone — rewrite the phase-38 story E2E to add local sources via the API instead.
## Work
1. `frontend/git-sources.html`:
- **Remove** the whole `#local-source-form` block (the phase-38 form: label, `#local-source-path` input, `#local-source-add` button, `#local-source-error` — including its phase-38 comment).
- **Add**, in its place (same `.git-source-error` / never-stale visual language as the git form):
```html
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form
replaces the phase-38 local-directory form — an uploaded
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and
scanned immediately; the same filename replaces the source in
place (no new folder, no duplicate row). -->
<form id="archive-upload-form">
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
<input id="archive-upload-file" name="file" type="file"
accept=".tar,.tar.gz,.tgz,.zip" required>
<button type="submit" id="archive-upload-btn">Upload &amp; scan</button>
<p class="git-source-error" id="archive-upload-error" role="alert" hidden></p>
<p class="git-source-result" id="archive-upload-result" role="status"
aria-live="polite" hidden></p>
</form>
```
(No `Content-Type` concerns: the JS posts a `FormData` and the browser sets the multipart boundary. The visible `<label>` satisfies the WCAG input-label rule for the file control.)
- **Update the hint** `#git-sources-hint`: uploads unpack + scan immediately and a same-name re-upload replaces in place; the Sync button still imports git checkouts and local directories together (union prune), and removing a source prunes on the next sync. Keep it one `role="note"` paragraph.
- **Update the table `<caption>`** (visually hidden): it reads "git repositories it clones and local directories it walks" — add uploaded archives (unpacked under the upload dir) to the description.
- Do **not** touch the page `<title>`/`<h1>` ("Git sources") — out of scope (flagged in phase 48); the nav link already reads "Sources" after phase 48.
2. `frontend/assets/styles.css` — one small block near the git-sources page styles: the file input (mono-ish, on-surface, ≥44px touch target, `:focus-visible` 3px outline like the other controls) and the `#archive-upload-result` success line (ink-soft on surface, ≥4.5:1 — reuse the existing palette; `prefers-reduced-motion` already global). Keep it minimal — no new layout regions.
3. `frontend/assets/git-sources.js`:
- Remove the local-form element refs (`localFormEl`, `pathInput`, `localAddBtn`, `localAddError`) and the second `wireAddForm(…)` call; update the module docstring (the page now wires the git add form + the archive upload).
- **Upload wiring** (one new `wireUploadForm`-style submit handler on `#archive-upload-form`), the §7.4 never-stale lifecycle:
- submit → `e.preventDefault()`; no file selected → inline error (the input is `required` too — browser prompt first); hide a previous result line; disable `#archive-upload-btn`, label **"Uploading…"**.
- `fetch("/api/git-sources/upload", { method: "POST", body: new FormData([["file", file]]) })` — **no** manual `Content-Type` header.
- **200**: clear the file input; hide the error; show `#archive-upload-result` with the counts in the sync-result shape (`2 added · 1 updated · 3 unchanged · 1 pruned` — omit zero parts, the `fmtSyncResult` convention from `sources.js`); announce through `#git-sources-announcer` (`"Archive uploaded: …"`); `await loadSources()` (the new/updated row lands with the Local badge; on a re-upload the row simply refreshes — no duplicate).
- **non-2xx**: inline the server detail via the existing `apiDetail(r, fallback)` (422 format/name/traversal, 413 size, 409 busy — the server messages are already user-safe); keep the file selection; restore the button (finally block, success AND failure).
- **network failure**: the `networkMessage` line, button restored.
4. **Rewrite `tests/e2e/test_local_directory_sources.py`** (phase-38 story — the form it drives is gone; its *acceptance* stands):
- Replace every form interaction (`page.fill("#local-source-path", …)` + `page.click("#local-source-add")`) with the authenticated API call the page's own JS no longer makes: `r = page.request.post(f"{app_url}/api/git-sources", json={"kind": "local", "path": str(local_dir)})` (the cookie rides the browser context — the established `page.request` pattern, cf. `test_git_sources_admin.py`).
- The "missing path → inline 422 naming the path + input kept" test becomes: the API returns **422** whose `detail` names the path (assert on the JSON body); drop the input-value assertions.
- Everything else stays: the Local badge on list rows, add/remove lifecycle, Sync importing the local dir (visible via `GET /api/docs` / the page), prune-on-file-deletion after re-sync, anonymous 403s, and the gate.
- The module docstring must note the phase-49 rewrite (form → API) so a future reader doesn't "restore" the form.
5. **UI Structure Check (AGENTS.md rule 5)** while in there: the new form is inside the existing `<main>` region, labeled, focus-visible, error `role=alert`, result `role=status`; no CDN (rule 6 — the no-CDN integration test re-proves it).
## Testing & Quality
- No new unit/integration logic (frontend + one E2E rewrite). `app/` coverage unaffected.
- The no-CDN integration test still passes (markup only, same-origin).
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov` green in isolation **after** the rewrite (DB up) — this is the proof the form removal caused no regression in the phase-38 story.
## Completion Criteria
- [ ] `#local-source-form` / `#local-source-path` / `#local-source-add` are gone from `frontend/git-sources.html` (and `git-sources.js`); `#archive-upload-form` with file input (accept set), button, `role=alert` error and `role=status` result line is in its place.
- [ ] Hint + caption updated; page `<title>`/`<h1>` untouched.
- [ ] `tests/e2e/test_local_directory_sources.py` green in isolation, fully API-driven for local adds.
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean (no Python behavior change expected in `app/` beyond nothing).
- [ ] Manual-feel check via the story E2E in task 04 (upload → counts → row) — this task leaves the page in a usable state on its own (independent viability).
.agent/phases/complete/49_archive_upload_sources/04_story_e2e_docs_commit.md
+34
View File
@@ -0,0 +1,34 @@
# Task 04 — Story E2E, README, regression suites, commit, phase move
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Prove the archive-upload story end to end with its dedicated Playwright suite (A16, in isolation), document the feature in the README, confirm no regression in the surrounding suites, and land the one atomic `--no-gpg-sign` commit.
## Work
1. **Create `tests/e2e/test_archive_upload_sources.py`** (one story, one file — conventions of `test_git_sources_admin.py` / `test_local_directory_sources.py`: shared `app_url` / `db_ready` fixtures, the `login(page, app_url, next=…)` helper, `page.request` for API assertions, desktop viewport 1280×800, sync `expect`). Build real archives in-test with Python's `tarfile` over `tmp_path` fixture files (markdown sentinels, e.g. `ALPHA-…` / `BETA-…` / `GAMMA-…`), named `e2e-upload.tar.gz` (source name will be `e2e-upload`).
Test cases:
- **form_swapped** — signed-in admin on `/git-sources.html`: `#local-source-form` count is 0; `#archive-upload-form` is visible with the file input (`accept` contains the four extensions) and the "Upload & scan" button; the hint mentions unpack/scan + in-place replace.
- **upload_scans_and_lists** — `page.set_input_files("#archive-upload-file", tarball_v1)` + click: the button shows "Uploading…" while in flight, then restores; `#archive-upload-result` shows the added count (2); the list gains exactly one row for `e2e-upload` with the **Local** badge; `page.request.get("/api/docs")` lists both sentinel files under source `e2e-upload`; on `/sources.html` (the RAG catalog) the table shows them (the admin sees the content where they expect it).
- **reupload_replaces_in_place** — back on the page, upload `tarball_v2` under the **same filename** (`e2e-upload.tar.gz`; v2: `alpha` modified, `beta` removed, `gamma` added): result line shows pruned ≥ 1; the list still has exactly **one** `e2e-upload` row (no duplicate — the row count for that source is invariant); `/api/docs` now shows `gamma` + the changed `alpha` and NOT `beta`.
- **bad_file_inline_error** — upload a `.txt` via the file input: `#archive-upload-error` (role=alert) shows the 422 detail naming the accepted formats; the button restores; the list is unchanged; a subsequent good upload still works (the form isn't wedged).
- **anonymous_gate** — anonymous on `/git-sources.html`: the gate (`#git-sources-gate`) shows, `#git-sources-content` (and thus the upload form) stays hidden, and `page.request.post(f"{app_url}/api/git-sources/upload", …)` is 403.
2. **README** — in the sources section (next to the phase-35/38 admin-sources docs): the upload form (accepted formats), the naming rule (filename minus archive suffix = source/folder name), the in-place replace + prune semantics on re-upload, the unpack destination (`BOR_UPLOAD_DIR`, default `~/bor-sources/uploads`) and the size cap (`BOR_UPLOAD_MAX_MB`, default 512, compressed + extracted), and the note that the local-directory *form* is gone but `POST /api/git-sources` with `kind=local` still works and existing Local rows are unchanged.
3. **Run the regression suites in isolation** (DB up, `--no-cov`): `test_git_sources_admin.py`, `test_local_directory_sources.py` (task 03's rewrite), `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (if phase 48 is complete — otherwise note it as the 48 gate), `test_smoke.py`, `test_shared_header.py`.
4. **Full gates**: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**; `uv run ruff check . && uv run pyright` clean.
5. **Commit** — one atomic commit staging exactly this phase's files (config, `.env.example`, `app/rag/archive_upload.py`, `app/api/git_sources.py`, `app/schemas.py`, `pyproject.toml` + lock, the two new test files, the rewritten `tests/e2e/test_local_directory_sources.py`, `frontend/git-sources.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, README):
`feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place` with a body citing the owner request (2026-08-28) + phase 49. `git commit --no-gpg-sign`.
6. **Move the phase directory**: `mv .agent/phases/todo/49_archive_upload_sources .agent/phases/complete/` and `git add -f` the moved directory + the story file `.agent/user_stories/archive-upload-sources.md` into the SAME commit (`.agent/` is gitignored by design — AGENTS.md rule 8).
## Testing & Quality
- Story E2E: `tests/e2e/test_archive_upload_sources.py` green **in isolation**.
- Coverage: `app/` >90% (the phase's Testing & Quality bar, re-proven on the final pass).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
- [ ] All regression suites green in isolation (list above).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README + `.env.example` document the upload semantics and both new settings.
- [ ] One `--no-gpg-sign` commit containing code, tests, story file, and the moved phase directory; `.agent/phases/todo/` no longer lists 49.
- [ ] No behavior change in completed phases (the suites above are the proof — incl. the Sync button and the `kind=local` API contract).
.agent/user_stories/archive-upload-sources.md
+75
View File
@@ -0,0 +1,75 @@
# Story: Archive upload sources
**Phase:** `49_archive_upload_sources` · **E2E:** `tests/e2e/test_archive_upload_sources.py`
## Narrative
As **the admin (owner)**, the "Add a local directory" form makes me type
server paths — but the directories I want to index often live on *another
machine*. I want to **upload a tarball or zipfile** on the Sources
(`/git-sources.html`) page: the server **unpacks it and scans it** (the
content is indexed, visible in the RAG catalog immediately after the
upload settles). Re-uploading the **same filename** must **replace that
source in place** — same folder, same list row, previous content
overwritten — never a second folder or a duplicate row.
- **Given** I am signed in as admin, on the Sources page, and I have
`homelab.tar.gz` (containing `k3s.md`, `gitlab.md`)
- **When** I upload it
- **Then** it is unpacked to a server folder named after the archive
(`homelab`), imported (added/updated/pruned counts shown), appears in
the source list with the Local badge, and its documents show up in the
RAG catalog (`/sources.html`).
- **Given** I later edit the tarball (drop `gitlab.md`, add `caddy.md`)
and re-upload **`homelab.tar.gz`** (same name)
- **When** the upload settles
- **Then** there is still exactly one `homelab` folder and one list row;
`gitlab.md` is pruned from the index, `caddy.md` is indexed, `k3s.md`
is unchanged.
## Acceptance criteria
1. `POST /api/git-sources/upload` (multipart `file`, admin-only —
anonymous 403 like the rest of the router): accepts
`.tar`, `.tar.gz`, `.tgz`, `.zip` (else 422 naming the accepted set);
derives the source name from the filename minus the archive suffix
(`homelab.tar.gz` → `homelab`); rejects empty/unsafe names (422);
caps compressed upload AND extracted bytes at `BOR_UPLOAD_MAX_MB`
(default 512 — new settings `BOR_UPLOAD_DIR`, default
`~/bor-sources/uploads`, and `BOR_UPLOAD_MAX_MB`); blocks zip-slip /
tar-slip (absolute members, `..`, symlink/hardlink escapes, device
files) with a 422 and **no partial state** — a failed upload never
touches an existing folder, row, or the KB.
2. Successful upload: unpack → atomic swap-in of the folder → the
`git_sources` row is upserted by path (`kind=local`, no new row when
the path exists) → fail-fast model check (phase 41) →
`import_sources([folder], prune=True)` (single source) →
`regenerate_overview` when the KB changed → **200 with the same count
keys as the sync detail** (`files, added, updated, unchanged, pruned,
errors, chunks, overview`); one upload at a time (409 while a run is
in flight — the phase-32 pattern); a per-upload log line (PLAN §9).
3. Re-upload of the same filename replaces the folder's content in place
(temp unpack + rename swap — no missing window) and prunes files that
left the archive; no duplicate folder, no duplicate row.
4. The page: the "Add a local directory" form is **removed**; a labeled
archive upload form (file input `accept=".tar,.tar.gz,.tgz,.zip"`,
inline error `role=alert`, result line `role=status`, never-stale
button per §7.4 showing the counts) takes its place; the hint and
table caption mention upload+scan and in-place replace; existing
Local rows (incl. pre-existing hand-added dirs) still list/remove.
The `POST /api/git-sources` `kind=local` API contract is unchanged
(the capability survives via the API — no regression).
5. The phase-38 story E2E (`test_local_directory_sources.py`) is
rewritten to add local sources via the API (`page.request.post`)
instead of the removed form — its other assertions (Local badge,
sync-import, prune-remove, 422 naming the path) stand.
6. Unit + integration green, `app/` coverage >90%, story E2E green in
isolation, ruff + pyright clean, one `--no-gpg-sign` commit.
## Playwright Mapping Rule
`tests/e2e/test_archive_upload_sources.py` — one story, one file, run in
isolation: the admin uploads a real (test-built) tarball through the
page's file input → counts shown + Local-badged row named after the
archive stem + documents visible via the catalog; re-upload of the same
filename (modified archive) → still one row/folder, dropped file pruned,
new file indexed; a non-archive file gets an inline error; the
local-directory form is absent; anonymous still gets the gate.
.env.example
+11
View File
@@ -66,6 +66,17 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
# Phase 49: uploaded source archives (the Sources page upload form).
# An uploaded .tar / .tar.gz / .tgz / .zip is unpacked to
# BOR_UPLOAD_DIR/<name>/ where <name> is the filename minus the archive
# suffix (homelab.tar.gz -> homelab/). Re-uploading the same name
# replaces the folder's content IN PLACE — one folder, one row, no
# missing window; a failed upload never touches the existing folder,
# row, or KB. Deliberately separate from BOR_SOURCES_DIR (git checkouts).
# BOR_UPLOAD_DIR=~/bor-sources/uploads
# BOR_UPLOAD_MAX_MB=512 # caps BOTH the compressed upload and the total
# extracted bytes (zip-bomb guard); must be > 0
# --- Admin & sign-in (single-admin password login; BOTH required) ---
# The app refuses to start while either is empty (names the missing
# variable(s) — README "Admin & sign-in"). Generate the secret with:
README.md
+88 -22
View File
@@ -63,7 +63,7 @@ uv run python -m scripts.llm_probe # sanity: models + 768-dim check
uv run python -m scripts.import_docs # import the configured sources (below)
```
Two managed source kinds (one page, one registry) plus a manual override:
Managed source kinds (one page, one registry) plus a manual override:
- **Git sources** — managed on the **admin Git sources page**
(`/git-sources.html`) and stored in Postgres (see
@@ -73,11 +73,19 @@ Two managed source kinds (one page, one registry) plus a manual override:
checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable
in `.env` is the fallback — the moment the page stores a source, the
variable is ignored.
- **Archive upload sources** (phase 49) — a `.tar`/`.tar.gz`/`.tgz`/`.zip`
uploaded on the *same* admin page (see
[Archive upload sources](#archive-upload-sources)). The archive is
unpacked under `BOR_UPLOAD_DIR/<name>/` and scanned immediately; it is
registered as a `kind=local` row, like a local directory.
- **Local directory sources** (phase 38) — an existing, non-git directory
on the server, registered on the *same* admin page (see
[Local directory sources](#local-directory-sources)). No clone, no
checkout copy: the directory is walked in place. There is **no env var
for local paths** — the DB is the registry.
for local paths** — the DB is the registry. Since phase 49 the page's
“Add a local directory” form is gone (the archive upload replaced it):
a plain directory is registered via `POST /api/git-sources` with
`kind=local`; existing Local rows are unchanged.
- **Manual directories** — `--source <path>` (repeatable) imports local
directories directly and *always wins* over the stored sources (git and
local) and the env fallback.
@@ -115,16 +123,24 @@ uv run uvicorn app.main:app --reload
open to everyone).
- **Git sources** (`/git-sources.html`) — the admin-managed source
registry: the git repositories the **Sync sources** button clones and
indexes, **and** existing local directories it imports directly
(phase 38 — one table with a `kind` discriminator, one page); **admin-only**
(the same sign-in gate as Sources). Add or remove sources here — no
`.env` editing, no restart. A local directory must be an absolute,
existing directory at add-time (a missing/relative path is rejected
inline, naming the path; so are duplicates); list rows carry a **Git**
or **Local** badge. Adding/removing does not clone or prune on its
own: the Sync button performs that (git + local together, one run,
prune over the union), and a removed source's documents leave the index
on the next sync.
indexes, uploaded archives it unpacks and scans, **and** existing local
directories it imports directly (one table with a `kind` discriminator,
one page); **admin-only** (the same sign-in gate as Sources). Add or
remove sources here — no `.env` editing, no restart. The **archive
upload form** (phase 49) accepts `.tar`, `.tar.gz`, `.tgz`, `.zip`: the
archive is unpacked under `BOR_UPLOAD_DIR/<name>/` (name = filename
minus the archive suffix) and scanned immediately — re-uploading the
same filename replaces that source **in place** (one folder, one row,
dropped files pruned; see
[Archive upload sources](#archive-upload-sources)). A local directory
must be an absolute, existing directory at add-time (a
missing/relative path is rejected, naming the path; so are duplicates)
— since phase 49 this is an API-only operation (`POST /api/git-sources`
with `kind=local`; the page's form was replaced by the upload form). List
rows carry a **Git** or **Local** badge. Adding/removing does not clone
or prune on its own: the Sync button performs that (git + local together,
one run, prune over the union), and a removed source's documents leave
the index on the next sync.
## Thinking
@@ -368,13 +384,17 @@ first-class source too (phase 38). It shares the git sources' **one
table** (the `git_sources` registry with a `kind` discriminator: `git` |
`local`, migration 0007), **one admin page**, and **one Sync button**:
- **Add it on the Git sources page** — the “Add a local directory” form
next to the git form. Add-time validation fails loud: the path is
trimmed, `~` is expanded, and must be an **absolute, existing directory
on the server** — anything else (missing, relative, a file) is rejected
with the path named inline; a duplicate path is rejected the same way.
There is **no env var for local paths** — the DB is the local-source
registry (`BOR_GIT_SOURCES` stays a git-only fallback).
- **Register it via the API (phase 49)** — the phase-38 “Add a local
directory” form on the Git sources page was replaced by the archive
upload form; adding a plain directory is now an **API-only** operation:
`POST /api/git-sources` with `{"kind": "local", "path": …}`. Add-time
validation fails loud: the path is trimmed, `~` is expanded, and must be
an **absolute, existing directory on the server** — anything else
(missing, relative, a file) is 422 with the path named; a duplicate path
is 409 the same way. **Existing Local rows are unchanged**: they still
list, remove, and sync exactly as before. There is **no env var for
local paths** — the DB is the local-source registry
(`BOR_GIT_SOURCES` stays a git-only fallback).
- **Sync walks it directly** — no clone, no checkout copy: each run
indexes the directory in place (A9 format filter, hidden-dir skip,
sha256 delta), together with the git checkouts in the **same run**.
@@ -394,6 +414,48 @@ table** (the `git_sources` registry with a `kind` discriminator: `git` |
`BOR_GIT_SOURCES` is the git-only fallback; no git rows, no local rows,
and no env URLs fails loudly ("no sources configured (git or local)").
### Archive upload sources
Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a
source (phase 49, owner permission 2026-08-28). The form on the admin
Git sources page — and the `POST /api/git-sources/upload` route behind
it — replaced the phase-38 “Add a local directory” form. An uploaded
source is registered as a `kind=local` row, so everything local
directory sources do (Sync, prune, remove) applies to it:
- **Accepted formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` — anything else
is 422 naming the accepted set. Unpacking is guarded: absolute member
paths, `..` traversal, symlink/hardlink targets escaping the unpack
folder, and device/FIFO members are rejected (422), and the total
*extracted* bytes count against the size cap (zip-bomb guard). A
zero-entry archive is 422; an archive with only non-A9 files is a
**valid replacement** (it indexes nothing and prunes the source's
previous documents).
- **Naming rule:** the source name is the **filename minus the archive
suffix** (`homelab.tar.gz` → `homelab`, case-sensitive). The name is
both the folder under `BOR_UPLOAD_DIR` and the row's identity; files
land in the KB exactly as packed (no auto-unwrap of a single top-level
folder).
- **In-place replace:** re-uploading the same filename creates **no
second folder and no second row** — the new content is unpacked to a
temp sibling and atomically renamed over the existing folder (no
missing window; a failed upload never touches the existing folder,
row, or KB), the row is upserted by path (`kind='local'`,
`added_at` preserved), and the source is re-scanned with `prune=True`
— files dropped from the archive leave the index in the same request.
- **The scan is synchronous in the request:** it fails fast on the
models (503 when they are down — the folder/row are already committed,
so the next sync or re-upload retries idempotently), then runs the
single-source import (embeddings + per-document summaries) and the
change-gated KB overview refresh, and answers 200 with the sync-style
counts (`added`, `updated`, `unchanged`, `pruned`, …) the page renders
as its result line. One upload at a time — a concurrent upload gets
409.
- **Where + how big:** archives unpack under `BOR_UPLOAD_DIR` (default
`~/bor-sources/uploads` — deliberately separate from the git checkouts
in `BOR_SOURCES_DIR`); `BOR_UPLOAD_MAX_MB` (default 512) caps **both**
the compressed upload and the total extracted bytes.
### Sync from the UI
The **Sync sources** button on the **Sources** page — visible to the
@@ -404,9 +466,11 @@ git-source refresh in one click, in-process:
admin-managed `git_sources` table; `BOR_GIT_SOURCES` only while that
list is empty) through the same `clone_or_pull` the CLI uses (shallow
clone on first run, `git pull --ff-only` afterwards), **and** the
local directories registered on the same page, walked directly
(re-verified to exist at sync time — a missing directory fails the run
loudly, naming the path);
local directories registered on the same page — including uploaded
archives (their `BOR_UPLOAD_DIR/<name>/` folders are `kind=local`
rows, *Archive upload sources*) — walked directly (re-verified to
exist at sync time — a missing directory fails the run loudly, naming
the path);
2. **re-import with prune** — the `--prune` equivalent, so files deleted
upstream leave the index (the button is the canonical "mirror the
repos" action); the sha256 delta still skips unchanged files, so an
@@ -651,6 +715,8 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | where uploaded source archives are unpacked — one subdirectory per source name (filename minus the archive suffix); separate from the git checkouts (see *Archive upload sources*) |
| `BOR_UPLOAD_MAX_MB` | `512` | cap (MiB) for uploaded source archives — bounds **both** the compressed upload and the total extracted bytes (zip-bomb guard); must be > 0 |
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | char budget for the `<knowledge_base>` (KB overview) prompt section |
app/api/git_sources.py
+227 -8
View File
@@ -17,9 +17,13 @@ Routes: ``GET`` (DB rows oldest-first, or the env list with
``path``, git rows — and env rows — report ``path: null``), ``POST``
(201, validated create; ``kind`` selects the validation: git → exactly
the phase-35 URL contract, local → an existing absolute directory, else
422 naming the path), ``DELETE /{source_id}`` (204). The whole router
sits behind :func:`app.core.auth.require_admin` — anonymous callers get
403 on every route.
422 naming the path), ``POST /upload`` (phase 49 — admin archive upload:
``.tar``/``.tar.gz``/``.tgz``/``.zip`` streamed with a size cap, safely
unpacked, atomically swapped in over an existing folder of the same
name, row upserted, then the synchronous single-source scan — see
:func:`upload_archive`), ``DELETE /{source_id}`` (204). The whole
router sits behind :func:`app.core.auth.require_admin` — anonymous
callers get 403 on every route.
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
masking discipline), so every git 409/422 detail is a fixed generic
@@ -27,27 +31,47 @@ string that never repeats the submitted URL. Local paths are not
secrets — the local 422/409 details name the (expanded) path so the
owner sees exactly which directory failed.
Scope boundary (phase locked decisions): adding or removing a source
does NOT clone, import, or prune anything — the existing Sync button
performs that (a removal prunes on the next sync, ``prune=True``).
Scope boundary (phase locked decisions): the CRUD routes do NOT
clone, import, or prune anything — the existing Sync button performs
that (a removal prunes on the next sync, ``prune=True``). The phase-49
upload route is the exception: it unpacks the archive and then scans
the single source synchronously in the request (``import_sources``
with ``prune=True`` + the change-gated overview refresh) and answers
with the sync-style counts.
"""
from __future__ import annotations
import logging
import re
import shutil
import time
import uuid
from pathlib import Path
from typing import Literal, cast
from fastapi import APIRouter, Depends, HTTPException, Response
from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.api.sync import _sanitize_error
from app.config import get_settings
from app.core.auth import require_admin
from app.db import get_db
from app.models import GitSource
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow
from app.rag.archive_upload import (
ARCHIVE_SUFFIXES,
ArchiveUploadError,
archive_source_name,
swap_in,
unpack_archive,
)
from app.rag.importer import import_sources
from app.rag.llm import LLMClient, ModelUnavailableError, check_models
from app.rag.overview import regenerate_overview
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow, UploadOut
logger = logging.getLogger("app.api.git_sources")
router = APIRouter(
prefix="/git-sources",
@@ -55,6 +79,18 @@ router = APIRouter(
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
)
#: One upload at a time (phase 49, task 02 — the phase-32 ``_task``
#: spirit): the flag is held from the name gate through the scan
#: response. A plain bool, not an ``asyncio.Lock`` — it is checked and
#: set with no await in between (a single app loop can never enter
#: twice), and it stays correct across requests that run on separate
#: event loops (the TestClient convention).
_upload_in_progress = False
#: Streaming read size while counting compressed upload bytes (1 MiB
#: chunks — the task-02 cap check granularity).
_STREAM_CHUNK = 1 << 20
#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
@@ -191,6 +227,189 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
)
@router.post("/upload", response_model=UploadOut)
async def upload_archive(
file: UploadFile = File(...), # noqa: B008
db: Session = Depends(get_db), # noqa: B008
) -> UploadOut:
"""Upload a source archive and scan it (phase 49, task 02).
The scan is **synchronous in the request** (phase locked decisions,
owner-confirmed) and mirrors the admin sync pipeline:
1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip``
(422 naming the accepted set) and a safe source name
(``archive_source_name`` — its message is the 422 detail);
2. one at a time — 409 ``an upload is already in progress``;
3. stream the upload in 1 MiB chunks into a dotfile temp with the
``upload_max_mb`` cap — 413 naming the cap, temp deleted;
4. unpack to a temp sibling (traversal/symlink/device/corrupt/
over-cap all 422 with the task-01 user-safe message, temps
deleted); a zero-entry archive is 422 ``the archive contains no
files`` — an archive with only non-A9 files is a VALID
replacement (the scan indexes nothing, prune removes the
source's docs);
5. atomic swap-in — a same-name re-upload replaces the previous
folder in place; a failure leaves the previous folder/row/KB
untouched (422);
6. upsert the row by ``path`` (``kind='local'``; an existing row is
left as-is — ``added_at`` preserved — and the unique index is
the 409 backstop);
7. fail-fast ``check_models`` — 503 with the sanitized
model-unavailable message; the folder/row are already committed,
so the next sync/re-upload retries idempotently;
8. ``import_sources([folder], llm, prune=True)`` + the change-gated
``regenerate_overview``;
9. one INFO log line (PLAN §9 / AGENTS.md rule 10);
10. 200 with the sync-detail count keys (``UploadOut``).
"""
started = time.monotonic()
settings = get_settings()
total = 0
# 1. Name/format gate — the accepted formats first (the 422 names
# them), then the task-01 safe-name derivation. A BARE suffix
# ("tar.gz") is an accepted format with no usable stem — it
# passes here and gets task-01's "no usable source name" 422.
# No upload dir is created for a rejected name.
filename = file.filename or ""
lowered = filename.lower()
if not any(
lowered.endswith(suffix) or lowered == suffix.lstrip(".")
for suffix in ARCHIVE_SUFFIXES
):
raise HTTPException(
status_code=422,
detail="only .tar, .tar.gz, .tgz or .zip archives are accepted",
)
try:
name = archive_source_name(filename)
except ArchiveUploadError as e:
raise HTTPException(status_code=422, detail=str(e)) from None
# 2. One at a time — the flag is checked and set with no await
# between, so the single app loop can never enter twice.
global _upload_in_progress
if _upload_in_progress:
raise HTTPException(status_code=409, detail="an upload is already in progress")
_upload_in_progress = True
upload_root = Path(settings.upload_dir).expanduser()
upload_root.mkdir(parents=True, exist_ok=True)
max_bytes = settings.upload_max_mb * 1024 * 1024
temp_upload = upload_root / f".{name}.{uuid.uuid4().hex}.upload"
temp_unpack = upload_root / f".{name}.{uuid.uuid4().hex}.unpack"
try:
# 3. Stream with the compressed-size cap — dotfile temps are
# hidden from the upload dir's listing.
try:
with open(temp_upload, "wb") as out:
while chunk := await file.read(_STREAM_CHUNK):
total += len(chunk)
if total > max_bytes:
raise HTTPException(
status_code=413,
detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit",
)
out.write(chunk)
except HTTPException:
temp_upload.unlink(missing_ok=True)
raise
# 4. Unpack to a temp sibling; the compressed bytes are no
# longer needed once unpacked (phase locked decision: only
# the unpacked content is kept).
try:
unpack_archive(temp_upload, temp_unpack, max_bytes)
except ArchiveUploadError as e:
temp_upload.unlink(missing_ok=True)
shutil.rmtree(temp_unpack, ignore_errors=True)
raise HTTPException(status_code=422, detail=str(e)) from None
temp_upload.unlink(missing_ok=True)
if not any(temp_unpack.iterdir()):
# Zero entries = a user error. (Only non-A9 files is NOT an
# error — it still has entries and is a valid replacement.)
shutil.rmtree(temp_unpack, ignore_errors=True)
raise HTTPException(status_code=422, detail="the archive contains no files")
# 5. Swap in — a same-name re-upload replaces the previous
# folder atomically; a failure leaves it, the row, and the
# KB untouched.
final_dir = upload_root / name
try:
swap_in(temp_unpack, final_dir)
except ArchiveUploadError as e:
shutil.rmtree(temp_unpack, ignore_errors=True)
raise HTTPException(status_code=422, detail=str(e)) from None
# 6. Upsert the row by path — no duplicates: an existing row is
# left exactly as it is (``added_at`` preserved); the unique
# index is the 409 backstop for a concurrent insert the
# pre-check missed.
path = str(final_dir)
if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
_commit_new(
GitSource(url=path, kind="local", path=path),
f"a local source with this path already exists: {path}",
db,
)
# Release the request session NOW — the handler never touches
# ``db`` again (the scan below uses its own sessions). If the
# session stayed open, its uncommitted transaction (the
# ``_commit_new`` refresh SELECT) would hold ``git_sources``
# locks for the whole scan, and any concurrent TRUNCATE of the
# KB tables (the E2E isolation fixtures) would deadlock against
# the scan's own document locks — a cycle Postgres cannot see.
# ``get_db``'s teardown close() is idempotent.
db.close()
# 7. Fail-fast models (phase 41) — 503 with the sanitized
# message; nothing else is rolled back (the folder/row are
# committed and the next sync/re-upload retries idempotently).
llm = LLMClient()
try:
await check_models(llm)
except ModelUnavailableError as e:
raise HTTPException(status_code=503, detail=_sanitize_error(str(e))) from None
# 8. Scan — single source, prune (dropped files leave the KB),
# then the change-gated overview refresh (phases 31/32).
summary = await import_sources([final_dir], llm, prune=True)
overview = False
if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm)
finally:
_upload_in_progress = False
# No temp may survive any failure path (defensive — each step
# already cleans its own; on success both are already gone).
temp_upload.unlink(missing_ok=True)
shutil.rmtree(temp_unpack, ignore_errors=True)
# 9. Per-upload log line (PLAN §9 / AGENTS.md rule 10).
logger.info(
"upload: name=%s file=%s bytes_in=%d files=%d added=%d updated=%d "
"unchanged=%d pruned=%d errors=%d overview=%s total_ms=%d",
name,
filename,
total,
summary.files,
summary.added,
summary.updated,
summary.unchanged,
summary.pruned,
summary.errors,
overview,
round((time.monotonic() - started) * 1000),
)
# 10. Respond 200 with the sync-style counts.
return UploadOut(
source=name,
files=summary.files,
added=summary.added,
updated=summary.updated,
unchanged=summary.unchanged,
pruned=summary.pruned,
errors=summary.errors,
chunks=summary.chunks,
overview=overview,
)
@router.delete("/{source_id}", status_code=204)
def delete_git_source(
source_id: uuid.UUID,
app/config.py
+20
View File
@@ -156,6 +156,18 @@ class Settings(BaseSettings):
#: 28). Stored as a raw string — ``Path.expanduser()`` is applied in
#: the import script, not here.
sources_dir: str = "~/bor-sources"
#: Where uploaded source archives are unpacked (phase 49) — one
#: subdirectory per source name (filename minus the archive suffix).
#: Deliberately kept **separate** from ``sources_dir`` (the git
#: checkouts). Raw string — ``Path.expanduser()`` is applied by the
#: upload endpoint, not here.
upload_dir: str = "~/bor-sources/uploads"
#: Cap in MiB for uploaded source archives (phase 49): it bounds BOTH
#: the compressed upload size and the total extracted bytes (the
#: zip-bomb guard). ``<= 0`` would reject every upload — a typo, so
#: the validator fails loudly at startup (the ``agent_max_rounds``
#: pattern).
upload_max_mb: int = 512
@field_validator("import_extensions")
@classmethod
@@ -181,6 +193,14 @@ class Settings(BaseSettings):
raise ValueError("agent_max_rounds must be >= 0 (0 = no tools)")
return v
@field_validator("upload_max_mb")
@classmethod
def _upload_max_mb_positive(cls, v: int) -> int:
"""``0``/negative would reject every upload — fail loud at startup."""
if v <= 0:
raise ValueError("upload_max_mb must be > 0 (MiB)")
return v
# Suggested questions (onboarding + empty state).
suggestions: list[str] = [
"How is my Kubernetes cluster set up?",
app/rag/archive_upload.py
+272
View File
@@ -0,0 +1,272 @@
"""Safe archive unpacking for uploaded sources (phase 49, task 01).
Pure file-system logic — no FastAPI/DB imports. The API layer
(``POST /api/git-sources/upload``, phase 49, task 02) calls these and
maps :class:`ArchiveUploadError` to status codes (422).
The guarantees (phase 49 locked decisions):
* :func:`archive_source_name` derives the source name from the uploaded
filename — ONE trailing archive suffix stripped, longest-first so
``homelab.tar.gz`` yields ``homelab`` (never ``homelab.tar``) — and
rejects anything that would not be a safe single folder name (the name
becomes a directory under ``BOR_UPLOAD_DIR`` and the ``git_sources``
row's ``path``).
* :func:`unpack_archive` extracts ``.zip`` or ``.tar`` (gz/bz2/xz
transparently) into a fresh directory, rejecting absolute member
paths, ``..`` traversal, symlink/hardlink targets that escape the
unpack directory, and device/FIFO members — and counting every
extracted byte against a cap (zip-bomb guard). Any failure removes the
partial ``target_dir`` so no half-unpacked tree survives.
* :func:`swap_in` makes ``new_dir`` become ``final_dir`` with **no
missing window**: the previous folder is renamed to a unique
same-filesystem ``.old-`` sibling first, the new folder is renamed
into place, then the sibling is deleted. A failed swap restores the
previous folder (best effort) and removes ``new_dir``.
"""
from __future__ import annotations
import os
import re
import shutil
import stat
import tarfile
import uuid
import zipfile
from pathlib import Path, PurePosixPath
from typing import IO
#: Accepted archive suffixes, LONGEST FIRST — ``.tar.gz`` must match
#: before ``.tar`` would (phase 49 locked decision: these four formats
#: only).
ARCHIVE_SUFFIXES: tuple[str, ...] = (".tar.gz", ".tgz", ".zip", ".tar")
#: A client-supplied filename never legitimately contains a path
#: separator — reject rather than strip (defense in depth; the upload
#: also carries a browsed-path, which the browser normalizes).
_SEPARATOR_RE = re.compile(r"[/\\]")
#: Windows drive-letter prefix (``C:``) — an absolute member path in a
#: cross-platform zip.
_DRIVE_RE = re.compile(r"^[A-Za-z]:")
#: Streaming read size while counting extracted bytes.
_CHUNK_SIZE = 1 << 20
class ArchiveUploadError(Exception):
"""A user-safe archive/unpack error (the API maps it to a status).
Messages name the problem — and the cap where relevant — never the
archive's content and never any path beyond the owner's own upload
directory.
"""
def archive_source_name(filename: str) -> str:
"""The source name for an uploaded archive filename.
The filename is used exactly as sent — a client can never
legitimately embed a ``/`` or ``\\``, so a separator is *rejected*
(the defensive basename step below is then a no-op, kept for the
contract). ONE trailing archive suffix from :data:`ARCHIVE_SUFFIXES`
is removed, matched case-insensitively and longest-first:
``homelab.tar.gz`` → ``homelab``, ``a.tar.gz`` → ``a`` (a single
compound strip, not a double one), ``a.zip.zip`` → ``a.zip``. The
stem keeps its original case — the name becomes a folder name on a
Linux filesystem (case-sensitive).
Raises:
ArchiveUploadError: the filename is empty, contains a path
separator, or the stripped stem is empty / ``.`` / ``..`` /
contains control characters (the API maps to 422).
"""
if not filename:
raise ArchiveUploadError("empty file name")
if _SEPARATOR_RE.search(filename):
raise ArchiveUploadError("file name contains a path separator")
# No separator above, so the basename is the name itself — kept
# explicit so the "take the basename" contract lives in one place.
base = filename.rsplit("/", 1)[-1].rsplit("\\", 1)[-1]
lowered = base.lower()
for suffix in ARCHIVE_SUFFIXES: # longest first
if lowered.endswith(suffix):
base = base[: -len(suffix)]
break
if lowered == suffix.lstrip("."): # bare suffix ("tar.gz") — no stem
base = ""
break
if not base or base in (".", ".."):
raise ArchiveUploadError("archive file name has no usable source name")
if any(ord(c) < 0x20 or ord(c) == 0x7F for c in base):
raise ArchiveUploadError("archive file name contains control characters")
return base
def _member_dest(name: str, target: Path) -> Path:
"""The destination path for one archive member, or raise.
Rejects empty names, absolute names (POSIX ``/``, Windows ``\\`` or
drive letters), and any ``..`` path component (zip names are
``/``-separated; backslash forms are normalized before the check).
The final path is resolved against the target (following any symlink
an earlier member may have created) and must stay inside ``target``.
"""
if not name:
raise ArchiveUploadError("archive member with an empty name")
if name.startswith(("/", "\\")) or _DRIVE_RE.match(name):
raise ArchiveUploadError("archive member with an absolute path")
if ".." in PurePosixPath(name).parts or ".." in name.replace("\\", "/").split("/"):
raise ArchiveUploadError("archive member path traversal")
dest = target / name
target_resolved = target.resolve()
dest_resolved = dest.resolve()
if dest_resolved != target_resolved and target_resolved not in dest_resolved.parents:
raise ArchiveUploadError("archive member path escapes the unpack directory")
return dest
def _link_target_resolved(linkname: str, link_dir: Path, target: Path) -> Path:
"""Resolve a symlink/hardlink target against its member's directory.
The resolved target must stay inside ``target`` — anything else
(absolute targets, ``..`` climbs out) is rejected. Returns the
resolved path (used directly for hardlinks).
"""
if not linkname:
raise ArchiveUploadError("archive link with an empty target")
# An absolute linkname wins over the join (pathlib semantics) and is
# then caught by the containment check below.
resolved = (link_dir / linkname).resolve()
target_resolved = target.resolve()
if resolved != target_resolved and target_resolved not in resolved.parents:
raise ArchiveUploadError("archive link target escapes the unpack directory")
return resolved
def _write_capped(src: IO[bytes], dst: Path, max_extract_bytes: int, total: list[int]) -> None:
"""Stream ``src`` to ``dst``, counting into ``total[0]``.
Raises as soon as the cumulative extracted bytes EXCEED
``max_extract_bytes`` (the cap itself is exactly reachable). The
error names the cap, not the archive content.
"""
dst.parent.mkdir(parents=True, exist_ok=True)
with open(dst, "wb") as out:
while chunk := src.read(_CHUNK_SIZE):
total[0] += len(chunk)
if total[0] > max_extract_bytes:
raise ArchiveUploadError(
f"archive exceeds the {max_extract_bytes}-byte extraction cap"
)
out.write(chunk)
def _unpack_zip(archive: Path, target: Path, max_extract_bytes: int) -> None:
total: list[int] = [0]
with zipfile.ZipFile(archive) as zf:
for member in zf.infolist():
# The high 16 bits of external_attr are the Unix mode when
# present. Modes may be 0 (Windows-made zips), bare permission
# bits (CPython ``writestr``: 0o600), or a full mode with the
# file-type bits — only the latter can prove a member is a
# symlink/device/FIFO, and only those are rejected; entries
# without type bits are decided by the member name.
mode = member.external_attr >> 16
if stat.S_ISLNK(mode):
raise ArchiveUploadError("zip archives with symlink entries are not allowed")
if mode & 0o170000 and not (stat.S_ISREG(mode) or stat.S_ISDIR(mode)):
raise ArchiveUploadError("zip archives with non-regular entries are not allowed")
dest = _member_dest(member.filename, target)
if member.filename.endswith("/") or (mode and stat.S_ISDIR(mode)):
dest.mkdir(parents=True, exist_ok=True)
else:
with zf.open(member) as src:
_write_capped(src, dest, max_extract_bytes, total)
def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
# ``r:*`` auto-detects plain/gz/bz2/xz compression.
total: list[int] = [0]
with tarfile.open(archive, mode="r:*") as tf:
for member in tf.getmembers():
dest = _member_dest(member.name, target)
if member.issym():
_link_target_resolved(member.linkname, dest.parent, target)
dest.parent.mkdir(parents=True, exist_ok=True)
os.symlink(member.linkname, dest)
elif member.islnk():
resolved = _link_target_resolved(member.linkname, dest.parent, target)
dest.parent.mkdir(parents=True, exist_ok=True)
os.link(resolved, dest)
elif member.isdir():
dest.mkdir(parents=True, exist_ok=True)
elif member.isreg():
src = tf.extractfile(member)
if src is None:
raise ArchiveUploadError("corrupt archive member")
_write_capped(src, dest, max_extract_bytes, total)
else: # char/block device, FIFO
raise ArchiveUploadError(
"tar archives with device or FIFO members are not allowed"
)
def unpack_archive(archive: Path, target_dir: Path, max_extract_bytes: int) -> None:
"""Extract ``archive`` into ``target_dir`` (created empty here).
``target_dir`` must NOT exist yet — the caller passes a fresh unique
path (a temp sibling of the final folder). On ANY failure — bad or
corrupt archive, unsafe member, cap exceeded, OS error — the partial
``target_dir`` is removed so no half-unpacked tree survives, and the
failure is raised as :class:`ArchiveUploadError` (the module's only
public exception type).
"""
if target_dir.exists() or target_dir.is_symlink():
raise ArchiveUploadError("the unpack target already exists")
target_dir.mkdir(parents=True)
try:
# Content-sniff the container: a .zip that is really a tar (or a
# truncated file) falls through to the tar reader and fails
# loudly there instead of half-extracting.
if zipfile.is_zipfile(archive):
_unpack_zip(archive, target_dir, max_extract_bytes)
else:
_unpack_tar(archive, target_dir, max_extract_bytes)
except Exception as exc:
shutil.rmtree(target_dir, ignore_errors=True)
if isinstance(exc, ArchiveUploadError):
raise
raise ArchiveUploadError("could not unpack the archive") from exc
def swap_in(new_dir: Path, final_dir: Path) -> None:
"""Atomically make ``new_dir`` become ``final_dir`` (no missing window).
If ``final_dir`` exists it is first renamed to a unique same-
filesystem sibling ``<name>.old-<hex>``, then ``new_dir`` is renamed
into place, and the ``.old-`` sibling is deleted (re-uploads replace
the previous content in place — one folder, no stale files). If it
does not exist, this is a plain rename.
On a rename failure the previous folder is restored (best effort) and
``new_dir`` removed, then the failure is raised as
:class:`ArchiveUploadError` — a failed upload never leaves the
previous folder, row, or KB in a mixed state.
"""
old_dir: Path | None = None
try:
if final_dir.exists():
old_dir = final_dir.with_name(final_dir.name + ".old-" + uuid.uuid4().hex)
os.rename(final_dir, old_dir)
os.rename(new_dir, final_dir)
except OSError as exc:
if old_dir is not None:
try:
os.rename(old_dir, final_dir)
except OSError:
shutil.rmtree(old_dir, ignore_errors=True)
shutil.rmtree(new_dir, ignore_errors=True)
raise ArchiveUploadError("could not replace the previous folder") from exc
if old_dir is not None:
shutil.rmtree(old_dir)
app/schemas.py
+22
View File
@@ -260,3 +260,25 @@ class GitSourceList(BaseModel):
sources: list[GitSourceRow]
from_env: bool
class UploadOut(BaseModel):
"""``POST /api/git-sources/upload`` response (phase 49, task 02).
The uploaded source's name (filename minus the archive suffix) plus
the SAME count keys as the admin sync's success ``detail``
(``files``, ``added``, ``updated``, ``unchanged``, ``pruned``,
``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the
``overview`` flag: the Sources page renders the same
"N added · N pruned" result line for both.
"""
source: str
files: int
added: int
updated: int
unchanged: int
pruned: int
errors: int
chunks: int
overview: bool
frontend/assets/git-sources.js
+136 -42
View File
@@ -1,10 +1,12 @@
/* Brain of Reese — Git sources admin page (phase 35, task 04;
* local directories, phase 38 task 04).
* archive uploads, phase 49 task 03).
*
* The page module for /git-sources.html: the admin-only manager for the
* stored source list (git-sources table, phase 35 tasks 01/02) — git
* repo URLs (kind "git") and existing local directories (kind
* "local", phase 38).
* repo URLs (kind "git") and uploaded archives unpacked under
* BOR_UPLOAD_DIR (kind "local", phase 49; the phase-38
* local-directory form is gone — the kind=local API POST is
* unchanged, the page just no longer offers it).
* This module is the single owner of the page's behaviour:
*
* • boot — initSharedHeader() (one cached whoami, shared with the
@@ -22,17 +24,30 @@
* credentials; phase 32's masking discipline). Non-2xx or a
* network failure renders the role="alert" load error with a
* retry button — never a stuck page.
* • add — #git-source-form submit → POST /api/git-sources {url};
* #local-source-form submit → POST /api/git-sources
* {kind: "local", path} (phase 38). ONE §7.4 never-stale
* lifecycle for both (wireAddForm): the button disables +
* relabels "Adding…" while the request is out, re-enables on
* success AND failure. 201 clears the input, reloads the list,
* and focuses the new row's Remove button (a11y); a failure (409
* duplicate, 422 validation) shows the server detail inline under
* the form (role="alert", 422 shape-aware like the tuning forms)
* and keeps the input — the instruction survives. Local 422/409
* details name the path (paths are not secrets, unlike URLs).
* • add — #git-source-form submit → POST /api/git-sources {url}.
* The §7.4 never-stale lifecycle (wireAddForm): the button
* disables + relabels "Adding…" while the request is out,
* re-enables on success AND failure. 201 clears the input,
* reloads the list, and focuses the new row's Remove button
* (a11y); a failure (409 duplicate, 422 validation) shows the
* server detail inline under the form (role="alert", 422
* shape-aware like the tuning forms) and keeps the input — the
* instruction survives. 409/422 details are fixed generic strings
* (credential safety — the URL is never echoed).
* • upload — #archive-upload-form submit (phase 49) → POST
* /api/git-sources/upload with a FormData file (NO manual
* Content-Type — the browser sets the multipart boundary). The
* SAME §7.4 never-stale lifecycle: the button disables +
* relabels "Uploading…" while the request is out and is restored
* on success AND failure. 200 clears the file input, shows the
* sync-style count line ("2 added · 1 pruned" — fmtUploadResult,
* sources.js's fmtSyncResult convention) in the role=status
* result line, announces "Archive uploaded: …" and reloads the
* list (the new/updated row lands with the Local badge; a
* re-upload simply refreshes the row — no duplicate). Non-2xx
* inlines the server detail (422 format/name/traversal, 413
* size, 409 busy — the messages are already user-safe) and KEEPS
* the file selection — the fix is one re-pick, not a re-type.
* • remove — a row's Remove button asks window.confirm first
* (removal prunes the documents only on the NEXT sync — the
* confirm says so). Cancel → nothing; ok → the row button
@@ -45,9 +60,12 @@
* aria-live=polite): the screen-reader confirmation for loads,
* adds, and removals.
*
* Scope boundary (phase locked decisions): adding or removing a repo
* does NOT clone, import, or prune — the sync service (server-side)
* performs that; the page's hint box says so.
* Scope boundary (phase locked decisions): adding a git repo or
* removing a source does NOT clone, import, or prune — the sync
* service (server-side) performs that; the page's hint box says so.
* The phase-49 upload is the exception: it unpacks and scans the
* single source in place, and its response counts render as the
* result line.
*
* The shared header module loads through this script's own relative
* import ("./header.js") — a hoisted import evaluated before this body
@@ -64,12 +82,14 @@ const formEl = document.querySelector("#git-source-form");
const urlInput = document.querySelector("#git-source-url");
const addBtn = document.querySelector("#git-source-add");
const addError = document.querySelector("#git-source-error");
/* Phase 38: the second add form — "Local directory" (same element
contract as the git form, own ids). */
const localFormEl = document.querySelector("#local-source-form");
const pathInput = document.querySelector("#local-source-path");
const localAddBtn = document.querySelector("#local-source-add");
const localAddError = document.querySelector("#local-source-error");
/* Phase 49: the archive upload form (replaces the phase-38 local
directory form — same card, a file input instead of a path input).
The response counts render in the role=status result line. */
const uploadFormEl = document.querySelector("#archive-upload-form");
const uploadFileInput = document.querySelector("#archive-upload-file");
const uploadBtn = document.querySelector("#archive-upload-btn");
const uploadError = document.querySelector("#archive-upload-error");
const uploadResult = document.querySelector("#archive-upload-result");
const loadErrorEl = document.querySelector("#git-sources-load-error");
const loadErrorText = document.querySelector("#git-sources-load-error-text");
const retryBtn = document.querySelector("#git-sources-retry");
@@ -255,18 +275,16 @@ async function removeSource(s, btn, rowError, kindLabel) {
}
}
/* ---------- add (POST /api/git-sources) — both forms, one lifecycle
* (the local form is phase 38) ----------
* The git form posts {url}; the local form posts {kind:"local",path}.
* wireAddForm gives both the §7.4 never-stale lifecycle: while the
* request is out the button disables + relabels "Adding…" and
/* ---------- add (POST /api/git-sources) — the git form ----------
* wireAddForm gives the form the §7.4 never-stale lifecycle: while
* the request is out the button disables + relabels "Adding…" and
* re-enables (idle label restored) on success AND failure. 201 clears
* the input, reloads the list, and focuses the new row's Remove button
* (a11y); a failure (409 duplicate, 422 validation) shows the server
* detail inline under the form (role="alert", 422 shape-aware via
* apiDetail) and keeps the input — the fix is one edit, not a re-type.
* Git 409/422 details are fixed generic strings (credential safety);
* local details name the path (not a secret). */
* 409/422 details are fixed generic strings (credential safety — the
* URL is never echoed). */
function wireAddForm(opts) {
const { form, input, btn, error } = opts;
if (!form || !input || !btn) return;
@@ -336,18 +354,94 @@ wireAddForm({
idleLabel: "Add source",
});
wireAddForm({
form: localFormEl,
input: pathInput,
btn: localAddBtn,
error: localAddError,
body: (path) => ({ kind: "local", path }),
emptyMessage: "Enter a directory path to add.",
failMessage: "Could not add the local directory — try again.",
networkMessage: "Could not add the local directory — is the app reachable?",
addedMessage: "Local source added.",
idleLabel: "Add directory",
});
/* ---------- upload (POST /api/git-sources/upload) — phase 49 -------
* The archive upload form: the file input's selection is posted as
* FormData (the browser sets the multipart boundary — no manual
* Content-Type). §7.4 never-stale: "Uploading…" while in flight,
* restored in the finally block on success AND failure. 200 → the
* input clears, the sync-style counts land in the role=status result
* line, the announcer confirms, and loadSources() re-renders the row
* (Local badge; a re-upload refreshes the existing row — no
* duplicate). Non-2xx → the server detail inline (role=alert; 422
* format/name/traversal, 413 size, 409 busy — user-safe as-is) with
* the file selection KEPT; network failure → the fixed line. */
/* The success line's text — the sync-result shape (sources.js's
fmtSyncResult convention): "N added" always leads, then updated /
unchanged / pruned — zero parts omitted (unchanged is shown
when nothing was added or updated). */
function fmtUploadResult(detail) {
const d = detail || {};
const added = d.added || 0;
const updated = d.updated || 0;
const parts = [`${added} added`];
if (updated > 0) parts.push(`${updated} updated`);
if ((d.unchanged || 0) > 0 || (added === 0 && updated === 0)) {
parts.push(`${d.unchanged || 0} unchanged`);
}
if ((d.pruned || 0) > 0) parts.push(`${d.pruned} pruned`);
return parts.join(" · ");
}
if (uploadFormEl && uploadFileInput && uploadBtn) {
uploadFormEl.addEventListener("submit", async (e) => {
e.preventDefault();
// Client-side no-file check (the input is `required` too — the
// browser's native prompt is the first line, this one the second).
const file = uploadFileInput.files && uploadFileInput.files[0];
if (!file) {
if (uploadError) {
uploadError.textContent = "Choose an archive file to upload.";
uploadError.hidden = false;
}
return;
}
if (uploadError) uploadError.hidden = true;
if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean
uploadBtn.disabled = true; // §7.4: one upload per click
uploadBtn.textContent = "Uploading…";
try {
// Multipart from the form itself (the file input's name is
// "file") — the browser sets the boundary; NO manual
// Content-Type header.
const r = await fetch("/api/git-sources/upload", {
method: "POST",
body: new FormData(uploadFormEl),
});
if (r.ok) {
let data = {};
try {
data = await r.json();
} catch {
/* the body is advisory — the counts line degrades gracefully */
}
uploadFileInput.value = ""; // 200: the archive is unpacked + scanned
if (uploadResult) {
uploadResult.textContent = fmtUploadResult(data);
uploadResult.hidden = false;
}
announce(`Archive uploaded: ${data.source || file.name}.`);
await loadSources(); // the new/updated row lands (Local badge)
return;
}
// 422 (format/name/traversal), 413 (size), 409 (busy): the server
// detail inline, the file selection KEPT — the fix is one
// re-pick, not a re-type.
if (uploadError) {
uploadError.textContent = await apiDetail(r, "Could not upload the archive — try again.");
uploadError.hidden = false;
}
} catch {
if (uploadError) {
uploadError.textContent = "Could not upload the archive — is the app reachable?";
uploadError.hidden = false;
}
} finally {
uploadBtn.disabled = false; // never stale — success OR failure
uploadBtn.textContent = "Upload & scan";
}
});
}
/* After a successful add, focus the new row's Remove button so the
keyboard/screen-reader user lands where the new data is. The 201
frontend/assets/styles.css
+108 -32
View File
@@ -335,9 +335,10 @@ html::after {
.auth-link:disabled { opacity: 0.6; cursor: wait; }
.auth-link svg { width: 16px; height: 16px; display: none; }
/* Mobile-only sign-out copy: hidden on desktop, revealed inside
the hamburger dropdown on mobile (phase 46 UX revision). */
.sign-out-mobile {
/* Mobile-only sign-out / sign-in copies: hidden on desktop,
revealed inside the hamburger dropdown on mobile (phase 46 UX revision). */
.sign-out-mobile,
.sign-in-mobile {
display: none;
}
@@ -1384,14 +1385,14 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
}
/* Add forms — the tuning form's surface as a single row: visible
label + mono location input (git URLs may embed credentials, local
paths may contain anything, so both inputs are mono) + the brand
button; wraps to a column at narrow widths (the <=640px block
below). Phase 38: the "Local directory" form (#local-source-form)
reuses the git form's rules VERBATIM — one form language for both
kinds. */
label + mono location input (git URLs may embed credentials, so the
input is mono) + the brand button; wraps to a column at narrow
widths (the <=640px block below). Phase 49: the archive upload form
(#archive-upload-form, replacing the phase-38 local-directory form)
reuses the git form's card + button rules VERBATIM — one form
language for both; its file input carries its own rules below. */
#git-source-form,
#local-source-form {
#archive-upload-form {
display: flex;
flex-wrap: wrap;
align-items: center;
@@ -1403,11 +1404,10 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
padding: 0.9rem 1rem 1rem;
}
#git-source-form:focus-within,
#local-source-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); }
#archive-upload-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); }
#git-source-form > label,
#local-source-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; }
#git-source-url,
#local-source-path {
#archive-upload-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; }
#git-source-url {
flex: 1;
min-width: 14rem;
min-height: 44px;
@@ -1419,12 +1419,10 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
border-radius: var(--radius-sm);
padding: 0.45rem 0.7rem;
}
#git-source-url::placeholder,
#local-source-path::placeholder { color: var(--ink-soft); }
#git-source-url:focus-visible,
#local-source-path:focus-visible { outline-offset: 0; border-color: var(--brand); }
#git-source-url::placeholder { color: var(--ink-soft); }
#git-source-url:focus-visible { outline-offset: 0; border-color: var(--brand); }
#git-source-add,
#local-source-add {
#archive-upload-btn {
display: inline-flex;
align-items: center;
justify-content: center;
@@ -1439,9 +1437,58 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
cursor: pointer;
}
#git-source-add:hover:not(:disabled),
#local-source-add:hover:not(:disabled) { background: #7d88f5; }
#archive-upload-btn:hover:not(:disabled) { background: #7d88f5; }
#git-source-add:disabled,
#local-source-add:disabled { opacity: 0.6; cursor: wait; }
#archive-upload-btn:disabled { opacity: 0.6; cursor: wait; }
/* Phase 49: the upload form's file control — mono-ish, on-surface, a
>=44px touch target, :focus-visible via the global 3px outline rule
(offset zeroed + brand border, exactly like the git URL input). The
chosen filename renders in mono; the selector button keeps a plain
surface chip. */
#archive-upload-file {
flex: 1;
min-width: 14rem;
min-height: 44px;
font-family: var(--mono);
font-size: 0.88rem;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
padding: 0.3rem 0.5rem;
}
#archive-upload-file:focus-visible { outline-offset: 0; border-color: var(--brand); }
#archive-upload-file::file-selector-button {
min-height: 34px;
margin-right: 0.6rem;
padding: 0.3rem 0.9rem;
border: 1px solid var(--line);
border-radius: var(--radius-sm);
background: var(--surface);
color: var(--ink); /* 13.8:1 on surface */
font: inherit;
font-size: 0.85rem;
font-weight: 600;
cursor: pointer;
}
#archive-upload-file::file-selector-button:hover { border-color: var(--brand); color: var(--brand-ink); }
/* The upload's success line (role=status): the sync-result shape
("2 added · 1 pruned") — ink-soft on surface (>=4.5:1), the dashed
hint-box border marks it as a result, not an error; it drops onto
its own row under the input like the error line. */
#archive-upload-result {
flex-basis: 100%;
margin: 0;
color: var(--ink-soft);
background: var(--surface);
border: 1px dashed var(--line);
border-radius: var(--radius-sm);
padding: 0.45rem 0.8rem;
font-size: 0.85rem;
font-weight: 600;
}
/* The add form's inline error (role=alert): the err pair (9.1:1);
flex-basis 100% drops it onto its own row under the input. */
@@ -1829,7 +1876,7 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
font-weight: 700;
letter-spacing: 0.08em;
text-transform: uppercase;
color: var(--brand-ink); /* #a5b4fc on --surface ≈8.7:1 */
color: var(--brand-ink); /* #fca5a5 on --surface ≈9.0:1 */
}
.doc-summary-text {
margin: 0;
@@ -2187,10 +2234,12 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.auth-link { padding: 0.4rem 0.3rem; }
.auth-label { display: none; }
.auth-link svg { display: block; }
/* Phase 46 UX revision: sign-out moves into the hamburger dropdown
on mobile — hide the bar copy, show the dropdown copy. */
/* Phase 46 UX revision: sign-out & sign-in move into the hamburger
dropdown on mobile — hide the bar copies, show the dropdown copies. */
.sign-out-mobile { display: inline-flex; }
#sign-out-btn { display: none !important; }
.sign-in-mobile { display: inline-flex; }
#sign-in-link { display: none !important; }
.nav-toggle { margin-left: auto; }
/* Dropdown-style sign-out: full-width row, error palette, label visible. */
#app-nav .sign-out-btn {
@@ -2217,6 +2266,33 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
flex: 0 0 auto;
}
#app-nav .sign-out-btn .auth-label { display: inline; }
/* Dropdown-style sign-in: matches sign-out row styling. */
#app-nav .sign-in-mobile {
display: flex;
width: 100%;
align-items: center;
gap: 0.6rem;
text-align: left;
padding: 0.75rem 1.25rem;
font-size: 1rem;
border: 0;
border-radius: 0;
background: transparent;
color: var(--err-ink);
font-weight: 600;
text-decoration: none;
min-height: 48px;
}
#app-nav .sign-in-mobile:hover {
background: var(--err-bg);
color: var(--err-ink);
}
#app-nav .sign-in-mobile svg {
width: 18px;
height: 18px;
flex: 0 0 auto;
}
#app-nav .sign-in-mobile .auth-label { display: inline; }
.steering-note { padding: 0.3rem 0.3rem 0.3rem 0.7rem; }
.app-main > .steering-panel { width: calc(100% - 1.8rem); }
.tune-btn { min-height: 44px; }
@@ -2248,18 +2324,18 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.doc-modal-meta { padding-inline: 0.9rem; }
.doc-modal-content { padding: 0.75rem 0.9rem 1.25rem; }
.composer { padding: 0.5rem; }
/* Phase 35 (phase 38: + the local directory form): the add forms
stack like the other cards — label, full-width mono input,
full-width button; the table wrapper's horizontal scroll already
covers long URLs/paths. */
/* Phase 35 (phase 49: + the archive upload form): the add forms
stack like the other cards — label, full-width input, full-width
button; the table wrapper's horizontal scroll already covers long
URLs/paths. */
#git-source-form,
#local-source-form { flex-direction: column; align-items: stretch; }
#archive-upload-form { flex-direction: column; align-items: stretch; }
#git-source-form > label,
#local-source-form > label { white-space: normal; }
#archive-upload-form > label { white-space: normal; }
#git-source-url,
#local-source-path { min-width: 0; }
#archive-upload-file { min-width: 0; }
#git-source-add,
#local-source-add { width: 100%; }
#archive-upload-btn { width: 100%; }
.footer-inner { flex-direction: column; gap: 0.2rem; text-align: center; }
main { padding-bottom: env(safe-area-inset-bottom, 0); }
/* Sync button goes icon-only on mobile; the label hides, aria-label
frontend/git-sources.html
+28 -30
View File
@@ -174,33 +174,29 @@
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
</form>
<!-- Phase 38: the second add form — "Local directory": an
existing directory on the server (NOT a git repo), walked
directly by Sync / import_docs. The SAME never-stale-button
+ inline-error pattern as the git form (PLAN §7.4): the
button disables + relabels "Adding…" while the POST is out
and recovers on success AND failure; on success the input
clears and the list re-fetches (the new row lands with the
Local badge). A missing/relative path 422s with the path
named inline (paths are not secrets, unlike git URLs). -->
<form id="local-source-form">
<label for="local-source-path">Add a local directory</label>
<input
id="local-source-path"
name="path"
type="text"
maxlength="2000"
autocomplete="off"
placeholder="~/Notes"
required
>
<button type="submit" id="local-source-add">Add directory</button>
<p class="git-source-error" id="local-source-error" role="alert" hidden></p>
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form
replaces the phase-38 local-directory form — an uploaded
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and
scanned immediately; the same filename replaces the source in
place (no new folder, no duplicate row). The file control is
labeled (visible <label for=…> — WCAG input-label rule); the
button runs the §7.4 never-stale lifecycle ("Uploading…"
while the POST is out, restored on success AND failure);
non-2xx shows the server detail inline (role=alert), 200
shows the sync-style counts (role=status). -->
<form id="archive-upload-form">
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
<input id="archive-upload-file" name="file" type="file"
accept=".tar,.tar.gz,.tgz,.zip" required>
<button type="submit" id="archive-upload-btn">Upload &amp; scan</button>
<p class="git-source-error" id="archive-upload-error" role="alert" hidden></p>
<p class="git-source-result" id="archive-upload-result" role="status"
aria-live="polite" hidden></p>
</form>
<div class="table-wrap" id="git-sources-table-wrap" role="region" aria-label="Sources" tabindex="0">
<table class="git-sources-table" id="git-sources-table">
<caption class="visually-hidden">Sources the Sync button imports — git repositories it clones and local directories it walks</caption>
<caption class="visually-hidden">Sources the Sync button imports — git repositories it clones, local directories it walks, and uploaded archives (unpacked under the upload directory)</caption>
<thead>
<tr>
<th scope="col">Source</th>
@@ -219,14 +215,16 @@
<!-- Scope boundary (phase locked decision): adding/removing a
source does NOT clone or prune — the Sync button performs
that. The hint says so (phase 38: git + local together,
files removed from a source pruned). -->
that; the phase-49 upload is the exception (it unpacks and
scans in place, and a same-name re-upload replaces the
source in place). -->
<p class="git-source-hint" id="git-sources-hint" role="note">
Sync clones/pulls the git repos and imports the local
directories together (files removed from a source are
pruned). Run the sync to clone/pull the git repos and import
the local directories — removing a source
prunes its documents from the index on the next sync.
Uploads unpack and scan immediately — re-uploading the same
filename replaces that source in place (no new folder, no
duplicate row). The Sync button still imports the git
checkouts and local directories together (files removed from
a source are pruned) — removing a source prunes its documents
from the index on the next sync.
</p>
</div>
<!-- Polite live region: the screen-reader confirmation for list
pyproject.toml
+4
View File
@@ -23,6 +23,10 @@ dependencies = [
# itsdangerous — an OPTIONAL starlette extra ("full") since starlette 1.x,
# so the app declares it directly (narrower than starlette[full]).
"itsdangerous>=2.2,<3.0",
# Phase 49: FastAPI's multipart parser for the archive upload form
# (POST /api/git-sources/upload) — an A2 FastAPI implementation detail,
# not a new architectural anchor (phase 49 locked decisions).
"python-multipart>=0.0.9,<0.1",
]
[dependency-groups]
tests/e2e/test_archive_upload_sources.py
+557
View File
@@ -0,0 +1,557 @@
"""Phase 49 story E2E (Playwright): archive upload sources.
Story: ``.agent/user_stories/archive-upload-sources.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov
The story gate for the **archive upload** form on the admin Sources page
(``/git-sources.html``, phase 49 — the phase-38 "Add a local directory"
form is gone, replaced by this form): an uploaded ``.tar``/``.tar.gz``/
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
(name = filename minus the archive suffix), the ``git_sources`` row is
upserted (``kind='local'``, no duplicates), and the source is **scanned
synchronously in the request** (single-source ``import_sources`` with
``prune=True`` + the change-gated overview refresh) — the real pipeline,
against the deterministic mock LLM (no real models, no network beyond
the app itself).
The archives are **built in-test** with Python's ``tarfile`` over
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
``BETA-…`` / ``GAMMA-…``) and are always named
``e2e-upload.tar.gz`` — so the source name is ``e2e-upload`` and
re-uploading under the same filename exercises the in-place replace
(one folder, one row, dropped files pruned from the KB). ``v1`` holds
``alpha.md`` + ``beta.md``; ``v2`` (same basename) modifies ``alpha``,
drops ``beta``, adds ``gamma``.
Per-module app env (the conftest pattern, module-scoped — as in
``test_git_sources_admin.py`` / ``test_local_directory_sources.py``):
``BOR_UPLOAD_DIR`` points at a scratch dir the suite can inspect from
the host (the app runs on the same machine), and
``BOR_GIT_SOURCES`` is forced empty so the dev ``.env``'s fallback URL
never renders as an env row on the (initially empty) table.
Contract under test:
* the **swap** (task 03): the phase-38 local form is gone (count 0);
the upload form is in its place with the labeled file input (accept
= the four archive extensions), the "Upload & scan" button, and the
hint explains unpack/scan + in-place replace;
* **upload → scan → list** (§7.4 never-stale): the button shows
"Uploading…" while the POST is in flight (the request is held in the
browser via ``page.route`` so the in-flight state is deterministic),
then restores; the result line shows the added count; the list gains
exactly one row for ``e2e-upload`` with the **Local** badge;
``GET /api/docs`` lists both sentinel files under source
``e2e-upload``; the RAG catalog (``/sources.html``) shows them;
* **re-upload, same filename** → in-place replace: the result line
shows the prune, the list still has exactly ONE ``e2e-upload`` row
(no duplicate), the KB shows the changed ``alpha`` + the new
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
only the new archive's files;
* **bad file** → inline 422 (role=alert) naming the accepted formats,
button restored, the file selection kept, the list unchanged, and a
subsequent good upload still works (the form is not wedged);
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
manager (and thus the upload form) stays hidden, and
``POST /api/git-sources/upload`` is 403.
Test → story mapping (Playwright Mapping Rule):
1. ``test_form_swapped``
2. ``test_upload_scans_and_lists``
3. ``test_reupload_replaces_in_place``
4. ``test_bad_file_inline_error``
5. ``test_anonymous_gate``
"""
from __future__ import annotations
import io
import os
import re
import subprocess
import sys
import tarfile
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
SOURCES_URL = "/sources.html"
#: The archive basename (both versions) — the source/folder name is the
#: filename minus the archive suffix (the phase's locked naming rule).
SOURCE_NAME = "e2e-upload"
#: v1: two sentinel docs. v2 (same filename): alpha CHANGED, beta DROPPED,
#: gamma ADDED — the in-place-replace subject.
ALPHA_SENTINEL_V1 = "ALPHA-TOKEN-v1-7f31"
ALPHA_SENTINEL_V2 = "ALPHA-TOKEN-v2-8b42"
BETA_SENTINEL_V1 = "BETA-TOKEN-v1-2c90"
GAMMA_SENTINEL_V2 = "GAMMA-TOKEN-v2-5e44"
V1_FILES: dict[str, str] = {
"alpha.md": (
"# Alpha note\n"
"\n"
"First version of the alpha note — it changes in v2.\n"
f"\nMarker: {ALPHA_SENTINEL_V1}\n"
),
"beta.md": (
"# Beta note\n"
"\n"
"Only present in v1 — v2 drops it (the prune subject).\n"
f"\nMarker: {BETA_SENTINEL_V1}\n"
),
}
V2_FILES: dict[str, str] = {
"alpha.md": (
"# Alpha note\n"
"\n"
"Second version of the alpha note — modified in place.\n"
f"\nMarker: {ALPHA_SENTINEL_V2}\n"
),
"gamma.md": (
"# Gamma note\n"
"\n"
"Brand new in v2 — the add subject of the re-upload.\n"
f"\nMarker: {GAMMA_SENTINEL_V2}\n"
),
}
#: The scan runs the full pipeline against the mock LLM (models probe +
#: embed batch + per-doc summaries + the change-gated overview) —
#: generous, like the sync suites; no client-side hard timeout.
UPLOAD_TIMEOUT_MS = 90_000
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
def _build_targz(path: Path, files: dict[str, str]) -> Path:
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
with tarfile.open(path, "w:gz") as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return path
@pytest.fixture(scope="module")
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
host-side assertions inspect (the app server runs on the same
machine). The app creates it on the first upload."""
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
@pytest.fixture(scope="module")
def tarball_v1(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""v1 — in its OWN subdirectory so v2 can reuse the same basename
(``e2e-upload.tar.gz``): the in-place-replace identity IS the
filename, and ``set_input_files`` sends the path's basename."""
root = tmp_path_factory.mktemp("bor_archive_v1")
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V1_FILES)
@pytest.fixture(scope="module")
def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""v2 — same basename as v1 (a different parent dir)."""
root = tmp_path_factory.mktemp("bor_archive_v2")
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
upload_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The real app under test — per-module env: uploads unpack into a
scratch dir and the env git list is forced empty (the dev ``.env``'s
``BOR_GIT_SOURCES`` must not render as env rows on the initially
empty table). No sync is triggered here — the upload's own scan is
the pipeline under test."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
env["BOR_GIT_SOURCES"] = ""
# Phase 49: unpack uploads into the suite's scratch dir (host-
# inspectable) and keep the (unused) git checkouts out of the dev
# location.
env["BOR_UPLOAD_DIR"] = str(upload_dir)
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern): the
upload's counts and every ``/api/docs`` assertion must be this
test's own doing. The E2E suites share one Postgres, and a leftover
git_sources row or document would corrupt the row-count and doc-list
assertions (and a leftover document under the same source name would
survive the re-upload's single-source prune)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
"""``GET /api/docs`` as the signed-in page → sorted (source, path)
pairs (the admin cookie rides the browser context)."""
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200, r.text
return sorted((d["source"], d["path"]) for d in r.json()["documents"])
def _upload_via_page(page: Page, archive: Path) -> str:
"""Pick the archive, submit the form, and wait for the result line
(the 200 path) — returns its text. The failing path is asserted
explicitly by the bad-file test, so any non-result outcome here is
a test error."""
page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result")
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
text = result.text_content()
assert text is not None
return text
def _hold_upload_request(page: Page, hold_s: float) -> None:
"""Intercept the upload POST and hold the REQUEST in the browser for
``hold_s`` seconds before letting it reach the server. While it is
held, the page's fetch is guaranteed pending — so the §7.4 in-flight
state (disabled button, "Uploading…" label) is observable
deterministically instead of racing the mock LLM's fast scan."""
def handle(route: Any) -> None:
time.sleep(hold_s)
route.continue_()
page.route("**/api/git-sources/upload", handle)
# ---------------------------------------------------------------------------
# 1. The swap: local form out, upload form in
# ---------------------------------------------------------------------------
def test_form_swapped(page: Page, app_url: str, db_ready: None) -> None:
"""The phase-38 "Add a local directory" form is GONE and the archive
upload form stands in its place: visible file input (accept = the
four archive extensions), the "Upload & scan" button, and a hint
that explains the unpack/scan + in-place-replace semantics."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# The phase-38 local form is gone (phase 49 replaced it)…
expect(page.locator("#local-source-form")).to_have_count(0)
expect(page.locator("#local-source-path")).to_have_count(0)
expect(page.locator("#local-source-add")).to_have_count(0)
# …and the upload form is in its place, visible with its parts.
expect(page.locator("#archive-upload-form")).to_be_visible()
file_input = page.locator("#archive-upload-file")
expect(file_input).to_be_visible()
accept = file_input.get_attribute("accept") or ""
for ext in (".tar", ".tar.gz", ".tgz", ".zip"):
assert ext in accept, f"accept={accept!r} is missing {ext!r}"
btn = page.locator("#archive-upload-btn")
expect(btn).to_be_visible()
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
# The error/result lines ship (hidden) with the right roles.
assert page.locator("#archive-upload-error").get_attribute("role") == "alert"
result = page.locator("#archive-upload-result")
assert result.get_attribute("role") == "status"
expect(result).to_be_hidden()
# The hint explains unpack/scan + in-place replace (task 03).
hint = page.locator("#git-sources-hint")
expect(hint).to_be_visible()
expect(hint).to_contain_text("unpack")
expect(hint).to_contain_text("scan")
expect(hint).to_contain_text("in place")
# ---------------------------------------------------------------------------
# 2. Upload → scan → list (the §7.4 in-flight state, the counts, the
# Local row, the KB, the RAG catalog)
# ---------------------------------------------------------------------------
def test_upload_scans_and_lists(
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
) -> None:
"""One real upload through the page: while the POST is in flight the
button is disabled and reads "Uploading…"; on the 200 it restores,
the result line shows the added count (2), the file input clears,
the list gains exactly ONE row for ``e2e-upload`` with the Local
badge, ``/api/docs`` lists both sentinel files under the source, and
the RAG catalog shows them where the admin expects them."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
btn = page.locator("#archive-upload-btn")
result = page.locator("#archive-upload-result")
# Hold the upload request in the browser: the in-flight state below
# cannot race the (fast) mock-LLM scan while it is held.
_hold_upload_request(page, hold_s=0.8)
page.set_input_files("#archive-upload-file", str(tarball_v1))
btn.click()
# In flight (§7.4): disabled + relabeled, no result yet.
expect(btn).to_be_disabled()
expect(btn).to_have_text("Uploading…")
expect(result).to_be_hidden()
# The request goes out, the server unpacks + scans (mock LLM) and
# answers 200 → the result line shows the added count.
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text("2 added")
# Never stale: the button restored on success and the input cleared.
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
expect(page.locator("#archive-upload-file")).to_have_value("")
# The list gained exactly one row — for the source, with the Local
# badge and the full unpacked path in the mono cell.
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
expect(row).to_have_count(1)
expect(row.locator("span.git-source-kind")).to_have_text("Local")
expect(row.locator("td.git-source-url-cell code")).to_have_text(
str(upload_dir / SOURCE_NAME)
)
# The KB: both sentinel files, under the source name e2e-upload.
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "beta.md")]
# The RAG catalog (admin sees it): both docs, under the source.
page.goto(app_url + SOURCES_URL)
expect(page.locator("#docs-tbody tr")).to_have_count(2)
expect(page.locator("#docs-tbody tr", has_text="alpha.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text="beta.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text=SOURCE_NAME)).to_have_count(2)
# ---------------------------------------------------------------------------
# 3. Re-upload, same filename → in-place replace (no duplicate row,
# dropped file pruned, changed/new file indexed)
# ---------------------------------------------------------------------------
def test_reupload_replaces_in_place(
page: Page,
app_url: str,
db_ready: None,
tarball_v1: Path,
tarball_v2: Path,
upload_dir: Path,
) -> None:
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
result line shows the prune, the list still has exactly ONE
``e2e-upload`` row (the row count for that source is invariant — no
duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
and NOT the dropped ``beta``, and the on-disk folder holds only the
new archive's files."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Baseline: v1 through the page (200 → "2 added", one row).
assert _upload_via_page(page, tarball_v1) == "2 added"
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
# Re-upload v2 — SAME basename, different parent dir (the file
# input's selection is replaced wholesale).
assert _upload_via_page(page, tarball_v2) is not None
result = page.locator("#archive-upload-result")
expect(result).to_have_text(re.compile(r"\d+ pruned"))
# No duplicate: exactly ONE row for that source (and one row total).
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# The registry agrees: one kind=local row, the unpacked path.
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert [(s["kind"], s["path"]) for s in body["sources"]] == [
("local", str(upload_dir / SOURCE_NAME))
]
# The KB: gamma + the CHANGED alpha, NOT the dropped beta.
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "gamma.md")]
# …and the indexed alpha is the v2 one (in-place replace, proven in
# the KB, not just the filesystem).
content = page.request.get(
f"{app_url}/api/documents/content?source={SOURCE_NAME}&path=alpha.md"
)
assert content.status == 200, content.text
assert ALPHA_SENTINEL_V2 in content.json()["content"]
assert ALPHA_SENTINEL_V1 not in content.json()["content"]
# The on-disk folder holds ONLY v2's files (the swap replaced the
# whole folder — no stale v1 file survived).
folder = upload_dir / SOURCE_NAME
assert {p.name for p in folder.iterdir()} == set(V2_FILES)
# ---------------------------------------------------------------------------
# 4. Bad file → inline 422; the form is not wedged
# ---------------------------------------------------------------------------
def test_bad_file_inline_error(
page: Page, app_url: str, db_ready: None, tarball_v1: Path, tmp_path: Path
) -> None:
"""A ``.txt`` through the file input: the role=alert line shows the
422 detail naming the accepted formats, the button restores, the
file selection is KEPT (the fix is one re-pick), the list is
unchanged — and a subsequent good upload still works (the form is
not wedged)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
bad = tmp_path / "notes.txt"
bad.write_text("I am not an archive.\n", encoding="utf-8")
error = page.locator("#archive-upload-error")
btn = page.locator("#archive-upload-btn")
page.set_input_files("#archive-upload-file", str(bad))
btn.click()
# The 422 detail inline (role=alert), naming the accepted formats.
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_contain_text("only .tar, .tar.gz, .tgz or .zip archives are accepted")
# Never stale + the selection kept + no result line + list unchanged.
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
# The selection is kept (the fix is one re-pick) — Chromium reports
# a fake path (``…/notes.txt``), so assert on the basename.
bad_value = page.locator("#archive-upload-file").input_value()
assert bad_value.endswith("notes.txt"), bad_value
expect(page.locator("#archive-upload-result")).to_be_hidden()
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
# The form is not wedged: a good upload right after still works.
assert _upload_via_page(page, tarball_v1) == "2 added"
expect(error).to_be_hidden()
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "beta.md")]
# ---------------------------------------------------------------------------
# 5. Anonymous: the gate, the hidden manager, the 403
# ---------------------------------------------------------------------------
def test_anonymous_gate(page: Page, app_url: str, db_ready: None) -> None:
"""Anonymous on the page: the sign-in gate shows, the manager (and
thus the upload form) stays hidden, and the upload route 403s
(``require_admin`` — A10)."""
page.set_default_timeout(30_000)
page.goto(app_url + GIT_SOURCES_URL)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
expect(page.locator("#sign-out-btn")).to_be_hidden()
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
# The manager is hidden — so is the upload form inside it.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#archive-upload-form")).to_be_hidden()
# The upload route 403s anonymous callers (require_admin runs before
# the multipart body is parsed — the body is a stand-in, the
# test_local_directory_sources.py pattern for this route).
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
assert r.status == 403
tests/e2e/test_local_directory_sources.py
+132 -99
View File
@@ -5,13 +5,27 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov
**Phase-49 rewrite (owner permission 2026-08-28): the page form this
story drove is GONE.** The "Add a local directory" form
(``#local-source-form``, phase 38) was removed from
``/git-sources.html`` and replaced by the archive upload form
(``#archive-upload-form``, phase 49 — its own story E2E is
``test_archive_upload_sources.py``). The phase-38 ACCEPTANCE stands —
a local directory can still be registered, imported, pruned, and
removed — so this suite now adds local sources through the
**authenticated API** the page's own JS no longer calls
(``POST /api/git-sources {"kind": "local", "path": …}``, the contract
the page used to wrap; the admin cookie rides the browser context via
``page.request``). Do NOT "restore" a form here: adding a plain
directory by hand is an API-only operation now.
The story gate for the **local directory** kind of the admin-managed
source registry (phase 38): the admin adds an existing, non-git
directory on the same admin page as the git repos (phase 35), and the
real Sync button (phase 32) imports it — with add-time fail-loud
validation (a missing/relative path is rejected inline, naming the
path) and union pruning (a file deleted from the directory leaves the
index on the next sync; removing the row stops it being a source).
source registry (phase 38): the admin registers an existing, non-git
directory (API — see above), and the real Sync button (phase 32)
imports it — with add-time fail-loud validation (a missing/relative
path is rejected with 422, naming the path in the JSON detail) and
union pruning (a file deleted from the directory leaves the index on
the next sync; removing the row stops it being a source).
The fixture is a **host temp dir** (``tmp_path_factory`` — the app
server runs on the same host, so the path is visible to it) containing
@@ -30,27 +44,27 @@ decision — local directories are DB-registered, no env var), so an
empty table means "no sources configured" until the admin adds the
directory through the real page.
Contract under test:
Contract under test (local adds are API-driven — phase-49 rewrite):
* anonymous: the sign-in gate (the phase-16/35 ``#git-sources-gate``
pattern), the manager hidden (list + BOTH add forms inert), NO
``/api/git-sources`` call, and 403 on the source routes + the sync
trigger (the phase-35 regression assertions, A10);
* admin: a missing path (``/nonexistent/bor-e2e``) 422s inline naming
the path with no row added and the button never stale; the host temp
dir adds (201 → row with the **Local** badge + the full path in a
mono cell, input cleared, button re-enabled); the same path again
409s inline ("already exists", path named) with no second row;
* admin: the header **Sync** button (the phase-32 lifecycle, "Syncing…"
→ "Synced HH:MM") imports the fixture file — it appears in
``GET /api/docs`` (and its sentinel is in ``GET
pattern), the manager hidden (list + git add form + archive upload
form inert), NO ``/api/git-sources`` call, and 403 on the source
routes (incl. the phase-49 upload route) + the sync trigger (the
phase-35 regression assertions, A10);
* admin: a missing path (``/nonexistent/bor-e2e``) 422s with the JSON
detail NAMING the path ("not a directory") and no row added; the
host temp dir adds (201 → the row renders with the **Local** badge +
the full path in a mono cell once the list re-renders); the same
path again 409s ("already exists", path named) with no second row;
* admin: the **Sync** button on the Sources page (the phase-32
lifecycle, "Syncing…" → "Synced HH:MM") imports the fixture file —
it appears in ``GET /api/docs`` (and its sentinel is in ``GET
/api/documents/content``); deleting the file and syncing again prunes
it (``pruned: 1``, gone from ``GET /api/docs`` — union prune); then
removing the row on the page makes it disappear (accept the confirm;
the empty state returns);
* the new local form's a11y basics (UI Structure Check, AGENTS.md rule
5): labeled input, role=alert error line, ≥44px target, 3px
focus-visible outline.
the empty state returns).
* (The phase-38 form's a11y assertions moved with the form: the
upload form's UI Structure Check lives in the phase-49 story E2E.)
Test → story mapping (Playwright Mapping Rule):
1. ``test_anonymous_soft_gate_and_403s``
@@ -223,20 +237,28 @@ def _admin_git_sources_page(page: Page, app_url: str) -> None:
expect(page.locator("#git-sources-content")).to_be_visible()
def _add_local_dir(page: Page, path: str) -> None:
"""Add a local directory through the real page form and wait for the
new row (the 201 → reload → row lifecycle of git-sources.js)."""
page.fill("#local-source-path", path)
page.click("#local-source-add")
expect(
page.locator("#git-sources-tbody tr", has_text=path)
).to_have_count(1, timeout=30_000)
def _add_local_dir_api(page: Page, app_url: str, path: str) -> None:
"""Register a local directory through the authenticated API
(phase-49 rewrite: the page form is gone — the ``kind=local``
POST contract the page used to wrap is unchanged, and the admin
cookie rides the browser context, cf. test_git_sources_admin.py).
``data`` with a dict is JSON-serialized by Playwright's Python API
(there is no ``json=`` kwarg — the JS API's shape is ``json``).
201 is the only success — the caller re-renders the page when it
needs the row in the table."""
r = page.request.post(
f"{app_url}/api/git-sources", data={"kind": "local", "path": path}
)
assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}"
def _click_sync(page: Page) -> None:
def _click_sync(page: Page, app_url: str) -> None:
"""The phase-32 button lifecycle: click → disabled + "Syncing…" →
"Synced HH:MM" (re-enabled — never stale). The server status poll
underneath is what the 2 s UI loop observes."""
underneath is what the 2 s UI loop observes. The button's home is
the Sources page (owner rework 2026-08-28 — it left the shared
navbar), so the helper visits it first."""
page.goto(app_url + "/sources.html")
btn = page.locator("#sync-btn")
expect(btn).to_be_visible()
btn.click()
@@ -277,10 +299,11 @@ def _docs(page: Page, app_url: str) -> list[dict[str, Any]]:
def test_anonymous_soft_gate_and_403s(
page: Page, app_url: str, db_ready: None
) -> None:
"""The phase-16/35 gate on this page (regression through the phase-38
form): anonymous visitors see the sign-in gate and a fully hidden
manager (list + git form + local form), the page never calls the
admin API, and every admin route 403s (A10)."""
"""The phase-16/35 gate on this page (regression through the
phase-49 form swap): anonymous visitors see the sign-in gate and a
fully hidden manager (list + git form + upload form — the phase-38
local form is gone), the page never calls the admin API, and every
admin route 403s (A10)."""
page.set_default_timeout(30_000)
api_calls: list[str] = []
@@ -299,18 +322,26 @@ def test_anonymous_soft_gate_and_403s(
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
# The manager is absent/inert: list, BOTH add forms, env note — all
# inside the hidden #git-sources-content.
# The manager is absent/inert: list, git add form, the phase-49
# archive upload form, env note — all inside the hidden
# #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#local-source-form")).to_be_hidden()
expect(page.locator("#archive-upload-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The phase-38 local-directory form is GONE (phase-49 rewrite) —
# the upload form replaced it.
expect(page.locator("#local-source-form")).to_have_count(0)
expect(page.locator("#local-source-path")).to_have_count(0)
expect(page.locator("#local-source-add")).to_have_count(0)
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers (the phase-35 assertions):
# all three source routes, for BOTH kinds, plus the sync trigger.
# …and the API 403s anonymous callers (the phase-35 assertions,
# plus the phase-49 upload route): all four source routes, for
# BOTH kinds, plus the sync trigger.
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
@@ -330,59 +361,66 @@ def test_anonymous_soft_gate_and_403s(
).status
== 403
)
# The phase-49 upload route 403s too (require_admin runs before the
# multipart body is ever parsed — the body here is a stand-in JSON
# payload, not a real multipart upload).
assert (
page.request.post(
f"{app_url}/api/git-sources/upload", data={"file": ""}
).status
== 403
)
assert page.request.post(f"{app_url}/api/sync").status == 403
# ---------------------------------------------------------------------------
# 2. Admin: add validation (missing path, dir, duplicate) + local form
# a11y basics
# 2. Admin: add validation (missing path, dir, duplicate) — API-driven
# (phase-49 rewrite: the form this drove is gone)
# ---------------------------------------------------------------------------
def test_admin_add_missing_path_then_dir_then_duplicate(
page: Page, app_url: str, local_dir: Path, db_ready: None
) -> None:
"""Add-time fail-loud validation on the real page: a missing path
422s inline NAMING the path (no row, never-stale button, the input
survives for one edit); the host temp dir adds (row with the Local
badge + full path, input cleared); the same path again 409s inline
("already exists", path named, no second row)."""
"""Add-time fail-loud validation (phase-49 rewrite: the page form is
gone, so the same contract is asserted on the API body the page
used to render): a missing path 422s NAMING the path in the JSON
detail (no row added); the host temp dir adds (201 → the row renders
with the Local badge + full path once the list re-renders); the same
path again 409s ("already exists", path named, no second row)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
error = page.locator("#local-source-error")
add_btn = page.locator("#local-source-add")
# --- missing path: 422 naming it in the JSON detail, NO row --------
r = page.request.post(
f"{app_url}/api/git-sources", data={"kind": "local", "path": MISSING_PATH}
)
assert r.status == 422, r.text
detail = r.json()["detail"]
assert MISSING_PATH in detail, f"detail does not name the path: {detail!r}"
assert "not a directory" in detail
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert r.json()["sources"] == []
# --- missing path: inline 422 naming it, NO row, button recovers ---
page.fill("#local-source-path", MISSING_PATH)
add_btn.click()
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_contain_text(MISSING_PATH)
expect(error).to_contain_text("not a directory")
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add directory")
expect(page.locator("#local-source-path")).to_have_value(MISSING_PATH)
# --- the temp dir: 201 → the row appears with the Local badge ------
_add_local_dir(page, str(local_dir))
# --- the temp dir: 201 → the row renders with the Local badge ------
_add_local_dir_api(page, app_url, str(local_dir))
# The API add is invisible to the open page (its JS no longer adds
# local dirs) — re-render the list, exactly as a fresh visit would.
page.reload()
expect(page.locator("#git-sources-content")).to_be_visible(timeout=30_000)
row = page.locator("#git-sources-tbody tr", has_text=str(local_dir))
expect(row).to_have_count(1)
expect(row).to_have_count(1, timeout=30_000)
badge = row.locator("span.git-source-kind")
expect(badge).to_have_text("Local")
expect(badge).to_have_class(re.compile(r"\bis-local\b"))
# The mono cell carries the full path (rendered as text)…
# The mono cell carries the full path (rendered as text)...
expect(row.locator("td.git-source-url-cell code")).to_have_text(str(local_dir))
# …and the row's Remove button is labeled with the kind + path.
expect(row.locator(".git-source-remove")).to_have_attribute(
"aria-label", f"Remove local source: {local_dir}"
)
# The 201 cleared the input and re-enabled the button (never stale).
expect(page.locator("#local-source-path")).to_have_value("")
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add directory")
# The API agrees: kind=local with the stored (expanded) path.
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
@@ -392,26 +430,17 @@ def test_admin_add_missing_path_then_dir_then_duplicate(
(s["kind"], s["path"]) for s in body["sources"]
] == [("local", str(local_dir))]
# --- duplicate: inline 409 naming the path, NO second row -----------
page.fill("#local-source-path", str(local_dir))
add_btn.click()
expect(error).to_be_visible(timeout=30_000)
expect(error).to_contain_text("already exists")
expect(error).to_contain_text(str(local_dir))
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add directory")
expect(page.locator("#local-source-path")).to_have_value(str(local_dir))
# --- the new form's a11y basics (UI Structure Check, AGENTS.md 5) ---
expect(page.get_by_label("Add a local directory")).to_have_count(1)
box = add_btn.bounding_box()
assert box is not None and box["height"] >= 44, f"target too small: {box}"
page.focus("#local-source-path")
outline = page.evaluate(
"() => getComputedStyle(document.querySelector('#local-source-path')).outlineWidth"
# --- duplicate: 409 naming the path, NO second row -----------------
r = page.request.post(
f"{app_url}/api/git-sources", data={"kind": "local", "path": str(local_dir)}
)
assert outline == "3px", f"focus-visible outline missing: {outline!r}"
assert r.status == 409, r.text
detail = r.json()["detail"]
assert "already exists" in detail
assert str(local_dir) in detail
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert len(r.json()["sources"]) == 1 # the row was NOT duplicated
# ---------------------------------------------------------------------------
@@ -423,20 +452,22 @@ def test_admin_add_missing_path_then_dir_then_duplicate(
def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
page: Page, app_url: str, local_dir: Path, db_ready: None
) -> None:
"""The phase-32 button drives the phase-38 pipeline: the header Sync
imports the local directory's fixture file (GET /api/docs shows it,
the sentinel is in its content); deleting the file and syncing again
prunes it (``pruned: 1`` — prune over the union); then removing the
row on the page makes it disappear (the empty state returns)."""
"""The phase-32 button drives the phase-38 pipeline: the Sources-
page Sync imports the local directory's fixture file (GET /api/docs
shows it, the sentinel is in its content); deleting the file and
syncing again prunes it (``pruned: 1`` — prune over the union); then
removing the row on the page makes it disappear (the empty state
returns). The local add is API-driven (phase-49 rewrite)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Fresh registry (the autouse fixture truncated it) — add the source
# through the real page, then run the sync lifecycle against it.
_add_local_dir(page, str(local_dir))
# Fresh registry (the autouse fixture truncated it) — register the
# source via the API (the page form is gone), then run the sync
# lifecycle against it.
_add_local_dir_api(page, app_url, str(local_dir))
# --- run 1: the real sync walks the local dir and imports the file -
_click_sync(page)
_click_sync(page, app_url)
body = _wait_sync_done(page, app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == 1, body["detail"]
@@ -457,7 +488,7 @@ def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
# --- run 2: file deleted → the next sync prunes it (union prune) ---
(local_dir / FIXTURE_REL).unlink()
_click_sync(page)
_click_sync(page, app_url)
body = _wait_sync_done(page, app_url)
assert body["state"] == "success", body
assert body["detail"]["pruned"] == 1, body["detail"]
@@ -468,6 +499,8 @@ def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
)
# --- remove the row: accept the confirm → it disappears ------------
# Back on the manager page (the sync clicks visited the Sources page).
page.goto(app_url + GIT_SOURCES_URL)
removes: list[str] = []
page.on(
"request",
tests/integration/test_git_sources_upload.py
+788
View File
@@ -0,0 +1,788 @@
"""Integration: the admin archive-upload API (phase 49, task 02).
Real Postgres (``podman compose up -d db``); the upload dir is pointed
at a fresh tmp dir per test by monkeypatching the router's
``get_settings`` (the ``test_git_sources_api.py`` pattern — the dev
``.env`` never leaks in), and the scan uses the deterministic
in-process ``FakeEmbedder`` (``test_sync_api.py``'s ``_real_llm``
pattern — real import, no network).
Contract under test:
* anonymous → 403 ``{"detail": "admin only"}`` (the router's
``require_admin`` covers the new route);
* name/format gate → 422: a non-archive extension names the accepted
formats; a ``..`` / separator / empty-stem name (including a bare
``tar.gz``) is rejected with the task-01 message — and the upload
dir is never created for a rejected name (control characters never
reach the app: the multipart transport percent-encodes them — the
task-01 branch for them is covered in ``test_archive_upload.py``);
* one upload at a time → 409 ``an upload is already in progress``
(while a run is in flight — the first request holds the flag through
its scan — and while the module-level flag seam is held);
* streaming cap → 413 naming the ``upload_max_mb`` cap; the temp
``.upload`` file is removed (no stray ``.`` files in the upload dir);
* unpack safety → 422 (zip-slip member, tar symlink escape, corrupt
archive, zero-entry archive) — and a failed upload **never** touches
the previous folder, row, or KB of an earlier good upload (the
no-partial-state locked decision, asserted explicitly);
* happy path → 200 with the sync-detail count keys (``source`` +
``files/added/updated/unchanged/pruned/errors/chunks/overview``), a
``kind=local`` row under the tmp ``upload_dir`` (the NOT-NULL
``url`` column carries the path — the phase-38 convention), the
unpacked folder, the KB via ``GET /api/docs``, and the per-upload
log line (PLAN §9 / AGENTS.md rule 10);
* re-upload, same name → the swap replaces the folder in place, the
row is NOT duplicated (``added_at`` preserved), dropped files are
pruned from the KB, added/changed files are indexed;
* an archive with only non-A9 files is a VALID replacement (indexes
nothing, prunes the previous docs, no overview refresh);
* dead models → 503 with the sanitized model-unavailable message; the
folder and row are already committed (the next sync/re-upload
retries idempotently).
``git_sources`` / ``documents`` / ``chunks`` / ``kb_overview`` are
global state: truncated around every test.
"""
from __future__ import annotations
import asyncio
import io
import logging
import os
import re
import tarfile
import threading
import zipfile
from collections.abc import Iterator
from pathlib import Path
from typing import Literal
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import git_sources as git_sources_api
from app.config import Settings
from app.db import SessionLocal
from app.main import app as fastapi_app
from app.models import GitSource
from app.rag.archive_upload import ArchiveUploadError
from app.rag.importer import ImportSummary
from app.rag.llm import ModelUnavailableError
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
@pytest.fixture(autouse=True)
def clean_git_sources(db: Session) -> Iterator[None]:
"""The stored list is global state: reset around every test."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def clean_documents(db: Session) -> Iterator[None]:
"""The happy path writes ``documents``/``chunks`` and (via the
change-gated overview refresh) ``kb_overview`` — global, truncated
around every test."""
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
db.commit()
@pytest.fixture()
def upload_client() -> Iterator[TestClient]:
"""Admin-signed client (the context-manager form keeps one event
loop across requests — the in-flight 409 test needs the first
request's run to survive while the second lands)."""
with TestClient(fastapi_app) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
yield client
def _point_at(monkeypatch: pytest.MonkeyPatch, upload_dir: Path, upload_max_mb: int = 512) -> None:
"""Fresh settings on the router's module: the tmp upload dir and
(optionally) a shrunk cap — the dev ``.env`` never leaks in."""
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
upload_dir=str(upload_dir),
upload_max_mb=upload_max_mb,
),
)
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
"""The pipeline's ``LLMClient`` becomes the deterministic
in-process ``FakeEmbedder`` (real import, no network); it also
implements ``embed_one``/``chat``, so the phase-41 probe passes."""
monkeypatch.setattr(git_sources_api, "LLMClient", lambda: FakeEmbedder())
#: tarfile write modes used by the tests (uncompressed + gzip).
_TAR_WRITE_MODES = Literal["w", "w:gz"]
def _tarball(path: Path, files: dict[str, str], compress: _TAR_WRITE_MODES = "w:gz") -> Path:
with tarfile.open(path, compress) as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return path
def _zip(path: Path, files: dict[str, str]) -> Path:
with zipfile.ZipFile(path, "w") as zf:
for rel, content in files.items():
zf.writestr(rel, content)
return path
def _targz_bytes(files: dict[str, str]) -> bytes:
"""A tarball in memory (no temp file on disk)."""
buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode="w:gz") as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return buf.getvalue()
def _post(client: TestClient, filename: str, payload: bytes):
"""One upload request (multipart, the page's exact shape)."""
return client.post(
"/api/git-sources/upload",
files={"file": (filename, payload, "application/octet-stream")},
)
def _row(db: Session, path: str) -> GitSource | None:
db.expire_all()
return db.scalar(select(GitSource).where(GitSource.path == path))
def _docs(client: TestClient) -> list[tuple[str, str]]:
body = client.get("/api/docs").json()
return [(d["source"], d["path"]) for d in body["documents"]]
def _count_rows(db: Session) -> int:
return db.execute(text("SELECT count(*) FROM git_sources")).scalar_one()
def _good_upload(
client: TestClient, name: str = "safe", files: dict[str, str] | None = None
) -> None:
"""A 200 upload of a two-sentinel tarball — the baseline state the
no-partial-state tests protect."""
payload_files = files or {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
r = _post(client, f"{name}.tar.gz", _targz_bytes(payload_files))
assert r.status_code == 200, r.text
def _assert_previous_intact(
client: TestClient, db: Session, upload_dir: Path, name: str, files: dict[str, str]
) -> None:
"""The no-partial-state locked decision, asserted explicitly: after a
FAILED upload the previous folder's content, the row (same
``added_at``), and the KB are all exactly as the good upload left
them — and no temp file survived."""
folder = upload_dir / name
assert {p.name: p.read_text(encoding="utf-8") for p in folder.iterdir()} == files
assert _row(db, str(folder)) is not None # the row is still there
assert _count_rows(db) == 1 # …and no second row appeared
assert _docs(client) == [(name, "alpha.md"), (name, "bravo.md")]
assert [p.name for p in upload_dir.iterdir()] == [name] # no stray temp
# --- anonymous -------------------------------------------------------------
def test_anonymous_upload_gets_403(client: TestClient, db: Session) -> None:
r = _post(client, "homelab.tar.gz", b"not an archive at all")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
assert _count_rows(db) == 0
# --- name / format gate -----------------------------------------------------
def test_non_archive_extension_gets_422_naming_formats(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
r = _post(upload_client, "notes.txt", b"hello world")
assert r.status_code == 422
assert r.json()["detail"] == "only .tar, .tar.gz, .tgz or .zip archives are accepted"
assert _count_rows(db) == 0
# The name gate runs before the dir is created — nothing on disk.
assert not uploads.exists()
@pytest.mark.parametrize(
"filename",
[
"../evil.zip", # ``..`` + separator
"sub/dir.tar.gz", # separator
"tar.gz", # bare suffix → empty stem
".tar.gz", # empty stem
# (control characters in the filename are percent-encoded by the
# multipart transport before the app ever sees them — the
# ``archive_source_name`` branch for them is covered in
# tests/unit/test_archive_upload.py)
],
)
def test_unsafe_name_gets_422(
upload_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
filename: str,
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
r = _post(upload_client, filename, b"content")
assert r.status_code == 422, f"{filename!r} must be rejected: {r.text}"
assert r.json()["detail"] != "only .tar, .tar.gz, .tgz or .zip archives are accepted"
assert _count_rows(db) == 0
assert not uploads.exists()
# --- one at a time (409) ----------------------------------------------------
def test_second_upload_while_one_is_in_flight_returns_409(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""The flag is held from the name gate through the scan response:
while the first run is inside its (blocked) scan, the second upload
is 409 — and after the first finishes, uploads are accepted again."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
archive = _tarball(tmp_path / "homelab.tar.gz", {"alpha.md": "# Alpha\nx\n"})
started = threading.Event()
release = threading.Event()
class BlockingImport:
async def __call__(self, sources, llm, **kwargs) -> ImportSummary:
started.set() # the run is in flight (the flag was set earlier)
await asyncio.to_thread(release.wait, 15.0)
return ImportSummary(files=1, unchanged=1)
monkeypatch.setattr(git_sources_api, "import_sources", BlockingImport())
first = {}
def do_first() -> None:
# A separate client/cookie jar: the two requests run on separate
# TestClient portals; only the module-level flag is shared.
with TestClient(fastapi_app) as c:
assert c.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
first["r"] = _post(c, "homelab.tar.gz", archive.read_bytes())
thread = threading.Thread(target=do_first)
thread.start()
try:
assert started.wait(15.0), "first upload did not reach its scan"
with TestClient(fastapi_app) as second:
assert second.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
r = _post(second, "homelab.tar.gz", archive.read_bytes())
assert r.status_code == 409
assert r.json() == {"detail": "an upload is already in progress"}
finally:
release.set()
thread.join(20)
assert first["r"].status_code == 200, first["r"].text
# The flag was released: the next upload goes through for real.
with TestClient(fastapi_app) as third:
assert third.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
r = _post(third, "homelab.tar.gz", archive.read_bytes())
assert r.status_code == 200, r.text
def test_upload_refused_while_flag_held(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""The smallest test seam: the module-level flag itself. Held →
409, nothing happens (no dir, no row)."""
_point_at(monkeypatch, tmp_path / "uploads")
monkeypatch.setattr(git_sources_api, "_upload_in_progress", True)
r = _post(upload_client, "a.tar.gz", b"x")
assert r.status_code == 409
assert r.json() == {"detail": "an upload is already in progress"}
assert _count_rows(db) == 0
assert not (tmp_path / "uploads").exists()
# --- in-flight upload must not hold the request session (regression, -----
# --- phase 49 task 03) ----------------------------------------------------
def test_in_flight_upload_does_not_block_a_concurrent_truncate(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""While the scan is in flight, a concurrent TRUNCATE of the KB +
registry tables (the E2E isolation-fixture pattern) must complete.
Regression for the phase-49 task-03 deadlock: the handler used to
keep its request ``db`` session open across the scan, and the
uncommitted ``_commit_new`` refresh transaction held
``git_sources`` locks for the whole scan. A concurrent TRUNCATE
(documents locked, git_sources pending) then deadlocked with the
scan's own document locks — a cycle spanning three connections
that Postgres's detector cannot see, hanging the app and the test
run forever. The handler now releases the session before the scan;
this pins it: with the scan held in flight, the TRUNCATE completes
well inside its 5 s ``lock_timeout`` (without the fix it times out
with a lock-not-available error)."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
archive = _tarball(tmp_path / "homelab.tar.gz", {"alpha.md": "# Alpha\nx\n"})
started = threading.Event()
release = threading.Event()
class BlockingImport:
async def __call__(self, sources, llm, **kwargs) -> ImportSummary:
started.set() # the run is in flight (the flag was set earlier)
await asyncio.to_thread(release.wait, 15.0)
return ImportSummary(files=1, unchanged=1)
monkeypatch.setattr(git_sources_api, "import_sources", BlockingImport())
first = {}
def do_first() -> None:
with TestClient(fastapi_app) as c:
assert c.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
first["r"] = _post(c, "homelab.tar.gz", archive.read_bytes())
thread = threading.Thread(target=do_first)
thread.start()
try:
assert started.wait(15.0), "first upload did not reach its scan"
# The E2E isolation TRUNCATE, exactly as the story fixtures run
# it — must complete while the scan is held in flight.
with SessionLocal() as tr, tr.begin():
tr.execute(text("SET LOCAL lock_timeout = '5s'"))
tr.execute(
text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources")
)
finally:
release.set()
thread.join(20)
# The scan finished once released — the 200 stands (the TRUNCATE ran
# after the row upsert and dropped it; the next re-upload is
# idempotent, and the autouse fixtures reset the tables anyway).
assert first["r"].status_code == 200, first["r"].text
# --- streaming cap (413) ----------------------------------------------------
def test_oversized_upload_gets_413_and_leaves_no_temp(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads, upload_max_mb=1)
# 2 MiB of incompressible bytes with a 1 MiB cap → 413 naming the cap.
big = os.urandom(2 * 1024 * 1024)
r = _post(upload_client, "big.zip", big)
assert r.status_code == 413
assert "1 MiB" in r.json()["detail"]
assert _count_rows(db) == 0
assert uploads.exists() # the dir was created before streaming
assert list(uploads.iterdir()) == [] # the temp .upload file is gone
# Boundary: EXACTLY the cap is not 413 (the check is strictly >) —
# the stream completes and the garbage bytes fail at unpack instead.
r = _post(upload_client, "exact.zip", os.urandom(1024 * 1024))
assert r.status_code == 422
assert "could not unpack the archive" in r.json()["detail"]
assert list(uploads.iterdir()) == []
# --- unpack safety: failed uploads leave the previous state intact ----------
def test_zip_slip_archive_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
evil = _zip(tmp_path / "safe.zip", {"../evil.txt": "pwned"})
r = _post(upload_client, "safe.zip", evil.read_bytes())
assert r.status_code == 422
assert "traversal" in r.json()["detail"]
assert not (tmp_path / "evil.txt").exists() # the escape never landed
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_tar_symlink_escape_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
evil = tmp_path / "safe.tar"
with tarfile.open(evil, "w") as tf:
info = tarfile.TarInfo("link")
info.type = tarfile.SYMTYPE
info.linkname = "/etc/passwd"
tf.addfile(info)
r = _post(upload_client, "safe.tar", evil.read_bytes())
assert r.status_code == 422
assert "escape" in r.json()["detail"]
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_corrupt_archive_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
# A truncated zip (the EOCD is cut off) is not a zip and not a tar.
good_zip = _zip(tmp_path / "good.zip", {"alpha.md": "# Alpha\nx\n"})
good_bytes = good_zip.read_bytes()
truncated = good_bytes[: len(good_bytes) // 2]
r = _post(upload_client, "safe.zip", truncated)
assert r.status_code == 422
assert "could not unpack the archive" in r.json()["detail"]
# A zero-byte "archive" fails the same way.
r = _post(upload_client, "safe.tar", b"")
assert r.status_code == 422
assert "could not unpack the archive" in r.json()["detail"]
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_swap_failure_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""A rename failure in ``swap_in`` (OS error) → 422 with its
message; the previous folder/row/KB are untouched and no temp
survives (the handler's finally cleans the temp sibling)."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
def failing_swap(new_dir: Path, final_dir: Path) -> None:
raise ArchiveUploadError("could not replace the previous folder")
monkeypatch.setattr(git_sources_api, "swap_in", failing_swap)
r = _post(upload_client, "safe.tar.gz", _targz_bytes({"alpha.md": "# A\nx\n"}))
assert r.status_code == 422
assert r.json()["detail"] == "could not replace the previous folder"
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_zero_entry_archive_gets_422(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""A completely empty archive (zero entries) is 422 — both
containers — with no folder, row, or temp file left behind."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
empty_targz = tmp_path / "empty.tar.gz"
with tarfile.open(empty_targz, "w:gz"):
pass
r = _post(upload_client, "empty.tar.gz", empty_targz.read_bytes())
assert r.status_code == 422
assert r.json()["detail"] == "the archive contains no files"
empty_zip = tmp_path / "empty.zip"
with zipfile.ZipFile(empty_zip, "w"):
pass
r = _post(upload_client, "empty.zip", empty_zip.read_bytes())
assert r.status_code == 422
assert r.json()["detail"] == "the archive contains no files"
assert _count_rows(db) == 0
assert not (uploads / "empty").exists()
assert list(uploads.iterdir()) == []
# --- happy path -------------------------------------------------------------
def test_happy_path_tar_gz(
upload_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {"alpha.md": "# Alpha\nfirst sentinel\n", "bravo.md": "# Bravo\nsecond sentinel\n"}
payload = _tarball(tmp_path / "homelab.tar.gz", files).read_bytes()
with caplog.at_level(logging.INFO, logger="app.api.git_sources"):
r = _post(upload_client, "homelab.tar.gz", payload)
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {
"source", "files", "added", "updated", "unchanged", "pruned", "errors", "chunks",
"overview",
}
assert body["source"] == "homelab" # filename minus the archive suffix
assert body["files"] == 2
assert body["added"] == 2
assert body["updated"] == 0
assert body["unchanged"] == 0
assert body["pruned"] == 0
assert body["errors"] == 0
assert body["chunks"] >= 2
assert body["overview"] is True # the KB changed → the overview refreshed
# Unpacked under the tmp upload dir, dotfile temps cleaned up.
folder = uploads / "homelab"
assert folder.is_dir()
assert {p.name: p.read_text(encoding="utf-8") for p in folder.iterdir()} == files
assert [p.name for p in uploads.iterdir()] == ["homelab"]
# One kind=local row; the NOT-NULL url column carries the path.
row = _row(db, str(folder))
assert row is not None
assert row.kind == "local"
assert row.url == str(folder)
assert row.path == str(folder)
assert _count_rows(db) == 1
# The KB lists both files under the source name.
assert _docs(upload_client) == [("homelab", "alpha.md"), ("homelab", "bravo.md")]
# The per-upload log line (PLAN §9 / AGENTS.md rule 10).
lines = [rec.getMessage() for rec in caplog.records if rec.getMessage().startswith("upload: ")]
assert len(lines) == 1, lines
match = re.match(
r"^upload: name=homelab file=homelab\.tar\.gz bytes_in=(\d+) files=2 added=2 "
r"updated=0 unchanged=0 pruned=0 errors=0 overview=True total_ms=\d+$",
lines[0],
)
assert match, lines[0]
assert int(match.group(1)) == len(payload) # the compressed bytes in
@pytest.mark.parametrize(
("filename", "builder"),
[
("notes.zip", "zip"),
("plain.tar", "tar"),
("tgz.tgz", "targz"),
],
)
def test_happy_path_other_accepted_formats(
upload_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
filename: str,
builder: str,
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {"alpha.md": "# Alpha\nsentinel\n"}
make = {"zip": _zip, "tar": lambda p, f: _tarball(p, f, "w"), "targz": _tarball}[builder]
payload = make(tmp_path / filename, files).read_bytes()
r = _post(upload_client, filename, payload)
assert r.status_code == 200, r.text
body = r.json()
expected_name = {"notes.zip": "notes", "plain.tar": "plain", "tgz.tgz": "tgz"}[filename]
assert body["source"] == expected_name
assert body["added"] == 1
assert _docs(upload_client) == [(expected_name, "alpha.md")]
assert _count_rows(db) == 1
# --- re-upload, same name: in-place replace ---------------------------------
def test_reupload_same_name_replaces_in_place(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
v1 = {
"alpha.md": "# Alpha\nv1 content\n",
"bravo.md": "# Bravo\nv1 content\n",
"charlie.md": "# Charlie\nv1 content\n",
}
r = _post(upload_client, "homelab.tar.gz", _tarball(tmp_path / "v1.tar.gz", v1).read_bytes())
assert r.status_code == 200, r.text
assert r.json()["added"] == 3
row_before = _row(db, str(uploads / "homelab"))
assert row_before is not None
added_at_before = row_before.added_at
# v2: alpha changed, bravo dropped, delta new.
v2 = {"alpha.md": "# Alpha\nv2 CHANGED content\n", "delta.md": "# Delta\nbrand new\n"}
r = _post(upload_client, "homelab.tar.gz", _tarball(tmp_path / "v2.tar.gz", v2).read_bytes())
assert r.status_code == 200, r.text
body = r.json()
assert body["source"] == "homelab"
assert body["files"] == 2
assert body["added"] == 1 # delta.md
assert body["updated"] == 1 # alpha.md (hash changed)
assert body["unchanged"] == 0
assert body["pruned"] == 2 # bravo.md + charlie.md left the folder → pruned
assert body["overview"] is True
# Exactly ONE row for the path, and its added_at survived (the
# upsert left the existing row alone).
assert _count_rows(db) == 1
row_after = _row(db, str(uploads / "homelab"))
assert row_after is not None
assert row_after.id == row_before.id
assert row_after.added_at == added_at_before
# The folder holds ONLY the new archive's files (full replacement —
# no stale files from v1), and the KB mirrors it.
folder = uploads / "homelab"
assert {p.name: p.read_text(encoding="utf-8") for p in folder.iterdir()} == v2
assert [p.name for p in uploads.iterdir()] == ["homelab"]
assert _docs(upload_client) == [("homelab", "alpha.md"), ("homelab", "delta.md")]
def test_non_a9_archive_is_a_valid_replacement(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""An archive with only non-A9 files is a valid replacement: the
swap happens, the scan indexes nothing, prune removes the source's
docs, and the overview is NOT refreshed (no KB change)."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
r = _post(
upload_client,
"notes.tar.gz",
_tarball(tmp_path / "a.tar.gz", {"readme.md": "# Readme\nv1\n"}).read_bytes(),
)
assert r.status_code == 200, r.text
assert _docs(upload_client) == [("notes", "readme.md")]
r = _post(
upload_client,
"notes.tar.gz",
_tarball(tmp_path / "b.tar.gz", {"binary.bin": "not an importable format"}).read_bytes(),
)
assert r.status_code == 200, r.text
body = r.json()
assert body["files"] == 0 # nothing matches the A9 filter
assert body["added"] == 0
assert body["updated"] == 0
assert body["pruned"] == 1 # readme.md left the KB
assert body["overview"] is False # added + updated == 0 → no refresh
assert {p.name for p in (uploads / "notes").iterdir()} == {"binary.bin"}
assert _count_rows(db) == 1
assert _docs(upload_client) == []
# --- fail-fast models (503) ---------------------------------------------------
def test_models_down_gets_503_and_leaves_folder_and_row(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""The folder and row are committed BEFORE the model probe: a dead
endpoint answers 503 (sanitized — credentials masked) and the next
sync/re-upload retries idempotently; the scan never ran."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
async def dead_probe(llm: object) -> None:
raise ModelUnavailableError(
"The embedding model ('embed') is not available — check the model "
"endpoint and retry. (embeddings request to "
"https://user:secret@aipi.reeseapps.com/v1 failed: connection refused)"
)
monkeypatch.setattr(git_sources_api, "check_models", dead_probe)
r = _post(
upload_client,
"homelab.tar.gz",
_tarball(tmp_path / "h.tar.gz", {"alpha.md": "# Alpha\nx\n"}).read_bytes(),
)
assert r.status_code == 503
detail = r.json()["detail"]
assert "The embedding model ('embed') is not available" in detail
assert "*****@aipi.reeseapps.com" in detail # the sanitizer masked the credentials
assert "user:secret" not in detail
assert "connection refused" in detail # the reason survives
# The folder and row are already committed (idempotent retry path).
folder = uploads / "homelab"
assert folder.is_dir()
row = _row(db, str(folder))
assert row is not None
assert row.kind == "local"
assert row.url == str(folder)
# The scan never ran: no docs, no stray temps.
assert _docs(upload_client) == []
assert [p.name for p in uploads.iterdir()] == ["homelab"]
tests/unit/test_archive_upload.py
+579
View File
@@ -0,0 +1,579 @@
"""Unit: phase 49 upload settings + the safe archive unpack utility.
Covers ``app.rag.archive_upload`` end to end:
* the name-derivation matrix (suffix stripping incl. the compound
``.tar.gz``, case handling, and the rejection list — ``..``,
separators, control chars, empty stems);
* safe extraction — valid zip/tar.gz archives extract byte-identically,
while zip-slip, absolute members, escaping symlinks/hardlinks, device
members, and the extracted-byte cap all raise
:class:`ArchiveUploadError` **and** leave no partial target behind;
* ``swap_in`` — fresh, in-place replace with full content replacement
(no interleave, no ``.old-`` leftovers), and restore-on-failure;
* the two new settings (``upload_dir`` / ``upload_max_mb``) with
env overrides and the fail-loud ``<= 0`` validator.
"""
from __future__ import annotations
import io
import os
import stat
import tarfile
import zipfile
from pathlib import Path
from typing import Any
import pytest
from pydantic import ValidationError
from app.config import Settings
from app.rag.archive_upload import (
ARCHIVE_SUFFIXES,
ArchiveUploadError,
_link_target_resolved, # pyright: ignore[reportPrivateUsage]
_member_dest, # pyright: ignore[reportPrivateUsage]
archive_source_name,
swap_in,
unpack_archive,
)
def _settings(**kwargs: Any) -> Settings:
"""Build Settings without reading a .env file (deterministic tests)."""
kwargs.setdefault("_env_file", None)
return Settings(**kwargs) # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
# ---------------------------------------------------------------------------
# Settings (phase 49, task 01)
# ---------------------------------------------------------------------------
def test_upload_settings_defaults(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("BOR_UPLOAD_DIR", raising=False)
monkeypatch.delenv("BOR_UPLOAD_MAX_MB", raising=False)
s = _settings()
# Deliberately separate from the git checkouts (``sources_dir``).
assert s.upload_dir == "~/bor-sources/uploads"
assert s.sources_dir == "~/bor-sources"
assert s.upload_max_mb == 512
def test_upload_settings_env_overrides(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("BOR_UPLOAD_DIR", "/data/bor/uploads")
monkeypatch.setenv("BOR_UPLOAD_MAX_MB", "128")
s = _settings()
assert s.upload_dir == "/data/bor/uploads"
assert s.upload_max_mb == 128
@pytest.mark.parametrize("value", ["0", "-1", "-512"])
def test_upload_max_mb_rejects_zero_and_negative(
monkeypatch: pytest.MonkeyPatch, value: str
) -> None:
"""``<= 0`` would reject every upload — the validator fails loudly at
startup (the ``agent_max_rounds`` pattern)."""
monkeypatch.setenv("BOR_UPLOAD_MAX_MB", value)
with pytest.raises(ValidationError, match="upload_max_mb"):
_settings()
# ---------------------------------------------------------------------------
# archive_source_name — the derivation matrix
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("filename", "expected"),
[
("homelab.tar.gz", "homelab"),
("notes.tgz", "notes"),
("a.zip", "a"),
("x.tar", "x"),
# Case-insensitive suffix match; the stem keeps its case (the name
# becomes a folder name on a case-sensitive Linux FS).
("upper.TAR.GZ", "upper"),
("Homelab.Zip", "Homelab"),
# ONE compound strip — never a double strip (``.tar.gz`` matches
# before ``.tar`` would; there is no ``.gz`` suffix at all).
("a.tar.gz", "a"),
# Only the LAST suffix is stripped.
("a.zip.zip", "a.zip"),
("y.tar.tgz", "y.tar"),
("café.tar", "café"),
],
)
def test_archive_source_name_strips_one_suffix(filename: str, expected: str) -> None:
assert archive_source_name(filename) == expected
def test_archive_suffixes_are_longest_first() -> None:
# ``.tar.gz`` must precede ``.tar`` or ``a.tar.gz`` would yield
# ``a.tar``.
assert ARCHIVE_SUFFIXES == (".tar.gz", ".tgz", ".zip", ".tar")
@pytest.mark.parametrize(
"filename",
[
"", # empty
"tar.gz", # empty stem (bare suffix, no leading dot)
"tgz", # empty stem (bare suffix)
"zip", # empty stem (bare suffix)
".zip", # empty stem (hidden-file suffix)
"..tar.gz", # ``..`` after the strip
"..", # no suffix, ``..`` stem
".", # no suffix, ``.`` stem
"a/b.tar", # forward-separator path
"a\\b.tar", # backslash path
"a\tb.zip", # tab control character
"a\x00b.tar", # NUL control character
"a\x1bb.tar", # escape-sequence control character
],
)
def test_archive_source_name_rejects_unsafe_names(filename: str) -> None:
with pytest.raises(ArchiveUploadError):
archive_source_name(filename)
# ---------------------------------------------------------------------------
# Archive builders (deterministic, in-memory)
# ---------------------------------------------------------------------------
def _make_zip(
path: Path,
files: dict[str, bytes] | None = None,
dirs: tuple[str, ...] = (),
extra_attrs: dict[str, int] | None = None,
) -> None:
with zipfile.ZipFile(path, "w") as zf:
for name in dirs:
info = zipfile.ZipInfo(name if name.endswith("/") else name + "/")
info.external_attr = (0o40755 << 16)
zf.writestr(info, b"")
for name, data in (files or {}).items():
info = zipfile.ZipInfo(name)
info.external_attr = (0o100644 << 16)
zf.writestr(info, data)
for name, attr in (extra_attrs or {}).items():
info = zipfile.ZipInfo(name)
info.external_attr = attr
zf.writestr(info, b"")
def _make_tar(
path: Path,
spec: list[tuple[str, bytes, str]],
gz: bool = False,
) -> None:
"""``spec`` entries: ``(name, payload, kind)`` with kind one of
``f`` (file), ``d`` (dir), ``sym`` (symlink, payload = target),
``lnk`` (hardlink, payload = target), ``chr`` (char device),
``fifo`` (FIFO)."""
mode = "w:gz" if gz else "w"
with tarfile.open(path, mode) as tf:
for name, payload, kind in spec:
ti = tarfile.TarInfo(name)
if kind == "f":
ti.size = len(payload)
ti.mode = 0o644
tf.addfile(ti, io.BytesIO(payload))
elif kind == "d":
ti.type = tarfile.DIRTYPE
ti.mode = 0o755
tf.addfile(ti)
elif kind in ("sym", "lnk"):
ti.type = tarfile.SYMTYPE if kind == "sym" else tarfile.LNKTYPE
ti.linkname = payload.decode()
tf.addfile(ti)
elif kind == "chr":
ti.type = tarfile.CHRTYPE
ti.devmajor, ti.devminor = 1, 3
tf.addfile(ti)
elif kind == "fifo":
ti.type = tarfile.FIFOTYPE
tf.addfile(ti)
# ---------------------------------------------------------------------------
# unpack_archive — valid archives extract byte-identically
# ---------------------------------------------------------------------------
def test_unpack_zip_valid_nested(tmp_path: Path) -> None:
archive = tmp_path / "notes.zip"
_make_zip(
archive,
files={"a/b.txt": b"hello\n", "c.txt": b"x" * 50, "a/deep/n.md": b"# deep"},
dirs=("a", "a/deep"),
)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "a/b.txt").read_bytes() == b"hello\n"
assert (target / "c.txt").read_bytes() == b"x" * 50
assert (target / "a/deep/n.md").read_bytes() == b"# deep"
assert (target / "a").is_dir()
assert (target / "a/deep").is_dir()
def test_unpack_zip_unknown_mode_entries_treated_as_files(tmp_path: Path) -> None:
"""Windows-made / ``writestr``-style zips carry no Unix mode bits
(external_attr 0) — decided by the member name, not rejected."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
zf.writestr("plain.txt", b"data") # default external_attr = 0
archive = tmp_path / "plain.zip"
archive.write_bytes(buf.getvalue())
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "plain.txt").read_bytes() == b"data"
def test_unpack_tar_gz_valid(tmp_path: Path) -> None:
archive = tmp_path / "notes.tar.gz"
_make_tar(
archive,
[
("dir/", b"", "d"),
("dir/a.md", b"# hi", "f"),
("top.txt", b"t", "f"),
],
gz=True,
)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "dir/a.md").read_bytes() == b"# hi"
assert (target / "top.txt").read_bytes() == b"t"
assert (target / "dir").is_dir()
def test_unpack_plain_tar_valid(tmp_path: Path) -> None:
"""``r:*`` handles uncompressed ``.tar`` too."""
archive = tmp_path / "notes.tar"
_make_tar(archive, [("only.txt", b"plain tar", "f")], gz=False)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "only.txt").read_bytes() == b"plain tar"
def test_unpack_internal_symlink_and_hardlink_allowed(tmp_path: Path) -> None:
"""Links that stay INSIDE the unpack directory are fine (the spec
resolves the target and rejects only escapes)."""
archive = tmp_path / "links.tar"
_make_tar(
archive,
[
("sub/", b"", "d"),
("sub/data.txt", b"inner", "f"),
("alias", b"sub/data.txt", "sym"),
("dup", b"sub/data.txt", "lnk"),
],
gz=False,
)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert target.joinpath("alias").is_symlink()
assert target.joinpath("alias").read_bytes() == b"inner"
assert target.joinpath("dup").read_bytes() == b"inner"
def test_extracted_bytes_exactly_at_cap_passes(tmp_path: Path) -> None:
"""The cap bounds EXCEEDING bytes — landing exactly on it is OK."""
archive = tmp_path / "exact.zip"
_make_zip(archive, files={"f.txt": b"a" * 10})
target = tmp_path / "out"
unpack_archive(archive, target, 10)
assert (target / "f.txt").read_bytes() == b"a" * 10
# ---------------------------------------------------------------------------
# unpack_archive — every guard raises AND leaves no partial target
# ---------------------------------------------------------------------------
def _assert_no_partial(target: Path) -> None:
assert not target.exists() and not target.is_symlink()
def test_member_dest_rejects_empty_name(tmp_path: Path) -> None:
target = tmp_path / "t"
target.mkdir()
with pytest.raises(ArchiveUploadError, match="empty name"):
_member_dest("", target)
def test_member_dest_dot_name_is_the_target_itself(tmp_path: Path) -> None:
"""``.`` resolves to the target itself — inside, so allowed (the
containment check's equality arm)."""
target = tmp_path / "t"
target.mkdir()
assert _member_dest(".", target) == target
def test_member_dest_rejects_resolution_escape(tmp_path: Path) -> None:
"""Defense in depth (the resolve-based containment check): a symlink
already inside the target that points OUT makes any member routed
through it escape once resolved."""
target = tmp_path / "t"
target.mkdir()
(target / "sneaky").symlink_to(tmp_path / "outside")
with pytest.raises(ArchiveUploadError, match="escapes"):
_member_dest("sneaky/evil.txt", target)
def test_link_target_rejects_empty_target(tmp_path: Path) -> None:
with pytest.raises(ArchiveUploadError, match="empty target"):
_link_target_resolved("", tmp_path / "d", tmp_path / "t")
def test_unpack_tar_extractfile_none_is_corrupt(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A REG member whose data tarfile cannot hand back (``extractfile``
→ None) is a corrupt member — rejected, partial state cleaned."""
archive = tmp_path / "corrupt.tar"
_make_tar(archive, [("f.txt", b"x", "f")], gz=False)
monkeypatch.setattr(tarfile.TarFile, "extractfile", lambda self, member: None)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="corrupt archive member"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_zip_slip_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "evil.zip"
_make_zip(archive, files={"../evil.txt": b"evil", "ok.txt": b"ok"})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="traversal"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
@pytest.mark.parametrize("name", ["/etc/x", "C:\\evil", "\\\\server\\share"])
def test_unpack_zip_absolute_member_rejected_and_cleaned(tmp_path: Path, name: str) -> None:
archive = tmp_path / "abs.zip"
_make_zip(archive, files={name: b"no"})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="absolute"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_zip_symlink_entry_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "slink.zip"
_make_zip(
archive,
extra_attrs={"link": (stat.S_IFLNK | 0o777) << 16},
)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="symlink"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_zip_non_regular_entry_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "dev.zip"
_make_zip(archive, extra_attrs={"dev": (stat.S_IFCHR | 0o644) << 16})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="non-regular"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_absolute_member_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "abs.tar"
_make_tar(archive, [("/etc/x", b"no", "f")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="absolute"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_symlink_escape_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "slink.tar"
_make_tar(archive, [("link", b"/etc/passwd", "sym")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="escapes"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_relative_symlink_escape_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "slink2.tar"
_make_tar(
archive,
[("sub/", b"", "d"), ("sub/escape", b"../../outside", "sym")],
gz=False,
)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="escapes"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_hardlink_escape_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "hlink.tar"
_make_tar(archive, [("hard", b"/etc/passwd", "lnk")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="escapes"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
@pytest.mark.parametrize("kind", ["chr", "fifo"])
def test_unpack_tar_device_and_fifo_rejected_and_cleaned(tmp_path: Path, kind: str) -> None:
archive = tmp_path / "dev.tar"
_make_tar(archive, [("dev", b"", kind)], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="device or FIFO"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
@pytest.mark.parametrize("kind", ["zip", "tar"])
def test_unpack_extracted_cap_exceeded_rejected_and_cleaned(tmp_path: Path, kind: str) -> None:
"""cap=10 with a 20-byte file → the cap (not the content) is named
and the partial tree is gone."""
archive = tmp_path / ("cap." + kind)
payload = b"b" * 20
if kind == "zip":
_make_zip(archive, files={"big.txt": payload})
else:
_make_tar(archive, [("big.txt", payload, "f")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="10-byte extraction cap"):
unpack_archive(archive, target, 10)
_assert_no_partial(target)
def test_unpack_cap_accumulates_across_members(tmp_path: Path) -> None:
"""6 + 6 bytes under a 10-byte cap: each file alone is under, the
total is not."""
archive = tmp_path / "acc.zip"
_make_zip(archive, files={"a.txt": b"a" * 6, "b.txt": b"b" * 6})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="extraction cap"):
unpack_archive(archive, target, 10)
_assert_no_partial(target)
def test_unpack_corrupt_archive_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "junk.bin"
archive.write_bytes(b"this is not an archive at all")
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="could not unpack"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_existing_target_rejected(tmp_path: Path) -> None:
archive = tmp_path / "ok.zip"
_make_zip(archive, files={"a.txt": b"1"})
target = tmp_path / "out"
target.mkdir()
with pytest.raises(ArchiveUploadError, match="already exists"):
unpack_archive(archive, target, 10_000)
# The pre-existing directory is left exactly as found.
assert target.is_dir() and not any(target.iterdir())
# ---------------------------------------------------------------------------
# swap_in — atomic in-place replacement
# ---------------------------------------------------------------------------
def test_swap_in_fresh(tmp_path: Path) -> None:
new = tmp_path / "new"
new.mkdir()
(new / "new.txt").write_text("new")
final = tmp_path / "final"
swap_in(new, final)
assert final.is_dir()
assert (final / "new.txt").read_text() == "new"
assert not new.exists()
def test_swap_in_replaces_existing_with_full_content(tmp_path: Path) -> None:
"""The previous content is fully gone, the new content complete —
no interleave — and no ``.old-`` sibling survives."""
final = tmp_path / "final"
final.mkdir()
(final / "old.txt").write_text("old")
(final / "keepdir/").mkdir()
(final / "keepdir" / "stale.txt").write_text("stale")
new = tmp_path / "new"
new.mkdir()
(new / "fresh.txt").write_text("fresh")
(new / "keepdir/").mkdir()
(new / "keepdir" / "v2.txt").write_text("v2")
swap_in(new, final)
assert not (final / "old.txt").exists()
assert not (final / "keepdir" / "stale.txt").exists()
assert (final / "fresh.txt").read_text() == "fresh"
assert (final / "keepdir" / "v2.txt").read_text() == "v2"
assert not new.exists()
assert not list(tmp_path.glob("final.old-*"))
def test_swap_in_restores_previous_folder_on_failure(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The second rename (new → final) fails: the previous folder comes
back intact, the new dir is cleaned, and the error is raised."""
final = tmp_path / "final"
final.mkdir()
(final / "old.txt").write_text("old")
new = tmp_path / "new"
new.mkdir()
(new / "fresh.txt").write_text("fresh")
real_rename = os.rename
def fake_rename(
src: str | os.PathLike[str], dst: str | os.PathLike[str], *args: Any, **kwargs: Any
) -> Any:
if Path(str(src)) == new: # the new → final rename fails
raise OSError("simulated swap failure")
return real_rename(src, dst, *args, **kwargs)
monkeypatch.setattr(os, "rename", fake_rename)
with pytest.raises(ArchiveUploadError, match="could not replace"):
swap_in(new, final)
# Previous folder intact, byte for byte.
assert (final / "old.txt").read_text() == "old"
# New dir cleaned, no orphaned .old sibling.
assert not new.exists()
assert not list(tmp_path.glob("final.old-*"))
def test_swap_in_double_failure_leaves_no_orphan(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Both renames fail (new→final AND the ``.old-``→final restore):
the best-effort path deletes the orphaned ``.old-`` sibling — the
previous folder is unrecoverable in this scenario, so at least
nothing is left mixed on disk."""
final = tmp_path / "final"
final.mkdir()
(final / "old.txt").write_text("old")
new = tmp_path / "new"
new.mkdir()
(new / "fresh.txt").write_text("fresh")
real_rename = os.rename
def fake_rename(
src: str | os.PathLike[str], dst: str | os.PathLike[str], *args: Any, **kwargs: Any
) -> Any:
s, d = Path(str(src)), Path(str(dst))
if s == new or d == final: # the swap rename and the restore both fail
raise OSError("simulated failure")
return real_rename(src, dst, *args, **kwargs)
monkeypatch.setattr(os, "rename", fake_rename)
with pytest.raises(ArchiveUploadError, match="could not replace"):
swap_in(new, final)
assert not new.exists()
assert not list(tmp_path.glob("final.old-*"))
uv.lock
Generated
+11
View File
@@ -62,6 +62,7 @@ dependencies = [
{ name = "pydantic" },
{ name = "pydantic-settings" },
{ name = "python-dotenv" },
{ name = "python-multipart" },
{ name = "sqlalchemy" },
{ name = "uvicorn", extra = ["standard"] },
]
@@ -88,6 +89,7 @@ requires-dist = [
{ name = "pydantic", specifier = ">=2.7,<3.0" },
{ name = "pydantic-settings", specifier = ">=2.3,<3.0" },
{ name = "python-dotenv", specifier = ">=1.0,<2.0" },
{ name = "python-multipart", specifier = ">=0.0.9,<0.1" },
{ name = "sqlalchemy", specifier = ">=2.0,<2.1" },
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.30,<1.0" },
]
@@ -896,6 +898,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0d/17/c5c6b53ddc18f297992099b3d9ec16c855c0ccc83263a21fe4d1c625ec6c/python_dotenv-1.2.3-py3-none-any.whl", hash = "sha256:904552145e8bfed22162c09dab1c2b9b54fefa7b23ba780f4f26ca0316b0f0d9", size = 22780, upload-time = "2026-08-16T16:54:52.473Z" },
]
[[package]]
name = "python-multipart"
version = "0.0.32"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/5b/42/55c32bb9b12693c092ad250a0e82edb5b31ddeda6eb772de5f308b3804ad/python_multipart-0.0.32.tar.gz", hash = "sha256:be54b7f3fa167bb83e4fcd936b887b708f4e57fe75911c02aebf53efaf8d938e", size = 46881, upload-time = "2026-06-04T16:18:58.647Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042, upload-time = "2026-06-04T16:18:57.319Z" },
]
[[package]]
name = "pyyaml"
version = "6.0.3"