Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place

Browse Source 
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
ducoterra
2026-08-28 15:57:59 -04:00
parent 872a07cee7
commit 03d26255c6
21 changed files with 3280 additions and 233 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/test_archive_upload_sources.py
+557
View File
@@ -0,0 +1,557 @@
"""Phase 49 story E2E (Playwright): archive upload sources.
Story: ``.agent/user_stories/archive-upload-sources.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov
The story gate for the **archive upload** form on the admin Sources page
(``/git-sources.html``, phase 49 — the phase-38 "Add a local directory"
form is gone, replaced by this form): an uploaded ``.tar``/``.tar.gz``/
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
(name = filename minus the archive suffix), the ``git_sources`` row is
upserted (``kind='local'``, no duplicates), and the source is **scanned
synchronously in the request** (single-source ``import_sources`` with
``prune=True`` + the change-gated overview refresh) — the real pipeline,
against the deterministic mock LLM (no real models, no network beyond
the app itself).
The archives are **built in-test** with Python's ``tarfile`` over
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
``BETA-…`` / ``GAMMA-…``) and are always named
``e2e-upload.tar.gz`` — so the source name is ``e2e-upload`` and
re-uploading under the same filename exercises the in-place replace
(one folder, one row, dropped files pruned from the KB). ``v1`` holds
``alpha.md`` + ``beta.md``; ``v2`` (same basename) modifies ``alpha``,
drops ``beta``, adds ``gamma``.
Per-module app env (the conftest pattern, module-scoped — as in
``test_git_sources_admin.py`` / ``test_local_directory_sources.py``):
``BOR_UPLOAD_DIR`` points at a scratch dir the suite can inspect from
the host (the app runs on the same machine), and
``BOR_GIT_SOURCES`` is forced empty so the dev ``.env``'s fallback URL
never renders as an env row on the (initially empty) table.
Contract under test:
* the **swap** (task 03): the phase-38 local form is gone (count 0);
the upload form is in its place with the labeled file input (accept
= the four archive extensions), the "Upload & scan" button, and the
hint explains unpack/scan + in-place replace;
* **upload → scan → list** (§7.4 never-stale): the button shows
"Uploading…" while the POST is in flight (the request is held in the
browser via ``page.route`` so the in-flight state is deterministic),
then restores; the result line shows the added count; the list gains
exactly one row for ``e2e-upload`` with the **Local** badge;
``GET /api/docs`` lists both sentinel files under source
``e2e-upload``; the RAG catalog (``/sources.html``) shows them;
* **re-upload, same filename** → in-place replace: the result line
shows the prune, the list still has exactly ONE ``e2e-upload`` row
(no duplicate), the KB shows the changed ``alpha`` + the new
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
only the new archive's files;
* **bad file** → inline 422 (role=alert) naming the accepted formats,
button restored, the file selection kept, the list unchanged, and a
subsequent good upload still works (the form is not wedged);
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
manager (and thus the upload form) stays hidden, and
``POST /api/git-sources/upload`` is 403.
Test → story mapping (Playwright Mapping Rule):
1. ``test_form_swapped``
2. ``test_upload_scans_and_lists``
3. ``test_reupload_replaces_in_place``
4. ``test_bad_file_inline_error``
5. ``test_anonymous_gate``
"""
from __future__ import annotations
import io
import os
import re
import subprocess
import sys
import tarfile
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
SOURCES_URL = "/sources.html"
#: The archive basename (both versions) — the source/folder name is the
#: filename minus the archive suffix (the phase's locked naming rule).
SOURCE_NAME = "e2e-upload"
#: v1: two sentinel docs. v2 (same filename): alpha CHANGED, beta DROPPED,
#: gamma ADDED — the in-place-replace subject.
ALPHA_SENTINEL_V1 = "ALPHA-TOKEN-v1-7f31"
ALPHA_SENTINEL_V2 = "ALPHA-TOKEN-v2-8b42"
BETA_SENTINEL_V1 = "BETA-TOKEN-v1-2c90"
GAMMA_SENTINEL_V2 = "GAMMA-TOKEN-v2-5e44"
V1_FILES: dict[str, str] = {
"alpha.md": (
"# Alpha note\n"
"\n"
"First version of the alpha note — it changes in v2.\n"
f"\nMarker: {ALPHA_SENTINEL_V1}\n"
),
"beta.md": (
"# Beta note\n"
"\n"
"Only present in v1 — v2 drops it (the prune subject).\n"
f"\nMarker: {BETA_SENTINEL_V1}\n"
),
}
V2_FILES: dict[str, str] = {
"alpha.md": (
"# Alpha note\n"
"\n"
"Second version of the alpha note — modified in place.\n"
f"\nMarker: {ALPHA_SENTINEL_V2}\n"
),
"gamma.md": (
"# Gamma note\n"
"\n"
"Brand new in v2 — the add subject of the re-upload.\n"
f"\nMarker: {GAMMA_SENTINEL_V2}\n"
),
}
#: The scan runs the full pipeline against the mock LLM (models probe +
#: embed batch + per-doc summaries + the change-gated overview) —
#: generous, like the sync suites; no client-side hard timeout.
UPLOAD_TIMEOUT_MS = 90_000
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
def _build_targz(path: Path, files: dict[str, str]) -> Path:
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
with tarfile.open(path, "w:gz") as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return path
@pytest.fixture(scope="module")
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
host-side assertions inspect (the app server runs on the same
machine). The app creates it on the first upload."""
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
@pytest.fixture(scope="module")
def tarball_v1(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""v1 — in its OWN subdirectory so v2 can reuse the same basename
(``e2e-upload.tar.gz``): the in-place-replace identity IS the
filename, and ``set_input_files`` sends the path's basename."""
root = tmp_path_factory.mktemp("bor_archive_v1")
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V1_FILES)
@pytest.fixture(scope="module")
def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""v2 — same basename as v1 (a different parent dir)."""
root = tmp_path_factory.mktemp("bor_archive_v2")
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
upload_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The real app under test — per-module env: uploads unpack into a
scratch dir and the env git list is forced empty (the dev ``.env``'s
``BOR_GIT_SOURCES`` must not render as env rows on the initially
empty table). No sync is triggered here — the upload's own scan is
the pipeline under test."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
env["BOR_GIT_SOURCES"] = ""
# Phase 49: unpack uploads into the suite's scratch dir (host-
# inspectable) and keep the (unused) git checkouts out of the dev
# location.
env["BOR_UPLOAD_DIR"] = str(upload_dir)
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern): the
upload's counts and every ``/api/docs`` assertion must be this
test's own doing. The E2E suites share one Postgres, and a leftover
git_sources row or document would corrupt the row-count and doc-list
assertions (and a leftover document under the same source name would
survive the re-upload's single-source prune)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
"""``GET /api/docs`` as the signed-in page → sorted (source, path)
pairs (the admin cookie rides the browser context)."""
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200, r.text
return sorted((d["source"], d["path"]) for d in r.json()["documents"])
def _upload_via_page(page: Page, archive: Path) -> str:
"""Pick the archive, submit the form, and wait for the result line
(the 200 path) — returns its text. The failing path is asserted
explicitly by the bad-file test, so any non-result outcome here is
a test error."""
page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result")
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
text = result.text_content()
assert text is not None
return text
def _hold_upload_request(page: Page, hold_s: float) -> None:
"""Intercept the upload POST and hold the REQUEST in the browser for
``hold_s`` seconds before letting it reach the server. While it is
held, the page's fetch is guaranteed pending — so the §7.4 in-flight
state (disabled button, "Uploading…" label) is observable
deterministically instead of racing the mock LLM's fast scan."""
def handle(route: Any) -> None:
time.sleep(hold_s)
route.continue_()
page.route("**/api/git-sources/upload", handle)
# ---------------------------------------------------------------------------
# 1. The swap: local form out, upload form in
# ---------------------------------------------------------------------------
def test_form_swapped(page: Page, app_url: str, db_ready: None) -> None:
"""The phase-38 "Add a local directory" form is GONE and the archive
upload form stands in its place: visible file input (accept = the
four archive extensions), the "Upload & scan" button, and a hint
that explains the unpack/scan + in-place-replace semantics."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# The phase-38 local form is gone (phase 49 replaced it)…
expect(page.locator("#local-source-form")).to_have_count(0)
expect(page.locator("#local-source-path")).to_have_count(0)
expect(page.locator("#local-source-add")).to_have_count(0)
# …and the upload form is in its place, visible with its parts.
expect(page.locator("#archive-upload-form")).to_be_visible()
file_input = page.locator("#archive-upload-file")
expect(file_input).to_be_visible()
accept = file_input.get_attribute("accept") or ""
for ext in (".tar", ".tar.gz", ".tgz", ".zip"):
assert ext in accept, f"accept={accept!r} is missing {ext!r}"
btn = page.locator("#archive-upload-btn")
expect(btn).to_be_visible()
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
# The error/result lines ship (hidden) with the right roles.
assert page.locator("#archive-upload-error").get_attribute("role") == "alert"
result = page.locator("#archive-upload-result")
assert result.get_attribute("role") == "status"
expect(result).to_be_hidden()
# The hint explains unpack/scan + in-place replace (task 03).
hint = page.locator("#git-sources-hint")
expect(hint).to_be_visible()
expect(hint).to_contain_text("unpack")
expect(hint).to_contain_text("scan")
expect(hint).to_contain_text("in place")
# ---------------------------------------------------------------------------
# 2. Upload → scan → list (the §7.4 in-flight state, the counts, the
# Local row, the KB, the RAG catalog)
# ---------------------------------------------------------------------------
def test_upload_scans_and_lists(
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
) -> None:
"""One real upload through the page: while the POST is in flight the
button is disabled and reads "Uploading…"; on the 200 it restores,
the result line shows the added count (2), the file input clears,
the list gains exactly ONE row for ``e2e-upload`` with the Local
badge, ``/api/docs`` lists both sentinel files under the source, and
the RAG catalog shows them where the admin expects them."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
btn = page.locator("#archive-upload-btn")
result = page.locator("#archive-upload-result")
# Hold the upload request in the browser: the in-flight state below
# cannot race the (fast) mock-LLM scan while it is held.
_hold_upload_request(page, hold_s=0.8)
page.set_input_files("#archive-upload-file", str(tarball_v1))
btn.click()
# In flight (§7.4): disabled + relabeled, no result yet.
expect(btn).to_be_disabled()
expect(btn).to_have_text("Uploading…")
expect(result).to_be_hidden()
# The request goes out, the server unpacks + scans (mock LLM) and
# answers 200 → the result line shows the added count.
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text("2 added")
# Never stale: the button restored on success and the input cleared.
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
expect(page.locator("#archive-upload-file")).to_have_value("")
# The list gained exactly one row — for the source, with the Local
# badge and the full unpacked path in the mono cell.
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
row = page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)
expect(row).to_have_count(1)
expect(row.locator("span.git-source-kind")).to_have_text("Local")
expect(row.locator("td.git-source-url-cell code")).to_have_text(
str(upload_dir / SOURCE_NAME)
)
# The KB: both sentinel files, under the source name e2e-upload.
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "beta.md")]
# The RAG catalog (admin sees it): both docs, under the source.
page.goto(app_url + SOURCES_URL)
expect(page.locator("#docs-tbody tr")).to_have_count(2)
expect(page.locator("#docs-tbody tr", has_text="alpha.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text="beta.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text=SOURCE_NAME)).to_have_count(2)
# ---------------------------------------------------------------------------
# 3. Re-upload, same filename → in-place replace (no duplicate row,
# dropped file pruned, changed/new file indexed)
# ---------------------------------------------------------------------------
def test_reupload_replaces_in_place(
page: Page,
app_url: str,
db_ready: None,
tarball_v1: Path,
tarball_v2: Path,
upload_dir: Path,
) -> None:
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
result line shows the prune, the list still has exactly ONE
``e2e-upload`` row (the row count for that source is invariant — no
duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
and NOT the dropped ``beta``, and the on-disk folder holds only the
new archive's files."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Baseline: v1 through the page (200 → "2 added", one row).
assert _upload_via_page(page, tarball_v1) == "2 added"
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
# Re-upload v2 — SAME basename, different parent dir (the file
# input's selection is replaced wholesale).
assert _upload_via_page(page, tarball_v2) is not None
result = page.locator("#archive-upload-result")
expect(result).to_have_text(re.compile(r"\d+ pruned"))
# No duplicate: exactly ONE row for that source (and one row total).
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# The registry agrees: one kind=local row, the unpacked path.
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert [(s["kind"], s["path"]) for s in body["sources"]] == [
("local", str(upload_dir / SOURCE_NAME))
]
# The KB: gamma + the CHANGED alpha, NOT the dropped beta.
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "gamma.md")]
# …and the indexed alpha is the v2 one (in-place replace, proven in
# the KB, not just the filesystem).
content = page.request.get(
f"{app_url}/api/documents/content?source={SOURCE_NAME}&path=alpha.md"
)
assert content.status == 200, content.text
assert ALPHA_SENTINEL_V2 in content.json()["content"]
assert ALPHA_SENTINEL_V1 not in content.json()["content"]
# The on-disk folder holds ONLY v2's files (the swap replaced the
# whole folder — no stale v1 file survived).
folder = upload_dir / SOURCE_NAME
assert {p.name for p in folder.iterdir()} == set(V2_FILES)
# ---------------------------------------------------------------------------
# 4. Bad file → inline 422; the form is not wedged
# ---------------------------------------------------------------------------
def test_bad_file_inline_error(
page: Page, app_url: str, db_ready: None, tarball_v1: Path, tmp_path: Path
) -> None:
"""A ``.txt`` through the file input: the role=alert line shows the
422 detail naming the accepted formats, the button restores, the
file selection is KEPT (the fix is one re-pick), the list is
unchanged — and a subsequent good upload still works (the form is
not wedged)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
bad = tmp_path / "notes.txt"
bad.write_text("I am not an archive.\n", encoding="utf-8")
error = page.locator("#archive-upload-error")
btn = page.locator("#archive-upload-btn")
page.set_input_files("#archive-upload-file", str(bad))
btn.click()
# The 422 detail inline (role=alert), naming the accepted formats.
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_contain_text("only .tar, .tar.gz, .tgz or .zip archives are accepted")
# Never stale + the selection kept + no result line + list unchanged.
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
# The selection is kept (the fix is one re-pick) — Chromium reports
# a fake path (``…/notes.txt``), so assert on the basename.
bad_value = page.locator("#archive-upload-file").input_value()
assert bad_value.endswith("notes.txt"), bad_value
expect(page.locator("#archive-upload-result")).to_be_hidden()
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
# The form is not wedged: a good upload right after still works.
assert _upload_via_page(page, tarball_v1) == "2 added"
expect(error).to_be_hidden()
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
assert _docs(page, app_url) == [(SOURCE_NAME, "alpha.md"), (SOURCE_NAME, "beta.md")]
# ---------------------------------------------------------------------------
# 5. Anonymous: the gate, the hidden manager, the 403
# ---------------------------------------------------------------------------
def test_anonymous_gate(page: Page, app_url: str, db_ready: None) -> None:
"""Anonymous on the page: the sign-in gate shows, the manager (and
thus the upload form) stays hidden, and the upload route 403s
(``require_admin`` — A10)."""
page.set_default_timeout(30_000)
page.goto(app_url + GIT_SOURCES_URL)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
expect(page.locator("#sign-out-btn")).to_be_hidden()
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
# The manager is hidden — so is the upload form inside it.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#archive-upload-form")).to_be_hidden()
# The upload route 403s anonymous callers (require_admin runs before
# the multipart body is parsed — the body is a stand-in, the
# test_local_directory_sources.py pattern for this route).
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
assert r.status == 403
tests/e2e/test_local_directory_sources.py
+132 -99
View File
@@ -5,13 +5,27 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov
**Phase-49 rewrite (owner permission 2026-08-28): the page form this
story drove is GONE.** The "Add a local directory" form
(``#local-source-form``, phase 38) was removed from
``/git-sources.html`` and replaced by the archive upload form
(``#archive-upload-form``, phase 49 — its own story E2E is
``test_archive_upload_sources.py``). The phase-38 ACCEPTANCE stands —
a local directory can still be registered, imported, pruned, and
removed — so this suite now adds local sources through the
**authenticated API** the page's own JS no longer calls
(``POST /api/git-sources {"kind": "local", "path": …}``, the contract
the page used to wrap; the admin cookie rides the browser context via
``page.request``). Do NOT "restore" a form here: adding a plain
directory by hand is an API-only operation now.
The story gate for the **local directory** kind of the admin-managed
source registry (phase 38): the admin adds an existing, non-git
directory on the same admin page as the git repos (phase 35), and the
real Sync button (phase 32) imports it — with add-time fail-loud
validation (a missing/relative path is rejected inline, naming the
path) and union pruning (a file deleted from the directory leaves the
index on the next sync; removing the row stops it being a source).
source registry (phase 38): the admin registers an existing, non-git
directory (API — see above), and the real Sync button (phase 32)
imports it — with add-time fail-loud validation (a missing/relative
path is rejected with 422, naming the path in the JSON detail) and
union pruning (a file deleted from the directory leaves the index on
the next sync; removing the row stops it being a source).
The fixture is a **host temp dir** (``tmp_path_factory`` — the app
server runs on the same host, so the path is visible to it) containing
@@ -30,27 +44,27 @@ decision — local directories are DB-registered, no env var), so an
empty table means "no sources configured" until the admin adds the
directory through the real page.
Contract under test:
Contract under test (local adds are API-driven — phase-49 rewrite):
* anonymous: the sign-in gate (the phase-16/35 ``#git-sources-gate``
pattern), the manager hidden (list + BOTH add forms inert), NO
``/api/git-sources`` call, and 403 on the source routes + the sync
trigger (the phase-35 regression assertions, A10);
* admin: a missing path (``/nonexistent/bor-e2e``) 422s inline naming
the path with no row added and the button never stale; the host temp
dir adds (201 → row with the **Local** badge + the full path in a
mono cell, input cleared, button re-enabled); the same path again
409s inline ("already exists", path named) with no second row;
* admin: the header **Sync** button (the phase-32 lifecycle, "Syncing…"
→ "Synced HH:MM") imports the fixture file — it appears in
``GET /api/docs`` (and its sentinel is in ``GET
pattern), the manager hidden (list + git add form + archive upload
form inert), NO ``/api/git-sources`` call, and 403 on the source
routes (incl. the phase-49 upload route) + the sync trigger (the
phase-35 regression assertions, A10);
* admin: a missing path (``/nonexistent/bor-e2e``) 422s with the JSON
detail NAMING the path ("not a directory") and no row added; the
host temp dir adds (201 → the row renders with the **Local** badge +
the full path in a mono cell once the list re-renders); the same
path again 409s ("already exists", path named) with no second row;
* admin: the **Sync** button on the Sources page (the phase-32
lifecycle, "Syncing…" → "Synced HH:MM") imports the fixture file —
it appears in ``GET /api/docs`` (and its sentinel is in ``GET
/api/documents/content``); deleting the file and syncing again prunes
it (``pruned: 1``, gone from ``GET /api/docs`` — union prune); then
removing the row on the page makes it disappear (accept the confirm;
the empty state returns);
* the new local form's a11y basics (UI Structure Check, AGENTS.md rule
5): labeled input, role=alert error line, ≥44px target, 3px
focus-visible outline.
the empty state returns).
* (The phase-38 form's a11y assertions moved with the form: the
upload form's UI Structure Check lives in the phase-49 story E2E.)
Test → story mapping (Playwright Mapping Rule):
1. ``test_anonymous_soft_gate_and_403s``
@@ -223,20 +237,28 @@ def _admin_git_sources_page(page: Page, app_url: str) -> None:
expect(page.locator("#git-sources-content")).to_be_visible()
def _add_local_dir(page: Page, path: str) -> None:
"""Add a local directory through the real page form and wait for the
new row (the 201 → reload → row lifecycle of git-sources.js)."""
page.fill("#local-source-path", path)
page.click("#local-source-add")
expect(
page.locator("#git-sources-tbody tr", has_text=path)
).to_have_count(1, timeout=30_000)
def _add_local_dir_api(page: Page, app_url: str, path: str) -> None:
"""Register a local directory through the authenticated API
(phase-49 rewrite: the page form is gone — the ``kind=local``
POST contract the page used to wrap is unchanged, and the admin
cookie rides the browser context, cf. test_git_sources_admin.py).
``data`` with a dict is JSON-serialized by Playwright's Python API
(there is no ``json=`` kwarg — the JS API's shape is ``json``).
201 is the only success — the caller re-renders the page when it
needs the row in the table."""
r = page.request.post(
f"{app_url}/api/git-sources", data={"kind": "local", "path": path}
)
assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}"
def _click_sync(page: Page) -> None:
def _click_sync(page: Page, app_url: str) -> None:
"""The phase-32 button lifecycle: click → disabled + "Syncing…" →
"Synced HH:MM" (re-enabled — never stale). The server status poll
underneath is what the 2 s UI loop observes."""
underneath is what the 2 s UI loop observes. The button's home is
the Sources page (owner rework 2026-08-28 — it left the shared
navbar), so the helper visits it first."""
page.goto(app_url + "/sources.html")
btn = page.locator("#sync-btn")
expect(btn).to_be_visible()
btn.click()
@@ -277,10 +299,11 @@ def _docs(page: Page, app_url: str) -> list[dict[str, Any]]:
def test_anonymous_soft_gate_and_403s(
page: Page, app_url: str, db_ready: None
) -> None:
"""The phase-16/35 gate on this page (regression through the phase-38
form): anonymous visitors see the sign-in gate and a fully hidden
manager (list + git form + local form), the page never calls the
admin API, and every admin route 403s (A10)."""
"""The phase-16/35 gate on this page (regression through the
phase-49 form swap): anonymous visitors see the sign-in gate and a
fully hidden manager (list + git form + upload form — the phase-38
local form is gone), the page never calls the admin API, and every
admin route 403s (A10)."""
page.set_default_timeout(30_000)
api_calls: list[str] = []
@@ -299,18 +322,26 @@ def test_anonymous_soft_gate_and_403s(
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
# The manager is absent/inert: list, BOTH add forms, env note — all
# inside the hidden #git-sources-content.
# The manager is absent/inert: list, git add form, the phase-49
# archive upload form, env note — all inside the hidden
# #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#local-source-form")).to_be_hidden()
expect(page.locator("#archive-upload-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The phase-38 local-directory form is GONE (phase-49 rewrite) —
# the upload form replaced it.
expect(page.locator("#local-source-form")).to_have_count(0)
expect(page.locator("#local-source-path")).to_have_count(0)
expect(page.locator("#local-source-add")).to_have_count(0)
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers (the phase-35 assertions):
# all three source routes, for BOTH kinds, plus the sync trigger.
# …and the API 403s anonymous callers (the phase-35 assertions,
# plus the phase-49 upload route): all four source routes, for
# BOTH kinds, plus the sync trigger.
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
@@ -330,59 +361,66 @@ def test_anonymous_soft_gate_and_403s(
).status
== 403
)
# The phase-49 upload route 403s too (require_admin runs before the
# multipart body is ever parsed — the body here is a stand-in JSON
# payload, not a real multipart upload).
assert (
page.request.post(
f"{app_url}/api/git-sources/upload", data={"file": ""}
).status
== 403
)
assert page.request.post(f"{app_url}/api/sync").status == 403
# ---------------------------------------------------------------------------
# 2. Admin: add validation (missing path, dir, duplicate) + local form
# a11y basics
# 2. Admin: add validation (missing path, dir, duplicate) — API-driven
# (phase-49 rewrite: the form this drove is gone)
# ---------------------------------------------------------------------------
def test_admin_add_missing_path_then_dir_then_duplicate(
page: Page, app_url: str, local_dir: Path, db_ready: None
) -> None:
"""Add-time fail-loud validation on the real page: a missing path
422s inline NAMING the path (no row, never-stale button, the input
survives for one edit); the host temp dir adds (row with the Local
badge + full path, input cleared); the same path again 409s inline
("already exists", path named, no second row)."""
"""Add-time fail-loud validation (phase-49 rewrite: the page form is
gone, so the same contract is asserted on the API body the page
used to render): a missing path 422s NAMING the path in the JSON
detail (no row added); the host temp dir adds (201 → the row renders
with the Local badge + full path once the list re-renders); the same
path again 409s ("already exists", path named, no second row)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
error = page.locator("#local-source-error")
add_btn = page.locator("#local-source-add")
# --- missing path: 422 naming it in the JSON detail, NO row --------
r = page.request.post(
f"{app_url}/api/git-sources", data={"kind": "local", "path": MISSING_PATH}
)
assert r.status == 422, r.text
detail = r.json()["detail"]
assert MISSING_PATH in detail, f"detail does not name the path: {detail!r}"
assert "not a directory" in detail
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert r.json()["sources"] == []
# --- missing path: inline 422 naming it, NO row, button recovers ---
page.fill("#local-source-path", MISSING_PATH)
add_btn.click()
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_contain_text(MISSING_PATH)
expect(error).to_contain_text("not a directory")
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add directory")
expect(page.locator("#local-source-path")).to_have_value(MISSING_PATH)
# --- the temp dir: 201 → the row appears with the Local badge ------
_add_local_dir(page, str(local_dir))
# --- the temp dir: 201 → the row renders with the Local badge ------
_add_local_dir_api(page, app_url, str(local_dir))
# The API add is invisible to the open page (its JS no longer adds
# local dirs) — re-render the list, exactly as a fresh visit would.
page.reload()
expect(page.locator("#git-sources-content")).to_be_visible(timeout=30_000)
row = page.locator("#git-sources-tbody tr", has_text=str(local_dir))
expect(row).to_have_count(1)
expect(row).to_have_count(1, timeout=30_000)
badge = row.locator("span.git-source-kind")
expect(badge).to_have_text("Local")
expect(badge).to_have_class(re.compile(r"\bis-local\b"))
# The mono cell carries the full path (rendered as text)…
# The mono cell carries the full path (rendered as text)...
expect(row.locator("td.git-source-url-cell code")).to_have_text(str(local_dir))
# …and the row's Remove button is labeled with the kind + path.
expect(row.locator(".git-source-remove")).to_have_attribute(
"aria-label", f"Remove local source: {local_dir}"
)
# The 201 cleared the input and re-enabled the button (never stale).
expect(page.locator("#local-source-path")).to_have_value("")
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add directory")
# The API agrees: kind=local with the stored (expanded) path.
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
@@ -392,26 +430,17 @@ def test_admin_add_missing_path_then_dir_then_duplicate(
(s["kind"], s["path"]) for s in body["sources"]
] == [("local", str(local_dir))]
# --- duplicate: inline 409 naming the path, NO second row -----------
page.fill("#local-source-path", str(local_dir))
add_btn.click()
expect(error).to_be_visible(timeout=30_000)
expect(error).to_contain_text("already exists")
expect(error).to_contain_text(str(local_dir))
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add directory")
expect(page.locator("#local-source-path")).to_have_value(str(local_dir))
# --- the new form's a11y basics (UI Structure Check, AGENTS.md 5) ---
expect(page.get_by_label("Add a local directory")).to_have_count(1)
box = add_btn.bounding_box()
assert box is not None and box["height"] >= 44, f"target too small: {box}"
page.focus("#local-source-path")
outline = page.evaluate(
"() => getComputedStyle(document.querySelector('#local-source-path')).outlineWidth"
# --- duplicate: 409 naming the path, NO second row -----------------
r = page.request.post(
f"{app_url}/api/git-sources", data={"kind": "local", "path": str(local_dir)}
)
assert outline == "3px", f"focus-visible outline missing: {outline!r}"
assert r.status == 409, r.text
detail = r.json()["detail"]
assert "already exists" in detail
assert str(local_dir) in detail
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert len(r.json()["sources"]) == 1 # the row was NOT duplicated
# ---------------------------------------------------------------------------
@@ -423,20 +452,22 @@ def test_admin_add_missing_path_then_dir_then_duplicate(
def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
page: Page, app_url: str, local_dir: Path, db_ready: None
) -> None:
"""The phase-32 button drives the phase-38 pipeline: the header Sync
imports the local directory's fixture file (GET /api/docs shows it,
the sentinel is in its content); deleting the file and syncing again
prunes it (``pruned: 1`` — prune over the union); then removing the
row on the page makes it disappear (the empty state returns)."""
"""The phase-32 button drives the phase-38 pipeline: the Sources-
page Sync imports the local directory's fixture file (GET /api/docs
shows it, the sentinel is in its content); deleting the file and
syncing again prunes it (``pruned: 1`` — prune over the union); then
removing the row on the page makes it disappear (the empty state
returns). The local add is API-driven (phase-49 rewrite)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Fresh registry (the autouse fixture truncated it) — add the source
# through the real page, then run the sync lifecycle against it.
_add_local_dir(page, str(local_dir))
# Fresh registry (the autouse fixture truncated it) — register the
# source via the API (the page form is gone), then run the sync
# lifecycle against it.
_add_local_dir_api(page, app_url, str(local_dir))
# --- run 1: the real sync walks the local dir and imports the file -
_click_sync(page)
_click_sync(page, app_url)
body = _wait_sync_done(page, app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == 1, body["detail"]
@@ -457,7 +488,7 @@ def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
# --- run 2: file deleted → the next sync prunes it (union prune) ---
(local_dir / FIXTURE_REL).unlink()
_click_sync(page)
_click_sync(page, app_url)
body = _wait_sync_done(page, app_url)
assert body["state"] == "success", body
assert body["detail"]["pruned"] == 1, body["detail"]
@@ -468,6 +499,8 @@ def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
)
# --- remove the row: accept the confirm → it disappears ------------
# Back on the manager page (the sync clicks visited the Sources page).
page.goto(app_url + GIT_SOURCES_URL)
removes: list[str] = []
page.on(
"request",
tests/integration/test_git_sources_upload.py
+788
View File
@@ -0,0 +1,788 @@
"""Integration: the admin archive-upload API (phase 49, task 02).
Real Postgres (``podman compose up -d db``); the upload dir is pointed
at a fresh tmp dir per test by monkeypatching the router's
``get_settings`` (the ``test_git_sources_api.py`` pattern — the dev
``.env`` never leaks in), and the scan uses the deterministic
in-process ``FakeEmbedder`` (``test_sync_api.py``'s ``_real_llm``
pattern — real import, no network).
Contract under test:
* anonymous → 403 ``{"detail": "admin only"}`` (the router's
``require_admin`` covers the new route);
* name/format gate → 422: a non-archive extension names the accepted
formats; a ``..`` / separator / empty-stem name (including a bare
``tar.gz``) is rejected with the task-01 message — and the upload
dir is never created for a rejected name (control characters never
reach the app: the multipart transport percent-encodes them — the
task-01 branch for them is covered in ``test_archive_upload.py``);
* one upload at a time → 409 ``an upload is already in progress``
(while a run is in flight — the first request holds the flag through
its scan — and while the module-level flag seam is held);
* streaming cap → 413 naming the ``upload_max_mb`` cap; the temp
``.upload`` file is removed (no stray ``.`` files in the upload dir);
* unpack safety → 422 (zip-slip member, tar symlink escape, corrupt
archive, zero-entry archive) — and a failed upload **never** touches
the previous folder, row, or KB of an earlier good upload (the
no-partial-state locked decision, asserted explicitly);
* happy path → 200 with the sync-detail count keys (``source`` +
``files/added/updated/unchanged/pruned/errors/chunks/overview``), a
``kind=local`` row under the tmp ``upload_dir`` (the NOT-NULL
``url`` column carries the path — the phase-38 convention), the
unpacked folder, the KB via ``GET /api/docs``, and the per-upload
log line (PLAN §9 / AGENTS.md rule 10);
* re-upload, same name → the swap replaces the folder in place, the
row is NOT duplicated (``added_at`` preserved), dropped files are
pruned from the KB, added/changed files are indexed;
* an archive with only non-A9 files is a VALID replacement (indexes
nothing, prunes the previous docs, no overview refresh);
* dead models → 503 with the sanitized model-unavailable message; the
folder and row are already committed (the next sync/re-upload
retries idempotently).
``git_sources`` / ``documents`` / ``chunks`` / ``kb_overview`` are
global state: truncated around every test.
"""
from __future__ import annotations
import asyncio
import io
import logging
import os
import re
import tarfile
import threading
import zipfile
from collections.abc import Iterator
from pathlib import Path
from typing import Literal
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import git_sources as git_sources_api
from app.config import Settings
from app.db import SessionLocal
from app.main import app as fastapi_app
from app.models import GitSource
from app.rag.archive_upload import ArchiveUploadError
from app.rag.importer import ImportSummary
from app.rag.llm import ModelUnavailableError
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
@pytest.fixture(autouse=True)
def clean_git_sources(db: Session) -> Iterator[None]:
"""The stored list is global state: reset around every test."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def clean_documents(db: Session) -> Iterator[None]:
"""The happy path writes ``documents``/``chunks`` and (via the
change-gated overview refresh) ``kb_overview`` — global, truncated
around every test."""
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
db.commit()
@pytest.fixture()
def upload_client() -> Iterator[TestClient]:
"""Admin-signed client (the context-manager form keeps one event
loop across requests — the in-flight 409 test needs the first
request's run to survive while the second lands)."""
with TestClient(fastapi_app) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
yield client
def _point_at(monkeypatch: pytest.MonkeyPatch, upload_dir: Path, upload_max_mb: int = 512) -> None:
"""Fresh settings on the router's module: the tmp upload dir and
(optionally) a shrunk cap — the dev ``.env`` never leaks in."""
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
upload_dir=str(upload_dir),
upload_max_mb=upload_max_mb,
),
)
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
"""The pipeline's ``LLMClient`` becomes the deterministic
in-process ``FakeEmbedder`` (real import, no network); it also
implements ``embed_one``/``chat``, so the phase-41 probe passes."""
monkeypatch.setattr(git_sources_api, "LLMClient", lambda: FakeEmbedder())
#: tarfile write modes used by the tests (uncompressed + gzip).
_TAR_WRITE_MODES = Literal["w", "w:gz"]
def _tarball(path: Path, files: dict[str, str], compress: _TAR_WRITE_MODES = "w:gz") -> Path:
with tarfile.open(path, compress) as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return path
def _zip(path: Path, files: dict[str, str]) -> Path:
with zipfile.ZipFile(path, "w") as zf:
for rel, content in files.items():
zf.writestr(rel, content)
return path
def _targz_bytes(files: dict[str, str]) -> bytes:
"""A tarball in memory (no temp file on disk)."""
buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode="w:gz") as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return buf.getvalue()
def _post(client: TestClient, filename: str, payload: bytes):
"""One upload request (multipart, the page's exact shape)."""
return client.post(
"/api/git-sources/upload",
files={"file": (filename, payload, "application/octet-stream")},
)
def _row(db: Session, path: str) -> GitSource | None:
db.expire_all()
return db.scalar(select(GitSource).where(GitSource.path == path))
def _docs(client: TestClient) -> list[tuple[str, str]]:
body = client.get("/api/docs").json()
return [(d["source"], d["path"]) for d in body["documents"]]
def _count_rows(db: Session) -> int:
return db.execute(text("SELECT count(*) FROM git_sources")).scalar_one()
def _good_upload(
client: TestClient, name: str = "safe", files: dict[str, str] | None = None
) -> None:
"""A 200 upload of a two-sentinel tarball — the baseline state the
no-partial-state tests protect."""
payload_files = files or {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
r = _post(client, f"{name}.tar.gz", _targz_bytes(payload_files))
assert r.status_code == 200, r.text
def _assert_previous_intact(
client: TestClient, db: Session, upload_dir: Path, name: str, files: dict[str, str]
) -> None:
"""The no-partial-state locked decision, asserted explicitly: after a
FAILED upload the previous folder's content, the row (same
``added_at``), and the KB are all exactly as the good upload left
them — and no temp file survived."""
folder = upload_dir / name
assert {p.name: p.read_text(encoding="utf-8") for p in folder.iterdir()} == files
assert _row(db, str(folder)) is not None # the row is still there
assert _count_rows(db) == 1 # …and no second row appeared
assert _docs(client) == [(name, "alpha.md"), (name, "bravo.md")]
assert [p.name for p in upload_dir.iterdir()] == [name] # no stray temp
# --- anonymous -------------------------------------------------------------
def test_anonymous_upload_gets_403(client: TestClient, db: Session) -> None:
r = _post(client, "homelab.tar.gz", b"not an archive at all")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
assert _count_rows(db) == 0
# --- name / format gate -----------------------------------------------------
def test_non_archive_extension_gets_422_naming_formats(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
r = _post(upload_client, "notes.txt", b"hello world")
assert r.status_code == 422
assert r.json()["detail"] == "only .tar, .tar.gz, .tgz or .zip archives are accepted"
assert _count_rows(db) == 0
# The name gate runs before the dir is created — nothing on disk.
assert not uploads.exists()
@pytest.mark.parametrize(
"filename",
[
"../evil.zip", # ``..`` + separator
"sub/dir.tar.gz", # separator
"tar.gz", # bare suffix → empty stem
".tar.gz", # empty stem
# (control characters in the filename are percent-encoded by the
# multipart transport before the app ever sees them — the
# ``archive_source_name`` branch for them is covered in
# tests/unit/test_archive_upload.py)
],
)
def test_unsafe_name_gets_422(
upload_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
filename: str,
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
r = _post(upload_client, filename, b"content")
assert r.status_code == 422, f"{filename!r} must be rejected: {r.text}"
assert r.json()["detail"] != "only .tar, .tar.gz, .tgz or .zip archives are accepted"
assert _count_rows(db) == 0
assert not uploads.exists()
# --- one at a time (409) ----------------------------------------------------
def test_second_upload_while_one_is_in_flight_returns_409(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""The flag is held from the name gate through the scan response:
while the first run is inside its (blocked) scan, the second upload
is 409 — and after the first finishes, uploads are accepted again."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
archive = _tarball(tmp_path / "homelab.tar.gz", {"alpha.md": "# Alpha\nx\n"})
started = threading.Event()
release = threading.Event()
class BlockingImport:
async def __call__(self, sources, llm, **kwargs) -> ImportSummary:
started.set() # the run is in flight (the flag was set earlier)
await asyncio.to_thread(release.wait, 15.0)
return ImportSummary(files=1, unchanged=1)
monkeypatch.setattr(git_sources_api, "import_sources", BlockingImport())
first = {}
def do_first() -> None:
# A separate client/cookie jar: the two requests run on separate
# TestClient portals; only the module-level flag is shared.
with TestClient(fastapi_app) as c:
assert c.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
first["r"] = _post(c, "homelab.tar.gz", archive.read_bytes())
thread = threading.Thread(target=do_first)
thread.start()
try:
assert started.wait(15.0), "first upload did not reach its scan"
with TestClient(fastapi_app) as second:
assert second.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
r = _post(second, "homelab.tar.gz", archive.read_bytes())
assert r.status_code == 409
assert r.json() == {"detail": "an upload is already in progress"}
finally:
release.set()
thread.join(20)
assert first["r"].status_code == 200, first["r"].text
# The flag was released: the next upload goes through for real.
with TestClient(fastapi_app) as third:
assert third.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
r = _post(third, "homelab.tar.gz", archive.read_bytes())
assert r.status_code == 200, r.text
def test_upload_refused_while_flag_held(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""The smallest test seam: the module-level flag itself. Held →
409, nothing happens (no dir, no row)."""
_point_at(monkeypatch, tmp_path / "uploads")
monkeypatch.setattr(git_sources_api, "_upload_in_progress", True)
r = _post(upload_client, "a.tar.gz", b"x")
assert r.status_code == 409
assert r.json() == {"detail": "an upload is already in progress"}
assert _count_rows(db) == 0
assert not (tmp_path / "uploads").exists()
# --- in-flight upload must not hold the request session (regression, -----
# --- phase 49 task 03) ----------------------------------------------------
def test_in_flight_upload_does_not_block_a_concurrent_truncate(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""While the scan is in flight, a concurrent TRUNCATE of the KB +
registry tables (the E2E isolation-fixture pattern) must complete.
Regression for the phase-49 task-03 deadlock: the handler used to
keep its request ``db`` session open across the scan, and the
uncommitted ``_commit_new`` refresh transaction held
``git_sources`` locks for the whole scan. A concurrent TRUNCATE
(documents locked, git_sources pending) then deadlocked with the
scan's own document locks — a cycle spanning three connections
that Postgres's detector cannot see, hanging the app and the test
run forever. The handler now releases the session before the scan;
this pins it: with the scan held in flight, the TRUNCATE completes
well inside its 5 s ``lock_timeout`` (without the fix it times out
with a lock-not-available error)."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
archive = _tarball(tmp_path / "homelab.tar.gz", {"alpha.md": "# Alpha\nx\n"})
started = threading.Event()
release = threading.Event()
class BlockingImport:
async def __call__(self, sources, llm, **kwargs) -> ImportSummary:
started.set() # the run is in flight (the flag was set earlier)
await asyncio.to_thread(release.wait, 15.0)
return ImportSummary(files=1, unchanged=1)
monkeypatch.setattr(git_sources_api, "import_sources", BlockingImport())
first = {}
def do_first() -> None:
with TestClient(fastapi_app) as c:
assert c.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
first["r"] = _post(c, "homelab.tar.gz", archive.read_bytes())
thread = threading.Thread(target=do_first)
thread.start()
try:
assert started.wait(15.0), "first upload did not reach its scan"
# The E2E isolation TRUNCATE, exactly as the story fixtures run
# it — must complete while the scan is held in flight.
with SessionLocal() as tr, tr.begin():
tr.execute(text("SET LOCAL lock_timeout = '5s'"))
tr.execute(
text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources")
)
finally:
release.set()
thread.join(20)
# The scan finished once released — the 200 stands (the TRUNCATE ran
# after the row upsert and dropped it; the next re-upload is
# idempotent, and the autouse fixtures reset the tables anyway).
assert first["r"].status_code == 200, first["r"].text
# --- streaming cap (413) ----------------------------------------------------
def test_oversized_upload_gets_413_and_leaves_no_temp(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads, upload_max_mb=1)
# 2 MiB of incompressible bytes with a 1 MiB cap → 413 naming the cap.
big = os.urandom(2 * 1024 * 1024)
r = _post(upload_client, "big.zip", big)
assert r.status_code == 413
assert "1 MiB" in r.json()["detail"]
assert _count_rows(db) == 0
assert uploads.exists() # the dir was created before streaming
assert list(uploads.iterdir()) == [] # the temp .upload file is gone
# Boundary: EXACTLY the cap is not 413 (the check is strictly >) —
# the stream completes and the garbage bytes fail at unpack instead.
r = _post(upload_client, "exact.zip", os.urandom(1024 * 1024))
assert r.status_code == 422
assert "could not unpack the archive" in r.json()["detail"]
assert list(uploads.iterdir()) == []
# --- unpack safety: failed uploads leave the previous state intact ----------
def test_zip_slip_archive_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
evil = _zip(tmp_path / "safe.zip", {"../evil.txt": "pwned"})
r = _post(upload_client, "safe.zip", evil.read_bytes())
assert r.status_code == 422
assert "traversal" in r.json()["detail"]
assert not (tmp_path / "evil.txt").exists() # the escape never landed
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_tar_symlink_escape_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
evil = tmp_path / "safe.tar"
with tarfile.open(evil, "w") as tf:
info = tarfile.TarInfo("link")
info.type = tarfile.SYMTYPE
info.linkname = "/etc/passwd"
tf.addfile(info)
r = _post(upload_client, "safe.tar", evil.read_bytes())
assert r.status_code == 422
assert "escape" in r.json()["detail"]
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_corrupt_archive_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
# A truncated zip (the EOCD is cut off) is not a zip and not a tar.
good_zip = _zip(tmp_path / "good.zip", {"alpha.md": "# Alpha\nx\n"})
good_bytes = good_zip.read_bytes()
truncated = good_bytes[: len(good_bytes) // 2]
r = _post(upload_client, "safe.zip", truncated)
assert r.status_code == 422
assert "could not unpack the archive" in r.json()["detail"]
# A zero-byte "archive" fails the same way.
r = _post(upload_client, "safe.tar", b"")
assert r.status_code == 422
assert "could not unpack the archive" in r.json()["detail"]
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_swap_failure_gets_422_and_previous_state_is_intact(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""A rename failure in ``swap_in`` (OS error) → 422 with its
message; the previous folder/row/KB are untouched and no temp
survives (the handler's finally cleans the temp sibling)."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\noriginal sentinel one\n",
"bravo.md": "# Bravo\noriginal sentinel two\n",
}
_good_upload(upload_client, "safe", files)
def failing_swap(new_dir: Path, final_dir: Path) -> None:
raise ArchiveUploadError("could not replace the previous folder")
monkeypatch.setattr(git_sources_api, "swap_in", failing_swap)
r = _post(upload_client, "safe.tar.gz", _targz_bytes({"alpha.md": "# A\nx\n"}))
assert r.status_code == 422
assert r.json()["detail"] == "could not replace the previous folder"
_assert_previous_intact(upload_client, db, uploads, "safe", files)
def test_zero_entry_archive_gets_422(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""A completely empty archive (zero entries) is 422 — both
containers — with no folder, row, or temp file left behind."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
empty_targz = tmp_path / "empty.tar.gz"
with tarfile.open(empty_targz, "w:gz"):
pass
r = _post(upload_client, "empty.tar.gz", empty_targz.read_bytes())
assert r.status_code == 422
assert r.json()["detail"] == "the archive contains no files"
empty_zip = tmp_path / "empty.zip"
with zipfile.ZipFile(empty_zip, "w"):
pass
r = _post(upload_client, "empty.zip", empty_zip.read_bytes())
assert r.status_code == 422
assert r.json()["detail"] == "the archive contains no files"
assert _count_rows(db) == 0
assert not (uploads / "empty").exists()
assert list(uploads.iterdir()) == []
# --- happy path -------------------------------------------------------------
def test_happy_path_tar_gz(
upload_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {"alpha.md": "# Alpha\nfirst sentinel\n", "bravo.md": "# Bravo\nsecond sentinel\n"}
payload = _tarball(tmp_path / "homelab.tar.gz", files).read_bytes()
with caplog.at_level(logging.INFO, logger="app.api.git_sources"):
r = _post(upload_client, "homelab.tar.gz", payload)
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {
"source", "files", "added", "updated", "unchanged", "pruned", "errors", "chunks",
"overview",
}
assert body["source"] == "homelab" # filename minus the archive suffix
assert body["files"] == 2
assert body["added"] == 2
assert body["updated"] == 0
assert body["unchanged"] == 0
assert body["pruned"] == 0
assert body["errors"] == 0
assert body["chunks"] >= 2
assert body["overview"] is True # the KB changed → the overview refreshed
# Unpacked under the tmp upload dir, dotfile temps cleaned up.
folder = uploads / "homelab"
assert folder.is_dir()
assert {p.name: p.read_text(encoding="utf-8") for p in folder.iterdir()} == files
assert [p.name for p in uploads.iterdir()] == ["homelab"]
# One kind=local row; the NOT-NULL url column carries the path.
row = _row(db, str(folder))
assert row is not None
assert row.kind == "local"
assert row.url == str(folder)
assert row.path == str(folder)
assert _count_rows(db) == 1
# The KB lists both files under the source name.
assert _docs(upload_client) == [("homelab", "alpha.md"), ("homelab", "bravo.md")]
# The per-upload log line (PLAN §9 / AGENTS.md rule 10).
lines = [rec.getMessage() for rec in caplog.records if rec.getMessage().startswith("upload: ")]
assert len(lines) == 1, lines
match = re.match(
r"^upload: name=homelab file=homelab\.tar\.gz bytes_in=(\d+) files=2 added=2 "
r"updated=0 unchanged=0 pruned=0 errors=0 overview=True total_ms=\d+$",
lines[0],
)
assert match, lines[0]
assert int(match.group(1)) == len(payload) # the compressed bytes in
@pytest.mark.parametrize(
("filename", "builder"),
[
("notes.zip", "zip"),
("plain.tar", "tar"),
("tgz.tgz", "targz"),
],
)
def test_happy_path_other_accepted_formats(
upload_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
filename: str,
builder: str,
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {"alpha.md": "# Alpha\nsentinel\n"}
make = {"zip": _zip, "tar": lambda p, f: _tarball(p, f, "w"), "targz": _tarball}[builder]
payload = make(tmp_path / filename, files).read_bytes()
r = _post(upload_client, filename, payload)
assert r.status_code == 200, r.text
body = r.json()
expected_name = {"notes.zip": "notes", "plain.tar": "plain", "tgz.tgz": "tgz"}[filename]
assert body["source"] == expected_name
assert body["added"] == 1
assert _docs(upload_client) == [(expected_name, "alpha.md")]
assert _count_rows(db) == 1
# --- re-upload, same name: in-place replace ---------------------------------
def test_reupload_same_name_replaces_in_place(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
v1 = {
"alpha.md": "# Alpha\nv1 content\n",
"bravo.md": "# Bravo\nv1 content\n",
"charlie.md": "# Charlie\nv1 content\n",
}
r = _post(upload_client, "homelab.tar.gz", _tarball(tmp_path / "v1.tar.gz", v1).read_bytes())
assert r.status_code == 200, r.text
assert r.json()["added"] == 3
row_before = _row(db, str(uploads / "homelab"))
assert row_before is not None
added_at_before = row_before.added_at
# v2: alpha changed, bravo dropped, delta new.
v2 = {"alpha.md": "# Alpha\nv2 CHANGED content\n", "delta.md": "# Delta\nbrand new\n"}
r = _post(upload_client, "homelab.tar.gz", _tarball(tmp_path / "v2.tar.gz", v2).read_bytes())
assert r.status_code == 200, r.text
body = r.json()
assert body["source"] == "homelab"
assert body["files"] == 2
assert body["added"] == 1 # delta.md
assert body["updated"] == 1 # alpha.md (hash changed)
assert body["unchanged"] == 0
assert body["pruned"] == 2 # bravo.md + charlie.md left the folder → pruned
assert body["overview"] is True
# Exactly ONE row for the path, and its added_at survived (the
# upsert left the existing row alone).
assert _count_rows(db) == 1
row_after = _row(db, str(uploads / "homelab"))
assert row_after is not None
assert row_after.id == row_before.id
assert row_after.added_at == added_at_before
# The folder holds ONLY the new archive's files (full replacement —
# no stale files from v1), and the KB mirrors it.
folder = uploads / "homelab"
assert {p.name: p.read_text(encoding="utf-8") for p in folder.iterdir()} == v2
assert [p.name for p in uploads.iterdir()] == ["homelab"]
assert _docs(upload_client) == [("homelab", "alpha.md"), ("homelab", "delta.md")]
def test_non_a9_archive_is_a_valid_replacement(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""An archive with only non-A9 files is a valid replacement: the
swap happens, the scan indexes nothing, prune removes the source's
docs, and the overview is NOT refreshed (no KB change)."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
r = _post(
upload_client,
"notes.tar.gz",
_tarball(tmp_path / "a.tar.gz", {"readme.md": "# Readme\nv1\n"}).read_bytes(),
)
assert r.status_code == 200, r.text
assert _docs(upload_client) == [("notes", "readme.md")]
r = _post(
upload_client,
"notes.tar.gz",
_tarball(tmp_path / "b.tar.gz", {"binary.bin": "not an importable format"}).read_bytes(),
)
assert r.status_code == 200, r.text
body = r.json()
assert body["files"] == 0 # nothing matches the A9 filter
assert body["added"] == 0
assert body["updated"] == 0
assert body["pruned"] == 1 # readme.md left the KB
assert body["overview"] is False # added + updated == 0 → no refresh
assert {p.name for p in (uploads / "notes").iterdir()} == {"binary.bin"}
assert _count_rows(db) == 1
assert _docs(upload_client) == []
# --- fail-fast models (503) ---------------------------------------------------
def test_models_down_gets_503_and_leaves_folder_and_row(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""The folder and row are committed BEFORE the model probe: a dead
endpoint answers 503 (sanitized — credentials masked) and the next
sync/re-upload retries idempotently; the scan never ran."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
async def dead_probe(llm: object) -> None:
raise ModelUnavailableError(
"The embedding model ('embed') is not available — check the model "
"endpoint and retry. (embeddings request to "
"https://user:secret@aipi.reeseapps.com/v1 failed: connection refused)"
)
monkeypatch.setattr(git_sources_api, "check_models", dead_probe)
r = _post(
upload_client,
"homelab.tar.gz",
_tarball(tmp_path / "h.tar.gz", {"alpha.md": "# Alpha\nx\n"}).read_bytes(),
)
assert r.status_code == 503
detail = r.json()["detail"]
assert "The embedding model ('embed') is not available" in detail
assert "*****@aipi.reeseapps.com" in detail # the sanitizer masked the credentials
assert "user:secret" not in detail
assert "connection refused" in detail # the reason survives
# The folder and row are already committed (idempotent retry path).
folder = uploads / "homelab"
assert folder.is_dir()
row = _row(db, str(folder))
assert row is not None
assert row.kind == "local"
assert row.url == str(folder)
# The scan never ran: no docs, no stray temps.
assert _docs(upload_client) == []
assert [p.name for p in uploads.iterdir()] == ["homelab"]
tests/unit/test_archive_upload.py
+579
View File
@@ -0,0 +1,579 @@
"""Unit: phase 49 upload settings + the safe archive unpack utility.
Covers ``app.rag.archive_upload`` end to end:
* the name-derivation matrix (suffix stripping incl. the compound
``.tar.gz``, case handling, and the rejection list — ``..``,
separators, control chars, empty stems);
* safe extraction — valid zip/tar.gz archives extract byte-identically,
while zip-slip, absolute members, escaping symlinks/hardlinks, device
members, and the extracted-byte cap all raise
:class:`ArchiveUploadError` **and** leave no partial target behind;
* ``swap_in`` — fresh, in-place replace with full content replacement
(no interleave, no ``.old-`` leftovers), and restore-on-failure;
* the two new settings (``upload_dir`` / ``upload_max_mb``) with
env overrides and the fail-loud ``<= 0`` validator.
"""
from __future__ import annotations
import io
import os
import stat
import tarfile
import zipfile
from pathlib import Path
from typing import Any
import pytest
from pydantic import ValidationError
from app.config import Settings
from app.rag.archive_upload import (
ARCHIVE_SUFFIXES,
ArchiveUploadError,
_link_target_resolved, # pyright: ignore[reportPrivateUsage]
_member_dest, # pyright: ignore[reportPrivateUsage]
archive_source_name,
swap_in,
unpack_archive,
)
def _settings(**kwargs: Any) -> Settings:
"""Build Settings without reading a .env file (deterministic tests)."""
kwargs.setdefault("_env_file", None)
return Settings(**kwargs) # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
# ---------------------------------------------------------------------------
# Settings (phase 49, task 01)
# ---------------------------------------------------------------------------
def test_upload_settings_defaults(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("BOR_UPLOAD_DIR", raising=False)
monkeypatch.delenv("BOR_UPLOAD_MAX_MB", raising=False)
s = _settings()
# Deliberately separate from the git checkouts (``sources_dir``).
assert s.upload_dir == "~/bor-sources/uploads"
assert s.sources_dir == "~/bor-sources"
assert s.upload_max_mb == 512
def test_upload_settings_env_overrides(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("BOR_UPLOAD_DIR", "/data/bor/uploads")
monkeypatch.setenv("BOR_UPLOAD_MAX_MB", "128")
s = _settings()
assert s.upload_dir == "/data/bor/uploads"
assert s.upload_max_mb == 128
@pytest.mark.parametrize("value", ["0", "-1", "-512"])
def test_upload_max_mb_rejects_zero_and_negative(
monkeypatch: pytest.MonkeyPatch, value: str
) -> None:
"""``<= 0`` would reject every upload — the validator fails loudly at
startup (the ``agent_max_rounds`` pattern)."""
monkeypatch.setenv("BOR_UPLOAD_MAX_MB", value)
with pytest.raises(ValidationError, match="upload_max_mb"):
_settings()
# ---------------------------------------------------------------------------
# archive_source_name — the derivation matrix
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("filename", "expected"),
[
("homelab.tar.gz", "homelab"),
("notes.tgz", "notes"),
("a.zip", "a"),
("x.tar", "x"),
# Case-insensitive suffix match; the stem keeps its case (the name
# becomes a folder name on a case-sensitive Linux FS).
("upper.TAR.GZ", "upper"),
("Homelab.Zip", "Homelab"),
# ONE compound strip — never a double strip (``.tar.gz`` matches
# before ``.tar`` would; there is no ``.gz`` suffix at all).
("a.tar.gz", "a"),
# Only the LAST suffix is stripped.
("a.zip.zip", "a.zip"),
("y.tar.tgz", "y.tar"),
("café.tar", "café"),
],
)
def test_archive_source_name_strips_one_suffix(filename: str, expected: str) -> None:
assert archive_source_name(filename) == expected
def test_archive_suffixes_are_longest_first() -> None:
# ``.tar.gz`` must precede ``.tar`` or ``a.tar.gz`` would yield
# ``a.tar``.
assert ARCHIVE_SUFFIXES == (".tar.gz", ".tgz", ".zip", ".tar")
@pytest.mark.parametrize(
"filename",
[
"", # empty
"tar.gz", # empty stem (bare suffix, no leading dot)
"tgz", # empty stem (bare suffix)
"zip", # empty stem (bare suffix)
".zip", # empty stem (hidden-file suffix)
"..tar.gz", # ``..`` after the strip
"..", # no suffix, ``..`` stem
".", # no suffix, ``.`` stem
"a/b.tar", # forward-separator path
"a\\b.tar", # backslash path
"a\tb.zip", # tab control character
"a\x00b.tar", # NUL control character
"a\x1bb.tar", # escape-sequence control character
],
)
def test_archive_source_name_rejects_unsafe_names(filename: str) -> None:
with pytest.raises(ArchiveUploadError):
archive_source_name(filename)
# ---------------------------------------------------------------------------
# Archive builders (deterministic, in-memory)
# ---------------------------------------------------------------------------
def _make_zip(
path: Path,
files: dict[str, bytes] | None = None,
dirs: tuple[str, ...] = (),
extra_attrs: dict[str, int] | None = None,
) -> None:
with zipfile.ZipFile(path, "w") as zf:
for name in dirs:
info = zipfile.ZipInfo(name if name.endswith("/") else name + "/")
info.external_attr = (0o40755 << 16)
zf.writestr(info, b"")
for name, data in (files or {}).items():
info = zipfile.ZipInfo(name)
info.external_attr = (0o100644 << 16)
zf.writestr(info, data)
for name, attr in (extra_attrs or {}).items():
info = zipfile.ZipInfo(name)
info.external_attr = attr
zf.writestr(info, b"")
def _make_tar(
path: Path,
spec: list[tuple[str, bytes, str]],
gz: bool = False,
) -> None:
"""``spec`` entries: ``(name, payload, kind)`` with kind one of
``f`` (file), ``d`` (dir), ``sym`` (symlink, payload = target),
``lnk`` (hardlink, payload = target), ``chr`` (char device),
``fifo`` (FIFO)."""
mode = "w:gz" if gz else "w"
with tarfile.open(path, mode) as tf:
for name, payload, kind in spec:
ti = tarfile.TarInfo(name)
if kind == "f":
ti.size = len(payload)
ti.mode = 0o644
tf.addfile(ti, io.BytesIO(payload))
elif kind == "d":
ti.type = tarfile.DIRTYPE
ti.mode = 0o755
tf.addfile(ti)
elif kind in ("sym", "lnk"):
ti.type = tarfile.SYMTYPE if kind == "sym" else tarfile.LNKTYPE
ti.linkname = payload.decode()
tf.addfile(ti)
elif kind == "chr":
ti.type = tarfile.CHRTYPE
ti.devmajor, ti.devminor = 1, 3
tf.addfile(ti)
elif kind == "fifo":
ti.type = tarfile.FIFOTYPE
tf.addfile(ti)
# ---------------------------------------------------------------------------
# unpack_archive — valid archives extract byte-identically
# ---------------------------------------------------------------------------
def test_unpack_zip_valid_nested(tmp_path: Path) -> None:
archive = tmp_path / "notes.zip"
_make_zip(
archive,
files={"a/b.txt": b"hello\n", "c.txt": b"x" * 50, "a/deep/n.md": b"# deep"},
dirs=("a", "a/deep"),
)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "a/b.txt").read_bytes() == b"hello\n"
assert (target / "c.txt").read_bytes() == b"x" * 50
assert (target / "a/deep/n.md").read_bytes() == b"# deep"
assert (target / "a").is_dir()
assert (target / "a/deep").is_dir()
def test_unpack_zip_unknown_mode_entries_treated_as_files(tmp_path: Path) -> None:
"""Windows-made / ``writestr``-style zips carry no Unix mode bits
(external_attr 0) — decided by the member name, not rejected."""
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
zf.writestr("plain.txt", b"data") # default external_attr = 0
archive = tmp_path / "plain.zip"
archive.write_bytes(buf.getvalue())
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "plain.txt").read_bytes() == b"data"
def test_unpack_tar_gz_valid(tmp_path: Path) -> None:
archive = tmp_path / "notes.tar.gz"
_make_tar(
archive,
[
("dir/", b"", "d"),
("dir/a.md", b"# hi", "f"),
("top.txt", b"t", "f"),
],
gz=True,
)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "dir/a.md").read_bytes() == b"# hi"
assert (target / "top.txt").read_bytes() == b"t"
assert (target / "dir").is_dir()
def test_unpack_plain_tar_valid(tmp_path: Path) -> None:
"""``r:*`` handles uncompressed ``.tar`` too."""
archive = tmp_path / "notes.tar"
_make_tar(archive, [("only.txt", b"plain tar", "f")], gz=False)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert (target / "only.txt").read_bytes() == b"plain tar"
def test_unpack_internal_symlink_and_hardlink_allowed(tmp_path: Path) -> None:
"""Links that stay INSIDE the unpack directory are fine (the spec
resolves the target and rejects only escapes)."""
archive = tmp_path / "links.tar"
_make_tar(
archive,
[
("sub/", b"", "d"),
("sub/data.txt", b"inner", "f"),
("alias", b"sub/data.txt", "sym"),
("dup", b"sub/data.txt", "lnk"),
],
gz=False,
)
target = tmp_path / "out"
unpack_archive(archive, target, 10_000)
assert target.joinpath("alias").is_symlink()
assert target.joinpath("alias").read_bytes() == b"inner"
assert target.joinpath("dup").read_bytes() == b"inner"
def test_extracted_bytes_exactly_at_cap_passes(tmp_path: Path) -> None:
"""The cap bounds EXCEEDING bytes — landing exactly on it is OK."""
archive = tmp_path / "exact.zip"
_make_zip(archive, files={"f.txt": b"a" * 10})
target = tmp_path / "out"
unpack_archive(archive, target, 10)
assert (target / "f.txt").read_bytes() == b"a" * 10
# ---------------------------------------------------------------------------
# unpack_archive — every guard raises AND leaves no partial target
# ---------------------------------------------------------------------------
def _assert_no_partial(target: Path) -> None:
assert not target.exists() and not target.is_symlink()
def test_member_dest_rejects_empty_name(tmp_path: Path) -> None:
target = tmp_path / "t"
target.mkdir()
with pytest.raises(ArchiveUploadError, match="empty name"):
_member_dest("", target)
def test_member_dest_dot_name_is_the_target_itself(tmp_path: Path) -> None:
"""``.`` resolves to the target itself — inside, so allowed (the
containment check's equality arm)."""
target = tmp_path / "t"
target.mkdir()
assert _member_dest(".", target) == target
def test_member_dest_rejects_resolution_escape(tmp_path: Path) -> None:
"""Defense in depth (the resolve-based containment check): a symlink
already inside the target that points OUT makes any member routed
through it escape once resolved."""
target = tmp_path / "t"
target.mkdir()
(target / "sneaky").symlink_to(tmp_path / "outside")
with pytest.raises(ArchiveUploadError, match="escapes"):
_member_dest("sneaky/evil.txt", target)
def test_link_target_rejects_empty_target(tmp_path: Path) -> None:
with pytest.raises(ArchiveUploadError, match="empty target"):
_link_target_resolved("", tmp_path / "d", tmp_path / "t")
def test_unpack_tar_extractfile_none_is_corrupt(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A REG member whose data tarfile cannot hand back (``extractfile``
→ None) is a corrupt member — rejected, partial state cleaned."""
archive = tmp_path / "corrupt.tar"
_make_tar(archive, [("f.txt", b"x", "f")], gz=False)
monkeypatch.setattr(tarfile.TarFile, "extractfile", lambda self, member: None)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="corrupt archive member"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_zip_slip_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "evil.zip"
_make_zip(archive, files={"../evil.txt": b"evil", "ok.txt": b"ok"})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="traversal"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
@pytest.mark.parametrize("name", ["/etc/x", "C:\\evil", "\\\\server\\share"])
def test_unpack_zip_absolute_member_rejected_and_cleaned(tmp_path: Path, name: str) -> None:
archive = tmp_path / "abs.zip"
_make_zip(archive, files={name: b"no"})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="absolute"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_zip_symlink_entry_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "slink.zip"
_make_zip(
archive,
extra_attrs={"link": (stat.S_IFLNK | 0o777) << 16},
)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="symlink"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_zip_non_regular_entry_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "dev.zip"
_make_zip(archive, extra_attrs={"dev": (stat.S_IFCHR | 0o644) << 16})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="non-regular"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_absolute_member_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "abs.tar"
_make_tar(archive, [("/etc/x", b"no", "f")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="absolute"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_symlink_escape_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "slink.tar"
_make_tar(archive, [("link", b"/etc/passwd", "sym")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="escapes"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_relative_symlink_escape_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "slink2.tar"
_make_tar(
archive,
[("sub/", b"", "d"), ("sub/escape", b"../../outside", "sym")],
gz=False,
)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="escapes"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_tar_hardlink_escape_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "hlink.tar"
_make_tar(archive, [("hard", b"/etc/passwd", "lnk")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="escapes"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
@pytest.mark.parametrize("kind", ["chr", "fifo"])
def test_unpack_tar_device_and_fifo_rejected_and_cleaned(tmp_path: Path, kind: str) -> None:
archive = tmp_path / "dev.tar"
_make_tar(archive, [("dev", b"", kind)], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="device or FIFO"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
@pytest.mark.parametrize("kind", ["zip", "tar"])
def test_unpack_extracted_cap_exceeded_rejected_and_cleaned(tmp_path: Path, kind: str) -> None:
"""cap=10 with a 20-byte file → the cap (not the content) is named
and the partial tree is gone."""
archive = tmp_path / ("cap." + kind)
payload = b"b" * 20
if kind == "zip":
_make_zip(archive, files={"big.txt": payload})
else:
_make_tar(archive, [("big.txt", payload, "f")], gz=False)
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="10-byte extraction cap"):
unpack_archive(archive, target, 10)
_assert_no_partial(target)
def test_unpack_cap_accumulates_across_members(tmp_path: Path) -> None:
"""6 + 6 bytes under a 10-byte cap: each file alone is under, the
total is not."""
archive = tmp_path / "acc.zip"
_make_zip(archive, files={"a.txt": b"a" * 6, "b.txt": b"b" * 6})
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="extraction cap"):
unpack_archive(archive, target, 10)
_assert_no_partial(target)
def test_unpack_corrupt_archive_rejected_and_cleaned(tmp_path: Path) -> None:
archive = tmp_path / "junk.bin"
archive.write_bytes(b"this is not an archive at all")
target = tmp_path / "out"
with pytest.raises(ArchiveUploadError, match="could not unpack"):
unpack_archive(archive, target, 10_000)
_assert_no_partial(target)
def test_unpack_existing_target_rejected(tmp_path: Path) -> None:
archive = tmp_path / "ok.zip"
_make_zip(archive, files={"a.txt": b"1"})
target = tmp_path / "out"
target.mkdir()
with pytest.raises(ArchiveUploadError, match="already exists"):
unpack_archive(archive, target, 10_000)
# The pre-existing directory is left exactly as found.
assert target.is_dir() and not any(target.iterdir())
# ---------------------------------------------------------------------------
# swap_in — atomic in-place replacement
# ---------------------------------------------------------------------------
def test_swap_in_fresh(tmp_path: Path) -> None:
new = tmp_path / "new"
new.mkdir()
(new / "new.txt").write_text("new")
final = tmp_path / "final"
swap_in(new, final)
assert final.is_dir()
assert (final / "new.txt").read_text() == "new"
assert not new.exists()
def test_swap_in_replaces_existing_with_full_content(tmp_path: Path) -> None:
"""The previous content is fully gone, the new content complete —
no interleave — and no ``.old-`` sibling survives."""
final = tmp_path / "final"
final.mkdir()
(final / "old.txt").write_text("old")
(final / "keepdir/").mkdir()
(final / "keepdir" / "stale.txt").write_text("stale")
new = tmp_path / "new"
new.mkdir()
(new / "fresh.txt").write_text("fresh")
(new / "keepdir/").mkdir()
(new / "keepdir" / "v2.txt").write_text("v2")
swap_in(new, final)
assert not (final / "old.txt").exists()
assert not (final / "keepdir" / "stale.txt").exists()
assert (final / "fresh.txt").read_text() == "fresh"
assert (final / "keepdir" / "v2.txt").read_text() == "v2"
assert not new.exists()
assert not list(tmp_path.glob("final.old-*"))
def test_swap_in_restores_previous_folder_on_failure(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The second rename (new → final) fails: the previous folder comes
back intact, the new dir is cleaned, and the error is raised."""
final = tmp_path / "final"
final.mkdir()
(final / "old.txt").write_text("old")
new = tmp_path / "new"
new.mkdir()
(new / "fresh.txt").write_text("fresh")
real_rename = os.rename
def fake_rename(
src: str | os.PathLike[str], dst: str | os.PathLike[str], *args: Any, **kwargs: Any
) -> Any:
if Path(str(src)) == new: # the new → final rename fails
raise OSError("simulated swap failure")
return real_rename(src, dst, *args, **kwargs)
monkeypatch.setattr(os, "rename", fake_rename)
with pytest.raises(ArchiveUploadError, match="could not replace"):
swap_in(new, final)
# Previous folder intact, byte for byte.
assert (final / "old.txt").read_text() == "old"
# New dir cleaned, no orphaned .old sibling.
assert not new.exists()
assert not list(tmp_path.glob("final.old-*"))
def test_swap_in_double_failure_leaves_no_orphan(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Both renames fail (new→final AND the ``.old-``→final restore):
the best-effort path deletes the orphaned ``.old-`` sibling — the
previous folder is unrecoverable in this scenario, so at least
nothing is left mixed on disk."""
final = tmp_path / "final"
final.mkdir()
(final / "old.txt").write_text("old")
new = tmp_path / "new"
new.mkdir()
(new / "fresh.txt").write_text("fresh")
real_rename = os.rename
def fake_rename(
src: str | os.PathLike[str], dst: str | os.PathLike[str], *args: Any, **kwargs: Any
) -> Any:
s, d = Path(str(src)), Path(str(dst))
if s == new or d == final: # the swap rename and the restore both fail
raise OSError("simulated failure")
return real_rename(src, dst, *args, **kwargs)
monkeypatch.setattr(os, "rename", fake_rename)
with pytest.raises(ArchiveUploadError, match="could not replace"):
swap_in(new, final)
assert not new.exists()
assert not list(tmp_path.glob("final.old-*"))