feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block.
This commit is contained in:
+227
-8
@@ -17,9 +17,13 @@ Routes: ``GET`` (DB rows oldest-first, or the env list with
|
||||
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
||||
(201, validated create; ``kind`` selects the validation: git → exactly
|
||||
the phase-35 URL contract, local → an existing absolute directory, else
|
||||
422 naming the path), ``DELETE /{source_id}`` (204). The whole router
|
||||
sits behind :func:`app.core.auth.require_admin` — anonymous callers get
|
||||
403 on every route.
|
||||
422 naming the path), ``POST /upload`` (phase 49 — admin archive upload:
|
||||
``.tar``/``.tar.gz``/``.tgz``/``.zip`` streamed with a size cap, safely
|
||||
unpacked, atomically swapped in over an existing folder of the same
|
||||
name, row upserted, then the synchronous single-source scan — see
|
||||
:func:`upload_archive`), ``DELETE /{source_id}`` (204). The whole
|
||||
router sits behind :func:`app.core.auth.require_admin` — anonymous
|
||||
callers get 403 on every route.
|
||||
|
||||
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so every git 409/422 detail is a fixed generic
|
||||
@@ -27,27 +31,47 @@ string that never repeats the submitted URL. Local paths are not
|
||||
secrets — the local 422/409 details name the (expanded) path so the
|
||||
owner sees exactly which directory failed.
|
||||
|
||||
Scope boundary (phase locked decisions): adding or removing a source
|
||||
does NOT clone, import, or prune anything — the existing Sync button
|
||||
performs that (a removal prunes on the next sync, ``prune=True``).
|
||||
Scope boundary (phase locked decisions): the CRUD routes do NOT
|
||||
clone, import, or prune anything — the existing Sync button performs
|
||||
that (a removal prunes on the next sync, ``prune=True``). The phase-49
|
||||
upload route is the exception: it unpacks the archive and then scans
|
||||
the single source synchronously in the request (``import_sources``
|
||||
with ``prune=True`` + the change-gated overview refresh) and answers
|
||||
with the sync-style counts.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.sync import _sanitize_error
|
||||
from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import get_db
|
||||
from app.models import GitSource
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow
|
||||
from app.rag.archive_upload import (
|
||||
ARCHIVE_SUFFIXES,
|
||||
ArchiveUploadError,
|
||||
archive_source_name,
|
||||
swap_in,
|
||||
unpack_archive,
|
||||
)
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import LLMClient, ModelUnavailableError, check_models
|
||||
from app.rag.overview import regenerate_overview
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow, UploadOut
|
||||
|
||||
logger = logging.getLogger("app.api.git_sources")
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/git-sources",
|
||||
@@ -55,6 +79,18 @@ router = APIRouter(
|
||||
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
|
||||
)
|
||||
|
||||
#: One upload at a time (phase 49, task 02 — the phase-32 ``_task``
|
||||
#: spirit): the flag is held from the name gate through the scan
|
||||
#: response. A plain bool, not an ``asyncio.Lock`` — it is checked and
|
||||
#: set with no await in between (a single app loop can never enter
|
||||
#: twice), and it stays correct across requests that run on separate
|
||||
#: event loops (the TestClient convention).
|
||||
_upload_in_progress = False
|
||||
|
||||
#: Streaming read size while counting compressed upload bytes (1 MiB
|
||||
#: chunks — the task-02 cap check granularity).
|
||||
_STREAM_CHUNK = 1 << 20
|
||||
|
||||
#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
|
||||
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
|
||||
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
|
||||
@@ -191,6 +227,189 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
)
|
||||
|
||||
|
||||
@router.post("/upload", response_model=UploadOut)
|
||||
async def upload_archive(
|
||||
file: UploadFile = File(...), # noqa: B008
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> UploadOut:
|
||||
"""Upload a source archive and scan it (phase 49, task 02).
|
||||
|
||||
The scan is **synchronous in the request** (phase locked decisions,
|
||||
owner-confirmed) and mirrors the admin sync pipeline:
|
||||
|
||||
1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip``
|
||||
(422 naming the accepted set) and a safe source name
|
||||
(``archive_source_name`` — its message is the 422 detail);
|
||||
2. one at a time — 409 ``an upload is already in progress``;
|
||||
3. stream the upload in 1 MiB chunks into a dotfile temp with the
|
||||
``upload_max_mb`` cap — 413 naming the cap, temp deleted;
|
||||
4. unpack to a temp sibling (traversal/symlink/device/corrupt/
|
||||
over-cap all 422 with the task-01 user-safe message, temps
|
||||
deleted); a zero-entry archive is 422 ``the archive contains no
|
||||
files`` — an archive with only non-A9 files is a VALID
|
||||
replacement (the scan indexes nothing, prune removes the
|
||||
source's docs);
|
||||
5. atomic swap-in — a same-name re-upload replaces the previous
|
||||
folder in place; a failure leaves the previous folder/row/KB
|
||||
untouched (422);
|
||||
6. upsert the row by ``path`` (``kind='local'``; an existing row is
|
||||
left as-is — ``added_at`` preserved — and the unique index is
|
||||
the 409 backstop);
|
||||
7. fail-fast ``check_models`` — 503 with the sanitized
|
||||
model-unavailable message; the folder/row are already committed,
|
||||
so the next sync/re-upload retries idempotently;
|
||||
8. ``import_sources([folder], llm, prune=True)`` + the change-gated
|
||||
``regenerate_overview``;
|
||||
9. one INFO log line (PLAN §9 / AGENTS.md rule 10);
|
||||
10. 200 with the sync-detail count keys (``UploadOut``).
|
||||
"""
|
||||
started = time.monotonic()
|
||||
settings = get_settings()
|
||||
total = 0
|
||||
|
||||
# 1. Name/format gate — the accepted formats first (the 422 names
|
||||
# them), then the task-01 safe-name derivation. A BARE suffix
|
||||
# ("tar.gz") is an accepted format with no usable stem — it
|
||||
# passes here and gets task-01's "no usable source name" 422.
|
||||
# No upload dir is created for a rejected name.
|
||||
filename = file.filename or ""
|
||||
lowered = filename.lower()
|
||||
if not any(
|
||||
lowered.endswith(suffix) or lowered == suffix.lstrip(".")
|
||||
for suffix in ARCHIVE_SUFFIXES
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="only .tar, .tar.gz, .tgz or .zip archives are accepted",
|
||||
)
|
||||
try:
|
||||
name = archive_source_name(filename)
|
||||
except ArchiveUploadError as e:
|
||||
raise HTTPException(status_code=422, detail=str(e)) from None
|
||||
|
||||
# 2. One at a time — the flag is checked and set with no await
|
||||
# between, so the single app loop can never enter twice.
|
||||
global _upload_in_progress
|
||||
if _upload_in_progress:
|
||||
raise HTTPException(status_code=409, detail="an upload is already in progress")
|
||||
_upload_in_progress = True
|
||||
|
||||
upload_root = Path(settings.upload_dir).expanduser()
|
||||
upload_root.mkdir(parents=True, exist_ok=True)
|
||||
max_bytes = settings.upload_max_mb * 1024 * 1024
|
||||
temp_upload = upload_root / f".{name}.{uuid.uuid4().hex}.upload"
|
||||
temp_unpack = upload_root / f".{name}.{uuid.uuid4().hex}.unpack"
|
||||
try:
|
||||
# 3. Stream with the compressed-size cap — dotfile temps are
|
||||
# hidden from the upload dir's listing.
|
||||
try:
|
||||
with open(temp_upload, "wb") as out:
|
||||
while chunk := await file.read(_STREAM_CHUNK):
|
||||
total += len(chunk)
|
||||
if total > max_bytes:
|
||||
raise HTTPException(
|
||||
status_code=413,
|
||||
detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit",
|
||||
)
|
||||
out.write(chunk)
|
||||
except HTTPException:
|
||||
temp_upload.unlink(missing_ok=True)
|
||||
raise
|
||||
# 4. Unpack to a temp sibling; the compressed bytes are no
|
||||
# longer needed once unpacked (phase locked decision: only
|
||||
# the unpacked content is kept).
|
||||
try:
|
||||
unpack_archive(temp_upload, temp_unpack, max_bytes)
|
||||
except ArchiveUploadError as e:
|
||||
temp_upload.unlink(missing_ok=True)
|
||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
||||
raise HTTPException(status_code=422, detail=str(e)) from None
|
||||
temp_upload.unlink(missing_ok=True)
|
||||
if not any(temp_unpack.iterdir()):
|
||||
# Zero entries = a user error. (Only non-A9 files is NOT an
|
||||
# error — it still has entries and is a valid replacement.)
|
||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
||||
raise HTTPException(status_code=422, detail="the archive contains no files")
|
||||
# 5. Swap in — a same-name re-upload replaces the previous
|
||||
# folder atomically; a failure leaves it, the row, and the
|
||||
# KB untouched.
|
||||
final_dir = upload_root / name
|
||||
try:
|
||||
swap_in(temp_unpack, final_dir)
|
||||
except ArchiveUploadError as e:
|
||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
||||
raise HTTPException(status_code=422, detail=str(e)) from None
|
||||
# 6. Upsert the row by path — no duplicates: an existing row is
|
||||
# left exactly as it is (``added_at`` preserved); the unique
|
||||
# index is the 409 backstop for a concurrent insert the
|
||||
# pre-check missed.
|
||||
path = str(final_dir)
|
||||
if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
|
||||
_commit_new(
|
||||
GitSource(url=path, kind="local", path=path),
|
||||
f"a local source with this path already exists: {path}",
|
||||
db,
|
||||
)
|
||||
# Release the request session NOW — the handler never touches
|
||||
# ``db`` again (the scan below uses its own sessions). If the
|
||||
# session stayed open, its uncommitted transaction (the
|
||||
# ``_commit_new`` refresh SELECT) would hold ``git_sources``
|
||||
# locks for the whole scan, and any concurrent TRUNCATE of the
|
||||
# KB tables (the E2E isolation fixtures) would deadlock against
|
||||
# the scan's own document locks — a cycle Postgres cannot see.
|
||||
# ``get_db``'s teardown close() is idempotent.
|
||||
db.close()
|
||||
# 7. Fail-fast models (phase 41) — 503 with the sanitized
|
||||
# message; nothing else is rolled back (the folder/row are
|
||||
# committed and the next sync/re-upload retries idempotently).
|
||||
llm = LLMClient()
|
||||
try:
|
||||
await check_models(llm)
|
||||
except ModelUnavailableError as e:
|
||||
raise HTTPException(status_code=503, detail=_sanitize_error(str(e))) from None
|
||||
# 8. Scan — single source, prune (dropped files leave the KB),
|
||||
# then the change-gated overview refresh (phases 31/32).
|
||||
summary = await import_sources([final_dir], llm, prune=True)
|
||||
overview = False
|
||||
if summary.added + summary.updated > 0:
|
||||
overview = await regenerate_overview(llm)
|
||||
finally:
|
||||
_upload_in_progress = False
|
||||
# No temp may survive any failure path (defensive — each step
|
||||
# already cleans its own; on success both are already gone).
|
||||
temp_upload.unlink(missing_ok=True)
|
||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
||||
|
||||
# 9. Per-upload log line (PLAN §9 / AGENTS.md rule 10).
|
||||
logger.info(
|
||||
"upload: name=%s file=%s bytes_in=%d files=%d added=%d updated=%d "
|
||||
"unchanged=%d pruned=%d errors=%d overview=%s total_ms=%d",
|
||||
name,
|
||||
filename,
|
||||
total,
|
||||
summary.files,
|
||||
summary.added,
|
||||
summary.updated,
|
||||
summary.unchanged,
|
||||
summary.pruned,
|
||||
summary.errors,
|
||||
overview,
|
||||
round((time.monotonic() - started) * 1000),
|
||||
)
|
||||
# 10. Respond 200 with the sync-style counts.
|
||||
return UploadOut(
|
||||
source=name,
|
||||
files=summary.files,
|
||||
added=summary.added,
|
||||
updated=summary.updated,
|
||||
unchanged=summary.unchanged,
|
||||
pruned=summary.pruned,
|
||||
errors=summary.errors,
|
||||
chunks=summary.chunks,
|
||||
overview=overview,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{source_id}", status_code=204)
|
||||
def delete_git_source(
|
||||
source_id: uuid.UUID,
|
||||
|
||||
@@ -156,6 +156,18 @@ class Settings(BaseSettings):
|
||||
#: 28). Stored as a raw string — ``Path.expanduser()`` is applied in
|
||||
#: the import script, not here.
|
||||
sources_dir: str = "~/bor-sources"
|
||||
#: Where uploaded source archives are unpacked (phase 49) — one
|
||||
#: subdirectory per source name (filename minus the archive suffix).
|
||||
#: Deliberately kept **separate** from ``sources_dir`` (the git
|
||||
#: checkouts). Raw string — ``Path.expanduser()`` is applied by the
|
||||
#: upload endpoint, not here.
|
||||
upload_dir: str = "~/bor-sources/uploads"
|
||||
#: Cap in MiB for uploaded source archives (phase 49): it bounds BOTH
|
||||
#: the compressed upload size and the total extracted bytes (the
|
||||
#: zip-bomb guard). ``<= 0`` would reject every upload — a typo, so
|
||||
#: the validator fails loudly at startup (the ``agent_max_rounds``
|
||||
#: pattern).
|
||||
upload_max_mb: int = 512
|
||||
|
||||
@field_validator("import_extensions")
|
||||
@classmethod
|
||||
@@ -181,6 +193,14 @@ class Settings(BaseSettings):
|
||||
raise ValueError("agent_max_rounds must be >= 0 (0 = no tools)")
|
||||
return v
|
||||
|
||||
@field_validator("upload_max_mb")
|
||||
@classmethod
|
||||
def _upload_max_mb_positive(cls, v: int) -> int:
|
||||
"""``0``/negative would reject every upload — fail loud at startup."""
|
||||
if v <= 0:
|
||||
raise ValueError("upload_max_mb must be > 0 (MiB)")
|
||||
return v
|
||||
|
||||
# Suggested questions (onboarding + empty state).
|
||||
suggestions: list[str] = [
|
||||
"How is my Kubernetes cluster set up?",
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
"""Safe archive unpacking for uploaded sources (phase 49, task 01).
|
||||
|
||||
Pure file-system logic — no FastAPI/DB imports. The API layer
|
||||
(``POST /api/git-sources/upload``, phase 49, task 02) calls these and
|
||||
maps :class:`ArchiveUploadError` to status codes (422).
|
||||
|
||||
The guarantees (phase 49 locked decisions):
|
||||
|
||||
* :func:`archive_source_name` derives the source name from the uploaded
|
||||
filename — ONE trailing archive suffix stripped, longest-first so
|
||||
``homelab.tar.gz`` yields ``homelab`` (never ``homelab.tar``) — and
|
||||
rejects anything that would not be a safe single folder name (the name
|
||||
becomes a directory under ``BOR_UPLOAD_DIR`` and the ``git_sources``
|
||||
row's ``path``).
|
||||
* :func:`unpack_archive` extracts ``.zip`` or ``.tar`` (gz/bz2/xz
|
||||
transparently) into a fresh directory, rejecting absolute member
|
||||
paths, ``..`` traversal, symlink/hardlink targets that escape the
|
||||
unpack directory, and device/FIFO members — and counting every
|
||||
extracted byte against a cap (zip-bomb guard). Any failure removes the
|
||||
partial ``target_dir`` so no half-unpacked tree survives.
|
||||
* :func:`swap_in` makes ``new_dir`` become ``final_dir`` with **no
|
||||
missing window**: the previous folder is renamed to a unique
|
||||
same-filesystem ``.old-`` sibling first, the new folder is renamed
|
||||
into place, then the sibling is deleted. A failed swap restores the
|
||||
previous folder (best effort) and removes ``new_dir``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import tarfile
|
||||
import uuid
|
||||
import zipfile
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import IO
|
||||
|
||||
#: Accepted archive suffixes, LONGEST FIRST — ``.tar.gz`` must match
|
||||
#: before ``.tar`` would (phase 49 locked decision: these four formats
|
||||
#: only).
|
||||
ARCHIVE_SUFFIXES: tuple[str, ...] = (".tar.gz", ".tgz", ".zip", ".tar")
|
||||
|
||||
#: A client-supplied filename never legitimately contains a path
|
||||
#: separator — reject rather than strip (defense in depth; the upload
|
||||
#: also carries a browsed-path, which the browser normalizes).
|
||||
_SEPARATOR_RE = re.compile(r"[/\\]")
|
||||
|
||||
#: Windows drive-letter prefix (``C:``) — an absolute member path in a
|
||||
#: cross-platform zip.
|
||||
_DRIVE_RE = re.compile(r"^[A-Za-z]:")
|
||||
|
||||
#: Streaming read size while counting extracted bytes.
|
||||
_CHUNK_SIZE = 1 << 20
|
||||
|
||||
|
||||
class ArchiveUploadError(Exception):
|
||||
"""A user-safe archive/unpack error (the API maps it to a status).
|
||||
|
||||
Messages name the problem — and the cap where relevant — never the
|
||||
archive's content and never any path beyond the owner's own upload
|
||||
directory.
|
||||
"""
|
||||
|
||||
|
||||
def archive_source_name(filename: str) -> str:
|
||||
"""The source name for an uploaded archive filename.
|
||||
|
||||
The filename is used exactly as sent — a client can never
|
||||
legitimately embed a ``/`` or ``\\``, so a separator is *rejected*
|
||||
(the defensive basename step below is then a no-op, kept for the
|
||||
contract). ONE trailing archive suffix from :data:`ARCHIVE_SUFFIXES`
|
||||
is removed, matched case-insensitively and longest-first:
|
||||
``homelab.tar.gz`` → ``homelab``, ``a.tar.gz`` → ``a`` (a single
|
||||
compound strip, not a double one), ``a.zip.zip`` → ``a.zip``. The
|
||||
stem keeps its original case — the name becomes a folder name on a
|
||||
Linux filesystem (case-sensitive).
|
||||
|
||||
Raises:
|
||||
ArchiveUploadError: the filename is empty, contains a path
|
||||
separator, or the stripped stem is empty / ``.`` / ``..`` /
|
||||
contains control characters (the API maps to 422).
|
||||
"""
|
||||
if not filename:
|
||||
raise ArchiveUploadError("empty file name")
|
||||
if _SEPARATOR_RE.search(filename):
|
||||
raise ArchiveUploadError("file name contains a path separator")
|
||||
# No separator above, so the basename is the name itself — kept
|
||||
# explicit so the "take the basename" contract lives in one place.
|
||||
base = filename.rsplit("/", 1)[-1].rsplit("\\", 1)[-1]
|
||||
lowered = base.lower()
|
||||
for suffix in ARCHIVE_SUFFIXES: # longest first
|
||||
if lowered.endswith(suffix):
|
||||
base = base[: -len(suffix)]
|
||||
break
|
||||
if lowered == suffix.lstrip("."): # bare suffix ("tar.gz") — no stem
|
||||
base = ""
|
||||
break
|
||||
if not base or base in (".", ".."):
|
||||
raise ArchiveUploadError("archive file name has no usable source name")
|
||||
if any(ord(c) < 0x20 or ord(c) == 0x7F for c in base):
|
||||
raise ArchiveUploadError("archive file name contains control characters")
|
||||
return base
|
||||
|
||||
|
||||
def _member_dest(name: str, target: Path) -> Path:
|
||||
"""The destination path for one archive member, or raise.
|
||||
|
||||
Rejects empty names, absolute names (POSIX ``/``, Windows ``\\`` or
|
||||
drive letters), and any ``..`` path component (zip names are
|
||||
``/``-separated; backslash forms are normalized before the check).
|
||||
The final path is resolved against the target (following any symlink
|
||||
an earlier member may have created) and must stay inside ``target``.
|
||||
"""
|
||||
if not name:
|
||||
raise ArchiveUploadError("archive member with an empty name")
|
||||
if name.startswith(("/", "\\")) or _DRIVE_RE.match(name):
|
||||
raise ArchiveUploadError("archive member with an absolute path")
|
||||
if ".." in PurePosixPath(name).parts or ".." in name.replace("\\", "/").split("/"):
|
||||
raise ArchiveUploadError("archive member path traversal")
|
||||
dest = target / name
|
||||
target_resolved = target.resolve()
|
||||
dest_resolved = dest.resolve()
|
||||
if dest_resolved != target_resolved and target_resolved not in dest_resolved.parents:
|
||||
raise ArchiveUploadError("archive member path escapes the unpack directory")
|
||||
return dest
|
||||
|
||||
|
||||
def _link_target_resolved(linkname: str, link_dir: Path, target: Path) -> Path:
|
||||
"""Resolve a symlink/hardlink target against its member's directory.
|
||||
|
||||
The resolved target must stay inside ``target`` — anything else
|
||||
(absolute targets, ``..`` climbs out) is rejected. Returns the
|
||||
resolved path (used directly for hardlinks).
|
||||
"""
|
||||
if not linkname:
|
||||
raise ArchiveUploadError("archive link with an empty target")
|
||||
# An absolute linkname wins over the join (pathlib semantics) and is
|
||||
# then caught by the containment check below.
|
||||
resolved = (link_dir / linkname).resolve()
|
||||
target_resolved = target.resolve()
|
||||
if resolved != target_resolved and target_resolved not in resolved.parents:
|
||||
raise ArchiveUploadError("archive link target escapes the unpack directory")
|
||||
return resolved
|
||||
|
||||
|
||||
def _write_capped(src: IO[bytes], dst: Path, max_extract_bytes: int, total: list[int]) -> None:
|
||||
"""Stream ``src`` to ``dst``, counting into ``total[0]``.
|
||||
|
||||
Raises as soon as the cumulative extracted bytes EXCEED
|
||||
``max_extract_bytes`` (the cap itself is exactly reachable). The
|
||||
error names the cap, not the archive content.
|
||||
"""
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(dst, "wb") as out:
|
||||
while chunk := src.read(_CHUNK_SIZE):
|
||||
total[0] += len(chunk)
|
||||
if total[0] > max_extract_bytes:
|
||||
raise ArchiveUploadError(
|
||||
f"archive exceeds the {max_extract_bytes}-byte extraction cap"
|
||||
)
|
||||
out.write(chunk)
|
||||
|
||||
|
||||
def _unpack_zip(archive: Path, target: Path, max_extract_bytes: int) -> None:
|
||||
total: list[int] = [0]
|
||||
with zipfile.ZipFile(archive) as zf:
|
||||
for member in zf.infolist():
|
||||
# The high 16 bits of external_attr are the Unix mode when
|
||||
# present. Modes may be 0 (Windows-made zips), bare permission
|
||||
# bits (CPython ``writestr``: 0o600), or a full mode with the
|
||||
# file-type bits — only the latter can prove a member is a
|
||||
# symlink/device/FIFO, and only those are rejected; entries
|
||||
# without type bits are decided by the member name.
|
||||
mode = member.external_attr >> 16
|
||||
if stat.S_ISLNK(mode):
|
||||
raise ArchiveUploadError("zip archives with symlink entries are not allowed")
|
||||
if mode & 0o170000 and not (stat.S_ISREG(mode) or stat.S_ISDIR(mode)):
|
||||
raise ArchiveUploadError("zip archives with non-regular entries are not allowed")
|
||||
dest = _member_dest(member.filename, target)
|
||||
if member.filename.endswith("/") or (mode and stat.S_ISDIR(mode)):
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
with zf.open(member) as src:
|
||||
_write_capped(src, dest, max_extract_bytes, total)
|
||||
|
||||
|
||||
def _unpack_tar(archive: Path, target: Path, max_extract_bytes: int) -> None:
|
||||
# ``r:*`` auto-detects plain/gz/bz2/xz compression.
|
||||
total: list[int] = [0]
|
||||
with tarfile.open(archive, mode="r:*") as tf:
|
||||
for member in tf.getmembers():
|
||||
dest = _member_dest(member.name, target)
|
||||
if member.issym():
|
||||
_link_target_resolved(member.linkname, dest.parent, target)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
os.symlink(member.linkname, dest)
|
||||
elif member.islnk():
|
||||
resolved = _link_target_resolved(member.linkname, dest.parent, target)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
os.link(resolved, dest)
|
||||
elif member.isdir():
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
elif member.isreg():
|
||||
src = tf.extractfile(member)
|
||||
if src is None:
|
||||
raise ArchiveUploadError("corrupt archive member")
|
||||
_write_capped(src, dest, max_extract_bytes, total)
|
||||
else: # char/block device, FIFO
|
||||
raise ArchiveUploadError(
|
||||
"tar archives with device or FIFO members are not allowed"
|
||||
)
|
||||
|
||||
|
||||
def unpack_archive(archive: Path, target_dir: Path, max_extract_bytes: int) -> None:
|
||||
"""Extract ``archive`` into ``target_dir`` (created empty here).
|
||||
|
||||
``target_dir`` must NOT exist yet — the caller passes a fresh unique
|
||||
path (a temp sibling of the final folder). On ANY failure — bad or
|
||||
corrupt archive, unsafe member, cap exceeded, OS error — the partial
|
||||
``target_dir`` is removed so no half-unpacked tree survives, and the
|
||||
failure is raised as :class:`ArchiveUploadError` (the module's only
|
||||
public exception type).
|
||||
"""
|
||||
if target_dir.exists() or target_dir.is_symlink():
|
||||
raise ArchiveUploadError("the unpack target already exists")
|
||||
target_dir.mkdir(parents=True)
|
||||
try:
|
||||
# Content-sniff the container: a .zip that is really a tar (or a
|
||||
# truncated file) falls through to the tar reader and fails
|
||||
# loudly there instead of half-extracting.
|
||||
if zipfile.is_zipfile(archive):
|
||||
_unpack_zip(archive, target_dir, max_extract_bytes)
|
||||
else:
|
||||
_unpack_tar(archive, target_dir, max_extract_bytes)
|
||||
except Exception as exc:
|
||||
shutil.rmtree(target_dir, ignore_errors=True)
|
||||
if isinstance(exc, ArchiveUploadError):
|
||||
raise
|
||||
raise ArchiveUploadError("could not unpack the archive") from exc
|
||||
|
||||
|
||||
def swap_in(new_dir: Path, final_dir: Path) -> None:
|
||||
"""Atomically make ``new_dir`` become ``final_dir`` (no missing window).
|
||||
|
||||
If ``final_dir`` exists it is first renamed to a unique same-
|
||||
filesystem sibling ``<name>.old-<hex>``, then ``new_dir`` is renamed
|
||||
into place, and the ``.old-`` sibling is deleted (re-uploads replace
|
||||
the previous content in place — one folder, no stale files). If it
|
||||
does not exist, this is a plain rename.
|
||||
|
||||
On a rename failure the previous folder is restored (best effort) and
|
||||
``new_dir`` removed, then the failure is raised as
|
||||
:class:`ArchiveUploadError` — a failed upload never leaves the
|
||||
previous folder, row, or KB in a mixed state.
|
||||
"""
|
||||
old_dir: Path | None = None
|
||||
try:
|
||||
if final_dir.exists():
|
||||
old_dir = final_dir.with_name(final_dir.name + ".old-" + uuid.uuid4().hex)
|
||||
os.rename(final_dir, old_dir)
|
||||
os.rename(new_dir, final_dir)
|
||||
except OSError as exc:
|
||||
if old_dir is not None:
|
||||
try:
|
||||
os.rename(old_dir, final_dir)
|
||||
except OSError:
|
||||
shutil.rmtree(old_dir, ignore_errors=True)
|
||||
shutil.rmtree(new_dir, ignore_errors=True)
|
||||
raise ArchiveUploadError("could not replace the previous folder") from exc
|
||||
if old_dir is not None:
|
||||
shutil.rmtree(old_dir)
|
||||
@@ -260,3 +260,25 @@ class GitSourceList(BaseModel):
|
||||
|
||||
sources: list[GitSourceRow]
|
||||
from_env: bool
|
||||
|
||||
|
||||
class UploadOut(BaseModel):
|
||||
"""``POST /api/git-sources/upload`` response (phase 49, task 02).
|
||||
|
||||
The uploaded source's name (filename minus the archive suffix) plus
|
||||
the SAME count keys as the admin sync's success ``detail``
|
||||
(``files``, ``added``, ``updated``, ``unchanged``, ``pruned``,
|
||||
``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the
|
||||
``overview`` flag: the Sources page renders the same
|
||||
"N added · N pruned" result line for both.
|
||||
"""
|
||||
|
||||
source: str
|
||||
files: int
|
||||
added: int
|
||||
updated: int
|
||||
unchanged: int
|
||||
pruned: int
|
||||
errors: int
|
||||
chunks: int
|
||||
overview: bool
|
||||
|
||||
Reference in New Issue
Block a user