brain-of-reese/app/api/git_sources.py

"""Admin-managed sources API (phase 35, task 02; local kind, phase 38).

Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10
extended — the public API surface stays stateless and the signed cookie
remains the only session state, same as ``/api/steering`` and
``/api/sync``): the ``git_sources`` table holds the sources the Sync
button (phase 32) and ``import_docs`` (phase 28) import — ``kind='git'``
rows carry the repo URL to clone/pull, ``kind='local'`` rows (phase 38)
carry an existing directory on the server to walk directly. DB rows win
over ``BOR_GIT_SOURCES``, which is a git-only fallback while the table is
empty (the phase's locked decision — ``from_env`` tells the UI which list
it is looking at, so the page can show the env note only while the
fallback is active).

Routes: ``GET`` (DB rows oldest-first, or the env list with
``from_env: true`` while the table is empty; rows carry ``kind`` +
``path``, git rows — and env rows — report ``path: null``; every row
reports ``ignore_paths`` — DB rows their stored normalized list, env
rows ``[]``, phase 89 — ``include_hidden`` — DB rows their stored
flag, env rows ``False``, phase 105 — and ``branch`` — DB rows their
stored value (NULL = the repo's default branch), env rows ``null``,
phase 135), ``POST``
(201, validated create; ``kind`` selects the validation: git → the
phase-35 URL contract + the ``file://`` local transport (phase 135,
task 05 — the E2E fixture repos the UI adds end to end), local → an
existing absolute directory, else 422 naming the path; both kinds
accept ``ignore_paths`` — optional, absent → ``[]``, stored
normalized, phase 89 — and ``include_hidden``
— optional, absent → ``False``, stored as sent, phase 105; git rows
accept ``branch`` — optional, absent/empty → NULL (the default
branch), non-empty validated by
:func:`app.rag.git_sources.validate_branch_name` with the fixed-detail
422 ``not a valid branch name`` (input never echoed, phase 135) — a
``branch`` on a LOCAL row is 422 ``branch is only valid for git
sources``),
``PATCH /{source_id}`` (phase 89, A5 + phase 105 + phase 121 + phase
135 — the ignore list (replace), the hidden-folders flag (phase 105),
the token (phase 121 tri-state), and the branch (phase 135 tri-state)
— each optional, present-wins: 404 unknown id; ``ignore_paths`` when
present is normalized + A4-validated with fixed-detail 422s and
REPLACES the row's list wholesale — an empty list clears all;
``include_hidden`` when present sets the flag; ``token`` and ``branch``
when present are tri-state — absent/None = no change, non-empty =
set/replace (``branch`` re-validated, ``local`` rows 422), empty
string = clear (stored NULL); all absent → 200 no-op; 200 → the
``GitSourceOut`` shape),
``POST /upload`` (phase 49, backgrounded in phase
64 task 03, scan deferred in phase 90 — admin archive upload: the
``.tar``/``.tar.gz``/``.tgz``/``.zip`` name/format gate + the 1 MiB-
chunk receive with the ``upload_max_mb`` cap run **inline** and
answered 202 the moment the archive is safely on disk; unpack → swap
→ row upsert then run in a **background task** — and nothing else:
no model check, no import, no overview refresh (phase 90, A1 — the
scan is the RAG page's "Sync sources" button's job) — see
:func:`upload_archive` and :func:`_run_upload`),
``GET /upload/status`` (the phase-32 ``SyncStatus``-shaped in-memory
state of that run — the phase-64 ``current_file`` / ``files_done`` /
``files_total`` keys stay in the set but null/0/0 for the whole run:
uploads have no file-level progress, phase 90 A2; navigating away from
the page mid-upload no longer aborts anything), ``DELETE /{source_id}``
(204 — total removal, phase 69: row + the source's documents (chunks +
embeddings) committed first, then the app-managed on-disk dir). The
whole router sits behind :func:`app.core.auth.require_admin` —
anonymous callers get 403 on every route.

No credential-echo path (phase 32's masking discipline, extended by
phase 121 — LOCKED A2): git URLs may embed ``user:pass@``, so (a)
every git 409/422 detail is a fixed generic string that never repeats
the submitted URL, and (b) every URL that LEAVES the API is masked
through :func:`app.rag.git_sources.sanitize_url` before it enters a
response (DB rows, env-fallback rows, POST 201, PATCH 200) — a legacy
row whose credential is still embedded in the stored ``url`` clones
fine (the stored value is untouched) but its API/UI output is
bare. Local paths are not secrets — the local 422/409 details name
the (expanded) path so the owner sees exactly which directory failed.

Scope boundary (phase locked decisions): the CRUD routes do NOT
clone or import anything — the existing Sync button performs that, and
its ``prune=True`` stays for **upstream file churn** (files deleted
upstream or dropped from a local dir), not for row removal: ``DELETE``
is a total removal in itself (phase 69) — the row + the source's
documents (chunks + embeddings via the ``all, delete-orphan`` cascade)
commit atomically **first** (the RAG is always consistent with the
registry), then the app-managed on-disk dir (git checkout / unpacked
upload folder) is deleted; an ``OSError`` there is logged, never fatal.
Foreign local dirs (the owner's own) are never touched on disk, a
sibling row sharing the source name keeps the shared documents +
files (only the row goes), and a pruned KB bumps ``sources_version``
exactly once (the phase-53 saved-chat invalidation) with a
best-effort overview refresh. The upload route is the other exception
(phase 64, task 03; phase 90): after the 202 receive
answer, its background task unpacks the archive, swaps it in, and
upserts the row — and **stops there**: no model probe, no import, no
overview refresh. The scan is the RAG page's Sync button's job
(phase 90, A1 — it gives the owner time to edit the new source's
ignore list first; the sync already imports ``kind='local'`` rows
with prune + each row's ignore list, A4). The terminal ``success``
carries the no-count payload ``{"message": "uploaded"}`` in the status
``detail`` (phase 90, A2 — the key set is unchanged; the UI composes
the user copy).
"""
from __future__ import annotations

import asyncio
import logging
import re
import shutil
import time
import uuid
from dataclasses import dataclass, field
from datetime import UTC, datetime
from pathlib import Path
from typing import Any, Literal, cast

from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session

from app.api.sync import _sanitize_error
from app.config import get_settings
from app.core.auth import require_admin
from app.db import SessionLocal, get_db
from app.models import Document, GitSource
from app.rag.archive_upload import (
    ARCHIVE_SUFFIXES,
    ArchiveUploadError,
    archive_source_name,
    swap_in,
    unpack_archive,
)
from app.rag.git_sources import (
    normalize_credential,
    sanitize_url,
    validate_branch_name,
)
from app.rag.importer import normalize_ignore_path
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
from app.rag.source_removal import (
    has_sibling,
    managed_dir_for,
    remove_managed_dir,
    resolve_source_name,
)
from app.rag.sources_meta import bump_sources_version
from app.schemas import (
    GitSourceIn,
    GitSourceList,
    GitSourceOut,
    GitSourcePatchIn,
    GitSourceRow,
    UploadAccepted,
)

logger = logging.getLogger("app.api.git_sources")

router = APIRouter(
    prefix="/git-sources",
    tags=["git-sources"],
    dependencies=[Depends(require_admin)],  # phase 16 pattern: admin-only surface
)

#: One upload at a time (phase 49, backgrounded in phase 64 task 03):
#: the flag is checked and set with **no await in between, BEFORE the
#: streaming receive** — the handler now awaits (the 1 MiB-chunk stream)
#: long before the background task exists, so a task-done check alone
#: would let a concurrent POST slip through the receive window and start
#: a second run. A plain bool, not an ``asyncio.Lock``: it is checked
#: and set with no await in between (a single app loop can never enter
#: twice), and it stays correct across requests that run on separate
#: event loops (the TestClient convention). Held until
#: :func:`_run_upload`'s ``finally`` (end of the background run) — or
#: cleared on the inline exception path where the task was never
#: created.
_upload_in_progress = False

#: Streaming read size while counting compressed upload bytes (1 MiB
#: chunks — the task-02 cap check granularity).
_STREAM_CHUNK = 1 << 20


@dataclass
class UploadStatus:
    """In-memory state of the (at most one) in-flight upload run.

    Mirrors :class:`app.api.sync.SyncStatus` (the phase-32 pattern,
    phase 64 task 03): ``state`` is the same four-state machine
    (``idle`` / ``running`` / ``success`` / ``failed``); terminal states
    carry the run's ``detail`` (success — the no-count
    ``{"message": "uploaded"}`` payload, phase 90 A2) or ``error``
    (failure — sanitized) so the UI can render the last result
    after a page reload (the re-attach behavior, task 05).

    Phase 64 (task 03) progress keys: ``current_file`` /
    ``files_done`` / ``files_total`` stay null/0/0 for the **whole**
    run (phase 90, A2 — the key set is unchanged, but uploads have no
    file-level progress: unpack has no per-file hook and the scan —
    the only thing that had one — moved to the sync, which keeps its
    live file label).
    """

    state: Literal["idle", "running", "success", "failed"] = "idle"
    started_at: datetime | None = None
    finished_at: datetime | None = None
    current_file: str | None = None
    files_done: int = 0
    files_total: int = 0
    detail: dict[str, Any] = field(default_factory=dict)
    error: str | None = None


_upload_status = UploadStatus()

#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
#: ``host:repo`` is deliberately rejected (422). ``file://`` (phase 135,
#: task 05) is the deterministic local-transport source the E2E fixture
#: repos use — a real ``git clone`` of a local repo (the phase-28/32/107
#: transport-true E2E idiom, promoted from the ``BOR_GIT_SOURCES`` env
#: fallback to a stored row the UI can add).
URL_RE = re.compile(r"^(https?://|ssh://|file://|git@)")

#: Phase 89, A4 — the per-source ignore-list limits, enforced in
#: :func:`_validate_ignore_paths` (shared by POST and PATCH): at most
#: 200 entries, each ≤500 chars after normalization. The 422 details
#: are fixed strings that never echo the input (the router's
#: credential-safety discipline, applied for consistency).
MAX_IGNORE_PATHS = 200
MAX_IGNORE_PATH_LENGTH = 500


@router.get("", response_model=GitSourceList)
def list_git_sources(
    db: Session = Depends(get_db),  # noqa: B008
) -> GitSourceList:
    """The effective source list (git + local rows, phase 38).

    DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
    same-timestamp inserts) with ``from_env: false`` — each row carries
    its ``kind``, its ``ignore_paths`` (phase 89 — the stored,
    normalized list; ``or []`` guards a row that predated the column),
    its ``include_hidden`` (phase 105 — the stored flag), its
    ``branch`` (phase 135 — the stored value; NULL = the repo's default
    branch), and, for local rows, the stored ``path`` (git rows and env
    rows report ``path: null``); while the table is empty, the
    ``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
    git-only) with null ``id``/``added_at``, ``ignore_paths: []``,
    ``include_hidden: False`` and ``branch: None`` (no DB row to store a
    list, a flag or a branch on), and ``from_env: true``.

    Every URL is masked on the way out (phase 121, LOCKED A2 —
    :func:`sanitize_url`): an env value or a legacy stored URL may
    embed ``user:pass@`` — the env value and the DB value are
    untouched, only the response is bare.
    """
    rows = db.scalars(
        select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
    ).all()
    if rows:
        return GitSourceList(
            sources=[
                # ``ck_git_sources_kind`` (migration 0007) guarantees the
                # value is 'git' or 'local' — the cast documents that.
                GitSourceRow(
                    id=row.id,
                    kind=cast(Literal["git", "local"], row.kind),
                    url=sanitize_url(row.url),  # phase 121: never echo userinfo
                    path=row.path,
                    added_at=row.added_at,
                    ignore_paths=row.ignore_paths or [],
                    include_hidden=row.include_hidden,
                    branch=row.branch,  # phase 135: NULL = default branch
                )
                for row in rows
            ],
            from_env=False,
        )
    return GitSourceList(
        sources=[
            GitSourceRow(
                id=None,
                kind="git",
                url=sanitize_url(url),  # phase 121: an env URL can embed a token
                path=None,
                added_at=None,
                ignore_paths=[],
                include_hidden=False,
                branch=None,  # phase 135: no DB row to store a branch on
            )
            for url in get_settings().git_source_list
        ],
        from_env=True,
    )


@router.post("", response_model=GitSourceOut, status_code=201)
def create_git_source(
    payload: GitSourceIn,
    db: Session = Depends(get_db),  # noqa: B008
) -> GitSourceOut:
    """Store one source (fields already trimmed by the schema).

    ``kind="git"`` (default) — exactly the phase-35 contract: 422 when
    the URL shape is not one of the accepted prefixes (generic detail —
    the input is never echoed), 409 when the trimmed URL is already
    stored (the unique index is the backstop against a concurrent insert
    the pre-check missed), 201 + the created row otherwise.

    ``kind="local"`` — ``path`` must expand (``~``) to an absolute,
    existing directory on the server: 422 naming the path otherwise
    (fail loud at add-time — the owner sees it immediately), 409 when
    the path is already stored (detail names the path), 201 + the stored
    row otherwise (``url`` holds the expanded path — the table's
    NOT-NULL location column).

    Wrong field combinations (git without url, local without path, both
    kinds' fields) are 422 with fixed, input-free details.

    ``ignore_paths`` (phase 89) — optional, both kinds: the raw box
    lines are normalized + A4-validated (``_validate_ignore_paths`` —
    the fixed-detail 422s) and the normalized list is what is stored and
    reported.

    ``include_hidden`` (phase 105) — optional, both kinds: absent →
    stored ``False`` (A4), present → stored as sent; the stored flag is
    what is reported.

    ``token`` (phase 121, LOCKED A2) — the masked private-repo
    credential: write-only, stored in the dedicated column, never
    echoed (the response has no token field by contract). Git rows
    are normalized on the way in (``normalize_credential``): an
    old-style embedded ``user:pass@`` URL is stored bare with the
    credential in the token column, an explicit ``token`` wins over
    the embedded one (LOCKED A6), and the duplicate check runs on the
    bare URL.

    ``branch`` (phase 135) — git rows only: absent/None → stored NULL
    (the repo's DEFAULT branch — the pre-phase-135 behavior), a
    trimmed-EMPTY value likewise stores NULL (at CREATE, empty is
    absent — the split from PATCH's clear leg, see :class:`GitSourceIn`),
    non-empty → validated by :func:`validate_branch_name` and stored
    (a ``ValueError`` → 422 ``not a valid branch name`` — FIXED detail,
    the input is never echoed). A ``branch`` on a LOCAL row is 422
    ``branch is only valid for git sources`` (local rows are walked,
    not cloned).
    """
    row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
    return GitSourceOut(
        id=row.id,
        url=sanitize_url(row.url),  # phase 121: the output mask, always
        added_at=row.added_at,
        ignore_paths=row.ignore_paths,
        include_hidden=row.include_hidden,
        branch=row.branch,  # phase 135: NULL = default branch
    )


def _validate_ignore_paths(raw: list[str] | None) -> list[str]:
    """Normalize + enforce the A4 limits (phase 89); the fixed 422
    details never echo the input (the credential-safety discipline,
    applied for consistency).

    Shared by POST and PATCH. The empty check runs FIRST: a
    whitespace-only entry must 422, not be silently dropped (the UI
    drops blank lines client-side; the API stays defensive).
    """
    entries = [normalize_ignore_path(e) for e in (raw or [])]
    if any(not e for e in entries):
        raise HTTPException(status_code=422, detail="ignore paths must be non-empty")
    if len(entries) > MAX_IGNORE_PATHS:
        raise HTTPException(status_code=422, detail="a source has at most 200 ignore paths")
    if any(len(e) > MAX_IGNORE_PATH_LENGTH for e in entries):
        raise HTTPException(status_code=422, detail="an ignore path exceeds 500 characters")
    return entries


def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
    """Insert ``row``; the unique index is the backstop — a concurrent
    insert the pre-check missed still yields the generic 409, never a
    500 (phase-35 convention, now shared by both kinds)."""
    db.add(row)
    try:
        db.commit()
    except IntegrityError:
        db.rollback()
        raise HTTPException(status_code=409, detail=duplicate_detail) from None
    db.refresh(row)
    return row


def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
    """``kind=git`` — the phase-35 URL contract, extended with the
    ``file://`` local transport (phase 135, task 05 — the E2E fixture
    repos the UI adds end to end); A10: no credential echo, so every
    detail is a fixed string."""
    if payload.path is not None:
        raise HTTPException(status_code=422, detail="a git source takes a url, not a path")
    if payload.url is None:
        raise HTTPException(status_code=422, detail="a git source requires a url")
    url = payload.url
    if not URL_RE.match(url):
        # The detail lists the accepted prefixes (fixed string — the
        # input is never echoed; phase 135 task 05 added file://).
        raise HTTPException(
            status_code=422,
            detail="not a valid git URL (expected https://, ssh://, file:// or git@…)",
        )
    # Phase 121 (task 02, LOCKED A6): normalize the credential — an
    # old-style embedded ``user:pass@`` URL is stored BARE and the
    # embedded credential moves to the token column; an explicit
    # ``token`` field wins over the embedded one. The duplicate check
    # below runs on the BARE URL, so the same repo pasted with a
    # different credential is the same source (409, not a second row).
    url, effective_token = normalize_credential(url, payload.token)
    if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
        raise HTTPException(status_code=409, detail="a git source with this URL already exists")
    # Phase 135: the branch pin (ASSUMPTION B2/B3) — absent or
    # trimmed-EMPTY stores NULL (the repo's default branch, the
    # pre-phase-135 behavior); non-empty is validated at the boundary
    # with the FIXED-detail 422 (the raw name never echoes).
    branch = (payload.branch or "").strip()
    if branch:
        try:
            branch = validate_branch_name(branch)
        except ValueError:
            raise HTTPException(
                status_code=422, detail="not a valid branch name"
            ) from None
    return _commit_new(
        GitSource(
            url=url,
            kind="git",
            ignore_paths=_validate_ignore_paths(payload.ignore_paths),
            include_hidden=bool(payload.include_hidden),
            token=effective_token or None,
            branch=branch or None,
        ),
        "a git source with this URL already exists",
        db,
    )


def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
    """``kind=local`` — fail-loud add-time validation (phase 38):
    trimmed → ``expanduser()`` → absolute + existing directory, else 422
    naming the path (not a secret, unlike a git URL). A ``branch`` on a
    local row (phase 135) is 422 — local rows are walked, not cloned."""
    if payload.url is not None:
        raise HTTPException(status_code=422, detail="a local source takes a path, not a url")
    if (payload.branch or "").strip():
        # Phase 135: local rows are walked, not cloned — a branch pin
        # has no meaning for them (the kind-field rules' precedent:
        # fixed detail, no input echo).
        raise HTTPException(
            status_code=422, detail="branch is only valid for git sources"
        )
    if payload.path is None:
        raise HTTPException(status_code=422, detail="a local source requires a path")
    expanded = Path(payload.path).expanduser()
    if not expanded.is_absolute() or not expanded.is_dir():
        raise HTTPException(
            status_code=422, detail=f"local source path is not a directory: {expanded}"
        )
    path = str(expanded)
    if db.scalar(select(GitSource).where(GitSource.path == path)) is not None:
        raise HTTPException(
            status_code=409, detail=f"a local source with this path already exists: {path}"
        )
    # ``url`` is the table's NOT-NULL location column (phase 38: local
    # rows carry the expanded path there too — git URL shapes and absolute
    # paths cannot collide). A ``token`` on a local row (phase 121) is
    # stored inert — local rows are walked, not cloned, so
    # ``clone_url_for`` never sees it — and, like on git rows, is never
    # echoed by any output shape.
    return _commit_new(
        GitSource(
            url=path,
            kind="local",
            path=path,
            ignore_paths=_validate_ignore_paths(payload.ignore_paths),
            include_hidden=bool(payload.include_hidden),
            token=payload.token or None,
        ),
        f"a local source with this path already exists: {path}",
        db,
    )


@router.patch("/{source_id}", response_model=GitSourceOut)
def patch_git_source(
    source_id: uuid.UUID,
    payload: GitSourcePatchIn,
    db: Session = Depends(get_db),  # noqa: B008
) -> GitSourceOut:
    """Edit one source's ignore list, hidden-folders flag, private-
    repo token, and/or branch.

    Phase 89 A5 (ignore list) + phase 105 (the flag) + phase 121
    (the token) + phase 135 (the branch): 404 unknown id; each PRESENT
    body field applies independently — ``ignore_paths`` REPLACES the
    list (normalized + A4-validated, fixed 422 details);
    ``include_hidden`` sets the flag; ``token`` is TRI-STATE (LOCKED
    A2): absent/None = no change (the row's stored credential survives
    an edit that does not touch the masked field), non-empty = replace,
    empty string = clear (stored NULL). A PRESENT token also
    re-normalizes the (current url, new token) pair with the POST
    write-path rules — a legacy embedded-token URL gets its userinfo
    stripped (moved to the column) the first time an explicit
    credential is written; a clean URL comes back untouched. The 409
    backstop: re-normalizing can make the stored URL collide with
    another row's bare URL (a legacy ``user:pass@`` row and a bare row
    for the same repo) — the unique index yields the generic 409,
    never a 500. ``branch`` (phase 135) is an independent PRESENT-
    wins field with the token's EXACT tri-state semantics:
    absent/None = no change (the row's stored branch survives an edit
    that does not touch it), non-empty = set (re-validated by
    :func:`validate_branch_name` — a ``ValueError`` is the FIXED-detail
    422 ``not a valid branch name``, input never echoed), empty string
    = clear (stored NULL = the repo's default branch). A PRESENT
    branch on a LOCAL row is 422 ``branch is only valid for git
    sources`` (local rows are walked, not cloned). Branch never
    touches a unique index — no Integrity interaction, no 409 path.
    Returns the updated row's public shape (id, url — masked,
    added_at, ignore_paths, include_hidden, branch); the token is
    never echoed.
    """
    row = db.get(GitSource, source_id)
    if row is None:
        raise HTTPException(status_code=404, detail="git source not found")
    if payload.ignore_paths is not None:
        row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
    if payload.include_hidden is not None:
        row.include_hidden = payload.include_hidden
    if payload.token is not None:
        # Phase 121 (task 02): the tri-state applies — "" clears
        # (stored NULL), non-empty replaces. Re-normalize the pair
        # (see the docstring): a legacy embedded-token URL becomes
        # bare + column credential.
        row.url, effective = normalize_credential(row.url, payload.token)
        row.token = effective or None
    if payload.branch is not None:
        # Phase 135: the branch tri-state (the token's exact semantics
        # — see the docstring). Local rows are walked, not cloned — a
        # branch pin is meaningless there (the same fixed 422 as POST).
        if row.kind != "git":
            raise HTTPException(
                status_code=422, detail="branch is only valid for git sources"
            )
        branch = payload.branch.strip()
        if not branch:
            row.branch = None  # the tri-state's clear leg (stored NULL)
        else:
            try:
                row.branch = validate_branch_name(branch)
            except ValueError:
                raise HTTPException(
                    status_code=422, detail="not a valid branch name"
                ) from None
    try:
        db.commit()
    except IntegrityError:
        # The re-normalized URL collided with another row's stored URL
        # (the legacy-embedded + bare sibling case) — the unique index
        # is the backstop: a generic 409, never a 500 (the phase-35
        # convention).
        db.rollback()
        raise HTTPException(
            status_code=409, detail="a git source with this URL already exists"
        ) from None
    db.refresh(row)
    return GitSourceOut(
        id=row.id,
        url=sanitize_url(row.url),  # phase 121: the output mask, always
        added_at=row.added_at,
        ignore_paths=row.ignore_paths,
        include_hidden=row.include_hidden,
        branch=row.branch,  # phase 135: NULL = default branch
    )


@router.post("/upload", response_model=UploadAccepted, status_code=202)
async def upload_archive(
    file: UploadFile = File(...),  # noqa: B008
) -> UploadAccepted:
    """Receive a source archive; unpack it and register the source row
    in the background — and nothing else (phase 49, backgrounded in
    phase 64 task 03, scan deferred in phase 90 — owner-locked A1/A2).

    The upload's job ends with the source row registered and the folder
    on disk: no model check, no import, no overview refresh (phase 90,
    A1 — the scan is the RAG page's "Sync sources" button's job, which
    gives the owner time to edit the new source's ignore list first;
    the sync already imports ``kind='local'`` rows with prune + the
    row's ignore list, A4).

    The **inline (request) work is exactly three gates** — steps 1–3 —
    everything else runs in a background task behind
    ``GET /upload/status`` (the phase-32 ``SyncStatus`` pattern), so
    navigating away mid-upload no longer aborts anything:

    1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip``
       (422 naming the accepted set) and a safe source name
       (``archive_source_name`` — its message is the 422 detail);
    2. one at a time — 409 ``an upload is already in progress`` while
       the flag is held (checked and set with no await in between,
       BEFORE the receive — see ``_upload_in_progress``);
    3. stream the upload in 1 MiB chunks into a dotfile temp with the
       ``upload_max_mb`` cap — 413 naming the cap, temp deleted.

    Then the archive is **safely on disk** — 202 + ``UploadAccepted``
    (the "successfully uploaded" moment the UI toasts on, A2) and
    :func:`_run_upload` runs the rest on the app's event loop:

    4. unpack to a temp sibling (traversal/symlink/device/corrupt/
       over-cap → ``failed`` with the task-01 user-safe message, temps
       deleted); a zero-entry archive is ``failed`` ``the archive
       contains no files`` — an archive with only non-importable files
       is a VALID replacement (the folder lands and the row registers;
       what the KB indexes with it is the sync's call);
    5. atomic swap-in — a same-name re-upload replaces the previous
       folder in place; a failure leaves the previous folder/row/KB
       untouched;
    6. upsert the row by ``path`` (``kind='local'``; an existing row is
       left as-is — ``added_at`` and ``ignore_paths`` preserved — and
       the unique index is the backstop: a concurrent insert lands
       ``failed`` with ``a local source with this path already exists:
       <path>``); the scan the sync later performs reads the row's
       ignore list straight off it (phase 89);
    7. one INFO log line (PLAN §9 / AGENTS.md rule 10 —
       ``upload: finished name=… file=… bytes=… total_ms=… state=…``;
       unpack+register only, no file counts — the state is ``success``
       or ``failed``, one line per run);
    8. ``success`` — ``detail = {"message": "uploaded"}`` (no count
       fields, phase 90 A2), ``current_file = None``,
       ``files_done = files_total = 0`` (the key set is unchanged —
       the UI composes the user copy).
    """
    # 1. Name/format gate — the accepted formats first (the 422 names
    #    them), then the task-01 safe-name derivation. A BARE suffix
    #    ("tar.gz") is an accepted format with no usable stem — it
    #    passes here and gets task-01's "no usable source name" 422.
    #    No upload dir is created for a rejected name.
    filename = file.filename or ""
    lowered = filename.lower()
    if not any(
        lowered.endswith(suffix) or lowered == suffix.lstrip(".")
        for suffix in ARCHIVE_SUFFIXES
    ):
        raise HTTPException(
            status_code=422,
            detail="only .tar, .tar.gz, .tgz or .zip archives are accepted",
        )
    try:
        name = archive_source_name(filename)
    except ArchiveUploadError as e:
        raise HTTPException(status_code=422, detail=str(e)) from None

    # 2. One at a time — the flag is checked and set with no await
    #    between, BEFORE the streaming receive: the background task
    #    does not exist yet, so the flag (not a task-done check) is the
    #    gate (see ``_upload_in_progress``).
    global _upload_in_progress
    if _upload_in_progress:
        raise HTTPException(status_code=409, detail="an upload is already in progress")
    _upload_in_progress = True

    settings = get_settings()
    upload_root = Path(settings.upload_dir).expanduser()
    upload_root.mkdir(parents=True, exist_ok=True)
    max_bytes = settings.upload_max_mb * 1024 * 1024
    temp_upload = upload_root / f".{name}.{uuid.uuid4().hex}.upload"
    temp_unpack = upload_root / f".{name}.{uuid.uuid4().hex}.unpack"
    try:
        # 3. Stream with the compressed-size cap — dotfile temps are
        #    hidden from the upload dir's listing.
        total = 0
        with open(temp_upload, "wb") as out:
            while chunk := await file.read(_STREAM_CHUNK):
                total += len(chunk)
                if total > max_bytes:
                    raise HTTPException(
                        status_code=413,
                        detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit",
                    )
                out.write(chunk)
        # The archive is safely on disk — 202 is the "successfully
        # uploaded" moment (phase 64 A2). Steps 4–8 (unpack → swap →
        # row upsert — no scan, phase 90) run in the background:
        asyncio.create_task(
            _run_upload(name, filename, total, upload_root, temp_upload, temp_unpack)
        )
    except BaseException:
        # The background task was never created (cap 413, a broken
        # pipe, cancellation, or create_task itself): release the flag
        # so the next upload is not refused, and make sure no temp
        # survives the failed receive.
        temp_upload.unlink(missing_ok=True)
        _upload_in_progress = False
        raise
    return UploadAccepted(name=name)


@router.get("/upload/status")
def upload_status() -> dict[str, Any]:
    """Current upload state (the UI polls this — the phase-32
    ``GET /api/sync/status`` contract, identical key set).

    ``started_at`` / ``finished_at`` are ISO-8601 strings or null.
    ``current_file`` / ``files_done`` / ``files_total`` stay null/0/0
    for the whole run (phase 90, A2 — the key set is unchanged, but
    uploads have no file-level progress: the scan the progress
    belonged to moved to the sync button, which keeps its live file
    label). The router dependency makes it admin-only like every other
    route here.
    """
    return {
        "state": _upload_status.state,
        "started_at": (
            _upload_status.started_at.isoformat() if _upload_status.started_at else None
        ),
        "finished_at": (
            _upload_status.finished_at.isoformat() if _upload_status.finished_at else None
        ),
        "detail": _upload_status.detail,
        "error": _upload_status.error,
        "current_file": _upload_status.current_file,
        "files_done": _upload_status.files_done,
        "files_total": _upload_status.files_total,
    }


async def _run_upload(
    name: str,
    filename: str,
    total_bytes: int,
    upload_root: Path,
    temp_upload: Path,
    temp_unpack: Path,
) -> None:
    """The post-202 upload pipeline, one in-process background task
    (the phase-32 ``_run_sync`` shape — phase 64 A1).

    Unpack → swap → row upsert — and nothing else (phase 90, A1: the
    model check, the import, and the overview refresh are the sync's
    job, not the upload's). Every failure mode (unpack, zero entries,
    swap, row, anything else) lands in the ``failed`` state with a
    sanitized ``error`` string — a background task must die in state,
    never as an unobserved exception (phase 64 A5: post-202 failures
    are status states, never HTTP errors). ``CancelledError`` is
    deliberately *not* caught: app shutdown cancels the task, and
    swallowing that would mask a real stop. The ``finally`` cleans both
    temps (defensive — each step already cleans its own) and clears
    ``_upload_in_progress``.
    """
    global _upload_in_progress
    started = time.monotonic()
    _upload_status.state = "running"
    _upload_status.started_at = datetime.now(UTC)
    _upload_status.finished_at = None
    _upload_status.current_file = None
    _upload_status.files_done = 0
    _upload_status.files_total = 0
    _upload_status.detail = {}
    _upload_status.error = None

    def _log_finished(state: str) -> None:
        # Per-upload log line (PLAN §9 / AGENTS.md rule 10) — unpack+
        # register only, no file counts (the scan's counts belong to
        # the sync, phase 90). One line per run, in BOTH terminal
        # states; ``total_ms`` is the background run's duration.
        logger.info(
            "upload: finished name=%s file=%s bytes=%d total_ms=%d state=%s",
            name,
            filename,
            total_bytes,
            round((time.monotonic() - started) * 1000),
            state,
        )

    try:
        settings = get_settings()
        max_bytes = settings.upload_max_mb * 1024 * 1024
        # Step 4 — unpack to a temp sibling; the compressed bytes are
        # no longer needed once unpacked (phase 49 locked decision:
        # only the unpacked content is kept).
        unpack_archive(temp_upload, temp_unpack, max_bytes)
        temp_upload.unlink(missing_ok=True)
        if not any(temp_unpack.iterdir()):
            # Zero entries = a user error. (Only non-importable files
            # is NOT an error — it still has entries and is a valid
            # replacement.)
            raise ArchiveUploadError("the archive contains no files")
        # Step 5 — swap in — a same-name re-upload replaces the
        # previous folder atomically; a failure leaves it, the row,
        # and the KB untouched (the ``failed`` state carries the
        # user-safe message).
        final_dir = upload_root / name
        swap_in(temp_unpack, final_dir)
        # Step 6 — upsert the row by path in a SHORT-LIVED session
        # (open/close around it — the ``effective_sources`` /
        # ``bump_sources_version`` pattern in ``app.api.sync``): the
        # background task has no request session to leak locks from
        # (the old inline ``db.close()`` discipline, now structural).
        # No duplicates: an existing row is left exactly as it is
        # (``added_at`` and ``ignore_paths`` preserved — the scan the
        # sync performs later reads the list straight off the row,
        # phase 89); the unique index is the backstop for a concurrent
        # insert the pre-check missed.
        path = str(final_dir)
        db = SessionLocal()
        try:
            row = db.scalar(select(GitSource).where(GitSource.path == path))
            if row is None:
                row = GitSource(url=path, kind="local", path=path)
                db.add(row)
                try:
                    db.commit()
                except IntegrityError:
                    db.rollback()
                    raise ValueError(
                        f"a local source with this path already exists: {path}"
                    ) from None
        finally:
            db.close()
        # Step 7 — the INFO line (``_log_finished`` — PLAN §9 /
        # AGENTS.md rule 10) lands together with the terminal state.
        # Step 8 — success: the no-count "uploaded" payload rides in
        # the status ``detail`` (phase 90 A2 — the key set is
        # unchanged; the scan's counts land in the SYNC's status when
        # the owner presses the button, and the UI composes the
        # user-facing result line from this payload).
        _upload_status.state = "success"
        _upload_status.finished_at = datetime.now(UTC)
        _upload_status.current_file = None
        _upload_status.files_done = 0
        _upload_status.files_total = 0
        _upload_status.detail = {"message": "uploaded"}
        _log_finished(_upload_status.state)
    except Exception as e:  # noqa: BLE001 — a background task dies in state, see above
        logger.exception("upload: failed")
        _upload_status.state = "failed"
        _upload_status.finished_at = datetime.now(UTC)
        _upload_status.error = _sanitize_error(str(e))
        _upload_status.current_file = None
        _log_finished(_upload_status.state)
    finally:
        _upload_in_progress = False
        # No temp may survive any failure path (defensive — each step
        # already cleans its own; on success both are already gone).
        temp_upload.unlink(missing_ok=True)
        shutil.rmtree(temp_unpack, ignore_errors=True)


@router.delete("/{source_id}", status_code=204)
async def delete_git_source(
    source_id: uuid.UUID,
    db: Session = Depends(get_db),  # noqa: B008
) -> Response:
    """Remove a stored source — **totally** (phase 69); 404 unknown id.

    Total removal, one action: the row, every indexed document of the
    source (chunks + embeddings dropped **in the database** by the
    ``chunks_document_id_fkey`` ``ON DELETE CASCADE`` — phase 183,
    ``Document.chunks`` is ``passive_deletes=True``, so the endpoint
    never loads a chunk row or its vector), and, for app-managed
    sources, the on-disk dir
    (the git checkout or the unpacked upload folder). Foreign local
    directories (the owner's own) are never touched on disk — their row
    + index entries still go.

    Locked order (phase 69):

    1. **Sibling guard** — another stored row resolves to the same
       source name (e.g. ``…/r`` and ``…/r.git``): it still owns the
       shared documents + files, so only this row is deleted (logged
       loudly); no index/disk work.
    2. **DB first** — in the request transaction: the source's
       documents are deleted, then the row, one ``commit``. A DB
       failure propagates as 500 **before any disk work** — the 204
       contract below never sees a half-removal.
    3. **Disk second** — the app-managed dir is removed after the
       commit; an ``OSError`` is logged, never fatal (a leftover dir
       is inert and self-heals on re-add; the reverse order is
       forbidden — a disk failure must never leave a row pointing at
       deleted files). Foreign local dir → skipped.
    4. **When documents were pruned** — the best-effort overview
       refresh (an LLM failure logs, never fails the 204; the next
       added/updated change refreshes it, as today) and then exactly
       one ``sources_version`` bump (phase 53 — the bump lands even if
       the overview failed, mirroring ``_run_sync``'s order). No docs
       pruned → no overview, no bump.

    The route contract is unchanged: 204 with no body; the per-
    operation INFO line (PLAN §9) carries the counts —
    ``files_removed`` is ``yes|no|skipped`` (``skipped`` for the
    sibling guard and for foreign local dirs).
    """
    started = time.monotonic()
    row = db.get(GitSource, source_id)
    if row is None:
        raise HTTPException(status_code=404, detail="git source not found")

    name = resolve_source_name(row)
    settings = get_settings()
    managed = managed_dir_for(row, Path(settings.sources_dir), Path(settings.upload_dir))

    if has_sibling(db, row):
        # Sibling guard: the shared documents + files still belong to
        # the sibling row — only this row goes (phase 69 locked
        # decision), so no index prune and no disk work.
        logger.warning(
            "source removal: row %s (url=%s) shares source name %s with "
            "another stored row — deleting only the row; the shared "
            "documents and files stay",
            row.id,
            row.url,
            name,
        )
        db.delete(row)
        db.commit()
        docs_pruned, files_removed, overview = 0, "skipped", False
    else:
        # DB first: prune every document of the source (the cascade
        # drops all chunks incl. embeddings) and the row itself in one
        # commit — the RAG is always consistent with the registry.
        docs = db.scalars(select(Document).where(Document.source == name)).all()
        for doc in docs:
            db.delete(doc)
        db.delete(row)
        db.commit()
        docs_pruned = len(docs)
        # Disk second: only the app-managed dir (git checkout / upload
        # folder) — ``managed is None`` is the foreign local dir (the
        # owner's own), never touched on disk.
        if managed is None:
            files_removed = "skipped"
        else:
            files_removed = "yes" if remove_managed_dir(managed) else "no"
        # KB changed (documents pruned) → best-effort overview refresh,
        # then exactly one sources_version bump — the bump in its own
        # short-lived session (the ``_run_sync`` pattern) so it lands
        # even if the best-effort overview failed.
        overview = False
        if docs_pruned > 0:
            try:
                overview = await regenerate_overview(LLMClient())
            except Exception:  # noqa: BLE001 — best-effort: never fail the 204
                logger.exception("source removal: overview regeneration failed (best-effort)")
                overview = False
            bump_db = SessionLocal()
            try:
                bump_sources_version(bump_db)
                bump_db.commit()
            finally:
                bump_db.close()

    logger.info(
        "source removed: kind=%s name=%s docs_pruned=%d files_removed=%s "
        "overview=%s total_ms=%d",
        row.kind,
        name,
        docs_pruned,
        files_removed,
        overview,
        round((time.monotonic() - started) * 1000),
    )
    return Response(status_code=204)