An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
210 lines
8.7 KiB
Python
210 lines
8.7 KiB
Python
"""Admin-managed sources API (phase 35, task 02; local kind, phase 38).
|
|
|
|
Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10
|
|
extended — the public API surface stays stateless and the signed cookie
|
|
remains the only session state, same as ``/api/steering`` and
|
|
``/api/sync``): the ``git_sources`` table holds the sources the Sync
|
|
button (phase 32) and ``import_docs`` (phase 28) import — ``kind='git'``
|
|
rows carry the repo URL to clone/pull, ``kind='local'`` rows (phase 38)
|
|
carry an existing directory on the server to walk directly. DB rows win
|
|
over ``BOR_GIT_SOURCES``, which is a git-only fallback while the table is
|
|
empty (the phase's locked decision — ``from_env`` tells the UI which list
|
|
it is looking at, so the page can show the env note only while the
|
|
fallback is active).
|
|
|
|
Routes: ``GET`` (DB rows oldest-first, or the env list with
|
|
``from_env: true`` while the table is empty; rows carry ``kind`` +
|
|
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
|
(201, validated create; ``kind`` selects the validation: git → exactly
|
|
the phase-35 URL contract, local → an existing absolute directory, else
|
|
422 naming the path), ``DELETE /{source_id}`` (204). The whole router
|
|
sits behind :func:`app.core.auth.require_admin` — anonymous callers get
|
|
403 on every route.
|
|
|
|
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
|
masking discipline), so every git 409/422 detail is a fixed generic
|
|
string that never repeats the submitted URL. Local paths are not
|
|
secrets — the local 422/409 details name the (expanded) path so the
|
|
owner sees exactly which directory failed.
|
|
|
|
Scope boundary (phase locked decisions): adding or removing a source
|
|
does NOT clone, import, or prune anything — the existing Sync button
|
|
performs that (a removal prunes on the next sync, ``prune=True``).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
import uuid
|
|
from pathlib import Path
|
|
from typing import Literal, cast
|
|
|
|
from fastapi import APIRouter, Depends, HTTPException, Response
|
|
from sqlalchemy import select
|
|
from sqlalchemy.exc import IntegrityError
|
|
from sqlalchemy.orm import Session
|
|
|
|
from app.config import get_settings
|
|
from app.core.auth import require_admin
|
|
from app.db import get_db
|
|
from app.models import GitSource
|
|
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow
|
|
|
|
router = APIRouter(
|
|
prefix="/git-sources",
|
|
tags=["git-sources"],
|
|
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
|
|
)
|
|
|
|
#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
|
|
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
|
|
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
|
|
#: accepted shapes are exactly these four prefixes.
|
|
URL_RE = re.compile(r"^(https?://|ssh://|git@)")
|
|
|
|
|
|
@router.get("", response_model=GitSourceList)
|
|
def list_git_sources(
|
|
db: Session = Depends(get_db), # noqa: B008
|
|
) -> GitSourceList:
|
|
"""The effective source list (git + local rows, phase 38).
|
|
|
|
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
|
|
same-timestamp inserts) with ``from_env: false`` — each row carries
|
|
its ``kind`` and, for local rows, the stored ``path`` (git rows and
|
|
env rows report ``path: null``); while the table is empty, the
|
|
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
|
|
git-only) with null ``id``/``added_at`` and ``from_env: true``.
|
|
"""
|
|
rows = db.scalars(
|
|
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
|
).all()
|
|
if rows:
|
|
return GitSourceList(
|
|
sources=[
|
|
# ``ck_git_sources_kind`` (migration 0007) guarantees the
|
|
# value is 'git' or 'local' — the cast documents that.
|
|
GitSourceRow(
|
|
id=row.id,
|
|
kind=cast(Literal["git", "local"], row.kind),
|
|
url=row.url,
|
|
path=row.path,
|
|
added_at=row.added_at,
|
|
)
|
|
for row in rows
|
|
],
|
|
from_env=False,
|
|
)
|
|
return GitSourceList(
|
|
sources=[
|
|
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None)
|
|
for url in get_settings().git_source_list
|
|
],
|
|
from_env=True,
|
|
)
|
|
|
|
|
|
@router.post("", response_model=GitSourceOut, status_code=201)
|
|
def create_git_source(
|
|
payload: GitSourceIn,
|
|
db: Session = Depends(get_db), # noqa: B008
|
|
) -> GitSourceOut:
|
|
"""Store one source (fields already trimmed by the schema).
|
|
|
|
``kind="git"`` (default) — exactly the phase-35 contract: 422 when
|
|
the URL shape is not one of the accepted prefixes (generic detail —
|
|
the input is never echoed), 409 when the trimmed URL is already
|
|
stored (the unique index is the backstop against a concurrent insert
|
|
the pre-check missed), 201 + the created row otherwise.
|
|
|
|
``kind="local"`` — ``path`` must expand (``~``) to an absolute,
|
|
existing directory on the server: 422 naming the path otherwise
|
|
(fail loud at add-time — the owner sees it immediately), 409 when
|
|
the path is already stored (detail names the path), 201 + the stored
|
|
row otherwise (``url`` holds the expanded path — the table's
|
|
NOT-NULL location column).
|
|
|
|
Wrong field combinations (git without url, local without path, both
|
|
kinds' fields) are 422 with fixed, input-free details.
|
|
"""
|
|
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
|
|
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
|
|
|
|
|
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
|
|
"""Insert ``row``; the unique index is the backstop — a concurrent
|
|
insert the pre-check missed still yields the generic 409, never a
|
|
500 (phase-35 convention, now shared by both kinds)."""
|
|
db.add(row)
|
|
try:
|
|
db.commit()
|
|
except IntegrityError:
|
|
db.rollback()
|
|
raise HTTPException(status_code=409, detail=duplicate_detail) from None
|
|
db.refresh(row)
|
|
return row
|
|
|
|
|
|
def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
|
|
"""``kind=git`` — the phase-35 URL contract, unchanged (A10: no
|
|
credential echo, so every detail is a fixed string)."""
|
|
if payload.path is not None:
|
|
raise HTTPException(status_code=422, detail="a git source takes a url, not a path")
|
|
if payload.url is None:
|
|
raise HTTPException(status_code=422, detail="a git source requires a url")
|
|
url = payload.url
|
|
if not URL_RE.match(url):
|
|
raise HTTPException(
|
|
status_code=422, detail="not a valid git URL (expected https://, ssh:// or git@…)"
|
|
)
|
|
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
|
|
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
|
|
return _commit_new(
|
|
GitSource(url=url, kind="git"), "a git source with this URL already exists", db
|
|
)
|
|
|
|
|
|
def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
|
"""``kind=local`` — fail-loud add-time validation (phase 38):
|
|
trimmed → ``expanduser()`` → absolute + existing directory, else 422
|
|
naming the path (not a secret, unlike a git URL)."""
|
|
if payload.url is not None:
|
|
raise HTTPException(status_code=422, detail="a local source takes a path, not a url")
|
|
if payload.path is None:
|
|
raise HTTPException(status_code=422, detail="a local source requires a path")
|
|
expanded = Path(payload.path).expanduser()
|
|
if not expanded.is_absolute() or not expanded.is_dir():
|
|
raise HTTPException(
|
|
status_code=422, detail=f"local source path is not a directory: {expanded}"
|
|
)
|
|
path = str(expanded)
|
|
if db.scalar(select(GitSource).where(GitSource.path == path)) is not None:
|
|
raise HTTPException(
|
|
status_code=409, detail=f"a local source with this path already exists: {path}"
|
|
)
|
|
# ``url`` is the table's NOT-NULL location column (phase 38: local
|
|
# rows carry the expanded path there too — git URL shapes and absolute
|
|
# paths cannot collide).
|
|
return _commit_new(
|
|
GitSource(url=path, kind="local", path=path),
|
|
f"a local source with this path already exists: {path}",
|
|
db,
|
|
)
|
|
|
|
|
|
@router.delete("/{source_id}", status_code=204)
|
|
def delete_git_source(
|
|
source_id: uuid.UUID,
|
|
db: Session = Depends(get_db), # noqa: B008
|
|
) -> Response:
|
|
"""Remove a stored row; 404 when the id is unknown.
|
|
|
|
Removing does not touch the clones or the index — the next Sync
|
|
(``prune=True``) prunes the dropped repo (phase scope boundary).
|
|
"""
|
|
row = db.get(GitSource, source_id)
|
|
if row is None:
|
|
raise HTTPException(status_code=404, detail="git source not found")
|
|
db.delete(row)
|
|
db.commit()
|
|
return Response(status_code=204)
|