feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
+117
-40
@@ -1,33 +1,42 @@
|
||||
"""Admin-managed git sources API (phase 35, task 02).
|
||||
"""Admin-managed sources API (phase 35, task 02; local kind, phase 38).
|
||||
|
||||
Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10
|
||||
extended — the public API surface stays stateless and the signed cookie
|
||||
remains the only session state, same as ``/api/steering`` and
|
||||
``/api/sync``): the ``git_sources`` table holds the repo URLs the Sync
|
||||
button (phase 32) and ``import_docs`` (phase 28) clone/pull. DB rows win
|
||||
over ``BOR_GIT_SOURCES``, which is a fallback while the table is empty
|
||||
(the phase's locked decision — ``from_env`` tells the UI which list it is
|
||||
looking at, so the page can show the env note only while the fallback is
|
||||
active).
|
||||
``/api/sync``): the ``git_sources`` table holds the sources the Sync
|
||||
button (phase 32) and ``import_docs`` (phase 28) import — ``kind='git'``
|
||||
rows carry the repo URL to clone/pull, ``kind='local'`` rows (phase 38)
|
||||
carry an existing directory on the server to walk directly. DB rows win
|
||||
over ``BOR_GIT_SOURCES``, which is a git-only fallback while the table is
|
||||
empty (the phase's locked decision — ``from_env`` tells the UI which list
|
||||
it is looking at, so the page can show the env note only while the
|
||||
fallback is active).
|
||||
|
||||
Routes: ``GET`` (DB rows oldest-first, or the env list with
|
||||
``from_env: true`` while the table is empty), ``POST`` (201, validated
|
||||
create), ``DELETE /{source_id}`` (204). The whole router sits behind
|
||||
:func:`app.core.auth.require_admin` — anonymous callers get 403 on every
|
||||
route.
|
||||
``from_env: true`` while the table is empty; rows carry ``kind`` +
|
||||
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
||||
(201, validated create; ``kind`` selects the validation: git → exactly
|
||||
the phase-35 URL contract, local → an existing absolute directory, else
|
||||
422 naming the path), ``DELETE /{source_id}`` (204). The whole router
|
||||
sits behind :func:`app.core.auth.require_admin` — anonymous callers get
|
||||
403 on every route.
|
||||
|
||||
No credential-echo path: URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so the 409/422 details are fixed generic strings
|
||||
that never repeat the submitted URL.
|
||||
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so every git 409/422 detail is a fixed generic
|
||||
string that never repeats the submitted URL. Local paths are not
|
||||
secrets — the local 422/409 details name the (expanded) path so the
|
||||
owner sees exactly which directory failed.
|
||||
|
||||
Scope boundary (phase locked decisions): adding or removing a repo does
|
||||
NOT clone, import, or prune anything — the existing Sync button performs
|
||||
that (a removal prunes on the next sync, ``prune=True``).
|
||||
Scope boundary (phase locked decisions): adding or removing a source
|
||||
does NOT clone, import, or prune anything — the existing Sync button
|
||||
performs that (a removal prunes on the next sync, ``prune=True``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from sqlalchemy import select
|
||||
@@ -38,7 +47,7 @@ from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import get_db
|
||||
from app.models import GitSource
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/git-sources",
|
||||
@@ -57,24 +66,37 @@ URL_RE = re.compile(r"^(https?://|ssh://|git@)")
|
||||
def list_git_sources(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> GitSourceList:
|
||||
"""The effective git source list.
|
||||
"""The effective source list (git + local rows, phase 38).
|
||||
|
||||
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
|
||||
same-timestamp inserts) with ``from_env: false``; while the table is
|
||||
empty, the ``BOR_GIT_SOURCES`` env URLs as rows with null
|
||||
``id``/``added_at`` and ``from_env: true``.
|
||||
same-timestamp inserts) with ``from_env: false`` — each row carries
|
||||
its ``kind`` and, for local rows, the stored ``path`` (git rows and
|
||||
env rows report ``path: null``); while the table is empty, the
|
||||
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
|
||||
git-only) with null ``id``/``added_at`` and ``from_env: true``.
|
||||
"""
|
||||
rows = db.scalars(
|
||||
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
||||
).all()
|
||||
if rows:
|
||||
return GitSourceList(
|
||||
sources=[GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) for row in rows],
|
||||
sources=[
|
||||
# ``ck_git_sources_kind`` (migration 0007) guarantees the
|
||||
# value is 'git' or 'local' — the cast documents that.
|
||||
GitSourceRow(
|
||||
id=row.id,
|
||||
kind=cast(Literal["git", "local"], row.kind),
|
||||
url=row.url,
|
||||
path=row.path,
|
||||
added_at=row.added_at,
|
||||
)
|
||||
for row in rows
|
||||
],
|
||||
from_env=False,
|
||||
)
|
||||
return GitSourceList(
|
||||
sources=[
|
||||
GitSourceOut(id=None, url=url, added_at=None)
|
||||
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None)
|
||||
for url in get_settings().git_source_list
|
||||
],
|
||||
from_env=True,
|
||||
@@ -86,14 +108,49 @@ def create_git_source(
|
||||
payload: GitSourceIn,
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> GitSourceOut:
|
||||
"""Store one repo URL (already trimmed by the schema).
|
||||
"""Store one source (fields already trimmed by the schema).
|
||||
|
||||
422 when the shape is not one of the accepted prefixes (generic
|
||||
detail — the input is never echoed); 409 when the trimmed URL is
|
||||
already stored (same; the unique index is the backstop against a
|
||||
concurrent insert the pre-check missed); 201 + the created row
|
||||
otherwise.
|
||||
``kind="git"`` (default) — exactly the phase-35 contract: 422 when
|
||||
the URL shape is not one of the accepted prefixes (generic detail —
|
||||
the input is never echoed), 409 when the trimmed URL is already
|
||||
stored (the unique index is the backstop against a concurrent insert
|
||||
the pre-check missed), 201 + the created row otherwise.
|
||||
|
||||
``kind="local"`` — ``path`` must expand (``~``) to an absolute,
|
||||
existing directory on the server: 422 naming the path otherwise
|
||||
(fail loud at add-time — the owner sees it immediately), 409 when
|
||||
the path is already stored (detail names the path), 201 + the stored
|
||||
row otherwise (``url`` holds the expanded path — the table's
|
||||
NOT-NULL location column).
|
||||
|
||||
Wrong field combinations (git without url, local without path, both
|
||||
kinds' fields) are 422 with fixed, input-free details.
|
||||
"""
|
||||
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
|
||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
||||
|
||||
|
||||
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
|
||||
"""Insert ``row``; the unique index is the backstop — a concurrent
|
||||
insert the pre-check missed still yields the generic 409, never a
|
||||
500 (phase-35 convention, now shared by both kinds)."""
|
||||
db.add(row)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise HTTPException(status_code=409, detail=duplicate_detail) from None
|
||||
db.refresh(row)
|
||||
return row
|
||||
|
||||
|
||||
def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
"""``kind=git`` — the phase-35 URL contract, unchanged (A10: no
|
||||
credential echo, so every detail is a fixed string)."""
|
||||
if payload.path is not None:
|
||||
raise HTTPException(status_code=422, detail="a git source takes a url, not a path")
|
||||
if payload.url is None:
|
||||
raise HTTPException(status_code=422, detail="a git source requires a url")
|
||||
url = payload.url
|
||||
if not URL_RE.match(url):
|
||||
raise HTTPException(
|
||||
@@ -101,17 +158,37 @@ def create_git_source(
|
||||
)
|
||||
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
|
||||
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
|
||||
row = GitSource(url=url)
|
||||
db.add(row)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
return _commit_new(
|
||||
GitSource(url=url, kind="git"), "a git source with this URL already exists", db
|
||||
)
|
||||
|
||||
|
||||
def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
"""``kind=local`` — fail-loud add-time validation (phase 38):
|
||||
trimmed → ``expanduser()`` → absolute + existing directory, else 422
|
||||
naming the path (not a secret, unlike a git URL)."""
|
||||
if payload.url is not None:
|
||||
raise HTTPException(status_code=422, detail="a local source takes a path, not a url")
|
||||
if payload.path is None:
|
||||
raise HTTPException(status_code=422, detail="a local source requires a path")
|
||||
expanded = Path(payload.path).expanduser()
|
||||
if not expanded.is_absolute() or not expanded.is_dir():
|
||||
raise HTTPException(
|
||||
status_code=409, detail="a git source with this URL already exists"
|
||||
) from None
|
||||
db.refresh(row)
|
||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
||||
status_code=422, detail=f"local source path is not a directory: {expanded}"
|
||||
)
|
||||
path = str(expanded)
|
||||
if db.scalar(select(GitSource).where(GitSource.path == path)) is not None:
|
||||
raise HTTPException(
|
||||
status_code=409, detail=f"a local source with this path already exists: {path}"
|
||||
)
|
||||
# ``url`` is the table's NOT-NULL location column (phase 38: local
|
||||
# rows carry the expanded path there too — git URL shapes and absolute
|
||||
# paths cannot collide).
|
||||
return _commit_new(
|
||||
GitSource(url=path, kind="local", path=path),
|
||||
f"a local source with this path already exists: {path}",
|
||||
db,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{source_id}", status_code=204)
|
||||
|
||||
Reference in New Issue
Block a user