feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
+117
-40
@@ -1,33 +1,42 @@
|
||||
"""Admin-managed git sources API (phase 35, task 02).
|
||||
"""Admin-managed sources API (phase 35, task 02; local kind, phase 38).
|
||||
|
||||
Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10
|
||||
extended — the public API surface stays stateless and the signed cookie
|
||||
remains the only session state, same as ``/api/steering`` and
|
||||
``/api/sync``): the ``git_sources`` table holds the repo URLs the Sync
|
||||
button (phase 32) and ``import_docs`` (phase 28) clone/pull. DB rows win
|
||||
over ``BOR_GIT_SOURCES``, which is a fallback while the table is empty
|
||||
(the phase's locked decision — ``from_env`` tells the UI which list it is
|
||||
looking at, so the page can show the env note only while the fallback is
|
||||
active).
|
||||
``/api/sync``): the ``git_sources`` table holds the sources the Sync
|
||||
button (phase 32) and ``import_docs`` (phase 28) import — ``kind='git'``
|
||||
rows carry the repo URL to clone/pull, ``kind='local'`` rows (phase 38)
|
||||
carry an existing directory on the server to walk directly. DB rows win
|
||||
over ``BOR_GIT_SOURCES``, which is a git-only fallback while the table is
|
||||
empty (the phase's locked decision — ``from_env`` tells the UI which list
|
||||
it is looking at, so the page can show the env note only while the
|
||||
fallback is active).
|
||||
|
||||
Routes: ``GET`` (DB rows oldest-first, or the env list with
|
||||
``from_env: true`` while the table is empty), ``POST`` (201, validated
|
||||
create), ``DELETE /{source_id}`` (204). The whole router sits behind
|
||||
:func:`app.core.auth.require_admin` — anonymous callers get 403 on every
|
||||
route.
|
||||
``from_env: true`` while the table is empty; rows carry ``kind`` +
|
||||
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
||||
(201, validated create; ``kind`` selects the validation: git → exactly
|
||||
the phase-35 URL contract, local → an existing absolute directory, else
|
||||
422 naming the path), ``DELETE /{source_id}`` (204). The whole router
|
||||
sits behind :func:`app.core.auth.require_admin` — anonymous callers get
|
||||
403 on every route.
|
||||
|
||||
No credential-echo path: URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so the 409/422 details are fixed generic strings
|
||||
that never repeat the submitted URL.
|
||||
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so every git 409/422 detail is a fixed generic
|
||||
string that never repeats the submitted URL. Local paths are not
|
||||
secrets — the local 422/409 details name the (expanded) path so the
|
||||
owner sees exactly which directory failed.
|
||||
|
||||
Scope boundary (phase locked decisions): adding or removing a repo does
|
||||
NOT clone, import, or prune anything — the existing Sync button performs
|
||||
that (a removal prunes on the next sync, ``prune=True``).
|
||||
Scope boundary (phase locked decisions): adding or removing a source
|
||||
does NOT clone, import, or prune anything — the existing Sync button
|
||||
performs that (a removal prunes on the next sync, ``prune=True``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from sqlalchemy import select
|
||||
@@ -38,7 +47,7 @@ from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import get_db
|
||||
from app.models import GitSource
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/git-sources",
|
||||
@@ -57,24 +66,37 @@ URL_RE = re.compile(r"^(https?://|ssh://|git@)")
|
||||
def list_git_sources(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> GitSourceList:
|
||||
"""The effective git source list.
|
||||
"""The effective source list (git + local rows, phase 38).
|
||||
|
||||
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
|
||||
same-timestamp inserts) with ``from_env: false``; while the table is
|
||||
empty, the ``BOR_GIT_SOURCES`` env URLs as rows with null
|
||||
``id``/``added_at`` and ``from_env: true``.
|
||||
same-timestamp inserts) with ``from_env: false`` — each row carries
|
||||
its ``kind`` and, for local rows, the stored ``path`` (git rows and
|
||||
env rows report ``path: null``); while the table is empty, the
|
||||
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
|
||||
git-only) with null ``id``/``added_at`` and ``from_env: true``.
|
||||
"""
|
||||
rows = db.scalars(
|
||||
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
||||
).all()
|
||||
if rows:
|
||||
return GitSourceList(
|
||||
sources=[GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) for row in rows],
|
||||
sources=[
|
||||
# ``ck_git_sources_kind`` (migration 0007) guarantees the
|
||||
# value is 'git' or 'local' — the cast documents that.
|
||||
GitSourceRow(
|
||||
id=row.id,
|
||||
kind=cast(Literal["git", "local"], row.kind),
|
||||
url=row.url,
|
||||
path=row.path,
|
||||
added_at=row.added_at,
|
||||
)
|
||||
for row in rows
|
||||
],
|
||||
from_env=False,
|
||||
)
|
||||
return GitSourceList(
|
||||
sources=[
|
||||
GitSourceOut(id=None, url=url, added_at=None)
|
||||
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None)
|
||||
for url in get_settings().git_source_list
|
||||
],
|
||||
from_env=True,
|
||||
@@ -86,14 +108,49 @@ def create_git_source(
|
||||
payload: GitSourceIn,
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> GitSourceOut:
|
||||
"""Store one repo URL (already trimmed by the schema).
|
||||
"""Store one source (fields already trimmed by the schema).
|
||||
|
||||
422 when the shape is not one of the accepted prefixes (generic
|
||||
detail — the input is never echoed); 409 when the trimmed URL is
|
||||
already stored (same; the unique index is the backstop against a
|
||||
concurrent insert the pre-check missed); 201 + the created row
|
||||
otherwise.
|
||||
``kind="git"`` (default) — exactly the phase-35 contract: 422 when
|
||||
the URL shape is not one of the accepted prefixes (generic detail —
|
||||
the input is never echoed), 409 when the trimmed URL is already
|
||||
stored (the unique index is the backstop against a concurrent insert
|
||||
the pre-check missed), 201 + the created row otherwise.
|
||||
|
||||
``kind="local"`` — ``path`` must expand (``~``) to an absolute,
|
||||
existing directory on the server: 422 naming the path otherwise
|
||||
(fail loud at add-time — the owner sees it immediately), 409 when
|
||||
the path is already stored (detail names the path), 201 + the stored
|
||||
row otherwise (``url`` holds the expanded path — the table's
|
||||
NOT-NULL location column).
|
||||
|
||||
Wrong field combinations (git without url, local without path, both
|
||||
kinds' fields) are 422 with fixed, input-free details.
|
||||
"""
|
||||
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
|
||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
||||
|
||||
|
||||
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
|
||||
"""Insert ``row``; the unique index is the backstop — a concurrent
|
||||
insert the pre-check missed still yields the generic 409, never a
|
||||
500 (phase-35 convention, now shared by both kinds)."""
|
||||
db.add(row)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise HTTPException(status_code=409, detail=duplicate_detail) from None
|
||||
db.refresh(row)
|
||||
return row
|
||||
|
||||
|
||||
def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
"""``kind=git`` — the phase-35 URL contract, unchanged (A10: no
|
||||
credential echo, so every detail is a fixed string)."""
|
||||
if payload.path is not None:
|
||||
raise HTTPException(status_code=422, detail="a git source takes a url, not a path")
|
||||
if payload.url is None:
|
||||
raise HTTPException(status_code=422, detail="a git source requires a url")
|
||||
url = payload.url
|
||||
if not URL_RE.match(url):
|
||||
raise HTTPException(
|
||||
@@ -101,17 +158,37 @@ def create_git_source(
|
||||
)
|
||||
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
|
||||
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
|
||||
row = GitSource(url=url)
|
||||
db.add(row)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
return _commit_new(
|
||||
GitSource(url=url, kind="git"), "a git source with this URL already exists", db
|
||||
)
|
||||
|
||||
|
||||
def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
"""``kind=local`` — fail-loud add-time validation (phase 38):
|
||||
trimmed → ``expanduser()`` → absolute + existing directory, else 422
|
||||
naming the path (not a secret, unlike a git URL)."""
|
||||
if payload.url is not None:
|
||||
raise HTTPException(status_code=422, detail="a local source takes a path, not a url")
|
||||
if payload.path is None:
|
||||
raise HTTPException(status_code=422, detail="a local source requires a path")
|
||||
expanded = Path(payload.path).expanduser()
|
||||
if not expanded.is_absolute() or not expanded.is_dir():
|
||||
raise HTTPException(
|
||||
status_code=409, detail="a git source with this URL already exists"
|
||||
) from None
|
||||
db.refresh(row)
|
||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
||||
status_code=422, detail=f"local source path is not a directory: {expanded}"
|
||||
)
|
||||
path = str(expanded)
|
||||
if db.scalar(select(GitSource).where(GitSource.path == path)) is not None:
|
||||
raise HTTPException(
|
||||
status_code=409, detail=f"a local source with this path already exists: {path}"
|
||||
)
|
||||
# ``url`` is the table's NOT-NULL location column (phase 38: local
|
||||
# rows carry the expanded path there too — git URL shapes and absolute
|
||||
# paths cannot collide).
|
||||
return _commit_new(
|
||||
GitSource(url=path, kind="local", path=path),
|
||||
f"a local source with this path already exists: {path}",
|
||||
db,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{source_id}", status_code=204)
|
||||
|
||||
+50
-26
@@ -12,20 +12,27 @@ plus a module-level :class:`SyncStatus` that the UI polls every 2 s
|
||||
409; the status object is authoritative, so the UI can never sit on a
|
||||
stale button state (§7.4 adaptation, phase locked decisions).
|
||||
|
||||
Pipeline (the canonical "mirror the repos" action — phase locked
|
||||
Pipeline (the canonical "mirror the sources" action — phase locked
|
||||
decisions):
|
||||
|
||||
1. resolve the effective git sources — the ``git_sources`` DB rows,
|
||||
else the ``BOR_GIT_SOURCES`` fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`, shared with the
|
||||
CLI) — empty on both origins fails loudly (``no git sources
|
||||
configured``) instead of silently importing the legacy local
|
||||
directories;
|
||||
2. :func:`scripts.git_sync.clone_or_pull` each repo into
|
||||
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
||||
re-implemented; a failing repo aborts before any import);
|
||||
3. ``import_sources(..., prune=True)`` — prune so files deleted
|
||||
upstream leave the index (the CLI's no-prune default is unchanged);
|
||||
1. resolve the effective sources — the ``git_sources`` DB rows (git
|
||||
**and** local, phase 38), else the ``BOR_GIT_SOURCES`` fallback
|
||||
(git-only)
|
||||
(:func:`app.rag.git_sources.effective_sources`, shared with the
|
||||
CLI) — empty on both origins (no git rows, no local rows, no env
|
||||
URLs) fails loudly (``no sources configured (git or local)``)
|
||||
instead of silently importing the legacy local directories;
|
||||
2. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull`
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
||||
re-implemented); ``kind=local`` → the stored directory, re-verified
|
||||
``.is_dir()`` **at sync time** (it may have moved/deleted since
|
||||
add-time) — a missing directory raises ``local source missing:
|
||||
<path>``; a failing clone or a missing local dir aborts before any
|
||||
import;
|
||||
3. ``import_sources(..., prune=True)`` over the single combined list
|
||||
(git checkouts + local dirs) — prune so files deleted upstream or
|
||||
out of a local dir leave the index (pruning covers the union; the
|
||||
CLI's no-prune default is unchanged);
|
||||
4. when the import changed the KB (added + updated > 0),
|
||||
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
||||
(phase 31 trigger, best-effort inside).
|
||||
@@ -48,7 +55,7 @@ from fastapi import APIRouter, Depends, HTTPException
|
||||
from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import SessionLocal
|
||||
from app.rag.git_sources import effective_git_sources
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.overview import regenerate_overview
|
||||
@@ -147,24 +154,41 @@ async def _run_sync() -> None:
|
||||
try:
|
||||
settings = get_settings()
|
||||
# The background task has no request session: open a short-lived
|
||||
# one around the shared phase-35 resolver (DB rows win, the
|
||||
# BOR_GIT_SOURCES list is a fallback while the table is empty).
|
||||
# one around the shared phase-35/38 resolver (DB rows of both
|
||||
# kinds win; the BOR_GIT_SOURCES git list is a fallback while
|
||||
# the table is empty).
|
||||
db = SessionLocal()
|
||||
try:
|
||||
git_urls, origin = effective_git_sources(db)
|
||||
rows, origin = effective_sources(db)
|
||||
finally:
|
||||
db.close()
|
||||
if not git_urls:
|
||||
# The button targets git sources only (manual --source dirs
|
||||
# have no repo to clone) — an empty config on *both* origins
|
||||
# fails loudly instead of silently importing the legacy
|
||||
# directories.
|
||||
raise GitSyncError(
|
||||
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
|
||||
)
|
||||
logger.info("sync: started repos=%d origin=%s", len(git_urls), origin)
|
||||
if not rows:
|
||||
# The button targets the admin-managed source registry
|
||||
# (manual --source dirs have no repo to clone) — an empty
|
||||
# config on *both* origins (no git rows, no local rows, no
|
||||
# env URLs) fails loudly instead of silently importing the
|
||||
# legacy directories.
|
||||
raise GitSyncError("no sources configured (git or local)")
|
||||
git_count = sum(1 for row in rows if row.kind == "git")
|
||||
logger.info(
|
||||
"sync: started repos=%d origin=%s git=%d local=%d",
|
||||
len(rows), origin, git_count, len(rows) - git_count,
|
||||
)
|
||||
sources_root = Path(settings.sources_dir).expanduser()
|
||||
sources = [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
|
||||
sources: list[Path] = []
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
# ``or`` keeps the type checker honest); re-verified at
|
||||
# sync time because the directory may have moved or been
|
||||
# deleted since add-time.
|
||||
path = Path(row.path or row.url).expanduser()
|
||||
if not path.is_dir():
|
||||
raise GitSyncError(f"local source missing: {path}")
|
||||
sources.append(path)
|
||||
llm = LLMClient()
|
||||
summary: ImportSummary = await import_sources(sources, llm, prune=True)
|
||||
overview = False
|
||||
|
||||
+18
-7
@@ -13,8 +13,10 @@ Data model — see ``.agent/PLAN.md`` §Data Model:
|
||||
* ``kb_overview`` — single-row lite-generated outline of the KB's basic
|
||||
categories, injected as the ``<knowledge_base>``
|
||||
section of every chat turn (phase 31).
|
||||
* ``git_sources`` — admin-managed git source URLs the Sync button and
|
||||
import_docs clone/pull (phase 35).
|
||||
* ``git_sources`` — admin-managed source registry (git URLs + local
|
||||
directories) the Sync button and import_docs
|
||||
import (phase 35; ``kind`` discriminator added in
|
||||
phase 38).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -138,16 +140,25 @@ class KbOverview(Base):
|
||||
|
||||
|
||||
class GitSource(Base):
|
||||
"""One admin-managed git source (phase 35).
|
||||
"""One admin-managed source (phase 35; kind discriminator, phase 38).
|
||||
|
||||
The UI-maintained list of repo URLs the Sync button (phase 32) and
|
||||
import_docs (phase 28) clone/pull. DB rows win over the
|
||||
BOR_GIT_SOURCES env var, which is a fallback while this table is
|
||||
empty (see app.rag.git_sources.effective_git_sources).
|
||||
The UI-maintained list the Sync button (phase 32) and import_docs
|
||||
(phase 28) import from. ``kind`` discriminates: ``git`` rows carry a
|
||||
repo ``url`` (cloned/pulled), ``local`` rows carry an existing
|
||||
directory ``path`` (walked directly). DB rows win over the
|
||||
BOR_GIT_SOURCES env var (git-only fallback), which is a fallback
|
||||
while this table is empty (see
|
||||
app.rag.git_sources.effective_git_sources).
|
||||
"""
|
||||
|
||||
__tablename__ = "git_sources"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
|
||||
#: Source-kind discriminator (phase 38): "git" (default) or "local"
|
||||
#: — enforced by the ``ck_git_sources_kind`` CHECK constraint.
|
||||
kind: Mapped[str] = mapped_column(Text, default="git", server_default="'git'")
|
||||
#: Absolute directory of a ``local`` source; NULL for git rows.
|
||||
#: Unique — Postgres treats NULLs as distinct under a unique index.
|
||||
path: Mapped[str | None] = mapped_column(Text, unique=True)
|
||||
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
+45
-20
@@ -1,19 +1,26 @@
|
||||
"""The shared git-source resolver (phase 35, task 03).
|
||||
"""The shared source resolver (phase 35, task 03; local kind, phase 38).
|
||||
|
||||
One function, two callers: the in-app Sync pipeline
|
||||
(``app/api/sync.py::_run_sync``) and the CLI
|
||||
(``scripts/import_docs.py::_resolve_sources``) — both resolve the repo
|
||||
URLs to clone/pull through :func:`effective_git_sources`, so the
|
||||
admin-managed ``git_sources`` table is what actually gets cloned and
|
||||
indexed from either entry point (the API's ``GET /api/git-sources``
|
||||
fallback list is the only other place that reads the env list — task 02).
|
||||
(``scripts/import_docs.py::_resolve_sources``) — both resolve the
|
||||
``git_sources`` rows to import through :func:`effective_sources`, so
|
||||
the admin-managed ``git_sources`` table is what actually gets cloned
|
||||
(git rows) or walked directly (local rows) from either entry point
|
||||
(the API's ``GET /api/git-sources`` fallback list is the only other
|
||||
place that reads the env list — task 02).
|
||||
|
||||
Precedence (the phase's locked decision — the env var is demoted, not
|
||||
removed): ``git_sources`` DB rows win, in ``(added_at, id)`` order;
|
||||
``BOR_GIT_SOURCES`` is a fallback only while the table is empty; both
|
||||
empty → ``([], "env")`` and each caller keeps its existing fail-loud
|
||||
behavior (sync: ``GitSyncError``; CLI: the legacy ``DEFAULT_SOURCES``
|
||||
fallback).
|
||||
removed): ``git_sources`` DB rows of **both kinds** win, in
|
||||
``(added_at, id)`` order; ``BOR_GIT_SOURCES`` is a fallback only while
|
||||
the table is empty (git URLs surface as synthetic ``kind='git'`` rows —
|
||||
the env fallback is git-only); both empty → ``([], "env")`` and each
|
||||
caller keeps its existing fail-loud behavior (sync: ``GitSyncError``;
|
||||
CLI: the legacy ``DEFAULT_SOURCES`` fallback).
|
||||
|
||||
:func:`effective_git_sources` is kept as the phase-35 back-compat alias
|
||||
(repo URLs of the effective git rows) so existing importers of the old
|
||||
name keep working; new code calls :func:`effective_sources` and
|
||||
branches on ``row.kind``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -26,19 +33,37 @@ from app.config import get_settings
|
||||
from app.models import GitSource
|
||||
|
||||
|
||||
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
|
||||
"""``(urls, origin)`` — the repo URLs to clone, and where they came from.
|
||||
def effective_sources(db: Session) -> tuple[list[GitSource], Literal["db", "env"]]:
|
||||
"""``(rows, origin)`` — the effective source rows (both kinds) and
|
||||
where they came from.
|
||||
|
||||
``"db"``: the ``git_sources`` rows in ``(added_at, id)`` order (oldest
|
||||
first, id tie-break for same-timestamp inserts) — once the table has
|
||||
any row, ``BOR_GIT_SOURCES`` is ignored entirely.
|
||||
``"db"``: the ``git_sources`` rows — ``kind='git'`` (repo URL to
|
||||
clone/pull) and ``kind='local'`` (existing directory to walk
|
||||
directly) — in ``(added_at, id)`` order (oldest first, id tie-break
|
||||
for same-timestamp inserts). Once the table has any row,
|
||||
``BOR_GIT_SOURCES`` is ignored entirely.
|
||||
``"env"``: the ``BOR_GIT_SOURCES`` list — ``Settings.git_source_list``,
|
||||
the phase-28 CSV parse, reused not re-implemented — used only while
|
||||
the table is empty; both empty → ``([], "env")``.
|
||||
the phase-28 CSV parse, reused not re-implemented — surfaced as
|
||||
synthetic ``kind='git'`` rows (``path`` NULL; the env fallback is
|
||||
git-only), used only while the table is empty; both empty →
|
||||
``([], "env")``.
|
||||
"""
|
||||
rows = db.scalars(
|
||||
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
||||
).all()
|
||||
if rows:
|
||||
return [row.url for row in rows], "db"
|
||||
return list(get_settings().git_source_list), "env"
|
||||
return list(rows), "db"
|
||||
return [GitSource(url=url, kind="git") for url in get_settings().git_source_list], "env"
|
||||
|
||||
|
||||
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
|
||||
"""Back-compat alias for the phase-35 name (existing importers).
|
||||
|
||||
Returns the repo URLs of the effective ``kind='git'`` rows only, with
|
||||
the same origin label — the env fallback is git-only, so its
|
||||
behavior is unchanged. Local rows carry no URL (they are walked
|
||||
directly); new code should call :func:`effective_sources` and branch
|
||||
on ``row.kind``.
|
||||
"""
|
||||
rows, origin = effective_sources(db)
|
||||
return [row.url for row in rows if row.kind == "git"], origin
|
||||
|
||||
+51
-12
@@ -181,28 +181,47 @@ class SteeringNoteList(BaseModel):
|
||||
|
||||
|
||||
class GitSourceIn(BaseModel):
|
||||
"""``POST /api/git-sources`` body: one repo URL (phase 35, task 02).
|
||||
"""``POST /api/git-sources`` body (phase 35, task 02; ``kind``, phase 38).
|
||||
|
||||
Mirrors :class:`SteeringNoteIn` — the URL is trimmed *before* the
|
||||
length constraints run, so a whitespace-only body is a 422 and a URL
|
||||
with surrounding spaces is stored clean. Shape validation
|
||||
(``https://``, ``ssh://``, ``git@``) happens in the API layer so the
|
||||
422 detail can be one generic string that never echoes the input.
|
||||
``kind`` selects the source kind and which field carries its location:
|
||||
|
||||
* ``"git"`` (default) — ``url`` is the repo URL. Mirrors the
|
||||
phase-35 contract: trimmed *before* the length constraints run, so a
|
||||
whitespace-only body is a 422 and a URL with surrounding spaces is
|
||||
stored clean. Shape validation (``https://``, ``ssh://``, ``git@``)
|
||||
and the kind-field rules (url present, no path) happen in the API
|
||||
layer so the 422/409 details stay fixed strings that never echo the
|
||||
input (credential safety).
|
||||
* ``"local"`` — ``path`` is an existing directory on the server.
|
||||
Trimmed here; the API layer then ``expanduser()``s it and requires an
|
||||
absolute existing directory (else 422 naming the path — the path is
|
||||
not a secret, unlike a git URL) and no ``url``.
|
||||
"""
|
||||
|
||||
url: str = Field(min_length=1, max_length=500)
|
||||
kind: Literal["git", "local"] = "git"
|
||||
url: str | None = Field(default=None, min_length=1, max_length=500)
|
||||
path: str | None = Field(default=None, min_length=1, max_length=2000)
|
||||
|
||||
@field_validator("url", mode="before")
|
||||
@classmethod
|
||||
def _trim_url(cls, v: object) -> object:
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@field_validator("path", mode="before")
|
||||
@classmethod
|
||||
def _trim_path(cls, v: object) -> object:
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
|
||||
class GitSourceOut(BaseModel):
|
||||
"""One git source as returned by the API (phase 35, task 02).
|
||||
"""One created git source as returned by ``POST`` (phase 35, task 02).
|
||||
|
||||
``id`` / ``added_at`` are nullable: env-fallback rows (table empty →
|
||||
the list comes from ``BOR_GIT_SOURCES``) carry neither, only a URL.
|
||||
``id`` / ``added_at`` are non-null for a stored row. ``url`` is the
|
||||
row's location column: the repo URL for ``kind=git`` rows and, for
|
||||
``kind=local`` rows, the stored (expanded) directory path — the
|
||||
phase-35 response shape is unchanged by phase 38, so a local 201
|
||||
reports its path in ``url`` and the full row (``kind`` + ``path``)
|
||||
via ``GET``.
|
||||
"""
|
||||
|
||||
id: uuid.UUID | None
|
||||
@@ -210,14 +229,34 @@ class GitSourceOut(BaseModel):
|
||||
added_at: datetime | None
|
||||
|
||||
|
||||
class GitSourceRow(BaseModel):
|
||||
"""One row of ``GET /api/git-sources`` (phase 35; ``kind``/``path``,
|
||||
phase 38, task 02).
|
||||
|
||||
``kind`` discriminates the row: git rows (and the git-only
|
||||
``BOR_GIT_SOURCES`` env-fallback rows) carry ``url`` and
|
||||
``path: null``; local rows carry ``path`` (the absolute directory,
|
||||
expanded) and the same string in ``url`` (the table's NOT-NULL
|
||||
location column). ``id`` / ``added_at`` are nullable: env-fallback
|
||||
rows (table empty) carry neither.
|
||||
"""
|
||||
|
||||
id: uuid.UUID | None
|
||||
kind: Literal["git", "local"]
|
||||
url: str
|
||||
path: str | None
|
||||
added_at: datetime | None
|
||||
|
||||
|
||||
class GitSourceList(BaseModel):
|
||||
"""``GET /api/git-sources`` response (phase 35, task 02).
|
||||
|
||||
``from_env`` is True only when the ``git_sources`` table is empty and
|
||||
the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback);
|
||||
the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback
|
||||
— env rows are git-only and report ``kind: "git"``, ``path: null``);
|
||||
once the table has rows the env var is ignored and ``from_env`` is
|
||||
False — the UI is the source of truth.
|
||||
"""
|
||||
|
||||
sources: list[GitSourceOut]
|
||||
sources: list[GitSourceRow]
|
||||
from_env: bool
|
||||
|
||||
Reference in New Issue
Block a user