feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
+18
-7
@@ -13,8 +13,10 @@ Data model — see ``.agent/PLAN.md`` §Data Model:
|
||||
* ``kb_overview`` — single-row lite-generated outline of the KB's basic
|
||||
categories, injected as the ``<knowledge_base>``
|
||||
section of every chat turn (phase 31).
|
||||
* ``git_sources`` — admin-managed git source URLs the Sync button and
|
||||
import_docs clone/pull (phase 35).
|
||||
* ``git_sources`` — admin-managed source registry (git URLs + local
|
||||
directories) the Sync button and import_docs
|
||||
import (phase 35; ``kind`` discriminator added in
|
||||
phase 38).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -138,16 +140,25 @@ class KbOverview(Base):
|
||||
|
||||
|
||||
class GitSource(Base):
|
||||
"""One admin-managed git source (phase 35).
|
||||
"""One admin-managed source (phase 35; kind discriminator, phase 38).
|
||||
|
||||
The UI-maintained list of repo URLs the Sync button (phase 32) and
|
||||
import_docs (phase 28) clone/pull. DB rows win over the
|
||||
BOR_GIT_SOURCES env var, which is a fallback while this table is
|
||||
empty (see app.rag.git_sources.effective_git_sources).
|
||||
The UI-maintained list the Sync button (phase 32) and import_docs
|
||||
(phase 28) import from. ``kind`` discriminates: ``git`` rows carry a
|
||||
repo ``url`` (cloned/pulled), ``local`` rows carry an existing
|
||||
directory ``path`` (walked directly). DB rows win over the
|
||||
BOR_GIT_SOURCES env var (git-only fallback), which is a fallback
|
||||
while this table is empty (see
|
||||
app.rag.git_sources.effective_git_sources).
|
||||
"""
|
||||
|
||||
__tablename__ = "git_sources"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
|
||||
#: Source-kind discriminator (phase 38): "git" (default) or "local"
|
||||
#: — enforced by the ``ck_git_sources_kind`` CHECK constraint.
|
||||
kind: Mapped[str] = mapped_column(Text, default="git", server_default="'git'")
|
||||
#: Absolute directory of a ``local`` source; NULL for git rows.
|
||||
#: Unique — Postgres treats NULLs as distinct under a unique index.
|
||||
path: Mapped[str | None] = mapped_column(Text, unique=True)
|
||||
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
Reference in New Issue
Block a user