feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
+50
-26
@@ -12,20 +12,27 @@ plus a module-level :class:`SyncStatus` that the UI polls every 2 s
|
||||
409; the status object is authoritative, so the UI can never sit on a
|
||||
stale button state (§7.4 adaptation, phase locked decisions).
|
||||
|
||||
Pipeline (the canonical "mirror the repos" action — phase locked
|
||||
Pipeline (the canonical "mirror the sources" action — phase locked
|
||||
decisions):
|
||||
|
||||
1. resolve the effective git sources — the ``git_sources`` DB rows,
|
||||
else the ``BOR_GIT_SOURCES`` fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`, shared with the
|
||||
CLI) — empty on both origins fails loudly (``no git sources
|
||||
configured``) instead of silently importing the legacy local
|
||||
directories;
|
||||
2. :func:`scripts.git_sync.clone_or_pull` each repo into
|
||||
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
||||
re-implemented; a failing repo aborts before any import);
|
||||
3. ``import_sources(..., prune=True)`` — prune so files deleted
|
||||
upstream leave the index (the CLI's no-prune default is unchanged);
|
||||
1. resolve the effective sources — the ``git_sources`` DB rows (git
|
||||
**and** local, phase 38), else the ``BOR_GIT_SOURCES`` fallback
|
||||
(git-only)
|
||||
(:func:`app.rag.git_sources.effective_sources`, shared with the
|
||||
CLI) — empty on both origins (no git rows, no local rows, no env
|
||||
URLs) fails loudly (``no sources configured (git or local)``)
|
||||
instead of silently importing the legacy local directories;
|
||||
2. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull`
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
||||
re-implemented); ``kind=local`` → the stored directory, re-verified
|
||||
``.is_dir()`` **at sync time** (it may have moved/deleted since
|
||||
add-time) — a missing directory raises ``local source missing:
|
||||
<path>``; a failing clone or a missing local dir aborts before any
|
||||
import;
|
||||
3. ``import_sources(..., prune=True)`` over the single combined list
|
||||
(git checkouts + local dirs) — prune so files deleted upstream or
|
||||
out of a local dir leave the index (pruning covers the union; the
|
||||
CLI's no-prune default is unchanged);
|
||||
4. when the import changed the KB (added + updated > 0),
|
||||
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
||||
(phase 31 trigger, best-effort inside).
|
||||
@@ -48,7 +55,7 @@ from fastapi import APIRouter, Depends, HTTPException
|
||||
from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import SessionLocal
|
||||
from app.rag.git_sources import effective_git_sources
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.overview import regenerate_overview
|
||||
@@ -147,24 +154,41 @@ async def _run_sync() -> None:
|
||||
try:
|
||||
settings = get_settings()
|
||||
# The background task has no request session: open a short-lived
|
||||
# one around the shared phase-35 resolver (DB rows win, the
|
||||
# BOR_GIT_SOURCES list is a fallback while the table is empty).
|
||||
# one around the shared phase-35/38 resolver (DB rows of both
|
||||
# kinds win; the BOR_GIT_SOURCES git list is a fallback while
|
||||
# the table is empty).
|
||||
db = SessionLocal()
|
||||
try:
|
||||
git_urls, origin = effective_git_sources(db)
|
||||
rows, origin = effective_sources(db)
|
||||
finally:
|
||||
db.close()
|
||||
if not git_urls:
|
||||
# The button targets git sources only (manual --source dirs
|
||||
# have no repo to clone) — an empty config on *both* origins
|
||||
# fails loudly instead of silently importing the legacy
|
||||
# directories.
|
||||
raise GitSyncError(
|
||||
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
|
||||
)
|
||||
logger.info("sync: started repos=%d origin=%s", len(git_urls), origin)
|
||||
if not rows:
|
||||
# The button targets the admin-managed source registry
|
||||
# (manual --source dirs have no repo to clone) — an empty
|
||||
# config on *both* origins (no git rows, no local rows, no
|
||||
# env URLs) fails loudly instead of silently importing the
|
||||
# legacy directories.
|
||||
raise GitSyncError("no sources configured (git or local)")
|
||||
git_count = sum(1 for row in rows if row.kind == "git")
|
||||
logger.info(
|
||||
"sync: started repos=%d origin=%s git=%d local=%d",
|
||||
len(rows), origin, git_count, len(rows) - git_count,
|
||||
)
|
||||
sources_root = Path(settings.sources_dir).expanduser()
|
||||
sources = [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
|
||||
sources: list[Path] = []
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
# ``or`` keeps the type checker honest); re-verified at
|
||||
# sync time because the directory may have moved or been
|
||||
# deleted since add-time.
|
||||
path = Path(row.path or row.url).expanduser()
|
||||
if not path.is_dir():
|
||||
raise GitSyncError(f"local source missing: {path}")
|
||||
sources.append(path)
|
||||
llm = LLMClient()
|
||||
summary: ImportSummary = await import_sources(sources, llm, prune=True)
|
||||
overview = False
|
||||
|
||||
Reference in New Issue
Block a user