Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges

Browse Source 
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.

Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.

API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).

Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.

Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.

Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.

Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
ducoterra
2026-08-27 01:04:16 -04:00
parent 15c1272828
commit 94d7228510
22 changed files with 2190 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/git_sources.py
+45 -20
View File
@@ -1,19 +1,26 @@
"""The shared git-source resolver (phase 35, task 03).
"""The shared source resolver (phase 35, task 03; local kind, phase 38).
One function, two callers: the in-app Sync pipeline
(``app/api/sync.py::_run_sync``) and the CLI
(``scripts/import_docs.py::_resolve_sources``) — both resolve the repo
URLs to clone/pull through :func:`effective_git_sources`, so the
admin-managed ``git_sources`` table is what actually gets cloned and
indexed from either entry point (the API's ``GET /api/git-sources``
fallback list is the only other place that reads the env list — task 02).
(``scripts/import_docs.py::_resolve_sources``) — both resolve the
``git_sources`` rows to import through :func:`effective_sources`, so
the admin-managed ``git_sources`` table is what actually gets cloned
(git rows) or walked directly (local rows) from either entry point
(the API's ``GET /api/git-sources`` fallback list is the only other
place that reads the env list — task 02).
Precedence (the phase's locked decision — the env var is demoted, not
removed): ``git_sources`` DB rows win, in ``(added_at, id)`` order;
``BOR_GIT_SOURCES`` is a fallback only while the table is empty; both
empty → ``([], "env")`` and each caller keeps its existing fail-loud
behavior (sync: ``GitSyncError``; CLI: the legacy ``DEFAULT_SOURCES``
fallback).
removed): ``git_sources`` DB rows of **both kinds** win, in
``(added_at, id)`` order; ``BOR_GIT_SOURCES`` is a fallback only while
the table is empty (git URLs surface as synthetic ``kind='git'`` rows —
the env fallback is git-only); both empty → ``([], "env")`` and each
caller keeps its existing fail-loud behavior (sync: ``GitSyncError``;
CLI: the legacy ``DEFAULT_SOURCES`` fallback).
:func:`effective_git_sources` is kept as the phase-35 back-compat alias
(repo URLs of the effective git rows) so existing importers of the old
name keep working; new code calls :func:`effective_sources` and
branches on ``row.kind``.
"""
from __future__ import annotations
@@ -26,19 +33,37 @@ from app.config import get_settings
from app.models import GitSource
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
"""``(urls, origin)`` — the repo URLs to clone, and where they came from.
def effective_sources(db: Session) -> tuple[list[GitSource], Literal["db", "env"]]:
"""``(rows, origin)`` — the effective source rows (both kinds) and
where they came from.
``"db"``: the ``git_sources`` rows in ``(added_at, id)`` order (oldest
first, id tie-break for same-timestamp inserts) — once the table has
any row, ``BOR_GIT_SOURCES`` is ignored entirely.
``"db"``: the ``git_sources`` rows — ``kind='git'`` (repo URL to
clone/pull) and ``kind='local'`` (existing directory to walk
directly) — in ``(added_at, id)`` order (oldest first, id tie-break
for same-timestamp inserts). Once the table has any row,
``BOR_GIT_SOURCES`` is ignored entirely.
``"env"``: the ``BOR_GIT_SOURCES`` list — ``Settings.git_source_list``,
the phase-28 CSV parse, reused not re-implemented — used only while
the table is empty; both empty → ``([], "env")``.
the phase-28 CSV parse, reused not re-implemented — surfaced as
synthetic ``kind='git'`` rows (``path`` NULL; the env fallback is
git-only), used only while the table is empty; both empty →
``([], "env")``.
"""
rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
).all()
if rows:
return [row.url for row in rows], "db"
return list(get_settings().git_source_list), "env"
return list(rows), "db"
return [GitSource(url=url, kind="git") for url in get_settings().git_source_list], "env"
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
"""Back-compat alias for the phase-35 name (existing importers).
Returns the repo URLs of the effective ``kind='git'`` rows only, with
the same origin label — the env fallback is git-only, so its
behavior is unchanged. Local rows carry no URL (they are walked
directly); new code should call :func:`effective_sources` and branch
on ``row.kind``.
"""
rows, origin = effective_sources(db)
return [row.url for row in rows if row.kind == "git"], origin