feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
+55
-28
@@ -7,20 +7,24 @@ Examples::
|
||||
uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files
|
||||
uv run python -m scripts.import_docs --limit 5 # debug: first 5 files only
|
||||
|
||||
Source resolution (phase 28, extended in phase 35), in precedence order:
|
||||
Source resolution (phase 28, extended in phases 35 and 38), in
|
||||
precedence order:
|
||||
|
||||
1. ``--source PATH`` — explicit manual directories (repeatable) always win;
|
||||
git sources are ignored when this flag is used.
|
||||
2. The effective git sources — the admin-managed ``git_sources`` table
|
||||
rows, else the ``BOR_GIT_SOURCES`` (comma-separated) fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`, the same shared
|
||||
resolver the in-app Sync button uses) — each repo is cloned (first
|
||||
run, shallow ``--depth 1``) or fast-forwarded (``git pull --ff-only``)
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` (default ``~/bor-sources``) and
|
||||
the resulting checkouts are imported. A failing clone/pull aborts the
|
||||
whole run *before* anything is imported.
|
||||
1. ``--source PATH`` — explicit manual directories (repeatable) always
|
||||
win; git and local sources are ignored when this flag is used.
|
||||
2. The effective sources — the admin-managed ``git_sources`` table rows
|
||||
(both kinds), else the ``BOR_GIT_SOURCES`` (comma-separated) git-only
|
||||
fallback (:func:`app.rag.git_sources.effective_sources`, the same
|
||||
shared resolver the in-app Sync button uses). Git rows are cloned
|
||||
(first run, shallow ``--depth 1``) or fast-forwarded
|
||||
(``git pull --ff-only``) into ``BOR_SOURCES_DIR/<repo-name>/``
|
||||
(default ``~/bor-sources``); local rows are the existing directories
|
||||
themselves, walked directly. A failing clone/pull — or a local
|
||||
directory that is missing at run time — aborts the whole run
|
||||
*before* anything is imported.
|
||||
3. Fallback — the legacy ``DEFAULT_SOURCES`` (``~/Homelab`` +
|
||||
``~/Deployments``), kept for backwards compatibility.
|
||||
``~/Deployments``), kept for backwards compatibility (reached only
|
||||
while both the table and ``BOR_GIT_SOURCES`` are empty).
|
||||
|
||||
Imported formats (PLAN anchor A9, revised): ``md, markdown, txt, yaml,
|
||||
yml, json, py`` (case-insensitive; narrow with ``BOR_IMPORT_EXTENSIONS``).
|
||||
@@ -56,7 +60,7 @@ from app.core.debugging import configure_debugging
|
||||
from app.core.logging import configure_logging
|
||||
from app.db import SessionLocal
|
||||
from app.models import KbOverview
|
||||
from app.rag.git_sources import effective_git_sources
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.overview import regenerate_overview
|
||||
@@ -78,8 +82,9 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
type=Path,
|
||||
metavar="PATH",
|
||||
help=(
|
||||
"directory to import (repeatable; always wins over BOR_GIT_SOURCES; "
|
||||
"default when neither is given: ~/Homelab ~/Deployments)"
|
||||
"directory to import (repeatable; always wins over the git_sources "
|
||||
"DB rows and BOR_GIT_SOURCES; default when neither is given: "
|
||||
"~/Homelab ~/Deployments)"
|
||||
),
|
||||
)
|
||||
p.add_argument(
|
||||
@@ -115,31 +120,53 @@ def repo_name(url: str) -> str:
|
||||
|
||||
|
||||
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
|
||||
"""Resolve the directories to import (phase 28, extended in phase 35).
|
||||
"""Resolve the directories to import (phase 28, extended in phases
|
||||
35 and 38).
|
||||
|
||||
Precedence: ``--source`` (explicit manual paths — always wins) >
|
||||
the effective git sources — the ``git_sources`` DB rows, else the
|
||||
``BOR_GIT_SOURCES`` fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`; the import needs
|
||||
the database anyway, so resolution opens a short session and there
|
||||
is no DB-down branch) — each URL cloned/pulled via
|
||||
the effective sources — the ``git_sources`` DB rows (git + local),
|
||||
else the ``BOR_GIT_SOURCES`` git-only fallback
|
||||
(:func:`app.rag.git_sources.effective_sources`; the import needs the
|
||||
database anyway, so resolution opens a short session and there is
|
||||
no DB-down branch) — git rows cloned/pulled via
|
||||
:func:`scripts.git_sync.clone_or_pull` into
|
||||
``BOR_SOURCES_DIR/<repo-name>/`` > the legacy ``DEFAULT_SOURCES``.
|
||||
``BOR_SOURCES_DIR/<repo-name>/``, local rows walked directly (the
|
||||
stored directory, re-verified ``.is_dir()`` at run time) > the
|
||||
legacy ``DEFAULT_SOURCES``.
|
||||
|
||||
A :class:`GitSyncError` from a failing clone/pull propagates to
|
||||
A :class:`GitSyncError` from a failing clone/pull — or a missing
|
||||
local directory (``local source missing: <path>``) — propagates to
|
||||
:func:`main`, which aborts the run before importing anything.
|
||||
"""
|
||||
if cli_sources:
|
||||
return [path.expanduser() for path in cli_sources]
|
||||
db = SessionLocal()
|
||||
try:
|
||||
git_urls, origin = effective_git_sources(db)
|
||||
rows, origin = effective_sources(db)
|
||||
finally:
|
||||
db.close()
|
||||
if git_urls:
|
||||
logger.info("git sources: %d repo(s) origin=%s", len(git_urls), origin)
|
||||
if rows:
|
||||
git_count = sum(1 for row in rows if row.kind == "git")
|
||||
logger.info(
|
||||
"sources: %d repo(s) git=%d local=%d origin=%s",
|
||||
len(rows), git_count, len(rows) - git_count, origin,
|
||||
)
|
||||
sources_root = Path(settings.sources_dir).expanduser()
|
||||
return [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
|
||||
sources: list[Path] = []
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
# ``or`` keeps the type checker honest); a missing
|
||||
# directory aborts before importing, the same pre-import
|
||||
# fail-loud as a failing git clone.
|
||||
path = Path(row.path or row.url).expanduser()
|
||||
if not path.is_dir():
|
||||
raise GitSyncError(f"local source missing: {path}")
|
||||
sources.append(path)
|
||||
return sources
|
||||
return [path.expanduser() for path in DEFAULT_SOURCES]
|
||||
|
||||
|
||||
@@ -167,7 +194,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
try:
|
||||
sources = _resolve_sources(args.source, settings)
|
||||
except GitSyncError as e:
|
||||
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
|
||||
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
logger.info(
|
||||
|
||||
Reference in New Issue
Block a user