feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
@@ -1,28 +1,36 @@
|
||||
"""Integration test: ``import_docs`` git-source resolution (phase 28, task 03).
|
||||
"""Integration test: ``import_docs`` source resolution (phase 28, task
|
||||
03; phase 35 re-points at the shared resolver; phase 38 adds the local
|
||||
kind).
|
||||
|
||||
Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull`` (no
|
||||
real git, no network) and a recording fake ``import_sources`` (no real
|
||||
DB), covering:
|
||||
Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull``
|
||||
(no real git, no network) and a recording fake ``import_sources`` (no
|
||||
real DB), covering:
|
||||
|
||||
- Effective git sources set (phase 35: the shared resolver — stubbed
|
||||
here, keeping this file's no-real-DB style) → each URL is cloned/pulled
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` and exactly those dirs are
|
||||
imported.
|
||||
- DB rows win over ``BOR_GIT_SOURCES`` (the resolver's ``db`` origin —
|
||||
the env list is ignored).
|
||||
- ``--source`` still wins over git sources (no git at all, no resolver
|
||||
- Effective sources set (phase 35: the shared resolver — stubbed here,
|
||||
keeping this file's no-real-DB style) → each git URL is cloned/pulled
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/``; local rows are their existing
|
||||
directories, walked directly; exactly those dirs are imported.
|
||||
- DB rows (both kinds) win over ``BOR_GIT_SOURCES`` (the resolver's
|
||||
``db`` origin — the env list is ignored; the env fallback stays
|
||||
git-only).
|
||||
- ``--source`` still wins over the DB rows (no git at all, no resolver
|
||||
call).
|
||||
- No git sources + no ``--source`` → the legacy ``DEFAULT_SOURCES``.
|
||||
- No sources configured + no ``--source`` → the legacy
|
||||
``DEFAULT_SOURCES``.
|
||||
- A failing git sync → exit code 1, an error naming the failing repo on
|
||||
stderr, and **zero** import attempts.
|
||||
- A missing local directory → the same pre-import fail-loud: exit code
|
||||
1, ``local source missing: <path>`` on stderr, zero import attempts.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import GitSource
|
||||
from app.rag.importer import ImportSummary
|
||||
from scripts import import_docs
|
||||
from scripts.git_sync import GitSyncError
|
||||
@@ -33,6 +41,16 @@ def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Sett
|
||||
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _git_row(url: str) -> GitSource:
|
||||
return GitSource(url=url, kind="git")
|
||||
|
||||
|
||||
def _local_row(path: str) -> GitSource:
|
||||
"""A local row as the phase-38 API stores it: the expanded path in
|
||||
both ``path`` and the NOT-NULL ``url`` location column."""
|
||||
return GitSource(url=path, kind="local", path=path)
|
||||
|
||||
|
||||
class FakeImportSources:
|
||||
"""Records every ``import_sources`` call instead of touching a DB."""
|
||||
|
||||
@@ -97,11 +115,15 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
|
||||
calls, fake = _fake_clone_factory()
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
# Phase 35: resolution goes through the shared resolver (stubbed —
|
||||
# this file keeps its no-real-DB style); the URLs are the env list.
|
||||
# this file keeps its no-real-DB style); the rows are the env list,
|
||||
# surfaced as synthetic git rows.
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_git_sources",
|
||||
lambda db: (["https://host/a/homelab.git", "git@host:user/deploy.git"], "env"),
|
||||
"effective_sources",
|
||||
lambda db: (
|
||||
[_git_row("https://host/a/homelab.git"), _git_row("git@host:user/deploy.git")],
|
||||
"env",
|
||||
),
|
||||
)
|
||||
settings = _settings(
|
||||
git_sources="https://host/a/homelab.git, git@host:user/deploy.git ,",
|
||||
@@ -140,8 +162,8 @@ def test_resolve_sources_db_rows_win_over_env(
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_git_sources",
|
||||
lambda db: (["https://db.example/only.git"], "db"),
|
||||
"effective_sources",
|
||||
lambda db: ([_git_row("https://db.example/only.git")], "db"),
|
||||
)
|
||||
settings = _settings(
|
||||
git_sources="https://env.example/ignored.git",
|
||||
@@ -158,7 +180,7 @@ def test_resolve_sources_defaults_when_nothing_configured(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
|
||||
monkeypatch.setattr(import_docs, "effective_git_sources", lambda db: ([], "env"))
|
||||
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
|
||||
sources = import_docs._resolve_sources(None, _settings())
|
||||
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
|
||||
|
||||
@@ -178,8 +200,11 @@ def test_main_git_sources_clone_then_import(
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_git_sources",
|
||||
lambda db: (["https://host/a/homelab.git", "https://host/a/deploy.git"], "env"),
|
||||
"effective_sources",
|
||||
lambda db: (
|
||||
[_git_row("https://host/a/homelab.git"), _git_row("https://host/a/deploy.git")],
|
||||
"env",
|
||||
),
|
||||
)
|
||||
calls, fake = _fake_clone_factory()
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
@@ -227,6 +252,55 @@ def test_main_cli_source_still_imports_manual_dir(
|
||||
assert fake_import.calls[0]["prune"] is False
|
||||
|
||||
|
||||
def test_resolve_sources_mixed_git_and_local(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 38: DB rows of both kinds — the git row is cloned into
|
||||
``BOR_SOURCES_DIR``, the local row is its existing directory itself
|
||||
(no clone), in row order; the env list is ignored."""
|
||||
calls, fake = _fake_clone_factory()
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "a.md").write_text("# A\nlocal fixture\n", encoding="utf-8")
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: (
|
||||
[_git_row("https://db.example/only.git"), _local_row(str(local_dir))],
|
||||
"db",
|
||||
),
|
||||
)
|
||||
settings = _settings(
|
||||
git_sources="https://env.example/ignored.git",
|
||||
sources_dir=str(tmp_path / "bor"),
|
||||
)
|
||||
|
||||
sources = import_docs._resolve_sources(None, settings)
|
||||
|
||||
assert sources == [tmp_path / "bor" / "only", local_dir]
|
||||
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
||||
|
||||
|
||||
def test_resolve_sources_missing_local_dir_aborts(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 38: a local row whose directory is gone at run time →
|
||||
``GitSyncError`` naming the path, before any import (the same
|
||||
pre-import fail-loud as a failing git clone)."""
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", _fake_clone_factory()[1])
|
||||
missing = tmp_path / "Gone"
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(missing))], "db"),
|
||||
)
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
|
||||
with pytest.raises(GitSyncError, match=f"local source missing: {re.escape(str(missing))}"):
|
||||
import_docs._resolve_sources(None, settings)
|
||||
|
||||
|
||||
def test_main_git_failure_aborts_before_import(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
@@ -236,7 +310,9 @@ def test_main_git_failure_aborts_before_import(
|
||||
)
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
monkeypatch.setattr(
|
||||
import_docs, "effective_git_sources", lambda db: (["https://host/a/bad.git"], "env")
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_git_row("https://host/a/bad.git")], "env"),
|
||||
)
|
||||
|
||||
def failing_clone(url: str, dest: Path | str) -> Path:
|
||||
@@ -253,7 +329,34 @@ def test_main_git_failure_aborts_before_import(
|
||||
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "import_docs: git sync failed" in err
|
||||
assert "import_docs: source sync failed" in err
|
||||
assert "bad.git" in err # the failing repo is named
|
||||
assert fake_import.calls == [] # no partial import
|
||||
assert not (tmp_path / "bor").exists()
|
||||
|
||||
|
||||
def test_main_missing_local_dir_aborts_before_import(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""Phase 38: a DB local row whose directory is missing → exit code
|
||||
1, ``local source missing: <path>`` on stderr, zero import attempts
|
||||
(no ``--source`` given, so the DB row is what should have been
|
||||
imported)."""
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
missing = tmp_path / "Gone"
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(missing))], "db"),
|
||||
)
|
||||
fake_import = FakeImportSources()
|
||||
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||
|
||||
rc = import_docs.main([])
|
||||
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "import_docs: source sync failed" in err
|
||||
assert f"local source missing: {missing}" in err # the path is named
|
||||
assert fake_import.calls == [] # no partial import
|
||||
|
||||
Reference in New Issue
Block a user