feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges

An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.

Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.

API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).

Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.

Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.

Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.

Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
2026-08-27 01:04:16 -04:00
parent 15c1272828
commit 94d7228510
22 changed files with 2190 additions and 323 deletions
+187 -17
View File
@@ -1,20 +1,31 @@
"""Integration: the admin sources-sync API (phase 32, task 01; phase 35,
task 03 re-points the URL resolution at the shared resolver).
task 03 re-points the URL resolution at the shared resolver; phase 38,
task 03 adds the local kind).
Covers the in-process sync runner end to end over HTTP: anonymous 403s
on both endpoints; admin idle → 202 → ``success`` with the full
ImportSummary detail; 409 on a double trigger while a run is in flight;
``GitSyncError`` → ``failed`` with the failing repo named and **zero**
import attempts; empty on *both* origins (``git_sources`` table +
``BOR_GIT_SOURCES``) → ``failed`` loudly; an embedding failure →
``failed`` with any credentials masked; the import always runs with
``prune=True``; and the phase-31 overview trigger is change-gated (no
``lite`` call on an unchanged KB).
import attempts; empty on *both* origins (no git rows, no local rows,
no env URLs) → ``failed`` loudly (``no sources configured
(git or local)``); an embedding failure → ``failed`` with any
credentials masked; the import always runs with ``prune=True``; and the
phase-31 overview trigger is change-gated (no ``lite`` call on an
unchanged KB).
Phase 35: the runner resolves the repos through
:func:`app.rag.git_sources.effective_git_sources` — the **real**
resolver against the **real** ``git_sources`` table (truncated around
every test), so DB-over-env and the env fallback go through the actual
Phase 38 (local kind): local-only, git-only, and mixed syncs over a
**host temp local dir** (the app server runs on the same host) — the
mixed run goes through the **real** ``import_sources`` (deterministic
in-process ``FakeEmbedder``, no network), so the local file verifiably
lands in the KB via ``GET /api/docs`` and union pruning holds (a file
deleted out of the local dir is pruned on the next sync while the git
doc survives); a local directory missing at sync time → ``failed`` with
``local source missing: <path>`` and **zero** import attempts.
Phase 35: the runner resolves the sources through
:func:`app.rag.git_sources.effective_sources` — the **real** resolver
against the **real** ``git_sources`` table (truncated around every
test), so DB-over-env and the env fallback go through the actual
indirection; the env list is driven by a fresh ``Settings`` on the
resolver's module (the dev ``.env`` never leaks in).
@@ -34,7 +45,7 @@ import asyncio
import logging
import time
from collections.abc import Iterator
from datetime import datetime
from datetime import UTC, datetime
from pathlib import Path
import pytest
@@ -51,6 +62,7 @@ from app.rag.importer import ImportSummary
from app.rag.llm import EmbeddingError, LLMClient
from scripts.git_sync import GitSyncError
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
@pytest.fixture(autouse=True)
@@ -110,6 +122,30 @@ def _seed(db: Session, url: str) -> None:
db.commit()
def _seed_local(db: Session, path: Path) -> None:
"""A ``kind=local`` row as the phase-38 API stores it: the expanded
absolute path in both ``path`` and the NOT-NULL ``url`` column."""
db.add(GitSource(url=str(path), kind="local", path=str(path)))
db.commit()
@pytest.fixture()
def clean_documents(db: Session) -> Iterator[None]:
"""The real-import tests write ``documents``/``chunks`` (the canonical
KB state) — global, truncated around every such test."""
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
"""The pipeline's ``LLMClient`` becomes the deterministic in-process
``FakeEmbedder`` (real import, no network)."""
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder())
def _login(client: TestClient) -> None:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
@@ -372,11 +408,12 @@ def test_git_failure_marks_failed_and_skips_import(
_poll(sync_client, "failed")
def test_no_git_sources_configured_fails_loudly(
def test_no_sources_configured_fails_loudly(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Both origins empty (the truncate fixture + a blank env) → the
fail-loud error names *both* (phase 35)."""
"""Both origins empty — no git rows, no local rows, no env URLs
(the truncate fixture + a blank env) → the fail-loud error
(phase 38: git-only message retired)."""
# Whitespace-only is just as unconfigured as empty.
_stub_env(monkeypatch, " , ")
monkeypatch.setattr(
@@ -393,9 +430,7 @@ def test_no_git_sources_configured_fails_loudly(
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert body["error"] == (
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
)
assert body["error"] == "no sources configured (git or local)"
assert clone_calls == [] # git is never touched
assert fake_import.sources == []
@@ -464,6 +499,141 @@ def test_env_fallback_when_table_empty(
assert any("sync: started repos=1 origin=env" in r.getMessage() for r in caplog.records)
# --- phase 38: the local kind (real import, host temp dirs) ---------------
def test_local_only_sync_imports_dir(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Local-only config: the host temp dir (one fixture ``.md``) is
walked directly — no clone at all — and the file lands in the KB
(``GET /api/docs`` as admin)."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "notes.md").write_text("# Local Notes\nthe local fixture\n", encoding="utf-8")
_seed_local(db, local_dir)
_stub_env(monkeypatch) # env must not matter once the table has a row
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert clone_calls == [] # nothing to clone — local is walked directly
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
# The local file is in the KB, sourced by the directory's basename.
docs = sync_client.get("/api/docs").json()["documents"]
assert [(d["source"], d["path"]) for d in docs] == [("LocalDocs", "notes.md")]
def test_mixed_git_local_sync_imports_both_and_prunes_union(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
clean_documents: None,
) -> None:
"""Mixed config: the git row is cloned, the local dir walked, and
both are imported in one run over the single combined list. The
started log carries the kind counts; a file deleted out of the
local dir is pruned on the next sync (union prune) while the git
doc survives."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "a.md").write_text("# A\nfirst local file\n", encoding="utf-8")
(local_dir / "b.md").write_text("# B\nsecond local file\n", encoding="utf-8")
_seed_local(db, local_dir)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_real_llm(monkeypatch)
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# Git cloned into BOR_SOURCES_DIR, local dir walked in row order.
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")]
assert body["detail"]["added"] == 3
assert any(
"sync: started repos=2 origin=db git=1 local=1" in r.getMessage() for r in caplog.records
)
docs = {(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]}
assert docs == {("repo", "notes.md"), ("LocalDocs", "a.md"), ("LocalDocs", "b.md")}
# Union prune: delete one local file → the next sync prunes exactly
# it; the git doc (and the surviving local file) stay.
(local_dir / "b.md").unlink()
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["detail"]["pruned"] == 1
assert body["detail"]["added"] == 0
docs = {(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]}
assert docs == {("repo", "notes.md"), ("LocalDocs", "a.md")}
def test_missing_local_dir_fails_loudly_and_imports_nothing(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""A local row whose directory is gone at sync time (moved/deleted
since add-time) → ``failed`` naming the path, **zero** import
attempts — the git row before it in row order was still cloned
(per-row walk; a clone is not an import)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
missing = tmp_path / "Gone"
db.add(
GitSource(url=repo_url, kind="git", added_at=datetime(2026, 1, 1, tzinfo=UTC))
)
db.add(
GitSource(url=str(missing), kind="local", path=str(missing),
added_at=datetime(2026, 1, 2, tzinfo=UTC))
)
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
fake_import = FakeImportSources(ImportSummary())
monkeypatch.setattr(sync_api, "import_sources", fake_import)
fake_overview = FakeOverview(ok=True)
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert f"local source missing: {missing}" in body["error"] # the path is named
assert body["detail"] == {}
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")] # git row walked first
assert fake_import.sources == [] # no partial import
assert fake_overview.llms == []
def test_import_error_is_reported_with_credentials_masked(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None: