Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges

Browse Source 
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.

Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.

API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).

Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.

Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.

Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.

Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
ducoterra
2026-08-27 01:04:16 -04:00
parent 15c1272828
commit 94d7228510
22 changed files with 2190 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/integration/test_git_sources_api.py
+245 -17
View File
@@ -1,4 +1,5 @@
"""Integration: the admin git-sources CRUD API (phase 35, task 02).
"""Integration: the admin sources CRUD API (phase 35, task 02; local
kind, phase 38, task 02).
Real Postgres (``podman compose up -d db``); the ``BOR_GIT_SOURCES``
fallback is exercised deterministically by monkeypatching the router's
@@ -7,17 +8,25 @@ fallback is exercised deterministically by monkeypatching the router's
Contract under test:
* anonymous → 403 ``{"detail": "admin only"}`` on GET, POST, and DELETE
(phase 16 pattern, same as ``/api/sync``);
* GET — empty table + env set → the env rows with ``from_env: true`` and
null ``id``/``added_at``; empty table + empty env → ``sources: []``
with ``from_env: true``; any DB rows → ``from_env: false`` and the env
var is ignored (the phase's locked decision); DB rows ordered by
``(added_at, id)``;
* POST — 201 stored trimmed; duplicate (even with different surrounding
whitespace) → 409 with a generic detail that never echoes the URL
(credential safety), including when only the DB unique index catches
it; bad shape / blank / >500 chars → 422, also input-free;
* anonymous → 403 ``{"detail": "admin only"}`` on GET, POST (git and
local), and DELETE (phase 16 pattern, same as ``/api/sync``);
* GET — rows carry ``kind`` + ``path`` (phase 38); empty table + env
set → the git-only env rows (``kind: "git"``, ``path: null``) with
``from_env: true`` and null ``id``/``added_at``; empty table + empty
env → ``sources: []`` with ``from_env: true``; any DB rows →
``from_env: false`` and the env var is ignored (the phase's locked
decision); DB rows ordered by ``(added_at, id)``;
* POST ``kind=git`` (default) — 201 stored trimmed; duplicate (even with
different surrounding whitespace) → 409 with a generic detail that
never echoes the URL (credential safety), including when only the DB
unique index catches it; bad shape / blank / >500 chars → 422, also
input-free (the phase-35 contract, unchanged);
* POST ``kind=local`` (phase 38) — existing directory → 201, stored row
carries ``kind=local`` + the path expanded (``~`` resolved) and
trimmed; relative / missing / not-a-directory path → 422 naming the
path (not a secret); duplicate path → 409 naming the path (unique
index as backstop); wrong field combinations (git without url, local
without path, both kinds' fields) → 422;
* DELETE — 204 and gone; an emptied table falls back to the env list
again; unknown id → 404.
@@ -28,6 +37,7 @@ from __future__ import annotations
import uuid
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
import pytest
@@ -66,6 +76,10 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
r = client.post("/api/git-sources", json={"url": "https://anon.example.com/x.git"})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# The phase-38 local kind is gated the same way.
r = client.post("/api/git-sources", json={"kind": "local", "path": "/tmp"})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
@@ -89,10 +103,23 @@ def test_get_empty_table_with_env_returns_env_rows(
assert r.status_code == 200
body = r.json()
assert body["from_env"] is True
# Whitespace-trimmed, empty entries dropped, order preserved; null ids.
# Whitespace-trimmed, empty entries dropped, order preserved; null
# ids; the env fallback is git-only (phase 38: kind + path fields).
assert body["sources"] == [
{"id": None, "url": "https://a.example.com/one.git", "added_at": None},
{"id": None, "url": "git@b.example.com:two.git", "added_at": None},
{
"id": None,
"kind": "git",
"url": "https://a.example.com/one.git",
"path": None,
"added_at": None,
},
{
"id": None,
"kind": "git",
"url": "git@b.example.com:two.git",
"path": None,
"added_at": None,
},
]
@@ -248,6 +275,200 @@ def test_post_rejects_blank_and_oversized_urls(admin_client: TestClient, db: Ses
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 1
# --- POST: local kind (phase 38, task 02) ----------------------------------
def test_post_local_creates_stored_row_with_expanded_path(
admin_client: TestClient,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``kind=local`` + an existing directory → 201; the stored row
carries ``kind=local`` and the path expanded (``~`` resolved via the
server's ``HOME``, whitespace trimmed)."""
monkeypatch.setenv("HOME", str(tmp_path / "home"))
real_dir = tmp_path / "home" / "notes"
real_dir.mkdir(parents=True)
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": " ~/notes\t"})
assert r.status_code == 201, r.text
body = r.json()
# The phase-35 response shape is unchanged — the local row reports
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET.
assert set(body) == {"id", "url", "added_at"}
uuid.UUID(body["id"])
assert body["url"] == str(real_dir)
assert body["added_at"] is not None
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["kind"] == "local"
assert row["path"] == str(real_dir)
assert row["url"] == str(real_dir)
assert row["id"] is not None
assert row["added_at"] is not None
def test_post_local_stores_trimmed_normalized_path(
admin_client: TestClient, tmp_path: Path
) -> None:
"""Absolute path with surrounding whitespace + trailing slash →
stored clean (trimmed, normalized)."""
real_dir = tmp_path / "plain"
real_dir.mkdir()
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": f" {real_dir}/ "})
assert r.status_code == 201, r.text
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["path"] == str(real_dir)
def test_post_local_relative_path_returns_422_naming_path(
admin_client: TestClient, db: Session
) -> None:
"""A relative path fails loud at add-time — 422 naming the path
(relative or not, it is never stored)."""
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": "relative/dir"})
assert r.status_code == 422
assert r.json()["detail"] == "local source path is not a directory: relative/dir"
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_post_local_missing_path_returns_422_naming_path(
admin_client: TestClient, db: Session
) -> None:
"""A missing (absolute) path is a user error → 422 naming the path so
the owner sees exactly which directory failed."""
missing = f"/nonexistent/bor-test-{uuid.uuid4()}"
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": missing})
assert r.status_code == 422
assert r.json()["detail"] == f"local source path is not a directory: {missing}"
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_post_local_file_not_dir_returns_422(
admin_client: TestClient, tmp_path: Path
) -> None:
"""An existing *file* is not a directory → 422 naming the path."""
a_file = tmp_path / "a-file.md"
a_file.write_text("not a directory")
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": str(a_file)})
assert r.status_code == 422
assert r.json()["detail"] == f"local source path is not a directory: {a_file}"
def test_post_local_duplicate_path_returns_409_naming_path(
admin_client: TestClient, tmp_path: Path
) -> None:
"""Duplicate path (even with different surrounding whitespace) → 409
naming the path (a path is not a secret, unlike a git URL)."""
real_dir = tmp_path / "dups"
real_dir.mkdir()
assert admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": f" {real_dir}\t"})
assert r.status_code == 409
detail = r.json()["detail"]
assert detail == f"a local source with this path already exists: {real_dir}"
# Exactly one row stored.
assert len(admin_client.get("/api/git-sources").json()["sources"]) == 1
def test_post_local_concurrent_insert_backstop_still_409(
admin_client: TestClient, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""If the duplicate pre-check misses (a concurrent insert lands
between the check and the commit), the DB unique index on ``path``
still yields the 409 naming the path — never a 500."""
real_dir = tmp_path / "backstop"
real_dir.mkdir()
assert (
admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
.status_code
== 201
)
real_select = git_sources_api.select
def blind_select(*args: Any, **kwargs: Any) -> Any:
if args and args[0] is GitSource: # the duplicate pre-check
# …now never matches — only the unique index can catch it.
return real_select(GitSource).where(GitSource.url == "zz-never-matches")
return real_select(*args, **kwargs)
monkeypatch.setattr(git_sources_api, "select", blind_select)
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
assert r.status_code == 409
assert r.json()["detail"] == f"a local source with this path already exists: {real_dir}"
def test_post_wrong_field_combinations_return_422(
admin_client: TestClient, db: Session, tmp_path: Path
) -> None:
"""git without url, local without path, and both kinds' fields are
422 with fixed details — nothing is stored."""
real_dir = tmp_path / "combo"
real_dir.mkdir()
assert admin_client.post("/api/git-sources", json={"kind": "git"}).status_code == 422
r = admin_client.post(
"/api/git-sources",
json={"kind": "git", "url": "https://example.com/both.git", "path": str(real_dir)},
)
assert r.status_code == 422
assert r.json()["detail"] == "a git source takes a url, not a path"
assert admin_client.post("/api/git-sources", json={"kind": "local"}).status_code == 422
# Whitespace-only path trims to empty → 422 as well (the schema's
# min-length guard).
assert (
admin_client.post("/api/git-sources", json={"kind": "local", "path": " "}).status_code
== 422
)
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "url": "https://example.com/both.git", "path": str(real_dir)},
)
assert r.status_code == 422
assert r.json()["detail"] == "a local source takes a path, not a url"
# Unknown kind and an oversized path are 422 too.
assert (
admin_client.post(
"/api/git-sources", json={"kind": "svn", "url": "https://example.com/x.git"}
).status_code
== 422
)
assert (
admin_client.post(
"/api/git-sources", json={"kind": "local", "path": "/" + "x" * 2000}
).status_code
== 422
)
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_get_mixed_kinds_in_added_order(
admin_client: TestClient, db: Session, tmp_path: Path
) -> None:
"""GET mixes git + local rows in ``(added_at, id)`` order; git rows
report ``path: null``, local rows their stored path."""
git_url = "https://example.com/mixed.git"
db.add(GitSource(url=git_url, kind="git", added_at=datetime.now(UTC) - timedelta(hours=1)))
db.commit()
real_dir = tmp_path / "mixed"
real_dir.mkdir()
assert admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is False
assert [s["url"] for s in body["sources"]] == [git_url, str(real_dir)]
git_row, local_row = body["sources"]
assert git_row["kind"] == "git"
assert git_row["path"] is None
assert git_row["id"] is not None
assert local_row["kind"] == "local"
assert local_row["path"] == str(real_dir)
assert local_row["id"] is not None
# --- DB rows win over env ---------------------------------------------------
@@ -284,11 +505,18 @@ def test_delete_removes_row_and_falls_back_to_env(
assert admin_client.delete(f"/api/git-sources/{created.json()['id']}").status_code == 204
# The table is empty again → the env fallback is live once more.
# The table is empty again → the env fallback is live once more
# (git-only rows, phase 38).
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is True
assert body["sources"] == [
{"id": None, "url": "https://env.example.com/env.git", "added_at": None}
{
"id": None,
"kind": "git",
"url": "https://env.example.com/env.git",
"path": None,
"added_at": None,
}
]