feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
@@ -0,0 +1,261 @@
|
||||
"""Integration: migration 0007 (git_sources.kind + path) schema contract.
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the style of
|
||||
``test_migration_0004.py`` / ``test_migration_0006.py``
|
||||
(information_schema assertions on the state the migration must leave).
|
||||
The tests target revision ``0007`` explicitly so later migrations
|
||||
cannot break them:
|
||||
|
||||
* upgrade 0006 → 0007 → ``git_sources`` gains ``kind TEXT NOT NULL``
|
||||
(server default ``'git'``, check constraint ``ck_git_sources_kind``:
|
||||
``kind IN ('git', 'local')``) and ``path TEXT`` (nullable) with the
|
||||
unique index ``uq_git_sources_path``; a row inserted before the
|
||||
upgrade (the pre-0007 insert shape) keeps ``kind='git'`` /
|
||||
``path=NULL`` after it;
|
||||
* the check constraint rejects any kind other than ``git``/``local``;
|
||||
* the unique index rejects duplicate local paths but tolerates NULL
|
||||
paths (git rows);
|
||||
* downgrade to 0006 → both columns, the constraint, and the index are
|
||||
gone;
|
||||
* upgrade back to 0007 → they are back (round-trip).
|
||||
|
||||
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||
fails or the process is interrupted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.db import db_available
|
||||
|
||||
URL_BASE = "https://git.example.com/mig0007"
|
||||
PATH_BASE = "/tmp/brain-of-reese-mig0007"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run); teardown upgrades
|
||||
to head no matter what happened, so the dev DB is never left below
|
||||
head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||
"""(data_type, is_nullable, column_default) for one git_sources column."""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT data_type, is_nullable, column_default"
|
||||
" FROM information_schema.columns"
|
||||
" WHERE table_name = 'git_sources' AND column_name = :c"
|
||||
),
|
||||
{"c": column},
|
||||
).fetchone()
|
||||
return tuple(row) if row is not None else None
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _check_constraint_def(db: Session) -> str | None:
|
||||
"""Definition of ``ck_git_sources_kind``, or None if it does not exist.
|
||||
|
||||
Only call while ``git_sources`` exists (the ``::regclass`` cast errors
|
||||
otherwise).
|
||||
"""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT pg_get_constraintdef(oid) FROM pg_constraint"
|
||||
" WHERE conname = 'ck_git_sources_kind'"
|
||||
" AND conrelid = 'git_sources'::regclass"
|
||||
)
|
||||
).fetchone()
|
||||
return row[0] if row is not None else None
|
||||
|
||||
|
||||
def _unique_path_index(db: Session) -> int:
|
||||
"""1 iff ``uq_git_sources_path`` exists as a UNIQUE index."""
|
||||
count: Any = db.execute(
|
||||
text(
|
||||
"SELECT count(*) FROM pg_indexes"
|
||||
" WHERE tablename = 'git_sources' AND indexname = 'uq_git_sources_path'"
|
||||
" AND indexdef ILIKE 'CREATE UNIQUE%'"
|
||||
)
|
||||
).scalar()
|
||||
assert count is not None, "pg_indexes count must be an int"
|
||||
return int(count)
|
||||
|
||||
|
||||
def _insert(db: Session, url: str, kind: str | None = None, path: str | None = None) -> None:
|
||||
"""Insert one git_sources row; kind/path omitted → pre-0007 shape."""
|
||||
if kind is None and path is None:
|
||||
db.execute(
|
||||
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
|
||||
{"u": url},
|
||||
)
|
||||
else:
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO git_sources (id, url, kind, path)"
|
||||
" VALUES (gen_random_uuid(), :u, :k, :p)"
|
||||
),
|
||||
{"u": url, "k": kind, "p": path},
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _delete_by_url(db: Session, url: str) -> None:
|
||||
db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url})
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_upgrade_to_0007_adds_kind_and_path(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0006 → 0007: both columns exist with the locked types,
|
||||
nullability, and defaults, plus the CHECK constraint and the unique
|
||||
path index."""
|
||||
command.downgrade(alembic, "0006") # start from the pre-0007 state
|
||||
assert _version(db) == "0006"
|
||||
|
||||
command.upgrade(alembic, "0007")
|
||||
assert _version(db) == "0007", "alembic_version must be at 0007"
|
||||
|
||||
kind = _column(db, "kind")
|
||||
assert kind is not None, "git_sources.kind is missing"
|
||||
assert kind[0] == "text", "git_sources.kind must be TEXT"
|
||||
assert kind[1] == "NO", "git_sources.kind must be NOT NULL"
|
||||
assert kind[2] is not None and "'git'" in kind[2], (
|
||||
"git_sources.kind must have server default 'git'"
|
||||
)
|
||||
|
||||
path = _column(db, "path")
|
||||
assert path is not None, "git_sources.path is missing"
|
||||
assert path[0] == "text", "git_sources.path must be TEXT"
|
||||
assert path[1] == "YES", "git_sources.path must be NULLABLE"
|
||||
|
||||
constraint = _check_constraint_def(db)
|
||||
assert constraint is not None, "ck_git_sources_kind is missing"
|
||||
assert "git" in constraint and "local" in constraint, (
|
||||
f"ck_git_sources_kind must restrict kind to git|local, got: {constraint}"
|
||||
)
|
||||
|
||||
assert _unique_path_index(db) == 1, "uq_git_sources_path unique index is missing"
|
||||
|
||||
|
||||
def test_pre_0007_row_reads_as_git(db: Session, alembic: Config) -> None:
|
||||
"""A row inserted before the upgrade (url only — the pre-0007 insert
|
||||
shape) reads as ``kind='git'``, ``path=NULL`` after it."""
|
||||
command.downgrade(alembic, "0006")
|
||||
url = f"{URL_BASE}/pre-existing.git"
|
||||
_insert(db, url) # no kind/path columns exist at 0006
|
||||
try:
|
||||
command.upgrade(alembic, "0007")
|
||||
kind, path = db.execute(
|
||||
text("SELECT kind, path FROM git_sources WHERE url = :u"), {"u": url}
|
||||
).one()
|
||||
assert kind == "git", "a pre-0007 row must read as kind='git'"
|
||||
assert path is None, "a pre-0007 row must keep path=NULL"
|
||||
finally:
|
||||
_delete_by_url(db, url)
|
||||
|
||||
|
||||
def test_kind_defaults_to_git_for_new_inserts(db: Session, alembic: Config) -> None:
|
||||
"""An insert that omits kind (the API's pre-phase-38 shape) lands as
|
||||
``kind='git'`` via the server default."""
|
||||
command.upgrade(alembic, "head")
|
||||
url = f"{URL_BASE}/default-kind.git"
|
||||
_insert(db, url)
|
||||
try:
|
||||
kind, path = db.execute(
|
||||
text("SELECT kind, path FROM git_sources WHERE url = :u"), {"u": url}
|
||||
).one()
|
||||
assert kind == "git", "git_sources.kind must default to 'git'"
|
||||
assert path is None, "git_sources.path must default to NULL"
|
||||
finally:
|
||||
_delete_by_url(db, url)
|
||||
|
||||
|
||||
def test_check_constraint_rejects_unknown_kind(db: Session, alembic: Config) -> None:
|
||||
"""``ck_git_sources_kind`` is what later API validation relies on:
|
||||
any kind other than git|local raises IntegrityError."""
|
||||
command.upgrade(alembic, "head")
|
||||
with pytest.raises(IntegrityError):
|
||||
_insert(db, f"{URL_BASE}/bogus-kind.git", kind="bogus")
|
||||
db.rollback() # the IntegrityError aborts the open transaction
|
||||
|
||||
|
||||
def test_duplicate_local_path_rejected(db: Session, alembic: Config) -> None:
|
||||
"""The unique path index is what the API's 409 relies on: two local
|
||||
rows with the same path are rejected (NULL paths stay distinct —
|
||||
git rows are unaffected)."""
|
||||
command.upgrade(alembic, "head")
|
||||
url_a = f"{URL_BASE}/dup-path-a.git"
|
||||
url_b = f"{URL_BASE}/dup-path-b.git"
|
||||
url_c = f"{URL_BASE}/dup-path-c.git"
|
||||
path = f"{PATH_BASE}/shared"
|
||||
try:
|
||||
_insert(db, url_a, kind="local", path=path)
|
||||
with pytest.raises(IntegrityError):
|
||||
_insert(db, url_b, kind="local", path=path)
|
||||
db.rollback()
|
||||
# NULL paths are distinct under the unique index (git rows).
|
||||
_insert(db, url_b)
|
||||
_insert(db, url_c)
|
||||
finally:
|
||||
db.rollback()
|
||||
_delete_by_url(db, url_a)
|
||||
_delete_by_url(db, url_b)
|
||||
_delete_by_url(db, url_c)
|
||||
|
||||
|
||||
def test_downgrade_to_0006_drops_columns(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0006: both columns, the CHECK constraint, and the
|
||||
unique index are dropped (A13 — reversible)."""
|
||||
command.downgrade(alembic, "0006")
|
||||
assert _version(db) == "0006"
|
||||
|
||||
assert _column(db, "kind") is None, "git_sources.kind must be dropped"
|
||||
assert _column(db, "path") is None, "git_sources.path must be dropped"
|
||||
assert _check_constraint_def(db) is None, "ck_git_sources_kind must be dropped"
|
||||
assert _unique_path_index(db) == 0, "uq_git_sources_path must be dropped"
|
||||
|
||||
|
||||
def test_upgrade_round_trip_restores_columns(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0006, then upgrade back to 0007: columns, default,
|
||||
constraint, and index are back."""
|
||||
command.downgrade(alembic, "0006")
|
||||
command.upgrade(alembic, "0007")
|
||||
assert _version(db) == "0007", "round-trip upgrade must land at 0007"
|
||||
|
||||
kind = _column(db, "kind")
|
||||
assert kind is not None, "git_sources.kind must be back"
|
||||
assert kind[1] == "NO" and kind[2] is not None and "'git'" in kind[2], (
|
||||
"git_sources.kind must keep its NOT NULL 'git' default after the round-trip"
|
||||
)
|
||||
|
||||
path = _column(db, "path")
|
||||
assert path is not None and path[1] == "YES", "git_sources.path must be back"
|
||||
|
||||
constraint = _check_constraint_def(db)
|
||||
assert constraint is not None, "ck_git_sources_kind must be back"
|
||||
|
||||
assert _unique_path_index(db) == 1, "uq_git_sources_path must be back"
|
||||
Reference in New Issue
Block a user