phase: 135_git_source_branch
Build and Push Containers / build-and-push-app (push) Successful in 2m34s
Build and Push Containers / build-and-push-db (push) Successful in 14s

All verification complete. No edits were needed by this pass — the phase work and docs were already consistent and green.

**Phase 135 final verification pass — report**

- Verified all 5 tasks' work present & correct: migration 0024 (reversible), `GitSource.branch` + 4 schema shapes, `validate_branch_name` (full check order per design), `clone_or_pull(…, branch)` (HEAD guard first, `--branch` clone, fetch+ff merge), both call sites pass `row.branch`, UI (add-form field, editor field, row tag), `file://` URL transport.
- Completion criteria: (1) branch-mirrored through Sync + CLI — **pass** (E2E + integration); (2) `branch=None` byte-identical — **pass** (`test_branch_none_argv_is_byte_identical_pre_phase_sequence` green); (3) fixed 422s / local-row 422 / tri-state — **pass**; (4) mismatch fails loudly with remove-and-re-add — **pass** (E2E); (5) coverage 99% >90%, E2E isolated green — **pass**; (6) no regressions — **pass**.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → 3223 passed, app/ 99%; `uv run pytest tests/e2e/test_git_source_branch.py -v --no-cov` → 4 passed (16.4 s); `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings.
- Documentation: README.md reviewed in full — phase-135 branch section accurate; the phase's README diff also correctly fixed a pre-existing stale line (shallow `--depth 1` clone vs full-history, phase 107). Routes, env defaults, feature lists verified against code. AGENTS.md — all 10 rules + quick-reference verified (referenced tests, PLAN.md, phases dirs, commands all accurate). **Verified consistent — no changes needed.**
- No live-infrastructure changes (dev DB already running and healthy; no live steps in this phase).
- Next pending phase: `.agents/phases/todo/136_force_push_reset`.
This commit is contained in:
2026-09-28 21:15:03 -04:00
parent fcac876d40
commit d3f8c7f2c8
50 changed files with 3955 additions and 140 deletions
+266 -10
View File
@@ -58,6 +58,22 @@ injected ``https://x-access-token:<token>@…`` URL with a
credential-free checkout path (mock ``clone_or_pull``) and a legacy
row's clone still receives its ORIGINAL stored URL (the credential
keeps working).
Phase 135 (task 01 — branch storage + reporting): the output shapes
gain ``branch`` (NULL = the repo's default branch) — list DB rows
report the stored value, env-fallback rows report ``null`` (no DB row
to store one on), and POST 201 / PATCH 200 report the stored value.
Phase 135 (task 02 — branch validation + editing): POST stores a
branch on GIT rows only (absent/empty → NULL = the default branch;
non-empty → :func:`app.rag.git_sources.validate_branch_name`, a
``ValueError`` → the FIXED-detail 422 ``not a valid branch name`` —
the input is never echoed, and the row is not created); a ``branch``
on a LOCAL row is 422 ``branch is only valid for git sources`` (POST
and PATCH alike — local rows are walked, not cloned); PATCH edits
the branch with the token's EXACT tri-state — absent/None = no
change, non-empty = set (re-validated), empty string = clear (stored
NULL).
"""
from __future__ import annotations
@@ -161,10 +177,11 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "https://a.example.com/one.git",
"path": None,
"added_at": None,
# Env rows have no DB row to store a list on (phase 89) or
# a flag on (phase 105).
# Env rows have no DB row to store a list on (phase 89), a
# flag on (phase 105) or a branch on (phase 135).
"ignore_paths": [],
"include_hidden": False,
"branch": None,
},
{
"id": None,
@@ -174,6 +191,7 @@ def test_get_empty_table_with_env_returns_env_rows(
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
"branch": None,
},
]
@@ -241,10 +259,11 @@ def test_post_creates_trimmed_and_list_stops_using_env(
assert body["added_at"] is not None
# Phase 89: the response gains ``ignore_paths`` — absent at create
# time → ``[]``; phase 105 adds ``include_hidden`` — absent →
# ``False`` (A4).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
# ``False`` (A4); phase 135 adds ``branch`` — absent → ``null``.
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden", "branch"}
assert body["ignore_paths"] == []
assert body["include_hidden"] is False
assert body["branch"] is None
# The DB row now wins: from_env False, the env URL is gone from the list.
body = admin_client.get("/api/git-sources").json()
@@ -258,6 +277,10 @@ def test_post_accepts_accepted_url_shapes(admin_client: TestClient) -> None:
"http://git.local/repo.git",
"ssh://git@example.com/repo.git",
"git@github.com:owner/repo.git",
# Phase 135 (task 05): the file:// local transport — the E2E
# fixture repos the UI adds end to end (the phase-28/32/107
# transport-true idiom, promoted from the env fallback).
"file:///srv/repos/homelab-notes",
):
r = admin_client.post("/api/git-sources", json={"url": url})
assert r.status_code == 201, f"{url} must be accepted: {r.text}"
@@ -356,10 +379,12 @@ def test_post_local_creates_stored_row_with_expanded_path(
# The phase-35 response shape is unchanged — the local row reports
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
# phase 89 adds ``ignore_paths`` (absent → ``[]``); phase 105 adds
# ``include_hidden`` (absent → ``False``, A4).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
# ``include_hidden`` (absent → ``False``, A4); phase 135 adds
# ``branch`` (absent → ``null``).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden", "branch"}
assert body["ignore_paths"] == []
assert body["include_hidden"] is False
assert body["branch"] is None
uuid.UUID(body["id"])
assert body["url"] == str(real_dir)
assert body["added_at"] is not None
@@ -592,7 +617,8 @@ def test_delete_removes_row_and_falls_back_to_env(
"path": None,
"added_at": None,
"ignore_paths": [], # env rows: no DB row to store a list on
"include_hidden": False, # … or a flag on (phase 105)
"include_hidden": False, # … a flag on (phase 105)
"branch": None, # … or a branch on (phase 135)
}
]
db.execute(text("TRUNCATE chunks, documents"))
@@ -709,6 +735,7 @@ def test_get_reports_stored_ignore_paths_and_env_rows_empty(
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
"branch": None, # env rows: no DB row to store a branch on (135)
}
]
@@ -728,9 +755,10 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden", "branch"}
assert body["ignore_paths"] == ["a", "b"] # normalized
assert body["include_hidden"] is False # untouched by a list-only PATCH
assert body["branch"] is None # untouched by a list-only PATCH
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Round-trip through GET.
@@ -988,7 +1016,8 @@ def test_post_with_token_stores_column_and_never_echoes(
assert r.status_code == 201, r.text
body = r.json()
assert body["url"] == "https://github.com/owner/private.git"
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden", "branch"}
assert body["branch"] is None # no branch sent → the default-branch NULL
assert TOKEN not in r.text
row = db.scalars(select(GitSource)).one()
@@ -1329,7 +1358,8 @@ def _stub_sync_stack(
clone_calls: list[tuple[str, Path]] = []
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
def fake_clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path:
# phase 135: the row's branch pin (None for this file's rows)
dest = Path(dest)
dest.mkdir(parents=True, exist_ok=True)
clone_calls.append((url, dest))
@@ -1421,3 +1451,229 @@ def test_sync_public_row_clones_with_verbatim_url(
assert _poll_sync(sync_admin_client, "success")["error"] is None
assert clone_calls == [(url, tmp_path / "sources" / "public")]
# --- branch: create + validation (phase 135, task 02) ----------------------
def test_post_git_source_with_branch_stores_and_reports(
admin_client: TestClient, db: Session
) -> None:
"""POST a git source with ``branch: "release"`` → 201 reports the
branch; the stored model row carries it; the GET round-trip
reports it too (201, list DB row)."""
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/branched.git", "branch": "release"},
)
assert r.status_code == 201, r.text
body = r.json()
assert body["branch"] == "release"
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden", "branch"}
row = db.scalars(select(GitSource)).one()
assert row.branch == "release" # the column, not response-only
listed = admin_client.get("/api/git-sources").json()["sources"][0]
assert listed["branch"] == "release"
def test_post_git_source_branch_is_trimmed(admin_client: TestClient) -> None:
"""Surrounding whitespace on the branch is trimmed (the schema's
``_trim_branch`` + the API's defensive strip) before storage."""
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/trim.git", "branch": " release/1.2\t"},
)
assert r.status_code == 201, r.text
assert r.json()["branch"] == "release/1.2"
def test_post_git_source_empty_branch_stores_null(admin_client: TestClient) -> None:
"""Absent/None AND trimmed-EMPTY branch at create → stored NULL
(the repo's default branch — the pre-phase-135 behavior): the
create-time split from PATCH's clear leg (GitSourceIn docstring)."""
for i, branch_value in enumerate((None, "", " ")):
payload: dict[str, object] = {
"url": f"https://example.com/def-{i}.git"
} # distinct URLs: the loop must not 409 on itself
if branch_value is not None:
payload["branch"] = branch_value
r = admin_client.post("/api/git-sources", json=payload)
assert r.status_code == 201, r.text
assert r.json()["branch"] is None
def test_post_git_source_invalid_branch_422_fixed_detail_no_row(
admin_client: TestClient, db: Session
) -> None:
"""Invalid branch names are 422 with the EXACT fixed detail
``not a valid branch name`` — the input is never echoed (the
credential-safety discipline, applied for consistency) — and no
row is created."""
for bad in ("a..b", "a b", "a@b", "-x", ".x", "/x", "x/", "x.", "HEAD", "head"):
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/bad-branch.git", "branch": bad},
)
assert r.status_code == 422, f"{bad!r} must be rejected: {r.text}"
assert r.json()["detail"] == "not a valid branch name" # EXACT
assert bad not in r.text, f"422 must not echo the input ({bad!r})"
# Nothing stored.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_post_git_source_oversized_branch_422(admin_client: TestClient, db: Session) -> None:
""">200 chars: the schema's ``max_length=200`` rejects it before
the handler (FastAPI's standard schema 422 — the same house
precedent as the oversized-URL 422, which this router also
answers with the framework's standard shape); no row is created.
The API-layer branch 422s (charset/sequence/leading/trailing/
HEAD) are the FIXED-detail ones — see the adjacent tests."""
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/long-branch.git", "branch": "a" * 201},
)
assert r.status_code == 422
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_post_local_source_with_branch_422(
admin_client: TestClient, tmp_path: Path, db: Session
) -> None:
"""A ``branch`` on a LOCAL row is 422 with the fixed detail (local
rows are walked, not cloned) — for BOTH valid and invalid branch
names (the kind check precedes validation); nothing is stored."""
real_dir = tmp_path / "local-branch"
real_dir.mkdir()
for bad in ("main", "a..b"):
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "path": str(real_dir), "branch": bad},
)
assert r.status_code == 422, f"{bad!r} must be rejected: {r.text}"
assert r.json()["detail"] == "branch is only valid for git sources"
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
# --- branch: PATCH tri-state (phase 135, task 02) ---------------------------
def test_patch_branch_set_change_clear_and_noop(
admin_client: TestClient, db: Session
) -> None:
"""The branch tri-state on a git row — the token's exact
semantics field-for-field:
* set: ``"hotfix/x"`` → 200 reports it (stored + GET round-trip);
* change: ``"hotfix/y"`` → 200 reports the new value;
* clear: ``""`` → 200 reports ``null`` (stored NULL = the
default branch); whitespace-only trims to empty → the same
clear leg;
* no-op: branch ABSENT (or explicit None) → 200, the stored
branch survives (a flag-only PATCH must not touch it).
"""
created = admin_client.post(
"/api/git-sources", json={"url": "https://example.com/patch-branch.git"}
)
assert created.status_code == 201
sid = created.json()["id"]
assert created.json()["branch"] is None # started on the default branch
def stored_branch() -> str | None:
# The endpoint commits in its own session — expire this one's
# identity map before reading back (the house pattern).
db.expire_all()
row = db.get(GitSource, uuid.UUID(sid))
assert row is not None
return row.branch
# SET — "hotfix/x".
r = admin_client.patch(f"/api/git-sources/{sid}", json={"branch": "hotfix/x"})
assert r.status_code == 200, r.text
assert r.json()["branch"] == "hotfix/x"
assert stored_branch() == "hotfix/x"
assert (
admin_client.get("/api/git-sources").json()["sources"][0]["branch"] == "hotfix/x"
)
# CHANGE — "hotfix/y".
r = admin_client.patch(f"/api/git-sources/{sid}", json={"branch": "hotfix/y"})
assert r.status_code == 200, r.text
assert r.json()["branch"] == "hotfix/y"
assert stored_branch() == "hotfix/y"
# NO-OP — an unrelated field only: the branch is UNTOUCHED.
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": True})
assert r.status_code == 200, r.text
assert r.json()["branch"] == "hotfix/y"
assert r.json()["include_hidden"] is True
assert stored_branch() == "hotfix/y"
# NO-OP — explicit None is absent too.
r = admin_client.patch(f"/api/git-sources/{sid}", json={"branch": None})
assert r.status_code == 200, r.text
assert r.json()["branch"] == "hotfix/y"
assert stored_branch() == "hotfix/y"
# CLEAR — "" (the tri-state's clear leg; a deliberate empty save).
r = admin_client.patch(f"/api/git-sources/{sid}", json={"branch": ""})
assert r.status_code == 200, r.text
assert r.json()["branch"] is None
assert stored_branch() is None
# Whitespace-only trims to empty → the same clear leg.
admin_client.patch(f"/api/git-sources/{sid}", json={"branch": "hotfix/x"})
r = admin_client.patch(f"/api/git-sources/{sid}", json={"branch": " "})
assert r.status_code == 200, r.text
assert r.json()["branch"] is None
assert stored_branch() is None
def test_patch_branch_invalid_422_leaves_row_unchanged(
admin_client: TestClient, db: Session
) -> None:
"""A PRESENT invalid branch 422s with the EXACT fixed detail —
and the stored branch is untouched (validation raises before the
assignment; the row's other fields in the same body never
half-apply)."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/patch-bad.git", "branch": "stable"},
)
assert created.status_code == 201
sid = created.json()["id"]
for bad in ("a..b", "HEAD", "x/"):
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"branch": bad, "include_hidden": True}
)
assert r.status_code == 422, f"{bad!r}: {r.text}"
assert r.json()["detail"] == "not a valid branch name"
assert bad not in r.text
# Nothing half-applied: the branch AND the flag are still the
# originals.
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["branch"] == "stable"
assert row["include_hidden"] is False
def test_patch_branch_on_local_row_422(admin_client: TestClient, tmp_path: Path) -> None:
"""A PRESENT branch on a LOCAL row is the same fixed 422 as POST
(local rows are walked, not cloned) — valid name or not; the row
is untouched."""
real_dir = tmp_path / "local-patch"
real_dir.mkdir()
created = admin_client.post(
"/api/git-sources", json={"kind": "local", "path": str(real_dir)}
)
assert created.status_code == 201
sid = created.json()["id"]
for branch_value in ("main", "a..b"):
r = admin_client.patch(f"/api/git-sources/{sid}", json={"branch": branch_value})
assert r.status_code == 422, f"{branch_value!r}: {r.text}"
assert r.json()["detail"] == "branch is only valid for git sources"
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["branch"] is None
assert row["kind"] == "local"
+219 -4
View File
@@ -23,10 +23,22 @@ is asserted against the canned value).
stderr, and **zero** import attempts.
- A missing local directory → the same pre-import fail-loud: exit code
1, ``local source missing: <path>`` on stderr, zero import attempts.
Phase 135 (the branch pin) adds the REAL-git section at the bottom:
a two-branch ``file://`` fixture repo (``main`` with file A, ``release``
with file B) drives the real :func:`scripts.git_sync.clone_or_pull` —
the fresh ``--branch`` checkout, the ff-merge pull of a new commit on
the pinned branch, the fail-loud branch MISMATCH (checkout untouched),
and the ``_resolve_sources`` seam with a ``branch='release'`` row
(importing file B, not file A) + an env-origin row without a branch
(the default branch, byte-identical pre-phase behavior). Skipped, not
failed, on a machine without the git CLI (the house pattern).
"""
from __future__ import annotations
import os
import re
import subprocess
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
@@ -39,7 +51,7 @@ from app.config import Settings
from app.models import GitSource
from app.rag.importer import ImportSummary
from scripts import import_docs
from scripts.git_sync import GitSyncError
from scripts.git_sync import GitSyncError, clone_or_pull
from tests.fakes import FakeEmbedder
@@ -106,10 +118,15 @@ class FakeImportSources:
def _fake_clone_factory() -> tuple[list[tuple[str, Path]], object]:
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
"""A ``clone_or_pull`` that materialises a checkout with one .md
file. Phase 135: accepts the row's branch pin (``None`` for every
row in this file's fixtures — the real-git branch behavior is
pinned by the phase-135 section below, which uses the REAL
``clone_or_pull``)."""
calls: list[tuple[str, Path]] = []
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
def fake_clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path:
assert branch is None # every row in this file's fixtures pins no branch
dest = Path(dest)
dest.mkdir(parents=True, exist_ok=True)
(dest / "notes.md").write_text(f"# {dest.name}\ncontent for the KB\n", encoding="utf-8")
@@ -687,7 +704,7 @@ def test_main_git_failure_aborts_before_import(
lambda db: ([_git_row("https://host/a/bad.git")], "env"),
)
def failing_clone(url: str, dest: Path | str) -> Path:
def failing_clone(url: str, dest: Path | str, branch: str | None = None) -> Path:
raise GitSyncError(
f"git clone {url} failed (exit 128): "
"fatal: repository not found"
@@ -732,3 +749,201 @@ def test_main_missing_local_dir_aborts_before_import(
assert "import_docs: source sync failed" in err
assert f"local source missing: {missing}" in err # the path is named
assert fake_import.calls == [] # no partial import
# --- phase 135: the branch pin (real git, two-branch file:// origin) -------
def _git_available() -> bool:
try:
proc = subprocess.run(["git", "--version"], capture_output=True, check=False)
return proc.returncode == 0
except (FileNotFoundError, OSError):
return False
#: Real ``git`` in the test environment — skipped cleanly without it
#: (the ``test_git_file_dates.py`` house pattern).
GIT = _git_available()
@pytest.fixture()
def two_branch_repo(tmp_path: Path) -> Path:
"""A real-git scratch origin (the phase-135 recipe): ``main`` holds
``a.md``, ``release`` holds ``b.md`` — different content per branch
(a branch checkout is the branch's tree, not main's)."""
repo = tmp_path / "origin"
repo.mkdir()
_git(repo, "init", "-q", "-b", "main")
_git(repo, "config", "commit.gpgsign", "false")
(repo / "a.md").write_text("# A\nmain branch content\n", encoding="utf-8")
_git(repo, "add", "a.md")
_git(repo, "commit", "-q", "-m", "main: add a.md")
_git(repo, "checkout", "-q", "-b", "release")
(repo / "a.md").unlink()
(repo / "b.md").write_text("# B\nrelease branch content v1\n", encoding="utf-8")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "release: add b.md")
_git(repo, "checkout", "-q", "main")
return repo
def _git(cwd: Path, *argv: str) -> None:
"""Run one git command for the test harness (fixture setup); a
non-zero exit fails the fixture, not the test under test."""
env = os.environ.copy()
env["GIT_AUTHOR_NAME"] = "T"
env["GIT_AUTHOR_EMAIL"] = "t@example.com"
env["GIT_COMMITTER_NAME"] = "T"
env["GIT_COMMITTER_EMAIL"] = "t@example.com"
proc = subprocess.run(
["git", *argv], cwd=cwd, env=env, capture_output=True, text=True, check=False
)
assert proc.returncode == 0, f"git {' '.join(argv)} failed: {proc.stderr}"
def _git_out(cwd: Path, *argv: str) -> str:
"""One harness git read (a non-zero exit fails the test)."""
proc = subprocess.run(
["git", *argv], cwd=cwd, capture_output=True, text=True, check=False
)
assert proc.returncode == 0, f"git {' '.join(argv)} failed: {proc.stderr}"
return proc.stdout
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_clone_or_pull_fresh_branch_checkout_is_on_that_branch(
tmp_path: Path, two_branch_repo: Path
) -> None:
"""Phase 135: a FRESH ``clone_or_pull(url, dest, 'release')``
(real git, ``file://``) checks out the pinned branch — HEAD is
``release``, file B present, file A (main's tree) absent."""
url = f"file://{two_branch_repo}"
dest = tmp_path / "bor" / "origin"
result = clone_or_pull(url, dest, "release")
assert result == dest
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "release"
assert (dest / "b.md").is_file()
assert (dest / "b.md").read_text(encoding="utf-8") == "# B\nrelease branch content v1\n"
assert not (dest / "a.md").exists()
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_clone_or_pull_existing_branch_checkout_ff_merges_new_commits(
tmp_path: Path, two_branch_repo: Path
) -> None:
"""Phase 135: a SECOND ``clone_or_pull`` of the same pinned branch
after the origin gains a commit on it → the guarded checkout is
ff-merged onto the new tip (file B updated, no error, still on
``release``)."""
url = f"file://{two_branch_repo}"
dest = tmp_path / "bor" / "origin"
clone_or_pull(url, dest, "release")
# the origin gains a commit on release (v2 of file B)
_git(two_branch_repo, "checkout", "-q", "release")
(two_branch_repo / "b.md").write_text("# B\nrelease branch content v2\n", encoding="utf-8")
_git(two_branch_repo, "add", "b.md")
_git(two_branch_repo, "commit", "-q", "-m", "release: bump b.md")
_git(two_branch_repo, "checkout", "-q", "main")
clone_or_pull(url, dest, "release")
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "release"
assert (dest / "b.md").read_text(encoding="utf-8") == "# B\nrelease branch content v2\n"
assert not (dest / "a.md").exists()
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_clone_or_pull_branch_mismatch_fails_loud_checkout_untouched(
tmp_path: Path, two_branch_repo: Path
) -> None:
"""Phase 135 (ASSUMPTION B5): a checkout on the DEFAULT branch
(fresh clone with no branch → ``main``) + a source row pinning
``release`` → the remove-and-re-add ``GitSyncError`` — and the
checkout is UNTOUCHED (still on main, still main's tree: no silent
branch switch)."""
url = f"file://{two_branch_repo}"
dest = tmp_path / "bor" / "origin"
clone_or_pull(url, dest) # no branch → the origin's default (main)
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "main"
assert (dest / "a.md").is_file()
with pytest.raises(GitSyncError) as excinfo:
clone_or_pull(url, dest, "release")
msg = str(excinfo.value)
assert "remove and re-add the source to switch branches" in msg
assert "'main'" in msg and "'release'" in msg
assert url not in msg # the guard message never carries the URL
# the checkout is untouched: still on main, still main's tree
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "main"
assert (dest / "a.md").is_file()
assert not (dest / "b.md").exists()
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_resolve_sources_git_row_with_branch_checks_out_that_branch(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, two_branch_repo: Path
) -> None:
"""Phase 135, the CLI seam: a ``git_sources`` row carrying
``branch='release'`` → ``_resolve_sources`` passes ``row.branch`` to
the REAL ``clone_or_pull`` (no fake — the branch is honored
end to end): the resolved root is the RELEASE tree (file B imported,
file A absent), and the date walk over the real checkout lists ONLY
the branch's tree (head-of-branch semantics, ``file_commit_dates``
untouched)."""
url = f"file://{two_branch_repo}"
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([GitSource(url=url, kind="git", branch="release")], "db"),
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
root = tmp_path / "bor" / "origin"
assert sources == [root]
assert ignore_map == {}
assert hidden_map == {str(root): False}
assert (root / "b.md").is_file() # the branch's file is what gets imported
assert not (root / "a.md").exists()
# the (real, untouched) date walk over the branch checkout yields
# TRUE dates for the branch's files (not the fail-soft ``{}`` — the
# checkout is a real full-history repo). The walk reads the
# reachable LOG (``git log --name-only``), so it also lists files
# from shared ancestor commits (``a.md`` — release was cut after
# main's first commit); the importer only consults the map for the
# files it actually finds in the working tree, so that is harmless.
dates = date_map[str(root)]
assert "b.md" in dates
assert isinstance(dates["b.md"], datetime)
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_resolve_sources_row_without_branch_clones_default_branch(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, two_branch_repo: Path
) -> None:
"""Phase 135, the byte-identical leg: a row WITHOUT a branch (the
env-fallback synthetic rows' shape — ``GitSource(url=…, kind="git")``
carries the model default ``branch=None``) → the real
``clone_or_pull`` takes the pre-phase path: the origin's DEFAULT
branch (``main``) is checked out — file A present, file B absent."""
url = f"file://{two_branch_repo}"
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_git_row(url)], "env"), # branch defaults to None
)
settings = _settings(git_sources=url, sources_dir=str(tmp_path / "bor"))
sources, *_ = import_docs._resolve_sources(None, settings)
root = tmp_path / "bor" / "origin"
assert sources == [root]
assert (root / "a.md").is_file() # main's file — the default branch
assert not (root / "b.md").exists()
+2 -4
View File
@@ -319,10 +319,8 @@ def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) ->
with the full contract (TEXT, NULLABLE, no server default) — a
clean upgrade → downgrade → upgrade round-trip (A13)."""
command.downgrade(alembic, "0022")
command.upgrade(alembic, "head")
assert _version(db) == "0023", (
"round-trip upgrade must land at head (0023)"
)
command.upgrade(alembic, "0023")
assert _version(db) == "0023", "round-trip upgrade must land at 0023"
context = _column(db, "image_context")
assert context is not None, "documents.image_context must be back"
+315
View File
@@ -0,0 +1,315 @@
"""Integration: migration 0024 (git_sources.branch) schema contract
(phase 135, task 01).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0021.py`` (information_schema assertions on the state
the migration must leave). The tests target the 0023 → 0024 step
explicitly so later migrations cannot break the pins:
* upgrade 0023 → 0024 → ``branch`` exists with the full contract —
TEXT, NULLABLE, no server default — while the 0023 ``git_sources``
schema (``url`` NOT NULL + the unique index, ``kind``, ``path``,
``ignore_paths``, ``include_hidden``, ``token``, ``added_at``)
survives;
* rows inserted while the DB is at 0023 backfill ``branch`` to NULL —
ALL of them (the byte-identical guarantee: a pre-phase-135 row is a
default-branch source by definition; no backfill, no default);
* the ORM contract agrees: a freshly inserted ``GitSource`` with an
explicit ``branch`` round-trips it through a fresh session, and a
row without one reads ``branch is None``;
* downgrade to 0023 → the column is GONE (A13) while the row + its
``url`` survive; upgrade back to 0024 → the column is back
(round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
import uuid
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.orm import Session
from alembic import command
from app.db import SessionLocal, db_available
from app.models import GitSource
URL_A = "https://github.com/mig0024/repo-a.git"
URL_B = "https://github.com/mig0024/repo-b.git"
BRANCH = "release/1.0"
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown
upgrades to head no matter what happened, so the dev DB is never
left below head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
# on ``git_sources``, which would deadlock the repair's
# ``ALTER TABLE`` (0024) forever.
db.rollback()
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one git_sources
column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'git_sources' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _insert_sql(db: Session, url: str, token: str | None = None) -> uuid.UUID:
"""Insert one git_sources row with the PRE-0024 column set (the
0023 shape — the branch column, when present, is omitted so a NULL
backfill is what the row reads)."""
row_id = uuid.uuid4()
db.execute(
text(
"INSERT INTO git_sources (id, url, kind, path, ignore_paths,"
" include_hidden, token)"
" VALUES (:id, :u, 'git', NULL, '[]', false, :t)"
),
{"id": row_id, "u": url, "t": token},
)
db.commit()
return row_id
def _delete(db: Session, row_id: uuid.UUID) -> None:
db.execute(text("DELETE FROM git_sources WHERE id = :id"), {"id": row_id})
db.commit()
def test_upgrade_to_0024_adds_branch(db: Session, alembic: Config) -> None:
"""Upgrade 0023 → 0024: ``branch`` exists with the full contract
(TEXT, NULLABLE, no server default — NULL = the repo's default
branch), is ABSENT at 0023, and every pre-0024 row (WITH and
WITHOUT a stored token) backfills ``branch`` to NULL — the
byte-identical guarantee, no backfill and no default — while the
0023 table contract (``url`` + unique index, ``kind``,
``ignore_paths``, ``include_hidden``, ``token``, ``added_at``)
survives."""
command.downgrade(alembic, "0023") # start from the pre-0024 state
assert _version(db) == "0023"
assert _column(db, "branch") is None, "branch must be absent at 0023"
pre_a = _insert_sql(db, URL_A) # the 0023 column set
pre_b = _insert_sql(db, URL_B, token="ghp_mig0024secret")
try:
command.upgrade(alembic, "0024")
assert _version(db) == "0024", "alembic_version must be at 0024"
branch = _column(db, "branch")
assert branch is not None, "git_sources.branch is missing"
assert branch[0] == "text", "branch must be TEXT"
assert branch[1] == "YES", "branch must be NULLABLE"
assert branch[2] is None, (
"branch must carry NO server default — NULL is the"
" default-branch value"
)
# Every pre-0024 row backfilled to NULL (the repo's default
# branch — the byte-identical guarantee for ALL existing rows).
for pre_id, url in ((pre_a, URL_A), (pre_b, URL_B)):
row = db.execute(
text("SELECT url, token, branch FROM git_sources WHERE id = :id"),
{"id": pre_id},
).fetchone()
assert row is not None and row[0] == url, (
"the pre-0024 row must survive the upgrade"
)
assert row[2] is None, (
"every pre-0024 row must backfill branch to NULL — no"
" backfill, no default"
)
# The tokened row kept its credential (the upgrade is
# branch-only).
assert db.execute(
text("SELECT token FROM git_sources WHERE id = :id"),
{"id": pre_b},
).scalar() == "ghp_mig0024secret", (
"a pre-0024 stored token must survive the branch upgrade"
)
# A row written without the column stays NULL (no default to
# fill it in — the Python-side default is None, same value).
new_id = _insert_sql(db, URL_A.replace("repo-a", "repo-c"))
try:
omitted = db.execute(
text("SELECT branch FROM git_sources WHERE id = :id"),
{"id": new_id},
).scalar()
assert omitted is None, "an omitted branch must stay NULL"
finally:
_delete(db, new_id)
# The 0023 schema survives the additive upgrade.
url = _column(db, "url")
assert url is not None and url[0] == "text" and url[1] == "NO", (
"git_sources.url (0006) must keep its 0023 contract"
)
kind = _column(db, "kind")
assert kind is not None and kind[0] == "text" and kind[1] == "NO", (
"git_sources.kind (0007) must survive the upgrade"
)
ignore = _column(db, "ignore_paths")
assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", (
"git_sources.ignore_paths (0013) must survive the upgrade"
)
hidden = _column(db, "include_hidden")
assert hidden is not None and hidden[0] == "boolean" and hidden[1] == "NO", (
"git_sources.include_hidden (0019) must survive the upgrade"
)
token = _column(db, "token")
assert token is not None and token[0] == "text" and token[1] == "YES", (
"git_sources.token (0021) must survive the upgrade"
)
added = _column(db, "added_at")
assert added is not None and added[0] == "timestamp with time zone", (
"git_sources.added_at (0006) must survive the upgrade"
)
assert added[1] == "NO" and "now()" in str(added[2]), (
"git_sources.added_at must keep its `now()` server default"
)
index = db.execute(
text(
"SELECT indexname FROM pg_indexes"
" WHERE tablename = 'git_sources'"
" AND indexname = 'uq_git_sources_url'"
)
).scalar()
assert index is not None, (
"the uq_git_sources_url unique index must survive the upgrade"
)
finally:
_delete(db, pre_a)
_delete(db, pre_b)
def test_orm_branch_round_trips(db: Session, alembic: Config) -> None:
"""The ORM contract agrees with the column contract: a freshly
inserted ``GitSource`` with an explicit ``branch`` round-trips the
value through a FRESH session, and a row without one reads
``branch is None`` (the NULL default-branch state)."""
command.upgrade(alembic, "head")
row_branched = GitSource(url=URL_A, kind="git", branch=BRANCH)
row_default = GitSource(url=URL_B, kind="git")
db.add(row_branched)
db.add(row_default)
db.commit()
try:
with SessionLocal() as fresh:
reloaded_branched = fresh.get(GitSource, row_branched.id)
assert reloaded_branched is not None, "the branched row must be readable"
assert reloaded_branched.branch == BRANCH, (
"the explicit branch must round-trip through the DB"
)
reloaded_default = fresh.get(GitSource, row_default.id)
assert reloaded_default is not None, "the default row must be readable"
assert reloaded_default.branch is None, (
"a row without a branch must read branch is None"
)
finally:
_delete(db, row_branched.id)
_delete(db, row_default.id)
def test_downgrade_to_0023_drops_the_column(db: Session, alembic: Config) -> None:
"""Downgrade 0024 → 0023: the branch column is gone (A13 — fully
reversible) while the row + its ``url`` (and ``token``) survive,
and the rest of the 0023 table contract (``url`` unique index,
``kind``, ``added_at``) is intact."""
command.upgrade(alembic, "head")
row = GitSource(url=URL_A, kind="git", branch=BRANCH, token="ghp_mig0024secret")
db.add(row)
db.commit()
try:
command.downgrade(alembic, "0023")
assert _version(db) == "0023"
assert _column(db, "branch") is None, "branch must be dropped"
surviving = db.execute(
text(
"SELECT url, kind, ignore_paths, include_hidden, token, added_at"
" FROM git_sources WHERE id = :id"
),
{"id": row.id},
).fetchone()
assert surviving is not None and surviving[0] == URL_A, (
"the row must survive the column drop"
)
assert surviving[1] == "git" and surviving[2] == [] and surviving[3] is False, (
"kind + ignore_paths + include_hidden must survive the drop"
)
assert surviving[4] == "ghp_mig0024secret", "token must survive the drop"
assert surviving[5] is not None, "added_at must survive the drop"
index = db.execute(
text(
"SELECT indexname FROM pg_indexes"
" WHERE tablename = 'git_sources'"
" AND indexname = 'uq_git_sources_url'"
)
).scalar()
assert index is not None, "the unique index must survive the downgrade"
finally:
_delete(db, row.id)
# Repair: the fixture teardown re-upgrades to head.
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
"""Downgrade to 0023, then upgrade back to 0024: ``branch`` is back
with the full contract (TEXT, NULLABLE, no server default), and a
row seeded at 0023 reads ``branch is None`` (the byte-identical
guarantee on the round-trip)."""
command.downgrade(alembic, "0023")
pre_id = _insert_sql(db, URL_B)
try:
command.upgrade(alembic, "0024")
assert _version(db) == "0024", "round-trip upgrade must land at 0024"
branch = _column(db, "branch")
assert branch is not None, "git_sources.branch must be back"
assert branch[0] == "text", "branch must be TEXT after the round-trip"
assert branch[1] == "YES", "branch must be NULLABLE after the round-trip"
assert branch[2] is None, (
"branch must still carry NO server default after the round-trip"
)
assert db.execute(
text("SELECT branch FROM git_sources WHERE id = :id"),
{"id": pre_id},
).scalar() is None, (
"a row seeded at 0023 must read branch NULL after the round-trip"
)
finally:
_delete(db, pre_id)
+147 -4
View File
@@ -91,6 +91,8 @@ from __future__ import annotations
import asyncio
import logging
import os
import subprocess
import threading
import time
from collections.abc import Callable, Iterator, Sequence
@@ -113,7 +115,7 @@ from app.rag.folder_summaries import affected_folder_keys
from app.rag.importer import ImportSummary
from app.rag.llm import EmbeddingError, LLMClient, ModelUnavailableError
from app.rag.sources_meta import current_sources_version
from scripts.git_sync import GitSyncError
from scripts.git_sync import GitSyncError, clone_or_pull
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
@@ -414,10 +416,12 @@ class FakeFolderSummaries:
def _fake_clone() -> tuple[list[tuple[str, Path]], object]:
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
"""A ``clone_or_pull`` that materialises a checkout with one .md
file. Phase 135: accepts the row's branch pin (the third argument
— ``None`` for every row this file seeds)."""
calls: list[tuple[str, Path]] = []
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
def fake_clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path:
dest = Path(dest)
dest.mkdir(parents=True, exist_ok=True)
(dest / "notes.md").write_text(f"# {dest.name}\ncontent for the KB\n", encoding="utf-8")
@@ -1193,7 +1197,7 @@ def test_git_failure_marks_failed_and_skips_import(
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
def failing_clone(url: str, dest: Path | str) -> Path:
def failing_clone(url: str, dest: Path | str, branch: str | None = None) -> Path:
raise GitSyncError(
f"git clone {url} failed (exit 128): "
"fatal: repository not found"
@@ -1993,3 +1997,142 @@ def test_cancel_mid_run_lands_in_cancelled_then_resyncs_clean(
assert current_sources_version(db) == 1
assert len(second_import.sources) == 1
assert second_import.prune_flags == [True]
# --- phase 135: the branch pin (real git, two-branch file:// origin) -------
def _git_available() -> bool:
try:
proc = subprocess.run(["git", "--version"], capture_output=True, check=False)
return proc.returncode == 0
except (FileNotFoundError, OSError):
return False
#: Real ``git`` in the test environment — skipped cleanly without it
#: (the ``test_git_file_dates.py`` house pattern).
GIT = _git_available()
def _git(cwd: Path, *argv: str) -> None:
"""Run one git command for the test harness (fixture setup); a
non-zero exit fails the fixture, not the test under test."""
env = os.environ.copy()
env["GIT_AUTHOR_NAME"] = "T"
env["GIT_AUTHOR_EMAIL"] = "t@example.com"
env["GIT_COMMITTER_NAME"] = "T"
env["GIT_COMMITTER_EMAIL"] = "t@example.com"
proc = subprocess.run(
["git", *argv], cwd=cwd, env=env, capture_output=True, text=True, check=False
)
assert proc.returncode == 0, f"git {' '.join(argv)} failed: {proc.stderr}"
def _two_branch_origin(tmp_path: Path) -> Path:
"""A real-git scratch origin (the phase-135 recipe): ``main`` holds
``a.md``, ``release`` holds ``b.md`` — different content per
branch (a branch checkout is the branch's tree, not main's)."""
origin = tmp_path / "origin"
origin.mkdir()
_git(origin, "init", "-q", "-b", "main")
_git(origin, "config", "commit.gpgsign", "false")
(origin / "a.md").write_text("# A\nmain branch content\n", encoding="utf-8")
_git(origin, "add", "a.md")
_git(origin, "commit", "-q", "-m", "main: add a.md")
_git(origin, "checkout", "-q", "-b", "release")
(origin / "a.md").unlink()
(origin / "b.md").write_text("# B\nrelease branch content\n", encoding="utf-8")
_git(origin, "add", "-A")
_git(origin, "commit", "-q", "-m", "release: add b.md")
_git(origin, "checkout", "-q", "main")
return origin
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_git_row_with_branch_button_path_clones_that_branch(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 135: the BUTTON path honors the row's branch pin — a
``git_sources`` row with ``branch='release'`` is cloned with the
REAL ``clone_or_pull`` (no fake in this test): the imported root
is the RELEASE tree (file B present, file A absent)."""
origin = _two_branch_origin(tmp_path)
db.add(GitSource(url=f"file://{origin}", kind="git", branch="release"))
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["error"] is None
root = tmp_path / "bor" / "origin"
assert fake_import.sources == [[root]]
assert (root / "b.md").is_file() # the branch's file is what got imported
assert not (root / "a.md").exists()
@pytest.mark.skipif(not GIT, reason="git CLI not available")
def test_git_row_branch_mismatch_button_path_fails_loud(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 135 (ASSUMPTION B5): a row pinning a branch the EXISTING
checkout does not sit on fails the button sync LOUDLY — the
remove-and-re-add message, zero import attempts, and the checkout
untouched (still on its branch: no silent switch)."""
origin = _two_branch_origin(tmp_path)
url = f"file://{origin}"
dest = tmp_path / "bor" / "origin"
# A pre-existing checkout on the default branch (main) — e.g. a
# row that existed before the branch was pinned (the sync's
# sources_dir is seeded with it before the run starts).
clone_or_pull(url, dest)
db.add(GitSource(url=url, kind="git", branch="release"))
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary())
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert "remove and re-add the source to switch branches" in body["error"]
assert "'main'" in body["error"] and "'release'" in body["error"]
assert body["detail"] == {}
assert fake_import.sources == [] # no partial import
# Phase 53: a FAILED sync never bumps.
assert current_sources_version(db) == 0
# the checkout is untouched: still on main, still main's tree
proc = subprocess.run(
["git", "rev-parse", "--abbrev-ref", "HEAD"],
cwd=dest, capture_output=True, text=True, check=False,
)
assert proc.returncode == 0 and proc.stdout.strip() == "main"
assert (dest / "a.md").is_file()
assert not (dest / "b.md").exists()