Everything is verified green and consistent. Final report: **Phase 136 — final verification pass (all tasks already complete in `complete/136_force_push_reset/`):** - **Verified implemented:** unified ff-or-reset update sequence in `scripts/git_sync.py::clone_or_pull` (phase-135 HEAD guard → shallow self-heal → `fetch origin <ref>` → `merge --ff-only` → on non-ff: SHA capture + `reset --hard FETCH_HEAD` + one WARNING line, returns normally); detached-HEAD fail-loud; `branch=None` legacy `git pull` deliberately gone - **Completion criteria — all met:** 1. ✅ Force push no longer fails sync (integration `test_clone_or_pull_force_push_resets_to_the_new_head` + API-seam `test_sync_pipeline_survives_a_force_push` + E2E run 2 asserts `success` and KB serves the V2 sentinel) 2. ✅ Unrelated-history force push handled by the same reset (`test_clone_or_pull_unrelated_history_force_push_resets`) 3. ✅ Broken fetch still fails loud (fetch-failure unit pins raise with no merge/reset argv; `test_main_git_failure_aborts_before_import`) 4. ✅ WARNING line pinned: one line, 7-char SHAs, checkout path, `origin/<ref>`, no URL (unit exact-string pin + integration caplog pin) 5. ✅ `uv run pytest --cov=app --cov-report=term-missing` → **3231 passed, app/ 99%**; `uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov` → **2 passed in isolation** (10.6 s) 6. ✅ Phase-135 pins preserved (mismatch guard, `--branch` clone argv, `branch=None` fresh-clone argv all green in the 53-test run) - **Test/lint results:** `uv run pytest tests/unit/test_git_sync.py tests/integration/test_import_docs_git.py -v` → 53 passed; full suite 3231 passed (72 s); `uv run ruff check . && uv run pyright` → clean (0 errors) - **Documentation:** README.md git-sources section already updated this phase to the ff-or-reset contract; all `.env.example` vars documented; pages table matches router; AGENTS.md rules and quick-reference commands verified accurate (rule 5 correctly documents the phase-100 §7.1 supersession) — **no changes needed** - **Notable:** E2E "teeth" verification (fails on pre-phase code) is recorded in the task report; nothing was committed per harness protocol - **Next pending phase:** `todo/137_pdf_image_context`
1334 lines
54 KiB
Python
1334 lines
54 KiB
Python
"""Integration test: ``import_docs`` source resolution (phase 28, task
|
|
03; phase 35 re-points at the shared resolver; phase 38 adds the local
|
|
kind).
|
|
|
|
Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull``
|
|
(no real git, no network) and a recording fake ``import_sources`` (no
|
|
real DB), covering: the phase-53 version bump is stubbed in the
|
|
``main()`` tests the same way (the ``sources_version=`` summary token
|
|
is asserted against the canned value).
|
|
|
|
- Effective sources set (phase 35: the shared resolver — stubbed here,
|
|
keeping this file's no-real-DB style) → each git URL is cloned/pulled
|
|
into ``BOR_SOURCES_DIR/<repo-name>/``; local rows are their existing
|
|
directories, walked directly; exactly those dirs are imported.
|
|
- DB rows (both kinds) win over ``BOR_GIT_SOURCES`` (the resolver's
|
|
``db`` origin — the env list is ignored; the env fallback stays
|
|
git-only).
|
|
- ``--source`` still wins over the DB rows (no git at all, no resolver
|
|
call).
|
|
- No sources configured + no ``--source`` → the legacy
|
|
``DEFAULT_SOURCES``.
|
|
- A failing git sync → exit code 1, an error naming the failing repo on
|
|
stderr, and **zero** import attempts.
|
|
- A missing local directory → the same pre-import fail-loud: exit code
|
|
1, ``local source missing: <path>`` on stderr, zero import attempts.
|
|
|
|
Phase 135 (the branch pin) adds the REAL-git section at the bottom:
|
|
a two-branch ``file://`` fixture repo (``main`` with file A, ``release``
|
|
with file B) drives the real :func:`scripts.git_sync.clone_or_pull` —
|
|
the fresh ``--branch`` checkout, the ff-merge pull of a new commit on
|
|
the pinned branch, the fail-loud branch MISMATCH (checkout untouched),
|
|
and the ``_resolve_sources`` seam with a ``branch='release'`` row
|
|
(importing file B, not file A) + an env-origin row without a branch
|
|
(the default branch, byte-identical pre-phase behavior). Skipped, not
|
|
failed, on a machine without the git CLI (the house pattern).
|
|
|
|
Phase 136 (the ff-or-reset) extends the REAL-git section: the owner's
|
|
force-push repro (the origin's history REWRITTEN to a new root commit
|
|
between two ``clone_or_pull`` runs — normal divergence AND fully
|
|
unrelated history) now resets the checkout to the origin's new head
|
|
instead of failing (the loud WARNING line pinned alongside), a
|
|
branched row's force push lands ``origin/<branch>``'s head (main's
|
|
content not involved), ``file_commit_dates`` over the post-reset
|
|
checkout reflects the NEW head's dates, and the SYNC PIPELINE END-
|
|
TO-END over the API seam (``POST /api/sync`` + the status poll — the
|
|
seam ``test_sync_api.py`` exercises) lands ``success`` after a force
|
|
push, with the checkout (and therefore the KB) holding the new head.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import logging
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import time
|
|
from collections.abc import Iterator
|
|
from datetime import datetime
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
from sqlalchemy import text
|
|
from sqlalchemy.orm import Session
|
|
|
|
from app.api import sync as sync_api
|
|
from app.config import Settings
|
|
from app.db import SessionLocal
|
|
from app.main import app as fastapi_app
|
|
from app.models import GitSource
|
|
from app.rag import git_sources as git_sources_resolver
|
|
from app.rag.importer import ImportSummary
|
|
from scripts import import_docs
|
|
from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates
|
|
from tests.conftest import ADMIN_PASSWORD
|
|
from tests.fakes import FakeEmbedder
|
|
|
|
|
|
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
|
|
"""Fresh settings (no .env file); explicit kwargs beat any env leaks."""
|
|
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
|
|
|
|
|
def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
|
|
return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
|
|
|
|
|
|
def _local_row(
|
|
path: str,
|
|
ignore_paths: list[str] | None = None,
|
|
include_hidden: bool = False,
|
|
) -> GitSource:
|
|
"""A local row as the phase-38 API stores it: the expanded path in
|
|
both ``path`` and the NOT-NULL ``url`` location column (plus the
|
|
phase-89 ignore list and the phase-105 hidden-folders flag, both
|
|
defaulting off — A4)."""
|
|
return GitSource(
|
|
url=path, kind="local", path=path,
|
|
ignore_paths=ignore_paths or [], include_hidden=include_hidden,
|
|
)
|
|
|
|
|
|
@pytest.fixture()
|
|
def clean_documents(db: Session) -> Iterator[None]:
|
|
"""Phase 105: the real-import CLI tests write ``documents``/
|
|
``chunks`` (the canonical KB state) — global, truncated around
|
|
every such test."""
|
|
db.execute(text("TRUNCATE chunks, documents"))
|
|
db.commit()
|
|
yield
|
|
db.execute(text("TRUNCATE chunks, documents"))
|
|
db.commit()
|
|
|
|
|
|
class FakeImportSources:
|
|
"""Records every ``import_sources`` call instead of touching a DB."""
|
|
|
|
def __init__(self) -> None:
|
|
self.calls: list[dict] = []
|
|
|
|
async def __call__(
|
|
self,
|
|
sources: list[Path],
|
|
llm: object,
|
|
*,
|
|
prune: bool = False,
|
|
limit: int | None = None,
|
|
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
|
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
|
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
|
) -> ImportSummary:
|
|
self.calls.append(
|
|
{"sources": list(sources), "prune": prune, "limit": limit,
|
|
"ignore_by_root": ignore_by_root,
|
|
"include_hidden_by_root": include_hidden_by_root,
|
|
"doc_dates_by_root": doc_dates_by_root}
|
|
)
|
|
return ImportSummary(files=1, added=1)
|
|
|
|
|
|
def _fake_clone_factory() -> tuple[list[tuple[str, Path]], object]:
|
|
"""A ``clone_or_pull`` that materialises a checkout with one .md
|
|
file. Phase 135: accepts the row's branch pin (``None`` for every
|
|
row in this file's fixtures — the real-git branch behavior is
|
|
pinned by the phase-135 section below, which uses the REAL
|
|
``clone_or_pull``)."""
|
|
calls: list[tuple[str, Path]] = []
|
|
|
|
def fake_clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path:
|
|
assert branch is None # every row in this file's fixtures pins no branch
|
|
dest = Path(dest)
|
|
dest.mkdir(parents=True, exist_ok=True)
|
|
(dest / "notes.md").write_text(f"# {dest.name}\ncontent for the KB\n", encoding="utf-8")
|
|
calls.append((url, dest))
|
|
return dest
|
|
|
|
return calls, fake_clone_or_pull
|
|
|
|
|
|
def _stub_bump(monkeypatch: pytest.MonkeyPatch) -> list[None]:
|
|
"""Stub the phase-53 version bump (this file keeps its no-real-DB
|
|
style for the counter — the fake import already avoids the KB
|
|
tables). Returns the call record; the canned new version is 1."""
|
|
bumps: list[None] = []
|
|
|
|
def fake_bump(session: object) -> int:
|
|
bumps.append(None)
|
|
return 1
|
|
|
|
monkeypatch.setattr(import_docs, "bump_sources_version", fake_bump)
|
|
return bumps
|
|
|
|
|
|
def _stub_folder_summaries(monkeypatch: pytest.MonkeyPatch) -> list[dict]:
|
|
"""Stub the phase-94 folder-summary regeneration (this file keeps
|
|
its no-real-DB / no-network style — the real generator would read
|
|
the global ``documents`` table and burn ``lite`` calls). Returns
|
|
the call record; the canned stats are the zero dict."""
|
|
calls: list[dict] = []
|
|
|
|
async def fake_generate(
|
|
db: object, llm: object, *, skip: bool = False, only_missing: bool = False
|
|
) -> dict[str, int]:
|
|
calls.append({"skip": skip, "only_missing": only_missing})
|
|
return {"generated": 0, "failed": 0, "pruned": 0}
|
|
|
|
monkeypatch.setattr(import_docs, "generate_folder_summaries", fake_generate)
|
|
return calls
|
|
|
|
|
|
# --- repo_name -------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("url", "name"),
|
|
[
|
|
("https://github.com/user/homelab.git", "homelab"),
|
|
("https://github.com/user/homelab", "homelab"),
|
|
("git@github.com:user/homelab.git", "homelab"),
|
|
("git@github.com:homelab.git", "homelab"),
|
|
("ssh://git@host:2222/group/deployments.git", "deployments"),
|
|
("https://git.reeseapps.com:8443/proj/notes", "notes"),
|
|
],
|
|
)
|
|
def test_repo_name(url: str, name: str) -> None:
|
|
assert import_docs.repo_name(url) == name
|
|
|
|
|
|
def test_repo_name_slug_fallback() -> None:
|
|
# No usable basename (path ends in the .git suffix itself) → slug.
|
|
assert import_docs.repo_name("https://host/.git") == "https-host"
|
|
|
|
|
|
# --- _resolve_sources ------------------------------------------------------
|
|
|
|
|
|
def test_resolve_sources_git_urls_cloned_into_sources_dir(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
# Phase 35: resolution goes through the shared resolver (stubbed —
|
|
# this file keeps its no-real-DB style); the rows are the env list,
|
|
# surfaced as synthetic git rows.
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: (
|
|
[_git_row("https://host/a/homelab.git"), _git_row("git@host:user/deploy.git")],
|
|
"env",
|
|
),
|
|
)
|
|
settings = _settings(
|
|
git_sources="https://host/a/homelab.git, git@host:user/deploy.git ,",
|
|
sources_dir=str(tmp_path / "bor"),
|
|
)
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
|
|
|
|
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
|
|
assert ignore_map == {} # phase 89: no row carries a list → empty map
|
|
# Phase 105: flag-off rows still contribute their root — with False
|
|
# (A4), keyed by the SAME root string the importer sees.
|
|
assert hidden_map == {
|
|
str(tmp_path / "bor" / "homelab"): False,
|
|
str(tmp_path / "bor" / "deploy"): False,
|
|
}
|
|
# Phase 106: git rows are listed with their checkout's date walk —
|
|
# the fake checkouts are not git repos, so the walk fails soft to
|
|
# ``{}`` (the importer would then take the mtime fallback).
|
|
assert date_map == {
|
|
str(tmp_path / "bor" / "homelab"): {},
|
|
str(tmp_path / "bor" / "deploy"): {},
|
|
}
|
|
assert calls == [
|
|
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
|
|
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
|
|
]
|
|
|
|
|
|
def test_resolve_sources_cli_source_wins(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
settings = _settings(git_sources="https://host/a/repo.git")
|
|
manual = tmp_path / "Manual"
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources([manual], settings)
|
|
|
|
assert sources == [manual]
|
|
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
|
|
assert hidden_map == {} # phase 105: manual dirs have no rows → hidden skipped
|
|
# Phase 106: manual dirs have no rows (no clone) → no date map
|
|
# entries (the importer's mtime fallback applies).
|
|
assert date_map == {}
|
|
assert calls == [] # git is never touched when --source is given
|
|
|
|
|
|
def test_resolve_sources_db_rows_win_over_env(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Phase 35: the resolver's ``db`` origin (table has rows) — the
|
|
``BOR_GIT_SOURCES`` list must be ignored; only the DB repo is cloned."""
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_git_row("https://db.example/only.git")], "db"),
|
|
)
|
|
settings = _settings(
|
|
git_sources="https://env.example/ignored.git",
|
|
sources_dir=str(tmp_path / "bor"),
|
|
)
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
|
|
|
|
assert sources == [tmp_path / "bor" / "only"]
|
|
assert ignore_map == {} # phase 89: no row carries a list → empty map
|
|
# Phase 105: the default-flag row contributes its root with False (A4).
|
|
assert hidden_map == {str(tmp_path / "bor" / "only"): False}
|
|
# Phase 106: the git row's (fake, non-repo) checkout fails soft → {}.
|
|
assert date_map == {str(tmp_path / "bor" / "only"): {}}
|
|
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
|
|
|
|
|
def test_resolve_sources_defaults_when_nothing_configured(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
|
|
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, _settings())
|
|
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
|
|
assert ignore_map == {} # phase 89: the legacy fallback has no rows
|
|
assert hidden_map == {} # phase 105: the legacy fallback has no rows
|
|
# Phase 106: the legacy fallback has no rows (no clone) → mtime
|
|
# fallback for every file.
|
|
assert date_map == {}
|
|
|
|
|
|
def test_resolve_sources_rows_branch_builds_ignore_map(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Phase 89: the rows branch returns each row's ignore list keyed by
|
|
the resolved root string — the local row's directory, the git row's
|
|
checkout dir; a row without a list contributes nothing to the map."""
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: (
|
|
[
|
|
_git_row("https://db.example/only.git"),
|
|
_local_row(str(local_dir), ignore_paths=["ignore/"]),
|
|
],
|
|
"db",
|
|
),
|
|
)
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
|
|
|
|
assert sources == [tmp_path / "bor" / "only", local_dir]
|
|
# Keyed by the SAME string the importer sees (the root, not the name).
|
|
assert ignore_map == {str(local_dir): ["ignore/"]}
|
|
# Phase 105: both rows are flag-off → per-root False entries (A4).
|
|
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
|
|
# Phase 106: ONLY the git row is listed (local rows take the mtime
|
|
# fallback); the fake checkout's date walk fails soft to ``{}``.
|
|
assert date_map == {str(tmp_path / "bor" / "only"): {}}
|
|
|
|
|
|
def test_resolve_sources_two_rows_sharing_root_string_extend(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Phase 89 collision rule: two rows resolving to the SAME root
|
|
string (the sibling/repo-name edge — ``…/shared`` and
|
|
``…/shared.git``) get the UNION of their lists (extend, not
|
|
replace), in row order."""
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: (
|
|
[
|
|
_git_row("https://a.example/shared", ignore_paths=["a/"]),
|
|
_git_row("https://a.example/shared.git", ignore_paths=["b"]),
|
|
],
|
|
"db",
|
|
),
|
|
)
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
|
|
|
|
shared = str(tmp_path / "bor" / "shared")
|
|
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
|
|
assert ignore_map == {shared: ["a/", "b"]} # union, row order
|
|
# Phase 105 collision: the shared root gets the OR of the flags —
|
|
# both rows off here, so one False entry for the one root string.
|
|
assert hidden_map == {shared: False}
|
|
# Phase 106 collision: both git rows resolve to the SAME root — one
|
|
# date walk for the one root string (last row's walk wins, both
|
|
# fail soft to ``{}`` for the fake checkout).
|
|
assert date_map == {shared: {}}
|
|
|
|
|
|
def test_main_rows_branch_passes_ignore_map_to_import(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
capsys: pytest.CaptureFixture[str],
|
|
) -> None:
|
|
"""Phase 89: a ``kind=local`` row carrying ``ignore_paths`` →
|
|
``main`` passes the per-root map to ``import_sources`` (keyed by
|
|
the directory string, prune flag unchanged)."""
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
|
|
(local_dir / "ignore").mkdir()
|
|
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
|
|
encoding="utf-8")
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_local_row(str(local_dir), ignore_paths=["ignore/"])], "db"),
|
|
)
|
|
fake_import = FakeImportSources()
|
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
|
_stub_bump(monkeypatch)
|
|
_stub_folder_summaries(monkeypatch)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 0
|
|
call = fake_import.calls[0]
|
|
assert call["sources"] == [local_dir]
|
|
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
|
|
# Phase 105: the default-flag row passes the per-root map too — a
|
|
# False entry, not an absent key (the importer reads it per root).
|
|
assert call["include_hidden_by_root"] == {str(local_dir): False}
|
|
# Phase 106: the local-only resolution contributes no date map
|
|
# (no clone — the importer's mtime fallback applies).
|
|
assert call["doc_dates_by_root"] == {}
|
|
assert call["prune"] is False # the CLI's no-prune default is unchanged
|
|
|
|
|
|
# --- phase 105: per-row hidden-folders flag ---------------------------------
|
|
|
|
|
|
def test_main_rows_branch_passes_include_hidden_map_to_import(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
capsys: pytest.CaptureFixture[str],
|
|
) -> None:
|
|
"""Phase 105: a ``kind=local`` row with ``include_hidden=True`` →
|
|
``main`` passes the per-root flag map to ``import_sources`` (keyed
|
|
by the directory string — the map is built, not lost)."""
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
|
(local_dir / ".hidden").mkdir()
|
|
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
|
|
)
|
|
fake_import = FakeImportSources()
|
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
|
_stub_bump(monkeypatch)
|
|
_stub_folder_summaries(monkeypatch)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 0
|
|
call = fake_import.calls[0]
|
|
assert call["sources"] == [local_dir]
|
|
assert call["include_hidden_by_root"] == {str(local_dir): True}
|
|
assert call["ignore_by_root"] == {} # the row carries no ignore list
|
|
assert call["prune"] is False # the CLI's no-prune default is unchanged
|
|
|
|
|
|
def _stub_overview(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Stub the phase-31 overview regeneration (the CLI's real-import
|
|
tests: the deterministic ``FakeEmbedder`` must not burn its canned
|
|
``chat`` on the KB outline — the import is what is under test)."""
|
|
|
|
async def fake_overview(llm: object, session: object = None) -> bool:
|
|
return False
|
|
|
|
monkeypatch.setattr(import_docs, "regenerate_overview", fake_overview)
|
|
|
|
|
|
def _kb_docs(db: Session) -> set[tuple[str, str]]:
|
|
"""The (source, path) pairs of the ``documents`` table."""
|
|
rows = db.execute(text("SELECT source, path FROM documents")).all()
|
|
return {(row.source, row.path) for row in rows}
|
|
|
|
|
|
def test_main_local_row_include_hidden_true_indexes_hidden_file(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
db: Session,
|
|
capsys: pytest.CaptureFixture[str],
|
|
clean_documents: None,
|
|
) -> None:
|
|
"""Phase 105 (A1): the CLI's DB-row path with the flag ON — the
|
|
file inside the hidden folder is indexed, embedded, and counted
|
|
like any visible file (real import over a host temp dir, the
|
|
deterministic ``FakeEmbedder``; the regression this phase most
|
|
plausibly breaks — the CLI's map built but lost — would leave it
|
|
out)."""
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
|
(local_dir / ".hidden").mkdir()
|
|
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
|
|
)
|
|
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
|
|
_stub_overview(monkeypatch)
|
|
_stub_bump(monkeypatch)
|
|
_stub_folder_summaries(monkeypatch)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 0
|
|
# The hidden file has a documents row next to the visible one.
|
|
assert _kb_docs(db) == {
|
|
("LocalDocs", "visible.md"),
|
|
("LocalDocs", ".hidden/note.md"),
|
|
}
|
|
out = capsys.readouterr().out
|
|
assert "added=2" in out # both files were imported
|
|
|
|
|
|
def test_main_local_row_include_hidden_false_skips_hidden_file(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
db: Session,
|
|
capsys: pytest.CaptureFixture[str],
|
|
clean_documents: None,
|
|
) -> None:
|
|
"""Phase 105 (A4): the same fixture with the default flag (off) —
|
|
the hidden-folder file never enters the KB (the byte-identical
|
|
pre-phase-105 walk): no documents row, and the summary line counts
|
|
only the visible file."""
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
|
(local_dir / ".hidden").mkdir()
|
|
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_local_row(str(local_dir))], "db"), # flag defaults to False
|
|
)
|
|
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
|
|
_stub_overview(monkeypatch)
|
|
_stub_bump(monkeypatch)
|
|
_stub_folder_summaries(monkeypatch)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 0
|
|
assert _kb_docs(db) == {("LocalDocs", "visible.md")}
|
|
out = capsys.readouterr().out
|
|
assert "added=1" in out # the hidden file was never walked
|
|
|
|
|
|
# --- main() ----------------------------------------------------------------
|
|
|
|
|
|
def test_main_git_sources_clone_then_import(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
capsys: pytest.CaptureFixture[str],
|
|
) -> None:
|
|
settings = _settings(
|
|
git_sources="https://host/a/homelab.git,https://host/a/deploy.git",
|
|
sources_dir=str(tmp_path / "bor"),
|
|
)
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: (
|
|
[_git_row("https://host/a/homelab.git"), _git_row("https://host/a/deploy.git")],
|
|
"env",
|
|
),
|
|
)
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
fake_import = FakeImportSources()
|
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
|
bumps = _stub_bump(monkeypatch)
|
|
_stub_folder_summaries(monkeypatch)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 0
|
|
assert [(url, dest) for url, dest in calls] == [
|
|
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
|
|
("https://host/a/deploy.git", tmp_path / "bor" / "deploy"),
|
|
]
|
|
assert len(fake_import.calls) == 1
|
|
# The cloned checkouts are exactly what gets imported.
|
|
assert fake_import.calls[0]["sources"] == [
|
|
tmp_path / "bor" / "homelab",
|
|
tmp_path / "bor" / "deploy",
|
|
]
|
|
# Phase 106: the git rows' (fake, non-repo) checkouts fail soft to
|
|
# ``{}`` — but the roots ARE listed (the CLI feeds the map).
|
|
assert fake_import.calls[0]["doc_dates_by_root"] == {
|
|
str(tmp_path / "bor" / "homelab"): {},
|
|
str(tmp_path / "bor" / "deploy"): {},
|
|
}
|
|
for dest in (tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"):
|
|
assert (dest / "notes.md").is_file()
|
|
# The final summary print reflects the import (added > 0).
|
|
out = capsys.readouterr().out
|
|
assert "added=1" in out
|
|
# Phase 53: a KB-changing run bumps the sources version exactly
|
|
# once and reports it (stubbed — this file keeps its no-real-DB
|
|
# style for the counter, like the fake import above).
|
|
assert len(bumps) == 1
|
|
assert "sources_version=1" in out
|
|
|
|
|
|
def test_main_cli_source_still_imports_manual_dir(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
capsys: pytest.CaptureFixture[str],
|
|
) -> None:
|
|
manual = tmp_path / "manual"
|
|
manual.mkdir()
|
|
(manual / "a.md").write_text("# A\nhi\n", encoding="utf-8")
|
|
# BOR_GIT_SOURCES is set but must be ignored — --source always wins.
|
|
settings = _settings(git_sources="https://host/a/repo.git")
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
fake_import = FakeImportSources()
|
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
|
bumps = _stub_bump(monkeypatch)
|
|
_stub_folder_summaries(monkeypatch)
|
|
|
|
rc = import_docs.main(["--source", str(manual)])
|
|
|
|
assert rc == 0
|
|
assert calls == []
|
|
assert fake_import.calls[0]["sources"] == [manual]
|
|
# Phase 106: manual --source has no rows (no clone) → no date map
|
|
# (the importer's mtime fallback applies).
|
|
assert fake_import.calls[0]["doc_dates_by_root"] == {}
|
|
assert fake_import.calls[0]["prune"] is False
|
|
# Phase 53: a manual --source run that changes the KB bumps exactly
|
|
# once (the CLI is the other canonical sync path).
|
|
assert len(bumps) == 1
|
|
assert "sources_version=1" in capsys.readouterr().out
|
|
|
|
|
|
def test_resolve_sources_mixed_git_and_local(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Phase 38: DB rows of both kinds — the git row is cloned into
|
|
``BOR_SOURCES_DIR``, the local row is its existing directory itself
|
|
(no clone), in row order; the env list is ignored."""
|
|
calls, fake = _fake_clone_factory()
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "a.md").write_text("# A\nlocal fixture\n", encoding="utf-8")
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: (
|
|
[_git_row("https://db.example/only.git"), _local_row(str(local_dir))],
|
|
"db",
|
|
),
|
|
)
|
|
settings = _settings(
|
|
git_sources="https://env.example/ignored.git",
|
|
sources_dir=str(tmp_path / "bor"),
|
|
)
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
|
|
|
|
assert sources == [tmp_path / "bor" / "only", local_dir]
|
|
assert ignore_map == {} # phase 89: neither row carries a list
|
|
# Phase 105: both rows default-flag → per-root False entries (A4).
|
|
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
|
|
# Phase 106: only the git row is listed (local takes the mtime
|
|
# fallback); the fake checkout's date walk fails soft to ``{}``.
|
|
assert date_map == {str(tmp_path / "bor" / "only"): {}}
|
|
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
|
|
|
|
|
def test_resolve_sources_missing_local_dir_aborts(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Phase 38: a local row whose directory is gone at run time →
|
|
``GitSyncError`` naming the path, before any import (the same
|
|
pre-import fail-loud as a failing git clone)."""
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", _fake_clone_factory()[1])
|
|
missing = tmp_path / "Gone"
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_local_row(str(missing))], "db"),
|
|
)
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
|
|
with pytest.raises(GitSyncError, match=f"local source missing: {re.escape(str(missing))}"):
|
|
import_docs._resolve_sources(None, settings)
|
|
|
|
|
|
def test_main_git_failure_aborts_before_import(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
|
) -> None:
|
|
settings = _settings(
|
|
git_sources="https://host/a/bad.git",
|
|
sources_dir=str(tmp_path / "bor"),
|
|
)
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_git_row("https://host/a/bad.git")], "env"),
|
|
)
|
|
|
|
def failing_clone(url: str, dest: Path | str, branch: str | None = None) -> Path:
|
|
raise GitSyncError(
|
|
f"git clone {url} failed (exit 128): "
|
|
"fatal: repository not found"
|
|
)
|
|
|
|
monkeypatch.setattr(import_docs, "clone_or_pull", failing_clone)
|
|
fake_import = FakeImportSources()
|
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 1
|
|
err = capsys.readouterr().err
|
|
assert "import_docs: source sync failed" in err
|
|
assert "bad.git" in err # the failing repo is named
|
|
assert fake_import.calls == [] # no partial import
|
|
assert not (tmp_path / "bor").exists()
|
|
|
|
|
|
def test_main_missing_local_dir_aborts_before_import(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
|
) -> None:
|
|
"""Phase 38: a DB local row whose directory is missing → exit code
|
|
1, ``local source missing: <path>`` on stderr, zero import attempts
|
|
(no ``--source`` given, so the DB row is what should have been
|
|
imported)."""
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
|
missing = tmp_path / "Gone"
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_local_row(str(missing))], "db"),
|
|
)
|
|
fake_import = FakeImportSources()
|
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
|
|
|
rc = import_docs.main([])
|
|
|
|
assert rc == 1
|
|
err = capsys.readouterr().err
|
|
assert "import_docs: source sync failed" in err
|
|
assert f"local source missing: {missing}" in err # the path is named
|
|
assert fake_import.calls == [] # no partial import
|
|
|
|
|
|
# --- phase 135: the branch pin (real git, two-branch file:// origin) -------
|
|
|
|
|
|
def _git_available() -> bool:
|
|
try:
|
|
proc = subprocess.run(["git", "--version"], capture_output=True, check=False)
|
|
return proc.returncode == 0
|
|
except (FileNotFoundError, OSError):
|
|
return False
|
|
|
|
|
|
#: Real ``git`` in the test environment — skipped cleanly without it
|
|
#: (the ``test_git_file_dates.py`` house pattern).
|
|
GIT = _git_available()
|
|
|
|
|
|
@pytest.fixture()
|
|
def two_branch_repo(tmp_path: Path) -> Path:
|
|
"""A real-git scratch origin (the phase-135 recipe): ``main`` holds
|
|
``a.md``, ``release`` holds ``b.md`` — different content per branch
|
|
(a branch checkout is the branch's tree, not main's)."""
|
|
repo = tmp_path / "origin"
|
|
repo.mkdir()
|
|
_git(repo, "init", "-q", "-b", "main")
|
|
_git(repo, "config", "commit.gpgsign", "false")
|
|
(repo / "a.md").write_text("# A\nmain branch content\n", encoding="utf-8")
|
|
_git(repo, "add", "a.md")
|
|
_git(repo, "commit", "-q", "-m", "main: add a.md")
|
|
_git(repo, "checkout", "-q", "-b", "release")
|
|
(repo / "a.md").unlink()
|
|
(repo / "b.md").write_text("# B\nrelease branch content v1\n", encoding="utf-8")
|
|
_git(repo, "add", "-A")
|
|
_git(repo, "commit", "-q", "-m", "release: add b.md")
|
|
_git(repo, "checkout", "-q", "main")
|
|
return repo
|
|
|
|
|
|
def _git(cwd: Path, *argv: str, date: str | None = None) -> None:
|
|
"""Run one git command for the test harness (fixture setup); a
|
|
non-zero exit fails the fixture, not the test under test.
|
|
|
|
``date`` pins the commit date (``GIT_COMMITTER_DATE`` =
|
|
``GIT_AUTHOR_DATE``) — deterministic per-file dates for the
|
|
phase-136 ``file_commit_dates`` pin (scratch commits would
|
|
otherwise share the wall clock's second)."""
|
|
env = os.environ.copy()
|
|
env["GIT_AUTHOR_NAME"] = "T"
|
|
env["GIT_AUTHOR_EMAIL"] = "t@example.com"
|
|
env["GIT_COMMITTER_NAME"] = "T"
|
|
env["GIT_COMMITTER_EMAIL"] = "t@example.com"
|
|
if date is not None:
|
|
env["GIT_AUTHOR_DATE"] = date
|
|
env["GIT_COMMITTER_DATE"] = date
|
|
proc = subprocess.run(
|
|
["git", *argv], cwd=cwd, env=env, capture_output=True, text=True, check=False
|
|
)
|
|
assert proc.returncode == 0, f"git {' '.join(argv)} failed: {proc.stderr}"
|
|
|
|
|
|
def _git_out(cwd: Path, *argv: str) -> str:
|
|
"""One harness git read (a non-zero exit fails the test)."""
|
|
proc = subprocess.run(
|
|
["git", *argv], cwd=cwd, capture_output=True, text=True, check=False
|
|
)
|
|
assert proc.returncode == 0, f"git {' '.join(argv)} failed: {proc.stderr}"
|
|
return proc.stdout
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_fresh_branch_checkout_is_on_that_branch(
|
|
tmp_path: Path, two_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 135: a FRESH ``clone_or_pull(url, dest, 'release')``
|
|
(real git, ``file://``) checks out the pinned branch — HEAD is
|
|
``release``, file B present, file A (main's tree) absent."""
|
|
url = f"file://{two_branch_repo}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
|
|
result = clone_or_pull(url, dest, "release")
|
|
|
|
assert result == dest
|
|
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "release"
|
|
assert (dest / "b.md").is_file()
|
|
assert (dest / "b.md").read_text(encoding="utf-8") == "# B\nrelease branch content v1\n"
|
|
assert not (dest / "a.md").exists()
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_existing_branch_checkout_ff_merges_new_commits(
|
|
tmp_path: Path, two_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 135: a SECOND ``clone_or_pull`` of the same pinned branch
|
|
after the origin gains a commit on it → the guarded checkout is
|
|
ff-merged onto the new tip (file B updated, no error, still on
|
|
``release``)."""
|
|
url = f"file://{two_branch_repo}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
|
|
clone_or_pull(url, dest, "release")
|
|
# the origin gains a commit on release (v2 of file B)
|
|
_git(two_branch_repo, "checkout", "-q", "release")
|
|
(two_branch_repo / "b.md").write_text("# B\nrelease branch content v2\n", encoding="utf-8")
|
|
_git(two_branch_repo, "add", "b.md")
|
|
_git(two_branch_repo, "commit", "-q", "-m", "release: bump b.md")
|
|
_git(two_branch_repo, "checkout", "-q", "main")
|
|
|
|
clone_or_pull(url, dest, "release")
|
|
|
|
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "release"
|
|
assert (dest / "b.md").read_text(encoding="utf-8") == "# B\nrelease branch content v2\n"
|
|
assert not (dest / "a.md").exists()
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_branch_mismatch_fails_loud_checkout_untouched(
|
|
tmp_path: Path, two_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 135 (ASSUMPTION B5): a checkout on the DEFAULT branch
|
|
(fresh clone with no branch → ``main``) + a source row pinning
|
|
``release`` → the remove-and-re-add ``GitSyncError`` — and the
|
|
checkout is UNTOUCHED (still on main, still main's tree: no silent
|
|
branch switch)."""
|
|
url = f"file://{two_branch_repo}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
|
|
clone_or_pull(url, dest) # no branch → the origin's default (main)
|
|
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "main"
|
|
assert (dest / "a.md").is_file()
|
|
|
|
with pytest.raises(GitSyncError) as excinfo:
|
|
clone_or_pull(url, dest, "release")
|
|
|
|
msg = str(excinfo.value)
|
|
assert "remove and re-add the source to switch branches" in msg
|
|
assert "'main'" in msg and "'release'" in msg
|
|
assert url not in msg # the guard message never carries the URL
|
|
# the checkout is untouched: still on main, still main's tree
|
|
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "main"
|
|
assert (dest / "a.md").is_file()
|
|
assert not (dest / "b.md").exists()
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_resolve_sources_git_row_with_branch_checks_out_that_branch(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, two_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 135, the CLI seam: a ``git_sources`` row carrying
|
|
``branch='release'`` → ``_resolve_sources`` passes ``row.branch`` to
|
|
the REAL ``clone_or_pull`` (no fake — the branch is honored
|
|
end to end): the resolved root is the RELEASE tree (file B imported,
|
|
file A absent), and the date walk over the real checkout lists ONLY
|
|
the branch's tree (head-of-branch semantics, ``file_commit_dates``
|
|
untouched)."""
|
|
url = f"file://{two_branch_repo}"
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([GitSource(url=url, kind="git", branch="release")], "db"),
|
|
)
|
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
|
|
|
sources, ignore_map, hidden_map, date_map = import_docs._resolve_sources(None, settings)
|
|
|
|
root = tmp_path / "bor" / "origin"
|
|
assert sources == [root]
|
|
assert ignore_map == {}
|
|
assert hidden_map == {str(root): False}
|
|
assert (root / "b.md").is_file() # the branch's file is what gets imported
|
|
assert not (root / "a.md").exists()
|
|
# the (real, untouched) date walk over the branch checkout yields
|
|
# TRUE dates for the branch's files (not the fail-soft ``{}`` — the
|
|
# checkout is a real full-history repo). The walk reads the
|
|
# reachable LOG (``git log --name-only``), so it also lists files
|
|
# from shared ancestor commits (``a.md`` — release was cut after
|
|
# main's first commit); the importer only consults the map for the
|
|
# files it actually finds in the working tree, so that is harmless.
|
|
dates = date_map[str(root)]
|
|
assert "b.md" in dates
|
|
assert isinstance(dates["b.md"], datetime)
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_resolve_sources_row_without_branch_clones_default_branch(
|
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, two_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 135, the byte-identical leg: a row WITHOUT a branch (the
|
|
env-fallback synthetic rows' shape — ``GitSource(url=…, kind="git")``
|
|
carries the model default ``branch=None``) → the real
|
|
``clone_or_pull`` takes the pre-phase path: the origin's DEFAULT
|
|
branch (``main``) is checked out — file A present, file B absent."""
|
|
url = f"file://{two_branch_repo}"
|
|
monkeypatch.setattr(
|
|
import_docs,
|
|
"effective_sources",
|
|
lambda db: ([_git_row(url)], "env"), # branch defaults to None
|
|
)
|
|
settings = _settings(git_sources=url, sources_dir=str(tmp_path / "bor"))
|
|
|
|
sources, *_ = import_docs._resolve_sources(None, settings)
|
|
|
|
root = tmp_path / "bor" / "origin"
|
|
assert sources == [root]
|
|
assert (root / "a.md").is_file() # main's file — the default branch
|
|
assert not (root / "b.md").exists()
|
|
|
|
|
|
# --- phase 136: the ff-or-reset (real git, the force-push repro) -----------
|
|
|
|
|
|
@pytest.fixture()
|
|
def single_branch_repo(tmp_path: Path) -> Path:
|
|
"""A real-git scratch origin on ``main`` with one file (``a.md``)
|
|
(the phase-136 base shape — the owner's single-branch source;
|
|
the root commit's date is pinned for the deterministic
|
|
``file_commit_dates`` pin below)."""
|
|
repo = tmp_path / "origin"
|
|
repo.mkdir()
|
|
_git(repo, "init", "-q", "-b", "main")
|
|
_git(repo, "config", "commit.gpgsign", "false")
|
|
(repo / "a.md").write_text("# A\noriginal content\n", encoding="utf-8")
|
|
_git(repo, "add", "a.md")
|
|
_git(repo, "commit", "-q", "-m", "root: add a.md", date="2026-01-05T10:00:00+00:00")
|
|
return repo
|
|
|
|
|
|
def _force_push_new_root(
|
|
origin: Path,
|
|
files: dict[str, str | None],
|
|
message: str,
|
|
*,
|
|
branch: str = "main",
|
|
date: str | None = None,
|
|
) -> None:
|
|
"""Rewrite the origin's ``branch`` to a NEW ROOT commit (the
|
|
phase-136 force push): ``git checkout --orphan`` + commit, so the
|
|
branch now points at a root with NO shared ancestry with the old
|
|
history — the strongest divergence, exactly what ``--ff-only``
|
|
refuses. ``files`` maps name → new content (the orphan commit
|
|
holds exactly that set — the old files are dropped)."""
|
|
_git(origin, "checkout", "--orphan", "rewritten")
|
|
for p in sorted(origin.iterdir()):
|
|
if p.is_file() and p.name != ".git":
|
|
p.unlink()
|
|
for name, content in files.items():
|
|
if content is not None:
|
|
(origin / name).write_text(content, encoding="utf-8")
|
|
_git(origin, "add", "-A")
|
|
_git(origin, "commit", "-q", "-m", message, date=date)
|
|
_git(origin, "branch", "-D", branch)
|
|
_git(origin, "branch", "-m", "rewritten", branch)
|
|
_git(origin, "checkout", "-q", "main")
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_normal_ff_updates_without_reset(
|
|
tmp_path: Path, single_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 136: a normal second commit (a true fast-forward) → the
|
|
unified sequence ff-merges: the checkout's HEAD is the origin's
|
|
head, the second commit's message is the HEAD subject, NO merge
|
|
commit (a pure ff), and the working tree holds the new content —
|
|
the reset tail never runs on a clean ff."""
|
|
origin = single_branch_repo
|
|
url = f"file://{origin}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
|
|
clone_or_pull(url, dest)
|
|
(origin / "b.md").write_text("# B\nsecond commit\n", encoding="utf-8")
|
|
_git(origin, "add", "b.md")
|
|
_git(origin, "commit", "-q", "-m", "second: add b.md", date="2026-02-01T09:00:00+00:00")
|
|
|
|
clone_or_pull(url, dest)
|
|
|
|
dest_head = _git_out(dest, "rev-parse", "HEAD").strip()
|
|
origin_head = _git_out(origin, "rev-parse", "main").strip()
|
|
assert dest_head == origin_head
|
|
subjects = _git_out(dest, "log", "--format=%s").splitlines()
|
|
assert subjects[0] == "second: add b.md" # the second commit's message is the HEAD
|
|
assert subjects == ["second: add b.md", "root: add a.md"] # pure ff: no merge commit
|
|
assert (dest / "b.md").read_text(encoding="utf-8") == "# B\nsecond commit\n"
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_force_push_resets_to_the_new_head(
|
|
tmp_path: Path, single_branch_repo: Path, caplog: pytest.LogCaptureFixture
|
|
) -> None:
|
|
"""Phase 136 (the owner's repro, TODO L5): after the first clone,
|
|
the origin's history is REWRITTEN to a NEW ROOT commit (same
|
|
file, new content — the branch now diverges; pre-phase, the next
|
|
sync died on the non-ff ``git pull --ff-only`` → the whole sync
|
|
landed ``failed``) → ``clone_or_pull`` NO LONGER raises: the
|
|
checkout is reset to the origin's NEW head (the checkout's HEAD ==
|
|
the origin's head), the working tree holds the NEW content, and
|
|
ONE WARNING line (PLAN §9) is the only side effect — no
|
|
user-facing error (the log carries the checkout path, never the
|
|
URL)."""
|
|
origin = single_branch_repo
|
|
url = f"file://{origin}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
new_content = "# A\nrewritten content\n"
|
|
|
|
clone_or_pull(url, dest)
|
|
_force_push_new_root(
|
|
origin, {"a.md": new_content}, "force: rewritten root",
|
|
date="2026-03-10T12:00:00+00:00",
|
|
)
|
|
|
|
with caplog.at_level(logging.WARNING, logger="scripts.git_sync"):
|
|
result = clone_or_pull(url, dest) # NO exception (the pre-phase break)
|
|
|
|
assert result == dest
|
|
dest_head = _git_out(dest, "rev-parse", "HEAD").strip()
|
|
origin_head = _git_out(origin, "rev-parse", "main").strip()
|
|
assert dest_head == origin_head
|
|
assert (dest / "a.md").read_text(encoding="utf-8") == new_content
|
|
# ONE WARNING line: the checkout path + the tracked ref, never the URL.
|
|
warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
|
|
assert len(warnings) == 1
|
|
msg = warnings[0].getMessage()
|
|
assert "git sync: divergent remote head on " in msg
|
|
assert str(dest) in msg
|
|
assert "to origin/main" in msg
|
|
assert url not in msg # credential-safety: the log never carries the URL
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_unrelated_history_force_push_resets(
|
|
tmp_path: Path, single_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 136: the rewritten origin has a DIFFERENT root commit
|
|
with a fully UNRELATED tree (a different file, NO shared ancestry
|
|
— the ff merge of an unrelated history is exactly what
|
|
``--ff-only`` refuses) → the SAME reset path handles it: the
|
|
checkout's HEAD is the origin's new head and the working tree IS
|
|
the new tree (the old file GONE — the reset is a hard one, not a
|
|
merge)."""
|
|
origin = single_branch_repo
|
|
url = f"file://{origin}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
|
|
clone_or_pull(url, dest)
|
|
_force_push_new_root(
|
|
origin, {"z.md": "# Z\nunrelated root\n"}, "force: unrelated root",
|
|
date="2026-03-11T12:00:00+00:00",
|
|
)
|
|
|
|
result = clone_or_pull(url, dest) # NO exception — the same reset path
|
|
|
|
assert result == dest
|
|
dest_head = _git_out(dest, "rev-parse", "HEAD").strip()
|
|
origin_head = _git_out(origin, "rev-parse", "main").strip()
|
|
assert dest_head == origin_head
|
|
assert (dest / "z.md").read_text(encoding="utf-8") == "# Z\nunrelated root\n"
|
|
assert not (dest / "a.md").exists() # the hard reset dropped the old tree's file
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_clone_or_pull_branched_force_push_resets_to_the_branch_head(
|
|
tmp_path: Path, two_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 136 + phase 135: a two-branch origin (the phase-135
|
|
shape), a row-shaped call ``clone_or_pull(url, dest,
|
|
"release")`` after a force push of ONLY ``release`` → the reset
|
|
lands ``origin/release``'s head (the owner's verbatim "reset
|
|
origin/<branch>") and ``main``'s content is not involved (file A
|
|
is still absent — the checkout holds the branch's tree only)."""
|
|
origin = two_branch_repo
|
|
url = f"file://{origin}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
forced = "# B\nrelease branch content FORCED\n"
|
|
|
|
clone_or_pull(url, dest, "release")
|
|
_force_push_new_root(
|
|
origin, {"b.md": forced}, "release: force-pushed root",
|
|
branch="release", date="2026-03-15T08:00:00+00:00",
|
|
)
|
|
|
|
clone_or_pull(url, dest, "release") # NO exception — the reset landed
|
|
|
|
assert _git_out(dest, "rev-parse", "--abbrev-ref", "HEAD").strip() == "release"
|
|
dest_head = _git_out(dest, "rev-parse", "HEAD").strip()
|
|
release_head = _git_out(origin, "rev-parse", "release").strip()
|
|
assert dest_head == release_head
|
|
assert (dest / "b.md").read_text(encoding="utf-8") == forced
|
|
assert not (dest / "a.md").exists() # main's content is not involved
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_file_commit_dates_after_reset_reflects_the_new_head(
|
|
tmp_path: Path, single_branch_repo: Path
|
|
) -> None:
|
|
"""Phase 136: ``file_commit_dates`` over the post-reset checkout
|
|
yields the NEW head's per-file dates — the new root commit's date
|
|
is what the walk returns for its files (the head-of-branch dates
|
|
the owner asked for: the KB is rebuilt from the reset checkout on
|
|
the same run, so its document dates are the NEW history's, not
|
|
the old one's)."""
|
|
origin = single_branch_repo
|
|
url = f"file://{origin}"
|
|
dest = tmp_path / "bor" / "origin"
|
|
|
|
clone_or_pull(url, dest)
|
|
old_dates = file_commit_dates(dest)
|
|
assert old_dates["a.md"] == datetime.fromisoformat("2026-01-05T10:00:00+00:00")
|
|
|
|
_force_push_new_root(
|
|
origin, {"a.md": "# A\nrewritten content\n"}, "force: rewritten root",
|
|
date="2026-03-10T12:00:00+00:00",
|
|
)
|
|
clone_or_pull(url, dest)
|
|
|
|
new_dates = file_commit_dates(dest)
|
|
assert set(new_dates) == {"a.md"} # the new tree's files only
|
|
assert new_dates["a.md"] == datetime.fromisoformat("2026-03-10T12:00:00+00:00")
|
|
assert new_dates["a.md"] != old_dates["a.md"] # the NEW head dates the file
|
|
|
|
|
|
# --- phase 136: the sync pipeline end to end (the API seam) ---------------
|
|
|
|
|
|
def _reset_sources_meta() -> None:
|
|
"""Pin the single-row ``sources_meta`` counter to the
|
|
migration-0010 seed (0) — the runner bumps it through its own
|
|
short-lived ``SessionLocal`` (the ``test_sync_api`` pattern)."""
|
|
session = SessionLocal()
|
|
try:
|
|
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
|
session.commit()
|
|
finally:
|
|
session.close()
|
|
|
|
|
|
@pytest.fixture()
|
|
def fresh_sync_state() -> Iterator[None]:
|
|
"""Phase 136 (the API-seam pin): the sync runner's module-level
|
|
status object + task + cancel event are process-global — reset
|
|
them around the test (the ``test_sync_api`` autouse pattern, the
|
|
full suite runs both files in one process)."""
|
|
sync_api._status = sync_api.SyncStatus()
|
|
sync_api._task = None
|
|
sync_api._cancel_event = asyncio.Event()
|
|
yield
|
|
sync_api._status = sync_api.SyncStatus()
|
|
sync_api._task = None
|
|
sync_api._cancel_event = asyncio.Event()
|
|
|
|
|
|
@pytest.fixture()
|
|
def clean_sync_seam(db: Session) -> Iterator[None]:
|
|
"""Phase 136 (the API-seam pin): the real sync runner writes
|
|
global tables — ``git_sources`` (seeded), ``documents``/
|
|
``chunks`` (the real import), and ``folder_summaries`` (the real
|
|
generator) — truncated around the test; the ``sources_meta``
|
|
counter is pinned to 0 before and after. Skips like the ``db``
|
|
fixture when Postgres is down."""
|
|
db.execute(text("TRUNCATE git_sources, chunks, documents, folder_summaries"))
|
|
db.commit()
|
|
_reset_sources_meta()
|
|
yield
|
|
db.execute(text("TRUNCATE git_sources, chunks, documents, folder_summaries"))
|
|
db.commit()
|
|
_reset_sources_meta()
|
|
|
|
|
|
@pytest.fixture()
|
|
def sync_client() -> Iterator[TestClient]:
|
|
"""Context-managed TestClient — one app event loop across
|
|
requests (the background sync task must survive between the POST
|
|
and the polls — the ``test_sync_api`` pattern; a TestClient
|
|
without the context manager would cancel the task on request
|
|
exit)."""
|
|
with TestClient(fastapi_app) as client:
|
|
yield client
|
|
|
|
|
|
def _poll_sync(client: TestClient, want: str, timeout: float = 10.0) -> dict:
|
|
"""Poll ``GET /api/sync/status`` until ``state == want``
|
|
(terminal). Any state other than ``running`` before the deadline
|
|
fails loudly — an unexpected ``failed`` must never be masked by
|
|
the wait (the pre-phase force push landed exactly that)."""
|
|
deadline = time.monotonic() + timeout
|
|
while time.monotonic() < deadline:
|
|
body = client.get("/api/sync/status").json()
|
|
if body["state"] == want:
|
|
return body
|
|
assert body["state"] == "running", (
|
|
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
|
|
)
|
|
time.sleep(0.05)
|
|
raise AssertionError(f"sync did not reach {want!r} within {timeout}s")
|
|
|
|
|
|
@pytest.mark.skipif(not GIT, reason="git CLI not available")
|
|
def test_sync_pipeline_survives_a_force_push(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
single_branch_repo: Path,
|
|
fresh_sync_state: None,
|
|
clean_sync_seam: None,
|
|
) -> None:
|
|
"""Phase 136, the SYNC PIPELINE END-TO-END (the seam
|
|
``tests/integration/test_sync_api.py`` already exercises — the
|
|
pre-phase force push landed the button sync in ``failed``; this
|
|
test is the API-seam regression pin that it now lands
|
|
``success``): a real-git ``file://`` origin + a seeded git row;
|
|
sync #1 clones and imports the original content; the origin's
|
|
history is REWRITTEN to a new root commit (the owner's exact
|
|
repro); sync #2 → ``state == "success"`` (NOT ``failed``), the
|
|
rewritten file is re-imported (``updated=1``), and the checkout
|
|
(and therefore the KB) holds the NEW head. The REAL
|
|
``clone_or_pull`` runs (no fake — the reset is what is under
|
|
test) over the real import + real overview/folder-summary steps
|
|
(deterministic in-process ``FakeEmbedder``, no network)."""
|
|
origin = single_branch_repo
|
|
url = f"file://{origin}"
|
|
db.add(GitSource(url=url, kind="git"))
|
|
db.commit()
|
|
# The resolver's env fallback pinned empty (the DB row wins — the
|
|
# dev ``.env`` never leaks in; the ``test_sync_api`` pattern).
|
|
monkeypatch.setattr(
|
|
git_sources_resolver,
|
|
"get_settings",
|
|
lambda: Settings(_env_file=None, git_sources=""), # pyright: ignore[reportCallIssue]
|
|
)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: Settings(_env_file=None, sources_dir=str(tmp_path / "bor")), # pyright: ignore[reportCallIssue]
|
|
)
|
|
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder()) # probe passes against it
|
|
|
|
r = sync_client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
|
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
|
|
|
# Sync #1: the fresh clone imports the original content.
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body1 = _poll_sync(sync_client, "success")
|
|
assert body1["detail"]["added"] == 1
|
|
|
|
# The owner's repro: force push the origin (a NEW ROOT commit).
|
|
new_content = "# A\nrewritten content\n"
|
|
_force_push_new_root(origin, {"a.md": new_content}, "force: rewritten root")
|
|
|
|
# Sync #2: pre-phase this landed ``failed`` (the non-ff
|
|
# ``git pull --ff-only`` died) — the reset is now self-healing
|
|
# and the run SUCCEEDS.
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body2 = _poll_sync(sync_client, "success") # NOT ``failed`` — the regression pin
|
|
assert body2["error"] is None
|
|
assert body2["detail"]["updated"] == 1 # the rewritten file is re-imported
|
|
|
|
# The checkout (and therefore the KB) holds the NEW head.
|
|
root = tmp_path / "bor" / "origin"
|
|
root_head = _git_out(root, "rev-parse", "HEAD").strip()
|
|
origin_head = _git_out(origin, "rev-parse", "main").strip()
|
|
assert root_head == origin_head
|
|
assert (root / "a.md").read_text(encoding="utf-8") == new_content
|