ducoterra
d731169b8b
All completion criteria verified. Everything is green. **Phase 105 final verification pass — all criteria verified** - Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page - Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified) - Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓ - Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation) - Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed - Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings) **Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched). **Deviations:** none — no defects found; no code changes were needed on this pass. **Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
1222 lines
49 KiB
Python
1222 lines
49 KiB
Python
"""Integration: the admin sources-sync API (phase 32, task 01; phase 35,
|
|
task 03 re-points the URL resolution at the shared resolver; phase 38,
|
|
task 03 adds the local kind).
|
|
|
|
Covers the in-process sync runner end to end over HTTP: anonymous 403s
|
|
on both endpoints; admin idle → 202 → ``success`` with the full
|
|
ImportSummary detail; 409 on a double trigger while a run is in flight;
|
|
``GitSyncError`` → ``failed`` with the failing repo named and **zero**
|
|
import attempts; empty on *both* origins (no git rows, no local rows,
|
|
no env URLs) → ``failed`` loudly (``no sources configured
|
|
(git or local)``); an embedding failure → ``failed`` with any
|
|
credentials masked; the import always runs with ``prune=True``; and the
|
|
phase-31 overview trigger is change-gated (no ``lite`` call on an
|
|
unchanged KB).
|
|
|
|
Phase 38 (local kind): local-only, git-only, and mixed syncs over a
|
|
**host temp local dir** (the app server runs on the same host) — the
|
|
mixed run goes through the **real** ``import_sources`` (deterministic
|
|
in-process ``FakeEmbedder``, no network), so the local file verifiably
|
|
lands in the KB via ``GET /api/docs`` and union pruning holds (a file
|
|
deleted out of the local dir is pruned on the next sync while the git
|
|
doc survives); a local directory missing at sync time → ``failed`` with
|
|
``local source missing: <path>`` and **zero** import attempts.
|
|
|
|
Phase 35: the runner resolves the sources through
|
|
:func:`app.rag.git_sources.effective_sources` — the **real** resolver
|
|
against the **real** ``git_sources`` table (truncated around every
|
|
test), so DB-over-env and the env fallback go through the actual
|
|
indirection; the env list is driven by a fresh ``Settings`` on the
|
|
resolver's module (the dev ``.env`` never leaks in).
|
|
|
|
Phase 41: the pre-sync model probe (``check_models``) — a dead model
|
|
endpoint fails the run **before any clone** (the model-naming error
|
|
lands in the ``failed`` state verbatim; the sanitizer is a no-op for
|
|
it); the probe runs before ``effective_sources``; a real probe against
|
|
a stubbed dead ``LLMClient`` names the embed model **and** masks the
|
|
credentials wrapped from the endpoint URL. Where the runner keeps a
|
|
real ``LLMClient`` the probe is stubbed (:func:`_stub_probe`) so no
|
|
test ever hits the network; the ``_real_llm`` tests get a passing
|
|
probe from ``FakeEmbedder.embed_one``/``chat``.
|
|
|
|
Phase 53 (task 02): the sources-version bump — a sync whose import
|
|
changed the KB (added + updated + pruned > 0) advances the single-row
|
|
``sources_meta`` counter exactly once (the new generation lands in the
|
|
``/api/sync/status`` detail as ``sources_version``); an unchanged
|
|
re-sync never bumps (the detail still reports the current generation),
|
|
and every failure path (git error, model down) never bumps. The
|
|
counter is pinned to the migration-0010 seed (0) around every test by
|
|
:func:`_reset_sources_version`.
|
|
|
|
The git / import / overview / folder-summary layers are monkeypatched
|
|
in ``app.api.sync`` (same fake style as ``test_import_docs_git.py``) —
|
|
no real git, no LLM: the runner's state machine and HTTP surface are
|
|
under test. Phase 94 (task 02): the folder-summary layer gets the same
|
|
treatment — the fake-import tests' canned summaries would otherwise
|
|
steer the REAL ``generate_folder_summaries`` at the global
|
|
``documents`` table and the real ``LLMClient`` (network); the
|
|
real-import tests (host temp dirs, deterministic ``FakeEmbedder``) keep
|
|
the real generator, with ``folder_summaries`` truncated around every
|
|
test (:func:`_clean_folder_summaries`).
|
|
|
|
The admin client is used **as a context manager** on purpose: the
|
|
background sync task lives on the app's event loop, so the loop must
|
|
survive across requests — exactly how the app runs under uvicorn.
|
|
(A TestClient without the context manager starts a fresh loop per
|
|
request and would cancel the task on request exit.)
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import logging
|
|
import time
|
|
from collections.abc import Callable, Iterator
|
|
from datetime import UTC, datetime
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
from sqlalchemy import select, text
|
|
from sqlalchemy.orm import Session
|
|
|
|
from app.api import sync as sync_api
|
|
from app.config import Settings
|
|
from app.db import SessionLocal, db_available
|
|
from app.main import app as fastapi_app
|
|
from app.models import GitSource
|
|
from app.rag import git_sources as git_sources_resolver
|
|
from app.rag.importer import ImportSummary
|
|
from app.rag.llm import EmbeddingError, LLMClient, ModelUnavailableError
|
|
from app.rag.sources_meta import current_sources_version
|
|
from scripts.git_sync import GitSyncError
|
|
from tests.conftest import ADMIN_PASSWORD
|
|
from tests.fakes import FakeEmbedder
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _fresh_sync_state() -> Iterator[None]:
|
|
"""The module-level status object + task are process-global: reset them
|
|
around every test (both before — a previous test's terminal state
|
|
would leak into the idle assertion — and after)."""
|
|
sync_api._status = sync_api.SyncStatus()
|
|
sync_api._task = None
|
|
yield
|
|
sync_api._status = sync_api.SyncStatus()
|
|
sync_api._task = None
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def clean_git_sources(db: Session) -> Iterator[None]:
|
|
"""Phase 35: the runner resolves through the real ``git_sources``
|
|
table — global state, truncated around every test (the ``db``
|
|
fixture skips the file when Postgres is down)."""
|
|
db.execute(text("TRUNCATE git_sources"))
|
|
db.commit()
|
|
yield
|
|
db.execute(text("TRUNCATE git_sources"))
|
|
db.commit()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_sources_version() -> Iterator[None]:
|
|
"""Phase 53: the sources version counter is global mutable state —
|
|
pin it to the migration-0010 seed (0) around every sync test so the
|
|
bump assertions start from a known generation (own session: the
|
|
runner bumps through its own short-lived ``SessionLocal``).
|
|
Skips like the ``db`` fixture when Postgres is down."""
|
|
if not db_available():
|
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
|
session = SessionLocal()
|
|
try:
|
|
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
|
session.commit()
|
|
yield
|
|
finally:
|
|
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
|
session.commit()
|
|
session.close()
|
|
|
|
|
|
@pytest.fixture()
|
|
def sync_client() -> Iterator[TestClient]:
|
|
"""Context-managed TestClient — one app event loop across requests
|
|
(the background task must survive between the POST and the polls)."""
|
|
with TestClient(fastapi_app) as client:
|
|
yield client
|
|
|
|
|
|
def _settings(sources_dir: str = "~/bor-sources") -> Settings:
|
|
"""Fresh settings (no .env file); explicit kwargs beat any env leaks.
|
|
|
|
(The ``git_sources`` kwarg is gone with phase 35 — the runner reads
|
|
the URLs from the resolver, not from its own settings; the env list
|
|
is stubbed on the resolver's module via :func:`_stub_env`.)
|
|
"""
|
|
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
|
|
|
|
|
def _stub_env(monkeypatch: pytest.MonkeyPatch, git_sources: str = "") -> None:
|
|
"""The resolver's env fallback, driven by a fresh ``Settings`` (the
|
|
dev ``.env`` never leaks in — the task-02 pattern)."""
|
|
monkeypatch.setattr(
|
|
git_sources_resolver,
|
|
"get_settings",
|
|
lambda: Settings(_env_file=None, git_sources=git_sources), # pyright: ignore[reportCallIssue]
|
|
)
|
|
|
|
|
|
def _seed(db: Session, url: str) -> None:
|
|
db.add(GitSource(url=url))
|
|
db.commit()
|
|
|
|
|
|
def _seed_local(db: Session, path: Path) -> None:
|
|
"""A ``kind=local`` row as the phase-38 API stores it: the expanded
|
|
absolute path in both ``path`` and the NOT-NULL ``url`` column."""
|
|
db.add(GitSource(url=str(path), kind="local", path=str(path)))
|
|
db.commit()
|
|
|
|
|
|
@pytest.fixture()
|
|
def clean_documents(db: Session) -> Iterator[None]:
|
|
"""The real-import tests write ``documents``/``chunks`` (the canonical
|
|
KB state) — global, truncated around every such test."""
|
|
db.execute(text("TRUNCATE chunks, documents"))
|
|
db.commit()
|
|
yield
|
|
db.execute(text("TRUNCATE chunks, documents"))
|
|
db.commit()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean_folder_summaries(db: Session) -> Iterator[None]:
|
|
"""Phase 94: the ``folder_summaries`` table is global state the real
|
|
generator (the real-import tests) writes — truncated around every
|
|
sync test so the change-gate / table-empty-gate assertions start
|
|
from a known (empty) table."""
|
|
db.execute(text("TRUNCATE folder_summaries"))
|
|
db.commit()
|
|
yield
|
|
db.execute(text("TRUNCATE folder_summaries"))
|
|
db.commit()
|
|
|
|
|
|
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The pipeline's ``LLMClient`` becomes the deterministic in-process
|
|
``FakeEmbedder`` (real import, no network). ``FakeEmbedder``
|
|
implements ``embed_one`` + ``chat``, so the phase-41 probe passes
|
|
against it without a stub."""
|
|
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder())
|
|
|
|
|
|
def _stub_probe(monkeypatch: pytest.MonkeyPatch) -> list[Any]:
|
|
"""Stub the phase-41 model probe (a real ``LLMClient`` in the runner
|
|
would hit the network). Returns the clients the probe received, so
|
|
tests can assert the probe and the import share one client."""
|
|
seen: list[Any] = []
|
|
|
|
async def fake_check_models(llm: LLMClient) -> None:
|
|
seen.append(llm)
|
|
|
|
monkeypatch.setattr(sync_api, "check_models", fake_check_models)
|
|
return seen
|
|
|
|
|
|
def _login(client: TestClient) -> None:
|
|
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
|
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
|
|
|
|
|
def _poll(client: TestClient, want: str, timeout: float = 5.0) -> dict:
|
|
"""Poll ``GET /api/sync/status`` until ``state == want`` (terminal).
|
|
|
|
Any state other than ``running`` before the deadline fails loudly —
|
|
an unexpected ``failed`` must never be masked by the wait.
|
|
"""
|
|
deadline = time.monotonic() + timeout
|
|
while time.monotonic() < deadline:
|
|
body = client.get("/api/sync/status").json()
|
|
if body["state"] == want:
|
|
return body
|
|
assert body["state"] == "running", (
|
|
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
|
|
)
|
|
time.sleep(0.05)
|
|
raise AssertionError(f"sync did not reach {want!r} within {timeout}s")
|
|
|
|
|
|
class FakeImportSources:
|
|
"""Records every ``import_sources`` call; returns a canned summary."""
|
|
|
|
def __init__(self, summary: ImportSummary, delay: float = 0.0) -> None:
|
|
self.summary = summary
|
|
self.delay = delay
|
|
self.sources: list[list[Path]] = []
|
|
self.llms: list[LLMClient] = []
|
|
self.prune_flags: list[bool] = []
|
|
# Phase 64 (task 02): the progress hook the runner passes (a live
|
|
# closure while wired, None if the wiring regresses).
|
|
self.progress_hooks: list[object] = []
|
|
# Phase 89: the per-root ignore map the runner builds from the
|
|
# rows' ``ignore_paths`` (keyed by the root string the importer
|
|
# sees; two rows sharing a root string get the union).
|
|
self.ignore_maps: list[dict[str, list[str]]] = []
|
|
# Phase 105: the per-root hidden-folders flag map the runner
|
|
# builds from the rows' ``include_hidden`` (same root-string
|
|
# keying; a shared-root collision ORs — if either row says
|
|
# "index hidden", the root does).
|
|
self.include_hidden_maps: list[dict[str, bool]] = []
|
|
|
|
async def __call__(
|
|
self,
|
|
sources: list[Path],
|
|
llm: LLMClient,
|
|
*,
|
|
prune: bool = False,
|
|
limit: int | None = None,
|
|
session: Session | None = None,
|
|
progress: Callable[[str, str, int, int], None] | None = None,
|
|
ignore_by_root: dict[str, list[str]] | None = None,
|
|
include_hidden_by_root: dict[str, bool] | None = None,
|
|
) -> ImportSummary:
|
|
self.sources.append(list(sources))
|
|
self.llms.append(llm)
|
|
self.prune_flags.append(prune)
|
|
self.progress_hooks.append(progress)
|
|
self.ignore_maps.append(ignore_by_root or {})
|
|
self.include_hidden_maps.append(include_hidden_by_root or {})
|
|
if self.delay:
|
|
await asyncio.sleep(self.delay)
|
|
return self.summary
|
|
|
|
|
|
class FakeOverview:
|
|
"""Records every ``regenerate_overview`` call; canned result."""
|
|
|
|
def __init__(self, ok: bool = True) -> None:
|
|
self.ok = ok
|
|
self.llms: list[LLMClient] = []
|
|
|
|
async def __call__(self, llm: LLMClient, session: Session | None = None) -> bool:
|
|
self.llms.append(llm)
|
|
return self.ok
|
|
|
|
|
|
class FakeFolderSummaries:
|
|
"""Records every ``generate_folder_summaries`` call; canned stats.
|
|
|
|
Phase 94 (task 02): keeps the fake-import tests at the deterministic
|
|
layer boundary — the real generator would read the global
|
|
``documents`` table and call the (real) ``LLMClient`` over the
|
|
network. The generator only flushes, so the fake honours the
|
|
``skip`` flag the same way (the zero stats, no side effects).
|
|
Phase 96 (task 03): the unchanged-walk gap path calls the
|
|
generator with ``only_missing=True`` — the fake records the flag
|
|
the same way it records ``skip`` (the real gap probe,
|
|
``missing_folder_summaries``, runs against the real tables).
|
|
"""
|
|
|
|
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
|
|
|
|
def __init__(self, stats: dict[str, int] | None = None) -> None:
|
|
self.stats = stats if stats is not None else dict(self.ZERO)
|
|
self.llms: list[LLMClient] = []
|
|
self.sessions: list[Session] = []
|
|
self.skip_flags: list[bool] = []
|
|
self.only_missing_flags: list[bool] = []
|
|
|
|
async def __call__(
|
|
self,
|
|
db: Session,
|
|
llm: LLMClient,
|
|
*,
|
|
skip: bool = False,
|
|
only_missing: bool = False,
|
|
) -> dict[str, int]:
|
|
self.skip_flags.append(skip)
|
|
self.only_missing_flags.append(only_missing)
|
|
if skip:
|
|
return dict(self.ZERO)
|
|
self.llms.append(llm)
|
|
self.sessions.append(db)
|
|
return dict(self.stats)
|
|
|
|
|
|
def _fake_clone() -> tuple[list[tuple[str, Path]], object]:
|
|
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
|
|
calls: list[tuple[str, Path]] = []
|
|
|
|
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
|
|
dest = Path(dest)
|
|
dest.mkdir(parents=True, exist_ok=True)
|
|
(dest / "notes.md").write_text(f"# {dest.name}\ncontent for the KB\n", encoding="utf-8")
|
|
calls.append((url, dest))
|
|
return dest
|
|
|
|
return calls, fake_clone_or_pull
|
|
|
|
|
|
# --- anonymous -------------------------------------------------------------
|
|
|
|
|
|
def test_anonymous_gets_403_on_both_endpoints(client: TestClient) -> None:
|
|
r = client.get("/api/sync/status")
|
|
assert r.status_code == 403
|
|
assert r.json() == {"detail": "admin only"}
|
|
r = client.post("/api/sync")
|
|
assert r.status_code == 403
|
|
assert r.json() == {"detail": "admin only"}
|
|
|
|
|
|
# --- admin: success --------------------------------------------------------
|
|
|
|
|
|
def test_admin_sync_success_reports_full_detail(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url) # the phase-35 resolver picks the DB row up
|
|
_stub_env(monkeypatch) # env must not matter once the table has a row
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
probe_seen = _stub_probe(monkeypatch) # real LLMClient — probe stubbed
|
|
summary = ImportSummary(
|
|
files=5, added=1, updated=2, unchanged=2, pruned=3, errors=0,
|
|
chunks=11, embed_batches=4, summaries=1, summary_errors=0,
|
|
)
|
|
fake_import = FakeImportSources(summary)
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
fake_overview = FakeOverview(ok=True)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
assert sync_client.get("/api/sync/status").json() == {
|
|
"state": "idle",
|
|
"started_at": None,
|
|
"finished_at": None,
|
|
"detail": {},
|
|
"error": None,
|
|
# Phase 64 (task 02): the per-file progress keys — null/0/0 idle.
|
|
"current_file": None,
|
|
"files_done": 0,
|
|
"files_total": 0,
|
|
}
|
|
|
|
r = sync_client.post("/api/sync")
|
|
assert r.status_code == 202
|
|
assert r.json() == {"detail": "sync started"}
|
|
|
|
body = _poll(sync_client, "success")
|
|
assert body["error"] is None
|
|
# ISO-8601 timestamps round-trip; finished after started.
|
|
started = datetime.fromisoformat(body["started_at"])
|
|
finished = datetime.fromisoformat(body["finished_at"])
|
|
assert finished >= started
|
|
# Every ImportSummary field + the overview flag, verbatim.
|
|
assert body["detail"] == {
|
|
"files": 5, "added": 1, "updated": 2, "unchanged": 2, "pruned": 3,
|
|
"errors": 0, "chunks": 11, "summaries": 1, "summary_errors": 0,
|
|
"overview": True,
|
|
"sources_version": 1, # phase 53: changed KB → exactly one bump (0 → 1)
|
|
}
|
|
# The bump committed: the counter advanced exactly once, not twice.
|
|
assert current_sources_version(db) == 1
|
|
# Git: the configured repo was cloned into BOR_SOURCES_DIR/<repo-name>/.
|
|
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")]
|
|
# Import: exactly the checkouts, with prune=True (the button is the
|
|
# canonical "mirror the repos" action) and a real LLMClient.
|
|
assert fake_import.sources == [[tmp_path / "bor" / "repo"]]
|
|
assert fake_import.prune_flags == [True]
|
|
assert len(fake_import.llms) == 1
|
|
assert isinstance(fake_import.llms[0], LLMClient)
|
|
# Phase 64 (task 02): the runner wires the per-file progress hook
|
|
# (the live closure the status endpoint reads while the import runs).
|
|
assert len(fake_import.progress_hooks) == 1
|
|
assert callable(fake_import.progress_hooks[0])
|
|
# Overview: refreshed (added + updated > 0) with the same client.
|
|
assert fake_overview.llms == [fake_import.llms[0]]
|
|
# Phase 41: the probe ran first and got the very client the import
|
|
# and the overview reuse.
|
|
assert probe_seen == [fake_import.llms[0]]
|
|
|
|
|
|
def test_unchanged_kb_skips_overview_refresh(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
"""Phase-31 trigger is change-gated: added + updated == 0 → no ``lite`` call."""
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
summary = ImportSummary(files=7, added=0, updated=0, unchanged=7, pruned=0)
|
|
fake_import = FakeImportSources(summary)
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
fake_overview = FakeOverview(ok=True)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
assert body["detail"]["overview"] is False
|
|
assert fake_overview.llms == [] # no wasted model call
|
|
assert len(fake_import.llms) == 1 # the import itself ran
|
|
# Phase 53: an unchanged re-sync never bumps — the detail reports
|
|
# the current (unadvanced) generation.
|
|
assert body["detail"]["sources_version"] == 0
|
|
assert current_sources_version(db) == 0
|
|
|
|
|
|
# --- admin: concurrency ----------------------------------------------------
|
|
|
|
|
|
def test_double_trigger_while_running_returns_409(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
# The in-flight run takes a while (asyncio.sleep) so the second POST
|
|
# lands while it is still running.
|
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1), delay=0.5)
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
r = sync_client.post("/api/sync") # second trigger while running
|
|
assert r.status_code == 409
|
|
assert r.json() == {"detail": "a sync is already running"}
|
|
|
|
body = sync_client.get("/api/sync/status").json()
|
|
assert body["state"] == "running"
|
|
assert body["started_at"] is not None
|
|
assert body["finished_at"] is None
|
|
assert body["error"] is None
|
|
|
|
# The (single) run completes; the import ran exactly once.
|
|
_poll(sync_client, "success")
|
|
assert len(fake_import.sources) == 1
|
|
|
|
|
|
# --- admin: failures -------------------------------------------------------
|
|
|
|
|
|
def test_git_failure_marks_failed_and_skips_import(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
repo_url = f"file://{tmp_path / 'bad.git'}"
|
|
_seed(db, repo_url)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
|
|
def failing_clone(url: str, dest: Path | str) -> Path:
|
|
raise GitSyncError(
|
|
f"git clone --depth 1 {url} failed (exit 128): "
|
|
"fatal: repository not found"
|
|
)
|
|
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", failing_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary())
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
fake_overview = FakeOverview(ok=True)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
body = _poll(sync_client, "failed")
|
|
assert "bad.git" in body["error"] # the failing repo is named
|
|
assert "fatal: repository not found" in body["error"]
|
|
assert body["detail"] == {}
|
|
assert body["finished_at"] is not None
|
|
assert fake_import.sources == [] # no partial import
|
|
assert fake_overview.llms == []
|
|
# Phase 53: a FAILED sync never bumps — the version is untouched.
|
|
assert current_sources_version(db) == 0
|
|
|
|
# A failed run leaves the system restartable: a new POST is accepted.
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
_poll(sync_client, "failed")
|
|
|
|
|
|
def test_no_sources_configured_fails_loudly(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Both origins empty — no git rows, no local rows, no env URLs
|
|
(the truncate fixture + a blank env) → the fail-loud error
|
|
(phase 38: git-only message retired)."""
|
|
# Whitespace-only is just as unconfigured as empty.
|
|
_stub_env(monkeypatch, " , ")
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary())
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
body = _poll(sync_client, "failed")
|
|
assert body["error"] == "no sources configured (git or local)"
|
|
assert clone_calls == [] # git is never touched
|
|
assert fake_import.sources == []
|
|
|
|
|
|
def test_db_rows_win_over_env(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
caplog: pytest.LogCaptureFixture,
|
|
) -> None:
|
|
"""Phase 35: a stored row beats ``BOR_GIT_SOURCES`` — only the DB URL
|
|
is cloned, and the started log names the origin."""
|
|
db_url = "https://db.example.com/managed.git"
|
|
_seed(db, db_url)
|
|
_stub_env(monkeypatch, "https://env.example.com/ignored.git")
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
fake_folders = FakeFolderSummaries()
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
|
|
|
_login(sync_client)
|
|
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
# Phase 94: the changed canned summary fired the (stubbed) folder
|
|
# step with the run's own session + the import's LLM client.
|
|
assert fake_folders.llms == [fake_import.llms[0]]
|
|
assert clone_calls == [(db_url, tmp_path / "bor" / "managed")]
|
|
assert fake_import.sources == [[tmp_path / "bor" / "managed"]]
|
|
assert "env.example.com" not in str(body) # the env URL never reaches the UI
|
|
assert any("sync: started repos=1 origin=db" in r.getMessage() for r in caplog.records)
|
|
|
|
|
|
def test_env_fallback_when_table_empty(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
caplog: pytest.LogCaptureFixture,
|
|
) -> None:
|
|
"""Phase 35: with the table empty (the truncate fixture), the
|
|
``BOR_GIT_SOURCES`` list is what gets cloned — origin ``env``."""
|
|
env_url = "https://env.example.com/fallback.git"
|
|
_stub_env(monkeypatch, env_url)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
_poll(sync_client, "success")
|
|
|
|
assert clone_calls == [(env_url, tmp_path / "bor" / "fallback")]
|
|
assert any("sync: started repos=1 origin=env" in r.getMessage() for r in caplog.records)
|
|
|
|
|
|
# --- phase 38: the local kind (real import, host temp dirs) ---------------
|
|
|
|
|
|
def test_local_only_sync_imports_dir(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
clean_documents: None,
|
|
) -> None:
|
|
"""Local-only config: the host temp dir (one fixture ``.md``) is
|
|
walked directly — no clone at all — and the file lands in the KB
|
|
(``GET /api/docs`` as admin)."""
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "notes.md").write_text("# Local Notes\nthe local fixture\n", encoding="utf-8")
|
|
_seed_local(db, local_dir)
|
|
_stub_env(monkeypatch) # env must not matter once the table has a row
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
assert clone_calls == [] # nothing to clone — local is walked directly
|
|
assert body["detail"]["added"] == 1
|
|
assert body["detail"]["errors"] == 0
|
|
# The local file is in the KB, sourced by the directory's basename.
|
|
docs = sync_client.get("/api/docs").json()["documents"]
|
|
assert [(d["source"], d["path"]) for d in docs] == [("LocalDocs", "notes.md")]
|
|
|
|
|
|
def test_mixed_git_local_sync_imports_both_and_prunes_union(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
caplog: pytest.LogCaptureFixture,
|
|
clean_documents: None,
|
|
) -> None:
|
|
"""Mixed config: the git row is cloned, the local dir walked, and
|
|
both are imported in one run over the single combined list. The
|
|
started log carries the kind counts; a file deleted out of the
|
|
local dir is pruned on the next sync (union prune) while the git
|
|
doc survives."""
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url)
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "a.md").write_text("# A\nfirst local file\n", encoding="utf-8")
|
|
(local_dir / "b.md").write_text("# B\nsecond local file\n", encoding="utf-8")
|
|
_seed_local(db, local_dir)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_real_llm(monkeypatch)
|
|
|
|
_login(sync_client)
|
|
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
# Git cloned into BOR_SOURCES_DIR, local dir walked in row order.
|
|
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")]
|
|
assert body["detail"]["added"] == 3
|
|
assert any(
|
|
"sync: started repos=2 origin=db git=1 local=1" in r.getMessage() for r in caplog.records
|
|
)
|
|
docs = {(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]}
|
|
assert docs == {("repo", "notes.md"), ("LocalDocs", "a.md"), ("LocalDocs", "b.md")}
|
|
|
|
# Union prune: delete one local file → the next sync prunes exactly
|
|
# it; the git doc (and the surviving local file) stay.
|
|
(local_dir / "b.md").unlink()
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
assert body["detail"]["pruned"] == 1
|
|
assert body["detail"]["added"] == 0
|
|
docs = {(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]}
|
|
assert docs == {("repo", "notes.md"), ("LocalDocs", "a.md")}
|
|
|
|
|
|
def test_missing_local_dir_fails_loudly_and_imports_nothing(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
"""A local row whose directory is gone at sync time (moved/deleted
|
|
since add-time) → ``failed`` naming the path, **zero** import
|
|
attempts — the git row before it in row order was still cloned
|
|
(per-row walk; a clone is not an import)."""
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
missing = tmp_path / "Gone"
|
|
db.add(
|
|
GitSource(url=repo_url, kind="git", added_at=datetime(2026, 1, 1, tzinfo=UTC))
|
|
)
|
|
db.add(
|
|
GitSource(url=str(missing), kind="local", path=str(missing),
|
|
added_at=datetime(2026, 1, 2, tzinfo=UTC))
|
|
)
|
|
db.commit()
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary())
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
fake_overview = FakeOverview(ok=True)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
body = _poll(sync_client, "failed")
|
|
assert f"local source missing: {missing}" in body["error"] # the path is named
|
|
assert body["detail"] == {}
|
|
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")] # git row walked first
|
|
assert fake_import.sources == [] # no partial import
|
|
assert fake_overview.llms == []
|
|
|
|
|
|
def test_import_error_is_reported_with_credentials_masked(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
|
|
async def failing_import(
|
|
sources: list[Path],
|
|
llm: LLMClient,
|
|
*,
|
|
prune: bool = False,
|
|
limit: int | None = None,
|
|
session: Session | None = None,
|
|
progress: Callable[[str, str, int, int], None] | None = None,
|
|
ignore_by_root: dict[str, list[str]] | None = None,
|
|
include_hidden_by_root: dict[str, bool] | None = None,
|
|
) -> ImportSummary:
|
|
raise EmbeddingError(
|
|
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
|
"failed: connection refused"
|
|
)
|
|
|
|
monkeypatch.setattr(sync_api, "import_sources", failing_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
body = _poll(sync_client, "failed")
|
|
assert "*****@aipi.reeseapps.com" in body["error"] # credentials masked
|
|
assert "user:secret" not in body["error"]
|
|
assert "connection refused" in body["error"] # the reason survives
|
|
|
|
|
|
# --- phase 41: model probe (fail fast before any clone) --------------------
|
|
|
|
|
|
class _DeadEmbedder:
|
|
"""Probe fake whose embedding call dies like a dead endpoint — with a
|
|
credential-bearing URL in the wrapped error (the sanitizer must
|
|
mask it). ``chat`` is recorded: it must never be reached."""
|
|
|
|
def __init__(self) -> None:
|
|
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
|
self.embed_batches = 0
|
|
self.chat_calls: list[list[dict[str, str]]] = []
|
|
|
|
async def embed_one(self, text: str) -> list[float]:
|
|
raise EmbeddingError(
|
|
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
|
"failed: connection refused"
|
|
)
|
|
|
|
async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str:
|
|
self.chat_calls.append(list(messages))
|
|
return "pong"
|
|
|
|
|
|
def test_model_down_fails_fast_before_any_clone(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
"""A dead model endpoint fails the run **before any clone**: the
|
|
model-naming error lands in the ``failed`` state verbatim (the
|
|
sanitizer is a no-op for it), and clone + import never run."""
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
fake_import = FakeImportSources(ImportSummary())
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
message = (
|
|
"The embedding model ('embed') is not available — "
|
|
"check the model endpoint and retry."
|
|
)
|
|
|
|
async def dead_probe(llm: LLMClient) -> None:
|
|
raise ModelUnavailableError(message)
|
|
|
|
monkeypatch.setattr(sync_api, "check_models", dead_probe)
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
body = _poll(sync_client, "failed")
|
|
assert body["error"] == message # the model is named, verbatim
|
|
# The sanitizer must leave the model-naming message untouched.
|
|
assert sync_api._sanitize_error(body["error"]) == body["error"]
|
|
assert body["detail"] == {}
|
|
assert clone_calls == [] # fail fast: before any clone
|
|
assert fake_import.sources == [] # and before any import
|
|
# Phase 53: a FAILED sync (model down) never bumps.
|
|
assert current_sources_version(db) == 0
|
|
|
|
|
|
def test_probe_names_dead_embed_model_and_masks_credentials(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
|
) -> None:
|
|
"""The **real** probe against a stubbed dead client: the error names
|
|
the embedding model, and the credentials wrapped from the endpoint
|
|
URL are masked by the sync sanitizer (the reason survives)."""
|
|
repo_url = f"file://{tmp_path / 'repo.git'}"
|
|
_seed(db, repo_url)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
dead = _DeadEmbedder()
|
|
monkeypatch.setattr(sync_api, "LLMClient", lambda: dead)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
fake_import = FakeImportSources(ImportSummary())
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
|
|
body = _poll(sync_client, "failed")
|
|
assert "The embedding model ('embed') is not available" in body["error"]
|
|
assert "*****@aipi.reeseapps.com" in body["error"] # credentials masked
|
|
assert "user:secret" not in body["error"]
|
|
assert "connection refused" in body["error"] # the reason survives
|
|
assert dead.chat_calls == [] # embed died first — chat never probed
|
|
assert clone_calls == []
|
|
assert fake_import.sources == []
|
|
|
|
|
|
def test_probe_runs_before_source_resolution(
|
|
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
|
) -> None:
|
|
"""Ordering: the probe is the **first** pipeline step — it runs
|
|
before ``effective_sources`` (and, transitively, before any
|
|
clone)."""
|
|
order: list[str] = []
|
|
|
|
async def spy_probe(llm: LLMClient) -> None:
|
|
order.append("probe")
|
|
|
|
def spy_effective_sources(session: Session) -> tuple[list[GitSource], str]:
|
|
order.append("effective_sources")
|
|
raise GitSyncError("short-circuit after the ordering spy")
|
|
|
|
monkeypatch.setattr(sync_api, "check_models", spy_probe)
|
|
monkeypatch.setattr(sync_api, "effective_sources", spy_effective_sources)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
clone_calls, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "failed")
|
|
|
|
assert order == ["probe", "effective_sources"]
|
|
assert clone_calls == []
|
|
assert "short-circuit" in body["error"] # the spy aborted the run
|
|
|
|
|
|
# --- phase 89: per-row ignore lists ------------------------------------------
|
|
|
|
|
|
def test_local_row_ignore_paths_excluded_from_sync(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
clean_documents: None,
|
|
) -> None:
|
|
"""Phase 89: a local row carrying ``ignore_paths`` — the button
|
|
sync skips every matching file: the ignored file never lands in the
|
|
KB (no document row — hence no embedding, no summary), and the
|
|
success detail's ``files``/``added`` counts exclude it, while the
|
|
kept file imports as usual."""
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
|
|
(local_dir / "ignore").mkdir()
|
|
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
|
|
encoding="utf-8")
|
|
db.add(
|
|
GitSource(
|
|
url=str(local_dir), kind="local", path=str(local_dir),
|
|
ignore_paths=["ignore/"], # any spelling — the importer normalizes
|
|
)
|
|
)
|
|
db.commit()
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
# The ignored file is not counted: only keep.md was walked.
|
|
assert body["detail"]["files"] == 1
|
|
assert body["detail"]["added"] == 1
|
|
assert body["detail"]["errors"] == 0
|
|
# The KB holds exactly the kept file — ignore/secret.md is absent.
|
|
docs = [(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]]
|
|
assert docs == [("LocalDocs", "keep.md")]
|
|
|
|
|
|
def test_sync_builds_ignore_map_by_root_string_with_union(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Phase 89 wiring (fake import): the runner keys the map by the
|
|
SAME root string the importer sees, and two rows sharing that root
|
|
string (the sibling/repo-name edge — ``…/shared`` and
|
|
``…/shared.git`` clone into the same checkout dir) get the UNION
|
|
of their lists, in row order; a row with an empty list contributes
|
|
nothing."""
|
|
url_a = f"file://{tmp_path / 'shared'}"
|
|
url_b = f"{url_a}.git" # same repo name → same checkout dir
|
|
# Distinct added_at: the resolver orders by (added_at, id) — a
|
|
# same-timestamp pair would tie-break on the random uuid.
|
|
db.add(GitSource(url=url_a, kind="git", ignore_paths=["a/"],
|
|
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
|
|
db.add(GitSource(url=url_b, kind="git", ignore_paths=["b"],
|
|
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
|
|
db.commit()
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
_poll(sync_client, "success")
|
|
|
|
shared = str(tmp_path / "bor" / "shared")
|
|
# Both rows resolve to the SAME checkout (the collision itself) and
|
|
# the map holds their union, keyed by that one root string.
|
|
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
|
|
assert fake_import.ignore_maps == [{shared: ["a/", "b"]}]
|
|
|
|
|
|
def test_sync_without_ignore_lists_passes_empty_map(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Phase 89 regression: rows without a list → the runner passes an
|
|
EMPTY map (the importer's byte-identical pre-phase-89 behavior)."""
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "notes.md").write_text("# Notes\nplain row\n", encoding="utf-8")
|
|
_seed_local(db, local_dir) # no ignore_paths → []
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
_poll(sync_client, "success")
|
|
|
|
assert fake_import.sources == [[local_dir]]
|
|
assert fake_import.ignore_maps == [{}] # no row carried a list
|
|
|
|
|
|
# --- phase 105: per-row hidden-folders flag --------------------------------
|
|
|
|
|
|
def test_local_row_hidden_flag_off_then_on_indexes_hidden_paths(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
clean_documents: None,
|
|
) -> None:
|
|
"""Phase 105 (A4 then A1): the SAME local row, synced first with the
|
|
default flag (off) — the file inside the hidden folder never lands
|
|
in the KB (no document row — hence no embedding, no summary), the
|
|
visible file imports as usual; then the row is flipped on (direct
|
|
model set — the PATCH round-trip is task 03's layer) and the next
|
|
sync walks the hidden file too: it is indexed, embedded, and counted
|
|
like any visible file."""
|
|
local_dir = tmp_path / "LocalDocs"
|
|
local_dir.mkdir()
|
|
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
|
(local_dir / ".hidden").mkdir()
|
|
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
|
_seed_local(db, local_dir) # include_hidden defaults to False (A4)
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
# Flag off (A4): only the visible file is walked and indexed — the
|
|
# hidden file has NO documents row.
|
|
assert body["detail"]["files"] == 1
|
|
assert body["detail"]["added"] == 1
|
|
assert body["detail"]["errors"] == 0
|
|
docs = {
|
|
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
|
|
}
|
|
assert docs == {("LocalDocs", "visible.md")}
|
|
|
|
# Flip the SAME row on — the next sync re-reads the flag per row.
|
|
row = db.execute(select(GitSource).where(GitSource.url == str(local_dir))).scalar_one()
|
|
row.include_hidden = True
|
|
db.commit()
|
|
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
body = _poll(sync_client, "success")
|
|
|
|
# Flag on (A1): the hidden file is walked (detail.files counts it),
|
|
# embedded, and indexed alongside the visible file.
|
|
assert body["detail"]["files"] == 2
|
|
assert body["detail"]["added"] == 1
|
|
assert body["detail"]["errors"] == 0
|
|
docs = {
|
|
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
|
|
}
|
|
assert docs == {("LocalDocs", "visible.md"), ("LocalDocs", ".hidden/note.md")}
|
|
|
|
|
|
def test_sync_builds_include_hidden_map_by_root_string_with_or(
|
|
sync_client: TestClient,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
db: Session,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Phase 105 wiring (fake import): the runner keys the flag map by
|
|
the SAME root string the importer sees, and two rows sharing that
|
|
root string (the sibling/repo-name edge — ``…/shared`` and
|
|
``…/shared.git`` clone into the same checkout dir) get the OR of
|
|
their flags — if EITHER row says "index hidden", the root does
|
|
(the ignore-map union's boolean mirror)."""
|
|
url_a = f"file://{tmp_path / 'shared'}"
|
|
url_b = f"{url_a}.git" # same repo name → same checkout dir
|
|
# Distinct added_at: the resolver orders by (added_at, id) — a
|
|
# same-timestamp pair would tie-break on the random uuid.
|
|
db.add(GitSource(url=url_a, kind="git", include_hidden=False,
|
|
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
|
|
db.add(GitSource(url=url_b, kind="git", include_hidden=True,
|
|
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
|
|
db.commit()
|
|
_stub_env(monkeypatch)
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
|
)
|
|
_, fake_clone = _fake_clone()
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
_stub_probe(monkeypatch)
|
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
|
|
|
_login(sync_client)
|
|
assert sync_client.post("/api/sync").status_code == 202
|
|
_poll(sync_client, "success")
|
|
|
|
shared = str(tmp_path / "bor" / "shared")
|
|
# Both rows resolve to the SAME checkout (the collision itself) and
|
|
# the map holds the OR of their flags, keyed by that one root
|
|
# string — the flag-off row's False is overridden by the True.
|
|
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
|
|
assert fake_import.include_hidden_maps == [{shared: True}]
|