phase: 89_source_ignore_paths
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed. ## Phase 89 — final verification pass: ALL GREEN **Verified (all 6 task files present in `complete/`):** - `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`) - Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources` - API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403) - Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py` - Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box **Test/lint results:** - `uv run pytest` → 1808 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%) - `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up) - Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules. **Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green). **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+37
-13
@@ -26,6 +26,11 @@ precedence order:
|
||||
``~/Deployments``), kept for backwards compatibility (reached only
|
||||
while both the table and ``BOR_GIT_SOURCES`` are empty).
|
||||
|
||||
Phase 89: resolution also returns each row's ignore paths, keyed by
|
||||
the resolved root string (the importer normalizes them); manual
|
||||
``--source`` dirs and the legacy fallback have no rows, so they import
|
||||
with no ignore.
|
||||
|
||||
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
|
||||
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
|
||||
family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add
|
||||
@@ -133,9 +138,11 @@ def repo_name(url: str) -> str:
|
||||
return slug or "repo"
|
||||
|
||||
|
||||
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
|
||||
def _resolve_sources(
|
||||
cli_sources: list[Path] | None, settings: Settings
|
||||
) -> tuple[list[Path], dict[str, list[str]]]:
|
||||
"""Resolve the directories to import (phase 28, extended in phases
|
||||
35 and 38).
|
||||
35 and 38; per-root ignore maps, phase 89).
|
||||
|
||||
Precedence: ``--source`` (explicit manual paths — always wins) >
|
||||
the effective sources — the ``git_sources`` DB rows (git + local),
|
||||
@@ -148,12 +155,18 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
|
||||
stored directory, re-verified ``.is_dir()`` at run time) > the
|
||||
legacy ``DEFAULT_SOURCES``.
|
||||
|
||||
Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed
|
||||
by the resolved root string, exactly as the importer sees it (two
|
||||
rows sharing a root string get the union — extend, not replace);
|
||||
manual ``--source`` dirs and the legacy fallback have no rows, so
|
||||
they import with an empty map (no ignore).
|
||||
|
||||
A :class:`GitSyncError` from a failing clone/pull — or a missing
|
||||
local directory (``local source missing: <path>``) — propagates to
|
||||
:func:`main`, which aborts the run before importing anything.
|
||||
"""
|
||||
if cli_sources:
|
||||
return [path.expanduser() for path in cli_sources]
|
||||
return [path.expanduser() for path in cli_sources], {}
|
||||
db = SessionLocal()
|
||||
try:
|
||||
rows, origin = effective_sources(db)
|
||||
@@ -167,21 +180,28 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
|
||||
)
|
||||
sources_root = Path(settings.sources_dir).expanduser()
|
||||
sources: list[Path] = []
|
||||
ignore_by_root: dict[str, list[str]] = {}
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
||||
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
# ``or`` keeps the type checker honest); a missing
|
||||
# directory aborts before importing, the same pre-import
|
||||
# fail-loud as a failing git clone.
|
||||
path = Path(row.path or row.url).expanduser()
|
||||
if not path.is_dir():
|
||||
raise GitSyncError(f"local source missing: {path}")
|
||||
sources.append(path)
|
||||
return sources
|
||||
return [path.expanduser() for path in DEFAULT_SOURCES]
|
||||
root = Path(row.path or row.url).expanduser()
|
||||
if not root.is_dir():
|
||||
raise GitSyncError(f"local source missing: {root}")
|
||||
sources.append(root)
|
||||
# Phase 89: the row's ignore list, keyed by the SAME root
|
||||
# string the importer sees; two rows sharing a root string
|
||||
# get the union (extend, not replace) — the sibling/repo-name
|
||||
# edge.
|
||||
if row.ignore_paths:
|
||||
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
|
||||
return sources, ignore_by_root
|
||||
return [path.expanduser() for path in DEFAULT_SOURCES], {}
|
||||
|
||||
|
||||
def _overview_row_exists() -> bool:
|
||||
@@ -204,9 +224,10 @@ def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
# Git sources resolve (and clone/pull) *before* any import: a failing
|
||||
# repo aborts the run with a non-zero exit, naming the failure — a bad
|
||||
# URL must never silently import partial junk.
|
||||
# URL must never silently import partial junk. The second element is
|
||||
# the phase-89 per-root ignore map (empty for manual/fallback paths).
|
||||
try:
|
||||
sources = _resolve_sources(args.source, settings)
|
||||
sources, ignore_by_root = _resolve_sources(args.source, settings)
|
||||
except GitSyncError as e:
|
||||
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
|
||||
return 1
|
||||
@@ -248,7 +269,10 @@ def main(argv: list[str] | None = None) -> int:
|
||||
runs and unchanged re-runs never bump. The returned token is
|
||||
the new version, or ``"skipped"``.
|
||||
"""
|
||||
summary = await import_sources(sources, llm, prune=args.prune, limit=args.limit)
|
||||
summary = await import_sources(
|
||||
sources, llm, prune=args.prune, limit=args.limit,
|
||||
ignore_by_root=ignore_by_root,
|
||||
)
|
||||
if args.limit is not None:
|
||||
# An incomplete walk is debug-only — it must never advance
|
||||
# the generation (mirrors the --limit overview skip below).
|
||||
|
||||
Reference in New Issue
Block a user