phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s

All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-09 01:45:42 -04:00
parent 0495e4e7e4
commit 8c706259e9
49 changed files with 3717 additions and 63 deletions
+37 -13
View File
@@ -26,6 +26,11 @@ precedence order:
``~/Deployments``), kept for backwards compatibility (reached only
while both the table and ``BOR_GIT_SOURCES`` are empty).
Phase 89: resolution also returns each row's ignore paths, keyed by
the resolved root string (the importer normalizes them); manual
``--source`` dirs and the legacy fallback have no rows, so they import
with no ignore.
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add
@@ -133,9 +138,11 @@ def repo_name(url: str) -> str:
return slug or "repo"
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
def _resolve_sources(
cli_sources: list[Path] | None, settings: Settings
) -> tuple[list[Path], dict[str, list[str]]]:
"""Resolve the directories to import (phase 28, extended in phases
35 and 38).
35 and 38; per-root ignore maps, phase 89).
Precedence: ``--source`` (explicit manual paths — always wins) >
the effective sources — the ``git_sources`` DB rows (git + local),
@@ -148,12 +155,18 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
stored directory, re-verified ``.is_dir()`` at run time) > the
legacy ``DEFAULT_SOURCES``.
Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed
by the resolved root string, exactly as the importer sees it (two
rows sharing a root string get the union — extend, not replace);
manual ``--source`` dirs and the legacy fallback have no rows, so
they import with an empty map (no ignore).
A :class:`GitSyncError` from a failing clone/pull — or a missing
local directory (``local source missing: <path>``) — propagates to
:func:`main`, which aborts the run before importing anything.
"""
if cli_sources:
return [path.expanduser() for path in cli_sources]
return [path.expanduser() for path in cli_sources], {}
db = SessionLocal()
try:
rows, origin = effective_sources(db)
@@ -167,21 +180,28 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
)
sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
for row in rows:
if row.kind == "git":
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
else:
# kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the
# ``or`` keeps the type checker honest); a missing
# directory aborts before importing, the same pre-import
# fail-loud as a failing git clone.
path = Path(row.path or row.url).expanduser()
if not path.is_dir():
raise GitSyncError(f"local source missing: {path}")
sources.append(path)
return sources
return [path.expanduser() for path in DEFAULT_SOURCES]
root = Path(row.path or row.url).expanduser()
if not root.is_dir():
raise GitSyncError(f"local source missing: {root}")
sources.append(root)
# Phase 89: the row's ignore list, keyed by the SAME root
# string the importer sees; two rows sharing a root string
# get the union (extend, not replace) — the sibling/repo-name
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
return sources, ignore_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}
def _overview_row_exists() -> bool:
@@ -204,9 +224,10 @@ def main(argv: list[str] | None = None) -> int:
# Git sources resolve (and clone/pull) *before* any import: a failing
# repo aborts the run with a non-zero exit, naming the failure — a bad
# URL must never silently import partial junk.
# URL must never silently import partial junk. The second element is
# the phase-89 per-root ignore map (empty for manual/fallback paths).
try:
sources = _resolve_sources(args.source, settings)
sources, ignore_by_root = _resolve_sources(args.source, settings)
except GitSyncError as e:
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
return 1
@@ -248,7 +269,10 @@ def main(argv: list[str] | None = None) -> int:
runs and unchanged re-runs never bump. The returned token is
the new version, or ``"skipped"``.
"""
summary = await import_sources(sources, llm, prune=args.prune, limit=args.limit)
summary = await import_sources(
sources, llm, prune=args.prune, limit=args.limit,
ignore_by_root=ignore_by_root,
)
if args.limit is not None:
# An incomplete walk is debug-only — it must never advance
# the generation (mirrors the --limit overview skip below).