phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s

All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
2026-09-09 01:45:42 -04:00
parent 0495e4e7e4
commit 8c706259e9
49 changed files with 3717 additions and 63 deletions
+73 -3
View File
@@ -118,17 +118,63 @@ class ImportSummary:
)
def normalize_ignore_path(entry: str) -> str:
"""One ignore-path entry → canonical form (phase 89, A1).
Trim surrounding whitespace, then strip ALL leading/trailing
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
all become ``"my/files"``. ``""`` / ``"//"``,
``" "`` normalize to ``""`` (callers drop empties).
"""
return entry.strip().strip("/")
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
"""Phase 89, A1 — the pure prefix rule, nothing else.
``rel`` is the source-relative POSIX path WITHOUT a leading
slash (the ``documents.path`` string). Match = ``rel`` STARTS
WITH a normalized entry: raw string prefix — deliberately NO
component-boundary check (``"my/files"`` also matches
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
"""
return any(rel.startswith(p) for p in prefixes)
def _ignore_for_root(
root: Path, ignore_by_root: dict[str, list[str]] | None
) -> tuple[str, ...]:
"""The normalized, non-empty prefix tuple for one root (phase 89).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (unambiguous when two rows share a
source *name* but different dirs). Callers may pass RAW box
lines: the importer normalizes + drops empties here, the single
choke point — stored lists (already normalized) normalize to
themselves.
"""
raw = (ignore_by_root or {}).get(str(root)) or []
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
def iter_importable_files(
root: Path,
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
) -> list[Path]:
"""All importable files under *root* (sorted), per the A9 scope rules.
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
Skips: any path with a dot-prefixed component (hidden dirs/files —
vendored caches like ``.esphome/.espressif/**``) and the well-known
non-content directories in *excluded*.
non-content directories in *excluded*. *ignore* (phase 89, A1) is a
tuple of ALREADY-normalized, non-empty source-relative path prefixes
(the importer's ``_ignore_for_root`` is the normalization choke point
— raw box lines never reach this function): a file is skipped when its
source-relative POSIX path starts with any entry; the default ``()``
keeps every existing caller byte-identical.
"""
if not root.is_dir():
return []
@@ -139,6 +185,8 @@ def iter_importable_files(
rel = path.relative_to(root)
if any(part.startswith(".") or part in excluded for part in rel.parts):
continue
if ignore and is_ignored(rel.as_posix(), ignore):
continue
if path.suffix.lower() not in extensions:
continue
files.append(path)
@@ -153,6 +201,7 @@ async def import_sources(
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring).
@@ -172,6 +221,18 @@ async def import_sources(
``limit``, the hook still fires per processed file only — ``done`` never
exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator).
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
string exactly as passed in *sources* — to that source's RAW ignore-path
lines (the importer normalizes them via ``_ignore_for_root``, the single
choke point): matching files are never walked, so they are never
embedded and never summarized, and the progress pre-walk uses the same
per-root tuple as the processing loop, so ``total`` never counts them.
A file that newly matches a pattern simply never enters ``seen``, so the
next ``prune=True`` run deletes its row automatically (A2 — the A9
junk-precedent). ``None`` (the default) changes nothing: the map is read
per root, unlisted roots get an empty tuple, and every existing caller
behaves byte-identically.
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -188,7 +249,13 @@ async def import_sources(
total = 0
if progress is not None:
for root in sources:
total += len(iter_importable_files(root, llm.settings.import_extension_set))
total += len(
iter_importable_files(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
)
)
try:
for root in sources:
if not root.is_dir():
@@ -198,7 +265,10 @@ async def import_sources(
break
source = root.name
source_names.add(source)
for path in iter_importable_files(root, llm.settings.import_extension_set):
ignore = _ignore_for_root(root, ignore_by_root)
for path in iter_importable_files(
root, llm.settings.import_extension_set, ignore=ignore
):
if limit is not None and summary.files >= limit:
break
rel = path.relative_to(root).as_posix()