phase: 89_source_ignore_paths
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed. ## Phase 89 — final verification pass: ALL GREEN **Verified (all 6 task files present in `complete/`):** - `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`) - Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources` - API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403) - Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py` - Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box **Test/lint results:** - `uv run pytest` → 1808 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%) - `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up) - Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules. **Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green). **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+73
-3
@@ -118,17 +118,63 @@ class ImportSummary:
|
||||
)
|
||||
|
||||
|
||||
def normalize_ignore_path(entry: str) -> str:
|
||||
"""One ignore-path entry → canonical form (phase 89, A1).
|
||||
|
||||
Trim surrounding whitespace, then strip ALL leading/trailing
|
||||
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
|
||||
all become ``"my/files"``. ``""`` / ``"//"``,
|
||||
``" "`` normalize to ``""`` (callers drop empties).
|
||||
"""
|
||||
return entry.strip().strip("/")
|
||||
|
||||
|
||||
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
|
||||
"""Phase 89, A1 — the pure prefix rule, nothing else.
|
||||
|
||||
``rel`` is the source-relative POSIX path WITHOUT a leading
|
||||
slash (the ``documents.path`` string). Match = ``rel`` STARTS
|
||||
WITH a normalized entry: raw string prefix — deliberately NO
|
||||
component-boundary check (``"my/files"`` also matches
|
||||
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
|
||||
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
|
||||
"""
|
||||
return any(rel.startswith(p) for p in prefixes)
|
||||
|
||||
|
||||
def _ignore_for_root(
|
||||
root: Path, ignore_by_root: dict[str, list[str]] | None
|
||||
) -> tuple[str, ...]:
|
||||
"""The normalized, non-empty prefix tuple for one root (phase 89).
|
||||
|
||||
Keyed by ``str(root)`` — the root string exactly as the caller
|
||||
passed it in ``sources`` (unambiguous when two rows share a
|
||||
source *name* but different dirs). Callers may pass RAW box
|
||||
lines: the importer normalizes + drops empties here, the single
|
||||
choke point — stored lists (already normalized) normalize to
|
||||
themselves.
|
||||
"""
|
||||
raw = (ignore_by_root or {}).get(str(root)) or []
|
||||
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
|
||||
|
||||
|
||||
def iter_importable_files(
|
||||
root: Path,
|
||||
extensions: frozenset[str],
|
||||
excluded: frozenset[str] = EXCLUDED_DIRS,
|
||||
ignore: tuple[str, ...] = (),
|
||||
) -> list[Path]:
|
||||
"""All importable files under *root* (sorted), per the A9 scope rules.
|
||||
|
||||
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
|
||||
Skips: any path with a dot-prefixed component (hidden dirs/files —
|
||||
vendored caches like ``.esphome/.espressif/**``) and the well-known
|
||||
non-content directories in *excluded*.
|
||||
non-content directories in *excluded*. *ignore* (phase 89, A1) is a
|
||||
tuple of ALREADY-normalized, non-empty source-relative path prefixes
|
||||
(the importer's ``_ignore_for_root`` is the normalization choke point
|
||||
— raw box lines never reach this function): a file is skipped when its
|
||||
source-relative POSIX path starts with any entry; the default ``()``
|
||||
keeps every existing caller byte-identical.
|
||||
"""
|
||||
if not root.is_dir():
|
||||
return []
|
||||
@@ -139,6 +185,8 @@ def iter_importable_files(
|
||||
rel = path.relative_to(root)
|
||||
if any(part.startswith(".") or part in excluded for part in rel.parts):
|
||||
continue
|
||||
if ignore and is_ignored(rel.as_posix(), ignore):
|
||||
continue
|
||||
if path.suffix.lower() not in extensions:
|
||||
continue
|
||||
files.append(path)
|
||||
@@ -153,6 +201,7 @@ async def import_sources(
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
) -> ImportSummary:
|
||||
"""Import every A9-format file under *sources* (see module docstring).
|
||||
|
||||
@@ -172,6 +221,18 @@ async def import_sources(
|
||||
``limit``, the hook still fires per processed file only — ``done`` never
|
||||
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||||
incomplete walk must not misreport the denominator).
|
||||
|
||||
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
|
||||
string exactly as passed in *sources* — to that source's RAW ignore-path
|
||||
lines (the importer normalizes them via ``_ignore_for_root``, the single
|
||||
choke point): matching files are never walked, so they are never
|
||||
embedded and never summarized, and the progress pre-walk uses the same
|
||||
per-root tuple as the processing loop, so ``total`` never counts them.
|
||||
A file that newly matches a pattern simply never enters ``seen``, so the
|
||||
next ``prune=True`` run deletes its row automatically (A2 — the A9
|
||||
junk-precedent). ``None`` (the default) changes nothing: the map is read
|
||||
per root, unlisted roots get an empty tuple, and every existing caller
|
||||
behaves byte-identically.
|
||||
"""
|
||||
if limit is not None and limit <= 0:
|
||||
raise ValueError("limit must be >= 1")
|
||||
@@ -188,7 +249,13 @@ async def import_sources(
|
||||
total = 0
|
||||
if progress is not None:
|
||||
for root in sources:
|
||||
total += len(iter_importable_files(root, llm.settings.import_extension_set))
|
||||
total += len(
|
||||
iter_importable_files(
|
||||
root,
|
||||
llm.settings.import_extension_set,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
)
|
||||
)
|
||||
try:
|
||||
for root in sources:
|
||||
if not root.is_dir():
|
||||
@@ -198,7 +265,10 @@ async def import_sources(
|
||||
break
|
||||
source = root.name
|
||||
source_names.add(source)
|
||||
for path in iter_importable_files(root, llm.settings.import_extension_set):
|
||||
ignore = _ignore_for_root(root, ignore_by_root)
|
||||
for path in iter_importable_files(
|
||||
root, llm.settings.import_extension_set, ignore=ignore
|
||||
):
|
||||
if limit is not None and summary.files >= limit:
|
||||
break
|
||||
rel = path.relative_to(root).as_posix()
|
||||
|
||||
Reference in New Issue
Block a user