diff --git a/.agents/phases/todo/89_source_ignore_paths/00_phase.md b/.agents/phases/todo/89_source_ignore_paths/00_phase.md new file mode 100644 index 0000000..161e845 --- /dev/null +++ b/.agents/phases/todo/89_source_ignore_paths/00_phase.md @@ -0,0 +1,72 @@ +# Phase 89 — Per-source ignore paths: exclude files/folders from import + +**Source:** `TODO.md` L3 — "The user should be able to ignore/exclude specific files and folders when importing from a source. They should be able to type these files and folders into a box on the sources page. Each source should support a list of ignored files and folders. These ignored files/folders should not be embedded and should not be summarized. The format of the ignored files/folders should be a path, so \"/my/files/\" or \"my/files/\" or \"my/files\" would ignore all files that start with the prefix \"my/files\". There shouldn't be any support for \"match in the middle\" of a path, so \"myfile.txt\" would match \"/myfile.txt\" but not \"some/path/myfile.txt\". The ignore is just a prefix ignore." +**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-08). +**Context:** Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources`). Every import entry point funnels through `import_sources` (`app/rag/importer.py`): the in-app Sync button (`app/api/sync.py::_run_sync`), the archive-upload background scan (`app/api/git_sources.py::_run_upload`), and the CLI (`scripts/import_docs.py`). `iter_importable_files` is the single walk choke point (extension filter + hidden-dir rule + `EXCLUDED_DIRS`); `import_sources` also runs a pre-walk with the same rules for the phase-64 progress `total`; ignored files must be excluded from BOTH. Non-markdown files get a `lite` summary in `_store_summary` (phase 30) — a file that never enters the walk is never embedded and never summarized, by construction. Pruning (`_prune`, `prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — so files that newly match an ignore pattern leave the index on the next sync, exactly like the A9 out-of-scope-junk behavior. The Sources page is admin-gated (phase 16 pattern); env-fallback rows (`from_env: true`, `id: null`) have no DB row to store a list on. + +## Objective +Each stored source carries an **ignore list** of source-relative path prefixes, edited in a per-row box on the Sources page; matching is pure prefix (no mid-path matching, no globs — the spec's examples verbatim); ignored files are never walked, embedded, or summarized, and previously indexed files that newly match an ignore pattern are pruned on the next sync. All three import entry points honor the lists; the API is admin-only like the rest of the router. + +## Dependencies +- `88_mobile_chat_hamburger_boot` (complete) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/upload/CLI pipelines, and the Sources view — none of which phase 88's pins reach; its suites must stay green unchanged). + +## Design (shared by all tasks — the executor reads this, not the chat) + +- **Match semantics (locked, A1).** One pure helper pair in `app/rag/importer.py`: + - `normalize_ignore_path(entry: str) -> str` — `entry.strip().strip("/")` (whitespace + all leading/trailing slashes — so `"/my/files/"`, `"my/files/"`, and `"my/files"` normalize to the same `"my/files"`; `"//"` and `""` normalize to `""` and are dropped). + - `is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool` — `any(rel.startswith(p) for p in prefixes)` where `rel` is the **source-relative POSIX path without a leading slash** (`path.relative_to(root).as_posix()` — the same string `documents.path` stores) and `prefixes` are the normalized, non-empty entries. + - **Raw string prefix — deliberately no component-boundary check** (A1, owner-confirmed): `"my/files"` also matches `"my/files2/x.md"`; `"myfile.txt"` matches `"myfile.txt"` (root-level file) but NOT `"some/path/myfile.txt"` (no mid-path matching — the string simply does not start with the pattern). No globs, no `**`, no case folding beyond what the walk already does for extensions. + - The importer normalizes its input itself (single choke point — callers may pass raw box lines): an internal `_ignore_for_root(root, ignore_by_root)` yields `tuple(p for p in (normalize_ignore_path(x) for x in raw) if p)`. +- **`iter_importable_files` / `import_sources` signature (task 02).** + - `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore: tuple[str, ...] = ())` — one more skip: `if ignore and is_ignored(rel.as_posix(), ignore): continue` (after the hidden/excluded check, before/after the extension check — either order is fine; keep the existing checks byte-identical). Default `()` keeps every existing caller (CLI manual `--source`, tests) behaving exactly as before. + - `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root: dict[str, list[str]] | None = None)` — the map is keyed by **`str(root)`** (the root path string exactly as passed in `sources` — unambiguous even when two rows resolve to the same source *name*; on a root-string collision the caller has already merged the lists, Design "Callers" below). The pre-walk that computes the phase-64 progress `total` and the processing loop MUST use the same per-root ignore tuple (same helper), so `files_total` never counts ignored files. `seen` is untouched in shape → `_prune` prunes newly-ignored documents automatically (A2). +- **Callers (task 04).** + - `app/api/sync.py::_run_sync` — while building `sources` per row, build `ignore_by_root: dict[str, list[str]]` in the same loop: git row → key `str(sources_root / repo_name(row.url))`, local row → key `str(Path(row.path or row.url).expanduser())`; value `row.ignore_paths` — **extend on key collision** (two rows sharing a root string get the union of their lists). Pass `ignore_by_root=…` to `import_sources`. + - `app/api/git_sources.py::_run_upload` — step 6 already fetches the row by path (existing or the newly created one); capture it and pass `ignore_by_root={str(final_dir): row.ignore_paths}` to the step-8 `import_sources` (a re-upload of an existing source respects the list the owner already saved). + - `scripts/import_docs.py` — `_resolve_sources` returns `tuple[list[Path], dict[str, list[str]]]` (manual `--source` → `({},)` empty map — manual dirs have no row; env-fallback rows have no lists). `main` unpacks and passes the map to `import_sources`; docstrings updated (module + `_resolve_sources`). +- **Storage (task 01).** `git_sources.ignore_paths` — JSONB NOT NULL, server default `'[]'`, `Mapped[list]` (the `saved_chats.messages` JSONB precedent, `app/models.py`). Alembic `0013_git_source_ignore_paths.py` (revises `0012`) — `op.add_column("git_sources", sa.Column("ignore_paths", postgresql.JSONB(astext_type=sa.Text()), server_default=sa.text("'[]'"), nullable=False))`; downgrade drops the column. Existing rows read `[]`. +- **API contract (task 03).** + - Schemas (`app/schemas.py`): `GitSourceIn.ignore_paths: list[str] | None = None` (create-time, optional — absent → `[]`); `GitSourceOut.ignore_paths: list[str]`; `GitSourceRow.ignore_paths: list[str]` (env-fallback rows report `[]`); new `GitSourceIgnoreIn` body model for PATCH with `ignore_paths: list[str]` (required — the box replaces the whole list, A5). + - `GET /api/git-sources` — each DB row reports its (normalized, non-empty) list; env rows `[]`. + - `POST /api/git-sources` — both kinds accept `ignore_paths`; stored **normalized** (each entry `normalize_ignore_path`, empties dropped) — the UI may send raw box lines and the stored list is canonical. + - `PATCH /api/git-sources/{source_id}` (NEW, behind the existing `require_admin` router dependency) — 404 unknown id; **replace** semantics (A5): the body list, normalized, becomes the row's list (empty list = clear all); 200 → `GitSourceOut` (id, url, added_at, ignore_paths). + - Validation (422, fixed details that never echo the input — the router's credential-safety discipline, applied for consistency): at most **200 entries** (A4) → `detail="a source has at most 200 ignore paths"`; an entry **empty after normalization** → `detail="ignore paths must be non-empty"`; an entry **>500 chars** (pre- or post-normalization — check post-normalization) → `detail="an ignore path exceeds 500 characters"`. +- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: an **"Ignore paths" button** (actions cell, left of Remove — same `.git-source-remove`-style button idiom, new class `.git-source-ignore`) whose `aria-label` is `Edit ignored paths for source: `; when `s.ignore_paths.length > 0` the row also shows a text count tag `N ignored` (text, never color alone) next to the location cell. Env-fallback rows (`id` null) get **no** button — the existing "from .env" tag stays (A3). + - The button opens a **page-local editor dialog** following the phase-69 `#remove-confirm-dialog` pattern exactly (new `#ignore-editor-dialog`, `role="dialog"`, dim backdrop, focus management: focus lands on Cancel — the safe default — Escape / Cancel / backdrop close as CANCEL and return focus to the trigger; the dim backdrop click handler is the same idiom). Inside: a visible `