Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: Experimental/brain-of-reese:1f0e4c6bb9f07bacacfc049685d02840a2ebf0b5
Branches Tags
Experimental/brain-of-reese:main
..
compare: Experimental/brain-of-reese:8c706259e98b264b0a7f25693567363204e36b04
Branches Tags
Experimental/brain-of-reese:main
2 Commits
1f0e4c6bb9 .. 8c706259e9
Author SHA1 Message Date
ducoterra 8c706259e9 phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details 
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
 2026-09-09 01:45:42 -04:00
ducoterra 0495e4e7e4 chore(agent): phase 89 roadmap from TODO.md — per-source ignore paths for imports 2026-09-08 22:55:58 -04:00
49 changed files with 4214 additions and 63 deletions
Expand all files Collapse all files
.agents/phases/complete/89_source_ignore_paths/00_phase.md
+72
View File
@@ -0,0 +1,72 @@
# Phase 89 — Per-source ignore paths: exclude files/folders from import
**Source:** `TODO.md` L3 — "The user should be able to ignore/exclude specific files and folders when importing from a source. They should be able to type these files and folders into a box on the sources page. Each source should support a list of ignored files and folders. These ignored files/folders should not be embedded and should not be summarized. The format of the ignored files/folders should be a path, so \"/my/files/\" or \"my/files/\" or \"my/files\" would ignore all files that start with the prefix \"my/files\". There shouldn't be any support for \"match in the middle\" of a path, so \"myfile.txt\" would match \"/myfile.txt\" but not \"some/path/myfile.txt\". The ignore is just a prefix ignore."
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-08).
**Context:** Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources`). Every import entry point funnels through `import_sources` (`app/rag/importer.py`): the in-app Sync button (`app/api/sync.py::_run_sync`), the archive-upload background scan (`app/api/git_sources.py::_run_upload`), and the CLI (`scripts/import_docs.py`). `iter_importable_files` is the single walk choke point (extension filter + hidden-dir rule + `EXCLUDED_DIRS`); `import_sources` also runs a pre-walk with the same rules for the phase-64 progress `total`; ignored files must be excluded from BOTH. Non-markdown files get a `lite` summary in `_store_summary` (phase 30) — a file that never enters the walk is never embedded and never summarized, by construction. Pruning (`_prune`, `prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — so files that newly match an ignore pattern leave the index on the next sync, exactly like the A9 out-of-scope-junk behavior. The Sources page is admin-gated (phase 16 pattern); env-fallback rows (`from_env: true`, `id: null`) have no DB row to store a list on.
## Objective
Each stored source carries an **ignore list** of source-relative path prefixes, edited in a per-row box on the Sources page; matching is pure prefix (no mid-path matching, no globs — the spec's examples verbatim); ignored files are never walked, embedded, or summarized, and previously indexed files that newly match an ignore pattern are pruned on the next sync. All three import entry points honor the lists; the API is admin-only like the rest of the router.
## Dependencies
- `88_mobile_chat_hamburger_boot` (complete) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/upload/CLI pipelines, and the Sources view — none of which phase 88's pins reach; its suites must stay green unchanged).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Match semantics (locked, A1).** One pure helper pair in `app/rag/importer.py`:
- `normalize_ignore_path(entry: str) -> str` — `entry.strip().strip("/")` (whitespace + all leading/trailing slashes — so `"/my/files/"`, `"my/files/"`, and `"my/files"` normalize to the same `"my/files"`; `"//"` and `""` normalize to `""` and are dropped).
- `is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool` — `any(rel.startswith(p) for p in prefixes)` where `rel` is the **source-relative POSIX path without a leading slash** (`path.relative_to(root).as_posix()` — the same string `documents.path` stores) and `prefixes` are the normalized, non-empty entries.
- **Raw string prefix — deliberately no component-boundary check** (A1, owner-confirmed): `"my/files"` also matches `"my/files2/x.md"`; `"myfile.txt"` matches `"myfile.txt"` (root-level file) but NOT `"some/path/myfile.txt"` (no mid-path matching — the string simply does not start with the pattern). No globs, no `**`, no case folding beyond what the walk already does for extensions.
- The importer normalizes its input itself (single choke point — callers may pass raw box lines): an internal `_ignore_for_root(root, ignore_by_root)` yields `tuple(p for p in (normalize_ignore_path(x) for x in raw) if p)`.
- **`iter_importable_files` / `import_sources` signature (task 02).**
- `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore: tuple[str, ...] = ())` — one more skip: `if ignore and is_ignored(rel.as_posix(), ignore): continue` (after the hidden/excluded check, before/after the extension check — either order is fine; keep the existing checks byte-identical). Default `()` keeps every existing caller (CLI manual `--source`, tests) behaving exactly as before.
- `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root: dict[str, list[str]] | None = None)` — the map is keyed by **`str(root)`** (the root path string exactly as passed in `sources` — unambiguous even when two rows resolve to the same source *name*; on a root-string collision the caller has already merged the lists, Design "Callers" below). The pre-walk that computes the phase-64 progress `total` and the processing loop MUST use the same per-root ignore tuple (same helper), so `files_total` never counts ignored files. `seen` is untouched in shape → `_prune` prunes newly-ignored documents automatically (A2).
- **Callers (task 04).**
- `app/api/sync.py::_run_sync` — while building `sources` per row, build `ignore_by_root: dict[str, list[str]]` in the same loop: git row → key `str(sources_root / repo_name(row.url))`, local row → key `str(Path(row.path or row.url).expanduser())`; value `row.ignore_paths` — **extend on key collision** (two rows sharing a root string get the union of their lists). Pass `ignore_by_root=…` to `import_sources`.
- `app/api/git_sources.py::_run_upload` — step 6 already fetches the row by path (existing or the newly created one); capture it and pass `ignore_by_root={str(final_dir): row.ignore_paths}` to the step-8 `import_sources` (a re-upload of an existing source respects the list the owner already saved).
- `scripts/import_docs.py` — `_resolve_sources` returns `tuple[list[Path], dict[str, list[str]]]` (manual `--source` → `({},)` empty map — manual dirs have no row; env-fallback rows have no lists). `main` unpacks and passes the map to `import_sources`; docstrings updated (module + `_resolve_sources`).
- **Storage (task 01).** `git_sources.ignore_paths` — JSONB NOT NULL, server default `'[]'`, `Mapped[list]` (the `saved_chats.messages` JSONB precedent, `app/models.py`). Alembic `0013_git_source_ignore_paths.py` (revises `0012`) — `op.add_column("git_sources", sa.Column("ignore_paths", postgresql.JSONB(astext_type=sa.Text()), server_default=sa.text("'[]'"), nullable=False))`; downgrade drops the column. Existing rows read `[]`.
- **API contract (task 03).**
- Schemas (`app/schemas.py`): `GitSourceIn.ignore_paths: list[str] | None = None` (create-time, optional — absent → `[]`); `GitSourceOut.ignore_paths: list[str]`; `GitSourceRow.ignore_paths: list[str]` (env-fallback rows report `[]`); new `GitSourceIgnoreIn` body model for PATCH with `ignore_paths: list[str]` (required — the box replaces the whole list, A5).
- `GET /api/git-sources` — each DB row reports its (normalized, non-empty) list; env rows `[]`.
- `POST /api/git-sources` — both kinds accept `ignore_paths`; stored **normalized** (each entry `normalize_ignore_path`, empties dropped) — the UI may send raw box lines and the stored list is canonical.
- `PATCH /api/git-sources/{source_id}` (NEW, behind the existing `require_admin` router dependency) — 404 unknown id; **replace** semantics (A5): the body list, normalized, becomes the row's list (empty list = clear all); 200 → `GitSourceOut` (id, url, added_at, ignore_paths).
- Validation (422, fixed details that never echo the input — the router's credential-safety discipline, applied for consistency): at most **200 entries** (A4) → `detail="a source has at most 200 ignore paths"`; an entry **empty after normalization** → `detail="ignore paths must be non-empty"`; an entry **>500 chars** (pre- or post-normalization — check post-normalization) → `detail="an ignore path exceeds 500 characters"`.
- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: an **"Ignore paths" button** (actions cell, left of Remove — same `.git-source-remove`-style button idiom, new class `.git-source-ignore`) whose `aria-label` is `Edit ignored paths for source: <value>`; when `s.ignore_paths.length > 0` the row also shows a text count tag `N ignored` (text, never color alone) next to the location cell. Env-fallback rows (`id` null) get **no** button — the existing "from .env" tag stays (A3).
- The button opens a **page-local editor dialog** following the phase-69 `#remove-confirm-dialog` pattern exactly (new `#ignore-editor-dialog`, `role="dialog"`, dim backdrop, focus management: focus lands on Cancel — the safe default — Escape / Cancel / backdrop close as CANCEL and return focus to the trigger; the dim backdrop click handler is the same idiom). Inside: a visible `<label for="ignore-editor-textarea">` ("Ignored files and folders — one path per line"), a `<textarea id="ignore-editor-textarea">` (mono, ~6 rows, `spellcheck="false"`, placeholder `my/files/`), a helper line explaining the prefix rule in plain words ("A file is ignored when its path in the source starts with any of these. No wildcards."), Save + Cancel buttons, and a `role="alert"` error line (server 422 details, same rendering as the remove modal's error line).
- `frontend/assets/git-sources.js` wiring (all scoped to the view root like everything else in the module): `openIgnoreEditor(s, triggerBtn)` — prefills `s.ignore_paths.join("\n")`; `saveIgnorePaths()` — the **§7.4 never-stale lifecycle**: Save disables + relabels "Saving…" while the PATCH is out, re-enables on success AND failure; on 200 → close the dialog, `loadSources()`, `announce("Ignored paths updated for <value>")`; on non-2xx → the server detail in the `role="alert"` line, textarea content KEPT (the instruction survives, tuning-form convention). Lines are parsed client-side: split on `\n`, trim, **drop empty lines** (a blank line in the box is a separator, not an entry) before submitting.
- `frontend/index.html` — the dialog skeleton inside `#view-git-sources` (next to `#remove-confirm-dialog`); `frontend/assets/styles.css` — dialog/textarea/button/count-tag rules reusing the remove-confirm-dialog and `.git-source-remove` idioms (house comment style, citing phase 89).
- **NOT touched:** the RAG view (`sources.js` — the catalog stays as-is; the Sync button already exists there), `app/rag/retriever.py`, the chunker, `mock_llm.py`/`slow_llm.py`, chat/SSE, `AGENTS.md`, any completed phase.
## Tasks
1. `01_ignore_paths_column.md` — `git_sources.ignore_paths` JSONB column (model + alembic `0013`) + unit tests.
2. `02_ignore_prefix_matching.md` — the pure matcher + `iter_importable_files`/`import_sources` `ignore_by_root` support (walk + progress pre-walk + prune interaction) + unit & integration tests.
3. `03_ignore_paths_api.md` — schemas + `GET`/`POST` fields + new admin-only `PATCH /api/git-sources/{id}` (replace semantics, 404/422 contracts) + integration tests.
4. `04_ignore_paths_in_import_pipelines.md` — wire the per-row lists into `_run_sync`, the `_run_upload` background scan, and `scripts/import_docs.py` + integration tests.
5. `05_ignore_paths_sources_ui.md` — the per-row "Ignore paths" box on the Sources page (dialog + §7.4 save lifecycle + a11y) + source-level unit pins.
6. `06_e2e_ignore_paths.md` — dedicated Playwright suite `tests/e2e/test_source_ignore_paths.py` (run in isolation), regressions, full gate, atomic commit.
## Testing & Quality
- Unit — `tests/unit/test_importer_ignore.py` (new, tasks 01–02): normalization table, the spec's prefix examples verbatim, no-mid-path, raw-prefix A1 edge, empty/drop rules, `iter_importable_files` on a tmp fixture tree; `tests/unit/test_source_ignore_paths.py` (new, task 05): source-level pins for the JS/HTML wiring (house pattern — read the assets as text).
- Integration — `tests/integration/test_importer_ignore.py` (new, task 02): `import_sources` against a fixture dir — ignored files produce no `Document`/`Chunk` rows (not embedded, not summarized), previously indexed file newly ignored → pruned, progress `total` excludes ignored files, sources not in the map behave exactly as before; `tests/integration/test_git_sources_api.py` (extended, task 03) and `tests/integration/test_sync_api.py` + the CLI/upload suites (extended, task 04).
- E2E (mandatory, A16) — `tests/e2e/test_source_ignore_paths.py` (task 06), run in isolation with the DB up: `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov`.
- Coverage: **>90%** on `app/` (the validate.sh gate — the importer/API additions are fully unit+integration covered).
## Completion Criteria
- [ ] A stored source's ignore list (git or local row) is editable in the box on the Sources page: one path per line; Save → `PATCH` 200 → the row shows the `N ignored` count; the list round-trips through `GET /api/git-sources`.
- [ ] A sync (button, upload scan, or CLI) with `ignore_paths: ["ignore/"]` (any of the three spellings) indexes nothing under `ignore/` — no `documents`/`chunks` rows, no embeddings, no summary calls for those files — and a previously indexed file under `ignore/` is pruned from the KB on that sync (the RAG catalog no longer lists it).
- [ ] The spec's matching examples hold end-to-end: `myfile.txt` ignores the root-level `myfile.txt` only, never `some/path/myfile.txt` (E2E proof).
- [ ] `PATCH` contracts hold: 404 unknown id; 422 fixed details for >200 entries / empty entry / >500-char entry; replace semantics (clearing works); anonymous 403.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` green in isolation (DB up); regression suites `test_git_sources_admin.py`, `test_archive_upload_sources.py`, `test_sync_button.py`, `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — raw prefix semantics (owner-confirmed 2026-09-08).** Normalize = trim whitespace + strip leading/trailing `/`; match = `rel.startswith(normalized)` on the source-relative POSIX path. No component boundary (`"my/files"` also ignores `"my/files2/x.md"`), no globs, no mid-path matching.
- **A2 — previously indexed files leave the index (owner-confirmed).** A file that newly matches an ignore pattern is pruned from the KB on the next sync (the `seen`-set mechanism, same as A9 out-of-scope junk) — "not embedded, not summarized" also cleans the existing index, not just future imports.
- **A3 — box placement (owner-confirmed).** The editor is a per-row control on the Sources page (`/git-sources.html`) for stored rows only; env-fallback rows have no DB row and get no editor.
- **A4 — limits (owner-confirmed).** At most 200 entries per source, each ≤500 chars after normalization; entries empty after normalization are rejected with a fixed 422 detail (the UI drops blank lines client-side; the API stays defensive).
- **A5 — replace semantics (owner-confirmed).** Saving the box replaces the whole list (textarea content = the list; empty box clears).
## Commit
```bash
git add app/ alembic/versions/0013_git_source_ignore_paths.py frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source ignore paths — prefix-excluded files are never embedded, summarized, or re-indexed"
```
.agents/phases/complete/89_source_ignore_paths/01_ignore_paths_column.md
+53
View File
@@ -0,0 +1,53 @@
# Task 01 — `git_sources.ignore_paths` column (model + migration 0013)
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "Each source should support a list of ignored files and folders."
**Story:** n/a (TODO-derived).
## Objective
Give every `git_sources` row a JSONB `ignore_paths` list (default empty) so the Sources page can store one ignore list per source — the storage half of the feature; no importer or API behavior changes in this task.
## Work
1. `app/models.py` — `GitSource` (between `SourcesMeta` and `DocDraft`): add the column after `path`, with a docstring comment in the house style (cite phase 89 + the A1 normalization):
```python
#: Per-source ignore paths (phase 89, A1/A4): the normalized,
#: non-empty source-relative path prefixes the owner types into the
#: box on the Sources page. A file is ignored when its
#: source-relative POSIX path starts with any entry (raw prefix —
#: no mid-path matching, no globs). Server default '[]' — every
#: pre-phase-89 row imports exactly as before.
ignore_paths: Mapped[list] = mapped_column(
JSONB, default=list, server_default=text("'[]'"), nullable=False
)
```
(`app/models.py` imports named symbols from `sqlalchemy` — add `text` to that existing import list; `JSONB` is already imported from `sqlalchemy.dialects.postgresql`.) Update the module docstring's `git_sources` bullet with one sentence: "``ignore_paths`` (phase 89 — JSONB list of normalized path prefixes, server default ``'[]'``)".
2. `alembic/versions/0013_git_source_ignore_paths.py` (NEW — house header style, mirror `0007_git_sources_kind.py`'s docstring shape: Revision ID `0013`, `Revises: 0012`, one-sentence rationale per column):
```python
def upgrade() -> None:
op.add_column(
"git_sources",
sa.Column(
"ignore_paths",
postgresql.JSONB(astext_type=sa.Text()),
server_default=sa.text("'[]'"),
nullable=False,
),
)
def downgrade() -> None:
op.drop_column("git_sources", "ignore_paths")
```
3. `tests/unit/test_models_git_source.py` (extend the existing model unit tests if one exists — check `tests/unit/` for a models test module first; otherwise create it) — or a focused new module `tests/unit/test_ignore_paths_column.py`:
- the `GitSource.__table__.c.ignore_paths` column exists, is NOT NULL, and its server default is `'[]'` (assert `col.server_default.arg.text == "'[]'"` and `col.nullable is False`);
- a freshly constructed `GitSource(url="https://example.com/r.git", kind="git")` has `ignore_paths == []` (the Python-side default) — the ORM default path, no DB needed.
4. Apply the migration to the dev DB so later tasks run against the real schema: `uv run alembic upgrade head` (record the head revision in the task report; the E2E/integration suites use the same DB via `podman compose up -d db`).
## Testing & Quality
- `uv run pytest tests/unit/ -q` green (the new pins + the untouched suite).
- Coverage: **>90%** on `app/` unaffected (model column + migration are config, not logic).
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies `0013` cleanly; `alembic downgrade -1 && alembic upgrade head` round-trips (reversible — the 0007 precedent).
- [ ] Existing `git_sources` rows (dev DB) read `ignore_paths == []` — no importer/API change yet: `uv run pytest tests/integration/test_git_sources_api.py -q` green UNCHANGED.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] `git diff --stat` limited to `app/models.py`, the new migration, and the unit test.
.agents/phases/complete/89_source_ignore_paths/02_ignore_prefix_matching.md
+88
View File
@@ -0,0 +1,88 @@
# Task 02 — Prefix matching + `ignore_by_root` in the importer
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "These ignored files/folders should not be embedded and should not be summarized… \"/my/files/\" or \"my/files/\" or \"my/files\" would ignore all files that start with the prefix \"my/files\"… \"myfile.txt\" would match \"/myfile.txt\" but not \"some/path/myfile.txt\". The ignore is just a prefix ignore."
**Story:** n/a (TODO-derived).
## Objective
Make the import walk honor per-source ignore lists: ignored files are never indexed (hence never embedded, never summarized), the phase-64 progress `total` never counts them, and previously indexed files that newly match an ignore pattern are pruned on the next `prune=True` run (A2) — while every caller that passes no map behaves byte-identically to today.
## Work
1. `app/rag/importer.py` — add the pure helpers (module level, near `iter_importable_files`; docstrings in the house style, each citing the spec example it encodes):
```python
def normalize_ignore_path(entry: str) -> str:
"""One ignore-path entry → canonical form (phase 89, A1).
Trim surrounding whitespace, then strip ALL leading/trailing
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
all become ``"my/files"``. ``""`` / ``"//"``,
``" "`` normalize to ``""`` (callers drop empties).
"""
return entry.strip().strip("/")
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
"""Phase 89, A1 — the pure prefix rule, nothing else.
``rel`` is the source-relative POSIX path WITHOUT a leading
slash (the ``documents.path`` string). Match = ``rel`` STARTS
WITH a normalized entry: raw string prefix — deliberately NO
component-boundary check (``"my/files"`` also matches
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
"""
return any(rel.startswith(p) for p in prefixes)
def _ignore_for_root(
root: Path, ignore_by_root: dict[str, list[str]] | None
) -> tuple[str, ...]:
"""The normalized, non-empty prefix tuple for one root (phase 89).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (unambiguous when two rows share a
source *name* but different dirs). Callers may pass RAW box
lines: the importer normalizes + drops empties here, the single
choke point — stored lists (already normalized) normalize to
themselves.
"""
raw = (ignore_by_root or {}).get(str(root)) or []
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
```
2. `app/rag/importer.py` — `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore: tuple[str, ...] = ())`: inside the `for path in sorted(root.rglob("*"))` loop, after the hidden/`excluded`-parts check and before/after the extension check (your choice — keep both existing checks byte-identical), add:
```python
if ignore and is_ignored(rel.as_posix(), ignore):
continue
```
Update the docstring: one sentence on the `ignore` param (phase 89, A1; default `()` = no behavior change).
3. `app/rag/importer.py` — `import_sources(…, ignore_by_root: dict[str, list[str]] | None = None)` (new keyword-only param, default `None`):
- compute `ignore_by_root` once per root inside the `for root in sources:` loop: `ignore = _ignore_for_root(root, ignore_by_root)`;
- the phase-64 pre-walk (`if progress is not None:` block) must use it — `iter_importable_files(root, llm.settings.import_extension_set, ignore=ignore_for_this_root)` (compute the tuple in the pre-walk too, or hoist: the pre-walk runs before the loop, so call `_ignore_for_root` there as well);
- the processing loop passes the same tuple to `iter_importable_files`;
- `seen`, `_prune`, and everything else are UNCHANGED — newly-ignored files simply never enter `seen`, so the next `prune=True` run deletes their rows (A2, the A9 junk-precedent);
- docstring: one paragraph — the map is keyed by `str(root)`, the pre-walk and the loop use the same tuple (so `files_total` excludes ignored files), and pruning follows automatically.
4. `tests/unit/test_importer_ignore.py` (NEW — the spec is the test table; import the three helpers from `app.rag.importer`):
- **normalize:** `"/my/files/"`, `"my/files/"`, `"my/files"`, `" my/files "` all → `"my/files"`; `"//"` → `""`; `""` → `""`; `" "` → `""`.
- **is_ignored (spec examples verbatim):** prefixes `("my/files",)` ignore `my/files/a.md`, `my/files/sub/b.md`, and `my/files` (a file literally named that); prefixes `("myfile.txt",)` ignore `myfile.txt` but NOT `some/path/myfile.txt` (no mid-path matching — the string does not start with the pattern) and NOT `xmyfile.txt` (no suffix/partial matching either). Also `("files",)` ignores `files/x.md` (a root-level dir — that IS prefix matching) but not `some/files/x.md`.
- **raw-prefix A1 edge (documented, owner-confirmed):** `("my/files",)` also ignores `my/files2/x.md`; `("a",)` ignores `ab.md` (raw string startswith — no component boundary).
- **empty prefixes:** `is_ignored(anything, ())` → False.
- **iter_importable_files:** a tmp_path fixture tree (`root/keep.md`, `root/ignore/secret.md`, `root/ignore/deep/x.yaml`, `root/notes.txt`, `root/myfiles.md`) with `extensions=frozenset({".md", ".txt", ".yaml"})`: default `ignore=()` → all five; `ignore=("ignore",)` → only `keep.md`, `notes.txt`, `myfiles.md`; `ignore=("//ignore/ ")` → same (normalization is the caller's job at this level — this function receives ALREADY-normalized tuples; pin that contract: passing raw `"//ignore/ "` does NOT match, proving normalization happens in `_ignore_for_root`, not here).
- **_ignore_for_root:** `{"str(root)": ["/a/", "b//", "", " "]} ` → `("a", "b")`; missing root → `()`; `None` map → `()`.
5. `tests/integration/test_importer_ignore.py` (NEW — mirror the fixture-tree + mock-LLM pattern of `tests/integration/test_importer_e2e.py`; use the session-scoped test DB and a `tmp_path` source dir named `IgnoreFix`):
- **never indexed:** tree `keep.md` + `ignore/secret.md` + `ignore/notes.txt` + `top.txt`; run `import_sources([root], fake_llm, ignore_by_root={str(root): ["ignore/"]})` → `ImportSummary.files == 2`; `documents` has exactly `keep.md` + `top.txt`; NO `chunks` row for `ignore/secret.md` (not embedded) and its `summary` column is NULL (not summarized); the fake LLM's `chat` call count (summary) excludes it.
- **prune (A2):** re-run with `prune=True` and a file previously indexed — pre-import `ignore/secret.md` WITHOUT the map (so it is indexed), then re-run WITH `ignore_by_root={str(root): ["ignore"]}` (no trailing slash — A1 normalization) → `summary.pruned == 1` and the doc row is gone from the DB.
- **progress pre-walk excludes ignored:** pass a `progress` hook + the map → `total` == count of non-ignored files only (2 in the first scenario).
- **no-map regression:** the same tree, `ignore_by_root=None` → all four files indexed (byte-identical behavior), `summary.files == 4`.
- **unlisted source unaffected:** two roots, map keys only the first → the second root imports everything.
- Clean up `documents` rows of the fixture source between tests (the house cleanup pattern in `test_importer_e2e.py`).
## Testing & Quality
- `uv run pytest tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -v` green.
- `uv run pytest -q` green UNCHANGED (every existing importer/sync/upload/CLI test passes no map — the default `()`/`None` contract).
- Coverage: **>90%** on `app/` (the helpers are exercised by the unit table; the importer branches by the integration suite).
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] The spec's own examples pass as unit tests verbatim (`/my/files/` ≡ `my/files/` ≡ `my/files`; `myfile.txt` matches root-level only, never mid-path).
- [ ] An ignored file yields no `documents` row, no `chunks` rows, no embedding call, and no summary call; a previously indexed file matching a new pattern is pruned on the next `prune=True` run.
- [ ] `files_total` (progress hook) never counts ignored files.
- [ ] `git diff --stat` limited to `app/rag/importer.py` + the two new test modules.
.agents/phases/complete/89_source_ignore_paths/03_ignore_paths_api.md
+75
View File
@@ -0,0 +1,75 @@
# Task 03 — API: expose + update ignore lists (GET / POST / PATCH)
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "They should be able to type these files and folders into a box on the sources page. Each source should support a list of ignored files and folders."
**Story:** n/a (TODO-derived).
## Objective
The admin-only sources API carries the ignore list: `GET` reports it per row, `POST` stores it at create time, and a new `PATCH /api/git-sources/{source_id}` replaces it (A5) — with the fixed-detail 422 validation of the A4 limits. No importer behavior changes here (the pipelines are wired in task 04).
## Work
1. `app/schemas.py` — extend the git-source schemas (house docstrings, cite phase 89):
- `GitSourceIn`: `ignore_paths: list[str] | None = Field(default=None)` — optional at create time (absent → `[]`); entries are the RAW box lines (trimmed/normalized by the API layer, not the schema — the 422 details must stay fixed strings, the router's credential-safety discipline applied for consistency).
- `GitSourceOut`: `ignore_paths: list[str]` (the stored, normalized list — non-null).
- `GitSourceRow`: `ignore_paths: list[str]` (env-fallback rows report `[]`).
- NEW `GitSourceIgnoreIn(BaseModel)`: `ignore_paths: list[str]` — the `PATCH` body; the list is REQUIRED (replace semantics, A5 — an absent field is a FastAPI 422 with the model's own detail; keep the A4 422s for the *value* checks).
2. `app/api/git_sources.py`:
- a module-level validation helper (private, near `_commit_new`), shared by POST and PATCH:
```python
def _validate_ignore_paths(raw: list[str] | None) -> list[str]:
"""Normalize + enforce the A4 limits; fixed details never echo input."""
entries = [normalize_ignore_path(e) for e in (raw or [])]
if any(not e for e in entries):
raise HTTPException(status_code=422, detail="ignore paths must be non-empty")
if len(entries) > MAX_IGNORE_PATHS: # = 200 (A4)
raise HTTPException(status_code=422, detail="a source has at most 200 ignore paths")
if any(len(e) > 500 for e in entries):
raise HTTPException(status_code=422, detail="an ignore path exceeds 500 characters")
return entries
```
(Import `normalize_ignore_path` from `app.rag.importer` — task 02 owns the helper; define `MAX_IGNORE_PATHS = 200` and the per-entry cap `500` as module constants with an A4 comment. NOTE the order: empty check FIRST — a whitespace-only entry must 422, not silently drop.)
- `list_git_sources` (GET): DB rows → `GitSourceRow(…, ignore_paths=row.ignore_paths or [])`; env-fallback rows → `ignore_paths=[]`.
- `_create_git_row` / `_create_local_row` (POST): `row = GitSource(…, ignore_paths=_validate_ignore_paths(payload.ignore_paths))` (both kinds — the stored value is always the normalized list).
- `GitSourceOut` responses in `create_git_source` gain `ignore_paths=row.ignore_paths`.
- NEW route (place it after the POST, before `/upload` — pathless params keep the `/upload` and `/upload/status` routes matching their own literal prefixes first; FastAPI matches in declaration order, so the `PATCH /{source_id}` declaration must NOT shadow anything — PATCH and POST/GET are different methods, and DELETE `/{source_id}` already coexists, so a plain `@router.patch("/{source_id}", response_model=GitSourceOut)` is safe):
```python
@router.patch("/{source_id}", response_model=GitSourceOut)
def patch_git_source(
source_id: uuid.UUID,
payload: GitSourceIgnoreIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Replace one source's ignore list (phase 89, A5).
404 unknown id. The body list (required) is normalized +
A4-validated (fixed 422 details) and REPLACES the row's list
wholesale — an empty list clears all. Returns the updated
row's public shape (id, url, added_at, ignore_paths).
"""
row = db.get(GitSource, source_id)
if row is None:
raise HTTPException(status_code=404, detail="git source not found")
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
db.commit()
db.refresh(row)
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at,
ignore_paths=row.ignore_paths)
```
- Router module docstring: add the PATCH route to the "Routes:" paragraph (house style — the existing paragraph already lists every route).
3. `tests/integration/test_git_sources_api.py` — extend (follow the existing test module's fixture/auth pattern — admin vs anonymous, DB truncation between tests):
- GET: a row created via POST without `ignore_paths` reports `ignore_paths == []`; env-fallback rows (empty table + `BOR_GIT_SOURCES` env) report `[]` per row and `from_env: true`.
- POST: `ignore_paths: ["/my/files/", " my/files2 ", "x"]` → 201, stored list `["my/files", "my/files2", "x"]` (normalized round-trip through GET); both kinds (git + local) accept it.
- PATCH: 200 replace semantics — set `["a/", "b"]` → GET shows `["a", "b"]`; then PATCH `[]` → GET shows `[]` (clear works); the row's `id`/`url`/`added_at` are unchanged; response shape is `GitSourceOut` incl. the new list.
- PATCH 404 on an unknown uuid; anonymous PATCH → 403 (the router dependency).
- PATCH 422s (fixed details, asserted as exact strings): `[" "]` → "ignore paths must be non-empty"; 201 entries → "a source has at most 200 ignore paths"; `["a" * 501]` → "an ignore path exceeds 500 characters"; 500-char entry is ACCEPTED (boundary).
- regression: every pre-existing test in the module passes UNCHANGED (the new fields are additive on the response models).
## Testing & Quality
- `uv run pytest tests/integration/test_git_sources_api.py -v` green (new + untouched tests).
- `uv run pytest -q` green (no pipeline wiring yet — the lists are stored but not yet consulted by imports).
- Coverage: **>90%** on `app/` (every new branch — 404/422/replace — is a tested path).
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `GET /api/git-sources` reports `ignore_paths` on every row (DB rows: their list; env rows: `[]`).
- [ ] `POST` stores the normalized list for both kinds; `PATCH /api/git-sources/{id}` replaces it (including clearing to `[]`), 404/403/422 contracts hold with the exact fixed details above.
- [ ] `git diff --stat` limited to `app/schemas.py`, `app/api/git_sources.py`, and the integration test module.
.agents/phases/complete/89_source_ignore_paths/04_ignore_paths_in_import_pipelines.md
+57
View File
@@ -0,0 +1,57 @@
# Task 04 — Wire per-row ignore lists into all three import entry points
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "The user should be able to ignore/exclude specific files and folders when importing from a source… These ignored files/folders should not be embedded and should not be summarized."
**Story:** n/a (TODO-derived).
## Objective
Every import entry point consults its source rows' ignore lists: the in-app Sync button, the archive-upload background scan, and the CLI. Rows with empty lists and manual `--source` dirs behave exactly as before.
## Work
1. `app/api/sync.py::_run_sync` — while the per-row loop builds `sources` (the `for row in rows:` block that appends `clone_or_pull(…)` for git rows and the re-verified dir for local rows), build the map in the SAME loop, per the phase overview's Design/Callers:
```python
ignore_by_root: dict[str, list[str]] = {}
…
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
else:
root = Path(row.path or row.url).expanduser()
if not root.is_dir():
raise GitSyncError(f"local source missing: {root}")
sources.append(root)
# phase 89: the row's ignore list, keyed by the SAME root string
# the importer sees; two rows sharing a root string get the
# union (extend, not replace) — the sibling/repo-name edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
```
(Refactor the existing loop body to the `root = …` shape above — behavior byte-identical; the existing re-verify-`.is_dir()` raise and its message are kept EXACTLY.) Then: `summary = await import_sources(sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root)`. Update the module docstring's pipeline step 4 with a clause: "honoring each row's `ignore_paths` (phase 89)".
2. `app/api/git_sources.py::_run_upload` — step 6 already opens a short-lived session and upserts the row by path; CAPTURE the row object on both branches (existing row from the `db.scalar(…)`; the newly created `GitSource` instance before `db.close()`) and keep it out of the session (the attributes are plain values — copy `row.ignore_paths or []` into a local `ignore_paths: list[str]`). Step 8 becomes:
```python
summary = await import_sources(
[final_dir], llm, prune=True, progress=_hook,
ignore_by_root={str(final_dir): ignore_paths},
)
```
Update the route's numbered docstring (steps 6/8) with one clause each (phase 89: a re-upload of an existing source honors the ignore list the owner already saved).
3. `scripts/import_docs.py` — `_resolve_sources` returns `tuple[list[Path], dict[str, list[str]]]`:
- manual `--source` branch → `([expanded paths], {})` (manual dirs have no row — no ignore applies);
- rows branch: same `root = …` per-row shape as `_run_sync`, building `ignore_by_root` with the same `setdefault(…).extend(…)` collision rule; return `(sources, ignore_by_root)`;
- legacy `DEFAULT_SOURCES` fallback → `([…], {})`;
- module + `_resolve_sources` docstrings: one sentence each (phase 89 — the map is keyed by the resolved root string; manual/fallback paths import with no ignore).
- `main`: `sources, ignore_by_root = _resolve_sources(args.source, settings)` and `import_sources(sources, llm, prune=args.prune, limit=args.limit, ignore_by_root=ignore_by_root)` (inside the `_run` closure — the variable is captured like `args` already is).
4. `tests/integration/test_sync_api.py` — extend (follow the module's existing local-source + mock-LLM fixture pattern; check how it seeds a `kind='local'` row and drives `POST /api/sync` + status polling):
- a local fixture dir with `keep.md` + `ignore/secret.md`; seed a `kind='local'` row with `ignore_paths=["ignore/"]`; run the sync to `success`; `documents` contains `keep.md`, NOT `ignore/secret.md`; the success `detail` counts are consistent (`files` excludes the ignored file).
- regression: the module's existing tests pass UNCHANGED (rows with `[]` lists → byte-identical sync).
5. `tests/integration/test_import_docs_git.py` (or `test_importer_e2e.py` if the CLI driver helper lives there — check which module already drives `import_docs.main` and extend THAT) — a rows-branch run with a `kind='local'` row carrying `ignore_paths` → the CLI import skips the ignored file (summary/files assertions); a `--source` manual run is unaffected (no map).
6. `tests/integration/test_git_sources_upload.py` — extend with the re-upload-respects-lists case: upload an archive (the module's existing fixture pattern) whose unpacked tree includes an ignored path; after the first scan, set the row's `ignore_paths` (PATCH or direct DB update — use the API), re-upload a same-name archive containing the file again → the background run lands `success` with the file NOT indexed; a second assertion: an upload of a NEW source name with no list imports everything (regression).
## Testing & Quality
- `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py tests/integration/test_git_sources_upload.py -v` green (new + existing).
- `uv run pytest -q` green; coverage **>90%** on `app/` (the new branches — map build, collision extend, upload capture — are covered by the integration tests; the CLI path by its test).
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `POST /api/sync` (button), the upload background scan, and `python -m scripts.import_docs` all skip the listed prefixes for their rows; nothing changes for empty lists or manual dirs.
- [ ] A re-uploaded existing source honors the ignore list saved on its row.
- [ ] `git diff --stat` limited to `app/api/sync.py`, `app/api/git_sources.py`, `scripts/import_docs.py`, and the extended integration modules.
.agents/phases/complete/89_source_ignore_paths/05_ignore_paths_sources_ui.md
+77
View File
@@ -0,0 +1,77 @@
# Task 05 — The per-row "Ignore paths" box on the Sources page
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "They should be able to type these files and folders into a box on the sources page."
**Story:** n/a (TODO-derived).
## Objective
A per-row "Ignore paths" control on the Sources page (`/git-sources.html`): a page-local editor dialog (the phase-69 `#remove-confirm-dialog` pattern) with a one-path-per-line textarea, the §7.4 never-stale save lifecycle, and full a11y (visible label, focus management, Escape, role=alert errors) — plus a `N ignored` count tag on rows that have a list.
## Work
1. `frontend/index.html` — inside `#view-git-sources` → `#git-sources-content`, directly AFTER the `#remove-confirm-dialog` block (the markup above — same placement logic), add the editor dialog (house comment header citing phase 89; ids/classes are the ones the JS + CSS + unit pins below key on):
```html
<!-- Phase 89: the per-source ignore-paths editor — the EXACT
#remove-confirm-dialog pattern (phase 69): page-local
alertdialog, dim backdrop, focus on Cancel (the safe default),
Escape/backdrop close as CANCEL, focus returns to the row's
trigger button. The box itself: one path per line (prefix
semantics — the phase-89 A1 rule, stated in plain words in
the helper line); Save runs the §7.4 never-stale lifecycle
(git-sources.js: "Saving…" while the PATCH is out). -->
<div class="ignore-editor" id="ignore-editor-dialog" role="alertdialog"
aria-modal="true" aria-labelledby="ignore-editor-title"
aria-describedby="ignore-editor-copy" hidden>
<div class="ignore-editor-backdrop" aria-hidden="true"></div>
<div class="ignore-editor-panel">
<h2 class="ignore-editor-title" id="ignore-editor-title">Ignored files and folders</h2>
<code class="ignore-editor-source" id="ignore-editor-source"></code>
<p class="ignore-editor-copy" id="ignore-editor-copy">
One path per line. A file is ignored when its path in the
source starts with any of these — <code>/my/files/</code>,
<code>my/files/</code> and <code>my/files</code> mean the same
thing. No wildcards, no matching in the middle of a path.
</p>
<label class="ignore-editor-label" for="ignore-editor-textarea">Ignored paths</label>
<textarea class="ignore-editor-textarea" id="ignore-editor-textarea"
rows="6" spellcheck="false"
placeholder="my/files/"></textarea>
<p class="ignore-editor-error" id="ignore-editor-error" role="alert" hidden></p>
<div class="ignore-editor-actions">
<button type="button" class="ignore-editor-btn ignore-editor-cancel"
id="ignore-editor-cancel">Cancel</button>
<button type="button" class="ignore-editor-btn ignore-editor-save"
id="ignore-editor-save">Save</button>
</div>
</div>
</div>
```
2. `frontend/assets/git-sources.js` — the wiring (every lookup scoped to the view `root` like the rest of the module; the module header docstring gains a phase-89 bullet in the house style):
- module-scope refs next to the remove-dialog refs: `ignoreDialog`, `ignoreSourceEl` (`#ignore-editor-source`), `ignoreTextarea` (`#ignore-editor-textarea`), `ignoreErrorEl` (`#ignore-editor-error`), `ignoreSaveBtn` (`#ignore-editor-save`), `ignoreCancelBtn` (`#ignore-editor-cancel`), `ignoreBackdrop` (`.ignore-editor-backdrop`); plus `let ignoreTarget = null` (the row object) and `let ignoreTriggerBtn = null` (for focus return).
- `makeRow(s)` — in the `s.id` branch of the actions cell, BEFORE the Remove button: an "Ignore paths" button (`type="button"`, class `git-source-ignore`, `aria-label` = `` `Edit ignored paths for ${kindLabel} source: ${value}` ``, built with `textContent`-safe content — the label is the only place `value` appears; follow the Remove button's `innerHTML` icon+text idiom ONLY if you reuse a static icon constant — never inject `value` via innerHTML). Its click → `openIgnoreEditor(s, btn)`. When `s.id && s.ignore_paths.length > 0`: append a count tag to the LOCATION cell (`urlTd`), after the `<code>`: `<span class="git-source-ignore-count">N ignored</span>` (text via textContent; N = `s.ignore_paths.length` — text + a distinct background, never color alone, WCAG).
- `openIgnoreEditor(s, triggerBtn)`: set `ignoreTarget`/`ignoreTriggerBtn`; `ignoreSourceEl.textContent = value` (the same value expression as makeRow); `ignoreTextarea.value = (s.ignore_paths || []).join("\n")`; clear the error line (`hidden` + empty text); show the dialog (`hidden = false`); focus `ignoreCancelBtn` (the safe default — the remove-dialog precedent); bind Escape via a keydown listener on the dialog (see the remove dialog's `onRemoveDialogKeydown` shape: Escape → `closeIgnoreEditor()`, return focus to the trigger).
- `closeIgnoreEditor()`: hide the dialog, reset the textarea value + error, `ignoreTarget = ignoreTriggerBtn = null`, return focus to the trigger button.
- Cancel button + backdrop click → `closeIgnoreEditor()` (the remove-dialog wiring shape).
- `saveIgnorePaths()` — the §7.4 lifecycle:
- parse: `ignoreTextarea.value.split("\n").map(l => l.trim()).filter(Boolean)` (a blank line is a separator, not an entry — the server still rejects empty entries defensively);
- disable Save + Cancel, relabel Save "Saving…";
- `fetch(`/api/git-sources/${ignoreTarget.id}`, {method: "PATCH", headers: {"Content-Type": "application/json"}, body: JSON.stringify({ignore_paths: lines})})` with the module's existing cookie/auth behavior (plain fetch — same as the add/remove calls);
- 200 → `closeIgnoreEditor()` (focus return included), `await loadSources()` (the `N ignored` tag lands), `announce(`Ignored paths updated for ${value}`)` (the existing `#git-sources-announcer`);
- non-2xx → clear Save/Cancel disabled states, restore the "Save" label, show the server detail in `#ignore-editor-error` (via the existing `apiDetail(r, fallback)` helper — 422 shape-aware like the tuning forms), KEEP the textarea content (the instruction survives);
- network throw → the same error-line path (the module's existing try/catch idiom for the other requests).
- env-fallback rows (`s.id` null): NO ignore button (A3) — the existing "from .env" tag stays as-is.
3. `frontend/assets/styles.css` — the ignore-editor rules (place next to the `.remove-confirm` rules; house comment style citing phase 89): `.ignore-editor` mirrors `.remove-confirm` (fixed overlay, dim backdrop `.ignore-editor-backdrop` = the `.remove-confirm-backdrop` treatment, centered panel `.ignore-editor-panel` = `.remove-confirm-panel` sizing); `.ignore-editor-source` = the `.remove-confirm-source` mono treatment; `.ignore-editor-label` a visible block label (WCAG — never `aria-label`-only); `.ignore-editor-textarea` — full panel width, `min-height: ~9rem`, `font-family: mono stack` (the existing code/mono variable), theme-consistent border + focus-visible ring (3px, the house focus convention); `.ignore-editor-error` = `.remove-confirm-error` treatment; `.ignore-editor-btn` pair = `.remove-confirm-btn` (Cancel secondary / Save brand — 44px min targets, the ≥44px rule); the row button `.git-source-ignore` = the `.git-source-remove` idiom (same size/spacing, a neutral/secondary fill so it reads distinct from the destructive Remove — text label included, never icon-only); `.git-source-ignore-count` — a small inline tag next to the location `<code>` (text + distinct background, 4.5:1 contrast on both theme surfaces).
4. `tests/unit/test_source_ignore_paths.py` (NEW — house source-level pattern: read the assets as text via `Path(__file__).parents[2] / "frontend" / "assets" / …`; module docstring cites TODO.md L3 + phase 89):
- `index.html`: `#ignore-editor-dialog` exists with `role="alertdialog"`, `aria-modal="true"`, `aria-labelledby="ignore-editor-title"`; `#ignore-editor-textarea` has a visible `<label … for="ignore-editor-textarea">` (the `ignore-editor-label` class); the error line carries `role="alert"`.
- `git-sources.js`: the PATCH call exists (`method: "PATCH"` + `` `/api/git-sources/${` ``); the §7.4 lifecycle markers exist (the "Saving…" relabel + disabling of the save button — assert the string `"Saving…"` appears in the save function's region); the parse chain `split("\n")` + trim + `filter(Boolean)`; `ignore_paths` round-trip markers (the GET rows render `(s.ignore_paths || [])` and the count tag class `git-source-ignore-count`); the env-row guard — the ignore button is created only in the `s.id` branch (assert the `openIgnoreEditor` binding sits after the `if (s.id)` — a string-position check, the phase-88 pin idiom).
- `styles.css`: the `.ignore-editor-textarea` rule exists (mono + width), and `git-source-ignore` / `git-source-ignore-count` rules exist.
- guard pin (the phase-46/76 no-collision contract): no id in the new markup duplicates an existing id — assert each `id="ignore-editor-…"` appears exactly once in `index.html`.
## Testing & Quality
- `uv run pytest tests/unit/test_source_ignore_paths.py -v` green; full `uv run pytest tests/unit/ -q` green UNCHANGED (no other suite reads `git-sources.js` — verify with `rg "git-sources" tests/unit` first; the `header.js`/`styles.css` pins must survive the additive CSS).
- Coverage: **>90%** on `app/` unaffected (frontend-only task).
- `uv run ruff check . && uv run pyright` clean.
- Manual smoke (the executor's own eyes, headless is fine): boot the app, add a local source, open the box, type `my/files/`, save → the tag appears; refresh the page → the box prefills.
## Completion Criteria
- [ ] Every stored row (git or local) shows "Ignore paths"; env-fallback rows do not. Rows with a list show `N ignored`.
- [ ] The dialog: visible label, mono textarea prefilled from the row, Escape/Cancel/backdrop close with focus returned to the trigger, Save runs §7.4 ("Saving…" while the PATCH is out; error line keeps the content on 422).
- [ ] `git diff --stat` limited to `frontend/index.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, and the new unit test module.
.agents/phases/complete/89_source_ignore_paths/06_e2e_ignore_paths.md
+75
View File
@@ -0,0 +1,75 @@
# Task 06 — E2E story suite, full gate, and the atomic commit
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 (the whole item — this task is its end-to-end proof).
**Story:** n/a (TODO-derived).
## Objective
Prove the feature end-to-end through the real page + real API + real sync pipeline (mock LLM, no git, no network — the `test_git_sources_admin.py` policy): the box on the Sources page sets a list, the sync honors it (nothing embedded/summarized), previously indexed files are pruned (A2), and the spec's no-mid-path rule holds. Then run the complete gate and land ONE atomic commit.
## Work
1. `tests/e2e/test_source_ignore_paths.py` (NEW story suite — conftest app-server + mock-LLM + admin-login pattern from `tests/e2e/test_git_sources_admin.py` / `tests/e2e/auth_helpers.py`; module docstring: story source `TODO.md` L3, isolation run command, the no-git/no-network policy, the contract-under-test list, the test→contract mapping):
- **Fixtures:** a module-scoped source dir (`tmp_path_factory` — same reasoning as `test_sync_button.py`'s module-scoped fixture note) with this tree: `keep.md` (a `# Keep` doc with a sentinel word), `notes.yaml` (non-markdown → exercises the summary path), `myfile.md` (root level), `sub/myfile.md`, `ignore/secret.md`, `ignore/deep/x.txt`. A helper that (re)seeds the DB row: truncate `git_sources`/`documents` between tests (the house `db_ready` truncate pattern, the `test_sync_button.py` module-env note shows the DSN), then `POST /api/git-sources {"kind": "local", "path": <dir>}` (201) and capture the row id. A `run_sync()` helper: `POST /api/sync` → poll `GET /api/sync/status` to a terminal state (the `test_sync_button.py` polling idiom, ~2 s ticks, 60 s budget).
- `test_anonymous_gate_and_403s` — anonymous visit to `/git-sources.html`: the sign-in gate visible, the manager hidden, NO `/api/git-sources` request on load (route listener), and the page's request client gets 403 on `GET/POST /api/git-sources` AND `PATCH /api/git-sources/<uuid>` (the `test_git_sources_admin.py` anonymous pin).
- `test_ignore_box_excludes_from_import` — admin: seed the row (no list); open the box on the row (the "Ignore paths" button), type `ignore/` (single line), Save → wait for 200 (the `N ignored` tag: "1 ignored" appears on the row); `run_sync()` → `success`; `GET /api/docs` (the RAG catalog's data source — or read the RAG table after a nav to `/sources.html`, whichever the existing suites use) contains `keep.md`, `notes.yaml`, `myfile.md`, `sub/myfile.md` and contains NO path starting with `ignore/`; the sync `detail` `files` count == 5 (the two `ignore/` files excluded from the walk).
- `test_prefix_rule_no_mid_path` — admin: fresh seed; set the list to `myfile.md` via the box; `run_sync()` → the catalog contains `sub/myfile.md` (NOT ignored — no mid-path matching, the spec's own example) and NOT `myfile.md`; the `ignore/` files ARE indexed this time (no list matches them — regression guard that the box replaced, not appended, the previous test's list).
- `test_newly_ignored_file_is_pruned` (A2) — admin: fresh seed; `run_sync()` once with NO list (all six files indexed — assert 6 in the catalog); then set the box to `ignore/`, Save, `run_sync()` again → `success`, `detail.pruned == 2`, the catalog has the four remaining paths and no `ignore/` path.
- `test_editor_a11y_and_error_path` — admin: open the box → the dialog `role="alertdialog"` is visible, the visible label ("Ignored paths") is present, focus is on Cancel; Escape closes and focus returns to the trigger button (the `aria-label` matches the row); re-open, type an entry >500 chars (501 a's), Save → the `role="alert"` error line shows `an ignore path exceeds 500 characters`, the textarea content is KEPT, the save button is re-enabled with its "Save" label; then clear it, type `keep.md`, Save → "1 ignored" tag (the happy path heals the error state).
- `test_env_fallback_rows_have_no_box` — admin: truncate `git_sources` (the app booted with a `BOR_GIT_SOURCES` env URL, the `test_git_sources_admin.py` module-env pattern) → the env row renders with the "from .env" tag and NO "Ignore paths" button; `#git-sources-env-note` visible (A3).
2. **Gate (all must pass; any failure → fix within this task, re-run):**
- `uv run pytest` (unit + integration, green).
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on `app/` (the new importer + API logic is covered by tasks 02–04's suites; verify the TOTAL line, not just the new files).
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` (the NEW story — in isolation, DB up: `podman compose up -d db`).
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` (the existing Sources-page story — in isolation).
- `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` (the upload pipeline — in isolation).
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` (the sync button story — in isolation).
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (in isolation).
- `uv run ruff check . && uv run pyright` clean.
3. **Diff hygiene:** `git diff --stat` must be limited to: `app/models.py`, `app/schemas.py`, `app/api/git_sources.py`, `app/api/sync.py`, `app/rag/importer.py`, `alembic/versions/0013_git_source_ignore_paths.py` (new), `scripts/import_docs.py`, `frontend/index.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, `tests/unit/test_ignore_paths_column.py` (or the extended models module), `tests/unit/test_importer_ignore.py` (new), `tests/unit/test_source_ignore_paths.py` (new), `tests/integration/test_importer_ignore.py` (new), the extended integration modules (`test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py`, `test_git_sources_upload.py`), `tests/e2e/test_source_ignore_paths.py` (new), `TODO.md` (cleared — Phase 4 of the conversion), and the `89_source_ignore_paths/` phase files. Anything else → stop and fix.
4. **Commit (one atomic, Conventional Commits, `--no-gpg-sign` per AGENTS.md rule 8):**
```
feat(sources): per-source ignore paths — prefix-excluded files are never embedded, summarized, or re-indexed
TODO.md L3: the owner can type ignored files/folders into a box on
the Sources page; each source stores a list of source-relative path
prefixes, and matching is a pure prefix (no mid-path matching, no
globs — "/my/files/" ≡ "my/files/"; "myfile.txt" never matches
"some/path/myfile.txt").
- git_sources.ignore_paths: JSONB NOT NULL, server default '[]'
(migration 0013, reversible);
- importer: normalize_ignore_path / is_ignored / iter_importable_files
ignore filter / import_sources ignore_by_root (walk AND the
progress pre-walk; previously indexed files that newly match are
pruned on the next prune=True sync, the A9 junk precedent);
- API: GET/POST carry the list; new admin-only
PATCH /api/git-sources/{id} (replace semantics; fixed 422 details
for >200 entries, empty entry, >500-char entry);
- pipelines: _run_sync, the _run_upload background scan (a re-upload
honors the saved list), and scripts/import_docs.py all pass the
per-row lists;
- Sources page: per-row "Ignore paths" editor (the phase-69
dialog pattern, one path per line, §7.4 save lifecycle, a11y) +
the "N ignored" row tag; env-fallback rows have no box.
Unit: tests/unit/test_ignore_paths_column.py,
tests/unit/test_importer_ignore.py,
tests/unit/test_source_ignore_paths.py.
Integration: tests/integration/test_importer_ignore.py + extended
git-sources/sync/upload/CLI suites.
E2E story: tests/e2e/test_source_ignore_paths.py (box → sync
exclusion, no-mid-path rule, prune of a newly ignored file, a11y +
422 path, env rows, anonymous gate).
Phase 89: .agents/phases/todo/89_source_ignore_paths/.
```
Stage ONLY the step-3 file set (the phase directory rides with the commit, AGENTS.md rule 8; `TODO.md` is cleared to the bare `# TODO` title as part of this commit — the conversion's Phase 4).
## Testing & Quality
- The full gate of step 2 IS this task's test contract; coverage **>90%** on `app/`.
- This task adds the E2E file only — everything else verifies and commits.
## Completion Criteria
- [ ] All five E2E tests pass in isolation (DB up); the four regression E2E suites pass in isolation; unit + integration green; TOTAL coverage >90%; ruff + pyright clean.
- [ ] `git diff --stat` (pre-commit) matches the step-3 file set exactly.
- [ ] One atomic `--no-gpg-sign` commit landed (message per step 4); `git log -1 --stat` confirms the file set.
- [ ] `TODO.md` contains exactly `# TODO` (cleared, never deleted).
- [ ] The phase directory is under `.agents/phases/todo/89_source_ignore_paths/` (the pipeline gate moves it to `complete/` after validate.sh passes).
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__00_phase.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__00_phase.a1.md
+23
View File
@@ -0,0 +1,23 @@
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.
## Phase 89 — final verification pass: ALL GREEN
**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box
**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)
**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.
**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).
**Next pending phase:** none — `todo/` holds only this phase.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__00_phase.a1.validate
+87
View File
@@ -0,0 +1,87 @@
........................................................................ [ 3%]
........................................................................ [ 7%]
........................................................................ [ 11%]
........................................................................ [ 15%]
........................................................................ [ 19%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 31%]
........................................................................ [ 35%]
........................................................................ [ 39%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 55%]
........................................................................ [ 59%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 95%]
........................................................................ [ 99%]
........ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__01_ignore_paths_column.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__01_ignore_paths_column.a1.md
+19
View File
@@ -0,0 +1,19 @@
All green. Task 01 complete.
## Report
**Implemented (task 01 — storage half of per-source ignore paths):**
- `app/models.py`: `GitSource.ignore_paths` — JSONB NOT NULL, `default=list` (INSERT-time) + server default `'[]'` (verbatim task spec); `text` added to the sqlalchemy imports; module-docstring `git_sources` bullet updated (phase 89).
- `alembic/versions/0013_git_source_ignore_paths.py` (new): `add_column` JSONB/`'[]'`/NOT NULL, `drop_column` downgrade — 0007 house-header style, revises 0012.
- `tests/unit/test_models.py` (extended, house pattern): column exists / NOT NULL / server default `arg.text == "'[]'"` (narrowed via `DefaultClause` + `TextClause` for pyright), plus the Python-side default contract test.
**Gates (exact commands):**
- `uv run alembic upgrade head` → `0012 -> 0013` clean; `alembic downgrade -1 && alembic upgrade head` round-trips; head = `0013`.
- `uv run pytest tests/integration/test_git_sources_api.py` → 23 passed, **unchanged** (dev DB `git_sources` empty; `'[]'` server default covers existing rows; a live ORM-insert probe confirmed `[]` is stored — probe row deleted).
- `uv run pytest --cov=app --cov-report=term-missing` → **1748 passed**, TOTAL **99%** (>90% ✓, `app/models.py` 100%).
- `uv run ruff check . && uv run pyright` → clean (0 errors).
- `git diff --stat`: only `app/models.py`, the new migration, and the unit test.
**Notable deviation:** the task's literal pin `GitSource(...).ignore_paths == []` cannot hold — SQLAlchemy 2.0 `default=list` is an INSERT-time `CallableColumnDefault` (unflushed attr is `None`; the builtin is wrapped in a ctx lambda, so `arg is list` also fails). The test instead pins the INSERT-time default resolving to `[]`; real-DB behavior is covered by the API integration suite. Model code matches the spec verbatim.
**Next pending task:** `.agents/phases/todo/89_source_ignore_paths/02_ignore_prefix_matching.md`
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__01_ignore_paths_column.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 37%]
........................................................................ [ 41%]
........................................................................ [ 45%]
........................................................................ [ 49%]
........................................................................ [ 53%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 74%]
........................................................................ [ 78%]
........................................................................ [ 82%]
........................................................................ [ 86%]
........................................................................ [ 90%]
........................................................................ [ 94%]
........................................................................ [ 98%]
.................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 212 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 98 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 232 0 100%
--------------------------------------------------
TOTAL 3238 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__02_ignore_prefix_matching.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__02_ignore_prefix_matching.a1.md
+10
View File
@@ -0,0 +1,10 @@
All green. Task complete.
**Report**
- Implemented in `app/rag/importer.py`: `normalize_ignore_path` / `is_ignored` / `_ignore_for_root` helpers (spec A1 raw-prefix semantics, single normalization choke point); `iter_importable_files(..., ignore=())` walk-level skip; `import_sources(..., ignore_by_root=None)` — pre-walk and processing loop use the same per-root tuple, `seen`/`_prune` untouched so newly-ignored files are pruned on next `prune=True` run (A2); docstrings per house style.
- New `tests/unit/test_importer_ignore.py` (24 tests): normalization table, spec examples verbatim (`/my/files/` ≡ `my/files/` ≡ `my/files`; `myfile.txt` root-level only, no mid-path), raw-prefix A1 edge (`my/files2`, `ab.md`), empty-prefix no-match, `iter_importable_files` on a 5-file tmp tree incl. the "walk receives already-normalized tuples" contract pin, `_ignore_for_root` normalize/drop/missing/None.
- New `tests/integration/test_importer_ignore.py` (5 tests): ignored files never indexed (no doc/chunk rows, no embed or chat call, kept `.txt` still summarized); A2 prune (`pruned == 1`, row gone, pattern given without trailing slash); progress `total` == 2 excludes ignored; no-map byte-identical (`files == 4`); unlisted second root imports everything.
- Test / lint / coverage: `uv run pytest tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -v` → 29 passed; `uv run pytest -q` → exit 0 (1777 collected, no failures); `uv run pytest --cov=app --cov-report=term-missing -q` → TOTAL 99% (>90% gate); `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
- Notable: (1) fixed the phase-64 counting-sentinel in `tests/unit/test_importer.py` to accept/forward the new `ignore` keyword (its monkeypatch stub broke on the new call site — the one extra file beyond the task's stated diff scope); (2) prune test uses a 3-file tree (one file under `ignore/`) so `pruned == 1` pins exactly, as the task specifies.
- Next pending task: `.agents/phases/todo/89_source_ignore_paths/03_ignore_paths_api.md`.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__02_ignore_prefix_matching.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 48%]
........................................................................ [ 52%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 76%]
........................................................................ [ 81%]
........................................................................ [ 85%]
........................................................................ [ 89%]
........................................................................ [ 93%]
........................................................................ [ 97%]
................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 212 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 98 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 232 0 100%
--------------------------------------------------
TOTAL 3248 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__03_ignore_paths_api.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__03_ignore_paths_api.a1.md
+17
View File
@@ -0,0 +1,17 @@
Everything is green. Final report:
**Task 03 (phase 89) complete — ignore-list API**
- `app/schemas.py`: `GitSourceIn.ignore_paths` (optional, raw lines), `GitSourceOut`/`GitSourceRow.ignore_paths` (non-null; env rows `[]`), new `GitSourceIgnoreIn` (required list, A5)
- `app/api/git_sources.py`: shared `_validate_ignore_paths` (normalize + A4 fixed-detail 422s, empty-check first; constants 200/500), GET reports per-row list, POST stores normalized for both kinds, new admin-only `PATCH /api/git-sources/{id}` (replace semantics, 404/422/403) placed after POST before `/upload`; module docstring updated
- `tests/integration/test_git_sources_api.py`: 10 new tests (normalize round-trip both kinds, absent→`[]`, POST+PATCH 422 exact details, missing-field 422, 404, invalid id, replace+clear, A4 boundaries, row-unchanged on reject); existing shape pins updated additively, anonymous test covers PATCH
**Gates:**
- `uv run pytest` → 1787 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 99% (>90% ✓; `git_sources.py` + `schemas.py` 100%)
- `uv run pytest tests/integration/test_git_sources_api.py --no-cov` → 33 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
**Notes:** no defects found in prior tasks; diff limited to the 3 task files. `.agents/PLAN.md` is absent in this repo — `00_phase.md` design (locked A1–A5) was the design source; no deviations.
**Next pending task:** `.agents/phases/todo/89_source_ignore_paths/04_ignore_paths_in_import_pipelines.md`
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__03_ignore_paths_api.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 48%]
........................................................................ [ 52%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 76%]
........................................................................ [ 80%]
........................................................................ [ 84%]
........................................................................ [ 88%]
........................................................................ [ 92%]
........................................................................ [ 96%]
........................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 98 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3273 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__04_ignore_paths_in_import_pipelines.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__04_ignore_paths_in_import_pipelines.a1.md
+19
View File
@@ -0,0 +1,19 @@
Task 04 complete. Report:
**Implemented**
- `app/api/sync.py::_run_sync` — per-row loop now also builds `ignore_by_root` (keyed by the same `str(root)` the importer sees; `setdefault().extend()` union on root-string collision) and passes it to `import_sources`; module docstring step 4 updated
- `app/api/git_sources.py::_run_upload` — step 6 captures the row's saved `ignore_paths` into a plain local (both existing/new-row branches, before `db.close()`); step 8 passes `ignore_by_root={str(final_dir): ignore_paths}`; route docstring steps 6/8 updated
- `scripts/import_docs.py` — `_resolve_sources` returns `(sources, ignore_by_root)` (manual `--source` and legacy fallback → `{}`); `main` unpacks and passes the map; module + function docstrings updated
- Tests: real-import sync test (ignored file excluded from KB + `files` count), fake-import union/empty-map wiring tests, CLI rows-branch/manual map tests, re-upload-respects-PATCHed-list test (pruned on re-scan) + new-source-no-list regression; `ignore_by_root` kwarg added to the 4 `import_sources` fakes
**Results**
- `uv run pytest -q` → 1795 passed, exit 0 (incl. `tests/integration/test_sync_api.py test_import_docs_git.py test_git_sources_upload.py` + `tests/unit/test_sync_button.py`: 105 passed)
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%); `app/api/sync.py` + `app/api/git_sources.py` 100%
- `uv run ruff check . && uv run pyright` → clean
- E2E regressions in isolation (`test_git_sources_admin`, `test_archive_upload_sources`, `test_sync_button`, `test_smoke`) → 17 passed
**Decisions**
- One-line mechanical addition to `tests/unit/test_sync_button.py` fake signature (outside the task's listed files) — required to keep the suite green once the runner passes the new kwarg
- Union-order test pins distinct `added_at` (same-timestamp rows tie-break on random uuid)
**Next pending task:** `05_ignore_paths_sources_ui.md`
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__04_ignore_paths_in_import_pipelines.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 48%]
........................................................................ [ 52%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 76%]
........................................................................ [ 80%]
........................................................................ [ 84%]
........................................................................ [ 88%]
........................................................................ [ 92%]
........................................................................ [ 96%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__05_ignore_paths_sources_ui.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__05_ignore_paths_sources_ui.a1.md
+18
View File
@@ -0,0 +1,18 @@
**Task 05 complete — per-row "Ignore paths" box on the Sources page.**
Implemented:
- `frontend/index.html`: `#ignore-editor-dialog` (phase-69 `#remove-confirm-dialog` pattern — alertdialog, dim backdrop, visible label, mono textarea, `role="alert"` error line, Cancel/Save) placed after the remove dialog in `#git-sources-content`
- `frontend/assets/git-sources.js`: per-row "Ignore paths" button (stored rows only, left of Remove, textContent-only), `N ignored` count tag, `openIgnoreEditor`/`closeIgnoreEditor`/`saveIgnorePaths` with the §7.4 "Saving…" lifecycle, focus-on-Cancel, Escape/Cancel/backdrop close with focus return, blank-line dropping, A5-replace PATCH, LAST-announcement order; module docstring phase-89 bullet
- `frontend/assets/styles.css`: `.ignore-editor-*` dialog family (house tokens, no blur, ≥44px, brand Save / ghost Cancel) + `.git-source-ignore` / `.git-source-ignore-count` row chrome
- `tests/unit/test_source_ignore_paths.py` (new): 13 source-level pins (markup, lifecycle, row guard, CSS, id no-collision)
Results:
- `uv run pytest tests/unit/test_source_ignore_paths.py --no-cov` → 13 passed
- `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, TOTAL 99%** (>90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- E2E regressions in isolation: `test_git_sources_admin.py` 6 passed; `test_smoke.py` + `test_archive_upload_sources.py` + `test_sync_button.py` 11 passed
- Headless manual smoke: add local source → open box → save (blank line dropped) → `2 ignored` tag + announcer → refresh prefills (normalized) → Escape closes with focus return → empty box clears (A5) — all verified, source cleaned up
Decisions: Escape/Cancel/backdrop are no-ops mid-PATCH (in-flight guard, remove-dialog idiom); Save uses the `.new-chat-btn` brand hover pair; count tag = ink-on-bg chip (16.7:1). One bug found+fixed in my own test (f-string doubled the announce phrase).
Next pending task: `.agents/phases/todo/89_source_ignore_paths/06_e2e_ignore_paths.md`.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__05_ignore_paths_sources_ui.a1.validate
+87
View File
@@ -0,0 +1,87 @@
........................................................................ [ 3%]
........................................................................ [ 7%]
........................................................................ [ 11%]
........................................................................ [ 15%]
........................................................................ [ 19%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 31%]
........................................................................ [ 35%]
........................................................................ [ 39%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 55%]
........................................................................ [ 59%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 95%]
........................................................................ [ 99%]
........ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__06_e2e_ignore_paths.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__06_e2e_ignore_paths.a1.md
+11
View File
@@ -0,0 +1,11 @@
**Task 06 complete — report**
- Implemented `tests/e2e/test_source_ignore_paths.py` (new story suite, 6 tests): anonymous gate + 403s (incl. new PATCH), box→sync exclusion (`ignore/` never walked/embedded/summarized, `files` count excludes it), spec's no-mid-path rule (`sub/myfile.md` kept, root `myfile.md` ignored, list *replaces* per A5), A2 prune of newly-ignored files (`pruned == 2`), editor a11y + 422 path (501-char entry, textarea kept, Save re-enabled, happy-path heal), env-fallback rows with no box + env note. Module app on port 8140 with `kind="local"` fixture dir (6 files) — no git, no network.
- Results (all green):
- `uv run pytest --cov=app --cov-report=term-missing` → 1808 passed, TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (run twice, stable)
- Regression E2E in isolation: `test_git_sources_admin.py` 6 passed, `test_archive_upload_sources.py` 5 passed, `test_sync_button.py` 3 passed, `test_smoke.py` 3 passed
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors
- Deviations: (1) task's `files == 5` assertion is an arithmetic slip against its own 6-file tree (6−2=4) — test asserts 4, documented in-code; (2) stored-list round-trip asserts the A1-normalized value (`ignore` for box line `ignore/`); (3) no `git add`/`commit` run — left in working tree per executor rules (harness makes the one atomic `--no-gpg-sign` commit; `TODO.md` already exactly `# TODO`).
- Diff hygiene: matches step-3 file set (the small `tests/unit/test_importer.py` / `test_sync_button.py` modifications pre-date this task, from completed tasks 02/04).
- Next pending task: none — this was the last task of phase 89; the pipeline gate moves the phase dir and commits.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__06_e2e_ignore_paths.a1.validate
+87
View File
@@ -0,0 +1,87 @@
........................................................................ [ 3%]
........................................................................ [ 7%]
........................................................................ [ 11%]
........................................................................ [ 15%]
........................................................................ [ 19%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 31%]
........................................................................ [ 35%]
........................................................................ [ 39%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 55%]
........................................................................ [ 59%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 95%]
........................................................................ [ 99%]
........ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
alembic/versions/0013_git_source_ignore_paths.py
+48
View File
@@ -0,0 +1,48 @@
"""git_sources.ignore_paths: per-source ignore-path list (phase 89)
Revision ID: 0013
Revises: 0012
Create Date: 2026-09-08
Phase 89 (per-source ignore paths: the owner types a list of
source-relative path prefixes per source — one path per line — into the
box on the Sources page; the importer never walks, embeds, or summarizes
a file whose source-relative path starts with one of them; A1 raw prefix
semantics, A4 limits, A5 replace semantics; one additive, reversible
column, A13):
* ``git_sources.ignore_paths`` — JSONB NOT NULL with server default
``'[]'``: the normalized, non-empty source-relative path prefixes. A
file is ignored when its source-relative POSIX path starts with any
entry (no mid-path matching, no globs). Existing rows read ``[]`` —
every pre-phase-89 row imports exactly as before.
"""
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
from alembic import op
revision = "0013"
down_revision = "0012"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"git_sources",
sa.Column(
"ignore_paths",
postgresql.JSONB(astext_type=sa.Text()),
server_default=sa.text("'[]'"),
nullable=False,
),
)
def downgrade() -> None:
# Safe order: the column is the only 0013 artefact — dropping it
# leaves 0012's schema byte-identical (A13, fully reversible).
op.drop_column("git_sources", "ignore_paths")
app/api/git_sources.py
+115 -17
View File
@@ -14,10 +14,18 @@ fallback is active).
Routes: ``GET`` (DB rows oldest-first, or the env list with
``from_env: true`` while the table is empty; rows carry ``kind`` +
``path``, git rows — and env rows — report ``path: null``), ``POST``
``path``, git rows — and env rows — report ``path: null``; every row
reports ``ignore_paths`` — DB rows their stored normalized list, env
rows ``[]``, phase 89), ``POST``
(201, validated create; ``kind`` selects the validation: git → exactly
the phase-35 URL contract, local → an existing absolute directory, else
422 naming the path), ``POST /upload`` (phase 49, backgrounded in phase
422 naming the path; both kinds accept ``ignore_paths`` — optional,
absent → ``[]``, stored normalized, phase 89),
``PATCH /{source_id}`` (phase 89, A5 — replace one source's ignore
list: 404 unknown id, the required body list is normalized + A4-
validated with fixed-detail 422s and REPLACES the row's list wholesale
— an empty list clears all; 200 → the ``GitSourceOut`` shape),
``POST /upload`` (phase 49, backgrounded in phase
64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/
``.zip`` name/format gate + the 1 MiB-chunk receive with the
``upload_max_mb`` cap run **inline** and answered 202 the moment the
@@ -90,7 +98,7 @@ from app.rag.archive_upload import (
swap_in,
unpack_archive,
)
from app.rag.importer import import_sources
from app.rag.importer import import_sources, normalize_ignore_path
from app.rag.llm import LLMClient, check_models
from app.rag.overview import regenerate_overview
from app.rag.source_removal import (
@@ -101,6 +109,7 @@ from app.rag.source_removal import (
)
from app.rag.sources_meta import bump_sources_version
from app.schemas import (
GitSourceIgnoreIn,
GitSourceIn,
GitSourceList,
GitSourceOut,
@@ -172,6 +181,14 @@ _upload_status = UploadStatus()
#: accepted shapes are exactly these four prefixes.
URL_RE = re.compile(r"^(https?://|ssh://|git@)")
#: Phase 89, A4 — the per-source ignore-list limits, enforced in
#: :func:`_validate_ignore_paths` (shared by POST and PATCH): at most
#: 200 entries, each ≤500 chars after normalization. The 422 details
#: are fixed strings that never echo the input (the router's
#: credential-safety discipline, applied for consistency).
MAX_IGNORE_PATHS = 200
MAX_IGNORE_PATH_LENGTH = 500
@router.get("", response_model=GitSourceList)
def list_git_sources(
@@ -181,10 +198,13 @@ def list_git_sources(
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
same-timestamp inserts) with ``from_env: false`` — each row carries
its ``kind`` and, for local rows, the stored ``path`` (git rows and
env rows report ``path: null``); while the table is empty, the
its ``kind``, its ``ignore_paths`` (phase 89 — the stored,
normalized list; ``or []`` guards a row that predated the column),
and, for local rows, the stored ``path`` (git rows and env rows
report ``path: null``); while the table is empty, the
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
git-only) with null ``id``/``added_at`` and ``from_env: true``.
git-only) with null ``id``/``added_at``, ``ignore_paths: []`` (no
DB row to store a list on), and ``from_env: true``.
"""
rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
@@ -200,6 +220,7 @@ def list_git_sources(
url=row.url,
path=row.path,
added_at=row.added_at,
ignore_paths=row.ignore_paths or [],
)
for row in rows
],
@@ -207,7 +228,9 @@ def list_git_sources(
)
return GitSourceList(
sources=[
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None)
GitSourceRow(
id=None, kind="git", url=url, path=None, added_at=None, ignore_paths=[]
)
for url in get_settings().git_source_list
],
from_env=True,
@@ -236,9 +259,35 @@ def create_git_source(
Wrong field combinations (git without url, local without path, both
kinds' fields) are 422 with fixed, input-free details.
``ignore_paths`` (phase 89) — optional, both kinds: the raw box
lines are normalized + A4-validated (``_validate_ignore_paths`` —
the fixed-detail 422s) and the normalized list is what is stored and
reported.
"""
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
return GitSourceOut(
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
)
def _validate_ignore_paths(raw: list[str] | None) -> list[str]:
"""Normalize + enforce the A4 limits (phase 89); the fixed 422
details never echo the input (the credential-safety discipline,
applied for consistency).
Shared by POST and PATCH. The empty check runs FIRST: a
whitespace-only entry must 422, not be silently dropped (the UI
drops blank lines client-side; the API stays defensive).
"""
entries = [normalize_ignore_path(e) for e in (raw or [])]
if any(not e for e in entries):
raise HTTPException(status_code=422, detail="ignore paths must be non-empty")
if len(entries) > MAX_IGNORE_PATHS:
raise HTTPException(status_code=422, detail="a source has at most 200 ignore paths")
if any(len(e) > MAX_IGNORE_PATH_LENGTH for e in entries):
raise HTTPException(status_code=422, detail="an ignore path exceeds 500 characters")
return entries
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
@@ -270,7 +319,13 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
return _commit_new(
GitSource(url=url, kind="git"), "a git source with this URL already exists", db
GitSource(
url=url,
kind="git",
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
),
"a git source with this URL already exists",
db,
)
@@ -296,12 +351,41 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
# rows carry the expanded path there too — git URL shapes and absolute
# paths cannot collide).
return _commit_new(
GitSource(url=path, kind="local", path=path),
GitSource(
url=path,
kind="local",
path=path,
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
),
f"a local source with this path already exists: {path}",
db,
)
@router.patch("/{source_id}", response_model=GitSourceOut)
def patch_git_source(
source_id: uuid.UUID,
payload: GitSourceIgnoreIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Replace one source's ignore list (phase 89, A5).
404 unknown id. The body list (required) is normalized +
A4-validated (fixed 422 details) and REPLACES the row's list
wholesale — an empty list clears all. Returns the updated
row's public shape (id, url, added_at, ignore_paths).
"""
row = db.get(GitSource, source_id)
if row is None:
raise HTTPException(status_code=404, detail="git source not found")
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
db.commit()
db.refresh(row)
return GitSourceOut(
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
)
@router.post("/upload", response_model=UploadAccepted, status_code=202)
async def upload_archive(
file: UploadFile = File(...), # noqa: B008
@@ -339,14 +423,18 @@ async def upload_archive(
6. upsert the row by ``path`` (``kind='local'``; an existing row is
left as-is — ``added_at`` preserved — and the unique index is
the backstop: a concurrent insert lands ``failed`` with
``a local source with this path already exists: <path>``);
``a local source with this path already exists: <path>``); the
row's saved ``ignore_paths`` are captured for the scan (phase
89: a re-upload of an existing source honors the list the owner
already saved);
7. fail-fast ``check_models`` — ``ModelUnavailableError`` →
``failed`` with the sanitized message (the phase-49 503 becomes
a status state, A5); the folder/row are already committed, so
the next sync/re-upload retries idempotently;
8. ``import_sources([folder], llm, prune=True, progress=<hook>)``
+ the change-gated ``regenerate_overview`` — the hook feeds the
status ``current_file`` / ``files_done`` / ``files_total``;
8. ``import_sources([folder], llm, prune=True, progress=<hook>,
ignore_by_root={folder: row's list})`` (phase 89) + the
change-gated ``regenerate_overview`` — the hook feeds the status
``current_file`` / ``files_done`` / ``files_total``;
9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is
the background run's duration);
10. ``success`` — ``detail`` = the ``UploadOut`` fields.
@@ -502,8 +590,10 @@ async def _run_upload(
path = str(final_dir)
db = SessionLocal()
try:
if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
db.add(GitSource(url=path, kind="local", path=path))
row = db.scalar(select(GitSource).where(GitSource.path == path))
if row is None:
row = GitSource(url=path, kind="local", path=path)
db.add(row)
try:
db.commit()
except IntegrityError:
@@ -511,6 +601,11 @@ async def _run_upload(
raise ValueError(
f"a local source with this path already exists: {path}"
) from None
# Phase 89: the row's saved ignore list, copied to plain
# values while the row is still usable in this session — a
# re-upload of an existing source honors the list the owner
# already saved; a fresh row has no list yet.
ignore_paths = list(row.ignore_paths or [])
finally:
db.close()
# Step 7 — fail-fast models (phase 41): ``ModelUnavailableError``
@@ -530,7 +625,10 @@ async def _run_upload(
_upload_status.files_done = done
_upload_status.files_total = total
summary = await import_sources([final_dir], llm, prune=True, progress=_hook)
summary = await import_sources(
[final_dir], llm, prune=True, progress=_hook,
ignore_by_root={str(final_dir): ignore_paths},
)
overview = False
if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm)
app/api/sync.py
+20 -9
View File
@@ -34,9 +34,11 @@ decisions):
<path>``; a failing clone or a missing local dir aborts before any
import;
4. ``import_sources(..., prune=True)`` over the single combined list
(git checkouts + local dirs) — prune so files deleted upstream or
out of a local dir leave the index (pruning covers the union; the
CLI's no-prune default is unchanged);
(git checkouts + local dirs), honoring each row's ``ignore_paths``
(phase 89 — the per-root ignore map is built in the same per-row
loop as the source list) — prune so files deleted upstream, out of
a local dir, or newly matching an ignore pattern leave the index
(pruning covers the union; the CLI's no-prune default is unchanged);
5. when the import changed the KB (added + updated > 0),
``regenerate_overview`` refreshes the single ``kb_overview`` row
(phase 31 trigger, best-effort inside);
@@ -214,19 +216,26 @@ async def _run_sync() -> None:
)
sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
for row in rows:
if row.kind == "git":
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
else:
# kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the
# ``or`` keeps the type checker honest); re-verified at
# sync time because the directory may have moved or been
# deleted since add-time.
path = Path(row.path or row.url).expanduser()
if not path.is_dir():
raise GitSyncError(f"local source missing: {path}")
sources.append(path)
root = Path(row.path or row.url).expanduser()
if not root.is_dir():
raise GitSyncError(f"local source missing: {root}")
sources.append(root)
# Phase 89: the row's ignore list, keyed by the SAME root
# string the importer sees; two rows sharing a root string
# get the union (extend, not replace) — the sibling/repo-name
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
# Phase 64 (task 02): the per-file progress hook — the status
# endpoint reports the file being processed right now. The
# closure captures the module ``_status`` exactly like the state
@@ -236,7 +245,9 @@ async def _run_sync() -> None:
_status.files_done = done
_status.files_total = total
summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook)
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root
)
overview = False
if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm)
app/models.py
+13 -1
View File
@@ -16,7 +16,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
* ``git_sources`` — admin-managed source registry (git URLs + local
directories) the Sync button and import_docs
import (phase 35; ``kind`` discriminator added in
phase 38).
phase 38; ``ignore_paths`` (phase 89 — JSONB list
of normalized path prefixes, server default
``'[]'``)).
* ``saved_chats`` — owner-saved chat conversations: one row per
explicitly Saved conversation (auto-``title`` +
the ``bor.chat.v1`` message list as JSONB, phase
@@ -72,6 +74,7 @@ from sqlalchemy import (
Text,
UniqueConstraint,
func,
text,
)
from sqlalchemy.dialects.postgresql import JSONB, UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -226,6 +229,15 @@ class GitSource(Base):
#: Absolute directory of a ``local`` source; NULL for git rows.
#: Unique — Postgres treats NULLs as distinct under a unique index.
path: Mapped[str | None] = mapped_column(Text, unique=True)
#: Per-source ignore paths (phase 89, A1/A4): the normalized,
#: non-empty source-relative path prefixes the owner types into the
#: box on the Sources page. A file is ignored when its
#: source-relative POSIX path starts with any entry (raw prefix —
#: no mid-path matching, no globs). Server default '[]' — every
#: pre-phase-89 row imports exactly as before.
ignore_paths: Mapped[list] = mapped_column(
JSONB, default=list, server_default=text("'[]'"), nullable=False
)
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
app/rag/importer.py
+73 -3
View File
@@ -118,17 +118,63 @@ class ImportSummary:
)
def normalize_ignore_path(entry: str) -> str:
"""One ignore-path entry → canonical form (phase 89, A1).
Trim surrounding whitespace, then strip ALL leading/trailing
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
all become ``"my/files"``. ``""`` / ``"//"``,
``" "`` normalize to ``""`` (callers drop empties).
"""
return entry.strip().strip("/")
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
"""Phase 89, A1 — the pure prefix rule, nothing else.
``rel`` is the source-relative POSIX path WITHOUT a leading
slash (the ``documents.path`` string). Match = ``rel`` STARTS
WITH a normalized entry: raw string prefix — deliberately NO
component-boundary check (``"my/files"`` also matches
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
"""
return any(rel.startswith(p) for p in prefixes)
def _ignore_for_root(
root: Path, ignore_by_root: dict[str, list[str]] | None
) -> tuple[str, ...]:
"""The normalized, non-empty prefix tuple for one root (phase 89).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (unambiguous when two rows share a
source *name* but different dirs). Callers may pass RAW box
lines: the importer normalizes + drops empties here, the single
choke point — stored lists (already normalized) normalize to
themselves.
"""
raw = (ignore_by_root or {}).get(str(root)) or []
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
def iter_importable_files(
root: Path,
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
) -> list[Path]:
"""All importable files under *root* (sorted), per the A9 scope rules.
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
Skips: any path with a dot-prefixed component (hidden dirs/files —
vendored caches like ``.esphome/.espressif/**``) and the well-known
non-content directories in *excluded*.
non-content directories in *excluded*. *ignore* (phase 89, A1) is a
tuple of ALREADY-normalized, non-empty source-relative path prefixes
(the importer's ``_ignore_for_root`` is the normalization choke point
— raw box lines never reach this function): a file is skipped when its
source-relative POSIX path starts with any entry; the default ``()``
keeps every existing caller byte-identical.
"""
if not root.is_dir():
return []
@@ -139,6 +185,8 @@ def iter_importable_files(
rel = path.relative_to(root)
if any(part.startswith(".") or part in excluded for part in rel.parts):
continue
if ignore and is_ignored(rel.as_posix(), ignore):
continue
if path.suffix.lower() not in extensions:
continue
files.append(path)
@@ -153,6 +201,7 @@ async def import_sources(
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring).
@@ -172,6 +221,18 @@ async def import_sources(
``limit``, the hook still fires per processed file only — ``done`` never
exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator).
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
string exactly as passed in *sources* — to that source's RAW ignore-path
lines (the importer normalizes them via ``_ignore_for_root``, the single
choke point): matching files are never walked, so they are never
embedded and never summarized, and the progress pre-walk uses the same
per-root tuple as the processing loop, so ``total`` never counts them.
A file that newly matches a pattern simply never enters ``seen``, so the
next ``prune=True`` run deletes its row automatically (A2 — the A9
junk-precedent). ``None`` (the default) changes nothing: the map is read
per root, unlisted roots get an empty tuple, and every existing caller
behaves byte-identically.
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -188,7 +249,13 @@ async def import_sources(
total = 0
if progress is not None:
for root in sources:
total += len(iter_importable_files(root, llm.settings.import_extension_set))
total += len(
iter_importable_files(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
)
)
try:
for root in sources:
if not root.is_dir():
@@ -198,7 +265,10 @@ async def import_sources(
break
source = root.name
source_names.add(source)
for path in iter_importable_files(root, llm.settings.import_extension_set):
ignore = _ignore_for_root(root, ignore_by_root)
for path in iter_importable_files(
root, llm.settings.import_extension_set, ignore=ignore
):
if limit is not None and summary.files >= limit:
break
rel = path.relative_to(root).as_posix()
app/schemas.py
+30 -2
View File
@@ -327,11 +327,18 @@ class GitSourceIn(BaseModel):
Trimmed here; the API layer then ``expanduser()``s it and requires an
absolute existing directory (else 422 naming the path — the path is
not a secret, unlike a git URL) and no ``url``.
``ignore_paths`` (phase 89) is optional at create time (absent →
``[]``) and carries the RAW box lines — trimming/normalization happens
in the API layer, not the schema, so the A4 422 details stay fixed
strings (the router's credential-safety discipline, applied for
consistency).
"""
kind: Literal["git", "local"] = "git"
url: str | None = Field(default=None, min_length=1, max_length=500)
path: str | None = Field(default=None, min_length=1, max_length=2000)
ignore_paths: list[str] | None = Field(default=None)
@field_validator("url", mode="before")
@classmethod
@@ -352,12 +359,14 @@ class GitSourceOut(BaseModel):
``kind=local`` rows, the stored (expanded) directory path — the
phase-35 response shape is unchanged by phase 38, so a local 201
reports its path in ``url`` and the full row (``kind`` + ``path``)
via ``GET``.
via ``GET``. ``ignore_paths`` (phase 89) is the stored, normalized
list — non-null (a row created without it reports ``[]``).
"""
id: uuid.UUID | None
url: str
added_at: datetime | None
ignore_paths: list[str]
class GitSourceRow(BaseModel):
@@ -369,7 +378,9 @@ class GitSourceRow(BaseModel):
``path: null``; local rows carry ``path`` (the absolute directory,
expanded) and the same string in ``url`` (the table's NOT-NULL
location column). ``id`` / ``added_at`` are nullable: env-fallback
rows (table empty) carry neither.
rows (table empty) carry neither. ``ignore_paths`` (phase 89) is the
row's stored, normalized list — env-fallback rows (no DB row to
store a list on) report ``[]``.
"""
id: uuid.UUID | None
@@ -377,6 +388,23 @@ class GitSourceRow(BaseModel):
url: str
path: str | None
added_at: datetime | None
ignore_paths: list[str]
class GitSourceIgnoreIn(BaseModel):
"""``PATCH /api/git-sources/{source_id}`` body (phase 89, A5).
``ignore_paths`` is REQUIRED — the box's lines in REPLACE semantics:
the body list (normalized + A4-validated in the API layer, with the
fixed-detail 422s) becomes the row's whole list — an empty list
clears all; an absent field is a 422 with the model's own detail.
Entries are RAW box lines: normalization + the A4 limits are
enforced in the API layer so the 422 details stay fixed strings
(the router's credential-safety discipline, applied for
consistency).
"""
ignore_paths: list[str]
class GitSourceList(BaseModel):
frontend/assets/git-sources.js
+228
View File
@@ -143,6 +143,35 @@
* gate and never fetches /api/git-sources (the Sources-page gate
* pattern).
*
* Phase 89 (task 05) — the per-source ignore-paths editor: every
* STORED row (s.id truthy — env-fallback rows carry NO button, A3:
* nothing is stored to edit) gets an "Ignore paths" button in the
* actions cell (left of Remove) that opens the page-local alertdialog
* (#ignore-editor-dialog — the EXACT #remove-confirm-dialog pattern,
* phase 69): #ignore-editor-source names the source (textContent ONLY
* — the same `value` expression makeRow uses — URLs may embed
* user:pass@ credentials, phase 32), the mono textarea prefills the
* row's stored list (one path per line — the phase-89 A1 prefix rule,
* stated in plain words in the helper copy), and focus lands on
* Cancel (the safe default); Escape / Cancel / the dim backdrop close
* as CANCEL (no request — focus returns to the row's trigger button).
* "Save" runs the §7.4 in-flight lifecycle: both buttons disable +
* the save relabels "Saving…" while the PATCH
* /api/git-sources/{id} is out (blank lines in the box are separators,
* not entries — split + trim + drop empty client-side; the server
* still rejects empty entries defensively, A4). 200 → close (focus
* return), loadSources (the row's "N ignored" count tag lands — the
* A5 replace semantics round-trip through GET), announce — the update
* confirmation is the LAST announcement (the reload's "N sources
* listed." must not overwrite it); non-2xx → the in-dialog
* role="alert" line (the server detail, apiDetail 422-shape-aware) +
* the buttons restored, the dialog STAYS open and the textarea
* content is KEPT (the instruction survives — tuning-form
* convention); network failure → the fixed reachable? line, same
* restore. Rows whose list is non-empty show the "N ignored" count
* tag next to the location <code> (text + a distinct background —
* never color alone, WCAG 1.4.1).
*
* Scope boundary (phase locked decisions): adding a git repo does
* NOT clone — the sync service (server-side) does that. Removing a
* source, however, performs the FULL cleanup server-side (phase 69):
@@ -203,6 +232,17 @@ export async function mount(root) {
const removeError = root.querySelector("#remove-confirm-error");
const removeCancelBtn = root.querySelector("#remove-confirm-cancel");
const removeRemoveBtn = root.querySelector("#remove-confirm-remove");
/* Phase 89: the per-source ignore-paths editor — the SAME page-local
alertdialog idiom as the remove-confirm modal (phase 69): static
markup in the shell's Sources view; this module owns the open /
cancel / save lifecycle. */
const ignoreDialog = root.querySelector("#ignore-editor-dialog");
const ignoreBackdrop = root.querySelector(".ignore-editor-backdrop");
const ignoreSourceEl = root.querySelector("#ignore-editor-source");
const ignoreTextarea = root.querySelector("#ignore-editor-textarea");
const ignoreErrorEl = root.querySelector("#ignore-editor-error");
const ignoreCancelBtn = root.querySelector("#ignore-editor-cancel");
const ignoreSaveBtn = root.querySelector("#ignore-editor-save");
/* Polite live region: the screen-reader confirmation for loads, adds,
and removals (the phase-15 announcer pattern). */
@@ -321,6 +361,16 @@ export async function mount(root) {
const code = document.createElement("code");
code.textContent = value; // rendered as text, never as HTML
urlTd.append(badge, code);
/* Phase 89: the "N ignored" count tag — stored rows with a
non-empty list only (text + a distinct background, never color
alone — WCAG 1.4.1). The list round-trips: (s.ignore_paths ||
[]) is the same expression openIgnoreEditor prefills from. */
if (s.id && (s.ignore_paths || []).length > 0) {
const count = document.createElement("span");
count.className = "git-source-ignore-count";
count.textContent = `${s.ignore_paths.length} ignored`;
urlTd.append(count);
}
tr.appendChild(urlTd);
const addedTd = document.createElement("td");
@@ -330,6 +380,22 @@ export async function mount(root) {
const actTd = document.createElement("td");
actTd.className = "git-source-actions-cell";
if (s.id) {
/* Phase 89: the per-row ignore-paths editor trigger (A3: stored
rows only — env-fallback rows fall through to the "from
.env" tag below). The aria-label is the ONLY place `value`
appears (setAttribute — never innerHTML); the text label is
static. Sits LEFT of Remove — the non-destructive action
first. */
const ignoreBtn = document.createElement("button");
ignoreBtn.type = "button";
ignoreBtn.className = "git-source-ignore";
ignoreBtn.setAttribute(
"aria-label",
`Edit ignored paths for ${kindLabel} source: ${value}`,
);
ignoreBtn.textContent = "Ignore paths";
ignoreBtn.addEventListener("click", () => openIgnoreEditor(s, ignoreBtn));
actTd.appendChild(ignoreBtn);
const btn = document.createElement("button");
btn.type = "button";
btn.className = "git-source-remove";
@@ -514,6 +580,168 @@ export async function mount(root) {
if (removeBackdrop) removeBackdrop.addEventListener("click", cancelRemoveConfirm);
if (removeRemoveBtn) removeRemoveBtn.addEventListener("click", confirmRemove);
/* ---------- ignore paths (PATCH /api/git-sources/{id}) — the per-row editor ----------
* Phase 89: a stored row's "Ignore paths" button (makeRow — left of
* Remove) opens the page-local alertdialog
* (#ignore-editor-dialog — the EXACT #remove-confirm-dialog idiom,
* phase 69) via openIgnoreEditor(s, triggerBtn): #ignore-editor-
* source names the source (textContent ONLY — the same `value`
* expression makeRow uses: s.path ?? s.url for local rows, s.url
* for git — URLs may embed user:pass@ credentials, phase 32), the
* mono textarea PREFILLS the row's stored list (one path per line
* — the phase-89 A1 prefix rule, stated in plain words in the
* helper copy), the error line clears, and focus lands on Cancel
* (the safe default). Escape / Cancel / the dim backdrop close as
* CANCEL: no request, focus returns to the row's trigger button.
*
* "Save" (saveIgnorePaths) runs the §7.4 never-stale lifecycle:
* both buttons disable and the save relabels "Saving…" while the
* PATCH is out — a blank line in the box is a separator, not an
* entry (split + trim + drop empty lines client-side; the server
* still rejects empty entries defensively, A4). 200 → close
* (focus return) → loadSources (the "N ignored" count tag lands —
* the A5 replace round-trips through GET) → announce (the update
* confirmation is the LAST announcement — the reload's "N sources
* listed." must not overwrite it); non-2xx → the in-dialog
* role="alert" line (the server detail, apiDetail 422-shape-aware)
* + the buttons restored — the dialog STAYS open and the textarea
* content is KEPT (the instruction survives — the tuning-form
* convention); network failure → the fixed reachable? line, same
* restore. Env-fallback rows (id null) carry NO button (A3 —
* nothing is stored to edit). */
let ignoreTarget = null; // the row object of the open editor
let ignoreTriggerBtn = null; // the row's button — focus returns here on close
let ignoreInFlight = false; // §7.4: a PATCH is out (both buttons disabled)
function openIgnoreEditor(s, triggerBtn) {
if (!ignoreDialog || !ignoreTextarea) return; // defensive — the markup ships with the page
if (ignoreInFlight) return; // one editor at a time
const isLocal = s.kind === "local";
// The same `value` expression makeRow uses — textContent ONLY.
if (ignoreSourceEl) ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url;
ignoreTextarea.value = (s.ignore_paths || []).join("\n");
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true; // a new attempt starts clean
}
ignoreInFlight = false;
ignoreTarget = s;
ignoreTriggerBtn = triggerBtn; // recorded for the focus return on close
ignoreDialog.hidden = false;
document.addEventListener("keydown", onIgnoreDialogKeydown);
// Cancel is the safe default — focus lands on it (the remove-
// dialog precedent; visibly: the global 3px :focus-visible
// outline).
if (ignoreCancelBtn) ignoreCancelBtn.focus();
}
/* Any close (cancel, success): hide the dialog, reset the textarea
+ error line, clear the target, detach the keydown handling, and
return focus to the row's "Ignore paths" button (WCAG 2.1). */
function closeIgnoreEditor() {
if (!ignoreDialog) return;
ignoreDialog.hidden = true;
ignoreInFlight = false;
if (ignoreTextarea) ignoreTextarea.value = "";
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true;
}
if (ignoreCancelBtn) ignoreCancelBtn.disabled = false;
if (ignoreSaveBtn) {
ignoreSaveBtn.disabled = false;
ignoreSaveBtn.textContent = "Save";
}
document.removeEventListener("keydown", onIgnoreDialogKeydown);
const trigger = ignoreTriggerBtn;
ignoreTarget = null;
ignoreTriggerBtn = null;
if (trigger) trigger.focus(); // focus returns to the row's button
}
/* Escape / Cancel / backdrop all close as cancel — NO request. A
cancel is a no-op while a PATCH is in flight (no half-cancel of
an in-progress save; the buttons are disabled anyway, the
Escape/backdrop paths need this guard). */
function cancelIgnoreEditor() {
if (ignoreInFlight) return;
closeIgnoreEditor();
}
/* While open (attached in openIgnoreEditor, detached in
closeIgnoreEditor): Escape closes as cancel. */
function onIgnoreDialogKeydown(e) {
if (e.key === "Escape") {
e.preventDefault();
cancelIgnoreEditor();
}
}
async function saveIgnorePaths() {
if (!ignoreTarget || ignoreInFlight) return; // one request at a time
// The box's lines: one path per line — a blank line is a
// separator, not an entry (trim + drop empty; the server still
// rejects empties defensively, A4).
const lines = ignoreTextarea.value.split("\n").map((l) => l.trim()).filter(Boolean);
const t = ignoreTarget;
const value = t.kind === "local" ? (t.path ?? t.url) : t.url;
ignoreInFlight = true;
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true;
}
if (ignoreCancelBtn) ignoreCancelBtn.disabled = true;
if (ignoreSaveBtn) {
ignoreSaveBtn.disabled = true;
ignoreSaveBtn.textContent = "Saving…"; // §7.4 in-flight label
}
try {
const r = await fetch(`/api/git-sources/${encodeURIComponent(t.id)}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ignore_paths: lines }),
});
if (r.ok) {
// 200: the server replaced the row's list (A5).
closeIgnoreEditor(); // focus returns to the row's button
await loadSources(); // the "N ignored" count tag lands
// The update confirmation is the LAST announcement: the
// reload's "N sources listed." must not overwrite it (the
// same order as the remove flow).
announce(`Ignored paths updated for ${value}`);
return;
}
// non-2xx: the in-dialog role="alert" line (the server detail)
// — the dialog STAYS open and the textarea content is KEPT:
// the fix is one edit + retry, not a re-type.
if (ignoreErrorEl) {
ignoreErrorEl.textContent = await apiDetail(r, "Could not save the ignored paths — try again.");
ignoreErrorEl.hidden = false;
}
} catch {
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "Could not save the ignored paths — is the app reachable?";
ignoreErrorEl.hidden = false;
}
} finally {
// Never stale (PLAN §7.4): the failure paths re-enable BOTH
// buttons + relabel the save; the success path already closed
// the dialog (which resets them) — the restore is a no-op
// there.
ignoreInFlight = false;
if (ignoreCancelBtn) ignoreCancelBtn.disabled = false;
if (ignoreSaveBtn) {
ignoreSaveBtn.disabled = false;
ignoreSaveBtn.textContent = "Save";
}
}
}
/* The dialog's own buttons (static markup — wired once). */
if (ignoreCancelBtn) ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor);
if (ignoreBackdrop) ignoreBackdrop.addEventListener("click", cancelIgnoreEditor);
if (ignoreSaveBtn) ignoreSaveBtn.addEventListener("click", saveIgnorePaths);
/* ---------- add (POST /api/git-sources) — the git form ----------
* wireAddForm gives the form the §7.4 never-stale lifecycle: while
* the request is out the button disables + relabels "Adding…" and
frontend/assets/styles.css
+215
View File
@@ -2183,6 +2183,173 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
color: var(--bg);
}
/* ---------- Per-source ignore-paths editor (phase 89, task 05) ----------
The shell's Sources view: the page-local alertdialog that edits
one source's ignore list (one path per line — the phase-89 A1
prefix rule, stated in plain words in the helper copy). The EXACT
#remove-confirm-dialog overlay contract (phase 69): a fixed
full-viewport dim backdrop + a centered panel (z-index 1000, above
the sticky header (20) + skip-link (100); NO blur — the phase-08
no-blur perf anchor), scaled to the 46rem chat-column width or the
viewport, whichever is narrower. The box: a VISIBLE block label
(WCAG — never aria-label-only) over a mono textarea (the box is a
data entry, not prose — the var(--mono) stack); the error line is
the err pair (err-ink on err-bg 9.3:1, the err-line border);
Cancel is the ghost ink-soft family (5.1:1 on --surface); Save is
the solid brand family (--bg text on --brand 5.2:1 — the
.new-chat-btn convention; the hover lightens the fill). Both
buttons >=44px; :focus-visible via the global 3px outline rule (no
local suppression — focus lands on Cancel at open and is visible);
no animation (reduced-motion safe by construction). AA pairs: title
/ copy / label are --ink (or ink-soft) on --surface (13.8:1 /
5.1:1); the source value is --ink on --bg (16.7:1). Phase-08
tokens only; system fonts; no CDN. */
.ignore-editor {
position: fixed;
inset: 0;
z-index: 1000;
display: flex; /* the panel is the only in-flow child — margin: auto centers it */
}
/* Explicit (the global [hidden] rule already wins — the documented,
testable contract for the skeleton). */
.ignore-editor[hidden] { display: none; }
.ignore-editor-backdrop {
position: fixed;
inset: 0;
/* --bg at 82% — the doc-modal dim, no backdrop-filter (no-blur). */
background: rgba(15, 10, 10, 0.82);
}
.ignore-editor-panel {
/* position:relative lifts the panel above the fixed backdrop
(positioned elements paint over in-flow siblings otherwise). */
position: relative;
z-index: 1;
margin: auto;
width: min(46rem, calc(100vw - 2rem));
padding: 1.5rem;
background: var(--surface);
border: 1px solid var(--line);
border-radius: var(--radius);
box-shadow: var(--shadow-lg);
}
.ignore-editor-title {
margin: 0 0 0.75rem;
font-size: 1.25rem;
line-height: 1.3;
color: var(--ink);
}
/* The source's value (git URL or local path) — mono, wrapped (a long
URL must not overflow the panel), --ink on --bg (16.7:1). */
.ignore-editor-source {
display: block;
margin: 0 0 0.75rem;
padding: 0.5rem 0.65rem;
font-family: var(--mono);
font-size: 0.85rem;
line-height: 1.5;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
overflow-wrap: anywhere;
}
.ignore-editor-copy {
margin: 0 0 1rem;
color: var(--ink); /* 13.8:1 on --surface */
}
.ignore-editor-copy code { font-family: var(--mono); font-size: 0.85em; }
/* Visible block label (WCAG — the textarea is NEVER labelled only via
aria-label); ink-soft on --surface (5.1:1). */
.ignore-editor-label {
display: block;
margin-bottom: 0.35rem;
color: var(--ink-soft);
font-size: 0.88rem;
font-weight: 600;
}
/* The box: full panel width, mono (the var(--mono) stack), theme-
consistent border on --bg; the focus-visible ring is the global
3px outline (no local suppression); vertical resize for long
lists (up to 200 entries, A4). */
.ignore-editor-textarea {
display: block;
width: 100%;
min-height: 9rem;
padding: 0.55rem 0.65rem;
font-family: var(--mono);
font-size: 0.85rem;
line-height: 1.5;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
resize: vertical;
}
/* The in-dialog failure line (role=alert): the err pair (err-ink on
err-bg 9.3:1, the err-line border) — the .remove-confirm-error
language. */
.ignore-editor-error {
margin: 0.75rem 0 0;
padding: 0.5rem 0.65rem;
color: var(--err-ink);
background: var(--err-bg);
border: 1px solid var(--err-line);
border-radius: var(--radius-sm);
}
.ignore-editor-actions {
display: flex;
justify-content: flex-end;
gap: 0.6rem;
margin-top: 1.1rem;
}
/* The two dialog buttons: >=44px targets, the house 3px :focus-visible
via the global outline rule (no local override). */
.ignore-editor-btn {
min-height: 44px;
min-width: 44px;
padding: 0.55rem 1.1rem;
border-radius: var(--radius-sm);
font: inherit;
font-weight: 600;
font-size: 0.9rem;
white-space: nowrap;
cursor: pointer;
}
.ignore-editor-btn:disabled { opacity: 0.5; cursor: wait; }
/* Cancel — the ghost ink-soft family (5.1:1 on --surface), like the
remove dialog's Cancel; the brand pair on hover (12.4:1 on
brand-soft). */
.ignore-editor-cancel {
border: 1px solid var(--line);
background: transparent;
color: var(--ink-soft);
}
.ignore-editor-cancel:hover:not(:disabled) {
background: var(--brand-soft);
color: var(--brand-ink);
}
/* "Save" — the solid brand family (the .new-chat-btn convention):
--bg text on --brand (5.2:1 — AA); the hover lightens the brand
fill (the .new-chat-btn hover pair). */
.ignore-editor-save {
border: 0;
background: var(--brand);
color: var(--bg);
}
.ignore-editor-save:hover:not(:disabled) { background: #f55a72; }
/* The list: the Sources page's table pattern — full width in the
72rem frame, surface card, horizontally scrollable wrapper (the
URL column never wraps or ellipsizes: long URLs, credentials
@@ -2266,6 +2433,54 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.git-source-remove:hover:not(:disabled) { background: var(--err-bg); color: var(--err-ink); border-color: var(--err-line); }
.git-source-remove:disabled { opacity: 0.5; cursor: wait; }
/* Phase 89: the per-row "Ignore paths" trigger (makeRow — left of
Remove): the .git-source-remove idiom (same size/spacing, >=44px
target) in a NEUTRAL secondary fill — it must read distinct from
the destructive Remove (text label included, never icon-only; the
hover takes the brand-soft pair, 12.4:1 — Remove hovers to the err
pair, 9.3:1). */
.git-source-ignore {
display: inline-flex;
align-items: center;
justify-content: center;
min-height: 44px;
min-width: 44px;
flex: 0 0 auto;
padding: 0.35rem 0.7rem;
margin-right: 0.4rem;
border: 1px solid var(--line);
border-radius: var(--radius-sm);
background: transparent;
color: var(--ink-soft);
font: inherit;
font-weight: 600;
font-size: 0.82rem;
white-space: nowrap;
cursor: pointer;
}
.git-source-ignore:hover:not(:disabled) {
background: var(--brand-soft);
color: var(--brand-ink);
}
/* Phase 89: the "N ignored" count tag — next to the location <code>
in the Source cell (TEXT + a distinct background, never color
alone — WCAG 1.4.1): --ink on --bg (16.7:1) inside the --surface
card; the line border keeps the chip legible when the row hover
swaps the cell onto --bg too. */
.git-source-ignore-count {
display: inline-block;
margin-left: 0.55rem;
padding: 0.08rem 0.5rem;
border: 1px solid var(--line);
border-radius: 999px;
font-size: 0.72rem;
font-weight: 600;
color: var(--ink);
background: var(--bg);
white-space: nowrap;
}
/* Env-fallback rows carry no Remove (nothing is stored to remove) —
the tag says where the row comes from (brand pair, 6.9:1). */
.git-source-env-tag {
frontend/index.html
+38
View File
@@ -647,6 +647,44 @@
</div>
</div>
</div>
<!-- Phase 89: the per-source ignore-paths editor — the EXACT
#remove-confirm-dialog pattern (phase 69): page-local
alertdialog, dim backdrop, focus on Cancel (the safe
default), Escape/backdrop close as CANCEL, focus returns
to the row's trigger button. The box itself: one path
per line (prefix semantics — the phase-89 A1 rule,
stated in plain words in the helper line); Save runs the
§7.4 never-stale lifecycle (git-sources.js: "Saving…"
while the PATCH is out). Stored rows only (A3) — the
per-row button lives in git-sources.js's makeRow; this
is the page-local dialog it opens. -->
<div class="ignore-editor" id="ignore-editor-dialog" role="alertdialog"
aria-modal="true" aria-labelledby="ignore-editor-title"
aria-describedby="ignore-editor-copy" hidden>
<div class="ignore-editor-backdrop" aria-hidden="true"></div>
<div class="ignore-editor-panel">
<h2 class="ignore-editor-title" id="ignore-editor-title">Ignored files and folders</h2>
<code class="ignore-editor-source" id="ignore-editor-source"></code>
<p class="ignore-editor-copy" id="ignore-editor-copy">
One path per line. A file is ignored when its path in the
source starts with any of these — <code>/my/files/</code>,
<code>my/files/</code> and <code>my/files</code> mean the same
thing. No wildcards, no matching in the middle of a path.
</p>
<label class="ignore-editor-label" for="ignore-editor-textarea">Ignored paths</label>
<textarea class="ignore-editor-textarea" id="ignore-editor-textarea"
rows="6" spellcheck="false"
placeholder="my/files/"></textarea>
<p class="ignore-editor-error" id="ignore-editor-error" role="alert" hidden></p>
<div class="ignore-editor-actions">
<button type="button" class="ignore-editor-btn ignore-editor-cancel"
id="ignore-editor-cancel">Cancel</button>
<button type="button" class="ignore-editor-btn ignore-editor-save"
id="ignore-editor-save">Save</button>
</div>
</div>
</div>
</div>
<!-- Polite live region: the screen-reader confirmation for list
loads, adds, and removals (git-sources.js owns the text). -->
scripts/import_docs.py
+37 -13
View File
@@ -26,6 +26,11 @@ precedence order:
``~/Deployments``), kept for backwards compatibility (reached only
while both the table and ``BOR_GIT_SOURCES`` are empty).
Phase 89: resolution also returns each row's ignore paths, keyed by
the resolved root string (the importer normalizes them); manual
``--source`` dirs and the legacy fallback have no rows, so they import
with no ignore.
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add
@@ -133,9 +138,11 @@ def repo_name(url: str) -> str:
return slug or "repo"
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
def _resolve_sources(
cli_sources: list[Path] | None, settings: Settings
) -> tuple[list[Path], dict[str, list[str]]]:
"""Resolve the directories to import (phase 28, extended in phases
35 and 38).
35 and 38; per-root ignore maps, phase 89).
Precedence: ``--source`` (explicit manual paths — always wins) >
the effective sources — the ``git_sources`` DB rows (git + local),
@@ -148,12 +155,18 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
stored directory, re-verified ``.is_dir()`` at run time) > the
legacy ``DEFAULT_SOURCES``.
Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed
by the resolved root string, exactly as the importer sees it (two
rows sharing a root string get the union — extend, not replace);
manual ``--source`` dirs and the legacy fallback have no rows, so
they import with an empty map (no ignore).
A :class:`GitSyncError` from a failing clone/pull — or a missing
local directory (``local source missing: <path>``) — propagates to
:func:`main`, which aborts the run before importing anything.
"""
if cli_sources:
return [path.expanduser() for path in cli_sources]
return [path.expanduser() for path in cli_sources], {}
db = SessionLocal()
try:
rows, origin = effective_sources(db)
@@ -167,21 +180,28 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
)
sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
for row in rows:
if row.kind == "git":
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
else:
# kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the
# ``or`` keeps the type checker honest); a missing
# directory aborts before importing, the same pre-import
# fail-loud as a failing git clone.
path = Path(row.path or row.url).expanduser()
if not path.is_dir():
raise GitSyncError(f"local source missing: {path}")
sources.append(path)
return sources
return [path.expanduser() for path in DEFAULT_SOURCES]
root = Path(row.path or row.url).expanduser()
if not root.is_dir():
raise GitSyncError(f"local source missing: {root}")
sources.append(root)
# Phase 89: the row's ignore list, keyed by the SAME root
# string the importer sees; two rows sharing a root string
# get the union (extend, not replace) — the sibling/repo-name
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
return sources, ignore_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}
def _overview_row_exists() -> bool:
@@ -204,9 +224,10 @@ def main(argv: list[str] | None = None) -> int:
# Git sources resolve (and clone/pull) *before* any import: a failing
# repo aborts the run with a non-zero exit, naming the failure — a bad
# URL must never silently import partial junk.
# URL must never silently import partial junk. The second element is
# the phase-89 per-root ignore map (empty for manual/fallback paths).
try:
sources = _resolve_sources(args.source, settings)
sources, ignore_by_root = _resolve_sources(args.source, settings)
except GitSyncError as e:
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
return 1
@@ -248,7 +269,10 @@ def main(argv: list[str] | None = None) -> int:
runs and unchanged re-runs never bump. The returned token is
the new version, or ``"skipped"``.
"""
summary = await import_sources(sources, llm, prune=args.prune, limit=args.limit)
summary = await import_sources(
sources, llm, prune=args.prune, limit=args.limit,
ignore_by_root=ignore_by_root,
)
if args.limit is not None:
# An incomplete walk is debug-only — it must never advance
# the generation (mirrors the --limit overview skip below).
tests/e2e/test_source_ignore_paths.py
+640
View File
@@ -0,0 +1,640 @@
"""Phase 89 story E2E (Playwright): per-source ignore paths — the box on
the Sources page excludes files/folders from the import (``/git-sources.html``).
Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-08 —
TODO-derived, no separate user-story file).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov
The story gate proves the whole item end to end through the REAL page +
REAL API + REAL sync pipeline (the in-app Sync's background task, mock
LLM). **No git, no network** — the suite's single source is a local
directory row (``kind="local"``, phase 38), so the sync walks the
fixture dir directly: no clone, no pull, no remote of any kind. The
module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL
(the ``test_git_sources_admin.py`` module-env pattern) that is NEVER
synced or cloned: every sync in this suite runs with the local DB row
in place (DB rows win over the env fallback), and the one
env-fallback test never triggers a sync.
Contract under test:
* anonymous: the sign-in gate (the ``#git-sources-gate`` pattern), the
manager hidden, NO ``/api/git-sources`` call on load, 403 on
``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}``
(the ``test_git_sources_admin.py`` anonymous pin, extended to the
phase-89 route);
* admin: the per-row "Ignore paths" button opens the page-local
alertdialog (the phase-69 ``#remove-confirm-dialog`` idiom); one path
per line; Save → ``PATCH`` 200 → the row shows the "N ignored" count
tag; the list round-trips through ``GET /api/git-sources``;
* the sync honors the list: with ``ignore/`` set, nothing under
``ignore/`` is indexed — no ``documents`` rows (and therefore no
chunks/embeddings, no summary calls) — and the sync's ``files``
count excludes the ignored files from the walk;
* the spec's no-mid-path rule: ``myfile.md`` ignores the root-level
``myfile.md`` only, never ``sub/myfile.md``; saving the box
REPLACES the list (A5 — the previous test's list is gone);
* A2: a previously indexed file that newly matches an ignore pattern
is PRUNED from the KB on the next sync (``detail.pruned``);
* a11y + the error path: the dialog is a real ``role="alertdialog"``,
the textarea has a visible label, focus lands on Cancel, Escape
closes and returns focus to the trigger; a 501-char entry 422s with
the fixed detail, the textarea content is KEPT and the Save button
re-enables with its "Save" label; the happy path heals the error
state;
* A3: env-fallback rows (table empty → ``BOR_GIT_SOURCES``) render the
"from .env" tag with NO "Ignore paths" button and the env note.
Test → contract mapping:
1. ``test_anonymous_gate_and_403s``
2. ``test_ignore_box_excludes_from_import``
3. ``test_prefix_rule_no_mid_path``
4. ``test_newly_ignored_file_is_pruned``
5. ``test_editor_a11y_and_error_path``
6. ``test_env_fallback_rows_have_no_box``
"""
from __future__ import annotations
import os
import subprocess
import sys
import time
import uuid
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_IGNORE", "8140"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is
#: NEVER cloned (the env-fallback test reads it; the sync tests all run
#: with the local DB row in place, so the env list never reaches a
#: clone).
ENV_SOURCE = "https://github.com/reese/env-alpha.git"
#: The sentinel inside the fixture's keeper doc — proof the (non-
#: ignored) markdown path was really imported.
SENTINEL = "RESE-IGNORE-KEEP-7f3a"
#: The six fixture files (source-relative POSIX paths — exactly the
#: strings ``documents.path`` stores).
KEEP_MD = "keep.md"
NOTES_YAML = "notes.yaml"
ROOT_MYFILE = "myfile.md"
SUB_MYFILE = "sub/myfile.md"
IGNORE_SECRET = "ignore/secret.md"
IGNORE_DEEP = "ignore/deep/x.txt"
ALL_SIX = (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE, IGNORE_SECRET, IGNORE_DEEP)
#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the
#: test_sync_button.py polling idiom) — real import of ≤6 small files
#: against the mock LLM; generous budget.
SYNC_TIMEOUT_S = 60.0
SYNC_TICK_S = 2.0
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The local source's on-disk tree (task 06 step 1).
Module-scoped (the same reasoning as ``test_sync_button.py``'s
module-scoped fixture note): ``tmp_path`` is function-scoped while
the module-scoped app + the seeded row reference the dir for the
module's lifetime, so it is built under ``tmp_path_factory`` (the
same pytest-managed temp area, module-safe). Plain files — no git:
a ``kind="local"`` row is walked directly by the sync (phase 38).
"""
root = tmp_path_factory.mktemp("ignore_src")
(root / "sub").mkdir()
(root / "ignore" / "deep").mkdir(parents=True)
(root / KEEP_MD).write_text(
"# Keep\n"
"\n"
"The keeper doc — the one markdown file the ignore list must\n"
"never touch.\n"
f"\n"
f"Marker: {SENTINEL}\n",
encoding="utf-8",
)
# Non-markdown → the phase-30 lite-summary path (proves "not
# summarized" applies to the non-md branch too when ignored).
(root / NOTES_YAML).write_text(
"title: Notes\nitems:\n - one\n - two\n", encoding="utf-8"
)
(root / ROOT_MYFILE).write_text(
"# Root myfile\n\nRoot-level file — matched by the bare "
"``myfile.md`` prefix (the spec's own example).\n",
encoding="utf-8",
)
(root / SUB_MYFILE).write_text(
"# Sub myfile\n\nA namesake in a subdirectory — NEVER matched "
"by ``myfile.md`` (no mid-path matching).\n",
encoding="utf-8",
)
(root / IGNORE_SECRET).write_text(
"# Secret\n\nIgnored by the ``ignore/`` prefix.\n", encoding="utf-8"
)
(root / IGNORE_DEEP).write_text("deep ignored text\n", encoding="utf-8")
return root
@pytest.fixture(scope="module")
def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]:
"""The real app under test — per-module env (the conftest pattern,
module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env
fallback's subject — never cloned), module-scratch checkouts/
upload dirs, the mock LLM. No git anywhere in this suite's path."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env
# shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The empty-table env fallback (only read while ``git_sources`` is
# EMPTY — the env-fallback test; never cloned in this suite).
env["BOR_GIT_SOURCES"] = ENV_SOURCE
# Module-scratch dirs (never reached by this suite's local-row
# syncs — kept explicit so a shared checkouts dir cannot leak rows
# into the walk).
scratch = source_dir.parent
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern, the
test_sync_button.py module-env DSN): the sync's counts and the
catalog must be each test's own doing."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources"
)
)
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""This suite owns ``git_sources`` AND the KB tables (the E2E
isolation pattern): suites run in isolation but share one
Postgres, and a leftover row would flip the app from the
``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover
document would skew the prune counts). Empty BOTH before and
after every test."""
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str:
"""(Re)seed the stored row for the module fixture dir via the REAL
admin API (201) — ``kind="local"`` (the POST contract is unchanged
by phase 89; no ``ignore_paths`` → the stored list is ``[]``) —
then reload the view so the page's table re-fetches the row (the
API seed is server-side: the mounted view does not know about it
without a re-show). Returns the stored (expanded) path — the
row's ``url`` location value, the same string the page renders
and aria-labels."""
r = page.request.post(
f"{app_url}/api/git-sources",
data={"kind": "local", "path": str(source_dir)},
)
assert r.status == 201, r.text
body = r.json()
assert body["ignore_paths"] == [] # the default list, round-tripped
page.reload()
_admin_git_sources_page(page, app_url)
return body["url"]
def run_sync(page: Page, app_url: str) -> dict[str, Any]:
"""``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal
state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s
budget). Returns the terminal status body."""
r = page.request.post(f"{app_url}/api/sync")
assert r.status == 202, r.text
deadline = time.monotonic() + SYNC_TIMEOUT_S
body: dict[str, Any] = {}
while True:
s = page.request.get(f"{app_url}/api/sync/status")
assert s.status == 200, s.text
body = s.json()
if body["state"] in ("success", "failed"):
return body
assert time.monotonic() < deadline, (
f"sync did not reach a terminal state: {body}"
)
time.sleep(SYNC_TICK_S)
def _catalog_paths(page: Page, app_url: str) -> list[str]:
"""The RAG catalog's data source (``GET /api/docs`` — the Sources
page's table): every ``documents.path`` (source-relative POSIX)."""
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200, r.text
return [d["path"] for d in r.json()["documents"]]
def _row(page: Page, value: str) -> Any:
"""The table row whose mono location cell shows ``value``."""
return page.locator("#git-sources-tbody tr", has_text=value)
def _open_ignore_editor(page: Page, value: str) -> Any:
"""Click the row's "Ignore paths" button; the alertdialog opens.
Returns the trigger button (focus returns to it on close)."""
btn = _row(page, value).locator(".git-source-ignore")
expect(btn).to_have_count(1)
btn.click()
dialog = page.locator("#ignore-editor-dialog")
expect(dialog).to_be_visible(timeout=30_000)
return btn
def _save_ignore_list(page: Page, value: str, entry: str) -> None:
"""Open the row's box, replace its content with one line (``entry``
— ``""`` clears) and Save; wait for the A5 round-trip: the dialog
closes and the row shows the "1 ignored" / cleared count state."""
_open_ignore_editor(page, value)
page.fill("#ignore-editor-textarea", entry)
page.click("#ignore-editor-save")
if entry.strip():
expect(_row(page, value).locator(".git-source-ignore-count")).to_have_text(
"1 ignored", timeout=30_000
)
else:
expect(_row(page, value).locator(".git-source-ignore-count")).to_have_count(0)
expect(page.locator("#ignore-editor-dialog")).to_be_hidden()
# ---------------------------------------------------------------------------
# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. PATCH)
# ---------------------------------------------------------------------------
def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None:
page.set_default_timeout(30_000)
# Track every /api/git-sources request the page itself makes — the
# gate must be reached WITHOUT touching the admin API (the
# test_git_sources_admin.py pattern).
api_calls: list[str] = []
page.on(
"request",
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
)
page.goto(app_url + GIT_SOURCES_URL)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
# The sign-in gate (the #sources-gate pattern, phase 16)…
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
expect(
gate.locator("a[href='/login.html?next=/git-sources.html']")
).to_have_count(1)
# …and the manager is absent/inert: table + form + env note, all
# inside the hidden #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers on GET/POST AND the
# phase-89 PATCH (the page context has no cookie — the
# test_git_sources_admin.py anonymous pin, extended).
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
f"{app_url}/api/git-sources", data={"url": ENV_SOURCE}
).status
== 403
)
assert (
page.request.patch(
f"{app_url}/api/git-sources/{uuid.uuid4()}",
data={"ignore_paths": ["ignore/"]},
).status
== 403
)
# ---------------------------------------------------------------------------
# 2. The box sets the list; the sync honors it (nothing under ignore/
# embedded/summarized; the walk's files count excludes it)
# ---------------------------------------------------------------------------
def test_ignore_box_excludes_from_import(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Seed the row (no list) through the real API…
stored_path = _seed_local_source(page, app_url, source_dir)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# …open the box, type "ignore/" (one line), Save → 200 → the row's
# "1 ignored" count tag lands (the A5 round-trip through GET).
_save_ignore_list(page, stored_path, "ignore/")
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
# A1: the raw box line is stored NORMALIZED ("ignore/" → "ignore")
# — the round-trip is canonical.
assert [s["ignore_paths"] for s in body["sources"]] == [["ignore"]]
assert body["from_env"] is False
# The real sync (mock LLM): success, and the walk counted the
# remaining FOUR files — the two ignore/ files never entered it
# (no documents → no chunks/embeddings, no summary calls for
# them). (6 fixture files − 2 ignored = 4; the task's "5" is a
# slip against its own six-file tree — the exclusion is what is
# under test, and it holds either way.)
sync = run_sync(page, app_url)
assert sync["state"] == "success", sync
detail = sync["detail"]
assert detail["files"] == 4, detail
assert detail["added"] == 4, detail
assert detail["errors"] == 0, detail
# The catalog holds exactly the four non-ignored paths.
paths = _catalog_paths(page, app_url)
for p in (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE):
assert p in paths, f"{p} missing from the catalog: {paths}"
assert not any(p.startswith("ignore/") for p in paths), (
f"ignored paths leaked into the catalog: {paths}"
)
# ---------------------------------------------------------------------------
# 3. The spec's no-mid-path rule: myfile.md ignores the root-level file
# only; the box REPLACES the previous test's list (A5)
# ---------------------------------------------------------------------------
def test_prefix_rule_no_mid_path(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# Bare filename prefix (no leading/trailing slash): the spec's own
# example — matches the root-level file, never the subdirectory
# namesake.
_save_ignore_list(page, stored_path, "myfile.md")
sync = run_sync(page, app_url)
assert sync["state"] == "success", sync
assert sync["detail"]["files"] == 5, sync["detail"]
paths = _catalog_paths(page, app_url)
# No mid-path matching: the namesake in sub/ is indexed…
assert SUB_MYFILE in paths, f"sub/myfile.md was wrongly ignored: {paths}"
# …the root-level file is NOT…
assert ROOT_MYFILE not in paths, f"root myfile.md must be ignored: {paths}"
# …and the ignore/ files ARE indexed this time — the box replaced
# (A5), not appended to, the previous test's "ignore/" list
# (regression guard).
assert IGNORE_SECRET in paths, f"ignore/secret.md must be indexed: {paths}"
assert IGNORE_DEEP in paths, f"ignore/deep/x.txt must be indexed: {paths}"
# ---------------------------------------------------------------------------
# 4. A2: a previously indexed file that newly matches is pruned
# ---------------------------------------------------------------------------
def test_newly_ignored_file_is_pruned(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# Sync 1 — no list: all six files are indexed.
first = run_sync(page, app_url)
assert first["state"] == "success", first
assert first["detail"]["files"] == 6, first["detail"]
assert first["detail"]["added"] == 6, first["detail"]
paths = _catalog_paths(page, app_url)
for p in ALL_SIX:
assert p in paths, f"{p} missing after the first sync: {paths}"
# New list: "ignore/" — the two previously indexed files now match.
_save_ignore_list(page, stored_path, "ignore/")
# Sync 2 — the prune=True run deletes the newly ignored documents
# (the seen-set mechanism, the A9 out-of-scope-junk precedent).
second = run_sync(page, app_url)
assert second["state"] == "success", second
detail = second["detail"]
assert detail["pruned"] == 2, detail
assert detail["files"] == 4, detail
paths = _catalog_paths(page, app_url)
for p in (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE):
assert p in paths, f"{p} missing after the prune: {paths}"
assert not any(p.startswith("ignore/") for p in paths), (
f"newly ignored paths survived the prune: {paths}"
)
# ---------------------------------------------------------------------------
# 5. The editor's a11y + the 422 error path (and the happy-path heal)
# ---------------------------------------------------------------------------
def test_editor_a11y_and_error_path(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# The trigger is the labeled per-row button (left of Remove)…
row = _row(page, stored_path)
btn = row.locator(".git-source-ignore")
expect(btn).to_have_count(1)
expect(btn).to_have_attribute(
"aria-label", f"Edit ignored paths for local source: {stored_path}"
)
expect(row.locator(".git-source-ignore-count")).to_have_count(0) # no list yet
# …and it opens a real alertdialog (the phase-69 idiom) with the
# visible label and the Cancel-safe focus default.
btn.click()
dialog = page.locator("#ignore-editor-dialog")
expect(dialog).to_be_visible(timeout=30_000)
assert dialog.get_attribute("role") == "alertdialog"
assert dialog.get_attribute("aria-modal") == "true"
# exact: the row button's aria-label ("Edit ignored paths for …")
# contains the substring — the visible <label> is the only exact
# "Ignored paths" name.
expect(page.get_by_label("Ignored paths", exact=True)).to_have_count(1)
# The dialog names the source (textContent — the local path).
expect(page.locator("#ignore-editor-source")).to_have_text(stored_path)
expect(page.locator("#ignore-editor-cancel")).to_be_focused()
# Escape closes as CANCEL — no request — and focus returns to the
# row's trigger button.
page.keyboard.press("Escape")
expect(dialog).to_be_hidden()
expect(btn).to_be_focused()
# The 422 path: one entry of 501 chars (> the A4 500-char limit) →
# the fixed detail, the textarea content KEPT, Save re-enabled with
# its idle label (never stale).
btn.click()
expect(dialog).to_be_visible(timeout=30_000)
bad = "a" * 501
page.fill("#ignore-editor-textarea", bad)
page.click("#ignore-editor-save")
error = page.locator("#ignore-editor-error")
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_have_text("an ignore path exceeds 500 characters")
expect(page.locator("#ignore-editor-textarea")).to_have_value(bad)
expect(dialog).to_be_visible() # the fix is one edit + retry
save_btn = page.locator("#ignore-editor-save")
expect(save_btn).to_be_enabled()
expect(save_btn).to_have_text("Save")
# Nothing was stored: the list is still empty (no count tag).
expect(row.locator(".git-source-ignore-count")).to_have_count(0)
# The happy path heals the error state: clear, one entry, Save →
# the dialog closes and the "1 ignored" tag lands.
page.fill("#ignore-editor-textarea", "keep.md")
save_btn.click()
expect(dialog).to_be_hidden(timeout=30_000)
expect(
page.locator("#git-sources-tbody tr .git-source-ignore-count")
).to_have_text("1 ignored", timeout=30_000)
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert [s["ignore_paths"] for s in r.json()["sources"]] == [["keep.md"]]
# ---------------------------------------------------------------------------
# 6. A3: env-fallback rows have no box (no DB row to store a list on)
# ---------------------------------------------------------------------------
def test_env_fallback_rows_have_no_box(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# The autouse fixture truncated git_sources — the table is EMPTY,
# so the module app's BOR_GIT_SOURCES URL is the effective list
# (the test_git_sources_admin.py module-env pattern).
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
row = page.locator("#git-sources-tbody tr")
expect(row).to_contain_text(ENV_SOURCE)
# The env-fallback surface: the "from .env" tag (never color
# alone)…
expect(row.locator(".git-source-env-tag")).to_have_count(1)
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
# …and NO per-row actions at all — neither Remove (nothing stored
# to remove) nor the phase-89 "Ignore paths" box (A3).
expect(row.locator(".git-source-remove")).to_have_count(0)
expect(row.locator(".git-source-ignore")).to_have_count(0)
expect(row.locator(".git-source-ignore-count")).to_have_count(0)
# The env-fallback note explains the active list's origin…
expect(page.locator("#git-sources-env-note")).to_be_visible()
# …and the API agrees: from_env true, null ids, the env URL, and an
# empty ignore list (no DB row to store a list on).
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert body["from_env"] is True
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE]
assert all(
s["id"] is None and s["ignore_paths"] == [] for s in body["sources"]
)
tests/integration/test_git_sources_api.py
+226 -4
View File
@@ -28,7 +28,14 @@ Contract under test:
index as backstop); wrong field combinations (git without url, local
without path, both kinds' fields) → 422;
* DELETE — 204 and gone; an emptied table falls back to the env list
again; unknown id → 404.
again; unknown id → 404;
* ignore paths (phase 89) — POST accepts the RAW box lines for both
kinds (optional, absent → ``[]``), stored normalized (A1) with the A4
fixed-detail 422s (shared gate with PATCH); GET reports each row's
stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only)
replaces the list wholesale (A5 — an empty list clears all), 404
unknown id, 422 fixed details for the A4 limits, the row otherwise
unchanged.
``git_sources`` is global state: truncated around every test.
"""
@@ -92,6 +99,10 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# The phase-89 ignore-list PATCH is gated the same way.
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# Nothing landed in the table.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
@@ -121,6 +132,8 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "https://a.example.com/one.git",
"path": None,
"added_at": None,
# Env rows have no DB row to store a list on (phase 89).
"ignore_paths": [],
},
{
"id": None,
@@ -128,6 +141,7 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "git@b.example.com:two.git",
"path": None,
"added_at": None,
"ignore_paths": [],
},
]
@@ -193,7 +207,10 @@ def test_post_creates_trimmed_and_list_stops_using_env(
assert body["url"] == "https://new.example.com/repo.git" # trimmed
uuid.UUID(body["id"])
assert body["added_at"] is not None
assert set(body) == {"id", "url", "added_at"}
# Phase 89: the response gains ``ignore_paths`` — absent at create
# time → ``[]``.
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert body["ignore_paths"] == []
# The DB row now wins: from_env False, the env URL is gone from the list.
body = admin_client.get("/api/git-sources").json()
@@ -303,8 +320,10 @@ def test_post_local_creates_stored_row_with_expanded_path(
assert r.status_code == 201, r.text
body = r.json()
# The phase-35 response shape is unchanged — the local row reports
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET.
assert set(body) == {"id", "url", "added_at"}
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
# phase 89 adds ``ignore_paths`` (absent → ``[]``).
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert body["ignore_paths"] == []
uuid.UUID(body["id"])
assert body["url"] == str(real_dir)
assert body["added_at"] is not None
@@ -536,6 +555,7 @@ def test_delete_removes_row_and_falls_back_to_env(
"url": "https://env.example.com/env.git",
"path": None,
"added_at": None,
"ignore_paths": [], # env rows: no DB row to store a list on
}
]
db.execute(text("TRUNCATE chunks, documents"))
@@ -550,3 +570,205 @@ def test_delete_unknown_id_returns_404(admin_client: TestClient) -> None:
def test_delete_invalid_id_returns_422(admin_client: TestClient) -> None:
assert admin_client.delete("/api/git-sources/not-a-uuid").status_code == 422
# --- ignore paths (phase 89) ----------------------------------------------
def test_post_stores_normalized_ignore_paths_both_kinds(
admin_client: TestClient, tmp_path: Path
) -> None:
"""POST accepts the RAW box lines for both kinds (A1 normalization
in the API layer) — the normalized list is stored and round-trips
through GET."""
r = admin_client.post(
"/api/git-sources",
json={
"url": "https://example.com/ig.git",
"ignore_paths": ["/my/files/", " my/files2 ", "x"],
},
)
assert r.status_code == 201, r.text
assert r.json()["ignore_paths"] == ["my/files", "my/files2", "x"]
real_dir = tmp_path / "ig"
real_dir.mkdir()
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "path": str(real_dir), "ignore_paths": ["//skip/", "keep"]},
)
assert r.status_code == 201, r.text
assert r.json()["ignore_paths"] == ["skip", "keep"]
# Round-trip through GET (keyed by url — (added_at, id) order of two
# same-millisecond inserts is not the point under test).
by_url = {
s["url"]: s["ignore_paths"] for s in admin_client.get("/api/git-sources").json()["sources"]
}
assert by_url["https://example.com/ig.git"] == ["my/files", "my/files2", "x"]
assert by_url[str(real_dir)] == ["skip", "keep"]
def test_post_without_ignore_paths_reports_empty_list(
admin_client: TestClient, db: Session
) -> None:
"""Absent ``ignore_paths`` at create time → ``[]`` — both in the 201
response and in the GET round-trip (the migration's server default)."""
r = admin_client.post("/api/git-sources", json={"url": "https://example.com/none.git"})
assert r.status_code == 201
assert r.json()["ignore_paths"] == []
body = admin_client.get("/api/git-sources").json()
assert body["sources"][0]["ignore_paths"] == []
# The stored value is the JSONB server default, not Python-only.
row = db.scalars(select(GitSource)).one()
assert row.ignore_paths == []
def test_post_rejects_invalid_ignore_paths_like_patch(
admin_client: TestClient, db: Session
) -> None:
"""POST shares ``_validate_ignore_paths`` with PATCH — the same A4
fixed 422 details; nothing is stored."""
for payload, detail in (
([" "], "ignore paths must be non-empty"),
([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"),
(["a" * 501], "an ignore path exceeds 500 characters"),
):
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/bad-ignore.git", "ignore_paths": payload},
)
assert r.status_code == 422, f"{detail!r}: {r.text}"
assert r.json()["detail"] == detail
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_get_reports_stored_ignore_paths_and_env_rows_empty(
admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch
) -> None:
"""GET reports each DB row's stored list; env-fallback rows (no DB
row to store a list on) report ``[]`` with ``from_env: true``."""
db.add(GitSource(url="https://example.com/stored.git", ignore_paths=["docs/drafts"]))
db.commit()
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is False
assert body["sources"][0]["ignore_paths"] == ["docs/drafts"]
db.execute(text("DELETE FROM git_sources"))
db.commit()
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: _settings("https://a.example.com/env.git"),
)
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is True
assert body["sources"] == [
{
"id": None,
"kind": "git",
"url": "https://a.example.com/env.git",
"path": None,
"added_at": None,
"ignore_paths": [],
}
]
def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -> None:
"""PATCH 200 — REPLACE semantics (A5): the body list, normalized,
becomes the row's whole list; an empty list clears all; ``id``/
``url``/``added_at`` are unchanged; the response is the
``GitSourceOut`` shape incl. the new list."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/patch.git", "ignore_paths": ["old/"]},
)
assert created.status_code == 201
before = created.json()
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert body["ignore_paths"] == ["a", "b"] # normalized
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Round-trip through GET.
assert (
admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a", "b"]
)
# An empty list clears all.
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": []})
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == []
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == []
def test_patch_missing_field_is_422(admin_client: TestClient) -> None:
"""``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent
field is a 422 with the model's own detail."""
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"})
assert created.status_code == 201
r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={})
assert r.status_code == 422
# The model's own (Pydantic) detail — the missing required field.
assert any(
item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"]
)
def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None:
r = admin_client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
assert r.status_code == 404
assert r.json() == {"detail": "git source not found"}
def test_patch_invalid_id_returns_422(admin_client: TestClient) -> None:
r = admin_client.patch("/api/git-sources/not-a-uuid", json={"ignore_paths": []})
assert r.status_code == 422
def test_patch_422s_are_fixed_details(admin_client: TestClient) -> None:
"""The A4 422s are exact fixed strings (never echoing the input),
and a rejected PATCH leaves the row's list unchanged."""
created = admin_client.post(
"/api/git-sources", json={"url": "https://example.com/v.git", "ignore_paths": ["keep"]}
)
assert created.status_code == 201
sid = created.json()["id"]
for payload, detail in (
# Whitespace-only → empty after normalization: 422, not a silent
# drop (the UI drops blank lines client-side; the API is
# defensive).
([" "], "ignore paths must be non-empty"),
(list(f"e{i}" for i in range(201)), "a source has at most 200 ignore paths"),
(["a" * 501], "an ignore path exceeds 500 characters"),
):
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": payload})
assert r.status_code == 422, f"{detail!r}: {r.text}"
assert r.json()["detail"] == detail
# Nothing changed by the failed PATCHes.
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["keep"]
def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None:
"""Exactly 200 entries and a 500-char entry (post-normalization)
are the accepted edge of A4."""
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/bnd.git"})
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": [f"e/{i}" for i in range(200)]}
)
assert r.status_code == 200, r.text
assert len(r.json()["ignore_paths"]) == 200
long_entry = "a" * 500
assert len(long_entry) == 500
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]})
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == [long_entry]
tests/integration/test_git_sources_upload.py
+90
View File
@@ -291,6 +291,7 @@ class _GatedImport:
limit: int | None = None,
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
) -> ImportSummary:
self.prune_flags.append(prune)
if progress is not None:
@@ -1352,3 +1353,92 @@ def test_models_down_fails_the_run_and_leaves_folder_and_row(
# The scan never ran: no docs, no stray temps.
assert _docs(upload_client) == []
assert [p.name for p in uploads.iterdir()] == ["homelab"]
# --- phase 89: re-uploads honor the row's saved ignore list -------------------
def test_reupload_honors_saved_ignore_list(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase 89: a re-upload of an EXISTING source honors the ignore
list saved on its row. First scan (fresh row, no list) indexes the
ignored file too; once the list is saved via the API, re-uploading
the same archive again scans only the kept file — the previously
indexed ignored file leaves the KB (prune), and the run lands
``success`` with counts that exclude it. The upload itself keeps
every file on disk: the ignore is about the index, not the folder."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"keep.md": "# Keep\nin scope\n",
"ignore/secret.md": "# Secret\nignored\n",
}
r = _post(upload_client, "docs.tar.gz", _targz_bytes(files))
assert r.status_code == 202, r.text
status = _wait_status(upload_client)
assert status["state"] == "success", status
# First scan: the fresh row has no list → both files are indexed.
assert status["detail"]["files"] == 2
assert _docs(upload_client) == [("docs", "ignore/secret.md"), ("docs", "keep.md")]
# Save the ignore list on the row via the API (phase 89, task 03).
folder = uploads / "docs"
row = _row(db, str(folder))
assert row is not None
r = upload_client.patch(
f"/api/git-sources/{row.id}", json={"ignore_paths": ["ignore/"]}
)
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == ["ignore"] # stored normalized
# Re-upload the SAME archive (same name → in-place replace).
r = _post(upload_client, "docs.tar.gz", _targz_bytes(files))
assert r.status_code == 202, r.text
status = _wait_status(upload_client)
assert status["state"] == "success", status
detail = status["detail"]
# The scan walked only keep.md (unchanged); the ignored file never
# entered the walk, so prune dropped the first scan's row for it.
assert detail["files"] == 1
assert detail["added"] == 0
assert detail["updated"] == 0
assert detail["unchanged"] == 1
assert detail["pruned"] == 1
assert _docs(upload_client) == [("docs", "keep.md")]
# The re-upload left the row's list in place (the existing row is
# untouched by the upsert), and the folder keeps every file.
row_after = _row(db, str(folder))
assert row_after is not None
assert row_after.ignore_paths == ["ignore"]
assert {p.name for p in folder.iterdir()} == {"keep.md", "ignore"}
def test_upload_new_source_without_list_imports_everything(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase 89 regression: an upload of a NEW source name (no row yet,
hence no ignore list) imports everything in the archive — including
nested files — exactly as before phase 89."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\nroot file\n",
"sub/deep.md": "# Deep\nnested file\n",
}
r = _post(upload_client, "fresh.tar.gz", _targz_bytes(files))
assert r.status_code == 202, r.text
status = _wait_status(upload_client)
assert status["state"] == "success", status
detail = status["detail"]
assert detail["files"] == 2
assert detail["added"] == 2
# The fresh row carries the server-default empty list: nothing was
# ignored.
row = _row(db, str(uploads / "fresh"))
assert row is not None
assert (row.ignore_paths or []) == []
assert _docs(upload_client) == [("fresh", "alpha.md"), ("fresh", "sub/deep.md")]
tests/integration/test_import_docs_git.py
+114 -11
View File
@@ -43,14 +43,15 @@ def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Sett
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _git_row(url: str) -> GitSource:
return GitSource(url=url, kind="git")
def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
def _local_row(path: str) -> GitSource:
def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource:
"""A local row as the phase-38 API stores it: the expanded path in
both ``path`` and the NOT-NULL ``url`` location column."""
return GitSource(url=path, kind="local", path=path)
both ``path`` and the NOT-NULL ``url`` location column (plus the
phase-89 ignore list, empty by default)."""
return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or [])
class FakeImportSources:
@@ -66,8 +67,12 @@ class FakeImportSources:
*,
prune: bool = False,
limit: int | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
) -> ImportSummary:
self.calls.append({"sources": list(sources), "prune": prune, "limit": limit})
self.calls.append(
{"sources": list(sources), "prune": prune, "limit": limit,
"ignore_by_root": ignore_by_root}
)
return ImportSummary(files=1, added=1)
@@ -146,9 +151,10 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
sources_dir=str(tmp_path / "bor"),
)
sources = import_docs._resolve_sources(None, settings)
sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
assert calls == [
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
@@ -163,9 +169,10 @@ def test_resolve_sources_cli_source_wins(
settings = _settings(git_sources="https://host/a/repo.git")
manual = tmp_path / "Manual"
sources = import_docs._resolve_sources([manual], settings)
sources, ignore_map = import_docs._resolve_sources([manual], settings)
assert sources == [manual]
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
assert calls == [] # git is never touched when --source is given
@@ -186,9 +193,10 @@ def test_resolve_sources_db_rows_win_over_env(
sources_dir=str(tmp_path / "bor"),
)
sources = import_docs._resolve_sources(None, settings)
sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
@@ -197,8 +205,102 @@ def test_resolve_sources_defaults_when_nothing_configured(
) -> None:
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
sources = import_docs._resolve_sources(None, _settings())
sources, ignore_map = import_docs._resolve_sources(None, _settings())
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
assert ignore_map == {} # phase 89: the legacy fallback has no rows
def test_resolve_sources_rows_branch_builds_ignore_map(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Phase 89: the rows branch returns each row's ignore list keyed by
the resolved root string — the local row's directory, the git row's
checkout dir; a row without a list contributes nothing to the map."""
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: (
[
_git_row("https://db.example/only.git"),
_local_row(str(local_dir), ignore_paths=["ignore/"]),
],
"db",
),
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
# Keyed by the SAME string the importer sees (the root, not the name).
assert ignore_map == {str(local_dir): ["ignore/"]}
def test_resolve_sources_two_rows_sharing_root_string_extend(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Phase 89 collision rule: two rows resolving to the SAME root
string (the sibling/repo-name edge — ``…/shared`` and
``…/shared.git``) get the UNION of their lists (extend, not
replace), in row order."""
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: (
[
_git_row("https://a.example/shared", ignore_paths=["a/"]),
_git_row("https://a.example/shared.git", ignore_paths=["b"]),
],
"db",
),
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
shared = str(tmp_path / "bor" / "shared")
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
assert ignore_map == {shared: ["a/", "b"]} # union, row order
def test_main_rows_branch_passes_ignore_map_to_import(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Phase 89: a ``kind=local`` row carrying ``ignore_paths`` →
``main`` passes the per-root map to ``import_sources`` (keyed by
the directory string, prune flag unchanged)."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
(local_dir / "ignore").mkdir()
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir), ignore_paths=["ignore/"])], "db"),
)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
_stub_bump(monkeypatch)
rc = import_docs.main([])
assert rc == 0
call = fake_import.calls[0]
assert call["sources"] == [local_dir]
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
assert call["prune"] is False # the CLI's no-prune default is unchanged
# --- main() ----------------------------------------------------------------
@@ -306,9 +408,10 @@ def test_resolve_sources_mixed_git_and_local(
sources_dir=str(tmp_path / "bor"),
)
sources = import_docs._resolve_sources(None, settings)
sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
assert ignore_map == {} # phase 89: neither row carries a list
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
tests/integration/test_importer_ignore.py
+202
View File
@@ -0,0 +1,202 @@
"""Integration: phase-89 ignore paths through the real import pipeline.
Phase 89 (TODO.md L3): per-source ignore lists — source-relative path
prefixes that are never walked, hence never embedded and never
summarized (A1), and previously indexed files that newly match a
pattern are pruned on the next ``prune=True`` run (A2). Mirrors the
fixture-tree + mock-LLM pattern of ``test_importer_e2e.py``: a
``tmp_path`` source dir named ``IgnoreFix`` run through the real
``import_sources`` into the compose Postgres, with
:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.models import Chunk, Document
from app.rag.importer import import_sources
from tests.fakes import FakeEmbedder
NAME = "IgnoreFix"
def _write(root: Path, rel: str, content: str) -> None:
path = root / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
def _tree(tmp_path: Path, name: str = NAME) -> Path:
"""The fixture tree: two kept files + one md and one txt under ``ignore/``."""
root = tmp_path / name
_write(root, "keep.md", "# Keep\n\nkept body\n")
_write(root, "ignore/secret.md", "# Secret\n\nSECRET-CONTENT\n")
_write(root, "ignore/notes.txt", "IGNORED-TEXT-CONTENT\n")
_write(root, "top.txt", "TOP-TEXT-CONTENT\n")
return root
def _reset(db: Session) -> None:
# House cleanup pattern (tests/integration/test_importer_e2e.py).
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
def test_ignored_files_never_indexed(db: Session, tmp_path: Path) -> None:
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
summary = asyncio.run(
import_sources(
[root], llm, session=db, ignore_by_root={str(root): ["ignore/"]}
)
)
# Only the two kept files are walked — the ignore/ subtree is
# invisible to the pipeline.
assert summary.files == 2
assert summary.errors == 0
# The kept non-markdown file IS summarized; the ignored .txt is not.
assert summary.summaries == 1
assert summary.summary_errors == 0
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
(NAME, "keep.md"),
(NAME, "top.txt"),
}
# No documents row for an ignored file — hence no chunks rows for it,
# no embedding call, and no summary column value, by construction.
for rel in ("ignore/secret.md", "ignore/notes.txt"):
assert not any(d.path == rel for d in docs)
assert not any("SECRET-CONTENT" in c.content for c in db.scalars(select(Chunk)).all())
for texts in llm.calls: # every embed batch
assert not any(
"SECRET-CONTENT" in t or "IGNORED-TEXT-CONTENT" in t for t in texts
)
# Exactly one summary call (top.txt) — the ignored files never reached
# the lite model.
assert len(llm.chat_calls) == 1
user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user")
assert "TOP-TEXT-CONTENT" in user
assert "SECRET-CONTENT" not in user and "IGNORED-TEXT-CONTENT" not in user
top = next(d for d in docs if d.path == "top.txt")
assert top.summary is not None
_reset(db)
def test_newly_ignored_file_pruned_on_next_prune_run(db: Session, tmp_path: Path) -> None:
_reset(db)
root = tmp_path / NAME
_write(root, "keep.md", "# Keep\n\nkept body\n")
_write(root, "top.txt", "TOP-TEXT-CONTENT\n")
# Exactly ONE file under ignore/ so the A2 prune count pins it.
_write(root, "ignore/secret.md", "# Secret\n\nSECRET-CONTENT\n")
llm = FakeEmbedder()
# First run — no map (omitted entirely): everything is indexed,
# including ignore/secret.md.
s1 = asyncio.run(import_sources([root], llm, session=db))
assert (s1.files, s1.added) == (3, 3)
secret = db.scalar(select(Document).where(Document.path == "ignore/secret.md"))
assert secret is not None
# Second run — the owner adds "ignore" (no trailing slash: A1
# normalization) and prunes. The file newly matches, never enters
# ``seen``, and leaves the index (A2 — the A9 junk-precedent).
s2 = asyncio.run(
import_sources(
[root],
llm,
session=db,
prune=True,
ignore_by_root={str(root): ["ignore"]},
)
)
assert s2.files == 2
assert s2.pruned == 1
assert (
db.scalar(select(Document).where(Document.path == "ignore/secret.md")) is None
)
_reset(db)
def test_progress_total_excludes_ignored(db: Session, tmp_path: Path) -> None:
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
calls: list[tuple[str, str, int, int]] = []
def progress(source: str, rel: str, done: int, total: int) -> None:
calls.append((source, rel, done, total))
summary = asyncio.run(
import_sources(
[root],
llm,
session=db,
progress=progress,
ignore_by_root={str(root): ["ignore/"]},
)
)
assert summary.files == 2
# The phase-64 pre-walk uses the same per-root tuple as the loop:
# ``total`` counts ONLY the non-ignored files, and the hook fired
# exactly once per imported file.
assert [c[2] for c in calls] == [1, 2] # done
assert {c[3] for c in calls} == {2} # total — never counts ignored files
assert {c[1] for c in calls} == {"keep.md", "top.txt"}
_reset(db)
def test_no_map_behavior_is_byte_identical(db: Session, tmp_path: Path) -> None:
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
# ``ignore_by_root=None`` (the default): all four files import exactly
# as pre-phase-89 callers see them.
summary = asyncio.run(import_sources([root], llm, session=db, ignore_by_root=None))
assert summary.files == 4
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
(NAME, "keep.md"),
(NAME, "top.txt"),
(NAME, "ignore/secret.md"),
(NAME, "ignore/notes.txt"),
}
_reset(db)
def test_unlisted_source_unaffected(db: Session, tmp_path: Path) -> None:
_reset(db)
root_a = _tree(tmp_path, name="IgnoreFixA")
root_b = tmp_path / "IgnoreFixB"
_write(root_b, "one.md", "# One\n\none body\n")
_write(root_b, "two.txt", "TWO-TEXT-CONTENT\n")
llm = FakeEmbedder()
# The map keys ONLY the first root — the second imports everything.
summary = asyncio.run(
import_sources(
[root_a, root_b],
llm,
session=db,
ignore_by_root={str(root_a): ["ignore"]},
)
)
# A: keep.md + top.txt (the whole ignore/ subtree is dropped) · B: one.md + two.txt
assert summary.files == 4
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
("IgnoreFixA", "keep.md"),
("IgnoreFixA", "top.txt"),
("IgnoreFixB", "one.md"),
("IgnoreFixB", "two.txt"),
}
assert not any(d.path == "ignore/secret.md" for d in docs)
_reset(db)
tests/integration/test_sync_api.py
+132
View File
@@ -238,6 +238,10 @@ class FakeImportSources:
# Phase 64 (task 02): the progress hook the runner passes (a live
# closure while wired, None if the wiring regresses).
self.progress_hooks: list[object] = []
# Phase 89: the per-root ignore map the runner builds from the
# rows' ``ignore_paths`` (keyed by the root string the importer
# sees; two rows sharing a root string get the union).
self.ignore_maps: list[dict[str, list[str]]] = []
async def __call__(
self,
@@ -248,11 +252,13 @@ class FakeImportSources:
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary:
self.sources.append(list(sources))
self.llms.append(llm)
self.prune_flags.append(prune)
self.progress_hooks.append(progress)
self.ignore_maps.append(ignore_by_root or {})
if self.delay:
await asyncio.sleep(self.delay)
return self.summary
@@ -747,6 +753,7 @@ def test_import_error_is_reported_with_credentials_masked(
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary:
raise EmbeddingError(
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
@@ -897,3 +904,128 @@ def test_probe_runs_before_source_resolution(
assert order == ["probe", "effective_sources"]
assert clone_calls == []
assert "short-circuit" in body["error"] # the spy aborted the run
# --- phase 89: per-row ignore lists ------------------------------------------
def test_local_row_ignore_paths_excluded_from_sync(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 89: a local row carrying ``ignore_paths`` — the button
sync skips every matching file: the ignored file never lands in the
KB (no document row — hence no embedding, no summary), and the
success detail's ``files``/``added`` counts exclude it, while the
kept file imports as usual."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
(local_dir / "ignore").mkdir()
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
encoding="utf-8")
db.add(
GitSource(
url=str(local_dir), kind="local", path=str(local_dir),
ignore_paths=["ignore/"], # any spelling — the importer normalizes
)
)
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# The ignored file is not counted: only keep.md was walked.
assert body["detail"]["files"] == 1
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
# The KB holds exactly the kept file — ignore/secret.md is absent.
docs = [(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]]
assert docs == [("LocalDocs", "keep.md")]
def test_sync_builds_ignore_map_by_root_string_with_union(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 89 wiring (fake import): the runner keys the map by the
SAME root string the importer sees, and two rows sharing that root
string (the sibling/repo-name edge — ``…/shared`` and
``…/shared.git`` clone into the same checkout dir) get the UNION
of their lists, in row order; a row with an empty list contributes
nothing."""
url_a = f"file://{tmp_path / 'shared'}"
url_b = f"{url_a}.git" # same repo name → same checkout dir
# Distinct added_at: the resolver orders by (added_at, id) — a
# same-timestamp pair would tie-break on the random uuid.
db.add(GitSource(url=url_a, kind="git", ignore_paths=["a/"],
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
db.add(GitSource(url=url_b, kind="git", ignore_paths=["b"],
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
shared = str(tmp_path / "bor" / "shared")
# Both rows resolve to the SAME checkout (the collision itself) and
# the map holds their union, keyed by that one root string.
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
assert fake_import.ignore_maps == [{shared: ["a/", "b"]}]
def test_sync_without_ignore_lists_passes_empty_map(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 89 regression: rows without a list → the runner passes an
EMPTY map (the importer's byte-identical pre-phase-89 behavior)."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "notes.md").write_text("# Notes\nplain row\n", encoding="utf-8")
_seed_local(db, local_dir) # no ignore_paths → []
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
assert fake_import.sources == [[local_dir]]
assert fake_import.ignore_maps == [{}] # no row carried a list
tests/unit/test_importer.py
+7 -2
View File
@@ -763,11 +763,16 @@ def test_no_progress_means_no_prewalk(
walk_calls = 0
def counting(
r: Path, extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS
r: Path,
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
) -> list[Path]:
# Phase 89: the walker gained the ``ignore`` keyword — the sentinel
# accepts (and forwards) it to stay a drop-in.
nonlocal walk_calls
walk_calls += 1
return real_walker(r, extensions, excluded)
return real_walker(r, extensions, excluded, ignore)
monkeypatch.setattr(importer, "iter_importable_files", counting)
try:
tests/unit/test_importer_ignore.py
+180
View File
@@ -0,0 +1,180 @@
"""Unit: the phase-89 ignore-path matcher (source-level pins).
Phase 89 (TODO.md L3 — "The ignore is just a prefix ignore."): each
stored source carries a list of source-relative path prefixes; a file
is ignored when its source-relative POSIX path (the ``documents.path``
string — NO leading slash) STARTS WITH a normalized entry. The spec's
own examples are pinned verbatim:
* ``"/my/files/"``, ``"my/files/"`` and ``"my/files"`` all normalize to
``"my/files"`` (whitespace + ALL leading/trailing slashes stripped);
* ``"myfile.txt"`` matches ``"myfile.txt"`` but NOT
``"some/path/myfile.txt"`` — no mid-path matching, no globs;
* raw string prefix, deliberately NO component boundary (A1,
owner-confirmed 2026-09-08): ``"my/files"`` also matches
``"my/files2/x.md"`` and ``"a"`` matches ``"ab.md"``.
Normalization happens exactly once — in ``_ignore_for_root``, the
single choke point — so ``iter_importable_files`` receives
ALREADY-normalized tuples (that contract is pinned too: a raw box line
does NOT match at the walk level). The DB-facing behavior (ignored
files never indexed/embedded/summarized; A2 prune) is integration-
gated by ``tests/integration/test_importer_ignore.py``.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from app.rag.importer import (
_ignore_for_root,
is_ignored,
iter_importable_files,
normalize_ignore_path,
)
#: The walk-level fixture uses an explicit extension set (not the A9
#: config default) so the test pins the matcher, not the config.
EXTS = frozenset({".md", ".txt", ".yaml"})
# --- normalize_ignore_path -------------------------------------------------
@pytest.mark.parametrize(
("raw", "expected"),
[
("/my/files/", "my/files"), # spec: all three spellings are equal
("my/files/", "my/files"),
("my/files", "my/files"),
(" my/files ", "my/files"), # surrounding whitespace trimmed
("//", ""), # nothing left after stripping
("", ""),
(" ", ""),
],
)
def test_normalize_ignore_path(raw: str, expected: str) -> None:
assert normalize_ignore_path(raw) == expected
# --- is_ignored: the spec's own examples, verbatim --------------------------
def test_is_ignored_myfile_txt_root_level_matches() -> None:
# Spec verbatim: "myfile.txt" would match "/myfile.txt" — the leading
# slash never exists in documents.path, so the root-level file matches.
assert is_ignored("myfile.txt", ("myfile.txt",)) is True
@pytest.mark.parametrize(
"rel",
[
"my/files/a.md", # spec: ignores everything under the prefix
"my/files/sub/b.md", # …and arbitrarily deep
"my/files", # a file literally named after the prefix
],
)
def test_is_ignored_prefix_matches(rel: str) -> None:
assert is_ignored(rel, ("my/files",)) is True
@pytest.mark.parametrize(
"rel",
[
"some/path/myfile.txt", # spec verbatim: NO mid-path matching —
# the string simply does not start with the pattern
"xmyfile.txt", # no suffix/partial matching either
],
)
def test_is_ignored_prefix_no_match(rel: str) -> None:
assert is_ignored(rel, ("myfile.txt",)) is False
def test_is_ignored_root_level_dir_is_prefix() -> None:
# A root-level dir IS prefix matching: "files" covers files/**.
assert is_ignored("files/x.md", ("files",)) is True
# …but not a same-named dir deeper down (no mid-path matching).
assert is_ignored("some/files/x.md", ("files",)) is False
# --- is_ignored: the raw-prefix A1 edge (documented, owner-confirmed) -------
def test_is_ignored_raw_prefix_no_component_boundary() -> None:
# A1: raw string startswith — "my/files" ALSO ignores "my/files2/x.md".
assert is_ignored("my/files2/x.md", ("my/files",)) is True
assert is_ignored("ab.md", ("a",)) is True
@pytest.mark.parametrize("rel", ["a.md", "some/a.md", "anything"])
def test_is_ignored_empty_prefixes_never_match(rel: str) -> None:
assert is_ignored(rel, ()) is False
# --- iter_importable_files: the walk-level skip ------------------------------
@pytest.fixture()
def tree(tmp_path: Path) -> Path:
root = tmp_path / "tree"
(root / "ignore" / "deep").mkdir(parents=True)
(root / "keep.md").write_text("keep\n", encoding="utf-8")
(root / "ignore" / "secret.md").write_text("secret\n", encoding="utf-8")
(root / "ignore" / "deep" / "x.yaml").write_text("x: 1\n", encoding="utf-8")
(root / "notes.txt").write_text("notes\n", encoding="utf-8")
(root / "myfiles.md").write_text("myfiles\n", encoding="utf-8")
return root
def _rels(paths: list[Path], root: Path) -> set[str]:
return {p.relative_to(root).as_posix() for p in paths}
ALL_FIVE = {
"keep.md",
"ignore/secret.md",
"ignore/deep/x.yaml",
"notes.txt",
"myfiles.md",
}
def test_iter_default_ignore_keeps_everything(tree: Path) -> None:
# Default ``ignore=()`` — every existing caller byte-identical.
assert _rels(iter_importable_files(tree, EXTS), tree) == ALL_FIVE
def test_iter_ignore_prefix_skips_matching_files(tree: Path) -> None:
got = _rels(iter_importable_files(tree, EXTS, ignore=("ignore",)), tree)
assert got == {"keep.md", "notes.txt", "myfiles.md"}
def test_iter_ignore_receives_already_normalized_tuples(tree: Path) -> None:
# Contract pin: normalization is the caller's job at THIS level — it
# happens in _ignore_for_root, not here. A raw box line does NOT
# match, so all five files still come back.
got = _rels(iter_importable_files(tree, EXTS, ignore=("//ignore/ ",)), tree)
assert got == ALL_FIVE
# --- _ignore_for_root: the single normalization choke point ------------------
def test_ignore_for_root_normalizes_and_drops_empties(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
got = _ignore_for_root(root, {str(root): ["/a/", "b//", "", " "]})
assert got == ("a", "b")
def test_ignore_for_root_missing_root_yields_empty(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _ignore_for_root(root, {"/other/root": ["a"]}) == ()
def test_ignore_for_root_none_map_yields_empty(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _ignore_for_root(root, None) == ()
tests/unit/test_models.py
+40 -1
View File
@@ -6,7 +6,8 @@ on; these tests lock the table/column contract (PLAN §5) without a live DB.
from __future__ import annotations
from pgvector.sqlalchemy import Vector
from sqlalchemy import UniqueConstraint
from sqlalchemy import TextClause, UniqueConstraint
from sqlalchemy.sql.schema import DefaultClause
import app.models # noqa: F401 (import registers all tables on Base.metadata)
from app.db import Base
@@ -62,6 +63,44 @@ def test_doc_drafts_token_is_unique_not_null() -> None:
assert uq, "doc_drafts must be unique on (token) — the URL credential"
def test_git_sources_ignore_paths_column_contract() -> None:
"""Phase 89: every source row carries its ignore list — JSONB, NOT
NULL, server default ``'[]'`` (a pre-phase-89 row reads ``[]``, so
every existing source imports exactly as before)."""
sources = Base.metadata.tables["git_sources"]
assert "ignore_paths" in sources.c, (
"git_sources must have the ignore_paths column (phase 89)"
)
col = sources.c["ignore_paths"]
assert col.nullable is False, "git_sources.ignore_paths must be NOT NULL"
sd = col.server_default
assert isinstance(sd, DefaultClause), "ignore_paths needs a server default"
assert isinstance(sd.arg, TextClause), (
"the server default must be the literal SQL text '[]'"
)
assert sd.arg.text == "'[]'", "ignore_paths server default must be '[]'"
def test_git_source_python_default_empty_list() -> None:
"""A freshly constructed row (no DB) resolves to an empty ignore
list via the Python-side default (``default=list``) — the ORM
INSERT-time default, so an ORM insert that omits the column inserts
``[]`` rather than NULL (the server default ``'[]'`` independently
covers non-ORM inserts)."""
from app.models import GitSource
row = GitSource(url="https://example.com/r.git", kind="git")
col = row.__table__.c["ignore_paths"]
assert col.default is not None, (
"ignore_paths needs a Python-side (INSERT-time) default"
)
# Invoked with the (unused) execution context at INSERT time — a
# freshly constructed row that omits the kwarg stores [] rather
# than NULL (the real-DB behaviour is pinned against the dev DB
# by the git-sources API integration tests).
assert col.default.arg(None) == [], "the default must resolve to []"
def test_doc_drafts_column_contract() -> None:
"""Phase 59: the editable triple (title/path/body) + status +
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
tests/unit/test_source_ignore_paths.py
+546
View File
@@ -0,0 +1,546 @@
"""Unit: the per-source ignore-paths editor on the Sources page
(phase 89, task 05).
``TODO.md`` L3: "They should be able to type these files and folders
into a box on the sources page." Phase 89 gives every STORED source
row (git or local) a one-path-per-line ignore box on
``/git-sources.html``: a page-local alertdialog (the EXACT
``#remove-confirm-dialog`` pattern, phase 69) with a visible label, a
mono textarea prefilled from the row, the §7.4 never-stale save
lifecycle ("Saving…" while the ``PATCH /api/git-sources/{id}`` is
out — the A5 replace round-trips through ``GET``), and full a11y
(focus on Cancel, Escape / backdrop close as CANCEL, focus return,
``role="alert"`` error line that keeps the textarea content on 422).
Rows with a list show the ``N ignored`` count tag; env-fallback rows
(``id`` null) get NO box (A3 — nothing is stored to edit).
The browser behavior itself is E2E-gated by the phase-89 story suite
(``tests/e2e/test_source_ignore_paths.py``, task 06); like the other
frontend-adjacent unit files (the
``test_remove_confirm_modal.py`` house pattern), this module pins the
source-level contract a silent regression would break:
* the static ``#ignore-editor-dialog`` markup — ``role=
"alertdialog"`` + ``aria-modal`` + ``aria-labelledby``, hidden by
default, inside the manager, the visible ``<label for=…>`` (never
aria-label-only), the ``role="alert"`` error line, real
``type="button"`` buttons, the mono textarea (``rows=6`` /
``spellcheck="false"``);
* the JS lifecycle — ``openIgnoreEditor`` (textContent-only source
population with makeRow's ``value`` expression, the textarea
prefilled from ``(s.ignore_paths || [])``, error cleared, focus on
Cancel), cancel = Escape / Cancel button / backdrop (no request;
focus returns to the trigger; a no-op while a PATCH is in flight),
``saveIgnorePaths`` (§7.4 in-flight state: both buttons disable +
"Saving…", blank lines dropped client-side, success → close /
reload / announce — the update confirmation LAST, non-2xx → the
in-dialog alert line with the textarea content KEPT, network →
the fixed reachable? line, re-enable in the finally);
* the row wiring — the "Ignore paths" button exists ONLY in the
``s.id`` branch of ``makeRow`` (left of Remove; the aria-label is
the only place the value appears — never innerHTML), the ``N
ignored`` count tag on rows with a list (text, never color alone);
* styles.css — the dialog / button / count-tag rules on the house
dark-tech palette (phase-08 tokens only, no CDN, no blur),
≥44px targets, the ``[hidden]`` override, the mono full-width
textarea;
* the no-collision guard — every ``id="ignore-editor-…"`` appears
EXACTLY ONCE in the shell (the phase-46/76 contract).
"""
from __future__ import annotations
import re
from pathlib import Path
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
# Phase 76 (task 02): git-sources.html is folded into the ONE-document
# shell — the dialog markup lives in the Sources view section of
# index.html (next to #remove-confirm-dialog).
SHELL_HTML = FRONTEND / "index.html"
JS = FRONTEND / "assets" / "git-sources.js"
CSS = FRONTEND / "assets" / "styles.css"
#: The dialog's static ids (the no-collision guard + the markup pins
#: key on exactly these).
DIALOG_IDS = (
"ignore-editor-dialog",
"ignore-editor-title",
"ignore-editor-source",
"ignore-editor-copy",
"ignore-editor-textarea",
"ignore-editor-error",
"ignore-editor-cancel",
"ignore-editor-save",
)
#: The §7.4 in-flight label + the idle label (pinned verbatim).
SAVING_LABEL = "Saving…"
IDLE_LABEL = "Save"
COUNT_CLASS = "git-source-ignore-count"
ROW_BTN_CLASS = "git-source-ignore"
ARIA_LABEL_TEMPLATE = "`Edit ignored paths for ${kindLabel} source: ${value}`"
#: The success announce — the update confirmation is the LAST
#: announcement (the reload's "N sources listed." must not overwrite
#: it, the same order as the remove flow).
ANNOUNCE_OK = "Ignored paths updated for ${value}"
def _text(path: Path) -> str:
return path.read_text(encoding="utf-8")
def _js() -> str:
return _text(JS)
def _css() -> str:
return _text(CSS)
def _fn(js: str, name: str) -> str:
"""The source of a (possibly async) top-level function via
balanced-brace counting (the test_remove_confirm_modal.py helper)."""
for prefix in ("async function ", "function "):
start = js.find(f"{prefix}{name}(")
if start != -1:
depth = 0
for i in range(js.find("{", start), len(js)):
if js[i] == "{":
depth += 1
elif js[i] == "}":
depth -= 1
if depth == 0:
return js[start : i + 1]
raise AssertionError(f"unbalanced braces in {name}()")
raise AssertionError(f"{name}() must exist in git-sources.js")
def _element_block(html: str, id_attr: str, tag: str = "div") -> str:
"""The <tag … id=…> element's full markup (a balanced-tag walk —
the dialog nests the backdrop / panel / actions divs)."""
marker = f'id="{id_attr}"'
i = html.find(marker)
assert i != -1, f"missing id={id_attr} in the shell's Sources view"
opens = [m.start() for m in re.finditer(rf"<{tag}\b", html[:i])]
assert opens, f"no <{tag}> owns id={id_attr}"
open_i = opens[-1]
depth = 0
for m in re.finditer(rf"<{tag}\b[^>]*>|</{tag}>", html[open_i :]):
if m.group(0).startswith(f"</{tag}>"):
depth -= 1
else:
depth += 1
if depth == 0:
return html[open_i : open_i + m.end()]
raise AssertionError(f"unbalanced <{tag}> for id={id_attr}")
def _css_rule(css: str, selector: str) -> str:
"""The declarations of a simple one-line-opening rule (comments
stripped first — a house comment may legally carry braces)."""
clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
start = clean.find(f"{selector} {{")
assert start != -1, f"missing rule {selector} in styles.css"
depth = 0
for i in range(clean.find("{", start), len(clean)):
if clean[i] == "{":
depth += 1
elif clean[i] == "}":
depth -= 1
if depth == 0:
return clean[start : i + 1]
raise AssertionError(f"unbalanced braces in {selector}")
# ---------- the static dialog markup ----------
def test_dialog_markup_is_the_locked_alertdialog() -> None:
"""#ignore-editor-dialog: role="alertdialog" + aria-modal +
aria-labelledby, hidden by default, INSIDE the manager
(#git-sources-content, directly after #remove-confirm-dialog —
the static-markup convention, stable E2E selectors); every
static id present; the error line is role="alert" + hidden;
both buttons are real type="button"; the textarea is mono-
capable (rows=6, spellcheck=false, the A1 prefix placeholder)."""
html = _text(SHELL_HTML)
frag = _element_block(html, "ignore-editor-dialog")
open_tag = frag[: frag.find(">") + 1]
assert 'role="alertdialog"' in open_tag
assert 'aria-modal="true"' in open_tag
assert 'aria-labelledby="ignore-editor-title"' in open_tag
assert 'aria-describedby="ignore-editor-copy"' in open_tag
assert "hidden" in open_tag, "the dialog ships hidden"
# Placed in the manager, right after the remove-confirm dialog
# (the same placement logic).
assert html.find('id="git-sources-content"') < html.find('id="ignore-editor-dialog"')
assert html.find('id="remove-confirm-dialog"') < html.find('id="ignore-editor-dialog"')
for child in DIALOG_IDS:
assert f'id="{child}"' in frag, f"missing #{child} in the dialog"
# The visible label (WCAG — never aria-label-only): a real
# <label for="ignore-editor-textarea"> with the house label class.
label = re.search(
r"<label[^>]*class=\"ignore-editor-label\"[^>]*for=\"ignore-editor-textarea\"[^>]*>(.*?)</label>",
frag,
re.S,
)
assert label and label.group(1).strip(), "a visible block label for the textarea"
# The error line: role="alert", hidden by default.
err = re.search(r"<p[^>]*id=\"ignore-editor-error\"[^>]*>", frag)
assert err and 'role="alert"' in err.group(0) and "hidden" in err.group(0)
# The mono box: rows + spellcheck off + the A1 placeholder.
ta = re.search(r"<textarea[^>]*id=\"ignore-editor-textarea\"[^>]*>", frag, re.S)
assert ta, "#ignore-editor-textarea is a real <textarea>"
for attr in ('rows="6"', 'spellcheck="false"', 'placeholder="my/files/"'):
assert attr in ta.group(0), f"the textarea carries {attr}"
for btn in ("ignore-editor-cancel", "ignore-editor-save"):
m = re.search(rf"<button[^>]*id=\"{btn}\"[^>]*>", frag)
assert m and 'type="button"' in m.group(0), f"#{btn} is a real type=button"
cancel = re.search(r'<button[^>]*id="ignore-editor-cancel"[^>]*>(.*?)</button>', frag, re.S)
save = re.search(r'<button[^>]*id="ignore-editor-save"[^>]*>(.*?)</button>', frag, re.S)
assert cancel and cancel.group(1).strip() == "Cancel"
assert save and save.group(1).strip() == IDLE_LABEL
# The helper copy states the A1 prefix rule in plain words.
copy = re.search(r"<p[^>]*id=\"ignore-editor-copy\"[^>]*>(.*?)</p>", frag, re.S)
assert copy, "the helper copy paragraph"
copy_norm = re.sub(r"\s+", " ", copy.group(1))
assert "No wildcards" in copy_norm and "middle of a path" in copy_norm
def test_dialog_ids_appear_exactly_once() -> None:
"""The no-collision guard (the phase-46/76 contract): every
``id="ignore-editor-…"`` in the NEW markup appears EXACTLY ONCE
in the whole shell — no duplicated id (the dialog is the only
owner of each id)."""
html = _text(SHELL_HTML)
for id_ in DIALOG_IDS:
n = html.count(f'id="{id_}"')
assert n == 1, f'id={id_} appears {n} times in index.html (must be exactly 1)'
# ---------- the row wiring in makeRow ----------
def test_row_button_exists_only_in_the_stored_row_branch() -> None:
"""makeRow: the "Ignore paths" button is created ONLY in the
``s.id`` branch (A3 — env-fallback rows get the "from .env" tag,
no button) and sits BEFORE the Remove button (left of Remove).
The button is class git-source-ignore, type=button, textContent
label (never innerHTML — the value appears ONLY in the aria-
label, built via setAttribute); its click opens the editor."""
make = _fn(_js(), "makeRow")
branch_i = make.find("if (s.id) {")
else_i = make.find("} else {", branch_i)
bind_i = make.find("openIgnoreEditor(s, ignoreBtn)")
assert -1 < branch_i < bind_i < else_i, (
"the ignore button is created in the s.id branch (the phase-88 pin idiom)"
)
remove_i = make.find('btn.className = "git-source-remove"', bind_i)
assert remove_i > bind_i, "the ignore button is LEFT of Remove"
assert f'ignoreBtn.className = "{ROW_BTN_CLASS}"' in make
assert "ignoreBtn.type = \"button\"" in make
assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)"
assert "ignoreBtn.textContent = \"Ignore paths\"" in make, (
"a static text label — never icon-only"
)
assert "ignoreBtn.innerHTML" not in _js(), "XSS contract: no innerHTML on the button"
# The env-fallback branch (else) has no ignore button.
else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)]
assert ROW_BTN_CLASS not in else_slice and "openIgnoreEditor" not in else_slice, (
"env-fallback rows get no ignore button (A3)"
)
def test_count_tag_shows_n_ignored_text_next_to_the_location() -> None:
"""makeRow: a stored row with a non-empty list gets the
``N ignored`` count tag in the LOCATION cell (after the
<code>) — text via textContent (never color alone, WCAG 1.4.1);
the list reads ``(s.ignore_paths || [])`` — the same expression
openIgnoreEditor prefills from (the GET round-trip marker)."""
make = _fn(_js(), "makeRow")
assert "s.id && (s.ignore_paths || []).length > 0" in make, (
"the count tag only for stored rows with a list"
)
assert f'count.className = "{COUNT_CLASS}"' in make
assert "count.textContent = `${s.ignore_paths.length} ignored`" in make, (
"N ignored — text, never color alone"
)
# It lands in the location cell (urlTd), not the actions cell.
append_i = make.find("urlTd.append(count)")
code_i = make.find("urlTd.append(badge, code)")
assert -1 < code_i < append_i, "the tag is appended to the location cell"
# ---------- the JS lifecycle ----------
def test_open_prefills_and_focuses_cancel() -> None:
"""openIgnoreEditor(s, triggerBtn): the source value is
textContent ONLY (never innerHTML — the credential-masking
discipline, phase 32) with makeRow's exact ``value`` expression;
the textarea prefills ``(s.ignore_paths || []).join("\\n")``
(one path per line — the round-trip marker); the error line
clears; the dialog unhides; the trigger is recorded; the keydown
handler attaches; and focus lands on Cancel — the safe default
(AFTER the unhide)."""
body = _fn(_js(), "openIgnoreEditor")
assert "s.kind === \"local\"" in body, "the kind-typed value branch"
assert (
"ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url" in body
), "the same `value` expression makeRow uses, via textContent"
assert "ignoreSourceEl.innerHTML" not in _js(), "XSS contract: textContent only"
assert 'ignoreTextarea.value = (s.ignore_paths || []).join("\\n")' in body, (
"the textarea prefills the stored list, one path per line"
)
assert "ignoreErrorEl.hidden = true" in body, "a new attempt starts clean"
assert "ignoreTriggerBtn = triggerBtn" in body, "the trigger is recorded"
trigger_i = body.find("ignoreTriggerBtn = triggerBtn")
unhide_i = body.find("ignoreDialog.hidden = false")
attach_i = body.find('document.addEventListener("keydown", onIgnoreDialogKeydown)')
focus_i = body.find("ignoreCancelBtn.focus()")
assert -1 < trigger_i < unhide_i < attach_i < focus_i, (
"record the trigger → unhide → attach keydown → focus Cancel"
)
def test_cancel_paths_close_without_a_request() -> None:
"""Cancel (Cancel button / Escape / backdrop) closes as cancel:
the dialog hides, the textarea + error line reset, the buttons
reset ("Save"), the keydown handler detaches, and focus RETURNS
to the recorded trigger — and the cancel path never fetches. A
cancel while a PATCH is in flight is a no-op (no half-cancel of
an in-progress save)."""
js = _js()
cancel = _fn(js, "cancelIgnoreEditor")
guard_i = cancel.find("if (ignoreInFlight) return")
close_i = cancel.find("closeIgnoreEditor()")
assert -1 < guard_i < close_i, "the in-flight guard precedes the close"
assert "fetch" not in cancel, "cancel never sends a request"
close = _fn(js, "closeIgnoreEditor")
hide_i = close.find("ignoreDialog.hidden = true")
reset_ta_i = close.find('ignoreTextarea.value = ""')
clear_i = close.find("ignoreErrorEl.hidden = true")
reset_btn_i = close.find('"Save"')
detach_i = close.find('document.removeEventListener("keydown", onIgnoreDialogKeydown)')
save_i = close.find("const trigger = ignoreTriggerBtn")
null_i = close.find("ignoreTriggerBtn = null")
focus_i = close.find("trigger.focus()")
assert -1 < hide_i < reset_ta_i < clear_i < reset_btn_i, (
"hide → reset textarea → clear error → reset buttons"
)
assert reset_btn_i < detach_i < save_i < null_i < focus_i, (
"→ detach → save trigger → focus return"
)
assert "ignoreCancelBtn.disabled = false" in close
assert "ignoreSaveBtn.disabled = false" in close
def test_escape_and_backdrop_and_cancel_button_all_cancel() -> None:
"""While open (handler attached on document in openIgnoreEditor,
detached in closeIgnoreEditor): Escape → preventDefault +
cancelIgnoreEditor; the dim backdrop AND the Cancel button wire
to cancelIgnoreEditor (only Save wires to saveIgnorePaths)."""
js = _js()
keydown = _fn(js, "onIgnoreDialogKeydown")
esc_i = keydown.find('e.key === "Escape"')
prevent_i = keydown.find("e.preventDefault()", esc_i)
cancel_i = keydown.find("cancelIgnoreEditor()", esc_i)
assert -1 < esc_i < prevent_i < cancel_i, "Escape: prevent + cancel"
assert 'ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor)' in js
assert 'ignoreBackdrop.addEventListener("click", cancelIgnoreEditor)' in js
assert 'ignoreSaveBtn.addEventListener("click", saveIgnorePaths)' in js
def test_save_runs_the_inflight_never_stale_lifecycle() -> None:
"""saveIgnorePaths: the box's lines are split on newlines,
trimmed, and empty lines DROPPED (a blank line is a separator,
not an entry — the server still rejects empties defensively,
A4). The §7.4 in-flight state precedes the PATCH: both buttons
disable + the save relabels "Saving…" — one
``PATCH /api/git-sources/{id}`` with the lines as the whole
body list (A5 replace). 200 → close (focus return) →
loadSources (the count tag lands) → announce (the update
confirmation is the LAST announcement). Non-2xx: the in-dialog
role=alert line (apiDetail, 422 shape-aware), the dialog STAYS
open and the textarea content is KEPT. Network failure: the
fixed reachable? line. The finally re-enables BOTH buttons +
relabels "Save" — never stale on any outcome."""
body = _fn(_js(), "saveIgnorePaths")
guard_i = body.find("if (!ignoreTarget || ignoreInFlight) return")
parse_i = body.find('.split("\\n")')
trim_i = body.find(".map((l) => l.trim())", parse_i)
drop_i = body.find(".filter(Boolean)", trim_i)
inflight_i = body.find("ignoreInFlight = true", drop_i)
dis_c = body.find("ignoreCancelBtn.disabled = true", inflight_i)
dis_s = body.find("ignoreSaveBtn.disabled = true", inflight_i)
label_i = body.find(f'"{SAVING_LABEL}"', dis_s)
fetch_i = body.find("`/api/git-sources/${", label_i)
method_i = body.find('method: "PATCH"', fetch_i)
body_i = body.find("JSON.stringify({ ignore_paths: lines })", method_i)
assert -1 < guard_i < parse_i < trim_i < drop_i < inflight_i, (
"guard → split + trim + drop blank lines → in-flight"
)
assert -1 < dis_c < dis_s < label_i < fetch_i < method_i < body_i, (
"disable both + 'Saving…' → the PATCH with the lines"
)
# Success: close → reload → announce (the exact order) — the
# update confirmation is the LAST announcement: the reload's
# "N sources listed." must not overwrite it.
ok_i = body.find("if (r.ok)")
close_i = body.find("closeIgnoreEditor()", ok_i)
reload_i = body.find("await loadSources()", close_i)
announce_i = body.find(f"announce(`{ANNOUNCE_OK}`)", reload_i)
assert -1 < ok_i < close_i < reload_i < announce_i, (
"200: close → loadSources → announce (LAST)"
)
assert body.count("closeIgnoreEditor()") == 1, (
"only the success path closes — failures stay open for one retry"
)
# Non-2xx: the in-dialog alert line (apiDetail, 422 shape-aware);
# the dialog stays open and the textarea content is KEPT.
catch_i = body.find("} catch {")
nonok_slice = body[ok_i:catch_i]
assert "await apiDetail(r," in nonok_slice, (
"the server detail is apiDetail-extracted (422 shape-aware)"
)
assert "ignoreErrorEl.hidden = false" in nonok_slice, "the error line shows"
assert "ignoreTextarea.value" not in nonok_slice.split("if (r.ok)")[1], (
"the textarea content is KEPT on failure (the instruction survives)"
)
# Network: the fixed reachable? line.
net_i = body.find("Could not save the ignored paths — is the app reachable?", catch_i)
assert -1 < net_i < body.find("finally"), "the network copy lands in the catch"
# Never stale: the finally re-enables BOTH buttons + relabels.
fin = body[body.find("finally"):]
assert "ignoreInFlight = false" in fin
assert "ignoreCancelBtn.disabled = false" in fin
assert "ignoreSaveBtn.disabled = false" in fin
assert f'ignoreSaveBtn.textContent = "{IDLE_LABEL}"' in fin
def test_module_docstring_carries_the_phase_89_contract() -> None:
"""The git-sources.js module docstring gained the phase-89
bullet: the editor is per-STORED-row (A3 env rows excluded),
one path per line (the A1 prefix rule), the §7.4 "Saving…"
lifecycle, and the A5 replace round-trip."""
doc = _js().split("*/", 2)[0] # the module docstring (first block)
for frag in (
"Phase 89 (task 05)",
"ignore-paths editor",
"#ignore-editor-dialog",
"one path per line",
f'"{SAVING_LABEL}"',
"LAST announcement",
"N ignored",
):
assert frag in doc, f"the module docstring lost: {frag!r}"
# ---------- styles.css ----------
def test_dialog_css_rules_present_and_house_tokens_only() -> None:
"""styles.css carries the ignore-editor class family on the house
dark-tech palette (phase-08 tokens): the overlay + backdrop +
panel (the EXACT .remove-confirm treatment), the title / source /
copy / visible label / mono textarea / error / actions / two-
button chrome; the [hidden] override; no blur (the phase-08
no-blur perf anchor); no CDN."""
css = _css()
for selector in (
".ignore-editor",
".ignore-editor-backdrop",
".ignore-editor-panel",
".ignore-editor-title",
".ignore-editor-source",
".ignore-editor-copy",
".ignore-editor-label",
".ignore-editor-textarea",
".ignore-editor-error",
".ignore-editor-actions",
".ignore-editor-btn",
".ignore-editor-cancel",
".ignore-editor-save",
):
assert f"{selector} " in css or f"{selector}." in css or f"{selector}[" in css, (
f"styles.css must style {selector}"
)
hidden = css.find(".ignore-editor[hidden]")
assert hidden != -1 and "display: none" in css[hidden : hidden + 60], (
"the hidden attr must beat the display rule"
)
# No blur (the phase-08 no-blur perf anchor) — the backdrop rule
# itself must not carry backdrop-filter (comments stripped).
backdrop = re.sub(r"/\*.*?\*/", "", _css_rule(css, ".ignore-editor-backdrop"), flags=re.S)
assert "backdrop-filter" not in backdrop, "no blur (phase-08 perf anchor)"
assert "url(http" not in css and "@import url(" not in css, (
"no CDN (AGENTS.md rule 6)"
)
def test_textarea_rule_is_mono_and_full_width() -> None:
"""The box itself: full panel width (width: 100%) + the mono
stack (var(--mono)) + a comfortable min-height — the house
comment cites phase 89 on the block."""
rule = _css_rule(_css(), ".ignore-editor-textarea")
assert "width: 100%" in rule, "full panel width"
assert "var(--mono)" in rule, "the mono stack"
assert "min-height" in rule, "a comfortable minimum height"
# The section comment cites phase 89 (house comment style).
css = _css()
header = css[css.rfind("/*", 0, css.find(".ignore-editor {")) : css.find(".ignore-editor {")]
assert "phase 89" in header, "the house comment cites phase 89"
def test_dialog_button_and_target_contrast_pairs() -> None:
"""The WCAG 2.1 AA basics in CSS: both dialog buttons >=44px;
Cancel is the ghost ink-soft family (5.1:1 on --surface) with the
brand-soft hover (12.4:1); Save is the solid brand family (--bg
text on --brand 5.2:1, the .new-chat-btn convention) with the
lightened hover; the error line is the err pair; the panel caps
at the 46rem chat-column width or the viewport; the visible
label is ink-soft (5.1:1) — never a label-less textarea."""
css = _css()
btn = _css_rule(css, ".ignore-editor-btn")
assert "min-height: 44px" in btn and "min-width: 44px" in btn
cancel = _css_rule(css, ".ignore-editor-cancel")
assert "var(--ink-soft)" in cancel and "transparent" in cancel
cancel_hover = _css_rule(css, ".ignore-editor-cancel:hover:not(:disabled)")
assert "var(--brand-soft)" in cancel_hover and "var(--brand-ink)" in cancel_hover
save = _css_rule(css, ".ignore-editor-save")
assert "var(--brand)" in save and "var(--bg)" in save, (
"Save: the solid brand family (--bg text on --brand, 5.2:1)"
)
save_hover = _css_rule(css, ".ignore-editor-save:hover:not(:disabled)")
assert "background" in save_hover, "the hover lightens the fill"
err = _css_rule(css, ".ignore-editor-error")
assert "var(--err-ink)" in err and "var(--err-bg)" in err
panel = _css_rule(css, ".ignore-editor-panel")
assert "min(46rem" in panel, "the 46rem chat-column cap (or the viewport)"
label = _css_rule(css, ".ignore-editor-label")
assert "display: block" in label and "var(--ink-soft)" in label, (
"a visible block label (WCAG — never aria-label-only)"
)
def test_row_button_and_count_tag_css_rules() -> None:
"""The row chrome: .git-source-ignore reuses the
.git-source-remove idiom (>=44px target, same size/spacing) in a
NEUTRAL secondary fill with the brand-soft hover (distinct from
the destructive Remove's err hover); .git-source-ignore-count is
a small inline tag — text + a distinct background (never color
alone), AA on both theme surfaces (--ink on --bg 16.7:1)."""
css = _css()
row = _css_rule(css, ".git-source-ignore")
assert "min-height: 44px" in row and "min-width: 44px" in row, "the >=44px rule"
assert "var(--ink-soft)" in row, "the neutral secondary resting fill"
row_hover = _css_rule(css, ".git-source-ignore:hover:not(:disabled)")
assert "var(--brand-soft)" in row_hover and "var(--brand-ink)" in row_hover, (
"the hover takes the brand pair (Remove hovers to the err pair)"
)
assert "var(--err-" not in row, "not the destructive err family"
count = _css_rule(css, ".git-source-ignore-count")
assert "var(--ink)" in count and "var(--bg)" in count, (
"text + a distinct background (never color alone)"
)
tests/unit/test_sync_button.py
+1
View File
@@ -727,6 +727,7 @@ class _GatedImport:
limit: int | None = None,
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
) -> ImportSummary:
self.prune_flags.append(prune)
if progress is not None: