Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
ducoterra
2026-09-09 01:45:42 -04:00
parent 0495e4e7e4
commit 8c706259e9
49 changed files with 3717 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/89_source_ignore_paths/00_phase.md → .agents/phases/complete/89_source_ignore_paths/00_phase.md
View File
.agents/phases/todo/89_source_ignore_paths/01_ignore_paths_column.md → .agents/phases/complete/89_source_ignore_paths/01_ignore_paths_column.md
View File
.agents/phases/todo/89_source_ignore_paths/02_ignore_prefix_matching.md → .agents/phases/complete/89_source_ignore_paths/02_ignore_prefix_matching.md
View File
.agents/phases/todo/89_source_ignore_paths/03_ignore_paths_api.md → .agents/phases/complete/89_source_ignore_paths/03_ignore_paths_api.md
View File
.agents/phases/todo/89_source_ignore_paths/04_ignore_paths_in_import_pipelines.md → .agents/phases/complete/89_source_ignore_paths/04_ignore_paths_in_import_pipelines.md
View File
.agents/phases/todo/89_source_ignore_paths/05_ignore_paths_sources_ui.md → .agents/phases/complete/89_source_ignore_paths/05_ignore_paths_sources_ui.md
View File
.agents/phases/todo/89_source_ignore_paths/06_e2e_ignore_paths.md → .agents/phases/complete/89_source_ignore_paths/06_e2e_ignore_paths.md
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__00_phase.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__00_phase.a1.md
+23
View File
@@ -0,0 +1,23 @@
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.
## Phase 89 — final verification pass: ALL GREEN
**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box
**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)
**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.
**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).
**Next pending phase:** none — `todo/` holds only this phase.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__00_phase.a1.validate
+87
View File
@@ -0,0 +1,87 @@
........................................................................ [ 3%]
........................................................................ [ 7%]
........................................................................ [ 11%]
........................................................................ [ 15%]
........................................................................ [ 19%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 31%]
........................................................................ [ 35%]
........................................................................ [ 39%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 55%]
........................................................................ [ 59%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 95%]
........................................................................ [ 99%]
........ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__01_ignore_paths_column.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__01_ignore_paths_column.a1.md
+19
View File
@@ -0,0 +1,19 @@
All green. Task 01 complete.
## Report
**Implemented (task 01 — storage half of per-source ignore paths):**
- `app/models.py`: `GitSource.ignore_paths` — JSONB NOT NULL, `default=list` (INSERT-time) + server default `'[]'` (verbatim task spec); `text` added to the sqlalchemy imports; module-docstring `git_sources` bullet updated (phase 89).
- `alembic/versions/0013_git_source_ignore_paths.py` (new): `add_column` JSONB/`'[]'`/NOT NULL, `drop_column` downgrade — 0007 house-header style, revises 0012.
- `tests/unit/test_models.py` (extended, house pattern): column exists / NOT NULL / server default `arg.text == "'[]'"` (narrowed via `DefaultClause` + `TextClause` for pyright), plus the Python-side default contract test.
**Gates (exact commands):**
- `uv run alembic upgrade head` → `0012 -> 0013` clean; `alembic downgrade -1 && alembic upgrade head` round-trips; head = `0013`.
- `uv run pytest tests/integration/test_git_sources_api.py` → 23 passed, **unchanged** (dev DB `git_sources` empty; `'[]'` server default covers existing rows; a live ORM-insert probe confirmed `[]` is stored — probe row deleted).
- `uv run pytest --cov=app --cov-report=term-missing` → **1748 passed**, TOTAL **99%** (>90% ✓, `app/models.py` 100%).
- `uv run ruff check . && uv run pyright` → clean (0 errors).
- `git diff --stat`: only `app/models.py`, the new migration, and the unit test.
**Notable deviation:** the task's literal pin `GitSource(...).ignore_paths == []` cannot hold — SQLAlchemy 2.0 `default=list` is an INSERT-time `CallableColumnDefault` (unflushed attr is `None`; the builtin is wrapped in a ctx lambda, so `arg is list` also fails). The test instead pins the INSERT-time default resolving to `[]`; real-DB behavior is covered by the API integration suite. Model code matches the spec verbatim.
**Next pending task:** `.agents/phases/todo/89_source_ignore_paths/02_ignore_prefix_matching.md`
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__01_ignore_paths_column.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 37%]
........................................................................ [ 41%]
........................................................................ [ 45%]
........................................................................ [ 49%]
........................................................................ [ 53%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 74%]
........................................................................ [ 78%]
........................................................................ [ 82%]
........................................................................ [ 86%]
........................................................................ [ 90%]
........................................................................ [ 94%]
........................................................................ [ 98%]
.................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 212 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 98 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 232 0 100%
--------------------------------------------------
TOTAL 3238 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__02_ignore_prefix_matching.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__02_ignore_prefix_matching.a1.md
+10
View File
@@ -0,0 +1,10 @@
All green. Task complete.
**Report**
- Implemented in `app/rag/importer.py`: `normalize_ignore_path` / `is_ignored` / `_ignore_for_root` helpers (spec A1 raw-prefix semantics, single normalization choke point); `iter_importable_files(..., ignore=())` walk-level skip; `import_sources(..., ignore_by_root=None)` — pre-walk and processing loop use the same per-root tuple, `seen`/`_prune` untouched so newly-ignored files are pruned on next `prune=True` run (A2); docstrings per house style.
- New `tests/unit/test_importer_ignore.py` (24 tests): normalization table, spec examples verbatim (`/my/files/` ≡ `my/files/` ≡ `my/files`; `myfile.txt` root-level only, no mid-path), raw-prefix A1 edge (`my/files2`, `ab.md`), empty-prefix no-match, `iter_importable_files` on a 5-file tmp tree incl. the "walk receives already-normalized tuples" contract pin, `_ignore_for_root` normalize/drop/missing/None.
- New `tests/integration/test_importer_ignore.py` (5 tests): ignored files never indexed (no doc/chunk rows, no embed or chat call, kept `.txt` still summarized); A2 prune (`pruned == 1`, row gone, pattern given without trailing slash); progress `total` == 2 excludes ignored; no-map byte-identical (`files == 4`); unlisted second root imports everything.
- Test / lint / coverage: `uv run pytest tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -v` → 29 passed; `uv run pytest -q` → exit 0 (1777 collected, no failures); `uv run pytest --cov=app --cov-report=term-missing -q` → TOTAL 99% (>90% gate); `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
- Notable: (1) fixed the phase-64 counting-sentinel in `tests/unit/test_importer.py` to accept/forward the new `ignore` keyword (its monkeypatch stub broke on the new call site — the one extra file beyond the task's stated diff scope); (2) prune test uses a 3-file tree (one file under `ignore/`) so `pruned == 1` pins exactly, as the task specifies.
- Next pending task: `.agents/phases/todo/89_source_ignore_paths/03_ignore_paths_api.md`.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__02_ignore_prefix_matching.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 48%]
........................................................................ [ 52%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 76%]
........................................................................ [ 81%]
........................................................................ [ 85%]
........................................................................ [ 89%]
........................................................................ [ 93%]
........................................................................ [ 97%]
................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 212 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 98 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 232 0 100%
--------------------------------------------------
TOTAL 3248 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__03_ignore_paths_api.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__03_ignore_paths_api.a1.md
+17
View File
@@ -0,0 +1,17 @@
Everything is green. Final report:
**Task 03 (phase 89) complete — ignore-list API**
- `app/schemas.py`: `GitSourceIn.ignore_paths` (optional, raw lines), `GitSourceOut`/`GitSourceRow.ignore_paths` (non-null; env rows `[]`), new `GitSourceIgnoreIn` (required list, A5)
- `app/api/git_sources.py`: shared `_validate_ignore_paths` (normalize + A4 fixed-detail 422s, empty-check first; constants 200/500), GET reports per-row list, POST stores normalized for both kinds, new admin-only `PATCH /api/git-sources/{id}` (replace semantics, 404/422/403) placed after POST before `/upload`; module docstring updated
- `tests/integration/test_git_sources_api.py`: 10 new tests (normalize round-trip both kinds, absent→`[]`, POST+PATCH 422 exact details, missing-field 422, 404, invalid id, replace+clear, A4 boundaries, row-unchanged on reject); existing shape pins updated additively, anonymous test covers PATCH
**Gates:**
- `uv run pytest` → 1787 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 99% (>90% ✓; `git_sources.py` + `schemas.py` 100%)
- `uv run pytest tests/integration/test_git_sources_api.py --no-cov` → 33 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
**Notes:** no defects found in prior tasks; diff limited to the 3 task files. `.agents/PLAN.md` is absent in this repo — `00_phase.md` design (locked A1–A5) was the design source; no deviations.
**Next pending task:** `.agents/phases/todo/89_source_ignore_paths/04_ignore_paths_in_import_pipelines.md`
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__03_ignore_paths_api.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 48%]
........................................................................ [ 52%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 76%]
........................................................................ [ 80%]
........................................................................ [ 84%]
........................................................................ [ 88%]
........................................................................ [ 92%]
........................................................................ [ 96%]
........................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 98 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3273 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__04_ignore_paths_in_import_pipelines.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__04_ignore_paths_in_import_pipelines.a1.md
+19
View File
@@ -0,0 +1,19 @@
Task 04 complete. Report:
**Implemented**
- `app/api/sync.py::_run_sync` — per-row loop now also builds `ignore_by_root` (keyed by the same `str(root)` the importer sees; `setdefault().extend()` union on root-string collision) and passes it to `import_sources`; module docstring step 4 updated
- `app/api/git_sources.py::_run_upload` — step 6 captures the row's saved `ignore_paths` into a plain local (both existing/new-row branches, before `db.close()`); step 8 passes `ignore_by_root={str(final_dir): ignore_paths}`; route docstring steps 6/8 updated
- `scripts/import_docs.py` — `_resolve_sources` returns `(sources, ignore_by_root)` (manual `--source` and legacy fallback → `{}`); `main` unpacks and passes the map; module + function docstrings updated
- Tests: real-import sync test (ignored file excluded from KB + `files` count), fake-import union/empty-map wiring tests, CLI rows-branch/manual map tests, re-upload-respects-PATCHed-list test (pruned on re-scan) + new-source-no-list regression; `ignore_by_root` kwarg added to the 4 `import_sources` fakes
**Results**
- `uv run pytest -q` → 1795 passed, exit 0 (incl. `tests/integration/test_sync_api.py test_import_docs_git.py test_git_sources_upload.py` + `tests/unit/test_sync_button.py`: 105 passed)
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%); `app/api/sync.py` + `app/api/git_sources.py` 100%
- `uv run ruff check . && uv run pyright` → clean
- E2E regressions in isolation (`test_git_sources_admin`, `test_archive_upload_sources`, `test_sync_button`, `test_smoke`) → 17 passed
**Decisions**
- One-line mechanical addition to `tests/unit/test_sync_button.py` fake signature (outside the task's listed files) — required to keep the suite green once the runner passes the new kwarg
- Union-order test pins distinct `added_at` (same-timestamp rows tie-break on random uuid)
**Next pending task:** `05_ignore_paths_sources_ui.md`
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__04_ignore_paths_in_import_pipelines.a1.validate
+86
View File
@@ -0,0 +1,86 @@
........................................................................ [ 4%]
........................................................................ [ 8%]
........................................................................ [ 12%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 24%]
........................................................................ [ 28%]
........................................................................ [ 32%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 48%]
........................................................................ [ 52%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 76%]
........................................................................ [ 80%]
........................................................................ [ 84%]
........................................................................ [ 88%]
........................................................................ [ 92%]
........................................................................ [ 96%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__05_ignore_paths_sources_ui.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__05_ignore_paths_sources_ui.a1.md
+18
View File
@@ -0,0 +1,18 @@
**Task 05 complete — per-row "Ignore paths" box on the Sources page.**
Implemented:
- `frontend/index.html`: `#ignore-editor-dialog` (phase-69 `#remove-confirm-dialog` pattern — alertdialog, dim backdrop, visible label, mono textarea, `role="alert"` error line, Cancel/Save) placed after the remove dialog in `#git-sources-content`
- `frontend/assets/git-sources.js`: per-row "Ignore paths" button (stored rows only, left of Remove, textContent-only), `N ignored` count tag, `openIgnoreEditor`/`closeIgnoreEditor`/`saveIgnorePaths` with the §7.4 "Saving…" lifecycle, focus-on-Cancel, Escape/Cancel/backdrop close with focus return, blank-line dropping, A5-replace PATCH, LAST-announcement order; module docstring phase-89 bullet
- `frontend/assets/styles.css`: `.ignore-editor-*` dialog family (house tokens, no blur, ≥44px, brand Save / ghost Cancel) + `.git-source-ignore` / `.git-source-ignore-count` row chrome
- `tests/unit/test_source_ignore_paths.py` (new): 13 source-level pins (markup, lifecycle, row guard, CSS, id no-collision)
Results:
- `uv run pytest tests/unit/test_source_ignore_paths.py --no-cov` → 13 passed
- `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, TOTAL 99%** (>90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- E2E regressions in isolation: `test_git_sources_admin.py` 6 passed; `test_smoke.py` + `test_archive_upload_sources.py` + `test_sync_button.py` 11 passed
- Headless manual smoke: add local source → open box → save (blank line dropped) → `2 ignored` tag + announcer → refresh prefills (normalized) → Escape closes with focus return → empty box clears (A5) — all verified, source cleaned up
Decisions: Escape/Cancel/backdrop are no-ops mid-PATCH (in-flight guard, remove-dialog idiom); Save uses the `.new-chat-btn` brand hover pair; count tag = ink-on-bg chip (16.7:1). One bug found+fixed in my own test (f-string doubled the announce phrase).
Next pending task: `.agents/phases/todo/89_source_ignore_paths/06_e2e_ignore_paths.md`.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__05_ignore_paths_sources_ui.a1.validate
+87
View File
@@ -0,0 +1,87 @@
........................................................................ [ 3%]
........................................................................ [ 7%]
........................................................................ [ 11%]
........................................................................ [ 15%]
........................................................................ [ 19%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 31%]
........................................................................ [ 35%]
........................................................................ [ 39%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 55%]
........................................................................ [ 59%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 95%]
........................................................................ [ 99%]
........ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__06_e2e_ignore_paths.a1.err
View File
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__06_e2e_ignore_paths.a1.md
+11
View File
@@ -0,0 +1,11 @@
**Task 06 complete — report**
- Implemented `tests/e2e/test_source_ignore_paths.py` (new story suite, 6 tests): anonymous gate + 403s (incl. new PATCH), box→sync exclusion (`ignore/` never walked/embedded/summarized, `files` count excludes it), spec's no-mid-path rule (`sub/myfile.md` kept, root `myfile.md` ignored, list *replaces* per A5), A2 prune of newly-ignored files (`pruned == 2`), editor a11y + 422 path (501-char entry, textarea kept, Save re-enabled, happy-path heal), env-fallback rows with no box + env note. Module app on port 8140 with `kind="local"` fixture dir (6 files) — no git, no network.
- Results (all green):
- `uv run pytest --cov=app --cov-report=term-missing` → 1808 passed, TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (run twice, stable)
- Regression E2E in isolation: `test_git_sources_admin.py` 6 passed, `test_archive_upload_sources.py` 5 passed, `test_sync_button.py` 3 passed, `test_smoke.py` 3 passed
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors
- Deviations: (1) task's `files == 5` assertion is an arithmetic slip against its own 6-file tree (6−2=4) — test asserts 4, documented in-code; (2) stored-list round-trip asserts the A1-normalized value (`ignore` for box line `ignore/`); (3) no `git add`/`commit` run — left in working tree per executor rules (harness makes the one atomic `--no-gpg-sign` commit; `TODO.md` already exactly `# TODO`).
- Diff hygiene: matches step-3 file set (the small `tests/unit/test_importer.py` / `test_sync_button.py` modifications pre-date this task, from completed tasks 02/04).
- Next pending task: none — this was the last task of phase 89; the pipeline gate moves the phase dir and commits.
.agents/reports/89_source_ignore_paths/89_source_ignore_paths__06_e2e_ignore_paths.a1.validate
+87
View File
@@ -0,0 +1,87 @@
........................................................................ [ 3%]
........................................................................ [ 7%]
........................................................................ [ 11%]
........................................................................ [ 15%]
........................................................................ [ 19%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 31%]
........................................................................ [ 35%]
........................................................................ [ 39%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 55%]
........................................................................ [ 59%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 95%]
........................................................................ [ 99%]
........ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 178 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 235 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 29 0 100%
app/api/sync.py 101 0 100%
app/api/tokens.py 28 0 100%
app/config.py 141 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 19 0 100%
app/core/tokens.py 33 0 100%
app/db.py 21 0 100%
app/main.py 64 0 100%
app/models.py 95 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 222 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 190 3 98%
app/rag/llm.py 217 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 237 0 100%
--------------------------------------------------
TOTAL 3279 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
alembic/versions/0013_git_source_ignore_paths.py
+48
View File
@@ -0,0 +1,48 @@
"""git_sources.ignore_paths: per-source ignore-path list (phase 89)
Revision ID: 0013
Revises: 0012
Create Date: 2026-09-08
Phase 89 (per-source ignore paths: the owner types a list of
source-relative path prefixes per source — one path per line — into the
box on the Sources page; the importer never walks, embeds, or summarizes
a file whose source-relative path starts with one of them; A1 raw prefix
semantics, A4 limits, A5 replace semantics; one additive, reversible
column, A13):
* ``git_sources.ignore_paths`` — JSONB NOT NULL with server default
``'[]'``: the normalized, non-empty source-relative path prefixes. A
file is ignored when its source-relative POSIX path starts with any
entry (no mid-path matching, no globs). Existing rows read ``[]`` —
every pre-phase-89 row imports exactly as before.
"""
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
from alembic import op
revision = "0013"
down_revision = "0012"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"git_sources",
sa.Column(
"ignore_paths",
postgresql.JSONB(astext_type=sa.Text()),
server_default=sa.text("'[]'"),
nullable=False,
),
)
def downgrade() -> None:
# Safe order: the column is the only 0013 artefact — dropping it
# leaves 0012's schema byte-identical (A13, fully reversible).
op.drop_column("git_sources", "ignore_paths")
app/api/git_sources.py
+115 -17
View File
@@ -14,10 +14,18 @@ fallback is active).
Routes: ``GET`` (DB rows oldest-first, or the env list with Routes: ``GET`` (DB rows oldest-first, or the env list with
``from_env: true`` while the table is empty; rows carry ``kind`` + ``from_env: true`` while the table is empty; rows carry ``kind`` +
``path``, git rows — and env rows — report ``path: null``), ``POST`` ``path``, git rows — and env rows — report ``path: null``; every row
reports ``ignore_paths`` — DB rows their stored normalized list, env
rows ``[]``, phase 89), ``POST``
(201, validated create; ``kind`` selects the validation: git → exactly (201, validated create; ``kind`` selects the validation: git → exactly
the phase-35 URL contract, local → an existing absolute directory, else the phase-35 URL contract, local → an existing absolute directory, else
422 naming the path), ``POST /upload`` (phase 49, backgrounded in phase 422 naming the path; both kinds accept ``ignore_paths`` — optional,
absent → ``[]``, stored normalized, phase 89),
``PATCH /{source_id}`` (phase 89, A5 — replace one source's ignore
list: 404 unknown id, the required body list is normalized + A4-
validated with fixed-detail 422s and REPLACES the row's list wholesale
— an empty list clears all; 200 → the ``GitSourceOut`` shape),
``POST /upload`` (phase 49, backgrounded in phase
64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/ 64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/
``.zip`` name/format gate + the 1 MiB-chunk receive with the ``.zip`` name/format gate + the 1 MiB-chunk receive with the
``upload_max_mb`` cap run **inline** and answered 202 the moment the ``upload_max_mb`` cap run **inline** and answered 202 the moment the
@@ -90,7 +98,7 @@ from app.rag.archive_upload import (
swap_in, swap_in,
unpack_archive, unpack_archive,
) )
from app.rag.importer import import_sources from app.rag.importer import import_sources, normalize_ignore_path
from app.rag.llm import LLMClient, check_models from app.rag.llm import LLMClient, check_models
from app.rag.overview import regenerate_overview from app.rag.overview import regenerate_overview
from app.rag.source_removal import ( from app.rag.source_removal import (
@@ -101,6 +109,7 @@ from app.rag.source_removal import (
) )
from app.rag.sources_meta import bump_sources_version from app.rag.sources_meta import bump_sources_version
from app.schemas import ( from app.schemas import (
GitSourceIgnoreIn,
GitSourceIn, GitSourceIn,
GitSourceList, GitSourceList,
GitSourceOut, GitSourceOut,
@@ -172,6 +181,14 @@ _upload_status = UploadStatus()
#: accepted shapes are exactly these four prefixes. #: accepted shapes are exactly these four prefixes.
URL_RE = re.compile(r"^(https?://|ssh://|git@)") URL_RE = re.compile(r"^(https?://|ssh://|git@)")
#: Phase 89, A4 — the per-source ignore-list limits, enforced in
#: :func:`_validate_ignore_paths` (shared by POST and PATCH): at most
#: 200 entries, each ≤500 chars after normalization. The 422 details
#: are fixed strings that never echo the input (the router's
#: credential-safety discipline, applied for consistency).
MAX_IGNORE_PATHS = 200
MAX_IGNORE_PATH_LENGTH = 500
@router.get("", response_model=GitSourceList) @router.get("", response_model=GitSourceList)
def list_git_sources( def list_git_sources(
@@ -181,10 +198,13 @@ def list_git_sources(
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
same-timestamp inserts) with ``from_env: false`` — each row carries same-timestamp inserts) with ``from_env: false`` — each row carries
its ``kind`` and, for local rows, the stored ``path`` (git rows and its ``kind``, its ``ignore_paths`` (phase 89 — the stored,
env rows report ``path: null``); while the table is empty, the normalized list; ``or []`` guards a row that predated the column),
and, for local rows, the stored ``path`` (git rows and env rows
report ``path: null``); while the table is empty, the
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is ``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
git-only) with null ``id``/``added_at`` and ``from_env: true``. git-only) with null ``id``/``added_at``, ``ignore_paths: []`` (no
DB row to store a list on), and ``from_env: true``.
""" """
rows = db.scalars( rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc()) select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
@@ -200,6 +220,7 @@ def list_git_sources(
url=row.url, url=row.url,
path=row.path, path=row.path,
added_at=row.added_at, added_at=row.added_at,
ignore_paths=row.ignore_paths or [],
) )
for row in rows for row in rows
], ],
@@ -207,7 +228,9 @@ def list_git_sources(
) )
return GitSourceList( return GitSourceList(
sources=[ sources=[
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None) GitSourceRow(
id=None, kind="git", url=url, path=None, added_at=None, ignore_paths=[]
)
for url in get_settings().git_source_list for url in get_settings().git_source_list
], ],
from_env=True, from_env=True,
@@ -236,9 +259,35 @@ def create_git_source(
Wrong field combinations (git without url, local without path, both Wrong field combinations (git without url, local without path, both
kinds' fields) are 422 with fixed, input-free details. kinds' fields) are 422 with fixed, input-free details.
``ignore_paths`` (phase 89) — optional, both kinds: the raw box
lines are normalized + A4-validated (``_validate_ignore_paths`` —
the fixed-detail 422s) and the normalized list is what is stored and
reported.
""" """
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db) row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) return GitSourceOut(
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
)
def _validate_ignore_paths(raw: list[str] | None) -> list[str]:
"""Normalize + enforce the A4 limits (phase 89); the fixed 422
details never echo the input (the credential-safety discipline,
applied for consistency).
Shared by POST and PATCH. The empty check runs FIRST: a
whitespace-only entry must 422, not be silently dropped (the UI
drops blank lines client-side; the API stays defensive).
"""
entries = [normalize_ignore_path(e) for e in (raw or [])]
if any(not e for e in entries):
raise HTTPException(status_code=422, detail="ignore paths must be non-empty")
if len(entries) > MAX_IGNORE_PATHS:
raise HTTPException(status_code=422, detail="a source has at most 200 ignore paths")
if any(len(e) > MAX_IGNORE_PATH_LENGTH for e in entries):
raise HTTPException(status_code=422, detail="an ignore path exceeds 500 characters")
return entries
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource: def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
@@ -270,7 +319,13 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None: if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
raise HTTPException(status_code=409, detail="a git source with this URL already exists") raise HTTPException(status_code=409, detail="a git source with this URL already exists")
return _commit_new( return _commit_new(
GitSource(url=url, kind="git"), "a git source with this URL already exists", db GitSource(
url=url,
kind="git",
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
),
"a git source with this URL already exists",
db,
) )
@@ -296,12 +351,41 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
# rows carry the expanded path there too — git URL shapes and absolute # rows carry the expanded path there too — git URL shapes and absolute
# paths cannot collide). # paths cannot collide).
return _commit_new( return _commit_new(
GitSource(url=path, kind="local", path=path), GitSource(
url=path,
kind="local",
path=path,
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
),
f"a local source with this path already exists: {path}", f"a local source with this path already exists: {path}",
db, db,
) )
@router.patch("/{source_id}", response_model=GitSourceOut)
def patch_git_source(
source_id: uuid.UUID,
payload: GitSourceIgnoreIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Replace one source's ignore list (phase 89, A5).
404 unknown id. The body list (required) is normalized +
A4-validated (fixed 422 details) and REPLACES the row's list
wholesale — an empty list clears all. Returns the updated
row's public shape (id, url, added_at, ignore_paths).
"""
row = db.get(GitSource, source_id)
if row is None:
raise HTTPException(status_code=404, detail="git source not found")
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
db.commit()
db.refresh(row)
return GitSourceOut(
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
)
@router.post("/upload", response_model=UploadAccepted, status_code=202) @router.post("/upload", response_model=UploadAccepted, status_code=202)
async def upload_archive( async def upload_archive(
file: UploadFile = File(...), # noqa: B008 file: UploadFile = File(...), # noqa: B008
@@ -339,14 +423,18 @@ async def upload_archive(
6. upsert the row by ``path`` (``kind='local'``; an existing row is 6. upsert the row by ``path`` (``kind='local'``; an existing row is
left as-is — ``added_at`` preserved — and the unique index is left as-is — ``added_at`` preserved — and the unique index is
the backstop: a concurrent insert lands ``failed`` with the backstop: a concurrent insert lands ``failed`` with
``a local source with this path already exists: <path>``); ``a local source with this path already exists: <path>``); the
row's saved ``ignore_paths`` are captured for the scan (phase
89: a re-upload of an existing source honors the list the owner
already saved);
7. fail-fast ``check_models`` — ``ModelUnavailableError`` → 7. fail-fast ``check_models`` — ``ModelUnavailableError`` →
``failed`` with the sanitized message (the phase-49 503 becomes ``failed`` with the sanitized message (the phase-49 503 becomes
a status state, A5); the folder/row are already committed, so a status state, A5); the folder/row are already committed, so
the next sync/re-upload retries idempotently; the next sync/re-upload retries idempotently;
8. ``import_sources([folder], llm, prune=True, progress=<hook>)`` 8. ``import_sources([folder], llm, prune=True, progress=<hook>,
+ the change-gated ``regenerate_overview`` — the hook feeds the ignore_by_root={folder: row's list})`` (phase 89) + the
status ``current_file`` / ``files_done`` / ``files_total``; change-gated ``regenerate_overview`` — the hook feeds the status
``current_file`` / ``files_done`` / ``files_total``;
9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is 9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is
the background run's duration); the background run's duration);
10. ``success`` — ``detail`` = the ``UploadOut`` fields. 10. ``success`` — ``detail`` = the ``UploadOut`` fields.
@@ -502,8 +590,10 @@ async def _run_upload(
path = str(final_dir) path = str(final_dir)
db = SessionLocal() db = SessionLocal()
try: try:
if db.scalar(select(GitSource).where(GitSource.path == path)) is None: row = db.scalar(select(GitSource).where(GitSource.path == path))
db.add(GitSource(url=path, kind="local", path=path)) if row is None:
row = GitSource(url=path, kind="local", path=path)
db.add(row)
try: try:
db.commit() db.commit()
except IntegrityError: except IntegrityError:
@@ -511,6 +601,11 @@ async def _run_upload(
raise ValueError( raise ValueError(
f"a local source with this path already exists: {path}" f"a local source with this path already exists: {path}"
) from None ) from None
# Phase 89: the row's saved ignore list, copied to plain
# values while the row is still usable in this session — a
# re-upload of an existing source honors the list the owner
# already saved; a fresh row has no list yet.
ignore_paths = list(row.ignore_paths or [])
finally: finally:
db.close() db.close()
# Step 7 — fail-fast models (phase 41): ``ModelUnavailableError`` # Step 7 — fail-fast models (phase 41): ``ModelUnavailableError``
@@ -530,7 +625,10 @@ async def _run_upload(
_upload_status.files_done = done _upload_status.files_done = done
_upload_status.files_total = total _upload_status.files_total = total
summary = await import_sources([final_dir], llm, prune=True, progress=_hook) summary = await import_sources(
[final_dir], llm, prune=True, progress=_hook,
ignore_by_root={str(final_dir): ignore_paths},
)
overview = False overview = False
if summary.added + summary.updated > 0: if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm) overview = await regenerate_overview(llm)
app/api/sync.py
+20 -9
View File
@@ -34,9 +34,11 @@ decisions):
<path>``; a failing clone or a missing local dir aborts before any <path>``; a failing clone or a missing local dir aborts before any
import; import;
4. ``import_sources(..., prune=True)`` over the single combined list 4. ``import_sources(..., prune=True)`` over the single combined list
(git checkouts + local dirs) — prune so files deleted upstream or (git checkouts + local dirs), honoring each row's ``ignore_paths``
out of a local dir leave the index (pruning covers the union; the (phase 89 — the per-root ignore map is built in the same per-row
CLI's no-prune default is unchanged); loop as the source list) — prune so files deleted upstream, out of
a local dir, or newly matching an ignore pattern leave the index
(pruning covers the union; the CLI's no-prune default is unchanged);
5. when the import changed the KB (added + updated > 0), 5. when the import changed the KB (added + updated > 0),
``regenerate_overview`` refreshes the single ``kb_overview`` row ``regenerate_overview`` refreshes the single ``kb_overview`` row
(phase 31 trigger, best-effort inside); (phase 31 trigger, best-effort inside);
@@ -214,19 +216,26 @@ async def _run_sync() -> None:
) )
sources_root = Path(settings.sources_dir).expanduser() sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = [] sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
for row in rows: for row in rows:
if row.kind == "git": if row.kind == "git":
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url))) root = clone_or_pull(row.url, sources_root / repo_name(row.url))
else: else:
# kind=local — the stored expanded path (phase 38 also # kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the # mirrors it in the NOT-NULL ``url`` location column, the
# ``or`` keeps the type checker honest); re-verified at # ``or`` keeps the type checker honest); re-verified at
# sync time because the directory may have moved or been # sync time because the directory may have moved or been
# deleted since add-time. # deleted since add-time.
path = Path(row.path or row.url).expanduser() root = Path(row.path or row.url).expanduser()
if not path.is_dir(): if not root.is_dir():
raise GitSyncError(f"local source missing: {path}") raise GitSyncError(f"local source missing: {root}")
sources.append(path) sources.append(root)
# Phase 89: the row's ignore list, keyed by the SAME root
# string the importer sees; two rows sharing a root string
# get the union (extend, not replace) — the sibling/repo-name
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
# Phase 64 (task 02): the per-file progress hook — the status # Phase 64 (task 02): the per-file progress hook — the status
# endpoint reports the file being processed right now. The # endpoint reports the file being processed right now. The
# closure captures the module ``_status`` exactly like the state # closure captures the module ``_status`` exactly like the state
@@ -236,7 +245,9 @@ async def _run_sync() -> None:
_status.files_done = done _status.files_done = done
_status.files_total = total _status.files_total = total
summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook) summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root
)
overview = False overview = False
if summary.added + summary.updated > 0: if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm) overview = await regenerate_overview(llm)
app/models.py
+13 -1
View File
@@ -16,7 +16,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
* ``git_sources`` — admin-managed source registry (git URLs + local * ``git_sources`` — admin-managed source registry (git URLs + local
directories) the Sync button and import_docs directories) the Sync button and import_docs
import (phase 35; ``kind`` discriminator added in import (phase 35; ``kind`` discriminator added in
phase 38). phase 38; ``ignore_paths`` (phase 89 — JSONB list
of normalized path prefixes, server default
``'[]'``)).
* ``saved_chats`` — owner-saved chat conversations: one row per * ``saved_chats`` — owner-saved chat conversations: one row per
explicitly Saved conversation (auto-``title`` + explicitly Saved conversation (auto-``title`` +
the ``bor.chat.v1`` message list as JSONB, phase the ``bor.chat.v1`` message list as JSONB, phase
@@ -72,6 +74,7 @@ from sqlalchemy import (
Text, Text,
UniqueConstraint, UniqueConstraint,
func, func,
text,
) )
from sqlalchemy.dialects.postgresql import JSONB, UUID from sqlalchemy.dialects.postgresql import JSONB, UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -226,6 +229,15 @@ class GitSource(Base):
#: Absolute directory of a ``local`` source; NULL for git rows. #: Absolute directory of a ``local`` source; NULL for git rows.
#: Unique — Postgres treats NULLs as distinct under a unique index. #: Unique — Postgres treats NULLs as distinct under a unique index.
path: Mapped[str | None] = mapped_column(Text, unique=True) path: Mapped[str | None] = mapped_column(Text, unique=True)
#: Per-source ignore paths (phase 89, A1/A4): the normalized,
#: non-empty source-relative path prefixes the owner types into the
#: box on the Sources page. A file is ignored when its
#: source-relative POSIX path starts with any entry (raw prefix —
#: no mid-path matching, no globs). Server default '[]' — every
#: pre-phase-89 row imports exactly as before.
ignore_paths: Mapped[list] = mapped_column(
JSONB, default=list, server_default=text("'[]'"), nullable=False
)
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
app/rag/importer.py
+73 -3
View File
@@ -118,17 +118,63 @@ class ImportSummary:
) )
def normalize_ignore_path(entry: str) -> str:
"""One ignore-path entry → canonical form (phase 89, A1).
Trim surrounding whitespace, then strip ALL leading/trailing
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
all become ``"my/files"``. ``""`` / ``"//"``,
``" "`` normalize to ``""`` (callers drop empties).
"""
return entry.strip().strip("/")
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
"""Phase 89, A1 — the pure prefix rule, nothing else.
``rel`` is the source-relative POSIX path WITHOUT a leading
slash (the ``documents.path`` string). Match = ``rel`` STARTS
WITH a normalized entry: raw string prefix — deliberately NO
component-boundary check (``"my/files"`` also matches
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
"""
return any(rel.startswith(p) for p in prefixes)
def _ignore_for_root(
root: Path, ignore_by_root: dict[str, list[str]] | None
) -> tuple[str, ...]:
"""The normalized, non-empty prefix tuple for one root (phase 89).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (unambiguous when two rows share a
source *name* but different dirs). Callers may pass RAW box
lines: the importer normalizes + drops empties here, the single
choke point — stored lists (already normalized) normalize to
themselves.
"""
raw = (ignore_by_root or {}).get(str(root)) or []
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
def iter_importable_files( def iter_importable_files(
root: Path, root: Path,
extensions: frozenset[str], extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS, excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
) -> list[Path]: ) -> list[Path]:
"""All importable files under *root* (sorted), per the A9 scope rules. """All importable files under *root* (sorted), per the A9 scope rules.
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``). *extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
Skips: any path with a dot-prefixed component (hidden dirs/files — Skips: any path with a dot-prefixed component (hidden dirs/files —
vendored caches like ``.esphome/.espressif/**``) and the well-known vendored caches like ``.esphome/.espressif/**``) and the well-known
non-content directories in *excluded*. non-content directories in *excluded*. *ignore* (phase 89, A1) is a
tuple of ALREADY-normalized, non-empty source-relative path prefixes
(the importer's ``_ignore_for_root`` is the normalization choke point
— raw box lines never reach this function): a file is skipped when its
source-relative POSIX path starts with any entry; the default ``()``
keeps every existing caller byte-identical.
""" """
if not root.is_dir(): if not root.is_dir():
return [] return []
@@ -139,6 +185,8 @@ def iter_importable_files(
rel = path.relative_to(root) rel = path.relative_to(root)
if any(part.startswith(".") or part in excluded for part in rel.parts): if any(part.startswith(".") or part in excluded for part in rel.parts):
continue continue
if ignore and is_ignored(rel.as_posix(), ignore):
continue
if path.suffix.lower() not in extensions: if path.suffix.lower() not in extensions:
continue continue
files.append(path) files.append(path)
@@ -153,6 +201,7 @@ async def import_sources(
limit: int | None = None, limit: int | None = None,
session: Session | None = None, session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None, progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary: ) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring). """Import every A9-format file under *sources* (see module docstring).
@@ -172,6 +221,18 @@ async def import_sources(
``limit``, the hook still fires per processed file only — ``done`` never ``limit``, the hook still fires per processed file only — ``done`` never
exceeds the limit, but ``total`` stays the full pre-walk count (an exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator). incomplete walk must not misreport the denominator).
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
string exactly as passed in *sources* — to that source's RAW ignore-path
lines (the importer normalizes them via ``_ignore_for_root``, the single
choke point): matching files are never walked, so they are never
embedded and never summarized, and the progress pre-walk uses the same
per-root tuple as the processing loop, so ``total`` never counts them.
A file that newly matches a pattern simply never enters ``seen``, so the
next ``prune=True`` run deletes its row automatically (A2 — the A9
junk-precedent). ``None`` (the default) changes nothing: the map is read
per root, unlisted roots get an empty tuple, and every existing caller
behaves byte-identically.
""" """
if limit is not None and limit <= 0: if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1") raise ValueError("limit must be >= 1")
@@ -188,7 +249,13 @@ async def import_sources(
total = 0 total = 0
if progress is not None: if progress is not None:
for root in sources: for root in sources:
total += len(iter_importable_files(root, llm.settings.import_extension_set)) total += len(
iter_importable_files(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
)
)
try: try:
for root in sources: for root in sources:
if not root.is_dir(): if not root.is_dir():
@@ -198,7 +265,10 @@ async def import_sources(
break break
source = root.name source = root.name
source_names.add(source) source_names.add(source)
for path in iter_importable_files(root, llm.settings.import_extension_set): ignore = _ignore_for_root(root, ignore_by_root)
for path in iter_importable_files(
root, llm.settings.import_extension_set, ignore=ignore
):
if limit is not None and summary.files >= limit: if limit is not None and summary.files >= limit:
break break
rel = path.relative_to(root).as_posix() rel = path.relative_to(root).as_posix()
app/schemas.py
+30 -2
View File
@@ -327,11 +327,18 @@ class GitSourceIn(BaseModel):
Trimmed here; the API layer then ``expanduser()``s it and requires an Trimmed here; the API layer then ``expanduser()``s it and requires an
absolute existing directory (else 422 naming the path — the path is absolute existing directory (else 422 naming the path — the path is
not a secret, unlike a git URL) and no ``url``. not a secret, unlike a git URL) and no ``url``.
``ignore_paths`` (phase 89) is optional at create time (absent →
``[]``) and carries the RAW box lines — trimming/normalization happens
in the API layer, not the schema, so the A4 422 details stay fixed
strings (the router's credential-safety discipline, applied for
consistency).
""" """
kind: Literal["git", "local"] = "git" kind: Literal["git", "local"] = "git"
url: str | None = Field(default=None, min_length=1, max_length=500) url: str | None = Field(default=None, min_length=1, max_length=500)
path: str | None = Field(default=None, min_length=1, max_length=2000) path: str | None = Field(default=None, min_length=1, max_length=2000)
ignore_paths: list[str] | None = Field(default=None)
@field_validator("url", mode="before") @field_validator("url", mode="before")
@classmethod @classmethod
@@ -352,12 +359,14 @@ class GitSourceOut(BaseModel):
``kind=local`` rows, the stored (expanded) directory path — the ``kind=local`` rows, the stored (expanded) directory path — the
phase-35 response shape is unchanged by phase 38, so a local 201 phase-35 response shape is unchanged by phase 38, so a local 201
reports its path in ``url`` and the full row (``kind`` + ``path``) reports its path in ``url`` and the full row (``kind`` + ``path``)
via ``GET``. via ``GET``. ``ignore_paths`` (phase 89) is the stored, normalized
list — non-null (a row created without it reports ``[]``).
""" """
id: uuid.UUID | None id: uuid.UUID | None
url: str url: str
added_at: datetime | None added_at: datetime | None
ignore_paths: list[str]
class GitSourceRow(BaseModel): class GitSourceRow(BaseModel):
@@ -369,7 +378,9 @@ class GitSourceRow(BaseModel):
``path: null``; local rows carry ``path`` (the absolute directory, ``path: null``; local rows carry ``path`` (the absolute directory,
expanded) and the same string in ``url`` (the table's NOT-NULL expanded) and the same string in ``url`` (the table's NOT-NULL
location column). ``id`` / ``added_at`` are nullable: env-fallback location column). ``id`` / ``added_at`` are nullable: env-fallback
rows (table empty) carry neither. rows (table empty) carry neither. ``ignore_paths`` (phase 89) is the
row's stored, normalized list — env-fallback rows (no DB row to
store a list on) report ``[]``.
""" """
id: uuid.UUID | None id: uuid.UUID | None
@@ -377,6 +388,23 @@ class GitSourceRow(BaseModel):
url: str url: str
path: str | None path: str | None
added_at: datetime | None added_at: datetime | None
ignore_paths: list[str]
class GitSourceIgnoreIn(BaseModel):
"""``PATCH /api/git-sources/{source_id}`` body (phase 89, A5).
``ignore_paths`` is REQUIRED — the box's lines in REPLACE semantics:
the body list (normalized + A4-validated in the API layer, with the
fixed-detail 422s) becomes the row's whole list — an empty list
clears all; an absent field is a 422 with the model's own detail.
Entries are RAW box lines: normalization + the A4 limits are
enforced in the API layer so the 422 details stay fixed strings
(the router's credential-safety discipline, applied for
consistency).
"""
ignore_paths: list[str]
class GitSourceList(BaseModel): class GitSourceList(BaseModel):
frontend/assets/git-sources.js
+228
View File
@@ -143,6 +143,35 @@
* gate and never fetches /api/git-sources (the Sources-page gate * gate and never fetches /api/git-sources (the Sources-page gate
* pattern). * pattern).
* *
* Phase 89 (task 05) — the per-source ignore-paths editor: every
* STORED row (s.id truthy — env-fallback rows carry NO button, A3:
* nothing is stored to edit) gets an "Ignore paths" button in the
* actions cell (left of Remove) that opens the page-local alertdialog
* (#ignore-editor-dialog — the EXACT #remove-confirm-dialog pattern,
* phase 69): #ignore-editor-source names the source (textContent ONLY
* — the same `value` expression makeRow uses — URLs may embed
* user:pass@ credentials, phase 32), the mono textarea prefills the
* row's stored list (one path per line — the phase-89 A1 prefix rule,
* stated in plain words in the helper copy), and focus lands on
* Cancel (the safe default); Escape / Cancel / the dim backdrop close
* as CANCEL (no request — focus returns to the row's trigger button).
* "Save" runs the §7.4 in-flight lifecycle: both buttons disable +
* the save relabels "Saving…" while the PATCH
* /api/git-sources/{id} is out (blank lines in the box are separators,
* not entries — split + trim + drop empty client-side; the server
* still rejects empty entries defensively, A4). 200 → close (focus
* return), loadSources (the row's "N ignored" count tag lands — the
* A5 replace semantics round-trip through GET), announce — the update
* confirmation is the LAST announcement (the reload's "N sources
* listed." must not overwrite it); non-2xx → the in-dialog
* role="alert" line (the server detail, apiDetail 422-shape-aware) +
* the buttons restored, the dialog STAYS open and the textarea
* content is KEPT (the instruction survives — tuning-form
* convention); network failure → the fixed reachable? line, same
* restore. Rows whose list is non-empty show the "N ignored" count
* tag next to the location <code> (text + a distinct background —
* never color alone, WCAG 1.4.1).
*
* Scope boundary (phase locked decisions): adding a git repo does * Scope boundary (phase locked decisions): adding a git repo does
* NOT clone — the sync service (server-side) does that. Removing a * NOT clone — the sync service (server-side) does that. Removing a
* source, however, performs the FULL cleanup server-side (phase 69): * source, however, performs the FULL cleanup server-side (phase 69):
@@ -203,6 +232,17 @@ export async function mount(root) {
const removeError = root.querySelector("#remove-confirm-error"); const removeError = root.querySelector("#remove-confirm-error");
const removeCancelBtn = root.querySelector("#remove-confirm-cancel"); const removeCancelBtn = root.querySelector("#remove-confirm-cancel");
const removeRemoveBtn = root.querySelector("#remove-confirm-remove"); const removeRemoveBtn = root.querySelector("#remove-confirm-remove");
/* Phase 89: the per-source ignore-paths editor — the SAME page-local
alertdialog idiom as the remove-confirm modal (phase 69): static
markup in the shell's Sources view; this module owns the open /
cancel / save lifecycle. */
const ignoreDialog = root.querySelector("#ignore-editor-dialog");
const ignoreBackdrop = root.querySelector(".ignore-editor-backdrop");
const ignoreSourceEl = root.querySelector("#ignore-editor-source");
const ignoreTextarea = root.querySelector("#ignore-editor-textarea");
const ignoreErrorEl = root.querySelector("#ignore-editor-error");
const ignoreCancelBtn = root.querySelector("#ignore-editor-cancel");
const ignoreSaveBtn = root.querySelector("#ignore-editor-save");
/* Polite live region: the screen-reader confirmation for loads, adds, /* Polite live region: the screen-reader confirmation for loads, adds,
and removals (the phase-15 announcer pattern). */ and removals (the phase-15 announcer pattern). */
@@ -321,6 +361,16 @@ export async function mount(root) {
const code = document.createElement("code"); const code = document.createElement("code");
code.textContent = value; // rendered as text, never as HTML code.textContent = value; // rendered as text, never as HTML
urlTd.append(badge, code); urlTd.append(badge, code);
/* Phase 89: the "N ignored" count tag — stored rows with a
non-empty list only (text + a distinct background, never color
alone — WCAG 1.4.1). The list round-trips: (s.ignore_paths ||
[]) is the same expression openIgnoreEditor prefills from. */
if (s.id && (s.ignore_paths || []).length > 0) {
const count = document.createElement("span");
count.className = "git-source-ignore-count";
count.textContent = `${s.ignore_paths.length} ignored`;
urlTd.append(count);
}
tr.appendChild(urlTd); tr.appendChild(urlTd);
const addedTd = document.createElement("td"); const addedTd = document.createElement("td");
@@ -330,6 +380,22 @@ export async function mount(root) {
const actTd = document.createElement("td"); const actTd = document.createElement("td");
actTd.className = "git-source-actions-cell"; actTd.className = "git-source-actions-cell";
if (s.id) { if (s.id) {
/* Phase 89: the per-row ignore-paths editor trigger (A3: stored
rows only — env-fallback rows fall through to the "from
.env" tag below). The aria-label is the ONLY place `value`
appears (setAttribute — never innerHTML); the text label is
static. Sits LEFT of Remove — the non-destructive action
first. */
const ignoreBtn = document.createElement("button");
ignoreBtn.type = "button";
ignoreBtn.className = "git-source-ignore";
ignoreBtn.setAttribute(
"aria-label",
`Edit ignored paths for ${kindLabel} source: ${value}`,
);
ignoreBtn.textContent = "Ignore paths";
ignoreBtn.addEventListener("click", () => openIgnoreEditor(s, ignoreBtn));
actTd.appendChild(ignoreBtn);
const btn = document.createElement("button"); const btn = document.createElement("button");
btn.type = "button"; btn.type = "button";
btn.className = "git-source-remove"; btn.className = "git-source-remove";
@@ -514,6 +580,168 @@ export async function mount(root) {
if (removeBackdrop) removeBackdrop.addEventListener("click", cancelRemoveConfirm); if (removeBackdrop) removeBackdrop.addEventListener("click", cancelRemoveConfirm);
if (removeRemoveBtn) removeRemoveBtn.addEventListener("click", confirmRemove); if (removeRemoveBtn) removeRemoveBtn.addEventListener("click", confirmRemove);
/* ---------- ignore paths (PATCH /api/git-sources/{id}) — the per-row editor ----------
* Phase 89: a stored row's "Ignore paths" button (makeRow — left of
* Remove) opens the page-local alertdialog
* (#ignore-editor-dialog — the EXACT #remove-confirm-dialog idiom,
* phase 69) via openIgnoreEditor(s, triggerBtn): #ignore-editor-
* source names the source (textContent ONLY — the same `value`
* expression makeRow uses: s.path ?? s.url for local rows, s.url
* for git — URLs may embed user:pass@ credentials, phase 32), the
* mono textarea PREFILLS the row's stored list (one path per line
* — the phase-89 A1 prefix rule, stated in plain words in the
* helper copy), the error line clears, and focus lands on Cancel
* (the safe default). Escape / Cancel / the dim backdrop close as
* CANCEL: no request, focus returns to the row's trigger button.
*
* "Save" (saveIgnorePaths) runs the §7.4 never-stale lifecycle:
* both buttons disable and the save relabels "Saving…" while the
* PATCH is out — a blank line in the box is a separator, not an
* entry (split + trim + drop empty lines client-side; the server
* still rejects empty entries defensively, A4). 200 → close
* (focus return) → loadSources (the "N ignored" count tag lands —
* the A5 replace round-trips through GET) → announce (the update
* confirmation is the LAST announcement — the reload's "N sources
* listed." must not overwrite it); non-2xx → the in-dialog
* role="alert" line (the server detail, apiDetail 422-shape-aware)
* + the buttons restored — the dialog STAYS open and the textarea
* content is KEPT (the instruction survives — the tuning-form
* convention); network failure → the fixed reachable? line, same
* restore. Env-fallback rows (id null) carry NO button (A3 —
* nothing is stored to edit). */
let ignoreTarget = null; // the row object of the open editor
let ignoreTriggerBtn = null; // the row's button — focus returns here on close
let ignoreInFlight = false; // §7.4: a PATCH is out (both buttons disabled)
function openIgnoreEditor(s, triggerBtn) {
if (!ignoreDialog || !ignoreTextarea) return; // defensive — the markup ships with the page
if (ignoreInFlight) return; // one editor at a time
const isLocal = s.kind === "local";
// The same `value` expression makeRow uses — textContent ONLY.
if (ignoreSourceEl) ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url;
ignoreTextarea.value = (s.ignore_paths || []).join("\n");
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true; // a new attempt starts clean
}
ignoreInFlight = false;
ignoreTarget = s;
ignoreTriggerBtn = triggerBtn; // recorded for the focus return on close
ignoreDialog.hidden = false;
document.addEventListener("keydown", onIgnoreDialogKeydown);
// Cancel is the safe default — focus lands on it (the remove-
// dialog precedent; visibly: the global 3px :focus-visible
// outline).
if (ignoreCancelBtn) ignoreCancelBtn.focus();
}
/* Any close (cancel, success): hide the dialog, reset the textarea
+ error line, clear the target, detach the keydown handling, and
return focus to the row's "Ignore paths" button (WCAG 2.1). */
function closeIgnoreEditor() {
if (!ignoreDialog) return;
ignoreDialog.hidden = true;
ignoreInFlight = false;
if (ignoreTextarea) ignoreTextarea.value = "";
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true;
}
if (ignoreCancelBtn) ignoreCancelBtn.disabled = false;
if (ignoreSaveBtn) {
ignoreSaveBtn.disabled = false;
ignoreSaveBtn.textContent = "Save";
}
document.removeEventListener("keydown", onIgnoreDialogKeydown);
const trigger = ignoreTriggerBtn;
ignoreTarget = null;
ignoreTriggerBtn = null;
if (trigger) trigger.focus(); // focus returns to the row's button
}
/* Escape / Cancel / backdrop all close as cancel — NO request. A
cancel is a no-op while a PATCH is in flight (no half-cancel of
an in-progress save; the buttons are disabled anyway, the
Escape/backdrop paths need this guard). */
function cancelIgnoreEditor() {
if (ignoreInFlight) return;
closeIgnoreEditor();
}
/* While open (attached in openIgnoreEditor, detached in
closeIgnoreEditor): Escape closes as cancel. */
function onIgnoreDialogKeydown(e) {
if (e.key === "Escape") {
e.preventDefault();
cancelIgnoreEditor();
}
}
async function saveIgnorePaths() {
if (!ignoreTarget || ignoreInFlight) return; // one request at a time
// The box's lines: one path per line — a blank line is a
// separator, not an entry (trim + drop empty; the server still
// rejects empties defensively, A4).
const lines = ignoreTextarea.value.split("\n").map((l) => l.trim()).filter(Boolean);
const t = ignoreTarget;
const value = t.kind === "local" ? (t.path ?? t.url) : t.url;
ignoreInFlight = true;
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true;
}
if (ignoreCancelBtn) ignoreCancelBtn.disabled = true;
if (ignoreSaveBtn) {
ignoreSaveBtn.disabled = true;
ignoreSaveBtn.textContent = "Saving…"; // §7.4 in-flight label
}
try {
const r = await fetch(`/api/git-sources/${encodeURIComponent(t.id)}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ignore_paths: lines }),
});
if (r.ok) {
// 200: the server replaced the row's list (A5).
closeIgnoreEditor(); // focus returns to the row's button
await loadSources(); // the "N ignored" count tag lands
// The update confirmation is the LAST announcement: the
// reload's "N sources listed." must not overwrite it (the
// same order as the remove flow).
announce(`Ignored paths updated for ${value}`);
return;
}
// non-2xx: the in-dialog role="alert" line (the server detail)
// — the dialog STAYS open and the textarea content is KEPT:
// the fix is one edit + retry, not a re-type.
if (ignoreErrorEl) {
ignoreErrorEl.textContent = await apiDetail(r, "Could not save the ignored paths — try again.");
ignoreErrorEl.hidden = false;
}
} catch {
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "Could not save the ignored paths — is the app reachable?";
ignoreErrorEl.hidden = false;
}
} finally {
// Never stale (PLAN §7.4): the failure paths re-enable BOTH
// buttons + relabel the save; the success path already closed
// the dialog (which resets them) — the restore is a no-op
// there.
ignoreInFlight = false;
if (ignoreCancelBtn) ignoreCancelBtn.disabled = false;
if (ignoreSaveBtn) {
ignoreSaveBtn.disabled = false;
ignoreSaveBtn.textContent = "Save";
}
}
}
/* The dialog's own buttons (static markup — wired once). */
if (ignoreCancelBtn) ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor);
if (ignoreBackdrop) ignoreBackdrop.addEventListener("click", cancelIgnoreEditor);
if (ignoreSaveBtn) ignoreSaveBtn.addEventListener("click", saveIgnorePaths);
/* ---------- add (POST /api/git-sources) — the git form ---------- /* ---------- add (POST /api/git-sources) — the git form ----------
* wireAddForm gives the form the §7.4 never-stale lifecycle: while * wireAddForm gives the form the §7.4 never-stale lifecycle: while
* the request is out the button disables + relabels "Adding…" and * the request is out the button disables + relabels "Adding…" and
frontend/assets/styles.css
+215
View File
@@ -2183,6 +2183,173 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
color: var(--bg); color: var(--bg);
} }
/* ---------- Per-source ignore-paths editor (phase 89, task 05) ----------
The shell's Sources view: the page-local alertdialog that edits
one source's ignore list (one path per line — the phase-89 A1
prefix rule, stated in plain words in the helper copy). The EXACT
#remove-confirm-dialog overlay contract (phase 69): a fixed
full-viewport dim backdrop + a centered panel (z-index 1000, above
the sticky header (20) + skip-link (100); NO blur — the phase-08
no-blur perf anchor), scaled to the 46rem chat-column width or the
viewport, whichever is narrower. The box: a VISIBLE block label
(WCAG — never aria-label-only) over a mono textarea (the box is a
data entry, not prose — the var(--mono) stack); the error line is
the err pair (err-ink on err-bg 9.3:1, the err-line border);
Cancel is the ghost ink-soft family (5.1:1 on --surface); Save is
the solid brand family (--bg text on --brand 5.2:1 — the
.new-chat-btn convention; the hover lightens the fill). Both
buttons >=44px; :focus-visible via the global 3px outline rule (no
local suppression — focus lands on Cancel at open and is visible);
no animation (reduced-motion safe by construction). AA pairs: title
/ copy / label are --ink (or ink-soft) on --surface (13.8:1 /
5.1:1); the source value is --ink on --bg (16.7:1). Phase-08
tokens only; system fonts; no CDN. */
.ignore-editor {
position: fixed;
inset: 0;
z-index: 1000;
display: flex; /* the panel is the only in-flow child — margin: auto centers it */
}
/* Explicit (the global [hidden] rule already wins — the documented,
testable contract for the skeleton). */
.ignore-editor[hidden] { display: none; }
.ignore-editor-backdrop {
position: fixed;
inset: 0;
/* --bg at 82% — the doc-modal dim, no backdrop-filter (no-blur). */
background: rgba(15, 10, 10, 0.82);
}
.ignore-editor-panel {
/* position:relative lifts the panel above the fixed backdrop
(positioned elements paint over in-flow siblings otherwise). */
position: relative;
z-index: 1;
margin: auto;
width: min(46rem, calc(100vw - 2rem));
padding: 1.5rem;
background: var(--surface);
border: 1px solid var(--line);
border-radius: var(--radius);
box-shadow: var(--shadow-lg);
}
.ignore-editor-title {
margin: 0 0 0.75rem;
font-size: 1.25rem;
line-height: 1.3;
color: var(--ink);
}
/* The source's value (git URL or local path) — mono, wrapped (a long
URL must not overflow the panel), --ink on --bg (16.7:1). */
.ignore-editor-source {
display: block;
margin: 0 0 0.75rem;
padding: 0.5rem 0.65rem;
font-family: var(--mono);
font-size: 0.85rem;
line-height: 1.5;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
overflow-wrap: anywhere;
}
.ignore-editor-copy {
margin: 0 0 1rem;
color: var(--ink); /* 13.8:1 on --surface */
}
.ignore-editor-copy code { font-family: var(--mono); font-size: 0.85em; }
/* Visible block label (WCAG — the textarea is NEVER labelled only via
aria-label); ink-soft on --surface (5.1:1). */
.ignore-editor-label {
display: block;
margin-bottom: 0.35rem;
color: var(--ink-soft);
font-size: 0.88rem;
font-weight: 600;
}
/* The box: full panel width, mono (the var(--mono) stack), theme-
consistent border on --bg; the focus-visible ring is the global
3px outline (no local suppression); vertical resize for long
lists (up to 200 entries, A4). */
.ignore-editor-textarea {
display: block;
width: 100%;
min-height: 9rem;
padding: 0.55rem 0.65rem;
font-family: var(--mono);
font-size: 0.85rem;
line-height: 1.5;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
resize: vertical;
}
/* The in-dialog failure line (role=alert): the err pair (err-ink on
err-bg 9.3:1, the err-line border) — the .remove-confirm-error
language. */
.ignore-editor-error {
margin: 0.75rem 0 0;
padding: 0.5rem 0.65rem;
color: var(--err-ink);
background: var(--err-bg);
border: 1px solid var(--err-line);
border-radius: var(--radius-sm);
}
.ignore-editor-actions {
display: flex;
justify-content: flex-end;
gap: 0.6rem;
margin-top: 1.1rem;
}
/* The two dialog buttons: >=44px targets, the house 3px :focus-visible
via the global outline rule (no local override). */
.ignore-editor-btn {
min-height: 44px;
min-width: 44px;
padding: 0.55rem 1.1rem;
border-radius: var(--radius-sm);
font: inherit;
font-weight: 600;
font-size: 0.9rem;
white-space: nowrap;
cursor: pointer;
}
.ignore-editor-btn:disabled { opacity: 0.5; cursor: wait; }
/* Cancel — the ghost ink-soft family (5.1:1 on --surface), like the
remove dialog's Cancel; the brand pair on hover (12.4:1 on
brand-soft). */
.ignore-editor-cancel {
border: 1px solid var(--line);
background: transparent;
color: var(--ink-soft);
}
.ignore-editor-cancel:hover:not(:disabled) {
background: var(--brand-soft);
color: var(--brand-ink);
}
/* "Save" — the solid brand family (the .new-chat-btn convention):
--bg text on --brand (5.2:1 — AA); the hover lightens the brand
fill (the .new-chat-btn hover pair). */
.ignore-editor-save {
border: 0;
background: var(--brand);
color: var(--bg);
}
.ignore-editor-save:hover:not(:disabled) { background: #f55a72; }
/* The list: the Sources page's table pattern — full width in the /* The list: the Sources page's table pattern — full width in the
72rem frame, surface card, horizontally scrollable wrapper (the 72rem frame, surface card, horizontally scrollable wrapper (the
URL column never wraps or ellipsizes: long URLs, credentials URL column never wraps or ellipsizes: long URLs, credentials
@@ -2266,6 +2433,54 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.git-source-remove:hover:not(:disabled) { background: var(--err-bg); color: var(--err-ink); border-color: var(--err-line); } .git-source-remove:hover:not(:disabled) { background: var(--err-bg); color: var(--err-ink); border-color: var(--err-line); }
.git-source-remove:disabled { opacity: 0.5; cursor: wait; } .git-source-remove:disabled { opacity: 0.5; cursor: wait; }
/* Phase 89: the per-row "Ignore paths" trigger (makeRow — left of
Remove): the .git-source-remove idiom (same size/spacing, >=44px
target) in a NEUTRAL secondary fill — it must read distinct from
the destructive Remove (text label included, never icon-only; the
hover takes the brand-soft pair, 12.4:1 — Remove hovers to the err
pair, 9.3:1). */
.git-source-ignore {
display: inline-flex;
align-items: center;
justify-content: center;
min-height: 44px;
min-width: 44px;
flex: 0 0 auto;
padding: 0.35rem 0.7rem;
margin-right: 0.4rem;
border: 1px solid var(--line);
border-radius: var(--radius-sm);
background: transparent;
color: var(--ink-soft);
font: inherit;
font-weight: 600;
font-size: 0.82rem;
white-space: nowrap;
cursor: pointer;
}
.git-source-ignore:hover:not(:disabled) {
background: var(--brand-soft);
color: var(--brand-ink);
}
/* Phase 89: the "N ignored" count tag — next to the location <code>
in the Source cell (TEXT + a distinct background, never color
alone — WCAG 1.4.1): --ink on --bg (16.7:1) inside the --surface
card; the line border keeps the chip legible when the row hover
swaps the cell onto --bg too. */
.git-source-ignore-count {
display: inline-block;
margin-left: 0.55rem;
padding: 0.08rem 0.5rem;
border: 1px solid var(--line);
border-radius: 999px;
font-size: 0.72rem;
font-weight: 600;
color: var(--ink);
background: var(--bg);
white-space: nowrap;
}
/* Env-fallback rows carry no Remove (nothing is stored to remove) — /* Env-fallback rows carry no Remove (nothing is stored to remove) —
the tag says where the row comes from (brand pair, 6.9:1). */ the tag says where the row comes from (brand pair, 6.9:1). */
.git-source-env-tag { .git-source-env-tag {
frontend/index.html
+38
View File
@@ -647,6 +647,44 @@
</div> </div>
</div> </div>
</div> </div>
<!-- Phase 89: the per-source ignore-paths editor — the EXACT
#remove-confirm-dialog pattern (phase 69): page-local
alertdialog, dim backdrop, focus on Cancel (the safe
default), Escape/backdrop close as CANCEL, focus returns
to the row's trigger button. The box itself: one path
per line (prefix semantics — the phase-89 A1 rule,
stated in plain words in the helper line); Save runs the
§7.4 never-stale lifecycle (git-sources.js: "Saving…"
while the PATCH is out). Stored rows only (A3) — the
per-row button lives in git-sources.js's makeRow; this
is the page-local dialog it opens. -->
<div class="ignore-editor" id="ignore-editor-dialog" role="alertdialog"
aria-modal="true" aria-labelledby="ignore-editor-title"
aria-describedby="ignore-editor-copy" hidden>
<div class="ignore-editor-backdrop" aria-hidden="true"></div>
<div class="ignore-editor-panel">
<h2 class="ignore-editor-title" id="ignore-editor-title">Ignored files and folders</h2>
<code class="ignore-editor-source" id="ignore-editor-source"></code>
<p class="ignore-editor-copy" id="ignore-editor-copy">
One path per line. A file is ignored when its path in the
source starts with any of these — <code>/my/files/</code>,
<code>my/files/</code> and <code>my/files</code> mean the same
thing. No wildcards, no matching in the middle of a path.
</p>
<label class="ignore-editor-label" for="ignore-editor-textarea">Ignored paths</label>
<textarea class="ignore-editor-textarea" id="ignore-editor-textarea"
rows="6" spellcheck="false"
placeholder="my/files/"></textarea>
<p class="ignore-editor-error" id="ignore-editor-error" role="alert" hidden></p>
<div class="ignore-editor-actions">
<button type="button" class="ignore-editor-btn ignore-editor-cancel"
id="ignore-editor-cancel">Cancel</button>
<button type="button" class="ignore-editor-btn ignore-editor-save"
id="ignore-editor-save">Save</button>
</div>
</div>
</div>
</div> </div>
<!-- Polite live region: the screen-reader confirmation for list <!-- Polite live region: the screen-reader confirmation for list
loads, adds, and removals (git-sources.js owns the text). --> loads, adds, and removals (git-sources.js owns the text). -->
scripts/import_docs.py
+37 -13
View File
@@ -26,6 +26,11 @@ precedence order:
``~/Deployments``), kept for backwards compatibility (reached only ``~/Deployments``), kept for backwards compatibility (reached only
while both the table and ``BOR_GIT_SOURCES`` are empty). while both the table and ``BOR_GIT_SOURCES`` are empty).
Phase 89: resolution also returns each row's ignore paths, keyed by
the resolved root string (the importer normalizes them); manual
``--source`` dirs and the legacy fallback have no rows, so they import
with no ignore.
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add
@@ -133,9 +138,11 @@ def repo_name(url: str) -> str:
return slug or "repo" return slug or "repo"
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]: def _resolve_sources(
cli_sources: list[Path] | None, settings: Settings
) -> tuple[list[Path], dict[str, list[str]]]:
"""Resolve the directories to import (phase 28, extended in phases """Resolve the directories to import (phase 28, extended in phases
35 and 38). 35 and 38; per-root ignore maps, phase 89).
Precedence: ``--source`` (explicit manual paths — always wins) > Precedence: ``--source`` (explicit manual paths — always wins) >
the effective sources — the ``git_sources`` DB rows (git + local), the effective sources — the ``git_sources`` DB rows (git + local),
@@ -148,12 +155,18 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
stored directory, re-verified ``.is_dir()`` at run time) > the stored directory, re-verified ``.is_dir()`` at run time) > the
legacy ``DEFAULT_SOURCES``. legacy ``DEFAULT_SOURCES``.
Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed
by the resolved root string, exactly as the importer sees it (two
rows sharing a root string get the union — extend, not replace);
manual ``--source`` dirs and the legacy fallback have no rows, so
they import with an empty map (no ignore).
A :class:`GitSyncError` from a failing clone/pull — or a missing A :class:`GitSyncError` from a failing clone/pull — or a missing
local directory (``local source missing: <path>``) — propagates to local directory (``local source missing: <path>``) — propagates to
:func:`main`, which aborts the run before importing anything. :func:`main`, which aborts the run before importing anything.
""" """
if cli_sources: if cli_sources:
return [path.expanduser() for path in cli_sources] return [path.expanduser() for path in cli_sources], {}
db = SessionLocal() db = SessionLocal()
try: try:
rows, origin = effective_sources(db) rows, origin = effective_sources(db)
@@ -167,21 +180,28 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
) )
sources_root = Path(settings.sources_dir).expanduser() sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = [] sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
for row in rows: for row in rows:
if row.kind == "git": if row.kind == "git":
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url))) root = clone_or_pull(row.url, sources_root / repo_name(row.url))
else: else:
# kind=local — the stored expanded path (phase 38 also # kind=local — the stored expanded path (phase 38 also
# mirrors it in the NOT-NULL ``url`` location column, the # mirrors it in the NOT-NULL ``url`` location column, the
# ``or`` keeps the type checker honest); a missing # ``or`` keeps the type checker honest); a missing
# directory aborts before importing, the same pre-import # directory aborts before importing, the same pre-import
# fail-loud as a failing git clone. # fail-loud as a failing git clone.
path = Path(row.path or row.url).expanduser() root = Path(row.path or row.url).expanduser()
if not path.is_dir(): if not root.is_dir():
raise GitSyncError(f"local source missing: {path}") raise GitSyncError(f"local source missing: {root}")
sources.append(path) sources.append(root)
return sources # Phase 89: the row's ignore list, keyed by the SAME root
return [path.expanduser() for path in DEFAULT_SOURCES] # string the importer sees; two rows sharing a root string
# get the union (extend, not replace) — the sibling/repo-name
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
return sources, ignore_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}
def _overview_row_exists() -> bool: def _overview_row_exists() -> bool:
@@ -204,9 +224,10 @@ def main(argv: list[str] | None = None) -> int:
# Git sources resolve (and clone/pull) *before* any import: a failing # Git sources resolve (and clone/pull) *before* any import: a failing
# repo aborts the run with a non-zero exit, naming the failure — a bad # repo aborts the run with a non-zero exit, naming the failure — a bad
# URL must never silently import partial junk. # URL must never silently import partial junk. The second element is
# the phase-89 per-root ignore map (empty for manual/fallback paths).
try: try:
sources = _resolve_sources(args.source, settings) sources, ignore_by_root = _resolve_sources(args.source, settings)
except GitSyncError as e: except GitSyncError as e:
print(f"import_docs: source sync failed: {e}", file=sys.stderr) print(f"import_docs: source sync failed: {e}", file=sys.stderr)
return 1 return 1
@@ -248,7 +269,10 @@ def main(argv: list[str] | None = None) -> int:
runs and unchanged re-runs never bump. The returned token is runs and unchanged re-runs never bump. The returned token is
the new version, or ``"skipped"``. the new version, or ``"skipped"``.
""" """
summary = await import_sources(sources, llm, prune=args.prune, limit=args.limit) summary = await import_sources(
sources, llm, prune=args.prune, limit=args.limit,
ignore_by_root=ignore_by_root,
)
if args.limit is not None: if args.limit is not None:
# An incomplete walk is debug-only — it must never advance # An incomplete walk is debug-only — it must never advance
# the generation (mirrors the --limit overview skip below). # the generation (mirrors the --limit overview skip below).
tests/e2e/test_source_ignore_paths.py
+640
View File
@@ -0,0 +1,640 @@
"""Phase 89 story E2E (Playwright): per-source ignore paths — the box on
the Sources page excludes files/folders from the import (``/git-sources.html``).
Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-08 —
TODO-derived, no separate user-story file).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov
The story gate proves the whole item end to end through the REAL page +
REAL API + REAL sync pipeline (the in-app Sync's background task, mock
LLM). **No git, no network** — the suite's single source is a local
directory row (``kind="local"``, phase 38), so the sync walks the
fixture dir directly: no clone, no pull, no remote of any kind. The
module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL
(the ``test_git_sources_admin.py`` module-env pattern) that is NEVER
synced or cloned: every sync in this suite runs with the local DB row
in place (DB rows win over the env fallback), and the one
env-fallback test never triggers a sync.
Contract under test:
* anonymous: the sign-in gate (the ``#git-sources-gate`` pattern), the
manager hidden, NO ``/api/git-sources`` call on load, 403 on
``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}``
(the ``test_git_sources_admin.py`` anonymous pin, extended to the
phase-89 route);
* admin: the per-row "Ignore paths" button opens the page-local
alertdialog (the phase-69 ``#remove-confirm-dialog`` idiom); one path
per line; Save → ``PATCH`` 200 → the row shows the "N ignored" count
tag; the list round-trips through ``GET /api/git-sources``;
* the sync honors the list: with ``ignore/`` set, nothing under
``ignore/`` is indexed — no ``documents`` rows (and therefore no
chunks/embeddings, no summary calls) — and the sync's ``files``
count excludes the ignored files from the walk;
* the spec's no-mid-path rule: ``myfile.md`` ignores the root-level
``myfile.md`` only, never ``sub/myfile.md``; saving the box
REPLACES the list (A5 — the previous test's list is gone);
* A2: a previously indexed file that newly matches an ignore pattern
is PRUNED from the KB on the next sync (``detail.pruned``);
* a11y + the error path: the dialog is a real ``role="alertdialog"``,
the textarea has a visible label, focus lands on Cancel, Escape
closes and returns focus to the trigger; a 501-char entry 422s with
the fixed detail, the textarea content is KEPT and the Save button
re-enables with its "Save" label; the happy path heals the error
state;
* A3: env-fallback rows (table empty → ``BOR_GIT_SOURCES``) render the
"from .env" tag with NO "Ignore paths" button and the env note.
Test → contract mapping:
1. ``test_anonymous_gate_and_403s``
2. ``test_ignore_box_excludes_from_import``
3. ``test_prefix_rule_no_mid_path``
4. ``test_newly_ignored_file_is_pruned``
5. ``test_editor_a11y_and_error_path``
6. ``test_env_fallback_rows_have_no_box``
"""
from __future__ import annotations
import os
import subprocess
import sys
import time
import uuid
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_IGNORE", "8140"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is
#: NEVER cloned (the env-fallback test reads it; the sync tests all run
#: with the local DB row in place, so the env list never reaches a
#: clone).
ENV_SOURCE = "https://github.com/reese/env-alpha.git"
#: The sentinel inside the fixture's keeper doc — proof the (non-
#: ignored) markdown path was really imported.
SENTINEL = "RESE-IGNORE-KEEP-7f3a"
#: The six fixture files (source-relative POSIX paths — exactly the
#: strings ``documents.path`` stores).
KEEP_MD = "keep.md"
NOTES_YAML = "notes.yaml"
ROOT_MYFILE = "myfile.md"
SUB_MYFILE = "sub/myfile.md"
IGNORE_SECRET = "ignore/secret.md"
IGNORE_DEEP = "ignore/deep/x.txt"
ALL_SIX = (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE, IGNORE_SECRET, IGNORE_DEEP)
#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the
#: test_sync_button.py polling idiom) — real import of ≤6 small files
#: against the mock LLM; generous budget.
SYNC_TIMEOUT_S = 60.0
SYNC_TICK_S = 2.0
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The local source's on-disk tree (task 06 step 1).
Module-scoped (the same reasoning as ``test_sync_button.py``'s
module-scoped fixture note): ``tmp_path`` is function-scoped while
the module-scoped app + the seeded row reference the dir for the
module's lifetime, so it is built under ``tmp_path_factory`` (the
same pytest-managed temp area, module-safe). Plain files — no git:
a ``kind="local"`` row is walked directly by the sync (phase 38).
"""
root = tmp_path_factory.mktemp("ignore_src")
(root / "sub").mkdir()
(root / "ignore" / "deep").mkdir(parents=True)
(root / KEEP_MD).write_text(
"# Keep\n"
"\n"
"The keeper doc — the one markdown file the ignore list must\n"
"never touch.\n"
f"\n"
f"Marker: {SENTINEL}\n",
encoding="utf-8",
)
# Non-markdown → the phase-30 lite-summary path (proves "not
# summarized" applies to the non-md branch too when ignored).
(root / NOTES_YAML).write_text(
"title: Notes\nitems:\n - one\n - two\n", encoding="utf-8"
)
(root / ROOT_MYFILE).write_text(
"# Root myfile\n\nRoot-level file — matched by the bare "
"``myfile.md`` prefix (the spec's own example).\n",
encoding="utf-8",
)
(root / SUB_MYFILE).write_text(
"# Sub myfile\n\nA namesake in a subdirectory — NEVER matched "
"by ``myfile.md`` (no mid-path matching).\n",
encoding="utf-8",
)
(root / IGNORE_SECRET).write_text(
"# Secret\n\nIgnored by the ``ignore/`` prefix.\n", encoding="utf-8"
)
(root / IGNORE_DEEP).write_text("deep ignored text\n", encoding="utf-8")
return root
@pytest.fixture(scope="module")
def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]:
"""The real app under test — per-module env (the conftest pattern,
module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env
fallback's subject — never cloned), module-scratch checkouts/
upload dirs, the mock LLM. No git anywhere in this suite's path."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env
# shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The empty-table env fallback (only read while ``git_sources`` is
# EMPTY — the env-fallback test; never cloned in this suite).
env["BOR_GIT_SOURCES"] = ENV_SOURCE
# Module-scratch dirs (never reached by this suite's local-row
# syncs — kept explicit so a shared checkouts dir cannot leak rows
# into the walk).
scratch = source_dir.parent
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern, the
test_sync_button.py module-env DSN): the sync's counts and the
catalog must be each test's own doing."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources"
)
)
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""This suite owns ``git_sources`` AND the KB tables (the E2E
isolation pattern): suites run in isolation but share one
Postgres, and a leftover row would flip the app from the
``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover
document would skew the prune counts). Empty BOTH before and
after every test."""
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str:
"""(Re)seed the stored row for the module fixture dir via the REAL
admin API (201) — ``kind="local"`` (the POST contract is unchanged
by phase 89; no ``ignore_paths`` → the stored list is ``[]``) —
then reload the view so the page's table re-fetches the row (the
API seed is server-side: the mounted view does not know about it
without a re-show). Returns the stored (expanded) path — the
row's ``url`` location value, the same string the page renders
and aria-labels."""
r = page.request.post(
f"{app_url}/api/git-sources",
data={"kind": "local", "path": str(source_dir)},
)
assert r.status == 201, r.text
body = r.json()
assert body["ignore_paths"] == [] # the default list, round-tripped
page.reload()
_admin_git_sources_page(page, app_url)
return body["url"]
def run_sync(page: Page, app_url: str) -> dict[str, Any]:
"""``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal
state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s
budget). Returns the terminal status body."""
r = page.request.post(f"{app_url}/api/sync")
assert r.status == 202, r.text
deadline = time.monotonic() + SYNC_TIMEOUT_S
body: dict[str, Any] = {}
while True:
s = page.request.get(f"{app_url}/api/sync/status")
assert s.status == 200, s.text
body = s.json()
if body["state"] in ("success", "failed"):
return body
assert time.monotonic() < deadline, (
f"sync did not reach a terminal state: {body}"
)
time.sleep(SYNC_TICK_S)
def _catalog_paths(page: Page, app_url: str) -> list[str]:
"""The RAG catalog's data source (``GET /api/docs`` — the Sources
page's table): every ``documents.path`` (source-relative POSIX)."""
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200, r.text
return [d["path"] for d in r.json()["documents"]]
def _row(page: Page, value: str) -> Any:
"""The table row whose mono location cell shows ``value``."""
return page.locator("#git-sources-tbody tr", has_text=value)
def _open_ignore_editor(page: Page, value: str) -> Any:
"""Click the row's "Ignore paths" button; the alertdialog opens.
Returns the trigger button (focus returns to it on close)."""
btn = _row(page, value).locator(".git-source-ignore")
expect(btn).to_have_count(1)
btn.click()
dialog = page.locator("#ignore-editor-dialog")
expect(dialog).to_be_visible(timeout=30_000)
return btn
def _save_ignore_list(page: Page, value: str, entry: str) -> None:
"""Open the row's box, replace its content with one line (``entry``
— ``""`` clears) and Save; wait for the A5 round-trip: the dialog
closes and the row shows the "1 ignored" / cleared count state."""
_open_ignore_editor(page, value)
page.fill("#ignore-editor-textarea", entry)
page.click("#ignore-editor-save")
if entry.strip():
expect(_row(page, value).locator(".git-source-ignore-count")).to_have_text(
"1 ignored", timeout=30_000
)
else:
expect(_row(page, value).locator(".git-source-ignore-count")).to_have_count(0)
expect(page.locator("#ignore-editor-dialog")).to_be_hidden()
# ---------------------------------------------------------------------------
# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. PATCH)
# ---------------------------------------------------------------------------
def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None:
page.set_default_timeout(30_000)
# Track every /api/git-sources request the page itself makes — the
# gate must be reached WITHOUT touching the admin API (the
# test_git_sources_admin.py pattern).
api_calls: list[str] = []
page.on(
"request",
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
)
page.goto(app_url + GIT_SOURCES_URL)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
# The sign-in gate (the #sources-gate pattern, phase 16)…
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
expect(
gate.locator("a[href='/login.html?next=/git-sources.html']")
).to_have_count(1)
# …and the manager is absent/inert: table + form + env note, all
# inside the hidden #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers on GET/POST AND the
# phase-89 PATCH (the page context has no cookie — the
# test_git_sources_admin.py anonymous pin, extended).
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
f"{app_url}/api/git-sources", data={"url": ENV_SOURCE}
).status
== 403
)
assert (
page.request.patch(
f"{app_url}/api/git-sources/{uuid.uuid4()}",
data={"ignore_paths": ["ignore/"]},
).status
== 403
)
# ---------------------------------------------------------------------------
# 2. The box sets the list; the sync honors it (nothing under ignore/
# embedded/summarized; the walk's files count excludes it)
# ---------------------------------------------------------------------------
def test_ignore_box_excludes_from_import(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# Seed the row (no list) through the real API…
stored_path = _seed_local_source(page, app_url, source_dir)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# …open the box, type "ignore/" (one line), Save → 200 → the row's
# "1 ignored" count tag lands (the A5 round-trip through GET).
_save_ignore_list(page, stored_path, "ignore/")
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
# A1: the raw box line is stored NORMALIZED ("ignore/" → "ignore")
# — the round-trip is canonical.
assert [s["ignore_paths"] for s in body["sources"]] == [["ignore"]]
assert body["from_env"] is False
# The real sync (mock LLM): success, and the walk counted the
# remaining FOUR files — the two ignore/ files never entered it
# (no documents → no chunks/embeddings, no summary calls for
# them). (6 fixture files − 2 ignored = 4; the task's "5" is a
# slip against its own six-file tree — the exclusion is what is
# under test, and it holds either way.)
sync = run_sync(page, app_url)
assert sync["state"] == "success", sync
detail = sync["detail"]
assert detail["files"] == 4, detail
assert detail["added"] == 4, detail
assert detail["errors"] == 0, detail
# The catalog holds exactly the four non-ignored paths.
paths = _catalog_paths(page, app_url)
for p in (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE):
assert p in paths, f"{p} missing from the catalog: {paths}"
assert not any(p.startswith("ignore/") for p in paths), (
f"ignored paths leaked into the catalog: {paths}"
)
# ---------------------------------------------------------------------------
# 3. The spec's no-mid-path rule: myfile.md ignores the root-level file
# only; the box REPLACES the previous test's list (A5)
# ---------------------------------------------------------------------------
def test_prefix_rule_no_mid_path(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# Bare filename prefix (no leading/trailing slash): the spec's own
# example — matches the root-level file, never the subdirectory
# namesake.
_save_ignore_list(page, stored_path, "myfile.md")
sync = run_sync(page, app_url)
assert sync["state"] == "success", sync
assert sync["detail"]["files"] == 5, sync["detail"]
paths = _catalog_paths(page, app_url)
# No mid-path matching: the namesake in sub/ is indexed…
assert SUB_MYFILE in paths, f"sub/myfile.md was wrongly ignored: {paths}"
# …the root-level file is NOT…
assert ROOT_MYFILE not in paths, f"root myfile.md must be ignored: {paths}"
# …and the ignore/ files ARE indexed this time — the box replaced
# (A5), not appended to, the previous test's "ignore/" list
# (regression guard).
assert IGNORE_SECRET in paths, f"ignore/secret.md must be indexed: {paths}"
assert IGNORE_DEEP in paths, f"ignore/deep/x.txt must be indexed: {paths}"
# ---------------------------------------------------------------------------
# 4. A2: a previously indexed file that newly matches is pruned
# ---------------------------------------------------------------------------
def test_newly_ignored_file_is_pruned(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# Sync 1 — no list: all six files are indexed.
first = run_sync(page, app_url)
assert first["state"] == "success", first
assert first["detail"]["files"] == 6, first["detail"]
assert first["detail"]["added"] == 6, first["detail"]
paths = _catalog_paths(page, app_url)
for p in ALL_SIX:
assert p in paths, f"{p} missing after the first sync: {paths}"
# New list: "ignore/" — the two previously indexed files now match.
_save_ignore_list(page, stored_path, "ignore/")
# Sync 2 — the prune=True run deletes the newly ignored documents
# (the seen-set mechanism, the A9 out-of-scope-junk precedent).
second = run_sync(page, app_url)
assert second["state"] == "success", second
detail = second["detail"]
assert detail["pruned"] == 2, detail
assert detail["files"] == 4, detail
paths = _catalog_paths(page, app_url)
for p in (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE):
assert p in paths, f"{p} missing after the prune: {paths}"
assert not any(p.startswith("ignore/") for p in paths), (
f"newly ignored paths survived the prune: {paths}"
)
# ---------------------------------------------------------------------------
# 5. The editor's a11y + the 422 error path (and the happy-path heal)
# ---------------------------------------------------------------------------
def test_editor_a11y_and_error_path(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# The trigger is the labeled per-row button (left of Remove)…
row = _row(page, stored_path)
btn = row.locator(".git-source-ignore")
expect(btn).to_have_count(1)
expect(btn).to_have_attribute(
"aria-label", f"Edit ignored paths for local source: {stored_path}"
)
expect(row.locator(".git-source-ignore-count")).to_have_count(0) # no list yet
# …and it opens a real alertdialog (the phase-69 idiom) with the
# visible label and the Cancel-safe focus default.
btn.click()
dialog = page.locator("#ignore-editor-dialog")
expect(dialog).to_be_visible(timeout=30_000)
assert dialog.get_attribute("role") == "alertdialog"
assert dialog.get_attribute("aria-modal") == "true"
# exact: the row button's aria-label ("Edit ignored paths for …")
# contains the substring — the visible <label> is the only exact
# "Ignored paths" name.
expect(page.get_by_label("Ignored paths", exact=True)).to_have_count(1)
# The dialog names the source (textContent — the local path).
expect(page.locator("#ignore-editor-source")).to_have_text(stored_path)
expect(page.locator("#ignore-editor-cancel")).to_be_focused()
# Escape closes as CANCEL — no request — and focus returns to the
# row's trigger button.
page.keyboard.press("Escape")
expect(dialog).to_be_hidden()
expect(btn).to_be_focused()
# The 422 path: one entry of 501 chars (> the A4 500-char limit) →
# the fixed detail, the textarea content KEPT, Save re-enabled with
# its idle label (never stale).
btn.click()
expect(dialog).to_be_visible(timeout=30_000)
bad = "a" * 501
page.fill("#ignore-editor-textarea", bad)
page.click("#ignore-editor-save")
error = page.locator("#ignore-editor-error")
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_have_text("an ignore path exceeds 500 characters")
expect(page.locator("#ignore-editor-textarea")).to_have_value(bad)
expect(dialog).to_be_visible() # the fix is one edit + retry
save_btn = page.locator("#ignore-editor-save")
expect(save_btn).to_be_enabled()
expect(save_btn).to_have_text("Save")
# Nothing was stored: the list is still empty (no count tag).
expect(row.locator(".git-source-ignore-count")).to_have_count(0)
# The happy path heals the error state: clear, one entry, Save →
# the dialog closes and the "1 ignored" tag lands.
page.fill("#ignore-editor-textarea", "keep.md")
save_btn.click()
expect(dialog).to_be_hidden(timeout=30_000)
expect(
page.locator("#git-sources-tbody tr .git-source-ignore-count")
).to_have_text("1 ignored", timeout=30_000)
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert [s["ignore_paths"] for s in r.json()["sources"]] == [["keep.md"]]
# ---------------------------------------------------------------------------
# 6. A3: env-fallback rows have no box (no DB row to store a list on)
# ---------------------------------------------------------------------------
def test_env_fallback_rows_have_no_box(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# The autouse fixture truncated git_sources — the table is EMPTY,
# so the module app's BOR_GIT_SOURCES URL is the effective list
# (the test_git_sources_admin.py module-env pattern).
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
row = page.locator("#git-sources-tbody tr")
expect(row).to_contain_text(ENV_SOURCE)
# The env-fallback surface: the "from .env" tag (never color
# alone)…
expect(row.locator(".git-source-env-tag")).to_have_count(1)
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
# …and NO per-row actions at all — neither Remove (nothing stored
# to remove) nor the phase-89 "Ignore paths" box (A3).
expect(row.locator(".git-source-remove")).to_have_count(0)
expect(row.locator(".git-source-ignore")).to_have_count(0)
expect(row.locator(".git-source-ignore-count")).to_have_count(0)
# The env-fallback note explains the active list's origin…
expect(page.locator("#git-sources-env-note")).to_be_visible()
# …and the API agrees: from_env true, null ids, the env URL, and an
# empty ignore list (no DB row to store a list on).
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert body["from_env"] is True
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE]
assert all(
s["id"] is None and s["ignore_paths"] == [] for s in body["sources"]
)
tests/integration/test_git_sources_api.py
+226 -4
View File
@@ -28,7 +28,14 @@ Contract under test:
index as backstop); wrong field combinations (git without url, local index as backstop); wrong field combinations (git without url, local
without path, both kinds' fields) → 422; without path, both kinds' fields) → 422;
* DELETE — 204 and gone; an emptied table falls back to the env list * DELETE — 204 and gone; an emptied table falls back to the env list
again; unknown id → 404. again; unknown id → 404;
* ignore paths (phase 89) — POST accepts the RAW box lines for both
kinds (optional, absent → ``[]``), stored normalized (A1) with the A4
fixed-detail 422s (shared gate with PATCH); GET reports each row's
stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only)
replaces the list wholesale (A5 — an empty list clears all), 404
unknown id, 422 fixed details for the A4 limits, the row otherwise
unchanged.
``git_sources`` is global state: truncated around every test. ``git_sources`` is global state: truncated around every test.
""" """
@@ -92,6 +99,10 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
r = client.delete(f"/api/git-sources/{uuid.uuid4()}") r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 403 assert r.status_code == 403
assert r.json() == {"detail": "admin only"} assert r.json() == {"detail": "admin only"}
# The phase-89 ignore-list PATCH is gated the same way.
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# Nothing landed in the table. # Nothing landed in the table.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0 assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
@@ -121,6 +132,8 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "https://a.example.com/one.git", "url": "https://a.example.com/one.git",
"path": None, "path": None,
"added_at": None, "added_at": None,
# Env rows have no DB row to store a list on (phase 89).
"ignore_paths": [],
}, },
{ {
"id": None, "id": None,
@@ -128,6 +141,7 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "git@b.example.com:two.git", "url": "git@b.example.com:two.git",
"path": None, "path": None,
"added_at": None, "added_at": None,
"ignore_paths": [],
}, },
] ]
@@ -193,7 +207,10 @@ def test_post_creates_trimmed_and_list_stops_using_env(
assert body["url"] == "https://new.example.com/repo.git" # trimmed assert body["url"] == "https://new.example.com/repo.git" # trimmed
uuid.UUID(body["id"]) uuid.UUID(body["id"])
assert body["added_at"] is not None assert body["added_at"] is not None
assert set(body) == {"id", "url", "added_at"} # Phase 89: the response gains ``ignore_paths`` — absent at create
# time → ``[]``.
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert body["ignore_paths"] == []
# The DB row now wins: from_env False, the env URL is gone from the list. # The DB row now wins: from_env False, the env URL is gone from the list.
body = admin_client.get("/api/git-sources").json() body = admin_client.get("/api/git-sources").json()
@@ -303,8 +320,10 @@ def test_post_local_creates_stored_row_with_expanded_path(
assert r.status_code == 201, r.text assert r.status_code == 201, r.text
body = r.json() body = r.json()
# The phase-35 response shape is unchanged — the local row reports # The phase-35 response shape is unchanged — the local row reports
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET. # its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
assert set(body) == {"id", "url", "added_at"} # phase 89 adds ``ignore_paths`` (absent → ``[]``).
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert body["ignore_paths"] == []
uuid.UUID(body["id"]) uuid.UUID(body["id"])
assert body["url"] == str(real_dir) assert body["url"] == str(real_dir)
assert body["added_at"] is not None assert body["added_at"] is not None
@@ -536,6 +555,7 @@ def test_delete_removes_row_and_falls_back_to_env(
"url": "https://env.example.com/env.git", "url": "https://env.example.com/env.git",
"path": None, "path": None,
"added_at": None, "added_at": None,
"ignore_paths": [], # env rows: no DB row to store a list on
} }
] ]
db.execute(text("TRUNCATE chunks, documents")) db.execute(text("TRUNCATE chunks, documents"))
@@ -550,3 +570,205 @@ def test_delete_unknown_id_returns_404(admin_client: TestClient) -> None:
def test_delete_invalid_id_returns_422(admin_client: TestClient) -> None: def test_delete_invalid_id_returns_422(admin_client: TestClient) -> None:
assert admin_client.delete("/api/git-sources/not-a-uuid").status_code == 422 assert admin_client.delete("/api/git-sources/not-a-uuid").status_code == 422
# --- ignore paths (phase 89) ----------------------------------------------
def test_post_stores_normalized_ignore_paths_both_kinds(
admin_client: TestClient, tmp_path: Path
) -> None:
"""POST accepts the RAW box lines for both kinds (A1 normalization
in the API layer) — the normalized list is stored and round-trips
through GET."""
r = admin_client.post(
"/api/git-sources",
json={
"url": "https://example.com/ig.git",
"ignore_paths": ["/my/files/", " my/files2 ", "x"],
},
)
assert r.status_code == 201, r.text
assert r.json()["ignore_paths"] == ["my/files", "my/files2", "x"]
real_dir = tmp_path / "ig"
real_dir.mkdir()
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "path": str(real_dir), "ignore_paths": ["//skip/", "keep"]},
)
assert r.status_code == 201, r.text
assert r.json()["ignore_paths"] == ["skip", "keep"]
# Round-trip through GET (keyed by url — (added_at, id) order of two
# same-millisecond inserts is not the point under test).
by_url = {
s["url"]: s["ignore_paths"] for s in admin_client.get("/api/git-sources").json()["sources"]
}
assert by_url["https://example.com/ig.git"] == ["my/files", "my/files2", "x"]
assert by_url[str(real_dir)] == ["skip", "keep"]
def test_post_without_ignore_paths_reports_empty_list(
admin_client: TestClient, db: Session
) -> None:
"""Absent ``ignore_paths`` at create time → ``[]`` — both in the 201
response and in the GET round-trip (the migration's server default)."""
r = admin_client.post("/api/git-sources", json={"url": "https://example.com/none.git"})
assert r.status_code == 201
assert r.json()["ignore_paths"] == []
body = admin_client.get("/api/git-sources").json()
assert body["sources"][0]["ignore_paths"] == []
# The stored value is the JSONB server default, not Python-only.
row = db.scalars(select(GitSource)).one()
assert row.ignore_paths == []
def test_post_rejects_invalid_ignore_paths_like_patch(
admin_client: TestClient, db: Session
) -> None:
"""POST shares ``_validate_ignore_paths`` with PATCH — the same A4
fixed 422 details; nothing is stored."""
for payload, detail in (
([" "], "ignore paths must be non-empty"),
([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"),
(["a" * 501], "an ignore path exceeds 500 characters"),
):
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/bad-ignore.git", "ignore_paths": payload},
)
assert r.status_code == 422, f"{detail!r}: {r.text}"
assert r.json()["detail"] == detail
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_get_reports_stored_ignore_paths_and_env_rows_empty(
admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch
) -> None:
"""GET reports each DB row's stored list; env-fallback rows (no DB
row to store a list on) report ``[]`` with ``from_env: true``."""
db.add(GitSource(url="https://example.com/stored.git", ignore_paths=["docs/drafts"]))
db.commit()
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is False
assert body["sources"][0]["ignore_paths"] == ["docs/drafts"]
db.execute(text("DELETE FROM git_sources"))
db.commit()
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: _settings("https://a.example.com/env.git"),
)
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is True
assert body["sources"] == [
{
"id": None,
"kind": "git",
"url": "https://a.example.com/env.git",
"path": None,
"added_at": None,
"ignore_paths": [],
}
]
def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -> None:
"""PATCH 200 — REPLACE semantics (A5): the body list, normalized,
becomes the row's whole list; an empty list clears all; ``id``/
``url``/``added_at`` are unchanged; the response is the
``GitSourceOut`` shape incl. the new list."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/patch.git", "ignore_paths": ["old/"]},
)
assert created.status_code == 201
before = created.json()
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert body["ignore_paths"] == ["a", "b"] # normalized
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Round-trip through GET.
assert (
admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a", "b"]
)
# An empty list clears all.
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": []})
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == []
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == []
def test_patch_missing_field_is_422(admin_client: TestClient) -> None:
"""``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent
field is a 422 with the model's own detail."""
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"})
assert created.status_code == 201
r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={})
assert r.status_code == 422
# The model's own (Pydantic) detail — the missing required field.
assert any(
item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"]
)
def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None:
r = admin_client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
assert r.status_code == 404
assert r.json() == {"detail": "git source not found"}
def test_patch_invalid_id_returns_422(admin_client: TestClient) -> None:
r = admin_client.patch("/api/git-sources/not-a-uuid", json={"ignore_paths": []})
assert r.status_code == 422
def test_patch_422s_are_fixed_details(admin_client: TestClient) -> None:
"""The A4 422s are exact fixed strings (never echoing the input),
and a rejected PATCH leaves the row's list unchanged."""
created = admin_client.post(
"/api/git-sources", json={"url": "https://example.com/v.git", "ignore_paths": ["keep"]}
)
assert created.status_code == 201
sid = created.json()["id"]
for payload, detail in (
# Whitespace-only → empty after normalization: 422, not a silent
# drop (the UI drops blank lines client-side; the API is
# defensive).
([" "], "ignore paths must be non-empty"),
(list(f"e{i}" for i in range(201)), "a source has at most 200 ignore paths"),
(["a" * 501], "an ignore path exceeds 500 characters"),
):
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": payload})
assert r.status_code == 422, f"{detail!r}: {r.text}"
assert r.json()["detail"] == detail
# Nothing changed by the failed PATCHes.
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["keep"]
def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None:
"""Exactly 200 entries and a 500-char entry (post-normalization)
are the accepted edge of A4."""
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/bnd.git"})
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": [f"e/{i}" for i in range(200)]}
)
assert r.status_code == 200, r.text
assert len(r.json()["ignore_paths"]) == 200
long_entry = "a" * 500
assert len(long_entry) == 500
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]})
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == [long_entry]
tests/integration/test_git_sources_upload.py
+90
View File
@@ -291,6 +291,7 @@ class _GatedImport:
limit: int | None = None, limit: int | None = None,
session: object = None, session: object = None,
progress: Callable[[str, str, int, int], None] | None = None, progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
) -> ImportSummary: ) -> ImportSummary:
self.prune_flags.append(prune) self.prune_flags.append(prune)
if progress is not None: if progress is not None:
@@ -1352,3 +1353,92 @@ def test_models_down_fails_the_run_and_leaves_folder_and_row(
# The scan never ran: no docs, no stray temps. # The scan never ran: no docs, no stray temps.
assert _docs(upload_client) == [] assert _docs(upload_client) == []
assert [p.name for p in uploads.iterdir()] == ["homelab"] assert [p.name for p in uploads.iterdir()] == ["homelab"]
# --- phase 89: re-uploads honor the row's saved ignore list -------------------
def test_reupload_honors_saved_ignore_list(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase 89: a re-upload of an EXISTING source honors the ignore
list saved on its row. First scan (fresh row, no list) indexes the
ignored file too; once the list is saved via the API, re-uploading
the same archive again scans only the kept file — the previously
indexed ignored file leaves the KB (prune), and the run lands
``success`` with counts that exclude it. The upload itself keeps
every file on disk: the ignore is about the index, not the folder."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"keep.md": "# Keep\nin scope\n",
"ignore/secret.md": "# Secret\nignored\n",
}
r = _post(upload_client, "docs.tar.gz", _targz_bytes(files))
assert r.status_code == 202, r.text
status = _wait_status(upload_client)
assert status["state"] == "success", status
# First scan: the fresh row has no list → both files are indexed.
assert status["detail"]["files"] == 2
assert _docs(upload_client) == [("docs", "ignore/secret.md"), ("docs", "keep.md")]
# Save the ignore list on the row via the API (phase 89, task 03).
folder = uploads / "docs"
row = _row(db, str(folder))
assert row is not None
r = upload_client.patch(
f"/api/git-sources/{row.id}", json={"ignore_paths": ["ignore/"]}
)
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == ["ignore"] # stored normalized
# Re-upload the SAME archive (same name → in-place replace).
r = _post(upload_client, "docs.tar.gz", _targz_bytes(files))
assert r.status_code == 202, r.text
status = _wait_status(upload_client)
assert status["state"] == "success", status
detail = status["detail"]
# The scan walked only keep.md (unchanged); the ignored file never
# entered the walk, so prune dropped the first scan's row for it.
assert detail["files"] == 1
assert detail["added"] == 0
assert detail["updated"] == 0
assert detail["unchanged"] == 1
assert detail["pruned"] == 1
assert _docs(upload_client) == [("docs", "keep.md")]
# The re-upload left the row's list in place (the existing row is
# untouched by the upsert), and the folder keeps every file.
row_after = _row(db, str(folder))
assert row_after is not None
assert row_after.ignore_paths == ["ignore"]
assert {p.name for p in folder.iterdir()} == {"keep.md", "ignore"}
def test_upload_new_source_without_list_imports_everything(
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase 89 regression: an upload of a NEW source name (no row yet,
hence no ignore list) imports everything in the archive — including
nested files — exactly as before phase 89."""
uploads = tmp_path / "uploads"
_point_at(monkeypatch, uploads)
_real_llm(monkeypatch)
files = {
"alpha.md": "# Alpha\nroot file\n",
"sub/deep.md": "# Deep\nnested file\n",
}
r = _post(upload_client, "fresh.tar.gz", _targz_bytes(files))
assert r.status_code == 202, r.text
status = _wait_status(upload_client)
assert status["state"] == "success", status
detail = status["detail"]
assert detail["files"] == 2
assert detail["added"] == 2
# The fresh row carries the server-default empty list: nothing was
# ignored.
row = _row(db, str(uploads / "fresh"))
assert row is not None
assert (row.ignore_paths or []) == []
assert _docs(upload_client) == [("fresh", "alpha.md"), ("fresh", "sub/deep.md")]
tests/integration/test_import_docs_git.py
+114 -11
View File
@@ -43,14 +43,15 @@ def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Sett
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue] return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _git_row(url: str) -> GitSource: def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
return GitSource(url=url, kind="git") return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
def _local_row(path: str) -> GitSource: def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource:
"""A local row as the phase-38 API stores it: the expanded path in """A local row as the phase-38 API stores it: the expanded path in
both ``path`` and the NOT-NULL ``url`` location column.""" both ``path`` and the NOT-NULL ``url`` location column (plus the
return GitSource(url=path, kind="local", path=path) phase-89 ignore list, empty by default)."""
return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or [])
class FakeImportSources: class FakeImportSources:
@@ -66,8 +67,12 @@ class FakeImportSources:
*, *,
prune: bool = False, prune: bool = False,
limit: int | None = None, limit: int | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
) -> ImportSummary: ) -> ImportSummary:
self.calls.append({"sources": list(sources), "prune": prune, "limit": limit}) self.calls.append(
{"sources": list(sources), "prune": prune, "limit": limit,
"ignore_by_root": ignore_by_root}
)
return ImportSummary(files=1, added=1) return ImportSummary(files=1, added=1)
@@ -146,9 +151,10 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
sources_dir=str(tmp_path / "bor"), sources_dir=str(tmp_path / "bor"),
) )
sources = import_docs._resolve_sources(None, settings) sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"] assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
assert calls == [ assert calls == [
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"), ("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"), ("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
@@ -163,9 +169,10 @@ def test_resolve_sources_cli_source_wins(
settings = _settings(git_sources="https://host/a/repo.git") settings = _settings(git_sources="https://host/a/repo.git")
manual = tmp_path / "Manual" manual = tmp_path / "Manual"
sources = import_docs._resolve_sources([manual], settings) sources, ignore_map = import_docs._resolve_sources([manual], settings)
assert sources == [manual] assert sources == [manual]
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
assert calls == [] # git is never touched when --source is given assert calls == [] # git is never touched when --source is given
@@ -186,9 +193,10 @@ def test_resolve_sources_db_rows_win_over_env(
sources_dir=str(tmp_path / "bor"), sources_dir=str(tmp_path / "bor"),
) )
sources = import_docs._resolve_sources(None, settings) sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only"] assert sources == [tmp_path / "bor" / "only"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")] assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
@@ -197,8 +205,102 @@ def test_resolve_sources_defaults_when_nothing_configured(
) -> None: ) -> None:
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs. # Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env")) monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
sources = import_docs._resolve_sources(None, _settings()) sources, ignore_map = import_docs._resolve_sources(None, _settings())
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES] assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
assert ignore_map == {} # phase 89: the legacy fallback has no rows
def test_resolve_sources_rows_branch_builds_ignore_map(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Phase 89: the rows branch returns each row's ignore list keyed by
the resolved root string — the local row's directory, the git row's
checkout dir; a row without a list contributes nothing to the map."""
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: (
[
_git_row("https://db.example/only.git"),
_local_row(str(local_dir), ignore_paths=["ignore/"]),
],
"db",
),
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
# Keyed by the SAME string the importer sees (the root, not the name).
assert ignore_map == {str(local_dir): ["ignore/"]}
def test_resolve_sources_two_rows_sharing_root_string_extend(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Phase 89 collision rule: two rows resolving to the SAME root
string (the sibling/repo-name edge — ``…/shared`` and
``…/shared.git``) get the UNION of their lists (extend, not
replace), in row order."""
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: (
[
_git_row("https://a.example/shared", ignore_paths=["a/"]),
_git_row("https://a.example/shared.git", ignore_paths=["b"]),
],
"db",
),
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
shared = str(tmp_path / "bor" / "shared")
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
assert ignore_map == {shared: ["a/", "b"]} # union, row order
def test_main_rows_branch_passes_ignore_map_to_import(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Phase 89: a ``kind=local`` row carrying ``ignore_paths`` →
``main`` passes the per-root map to ``import_sources`` (keyed by
the directory string, prune flag unchanged)."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
(local_dir / "ignore").mkdir()
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir), ignore_paths=["ignore/"])], "db"),
)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
_stub_bump(monkeypatch)
rc = import_docs.main([])
assert rc == 0
call = fake_import.calls[0]
assert call["sources"] == [local_dir]
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
assert call["prune"] is False # the CLI's no-prune default is unchanged
# --- main() ---------------------------------------------------------------- # --- main() ----------------------------------------------------------------
@@ -306,9 +408,10 @@ def test_resolve_sources_mixed_git_and_local(
sources_dir=str(tmp_path / "bor"), sources_dir=str(tmp_path / "bor"),
) )
sources = import_docs._resolve_sources(None, settings) sources, ignore_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir] assert sources == [tmp_path / "bor" / "only", local_dir]
assert ignore_map == {} # phase 89: neither row carries a list
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")] assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
tests/integration/test_importer_ignore.py
+202
View File
@@ -0,0 +1,202 @@
"""Integration: phase-89 ignore paths through the real import pipeline.
Phase 89 (TODO.md L3): per-source ignore lists — source-relative path
prefixes that are never walked, hence never embedded and never
summarized (A1), and previously indexed files that newly match a
pattern are pruned on the next ``prune=True`` run (A2). Mirrors the
fixture-tree + mock-LLM pattern of ``test_importer_e2e.py``: a
``tmp_path`` source dir named ``IgnoreFix`` run through the real
``import_sources`` into the compose Postgres, with
:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.models import Chunk, Document
from app.rag.importer import import_sources
from tests.fakes import FakeEmbedder
NAME = "IgnoreFix"
def _write(root: Path, rel: str, content: str) -> None:
path = root / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
def _tree(tmp_path: Path, name: str = NAME) -> Path:
"""The fixture tree: two kept files + one md and one txt under ``ignore/``."""
root = tmp_path / name
_write(root, "keep.md", "# Keep\n\nkept body\n")
_write(root, "ignore/secret.md", "# Secret\n\nSECRET-CONTENT\n")
_write(root, "ignore/notes.txt", "IGNORED-TEXT-CONTENT\n")
_write(root, "top.txt", "TOP-TEXT-CONTENT\n")
return root
def _reset(db: Session) -> None:
# House cleanup pattern (tests/integration/test_importer_e2e.py).
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
def test_ignored_files_never_indexed(db: Session, tmp_path: Path) -> None:
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
summary = asyncio.run(
import_sources(
[root], llm, session=db, ignore_by_root={str(root): ["ignore/"]}
)
)
# Only the two kept files are walked — the ignore/ subtree is
# invisible to the pipeline.
assert summary.files == 2
assert summary.errors == 0
# The kept non-markdown file IS summarized; the ignored .txt is not.
assert summary.summaries == 1
assert summary.summary_errors == 0
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
(NAME, "keep.md"),
(NAME, "top.txt"),
}
# No documents row for an ignored file — hence no chunks rows for it,
# no embedding call, and no summary column value, by construction.
for rel in ("ignore/secret.md", "ignore/notes.txt"):
assert not any(d.path == rel for d in docs)
assert not any("SECRET-CONTENT" in c.content for c in db.scalars(select(Chunk)).all())
for texts in llm.calls: # every embed batch
assert not any(
"SECRET-CONTENT" in t or "IGNORED-TEXT-CONTENT" in t for t in texts
)
# Exactly one summary call (top.txt) — the ignored files never reached
# the lite model.
assert len(llm.chat_calls) == 1
user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user")
assert "TOP-TEXT-CONTENT" in user
assert "SECRET-CONTENT" not in user and "IGNORED-TEXT-CONTENT" not in user
top = next(d for d in docs if d.path == "top.txt")
assert top.summary is not None
_reset(db)
def test_newly_ignored_file_pruned_on_next_prune_run(db: Session, tmp_path: Path) -> None:
_reset(db)
root = tmp_path / NAME
_write(root, "keep.md", "# Keep\n\nkept body\n")
_write(root, "top.txt", "TOP-TEXT-CONTENT\n")
# Exactly ONE file under ignore/ so the A2 prune count pins it.
_write(root, "ignore/secret.md", "# Secret\n\nSECRET-CONTENT\n")
llm = FakeEmbedder()
# First run — no map (omitted entirely): everything is indexed,
# including ignore/secret.md.
s1 = asyncio.run(import_sources([root], llm, session=db))
assert (s1.files, s1.added) == (3, 3)
secret = db.scalar(select(Document).where(Document.path == "ignore/secret.md"))
assert secret is not None
# Second run — the owner adds "ignore" (no trailing slash: A1
# normalization) and prunes. The file newly matches, never enters
# ``seen``, and leaves the index (A2 — the A9 junk-precedent).
s2 = asyncio.run(
import_sources(
[root],
llm,
session=db,
prune=True,
ignore_by_root={str(root): ["ignore"]},
)
)
assert s2.files == 2
assert s2.pruned == 1
assert (
db.scalar(select(Document).where(Document.path == "ignore/secret.md")) is None
)
_reset(db)
def test_progress_total_excludes_ignored(db: Session, tmp_path: Path) -> None:
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
calls: list[tuple[str, str, int, int]] = []
def progress(source: str, rel: str, done: int, total: int) -> None:
calls.append((source, rel, done, total))
summary = asyncio.run(
import_sources(
[root],
llm,
session=db,
progress=progress,
ignore_by_root={str(root): ["ignore/"]},
)
)
assert summary.files == 2
# The phase-64 pre-walk uses the same per-root tuple as the loop:
# ``total`` counts ONLY the non-ignored files, and the hook fired
# exactly once per imported file.
assert [c[2] for c in calls] == [1, 2] # done
assert {c[3] for c in calls} == {2} # total — never counts ignored files
assert {c[1] for c in calls} == {"keep.md", "top.txt"}
_reset(db)
def test_no_map_behavior_is_byte_identical(db: Session, tmp_path: Path) -> None:
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
# ``ignore_by_root=None`` (the default): all four files import exactly
# as pre-phase-89 callers see them.
summary = asyncio.run(import_sources([root], llm, session=db, ignore_by_root=None))
assert summary.files == 4
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
(NAME, "keep.md"),
(NAME, "top.txt"),
(NAME, "ignore/secret.md"),
(NAME, "ignore/notes.txt"),
}
_reset(db)
def test_unlisted_source_unaffected(db: Session, tmp_path: Path) -> None:
_reset(db)
root_a = _tree(tmp_path, name="IgnoreFixA")
root_b = tmp_path / "IgnoreFixB"
_write(root_b, "one.md", "# One\n\none body\n")
_write(root_b, "two.txt", "TWO-TEXT-CONTENT\n")
llm = FakeEmbedder()
# The map keys ONLY the first root — the second imports everything.
summary = asyncio.run(
import_sources(
[root_a, root_b],
llm,
session=db,
ignore_by_root={str(root_a): ["ignore"]},
)
)
# A: keep.md + top.txt (the whole ignore/ subtree is dropped) · B: one.md + two.txt
assert summary.files == 4
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
("IgnoreFixA", "keep.md"),
("IgnoreFixA", "top.txt"),
("IgnoreFixB", "one.md"),
("IgnoreFixB", "two.txt"),
}
assert not any(d.path == "ignore/secret.md" for d in docs)
_reset(db)
tests/integration/test_sync_api.py
+132
View File
@@ -238,6 +238,10 @@ class FakeImportSources:
# Phase 64 (task 02): the progress hook the runner passes (a live # Phase 64 (task 02): the progress hook the runner passes (a live
# closure while wired, None if the wiring regresses). # closure while wired, None if the wiring regresses).
self.progress_hooks: list[object] = [] self.progress_hooks: list[object] = []
# Phase 89: the per-root ignore map the runner builds from the
# rows' ``ignore_paths`` (keyed by the root string the importer
# sees; two rows sharing a root string get the union).
self.ignore_maps: list[dict[str, list[str]]] = []
async def __call__( async def __call__(
self, self,
@@ -248,11 +252,13 @@ class FakeImportSources:
limit: int | None = None, limit: int | None = None,
session: Session | None = None, session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None, progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary: ) -> ImportSummary:
self.sources.append(list(sources)) self.sources.append(list(sources))
self.llms.append(llm) self.llms.append(llm)
self.prune_flags.append(prune) self.prune_flags.append(prune)
self.progress_hooks.append(progress) self.progress_hooks.append(progress)
self.ignore_maps.append(ignore_by_root or {})
if self.delay: if self.delay:
await asyncio.sleep(self.delay) await asyncio.sleep(self.delay)
return self.summary return self.summary
@@ -747,6 +753,7 @@ def test_import_error_is_reported_with_credentials_masked(
limit: int | None = None, limit: int | None = None,
session: Session | None = None, session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None, progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
) -> ImportSummary: ) -> ImportSummary:
raise EmbeddingError( raise EmbeddingError(
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 " "embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
@@ -897,3 +904,128 @@ def test_probe_runs_before_source_resolution(
assert order == ["probe", "effective_sources"] assert order == ["probe", "effective_sources"]
assert clone_calls == [] assert clone_calls == []
assert "short-circuit" in body["error"] # the spy aborted the run assert "short-circuit" in body["error"] # the spy aborted the run
# --- phase 89: per-row ignore lists ------------------------------------------
def test_local_row_ignore_paths_excluded_from_sync(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 89: a local row carrying ``ignore_paths`` — the button
sync skips every matching file: the ignored file never lands in the
KB (no document row — hence no embedding, no summary), and the
success detail's ``files``/``added`` counts exclude it, while the
kept file imports as usual."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
(local_dir / "ignore").mkdir()
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
encoding="utf-8")
db.add(
GitSource(
url=str(local_dir), kind="local", path=str(local_dir),
ignore_paths=["ignore/"], # any spelling — the importer normalizes
)
)
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# The ignored file is not counted: only keep.md was walked.
assert body["detail"]["files"] == 1
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
# The KB holds exactly the kept file — ignore/secret.md is absent.
docs = [(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]]
assert docs == [("LocalDocs", "keep.md")]
def test_sync_builds_ignore_map_by_root_string_with_union(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 89 wiring (fake import): the runner keys the map by the
SAME root string the importer sees, and two rows sharing that root
string (the sibling/repo-name edge — ``…/shared`` and
``…/shared.git`` clone into the same checkout dir) get the UNION
of their lists, in row order; a row with an empty list contributes
nothing."""
url_a = f"file://{tmp_path / 'shared'}"
url_b = f"{url_a}.git" # same repo name → same checkout dir
# Distinct added_at: the resolver orders by (added_at, id) — a
# same-timestamp pair would tie-break on the random uuid.
db.add(GitSource(url=url_a, kind="git", ignore_paths=["a/"],
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
db.add(GitSource(url=url_b, kind="git", ignore_paths=["b"],
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
shared = str(tmp_path / "bor" / "shared")
# Both rows resolve to the SAME checkout (the collision itself) and
# the map holds their union, keyed by that one root string.
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
assert fake_import.ignore_maps == [{shared: ["a/", "b"]}]
def test_sync_without_ignore_lists_passes_empty_map(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 89 regression: rows without a list → the runner passes an
EMPTY map (the importer's byte-identical pre-phase-89 behavior)."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "notes.md").write_text("# Notes\nplain row\n", encoding="utf-8")
_seed_local(db, local_dir) # no ignore_paths → []
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
assert fake_import.sources == [[local_dir]]
assert fake_import.ignore_maps == [{}] # no row carried a list
tests/unit/test_importer.py
+7 -2
View File
@@ -763,11 +763,16 @@ def test_no_progress_means_no_prewalk(
walk_calls = 0 walk_calls = 0
def counting( def counting(
r: Path, extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS r: Path,
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
) -> list[Path]: ) -> list[Path]:
# Phase 89: the walker gained the ``ignore`` keyword — the sentinel
# accepts (and forwards) it to stay a drop-in.
nonlocal walk_calls nonlocal walk_calls
walk_calls += 1 walk_calls += 1
return real_walker(r, extensions, excluded) return real_walker(r, extensions, excluded, ignore)
monkeypatch.setattr(importer, "iter_importable_files", counting) monkeypatch.setattr(importer, "iter_importable_files", counting)
try: try:
tests/unit/test_importer_ignore.py
+180
View File
@@ -0,0 +1,180 @@
"""Unit: the phase-89 ignore-path matcher (source-level pins).
Phase 89 (TODO.md L3 — "The ignore is just a prefix ignore."): each
stored source carries a list of source-relative path prefixes; a file
is ignored when its source-relative POSIX path (the ``documents.path``
string — NO leading slash) STARTS WITH a normalized entry. The spec's
own examples are pinned verbatim:
* ``"/my/files/"``, ``"my/files/"`` and ``"my/files"`` all normalize to
``"my/files"`` (whitespace + ALL leading/trailing slashes stripped);
* ``"myfile.txt"`` matches ``"myfile.txt"`` but NOT
``"some/path/myfile.txt"`` — no mid-path matching, no globs;
* raw string prefix, deliberately NO component boundary (A1,
owner-confirmed 2026-09-08): ``"my/files"`` also matches
``"my/files2/x.md"`` and ``"a"`` matches ``"ab.md"``.
Normalization happens exactly once — in ``_ignore_for_root``, the
single choke point — so ``iter_importable_files`` receives
ALREADY-normalized tuples (that contract is pinned too: a raw box line
does NOT match at the walk level). The DB-facing behavior (ignored
files never indexed/embedded/summarized; A2 prune) is integration-
gated by ``tests/integration/test_importer_ignore.py``.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from app.rag.importer import (
_ignore_for_root,
is_ignored,
iter_importable_files,
normalize_ignore_path,
)
#: The walk-level fixture uses an explicit extension set (not the A9
#: config default) so the test pins the matcher, not the config.
EXTS = frozenset({".md", ".txt", ".yaml"})
# --- normalize_ignore_path -------------------------------------------------
@pytest.mark.parametrize(
("raw", "expected"),
[
("/my/files/", "my/files"), # spec: all three spellings are equal
("my/files/", "my/files"),
("my/files", "my/files"),
(" my/files ", "my/files"), # surrounding whitespace trimmed
("//", ""), # nothing left after stripping
("", ""),
(" ", ""),
],
)
def test_normalize_ignore_path(raw: str, expected: str) -> None:
assert normalize_ignore_path(raw) == expected
# --- is_ignored: the spec's own examples, verbatim --------------------------
def test_is_ignored_myfile_txt_root_level_matches() -> None:
# Spec verbatim: "myfile.txt" would match "/myfile.txt" — the leading
# slash never exists in documents.path, so the root-level file matches.
assert is_ignored("myfile.txt", ("myfile.txt",)) is True
@pytest.mark.parametrize(
"rel",
[
"my/files/a.md", # spec: ignores everything under the prefix
"my/files/sub/b.md", # …and arbitrarily deep
"my/files", # a file literally named after the prefix
],
)
def test_is_ignored_prefix_matches(rel: str) -> None:
assert is_ignored(rel, ("my/files",)) is True
@pytest.mark.parametrize(
"rel",
[
"some/path/myfile.txt", # spec verbatim: NO mid-path matching —
# the string simply does not start with the pattern
"xmyfile.txt", # no suffix/partial matching either
],
)
def test_is_ignored_prefix_no_match(rel: str) -> None:
assert is_ignored(rel, ("myfile.txt",)) is False
def test_is_ignored_root_level_dir_is_prefix() -> None:
# A root-level dir IS prefix matching: "files" covers files/**.
assert is_ignored("files/x.md", ("files",)) is True
# …but not a same-named dir deeper down (no mid-path matching).
assert is_ignored("some/files/x.md", ("files",)) is False
# --- is_ignored: the raw-prefix A1 edge (documented, owner-confirmed) -------
def test_is_ignored_raw_prefix_no_component_boundary() -> None:
# A1: raw string startswith — "my/files" ALSO ignores "my/files2/x.md".
assert is_ignored("my/files2/x.md", ("my/files",)) is True
assert is_ignored("ab.md", ("a",)) is True
@pytest.mark.parametrize("rel", ["a.md", "some/a.md", "anything"])
def test_is_ignored_empty_prefixes_never_match(rel: str) -> None:
assert is_ignored(rel, ()) is False
# --- iter_importable_files: the walk-level skip ------------------------------
@pytest.fixture()
def tree(tmp_path: Path) -> Path:
root = tmp_path / "tree"
(root / "ignore" / "deep").mkdir(parents=True)
(root / "keep.md").write_text("keep\n", encoding="utf-8")
(root / "ignore" / "secret.md").write_text("secret\n", encoding="utf-8")
(root / "ignore" / "deep" / "x.yaml").write_text("x: 1\n", encoding="utf-8")
(root / "notes.txt").write_text("notes\n", encoding="utf-8")
(root / "myfiles.md").write_text("myfiles\n", encoding="utf-8")
return root
def _rels(paths: list[Path], root: Path) -> set[str]:
return {p.relative_to(root).as_posix() for p in paths}
ALL_FIVE = {
"keep.md",
"ignore/secret.md",
"ignore/deep/x.yaml",
"notes.txt",
"myfiles.md",
}
def test_iter_default_ignore_keeps_everything(tree: Path) -> None:
# Default ``ignore=()`` — every existing caller byte-identical.
assert _rels(iter_importable_files(tree, EXTS), tree) == ALL_FIVE
def test_iter_ignore_prefix_skips_matching_files(tree: Path) -> None:
got = _rels(iter_importable_files(tree, EXTS, ignore=("ignore",)), tree)
assert got == {"keep.md", "notes.txt", "myfiles.md"}
def test_iter_ignore_receives_already_normalized_tuples(tree: Path) -> None:
# Contract pin: normalization is the caller's job at THIS level — it
# happens in _ignore_for_root, not here. A raw box line does NOT
# match, so all five files still come back.
got = _rels(iter_importable_files(tree, EXTS, ignore=("//ignore/ ",)), tree)
assert got == ALL_FIVE
# --- _ignore_for_root: the single normalization choke point ------------------
def test_ignore_for_root_normalizes_and_drops_empties(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
got = _ignore_for_root(root, {str(root): ["/a/", "b//", "", " "]})
assert got == ("a", "b")
def test_ignore_for_root_missing_root_yields_empty(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _ignore_for_root(root, {"/other/root": ["a"]}) == ()
def test_ignore_for_root_none_map_yields_empty(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _ignore_for_root(root, None) == ()
tests/unit/test_models.py
+40 -1
View File
@@ -6,7 +6,8 @@ on; these tests lock the table/column contract (PLAN §5) without a live DB.
from __future__ import annotations from __future__ import annotations
from pgvector.sqlalchemy import Vector from pgvector.sqlalchemy import Vector
from sqlalchemy import UniqueConstraint from sqlalchemy import TextClause, UniqueConstraint
from sqlalchemy.sql.schema import DefaultClause
import app.models # noqa: F401 (import registers all tables on Base.metadata) import app.models # noqa: F401 (import registers all tables on Base.metadata)
from app.db import Base from app.db import Base
@@ -62,6 +63,44 @@ def test_doc_drafts_token_is_unique_not_null() -> None:
assert uq, "doc_drafts must be unique on (token) — the URL credential" assert uq, "doc_drafts must be unique on (token) — the URL credential"
def test_git_sources_ignore_paths_column_contract() -> None:
"""Phase 89: every source row carries its ignore list — JSONB, NOT
NULL, server default ``'[]'`` (a pre-phase-89 row reads ``[]``, so
every existing source imports exactly as before)."""
sources = Base.metadata.tables["git_sources"]
assert "ignore_paths" in sources.c, (
"git_sources must have the ignore_paths column (phase 89)"
)
col = sources.c["ignore_paths"]
assert col.nullable is False, "git_sources.ignore_paths must be NOT NULL"
sd = col.server_default
assert isinstance(sd, DefaultClause), "ignore_paths needs a server default"
assert isinstance(sd.arg, TextClause), (
"the server default must be the literal SQL text '[]'"
)
assert sd.arg.text == "'[]'", "ignore_paths server default must be '[]'"
def test_git_source_python_default_empty_list() -> None:
"""A freshly constructed row (no DB) resolves to an empty ignore
list via the Python-side default (``default=list``) — the ORM
INSERT-time default, so an ORM insert that omits the column inserts
``[]`` rather than NULL (the server default ``'[]'`` independently
covers non-ORM inserts)."""
from app.models import GitSource
row = GitSource(url="https://example.com/r.git", kind="git")
col = row.__table__.c["ignore_paths"]
assert col.default is not None, (
"ignore_paths needs a Python-side (INSERT-time) default"
)
# Invoked with the (unused) execution context at INSERT time — a
# freshly constructed row that omits the kwarg stores [] rather
# than NULL (the real-DB behaviour is pinned against the dev DB
# by the git-sources API integration tests).
assert col.default.arg(None) == [], "the default must resolve to []"
def test_doc_drafts_column_contract() -> None: def test_doc_drafts_column_contract() -> None:
"""Phase 59: the editable triple (title/path/body) + status + """Phase 59: the editable triple (title/path/body) + status +
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
tests/unit/test_source_ignore_paths.py
+546
View File
@@ -0,0 +1,546 @@
"""Unit: the per-source ignore-paths editor on the Sources page
(phase 89, task 05).
``TODO.md`` L3: "They should be able to type these files and folders
into a box on the sources page." Phase 89 gives every STORED source
row (git or local) a one-path-per-line ignore box on
``/git-sources.html``: a page-local alertdialog (the EXACT
``#remove-confirm-dialog`` pattern, phase 69) with a visible label, a
mono textarea prefilled from the row, the §7.4 never-stale save
lifecycle ("Saving…" while the ``PATCH /api/git-sources/{id}`` is
out — the A5 replace round-trips through ``GET``), and full a11y
(focus on Cancel, Escape / backdrop close as CANCEL, focus return,
``role="alert"`` error line that keeps the textarea content on 422).
Rows with a list show the ``N ignored`` count tag; env-fallback rows
(``id`` null) get NO box (A3 — nothing is stored to edit).
The browser behavior itself is E2E-gated by the phase-89 story suite
(``tests/e2e/test_source_ignore_paths.py``, task 06); like the other
frontend-adjacent unit files (the
``test_remove_confirm_modal.py`` house pattern), this module pins the
source-level contract a silent regression would break:
* the static ``#ignore-editor-dialog`` markup — ``role=
"alertdialog"`` + ``aria-modal`` + ``aria-labelledby``, hidden by
default, inside the manager, the visible ``<label for=…>`` (never
aria-label-only), the ``role="alert"`` error line, real
``type="button"`` buttons, the mono textarea (``rows=6`` /
``spellcheck="false"``);
* the JS lifecycle — ``openIgnoreEditor`` (textContent-only source
population with makeRow's ``value`` expression, the textarea
prefilled from ``(s.ignore_paths || [])``, error cleared, focus on
Cancel), cancel = Escape / Cancel button / backdrop (no request;
focus returns to the trigger; a no-op while a PATCH is in flight),
``saveIgnorePaths`` (§7.4 in-flight state: both buttons disable +
"Saving…", blank lines dropped client-side, success → close /
reload / announce — the update confirmation LAST, non-2xx → the
in-dialog alert line with the textarea content KEPT, network →
the fixed reachable? line, re-enable in the finally);
* the row wiring — the "Ignore paths" button exists ONLY in the
``s.id`` branch of ``makeRow`` (left of Remove; the aria-label is
the only place the value appears — never innerHTML), the ``N
ignored`` count tag on rows with a list (text, never color alone);
* styles.css — the dialog / button / count-tag rules on the house
dark-tech palette (phase-08 tokens only, no CDN, no blur),
≥44px targets, the ``[hidden]`` override, the mono full-width
textarea;
* the no-collision guard — every ``id="ignore-editor-…"`` appears
EXACTLY ONCE in the shell (the phase-46/76 contract).
"""
from __future__ import annotations
import re
from pathlib import Path
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
# Phase 76 (task 02): git-sources.html is folded into the ONE-document
# shell — the dialog markup lives in the Sources view section of
# index.html (next to #remove-confirm-dialog).
SHELL_HTML = FRONTEND / "index.html"
JS = FRONTEND / "assets" / "git-sources.js"
CSS = FRONTEND / "assets" / "styles.css"
#: The dialog's static ids (the no-collision guard + the markup pins
#: key on exactly these).
DIALOG_IDS = (
"ignore-editor-dialog",
"ignore-editor-title",
"ignore-editor-source",
"ignore-editor-copy",
"ignore-editor-textarea",
"ignore-editor-error",
"ignore-editor-cancel",
"ignore-editor-save",
)
#: The §7.4 in-flight label + the idle label (pinned verbatim).
SAVING_LABEL = "Saving…"
IDLE_LABEL = "Save"
COUNT_CLASS = "git-source-ignore-count"
ROW_BTN_CLASS = "git-source-ignore"
ARIA_LABEL_TEMPLATE = "`Edit ignored paths for ${kindLabel} source: ${value}`"
#: The success announce — the update confirmation is the LAST
#: announcement (the reload's "N sources listed." must not overwrite
#: it, the same order as the remove flow).
ANNOUNCE_OK = "Ignored paths updated for ${value}"
def _text(path: Path) -> str:
return path.read_text(encoding="utf-8")
def _js() -> str:
return _text(JS)
def _css() -> str:
return _text(CSS)
def _fn(js: str, name: str) -> str:
"""The source of a (possibly async) top-level function via
balanced-brace counting (the test_remove_confirm_modal.py helper)."""
for prefix in ("async function ", "function "):
start = js.find(f"{prefix}{name}(")
if start != -1:
depth = 0
for i in range(js.find("{", start), len(js)):
if js[i] == "{":
depth += 1
elif js[i] == "}":
depth -= 1
if depth == 0:
return js[start : i + 1]
raise AssertionError(f"unbalanced braces in {name}()")
raise AssertionError(f"{name}() must exist in git-sources.js")
def _element_block(html: str, id_attr: str, tag: str = "div") -> str:
"""The <tag … id=…> element's full markup (a balanced-tag walk —
the dialog nests the backdrop / panel / actions divs)."""
marker = f'id="{id_attr}"'
i = html.find(marker)
assert i != -1, f"missing id={id_attr} in the shell's Sources view"
opens = [m.start() for m in re.finditer(rf"<{tag}\b", html[:i])]
assert opens, f"no <{tag}> owns id={id_attr}"
open_i = opens[-1]
depth = 0
for m in re.finditer(rf"<{tag}\b[^>]*>|</{tag}>", html[open_i :]):
if m.group(0).startswith(f"</{tag}>"):
depth -= 1
else:
depth += 1
if depth == 0:
return html[open_i : open_i + m.end()]
raise AssertionError(f"unbalanced <{tag}> for id={id_attr}")
def _css_rule(css: str, selector: str) -> str:
"""The declarations of a simple one-line-opening rule (comments
stripped first — a house comment may legally carry braces)."""
clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
start = clean.find(f"{selector} {{")
assert start != -1, f"missing rule {selector} in styles.css"
depth = 0
for i in range(clean.find("{", start), len(clean)):
if clean[i] == "{":
depth += 1
elif clean[i] == "}":
depth -= 1
if depth == 0:
return clean[start : i + 1]
raise AssertionError(f"unbalanced braces in {selector}")
# ---------- the static dialog markup ----------
def test_dialog_markup_is_the_locked_alertdialog() -> None:
"""#ignore-editor-dialog: role="alertdialog" + aria-modal +
aria-labelledby, hidden by default, INSIDE the manager
(#git-sources-content, directly after #remove-confirm-dialog —
the static-markup convention, stable E2E selectors); every
static id present; the error line is role="alert" + hidden;
both buttons are real type="button"; the textarea is mono-
capable (rows=6, spellcheck=false, the A1 prefix placeholder)."""
html = _text(SHELL_HTML)
frag = _element_block(html, "ignore-editor-dialog")
open_tag = frag[: frag.find(">") + 1]
assert 'role="alertdialog"' in open_tag
assert 'aria-modal="true"' in open_tag
assert 'aria-labelledby="ignore-editor-title"' in open_tag
assert 'aria-describedby="ignore-editor-copy"' in open_tag
assert "hidden" in open_tag, "the dialog ships hidden"
# Placed in the manager, right after the remove-confirm dialog
# (the same placement logic).
assert html.find('id="git-sources-content"') < html.find('id="ignore-editor-dialog"')
assert html.find('id="remove-confirm-dialog"') < html.find('id="ignore-editor-dialog"')
for child in DIALOG_IDS:
assert f'id="{child}"' in frag, f"missing #{child} in the dialog"
# The visible label (WCAG — never aria-label-only): a real
# <label for="ignore-editor-textarea"> with the house label class.
label = re.search(
r"<label[^>]*class=\"ignore-editor-label\"[^>]*for=\"ignore-editor-textarea\"[^>]*>(.*?)</label>",
frag,
re.S,
)
assert label and label.group(1).strip(), "a visible block label for the textarea"
# The error line: role="alert", hidden by default.
err = re.search(r"<p[^>]*id=\"ignore-editor-error\"[^>]*>", frag)
assert err and 'role="alert"' in err.group(0) and "hidden" in err.group(0)
# The mono box: rows + spellcheck off + the A1 placeholder.
ta = re.search(r"<textarea[^>]*id=\"ignore-editor-textarea\"[^>]*>", frag, re.S)
assert ta, "#ignore-editor-textarea is a real <textarea>"
for attr in ('rows="6"', 'spellcheck="false"', 'placeholder="my/files/"'):
assert attr in ta.group(0), f"the textarea carries {attr}"
for btn in ("ignore-editor-cancel", "ignore-editor-save"):
m = re.search(rf"<button[^>]*id=\"{btn}\"[^>]*>", frag)
assert m and 'type="button"' in m.group(0), f"#{btn} is a real type=button"
cancel = re.search(r'<button[^>]*id="ignore-editor-cancel"[^>]*>(.*?)</button>', frag, re.S)
save = re.search(r'<button[^>]*id="ignore-editor-save"[^>]*>(.*?)</button>', frag, re.S)
assert cancel and cancel.group(1).strip() == "Cancel"
assert save and save.group(1).strip() == IDLE_LABEL
# The helper copy states the A1 prefix rule in plain words.
copy = re.search(r"<p[^>]*id=\"ignore-editor-copy\"[^>]*>(.*?)</p>", frag, re.S)
assert copy, "the helper copy paragraph"
copy_norm = re.sub(r"\s+", " ", copy.group(1))
assert "No wildcards" in copy_norm and "middle of a path" in copy_norm
def test_dialog_ids_appear_exactly_once() -> None:
"""The no-collision guard (the phase-46/76 contract): every
``id="ignore-editor-…"`` in the NEW markup appears EXACTLY ONCE
in the whole shell — no duplicated id (the dialog is the only
owner of each id)."""
html = _text(SHELL_HTML)
for id_ in DIALOG_IDS:
n = html.count(f'id="{id_}"')
assert n == 1, f'id={id_} appears {n} times in index.html (must be exactly 1)'
# ---------- the row wiring in makeRow ----------
def test_row_button_exists_only_in_the_stored_row_branch() -> None:
"""makeRow: the "Ignore paths" button is created ONLY in the
``s.id`` branch (A3 — env-fallback rows get the "from .env" tag,
no button) and sits BEFORE the Remove button (left of Remove).
The button is class git-source-ignore, type=button, textContent
label (never innerHTML — the value appears ONLY in the aria-
label, built via setAttribute); its click opens the editor."""
make = _fn(_js(), "makeRow")
branch_i = make.find("if (s.id) {")
else_i = make.find("} else {", branch_i)
bind_i = make.find("openIgnoreEditor(s, ignoreBtn)")
assert -1 < branch_i < bind_i < else_i, (
"the ignore button is created in the s.id branch (the phase-88 pin idiom)"
)
remove_i = make.find('btn.className = "git-source-remove"', bind_i)
assert remove_i > bind_i, "the ignore button is LEFT of Remove"
assert f'ignoreBtn.className = "{ROW_BTN_CLASS}"' in make
assert "ignoreBtn.type = \"button\"" in make
assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)"
assert "ignoreBtn.textContent = \"Ignore paths\"" in make, (
"a static text label — never icon-only"
)
assert "ignoreBtn.innerHTML" not in _js(), "XSS contract: no innerHTML on the button"
# The env-fallback branch (else) has no ignore button.
else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)]
assert ROW_BTN_CLASS not in else_slice and "openIgnoreEditor" not in else_slice, (
"env-fallback rows get no ignore button (A3)"
)
def test_count_tag_shows_n_ignored_text_next_to_the_location() -> None:
"""makeRow: a stored row with a non-empty list gets the
``N ignored`` count tag in the LOCATION cell (after the
<code>) — text via textContent (never color alone, WCAG 1.4.1);
the list reads ``(s.ignore_paths || [])`` — the same expression
openIgnoreEditor prefills from (the GET round-trip marker)."""
make = _fn(_js(), "makeRow")
assert "s.id && (s.ignore_paths || []).length > 0" in make, (
"the count tag only for stored rows with a list"
)
assert f'count.className = "{COUNT_CLASS}"' in make
assert "count.textContent = `${s.ignore_paths.length} ignored`" in make, (
"N ignored — text, never color alone"
)
# It lands in the location cell (urlTd), not the actions cell.
append_i = make.find("urlTd.append(count)")
code_i = make.find("urlTd.append(badge, code)")
assert -1 < code_i < append_i, "the tag is appended to the location cell"
# ---------- the JS lifecycle ----------
def test_open_prefills_and_focuses_cancel() -> None:
"""openIgnoreEditor(s, triggerBtn): the source value is
textContent ONLY (never innerHTML — the credential-masking
discipline, phase 32) with makeRow's exact ``value`` expression;
the textarea prefills ``(s.ignore_paths || []).join("\\n")``
(one path per line — the round-trip marker); the error line
clears; the dialog unhides; the trigger is recorded; the keydown
handler attaches; and focus lands on Cancel — the safe default
(AFTER the unhide)."""
body = _fn(_js(), "openIgnoreEditor")
assert "s.kind === \"local\"" in body, "the kind-typed value branch"
assert (
"ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url" in body
), "the same `value` expression makeRow uses, via textContent"
assert "ignoreSourceEl.innerHTML" not in _js(), "XSS contract: textContent only"
assert 'ignoreTextarea.value = (s.ignore_paths || []).join("\\n")' in body, (
"the textarea prefills the stored list, one path per line"
)
assert "ignoreErrorEl.hidden = true" in body, "a new attempt starts clean"
assert "ignoreTriggerBtn = triggerBtn" in body, "the trigger is recorded"
trigger_i = body.find("ignoreTriggerBtn = triggerBtn")
unhide_i = body.find("ignoreDialog.hidden = false")
attach_i = body.find('document.addEventListener("keydown", onIgnoreDialogKeydown)')
focus_i = body.find("ignoreCancelBtn.focus()")
assert -1 < trigger_i < unhide_i < attach_i < focus_i, (
"record the trigger → unhide → attach keydown → focus Cancel"
)
def test_cancel_paths_close_without_a_request() -> None:
"""Cancel (Cancel button / Escape / backdrop) closes as cancel:
the dialog hides, the textarea + error line reset, the buttons
reset ("Save"), the keydown handler detaches, and focus RETURNS
to the recorded trigger — and the cancel path never fetches. A
cancel while a PATCH is in flight is a no-op (no half-cancel of
an in-progress save)."""
js = _js()
cancel = _fn(js, "cancelIgnoreEditor")
guard_i = cancel.find("if (ignoreInFlight) return")
close_i = cancel.find("closeIgnoreEditor()")
assert -1 < guard_i < close_i, "the in-flight guard precedes the close"
assert "fetch" not in cancel, "cancel never sends a request"
close = _fn(js, "closeIgnoreEditor")
hide_i = close.find("ignoreDialog.hidden = true")
reset_ta_i = close.find('ignoreTextarea.value = ""')
clear_i = close.find("ignoreErrorEl.hidden = true")
reset_btn_i = close.find('"Save"')
detach_i = close.find('document.removeEventListener("keydown", onIgnoreDialogKeydown)')
save_i = close.find("const trigger = ignoreTriggerBtn")
null_i = close.find("ignoreTriggerBtn = null")
focus_i = close.find("trigger.focus()")
assert -1 < hide_i < reset_ta_i < clear_i < reset_btn_i, (
"hide → reset textarea → clear error → reset buttons"
)
assert reset_btn_i < detach_i < save_i < null_i < focus_i, (
"→ detach → save trigger → focus return"
)
assert "ignoreCancelBtn.disabled = false" in close
assert "ignoreSaveBtn.disabled = false" in close
def test_escape_and_backdrop_and_cancel_button_all_cancel() -> None:
"""While open (handler attached on document in openIgnoreEditor,
detached in closeIgnoreEditor): Escape → preventDefault +
cancelIgnoreEditor; the dim backdrop AND the Cancel button wire
to cancelIgnoreEditor (only Save wires to saveIgnorePaths)."""
js = _js()
keydown = _fn(js, "onIgnoreDialogKeydown")
esc_i = keydown.find('e.key === "Escape"')
prevent_i = keydown.find("e.preventDefault()", esc_i)
cancel_i = keydown.find("cancelIgnoreEditor()", esc_i)
assert -1 < esc_i < prevent_i < cancel_i, "Escape: prevent + cancel"
assert 'ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor)' in js
assert 'ignoreBackdrop.addEventListener("click", cancelIgnoreEditor)' in js
assert 'ignoreSaveBtn.addEventListener("click", saveIgnorePaths)' in js
def test_save_runs_the_inflight_never_stale_lifecycle() -> None:
"""saveIgnorePaths: the box's lines are split on newlines,
trimmed, and empty lines DROPPED (a blank line is a separator,
not an entry — the server still rejects empties defensively,
A4). The §7.4 in-flight state precedes the PATCH: both buttons
disable + the save relabels "Saving…" — one
``PATCH /api/git-sources/{id}`` with the lines as the whole
body list (A5 replace). 200 → close (focus return) →
loadSources (the count tag lands) → announce (the update
confirmation is the LAST announcement). Non-2xx: the in-dialog
role=alert line (apiDetail, 422 shape-aware), the dialog STAYS
open and the textarea content is KEPT. Network failure: the
fixed reachable? line. The finally re-enables BOTH buttons +
relabels "Save" — never stale on any outcome."""
body = _fn(_js(), "saveIgnorePaths")
guard_i = body.find("if (!ignoreTarget || ignoreInFlight) return")
parse_i = body.find('.split("\\n")')
trim_i = body.find(".map((l) => l.trim())", parse_i)
drop_i = body.find(".filter(Boolean)", trim_i)
inflight_i = body.find("ignoreInFlight = true", drop_i)
dis_c = body.find("ignoreCancelBtn.disabled = true", inflight_i)
dis_s = body.find("ignoreSaveBtn.disabled = true", inflight_i)
label_i = body.find(f'"{SAVING_LABEL}"', dis_s)
fetch_i = body.find("`/api/git-sources/${", label_i)
method_i = body.find('method: "PATCH"', fetch_i)
body_i = body.find("JSON.stringify({ ignore_paths: lines })", method_i)
assert -1 < guard_i < parse_i < trim_i < drop_i < inflight_i, (
"guard → split + trim + drop blank lines → in-flight"
)
assert -1 < dis_c < dis_s < label_i < fetch_i < method_i < body_i, (
"disable both + 'Saving…' → the PATCH with the lines"
)
# Success: close → reload → announce (the exact order) — the
# update confirmation is the LAST announcement: the reload's
# "N sources listed." must not overwrite it.
ok_i = body.find("if (r.ok)")
close_i = body.find("closeIgnoreEditor()", ok_i)
reload_i = body.find("await loadSources()", close_i)
announce_i = body.find(f"announce(`{ANNOUNCE_OK}`)", reload_i)
assert -1 < ok_i < close_i < reload_i < announce_i, (
"200: close → loadSources → announce (LAST)"
)
assert body.count("closeIgnoreEditor()") == 1, (
"only the success path closes — failures stay open for one retry"
)
# Non-2xx: the in-dialog alert line (apiDetail, 422 shape-aware);
# the dialog stays open and the textarea content is KEPT.
catch_i = body.find("} catch {")
nonok_slice = body[ok_i:catch_i]
assert "await apiDetail(r," in nonok_slice, (
"the server detail is apiDetail-extracted (422 shape-aware)"
)
assert "ignoreErrorEl.hidden = false" in nonok_slice, "the error line shows"
assert "ignoreTextarea.value" not in nonok_slice.split("if (r.ok)")[1], (
"the textarea content is KEPT on failure (the instruction survives)"
)
# Network: the fixed reachable? line.
net_i = body.find("Could not save the ignored paths — is the app reachable?", catch_i)
assert -1 < net_i < body.find("finally"), "the network copy lands in the catch"
# Never stale: the finally re-enables BOTH buttons + relabels.
fin = body[body.find("finally"):]
assert "ignoreInFlight = false" in fin
assert "ignoreCancelBtn.disabled = false" in fin
assert "ignoreSaveBtn.disabled = false" in fin
assert f'ignoreSaveBtn.textContent = "{IDLE_LABEL}"' in fin
def test_module_docstring_carries_the_phase_89_contract() -> None:
"""The git-sources.js module docstring gained the phase-89
bullet: the editor is per-STORED-row (A3 env rows excluded),
one path per line (the A1 prefix rule), the §7.4 "Saving…"
lifecycle, and the A5 replace round-trip."""
doc = _js().split("*/", 2)[0] # the module docstring (first block)
for frag in (
"Phase 89 (task 05)",
"ignore-paths editor",
"#ignore-editor-dialog",
"one path per line",
f'"{SAVING_LABEL}"',
"LAST announcement",
"N ignored",
):
assert frag in doc, f"the module docstring lost: {frag!r}"
# ---------- styles.css ----------
def test_dialog_css_rules_present_and_house_tokens_only() -> None:
"""styles.css carries the ignore-editor class family on the house
dark-tech palette (phase-08 tokens): the overlay + backdrop +
panel (the EXACT .remove-confirm treatment), the title / source /
copy / visible label / mono textarea / error / actions / two-
button chrome; the [hidden] override; no blur (the phase-08
no-blur perf anchor); no CDN."""
css = _css()
for selector in (
".ignore-editor",
".ignore-editor-backdrop",
".ignore-editor-panel",
".ignore-editor-title",
".ignore-editor-source",
".ignore-editor-copy",
".ignore-editor-label",
".ignore-editor-textarea",
".ignore-editor-error",
".ignore-editor-actions",
".ignore-editor-btn",
".ignore-editor-cancel",
".ignore-editor-save",
):
assert f"{selector} " in css or f"{selector}." in css or f"{selector}[" in css, (
f"styles.css must style {selector}"
)
hidden = css.find(".ignore-editor[hidden]")
assert hidden != -1 and "display: none" in css[hidden : hidden + 60], (
"the hidden attr must beat the display rule"
)
# No blur (the phase-08 no-blur perf anchor) — the backdrop rule
# itself must not carry backdrop-filter (comments stripped).
backdrop = re.sub(r"/\*.*?\*/", "", _css_rule(css, ".ignore-editor-backdrop"), flags=re.S)
assert "backdrop-filter" not in backdrop, "no blur (phase-08 perf anchor)"
assert "url(http" not in css and "@import url(" not in css, (
"no CDN (AGENTS.md rule 6)"
)
def test_textarea_rule_is_mono_and_full_width() -> None:
"""The box itself: full panel width (width: 100%) + the mono
stack (var(--mono)) + a comfortable min-height — the house
comment cites phase 89 on the block."""
rule = _css_rule(_css(), ".ignore-editor-textarea")
assert "width: 100%" in rule, "full panel width"
assert "var(--mono)" in rule, "the mono stack"
assert "min-height" in rule, "a comfortable minimum height"
# The section comment cites phase 89 (house comment style).
css = _css()
header = css[css.rfind("/*", 0, css.find(".ignore-editor {")) : css.find(".ignore-editor {")]
assert "phase 89" in header, "the house comment cites phase 89"
def test_dialog_button_and_target_contrast_pairs() -> None:
"""The WCAG 2.1 AA basics in CSS: both dialog buttons >=44px;
Cancel is the ghost ink-soft family (5.1:1 on --surface) with the
brand-soft hover (12.4:1); Save is the solid brand family (--bg
text on --brand 5.2:1, the .new-chat-btn convention) with the
lightened hover; the error line is the err pair; the panel caps
at the 46rem chat-column width or the viewport; the visible
label is ink-soft (5.1:1) — never a label-less textarea."""
css = _css()
btn = _css_rule(css, ".ignore-editor-btn")
assert "min-height: 44px" in btn and "min-width: 44px" in btn
cancel = _css_rule(css, ".ignore-editor-cancel")
assert "var(--ink-soft)" in cancel and "transparent" in cancel
cancel_hover = _css_rule(css, ".ignore-editor-cancel:hover:not(:disabled)")
assert "var(--brand-soft)" in cancel_hover and "var(--brand-ink)" in cancel_hover
save = _css_rule(css, ".ignore-editor-save")
assert "var(--brand)" in save and "var(--bg)" in save, (
"Save: the solid brand family (--bg text on --brand, 5.2:1)"
)
save_hover = _css_rule(css, ".ignore-editor-save:hover:not(:disabled)")
assert "background" in save_hover, "the hover lightens the fill"
err = _css_rule(css, ".ignore-editor-error")
assert "var(--err-ink)" in err and "var(--err-bg)" in err
panel = _css_rule(css, ".ignore-editor-panel")
assert "min(46rem" in panel, "the 46rem chat-column cap (or the viewport)"
label = _css_rule(css, ".ignore-editor-label")
assert "display: block" in label and "var(--ink-soft)" in label, (
"a visible block label (WCAG — never aria-label-only)"
)
def test_row_button_and_count_tag_css_rules() -> None:
"""The row chrome: .git-source-ignore reuses the
.git-source-remove idiom (>=44px target, same size/spacing) in a
NEUTRAL secondary fill with the brand-soft hover (distinct from
the destructive Remove's err hover); .git-source-ignore-count is
a small inline tag — text + a distinct background (never color
alone), AA on both theme surfaces (--ink on --bg 16.7:1)."""
css = _css()
row = _css_rule(css, ".git-source-ignore")
assert "min-height: 44px" in row and "min-width: 44px" in row, "the >=44px rule"
assert "var(--ink-soft)" in row, "the neutral secondary resting fill"
row_hover = _css_rule(css, ".git-source-ignore:hover:not(:disabled)")
assert "var(--brand-soft)" in row_hover and "var(--brand-ink)" in row_hover, (
"the hover takes the brand pair (Remove hovers to the err pair)"
)
assert "var(--err-" not in row, "not the destructive err family"
count = _css_rule(css, ".git-source-ignore-count")
assert "var(--ink)" in count and "var(--bg)" in count, (
"text + a distinct background (never color alone)"
)
tests/unit/test_sync_button.py
+1
View File
@@ -727,6 +727,7 @@ class _GatedImport:
limit: int | None = None, limit: int | None = None,
session: object = None, session: object = None,
progress: Callable[[str, str, int, int], None] | None = None, progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
) -> ImportSummary: ) -> ImportSummary:
self.prune_flags.append(prune) self.prune_flags.append(prune)
if progress is not None: if progress is not None: