Compare commits
2
Commits
1f0e4c6bb9
...
8c706259e9
| Author | SHA1 | Date | |
|---|---|---|---|
 ducoterra
|
8c706259e9 | ||
|
ducoterra
|
0495e4e7e4 |
@@ -0,0 +1,72 @@
|
|||||||
|
# Phase 89 — Per-source ignore paths: exclude files/folders from import
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "The user should be able to ignore/exclude specific files and folders when importing from a source. They should be able to type these files and folders into a box on the sources page. Each source should support a list of ignored files and folders. These ignored files/folders should not be embedded and should not be summarized. The format of the ignored files/folders should be a path, so \"/my/files/\" or \"my/files/\" or \"my/files\" would ignore all files that start with the prefix \"my/files\". There shouldn't be any support for \"match in the middle\" of a path, so \"myfile.txt\" would match \"/myfile.txt\" but not \"some/path/myfile.txt\". The ignore is just a prefix ignore."
|
||||||
|
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-08).
|
||||||
|
**Context:** Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources`). Every import entry point funnels through `import_sources` (`app/rag/importer.py`): the in-app Sync button (`app/api/sync.py::_run_sync`), the archive-upload background scan (`app/api/git_sources.py::_run_upload`), and the CLI (`scripts/import_docs.py`). `iter_importable_files` is the single walk choke point (extension filter + hidden-dir rule + `EXCLUDED_DIRS`); `import_sources` also runs a pre-walk with the same rules for the phase-64 progress `total`; ignored files must be excluded from BOTH. Non-markdown files get a `lite` summary in `_store_summary` (phase 30) — a file that never enters the walk is never embedded and never summarized, by construction. Pruning (`_prune`, `prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — so files that newly match an ignore pattern leave the index on the next sync, exactly like the A9 out-of-scope-junk behavior. The Sources page is admin-gated (phase 16 pattern); env-fallback rows (`from_env: true`, `id: null`) have no DB row to store a list on.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Each stored source carries an **ignore list** of source-relative path prefixes, edited in a per-row box on the Sources page; matching is pure prefix (no mid-path matching, no globs — the spec's examples verbatim); ignored files are never walked, embedded, or summarized, and previously indexed files that newly match an ignore pattern are pruned on the next sync. All three import entry points honor the lists; the API is admin-only like the rest of the router.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `88_mobile_chat_hamburger_boot` (complete) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/upload/CLI pipelines, and the Sources view — none of which phase 88's pins reach; its suites must stay green unchanged).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **Match semantics (locked, A1).** One pure helper pair in `app/rag/importer.py`:
|
||||||
|
- `normalize_ignore_path(entry: str) -> str` — `entry.strip().strip("/")` (whitespace + all leading/trailing slashes — so `"/my/files/"`, `"my/files/"`, and `"my/files"` normalize to the same `"my/files"`; `"//"` and `""` normalize to `""` and are dropped).
|
||||||
|
- `is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool` — `any(rel.startswith(p) for p in prefixes)` where `rel` is the **source-relative POSIX path without a leading slash** (`path.relative_to(root).as_posix()` — the same string `documents.path` stores) and `prefixes` are the normalized, non-empty entries.
|
||||||
|
- **Raw string prefix — deliberately no component-boundary check** (A1, owner-confirmed): `"my/files"` also matches `"my/files2/x.md"`; `"myfile.txt"` matches `"myfile.txt"` (root-level file) but NOT `"some/path/myfile.txt"` (no mid-path matching — the string simply does not start with the pattern). No globs, no `**`, no case folding beyond what the walk already does for extensions.
|
||||||
|
- The importer normalizes its input itself (single choke point — callers may pass raw box lines): an internal `_ignore_for_root(root, ignore_by_root)` yields `tuple(p for p in (normalize_ignore_path(x) for x in raw) if p)`.
|
||||||
|
- **`iter_importable_files` / `import_sources` signature (task 02).**
|
||||||
|
- `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore: tuple[str, ...] = ())` — one more skip: `if ignore and is_ignored(rel.as_posix(), ignore): continue` (after the hidden/excluded check, before/after the extension check — either order is fine; keep the existing checks byte-identical). Default `()` keeps every existing caller (CLI manual `--source`, tests) behaving exactly as before.
|
||||||
|
- `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root: dict[str, list[str]] | None = None)` — the map is keyed by **`str(root)`** (the root path string exactly as passed in `sources` — unambiguous even when two rows resolve to the same source *name*; on a root-string collision the caller has already merged the lists, Design "Callers" below). The pre-walk that computes the phase-64 progress `total` and the processing loop MUST use the same per-root ignore tuple (same helper), so `files_total` never counts ignored files. `seen` is untouched in shape → `_prune` prunes newly-ignored documents automatically (A2).
|
||||||
|
- **Callers (task 04).**
|
||||||
|
- `app/api/sync.py::_run_sync` — while building `sources` per row, build `ignore_by_root: dict[str, list[str]]` in the same loop: git row → key `str(sources_root / repo_name(row.url))`, local row → key `str(Path(row.path or row.url).expanduser())`; value `row.ignore_paths` — **extend on key collision** (two rows sharing a root string get the union of their lists). Pass `ignore_by_root=…` to `import_sources`.
|
||||||
|
- `app/api/git_sources.py::_run_upload` — step 6 already fetches the row by path (existing or the newly created one); capture it and pass `ignore_by_root={str(final_dir): row.ignore_paths}` to the step-8 `import_sources` (a re-upload of an existing source respects the list the owner already saved).
|
||||||
|
- `scripts/import_docs.py` — `_resolve_sources` returns `tuple[list[Path], dict[str, list[str]]]` (manual `--source` → `({},)` empty map — manual dirs have no row; env-fallback rows have no lists). `main` unpacks and passes the map to `import_sources`; docstrings updated (module + `_resolve_sources`).
|
||||||
|
- **Storage (task 01).** `git_sources.ignore_paths` — JSONB NOT NULL, server default `'[]'`, `Mapped[list]` (the `saved_chats.messages` JSONB precedent, `app/models.py`). Alembic `0013_git_source_ignore_paths.py` (revises `0012`) — `op.add_column("git_sources", sa.Column("ignore_paths", postgresql.JSONB(astext_type=sa.Text()), server_default=sa.text("'[]'"), nullable=False))`; downgrade drops the column. Existing rows read `[]`.
|
||||||
|
- **API contract (task 03).**
|
||||||
|
- Schemas (`app/schemas.py`): `GitSourceIn.ignore_paths: list[str] | None = None` (create-time, optional — absent → `[]`); `GitSourceOut.ignore_paths: list[str]`; `GitSourceRow.ignore_paths: list[str]` (env-fallback rows report `[]`); new `GitSourceIgnoreIn` body model for PATCH with `ignore_paths: list[str]` (required — the box replaces the whole list, A5).
|
||||||
|
- `GET /api/git-sources` — each DB row reports its (normalized, non-empty) list; env rows `[]`.
|
||||||
|
- `POST /api/git-sources` — both kinds accept `ignore_paths`; stored **normalized** (each entry `normalize_ignore_path`, empties dropped) — the UI may send raw box lines and the stored list is canonical.
|
||||||
|
- `PATCH /api/git-sources/{source_id}` (NEW, behind the existing `require_admin` router dependency) — 404 unknown id; **replace** semantics (A5): the body list, normalized, becomes the row's list (empty list = clear all); 200 → `GitSourceOut` (id, url, added_at, ignore_paths).
|
||||||
|
- Validation (422, fixed details that never echo the input — the router's credential-safety discipline, applied for consistency): at most **200 entries** (A4) → `detail="a source has at most 200 ignore paths"`; an entry **empty after normalization** → `detail="ignore paths must be non-empty"`; an entry **>500 chars** (pre- or post-normalization — check post-normalization) → `detail="an ignore path exceeds 500 characters"`.
|
||||||
|
- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: an **"Ignore paths" button** (actions cell, left of Remove — same `.git-source-remove`-style button idiom, new class `.git-source-ignore`) whose `aria-label` is `Edit ignored paths for source: <value>`; when `s.ignore_paths.length > 0` the row also shows a text count tag `N ignored` (text, never color alone) next to the location cell. Env-fallback rows (`id` null) get **no** button — the existing "from .env" tag stays (A3).
|
||||||
|
- The button opens a **page-local editor dialog** following the phase-69 `#remove-confirm-dialog` pattern exactly (new `#ignore-editor-dialog`, `role="dialog"`, dim backdrop, focus management: focus lands on Cancel — the safe default — Escape / Cancel / backdrop close as CANCEL and return focus to the trigger; the dim backdrop click handler is the same idiom). Inside: a visible `<label for="ignore-editor-textarea">` ("Ignored files and folders — one path per line"), a `<textarea id="ignore-editor-textarea">` (mono, ~6 rows, `spellcheck="false"`, placeholder `my/files/`), a helper line explaining the prefix rule in plain words ("A file is ignored when its path in the source starts with any of these. No wildcards."), Save + Cancel buttons, and a `role="alert"` error line (server 422 details, same rendering as the remove modal's error line).
|
||||||
|
- `frontend/assets/git-sources.js` wiring (all scoped to the view root like everything else in the module): `openIgnoreEditor(s, triggerBtn)` — prefills `s.ignore_paths.join("\n")`; `saveIgnorePaths()` — the **§7.4 never-stale lifecycle**: Save disables + relabels "Saving…" while the PATCH is out, re-enables on success AND failure; on 200 → close the dialog, `loadSources()`, `announce("Ignored paths updated for <value>")`; on non-2xx → the server detail in the `role="alert"` line, textarea content KEPT (the instruction survives, tuning-form convention). Lines are parsed client-side: split on `\n`, trim, **drop empty lines** (a blank line in the box is a separator, not an entry) before submitting.
|
||||||
|
- `frontend/index.html` — the dialog skeleton inside `#view-git-sources` (next to `#remove-confirm-dialog`); `frontend/assets/styles.css` — dialog/textarea/button/count-tag rules reusing the remove-confirm-dialog and `.git-source-remove` idioms (house comment style, citing phase 89).
|
||||||
|
- **NOT touched:** the RAG view (`sources.js` — the catalog stays as-is; the Sync button already exists there), `app/rag/retriever.py`, the chunker, `mock_llm.py`/`slow_llm.py`, chat/SSE, `AGENTS.md`, any completed phase.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_ignore_paths_column.md` — `git_sources.ignore_paths` JSONB column (model + alembic `0013`) + unit tests.
|
||||||
|
2. `02_ignore_prefix_matching.md` — the pure matcher + `iter_importable_files`/`import_sources` `ignore_by_root` support (walk + progress pre-walk + prune interaction) + unit & integration tests.
|
||||||
|
3. `03_ignore_paths_api.md` — schemas + `GET`/`POST` fields + new admin-only `PATCH /api/git-sources/{id}` (replace semantics, 404/422 contracts) + integration tests.
|
||||||
|
4. `04_ignore_paths_in_import_pipelines.md` — wire the per-row lists into `_run_sync`, the `_run_upload` background scan, and `scripts/import_docs.py` + integration tests.
|
||||||
|
5. `05_ignore_paths_sources_ui.md` — the per-row "Ignore paths" box on the Sources page (dialog + §7.4 save lifecycle + a11y) + source-level unit pins.
|
||||||
|
6. `06_e2e_ignore_paths.md` — dedicated Playwright suite `tests/e2e/test_source_ignore_paths.py` (run in isolation), regressions, full gate, atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_importer_ignore.py` (new, tasks 01–02): normalization table, the spec's prefix examples verbatim, no-mid-path, raw-prefix A1 edge, empty/drop rules, `iter_importable_files` on a tmp fixture tree; `tests/unit/test_source_ignore_paths.py` (new, task 05): source-level pins for the JS/HTML wiring (house pattern — read the assets as text).
|
||||||
|
- Integration — `tests/integration/test_importer_ignore.py` (new, task 02): `import_sources` against a fixture dir — ignored files produce no `Document`/`Chunk` rows (not embedded, not summarized), previously indexed file newly ignored → pruned, progress `total` excludes ignored files, sources not in the map behave exactly as before; `tests/integration/test_git_sources_api.py` (extended, task 03) and `tests/integration/test_sync_api.py` + the CLI/upload suites (extended, task 04).
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_source_ignore_paths.py` (task 06), run in isolation with the DB up: `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov`.
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate — the importer/API additions are fully unit+integration covered).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A stored source's ignore list (git or local row) is editable in the box on the Sources page: one path per line; Save → `PATCH` 200 → the row shows the `N ignored` count; the list round-trips through `GET /api/git-sources`.
|
||||||
|
- [ ] A sync (button, upload scan, or CLI) with `ignore_paths: ["ignore/"]` (any of the three spellings) indexes nothing under `ignore/` — no `documents`/`chunks` rows, no embeddings, no summary calls for those files — and a previously indexed file under `ignore/` is pruned from the KB on that sync (the RAG catalog no longer lists it).
|
||||||
|
- [ ] The spec's matching examples hold end-to-end: `myfile.txt` ignores the root-level `myfile.txt` only, never `some/path/myfile.txt` (E2E proof).
|
||||||
|
- [ ] `PATCH` contracts hold: 404 unknown id; 422 fixed details for >200 entries / empty entry / >500-char entry; replace semantics (clearing works); anonymous 403.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` green in isolation (DB up); regression suites `test_git_sources_admin.py`, `test_archive_upload_sources.py`, `test_sync_button.py`, `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — raw prefix semantics (owner-confirmed 2026-09-08).** Normalize = trim whitespace + strip leading/trailing `/`; match = `rel.startswith(normalized)` on the source-relative POSIX path. No component boundary (`"my/files"` also ignores `"my/files2/x.md"`), no globs, no mid-path matching.
|
||||||
|
- **A2 — previously indexed files leave the index (owner-confirmed).** A file that newly matches an ignore pattern is pruned from the KB on the next sync (the `seen`-set mechanism, same as A9 out-of-scope junk) — "not embedded, not summarized" also cleans the existing index, not just future imports.
|
||||||
|
- **A3 — box placement (owner-confirmed).** The editor is a per-row control on the Sources page (`/git-sources.html`) for stored rows only; env-fallback rows have no DB row and get no editor.
|
||||||
|
- **A4 — limits (owner-confirmed).** At most 200 entries per source, each ≤500 chars after normalization; entries empty after normalization are rejected with a fixed 422 detail (the UI drops blank lines client-side; the API stays defensive).
|
||||||
|
- **A5 — replace semantics (owner-confirmed).** Saving the box replaces the whole list (textarea content = the list; empty box clears).
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/versions/0013_git_source_ignore_paths.py frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source ignore paths — prefix-excluded files are never embedded, summarized, or re-indexed"
|
||||||
|
```
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Task 01 — `git_sources.ignore_paths` column (model + migration 0013)
|
||||||
|
|
||||||
|
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "Each source should support a list of ignored files and folders."
|
||||||
|
**Story:** n/a (TODO-derived).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Give every `git_sources` row a JSONB `ignore_paths` list (default empty) so the Sources page can store one ignore list per source — the storage half of the feature; no importer or API behavior changes in this task.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — `GitSource` (between `SourcesMeta` and `DocDraft`): add the column after `path`, with a docstring comment in the house style (cite phase 89 + the A1 normalization):
|
||||||
|
```python
|
||||||
|
#: Per-source ignore paths (phase 89, A1/A4): the normalized,
|
||||||
|
#: non-empty source-relative path prefixes the owner types into the
|
||||||
|
#: box on the Sources page. A file is ignored when its
|
||||||
|
#: source-relative POSIX path starts with any entry (raw prefix —
|
||||||
|
#: no mid-path matching, no globs). Server default '[]' — every
|
||||||
|
#: pre-phase-89 row imports exactly as before.
|
||||||
|
ignore_paths: Mapped[list] = mapped_column(
|
||||||
|
JSONB, default=list, server_default=text("'[]'"), nullable=False
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(`app/models.py` imports named symbols from `sqlalchemy` — add `text` to that existing import list; `JSONB` is already imported from `sqlalchemy.dialects.postgresql`.) Update the module docstring's `git_sources` bullet with one sentence: "``ignore_paths`` (phase 89 — JSONB list of normalized path prefixes, server default ``'[]'``)".
|
||||||
|
2. `alembic/versions/0013_git_source_ignore_paths.py` (NEW — house header style, mirror `0007_git_sources_kind.py`'s docstring shape: Revision ID `0013`, `Revises: 0012`, one-sentence rationale per column):
|
||||||
|
```python
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"git_sources",
|
||||||
|
sa.Column(
|
||||||
|
"ignore_paths",
|
||||||
|
postgresql.JSONB(astext_type=sa.Text()),
|
||||||
|
server_default=sa.text("'[]'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
op.drop_column("git_sources", "ignore_paths")
|
||||||
|
```
|
||||||
|
3. `tests/unit/test_models_git_source.py` (extend the existing model unit tests if one exists — check `tests/unit/` for a models test module first; otherwise create it) — or a focused new module `tests/unit/test_ignore_paths_column.py`:
|
||||||
|
- the `GitSource.__table__.c.ignore_paths` column exists, is NOT NULL, and its server default is `'[]'` (assert `col.server_default.arg.text == "'[]'"` and `col.nullable is False`);
|
||||||
|
- a freshly constructed `GitSource(url="https://example.com/r.git", kind="git")` has `ignore_paths == []` (the Python-side default) — the ORM default path, no DB needed.
|
||||||
|
4. Apply the migration to the dev DB so later tasks run against the real schema: `uv run alembic upgrade head` (record the head revision in the task report; the E2E/integration suites use the same DB via `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/unit/ -q` green (the new pins + the untouched suite).
|
||||||
|
- Coverage: **>90%** on `app/` unaffected (model column + migration are config, not logic).
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run alembic upgrade head` applies `0013` cleanly; `alembic downgrade -1 && alembic upgrade head` round-trips (reversible — the 0007 precedent).
|
||||||
|
- [ ] Existing `git_sources` rows (dev DB) read `ignore_paths == []` — no importer/API change yet: `uv run pytest tests/integration/test_git_sources_api.py -q` green UNCHANGED.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `git diff --stat` limited to `app/models.py`, the new migration, and the unit test.
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Task 02 — Prefix matching + `ignore_by_root` in the importer
|
||||||
|
|
||||||
|
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "These ignored files/folders should not be embedded and should not be summarized… \"/my/files/\" or \"my/files/\" or \"my/files\" would ignore all files that start with the prefix \"my/files\"… \"myfile.txt\" would match \"/myfile.txt\" but not \"some/path/myfile.txt\". The ignore is just a prefix ignore."
|
||||||
|
**Story:** n/a (TODO-derived).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make the import walk honor per-source ignore lists: ignored files are never indexed (hence never embedded, never summarized), the phase-64 progress `total` never counts them, and previously indexed files that newly match an ignore pattern are pruned on the next `prune=True` run (A2) — while every caller that passes no map behaves byte-identically to today.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — add the pure helpers (module level, near `iter_importable_files`; docstrings in the house style, each citing the spec example it encodes):
|
||||||
|
```python
|
||||||
|
def normalize_ignore_path(entry: str) -> str:
|
||||||
|
"""One ignore-path entry → canonical form (phase 89, A1).
|
||||||
|
|
||||||
|
Trim surrounding whitespace, then strip ALL leading/trailing
|
||||||
|
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
|
||||||
|
all become ``"my/files"``. ``""`` / ``"//"``,
|
||||||
|
``" "`` normalize to ``""`` (callers drop empties).
|
||||||
|
"""
|
||||||
|
return entry.strip().strip("/")
|
||||||
|
|
||||||
|
|
||||||
|
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
|
||||||
|
"""Phase 89, A1 — the pure prefix rule, nothing else.
|
||||||
|
|
||||||
|
``rel`` is the source-relative POSIX path WITHOUT a leading
|
||||||
|
slash (the ``documents.path`` string). Match = ``rel`` STARTS
|
||||||
|
WITH a normalized entry: raw string prefix — deliberately NO
|
||||||
|
component-boundary check (``"my/files"`` also matches
|
||||||
|
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
|
||||||
|
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
|
||||||
|
"""
|
||||||
|
return any(rel.startswith(p) for p in prefixes)
|
||||||
|
|
||||||
|
|
||||||
|
def _ignore_for_root(
|
||||||
|
root: Path, ignore_by_root: dict[str, list[str]] | None
|
||||||
|
) -> tuple[str, ...]:
|
||||||
|
"""The normalized, non-empty prefix tuple for one root (phase 89).
|
||||||
|
|
||||||
|
Keyed by ``str(root)`` — the root string exactly as the caller
|
||||||
|
passed it in ``sources`` (unambiguous when two rows share a
|
||||||
|
source *name* but different dirs). Callers may pass RAW box
|
||||||
|
lines: the importer normalizes + drops empties here, the single
|
||||||
|
choke point — stored lists (already normalized) normalize to
|
||||||
|
themselves.
|
||||||
|
"""
|
||||||
|
raw = (ignore_by_root or {}).get(str(root)) or []
|
||||||
|
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
|
||||||
|
```
|
||||||
|
2. `app/rag/importer.py` — `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore: tuple[str, ...] = ())`: inside the `for path in sorted(root.rglob("*"))` loop, after the hidden/`excluded`-parts check and before/after the extension check (your choice — keep both existing checks byte-identical), add:
|
||||||
|
```python
|
||||||
|
if ignore and is_ignored(rel.as_posix(), ignore):
|
||||||
|
continue
|
||||||
|
```
|
||||||
|
Update the docstring: one sentence on the `ignore` param (phase 89, A1; default `()` = no behavior change).
|
||||||
|
3. `app/rag/importer.py` — `import_sources(…, ignore_by_root: dict[str, list[str]] | None = None)` (new keyword-only param, default `None`):
|
||||||
|
- compute `ignore_by_root` once per root inside the `for root in sources:` loop: `ignore = _ignore_for_root(root, ignore_by_root)`;
|
||||||
|
- the phase-64 pre-walk (`if progress is not None:` block) must use it — `iter_importable_files(root, llm.settings.import_extension_set, ignore=ignore_for_this_root)` (compute the tuple in the pre-walk too, or hoist: the pre-walk runs before the loop, so call `_ignore_for_root` there as well);
|
||||||
|
- the processing loop passes the same tuple to `iter_importable_files`;
|
||||||
|
- `seen`, `_prune`, and everything else are UNCHANGED — newly-ignored files simply never enter `seen`, so the next `prune=True` run deletes their rows (A2, the A9 junk-precedent);
|
||||||
|
- docstring: one paragraph — the map is keyed by `str(root)`, the pre-walk and the loop use the same tuple (so `files_total` excludes ignored files), and pruning follows automatically.
|
||||||
|
4. `tests/unit/test_importer_ignore.py` (NEW — the spec is the test table; import the three helpers from `app.rag.importer`):
|
||||||
|
- **normalize:** `"/my/files/"`, `"my/files/"`, `"my/files"`, `" my/files "` all → `"my/files"`; `"//"` → `""`; `""` → `""`; `" "` → `""`.
|
||||||
|
- **is_ignored (spec examples verbatim):** prefixes `("my/files",)` ignore `my/files/a.md`, `my/files/sub/b.md`, and `my/files` (a file literally named that); prefixes `("myfile.txt",)` ignore `myfile.txt` but NOT `some/path/myfile.txt` (no mid-path matching — the string does not start with the pattern) and NOT `xmyfile.txt` (no suffix/partial matching either). Also `("files",)` ignores `files/x.md` (a root-level dir — that IS prefix matching) but not `some/files/x.md`.
|
||||||
|
- **raw-prefix A1 edge (documented, owner-confirmed):** `("my/files",)` also ignores `my/files2/x.md`; `("a",)` ignores `ab.md` (raw string startswith — no component boundary).
|
||||||
|
- **empty prefixes:** `is_ignored(anything, ())` → False.
|
||||||
|
- **iter_importable_files:** a tmp_path fixture tree (`root/keep.md`, `root/ignore/secret.md`, `root/ignore/deep/x.yaml`, `root/notes.txt`, `root/myfiles.md`) with `extensions=frozenset({".md", ".txt", ".yaml"})`: default `ignore=()` → all five; `ignore=("ignore",)` → only `keep.md`, `notes.txt`, `myfiles.md`; `ignore=("//ignore/ ")` → same (normalization is the caller's job at this level — this function receives ALREADY-normalized tuples; pin that contract: passing raw `"//ignore/ "` does NOT match, proving normalization happens in `_ignore_for_root`, not here).
|
||||||
|
- **_ignore_for_root:** `{"str(root)": ["/a/", "b//", "", " "]} ` → `("a", "b")`; missing root → `()`; `None` map → `()`.
|
||||||
|
5. `tests/integration/test_importer_ignore.py` (NEW — mirror the fixture-tree + mock-LLM pattern of `tests/integration/test_importer_e2e.py`; use the session-scoped test DB and a `tmp_path` source dir named `IgnoreFix`):
|
||||||
|
- **never indexed:** tree `keep.md` + `ignore/secret.md` + `ignore/notes.txt` + `top.txt`; run `import_sources([root], fake_llm, ignore_by_root={str(root): ["ignore/"]})` → `ImportSummary.files == 2`; `documents` has exactly `keep.md` + `top.txt`; NO `chunks` row for `ignore/secret.md` (not embedded) and its `summary` column is NULL (not summarized); the fake LLM's `chat` call count (summary) excludes it.
|
||||||
|
- **prune (A2):** re-run with `prune=True` and a file previously indexed — pre-import `ignore/secret.md` WITHOUT the map (so it is indexed), then re-run WITH `ignore_by_root={str(root): ["ignore"]}` (no trailing slash — A1 normalization) → `summary.pruned == 1` and the doc row is gone from the DB.
|
||||||
|
- **progress pre-walk excludes ignored:** pass a `progress` hook + the map → `total` == count of non-ignored files only (2 in the first scenario).
|
||||||
|
- **no-map regression:** the same tree, `ignore_by_root=None` → all four files indexed (byte-identical behavior), `summary.files == 4`.
|
||||||
|
- **unlisted source unaffected:** two roots, map keys only the first → the second root imports everything.
|
||||||
|
- Clean up `documents` rows of the fixture source between tests (the house cleanup pattern in `test_importer_e2e.py`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -v` green.
|
||||||
|
- `uv run pytest -q` green UNCHANGED (every existing importer/sync/upload/CLI test passes no map — the default `()`/`None` contract).
|
||||||
|
- Coverage: **>90%** on `app/` (the helpers are exercised by the unit table; the importer branches by the integration suite).
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The spec's own examples pass as unit tests verbatim (`/my/files/` ≡ `my/files/` ≡ `my/files`; `myfile.txt` matches root-level only, never mid-path).
|
||||||
|
- [ ] An ignored file yields no `documents` row, no `chunks` rows, no embedding call, and no summary call; a previously indexed file matching a new pattern is pruned on the next `prune=True` run.
|
||||||
|
- [ ] `files_total` (progress hook) never counts ignored files.
|
||||||
|
- [ ] `git diff --stat` limited to `app/rag/importer.py` + the two new test modules.
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# Task 03 — API: expose + update ignore lists (GET / POST / PATCH)
|
||||||
|
|
||||||
|
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "They should be able to type these files and folders into a box on the sources page. Each source should support a list of ignored files and folders."
|
||||||
|
**Story:** n/a (TODO-derived).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The admin-only sources API carries the ignore list: `GET` reports it per row, `POST` stores it at create time, and a new `PATCH /api/git-sources/{source_id}` replaces it (A5) — with the fixed-detail 422 validation of the A4 limits. No importer behavior changes here (the pipelines are wired in task 04).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py` — extend the git-source schemas (house docstrings, cite phase 89):
|
||||||
|
- `GitSourceIn`: `ignore_paths: list[str] | None = Field(default=None)` — optional at create time (absent → `[]`); entries are the RAW box lines (trimmed/normalized by the API layer, not the schema — the 422 details must stay fixed strings, the router's credential-safety discipline applied for consistency).
|
||||||
|
- `GitSourceOut`: `ignore_paths: list[str]` (the stored, normalized list — non-null).
|
||||||
|
- `GitSourceRow`: `ignore_paths: list[str]` (env-fallback rows report `[]`).
|
||||||
|
- NEW `GitSourceIgnoreIn(BaseModel)`: `ignore_paths: list[str]` — the `PATCH` body; the list is REQUIRED (replace semantics, A5 — an absent field is a FastAPI 422 with the model's own detail; keep the A4 422s for the *value* checks).
|
||||||
|
2. `app/api/git_sources.py`:
|
||||||
|
- a module-level validation helper (private, near `_commit_new`), shared by POST and PATCH:
|
||||||
|
```python
|
||||||
|
def _validate_ignore_paths(raw: list[str] | None) -> list[str]:
|
||||||
|
"""Normalize + enforce the A4 limits; fixed details never echo input."""
|
||||||
|
entries = [normalize_ignore_path(e) for e in (raw or [])]
|
||||||
|
if any(not e for e in entries):
|
||||||
|
raise HTTPException(status_code=422, detail="ignore paths must be non-empty")
|
||||||
|
if len(entries) > MAX_IGNORE_PATHS: # = 200 (A4)
|
||||||
|
raise HTTPException(status_code=422, detail="a source has at most 200 ignore paths")
|
||||||
|
if any(len(e) > 500 for e in entries):
|
||||||
|
raise HTTPException(status_code=422, detail="an ignore path exceeds 500 characters")
|
||||||
|
return entries
|
||||||
|
```
|
||||||
|
(Import `normalize_ignore_path` from `app.rag.importer` — task 02 owns the helper; define `MAX_IGNORE_PATHS = 200` and the per-entry cap `500` as module constants with an A4 comment. NOTE the order: empty check FIRST — a whitespace-only entry must 422, not silently drop.)
|
||||||
|
- `list_git_sources` (GET): DB rows → `GitSourceRow(…, ignore_paths=row.ignore_paths or [])`; env-fallback rows → `ignore_paths=[]`.
|
||||||
|
- `_create_git_row` / `_create_local_row` (POST): `row = GitSource(…, ignore_paths=_validate_ignore_paths(payload.ignore_paths))` (both kinds — the stored value is always the normalized list).
|
||||||
|
- `GitSourceOut` responses in `create_git_source` gain `ignore_paths=row.ignore_paths`.
|
||||||
|
- NEW route (place it after the POST, before `/upload` — pathless params keep the `/upload` and `/upload/status` routes matching their own literal prefixes first; FastAPI matches in declaration order, so the `PATCH /{source_id}` declaration must NOT shadow anything — PATCH and POST/GET are different methods, and DELETE `/{source_id}` already coexists, so a plain `@router.patch("/{source_id}", response_model=GitSourceOut)` is safe):
|
||||||
|
```python
|
||||||
|
@router.patch("/{source_id}", response_model=GitSourceOut)
|
||||||
|
def patch_git_source(
|
||||||
|
source_id: uuid.UUID,
|
||||||
|
payload: GitSourceIgnoreIn,
|
||||||
|
db: Session = Depends(get_db), # noqa: B008
|
||||||
|
) -> GitSourceOut:
|
||||||
|
"""Replace one source's ignore list (phase 89, A5).
|
||||||
|
|
||||||
|
404 unknown id. The body list (required) is normalized +
|
||||||
|
A4-validated (fixed 422 details) and REPLACES the row's list
|
||||||
|
wholesale — an empty list clears all. Returns the updated
|
||||||
|
row's public shape (id, url, added_at, ignore_paths).
|
||||||
|
"""
|
||||||
|
row = db.get(GitSource, source_id)
|
||||||
|
if row is None:
|
||||||
|
raise HTTPException(status_code=404, detail="git source not found")
|
||||||
|
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(row)
|
||||||
|
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at,
|
||||||
|
ignore_paths=row.ignore_paths)
|
||||||
|
```
|
||||||
|
- Router module docstring: add the PATCH route to the "Routes:" paragraph (house style — the existing paragraph already lists every route).
|
||||||
|
3. `tests/integration/test_git_sources_api.py` — extend (follow the existing test module's fixture/auth pattern — admin vs anonymous, DB truncation between tests):
|
||||||
|
- GET: a row created via POST without `ignore_paths` reports `ignore_paths == []`; env-fallback rows (empty table + `BOR_GIT_SOURCES` env) report `[]` per row and `from_env: true`.
|
||||||
|
- POST: `ignore_paths: ["/my/files/", " my/files2 ", "x"]` → 201, stored list `["my/files", "my/files2", "x"]` (normalized round-trip through GET); both kinds (git + local) accept it.
|
||||||
|
- PATCH: 200 replace semantics — set `["a/", "b"]` → GET shows `["a", "b"]`; then PATCH `[]` → GET shows `[]` (clear works); the row's `id`/`url`/`added_at` are unchanged; response shape is `GitSourceOut` incl. the new list.
|
||||||
|
- PATCH 404 on an unknown uuid; anonymous PATCH → 403 (the router dependency).
|
||||||
|
- PATCH 422s (fixed details, asserted as exact strings): `[" "]` → "ignore paths must be non-empty"; 201 entries → "a source has at most 200 ignore paths"; `["a" * 501]` → "an ignore path exceeds 500 characters"; 500-char entry is ACCEPTED (boundary).
|
||||||
|
- regression: every pre-existing test in the module passes UNCHANGED (the new fields are additive on the response models).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/integration/test_git_sources_api.py -v` green (new + untouched tests).
|
||||||
|
- `uv run pytest -q` green (no pipeline wiring yet — the lists are stored but not yet consulted by imports).
|
||||||
|
- Coverage: **>90%** on `app/` (every new branch — 404/422/replace — is a tested path).
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/git-sources` reports `ignore_paths` on every row (DB rows: their list; env rows: `[]`).
|
||||||
|
- [ ] `POST` stores the normalized list for both kinds; `PATCH /api/git-sources/{id}` replaces it (including clearing to `[]`), 404/403/422 contracts hold with the exact fixed details above.
|
||||||
|
- [ ] `git diff --stat` limited to `app/schemas.py`, `app/api/git_sources.py`, and the integration test module.
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# Task 04 — Wire per-row ignore lists into all three import entry points
|
||||||
|
|
||||||
|
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "The user should be able to ignore/exclude specific files and folders when importing from a source… These ignored files/folders should not be embedded and should not be summarized."
|
||||||
|
**Story:** n/a (TODO-derived).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every import entry point consults its source rows' ignore lists: the in-app Sync button, the archive-upload background scan, and the CLI. Rows with empty lists and manual `--source` dirs behave exactly as before.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/sync.py::_run_sync` — while the per-row loop builds `sources` (the `for row in rows:` block that appends `clone_or_pull(…)` for git rows and the re-verified dir for local rows), build the map in the SAME loop, per the phase overview's Design/Callers:
|
||||||
|
```python
|
||||||
|
ignore_by_root: dict[str, list[str]] = {}
|
||||||
|
…
|
||||||
|
for row in rows:
|
||||||
|
if row.kind == "git":
|
||||||
|
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
|
||||||
|
else:
|
||||||
|
root = Path(row.path or row.url).expanduser()
|
||||||
|
if not root.is_dir():
|
||||||
|
raise GitSyncError(f"local source missing: {root}")
|
||||||
|
sources.append(root)
|
||||||
|
# phase 89: the row's ignore list, keyed by the SAME root string
|
||||||
|
# the importer sees; two rows sharing a root string get the
|
||||||
|
# union (extend, not replace) — the sibling/repo-name edge.
|
||||||
|
if row.ignore_paths:
|
||||||
|
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
|
||||||
|
```
|
||||||
|
(Refactor the existing loop body to the `root = …` shape above — behavior byte-identical; the existing re-verify-`.is_dir()` raise and its message are kept EXACTLY.) Then: `summary = await import_sources(sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root)`. Update the module docstring's pipeline step 4 with a clause: "honoring each row's `ignore_paths` (phase 89)".
|
||||||
|
2. `app/api/git_sources.py::_run_upload` — step 6 already opens a short-lived session and upserts the row by path; CAPTURE the row object on both branches (existing row from the `db.scalar(…)`; the newly created `GitSource` instance before `db.close()`) and keep it out of the session (the attributes are plain values — copy `row.ignore_paths or []` into a local `ignore_paths: list[str]`). Step 8 becomes:
|
||||||
|
```python
|
||||||
|
summary = await import_sources(
|
||||||
|
[final_dir], llm, prune=True, progress=_hook,
|
||||||
|
ignore_by_root={str(final_dir): ignore_paths},
|
||||||
|
)
|
||||||
|
```
|
||||||
|
Update the route's numbered docstring (steps 6/8) with one clause each (phase 89: a re-upload of an existing source honors the ignore list the owner already saved).
|
||||||
|
3. `scripts/import_docs.py` — `_resolve_sources` returns `tuple[list[Path], dict[str, list[str]]]`:
|
||||||
|
- manual `--source` branch → `([expanded paths], {})` (manual dirs have no row — no ignore applies);
|
||||||
|
- rows branch: same `root = …` per-row shape as `_run_sync`, building `ignore_by_root` with the same `setdefault(…).extend(…)` collision rule; return `(sources, ignore_by_root)`;
|
||||||
|
- legacy `DEFAULT_SOURCES` fallback → `([…], {})`;
|
||||||
|
- module + `_resolve_sources` docstrings: one sentence each (phase 89 — the map is keyed by the resolved root string; manual/fallback paths import with no ignore).
|
||||||
|
- `main`: `sources, ignore_by_root = _resolve_sources(args.source, settings)` and `import_sources(sources, llm, prune=args.prune, limit=args.limit, ignore_by_root=ignore_by_root)` (inside the `_run` closure — the variable is captured like `args` already is).
|
||||||
|
4. `tests/integration/test_sync_api.py` — extend (follow the module's existing local-source + mock-LLM fixture pattern; check how it seeds a `kind='local'` row and drives `POST /api/sync` + status polling):
|
||||||
|
- a local fixture dir with `keep.md` + `ignore/secret.md`; seed a `kind='local'` row with `ignore_paths=["ignore/"]`; run the sync to `success`; `documents` contains `keep.md`, NOT `ignore/secret.md`; the success `detail` counts are consistent (`files` excludes the ignored file).
|
||||||
|
- regression: the module's existing tests pass UNCHANGED (rows with `[]` lists → byte-identical sync).
|
||||||
|
5. `tests/integration/test_import_docs_git.py` (or `test_importer_e2e.py` if the CLI driver helper lives there — check which module already drives `import_docs.main` and extend THAT) — a rows-branch run with a `kind='local'` row carrying `ignore_paths` → the CLI import skips the ignored file (summary/files assertions); a `--source` manual run is unaffected (no map).
|
||||||
|
6. `tests/integration/test_git_sources_upload.py` — extend with the re-upload-respects-lists case: upload an archive (the module's existing fixture pattern) whose unpacked tree includes an ignored path; after the first scan, set the row's `ignore_paths` (PATCH or direct DB update — use the API), re-upload a same-name archive containing the file again → the background run lands `success` with the file NOT indexed; a second assertion: an upload of a NEW source name with no list imports everything (regression).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py tests/integration/test_git_sources_upload.py -v` green (new + existing).
|
||||||
|
- `uv run pytest -q` green; coverage **>90%** on `app/` (the new branches — map build, collision extend, upload capture — are covered by the integration tests; the CLI path by its test).
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `POST /api/sync` (button), the upload background scan, and `python -m scripts.import_docs` all skip the listed prefixes for their rows; nothing changes for empty lists or manual dirs.
|
||||||
|
- [ ] A re-uploaded existing source honors the ignore list saved on its row.
|
||||||
|
- [ ] `git diff --stat` limited to `app/api/sync.py`, `app/api/git_sources.py`, `scripts/import_docs.py`, and the extended integration modules.
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# Task 05 — The per-row "Ignore paths" box on the Sources page
|
||||||
|
|
||||||
|
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 — "They should be able to type these files and folders into a box on the sources page."
|
||||||
|
**Story:** n/a (TODO-derived).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A per-row "Ignore paths" control on the Sources page (`/git-sources.html`): a page-local editor dialog (the phase-69 `#remove-confirm-dialog` pattern) with a one-path-per-line textarea, the §7.4 never-stale save lifecycle, and full a11y (visible label, focus management, Escape, role=alert errors) — plus a `N ignored` count tag on rows that have a list.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — inside `#view-git-sources` → `#git-sources-content`, directly AFTER the `#remove-confirm-dialog` block (the markup above — same placement logic), add the editor dialog (house comment header citing phase 89; ids/classes are the ones the JS + CSS + unit pins below key on):
|
||||||
|
```html
|
||||||
|
<!-- Phase 89: the per-source ignore-paths editor — the EXACT
|
||||||
|
#remove-confirm-dialog pattern (phase 69): page-local
|
||||||
|
alertdialog, dim backdrop, focus on Cancel (the safe default),
|
||||||
|
Escape/backdrop close as CANCEL, focus returns to the row's
|
||||||
|
trigger button. The box itself: one path per line (prefix
|
||||||
|
semantics — the phase-89 A1 rule, stated in plain words in
|
||||||
|
the helper line); Save runs the §7.4 never-stale lifecycle
|
||||||
|
(git-sources.js: "Saving…" while the PATCH is out). -->
|
||||||
|
<div class="ignore-editor" id="ignore-editor-dialog" role="alertdialog"
|
||||||
|
aria-modal="true" aria-labelledby="ignore-editor-title"
|
||||||
|
aria-describedby="ignore-editor-copy" hidden>
|
||||||
|
<div class="ignore-editor-backdrop" aria-hidden="true"></div>
|
||||||
|
<div class="ignore-editor-panel">
|
||||||
|
<h2 class="ignore-editor-title" id="ignore-editor-title">Ignored files and folders</h2>
|
||||||
|
<code class="ignore-editor-source" id="ignore-editor-source"></code>
|
||||||
|
<p class="ignore-editor-copy" id="ignore-editor-copy">
|
||||||
|
One path per line. A file is ignored when its path in the
|
||||||
|
source starts with any of these — <code>/my/files/</code>,
|
||||||
|
<code>my/files/</code> and <code>my/files</code> mean the same
|
||||||
|
thing. No wildcards, no matching in the middle of a path.
|
||||||
|
</p>
|
||||||
|
<label class="ignore-editor-label" for="ignore-editor-textarea">Ignored paths</label>
|
||||||
|
<textarea class="ignore-editor-textarea" id="ignore-editor-textarea"
|
||||||
|
rows="6" spellcheck="false"
|
||||||
|
placeholder="my/files/"></textarea>
|
||||||
|
<p class="ignore-editor-error" id="ignore-editor-error" role="alert" hidden></p>
|
||||||
|
<div class="ignore-editor-actions">
|
||||||
|
<button type="button" class="ignore-editor-btn ignore-editor-cancel"
|
||||||
|
id="ignore-editor-cancel">Cancel</button>
|
||||||
|
<button type="button" class="ignore-editor-btn ignore-editor-save"
|
||||||
|
id="ignore-editor-save">Save</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
```
|
||||||
|
2. `frontend/assets/git-sources.js` — the wiring (every lookup scoped to the view `root` like the rest of the module; the module header docstring gains a phase-89 bullet in the house style):
|
||||||
|
- module-scope refs next to the remove-dialog refs: `ignoreDialog`, `ignoreSourceEl` (`#ignore-editor-source`), `ignoreTextarea` (`#ignore-editor-textarea`), `ignoreErrorEl` (`#ignore-editor-error`), `ignoreSaveBtn` (`#ignore-editor-save`), `ignoreCancelBtn` (`#ignore-editor-cancel`), `ignoreBackdrop` (`.ignore-editor-backdrop`); plus `let ignoreTarget = null` (the row object) and `let ignoreTriggerBtn = null` (for focus return).
|
||||||
|
- `makeRow(s)` — in the `s.id` branch of the actions cell, BEFORE the Remove button: an "Ignore paths" button (`type="button"`, class `git-source-ignore`, `aria-label` = `` `Edit ignored paths for ${kindLabel} source: ${value}` ``, built with `textContent`-safe content — the label is the only place `value` appears; follow the Remove button's `innerHTML` icon+text idiom ONLY if you reuse a static icon constant — never inject `value` via innerHTML). Its click → `openIgnoreEditor(s, btn)`. When `s.id && s.ignore_paths.length > 0`: append a count tag to the LOCATION cell (`urlTd`), after the `<code>`: `<span class="git-source-ignore-count">N ignored</span>` (text via textContent; N = `s.ignore_paths.length` — text + a distinct background, never color alone, WCAG).
|
||||||
|
- `openIgnoreEditor(s, triggerBtn)`: set `ignoreTarget`/`ignoreTriggerBtn`; `ignoreSourceEl.textContent = value` (the same value expression as makeRow); `ignoreTextarea.value = (s.ignore_paths || []).join("\n")`; clear the error line (`hidden` + empty text); show the dialog (`hidden = false`); focus `ignoreCancelBtn` (the safe default — the remove-dialog precedent); bind Escape via a keydown listener on the dialog (see the remove dialog's `onRemoveDialogKeydown` shape: Escape → `closeIgnoreEditor()`, return focus to the trigger).
|
||||||
|
- `closeIgnoreEditor()`: hide the dialog, reset the textarea value + error, `ignoreTarget = ignoreTriggerBtn = null`, return focus to the trigger button.
|
||||||
|
- Cancel button + backdrop click → `closeIgnoreEditor()` (the remove-dialog wiring shape).
|
||||||
|
- `saveIgnorePaths()` — the §7.4 lifecycle:
|
||||||
|
- parse: `ignoreTextarea.value.split("\n").map(l => l.trim()).filter(Boolean)` (a blank line is a separator, not an entry — the server still rejects empty entries defensively);
|
||||||
|
- disable Save + Cancel, relabel Save "Saving…";
|
||||||
|
- `fetch(`/api/git-sources/${ignoreTarget.id}`, {method: "PATCH", headers: {"Content-Type": "application/json"}, body: JSON.stringify({ignore_paths: lines})})` with the module's existing cookie/auth behavior (plain fetch — same as the add/remove calls);
|
||||||
|
- 200 → `closeIgnoreEditor()` (focus return included), `await loadSources()` (the `N ignored` tag lands), `announce(`Ignored paths updated for ${value}`)` (the existing `#git-sources-announcer`);
|
||||||
|
- non-2xx → clear Save/Cancel disabled states, restore the "Save" label, show the server detail in `#ignore-editor-error` (via the existing `apiDetail(r, fallback)` helper — 422 shape-aware like the tuning forms), KEEP the textarea content (the instruction survives);
|
||||||
|
- network throw → the same error-line path (the module's existing try/catch idiom for the other requests).
|
||||||
|
- env-fallback rows (`s.id` null): NO ignore button (A3) — the existing "from .env" tag stays as-is.
|
||||||
|
3. `frontend/assets/styles.css` — the ignore-editor rules (place next to the `.remove-confirm` rules; house comment style citing phase 89): `.ignore-editor` mirrors `.remove-confirm` (fixed overlay, dim backdrop `.ignore-editor-backdrop` = the `.remove-confirm-backdrop` treatment, centered panel `.ignore-editor-panel` = `.remove-confirm-panel` sizing); `.ignore-editor-source` = the `.remove-confirm-source` mono treatment; `.ignore-editor-label` a visible block label (WCAG — never `aria-label`-only); `.ignore-editor-textarea` — full panel width, `min-height: ~9rem`, `font-family: mono stack` (the existing code/mono variable), theme-consistent border + focus-visible ring (3px, the house focus convention); `.ignore-editor-error` = `.remove-confirm-error` treatment; `.ignore-editor-btn` pair = `.remove-confirm-btn` (Cancel secondary / Save brand — 44px min targets, the ≥44px rule); the row button `.git-source-ignore` = the `.git-source-remove` idiom (same size/spacing, a neutral/secondary fill so it reads distinct from the destructive Remove — text label included, never icon-only); `.git-source-ignore-count` — a small inline tag next to the location `<code>` (text + distinct background, 4.5:1 contrast on both theme surfaces).
|
||||||
|
4. `tests/unit/test_source_ignore_paths.py` (NEW — house source-level pattern: read the assets as text via `Path(__file__).parents[2] / "frontend" / "assets" / …`; module docstring cites TODO.md L3 + phase 89):
|
||||||
|
- `index.html`: `#ignore-editor-dialog` exists with `role="alertdialog"`, `aria-modal="true"`, `aria-labelledby="ignore-editor-title"`; `#ignore-editor-textarea` has a visible `<label … for="ignore-editor-textarea">` (the `ignore-editor-label` class); the error line carries `role="alert"`.
|
||||||
|
- `git-sources.js`: the PATCH call exists (`method: "PATCH"` + `` `/api/git-sources/${` ``); the §7.4 lifecycle markers exist (the "Saving…" relabel + disabling of the save button — assert the string `"Saving…"` appears in the save function's region); the parse chain `split("\n")` + trim + `filter(Boolean)`; `ignore_paths` round-trip markers (the GET rows render `(s.ignore_paths || [])` and the count tag class `git-source-ignore-count`); the env-row guard — the ignore button is created only in the `s.id` branch (assert the `openIgnoreEditor` binding sits after the `if (s.id)` — a string-position check, the phase-88 pin idiom).
|
||||||
|
- `styles.css`: the `.ignore-editor-textarea` rule exists (mono + width), and `git-source-ignore` / `git-source-ignore-count` rules exist.
|
||||||
|
- guard pin (the phase-46/76 no-collision contract): no id in the new markup duplicates an existing id — assert each `id="ignore-editor-…"` appears exactly once in `index.html`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/unit/test_source_ignore_paths.py -v` green; full `uv run pytest tests/unit/ -q` green UNCHANGED (no other suite reads `git-sources.js` — verify with `rg "git-sources" tests/unit` first; the `header.js`/`styles.css` pins must survive the additive CSS).
|
||||||
|
- Coverage: **>90%** on `app/` unaffected (frontend-only task).
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- Manual smoke (the executor's own eyes, headless is fine): boot the app, add a local source, open the box, type `my/files/`, save → the tag appears; refresh the page → the box prefills.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Every stored row (git or local) shows "Ignore paths"; env-fallback rows do not. Rows with a list show `N ignored`.
|
||||||
|
- [ ] The dialog: visible label, mono textarea prefilled from the row, Escape/Cancel/backdrop close with focus returned to the trigger, Save runs §7.4 ("Saving…" while the PATCH is out; error line keeps the content on 422).
|
||||||
|
- [ ] `git diff --stat` limited to `frontend/index.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, and the new unit test module.
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# Task 06 — E2E story suite, full gate, and the atomic commit
|
||||||
|
|
||||||
|
**Phase:** `89_source_ignore_paths` · **Source:** `TODO.md` L3 (the whole item — this task is its end-to-end proof).
|
||||||
|
**Story:** n/a (TODO-derived).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove the feature end-to-end through the real page + real API + real sync pipeline (mock LLM, no git, no network — the `test_git_sources_admin.py` policy): the box on the Sources page sets a list, the sync honors it (nothing embedded/summarized), previously indexed files are pruned (A2), and the spec's no-mid-path rule holds. Then run the complete gate and land ONE atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_source_ignore_paths.py` (NEW story suite — conftest app-server + mock-LLM + admin-login pattern from `tests/e2e/test_git_sources_admin.py` / `tests/e2e/auth_helpers.py`; module docstring: story source `TODO.md` L3, isolation run command, the no-git/no-network policy, the contract-under-test list, the test→contract mapping):
|
||||||
|
- **Fixtures:** a module-scoped source dir (`tmp_path_factory` — same reasoning as `test_sync_button.py`'s module-scoped fixture note) with this tree: `keep.md` (a `# Keep` doc with a sentinel word), `notes.yaml` (non-markdown → exercises the summary path), `myfile.md` (root level), `sub/myfile.md`, `ignore/secret.md`, `ignore/deep/x.txt`. A helper that (re)seeds the DB row: truncate `git_sources`/`documents` between tests (the house `db_ready` truncate pattern, the `test_sync_button.py` module-env note shows the DSN), then `POST /api/git-sources {"kind": "local", "path": <dir>}` (201) and capture the row id. A `run_sync()` helper: `POST /api/sync` → poll `GET /api/sync/status` to a terminal state (the `test_sync_button.py` polling idiom, ~2 s ticks, 60 s budget).
|
||||||
|
- `test_anonymous_gate_and_403s` — anonymous visit to `/git-sources.html`: the sign-in gate visible, the manager hidden, NO `/api/git-sources` request on load (route listener), and the page's request client gets 403 on `GET/POST /api/git-sources` AND `PATCH /api/git-sources/<uuid>` (the `test_git_sources_admin.py` anonymous pin).
|
||||||
|
- `test_ignore_box_excludes_from_import` — admin: seed the row (no list); open the box on the row (the "Ignore paths" button), type `ignore/` (single line), Save → wait for 200 (the `N ignored` tag: "1 ignored" appears on the row); `run_sync()` → `success`; `GET /api/docs` (the RAG catalog's data source — or read the RAG table after a nav to `/sources.html`, whichever the existing suites use) contains `keep.md`, `notes.yaml`, `myfile.md`, `sub/myfile.md` and contains NO path starting with `ignore/`; the sync `detail` `files` count == 5 (the two `ignore/` files excluded from the walk).
|
||||||
|
- `test_prefix_rule_no_mid_path` — admin: fresh seed; set the list to `myfile.md` via the box; `run_sync()` → the catalog contains `sub/myfile.md` (NOT ignored — no mid-path matching, the spec's own example) and NOT `myfile.md`; the `ignore/` files ARE indexed this time (no list matches them — regression guard that the box replaced, not appended, the previous test's list).
|
||||||
|
- `test_newly_ignored_file_is_pruned` (A2) — admin: fresh seed; `run_sync()` once with NO list (all six files indexed — assert 6 in the catalog); then set the box to `ignore/`, Save, `run_sync()` again → `success`, `detail.pruned == 2`, the catalog has the four remaining paths and no `ignore/` path.
|
||||||
|
- `test_editor_a11y_and_error_path` — admin: open the box → the dialog `role="alertdialog"` is visible, the visible label ("Ignored paths") is present, focus is on Cancel; Escape closes and focus returns to the trigger button (the `aria-label` matches the row); re-open, type an entry >500 chars (501 a's), Save → the `role="alert"` error line shows `an ignore path exceeds 500 characters`, the textarea content is KEPT, the save button is re-enabled with its "Save" label; then clear it, type `keep.md`, Save → "1 ignored" tag (the happy path heals the error state).
|
||||||
|
- `test_env_fallback_rows_have_no_box` — admin: truncate `git_sources` (the app booted with a `BOR_GIT_SOURCES` env URL, the `test_git_sources_admin.py` module-env pattern) → the env row renders with the "from .env" tag and NO "Ignore paths" button; `#git-sources-env-note` visible (A3).
|
||||||
|
2. **Gate (all must pass; any failure → fix within this task, re-run):**
|
||||||
|
- `uv run pytest` (unit + integration, green).
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on `app/` (the new importer + API logic is covered by tasks 02–04's suites; verify the TOTAL line, not just the new files).
|
||||||
|
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` (the NEW story — in isolation, DB up: `podman compose up -d db`).
|
||||||
|
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` (the existing Sources-page story — in isolation).
|
||||||
|
- `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` (the upload pipeline — in isolation).
|
||||||
|
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` (the sync button story — in isolation).
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (in isolation).
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
3. **Diff hygiene:** `git diff --stat` must be limited to: `app/models.py`, `app/schemas.py`, `app/api/git_sources.py`, `app/api/sync.py`, `app/rag/importer.py`, `alembic/versions/0013_git_source_ignore_paths.py` (new), `scripts/import_docs.py`, `frontend/index.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, `tests/unit/test_ignore_paths_column.py` (or the extended models module), `tests/unit/test_importer_ignore.py` (new), `tests/unit/test_source_ignore_paths.py` (new), `tests/integration/test_importer_ignore.py` (new), the extended integration modules (`test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py`, `test_git_sources_upload.py`), `tests/e2e/test_source_ignore_paths.py` (new), `TODO.md` (cleared — Phase 4 of the conversion), and the `89_source_ignore_paths/` phase files. Anything else → stop and fix.
|
||||||
|
4. **Commit (one atomic, Conventional Commits, `--no-gpg-sign` per AGENTS.md rule 8):**
|
||||||
|
```
|
||||||
|
feat(sources): per-source ignore paths — prefix-excluded files are never embedded, summarized, or re-indexed
|
||||||
|
|
||||||
|
TODO.md L3: the owner can type ignored files/folders into a box on
|
||||||
|
the Sources page; each source stores a list of source-relative path
|
||||||
|
prefixes, and matching is a pure prefix (no mid-path matching, no
|
||||||
|
globs — "/my/files/" ≡ "my/files/"; "myfile.txt" never matches
|
||||||
|
"some/path/myfile.txt").
|
||||||
|
|
||||||
|
- git_sources.ignore_paths: JSONB NOT NULL, server default '[]'
|
||||||
|
(migration 0013, reversible);
|
||||||
|
- importer: normalize_ignore_path / is_ignored / iter_importable_files
|
||||||
|
ignore filter / import_sources ignore_by_root (walk AND the
|
||||||
|
progress pre-walk; previously indexed files that newly match are
|
||||||
|
pruned on the next prune=True sync, the A9 junk precedent);
|
||||||
|
- API: GET/POST carry the list; new admin-only
|
||||||
|
PATCH /api/git-sources/{id} (replace semantics; fixed 422 details
|
||||||
|
for >200 entries, empty entry, >500-char entry);
|
||||||
|
- pipelines: _run_sync, the _run_upload background scan (a re-upload
|
||||||
|
honors the saved list), and scripts/import_docs.py all pass the
|
||||||
|
per-row lists;
|
||||||
|
- Sources page: per-row "Ignore paths" editor (the phase-69
|
||||||
|
dialog pattern, one path per line, §7.4 save lifecycle, a11y) +
|
||||||
|
the "N ignored" row tag; env-fallback rows have no box.
|
||||||
|
|
||||||
|
Unit: tests/unit/test_ignore_paths_column.py,
|
||||||
|
tests/unit/test_importer_ignore.py,
|
||||||
|
tests/unit/test_source_ignore_paths.py.
|
||||||
|
Integration: tests/integration/test_importer_ignore.py + extended
|
||||||
|
git-sources/sync/upload/CLI suites.
|
||||||
|
E2E story: tests/e2e/test_source_ignore_paths.py (box → sync
|
||||||
|
exclusion, no-mid-path rule, prune of a newly ignored file, a11y +
|
||||||
|
422 path, env rows, anonymous gate).
|
||||||
|
Phase 89: .agents/phases/todo/89_source_ignore_paths/.
|
||||||
|
```
|
||||||
|
Stage ONLY the step-3 file set (the phase directory rides with the commit, AGENTS.md rule 8; `TODO.md` is cleared to the bare `# TODO` title as part of this commit — the conversion's Phase 4).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The full gate of step 2 IS this task's test contract; coverage **>90%** on `app/`.
|
||||||
|
- This task adds the E2E file only — everything else verifies and commits.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All five E2E tests pass in isolation (DB up); the four regression E2E suites pass in isolation; unit + integration green; TOTAL coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] `git diff --stat` (pre-commit) matches the step-3 file set exactly.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` commit landed (message per step 4); `git log -1 --stat` confirms the file set.
|
||||||
|
- [ ] `TODO.md` contains exactly `# TODO` (cleared, never deleted).
|
||||||
|
- [ ] The phase directory is under `.agents/phases/todo/89_source_ignore_paths/` (the pipeline gate moves it to `complete/` after validate.sh passes).
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.
|
||||||
|
|
||||||
|
## Phase 89 — final verification pass: ALL GREEN
|
||||||
|
|
||||||
|
**Verified (all 6 task files present in `complete/`):**
|
||||||
|
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
|
||||||
|
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
|
||||||
|
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
|
||||||
|
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
|
||||||
|
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box
|
||||||
|
|
||||||
|
**Test/lint results:**
|
||||||
|
- `uv run pytest` → 1808 passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
|
||||||
|
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
|
||||||
|
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
|
||||||
|
**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.
|
||||||
|
|
||||||
|
**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).
|
||||||
|
|
||||||
|
**Next pending phase:** none — `todo/` holds only this phase.
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 19%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
........ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 235 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 237 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3279 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+19
@@ -0,0 +1,19 @@
|
|||||||
|
All green. Task 01 complete.
|
||||||
|
|
||||||
|
## Report
|
||||||
|
|
||||||
|
**Implemented (task 01 — storage half of per-source ignore paths):**
|
||||||
|
- `app/models.py`: `GitSource.ignore_paths` — JSONB NOT NULL, `default=list` (INSERT-time) + server default `'[]'` (verbatim task spec); `text` added to the sqlalchemy imports; module-docstring `git_sources` bullet updated (phase 89).
|
||||||
|
- `alembic/versions/0013_git_source_ignore_paths.py` (new): `add_column` JSONB/`'[]'`/NOT NULL, `drop_column` downgrade — 0007 house-header style, revises 0012.
|
||||||
|
- `tests/unit/test_models.py` (extended, house pattern): column exists / NOT NULL / server default `arg.text == "'[]'"` (narrowed via `DefaultClause` + `TextClause` for pyright), plus the Python-side default contract test.
|
||||||
|
|
||||||
|
**Gates (exact commands):**
|
||||||
|
- `uv run alembic upgrade head` → `0012 -> 0013` clean; `alembic downgrade -1 && alembic upgrade head` round-trips; head = `0013`.
|
||||||
|
- `uv run pytest tests/integration/test_git_sources_api.py` → 23 passed, **unchanged** (dev DB `git_sources` empty; `'[]'` server default covers existing rows; a live ORM-insert probe confirmed `[]` is stored — probe row deleted).
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1748 passed**, TOTAL **99%** (>90% ✓, `app/models.py` 100%).
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors).
|
||||||
|
- `git diff --stat`: only `app/models.py`, the new migration, and the unit test.
|
||||||
|
|
||||||
|
**Notable deviation:** the task's literal pin `GitSource(...).ignore_paths == []` cannot hold — SQLAlchemy 2.0 `default=list` is an INSERT-time `CallableColumnDefault` (unflushed attr is `None`; the builtin is wrapped in a ctx lambda, so `arg is list` also fails). The test instead pins the INSERT-time default resolving to `[]`; real-DB behavior is covered by the API integration suite. Model code matches the spec verbatim.
|
||||||
|
|
||||||
|
**Next pending task:** `.agents/phases/todo/89_source_ignore_paths/02_ignore_prefix_matching.md`
|
||||||
+86
@@ -0,0 +1,86 @@
|
|||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
.................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 212 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 98 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 180 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 232 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3238 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+10
@@ -0,0 +1,10 @@
|
|||||||
|
All green. Task complete.
|
||||||
|
|
||||||
|
**Report**
|
||||||
|
|
||||||
|
- Implemented in `app/rag/importer.py`: `normalize_ignore_path` / `is_ignored` / `_ignore_for_root` helpers (spec A1 raw-prefix semantics, single normalization choke point); `iter_importable_files(..., ignore=())` walk-level skip; `import_sources(..., ignore_by_root=None)` — pre-walk and processing loop use the same per-root tuple, `seen`/`_prune` untouched so newly-ignored files are pruned on next `prune=True` run (A2); docstrings per house style.
|
||||||
|
- New `tests/unit/test_importer_ignore.py` (24 tests): normalization table, spec examples verbatim (`/my/files/` ≡ `my/files/` ≡ `my/files`; `myfile.txt` root-level only, no mid-path), raw-prefix A1 edge (`my/files2`, `ab.md`), empty-prefix no-match, `iter_importable_files` on a 5-file tmp tree incl. the "walk receives already-normalized tuples" contract pin, `_ignore_for_root` normalize/drop/missing/None.
|
||||||
|
- New `tests/integration/test_importer_ignore.py` (5 tests): ignored files never indexed (no doc/chunk rows, no embed or chat call, kept `.txt` still summarized); A2 prune (`pruned == 1`, row gone, pattern given without trailing slash); progress `total` == 2 excludes ignored; no-map byte-identical (`files == 4`); unlisted second root imports everything.
|
||||||
|
- Test / lint / coverage: `uv run pytest tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -v` → 29 passed; `uv run pytest -q` → exit 0 (1777 collected, no failures); `uv run pytest --cov=app --cov-report=term-missing -q` → TOTAL 99% (>90% gate); `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
|
||||||
|
- Notable: (1) fixed the phase-64 counting-sentinel in `tests/unit/test_importer.py` to accept/forward the new `ignore` keyword (its monkeypatch stub broke on the new call site — the one extra file beyond the task's stated diff scope); (2) prune test uses a 3-file tree (one file under `ignore/`) so `pruned == 1` pins exactly, as the task specifies.
|
||||||
|
- Next pending task: `.agents/phases/todo/89_source_ignore_paths/03_ignore_paths_api.md`.
|
||||||
+86
@@ -0,0 +1,86 @@
|
|||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 212 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 98 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 232 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3248 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+17
@@ -0,0 +1,17 @@
|
|||||||
|
Everything is green. Final report:
|
||||||
|
|
||||||
|
**Task 03 (phase 89) complete — ignore-list API**
|
||||||
|
|
||||||
|
- `app/schemas.py`: `GitSourceIn.ignore_paths` (optional, raw lines), `GitSourceOut`/`GitSourceRow.ignore_paths` (non-null; env rows `[]`), new `GitSourceIgnoreIn` (required list, A5)
|
||||||
|
- `app/api/git_sources.py`: shared `_validate_ignore_paths` (normalize + A4 fixed-detail 422s, empty-check first; constants 200/500), GET reports per-row list, POST stores normalized for both kinds, new admin-only `PATCH /api/git-sources/{id}` (replace semantics, 404/422/403) placed after POST before `/upload`; module docstring updated
|
||||||
|
- `tests/integration/test_git_sources_api.py`: 10 new tests (normalize round-trip both kinds, absent→`[]`, POST+PATCH 422 exact details, missing-field 422, 404, invalid id, replace+clear, A4 boundaries, row-unchanged on reject); existing shape pins updated additively, anonymous test covers PATCH
|
||||||
|
|
||||||
|
**Gates:**
|
||||||
|
- `uv run pytest` → 1787 passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 99% (>90% ✓; `git_sources.py` + `schemas.py` 100%)
|
||||||
|
- `uv run pytest tests/integration/test_git_sources_api.py --no-cov` → 33 passed
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||||
|
|
||||||
|
**Notes:** no defects found in prior tasks; diff limited to the 3 task files. `.agents/PLAN.md` is absent in this repo — `00_phase.md` design (locked A1–A5) was the design source; no deviations.
|
||||||
|
|
||||||
|
**Next pending task:** `.agents/phases/todo/89_source_ignore_paths/04_ignore_paths_in_import_pipelines.md`
|
||||||
+86
@@ -0,0 +1,86 @@
|
|||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 232 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 98 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 237 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3273 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+19
@@ -0,0 +1,19 @@
|
|||||||
|
Task 04 complete. Report:
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `app/api/sync.py::_run_sync` — per-row loop now also builds `ignore_by_root` (keyed by the same `str(root)` the importer sees; `setdefault().extend()` union on root-string collision) and passes it to `import_sources`; module docstring step 4 updated
|
||||||
|
- `app/api/git_sources.py::_run_upload` — step 6 captures the row's saved `ignore_paths` into a plain local (both existing/new-row branches, before `db.close()`); step 8 passes `ignore_by_root={str(final_dir): ignore_paths}`; route docstring steps 6/8 updated
|
||||||
|
- `scripts/import_docs.py` — `_resolve_sources` returns `(sources, ignore_by_root)` (manual `--source` and legacy fallback → `{}`); `main` unpacks and passes the map; module + function docstrings updated
|
||||||
|
- Tests: real-import sync test (ignored file excluded from KB + `files` count), fake-import union/empty-map wiring tests, CLI rows-branch/manual map tests, re-upload-respects-PATCHed-list test (pruned on re-scan) + new-source-no-list regression; `ignore_by_root` kwarg added to the 4 `import_sources` fakes
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest -q` → 1795 passed, exit 0 (incl. `tests/integration/test_sync_api.py test_import_docs_git.py test_git_sources_upload.py` + `tests/unit/test_sync_button.py`: 105 passed)
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%); `app/api/sync.py` + `app/api/git_sources.py` 100%
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean
|
||||||
|
- E2E regressions in isolation (`test_git_sources_admin`, `test_archive_upload_sources`, `test_sync_button`, `test_smoke`) → 17 passed
|
||||||
|
|
||||||
|
**Decisions**
|
||||||
|
- One-line mechanical addition to `tests/unit/test_sync_button.py` fake signature (outside the task's listed files) — required to keep the suite green once the runner passes the new kwarg
|
||||||
|
- Union-order test pins distinct `added_at` (same-timestamp rows tie-break on random uuid)
|
||||||
|
|
||||||
|
**Next pending task:** `05_ignore_paths_sources_ui.md`
|
||||||
+86
@@ -0,0 +1,86 @@
|
|||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 235 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 237 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3279 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
**Task 05 complete — per-row "Ignore paths" box on the Sources page.**
|
||||||
|
|
||||||
|
Implemented:
|
||||||
|
- `frontend/index.html`: `#ignore-editor-dialog` (phase-69 `#remove-confirm-dialog` pattern — alertdialog, dim backdrop, visible label, mono textarea, `role="alert"` error line, Cancel/Save) placed after the remove dialog in `#git-sources-content`
|
||||||
|
- `frontend/assets/git-sources.js`: per-row "Ignore paths" button (stored rows only, left of Remove, textContent-only), `N ignored` count tag, `openIgnoreEditor`/`closeIgnoreEditor`/`saveIgnorePaths` with the §7.4 "Saving…" lifecycle, focus-on-Cancel, Escape/Cancel/backdrop close with focus return, blank-line dropping, A5-replace PATCH, LAST-announcement order; module docstring phase-89 bullet
|
||||||
|
- `frontend/assets/styles.css`: `.ignore-editor-*` dialog family (house tokens, no blur, ≥44px, brand Save / ghost Cancel) + `.git-source-ignore` / `.git-source-ignore-count` row chrome
|
||||||
|
- `tests/unit/test_source_ignore_paths.py` (new): 13 source-level pins (markup, lifecycle, row guard, CSS, id no-collision)
|
||||||
|
|
||||||
|
Results:
|
||||||
|
- `uv run pytest tests/unit/test_source_ignore_paths.py --no-cov` → 13 passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, TOTAL 99%** (>90%)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- E2E regressions in isolation: `test_git_sources_admin.py` 6 passed; `test_smoke.py` + `test_archive_upload_sources.py` + `test_sync_button.py` 11 passed
|
||||||
|
- Headless manual smoke: add local source → open box → save (blank line dropped) → `2 ignored` tag + announcer → refresh prefills (normalized) → Escape closes with focus return → empty box clears (A5) — all verified, source cleaned up
|
||||||
|
|
||||||
|
Decisions: Escape/Cancel/backdrop are no-ops mid-PATCH (in-flight guard, remove-dialog idiom); Save uses the `.new-chat-btn` brand hover pair; count tag = ink-on-bg chip (16.7:1). One bug found+fixed in my own test (f-string doubled the announce phrase).
|
||||||
|
|
||||||
|
Next pending task: `.agents/phases/todo/89_source_ignore_paths/06_e2e_ignore_paths.md`.
|
||||||
+87
@@ -0,0 +1,87 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 19%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
........ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 235 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 237 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3279 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+11
@@ -0,0 +1,11 @@
|
|||||||
|
**Task 06 complete — report**
|
||||||
|
|
||||||
|
- Implemented `tests/e2e/test_source_ignore_paths.py` (new story suite, 6 tests): anonymous gate + 403s (incl. new PATCH), box→sync exclusion (`ignore/` never walked/embedded/summarized, `files` count excludes it), spec's no-mid-path rule (`sub/myfile.md` kept, root `myfile.md` ignored, list *replaces* per A5), A2 prune of newly-ignored files (`pruned == 2`), editor a11y + 422 path (501-char entry, textarea kept, Save re-enabled, happy-path heal), env-fallback rows with no box + env note. Module app on port 8140 with `kind="local"` fixture dir (6 files) — no git, no network.
|
||||||
|
- Results (all green):
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 1808 passed, TOTAL **99%** (>90%)
|
||||||
|
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (run twice, stable)
|
||||||
|
- Regression E2E in isolation: `test_git_sources_admin.py` 6 passed, `test_archive_upload_sources.py` 5 passed, `test_sync_button.py` 3 passed, `test_smoke.py` 3 passed
|
||||||
|
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors
|
||||||
|
- Deviations: (1) task's `files == 5` assertion is an arithmetic slip against its own 6-file tree (6−2=4) — test asserts 4, documented in-code; (2) stored-list round-trip asserts the A1-normalized value (`ignore` for box line `ignore/`); (3) no `git add`/`commit` run — left in working tree per executor rules (harness makes the one atomic `--no-gpg-sign` commit; `TODO.md` already exactly `# TODO`).
|
||||||
|
- Diff hygiene: matches step-3 file set (the small `tests/unit/test_importer.py` / `test_sync_button.py` modifications pre-date this task, from completed tasks 02/04).
|
||||||
|
- Next pending task: none — this was the last task of phase 89; the pipeline gate moves the phase dir and commits.
|
||||||
+87
@@ -0,0 +1,87 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 19%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
........ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 235 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/config.py 141 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 19 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 64 0 100%
|
||||||
|
app/models.py 95 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 237 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3279 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
"""git_sources.ignore_paths: per-source ignore-path list (phase 89)
|
||||||
|
|
||||||
|
Revision ID: 0013
|
||||||
|
Revises: 0012
|
||||||
|
Create Date: 2026-09-08
|
||||||
|
|
||||||
|
Phase 89 (per-source ignore paths: the owner types a list of
|
||||||
|
source-relative path prefixes per source — one path per line — into the
|
||||||
|
box on the Sources page; the importer never walks, embeds, or summarizes
|
||||||
|
a file whose source-relative path starts with one of them; A1 raw prefix
|
||||||
|
semantics, A4 limits, A5 replace semantics; one additive, reversible
|
||||||
|
column, A13):
|
||||||
|
|
||||||
|
* ``git_sources.ignore_paths`` — JSONB NOT NULL with server default
|
||||||
|
``'[]'``: the normalized, non-empty source-relative path prefixes. A
|
||||||
|
file is ignored when its source-relative POSIX path starts with any
|
||||||
|
entry (no mid-path matching, no globs). Existing rows read ``[]`` —
|
||||||
|
every pre-phase-89 row imports exactly as before.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from sqlalchemy.dialects import postgresql
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision = "0013"
|
||||||
|
down_revision = "0012"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.add_column(
|
||||||
|
"git_sources",
|
||||||
|
sa.Column(
|
||||||
|
"ignore_paths",
|
||||||
|
postgresql.JSONB(astext_type=sa.Text()),
|
||||||
|
server_default=sa.text("'[]'"),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# Safe order: the column is the only 0013 artefact — dropping it
|
||||||
|
# leaves 0012's schema byte-identical (A13, fully reversible).
|
||||||
|
op.drop_column("git_sources", "ignore_paths")
|
||||||
+115
-17
@@ -14,10 +14,18 @@ fallback is active).
|
|||||||
|
|
||||||
Routes: ``GET`` (DB rows oldest-first, or the env list with
|
Routes: ``GET`` (DB rows oldest-first, or the env list with
|
||||||
``from_env: true`` while the table is empty; rows carry ``kind`` +
|
``from_env: true`` while the table is empty; rows carry ``kind`` +
|
||||||
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
``path``, git rows — and env rows — report ``path: null``; every row
|
||||||
|
reports ``ignore_paths`` — DB rows their stored normalized list, env
|
||||||
|
rows ``[]``, phase 89), ``POST``
|
||||||
(201, validated create; ``kind`` selects the validation: git → exactly
|
(201, validated create; ``kind`` selects the validation: git → exactly
|
||||||
the phase-35 URL contract, local → an existing absolute directory, else
|
the phase-35 URL contract, local → an existing absolute directory, else
|
||||||
422 naming the path), ``POST /upload`` (phase 49, backgrounded in phase
|
422 naming the path; both kinds accept ``ignore_paths`` — optional,
|
||||||
|
absent → ``[]``, stored normalized, phase 89),
|
||||||
|
``PATCH /{source_id}`` (phase 89, A5 — replace one source's ignore
|
||||||
|
list: 404 unknown id, the required body list is normalized + A4-
|
||||||
|
validated with fixed-detail 422s and REPLACES the row's list wholesale
|
||||||
|
— an empty list clears all; 200 → the ``GitSourceOut`` shape),
|
||||||
|
``POST /upload`` (phase 49, backgrounded in phase
|
||||||
64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/
|
64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/
|
||||||
``.zip`` name/format gate + the 1 MiB-chunk receive with the
|
``.zip`` name/format gate + the 1 MiB-chunk receive with the
|
||||||
``upload_max_mb`` cap run **inline** and answered 202 the moment the
|
``upload_max_mb`` cap run **inline** and answered 202 the moment the
|
||||||
@@ -90,7 +98,7 @@ from app.rag.archive_upload import (
|
|||||||
swap_in,
|
swap_in,
|
||||||
unpack_archive,
|
unpack_archive,
|
||||||
)
|
)
|
||||||
from app.rag.importer import import_sources
|
from app.rag.importer import import_sources, normalize_ignore_path
|
||||||
from app.rag.llm import LLMClient, check_models
|
from app.rag.llm import LLMClient, check_models
|
||||||
from app.rag.overview import regenerate_overview
|
from app.rag.overview import regenerate_overview
|
||||||
from app.rag.source_removal import (
|
from app.rag.source_removal import (
|
||||||
@@ -101,6 +109,7 @@ from app.rag.source_removal import (
|
|||||||
)
|
)
|
||||||
from app.rag.sources_meta import bump_sources_version
|
from app.rag.sources_meta import bump_sources_version
|
||||||
from app.schemas import (
|
from app.schemas import (
|
||||||
|
GitSourceIgnoreIn,
|
||||||
GitSourceIn,
|
GitSourceIn,
|
||||||
GitSourceList,
|
GitSourceList,
|
||||||
GitSourceOut,
|
GitSourceOut,
|
||||||
@@ -172,6 +181,14 @@ _upload_status = UploadStatus()
|
|||||||
#: accepted shapes are exactly these four prefixes.
|
#: accepted shapes are exactly these four prefixes.
|
||||||
URL_RE = re.compile(r"^(https?://|ssh://|git@)")
|
URL_RE = re.compile(r"^(https?://|ssh://|git@)")
|
||||||
|
|
||||||
|
#: Phase 89, A4 — the per-source ignore-list limits, enforced in
|
||||||
|
#: :func:`_validate_ignore_paths` (shared by POST and PATCH): at most
|
||||||
|
#: 200 entries, each ≤500 chars after normalization. The 422 details
|
||||||
|
#: are fixed strings that never echo the input (the router's
|
||||||
|
#: credential-safety discipline, applied for consistency).
|
||||||
|
MAX_IGNORE_PATHS = 200
|
||||||
|
MAX_IGNORE_PATH_LENGTH = 500
|
||||||
|
|
||||||
|
|
||||||
@router.get("", response_model=GitSourceList)
|
@router.get("", response_model=GitSourceList)
|
||||||
def list_git_sources(
|
def list_git_sources(
|
||||||
@@ -181,10 +198,13 @@ def list_git_sources(
|
|||||||
|
|
||||||
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
|
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
|
||||||
same-timestamp inserts) with ``from_env: false`` — each row carries
|
same-timestamp inserts) with ``from_env: false`` — each row carries
|
||||||
its ``kind`` and, for local rows, the stored ``path`` (git rows and
|
its ``kind``, its ``ignore_paths`` (phase 89 — the stored,
|
||||||
env rows report ``path: null``); while the table is empty, the
|
normalized list; ``or []`` guards a row that predated the column),
|
||||||
|
and, for local rows, the stored ``path`` (git rows and env rows
|
||||||
|
report ``path: null``); while the table is empty, the
|
||||||
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
|
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
|
||||||
git-only) with null ``id``/``added_at`` and ``from_env: true``.
|
git-only) with null ``id``/``added_at``, ``ignore_paths: []`` (no
|
||||||
|
DB row to store a list on), and ``from_env: true``.
|
||||||
"""
|
"""
|
||||||
rows = db.scalars(
|
rows = db.scalars(
|
||||||
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
||||||
@@ -200,6 +220,7 @@ def list_git_sources(
|
|||||||
url=row.url,
|
url=row.url,
|
||||||
path=row.path,
|
path=row.path,
|
||||||
added_at=row.added_at,
|
added_at=row.added_at,
|
||||||
|
ignore_paths=row.ignore_paths or [],
|
||||||
)
|
)
|
||||||
for row in rows
|
for row in rows
|
||||||
],
|
],
|
||||||
@@ -207,7 +228,9 @@ def list_git_sources(
|
|||||||
)
|
)
|
||||||
return GitSourceList(
|
return GitSourceList(
|
||||||
sources=[
|
sources=[
|
||||||
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None)
|
GitSourceRow(
|
||||||
|
id=None, kind="git", url=url, path=None, added_at=None, ignore_paths=[]
|
||||||
|
)
|
||||||
for url in get_settings().git_source_list
|
for url in get_settings().git_source_list
|
||||||
],
|
],
|
||||||
from_env=True,
|
from_env=True,
|
||||||
@@ -236,9 +259,35 @@ def create_git_source(
|
|||||||
|
|
||||||
Wrong field combinations (git without url, local without path, both
|
Wrong field combinations (git without url, local without path, both
|
||||||
kinds' fields) are 422 with fixed, input-free details.
|
kinds' fields) are 422 with fixed, input-free details.
|
||||||
|
|
||||||
|
``ignore_paths`` (phase 89) — optional, both kinds: the raw box
|
||||||
|
lines are normalized + A4-validated (``_validate_ignore_paths`` —
|
||||||
|
the fixed-detail 422s) and the normalized list is what is stored and
|
||||||
|
reported.
|
||||||
"""
|
"""
|
||||||
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
|
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
|
||||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
return GitSourceOut(
|
||||||
|
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_ignore_paths(raw: list[str] | None) -> list[str]:
|
||||||
|
"""Normalize + enforce the A4 limits (phase 89); the fixed 422
|
||||||
|
details never echo the input (the credential-safety discipline,
|
||||||
|
applied for consistency).
|
||||||
|
|
||||||
|
Shared by POST and PATCH. The empty check runs FIRST: a
|
||||||
|
whitespace-only entry must 422, not be silently dropped (the UI
|
||||||
|
drops blank lines client-side; the API stays defensive).
|
||||||
|
"""
|
||||||
|
entries = [normalize_ignore_path(e) for e in (raw or [])]
|
||||||
|
if any(not e for e in entries):
|
||||||
|
raise HTTPException(status_code=422, detail="ignore paths must be non-empty")
|
||||||
|
if len(entries) > MAX_IGNORE_PATHS:
|
||||||
|
raise HTTPException(status_code=422, detail="a source has at most 200 ignore paths")
|
||||||
|
if any(len(e) > MAX_IGNORE_PATH_LENGTH for e in entries):
|
||||||
|
raise HTTPException(status_code=422, detail="an ignore path exceeds 500 characters")
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
|
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
|
||||||
@@ -270,7 +319,13 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
|
|||||||
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
|
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
|
||||||
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
|
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
|
||||||
return _commit_new(
|
return _commit_new(
|
||||||
GitSource(url=url, kind="git"), "a git source with this URL already exists", db
|
GitSource(
|
||||||
|
url=url,
|
||||||
|
kind="git",
|
||||||
|
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
|
||||||
|
),
|
||||||
|
"a git source with this URL already exists",
|
||||||
|
db,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -296,12 +351,41 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
|||||||
# rows carry the expanded path there too — git URL shapes and absolute
|
# rows carry the expanded path there too — git URL shapes and absolute
|
||||||
# paths cannot collide).
|
# paths cannot collide).
|
||||||
return _commit_new(
|
return _commit_new(
|
||||||
GitSource(url=path, kind="local", path=path),
|
GitSource(
|
||||||
|
url=path,
|
||||||
|
kind="local",
|
||||||
|
path=path,
|
||||||
|
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
|
||||||
|
),
|
||||||
f"a local source with this path already exists: {path}",
|
f"a local source with this path already exists: {path}",
|
||||||
db,
|
db,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/{source_id}", response_model=GitSourceOut)
|
||||||
|
def patch_git_source(
|
||||||
|
source_id: uuid.UUID,
|
||||||
|
payload: GitSourceIgnoreIn,
|
||||||
|
db: Session = Depends(get_db), # noqa: B008
|
||||||
|
) -> GitSourceOut:
|
||||||
|
"""Replace one source's ignore list (phase 89, A5).
|
||||||
|
|
||||||
|
404 unknown id. The body list (required) is normalized +
|
||||||
|
A4-validated (fixed 422 details) and REPLACES the row's list
|
||||||
|
wholesale — an empty list clears all. Returns the updated
|
||||||
|
row's public shape (id, url, added_at, ignore_paths).
|
||||||
|
"""
|
||||||
|
row = db.get(GitSource, source_id)
|
||||||
|
if row is None:
|
||||||
|
raise HTTPException(status_code=404, detail="git source not found")
|
||||||
|
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(row)
|
||||||
|
return GitSourceOut(
|
||||||
|
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/upload", response_model=UploadAccepted, status_code=202)
|
@router.post("/upload", response_model=UploadAccepted, status_code=202)
|
||||||
async def upload_archive(
|
async def upload_archive(
|
||||||
file: UploadFile = File(...), # noqa: B008
|
file: UploadFile = File(...), # noqa: B008
|
||||||
@@ -339,14 +423,18 @@ async def upload_archive(
|
|||||||
6. upsert the row by ``path`` (``kind='local'``; an existing row is
|
6. upsert the row by ``path`` (``kind='local'``; an existing row is
|
||||||
left as-is — ``added_at`` preserved — and the unique index is
|
left as-is — ``added_at`` preserved — and the unique index is
|
||||||
the backstop: a concurrent insert lands ``failed`` with
|
the backstop: a concurrent insert lands ``failed`` with
|
||||||
``a local source with this path already exists: <path>``);
|
``a local source with this path already exists: <path>``); the
|
||||||
|
row's saved ``ignore_paths`` are captured for the scan (phase
|
||||||
|
89: a re-upload of an existing source honors the list the owner
|
||||||
|
already saved);
|
||||||
7. fail-fast ``check_models`` — ``ModelUnavailableError`` →
|
7. fail-fast ``check_models`` — ``ModelUnavailableError`` →
|
||||||
``failed`` with the sanitized message (the phase-49 503 becomes
|
``failed`` with the sanitized message (the phase-49 503 becomes
|
||||||
a status state, A5); the folder/row are already committed, so
|
a status state, A5); the folder/row are already committed, so
|
||||||
the next sync/re-upload retries idempotently;
|
the next sync/re-upload retries idempotently;
|
||||||
8. ``import_sources([folder], llm, prune=True, progress=<hook>)``
|
8. ``import_sources([folder], llm, prune=True, progress=<hook>,
|
||||||
+ the change-gated ``regenerate_overview`` — the hook feeds the
|
ignore_by_root={folder: row's list})`` (phase 89) + the
|
||||||
status ``current_file`` / ``files_done`` / ``files_total``;
|
change-gated ``regenerate_overview`` — the hook feeds the status
|
||||||
|
``current_file`` / ``files_done`` / ``files_total``;
|
||||||
9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is
|
9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is
|
||||||
the background run's duration);
|
the background run's duration);
|
||||||
10. ``success`` — ``detail`` = the ``UploadOut`` fields.
|
10. ``success`` — ``detail`` = the ``UploadOut`` fields.
|
||||||
@@ -502,8 +590,10 @@ async def _run_upload(
|
|||||||
path = str(final_dir)
|
path = str(final_dir)
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
|
row = db.scalar(select(GitSource).where(GitSource.path == path))
|
||||||
db.add(GitSource(url=path, kind="local", path=path))
|
if row is None:
|
||||||
|
row = GitSource(url=path, kind="local", path=path)
|
||||||
|
db.add(row)
|
||||||
try:
|
try:
|
||||||
db.commit()
|
db.commit()
|
||||||
except IntegrityError:
|
except IntegrityError:
|
||||||
@@ -511,6 +601,11 @@ async def _run_upload(
|
|||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"a local source with this path already exists: {path}"
|
f"a local source with this path already exists: {path}"
|
||||||
) from None
|
) from None
|
||||||
|
# Phase 89: the row's saved ignore list, copied to plain
|
||||||
|
# values while the row is still usable in this session — a
|
||||||
|
# re-upload of an existing source honors the list the owner
|
||||||
|
# already saved; a fresh row has no list yet.
|
||||||
|
ignore_paths = list(row.ignore_paths or [])
|
||||||
finally:
|
finally:
|
||||||
db.close()
|
db.close()
|
||||||
# Step 7 — fail-fast models (phase 41): ``ModelUnavailableError``
|
# Step 7 — fail-fast models (phase 41): ``ModelUnavailableError``
|
||||||
@@ -530,7 +625,10 @@ async def _run_upload(
|
|||||||
_upload_status.files_done = done
|
_upload_status.files_done = done
|
||||||
_upload_status.files_total = total
|
_upload_status.files_total = total
|
||||||
|
|
||||||
summary = await import_sources([final_dir], llm, prune=True, progress=_hook)
|
summary = await import_sources(
|
||||||
|
[final_dir], llm, prune=True, progress=_hook,
|
||||||
|
ignore_by_root={str(final_dir): ignore_paths},
|
||||||
|
)
|
||||||
overview = False
|
overview = False
|
||||||
if summary.added + summary.updated > 0:
|
if summary.added + summary.updated > 0:
|
||||||
overview = await regenerate_overview(llm)
|
overview = await regenerate_overview(llm)
|
||||||
|
|||||||
+20
-9
@@ -34,9 +34,11 @@ decisions):
|
|||||||
<path>``; a failing clone or a missing local dir aborts before any
|
<path>``; a failing clone or a missing local dir aborts before any
|
||||||
import;
|
import;
|
||||||
4. ``import_sources(..., prune=True)`` over the single combined list
|
4. ``import_sources(..., prune=True)`` over the single combined list
|
||||||
(git checkouts + local dirs) — prune so files deleted upstream or
|
(git checkouts + local dirs), honoring each row's ``ignore_paths``
|
||||||
out of a local dir leave the index (pruning covers the union; the
|
(phase 89 — the per-root ignore map is built in the same per-row
|
||||||
CLI's no-prune default is unchanged);
|
loop as the source list) — prune so files deleted upstream, out of
|
||||||
|
a local dir, or newly matching an ignore pattern leave the index
|
||||||
|
(pruning covers the union; the CLI's no-prune default is unchanged);
|
||||||
5. when the import changed the KB (added + updated > 0),
|
5. when the import changed the KB (added + updated > 0),
|
||||||
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
||||||
(phase 31 trigger, best-effort inside);
|
(phase 31 trigger, best-effort inside);
|
||||||
@@ -214,19 +216,26 @@ async def _run_sync() -> None:
|
|||||||
)
|
)
|
||||||
sources_root = Path(settings.sources_dir).expanduser()
|
sources_root = Path(settings.sources_dir).expanduser()
|
||||||
sources: list[Path] = []
|
sources: list[Path] = []
|
||||||
|
ignore_by_root: dict[str, list[str]] = {}
|
||||||
for row in rows:
|
for row in rows:
|
||||||
if row.kind == "git":
|
if row.kind == "git":
|
||||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
|
||||||
else:
|
else:
|
||||||
# kind=local — the stored expanded path (phase 38 also
|
# kind=local — the stored expanded path (phase 38 also
|
||||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||||
# ``or`` keeps the type checker honest); re-verified at
|
# ``or`` keeps the type checker honest); re-verified at
|
||||||
# sync time because the directory may have moved or been
|
# sync time because the directory may have moved or been
|
||||||
# deleted since add-time.
|
# deleted since add-time.
|
||||||
path = Path(row.path or row.url).expanduser()
|
root = Path(row.path or row.url).expanduser()
|
||||||
if not path.is_dir():
|
if not root.is_dir():
|
||||||
raise GitSyncError(f"local source missing: {path}")
|
raise GitSyncError(f"local source missing: {root}")
|
||||||
sources.append(path)
|
sources.append(root)
|
||||||
|
# Phase 89: the row's ignore list, keyed by the SAME root
|
||||||
|
# string the importer sees; two rows sharing a root string
|
||||||
|
# get the union (extend, not replace) — the sibling/repo-name
|
||||||
|
# edge.
|
||||||
|
if row.ignore_paths:
|
||||||
|
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
|
||||||
# Phase 64 (task 02): the per-file progress hook — the status
|
# Phase 64 (task 02): the per-file progress hook — the status
|
||||||
# endpoint reports the file being processed right now. The
|
# endpoint reports the file being processed right now. The
|
||||||
# closure captures the module ``_status`` exactly like the state
|
# closure captures the module ``_status`` exactly like the state
|
||||||
@@ -236,7 +245,9 @@ async def _run_sync() -> None:
|
|||||||
_status.files_done = done
|
_status.files_done = done
|
||||||
_status.files_total = total
|
_status.files_total = total
|
||||||
|
|
||||||
summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook)
|
summary: ImportSummary = await import_sources(
|
||||||
|
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root
|
||||||
|
)
|
||||||
overview = False
|
overview = False
|
||||||
if summary.added + summary.updated > 0:
|
if summary.added + summary.updated > 0:
|
||||||
overview = await regenerate_overview(llm)
|
overview = await regenerate_overview(llm)
|
||||||
|
|||||||
+13
-1
@@ -16,7 +16,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
|
|||||||
* ``git_sources`` — admin-managed source registry (git URLs + local
|
* ``git_sources`` — admin-managed source registry (git URLs + local
|
||||||
directories) the Sync button and import_docs
|
directories) the Sync button and import_docs
|
||||||
import (phase 35; ``kind`` discriminator added in
|
import (phase 35; ``kind`` discriminator added in
|
||||||
phase 38).
|
phase 38; ``ignore_paths`` (phase 89 — JSONB list
|
||||||
|
of normalized path prefixes, server default
|
||||||
|
``'[]'``)).
|
||||||
* ``saved_chats`` — owner-saved chat conversations: one row per
|
* ``saved_chats`` — owner-saved chat conversations: one row per
|
||||||
explicitly Saved conversation (auto-``title`` +
|
explicitly Saved conversation (auto-``title`` +
|
||||||
the ``bor.chat.v1`` message list as JSONB, phase
|
the ``bor.chat.v1`` message list as JSONB, phase
|
||||||
@@ -72,6 +74,7 @@ from sqlalchemy import (
|
|||||||
Text,
|
Text,
|
||||||
UniqueConstraint,
|
UniqueConstraint,
|
||||||
func,
|
func,
|
||||||
|
text,
|
||||||
)
|
)
|
||||||
from sqlalchemy.dialects.postgresql import JSONB, UUID
|
from sqlalchemy.dialects.postgresql import JSONB, UUID
|
||||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||||
@@ -226,6 +229,15 @@ class GitSource(Base):
|
|||||||
#: Absolute directory of a ``local`` source; NULL for git rows.
|
#: Absolute directory of a ``local`` source; NULL for git rows.
|
||||||
#: Unique — Postgres treats NULLs as distinct under a unique index.
|
#: Unique — Postgres treats NULLs as distinct under a unique index.
|
||||||
path: Mapped[str | None] = mapped_column(Text, unique=True)
|
path: Mapped[str | None] = mapped_column(Text, unique=True)
|
||||||
|
#: Per-source ignore paths (phase 89, A1/A4): the normalized,
|
||||||
|
#: non-empty source-relative path prefixes the owner types into the
|
||||||
|
#: box on the Sources page. A file is ignored when its
|
||||||
|
#: source-relative POSIX path starts with any entry (raw prefix —
|
||||||
|
#: no mid-path matching, no globs). Server default '[]' — every
|
||||||
|
#: pre-phase-89 row imports exactly as before.
|
||||||
|
ignore_paths: Mapped[list] = mapped_column(
|
||||||
|
JSONB, default=list, server_default=text("'[]'"), nullable=False
|
||||||
|
)
|
||||||
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+73
-3
@@ -118,17 +118,63 @@ class ImportSummary:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_ignore_path(entry: str) -> str:
|
||||||
|
"""One ignore-path entry → canonical form (phase 89, A1).
|
||||||
|
|
||||||
|
Trim surrounding whitespace, then strip ALL leading/trailing
|
||||||
|
``/`` — so ``"/my/files/"``, ``"my/files/"`` and ``"my/files"``
|
||||||
|
all become ``"my/files"``. ``""`` / ``"//"``,
|
||||||
|
``" "`` normalize to ``""`` (callers drop empties).
|
||||||
|
"""
|
||||||
|
return entry.strip().strip("/")
|
||||||
|
|
||||||
|
|
||||||
|
def is_ignored(rel: str, prefixes: tuple[str, ...]) -> bool:
|
||||||
|
"""Phase 89, A1 — the pure prefix rule, nothing else.
|
||||||
|
|
||||||
|
``rel`` is the source-relative POSIX path WITHOUT a leading
|
||||||
|
slash (the ``documents.path`` string). Match = ``rel`` STARTS
|
||||||
|
WITH a normalized entry: raw string prefix — deliberately NO
|
||||||
|
component-boundary check (``"my/files"`` also matches
|
||||||
|
``"my/files2/x.md"``) and NO mid-path matching (``"myfile.txt"``
|
||||||
|
matches ``"myfile.txt"`` but not ``"some/path/myfile.txt"``).
|
||||||
|
"""
|
||||||
|
return any(rel.startswith(p) for p in prefixes)
|
||||||
|
|
||||||
|
|
||||||
|
def _ignore_for_root(
|
||||||
|
root: Path, ignore_by_root: dict[str, list[str]] | None
|
||||||
|
) -> tuple[str, ...]:
|
||||||
|
"""The normalized, non-empty prefix tuple for one root (phase 89).
|
||||||
|
|
||||||
|
Keyed by ``str(root)`` — the root string exactly as the caller
|
||||||
|
passed it in ``sources`` (unambiguous when two rows share a
|
||||||
|
source *name* but different dirs). Callers may pass RAW box
|
||||||
|
lines: the importer normalizes + drops empties here, the single
|
||||||
|
choke point — stored lists (already normalized) normalize to
|
||||||
|
themselves.
|
||||||
|
"""
|
||||||
|
raw = (ignore_by_root or {}).get(str(root)) or []
|
||||||
|
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
|
||||||
|
|
||||||
|
|
||||||
def iter_importable_files(
|
def iter_importable_files(
|
||||||
root: Path,
|
root: Path,
|
||||||
extensions: frozenset[str],
|
extensions: frozenset[str],
|
||||||
excluded: frozenset[str] = EXCLUDED_DIRS,
|
excluded: frozenset[str] = EXCLUDED_DIRS,
|
||||||
|
ignore: tuple[str, ...] = (),
|
||||||
) -> list[Path]:
|
) -> list[Path]:
|
||||||
"""All importable files under *root* (sorted), per the A9 scope rules.
|
"""All importable files under *root* (sorted), per the A9 scope rules.
|
||||||
|
|
||||||
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
|
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
|
||||||
Skips: any path with a dot-prefixed component (hidden dirs/files —
|
Skips: any path with a dot-prefixed component (hidden dirs/files —
|
||||||
vendored caches like ``.esphome/.espressif/**``) and the well-known
|
vendored caches like ``.esphome/.espressif/**``) and the well-known
|
||||||
non-content directories in *excluded*.
|
non-content directories in *excluded*. *ignore* (phase 89, A1) is a
|
||||||
|
tuple of ALREADY-normalized, non-empty source-relative path prefixes
|
||||||
|
(the importer's ``_ignore_for_root`` is the normalization choke point
|
||||||
|
— raw box lines never reach this function): a file is skipped when its
|
||||||
|
source-relative POSIX path starts with any entry; the default ``()``
|
||||||
|
keeps every existing caller byte-identical.
|
||||||
"""
|
"""
|
||||||
if not root.is_dir():
|
if not root.is_dir():
|
||||||
return []
|
return []
|
||||||
@@ -139,6 +185,8 @@ def iter_importable_files(
|
|||||||
rel = path.relative_to(root)
|
rel = path.relative_to(root)
|
||||||
if any(part.startswith(".") or part in excluded for part in rel.parts):
|
if any(part.startswith(".") or part in excluded for part in rel.parts):
|
||||||
continue
|
continue
|
||||||
|
if ignore and is_ignored(rel.as_posix(), ignore):
|
||||||
|
continue
|
||||||
if path.suffix.lower() not in extensions:
|
if path.suffix.lower() not in extensions:
|
||||||
continue
|
continue
|
||||||
files.append(path)
|
files.append(path)
|
||||||
@@ -153,6 +201,7 @@ async def import_sources(
|
|||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: Session | None = None,
|
session: Session | None = None,
|
||||||
progress: Callable[[str, str, int, int], None] | None = None,
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
|
ignore_by_root: dict[str, list[str]] | None = None,
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
"""Import every A9-format file under *sources* (see module docstring).
|
"""Import every A9-format file under *sources* (see module docstring).
|
||||||
|
|
||||||
@@ -172,6 +221,18 @@ async def import_sources(
|
|||||||
``limit``, the hook still fires per processed file only — ``done`` never
|
``limit``, the hook still fires per processed file only — ``done`` never
|
||||||
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||||||
incomplete walk must not misreport the denominator).
|
incomplete walk must not misreport the denominator).
|
||||||
|
|
||||||
|
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
|
||||||
|
string exactly as passed in *sources* — to that source's RAW ignore-path
|
||||||
|
lines (the importer normalizes them via ``_ignore_for_root``, the single
|
||||||
|
choke point): matching files are never walked, so they are never
|
||||||
|
embedded and never summarized, and the progress pre-walk uses the same
|
||||||
|
per-root tuple as the processing loop, so ``total`` never counts them.
|
||||||
|
A file that newly matches a pattern simply never enters ``seen``, so the
|
||||||
|
next ``prune=True`` run deletes its row automatically (A2 — the A9
|
||||||
|
junk-precedent). ``None`` (the default) changes nothing: the map is read
|
||||||
|
per root, unlisted roots get an empty tuple, and every existing caller
|
||||||
|
behaves byte-identically.
|
||||||
"""
|
"""
|
||||||
if limit is not None and limit <= 0:
|
if limit is not None and limit <= 0:
|
||||||
raise ValueError("limit must be >= 1")
|
raise ValueError("limit must be >= 1")
|
||||||
@@ -188,7 +249,13 @@ async def import_sources(
|
|||||||
total = 0
|
total = 0
|
||||||
if progress is not None:
|
if progress is not None:
|
||||||
for root in sources:
|
for root in sources:
|
||||||
total += len(iter_importable_files(root, llm.settings.import_extension_set))
|
total += len(
|
||||||
|
iter_importable_files(
|
||||||
|
root,
|
||||||
|
llm.settings.import_extension_set,
|
||||||
|
ignore=_ignore_for_root(root, ignore_by_root),
|
||||||
|
)
|
||||||
|
)
|
||||||
try:
|
try:
|
||||||
for root in sources:
|
for root in sources:
|
||||||
if not root.is_dir():
|
if not root.is_dir():
|
||||||
@@ -198,7 +265,10 @@ async def import_sources(
|
|||||||
break
|
break
|
||||||
source = root.name
|
source = root.name
|
||||||
source_names.add(source)
|
source_names.add(source)
|
||||||
for path in iter_importable_files(root, llm.settings.import_extension_set):
|
ignore = _ignore_for_root(root, ignore_by_root)
|
||||||
|
for path in iter_importable_files(
|
||||||
|
root, llm.settings.import_extension_set, ignore=ignore
|
||||||
|
):
|
||||||
if limit is not None and summary.files >= limit:
|
if limit is not None and summary.files >= limit:
|
||||||
break
|
break
|
||||||
rel = path.relative_to(root).as_posix()
|
rel = path.relative_to(root).as_posix()
|
||||||
|
|||||||
+30
-2
@@ -327,11 +327,18 @@ class GitSourceIn(BaseModel):
|
|||||||
Trimmed here; the API layer then ``expanduser()``s it and requires an
|
Trimmed here; the API layer then ``expanduser()``s it and requires an
|
||||||
absolute existing directory (else 422 naming the path — the path is
|
absolute existing directory (else 422 naming the path — the path is
|
||||||
not a secret, unlike a git URL) and no ``url``.
|
not a secret, unlike a git URL) and no ``url``.
|
||||||
|
|
||||||
|
``ignore_paths`` (phase 89) is optional at create time (absent →
|
||||||
|
``[]``) and carries the RAW box lines — trimming/normalization happens
|
||||||
|
in the API layer, not the schema, so the A4 422 details stay fixed
|
||||||
|
strings (the router's credential-safety discipline, applied for
|
||||||
|
consistency).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
kind: Literal["git", "local"] = "git"
|
kind: Literal["git", "local"] = "git"
|
||||||
url: str | None = Field(default=None, min_length=1, max_length=500)
|
url: str | None = Field(default=None, min_length=1, max_length=500)
|
||||||
path: str | None = Field(default=None, min_length=1, max_length=2000)
|
path: str | None = Field(default=None, min_length=1, max_length=2000)
|
||||||
|
ignore_paths: list[str] | None = Field(default=None)
|
||||||
|
|
||||||
@field_validator("url", mode="before")
|
@field_validator("url", mode="before")
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -352,12 +359,14 @@ class GitSourceOut(BaseModel):
|
|||||||
``kind=local`` rows, the stored (expanded) directory path — the
|
``kind=local`` rows, the stored (expanded) directory path — the
|
||||||
phase-35 response shape is unchanged by phase 38, so a local 201
|
phase-35 response shape is unchanged by phase 38, so a local 201
|
||||||
reports its path in ``url`` and the full row (``kind`` + ``path``)
|
reports its path in ``url`` and the full row (``kind`` + ``path``)
|
||||||
via ``GET``.
|
via ``GET``. ``ignore_paths`` (phase 89) is the stored, normalized
|
||||||
|
list — non-null (a row created without it reports ``[]``).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
id: uuid.UUID | None
|
id: uuid.UUID | None
|
||||||
url: str
|
url: str
|
||||||
added_at: datetime | None
|
added_at: datetime | None
|
||||||
|
ignore_paths: list[str]
|
||||||
|
|
||||||
|
|
||||||
class GitSourceRow(BaseModel):
|
class GitSourceRow(BaseModel):
|
||||||
@@ -369,7 +378,9 @@ class GitSourceRow(BaseModel):
|
|||||||
``path: null``; local rows carry ``path`` (the absolute directory,
|
``path: null``; local rows carry ``path`` (the absolute directory,
|
||||||
expanded) and the same string in ``url`` (the table's NOT-NULL
|
expanded) and the same string in ``url`` (the table's NOT-NULL
|
||||||
location column). ``id`` / ``added_at`` are nullable: env-fallback
|
location column). ``id`` / ``added_at`` are nullable: env-fallback
|
||||||
rows (table empty) carry neither.
|
rows (table empty) carry neither. ``ignore_paths`` (phase 89) is the
|
||||||
|
row's stored, normalized list — env-fallback rows (no DB row to
|
||||||
|
store a list on) report ``[]``.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
id: uuid.UUID | None
|
id: uuid.UUID | None
|
||||||
@@ -377,6 +388,23 @@ class GitSourceRow(BaseModel):
|
|||||||
url: str
|
url: str
|
||||||
path: str | None
|
path: str | None
|
||||||
added_at: datetime | None
|
added_at: datetime | None
|
||||||
|
ignore_paths: list[str]
|
||||||
|
|
||||||
|
|
||||||
|
class GitSourceIgnoreIn(BaseModel):
|
||||||
|
"""``PATCH /api/git-sources/{source_id}`` body (phase 89, A5).
|
||||||
|
|
||||||
|
``ignore_paths`` is REQUIRED — the box's lines in REPLACE semantics:
|
||||||
|
the body list (normalized + A4-validated in the API layer, with the
|
||||||
|
fixed-detail 422s) becomes the row's whole list — an empty list
|
||||||
|
clears all; an absent field is a 422 with the model's own detail.
|
||||||
|
Entries are RAW box lines: normalization + the A4 limits are
|
||||||
|
enforced in the API layer so the 422 details stay fixed strings
|
||||||
|
(the router's credential-safety discipline, applied for
|
||||||
|
consistency).
|
||||||
|
"""
|
||||||
|
|
||||||
|
ignore_paths: list[str]
|
||||||
|
|
||||||
|
|
||||||
class GitSourceList(BaseModel):
|
class GitSourceList(BaseModel):
|
||||||
|
|||||||
@@ -143,6 +143,35 @@
|
|||||||
* gate and never fetches /api/git-sources (the Sources-page gate
|
* gate and never fetches /api/git-sources (the Sources-page gate
|
||||||
* pattern).
|
* pattern).
|
||||||
*
|
*
|
||||||
|
* Phase 89 (task 05) — the per-source ignore-paths editor: every
|
||||||
|
* STORED row (s.id truthy — env-fallback rows carry NO button, A3:
|
||||||
|
* nothing is stored to edit) gets an "Ignore paths" button in the
|
||||||
|
* actions cell (left of Remove) that opens the page-local alertdialog
|
||||||
|
* (#ignore-editor-dialog — the EXACT #remove-confirm-dialog pattern,
|
||||||
|
* phase 69): #ignore-editor-source names the source (textContent ONLY
|
||||||
|
* — the same `value` expression makeRow uses — URLs may embed
|
||||||
|
* user:pass@ credentials, phase 32), the mono textarea prefills the
|
||||||
|
* row's stored list (one path per line — the phase-89 A1 prefix rule,
|
||||||
|
* stated in plain words in the helper copy), and focus lands on
|
||||||
|
* Cancel (the safe default); Escape / Cancel / the dim backdrop close
|
||||||
|
* as CANCEL (no request — focus returns to the row's trigger button).
|
||||||
|
* "Save" runs the §7.4 in-flight lifecycle: both buttons disable +
|
||||||
|
* the save relabels "Saving…" while the PATCH
|
||||||
|
* /api/git-sources/{id} is out (blank lines in the box are separators,
|
||||||
|
* not entries — split + trim + drop empty client-side; the server
|
||||||
|
* still rejects empty entries defensively, A4). 200 → close (focus
|
||||||
|
* return), loadSources (the row's "N ignored" count tag lands — the
|
||||||
|
* A5 replace semantics round-trip through GET), announce — the update
|
||||||
|
* confirmation is the LAST announcement (the reload's "N sources
|
||||||
|
* listed." must not overwrite it); non-2xx → the in-dialog
|
||||||
|
* role="alert" line (the server detail, apiDetail 422-shape-aware) +
|
||||||
|
* the buttons restored, the dialog STAYS open and the textarea
|
||||||
|
* content is KEPT (the instruction survives — tuning-form
|
||||||
|
* convention); network failure → the fixed reachable? line, same
|
||||||
|
* restore. Rows whose list is non-empty show the "N ignored" count
|
||||||
|
* tag next to the location <code> (text + a distinct background —
|
||||||
|
* never color alone, WCAG 1.4.1).
|
||||||
|
*
|
||||||
* Scope boundary (phase locked decisions): adding a git repo does
|
* Scope boundary (phase locked decisions): adding a git repo does
|
||||||
* NOT clone — the sync service (server-side) does that. Removing a
|
* NOT clone — the sync service (server-side) does that. Removing a
|
||||||
* source, however, performs the FULL cleanup server-side (phase 69):
|
* source, however, performs the FULL cleanup server-side (phase 69):
|
||||||
@@ -203,6 +232,17 @@ export async function mount(root) {
|
|||||||
const removeError = root.querySelector("#remove-confirm-error");
|
const removeError = root.querySelector("#remove-confirm-error");
|
||||||
const removeCancelBtn = root.querySelector("#remove-confirm-cancel");
|
const removeCancelBtn = root.querySelector("#remove-confirm-cancel");
|
||||||
const removeRemoveBtn = root.querySelector("#remove-confirm-remove");
|
const removeRemoveBtn = root.querySelector("#remove-confirm-remove");
|
||||||
|
/* Phase 89: the per-source ignore-paths editor — the SAME page-local
|
||||||
|
alertdialog idiom as the remove-confirm modal (phase 69): static
|
||||||
|
markup in the shell's Sources view; this module owns the open /
|
||||||
|
cancel / save lifecycle. */
|
||||||
|
const ignoreDialog = root.querySelector("#ignore-editor-dialog");
|
||||||
|
const ignoreBackdrop = root.querySelector(".ignore-editor-backdrop");
|
||||||
|
const ignoreSourceEl = root.querySelector("#ignore-editor-source");
|
||||||
|
const ignoreTextarea = root.querySelector("#ignore-editor-textarea");
|
||||||
|
const ignoreErrorEl = root.querySelector("#ignore-editor-error");
|
||||||
|
const ignoreCancelBtn = root.querySelector("#ignore-editor-cancel");
|
||||||
|
const ignoreSaveBtn = root.querySelector("#ignore-editor-save");
|
||||||
|
|
||||||
/* Polite live region: the screen-reader confirmation for loads, adds,
|
/* Polite live region: the screen-reader confirmation for loads, adds,
|
||||||
and removals (the phase-15 announcer pattern). */
|
and removals (the phase-15 announcer pattern). */
|
||||||
@@ -321,6 +361,16 @@ export async function mount(root) {
|
|||||||
const code = document.createElement("code");
|
const code = document.createElement("code");
|
||||||
code.textContent = value; // rendered as text, never as HTML
|
code.textContent = value; // rendered as text, never as HTML
|
||||||
urlTd.append(badge, code);
|
urlTd.append(badge, code);
|
||||||
|
/* Phase 89: the "N ignored" count tag — stored rows with a
|
||||||
|
non-empty list only (text + a distinct background, never color
|
||||||
|
alone — WCAG 1.4.1). The list round-trips: (s.ignore_paths ||
|
||||||
|
[]) is the same expression openIgnoreEditor prefills from. */
|
||||||
|
if (s.id && (s.ignore_paths || []).length > 0) {
|
||||||
|
const count = document.createElement("span");
|
||||||
|
count.className = "git-source-ignore-count";
|
||||||
|
count.textContent = `${s.ignore_paths.length} ignored`;
|
||||||
|
urlTd.append(count);
|
||||||
|
}
|
||||||
tr.appendChild(urlTd);
|
tr.appendChild(urlTd);
|
||||||
|
|
||||||
const addedTd = document.createElement("td");
|
const addedTd = document.createElement("td");
|
||||||
@@ -330,6 +380,22 @@ export async function mount(root) {
|
|||||||
const actTd = document.createElement("td");
|
const actTd = document.createElement("td");
|
||||||
actTd.className = "git-source-actions-cell";
|
actTd.className = "git-source-actions-cell";
|
||||||
if (s.id) {
|
if (s.id) {
|
||||||
|
/* Phase 89: the per-row ignore-paths editor trigger (A3: stored
|
||||||
|
rows only — env-fallback rows fall through to the "from
|
||||||
|
.env" tag below). The aria-label is the ONLY place `value`
|
||||||
|
appears (setAttribute — never innerHTML); the text label is
|
||||||
|
static. Sits LEFT of Remove — the non-destructive action
|
||||||
|
first. */
|
||||||
|
const ignoreBtn = document.createElement("button");
|
||||||
|
ignoreBtn.type = "button";
|
||||||
|
ignoreBtn.className = "git-source-ignore";
|
||||||
|
ignoreBtn.setAttribute(
|
||||||
|
"aria-label",
|
||||||
|
`Edit ignored paths for ${kindLabel} source: ${value}`,
|
||||||
|
);
|
||||||
|
ignoreBtn.textContent = "Ignore paths";
|
||||||
|
ignoreBtn.addEventListener("click", () => openIgnoreEditor(s, ignoreBtn));
|
||||||
|
actTd.appendChild(ignoreBtn);
|
||||||
const btn = document.createElement("button");
|
const btn = document.createElement("button");
|
||||||
btn.type = "button";
|
btn.type = "button";
|
||||||
btn.className = "git-source-remove";
|
btn.className = "git-source-remove";
|
||||||
@@ -514,6 +580,168 @@ export async function mount(root) {
|
|||||||
if (removeBackdrop) removeBackdrop.addEventListener("click", cancelRemoveConfirm);
|
if (removeBackdrop) removeBackdrop.addEventListener("click", cancelRemoveConfirm);
|
||||||
if (removeRemoveBtn) removeRemoveBtn.addEventListener("click", confirmRemove);
|
if (removeRemoveBtn) removeRemoveBtn.addEventListener("click", confirmRemove);
|
||||||
|
|
||||||
|
/* ---------- ignore paths (PATCH /api/git-sources/{id}) — the per-row editor ----------
|
||||||
|
* Phase 89: a stored row's "Ignore paths" button (makeRow — left of
|
||||||
|
* Remove) opens the page-local alertdialog
|
||||||
|
* (#ignore-editor-dialog — the EXACT #remove-confirm-dialog idiom,
|
||||||
|
* phase 69) via openIgnoreEditor(s, triggerBtn): #ignore-editor-
|
||||||
|
* source names the source (textContent ONLY — the same `value`
|
||||||
|
* expression makeRow uses: s.path ?? s.url for local rows, s.url
|
||||||
|
* for git — URLs may embed user:pass@ credentials, phase 32), the
|
||||||
|
* mono textarea PREFILLS the row's stored list (one path per line
|
||||||
|
* — the phase-89 A1 prefix rule, stated in plain words in the
|
||||||
|
* helper copy), the error line clears, and focus lands on Cancel
|
||||||
|
* (the safe default). Escape / Cancel / the dim backdrop close as
|
||||||
|
* CANCEL: no request, focus returns to the row's trigger button.
|
||||||
|
*
|
||||||
|
* "Save" (saveIgnorePaths) runs the §7.4 never-stale lifecycle:
|
||||||
|
* both buttons disable and the save relabels "Saving…" while the
|
||||||
|
* PATCH is out — a blank line in the box is a separator, not an
|
||||||
|
* entry (split + trim + drop empty lines client-side; the server
|
||||||
|
* still rejects empty entries defensively, A4). 200 → close
|
||||||
|
* (focus return) → loadSources (the "N ignored" count tag lands —
|
||||||
|
* the A5 replace round-trips through GET) → announce (the update
|
||||||
|
* confirmation is the LAST announcement — the reload's "N sources
|
||||||
|
* listed." must not overwrite it); non-2xx → the in-dialog
|
||||||
|
* role="alert" line (the server detail, apiDetail 422-shape-aware)
|
||||||
|
* + the buttons restored — the dialog STAYS open and the textarea
|
||||||
|
* content is KEPT (the instruction survives — the tuning-form
|
||||||
|
* convention); network failure → the fixed reachable? line, same
|
||||||
|
* restore. Env-fallback rows (id null) carry NO button (A3 —
|
||||||
|
* nothing is stored to edit). */
|
||||||
|
let ignoreTarget = null; // the row object of the open editor
|
||||||
|
let ignoreTriggerBtn = null; // the row's button — focus returns here on close
|
||||||
|
let ignoreInFlight = false; // §7.4: a PATCH is out (both buttons disabled)
|
||||||
|
|
||||||
|
function openIgnoreEditor(s, triggerBtn) {
|
||||||
|
if (!ignoreDialog || !ignoreTextarea) return; // defensive — the markup ships with the page
|
||||||
|
if (ignoreInFlight) return; // one editor at a time
|
||||||
|
const isLocal = s.kind === "local";
|
||||||
|
// The same `value` expression makeRow uses — textContent ONLY.
|
||||||
|
if (ignoreSourceEl) ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url;
|
||||||
|
ignoreTextarea.value = (s.ignore_paths || []).join("\n");
|
||||||
|
if (ignoreErrorEl) {
|
||||||
|
ignoreErrorEl.textContent = "";
|
||||||
|
ignoreErrorEl.hidden = true; // a new attempt starts clean
|
||||||
|
}
|
||||||
|
ignoreInFlight = false;
|
||||||
|
ignoreTarget = s;
|
||||||
|
ignoreTriggerBtn = triggerBtn; // recorded for the focus return on close
|
||||||
|
ignoreDialog.hidden = false;
|
||||||
|
document.addEventListener("keydown", onIgnoreDialogKeydown);
|
||||||
|
// Cancel is the safe default — focus lands on it (the remove-
|
||||||
|
// dialog precedent; visibly: the global 3px :focus-visible
|
||||||
|
// outline).
|
||||||
|
if (ignoreCancelBtn) ignoreCancelBtn.focus();
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Any close (cancel, success): hide the dialog, reset the textarea
|
||||||
|
+ error line, clear the target, detach the keydown handling, and
|
||||||
|
return focus to the row's "Ignore paths" button (WCAG 2.1). */
|
||||||
|
function closeIgnoreEditor() {
|
||||||
|
if (!ignoreDialog) return;
|
||||||
|
ignoreDialog.hidden = true;
|
||||||
|
ignoreInFlight = false;
|
||||||
|
if (ignoreTextarea) ignoreTextarea.value = "";
|
||||||
|
if (ignoreErrorEl) {
|
||||||
|
ignoreErrorEl.textContent = "";
|
||||||
|
ignoreErrorEl.hidden = true;
|
||||||
|
}
|
||||||
|
if (ignoreCancelBtn) ignoreCancelBtn.disabled = false;
|
||||||
|
if (ignoreSaveBtn) {
|
||||||
|
ignoreSaveBtn.disabled = false;
|
||||||
|
ignoreSaveBtn.textContent = "Save";
|
||||||
|
}
|
||||||
|
document.removeEventListener("keydown", onIgnoreDialogKeydown);
|
||||||
|
const trigger = ignoreTriggerBtn;
|
||||||
|
ignoreTarget = null;
|
||||||
|
ignoreTriggerBtn = null;
|
||||||
|
if (trigger) trigger.focus(); // focus returns to the row's button
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Escape / Cancel / backdrop all close as cancel — NO request. A
|
||||||
|
cancel is a no-op while a PATCH is in flight (no half-cancel of
|
||||||
|
an in-progress save; the buttons are disabled anyway, the
|
||||||
|
Escape/backdrop paths need this guard). */
|
||||||
|
function cancelIgnoreEditor() {
|
||||||
|
if (ignoreInFlight) return;
|
||||||
|
closeIgnoreEditor();
|
||||||
|
}
|
||||||
|
|
||||||
|
/* While open (attached in openIgnoreEditor, detached in
|
||||||
|
closeIgnoreEditor): Escape closes as cancel. */
|
||||||
|
function onIgnoreDialogKeydown(e) {
|
||||||
|
if (e.key === "Escape") {
|
||||||
|
e.preventDefault();
|
||||||
|
cancelIgnoreEditor();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveIgnorePaths() {
|
||||||
|
if (!ignoreTarget || ignoreInFlight) return; // one request at a time
|
||||||
|
// The box's lines: one path per line — a blank line is a
|
||||||
|
// separator, not an entry (trim + drop empty; the server still
|
||||||
|
// rejects empties defensively, A4).
|
||||||
|
const lines = ignoreTextarea.value.split("\n").map((l) => l.trim()).filter(Boolean);
|
||||||
|
const t = ignoreTarget;
|
||||||
|
const value = t.kind === "local" ? (t.path ?? t.url) : t.url;
|
||||||
|
ignoreInFlight = true;
|
||||||
|
if (ignoreErrorEl) {
|
||||||
|
ignoreErrorEl.textContent = "";
|
||||||
|
ignoreErrorEl.hidden = true;
|
||||||
|
}
|
||||||
|
if (ignoreCancelBtn) ignoreCancelBtn.disabled = true;
|
||||||
|
if (ignoreSaveBtn) {
|
||||||
|
ignoreSaveBtn.disabled = true;
|
||||||
|
ignoreSaveBtn.textContent = "Saving…"; // §7.4 in-flight label
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const r = await fetch(`/api/git-sources/${encodeURIComponent(t.id)}`, {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ ignore_paths: lines }),
|
||||||
|
});
|
||||||
|
if (r.ok) {
|
||||||
|
// 200: the server replaced the row's list (A5).
|
||||||
|
closeIgnoreEditor(); // focus returns to the row's button
|
||||||
|
await loadSources(); // the "N ignored" count tag lands
|
||||||
|
// The update confirmation is the LAST announcement: the
|
||||||
|
// reload's "N sources listed." must not overwrite it (the
|
||||||
|
// same order as the remove flow).
|
||||||
|
announce(`Ignored paths updated for ${value}`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// non-2xx: the in-dialog role="alert" line (the server detail)
|
||||||
|
// — the dialog STAYS open and the textarea content is KEPT:
|
||||||
|
// the fix is one edit + retry, not a re-type.
|
||||||
|
if (ignoreErrorEl) {
|
||||||
|
ignoreErrorEl.textContent = await apiDetail(r, "Could not save the ignored paths — try again.");
|
||||||
|
ignoreErrorEl.hidden = false;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
if (ignoreErrorEl) {
|
||||||
|
ignoreErrorEl.textContent = "Could not save the ignored paths — is the app reachable?";
|
||||||
|
ignoreErrorEl.hidden = false;
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
// Never stale (PLAN §7.4): the failure paths re-enable BOTH
|
||||||
|
// buttons + relabel the save; the success path already closed
|
||||||
|
// the dialog (which resets them) — the restore is a no-op
|
||||||
|
// there.
|
||||||
|
ignoreInFlight = false;
|
||||||
|
if (ignoreCancelBtn) ignoreCancelBtn.disabled = false;
|
||||||
|
if (ignoreSaveBtn) {
|
||||||
|
ignoreSaveBtn.disabled = false;
|
||||||
|
ignoreSaveBtn.textContent = "Save";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The dialog's own buttons (static markup — wired once). */
|
||||||
|
if (ignoreCancelBtn) ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor);
|
||||||
|
if (ignoreBackdrop) ignoreBackdrop.addEventListener("click", cancelIgnoreEditor);
|
||||||
|
if (ignoreSaveBtn) ignoreSaveBtn.addEventListener("click", saveIgnorePaths);
|
||||||
|
|
||||||
/* ---------- add (POST /api/git-sources) — the git form ----------
|
/* ---------- add (POST /api/git-sources) — the git form ----------
|
||||||
* wireAddForm gives the form the §7.4 never-stale lifecycle: while
|
* wireAddForm gives the form the §7.4 never-stale lifecycle: while
|
||||||
* the request is out the button disables + relabels "Adding…" and
|
* the request is out the button disables + relabels "Adding…" and
|
||||||
|
|||||||
@@ -2183,6 +2183,173 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
|||||||
color: var(--bg);
|
color: var(--bg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---------- Per-source ignore-paths editor (phase 89, task 05) ----------
|
||||||
|
The shell's Sources view: the page-local alertdialog that edits
|
||||||
|
one source's ignore list (one path per line — the phase-89 A1
|
||||||
|
prefix rule, stated in plain words in the helper copy). The EXACT
|
||||||
|
#remove-confirm-dialog overlay contract (phase 69): a fixed
|
||||||
|
full-viewport dim backdrop + a centered panel (z-index 1000, above
|
||||||
|
the sticky header (20) + skip-link (100); NO blur — the phase-08
|
||||||
|
no-blur perf anchor), scaled to the 46rem chat-column width or the
|
||||||
|
viewport, whichever is narrower. The box: a VISIBLE block label
|
||||||
|
(WCAG — never aria-label-only) over a mono textarea (the box is a
|
||||||
|
data entry, not prose — the var(--mono) stack); the error line is
|
||||||
|
the err pair (err-ink on err-bg 9.3:1, the err-line border);
|
||||||
|
Cancel is the ghost ink-soft family (5.1:1 on --surface); Save is
|
||||||
|
the solid brand family (--bg text on --brand 5.2:1 — the
|
||||||
|
.new-chat-btn convention; the hover lightens the fill). Both
|
||||||
|
buttons >=44px; :focus-visible via the global 3px outline rule (no
|
||||||
|
local suppression — focus lands on Cancel at open and is visible);
|
||||||
|
no animation (reduced-motion safe by construction). AA pairs: title
|
||||||
|
/ copy / label are --ink (or ink-soft) on --surface (13.8:1 /
|
||||||
|
5.1:1); the source value is --ink on --bg (16.7:1). Phase-08
|
||||||
|
tokens only; system fonts; no CDN. */
|
||||||
|
.ignore-editor {
|
||||||
|
position: fixed;
|
||||||
|
inset: 0;
|
||||||
|
z-index: 1000;
|
||||||
|
display: flex; /* the panel is the only in-flow child — margin: auto centers it */
|
||||||
|
}
|
||||||
|
/* Explicit (the global [hidden] rule already wins — the documented,
|
||||||
|
testable contract for the skeleton). */
|
||||||
|
.ignore-editor[hidden] { display: none; }
|
||||||
|
|
||||||
|
.ignore-editor-backdrop {
|
||||||
|
position: fixed;
|
||||||
|
inset: 0;
|
||||||
|
/* --bg at 82% — the doc-modal dim, no backdrop-filter (no-blur). */
|
||||||
|
background: rgba(15, 10, 10, 0.82);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ignore-editor-panel {
|
||||||
|
/* position:relative lifts the panel above the fixed backdrop
|
||||||
|
(positioned elements paint over in-flow siblings otherwise). */
|
||||||
|
position: relative;
|
||||||
|
z-index: 1;
|
||||||
|
margin: auto;
|
||||||
|
width: min(46rem, calc(100vw - 2rem));
|
||||||
|
padding: 1.5rem;
|
||||||
|
background: var(--surface);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: var(--radius);
|
||||||
|
box-shadow: var(--shadow-lg);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ignore-editor-title {
|
||||||
|
margin: 0 0 0.75rem;
|
||||||
|
font-size: 1.25rem;
|
||||||
|
line-height: 1.3;
|
||||||
|
color: var(--ink);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The source's value (git URL or local path) — mono, wrapped (a long
|
||||||
|
URL must not overflow the panel), --ink on --bg (16.7:1). */
|
||||||
|
.ignore-editor-source {
|
||||||
|
display: block;
|
||||||
|
margin: 0 0 0.75rem;
|
||||||
|
padding: 0.5rem 0.65rem;
|
||||||
|
font-family: var(--mono);
|
||||||
|
font-size: 0.85rem;
|
||||||
|
line-height: 1.5;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--bg);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: var(--radius-sm);
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
|
||||||
|
.ignore-editor-copy {
|
||||||
|
margin: 0 0 1rem;
|
||||||
|
color: var(--ink); /* 13.8:1 on --surface */
|
||||||
|
}
|
||||||
|
.ignore-editor-copy code { font-family: var(--mono); font-size: 0.85em; }
|
||||||
|
|
||||||
|
/* Visible block label (WCAG — the textarea is NEVER labelled only via
|
||||||
|
aria-label); ink-soft on --surface (5.1:1). */
|
||||||
|
.ignore-editor-label {
|
||||||
|
display: block;
|
||||||
|
margin-bottom: 0.35rem;
|
||||||
|
color: var(--ink-soft);
|
||||||
|
font-size: 0.88rem;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The box: full panel width, mono (the var(--mono) stack), theme-
|
||||||
|
consistent border on --bg; the focus-visible ring is the global
|
||||||
|
3px outline (no local suppression); vertical resize for long
|
||||||
|
lists (up to 200 entries, A4). */
|
||||||
|
.ignore-editor-textarea {
|
||||||
|
display: block;
|
||||||
|
width: 100%;
|
||||||
|
min-height: 9rem;
|
||||||
|
padding: 0.55rem 0.65rem;
|
||||||
|
font-family: var(--mono);
|
||||||
|
font-size: 0.85rem;
|
||||||
|
line-height: 1.5;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--bg);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: var(--radius-sm);
|
||||||
|
resize: vertical;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The in-dialog failure line (role=alert): the err pair (err-ink on
|
||||||
|
err-bg 9.3:1, the err-line border) — the .remove-confirm-error
|
||||||
|
language. */
|
||||||
|
.ignore-editor-error {
|
||||||
|
margin: 0.75rem 0 0;
|
||||||
|
padding: 0.5rem 0.65rem;
|
||||||
|
color: var(--err-ink);
|
||||||
|
background: var(--err-bg);
|
||||||
|
border: 1px solid var(--err-line);
|
||||||
|
border-radius: var(--radius-sm);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ignore-editor-actions {
|
||||||
|
display: flex;
|
||||||
|
justify-content: flex-end;
|
||||||
|
gap: 0.6rem;
|
||||||
|
margin-top: 1.1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The two dialog buttons: >=44px targets, the house 3px :focus-visible
|
||||||
|
via the global outline rule (no local override). */
|
||||||
|
.ignore-editor-btn {
|
||||||
|
min-height: 44px;
|
||||||
|
min-width: 44px;
|
||||||
|
padding: 0.55rem 1.1rem;
|
||||||
|
border-radius: var(--radius-sm);
|
||||||
|
font: inherit;
|
||||||
|
font-weight: 600;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
white-space: nowrap;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.ignore-editor-btn:disabled { opacity: 0.5; cursor: wait; }
|
||||||
|
|
||||||
|
/* Cancel — the ghost ink-soft family (5.1:1 on --surface), like the
|
||||||
|
remove dialog's Cancel; the brand pair on hover (12.4:1 on
|
||||||
|
brand-soft). */
|
||||||
|
.ignore-editor-cancel {
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: transparent;
|
||||||
|
color: var(--ink-soft);
|
||||||
|
}
|
||||||
|
.ignore-editor-cancel:hover:not(:disabled) {
|
||||||
|
background: var(--brand-soft);
|
||||||
|
color: var(--brand-ink);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* "Save" — the solid brand family (the .new-chat-btn convention):
|
||||||
|
--bg text on --brand (5.2:1 — AA); the hover lightens the brand
|
||||||
|
fill (the .new-chat-btn hover pair). */
|
||||||
|
.ignore-editor-save {
|
||||||
|
border: 0;
|
||||||
|
background: var(--brand);
|
||||||
|
color: var(--bg);
|
||||||
|
}
|
||||||
|
.ignore-editor-save:hover:not(:disabled) { background: #f55a72; }
|
||||||
|
|
||||||
/* The list: the Sources page's table pattern — full width in the
|
/* The list: the Sources page's table pattern — full width in the
|
||||||
72rem frame, surface card, horizontally scrollable wrapper (the
|
72rem frame, surface card, horizontally scrollable wrapper (the
|
||||||
URL column never wraps or ellipsizes: long URLs, credentials
|
URL column never wraps or ellipsizes: long URLs, credentials
|
||||||
@@ -2266,6 +2433,54 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
|||||||
.git-source-remove:hover:not(:disabled) { background: var(--err-bg); color: var(--err-ink); border-color: var(--err-line); }
|
.git-source-remove:hover:not(:disabled) { background: var(--err-bg); color: var(--err-ink); border-color: var(--err-line); }
|
||||||
.git-source-remove:disabled { opacity: 0.5; cursor: wait; }
|
.git-source-remove:disabled { opacity: 0.5; cursor: wait; }
|
||||||
|
|
||||||
|
/* Phase 89: the per-row "Ignore paths" trigger (makeRow — left of
|
||||||
|
Remove): the .git-source-remove idiom (same size/spacing, >=44px
|
||||||
|
target) in a NEUTRAL secondary fill — it must read distinct from
|
||||||
|
the destructive Remove (text label included, never icon-only; the
|
||||||
|
hover takes the brand-soft pair, 12.4:1 — Remove hovers to the err
|
||||||
|
pair, 9.3:1). */
|
||||||
|
.git-source-ignore {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
min-height: 44px;
|
||||||
|
min-width: 44px;
|
||||||
|
flex: 0 0 auto;
|
||||||
|
padding: 0.35rem 0.7rem;
|
||||||
|
margin-right: 0.4rem;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: var(--radius-sm);
|
||||||
|
background: transparent;
|
||||||
|
color: var(--ink-soft);
|
||||||
|
font: inherit;
|
||||||
|
font-weight: 600;
|
||||||
|
font-size: 0.82rem;
|
||||||
|
white-space: nowrap;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.git-source-ignore:hover:not(:disabled) {
|
||||||
|
background: var(--brand-soft);
|
||||||
|
color: var(--brand-ink);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Phase 89: the "N ignored" count tag — next to the location <code>
|
||||||
|
in the Source cell (TEXT + a distinct background, never color
|
||||||
|
alone — WCAG 1.4.1): --ink on --bg (16.7:1) inside the --surface
|
||||||
|
card; the line border keeps the chip legible when the row hover
|
||||||
|
swaps the cell onto --bg too. */
|
||||||
|
.git-source-ignore-count {
|
||||||
|
display: inline-block;
|
||||||
|
margin-left: 0.55rem;
|
||||||
|
padding: 0.08rem 0.5rem;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 999px;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
font-weight: 600;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--bg);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
/* Env-fallback rows carry no Remove (nothing is stored to remove) —
|
/* Env-fallback rows carry no Remove (nothing is stored to remove) —
|
||||||
the tag says where the row comes from (brand pair, 6.9:1). */
|
the tag says where the row comes from (brand pair, 6.9:1). */
|
||||||
.git-source-env-tag {
|
.git-source-env-tag {
|
||||||
|
|||||||
@@ -647,6 +647,44 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<!-- Phase 89: the per-source ignore-paths editor — the EXACT
|
||||||
|
#remove-confirm-dialog pattern (phase 69): page-local
|
||||||
|
alertdialog, dim backdrop, focus on Cancel (the safe
|
||||||
|
default), Escape/backdrop close as CANCEL, focus returns
|
||||||
|
to the row's trigger button. The box itself: one path
|
||||||
|
per line (prefix semantics — the phase-89 A1 rule,
|
||||||
|
stated in plain words in the helper line); Save runs the
|
||||||
|
§7.4 never-stale lifecycle (git-sources.js: "Saving…"
|
||||||
|
while the PATCH is out). Stored rows only (A3) — the
|
||||||
|
per-row button lives in git-sources.js's makeRow; this
|
||||||
|
is the page-local dialog it opens. -->
|
||||||
|
<div class="ignore-editor" id="ignore-editor-dialog" role="alertdialog"
|
||||||
|
aria-modal="true" aria-labelledby="ignore-editor-title"
|
||||||
|
aria-describedby="ignore-editor-copy" hidden>
|
||||||
|
<div class="ignore-editor-backdrop" aria-hidden="true"></div>
|
||||||
|
<div class="ignore-editor-panel">
|
||||||
|
<h2 class="ignore-editor-title" id="ignore-editor-title">Ignored files and folders</h2>
|
||||||
|
<code class="ignore-editor-source" id="ignore-editor-source"></code>
|
||||||
|
<p class="ignore-editor-copy" id="ignore-editor-copy">
|
||||||
|
One path per line. A file is ignored when its path in the
|
||||||
|
source starts with any of these — <code>/my/files/</code>,
|
||||||
|
<code>my/files/</code> and <code>my/files</code> mean the same
|
||||||
|
thing. No wildcards, no matching in the middle of a path.
|
||||||
|
</p>
|
||||||
|
<label class="ignore-editor-label" for="ignore-editor-textarea">Ignored paths</label>
|
||||||
|
<textarea class="ignore-editor-textarea" id="ignore-editor-textarea"
|
||||||
|
rows="6" spellcheck="false"
|
||||||
|
placeholder="my/files/"></textarea>
|
||||||
|
<p class="ignore-editor-error" id="ignore-editor-error" role="alert" hidden></p>
|
||||||
|
<div class="ignore-editor-actions">
|
||||||
|
<button type="button" class="ignore-editor-btn ignore-editor-cancel"
|
||||||
|
id="ignore-editor-cancel">Cancel</button>
|
||||||
|
<button type="button" class="ignore-editor-btn ignore-editor-save"
|
||||||
|
id="ignore-editor-save">Save</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<!-- Polite live region: the screen-reader confirmation for list
|
<!-- Polite live region: the screen-reader confirmation for list
|
||||||
loads, adds, and removals (git-sources.js owns the text). -->
|
loads, adds, and removals (git-sources.js owns the text). -->
|
||||||
|
|||||||
+37
-13
@@ -26,6 +26,11 @@ precedence order:
|
|||||||
``~/Deployments``), kept for backwards compatibility (reached only
|
``~/Deployments``), kept for backwards compatibility (reached only
|
||||||
while both the table and ``BOR_GIT_SOURCES`` are empty).
|
while both the table and ``BOR_GIT_SOURCES`` are empty).
|
||||||
|
|
||||||
|
Phase 89: resolution also returns each row's ignore paths, keyed by
|
||||||
|
the resolved root string (the importer normalizes them); manual
|
||||||
|
``--source`` dirs and the legacy fallback have no rows, so they import
|
||||||
|
with no ignore.
|
||||||
|
|
||||||
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
|
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
|
||||||
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
|
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
|
||||||
family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add
|
family and ``j2`` (case-insensitive). ``BOR_IMPORT_EXTENSIONS`` may add
|
||||||
@@ -133,9 +138,11 @@ def repo_name(url: str) -> str:
|
|||||||
return slug or "repo"
|
return slug or "repo"
|
||||||
|
|
||||||
|
|
||||||
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
|
def _resolve_sources(
|
||||||
|
cli_sources: list[Path] | None, settings: Settings
|
||||||
|
) -> tuple[list[Path], dict[str, list[str]]]:
|
||||||
"""Resolve the directories to import (phase 28, extended in phases
|
"""Resolve the directories to import (phase 28, extended in phases
|
||||||
35 and 38).
|
35 and 38; per-root ignore maps, phase 89).
|
||||||
|
|
||||||
Precedence: ``--source`` (explicit manual paths — always wins) >
|
Precedence: ``--source`` (explicit manual paths — always wins) >
|
||||||
the effective sources — the ``git_sources`` DB rows (git + local),
|
the effective sources — the ``git_sources`` DB rows (git + local),
|
||||||
@@ -148,12 +155,18 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
|
|||||||
stored directory, re-verified ``.is_dir()`` at run time) > the
|
stored directory, re-verified ``.is_dir()`` at run time) > the
|
||||||
legacy ``DEFAULT_SOURCES``.
|
legacy ``DEFAULT_SOURCES``.
|
||||||
|
|
||||||
|
Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed
|
||||||
|
by the resolved root string, exactly as the importer sees it (two
|
||||||
|
rows sharing a root string get the union — extend, not replace);
|
||||||
|
manual ``--source`` dirs and the legacy fallback have no rows, so
|
||||||
|
they import with an empty map (no ignore).
|
||||||
|
|
||||||
A :class:`GitSyncError` from a failing clone/pull — or a missing
|
A :class:`GitSyncError` from a failing clone/pull — or a missing
|
||||||
local directory (``local source missing: <path>``) — propagates to
|
local directory (``local source missing: <path>``) — propagates to
|
||||||
:func:`main`, which aborts the run before importing anything.
|
:func:`main`, which aborts the run before importing anything.
|
||||||
"""
|
"""
|
||||||
if cli_sources:
|
if cli_sources:
|
||||||
return [path.expanduser() for path in cli_sources]
|
return [path.expanduser() for path in cli_sources], {}
|
||||||
db = SessionLocal()
|
db = SessionLocal()
|
||||||
try:
|
try:
|
||||||
rows, origin = effective_sources(db)
|
rows, origin = effective_sources(db)
|
||||||
@@ -167,21 +180,28 @@ def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list
|
|||||||
)
|
)
|
||||||
sources_root = Path(settings.sources_dir).expanduser()
|
sources_root = Path(settings.sources_dir).expanduser()
|
||||||
sources: list[Path] = []
|
sources: list[Path] = []
|
||||||
|
ignore_by_root: dict[str, list[str]] = {}
|
||||||
for row in rows:
|
for row in rows:
|
||||||
if row.kind == "git":
|
if row.kind == "git":
|
||||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
|
||||||
else:
|
else:
|
||||||
# kind=local — the stored expanded path (phase 38 also
|
# kind=local — the stored expanded path (phase 38 also
|
||||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||||
# ``or`` keeps the type checker honest); a missing
|
# ``or`` keeps the type checker honest); a missing
|
||||||
# directory aborts before importing, the same pre-import
|
# directory aborts before importing, the same pre-import
|
||||||
# fail-loud as a failing git clone.
|
# fail-loud as a failing git clone.
|
||||||
path = Path(row.path or row.url).expanduser()
|
root = Path(row.path or row.url).expanduser()
|
||||||
if not path.is_dir():
|
if not root.is_dir():
|
||||||
raise GitSyncError(f"local source missing: {path}")
|
raise GitSyncError(f"local source missing: {root}")
|
||||||
sources.append(path)
|
sources.append(root)
|
||||||
return sources
|
# Phase 89: the row's ignore list, keyed by the SAME root
|
||||||
return [path.expanduser() for path in DEFAULT_SOURCES]
|
# string the importer sees; two rows sharing a root string
|
||||||
|
# get the union (extend, not replace) — the sibling/repo-name
|
||||||
|
# edge.
|
||||||
|
if row.ignore_paths:
|
||||||
|
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
|
||||||
|
return sources, ignore_by_root
|
||||||
|
return [path.expanduser() for path in DEFAULT_SOURCES], {}
|
||||||
|
|
||||||
|
|
||||||
def _overview_row_exists() -> bool:
|
def _overview_row_exists() -> bool:
|
||||||
@@ -204,9 +224,10 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
|
|
||||||
# Git sources resolve (and clone/pull) *before* any import: a failing
|
# Git sources resolve (and clone/pull) *before* any import: a failing
|
||||||
# repo aborts the run with a non-zero exit, naming the failure — a bad
|
# repo aborts the run with a non-zero exit, naming the failure — a bad
|
||||||
# URL must never silently import partial junk.
|
# URL must never silently import partial junk. The second element is
|
||||||
|
# the phase-89 per-root ignore map (empty for manual/fallback paths).
|
||||||
try:
|
try:
|
||||||
sources = _resolve_sources(args.source, settings)
|
sources, ignore_by_root = _resolve_sources(args.source, settings)
|
||||||
except GitSyncError as e:
|
except GitSyncError as e:
|
||||||
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
|
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
@@ -248,7 +269,10 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
runs and unchanged re-runs never bump. The returned token is
|
runs and unchanged re-runs never bump. The returned token is
|
||||||
the new version, or ``"skipped"``.
|
the new version, or ``"skipped"``.
|
||||||
"""
|
"""
|
||||||
summary = await import_sources(sources, llm, prune=args.prune, limit=args.limit)
|
summary = await import_sources(
|
||||||
|
sources, llm, prune=args.prune, limit=args.limit,
|
||||||
|
ignore_by_root=ignore_by_root,
|
||||||
|
)
|
||||||
if args.limit is not None:
|
if args.limit is not None:
|
||||||
# An incomplete walk is debug-only — it must never advance
|
# An incomplete walk is debug-only — it must never advance
|
||||||
# the generation (mirrors the --limit overview skip below).
|
# the generation (mirrors the --limit overview skip below).
|
||||||
|
|||||||
@@ -0,0 +1,640 @@
|
|||||||
|
"""Phase 89 story E2E (Playwright): per-source ignore paths — the box on
|
||||||
|
the Sources page excludes files/folders from the import (``/git-sources.html``).
|
||||||
|
|
||||||
|
Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-08 —
|
||||||
|
TODO-derived, no separate user-story file).
|
||||||
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov
|
||||||
|
|
||||||
|
The story gate proves the whole item end to end through the REAL page +
|
||||||
|
REAL API + REAL sync pipeline (the in-app Sync's background task, mock
|
||||||
|
LLM). **No git, no network** — the suite's single source is a local
|
||||||
|
directory row (``kind="local"``, phase 38), so the sync walks the
|
||||||
|
fixture dir directly: no clone, no pull, no remote of any kind. The
|
||||||
|
module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL
|
||||||
|
(the ``test_git_sources_admin.py`` module-env pattern) that is NEVER
|
||||||
|
synced or cloned: every sync in this suite runs with the local DB row
|
||||||
|
in place (DB rows win over the env fallback), and the one
|
||||||
|
env-fallback test never triggers a sync.
|
||||||
|
|
||||||
|
Contract under test:
|
||||||
|
|
||||||
|
* anonymous: the sign-in gate (the ``#git-sources-gate`` pattern), the
|
||||||
|
manager hidden, NO ``/api/git-sources`` call on load, 403 on
|
||||||
|
``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}``
|
||||||
|
(the ``test_git_sources_admin.py`` anonymous pin, extended to the
|
||||||
|
phase-89 route);
|
||||||
|
* admin: the per-row "Ignore paths" button opens the page-local
|
||||||
|
alertdialog (the phase-69 ``#remove-confirm-dialog`` idiom); one path
|
||||||
|
per line; Save → ``PATCH`` 200 → the row shows the "N ignored" count
|
||||||
|
tag; the list round-trips through ``GET /api/git-sources``;
|
||||||
|
* the sync honors the list: with ``ignore/`` set, nothing under
|
||||||
|
``ignore/`` is indexed — no ``documents`` rows (and therefore no
|
||||||
|
chunks/embeddings, no summary calls) — and the sync's ``files``
|
||||||
|
count excludes the ignored files from the walk;
|
||||||
|
* the spec's no-mid-path rule: ``myfile.md`` ignores the root-level
|
||||||
|
``myfile.md`` only, never ``sub/myfile.md``; saving the box
|
||||||
|
REPLACES the list (A5 — the previous test's list is gone);
|
||||||
|
* A2: a previously indexed file that newly matches an ignore pattern
|
||||||
|
is PRUNED from the KB on the next sync (``detail.pruned``);
|
||||||
|
* a11y + the error path: the dialog is a real ``role="alertdialog"``,
|
||||||
|
the textarea has a visible label, focus lands on Cancel, Escape
|
||||||
|
closes and returns focus to the trigger; a 501-char entry 422s with
|
||||||
|
the fixed detail, the textarea content is KEPT and the Save button
|
||||||
|
re-enables with its "Save" label; the happy path heals the error
|
||||||
|
state;
|
||||||
|
* A3: env-fallback rows (table empty → ``BOR_GIT_SOURCES``) render the
|
||||||
|
"from .env" tag with NO "Ignore paths" button and the env note.
|
||||||
|
|
||||||
|
Test → contract mapping:
|
||||||
|
1. ``test_anonymous_gate_and_403s``
|
||||||
|
2. ``test_ignore_box_excludes_from_import``
|
||||||
|
3. ``test_prefix_rule_no_mid_path``
|
||||||
|
4. ``test_newly_ignored_file_is_pruned``
|
||||||
|
5. ``test_editor_a11y_and_error_path``
|
||||||
|
6. ``test_env_fallback_rows_have_no_box``
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from playwright.sync_api import Page, expect
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from e2e.auth_helpers import login
|
||||||
|
from e2e.conftest import (
|
||||||
|
ADMIN_PASSWORD,
|
||||||
|
SESSION_SECRET,
|
||||||
|
USE_REAL_LLM,
|
||||||
|
_wait_http,
|
||||||
|
)
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
|
||||||
|
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||||
|
# port in a combined run — this module app binds its own port instead
|
||||||
|
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||||
|
# server). Env-overridable.
|
||||||
|
APP_PORT = int(os.environ.get("E2E_APP_PORT_IGNORE", "8140"))
|
||||||
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||||
|
|
||||||
|
GIT_SOURCES_URL = "/git-sources.html"
|
||||||
|
|
||||||
|
#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is
|
||||||
|
#: NEVER cloned (the env-fallback test reads it; the sync tests all run
|
||||||
|
#: with the local DB row in place, so the env list never reaches a
|
||||||
|
#: clone).
|
||||||
|
ENV_SOURCE = "https://github.com/reese/env-alpha.git"
|
||||||
|
|
||||||
|
#: The sentinel inside the fixture's keeper doc — proof the (non-
|
||||||
|
#: ignored) markdown path was really imported.
|
||||||
|
SENTINEL = "RESE-IGNORE-KEEP-7f3a"
|
||||||
|
|
||||||
|
#: The six fixture files (source-relative POSIX paths — exactly the
|
||||||
|
#: strings ``documents.path`` stores).
|
||||||
|
KEEP_MD = "keep.md"
|
||||||
|
NOTES_YAML = "notes.yaml"
|
||||||
|
ROOT_MYFILE = "myfile.md"
|
||||||
|
SUB_MYFILE = "sub/myfile.md"
|
||||||
|
IGNORE_SECRET = "ignore/secret.md"
|
||||||
|
IGNORE_DEEP = "ignore/deep/x.txt"
|
||||||
|
ALL_SIX = (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE, IGNORE_SECRET, IGNORE_DEEP)
|
||||||
|
|
||||||
|
#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the
|
||||||
|
#: test_sync_button.py polling idiom) — real import of ≤6 small files
|
||||||
|
#: against the mock LLM; generous budget.
|
||||||
|
SYNC_TIMEOUT_S = 60.0
|
||||||
|
SYNC_TICK_S = 2.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fixtures
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||||
|
"""The local source's on-disk tree (task 06 step 1).
|
||||||
|
|
||||||
|
Module-scoped (the same reasoning as ``test_sync_button.py``'s
|
||||||
|
module-scoped fixture note): ``tmp_path`` is function-scoped while
|
||||||
|
the module-scoped app + the seeded row reference the dir for the
|
||||||
|
module's lifetime, so it is built under ``tmp_path_factory`` (the
|
||||||
|
same pytest-managed temp area, module-safe). Plain files — no git:
|
||||||
|
a ``kind="local"`` row is walked directly by the sync (phase 38).
|
||||||
|
"""
|
||||||
|
root = tmp_path_factory.mktemp("ignore_src")
|
||||||
|
(root / "sub").mkdir()
|
||||||
|
(root / "ignore" / "deep").mkdir(parents=True)
|
||||||
|
(root / KEEP_MD).write_text(
|
||||||
|
"# Keep\n"
|
||||||
|
"\n"
|
||||||
|
"The keeper doc — the one markdown file the ignore list must\n"
|
||||||
|
"never touch.\n"
|
||||||
|
f"\n"
|
||||||
|
f"Marker: {SENTINEL}\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
# Non-markdown → the phase-30 lite-summary path (proves "not
|
||||||
|
# summarized" applies to the non-md branch too when ignored).
|
||||||
|
(root / NOTES_YAML).write_text(
|
||||||
|
"title: Notes\nitems:\n - one\n - two\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
(root / ROOT_MYFILE).write_text(
|
||||||
|
"# Root myfile\n\nRoot-level file — matched by the bare "
|
||||||
|
"``myfile.md`` prefix (the spec's own example).\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(root / SUB_MYFILE).write_text(
|
||||||
|
"# Sub myfile\n\nA namesake in a subdirectory — NEVER matched "
|
||||||
|
"by ``myfile.md`` (no mid-path matching).\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(root / IGNORE_SECRET).write_text(
|
||||||
|
"# Secret\n\nIgnored by the ``ignore/`` prefix.\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
(root / IGNORE_DEEP).write_text("deep ignored text\n", encoding="utf-8")
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]:
|
||||||
|
"""The real app under test — per-module env (the conftest pattern,
|
||||||
|
module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env
|
||||||
|
fallback's subject — never cloned), module-scratch checkouts/
|
||||||
|
upload dirs, the mock LLM. No git anywhere in this suite's path."""
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
env["BOR_ENVIRONMENT"] = "e2e"
|
||||||
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||||
|
env["BOR_LLM_BASE_URL"] = (
|
||||||
|
"https://aipi.reeseapps.com/v1"
|
||||||
|
if USE_REAL_LLM
|
||||||
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||||
|
)
|
||||||
|
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
||||||
|
# ever sent in this suite, but the app boots with the same env
|
||||||
|
# shape.
|
||||||
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||||
|
env.setdefault(
|
||||||
|
"BOR_DATABASE_URL",
|
||||||
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||||
|
)
|
||||||
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||||
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||||
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||||
|
# The empty-table env fallback (only read while ``git_sources`` is
|
||||||
|
# EMPTY — the env-fallback test; never cloned in this suite).
|
||||||
|
env["BOR_GIT_SOURCES"] = ENV_SOURCE
|
||||||
|
# Module-scratch dirs (never reached by this suite's local-row
|
||||||
|
# syncs — kept explicit so a shared checkouts dir cannot leak rows
|
||||||
|
# into the walk).
|
||||||
|
scratch = source_dir.parent
|
||||||
|
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||||
|
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"{APP_URL}/api/health")
|
||||||
|
yield APP_URL
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_url(app_server: str) -> str:
|
||||||
|
return app_server
|
||||||
|
|
||||||
|
|
||||||
|
def _truncate_all() -> None:
|
||||||
|
"""Fresh registry + KB per test (the E2E isolation pattern, the
|
||||||
|
test_sync_button.py module-env DSN): the sync's counts and the
|
||||||
|
catalog must be each test's own doing."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||||
|
"git_sources"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean(db_ready: None) -> Iterator[None]:
|
||||||
|
"""This suite owns ``git_sources`` AND the KB tables (the E2E
|
||||||
|
isolation pattern): suites run in isolation but share one
|
||||||
|
Postgres, and a leftover row would flip the app from the
|
||||||
|
``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover
|
||||||
|
document would skew the prune counts). Empty BOTH before and
|
||||||
|
after every test."""
|
||||||
|
_truncate_all()
|
||||||
|
yield
|
||||||
|
_truncate_all()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
||||||
|
"""Real form login landing on the git sources page (admin settled:
|
||||||
|
Sign out visible, the manager revealed by the page module)."""
|
||||||
|
login(page, app_url, next=GIT_SOURCES_URL)
|
||||||
|
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||||
|
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||||
|
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
||||||
|
expect(page.locator("#git-sources-content")).to_be_visible()
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str:
|
||||||
|
"""(Re)seed the stored row for the module fixture dir via the REAL
|
||||||
|
admin API (201) — ``kind="local"`` (the POST contract is unchanged
|
||||||
|
by phase 89; no ``ignore_paths`` → the stored list is ``[]``) —
|
||||||
|
then reload the view so the page's table re-fetches the row (the
|
||||||
|
API seed is server-side: the mounted view does not know about it
|
||||||
|
without a re-show). Returns the stored (expanded) path — the
|
||||||
|
row's ``url`` location value, the same string the page renders
|
||||||
|
and aria-labels."""
|
||||||
|
r = page.request.post(
|
||||||
|
f"{app_url}/api/git-sources",
|
||||||
|
data={"kind": "local", "path": str(source_dir)},
|
||||||
|
)
|
||||||
|
assert r.status == 201, r.text
|
||||||
|
body = r.json()
|
||||||
|
assert body["ignore_paths"] == [] # the default list, round-tripped
|
||||||
|
page.reload()
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
return body["url"]
|
||||||
|
|
||||||
|
|
||||||
|
def run_sync(page: Page, app_url: str) -> dict[str, Any]:
|
||||||
|
"""``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal
|
||||||
|
state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s
|
||||||
|
budget). Returns the terminal status body."""
|
||||||
|
r = page.request.post(f"{app_url}/api/sync")
|
||||||
|
assert r.status == 202, r.text
|
||||||
|
deadline = time.monotonic() + SYNC_TIMEOUT_S
|
||||||
|
body: dict[str, Any] = {}
|
||||||
|
while True:
|
||||||
|
s = page.request.get(f"{app_url}/api/sync/status")
|
||||||
|
assert s.status == 200, s.text
|
||||||
|
body = s.json()
|
||||||
|
if body["state"] in ("success", "failed"):
|
||||||
|
return body
|
||||||
|
assert time.monotonic() < deadline, (
|
||||||
|
f"sync did not reach a terminal state: {body}"
|
||||||
|
)
|
||||||
|
time.sleep(SYNC_TICK_S)
|
||||||
|
|
||||||
|
|
||||||
|
def _catalog_paths(page: Page, app_url: str) -> list[str]:
|
||||||
|
"""The RAG catalog's data source (``GET /api/docs`` — the Sources
|
||||||
|
page's table): every ``documents.path`` (source-relative POSIX)."""
|
||||||
|
r = page.request.get(f"{app_url}/api/docs")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
return [d["path"] for d in r.json()["documents"]]
|
||||||
|
|
||||||
|
|
||||||
|
def _row(page: Page, value: str) -> Any:
|
||||||
|
"""The table row whose mono location cell shows ``value``."""
|
||||||
|
return page.locator("#git-sources-tbody tr", has_text=value)
|
||||||
|
|
||||||
|
|
||||||
|
def _open_ignore_editor(page: Page, value: str) -> Any:
|
||||||
|
"""Click the row's "Ignore paths" button; the alertdialog opens.
|
||||||
|
Returns the trigger button (focus returns to it on close)."""
|
||||||
|
btn = _row(page, value).locator(".git-source-ignore")
|
||||||
|
expect(btn).to_have_count(1)
|
||||||
|
btn.click()
|
||||||
|
dialog = page.locator("#ignore-editor-dialog")
|
||||||
|
expect(dialog).to_be_visible(timeout=30_000)
|
||||||
|
return btn
|
||||||
|
|
||||||
|
|
||||||
|
def _save_ignore_list(page: Page, value: str, entry: str) -> None:
|
||||||
|
"""Open the row's box, replace its content with one line (``entry``
|
||||||
|
— ``""`` clears) and Save; wait for the A5 round-trip: the dialog
|
||||||
|
closes and the row shows the "1 ignored" / cleared count state."""
|
||||||
|
_open_ignore_editor(page, value)
|
||||||
|
page.fill("#ignore-editor-textarea", entry)
|
||||||
|
page.click("#ignore-editor-save")
|
||||||
|
if entry.strip():
|
||||||
|
expect(_row(page, value).locator(".git-source-ignore-count")).to_have_text(
|
||||||
|
"1 ignored", timeout=30_000
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
expect(_row(page, value).locator(".git-source-ignore-count")).to_have_count(0)
|
||||||
|
expect(page.locator("#ignore-editor-dialog")).to_be_hidden()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. PATCH)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
|
||||||
|
# Track every /api/git-sources request the page itself makes — the
|
||||||
|
# gate must be reached WITHOUT touching the admin API (the
|
||||||
|
# test_git_sources_admin.py pattern).
|
||||||
|
api_calls: list[str] = []
|
||||||
|
page.on(
|
||||||
|
"request",
|
||||||
|
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
page.goto(app_url + GIT_SOURCES_URL)
|
||||||
|
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
|
||||||
|
|
||||||
|
# The sign-in gate (the #sources-gate pattern, phase 16)…
|
||||||
|
gate = page.locator("#git-sources-gate")
|
||||||
|
expect(gate).to_be_visible()
|
||||||
|
expect(gate).to_contain_text("Sign in to manage the git sources")
|
||||||
|
expect(
|
||||||
|
gate.locator("a[href='/login.html?next=/git-sources.html']")
|
||||||
|
).to_have_count(1)
|
||||||
|
# …and the manager is absent/inert: table + form + env note, all
|
||||||
|
# inside the hidden #git-sources-content.
|
||||||
|
expect(page.locator("#git-sources-content")).to_be_hidden()
|
||||||
|
expect(page.locator("#git-sources-table")).to_be_hidden()
|
||||||
|
expect(page.locator("#git-source-form")).to_be_hidden()
|
||||||
|
expect(page.locator("#git-sources-env-note")).to_be_hidden()
|
||||||
|
|
||||||
|
# The gate never called the admin API…
|
||||||
|
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
|
||||||
|
# …and the API 403s anonymous callers on GET/POST AND the
|
||||||
|
# phase-89 PATCH (the page context has no cookie — the
|
||||||
|
# test_git_sources_admin.py anonymous pin, extended).
|
||||||
|
assert page.request.get(f"{app_url}/api/git-sources").status == 403
|
||||||
|
assert (
|
||||||
|
page.request.post(
|
||||||
|
f"{app_url}/api/git-sources", data={"url": ENV_SOURCE}
|
||||||
|
).status
|
||||||
|
== 403
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
page.request.patch(
|
||||||
|
f"{app_url}/api/git-sources/{uuid.uuid4()}",
|
||||||
|
data={"ignore_paths": ["ignore/"]},
|
||||||
|
).status
|
||||||
|
== 403
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. The box sets the list; the sync honors it (nothing under ignore/
|
||||||
|
# embedded/summarized; the walk's files count excludes it)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_ignore_box_excludes_from_import(
|
||||||
|
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
|
||||||
|
# Seed the row (no list) through the real API…
|
||||||
|
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||||
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||||
|
|
||||||
|
# …open the box, type "ignore/" (one line), Save → 200 → the row's
|
||||||
|
# "1 ignored" count tag lands (the A5 round-trip through GET).
|
||||||
|
_save_ignore_list(page, stored_path, "ignore/")
|
||||||
|
r = page.request.get(f"{app_url}/api/git-sources")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
# A1: the raw box line is stored NORMALIZED ("ignore/" → "ignore")
|
||||||
|
# — the round-trip is canonical.
|
||||||
|
assert [s["ignore_paths"] for s in body["sources"]] == [["ignore"]]
|
||||||
|
assert body["from_env"] is False
|
||||||
|
|
||||||
|
# The real sync (mock LLM): success, and the walk counted the
|
||||||
|
# remaining FOUR files — the two ignore/ files never entered it
|
||||||
|
# (no documents → no chunks/embeddings, no summary calls for
|
||||||
|
# them). (6 fixture files − 2 ignored = 4; the task's "5" is a
|
||||||
|
# slip against its own six-file tree — the exclusion is what is
|
||||||
|
# under test, and it holds either way.)
|
||||||
|
sync = run_sync(page, app_url)
|
||||||
|
assert sync["state"] == "success", sync
|
||||||
|
detail = sync["detail"]
|
||||||
|
assert detail["files"] == 4, detail
|
||||||
|
assert detail["added"] == 4, detail
|
||||||
|
assert detail["errors"] == 0, detail
|
||||||
|
|
||||||
|
# The catalog holds exactly the four non-ignored paths.
|
||||||
|
paths = _catalog_paths(page, app_url)
|
||||||
|
for p in (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE):
|
||||||
|
assert p in paths, f"{p} missing from the catalog: {paths}"
|
||||||
|
assert not any(p.startswith("ignore/") for p in paths), (
|
||||||
|
f"ignored paths leaked into the catalog: {paths}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. The spec's no-mid-path rule: myfile.md ignores the root-level file
|
||||||
|
# only; the box REPLACES the previous test's list (A5)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefix_rule_no_mid_path(
|
||||||
|
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||||
|
|
||||||
|
# Bare filename prefix (no leading/trailing slash): the spec's own
|
||||||
|
# example — matches the root-level file, never the subdirectory
|
||||||
|
# namesake.
|
||||||
|
_save_ignore_list(page, stored_path, "myfile.md")
|
||||||
|
|
||||||
|
sync = run_sync(page, app_url)
|
||||||
|
assert sync["state"] == "success", sync
|
||||||
|
assert sync["detail"]["files"] == 5, sync["detail"]
|
||||||
|
|
||||||
|
paths = _catalog_paths(page, app_url)
|
||||||
|
# No mid-path matching: the namesake in sub/ is indexed…
|
||||||
|
assert SUB_MYFILE in paths, f"sub/myfile.md was wrongly ignored: {paths}"
|
||||||
|
# …the root-level file is NOT…
|
||||||
|
assert ROOT_MYFILE not in paths, f"root myfile.md must be ignored: {paths}"
|
||||||
|
# …and the ignore/ files ARE indexed this time — the box replaced
|
||||||
|
# (A5), not appended to, the previous test's "ignore/" list
|
||||||
|
# (regression guard).
|
||||||
|
assert IGNORE_SECRET in paths, f"ignore/secret.md must be indexed: {paths}"
|
||||||
|
assert IGNORE_DEEP in paths, f"ignore/deep/x.txt must be indexed: {paths}"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. A2: a previously indexed file that newly matches is pruned
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_newly_ignored_file_is_pruned(
|
||||||
|
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||||
|
|
||||||
|
# Sync 1 — no list: all six files are indexed.
|
||||||
|
first = run_sync(page, app_url)
|
||||||
|
assert first["state"] == "success", first
|
||||||
|
assert first["detail"]["files"] == 6, first["detail"]
|
||||||
|
assert first["detail"]["added"] == 6, first["detail"]
|
||||||
|
paths = _catalog_paths(page, app_url)
|
||||||
|
for p in ALL_SIX:
|
||||||
|
assert p in paths, f"{p} missing after the first sync: {paths}"
|
||||||
|
|
||||||
|
# New list: "ignore/" — the two previously indexed files now match.
|
||||||
|
_save_ignore_list(page, stored_path, "ignore/")
|
||||||
|
|
||||||
|
# Sync 2 — the prune=True run deletes the newly ignored documents
|
||||||
|
# (the seen-set mechanism, the A9 out-of-scope-junk precedent).
|
||||||
|
second = run_sync(page, app_url)
|
||||||
|
assert second["state"] == "success", second
|
||||||
|
detail = second["detail"]
|
||||||
|
assert detail["pruned"] == 2, detail
|
||||||
|
assert detail["files"] == 4, detail
|
||||||
|
|
||||||
|
paths = _catalog_paths(page, app_url)
|
||||||
|
for p in (KEEP_MD, NOTES_YAML, ROOT_MYFILE, SUB_MYFILE):
|
||||||
|
assert p in paths, f"{p} missing after the prune: {paths}"
|
||||||
|
assert not any(p.startswith("ignore/") for p in paths), (
|
||||||
|
f"newly ignored paths survived the prune: {paths}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 5. The editor's a11y + the 422 error path (and the happy-path heal)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_editor_a11y_and_error_path(
|
||||||
|
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||||
|
|
||||||
|
# The trigger is the labeled per-row button (left of Remove)…
|
||||||
|
row = _row(page, stored_path)
|
||||||
|
btn = row.locator(".git-source-ignore")
|
||||||
|
expect(btn).to_have_count(1)
|
||||||
|
expect(btn).to_have_attribute(
|
||||||
|
"aria-label", f"Edit ignored paths for local source: {stored_path}"
|
||||||
|
)
|
||||||
|
expect(row.locator(".git-source-ignore-count")).to_have_count(0) # no list yet
|
||||||
|
|
||||||
|
# …and it opens a real alertdialog (the phase-69 idiom) with the
|
||||||
|
# visible label and the Cancel-safe focus default.
|
||||||
|
btn.click()
|
||||||
|
dialog = page.locator("#ignore-editor-dialog")
|
||||||
|
expect(dialog).to_be_visible(timeout=30_000)
|
||||||
|
assert dialog.get_attribute("role") == "alertdialog"
|
||||||
|
assert dialog.get_attribute("aria-modal") == "true"
|
||||||
|
# exact: the row button's aria-label ("Edit ignored paths for …")
|
||||||
|
# contains the substring — the visible <label> is the only exact
|
||||||
|
# "Ignored paths" name.
|
||||||
|
expect(page.get_by_label("Ignored paths", exact=True)).to_have_count(1)
|
||||||
|
# The dialog names the source (textContent — the local path).
|
||||||
|
expect(page.locator("#ignore-editor-source")).to_have_text(stored_path)
|
||||||
|
expect(page.locator("#ignore-editor-cancel")).to_be_focused()
|
||||||
|
|
||||||
|
# Escape closes as CANCEL — no request — and focus returns to the
|
||||||
|
# row's trigger button.
|
||||||
|
page.keyboard.press("Escape")
|
||||||
|
expect(dialog).to_be_hidden()
|
||||||
|
expect(btn).to_be_focused()
|
||||||
|
|
||||||
|
# The 422 path: one entry of 501 chars (> the A4 500-char limit) →
|
||||||
|
# the fixed detail, the textarea content KEPT, Save re-enabled with
|
||||||
|
# its idle label (never stale).
|
||||||
|
btn.click()
|
||||||
|
expect(dialog).to_be_visible(timeout=30_000)
|
||||||
|
bad = "a" * 501
|
||||||
|
page.fill("#ignore-editor-textarea", bad)
|
||||||
|
page.click("#ignore-editor-save")
|
||||||
|
error = page.locator("#ignore-editor-error")
|
||||||
|
expect(error).to_be_visible(timeout=30_000)
|
||||||
|
assert error.get_attribute("role") == "alert"
|
||||||
|
expect(error).to_have_text("an ignore path exceeds 500 characters")
|
||||||
|
expect(page.locator("#ignore-editor-textarea")).to_have_value(bad)
|
||||||
|
expect(dialog).to_be_visible() # the fix is one edit + retry
|
||||||
|
save_btn = page.locator("#ignore-editor-save")
|
||||||
|
expect(save_btn).to_be_enabled()
|
||||||
|
expect(save_btn).to_have_text("Save")
|
||||||
|
# Nothing was stored: the list is still empty (no count tag).
|
||||||
|
expect(row.locator(".git-source-ignore-count")).to_have_count(0)
|
||||||
|
|
||||||
|
# The happy path heals the error state: clear, one entry, Save →
|
||||||
|
# the dialog closes and the "1 ignored" tag lands.
|
||||||
|
page.fill("#ignore-editor-textarea", "keep.md")
|
||||||
|
save_btn.click()
|
||||||
|
expect(dialog).to_be_hidden(timeout=30_000)
|
||||||
|
expect(
|
||||||
|
page.locator("#git-sources-tbody tr .git-source-ignore-count")
|
||||||
|
).to_have_text("1 ignored", timeout=30_000)
|
||||||
|
r = page.request.get(f"{app_url}/api/git-sources")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
assert [s["ignore_paths"] for s in r.json()["sources"]] == [["keep.md"]]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 6. A3: env-fallback rows have no box (no DB row to store a list on)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_fallback_rows_have_no_box(
|
||||||
|
page: Page, app_url: str, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
# The autouse fixture truncated git_sources — the table is EMPTY,
|
||||||
|
# so the module app's BOR_GIT_SOURCES URL is the effective list
|
||||||
|
# (the test_git_sources_admin.py module-env pattern).
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
|
||||||
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||||
|
row = page.locator("#git-sources-tbody tr")
|
||||||
|
expect(row).to_contain_text(ENV_SOURCE)
|
||||||
|
# The env-fallback surface: the "from .env" tag (never color
|
||||||
|
# alone)…
|
||||||
|
expect(row.locator(".git-source-env-tag")).to_have_count(1)
|
||||||
|
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
|
||||||
|
# …and NO per-row actions at all — neither Remove (nothing stored
|
||||||
|
# to remove) nor the phase-89 "Ignore paths" box (A3).
|
||||||
|
expect(row.locator(".git-source-remove")).to_have_count(0)
|
||||||
|
expect(row.locator(".git-source-ignore")).to_have_count(0)
|
||||||
|
expect(row.locator(".git-source-ignore-count")).to_have_count(0)
|
||||||
|
# The env-fallback note explains the active list's origin…
|
||||||
|
expect(page.locator("#git-sources-env-note")).to_be_visible()
|
||||||
|
# …and the API agrees: from_env true, null ids, the env URL, and an
|
||||||
|
# empty ignore list (no DB row to store a list on).
|
||||||
|
r = page.request.get(f"{app_url}/api/git-sources")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
assert body["from_env"] is True
|
||||||
|
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE]
|
||||||
|
assert all(
|
||||||
|
s["id"] is None and s["ignore_paths"] == [] for s in body["sources"]
|
||||||
|
)
|
||||||
@@ -28,7 +28,14 @@ Contract under test:
|
|||||||
index as backstop); wrong field combinations (git without url, local
|
index as backstop); wrong field combinations (git without url, local
|
||||||
without path, both kinds' fields) → 422;
|
without path, both kinds' fields) → 422;
|
||||||
* DELETE — 204 and gone; an emptied table falls back to the env list
|
* DELETE — 204 and gone; an emptied table falls back to the env list
|
||||||
again; unknown id → 404.
|
again; unknown id → 404;
|
||||||
|
* ignore paths (phase 89) — POST accepts the RAW box lines for both
|
||||||
|
kinds (optional, absent → ``[]``), stored normalized (A1) with the A4
|
||||||
|
fixed-detail 422s (shared gate with PATCH); GET reports each row's
|
||||||
|
stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only)
|
||||||
|
replaces the list wholesale (A5 — an empty list clears all), 404
|
||||||
|
unknown id, 422 fixed details for the A4 limits, the row otherwise
|
||||||
|
unchanged.
|
||||||
|
|
||||||
``git_sources`` is global state: truncated around every test.
|
``git_sources`` is global state: truncated around every test.
|
||||||
"""
|
"""
|
||||||
@@ -92,6 +99,10 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
|
|||||||
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
|
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
|
||||||
assert r.status_code == 403
|
assert r.status_code == 403
|
||||||
assert r.json() == {"detail": "admin only"}
|
assert r.json() == {"detail": "admin only"}
|
||||||
|
# The phase-89 ignore-list PATCH is gated the same way.
|
||||||
|
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
|
||||||
|
assert r.status_code == 403
|
||||||
|
assert r.json() == {"detail": "admin only"}
|
||||||
# Nothing landed in the table.
|
# Nothing landed in the table.
|
||||||
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
||||||
|
|
||||||
@@ -121,6 +132,8 @@ def test_get_empty_table_with_env_returns_env_rows(
|
|||||||
"url": "https://a.example.com/one.git",
|
"url": "https://a.example.com/one.git",
|
||||||
"path": None,
|
"path": None,
|
||||||
"added_at": None,
|
"added_at": None,
|
||||||
|
# Env rows have no DB row to store a list on (phase 89).
|
||||||
|
"ignore_paths": [],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": None,
|
"id": None,
|
||||||
@@ -128,6 +141,7 @@ def test_get_empty_table_with_env_returns_env_rows(
|
|||||||
"url": "git@b.example.com:two.git",
|
"url": "git@b.example.com:two.git",
|
||||||
"path": None,
|
"path": None,
|
||||||
"added_at": None,
|
"added_at": None,
|
||||||
|
"ignore_paths": [],
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -193,7 +207,10 @@ def test_post_creates_trimmed_and_list_stops_using_env(
|
|||||||
assert body["url"] == "https://new.example.com/repo.git" # trimmed
|
assert body["url"] == "https://new.example.com/repo.git" # trimmed
|
||||||
uuid.UUID(body["id"])
|
uuid.UUID(body["id"])
|
||||||
assert body["added_at"] is not None
|
assert body["added_at"] is not None
|
||||||
assert set(body) == {"id", "url", "added_at"}
|
# Phase 89: the response gains ``ignore_paths`` — absent at create
|
||||||
|
# time → ``[]``.
|
||||||
|
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
|
||||||
|
assert body["ignore_paths"] == []
|
||||||
|
|
||||||
# The DB row now wins: from_env False, the env URL is gone from the list.
|
# The DB row now wins: from_env False, the env URL is gone from the list.
|
||||||
body = admin_client.get("/api/git-sources").json()
|
body = admin_client.get("/api/git-sources").json()
|
||||||
@@ -303,8 +320,10 @@ def test_post_local_creates_stored_row_with_expanded_path(
|
|||||||
assert r.status_code == 201, r.text
|
assert r.status_code == 201, r.text
|
||||||
body = r.json()
|
body = r.json()
|
||||||
# The phase-35 response shape is unchanged — the local row reports
|
# The phase-35 response shape is unchanged — the local row reports
|
||||||
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET.
|
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
|
||||||
assert set(body) == {"id", "url", "added_at"}
|
# phase 89 adds ``ignore_paths`` (absent → ``[]``).
|
||||||
|
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
|
||||||
|
assert body["ignore_paths"] == []
|
||||||
uuid.UUID(body["id"])
|
uuid.UUID(body["id"])
|
||||||
assert body["url"] == str(real_dir)
|
assert body["url"] == str(real_dir)
|
||||||
assert body["added_at"] is not None
|
assert body["added_at"] is not None
|
||||||
@@ -536,6 +555,7 @@ def test_delete_removes_row_and_falls_back_to_env(
|
|||||||
"url": "https://env.example.com/env.git",
|
"url": "https://env.example.com/env.git",
|
||||||
"path": None,
|
"path": None,
|
||||||
"added_at": None,
|
"added_at": None,
|
||||||
|
"ignore_paths": [], # env rows: no DB row to store a list on
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
db.execute(text("TRUNCATE chunks, documents"))
|
db.execute(text("TRUNCATE chunks, documents"))
|
||||||
@@ -550,3 +570,205 @@ def test_delete_unknown_id_returns_404(admin_client: TestClient) -> None:
|
|||||||
|
|
||||||
def test_delete_invalid_id_returns_422(admin_client: TestClient) -> None:
|
def test_delete_invalid_id_returns_422(admin_client: TestClient) -> None:
|
||||||
assert admin_client.delete("/api/git-sources/not-a-uuid").status_code == 422
|
assert admin_client.delete("/api/git-sources/not-a-uuid").status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
# --- ignore paths (phase 89) ----------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_stores_normalized_ignore_paths_both_kinds(
|
||||||
|
admin_client: TestClient, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""POST accepts the RAW box lines for both kinds (A1 normalization
|
||||||
|
in the API layer) — the normalized list is stored and round-trips
|
||||||
|
through GET."""
|
||||||
|
r = admin_client.post(
|
||||||
|
"/api/git-sources",
|
||||||
|
json={
|
||||||
|
"url": "https://example.com/ig.git",
|
||||||
|
"ignore_paths": ["/my/files/", " my/files2 ", "x"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert r.status_code == 201, r.text
|
||||||
|
assert r.json()["ignore_paths"] == ["my/files", "my/files2", "x"]
|
||||||
|
|
||||||
|
real_dir = tmp_path / "ig"
|
||||||
|
real_dir.mkdir()
|
||||||
|
r = admin_client.post(
|
||||||
|
"/api/git-sources",
|
||||||
|
json={"kind": "local", "path": str(real_dir), "ignore_paths": ["//skip/", "keep"]},
|
||||||
|
)
|
||||||
|
assert r.status_code == 201, r.text
|
||||||
|
assert r.json()["ignore_paths"] == ["skip", "keep"]
|
||||||
|
|
||||||
|
# Round-trip through GET (keyed by url — (added_at, id) order of two
|
||||||
|
# same-millisecond inserts is not the point under test).
|
||||||
|
by_url = {
|
||||||
|
s["url"]: s["ignore_paths"] for s in admin_client.get("/api/git-sources").json()["sources"]
|
||||||
|
}
|
||||||
|
assert by_url["https://example.com/ig.git"] == ["my/files", "my/files2", "x"]
|
||||||
|
assert by_url[str(real_dir)] == ["skip", "keep"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_without_ignore_paths_reports_empty_list(
|
||||||
|
admin_client: TestClient, db: Session
|
||||||
|
) -> None:
|
||||||
|
"""Absent ``ignore_paths`` at create time → ``[]`` — both in the 201
|
||||||
|
response and in the GET round-trip (the migration's server default)."""
|
||||||
|
r = admin_client.post("/api/git-sources", json={"url": "https://example.com/none.git"})
|
||||||
|
assert r.status_code == 201
|
||||||
|
assert r.json()["ignore_paths"] == []
|
||||||
|
body = admin_client.get("/api/git-sources").json()
|
||||||
|
assert body["sources"][0]["ignore_paths"] == []
|
||||||
|
# The stored value is the JSONB server default, not Python-only.
|
||||||
|
row = db.scalars(select(GitSource)).one()
|
||||||
|
assert row.ignore_paths == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_rejects_invalid_ignore_paths_like_patch(
|
||||||
|
admin_client: TestClient, db: Session
|
||||||
|
) -> None:
|
||||||
|
"""POST shares ``_validate_ignore_paths`` with PATCH — the same A4
|
||||||
|
fixed 422 details; nothing is stored."""
|
||||||
|
for payload, detail in (
|
||||||
|
([" "], "ignore paths must be non-empty"),
|
||||||
|
([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"),
|
||||||
|
(["a" * 501], "an ignore path exceeds 500 characters"),
|
||||||
|
):
|
||||||
|
r = admin_client.post(
|
||||||
|
"/api/git-sources",
|
||||||
|
json={"url": "https://example.com/bad-ignore.git", "ignore_paths": payload},
|
||||||
|
)
|
||||||
|
assert r.status_code == 422, f"{detail!r}: {r.text}"
|
||||||
|
assert r.json()["detail"] == detail
|
||||||
|
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_reports_stored_ignore_paths_and_env_rows_empty(
|
||||||
|
admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""GET reports each DB row's stored list; env-fallback rows (no DB
|
||||||
|
row to store a list on) report ``[]`` with ``from_env: true``."""
|
||||||
|
db.add(GitSource(url="https://example.com/stored.git", ignore_paths=["docs/drafts"]))
|
||||||
|
db.commit()
|
||||||
|
body = admin_client.get("/api/git-sources").json()
|
||||||
|
assert body["from_env"] is False
|
||||||
|
assert body["sources"][0]["ignore_paths"] == ["docs/drafts"]
|
||||||
|
|
||||||
|
db.execute(text("DELETE FROM git_sources"))
|
||||||
|
db.commit()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
git_sources_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings("https://a.example.com/env.git"),
|
||||||
|
)
|
||||||
|
body = admin_client.get("/api/git-sources").json()
|
||||||
|
assert body["from_env"] is True
|
||||||
|
assert body["sources"] == [
|
||||||
|
{
|
||||||
|
"id": None,
|
||||||
|
"kind": "git",
|
||||||
|
"url": "https://a.example.com/env.git",
|
||||||
|
"path": None,
|
||||||
|
"added_at": None,
|
||||||
|
"ignore_paths": [],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -> None:
|
||||||
|
"""PATCH 200 — REPLACE semantics (A5): the body list, normalized,
|
||||||
|
becomes the row's whole list; an empty list clears all; ``id``/
|
||||||
|
``url``/``added_at`` are unchanged; the response is the
|
||||||
|
``GitSourceOut`` shape incl. the new list."""
|
||||||
|
created = admin_client.post(
|
||||||
|
"/api/git-sources",
|
||||||
|
json={"url": "https://example.com/patch.git", "ignore_paths": ["old/"]},
|
||||||
|
)
|
||||||
|
assert created.status_code == 201
|
||||||
|
before = created.json()
|
||||||
|
|
||||||
|
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
|
||||||
|
assert body["ignore_paths"] == ["a", "b"] # normalized
|
||||||
|
for key in ("id", "url", "added_at"):
|
||||||
|
assert body[key] == before[key]
|
||||||
|
# Round-trip through GET.
|
||||||
|
assert (
|
||||||
|
admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a", "b"]
|
||||||
|
)
|
||||||
|
|
||||||
|
# An empty list clears all.
|
||||||
|
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": []})
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
assert r.json()["ignore_paths"] == []
|
||||||
|
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_missing_field_is_422(admin_client: TestClient) -> None:
|
||||||
|
"""``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent
|
||||||
|
field is a 422 with the model's own detail."""
|
||||||
|
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"})
|
||||||
|
assert created.status_code == 201
|
||||||
|
r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={})
|
||||||
|
assert r.status_code == 422
|
||||||
|
# The model's own (Pydantic) detail — the missing required field.
|
||||||
|
assert any(
|
||||||
|
item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None:
|
||||||
|
r = admin_client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
|
||||||
|
assert r.status_code == 404
|
||||||
|
assert r.json() == {"detail": "git source not found"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_invalid_id_returns_422(admin_client: TestClient) -> None:
|
||||||
|
r = admin_client.patch("/api/git-sources/not-a-uuid", json={"ignore_paths": []})
|
||||||
|
assert r.status_code == 422
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_422s_are_fixed_details(admin_client: TestClient) -> None:
|
||||||
|
"""The A4 422s are exact fixed strings (never echoing the input),
|
||||||
|
and a rejected PATCH leaves the row's list unchanged."""
|
||||||
|
created = admin_client.post(
|
||||||
|
"/api/git-sources", json={"url": "https://example.com/v.git", "ignore_paths": ["keep"]}
|
||||||
|
)
|
||||||
|
assert created.status_code == 201
|
||||||
|
sid = created.json()["id"]
|
||||||
|
|
||||||
|
for payload, detail in (
|
||||||
|
# Whitespace-only → empty after normalization: 422, not a silent
|
||||||
|
# drop (the UI drops blank lines client-side; the API is
|
||||||
|
# defensive).
|
||||||
|
([" "], "ignore paths must be non-empty"),
|
||||||
|
(list(f"e{i}" for i in range(201)), "a source has at most 200 ignore paths"),
|
||||||
|
(["a" * 501], "an ignore path exceeds 500 characters"),
|
||||||
|
):
|
||||||
|
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": payload})
|
||||||
|
assert r.status_code == 422, f"{detail!r}: {r.text}"
|
||||||
|
assert r.json()["detail"] == detail
|
||||||
|
# Nothing changed by the failed PATCHes.
|
||||||
|
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["keep"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None:
|
||||||
|
"""Exactly 200 entries and a 500-char entry (post-normalization)
|
||||||
|
are the accepted edge of A4."""
|
||||||
|
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/bnd.git"})
|
||||||
|
assert created.status_code == 201
|
||||||
|
sid = created.json()["id"]
|
||||||
|
|
||||||
|
r = admin_client.patch(
|
||||||
|
f"/api/git-sources/{sid}", json={"ignore_paths": [f"e/{i}" for i in range(200)]}
|
||||||
|
)
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
assert len(r.json()["ignore_paths"]) == 200
|
||||||
|
|
||||||
|
long_entry = "a" * 500
|
||||||
|
assert len(long_entry) == 500
|
||||||
|
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]})
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
assert r.json()["ignore_paths"] == [long_entry]
|
||||||
|
|||||||
@@ -291,6 +291,7 @@ class _GatedImport:
|
|||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: object = None,
|
session: object = None,
|
||||||
progress: Callable[[str, str, int, int], None] | None = None,
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
|
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
self.prune_flags.append(prune)
|
self.prune_flags.append(prune)
|
||||||
if progress is not None:
|
if progress is not None:
|
||||||
@@ -1352,3 +1353,92 @@ def test_models_down_fails_the_run_and_leaves_folder_and_row(
|
|||||||
# The scan never ran: no docs, no stray temps.
|
# The scan never ran: no docs, no stray temps.
|
||||||
assert _docs(upload_client) == []
|
assert _docs(upload_client) == []
|
||||||
assert [p.name for p in uploads.iterdir()] == ["homelab"]
|
assert [p.name for p in uploads.iterdir()] == ["homelab"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- phase 89: re-uploads honor the row's saved ignore list -------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_reupload_honors_saved_ignore_list(
|
||||||
|
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89: a re-upload of an EXISTING source honors the ignore
|
||||||
|
list saved on its row. First scan (fresh row, no list) indexes the
|
||||||
|
ignored file too; once the list is saved via the API, re-uploading
|
||||||
|
the same archive again scans only the kept file — the previously
|
||||||
|
indexed ignored file leaves the KB (prune), and the run lands
|
||||||
|
``success`` with counts that exclude it. The upload itself keeps
|
||||||
|
every file on disk: the ignore is about the index, not the folder."""
|
||||||
|
uploads = tmp_path / "uploads"
|
||||||
|
_point_at(monkeypatch, uploads)
|
||||||
|
_real_llm(monkeypatch)
|
||||||
|
files = {
|
||||||
|
"keep.md": "# Keep\nin scope\n",
|
||||||
|
"ignore/secret.md": "# Secret\nignored\n",
|
||||||
|
}
|
||||||
|
r = _post(upload_client, "docs.tar.gz", _targz_bytes(files))
|
||||||
|
assert r.status_code == 202, r.text
|
||||||
|
status = _wait_status(upload_client)
|
||||||
|
assert status["state"] == "success", status
|
||||||
|
# First scan: the fresh row has no list → both files are indexed.
|
||||||
|
assert status["detail"]["files"] == 2
|
||||||
|
assert _docs(upload_client) == [("docs", "ignore/secret.md"), ("docs", "keep.md")]
|
||||||
|
|
||||||
|
# Save the ignore list on the row via the API (phase 89, task 03).
|
||||||
|
folder = uploads / "docs"
|
||||||
|
row = _row(db, str(folder))
|
||||||
|
assert row is not None
|
||||||
|
r = upload_client.patch(
|
||||||
|
f"/api/git-sources/{row.id}", json={"ignore_paths": ["ignore/"]}
|
||||||
|
)
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
assert r.json()["ignore_paths"] == ["ignore"] # stored normalized
|
||||||
|
|
||||||
|
# Re-upload the SAME archive (same name → in-place replace).
|
||||||
|
r = _post(upload_client, "docs.tar.gz", _targz_bytes(files))
|
||||||
|
assert r.status_code == 202, r.text
|
||||||
|
status = _wait_status(upload_client)
|
||||||
|
assert status["state"] == "success", status
|
||||||
|
detail = status["detail"]
|
||||||
|
# The scan walked only keep.md (unchanged); the ignored file never
|
||||||
|
# entered the walk, so prune dropped the first scan's row for it.
|
||||||
|
assert detail["files"] == 1
|
||||||
|
assert detail["added"] == 0
|
||||||
|
assert detail["updated"] == 0
|
||||||
|
assert detail["unchanged"] == 1
|
||||||
|
assert detail["pruned"] == 1
|
||||||
|
assert _docs(upload_client) == [("docs", "keep.md")]
|
||||||
|
|
||||||
|
# The re-upload left the row's list in place (the existing row is
|
||||||
|
# untouched by the upsert), and the folder keeps every file.
|
||||||
|
row_after = _row(db, str(folder))
|
||||||
|
assert row_after is not None
|
||||||
|
assert row_after.ignore_paths == ["ignore"]
|
||||||
|
assert {p.name for p in folder.iterdir()} == {"keep.md", "ignore"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_new_source_without_list_imports_everything(
|
||||||
|
upload_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89 regression: an upload of a NEW source name (no row yet,
|
||||||
|
hence no ignore list) imports everything in the archive — including
|
||||||
|
nested files — exactly as before phase 89."""
|
||||||
|
uploads = tmp_path / "uploads"
|
||||||
|
_point_at(monkeypatch, uploads)
|
||||||
|
_real_llm(monkeypatch)
|
||||||
|
files = {
|
||||||
|
"alpha.md": "# Alpha\nroot file\n",
|
||||||
|
"sub/deep.md": "# Deep\nnested file\n",
|
||||||
|
}
|
||||||
|
r = _post(upload_client, "fresh.tar.gz", _targz_bytes(files))
|
||||||
|
assert r.status_code == 202, r.text
|
||||||
|
status = _wait_status(upload_client)
|
||||||
|
assert status["state"] == "success", status
|
||||||
|
detail = status["detail"]
|
||||||
|
assert detail["files"] == 2
|
||||||
|
assert detail["added"] == 2
|
||||||
|
# The fresh row carries the server-default empty list: nothing was
|
||||||
|
# ignored.
|
||||||
|
row = _row(db, str(uploads / "fresh"))
|
||||||
|
assert row is not None
|
||||||
|
assert (row.ignore_paths or []) == []
|
||||||
|
assert _docs(upload_client) == [("fresh", "alpha.md"), ("fresh", "sub/deep.md")]
|
||||||
|
|||||||
@@ -43,14 +43,15 @@ def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Sett
|
|||||||
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
||||||
|
|
||||||
|
|
||||||
def _git_row(url: str) -> GitSource:
|
def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
|
||||||
return GitSource(url=url, kind="git")
|
return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
|
||||||
|
|
||||||
|
|
||||||
def _local_row(path: str) -> GitSource:
|
def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource:
|
||||||
"""A local row as the phase-38 API stores it: the expanded path in
|
"""A local row as the phase-38 API stores it: the expanded path in
|
||||||
both ``path`` and the NOT-NULL ``url`` location column."""
|
both ``path`` and the NOT-NULL ``url`` location column (plus the
|
||||||
return GitSource(url=path, kind="local", path=path)
|
phase-89 ignore list, empty by default)."""
|
||||||
|
return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or [])
|
||||||
|
|
||||||
|
|
||||||
class FakeImportSources:
|
class FakeImportSources:
|
||||||
@@ -66,8 +67,12 @@ class FakeImportSources:
|
|||||||
*,
|
*,
|
||||||
prune: bool = False,
|
prune: bool = False,
|
||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
|
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
self.calls.append({"sources": list(sources), "prune": prune, "limit": limit})
|
self.calls.append(
|
||||||
|
{"sources": list(sources), "prune": prune, "limit": limit,
|
||||||
|
"ignore_by_root": ignore_by_root}
|
||||||
|
)
|
||||||
return ImportSummary(files=1, added=1)
|
return ImportSummary(files=1, added=1)
|
||||||
|
|
||||||
|
|
||||||
@@ -146,9 +151,10 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
|
|||||||
sources_dir=str(tmp_path / "bor"),
|
sources_dir=str(tmp_path / "bor"),
|
||||||
)
|
)
|
||||||
|
|
||||||
sources = import_docs._resolve_sources(None, settings)
|
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||||
|
|
||||||
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
|
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
|
||||||
|
assert ignore_map == {} # phase 89: no row carries a list → empty map
|
||||||
assert calls == [
|
assert calls == [
|
||||||
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
|
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
|
||||||
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
|
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
|
||||||
@@ -163,9 +169,10 @@ def test_resolve_sources_cli_source_wins(
|
|||||||
settings = _settings(git_sources="https://host/a/repo.git")
|
settings = _settings(git_sources="https://host/a/repo.git")
|
||||||
manual = tmp_path / "Manual"
|
manual = tmp_path / "Manual"
|
||||||
|
|
||||||
sources = import_docs._resolve_sources([manual], settings)
|
sources, ignore_map = import_docs._resolve_sources([manual], settings)
|
||||||
|
|
||||||
assert sources == [manual]
|
assert sources == [manual]
|
||||||
|
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
|
||||||
assert calls == [] # git is never touched when --source is given
|
assert calls == [] # git is never touched when --source is given
|
||||||
|
|
||||||
|
|
||||||
@@ -186,9 +193,10 @@ def test_resolve_sources_db_rows_win_over_env(
|
|||||||
sources_dir=str(tmp_path / "bor"),
|
sources_dir=str(tmp_path / "bor"),
|
||||||
)
|
)
|
||||||
|
|
||||||
sources = import_docs._resolve_sources(None, settings)
|
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||||
|
|
||||||
assert sources == [tmp_path / "bor" / "only"]
|
assert sources == [tmp_path / "bor" / "only"]
|
||||||
|
assert ignore_map == {} # phase 89: no row carries a list → empty map
|
||||||
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
||||||
|
|
||||||
|
|
||||||
@@ -197,8 +205,102 @@ def test_resolve_sources_defaults_when_nothing_configured(
|
|||||||
) -> None:
|
) -> None:
|
||||||
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
|
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
|
||||||
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
|
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
|
||||||
sources = import_docs._resolve_sources(None, _settings())
|
sources, ignore_map = import_docs._resolve_sources(None, _settings())
|
||||||
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
|
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
|
||||||
|
assert ignore_map == {} # phase 89: the legacy fallback has no rows
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_sources_rows_branch_builds_ignore_map(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89: the rows branch returns each row's ignore list keyed by
|
||||||
|
the resolved root string — the local row's directory, the git row's
|
||||||
|
checkout dir; a row without a list contributes nothing to the map."""
|
||||||
|
calls, fake = _fake_clone_factory()
|
||||||
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||||
|
local_dir = tmp_path / "LocalDocs"
|
||||||
|
local_dir.mkdir()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
import_docs,
|
||||||
|
"effective_sources",
|
||||||
|
lambda db: (
|
||||||
|
[
|
||||||
|
_git_row("https://db.example/only.git"),
|
||||||
|
_local_row(str(local_dir), ignore_paths=["ignore/"]),
|
||||||
|
],
|
||||||
|
"db",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||||
|
|
||||||
|
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||||
|
|
||||||
|
assert sources == [tmp_path / "bor" / "only", local_dir]
|
||||||
|
# Keyed by the SAME string the importer sees (the root, not the name).
|
||||||
|
assert ignore_map == {str(local_dir): ["ignore/"]}
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_sources_two_rows_sharing_root_string_extend(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89 collision rule: two rows resolving to the SAME root
|
||||||
|
string (the sibling/repo-name edge — ``…/shared`` and
|
||||||
|
``…/shared.git``) get the UNION of their lists (extend, not
|
||||||
|
replace), in row order."""
|
||||||
|
calls, fake = _fake_clone_factory()
|
||||||
|
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
import_docs,
|
||||||
|
"effective_sources",
|
||||||
|
lambda db: (
|
||||||
|
[
|
||||||
|
_git_row("https://a.example/shared", ignore_paths=["a/"]),
|
||||||
|
_git_row("https://a.example/shared.git", ignore_paths=["b"]),
|
||||||
|
],
|
||||||
|
"db",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||||
|
|
||||||
|
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||||
|
|
||||||
|
shared = str(tmp_path / "bor" / "shared")
|
||||||
|
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
|
||||||
|
assert ignore_map == {shared: ["a/", "b"]} # union, row order
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_rows_branch_passes_ignore_map_to_import(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
tmp_path: Path,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89: a ``kind=local`` row carrying ``ignore_paths`` →
|
||||||
|
``main`` passes the per-root map to ``import_sources`` (keyed by
|
||||||
|
the directory string, prune flag unchanged)."""
|
||||||
|
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||||
|
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||||
|
local_dir = tmp_path / "LocalDocs"
|
||||||
|
local_dir.mkdir()
|
||||||
|
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
|
||||||
|
(local_dir / "ignore").mkdir()
|
||||||
|
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
|
||||||
|
encoding="utf-8")
|
||||||
|
monkeypatch.setattr(
|
||||||
|
import_docs,
|
||||||
|
"effective_sources",
|
||||||
|
lambda db: ([_local_row(str(local_dir), ignore_paths=["ignore/"])], "db"),
|
||||||
|
)
|
||||||
|
fake_import = FakeImportSources()
|
||||||
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||||
|
_stub_bump(monkeypatch)
|
||||||
|
|
||||||
|
rc = import_docs.main([])
|
||||||
|
|
||||||
|
assert rc == 0
|
||||||
|
call = fake_import.calls[0]
|
||||||
|
assert call["sources"] == [local_dir]
|
||||||
|
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
|
||||||
|
assert call["prune"] is False # the CLI's no-prune default is unchanged
|
||||||
|
|
||||||
|
|
||||||
# --- main() ----------------------------------------------------------------
|
# --- main() ----------------------------------------------------------------
|
||||||
@@ -306,9 +408,10 @@ def test_resolve_sources_mixed_git_and_local(
|
|||||||
sources_dir=str(tmp_path / "bor"),
|
sources_dir=str(tmp_path / "bor"),
|
||||||
)
|
)
|
||||||
|
|
||||||
sources = import_docs._resolve_sources(None, settings)
|
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||||
|
|
||||||
assert sources == [tmp_path / "bor" / "only", local_dir]
|
assert sources == [tmp_path / "bor" / "only", local_dir]
|
||||||
|
assert ignore_map == {} # phase 89: neither row carries a list
|
||||||
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,202 @@
|
|||||||
|
"""Integration: phase-89 ignore paths through the real import pipeline.
|
||||||
|
|
||||||
|
Phase 89 (TODO.md L3): per-source ignore lists — source-relative path
|
||||||
|
prefixes that are never walked, hence never embedded and never
|
||||||
|
summarized (A1), and previously indexed files that newly match a
|
||||||
|
pattern are pruned on the next ``prune=True`` run (A2). Mirrors the
|
||||||
|
fixture-tree + mock-LLM pattern of ``test_importer_e2e.py``: a
|
||||||
|
``tmp_path`` source dir named ``IgnoreFix`` run through the real
|
||||||
|
``import_sources`` into the compose Postgres, with
|
||||||
|
:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from sqlalchemy import select, text
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.models import Chunk, Document
|
||||||
|
from app.rag.importer import import_sources
|
||||||
|
from tests.fakes import FakeEmbedder
|
||||||
|
|
||||||
|
NAME = "IgnoreFix"
|
||||||
|
|
||||||
|
|
||||||
|
def _write(root: Path, rel: str, content: str) -> None:
|
||||||
|
path = root / rel
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(content, encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _tree(tmp_path: Path, name: str = NAME) -> Path:
|
||||||
|
"""The fixture tree: two kept files + one md and one txt under ``ignore/``."""
|
||||||
|
root = tmp_path / name
|
||||||
|
_write(root, "keep.md", "# Keep\n\nkept body\n")
|
||||||
|
_write(root, "ignore/secret.md", "# Secret\n\nSECRET-CONTENT\n")
|
||||||
|
_write(root, "ignore/notes.txt", "IGNORED-TEXT-CONTENT\n")
|
||||||
|
_write(root, "top.txt", "TOP-TEXT-CONTENT\n")
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _reset(db: Session) -> None:
|
||||||
|
# House cleanup pattern (tests/integration/test_importer_e2e.py).
|
||||||
|
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_ignored_files_never_indexed(db: Session, tmp_path: Path) -> None:
|
||||||
|
_reset(db)
|
||||||
|
root = _tree(tmp_path)
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
|
||||||
|
summary = asyncio.run(
|
||||||
|
import_sources(
|
||||||
|
[root], llm, session=db, ignore_by_root={str(root): ["ignore/"]}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# Only the two kept files are walked — the ignore/ subtree is
|
||||||
|
# invisible to the pipeline.
|
||||||
|
assert summary.files == 2
|
||||||
|
assert summary.errors == 0
|
||||||
|
# The kept non-markdown file IS summarized; the ignored .txt is not.
|
||||||
|
assert summary.summaries == 1
|
||||||
|
assert summary.summary_errors == 0
|
||||||
|
|
||||||
|
docs = db.scalars(select(Document)).all()
|
||||||
|
assert {(d.source, d.path) for d in docs} == {
|
||||||
|
(NAME, "keep.md"),
|
||||||
|
(NAME, "top.txt"),
|
||||||
|
}
|
||||||
|
# No documents row for an ignored file — hence no chunks rows for it,
|
||||||
|
# no embedding call, and no summary column value, by construction.
|
||||||
|
for rel in ("ignore/secret.md", "ignore/notes.txt"):
|
||||||
|
assert not any(d.path == rel for d in docs)
|
||||||
|
assert not any("SECRET-CONTENT" in c.content for c in db.scalars(select(Chunk)).all())
|
||||||
|
for texts in llm.calls: # every embed batch
|
||||||
|
assert not any(
|
||||||
|
"SECRET-CONTENT" in t or "IGNORED-TEXT-CONTENT" in t for t in texts
|
||||||
|
)
|
||||||
|
# Exactly one summary call (top.txt) — the ignored files never reached
|
||||||
|
# the lite model.
|
||||||
|
assert len(llm.chat_calls) == 1
|
||||||
|
user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user")
|
||||||
|
assert "TOP-TEXT-CONTENT" in user
|
||||||
|
assert "SECRET-CONTENT" not in user and "IGNORED-TEXT-CONTENT" not in user
|
||||||
|
top = next(d for d in docs if d.path == "top.txt")
|
||||||
|
assert top.summary is not None
|
||||||
|
_reset(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_newly_ignored_file_pruned_on_next_prune_run(db: Session, tmp_path: Path) -> None:
|
||||||
|
_reset(db)
|
||||||
|
root = tmp_path / NAME
|
||||||
|
_write(root, "keep.md", "# Keep\n\nkept body\n")
|
||||||
|
_write(root, "top.txt", "TOP-TEXT-CONTENT\n")
|
||||||
|
# Exactly ONE file under ignore/ so the A2 prune count pins it.
|
||||||
|
_write(root, "ignore/secret.md", "# Secret\n\nSECRET-CONTENT\n")
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
|
||||||
|
# First run — no map (omitted entirely): everything is indexed,
|
||||||
|
# including ignore/secret.md.
|
||||||
|
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (s1.files, s1.added) == (3, 3)
|
||||||
|
secret = db.scalar(select(Document).where(Document.path == "ignore/secret.md"))
|
||||||
|
assert secret is not None
|
||||||
|
|
||||||
|
# Second run — the owner adds "ignore" (no trailing slash: A1
|
||||||
|
# normalization) and prunes. The file newly matches, never enters
|
||||||
|
# ``seen``, and leaves the index (A2 — the A9 junk-precedent).
|
||||||
|
s2 = asyncio.run(
|
||||||
|
import_sources(
|
||||||
|
[root],
|
||||||
|
llm,
|
||||||
|
session=db,
|
||||||
|
prune=True,
|
||||||
|
ignore_by_root={str(root): ["ignore"]},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert s2.files == 2
|
||||||
|
assert s2.pruned == 1
|
||||||
|
assert (
|
||||||
|
db.scalar(select(Document).where(Document.path == "ignore/secret.md")) is None
|
||||||
|
)
|
||||||
|
_reset(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_progress_total_excludes_ignored(db: Session, tmp_path: Path) -> None:
|
||||||
|
_reset(db)
|
||||||
|
root = _tree(tmp_path)
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
calls: list[tuple[str, str, int, int]] = []
|
||||||
|
|
||||||
|
def progress(source: str, rel: str, done: int, total: int) -> None:
|
||||||
|
calls.append((source, rel, done, total))
|
||||||
|
|
||||||
|
summary = asyncio.run(
|
||||||
|
import_sources(
|
||||||
|
[root],
|
||||||
|
llm,
|
||||||
|
session=db,
|
||||||
|
progress=progress,
|
||||||
|
ignore_by_root={str(root): ["ignore/"]},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert summary.files == 2
|
||||||
|
# The phase-64 pre-walk uses the same per-root tuple as the loop:
|
||||||
|
# ``total`` counts ONLY the non-ignored files, and the hook fired
|
||||||
|
# exactly once per imported file.
|
||||||
|
assert [c[2] for c in calls] == [1, 2] # done
|
||||||
|
assert {c[3] for c in calls} == {2} # total — never counts ignored files
|
||||||
|
assert {c[1] for c in calls} == {"keep.md", "top.txt"}
|
||||||
|
_reset(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_map_behavior_is_byte_identical(db: Session, tmp_path: Path) -> None:
|
||||||
|
_reset(db)
|
||||||
|
root = _tree(tmp_path)
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
|
||||||
|
# ``ignore_by_root=None`` (the default): all four files import exactly
|
||||||
|
# as pre-phase-89 callers see them.
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db, ignore_by_root=None))
|
||||||
|
assert summary.files == 4
|
||||||
|
docs = db.scalars(select(Document)).all()
|
||||||
|
assert {(d.source, d.path) for d in docs} == {
|
||||||
|
(NAME, "keep.md"),
|
||||||
|
(NAME, "top.txt"),
|
||||||
|
(NAME, "ignore/secret.md"),
|
||||||
|
(NAME, "ignore/notes.txt"),
|
||||||
|
}
|
||||||
|
_reset(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unlisted_source_unaffected(db: Session, tmp_path: Path) -> None:
|
||||||
|
_reset(db)
|
||||||
|
root_a = _tree(tmp_path, name="IgnoreFixA")
|
||||||
|
root_b = tmp_path / "IgnoreFixB"
|
||||||
|
_write(root_b, "one.md", "# One\n\none body\n")
|
||||||
|
_write(root_b, "two.txt", "TWO-TEXT-CONTENT\n")
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
|
||||||
|
# The map keys ONLY the first root — the second imports everything.
|
||||||
|
summary = asyncio.run(
|
||||||
|
import_sources(
|
||||||
|
[root_a, root_b],
|
||||||
|
llm,
|
||||||
|
session=db,
|
||||||
|
ignore_by_root={str(root_a): ["ignore"]},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# A: keep.md + top.txt (the whole ignore/ subtree is dropped) · B: one.md + two.txt
|
||||||
|
assert summary.files == 4
|
||||||
|
docs = db.scalars(select(Document)).all()
|
||||||
|
assert {(d.source, d.path) for d in docs} == {
|
||||||
|
("IgnoreFixA", "keep.md"),
|
||||||
|
("IgnoreFixA", "top.txt"),
|
||||||
|
("IgnoreFixB", "one.md"),
|
||||||
|
("IgnoreFixB", "two.txt"),
|
||||||
|
}
|
||||||
|
assert not any(d.path == "ignore/secret.md" for d in docs)
|
||||||
|
_reset(db)
|
||||||
@@ -238,6 +238,10 @@ class FakeImportSources:
|
|||||||
# Phase 64 (task 02): the progress hook the runner passes (a live
|
# Phase 64 (task 02): the progress hook the runner passes (a live
|
||||||
# closure while wired, None if the wiring regresses).
|
# closure while wired, None if the wiring regresses).
|
||||||
self.progress_hooks: list[object] = []
|
self.progress_hooks: list[object] = []
|
||||||
|
# Phase 89: the per-root ignore map the runner builds from the
|
||||||
|
# rows' ``ignore_paths`` (keyed by the root string the importer
|
||||||
|
# sees; two rows sharing a root string get the union).
|
||||||
|
self.ignore_maps: list[dict[str, list[str]]] = []
|
||||||
|
|
||||||
async def __call__(
|
async def __call__(
|
||||||
self,
|
self,
|
||||||
@@ -248,11 +252,13 @@ class FakeImportSources:
|
|||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: Session | None = None,
|
session: Session | None = None,
|
||||||
progress: Callable[[str, str, int, int], None] | None = None,
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
|
ignore_by_root: dict[str, list[str]] | None = None,
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
self.sources.append(list(sources))
|
self.sources.append(list(sources))
|
||||||
self.llms.append(llm)
|
self.llms.append(llm)
|
||||||
self.prune_flags.append(prune)
|
self.prune_flags.append(prune)
|
||||||
self.progress_hooks.append(progress)
|
self.progress_hooks.append(progress)
|
||||||
|
self.ignore_maps.append(ignore_by_root or {})
|
||||||
if self.delay:
|
if self.delay:
|
||||||
await asyncio.sleep(self.delay)
|
await asyncio.sleep(self.delay)
|
||||||
return self.summary
|
return self.summary
|
||||||
@@ -747,6 +753,7 @@ def test_import_error_is_reported_with_credentials_masked(
|
|||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: Session | None = None,
|
session: Session | None = None,
|
||||||
progress: Callable[[str, str, int, int], None] | None = None,
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
|
ignore_by_root: dict[str, list[str]] | None = None,
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
raise EmbeddingError(
|
raise EmbeddingError(
|
||||||
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
||||||
@@ -897,3 +904,128 @@ def test_probe_runs_before_source_resolution(
|
|||||||
assert order == ["probe", "effective_sources"]
|
assert order == ["probe", "effective_sources"]
|
||||||
assert clone_calls == []
|
assert clone_calls == []
|
||||||
assert "short-circuit" in body["error"] # the spy aborted the run
|
assert "short-circuit" in body["error"] # the spy aborted the run
|
||||||
|
|
||||||
|
|
||||||
|
# --- phase 89: per-row ignore lists ------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_local_row_ignore_paths_excluded_from_sync(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
tmp_path: Path,
|
||||||
|
clean_documents: None,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89: a local row carrying ``ignore_paths`` — the button
|
||||||
|
sync skips every matching file: the ignored file never lands in the
|
||||||
|
KB (no document row — hence no embedding, no summary), and the
|
||||||
|
success detail's ``files``/``added`` counts exclude it, while the
|
||||||
|
kept file imports as usual."""
|
||||||
|
local_dir = tmp_path / "LocalDocs"
|
||||||
|
local_dir.mkdir()
|
||||||
|
(local_dir / "keep.md").write_text("# Keep\nin scope\n", encoding="utf-8")
|
||||||
|
(local_dir / "ignore").mkdir()
|
||||||
|
(local_dir / "ignore" / "secret.md").write_text("# Secret\nignored\n",
|
||||||
|
encoding="utf-8")
|
||||||
|
db.add(
|
||||||
|
GitSource(
|
||||||
|
url=str(local_dir), kind="local", path=str(local_dir),
|
||||||
|
ignore_paths=["ignore/"], # any spelling — the importer normalizes
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||||
|
)
|
||||||
|
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
body = _poll(sync_client, "success")
|
||||||
|
|
||||||
|
# The ignored file is not counted: only keep.md was walked.
|
||||||
|
assert body["detail"]["files"] == 1
|
||||||
|
assert body["detail"]["added"] == 1
|
||||||
|
assert body["detail"]["errors"] == 0
|
||||||
|
# The KB holds exactly the kept file — ignore/secret.md is absent.
|
||||||
|
docs = [(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]]
|
||||||
|
assert docs == [("LocalDocs", "keep.md")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_builds_ignore_map_by_root_string_with_union(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89 wiring (fake import): the runner keys the map by the
|
||||||
|
SAME root string the importer sees, and two rows sharing that root
|
||||||
|
string (the sibling/repo-name edge — ``…/shared`` and
|
||||||
|
``…/shared.git`` clone into the same checkout dir) get the UNION
|
||||||
|
of their lists, in row order; a row with an empty list contributes
|
||||||
|
nothing."""
|
||||||
|
url_a = f"file://{tmp_path / 'shared'}"
|
||||||
|
url_b = f"{url_a}.git" # same repo name → same checkout dir
|
||||||
|
# Distinct added_at: the resolver orders by (added_at, id) — a
|
||||||
|
# same-timestamp pair would tie-break on the random uuid.
|
||||||
|
db.add(GitSource(url=url_a, kind="git", ignore_paths=["a/"],
|
||||||
|
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
|
||||||
|
db.add(GitSource(url=url_b, kind="git", ignore_paths=["b"],
|
||||||
|
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
|
||||||
|
db.commit()
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||||
|
)
|
||||||
|
_, fake_clone = _fake_clone()
|
||||||
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||||
|
_stub_probe(monkeypatch)
|
||||||
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||||
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
_poll(sync_client, "success")
|
||||||
|
|
||||||
|
shared = str(tmp_path / "bor" / "shared")
|
||||||
|
# Both rows resolve to the SAME checkout (the collision itself) and
|
||||||
|
# the map holds their union, keyed by that one root string.
|
||||||
|
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
|
||||||
|
assert fake_import.ignore_maps == [{shared: ["a/", "b"]}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_without_ignore_lists_passes_empty_map(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 89 regression: rows without a list → the runner passes an
|
||||||
|
EMPTY map (the importer's byte-identical pre-phase-89 behavior)."""
|
||||||
|
local_dir = tmp_path / "LocalDocs"
|
||||||
|
local_dir.mkdir()
|
||||||
|
(local_dir / "notes.md").write_text("# Notes\nplain row\n", encoding="utf-8")
|
||||||
|
_seed_local(db, local_dir) # no ignore_paths → []
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||||
|
)
|
||||||
|
_stub_probe(monkeypatch)
|
||||||
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||||
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
_poll(sync_client, "success")
|
||||||
|
|
||||||
|
assert fake_import.sources == [[local_dir]]
|
||||||
|
assert fake_import.ignore_maps == [{}] # no row carried a list
|
||||||
|
|||||||
@@ -763,11 +763,16 @@ def test_no_progress_means_no_prewalk(
|
|||||||
walk_calls = 0
|
walk_calls = 0
|
||||||
|
|
||||||
def counting(
|
def counting(
|
||||||
r: Path, extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS
|
r: Path,
|
||||||
|
extensions: frozenset[str],
|
||||||
|
excluded: frozenset[str] = EXCLUDED_DIRS,
|
||||||
|
ignore: tuple[str, ...] = (),
|
||||||
) -> list[Path]:
|
) -> list[Path]:
|
||||||
|
# Phase 89: the walker gained the ``ignore`` keyword — the sentinel
|
||||||
|
# accepts (and forwards) it to stay a drop-in.
|
||||||
nonlocal walk_calls
|
nonlocal walk_calls
|
||||||
walk_calls += 1
|
walk_calls += 1
|
||||||
return real_walker(r, extensions, excluded)
|
return real_walker(r, extensions, excluded, ignore)
|
||||||
|
|
||||||
monkeypatch.setattr(importer, "iter_importable_files", counting)
|
monkeypatch.setattr(importer, "iter_importable_files", counting)
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -0,0 +1,180 @@
|
|||||||
|
"""Unit: the phase-89 ignore-path matcher (source-level pins).
|
||||||
|
|
||||||
|
Phase 89 (TODO.md L3 — "The ignore is just a prefix ignore."): each
|
||||||
|
stored source carries a list of source-relative path prefixes; a file
|
||||||
|
is ignored when its source-relative POSIX path (the ``documents.path``
|
||||||
|
string — NO leading slash) STARTS WITH a normalized entry. The spec's
|
||||||
|
own examples are pinned verbatim:
|
||||||
|
|
||||||
|
* ``"/my/files/"``, ``"my/files/"`` and ``"my/files"`` all normalize to
|
||||||
|
``"my/files"`` (whitespace + ALL leading/trailing slashes stripped);
|
||||||
|
* ``"myfile.txt"`` matches ``"myfile.txt"`` but NOT
|
||||||
|
``"some/path/myfile.txt"`` — no mid-path matching, no globs;
|
||||||
|
* raw string prefix, deliberately NO component boundary (A1,
|
||||||
|
owner-confirmed 2026-09-08): ``"my/files"`` also matches
|
||||||
|
``"my/files2/x.md"`` and ``"a"`` matches ``"ab.md"``.
|
||||||
|
|
||||||
|
Normalization happens exactly once — in ``_ignore_for_root``, the
|
||||||
|
single choke point — so ``iter_importable_files`` receives
|
||||||
|
ALREADY-normalized tuples (that contract is pinned too: a raw box line
|
||||||
|
does NOT match at the walk level). The DB-facing behavior (ignored
|
||||||
|
files never indexed/embedded/summarized; A2 prune) is integration-
|
||||||
|
gated by ``tests/integration/test_importer_ignore.py``.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.rag.importer import (
|
||||||
|
_ignore_for_root,
|
||||||
|
is_ignored,
|
||||||
|
iter_importable_files,
|
||||||
|
normalize_ignore_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The walk-level fixture uses an explicit extension set (not the A9
|
||||||
|
#: config default) so the test pins the matcher, not the config.
|
||||||
|
EXTS = frozenset({".md", ".txt", ".yaml"})
|
||||||
|
|
||||||
|
|
||||||
|
# --- normalize_ignore_path -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("raw", "expected"),
|
||||||
|
[
|
||||||
|
("/my/files/", "my/files"), # spec: all three spellings are equal
|
||||||
|
("my/files/", "my/files"),
|
||||||
|
("my/files", "my/files"),
|
||||||
|
(" my/files ", "my/files"), # surrounding whitespace trimmed
|
||||||
|
("//", ""), # nothing left after stripping
|
||||||
|
("", ""),
|
||||||
|
(" ", ""),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_normalize_ignore_path(raw: str, expected: str) -> None:
|
||||||
|
assert normalize_ignore_path(raw) == expected
|
||||||
|
|
||||||
|
|
||||||
|
# --- is_ignored: the spec's own examples, verbatim --------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_ignored_myfile_txt_root_level_matches() -> None:
|
||||||
|
# Spec verbatim: "myfile.txt" would match "/myfile.txt" — the leading
|
||||||
|
# slash never exists in documents.path, so the root-level file matches.
|
||||||
|
assert is_ignored("myfile.txt", ("myfile.txt",)) is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"rel",
|
||||||
|
[
|
||||||
|
"my/files/a.md", # spec: ignores everything under the prefix
|
||||||
|
"my/files/sub/b.md", # …and arbitrarily deep
|
||||||
|
"my/files", # a file literally named after the prefix
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_is_ignored_prefix_matches(rel: str) -> None:
|
||||||
|
assert is_ignored(rel, ("my/files",)) is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"rel",
|
||||||
|
[
|
||||||
|
"some/path/myfile.txt", # spec verbatim: NO mid-path matching —
|
||||||
|
# the string simply does not start with the pattern
|
||||||
|
"xmyfile.txt", # no suffix/partial matching either
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_is_ignored_prefix_no_match(rel: str) -> None:
|
||||||
|
assert is_ignored(rel, ("myfile.txt",)) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_ignored_root_level_dir_is_prefix() -> None:
|
||||||
|
# A root-level dir IS prefix matching: "files" covers files/**.
|
||||||
|
assert is_ignored("files/x.md", ("files",)) is True
|
||||||
|
# …but not a same-named dir deeper down (no mid-path matching).
|
||||||
|
assert is_ignored("some/files/x.md", ("files",)) is False
|
||||||
|
|
||||||
|
|
||||||
|
# --- is_ignored: the raw-prefix A1 edge (documented, owner-confirmed) -------
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_ignored_raw_prefix_no_component_boundary() -> None:
|
||||||
|
# A1: raw string startswith — "my/files" ALSO ignores "my/files2/x.md".
|
||||||
|
assert is_ignored("my/files2/x.md", ("my/files",)) is True
|
||||||
|
assert is_ignored("ab.md", ("a",)) is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("rel", ["a.md", "some/a.md", "anything"])
|
||||||
|
def test_is_ignored_empty_prefixes_never_match(rel: str) -> None:
|
||||||
|
assert is_ignored(rel, ()) is False
|
||||||
|
|
||||||
|
|
||||||
|
# --- iter_importable_files: the walk-level skip ------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def tree(tmp_path: Path) -> Path:
|
||||||
|
root = tmp_path / "tree"
|
||||||
|
(root / "ignore" / "deep").mkdir(parents=True)
|
||||||
|
(root / "keep.md").write_text("keep\n", encoding="utf-8")
|
||||||
|
(root / "ignore" / "secret.md").write_text("secret\n", encoding="utf-8")
|
||||||
|
(root / "ignore" / "deep" / "x.yaml").write_text("x: 1\n", encoding="utf-8")
|
||||||
|
(root / "notes.txt").write_text("notes\n", encoding="utf-8")
|
||||||
|
(root / "myfiles.md").write_text("myfiles\n", encoding="utf-8")
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _rels(paths: list[Path], root: Path) -> set[str]:
|
||||||
|
return {p.relative_to(root).as_posix() for p in paths}
|
||||||
|
|
||||||
|
|
||||||
|
ALL_FIVE = {
|
||||||
|
"keep.md",
|
||||||
|
"ignore/secret.md",
|
||||||
|
"ignore/deep/x.yaml",
|
||||||
|
"notes.txt",
|
||||||
|
"myfiles.md",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_iter_default_ignore_keeps_everything(tree: Path) -> None:
|
||||||
|
# Default ``ignore=()`` — every existing caller byte-identical.
|
||||||
|
assert _rels(iter_importable_files(tree, EXTS), tree) == ALL_FIVE
|
||||||
|
|
||||||
|
|
||||||
|
def test_iter_ignore_prefix_skips_matching_files(tree: Path) -> None:
|
||||||
|
got = _rels(iter_importable_files(tree, EXTS, ignore=("ignore",)), tree)
|
||||||
|
assert got == {"keep.md", "notes.txt", "myfiles.md"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_iter_ignore_receives_already_normalized_tuples(tree: Path) -> None:
|
||||||
|
# Contract pin: normalization is the caller's job at THIS level — it
|
||||||
|
# happens in _ignore_for_root, not here. A raw box line does NOT
|
||||||
|
# match, so all five files still come back.
|
||||||
|
got = _rels(iter_importable_files(tree, EXTS, ignore=("//ignore/ ",)), tree)
|
||||||
|
assert got == ALL_FIVE
|
||||||
|
|
||||||
|
|
||||||
|
# --- _ignore_for_root: the single normalization choke point ------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_ignore_for_root_normalizes_and_drops_empties(tmp_path: Path) -> None:
|
||||||
|
root = tmp_path / "src"
|
||||||
|
root.mkdir()
|
||||||
|
got = _ignore_for_root(root, {str(root): ["/a/", "b//", "", " "]})
|
||||||
|
assert got == ("a", "b")
|
||||||
|
|
||||||
|
|
||||||
|
def test_ignore_for_root_missing_root_yields_empty(tmp_path: Path) -> None:
|
||||||
|
root = tmp_path / "src"
|
||||||
|
root.mkdir()
|
||||||
|
assert _ignore_for_root(root, {"/other/root": ["a"]}) == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_ignore_for_root_none_map_yields_empty(tmp_path: Path) -> None:
|
||||||
|
root = tmp_path / "src"
|
||||||
|
root.mkdir()
|
||||||
|
assert _ignore_for_root(root, None) == ()
|
||||||
@@ -6,7 +6,8 @@ on; these tests lock the table/column contract (PLAN §5) without a live DB.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from pgvector.sqlalchemy import Vector
|
from pgvector.sqlalchemy import Vector
|
||||||
from sqlalchemy import UniqueConstraint
|
from sqlalchemy import TextClause, UniqueConstraint
|
||||||
|
from sqlalchemy.sql.schema import DefaultClause
|
||||||
|
|
||||||
import app.models # noqa: F401 (import registers all tables on Base.metadata)
|
import app.models # noqa: F401 (import registers all tables on Base.metadata)
|
||||||
from app.db import Base
|
from app.db import Base
|
||||||
@@ -62,6 +63,44 @@ def test_doc_drafts_token_is_unique_not_null() -> None:
|
|||||||
assert uq, "doc_drafts must be unique on (token) — the URL credential"
|
assert uq, "doc_drafts must be unique on (token) — the URL credential"
|
||||||
|
|
||||||
|
|
||||||
|
def test_git_sources_ignore_paths_column_contract() -> None:
|
||||||
|
"""Phase 89: every source row carries its ignore list — JSONB, NOT
|
||||||
|
NULL, server default ``'[]'`` (a pre-phase-89 row reads ``[]``, so
|
||||||
|
every existing source imports exactly as before)."""
|
||||||
|
sources = Base.metadata.tables["git_sources"]
|
||||||
|
assert "ignore_paths" in sources.c, (
|
||||||
|
"git_sources must have the ignore_paths column (phase 89)"
|
||||||
|
)
|
||||||
|
col = sources.c["ignore_paths"]
|
||||||
|
assert col.nullable is False, "git_sources.ignore_paths must be NOT NULL"
|
||||||
|
sd = col.server_default
|
||||||
|
assert isinstance(sd, DefaultClause), "ignore_paths needs a server default"
|
||||||
|
assert isinstance(sd.arg, TextClause), (
|
||||||
|
"the server default must be the literal SQL text '[]'"
|
||||||
|
)
|
||||||
|
assert sd.arg.text == "'[]'", "ignore_paths server default must be '[]'"
|
||||||
|
|
||||||
|
|
||||||
|
def test_git_source_python_default_empty_list() -> None:
|
||||||
|
"""A freshly constructed row (no DB) resolves to an empty ignore
|
||||||
|
list via the Python-side default (``default=list``) — the ORM
|
||||||
|
INSERT-time default, so an ORM insert that omits the column inserts
|
||||||
|
``[]`` rather than NULL (the server default ``'[]'`` independently
|
||||||
|
covers non-ORM inserts)."""
|
||||||
|
from app.models import GitSource
|
||||||
|
|
||||||
|
row = GitSource(url="https://example.com/r.git", kind="git")
|
||||||
|
col = row.__table__.c["ignore_paths"]
|
||||||
|
assert col.default is not None, (
|
||||||
|
"ignore_paths needs a Python-side (INSERT-time) default"
|
||||||
|
)
|
||||||
|
# Invoked with the (unused) execution context at INSERT time — a
|
||||||
|
# freshly constructed row that omits the kwarg stores [] rather
|
||||||
|
# than NULL (the real-DB behaviour is pinned against the dev DB
|
||||||
|
# by the git-sources API integration tests).
|
||||||
|
assert col.default.arg(None) == [], "the default must resolve to []"
|
||||||
|
|
||||||
|
|
||||||
def test_doc_drafts_column_contract() -> None:
|
def test_doc_drafts_column_contract() -> None:
|
||||||
"""Phase 59: the editable triple (title/path/body) + status +
|
"""Phase 59: the editable triple (title/path/body) + status +
|
||||||
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
|
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
|
||||||
|
|||||||
@@ -0,0 +1,546 @@
|
|||||||
|
"""Unit: the per-source ignore-paths editor on the Sources page
|
||||||
|
(phase 89, task 05).
|
||||||
|
|
||||||
|
``TODO.md`` L3: "They should be able to type these files and folders
|
||||||
|
into a box on the sources page." Phase 89 gives every STORED source
|
||||||
|
row (git or local) a one-path-per-line ignore box on
|
||||||
|
``/git-sources.html``: a page-local alertdialog (the EXACT
|
||||||
|
``#remove-confirm-dialog`` pattern, phase 69) with a visible label, a
|
||||||
|
mono textarea prefilled from the row, the §7.4 never-stale save
|
||||||
|
lifecycle ("Saving…" while the ``PATCH /api/git-sources/{id}`` is
|
||||||
|
out — the A5 replace round-trips through ``GET``), and full a11y
|
||||||
|
(focus on Cancel, Escape / backdrop close as CANCEL, focus return,
|
||||||
|
``role="alert"`` error line that keeps the textarea content on 422).
|
||||||
|
Rows with a list show the ``N ignored`` count tag; env-fallback rows
|
||||||
|
(``id`` null) get NO box (A3 — nothing is stored to edit).
|
||||||
|
|
||||||
|
The browser behavior itself is E2E-gated by the phase-89 story suite
|
||||||
|
(``tests/e2e/test_source_ignore_paths.py``, task 06); like the other
|
||||||
|
frontend-adjacent unit files (the
|
||||||
|
``test_remove_confirm_modal.py`` house pattern), this module pins the
|
||||||
|
source-level contract a silent regression would break:
|
||||||
|
|
||||||
|
* the static ``#ignore-editor-dialog`` markup — ``role=
|
||||||
|
"alertdialog"`` + ``aria-modal`` + ``aria-labelledby``, hidden by
|
||||||
|
default, inside the manager, the visible ``<label for=…>`` (never
|
||||||
|
aria-label-only), the ``role="alert"`` error line, real
|
||||||
|
``type="button"`` buttons, the mono textarea (``rows=6`` /
|
||||||
|
``spellcheck="false"``);
|
||||||
|
* the JS lifecycle — ``openIgnoreEditor`` (textContent-only source
|
||||||
|
population with makeRow's ``value`` expression, the textarea
|
||||||
|
prefilled from ``(s.ignore_paths || [])``, error cleared, focus on
|
||||||
|
Cancel), cancel = Escape / Cancel button / backdrop (no request;
|
||||||
|
focus returns to the trigger; a no-op while a PATCH is in flight),
|
||||||
|
``saveIgnorePaths`` (§7.4 in-flight state: both buttons disable +
|
||||||
|
"Saving…", blank lines dropped client-side, success → close /
|
||||||
|
reload / announce — the update confirmation LAST, non-2xx → the
|
||||||
|
in-dialog alert line with the textarea content KEPT, network →
|
||||||
|
the fixed reachable? line, re-enable in the finally);
|
||||||
|
* the row wiring — the "Ignore paths" button exists ONLY in the
|
||||||
|
``s.id`` branch of ``makeRow`` (left of Remove; the aria-label is
|
||||||
|
the only place the value appears — never innerHTML), the ``N
|
||||||
|
ignored`` count tag on rows with a list (text, never color alone);
|
||||||
|
* styles.css — the dialog / button / count-tag rules on the house
|
||||||
|
dark-tech palette (phase-08 tokens only, no CDN, no blur),
|
||||||
|
≥44px targets, the ``[hidden]`` override, the mono full-width
|
||||||
|
textarea;
|
||||||
|
* the no-collision guard — every ``id="ignore-editor-…"`` appears
|
||||||
|
EXACTLY ONCE in the shell (the phase-46/76 contract).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||||
|
# Phase 76 (task 02): git-sources.html is folded into the ONE-document
|
||||||
|
# shell — the dialog markup lives in the Sources view section of
|
||||||
|
# index.html (next to #remove-confirm-dialog).
|
||||||
|
SHELL_HTML = FRONTEND / "index.html"
|
||||||
|
JS = FRONTEND / "assets" / "git-sources.js"
|
||||||
|
CSS = FRONTEND / "assets" / "styles.css"
|
||||||
|
|
||||||
|
#: The dialog's static ids (the no-collision guard + the markup pins
|
||||||
|
#: key on exactly these).
|
||||||
|
DIALOG_IDS = (
|
||||||
|
"ignore-editor-dialog",
|
||||||
|
"ignore-editor-title",
|
||||||
|
"ignore-editor-source",
|
||||||
|
"ignore-editor-copy",
|
||||||
|
"ignore-editor-textarea",
|
||||||
|
"ignore-editor-error",
|
||||||
|
"ignore-editor-cancel",
|
||||||
|
"ignore-editor-save",
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The §7.4 in-flight label + the idle label (pinned verbatim).
|
||||||
|
SAVING_LABEL = "Saving…"
|
||||||
|
IDLE_LABEL = "Save"
|
||||||
|
COUNT_CLASS = "git-source-ignore-count"
|
||||||
|
ROW_BTN_CLASS = "git-source-ignore"
|
||||||
|
ARIA_LABEL_TEMPLATE = "`Edit ignored paths for ${kindLabel} source: ${value}`"
|
||||||
|
|
||||||
|
#: The success announce — the update confirmation is the LAST
|
||||||
|
#: announcement (the reload's "N sources listed." must not overwrite
|
||||||
|
#: it, the same order as the remove flow).
|
||||||
|
ANNOUNCE_OK = "Ignored paths updated for ${value}"
|
||||||
|
|
||||||
|
|
||||||
|
def _text(path: Path) -> str:
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _js() -> str:
|
||||||
|
return _text(JS)
|
||||||
|
|
||||||
|
|
||||||
|
def _css() -> str:
|
||||||
|
return _text(CSS)
|
||||||
|
|
||||||
|
|
||||||
|
def _fn(js: str, name: str) -> str:
|
||||||
|
"""The source of a (possibly async) top-level function via
|
||||||
|
balanced-brace counting (the test_remove_confirm_modal.py helper)."""
|
||||||
|
for prefix in ("async function ", "function "):
|
||||||
|
start = js.find(f"{prefix}{name}(")
|
||||||
|
if start != -1:
|
||||||
|
depth = 0
|
||||||
|
for i in range(js.find("{", start), len(js)):
|
||||||
|
if js[i] == "{":
|
||||||
|
depth += 1
|
||||||
|
elif js[i] == "}":
|
||||||
|
depth -= 1
|
||||||
|
if depth == 0:
|
||||||
|
return js[start : i + 1]
|
||||||
|
raise AssertionError(f"unbalanced braces in {name}()")
|
||||||
|
raise AssertionError(f"{name}() must exist in git-sources.js")
|
||||||
|
|
||||||
|
|
||||||
|
def _element_block(html: str, id_attr: str, tag: str = "div") -> str:
|
||||||
|
"""The <tag … id=…> element's full markup (a balanced-tag walk —
|
||||||
|
the dialog nests the backdrop / panel / actions divs)."""
|
||||||
|
marker = f'id="{id_attr}"'
|
||||||
|
i = html.find(marker)
|
||||||
|
assert i != -1, f"missing id={id_attr} in the shell's Sources view"
|
||||||
|
opens = [m.start() for m in re.finditer(rf"<{tag}\b", html[:i])]
|
||||||
|
assert opens, f"no <{tag}> owns id={id_attr}"
|
||||||
|
open_i = opens[-1]
|
||||||
|
depth = 0
|
||||||
|
for m in re.finditer(rf"<{tag}\b[^>]*>|</{tag}>", html[open_i :]):
|
||||||
|
if m.group(0).startswith(f"</{tag}>"):
|
||||||
|
depth -= 1
|
||||||
|
else:
|
||||||
|
depth += 1
|
||||||
|
if depth == 0:
|
||||||
|
return html[open_i : open_i + m.end()]
|
||||||
|
raise AssertionError(f"unbalanced <{tag}> for id={id_attr}")
|
||||||
|
|
||||||
|
|
||||||
|
def _css_rule(css: str, selector: str) -> str:
|
||||||
|
"""The declarations of a simple one-line-opening rule (comments
|
||||||
|
stripped first — a house comment may legally carry braces)."""
|
||||||
|
clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
|
||||||
|
start = clean.find(f"{selector} {{")
|
||||||
|
assert start != -1, f"missing rule {selector} in styles.css"
|
||||||
|
depth = 0
|
||||||
|
for i in range(clean.find("{", start), len(clean)):
|
||||||
|
if clean[i] == "{":
|
||||||
|
depth += 1
|
||||||
|
elif clean[i] == "}":
|
||||||
|
depth -= 1
|
||||||
|
if depth == 0:
|
||||||
|
return clean[start : i + 1]
|
||||||
|
raise AssertionError(f"unbalanced braces in {selector}")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the static dialog markup ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_dialog_markup_is_the_locked_alertdialog() -> None:
|
||||||
|
"""#ignore-editor-dialog: role="alertdialog" + aria-modal +
|
||||||
|
aria-labelledby, hidden by default, INSIDE the manager
|
||||||
|
(#git-sources-content, directly after #remove-confirm-dialog —
|
||||||
|
the static-markup convention, stable E2E selectors); every
|
||||||
|
static id present; the error line is role="alert" + hidden;
|
||||||
|
both buttons are real type="button"; the textarea is mono-
|
||||||
|
capable (rows=6, spellcheck=false, the A1 prefix placeholder)."""
|
||||||
|
html = _text(SHELL_HTML)
|
||||||
|
frag = _element_block(html, "ignore-editor-dialog")
|
||||||
|
open_tag = frag[: frag.find(">") + 1]
|
||||||
|
assert 'role="alertdialog"' in open_tag
|
||||||
|
assert 'aria-modal="true"' in open_tag
|
||||||
|
assert 'aria-labelledby="ignore-editor-title"' in open_tag
|
||||||
|
assert 'aria-describedby="ignore-editor-copy"' in open_tag
|
||||||
|
assert "hidden" in open_tag, "the dialog ships hidden"
|
||||||
|
# Placed in the manager, right after the remove-confirm dialog
|
||||||
|
# (the same placement logic).
|
||||||
|
assert html.find('id="git-sources-content"') < html.find('id="ignore-editor-dialog"')
|
||||||
|
assert html.find('id="remove-confirm-dialog"') < html.find('id="ignore-editor-dialog"')
|
||||||
|
for child in DIALOG_IDS:
|
||||||
|
assert f'id="{child}"' in frag, f"missing #{child} in the dialog"
|
||||||
|
# The visible label (WCAG — never aria-label-only): a real
|
||||||
|
# <label for="ignore-editor-textarea"> with the house label class.
|
||||||
|
label = re.search(
|
||||||
|
r"<label[^>]*class=\"ignore-editor-label\"[^>]*for=\"ignore-editor-textarea\"[^>]*>(.*?)</label>",
|
||||||
|
frag,
|
||||||
|
re.S,
|
||||||
|
)
|
||||||
|
assert label and label.group(1).strip(), "a visible block label for the textarea"
|
||||||
|
# The error line: role="alert", hidden by default.
|
||||||
|
err = re.search(r"<p[^>]*id=\"ignore-editor-error\"[^>]*>", frag)
|
||||||
|
assert err and 'role="alert"' in err.group(0) and "hidden" in err.group(0)
|
||||||
|
# The mono box: rows + spellcheck off + the A1 placeholder.
|
||||||
|
ta = re.search(r"<textarea[^>]*id=\"ignore-editor-textarea\"[^>]*>", frag, re.S)
|
||||||
|
assert ta, "#ignore-editor-textarea is a real <textarea>"
|
||||||
|
for attr in ('rows="6"', 'spellcheck="false"', 'placeholder="my/files/"'):
|
||||||
|
assert attr in ta.group(0), f"the textarea carries {attr}"
|
||||||
|
for btn in ("ignore-editor-cancel", "ignore-editor-save"):
|
||||||
|
m = re.search(rf"<button[^>]*id=\"{btn}\"[^>]*>", frag)
|
||||||
|
assert m and 'type="button"' in m.group(0), f"#{btn} is a real type=button"
|
||||||
|
cancel = re.search(r'<button[^>]*id="ignore-editor-cancel"[^>]*>(.*?)</button>', frag, re.S)
|
||||||
|
save = re.search(r'<button[^>]*id="ignore-editor-save"[^>]*>(.*?)</button>', frag, re.S)
|
||||||
|
assert cancel and cancel.group(1).strip() == "Cancel"
|
||||||
|
assert save and save.group(1).strip() == IDLE_LABEL
|
||||||
|
# The helper copy states the A1 prefix rule in plain words.
|
||||||
|
copy = re.search(r"<p[^>]*id=\"ignore-editor-copy\"[^>]*>(.*?)</p>", frag, re.S)
|
||||||
|
assert copy, "the helper copy paragraph"
|
||||||
|
copy_norm = re.sub(r"\s+", " ", copy.group(1))
|
||||||
|
assert "No wildcards" in copy_norm and "middle of a path" in copy_norm
|
||||||
|
|
||||||
|
|
||||||
|
def test_dialog_ids_appear_exactly_once() -> None:
|
||||||
|
"""The no-collision guard (the phase-46/76 contract): every
|
||||||
|
``id="ignore-editor-…"`` in the NEW markup appears EXACTLY ONCE
|
||||||
|
in the whole shell — no duplicated id (the dialog is the only
|
||||||
|
owner of each id)."""
|
||||||
|
html = _text(SHELL_HTML)
|
||||||
|
for id_ in DIALOG_IDS:
|
||||||
|
n = html.count(f'id="{id_}"')
|
||||||
|
assert n == 1, f'id={id_} appears {n} times in index.html (must be exactly 1)'
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the row wiring in makeRow ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_button_exists_only_in_the_stored_row_branch() -> None:
|
||||||
|
"""makeRow: the "Ignore paths" button is created ONLY in the
|
||||||
|
``s.id`` branch (A3 — env-fallback rows get the "from .env" tag,
|
||||||
|
no button) and sits BEFORE the Remove button (left of Remove).
|
||||||
|
The button is class git-source-ignore, type=button, textContent
|
||||||
|
label (never innerHTML — the value appears ONLY in the aria-
|
||||||
|
label, built via setAttribute); its click opens the editor."""
|
||||||
|
make = _fn(_js(), "makeRow")
|
||||||
|
branch_i = make.find("if (s.id) {")
|
||||||
|
else_i = make.find("} else {", branch_i)
|
||||||
|
bind_i = make.find("openIgnoreEditor(s, ignoreBtn)")
|
||||||
|
assert -1 < branch_i < bind_i < else_i, (
|
||||||
|
"the ignore button is created in the s.id branch (the phase-88 pin idiom)"
|
||||||
|
)
|
||||||
|
remove_i = make.find('btn.className = "git-source-remove"', bind_i)
|
||||||
|
assert remove_i > bind_i, "the ignore button is LEFT of Remove"
|
||||||
|
assert f'ignoreBtn.className = "{ROW_BTN_CLASS}"' in make
|
||||||
|
assert "ignoreBtn.type = \"button\"" in make
|
||||||
|
assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)"
|
||||||
|
assert "ignoreBtn.textContent = \"Ignore paths\"" in make, (
|
||||||
|
"a static text label — never icon-only"
|
||||||
|
)
|
||||||
|
assert "ignoreBtn.innerHTML" not in _js(), "XSS contract: no innerHTML on the button"
|
||||||
|
# The env-fallback branch (else) has no ignore button.
|
||||||
|
else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)]
|
||||||
|
assert ROW_BTN_CLASS not in else_slice and "openIgnoreEditor" not in else_slice, (
|
||||||
|
"env-fallback rows get no ignore button (A3)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_tag_shows_n_ignored_text_next_to_the_location() -> None:
|
||||||
|
"""makeRow: a stored row with a non-empty list gets the
|
||||||
|
``N ignored`` count tag in the LOCATION cell (after the
|
||||||
|
<code>) — text via textContent (never color alone, WCAG 1.4.1);
|
||||||
|
the list reads ``(s.ignore_paths || [])`` — the same expression
|
||||||
|
openIgnoreEditor prefills from (the GET round-trip marker)."""
|
||||||
|
make = _fn(_js(), "makeRow")
|
||||||
|
assert "s.id && (s.ignore_paths || []).length > 0" in make, (
|
||||||
|
"the count tag only for stored rows with a list"
|
||||||
|
)
|
||||||
|
assert f'count.className = "{COUNT_CLASS}"' in make
|
||||||
|
assert "count.textContent = `${s.ignore_paths.length} ignored`" in make, (
|
||||||
|
"N ignored — text, never color alone"
|
||||||
|
)
|
||||||
|
# It lands in the location cell (urlTd), not the actions cell.
|
||||||
|
append_i = make.find("urlTd.append(count)")
|
||||||
|
code_i = make.find("urlTd.append(badge, code)")
|
||||||
|
assert -1 < code_i < append_i, "the tag is appended to the location cell"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the JS lifecycle ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_open_prefills_and_focuses_cancel() -> None:
|
||||||
|
"""openIgnoreEditor(s, triggerBtn): the source value is
|
||||||
|
textContent ONLY (never innerHTML — the credential-masking
|
||||||
|
discipline, phase 32) with makeRow's exact ``value`` expression;
|
||||||
|
the textarea prefills ``(s.ignore_paths || []).join("\\n")``
|
||||||
|
(one path per line — the round-trip marker); the error line
|
||||||
|
clears; the dialog unhides; the trigger is recorded; the keydown
|
||||||
|
handler attaches; and focus lands on Cancel — the safe default
|
||||||
|
(AFTER the unhide)."""
|
||||||
|
body = _fn(_js(), "openIgnoreEditor")
|
||||||
|
assert "s.kind === \"local\"" in body, "the kind-typed value branch"
|
||||||
|
assert (
|
||||||
|
"ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url" in body
|
||||||
|
), "the same `value` expression makeRow uses, via textContent"
|
||||||
|
assert "ignoreSourceEl.innerHTML" not in _js(), "XSS contract: textContent only"
|
||||||
|
assert 'ignoreTextarea.value = (s.ignore_paths || []).join("\\n")' in body, (
|
||||||
|
"the textarea prefills the stored list, one path per line"
|
||||||
|
)
|
||||||
|
assert "ignoreErrorEl.hidden = true" in body, "a new attempt starts clean"
|
||||||
|
assert "ignoreTriggerBtn = triggerBtn" in body, "the trigger is recorded"
|
||||||
|
trigger_i = body.find("ignoreTriggerBtn = triggerBtn")
|
||||||
|
unhide_i = body.find("ignoreDialog.hidden = false")
|
||||||
|
attach_i = body.find('document.addEventListener("keydown", onIgnoreDialogKeydown)')
|
||||||
|
focus_i = body.find("ignoreCancelBtn.focus()")
|
||||||
|
assert -1 < trigger_i < unhide_i < attach_i < focus_i, (
|
||||||
|
"record the trigger → unhide → attach keydown → focus Cancel"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cancel_paths_close_without_a_request() -> None:
|
||||||
|
"""Cancel (Cancel button / Escape / backdrop) closes as cancel:
|
||||||
|
the dialog hides, the textarea + error line reset, the buttons
|
||||||
|
reset ("Save"), the keydown handler detaches, and focus RETURNS
|
||||||
|
to the recorded trigger — and the cancel path never fetches. A
|
||||||
|
cancel while a PATCH is in flight is a no-op (no half-cancel of
|
||||||
|
an in-progress save)."""
|
||||||
|
js = _js()
|
||||||
|
cancel = _fn(js, "cancelIgnoreEditor")
|
||||||
|
guard_i = cancel.find("if (ignoreInFlight) return")
|
||||||
|
close_i = cancel.find("closeIgnoreEditor()")
|
||||||
|
assert -1 < guard_i < close_i, "the in-flight guard precedes the close"
|
||||||
|
assert "fetch" not in cancel, "cancel never sends a request"
|
||||||
|
close = _fn(js, "closeIgnoreEditor")
|
||||||
|
hide_i = close.find("ignoreDialog.hidden = true")
|
||||||
|
reset_ta_i = close.find('ignoreTextarea.value = ""')
|
||||||
|
clear_i = close.find("ignoreErrorEl.hidden = true")
|
||||||
|
reset_btn_i = close.find('"Save"')
|
||||||
|
detach_i = close.find('document.removeEventListener("keydown", onIgnoreDialogKeydown)')
|
||||||
|
save_i = close.find("const trigger = ignoreTriggerBtn")
|
||||||
|
null_i = close.find("ignoreTriggerBtn = null")
|
||||||
|
focus_i = close.find("trigger.focus()")
|
||||||
|
assert -1 < hide_i < reset_ta_i < clear_i < reset_btn_i, (
|
||||||
|
"hide → reset textarea → clear error → reset buttons"
|
||||||
|
)
|
||||||
|
assert reset_btn_i < detach_i < save_i < null_i < focus_i, (
|
||||||
|
"→ detach → save trigger → focus return"
|
||||||
|
)
|
||||||
|
assert "ignoreCancelBtn.disabled = false" in close
|
||||||
|
assert "ignoreSaveBtn.disabled = false" in close
|
||||||
|
|
||||||
|
|
||||||
|
def test_escape_and_backdrop_and_cancel_button_all_cancel() -> None:
|
||||||
|
"""While open (handler attached on document in openIgnoreEditor,
|
||||||
|
detached in closeIgnoreEditor): Escape → preventDefault +
|
||||||
|
cancelIgnoreEditor; the dim backdrop AND the Cancel button wire
|
||||||
|
to cancelIgnoreEditor (only Save wires to saveIgnorePaths)."""
|
||||||
|
js = _js()
|
||||||
|
keydown = _fn(js, "onIgnoreDialogKeydown")
|
||||||
|
esc_i = keydown.find('e.key === "Escape"')
|
||||||
|
prevent_i = keydown.find("e.preventDefault()", esc_i)
|
||||||
|
cancel_i = keydown.find("cancelIgnoreEditor()", esc_i)
|
||||||
|
assert -1 < esc_i < prevent_i < cancel_i, "Escape: prevent + cancel"
|
||||||
|
assert 'ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor)' in js
|
||||||
|
assert 'ignoreBackdrop.addEventListener("click", cancelIgnoreEditor)' in js
|
||||||
|
assert 'ignoreSaveBtn.addEventListener("click", saveIgnorePaths)' in js
|
||||||
|
|
||||||
|
|
||||||
|
def test_save_runs_the_inflight_never_stale_lifecycle() -> None:
|
||||||
|
"""saveIgnorePaths: the box's lines are split on newlines,
|
||||||
|
trimmed, and empty lines DROPPED (a blank line is a separator,
|
||||||
|
not an entry — the server still rejects empties defensively,
|
||||||
|
A4). The §7.4 in-flight state precedes the PATCH: both buttons
|
||||||
|
disable + the save relabels "Saving…" — one
|
||||||
|
``PATCH /api/git-sources/{id}`` with the lines as the whole
|
||||||
|
body list (A5 replace). 200 → close (focus return) →
|
||||||
|
loadSources (the count tag lands) → announce (the update
|
||||||
|
confirmation is the LAST announcement). Non-2xx: the in-dialog
|
||||||
|
role=alert line (apiDetail, 422 shape-aware), the dialog STAYS
|
||||||
|
open and the textarea content is KEPT. Network failure: the
|
||||||
|
fixed reachable? line. The finally re-enables BOTH buttons +
|
||||||
|
relabels "Save" — never stale on any outcome."""
|
||||||
|
body = _fn(_js(), "saveIgnorePaths")
|
||||||
|
guard_i = body.find("if (!ignoreTarget || ignoreInFlight) return")
|
||||||
|
parse_i = body.find('.split("\\n")')
|
||||||
|
trim_i = body.find(".map((l) => l.trim())", parse_i)
|
||||||
|
drop_i = body.find(".filter(Boolean)", trim_i)
|
||||||
|
inflight_i = body.find("ignoreInFlight = true", drop_i)
|
||||||
|
dis_c = body.find("ignoreCancelBtn.disabled = true", inflight_i)
|
||||||
|
dis_s = body.find("ignoreSaveBtn.disabled = true", inflight_i)
|
||||||
|
label_i = body.find(f'"{SAVING_LABEL}"', dis_s)
|
||||||
|
fetch_i = body.find("`/api/git-sources/${", label_i)
|
||||||
|
method_i = body.find('method: "PATCH"', fetch_i)
|
||||||
|
body_i = body.find("JSON.stringify({ ignore_paths: lines })", method_i)
|
||||||
|
assert -1 < guard_i < parse_i < trim_i < drop_i < inflight_i, (
|
||||||
|
"guard → split + trim + drop blank lines → in-flight"
|
||||||
|
)
|
||||||
|
assert -1 < dis_c < dis_s < label_i < fetch_i < method_i < body_i, (
|
||||||
|
"disable both + 'Saving…' → the PATCH with the lines"
|
||||||
|
)
|
||||||
|
# Success: close → reload → announce (the exact order) — the
|
||||||
|
# update confirmation is the LAST announcement: the reload's
|
||||||
|
# "N sources listed." must not overwrite it.
|
||||||
|
ok_i = body.find("if (r.ok)")
|
||||||
|
close_i = body.find("closeIgnoreEditor()", ok_i)
|
||||||
|
reload_i = body.find("await loadSources()", close_i)
|
||||||
|
announce_i = body.find(f"announce(`{ANNOUNCE_OK}`)", reload_i)
|
||||||
|
assert -1 < ok_i < close_i < reload_i < announce_i, (
|
||||||
|
"200: close → loadSources → announce (LAST)"
|
||||||
|
)
|
||||||
|
assert body.count("closeIgnoreEditor()") == 1, (
|
||||||
|
"only the success path closes — failures stay open for one retry"
|
||||||
|
)
|
||||||
|
# Non-2xx: the in-dialog alert line (apiDetail, 422 shape-aware);
|
||||||
|
# the dialog stays open and the textarea content is KEPT.
|
||||||
|
catch_i = body.find("} catch {")
|
||||||
|
nonok_slice = body[ok_i:catch_i]
|
||||||
|
assert "await apiDetail(r," in nonok_slice, (
|
||||||
|
"the server detail is apiDetail-extracted (422 shape-aware)"
|
||||||
|
)
|
||||||
|
assert "ignoreErrorEl.hidden = false" in nonok_slice, "the error line shows"
|
||||||
|
assert "ignoreTextarea.value" not in nonok_slice.split("if (r.ok)")[1], (
|
||||||
|
"the textarea content is KEPT on failure (the instruction survives)"
|
||||||
|
)
|
||||||
|
# Network: the fixed reachable? line.
|
||||||
|
net_i = body.find("Could not save the ignored paths — is the app reachable?", catch_i)
|
||||||
|
assert -1 < net_i < body.find("finally"), "the network copy lands in the catch"
|
||||||
|
# Never stale: the finally re-enables BOTH buttons + relabels.
|
||||||
|
fin = body[body.find("finally"):]
|
||||||
|
assert "ignoreInFlight = false" in fin
|
||||||
|
assert "ignoreCancelBtn.disabled = false" in fin
|
||||||
|
assert "ignoreSaveBtn.disabled = false" in fin
|
||||||
|
assert f'ignoreSaveBtn.textContent = "{IDLE_LABEL}"' in fin
|
||||||
|
|
||||||
|
|
||||||
|
def test_module_docstring_carries_the_phase_89_contract() -> None:
|
||||||
|
"""The git-sources.js module docstring gained the phase-89
|
||||||
|
bullet: the editor is per-STORED-row (A3 env rows excluded),
|
||||||
|
one path per line (the A1 prefix rule), the §7.4 "Saving…"
|
||||||
|
lifecycle, and the A5 replace round-trip."""
|
||||||
|
doc = _js().split("*/", 2)[0] # the module docstring (first block)
|
||||||
|
for frag in (
|
||||||
|
"Phase 89 (task 05)",
|
||||||
|
"ignore-paths editor",
|
||||||
|
"#ignore-editor-dialog",
|
||||||
|
"one path per line",
|
||||||
|
f'"{SAVING_LABEL}"',
|
||||||
|
"LAST announcement",
|
||||||
|
"N ignored",
|
||||||
|
):
|
||||||
|
assert frag in doc, f"the module docstring lost: {frag!r}"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- styles.css ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_dialog_css_rules_present_and_house_tokens_only() -> None:
|
||||||
|
"""styles.css carries the ignore-editor class family on the house
|
||||||
|
dark-tech palette (phase-08 tokens): the overlay + backdrop +
|
||||||
|
panel (the EXACT .remove-confirm treatment), the title / source /
|
||||||
|
copy / visible label / mono textarea / error / actions / two-
|
||||||
|
button chrome; the [hidden] override; no blur (the phase-08
|
||||||
|
no-blur perf anchor); no CDN."""
|
||||||
|
css = _css()
|
||||||
|
for selector in (
|
||||||
|
".ignore-editor",
|
||||||
|
".ignore-editor-backdrop",
|
||||||
|
".ignore-editor-panel",
|
||||||
|
".ignore-editor-title",
|
||||||
|
".ignore-editor-source",
|
||||||
|
".ignore-editor-copy",
|
||||||
|
".ignore-editor-label",
|
||||||
|
".ignore-editor-textarea",
|
||||||
|
".ignore-editor-error",
|
||||||
|
".ignore-editor-actions",
|
||||||
|
".ignore-editor-btn",
|
||||||
|
".ignore-editor-cancel",
|
||||||
|
".ignore-editor-save",
|
||||||
|
):
|
||||||
|
assert f"{selector} " in css or f"{selector}." in css or f"{selector}[" in css, (
|
||||||
|
f"styles.css must style {selector}"
|
||||||
|
)
|
||||||
|
hidden = css.find(".ignore-editor[hidden]")
|
||||||
|
assert hidden != -1 and "display: none" in css[hidden : hidden + 60], (
|
||||||
|
"the hidden attr must beat the display rule"
|
||||||
|
)
|
||||||
|
# No blur (the phase-08 no-blur perf anchor) — the backdrop rule
|
||||||
|
# itself must not carry backdrop-filter (comments stripped).
|
||||||
|
backdrop = re.sub(r"/\*.*?\*/", "", _css_rule(css, ".ignore-editor-backdrop"), flags=re.S)
|
||||||
|
assert "backdrop-filter" not in backdrop, "no blur (phase-08 perf anchor)"
|
||||||
|
assert "url(http" not in css and "@import url(" not in css, (
|
||||||
|
"no CDN (AGENTS.md rule 6)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_textarea_rule_is_mono_and_full_width() -> None:
|
||||||
|
"""The box itself: full panel width (width: 100%) + the mono
|
||||||
|
stack (var(--mono)) + a comfortable min-height — the house
|
||||||
|
comment cites phase 89 on the block."""
|
||||||
|
rule = _css_rule(_css(), ".ignore-editor-textarea")
|
||||||
|
assert "width: 100%" in rule, "full panel width"
|
||||||
|
assert "var(--mono)" in rule, "the mono stack"
|
||||||
|
assert "min-height" in rule, "a comfortable minimum height"
|
||||||
|
# The section comment cites phase 89 (house comment style).
|
||||||
|
css = _css()
|
||||||
|
header = css[css.rfind("/*", 0, css.find(".ignore-editor {")) : css.find(".ignore-editor {")]
|
||||||
|
assert "phase 89" in header, "the house comment cites phase 89"
|
||||||
|
|
||||||
|
|
||||||
|
def test_dialog_button_and_target_contrast_pairs() -> None:
|
||||||
|
"""The WCAG 2.1 AA basics in CSS: both dialog buttons >=44px;
|
||||||
|
Cancel is the ghost ink-soft family (5.1:1 on --surface) with the
|
||||||
|
brand-soft hover (12.4:1); Save is the solid brand family (--bg
|
||||||
|
text on --brand 5.2:1, the .new-chat-btn convention) with the
|
||||||
|
lightened hover; the error line is the err pair; the panel caps
|
||||||
|
at the 46rem chat-column width or the viewport; the visible
|
||||||
|
label is ink-soft (5.1:1) — never a label-less textarea."""
|
||||||
|
css = _css()
|
||||||
|
btn = _css_rule(css, ".ignore-editor-btn")
|
||||||
|
assert "min-height: 44px" in btn and "min-width: 44px" in btn
|
||||||
|
cancel = _css_rule(css, ".ignore-editor-cancel")
|
||||||
|
assert "var(--ink-soft)" in cancel and "transparent" in cancel
|
||||||
|
cancel_hover = _css_rule(css, ".ignore-editor-cancel:hover:not(:disabled)")
|
||||||
|
assert "var(--brand-soft)" in cancel_hover and "var(--brand-ink)" in cancel_hover
|
||||||
|
save = _css_rule(css, ".ignore-editor-save")
|
||||||
|
assert "var(--brand)" in save and "var(--bg)" in save, (
|
||||||
|
"Save: the solid brand family (--bg text on --brand, 5.2:1)"
|
||||||
|
)
|
||||||
|
save_hover = _css_rule(css, ".ignore-editor-save:hover:not(:disabled)")
|
||||||
|
assert "background" in save_hover, "the hover lightens the fill"
|
||||||
|
err = _css_rule(css, ".ignore-editor-error")
|
||||||
|
assert "var(--err-ink)" in err and "var(--err-bg)" in err
|
||||||
|
panel = _css_rule(css, ".ignore-editor-panel")
|
||||||
|
assert "min(46rem" in panel, "the 46rem chat-column cap (or the viewport)"
|
||||||
|
label = _css_rule(css, ".ignore-editor-label")
|
||||||
|
assert "display: block" in label and "var(--ink-soft)" in label, (
|
||||||
|
"a visible block label (WCAG — never aria-label-only)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_button_and_count_tag_css_rules() -> None:
|
||||||
|
"""The row chrome: .git-source-ignore reuses the
|
||||||
|
.git-source-remove idiom (>=44px target, same size/spacing) in a
|
||||||
|
NEUTRAL secondary fill with the brand-soft hover (distinct from
|
||||||
|
the destructive Remove's err hover); .git-source-ignore-count is
|
||||||
|
a small inline tag — text + a distinct background (never color
|
||||||
|
alone), AA on both theme surfaces (--ink on --bg 16.7:1)."""
|
||||||
|
css = _css()
|
||||||
|
row = _css_rule(css, ".git-source-ignore")
|
||||||
|
assert "min-height: 44px" in row and "min-width: 44px" in row, "the >=44px rule"
|
||||||
|
assert "var(--ink-soft)" in row, "the neutral secondary resting fill"
|
||||||
|
row_hover = _css_rule(css, ".git-source-ignore:hover:not(:disabled)")
|
||||||
|
assert "var(--brand-soft)" in row_hover and "var(--brand-ink)" in row_hover, (
|
||||||
|
"the hover takes the brand pair (Remove hovers to the err pair)"
|
||||||
|
)
|
||||||
|
assert "var(--err-" not in row, "not the destructive err family"
|
||||||
|
count = _css_rule(css, ".git-source-ignore-count")
|
||||||
|
assert "var(--ink)" in count and "var(--bg)" in count, (
|
||||||
|
"text + a distinct background (never color alone)"
|
||||||
|
)
|
||||||
@@ -727,6 +727,7 @@ class _GatedImport:
|
|||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: object = None,
|
session: object = None,
|
||||||
progress: Callable[[str, str, int, int], None] | None = None,
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
|
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
self.prune_flags.append(prune)
|
self.prune_flags.append(prune)
|
||||||
if progress is not None:
|
if progress is not None:
|
||||||
|
|||||||
Reference in New Issue
Block a user