chore(agent): phase 105 (hidden-folders toggle) from TODO.md, clear the list
This commit is contained in:
@@ -0,0 +1,78 @@
|
|||||||
|
# Phase 105 — Per-source hidden-folders toggle: dot-prefixed paths become indexable per input
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
|
||||||
|
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-14).
|
||||||
|
**Context:** The single filesystem walk choke point is `iter_importable_files` (`app/rag/importer.py` L161-194): it skips any path with a dot-prefixed component — hidden dirs (vendored caches like `.esphome/.espressif/**`) AND hidden files — plus the well-known `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`); the extension filter (`BOR_IMPORT_EXTENSIONS`, the A9 family) then admits the rest. `import_sources` (L196+) walks every root twice when `progress` is set (the phase-64 pre-walk for the `total` denominator uses the EXACT same rules), collects `(source, rel)` into `seen`, and `_prune` (`prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — the same mechanism phase 89 uses so newly-ignored files leave the index. Both live import entry points build a per-root `ignore_by_root: dict[str, list[str]]` map from the `git_sources` rows in one loop — the Sync button (`app/api/sync.py::_run_sync` L233-263) and the CLI (`scripts/import_docs.py::_resolve_sources` L181-226, consumed at L269/L332-334); the map is keyed by `str(root)` — the root path string exactly as passed to `import_sources` — with extend-on-collision for shared roots. `scripts/load_test_kb.py` calls `import_sources` with defaults (untouched). The archive-upload background run no longer scans (phase 90), so no map is needed there. Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources` L514+); the per-row "Ignore paths" button (`makeRow` L357+, `ignoreBtn` ~L393-408) sits in the actions cell LEFT of Remove, stored rows only (env-fallback rows, `id` null, get no control — phase 89 A3); non-empty lists render a `N ignored` text tag in the source cell (`.git-source-ignore-count`, `frontend/assets/styles.css` L2686, text + background, never color alone). The admin-only `PATCH /api/git-sources/{source_id}` (phase 89 A5, `app/api/git_sources.py` L374-393) today takes the REQUIRED `ignore_paths` replace list; the read shapes are `GitSourceRow` (`app/schemas.py` L508) and `GitSourceOut` (L490), the create body `GitSourceIn` (L450, optional `ignore_paths`). Alembic head is `0018`. The KB tree/catalog (phase 97) and the agent tools read the DB, so newly indexed hidden documents appear in them automatically — no change needed.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Each stored source carries an **index-hidden-folders flag**, toggled by a per-row checkbox next to the "Ignore paths" button on the Sources page. When ON for a source, the walk no longer skips dot-prefixed components for that source — files inside hidden folders (and hidden files with an importable extension) are indexed, embedded, and summarized exactly like visible files; when OFF (the default for every existing row), behavior is byte-identical to today. `EXCLUDED_DIRS` stays excluded in both states, the extension filter always applies, and the flag — like the ignore list — takes effect on the next sync, with previously indexed hidden files pruned when it is switched off. All entry points (Sync button, CLI) honor the flag; the API stays admin-only.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `104_chip_sizing_question_cap` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/CLI pipelines, and the Sources view — none of which phase 104's pins reach; its suites must stay green unchanged).
|
||||||
|
- `89_source_ignore_paths` (complete) — the per-root map, the actions-cell control idiom, the count-tag idiom, and the `PATCH` route this phase extends.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **Flag semantics (locked, A1).** `include_hidden=True` lifts ONLY the dot-prefixed-component skip in `iter_importable_files`: the existing check `any(part.startswith(".") or part in excluded for part in rel.parts)` becomes dot-aware only when the flag is False — e.g. `any((not include_hidden and part.startswith(".")) or part in excluded for part in rel.parts)`. Consequences, all deliberate:
|
||||||
|
- Files INSIDE hidden dirs become importable (`.esphome/esp.md` indexed when ON).
|
||||||
|
- Hidden files with an importable extension also become importable (`.notes.md` — the dot check covers components, not "the folder of the file", so one rule covers both; the extension filter is the real content gate, and a secret-flavoured file like `.env` has no A9 extension and is never indexed).
|
||||||
|
- `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) are skipped in BOTH states — caches/VCS internals are never content.
|
||||||
|
- The `ignore` tuple (phase 89) composes additively with the flag: an ignored prefix still skips a file when `include_hidden=True`.
|
||||||
|
- **Storage (task 01).** `git_sources.include_hidden` — BOOLEAN NOT NULL, server default `false`, `Mapped[bool]` (the `documents.is_summary` Boolean precedent, `app/models.py` L136). Alembic `0019_git_source_include_hidden.py` (revises `0018`): `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`; downgrade drops the column. Existing rows read `False` (A4).
|
||||||
|
- **Importer signature (task 02).**
|
||||||
|
- `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore=(), include_hidden: bool = False)` — default `False` keeps every existing caller byte-identical; the docstring's skip sentence gains the flag clause.
|
||||||
|
- `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root=None, include_hidden_by_root: dict[str, bool] | None = None)` — the map is keyed by **`str(root)`** with the SAME keying convention as `ignore_by_root`; an internal `_include_hidden_for_root(root, include_hidden_by_root) -> bool` (default `False`) is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop, so `files_total` never disagrees with the walk. `seen` is untouched in shape → `_prune` prunes hidden documents automatically when the flag flips OFF (A2 — the A9/phase-89 precedent). Module docstring "Scope" paragraph updated.
|
||||||
|
- **API contract (task 03).**
|
||||||
|
- Schemas (`app/schemas.py`): `GitSourceIn.include_hidden: bool | None = Field(default=None)` (create-time, optional — absent → stored `False`); `GitSourceOut.include_hidden: bool`; `GitSourceRow.include_hidden: bool` (env-fallback rows report `False` — no DB row to store a flag on).
|
||||||
|
- The PATCH body model is RENAMED `GitSourceIgnoreIn` → `GitSourcePatchIn` (grep-verified: referenced only in `app/schemas.py` and `app/api/git_sources.py` — import L119 + `patch_git_source` L376) and gains:
|
||||||
|
- `ignore_paths: list[str] | None = Field(default=None)` — **absent/None = the row's list is unchanged; PRESENT = replace semantics exactly as phase 89 A5** (normalization + the A4 fixed-detail 422s run only when present). Every existing client always sends the list, so their behavior is byte-identical; the toggle's PATCH sends only the bool.
|
||||||
|
- `include_hidden: bool | None = Field(default=None)` — absent/None = unchanged; present = set.
|
||||||
|
- Both absent → 200 no-op (row untouched).
|
||||||
|
- `GET /api/git-sources` — DB rows report the stored flag; env rows `False`. `POST /api/git-sources` — both kinds accept `include_hidden`; stored `bool(payload.include_hidden)`.
|
||||||
|
- `PATCH /api/git-sources/{source_id}` (existing route, still behind `require_admin`) — applies each PRESENT field independently (404 unknown id unchanged); 200 → `GitSourceOut` (id, url, added_at, ignore_paths, include_hidden).
|
||||||
|
- **Callers (task 04).**
|
||||||
|
- `app/api/sync.py::_run_sync` — in the existing per-row loop that builds `ignore_by_root` (L233-252), build `include_hidden_by_root: dict[str, bool]` with the SAME `str(root)` keying: `include_hidden_by_root[str(root)] = include_hidden_by_root.get(str(root), False) or bool(row.include_hidden)` (collision → OR — the mirror of the ignore-map union: if either row says "index hidden", the shared root does). Pass `include_hidden_by_root=…` to `import_sources` (L263). Module docstring (L36-40) updated.
|
||||||
|
- `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple `(sources, ignore_by_root, include_hidden_by_root)` (manual `--source` → `(sources, {}, {})` — manual dirs have no row; env-fallback rows have no flags); the OR-collision rule is the same; `main` unpacks (L269) and passes the map (L332-334); docstrings updated (module + `_resolve_sources` L181).
|
||||||
|
- `scripts/load_test_kb.py` — untouched (defaults).
|
||||||
|
- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: a **native labeled checkbox** in the actions cell, DOM order **Hidden · Ignore paths · Remove** (the toggle sits next to — left of — the "Ignore paths" button, per the owner's wording; Remove stays last):
|
||||||
|
- `<label class="git-source-hidden">` wrapping `<input type="checkbox" class="git-source-hidden-box">` + visible text "Hidden"; the checkbox's `aria-label` is `Index hidden folders for ${kindLabel} source: ${value}` (setAttribute — never innerHTML; `value` is the git URL or local path, credential-safety discipline), `checked = s.include_hidden === true`; a `title` on the label explains in plain words ("When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.").
|
||||||
|
- When `s.include_hidden === true`, the source cell also shows a **"hidden on" text tag** (`.git-source-hidden-count` — the `.git-source-ignore-count` idiom: text + background, never color alone, WCAG 1.4.1), next to the `N ignored` tag.
|
||||||
|
- **§7.4 never-stale lifecycle** — `toggleHidden(s, box)`: on `change`, the box disables immediately (no double-flip); `PATCH /api/git-sources/${s.id}` with `{"include_hidden": box.checked}`; on 200 → clear the error line, `await loadSources()` (the row re-renders from the server), THEN `announce("Hidden folders enabled|disabled for <value>.")` (the phase-89 last-announce order — the confirmation lands after the reload's "N sources listed."); on non-2xx or network failure → the server detail (or the canned "Could not reach the server — the setting was not changed.") into the new page-level `role="alert"` line, and the box **reverts to the server state** (`box.checked = s.include_hidden === true`) and re-enables — the UI never claims a state the server didn't save.
|
||||||
|
- `frontend/index.html` — one new element after the `#git-sources-table-wrap` region: `<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>` (reuses the existing `.git-source-error` styling). `frontend/assets/styles.css` — near the phase-89 block (~L2657-2695): `.git-source-hidden` (inline-flex, ~44px hit height matching the action buttons, visible label), `.git-source-hidden input[type="checkbox"]` (sized, `accent-color` on the brand pair — verify + record the AA ratio in the comment, house style), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), focus ring via the GLOBAL `:focus-visible` rule (L146 — no per-control rule needed), and `.git-source-hidden-count` (copy of the `.git-source-ignore-count` rule, provenance comment citing phase 105).
|
||||||
|
- Env-fallback rows (`id` null) get **no** checkbox — the existing "from .env" tag stays (A3).
|
||||||
|
- **NOT touched:** the RAG view (`sources.js`), `app/rag/retriever.py`, the chunker, the KB tree/catalog + agent tools (DB-driven — they pick up newly indexed hidden docs for free), `app/rag/git_sources.py` (clone/pull only, no walk), the upload run (phase 90 — no scan), `AGENTS.md`, `.agents/PLAN.md`, any completed phase.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_include_hidden_column.md` — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`) + default/round-trip tests.
|
||||||
|
2. `02_importer_include_hidden.md` — `iter_importable_files`/`import_sources` flag support (walk + progress pre-walk + prune interaction + ignore composition) + unit & integration tests.
|
||||||
|
3. `03_include_hidden_api.md` — schemas (`In`/`Out`/`Row` + the `GitSourcePatchIn` rename with optional fields) + GET/POST/PATCH wiring + integration tests.
|
||||||
|
4. `04_include_hidden_pipelines.md` — wire the per-row flag into `_run_sync` and `scripts/import_docs.py` + integration tests.
|
||||||
|
5. `05_hidden_toggle_sources_ui.md` — the per-row "Hidden" checkbox on the Sources page (tag + §7.4 lifecycle + error line + a11y) + source-level unit pins.
|
||||||
|
6. `06_e2e_hidden_folders_toggle.md` — dedicated Playwright suite `tests/e2e/test_hidden_folders_toggle.py` (run in isolation), regressions, full gate, atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_importer_include_hidden.py` (new, task 02): `iter_importable_files` on a tmp fixture tree — default OFF pins today's behavior byte-identically (hidden dir + hidden file skipped), ON admits both, `EXCLUDED_DIRS` skipped in BOTH states, `ignore` tuple still bites when ON, extension filter unchanged (`.env` never indexed); the `str(root)` keying + default-False for unlisted roots; `tests/unit/test_hidden_folders_toggle.py` (new, task 05): source-level pins for the JS/HTML/CSS wiring (house pattern — read the assets as text), incl. the single-source cross-file check that the aria-label template names the source value; task 01's pins extend the existing model/migration test surfaces (a fresh row reads `include_hidden is False`; an explicit `True` round-trips).
|
||||||
|
- Integration — `tests/integration/test_importer_include_hidden.py` (new, task 02): `import_sources` against a fixture dir — hidden file produces NO `Document`/`Chunk` rows by default; WITH the map it is embedded + summarized normally; previously indexed hidden file + flag OFF → pruned on the next run; progress `total` agrees with the walk in both states; sources not in the map behave exactly as before. `tests/integration/test_git_sources_api.py` (extended, task 03): GET reports `False` default / stored `True`; POST create round-trip; PATCH bool-only, list-only, both, neither (no-op 200), the phase-89 422s unchanged for present lists, 404, anonymous 403. `tests/integration/test_sync_api.py` (extended, task 04): a local row with a hidden dir — flag False syncs zero hidden docs, True syncs them; `tests/integration/test_import_docs_git.py` (extended, task 04): the CLI DB-row path with the flag set.
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_hidden_folders_toggle.py` (task 06), run in isolation with the DB up: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov`.
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate — the importer/API/pipeline additions are fully unit+integration covered).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A stored source (git, local, or uploaded) has a "Hidden" checkbox next to its "Ignore paths" button: flipping it on persists (`PATCH` 200, the source cell shows the "hidden on" tag, `GET /api/git-sources` round-trips `include_hidden: true`); the failure path reverts the box and announces the error in a `role="alert"` line.
|
||||||
|
- [ ] A sync (button or CLI) with the flag OFF indexes nothing with a dot-prefixed component (no `documents`/`chunks` rows — the byte-identical default); with the flag ON, `.hidden/note.md` is indexed, embedded, and summarized like any visible file and shows up in the KB catalog; `EXCLUDED_DIRS` content is excluded in both states.
|
||||||
|
- [ ] A2: a previously indexed hidden file is PRUNED from the KB on the next sync after the flag flips OFF (`detail.pruned` increments; the catalog no longer lists it).
|
||||||
|
- [ ] API contracts hold: PATCH bool-only / list-only / both / neither; the phase-89 fixed-detail 422s unchanged for present lists; 404 unknown id; anonymous 403 on the route.
|
||||||
|
- [ ] Env-fallback rows render no checkbox (the "from .env" tag stays); the Sources page stays WCAG-clean (visible label, focus-visible, text never color alone).
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation (DB up); regression suites `test_source_ignore_paths.py`, `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — what "on" means (owner-confirmed 2026-09-14).** The flag lifts the dot-prefixed-component skip for that source — files inside hidden folders AND hidden files with an importable extension become indexable; `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) stay excluded regardless; the extension filter always applies.
|
||||||
|
- **A2 — toggling off prunes (owner-confirmed).** Previously indexed hidden files leave the KB on the next sync (the `seen`-set prune — the phase-89 A2 / A9 precedent), exactly like newly-ignored files.
|
||||||
|
- **A3 — per stored row only (owner-confirmed).** Every stored `git_sources` row (git, local, uploaded) gets the toggle; env-fallback rows have no DB row and get no control (the phase-89 A3 precedent).
|
||||||
|
- **A4 — default off (owner-confirmed).** `include_hidden` defaults to `false` for all existing rows — byte-identical behavior until the owner flips it; the flag takes effect on the NEXT sync (no auto-sync, same as the ignore list).
|
||||||
|
- **A5 — control idiom (owner-confirmed).** A native labeled checkbox ("Hidden") in the actions cell, left of the "Ignore paths" button (Remove stays last), with a "hidden on" text tag in the source cell when enabled — not a styled switch.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
|
||||||
|
```
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Task 01 — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`)
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "…There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Persist the per-source hidden-folders flag: one additive, reversible BOOLEAN column on `git_sources`, server-defaulted to `false` so every pre-phase-105 row imports byte-identically (A4).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — the `GitSource` class (L220-259): add the column directly AFTER `ignore_paths` (L248-251), mirroring its docstring/provenance style (`Boolean` is already imported, L78):
|
||||||
|
```python
|
||||||
|
#: Index hidden (dot-prefixed) paths from this source (phase 105,
|
||||||
|
#: A1): True → the walk (app.rag.importer.iter_importable_files)
|
||||||
|
#: does not skip dot-prefixed components — files inside hidden
|
||||||
|
#: folders AND hidden files with an importable extension are
|
||||||
|
#: indexed; ``EXCLUDED_DIRS`` (``.venv``, ``node_modules``,
|
||||||
|
#: ``.git``, …) are excluded in BOTH states, and the extension
|
||||||
|
#: filter always applies. Takes effect on the next sync (no
|
||||||
|
#: auto-sync — the ignore-list precedent, phase 89). Server
|
||||||
|
#: default False: every pre-phase-105 row imports exactly as
|
||||||
|
#: before (A4).
|
||||||
|
include_hidden: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=False, server_default=text("false"), nullable=False
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(Extend the class docstring's one-line field inventory if it names `ignore_paths` — the module header L19 does: add `include_hidden` (phase 105) to the parenthetical.)
|
||||||
|
2. `alembic/versions/0019_git_source_include_hidden.py` (NEW — the house format of `0013_git_source_ignore_paths.py`, one additive reversible column):
|
||||||
|
- `revision = "0019"`, `down_revision = "0018"`.
|
||||||
|
- `upgrade()`: `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`.
|
||||||
|
- `downgrade()`: `op.drop_column("git_sources", "include_hidden")`.
|
||||||
|
- Module docstring: the phase-89-0013 provenance style (what the flag is, A1/A4, one additive reversible column).
|
||||||
|
3. Tests — extend the existing model/migration test surfaces (find the current `GitSource` default-pinning tests — the phase-89 column tests live in the `git_sources` unit/integration suites; add alongside them):
|
||||||
|
- A freshly inserted `GitSource` row (no `include_hidden` passed) reads `include_hidden is False` (the Python `default=False` AND the server default agree).
|
||||||
|
- An explicit `include_hidden=True` round-trips through the DB (`session.add` → `commit` → fresh session → `True`).
|
||||||
|
- The migration applies on the test DB from head (`alembic upgrade head` is part of the standard test-db fixture setup — if the suite asserts the column set of `git_sources`, add `include_hidden` to the expected set).
|
||||||
|
4. Run `uv run pytest tests/unit/ -q && uv run alembic upgrade head` (against the dev/test DB per the house quick reference) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the default + round-trip pins above ARE this task's layer (no importer behavior yet — that is task 02).
|
||||||
|
- Coverage: **>90%** on `app/` (model-only change; the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GitSource.include_hidden` exists with `server_default=text("false")`, `nullable=False`, and the A1/A4 provenance comment
|
||||||
|
- [ ] `alembic/versions/0019_git_source_include_hidden.py` upgrades from `0018` and downgrades cleanly; the dev/test DB is at head
|
||||||
|
- [ ] Fresh-row-default-False and explicit-True round-trip tests pass; existing `git_sources` suites stay green
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Task 02 — Importer flag support: `iter_importable_files` / `import_sources` honor `include_hidden` per root
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. …to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make the walk choke point flag-aware: `iter_importable_files` gains `include_hidden` (default `False` — every existing caller byte-identical) and `import_sources` gains `include_hidden_by_root` (same `str(root)` keying as phase 89's `ignore_by_root`), used by BOTH the phase-64 progress pre-walk and the processing loop. Pruning falls out for free through the untouched `seen` set (A2).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — `iter_importable_files` (L161-194):
|
||||||
|
- Signature: `def iter_importable_files(root: Path, extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS, ignore: tuple[str, ...] = (), include_hidden: bool = False) -> list[Path]:`
|
||||||
|
- The skip check (L184) becomes flag-aware — ONE expression, byte-identical when the flag is False:
|
||||||
|
```python
|
||||||
|
if any(
|
||||||
|
(not include_hidden and part.startswith(".")) or part in excluded
|
||||||
|
for part in rel.parts
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
```
|
||||||
|
- Docstring: the "Skips:" sentence gains — "…when ``include_hidden`` is False (the default): any path with a dot-prefixed component; when True, dot-prefixed components are ADMITTED (files inside hidden folders, and hidden files) and only *excluded* is consulted (A1 — caches/VCS internals are never content). The *ignore* tuple composes additively in both states."
|
||||||
|
2. `app/rag/importer.py` — `import_sources` (L196+):
|
||||||
|
- Keyword-only param after `ignore_by_root`: `include_hidden_by_root: dict[str, bool] | None = None`.
|
||||||
|
- New private helper next to `_ignore_for_root` (L145-160), same style:
|
||||||
|
```python
|
||||||
|
def _include_hidden_for_root(
|
||||||
|
root: Path, include_hidden_by_root: dict[str, bool] | None
|
||||||
|
) -> bool:
|
||||||
|
"""The per-root hidden-folders flag (phase 105, A1).
|
||||||
|
|
||||||
|
Keyed by ``str(root)`` — the root string exactly as the caller
|
||||||
|
passed it in ``sources`` (the ``_ignore_for_root`` convention,
|
||||||
|
phase 89): ``True`` only for roots the caller lists as True;
|
||||||
|
unlisted/``None`` roots are ``False`` — every existing caller
|
||||||
|
behaves byte-identically (A4).
|
||||||
|
"""
|
||||||
|
return bool((include_hidden_by_root or {}).get(str(root), False))
|
||||||
|
```
|
||||||
|
- The progress pre-walk (the `if progress is not None:` block, ~L244-252) passes `include_hidden=_include_hidden_for_root(root, include_hidden_by_root)` to its `iter_importable_files` call — `files_total` must agree with the walk in both states.
|
||||||
|
- The processing loop: alongside `ignore = _ignore_for_root(root, ignore_by_root)` (~L272), add `include_hidden = _include_hidden_for_root(root, include_hidden_by_root)` and pass both into the `iter_importable_files` call (~L273-275).
|
||||||
|
- `import_sources` docstring: after the `ignore_by_root` paragraph, the mirror paragraph: "``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the stored flag: ``True`` admits dot-prefixed components for that root (``EXCLUDED_DIRS`` and the extension filter still apply; the ignore tuple composes additively). Unlisted/``None`` roots are ``False`` — byte-identical to pre-phase-105. A file that was indexed with the flag ON and is walked again with it OFF simply never enters ``seen``, so the next ``prune=True`` run deletes its row automatically (A2 — the A9/phase-89 precedent)."
|
||||||
|
- Module docstring "Scope" paragraph (L20-22): append the flag clause ("…skipped, plus the well-known exclusion list — UNLESS the source's phase-105 hidden-folders flag admits dot-prefixed paths; the exclusion list always applies").
|
||||||
|
3. `tests/unit/test_importer_include_hidden.py` (NEW) — `iter_importable_files` on a `tmp_path` tree (extensions `frozenset({".md"})`), fixture layout:
|
||||||
|
```
|
||||||
|
visible.md .hidden/note.md .notes.md
|
||||||
|
.venv/junk.md node_modules/x.md .hidden/.deep.md
|
||||||
|
keep/ok.md
|
||||||
|
```
|
||||||
|
- **Default (flag False) — today's behavior pinned byte-identically:** result == `[keep/ok.md, visible.md]` (sorted); hidden dir, hidden file, `.venv`, `node_modules` all absent.
|
||||||
|
- **Flag True (A1):** result contains `.hidden/note.md`, `.notes.md`, `.hidden/.deep.md`, `visible.md`, `keep/ok.md`; STILL excludes `.venv/junk.md` and `node_modules/x.md` (EXCLUDED_DIRS in both states).
|
||||||
|
- **Composition with `ignore`:** flag True + `ignore=(".hidden",)` → `.hidden/*` gone, `.notes.md` present (additive).
|
||||||
|
- **Extension filter unchanged:** flag True with `.env`-like file `.env` (no A9 extension) → never listed.
|
||||||
|
- `_include_hidden_for_root` unit pins: `None` map → False; unlisted root → False; listed `True` → True; listed `False` → False; keying is `str(root)` (two `Path` objects, equal strings, same answer).
|
||||||
|
4. `tests/integration/test_importer_include_hidden.py` (NEW) — `import_sources` against a fixture dir (the `tests/integration/test_importer_ignore.py` harness — fake `Embedder` from `tests/fakes.py` or the module's own fake, real test DB):
|
||||||
|
- Layout: `visible.md`, `.hidden/note.md` (non-markdown `.hidden/data.yaml` too, to prove the summary path runs for admitted hidden files), `.venv/junk.md`.
|
||||||
|
- **Run 1 (default, no map):** `Document` rows exist for `visible.md` ONLY — no row for `.hidden/note.md` (A4 byte-identical); `summary` stats unchanged.
|
||||||
|
- **Run 2 (`include_hidden_by_root={str(root): True}`):** `.hidden/note.md` + `.hidden/data.yaml` get `Document` + `Chunk` rows (embedded via the fake), the yaml gets a `summary` row (the phase-30 path), `.venv/junk.md` still absent (A1).
|
||||||
|
- **A2 prune:** with the rows from run 2 committed, run 3 with `prune=True` and NO map (flag off) → `.hidden/*` rows deleted (`summary.pruned >= 2`), `visible.md` untouched.
|
||||||
|
- **Progress agreement:** with `progress=…` set, `total` equals the visible-only count in run 1's rules and the extended count in run 2's rules (the pre-walk uses the same flag).
|
||||||
|
- **Unlisted roots untouched:** two roots, map lists only one as True — the other root's hidden file stays out (per-root, not global).
|
||||||
|
5. Run `uv run pytest tests/unit/test_importer_include_hidden.py tests/integration/test_importer_include_hidden.py tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -q` — green (the phase-89 suites are the byte-identical regression proof).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit + integration as above are this task's layer; API/pipeline/UI come later.
|
||||||
|
- Coverage: **>90%** on `app/` (the new helper + both call sites are fully covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `iter_importable_files(..., include_hidden=False)` is the default and its default-state result is byte-identical to pre-task (phase-89 importer suites green)
|
||||||
|
- [ ] `include_hidden=True` admits dot-prefixed components, keeps `EXCLUDED_DIRS` + extension filter + `ignore` tuple in force (A1)
|
||||||
|
- [ ] `import_sources(include_hidden_by_root=…)` drives BOTH the pre-walk and the loop through `_include_hidden_for_root`; unlisted/`None` → False
|
||||||
|
- [ ] Flag-off re-run prunes previously indexed hidden docs (A2); progress `total` agrees with the walk in both states
|
||||||
|
- [ ] `uv run pytest tests/unit/ -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# Task 03 — API: `include_hidden` on GET/POST/PATCH (the `GitSourcePatchIn` rename)
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "There should be a toggle per input (next to the ignores button)…"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Expose the flag through the admin API: `GET`/`POST` carry it, and the existing `PATCH /api/git-sources/{id}` becomes the single per-row edit endpoint for BOTH the ignore list and the toggle — each field optional, present-field-wins, so the toggle's PATCH sends only `{"include_hidden": …}` and the phase-89 dialog's PATCH (which always sends the list) keeps byte-identical semantics.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py`:
|
||||||
|
- `GitSourceIn` (L450-477): add after `ignore_paths`:
|
||||||
|
```python
|
||||||
|
include_hidden: bool | None = Field(default=None)
|
||||||
|
```
|
||||||
|
+ docstring clause: "``include_hidden`` (phase 105) is optional at create time (absent → stored ``False`` — A4)."
|
||||||
|
- `GitSourceOut` (L490-506): add `include_hidden: bool` (after `ignore_paths`) + docstring clause (the stored flag; `False` for a row created without it).
|
||||||
|
- `GitSourceRow` (L508-528): add `include_hidden: bool` + docstring clause (**env-fallback rows report `False`** — no DB row to store a flag on, the `ignore_paths: []` precedent).
|
||||||
|
- **RENAME** `GitSourceIgnoreIn` (L530-543) → `GitSourcePatchIn` and rework its body (grep-verified references: only `app/schemas.py` + `app/api/git_sources.py` L119/L376 — no tests import it):
|
||||||
|
```python
|
||||||
|
class GitSourcePatchIn(BaseModel):
|
||||||
|
"""``PATCH /api/git-sources/{source_id}`` body (phase 89 A5;
|
||||||
|
extended phase 105).
|
||||||
|
|
||||||
|
Each field is independent and OPTIONAL: absent/None leaves the
|
||||||
|
row's value unchanged; PRESENT applies. ``ignore_paths`` when
|
||||||
|
present keeps the phase-89 A5 REPLACE semantics (the body list,
|
||||||
|
normalized + A4-validated, becomes the row's whole list — empty
|
||||||
|
list clears all; every pre-phase-105 client always sends the
|
||||||
|
list, so their behavior is byte-identical). ``include_hidden``
|
||||||
|
(phase 105) when present sets the stored flag. Both absent →
|
||||||
|
200 no-op (the row is untouched).
|
||||||
|
"""
|
||||||
|
|
||||||
|
ignore_paths: list[str] | None = Field(default=None)
|
||||||
|
include_hidden: bool | None = Field(default=None)
|
||||||
|
```
|
||||||
|
2. `app/api/git_sources.py`:
|
||||||
|
- Import L119: `GitSourceIgnoreIn` → `GitSourcePatchIn`.
|
||||||
|
- `list_git_sources` (L202-245): DB row construction (~L226-233) gains `include_hidden=row.include_hidden`; the env-fallback row (~L240) gains `include_hidden=False`; the endpoint docstring's field list mentions it.
|
||||||
|
- `create_git_source` (L249-280): response construction (L278) gains `include_hidden=row.include_hidden`.
|
||||||
|
- `_create_git_row` (L315, row construction ~L326-335) and `_create_local_row` (L340, ~L358-368): both gain `include_hidden=bool(payload.include_hidden)` (absent → `False`, A4).
|
||||||
|
- `patch_git_source` (L374-393) — new body:
|
||||||
|
```python
|
||||||
|
def patch_git_source(
|
||||||
|
source_id: uuid.UUID,
|
||||||
|
payload: GitSourcePatchIn,
|
||||||
|
db: Session = Depends(get_db), # noqa: B008
|
||||||
|
) -> GitSourceOut:
|
||||||
|
"""Edit one source's ignore list and/or hidden-folders flag.
|
||||||
|
|
||||||
|
Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
|
||||||
|
id; each PRESENT body field applies independently —
|
||||||
|
``ignore_paths`` REPLACES the list (normalized + A4-validated,
|
||||||
|
fixed 422 details); ``include_hidden`` sets the flag. Both
|
||||||
|
absent → 200 no-op. Returns the updated row's public shape
|
||||||
|
(id, url, added_at, ignore_paths, include_hidden).
|
||||||
|
"""
|
||||||
|
row = db.get(GitSource, source_id)
|
||||||
|
if row is None:
|
||||||
|
raise HTTPException(status_code=404, detail="git source not found")
|
||||||
|
if payload.ignore_paths is not None:
|
||||||
|
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
|
||||||
|
if payload.include_hidden is not None:
|
||||||
|
row.include_hidden = payload.include_hidden
|
||||||
|
db.commit()
|
||||||
|
db.refresh(row)
|
||||||
|
return GitSourceOut(
|
||||||
|
id=row.id,
|
||||||
|
url=row.url,
|
||||||
|
added_at=row.added_at,
|
||||||
|
ignore_paths=row.ignore_paths,
|
||||||
|
include_hidden=row.include_hidden,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
- Router module docstring (L1-30ish — the contract list): the PATCH line extends to "the ignore list (replace) and/or the hidden-folders flag (phase 105) — each optional, present-wins".
|
||||||
|
3. `tests/integration/test_git_sources_api.py` — extend the phase-89 PATCH section (the suite already pins 404 + anonymous 403 + the A4 fixed-detail 422s):
|
||||||
|
- `GET` — a fresh stored row reports `include_hidden: false`; an env-fallback row (table empty + `BOR_GIT_SOURCES` monkeypatched, the suite's existing pattern) reports `include_hidden: false`.
|
||||||
|
- `POST` — `kind="local"` create with `include_hidden: true` → 201 body `include_hidden: true`; without the field → `false` (A4).
|
||||||
|
- `PATCH` matrix (one stored row, list `["a/b"]`, flag `false`):
|
||||||
|
- `{"include_hidden": true}` → 200, flag `true`, list UNCHANGED `["a/b"]` (the toggle's exact payload).
|
||||||
|
- `{"ignore_paths": ["c/d"]}` → 200, list REPLACED, flag UNCHANGED `false` (the dialog's exact payload — byte-identical to phase 89).
|
||||||
|
- `{"ignore_paths": [], "include_hidden": true}` → both applied.
|
||||||
|
- `{}` → 200 no-op (list + flag unchanged).
|
||||||
|
- the phase-89 422s still fire for a PRESENT bad list (>200 entries / empty-after-normalization / >500-char entry — the fixed details), and a bad list does NOT half-apply the flag (assert the flag is untouched after a 422 with both fields present).
|
||||||
|
- 404 unknown id; anonymous `PATCH` 403 (the existing pin already covers the route — extend it to include a bool-only body so the pin proves the toggle path is gated too).
|
||||||
|
4. Run `uv run pytest tests/integration/test_git_sources_api.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration as above is this task's layer (the unit schema pins ride on the integration 422/shape checks, house pattern for this router).
|
||||||
|
- Coverage: **>90%** on `app/` (every new/changed branch — both field-present/absent combos — is exercised — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET` reports the flag for DB rows and `false` for env rows; `POST` create round-trips it (absent → `false`)
|
||||||
|
- [ ] `GitSourcePatchIn` exists with two optional fields; `GitSourceIgnoreIn` is gone repo-wide (grep-verified)
|
||||||
|
- [ ] PATCH: bool-only, list-only, both, neither (no-op 200); the phase-89 A4 fixed-detail 422s unchanged for present lists; a 422 never half-applies the other field; 404 + anonymous 403 hold
|
||||||
|
- [ ] `uv run pytest tests/integration/test_git_sources_api.py -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# Task 04 — Pipelines: the Sync button + the CLI honor the per-row flag
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. …to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Build the `include_hidden_by_root` map at the two live import entry points — the in-app Sync (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py::_resolve_sources`) — with the SAME `str(root)` keying and collision rule as the phase-89 `ignore_by_root`, and pass it to `import_sources`. The upload run needs nothing (phase 90 — no scan) and `scripts/load_test_kb.py` keeps its defaults.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/sync.py` — `_run_sync` (the per-row loop L233-252 + the `import_sources` call L262-264):
|
||||||
|
- After `ignore_by_root: dict[str, list[str]] = {}` (L233): `include_hidden_by_root: dict[str, bool] = {}`
|
||||||
|
- Inside the loop, next to the phase-89 `ignore_by_root.setdefault(...)` (L251-252) — the comment cites phase 105 + the same sibling/repo-name collision note:
|
||||||
|
```python
|
||||||
|
# Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
|
||||||
|
# the SAME root string the importer sees; a shared-root
|
||||||
|
# collision ORs — if EITHER row says "index hidden", the
|
||||||
|
# root does (the ignore-map union's boolean mirror).
|
||||||
|
include_hidden_by_root[str(root)] = (
|
||||||
|
include_hidden_by_root.get(str(root), False)
|
||||||
|
or bool(row.include_hidden)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
- The call (L262-264): add the kwarg — `summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root, include_hidden_by_root=include_hidden_by_root)`
|
||||||
|
- Module docstring (L30-45, the pipeline list that names "honoring each row's ``ignore_paths`` (phase 89…)"): extend the clause with "and its ``include_hidden`` flag (phase 105 — the per-root hidden-folders map, same per-row construction)".
|
||||||
|
2. `scripts/import_docs.py`:
|
||||||
|
- `_resolve_sources` docstring (L181-185): "Returns ``(sources, ignore_by_root)``" → "Returns ``(sources, ignore_by_root, include_hidden_by_root)`` (phase 89; phase 105 adds the per-root flag map — the flag is stored per row, manual ``--source`` dirs and the legacy fallback have no rows and import with the empty map: hidden paths skipped, A4)."
|
||||||
|
- All THREE return paths become 3-tuples:
|
||||||
|
- the `cli_sources` early return (~L190): `return [path.expanduser() for path in cli_sources], {}, {}`
|
||||||
|
- the rows branch: after the `ignore_by_root` dict init (L206), add `include_hidden_by_root: dict[str, bool] = {}`; inside the loop next to the phase-89 extend (L224-225), the same OR assignment as sync.py (comment: phase 105); the return (L226): `return sources, ignore_by_root, include_hidden_by_root`
|
||||||
|
- the legacy fallback return (~L227): `return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}`
|
||||||
|
- `main` (L269): `sources, ignore_by_root, include_hidden_by_root = _resolve_sources(args.source, settings)` (the comment above it, L258, extends to name the flag map).
|
||||||
|
- The `import_sources` call (L332-335): add `include_hidden_by_root=include_hidden_by_root,`.
|
||||||
|
- Module docstring: the phase-89 line about the per-root ignore map gains the flag clause.
|
||||||
|
3. Tests:
|
||||||
|
- `tests/integration/test_sync_api.py` — extend with the flag (the suite's local-row + fixture-dir pattern; the fixture dir gains `.hidden/note.md` + a visible file):
|
||||||
|
- Row with `include_hidden=False` (default) + `POST /api/sync` → terminal status `success`, the hidden file has NO `documents` row (A4).
|
||||||
|
- Same row flipped to `True` (task-03 PATCH or direct model set, the suite's idiom) + sync → the hidden file IS indexed (`documents` row present; `detail.files` counts it).
|
||||||
|
- `tests/integration/test_import_docs_git.py` (the CLI DB-row suite) — one test: a local row with a hidden file + `include_hidden=True` → running the CLI main path (the suite's existing invocation harness) indexes the hidden file; `False` → not (the CLI's map is built, not lost — the regression this phase most plausibly breaks is here).
|
||||||
|
4. Run `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration as above is this task's layer; the importer itself is task 02's, the UI task 05's.
|
||||||
|
- Coverage: **>90%** on `app/` (the sync map-building branch is exercised; `scripts/` is outside the `--cov=app` gate — the CLI test is the behavioral pin, house pattern).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `_run_sync` builds `include_hidden_by_root` in the existing per-row loop (OR on collision) and passes it to `import_sources`
|
||||||
|
- [ ] `_resolve_sources` returns the 3-tuple on ALL three return paths (manual, rows, legacy fallback) and `main` passes the map through
|
||||||
|
- [ ] Sync with the flag off indexes no hidden docs; with it on, does — E2E-provable through `detail` + `documents` rows; the CLI behaves the same
|
||||||
|
- [ ] `uv run pytest tests/integration/ -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
# Task 05 — The per-row "Hidden" toggle on the Sources page (checkbox + tag + §7.4 lifecycle + a11y)
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The owner-visible half of the feature: every stored row on the Sources page gets a labeled **Hidden** checkbox in the actions cell, left of its "Ignore paths" button (DOM order Hidden · Ignore paths · Remove — A5); when on, the source cell shows a "hidden on" text tag; the flip PATCHes the flag with the §7.4 never-stale lifecycle and reverts to the server state on failure.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — inside `#view-git-sources`, directly AFTER the `#git-sources-table-wrap` region closes (~L632), the page-level error line for the toggle (the ignore dialog carries its own error INSIDE the modal; the checkbox lives in the table, so its error lives at page level — reuses the existing `.git-source-error` styling):
|
||||||
|
```html
|
||||||
|
<!-- Phase 105: the per-row "Hidden" toggle's error line — the
|
||||||
|
checkbox is a table-cell control (no dialog of its own), so
|
||||||
|
its failure announces here (role=alert; git-sources.js
|
||||||
|
showHiddenError). Hidden until a PATCH fails. -->
|
||||||
|
<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>
|
||||||
|
```
|
||||||
|
2. `frontend/assets/git-sources.js` — `makeRow` (L357+):
|
||||||
|
- **State tag** — in the `urlTd` block, right after the phase-89 `N ignored` count-tag block (~L374-382), the same idiom:
|
||||||
|
```js
|
||||||
|
/* Phase 105 (A5): the "hidden on" state tag — the
|
||||||
|
.git-source-ignore-count idiom (TEXT + background, never
|
||||||
|
color alone — WCAG 1.4.1), so the flag is readable at a
|
||||||
|
glance without hovering the checkbox. */
|
||||||
|
if (s.id && s.include_hidden === true) {
|
||||||
|
const hiddenTag = document.createElement("span");
|
||||||
|
hiddenTag.className = "git-source-hidden-count";
|
||||||
|
hiddenTag.textContent = "hidden on";
|
||||||
|
urlTd.append(hiddenTag);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- **The checkbox** — in the `if (s.id) {` actions-cell branch (~L390), BEFORE the `ignoreBtn` construction (~L399), so DOM order is Hidden · Ignore paths · Remove:
|
||||||
|
```js
|
||||||
|
/* Phase 105 (A5): the per-row hidden-folders toggle — a native
|
||||||
|
labeled checkbox (the WCAG focus/label idiom) LEFT of the
|
||||||
|
"Ignore paths" button; Remove stays last. Stored rows only
|
||||||
|
(A3 — env-fallback rows fall through to the "from .env"
|
||||||
|
tag). The aria-label is the ONLY place `value` appears
|
||||||
|
(setAttribute — never innerHTML). Checked state comes from
|
||||||
|
the SERVER row (s.include_hidden), never from a prior local
|
||||||
|
flip (§7.4 — makeRow only ever renders server state). */
|
||||||
|
const hiddenLabel = document.createElement("label");
|
||||||
|
hiddenLabel.className = "git-source-hidden";
|
||||||
|
hiddenLabel.title =
|
||||||
|
"When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.";
|
||||||
|
const hiddenBox = document.createElement("input");
|
||||||
|
hiddenBox.type = "checkbox";
|
||||||
|
hiddenBox.className = "git-source-hidden-box";
|
||||||
|
hiddenBox.checked = s.include_hidden === true;
|
||||||
|
hiddenBox.setAttribute(
|
||||||
|
"aria-label",
|
||||||
|
`Index hidden folders for ${kindLabel} source: ${value}`,
|
||||||
|
);
|
||||||
|
hiddenLabel.append(hiddenBox, document.createTextNode("Hidden"));
|
||||||
|
hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox));
|
||||||
|
actTd.appendChild(hiddenLabel);
|
||||||
|
```
|
||||||
|
- **`toggleHidden(s, box)`** — new function next to `saveIgnorePaths` (the phase-89 §7.4 section, ~L593+), same announce/reload discipline:
|
||||||
|
```js
|
||||||
|
/* Phase 105: the hidden-folders toggle — PATCH { include_hidden }
|
||||||
|
only (the row's list is untouched — the PATCH body's optional
|
||||||
|
fields, task 03). §7.4 never-stale: the box disables at once
|
||||||
|
(no double-flip while the PATCH is out); on 200 the row
|
||||||
|
re-renders from the server (loadSources) and the
|
||||||
|
CONFIRMATION is the LAST announcement (the reload's "N
|
||||||
|
sources listed." lands first — the phase-89 order); on
|
||||||
|
failure the box REVERTS to the server state and the detail
|
||||||
|
lands in #git-sources-hidden-error (role=alert). */
|
||||||
|
function toggleHidden(s, box) {
|
||||||
|
const value = s.kind === "local" ? (s.path ?? s.url) : s.url;
|
||||||
|
const wanted = box.checked;
|
||||||
|
box.disabled = true; // a PATCH is out — the box must not flip twice
|
||||||
|
fetch(`/api/git-sources/${s.id}`, {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
credentials: "same-origin",
|
||||||
|
body: JSON.stringify({ include_hidden: wanted }),
|
||||||
|
})
|
||||||
|
.then(async (r) => {
|
||||||
|
if (r.ok) {
|
||||||
|
hideHiddenError();
|
||||||
|
await loadSources();
|
||||||
|
announce(`Hidden folders ${wanted ? "enabled" : "disabled"} for ${value}.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const detail = await apiDetail(
|
||||||
|
r, `Could not update the hidden-folders setting (${r.status}).`,
|
||||||
|
);
|
||||||
|
showHiddenError(detail);
|
||||||
|
box.checked = s.include_hidden === true; // revert to server state
|
||||||
|
box.disabled = false;
|
||||||
|
})
|
||||||
|
.catch(() => {
|
||||||
|
showHiddenError("Could not reach the server — the setting was not changed.");
|
||||||
|
box.checked = s.include_hidden === true;
|
||||||
|
box.disabled = false;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function showHiddenError(message) {
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.textContent = message;
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.hidden = false;
|
||||||
|
}
|
||||||
|
function hideHiddenError() {
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.textContent = "";
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.hidden = true;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
(Also call `hideHiddenError()` at the top of `loadSources`'s success path (~L315, after `hideLoadError()`) so a healed list clears the stale line — the phase-89 "happy path heals the error state" precedent.)
|
||||||
|
- Element grabber next to the other page-local grabs (~L245-255): `const hiddenErrorEl = root.querySelector("#git-sources-hidden-error");`
|
||||||
|
- Module header comment block (L1-177ish, the contract list): add the phase-105 entry — "the per-row Hidden checkbox (makeRow) → PATCH {include_hidden} (task 03's optional field) → loadSources + announce; failure reverts the box + #git-sources-hidden-error (role=alert); env-fallback rows get no checkbox (A3)".
|
||||||
|
3. `frontend/assets/styles.css` — after the phase-89 block (the `.git-source-ignore-count` rule ends ~L2695):
|
||||||
|
```css
|
||||||
|
/* Phase 105 (A5): the per-row "Hidden" checkbox — the actions
|
||||||
|
cell, LEFT of the "Ignore paths" button (JS builds it,
|
||||||
|
git-sources.js makeRow). A native labeled checkbox: the
|
||||||
|
visible "Hidden" text + the checkbox's own aria-label
|
||||||
|
(full source value). ~44px hit height matches the action
|
||||||
|
buttons; :focus-visible via the GLOBAL rule (L146) — no
|
||||||
|
per-control ring needed. */
|
||||||
|
.git-source-hidden {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.4rem;
|
||||||
|
height: 44px;
|
||||||
|
padding: 0 0.5rem;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
color: var(--ink);
|
||||||
|
cursor: pointer;
|
||||||
|
user-select: none;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
/* --brand checkbox fill: the native check renders in --bg on
|
||||||
|
--brand (the house 5.2:1 brand pairing); [executor: verify the
|
||||||
|
checked-state contrast of the native widget in both themes +
|
||||||
|
record the ratio here — house style]. */
|
||||||
|
.git-source-hidden input[type="checkbox"] {
|
||||||
|
width: 1.05rem;
|
||||||
|
height: 1.05rem;
|
||||||
|
margin: 0;
|
||||||
|
accent-color: var(--brand);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.git-source-hidden:disabled { opacity: 0.5; cursor: wait; }
|
||||||
|
/* Phase 105 (A5): the "hidden on" state tag — a copy of the
|
||||||
|
.git-source-ignore-count idiom (TEXT + background, never
|
||||||
|
color alone — WCAG 1.4.1: --ink on --bg 16.7:1). */
|
||||||
|
.git-source-hidden-count {
|
||||||
|
display: inline-block;
|
||||||
|
margin-left: 0.55rem;
|
||||||
|
padding: 0.08rem 0.5rem;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 999px;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
font-weight: 600;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--bg);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
4. `tests/unit/test_hidden_folders_toggle.py` (NEW) — source-level pins (the house pattern: read the asset files as text; mirror `tests/unit/test_source_ignore_paths.py`'s structure if it exists, else the closest JS-pinning suite):
|
||||||
|
- `git-sources.js`: `makeRow` contains the checkbox construction — class `git-source-hidden-box`, `type` checkbox, the aria-label template `` `Index hidden folders for ${kindLabel} source: ${value}` ``; `checked = s.include_hidden === true`; the label is appended to `actTd` BEFORE the ignore button (pin the slice order: the hidden-label block precedes the `ignoreBtn` construction); `toggleHidden` defined with `PATCH` + `body: JSON.stringify({ include_hidden: wanted })`; the failure branch reverts (`box.checked = s.include_hidden === true`) AND re-enables; the success branch announces AFTER `await loadSources()` (pin the order in the source slice).
|
||||||
|
- **Cross-file pin (single source of truth for the field name):** the JS body `include_hidden` key and the Python `GitSourcePatchIn.include_hidden` field (regex-parsed from `app/schemas.py`) are the SAME string — a rename on either side breaks the wire contract and this test.
|
||||||
|
- `index.html`: `#git-sources-hidden-error` exists, carries `role="alert"`, `hidden`, class `git-source-error`, and sits INSIDE `#view-git-sources` after `#git-sources-table-wrap` (source order).
|
||||||
|
- `styles.css`: the `.git-source-hidden`, `.git-source-hidden input[type="checkbox"]`, `.git-source-hidden:disabled`, `.git-source-hidden-count` rules exist; the checkbox rule sets `accent-color`.
|
||||||
|
- The `N ignored`-style tag text pin: `hidden on` literal present in the JS (the tag copy is the state — text, never color alone).
|
||||||
|
5. Run `uv run pytest tests/unit/test_hidden_folders_toggle.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (source-level pins) is this task's layer; the behavioral E2E is task 06.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` changes in this task — the gate is unaffected; keep it green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Every stored row renders Hidden · Ignore paths · Remove in the actions cell; env-fallback rows render no checkbox
|
||||||
|
- [ ] `checked` comes only from server state; a "hidden on" tag appears in the source cell iff the flag is on
|
||||||
|
- [ ] Flip → box disables → `PATCH {"include_hidden": …}` → 200: error cleared, list reloaded, confirmation announced LAST; failure: box reverts + re-enables, detail in the `role="alert"` line; the line heals on a successful load
|
||||||
|
- [ ] WCAG: visible label + full-value `aria-label` on the checkbox, global `:focus-visible` ring, tag is text (never color alone), checkbox checked-state contrast verified + recorded in the CSS comment
|
||||||
|
- [ ] The JS field-name pin matches `app/schemas.py` (cross-file test); `uv run pytest tests/unit/ -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Task 06 — E2E: `tests/e2e/test_hidden_folders_toggle.py` (isolation) + regressions + full gate + commit
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — the whole item, proven end to end.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One dedicated Playwright suite proving the TODO item through the REAL page + REAL API + REAL sync pipeline (mock LLM, no git, no network — a `kind="local"` row over a fixture dir, the `test_source_ignore_paths.py` module pattern verbatim): default-off byte-identity, toggle-on indexes hidden folders, toggle-off prunes them (A2), env rows have no control, a11y + the error line. Then the phase's full gate and the single atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_hidden_folders_toggle.py` (NEW) — copy the module scaffolding of `tests/e2e/test_source_ignore_paths.py` (the module-env `BOR_GIT_SOURCES` URL that is NEVER synced/cloned; `e2e/auth_helpers.login`; `e2e/conftest.py` `ADMIN_PASSWORD`/`SESSION_SECRET`/`USE_REAL_LLM`/`_wait_http`; the sync helper: `POST /api/sync` → poll `GET /api/sync/status` to terminal; the `source_dir` fixture built under `tmp_path_factory`) with the fixture tree:
|
||||||
|
```
|
||||||
|
visible.md
|
||||||
|
.hidden/note.md <- the phase-105 subject
|
||||||
|
.venv/junk.md <- EXCLUDED_DIRS: never indexed, both states (A1)
|
||||||
|
```
|
||||||
|
Contract under test (docstring):
|
||||||
|
- anonymous: the `#git-sources-gate` sign-in gate, the manager hidden, NO `/api/git-sources` call on load, 403 on `GET`/`POST /api/git-sources` AND `PATCH /api/git-sources/{id}` with a bool-only body (the phase-89 anonymous pin extended to the toggle payload);
|
||||||
|
- A4: with the default row, a sync indexes `visible.md` ONLY — `detail.files` counts one, `.hidden/note.md` has no `documents` row, the checkbox renders UNCHECKED and no "hidden on" tag;
|
||||||
|
- A1: flipping the checkbox on (the real click) → the PATCH 200 lands (the "hidden on" tag appears, the announcer `role=status` fires the confirmation AFTER the reload line) → sync → `.hidden/note.md` IS indexed (`documents` row present; the KB catalog lists it — the tree/catalog is DB-driven, no extra surface); `.venv/junk.md` STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders CHECKED (server state);
|
||||||
|
- A2: flipping it OFF (real click) → sync → `detail.pruned` includes the hidden doc, the catalog no longer lists it, the tag is gone;
|
||||||
|
- A3: the env-fallback view (the table-empty state) renders the "from .env" tag with NO Hidden checkbox and NO "Ignore paths" button;
|
||||||
|
- a11y + error surface: the checkbox has a full accessible name containing the source location (`Index hidden folders for local source: …`), is keyboard-focusable (Tab reaches it, the global `:focus-visible` ring applies), the tag text is "hidden on" (never color alone); `#git-sources-hidden-error` exists with `role="alert"` and stays `hidden` through the happy path.
|
||||||
|
Test → contract mapping (one test per bullet, the phase-89 suite's shape):
|
||||||
|
1. `test_anonymous_gate_and_403s`
|
||||||
|
2. `test_hidden_off_by_default`
|
||||||
|
3. `test_toggle_on_indexes_hidden_folders`
|
||||||
|
4. `test_toggle_off_prunes_hidden`
|
||||||
|
5. `test_env_fallback_rows_have_no_toggle`
|
||||||
|
6. `test_toggle_a11y_and_error_surface`
|
||||||
|
2. **Regressions** — each in isolation (DB up), all green:
|
||||||
|
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` (the phase-89 suite — the PATCH body rename + the actions-cell order must not break it; its dialog still sends the list and gets byte-identical replace semantics)
|
||||||
|
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
|
||||||
|
3. **Full gate** (AGENTS.md rule 9 — non-negotiable):
|
||||||
|
- `uv run pytest` (unit + integration) green
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **>90%**
|
||||||
|
- `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation
|
||||||
|
- `uv run ruff check . && uv run pyright` clean
|
||||||
|
4. **Commit** — one atomic Conventional-Commits commit, `--no-gpg-sign` (AGENTS.md rule 8), per the phase overview's Commit block:
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
|
||||||
|
```
|
||||||
|
(`TODO.md` is cleared to the bare `# TODO` by the conversion step BEFORE this commit lands — the items now live in this phase; if the pipeline commits per task instead, fold `TODO.md` into this phase's final commit and move the phase dir to `.agents/phases/complete/105_hidden_folders_toggle/` as the last action, per the pipeline gate.)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the phase's E2E + gate + commit; no new application code (the E2E may reveal a one-line fix in an earlier task's work — fix it IN that task's file, keep the suite's contract as written).
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate — enforced here, not assumed).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `tests/e2e/test_hidden_folders_toggle.py` exists, maps 1:1 to the six contract bullets, and passes in isolation (`--no-cov`, DB up)
|
||||||
|
- [ ] The five regression E2E suites pass in isolation (the phase-89 suite first — it is the rename's canary)
|
||||||
|
- [ ] The full gate is green: unit+integration, TOTAL coverage >90%, ruff + pyright clean
|
||||||
|
- [ ] One `--no-gpg-sign` commit contains the whole phase (app + alembic 0019 + scripts + frontend + tests + the cleared `TODO.md` + the phase files)
|
||||||
|
- [ ] The TODO item is done: the owner can flip "Hidden" per input next to its Ignore paths button, and the next sync indexes (or prunes, when off) the dot-prefixed paths of that source — with `.venv`/`node_modules`/`.git`/… always excluded
|
||||||
Reference in New Issue
Block a user