phase: 105_hidden_folders_toggle

All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.

.agents/phases/complete/105_hidden_folders_toggle/00_phase.md

@@ -0,0 +1,78 @@
+# Phase 105 — Per-source hidden-folders toggle: dot-prefixed paths become indexable per input
+
+**Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
+**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-14).
+**Context:** The single filesystem walk choke point is `iter_importable_files` (`app/rag/importer.py` L161-194): it skips any path with a dot-prefixed component — hidden dirs (vendored caches like `.esphome/.espressif/**`) AND hidden files — plus the well-known `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`); the extension filter (`BOR_IMPORT_EXTENSIONS`, the A9 family) then admits the rest. `import_sources` (L196+) walks every root twice when `progress` is set (the phase-64 pre-walk for the `total` denominator uses the EXACT same rules), collects `(source, rel)` into `seen`, and `_prune` (`prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — the same mechanism phase 89 uses so newly-ignored files leave the index. Both live import entry points build a per-root `ignore_by_root: dict[str, list[str]]` map from the `git_sources` rows in one loop — the Sync button (`app/api/sync.py::_run_sync` L233-263) and the CLI (`scripts/import_docs.py::_resolve_sources` L181-226, consumed at L269/L332-334); the map is keyed by `str(root)` — the root path string exactly as passed to `import_sources` — with extend-on-collision for shared roots. `scripts/load_test_kb.py` calls `import_sources` with defaults (untouched). The archive-upload background run no longer scans (phase 90), so no map is needed there. Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources` L514+); the per-row "Ignore paths" button (`makeRow` L357+, `ignoreBtn` ~L393-408) sits in the actions cell LEFT of Remove, stored rows only (env-fallback rows, `id` null, get no control — phase 89 A3); non-empty lists render a `N ignored` text tag in the source cell (`.git-source-ignore-count`, `frontend/assets/styles.css` L2686, text + background, never color alone). The admin-only `PATCH /api/git-sources/{source_id}` (phase 89 A5, `app/api/git_sources.py` L374-393) today takes the REQUIRED `ignore_paths` replace list; the read shapes are `GitSourceRow` (`app/schemas.py` L508) and `GitSourceOut` (L490), the create body `GitSourceIn` (L450, optional `ignore_paths`). Alembic head is `0018`. The KB tree/catalog (phase 97) and the agent tools read the DB, so newly indexed hidden documents appear in them automatically — no change needed.
+
+## Objective
+Each stored source carries an **index-hidden-folders flag**, toggled by a per-row checkbox next to the "Ignore paths" button on the Sources page. When ON for a source, the walk no longer skips dot-prefixed components for that source — files inside hidden folders (and hidden files with an importable extension) are indexed, embedded, and summarized exactly like visible files; when OFF (the default for every existing row), behavior is byte-identical to today. `EXCLUDED_DIRS` stays excluded in both states, the extension filter always applies, and the flag — like the ignore list — takes effect on the next sync, with previously indexed hidden files pruned when it is switched off. All entry points (Sync button, CLI) honor the flag; the API stays admin-only.
+
+## Dependencies
+- `104_chip_sizing_question_cap` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/CLI pipelines, and the Sources view — none of which phase 104's pins reach; its suites must stay green unchanged).
+- `89_source_ignore_paths` (complete) — the per-root map, the actions-cell control idiom, the count-tag idiom, and the `PATCH` route this phase extends.
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+
+- **Flag semantics (locked, A1).** `include_hidden=True` lifts ONLY the dot-prefixed-component skip in `iter_importable_files`: the existing check `any(part.startswith(".") or part in excluded for part in rel.parts)` becomes dot-aware only when the flag is False — e.g. `any((not include_hidden and part.startswith(".")) or part in excluded for part in rel.parts)`. Consequences, all deliberate:
+  - Files INSIDE hidden dirs become importable (`.esphome/esp.md` indexed when ON).
+  - Hidden files with an importable extension also become importable (`.notes.md` — the dot check covers components, not "the folder of the file", so one rule covers both; the extension filter is the real content gate, and a secret-flavoured file like `.env` has no A9 extension and is never indexed).
+  - `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) are skipped in BOTH states — caches/VCS internals are never content.
+  - The `ignore` tuple (phase 89) composes additively with the flag: an ignored prefix still skips a file when `include_hidden=True`.
+- **Storage (task 01).** `git_sources.include_hidden` — BOOLEAN NOT NULL, server default `false`, `Mapped[bool]` (the `documents.is_summary` Boolean precedent, `app/models.py` L136). Alembic `0019_git_source_include_hidden.py` (revises `0018`): `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`; downgrade drops the column. Existing rows read `False` (A4).
+- **Importer signature (task 02).**
+  - `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore=(), include_hidden: bool = False)` — default `False` keeps every existing caller byte-identical; the docstring's skip sentence gains the flag clause.
+  - `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root=None, include_hidden_by_root: dict[str, bool] | None = None)` — the map is keyed by **`str(root)`** with the SAME keying convention as `ignore_by_root`; an internal `_include_hidden_for_root(root, include_hidden_by_root) -> bool` (default `False`) is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop, so `files_total` never disagrees with the walk. `seen` is untouched in shape → `_prune` prunes hidden documents automatically when the flag flips OFF (A2 — the A9/phase-89 precedent). Module docstring "Scope" paragraph updated.
+- **API contract (task 03).**
+  - Schemas (`app/schemas.py`): `GitSourceIn.include_hidden: bool | None = Field(default=None)` (create-time, optional — absent → stored `False`); `GitSourceOut.include_hidden: bool`; `GitSourceRow.include_hidden: bool` (env-fallback rows report `False` — no DB row to store a flag on).
+  - The PATCH body model is RENAMED `GitSourceIgnoreIn` → `GitSourcePatchIn` (grep-verified: referenced only in `app/schemas.py` and `app/api/git_sources.py` — import L119 + `patch_git_source` L376) and gains:
+    - `ignore_paths: list[str] | None = Field(default=None)` — **absent/None = the row's list is unchanged; PRESENT = replace semantics exactly as phase 89 A5** (normalization + the A4 fixed-detail 422s run only when present). Every existing client always sends the list, so their behavior is byte-identical; the toggle's PATCH sends only the bool.
+    - `include_hidden: bool | None = Field(default=None)` — absent/None = unchanged; present = set.
+    - Both absent → 200 no-op (row untouched).
+  - `GET /api/git-sources` — DB rows report the stored flag; env rows `False`. `POST /api/git-sources` — both kinds accept `include_hidden`; stored `bool(payload.include_hidden)`.
+  - `PATCH /api/git-sources/{source_id}` (existing route, still behind `require_admin`) — applies each PRESENT field independently (404 unknown id unchanged); 200 → `GitSourceOut` (id, url, added_at, ignore_paths, include_hidden).
+- **Callers (task 04).**
+  - `app/api/sync.py::_run_sync` — in the existing per-row loop that builds `ignore_by_root` (L233-252), build `include_hidden_by_root: dict[str, bool]` with the SAME `str(root)` keying: `include_hidden_by_root[str(root)] = include_hidden_by_root.get(str(root), False) or bool(row.include_hidden)` (collision → OR — the mirror of the ignore-map union: if either row says "index hidden", the shared root does). Pass `include_hidden_by_root=…` to `import_sources` (L263). Module docstring (L36-40) updated.
+  - `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple `(sources, ignore_by_root, include_hidden_by_root)` (manual `--source` → `(sources, {}, {})` — manual dirs have no row; env-fallback rows have no flags); the OR-collision rule is the same; `main` unpacks (L269) and passes the map (L332-334); docstrings updated (module + `_resolve_sources` L181).
+  - `scripts/load_test_kb.py` — untouched (defaults).
+- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: a **native labeled checkbox** in the actions cell, DOM order **Hidden · Ignore paths · Remove** (the toggle sits next to — left of — the "Ignore paths" button, per the owner's wording; Remove stays last):
+  - `<label class="git-source-hidden">` wrapping `<input type="checkbox" class="git-source-hidden-box">` + visible text "Hidden"; the checkbox's `aria-label` is `Index hidden folders for ${kindLabel} source: ${value}` (setAttribute — never innerHTML; `value` is the git URL or local path, credential-safety discipline), `checked = s.include_hidden === true`; a `title` on the label explains in plain words ("When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.").
+  - When `s.include_hidden === true`, the source cell also shows a **"hidden on" text tag** (`.git-source-hidden-count` — the `.git-source-ignore-count` idiom: text + background, never color alone, WCAG 1.4.1), next to the `N ignored` tag.
+  - **§7.4 never-stale lifecycle** — `toggleHidden(s, box)`: on `change`, the box disables immediately (no double-flip); `PATCH /api/git-sources/${s.id}` with `{"include_hidden": box.checked}`; on 200 → clear the error line, `await loadSources()` (the row re-renders from the server), THEN `announce("Hidden folders enabled|disabled for <value>.")` (the phase-89 last-announce order — the confirmation lands after the reload's "N sources listed."); on non-2xx or network failure → the server detail (or the canned "Could not reach the server — the setting was not changed.") into the new page-level `role="alert"` line, and the box **reverts to the server state** (`box.checked = s.include_hidden === true`) and re-enables — the UI never claims a state the server didn't save.
+  - `frontend/index.html` — one new element after the `#git-sources-table-wrap` region: `<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>` (reuses the existing `.git-source-error` styling). `frontend/assets/styles.css` — near the phase-89 block (~L2657-2695): `.git-source-hidden` (inline-flex, ~44px hit height matching the action buttons, visible label), `.git-source-hidden input[type="checkbox"]` (sized, `accent-color` on the brand pair — verify + record the AA ratio in the comment, house style), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), focus ring via the GLOBAL `:focus-visible` rule (L146 — no per-control rule needed), and `.git-source-hidden-count` (copy of the `.git-source-ignore-count` rule, provenance comment citing phase 105).
+  - Env-fallback rows (`id` null) get **no** checkbox — the existing "from .env" tag stays (A3).
+- **NOT touched:** the RAG view (`sources.js`), `app/rag/retriever.py`, the chunker, the KB tree/catalog + agent tools (DB-driven — they pick up newly indexed hidden docs for free), `app/rag/git_sources.py` (clone/pull only, no walk), the upload run (phase 90 — no scan), `AGENTS.md`, `.agents/PLAN.md`, any completed phase.
+
+## Tasks
+1. `01_include_hidden_column.md` — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`) + default/round-trip tests.
+2. `02_importer_include_hidden.md` — `iter_importable_files`/`import_sources` flag support (walk + progress pre-walk + prune interaction + ignore composition) + unit & integration tests.
+3. `03_include_hidden_api.md` — schemas (`In`/`Out`/`Row` + the `GitSourcePatchIn` rename with optional fields) + GET/POST/PATCH wiring + integration tests.
+4. `04_include_hidden_pipelines.md` — wire the per-row flag into `_run_sync` and `scripts/import_docs.py` + integration tests.
+5. `05_hidden_toggle_sources_ui.md` — the per-row "Hidden" checkbox on the Sources page (tag + §7.4 lifecycle + error line + a11y) + source-level unit pins.
+6. `06_e2e_hidden_folders_toggle.md` — dedicated Playwright suite `tests/e2e/test_hidden_folders_toggle.py` (run in isolation), regressions, full gate, atomic commit.
+
+## Testing & Quality
+- Unit — `tests/unit/test_importer_include_hidden.py` (new, task 02): `iter_importable_files` on a tmp fixture tree — default OFF pins today's behavior byte-identically (hidden dir + hidden file skipped), ON admits both, `EXCLUDED_DIRS` skipped in BOTH states, `ignore` tuple still bites when ON, extension filter unchanged (`.env` never indexed); the `str(root)` keying + default-False for unlisted roots; `tests/unit/test_hidden_folders_toggle.py` (new, task 05): source-level pins for the JS/HTML/CSS wiring (house pattern — read the assets as text), incl. the single-source cross-file check that the aria-label template names the source value; task 01's pins extend the existing model/migration test surfaces (a fresh row reads `include_hidden is False`; an explicit `True` round-trips).
+- Integration — `tests/integration/test_importer_include_hidden.py` (new, task 02): `import_sources` against a fixture dir — hidden file produces NO `Document`/`Chunk` rows by default; WITH the map it is embedded + summarized normally; previously indexed hidden file + flag OFF → pruned on the next run; progress `total` agrees with the walk in both states; sources not in the map behave exactly as before. `tests/integration/test_git_sources_api.py` (extended, task 03): GET reports `False` default / stored `True`; POST create round-trip; PATCH bool-only, list-only, both, neither (no-op 200), the phase-89 422s unchanged for present lists, 404, anonymous 403. `tests/integration/test_sync_api.py` (extended, task 04): a local row with a hidden dir — flag False syncs zero hidden docs, True syncs them; `tests/integration/test_import_docs_git.py` (extended, task 04): the CLI DB-row path with the flag set.
+- E2E (mandatory, A16) — `tests/e2e/test_hidden_folders_toggle.py` (task 06), run in isolation with the DB up: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov`.
+- Coverage: **>90%** on `app/` (the validate.sh gate — the importer/API/pipeline additions are fully unit+integration covered).
+
+## Completion Criteria
+- [ ] A stored source (git, local, or uploaded) has a "Hidden" checkbox next to its "Ignore paths" button: flipping it on persists (`PATCH` 200, the source cell shows the "hidden on" tag, `GET /api/git-sources` round-trips `include_hidden: true`); the failure path reverts the box and announces the error in a `role="alert"` line.
+- [ ] A sync (button or CLI) with the flag OFF indexes nothing with a dot-prefixed component (no `documents`/`chunks` rows — the byte-identical default); with the flag ON, `.hidden/note.md` is indexed, embedded, and summarized like any visible file and shows up in the KB catalog; `EXCLUDED_DIRS` content is excluded in both states.
+- [ ] A2: a previously indexed hidden file is PRUNED from the KB on the next sync after the flag flips OFF (`detail.pruned` increments; the catalog no longer lists it).
+- [ ] API contracts hold: PATCH bool-only / list-only / both / neither; the phase-89 fixed-detail 422s unchanged for present lists; 404 unknown id; anonymous 403 on the route.
+- [ ] Env-fallback rows render no checkbox (the "from .env" tag stays); the Sources page stays WCAG-clean (visible label, focus-visible, text never color alone).
+- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation (DB up); regression suites `test_source_ignore_paths.py`, `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
+- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
+
+## Locked decisions
+- **A1 — what "on" means (owner-confirmed 2026-09-14).** The flag lifts the dot-prefixed-component skip for that source — files inside hidden folders AND hidden files with an importable extension become indexable; `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) stay excluded regardless; the extension filter always applies.
+- **A2 — toggling off prunes (owner-confirmed).** Previously indexed hidden files leave the KB on the next sync (the `seen`-set prune — the phase-89 A2 / A9 precedent), exactly like newly-ignored files.
+- **A3 — per stored row only (owner-confirmed).** Every stored `git_sources` row (git, local, uploaded) gets the toggle; env-fallback rows have no DB row and get no control (the phase-89 A3 precedent).
+- **A4 — default off (owner-confirmed).** `include_hidden` defaults to `false` for all existing rows — byte-identical behavior until the owner flips it; the flag takes effect on the NEXT sync (no auto-sync, same as the ignore list).
+- **A5 — control idiom (owner-confirmed).** A native labeled checkbox ("Hidden") in the actions cell, left of the "Ignore paths" button (Remove stays last), with a "hidden on" text tag in the source cell when enabled — not a styled switch.
+
+## Commit
+```bash
+git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
+```