brain-of-reese/.agents/phases/complete/105_hidden_folders_toggle/00_phase.md

Phase 105 — Per-source hidden-folders toggle: dot-prefixed paths become indexable per input

Source: TODO.md L3 — "hidden dot folders aren't being indexed. There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders." Story: n/a (TODO-derived — owner roadmap confirmation 2026-09-14). Context: The single filesystem walk choke point is iter_importable_files (app/rag/importer.py L161-194): it skips any path with a dot-prefixed component — hidden dirs (vendored caches like .esphome/.espressif/**) AND hidden files — plus the well-known EXCLUDED_DIRS (.venv, node_modules, .git, __pycache__, .pytest_cache, dist, build); the extension filter (BOR_IMPORT_EXTENSIONS, the A9 family) then admits the rest. import_sources (L196+) walks every root twice when progress is set (the phase-64 pre-walk for the total denominator uses the EXACT same rules), collects (source, rel) into seen, and _prune (prune=True) deletes every indexed document of the imported sources whose (source, rel) is not in seen — the same mechanism phase 89 uses so newly-ignored files leave the index. Both live import entry points build a per-root ignore_by_root: dict[str, list[str]] map from the git_sources rows in one loop — the Sync button (app/api/sync.py::_run_sync L233-263) and the CLI (scripts/import_docs.py::_resolve_sources L181-226, consumed at L269/L332-334); the map is keyed by str(root) — the root path string exactly as passed to import_sources — with extend-on-collision for shared roots. scripts/load_test_kb.py calls import_sources with defaults (untouched). The archive-upload background run no longer scans (phase 90), so no map is needed there. Sources are the git_sources rows (kind git / local, phase 35/38) managed on the Sources page (/git-sources.html, view module frontend/assets/git-sources.js, skeleton in frontend/index.html #view-git-sources L514+); the per-row "Ignore paths" button (makeRow L357+, ignoreBtn ~L393-408) sits in the actions cell LEFT of Remove, stored rows only (env-fallback rows, id null, get no control — phase 89 A3); non-empty lists render a N ignored text tag in the source cell (.git-source-ignore-count, frontend/assets/styles.css L2686, text + background, never color alone). The admin-only PATCH /api/git-sources/{source_id} (phase 89 A5, app/api/git_sources.py L374-393) today takes the REQUIRED ignore_paths replace list; the read shapes are GitSourceRow (app/schemas.py L508) and GitSourceOut (L490), the create body GitSourceIn (L450, optional ignore_paths). Alembic head is 0018. The KB tree/catalog (phase 97) and the agent tools read the DB, so newly indexed hidden documents appear in them automatically — no change needed.

Objective

Each stored source carries an index-hidden-folders flag, toggled by a per-row checkbox next to the "Ignore paths" button on the Sources page. When ON for a source, the walk no longer skips dot-prefixed components for that source — files inside hidden folders (and hidden files with an importable extension) are indexed, embedded, and summarized exactly like visible files; when OFF (the default for every existing row), behavior is byte-identical to today. EXCLUDED_DIRS stays excluded in both states, the extension filter always applies, and the flag — like the ignore list — takes effect on the next sync, with previously indexed hidden files pruned when it is switched off. All entry points (Sync button, CLI) honor the flag; the API stays admin-only.

Dependencies

• 104_chip_sizing_question_cap (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/CLI pipelines, and the Sources view — none of which phase 104's pins reach; its suites must stay green unchanged).
• 89_source_ignore_paths (complete) — the per-root map, the actions-cell control idiom, the count-tag idiom, and the PATCH route this phase extends.

Design (shared by all tasks — the executor reads this, not the chat)

• Flag semantics (locked, A1). include_hidden=True lifts ONLY the dot-prefixed-component skip in iter_importable_files: the existing check any(part.startswith(".") or part in excluded for part in rel.parts) becomes dot-aware only when the flag is False — e.g. any((not include_hidden and part.startswith(".")) or part in excluded for part in rel.parts). Consequences, all deliberate:
  • Files INSIDE hidden dirs become importable (.esphome/esp.md indexed when ON).
  • Hidden files with an importable extension also become importable (.notes.md — the dot check covers components, not "the folder of the file", so one rule covers both; the extension filter is the real content gate, and a secret-flavoured file like .env has no A9 extension and is never indexed).
  • EXCLUDED_DIRS (.venv, node_modules, .git, __pycache__, .pytest_cache, dist, build) are skipped in BOTH states — caches/VCS internals are never content.
  • The ignore tuple (phase 89) composes additively with the flag: an ignored prefix still skips a file when include_hidden=True.
• Storage (task 01). git_sources.include_hidden — BOOLEAN NOT NULL, server default false, Mapped[bool] (the documents.is_summary Boolean precedent, app/models.py L136). Alembic 0019_git_source_include_hidden.py (revises 0018): op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False)); downgrade drops the column. Existing rows read False (A4).
• Importer signature (task 02).
  • iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore=(), include_hidden: bool = False) — default False keeps every existing caller byte-identical; the docstring's skip sentence gains the flag clause.
  • import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root=None, include_hidden_by_root: dict[str, bool] | None = None) — the map is keyed by str(root) with the SAME keying convention as ignore_by_root; an internal _include_hidden_for_root(root, include_hidden_by_root) -> bool (default False) is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop, so files_total never disagrees with the walk. seen is untouched in shape → _prune prunes hidden documents automatically when the flag flips OFF (A2 — the A9/phase-89 precedent). Module docstring "Scope" paragraph updated.
• API contract (task 03).
  • Schemas (app/schemas.py): GitSourceIn.include_hidden: bool | None = Field(default=None) (create-time, optional — absent → stored False); GitSourceOut.include_hidden: bool; GitSourceRow.include_hidden: bool (env-fallback rows report False — no DB row to store a flag on).
  • The PATCH body model is RENAMED GitSourceIgnoreIn → GitSourcePatchIn (grep-verified: referenced only in app/schemas.py and app/api/git_sources.py — import L119 + patch_git_source L376) and gains:
    • ignore_paths: list[str] | None = Field(default=None) — absent/None = the row's list is unchanged; PRESENT = replace semantics exactly as phase 89 A5 (normalization + the A4 fixed-detail 422s run only when present). Every existing client always sends the list, so their behavior is byte-identical; the toggle's PATCH sends only the bool.
    • include_hidden: bool | None = Field(default=None) — absent/None = unchanged; present = set.
    • Both absent → 200 no-op (row untouched).
  • GET /api/git-sources — DB rows report the stored flag; env rows False. POST /api/git-sources — both kinds accept include_hidden; stored bool(payload.include_hidden).
  • PATCH /api/git-sources/{source_id} (existing route, still behind require_admin) — applies each PRESENT field independently (404 unknown id unchanged); 200 → GitSourceOut (id, url, added_at, ignore_paths, include_hidden).
• Callers (task 04).
  • app/api/sync.py::_run_sync — in the existing per-row loop that builds ignore_by_root (L233-252), build include_hidden_by_root: dict[str, bool] with the SAME str(root) keying: include_hidden_by_root[str(root)] = include_hidden_by_root.get(str(root), False) or bool(row.include_hidden) (collision → OR — the mirror of the ignore-map union: if either row says "index hidden", the shared root does). Pass include_hidden_by_root=… to import_sources (L263). Module docstring (L36-40) updated.
  • scripts/import_docs.py — _resolve_sources returns the 3-tuple (sources, ignore_by_root, include_hidden_by_root) (manual --source → (sources, {}, {}) — manual dirs have no row; env-fallback rows have no flags); the OR-collision rule is the same; main unpacks (L269) and passes the map (L332-334); docstrings updated (module + _resolve_sources L181).
  • scripts/load_test_kb.py — untouched (defaults).
• UI (task 05). Sources page = the git-sources view. Per stored row (s.id truthy) in makeRow: a native labeled checkbox in the actions cell, DOM order Hidden · Ignore paths · Remove (the toggle sits next to — left of — the "Ignore paths" button, per the owner's wording; Remove stays last):
  • <label class="git-source-hidden"> wrapping <input type="checkbox" class="git-source-hidden-box"> + visible text "Hidden"; the checkbox's aria-label is Index hidden folders for ${kindLabel} source: ${value} (setAttribute — never innerHTML; value is the git URL or local path, credential-safety discipline), checked = s.include_hidden === true; a title on the label explains in plain words ("When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.").
  • When s.include_hidden === true, the source cell also shows a "hidden on" text tag (.git-source-hidden-count — the .git-source-ignore-count idiom: text + background, never color alone, WCAG 1.4.1), next to the N ignored tag.
  • §7.4 never-stale lifecycle — toggleHidden(s, box): on change, the box disables immediately (no double-flip); PATCH /api/git-sources/${s.id} with {"include_hidden": box.checked}; on 200 → clear the error line, await loadSources() (the row re-renders from the server), THEN announce("Hidden folders enabled|disabled for <value>.") (the phase-89 last-announce order — the confirmation lands after the reload's "N sources listed."); on non-2xx or network failure → the server detail (or the canned "Could not reach the server — the setting was not changed.") into the new page-level role="alert" line, and the box reverts to the server state (box.checked = s.include_hidden === true) and re-enables — the UI never claims a state the server didn't save.
  • frontend/index.html — one new element after the #git-sources-table-wrap region: <p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p> (reuses the existing .git-source-error styling). frontend/assets/styles.css — near the phase-89 block (~L2657-2695): .git-source-hidden (inline-flex, ~44px hit height matching the action buttons, visible label), .git-source-hidden input[type="checkbox"] (sized, accent-color on the brand pair — verify + record the AA ratio in the comment, house style), :disabled (opacity + cursor: wait — the .git-source-remove:disabled idiom), focus ring via the GLOBAL :focus-visible rule (L146 — no per-control rule needed), and .git-source-hidden-count (copy of the .git-source-ignore-count rule, provenance comment citing phase 105).
  • Env-fallback rows (id null) get no checkbox — the existing "from .env" tag stays (A3).
• NOT touched: the RAG view (sources.js), app/rag/retriever.py, the chunker, the KB tree/catalog + agent tools (DB-driven — they pick up newly indexed hidden docs for free), app/rag/git_sources.py (clone/pull only, no walk), the upload run (phase 90 — no scan), AGENTS.md, .agents/PLAN.md, any completed phase.

Tasks

1. 01_include_hidden_column.md — git_sources.include_hidden BOOLEAN column (model + alembic 0019) + default/round-trip tests.
2. 02_importer_include_hidden.md — iter_importable_files/import_sources flag support (walk + progress pre-walk + prune interaction + ignore composition) + unit & integration tests.
3. 03_include_hidden_api.md — schemas (In/Out/Row + the GitSourcePatchIn rename with optional fields) + GET/POST/PATCH wiring + integration tests.
4. 04_include_hidden_pipelines.md — wire the per-row flag into _run_sync and scripts/import_docs.py + integration tests.
5. 05_hidden_toggle_sources_ui.md — the per-row "Hidden" checkbox on the Sources page (tag + §7.4 lifecycle + error line + a11y) + source-level unit pins.
6. 06_e2e_hidden_folders_toggle.md — dedicated Playwright suite tests/e2e/test_hidden_folders_toggle.py (run in isolation), regressions, full gate, atomic commit.

Testing & Quality

• Unit — tests/unit/test_importer_include_hidden.py (new, task 02): iter_importable_files on a tmp fixture tree — default OFF pins today's behavior byte-identically (hidden dir + hidden file skipped), ON admits both, EXCLUDED_DIRS skipped in BOTH states, ignore tuple still bites when ON, extension filter unchanged (.env never indexed); the str(root) keying + default-False for unlisted roots; tests/unit/test_hidden_folders_toggle.py (new, task 05): source-level pins for the JS/HTML/CSS wiring (house pattern — read the assets as text), incl. the single-source cross-file check that the aria-label template names the source value; task 01's pins extend the existing model/migration test surfaces (a fresh row reads include_hidden is False; an explicit True round-trips).
• Integration — tests/integration/test_importer_include_hidden.py (new, task 02): import_sources against a fixture dir — hidden file produces NO Document/Chunk rows by default; WITH the map it is embedded + summarized normally; previously indexed hidden file + flag OFF → pruned on the next run; progress total agrees with the walk in both states; sources not in the map behave exactly as before. tests/integration/test_git_sources_api.py (extended, task 03): GET reports False default / stored True; POST create round-trip; PATCH bool-only, list-only, both, neither (no-op 200), the phase-89 422s unchanged for present lists, 404, anonymous 403. tests/integration/test_sync_api.py (extended, task 04): a local row with a hidden dir — flag False syncs zero hidden docs, True syncs them; tests/integration/test_import_docs_git.py (extended, task 04): the CLI DB-row path with the flag set.
• E2E (mandatory, A16) — tests/e2e/test_hidden_folders_toggle.py (task 06), run in isolation with the DB up: uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov.
• Coverage: >90% on app/ (the validate.sh gate — the importer/API/pipeline additions are fully unit+integration covered).

Completion Criteria

• A stored source (git, local, or uploaded) has a "Hidden" checkbox next to its "Ignore paths" button: flipping it on persists (PATCH 200, the source cell shows the "hidden on" tag, GET /api/git-sources round-trips include_hidden: true); the failure path reverts the box and announces the error in a role="alert" line.
• A sync (button or CLI) with the flag OFF indexes nothing with a dot-prefixed component (no documents/chunks rows — the byte-identical default); with the flag ON, .hidden/note.md is indexed, embedded, and summarized like any visible file and shows up in the KB catalog; EXCLUDED_DIRS content is excluded in both states.
• A2: a previously indexed hidden file is PRUNED from the KB on the next sync after the flag flips OFF (detail.pruned increments; the catalog no longer lists it).
• API contracts hold: PATCH bool-only / list-only / both / neither; the phase-89 fixed-detail 422s unchanged for present lists; 404 unknown id; anonymous 403 on the route.
• Env-fallback rows render no checkbox (the "from .env" tag stays); the Sources page stays WCAG-clean (visible label, focus-visible, text never color alone).
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov green in isolation (DB up); regression suites test_source_ignore_paths.py, test_git_sources_admin.py, test_local_directory_sources.py, test_sync_button.py, test_smoke.py green in isolation; uv run ruff check . && uv run pyright clean.
• One --no-gpg-sign commit; phase dir moved to .agents/phases/complete/ by the pipeline gate.

Locked decisions

• A1 — what "on" means (owner-confirmed 2026-09-14). The flag lifts the dot-prefixed-component skip for that source — files inside hidden folders AND hidden files with an importable extension become indexable; EXCLUDED_DIRS (.venv, node_modules, .git, __pycache__, .pytest_cache, dist, build) stay excluded regardless; the extension filter always applies.
• A2 — toggling off prunes (owner-confirmed). Previously indexed hidden files leave the KB on the next sync (the seen-set prune — the phase-89 A2 / A9 precedent), exactly like newly-ignored files.
• A3 — per stored row only (owner-confirmed). Every stored git_sources row (git, local, uploaded) gets the toggle; env-fallback rows have no DB row and get no control (the phase-89 A3 precedent).
• A4 — default off (owner-confirmed). include_hidden defaults to false for all existing rows — byte-identical behavior until the owner flips it; the flag takes effect on the NEXT sync (no auto-sync, same as the ignore list).
• A5 — control idiom (owner-confirmed). A native labeled checkbox ("Hidden") in the actions cell, left of the "Ignore paths" button (Remove stays last), with a "hidden on" text tag in the source cell when enabled — not a styled switch.

Commit

git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"