diff --git a/.agents/phases/todo/105_hidden_folders_toggle/00_phase.md b/.agents/phases/todo/105_hidden_folders_toggle/00_phase.md new file mode 100644 index 0000000..0b8bfd4 --- /dev/null +++ b/.agents/phases/todo/105_hidden_folders_toggle/00_phase.md @@ -0,0 +1,78 @@ +# Phase 105 — Per-source hidden-folders toggle: dot-prefixed paths become indexable per input + +**Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders." +**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-14). +**Context:** The single filesystem walk choke point is `iter_importable_files` (`app/rag/importer.py` L161-194): it skips any path with a dot-prefixed component — hidden dirs (vendored caches like `.esphome/.espressif/**`) AND hidden files — plus the well-known `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`); the extension filter (`BOR_IMPORT_EXTENSIONS`, the A9 family) then admits the rest. `import_sources` (L196+) walks every root twice when `progress` is set (the phase-64 pre-walk for the `total` denominator uses the EXACT same rules), collects `(source, rel)` into `seen`, and `_prune` (`prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — the same mechanism phase 89 uses so newly-ignored files leave the index. Both live import entry points build a per-root `ignore_by_root: dict[str, list[str]]` map from the `git_sources` rows in one loop — the Sync button (`app/api/sync.py::_run_sync` L233-263) and the CLI (`scripts/import_docs.py::_resolve_sources` L181-226, consumed at L269/L332-334); the map is keyed by `str(root)` — the root path string exactly as passed to `import_sources` — with extend-on-collision for shared roots. `scripts/load_test_kb.py` calls `import_sources` with defaults (untouched). The archive-upload background run no longer scans (phase 90), so no map is needed there. Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources` L514+); the per-row "Ignore paths" button (`makeRow` L357+, `ignoreBtn` ~L393-408) sits in the actions cell LEFT of Remove, stored rows only (env-fallback rows, `id` null, get no control — phase 89 A3); non-empty lists render a `N ignored` text tag in the source cell (`.git-source-ignore-count`, `frontend/assets/styles.css` L2686, text + background, never color alone). The admin-only `PATCH /api/git-sources/{source_id}` (phase 89 A5, `app/api/git_sources.py` L374-393) today takes the REQUIRED `ignore_paths` replace list; the read shapes are `GitSourceRow` (`app/schemas.py` L508) and `GitSourceOut` (L490), the create body `GitSourceIn` (L450, optional `ignore_paths`). Alembic head is `0018`. The KB tree/catalog (phase 97) and the agent tools read the DB, so newly indexed hidden documents appear in them automatically — no change needed. + +## Objective +Each stored source carries an **index-hidden-folders flag**, toggled by a per-row checkbox next to the "Ignore paths" button on the Sources page. When ON for a source, the walk no longer skips dot-prefixed components for that source — files inside hidden folders (and hidden files with an importable extension) are indexed, embedded, and summarized exactly like visible files; when OFF (the default for every existing row), behavior is byte-identical to today. `EXCLUDED_DIRS` stays excluded in both states, the extension filter always applies, and the flag — like the ignore list — takes effect on the next sync, with previously indexed hidden files pruned when it is switched off. All entry points (Sync button, CLI) honor the flag; the API stays admin-only. + +## Dependencies +- `104_chip_sizing_question_cap` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/CLI pipelines, and the Sources view — none of which phase 104's pins reach; its suites must stay green unchanged). +- `89_source_ignore_paths` (complete) — the per-root map, the actions-cell control idiom, the count-tag idiom, and the `PATCH` route this phase extends. + +## Design (shared by all tasks — the executor reads this, not the chat) + +- **Flag semantics (locked, A1).** `include_hidden=True` lifts ONLY the dot-prefixed-component skip in `iter_importable_files`: the existing check `any(part.startswith(".") or part in excluded for part in rel.parts)` becomes dot-aware only when the flag is False — e.g. `any((not include_hidden and part.startswith(".")) or part in excluded for part in rel.parts)`. Consequences, all deliberate: + - Files INSIDE hidden dirs become importable (`.esphome/esp.md` indexed when ON). + - Hidden files with an importable extension also become importable (`.notes.md` — the dot check covers components, not "the folder of the file", so one rule covers both; the extension filter is the real content gate, and a secret-flavoured file like `.env` has no A9 extension and is never indexed). + - `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) are skipped in BOTH states — caches/VCS internals are never content. + - The `ignore` tuple (phase 89) composes additively with the flag: an ignored prefix still skips a file when `include_hidden=True`. +- **Storage (task 01).** `git_sources.include_hidden` — BOOLEAN NOT NULL, server default `false`, `Mapped[bool]` (the `documents.is_summary` Boolean precedent, `app/models.py` L136). Alembic `0019_git_source_include_hidden.py` (revises `0018`): `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`; downgrade drops the column. Existing rows read `False` (A4). +- **Importer signature (task 02).** + - `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore=(), include_hidden: bool = False)` — default `False` keeps every existing caller byte-identical; the docstring's skip sentence gains the flag clause. + - `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root=None, include_hidden_by_root: dict[str, bool] | None = None)` — the map is keyed by **`str(root)`** with the SAME keying convention as `ignore_by_root`; an internal `_include_hidden_for_root(root, include_hidden_by_root) -> bool` (default `False`) is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop, so `files_total` never disagrees with the walk. `seen` is untouched in shape → `_prune` prunes hidden documents automatically when the flag flips OFF (A2 — the A9/phase-89 precedent). Module docstring "Scope" paragraph updated. +- **API contract (task 03).** + - Schemas (`app/schemas.py`): `GitSourceIn.include_hidden: bool | None = Field(default=None)` (create-time, optional — absent → stored `False`); `GitSourceOut.include_hidden: bool`; `GitSourceRow.include_hidden: bool` (env-fallback rows report `False` — no DB row to store a flag on). + - The PATCH body model is RENAMED `GitSourceIgnoreIn` → `GitSourcePatchIn` (grep-verified: referenced only in `app/schemas.py` and `app/api/git_sources.py` — import L119 + `patch_git_source` L376) and gains: + - `ignore_paths: list[str] | None = Field(default=None)` — **absent/None = the row's list is unchanged; PRESENT = replace semantics exactly as phase 89 A5** (normalization + the A4 fixed-detail 422s run only when present). Every existing client always sends the list, so their behavior is byte-identical; the toggle's PATCH sends only the bool. + - `include_hidden: bool | None = Field(default=None)` — absent/None = unchanged; present = set. + - Both absent → 200 no-op (row untouched). + - `GET /api/git-sources` — DB rows report the stored flag; env rows `False`. `POST /api/git-sources` — both kinds accept `include_hidden`; stored `bool(payload.include_hidden)`. + - `PATCH /api/git-sources/{source_id}` (existing route, still behind `require_admin`) — applies each PRESENT field independently (404 unknown id unchanged); 200 → `GitSourceOut` (id, url, added_at, ignore_paths, include_hidden). +- **Callers (task 04).** + - `app/api/sync.py::_run_sync` — in the existing per-row loop that builds `ignore_by_root` (L233-252), build `include_hidden_by_root: dict[str, bool]` with the SAME `str(root)` keying: `include_hidden_by_root[str(root)] = include_hidden_by_root.get(str(root), False) or bool(row.include_hidden)` (collision → OR — the mirror of the ignore-map union: if either row says "index hidden", the shared root does). Pass `include_hidden_by_root=…` to `import_sources` (L263). Module docstring (L36-40) updated. + - `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple `(sources, ignore_by_root, include_hidden_by_root)` (manual `--source` → `(sources, {}, {})` — manual dirs have no row; env-fallback rows have no flags); the OR-collision rule is the same; `main` unpacks (L269) and passes the map (L332-334); docstrings updated (module + `_resolve_sources` L181). + - `scripts/load_test_kb.py` — untouched (defaults). +- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: a **native labeled checkbox** in the actions cell, DOM order **Hidden · Ignore paths · Remove** (the toggle sits next to — left of — the "Ignore paths" button, per the owner's wording; Remove stays last): + - `