phase: 105_hidden_folders_toggle

All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.

app/api/git_sources.py

@@ -16,15 +16,20 @@ Routes: ``GET`` (DB rows oldest-first, or the env list with
 ``from_env: true`` while the table is empty; rows carry ``kind`` +
 ``path``, git rows — and env rows — report ``path: null``; every row
 reports ``ignore_paths`` — DB rows their stored normalized list, env
-rows ``[]``, phase 89), ``POST``
+rows ``[]``, phase 89 — and ``include_hidden`` — DB rows their stored
+flag, env rows ``False``, phase 105), ``POST``
 (201, validated create; ``kind`` selects the validation: git → exactly
 the phase-35 URL contract, local → an existing absolute directory, else
 422 naming the path; both kinds accept ``ignore_paths`` — optional,
-absent → ``[]``, stored normalized, phase 89),
-``PATCH /{source_id}`` (phase 89, A5 — replace one source's ignore
-list: 404 unknown id, the required body list is normalized + A4-
-validated with fixed-detail 422s and REPLACES the row's list wholesale
-— an empty list clears all; 200 → the ``GitSourceOut`` shape),
+absent → ``[]``, stored normalized, phase 89 — and ``include_hidden``
+— optional, absent → ``False``, stored as sent, phase 105),
+``PATCH /{source_id}`` (phase 89, A5 + phase 105 — the ignore list
+(replace) and/or the hidden-folders flag (phase 105) — each optional,
+present-wins: 404 unknown id; ``ignore_paths`` when present is
+normalized + A4-validated with fixed-detail 422s and REPLACES the
+row's list wholesale — an empty list clears all; ``include_hidden``
+when present sets the flag; both absent → 200 no-op; 200 → the
+``GitSourceOut`` shape),
 ``POST /upload`` (phase 49, backgrounded in phase
 64 task 03, scan deferred in phase 90 — admin archive upload: the
 ``.tar``/``.tar.gz``/``.tgz``/``.zip`` name/format gate + the 1 MiB-
@@ -116,10 +121,10 @@ from app.rag.source_removal import (
 )
 from app.rag.sources_meta import bump_sources_version
 from app.schemas import (
-    GitSourceIgnoreIn,
     GitSourceIn,
     GitSourceList,
     GitSourceOut,
+    GitSourcePatchIn,
     GitSourceRow,
     UploadAccepted,
 )
@@ -208,11 +213,13 @@ def list_git_sources(
     same-timestamp inserts) with ``from_env: false`` — each row carries
     its ``kind``, its ``ignore_paths`` (phase 89 — the stored,
     normalized list; ``or []`` guards a row that predated the column),
-    and, for local rows, the stored ``path`` (git rows and env rows
-    report ``path: null``); while the table is empty, the
+    its ``include_hidden`` (phase 105 — the stored flag), and, for
+    local rows, the stored ``path`` (git rows and env rows report
+    ``path: null``); while the table is empty, the
     ``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
-    git-only) with null ``id``/``added_at``, ``ignore_paths: []`` (no
-    DB row to store a list on), and ``from_env: true``.
+    git-only) with null ``id``/``added_at``, ``ignore_paths: []`` and
+    ``include_hidden: False`` (no DB row to store a list or a flag on),
+    and ``from_env: true``.
     """
     rows = db.scalars(
         select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
@@ -229,6 +236,7 @@ def list_git_sources(
                     path=row.path,
                     added_at=row.added_at,
                     ignore_paths=row.ignore_paths or [],
+                    include_hidden=row.include_hidden,
                 )
                 for row in rows
             ],
@@ -237,7 +245,13 @@ def list_git_sources(
     return GitSourceList(
         sources=[
             GitSourceRow(
-                id=None, kind="git", url=url, path=None, added_at=None, ignore_paths=[]
+                id=None,
+                kind="git",
+                url=url,
+                path=None,
+                added_at=None,
+                ignore_paths=[],
+                include_hidden=False,
             )
             for url in get_settings().git_source_list
         ],
@@ -272,10 +286,18 @@ def create_git_source(
     lines are normalized + A4-validated (``_validate_ignore_paths`` —
     the fixed-detail 422s) and the normalized list is what is stored and
     reported.
+
+    ``include_hidden`` (phase 105) — optional, both kinds: absent →
+    stored ``False`` (A4), present → stored as sent; the stored flag is
+    what is reported.
     """
     row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
     return GitSourceOut(
-        id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
+        id=row.id,
+        url=row.url,
+        added_at=row.added_at,
+        ignore_paths=row.ignore_paths,
+        include_hidden=row.include_hidden,
     )
 
 
@@ -331,6 +353,7 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
             url=url,
             kind="git",
             ignore_paths=_validate_ignore_paths(payload.ignore_paths),
+            include_hidden=bool(payload.include_hidden),
         ),
         "a git source with this URL already exists",
         db,
@@ -364,6 +387,7 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
             kind="local",
             path=path,
             ignore_paths=_validate_ignore_paths(payload.ignore_paths),
+            include_hidden=bool(payload.include_hidden),
         ),
         f"a local source with this path already exists: {path}",
         db,
@@ -373,24 +397,33 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
 @router.patch("/{source_id}", response_model=GitSourceOut)
 def patch_git_source(
     source_id: uuid.UUID,
-    payload: GitSourceIgnoreIn,
+    payload: GitSourcePatchIn,
     db: Session = Depends(get_db),  # noqa: B008
 ) -> GitSourceOut:
-    """Replace one source's ignore list (phase 89, A5).
+    """Edit one source's ignore list and/or hidden-folders flag.
 
-    404 unknown id. The body list (required) is normalized +
-    A4-validated (fixed 422 details) and REPLACES the row's list
-    wholesale — an empty list clears all. Returns the updated
-    row's public shape (id, url, added_at, ignore_paths).
+    Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
+    id; each PRESENT body field applies independently —
+    ``ignore_paths`` REPLACES the list (normalized + A4-validated,
+    fixed 422 details); ``include_hidden`` sets the flag. Both
+    absent → 200 no-op. Returns the updated row's public shape
+    (id, url, added_at, ignore_paths, include_hidden).
     """
     row = db.get(GitSource, source_id)
     if row is None:
         raise HTTPException(status_code=404, detail="git source not found")
-    row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
+    if payload.ignore_paths is not None:
+        row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
+    if payload.include_hidden is not None:
+        row.include_hidden = payload.include_hidden
     db.commit()
     db.refresh(row)
     return GitSourceOut(
-        id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
+        id=row.id,
+        url=row.url,
+        added_at=row.added_at,
+        ignore_paths=row.ignore_paths,
+        include_hidden=row.include_hidden,
     )
 
 

app/api/sync.py

@@ -36,7 +36,9 @@ decisions):
 4. ``import_sources(..., prune=True)`` over the single combined list
    (git checkouts + local dirs), honoring each row's ``ignore_paths``
    (phase 89 — the per-root ignore map is built in the same per-row
-   loop as the source list) — prune so files deleted upstream, out of
+   loop as the source list) and its ``include_hidden`` flag (phase 105
+   — the per-root hidden-folders map, same per-row construction) —
+   prune so files deleted upstream, out of
    a local dir, or newly matching an ignore pattern leave the index
    (pruning covers the union; the CLI's no-prune default is unchanged);
 5. when the import changed the KB (added + updated > 0),
@@ -231,6 +233,7 @@ async def _run_sync() -> None:
         sources_root = Path(settings.sources_dir).expanduser()
         sources: list[Path] = []
         ignore_by_root: dict[str, list[str]] = {}
+        include_hidden_by_root: dict[str, bool] = {}
         for row in rows:
             if row.kind == "git":
                 root = clone_or_pull(row.url, sources_root / repo_name(row.url))
@@ -250,6 +253,14 @@ async def _run_sync() -> None:
             # edge.
             if row.ignore_paths:
                 ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
+            # Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
+            # the SAME root string the importer sees; a shared-root
+            # collision ORs — if EITHER row says "index hidden", the
+            # root does (the ignore-map union's boolean mirror).
+            include_hidden_by_root[str(root)] = (
+                include_hidden_by_root.get(str(root), False)
+                or bool(row.include_hidden)
+            )
         # Phase 64 (task 02): the per-file progress hook — the status
         # endpoint reports the file being processed right now. The
         # closure captures the module ``_status`` exactly like the state
@@ -260,7 +271,8 @@ async def _run_sync() -> None:
             _status.files_total = total
 
         summary: ImportSummary = await import_sources(
-            sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root
+            sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
+            include_hidden_by_root=include_hidden_by_root,
         )
         overview = False
         if summary.added + summary.updated > 0:

app/models.py

@@ -18,7 +18,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
                       import (phase 35; ``kind`` discriminator added in
                       phase 38; ``ignore_paths`` (phase 89 — JSONB list
                       of normalized path prefixes, server default
-                      ``'[]'``)).
+                      ``'[]'``); ``include_hidden`` (phase 105 — index
+                      dot-prefixed paths from this source, server
+                      default ``false``)).
 * ``saved_chats``    — owner-saved chat conversations: one row per
                       explicitly Saved conversation (auto-``title`` +
                       the ``bor.chat.v1`` message list as JSONB, phase
@@ -248,6 +250,19 @@ class GitSource(Base):
     ignore_paths: Mapped[list] = mapped_column(
         JSONB, default=list, server_default=text("'[]'"), nullable=False
     )
+    #: Index hidden (dot-prefixed) paths from this source (phase 105,
+    #: A1): True → the walk (app.rag.importer.iter_importable_files)
+    #: does not skip dot-prefixed components — files inside hidden
+    #: folders AND hidden files with an importable extension are
+    #: indexed; ``EXCLUDED_DIRS`` (``.venv``, ``node_modules``,
+    #: ``.git``, …) are excluded in BOTH states, and the extension
+    #: filter always applies. Takes effect on the next sync (no
+    #: auto-sync — the ignore-list precedent, phase 89). Server
+    #: default False: every pre-phase-105 row imports exactly as
+    #: before (A4).
+    include_hidden: Mapped[bool] = mapped_column(
+        Boolean, default=False, server_default=text("false"), nullable=False
+    )
     added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
 
 

app/rag/importer.py

@@ -19,9 +19,11 @@ the two-phase upsert:
 
 Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
 component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
-or hidden files) is skipped, plus the well-known exclusion list. A token
-may also name extensionless files by their exact lowercased full filename
-(``Dockerfile`` under the ``dockerfile`` token — phase 102).
+or hidden files) is skipped, plus the well-known exclusion list — UNLESS
+the source's phase-105 hidden-folders flag admits dot-prefixed paths; the
+exclusion list always applies. A token may also name extensionless files
+by their exact lowercased full filename (``Dockerfile`` under the
+``dockerfile`` token — phase 102).
 
 ``prune=True`` deletes documents (of the imported sources only) whose files
 no longer exist **or no longer match the format filter** — this is how
@@ -160,6 +162,20 @@ def _ignore_for_root(
     return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
 
 
+def _include_hidden_for_root(
+    root: Path, include_hidden_by_root: dict[str, bool] | None
+) -> bool:
+    """The per-root hidden-folders flag (phase 105, A1).
+
+    Keyed by ``str(root)`` — the root string exactly as the caller
+    passed it in ``sources`` (the ``_ignore_for_root`` convention,
+    phase 89): ``True`` only for roots the caller lists as True;
+    unlisted/``None`` roots are ``False`` — every existing caller
+    behaves byte-identically (A4).
+    """
+    return bool((include_hidden_by_root or {}).get(str(root), False))
+
+
 def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
     """The bare lowercased token *path* imports under, or ``None``.
 
@@ -183,18 +199,24 @@ def iter_importable_files(
     extensions: frozenset[str],
     excluded: frozenset[str] = EXCLUDED_DIRS,
     ignore: tuple[str, ...] = (),
+    include_hidden: bool = False,
 ) -> list[Path]:
     """All importable files under *root* (sorted), per the A9 scope rules.
 
     *extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
-    Skips: any path with a dot-prefixed component (hidden dirs/files —
-    vendored caches like ``.esphome/.espressif/**``) and the well-known
-    non-content directories in *excluded*. *ignore* (phase 89, A1) is a
+    Skips: when ``include_hidden`` is False (the default), any path with a
+    dot-prefixed component (hidden dirs/files — vendored caches like
+    ``.esphome/.espressif/**``); when True, dot-prefixed components are
+    ADMITTED (files inside hidden folders, and hidden files) and only
+    *excluded* is consulted (A1 — caches/VCS internals are never content).
+    The well-known non-content directories in *excluded* are skipped in
+    BOTH states. *ignore* (phase 89, A1) is a
     tuple of ALREADY-normalized, non-empty source-relative path prefixes
     (the importer's ``_ignore_for_root`` is the normalization choke point
     — raw box lines never reach this function): a file is skipped when its
     source-relative POSIX path starts with any entry; the default ``()``
-    keeps every existing caller byte-identical.
+    keeps every existing caller byte-identical. The *ignore* tuple
+    composes additively in both states.
     """
     if not root.is_dir():
         return []
@@ -203,7 +225,10 @@ def iter_importable_files(
         if not path.is_file():
             continue
         rel = path.relative_to(root)
-        if any(part.startswith(".") or part in excluded for part in rel.parts):
+        if any(
+            (not include_hidden and part.startswith(".")) or part in excluded
+            for part in rel.parts
+        ):
             continue
         if ignore and is_ignored(rel.as_posix(), ignore):
             continue
@@ -222,6 +247,7 @@ async def import_sources(
     session: Session | None = None,
     progress: Callable[[str, str, int, int], None] | None = None,
     ignore_by_root: dict[str, list[str]] | None = None,
+    include_hidden_by_root: dict[str, bool] | None = None,
 ) -> ImportSummary:
     """Import every A9-format file under *sources* (see module docstring).
 
@@ -253,6 +279,15 @@ async def import_sources(
     junk-precedent). ``None`` (the default) changes nothing: the map is read
     per root, unlisted roots get an empty tuple, and every existing caller
     behaves byte-identically.
+
+    ``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the
+    stored flag: ``True`` admits dot-prefixed components for that root
+    (``EXCLUDED_DIRS`` and the extension filter still apply; the ignore
+    tuple composes additively). Unlisted/``None`` roots are ``False`` —
+    byte-identical to pre-phase-105. A file that was indexed with the
+    flag ON and is walked again with it OFF simply never enters
+    ``seen``, so the next ``prune=True`` run deletes its row
+    automatically (A2 — the A9/phase-89 precedent).
     """
     if limit is not None and limit <= 0:
         raise ValueError("limit must be >= 1")
@@ -274,6 +309,9 @@ async def import_sources(
                     root,
                     llm.settings.import_extension_set,
                     ignore=_ignore_for_root(root, ignore_by_root),
+                    include_hidden=_include_hidden_for_root(
+                        root, include_hidden_by_root
+                    ),
                 )
             )
     try:
@@ -286,8 +324,12 @@ async def import_sources(
             source = root.name
             source_names.add(source)
             ignore = _ignore_for_root(root, ignore_by_root)
+            include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
             for path in iter_importable_files(
-                root, llm.settings.import_extension_set, ignore=ignore
+                root,
+                llm.settings.import_extension_set,
+                ignore=ignore,
+                include_hidden=include_hidden,
             ):
                 if limit is not None and summary.files >= limit:
                     break

app/schemas.py

@@ -469,12 +469,16 @@ class GitSourceIn(BaseModel):
     in the API layer, not the schema, so the A4 422 details stay fixed
     strings (the router's credential-safety discipline, applied for
     consistency).
+
+    ``include_hidden`` (phase 105) is optional at create time (absent →
+    stored ``False`` — A4).
     """
 
     kind: Literal["git", "local"] = "git"
     url: str | None = Field(default=None, min_length=1, max_length=500)
     path: str | None = Field(default=None, min_length=1, max_length=2000)
     ignore_paths: list[str] | None = Field(default=None)
+    include_hidden: bool | None = Field(default=None)
 
     @field_validator("url", mode="before")
     @classmethod
@@ -497,12 +501,15 @@ class GitSourceOut(BaseModel):
     reports its path in ``url`` and the full row (``kind`` + ``path``)
     via ``GET``. ``ignore_paths`` (phase 89) is the stored, normalized
     list — non-null (a row created without it reports ``[]``).
+    ``include_hidden`` (phase 105) is the stored flag — a row created
+    without it reports ``False`` (A4).
     """
 
     id: uuid.UUID | None
     url: str
     added_at: datetime | None
     ignore_paths: list[str]
+    include_hidden: bool
 
 
 class GitSourceRow(BaseModel):
@@ -516,7 +523,9 @@ class GitSourceRow(BaseModel):
     location column). ``id`` / ``added_at`` are nullable: env-fallback
     rows (table empty) carry neither. ``ignore_paths`` (phase 89) is the
     row's stored, normalized list — env-fallback rows (no DB row to
-    store a list on) report ``[]``.
+    store a list on) report ``[]``. ``include_hidden`` (phase 105) is
+    the row's stored flag — env-fallback rows (no DB row to store a flag
+    on) report ``False`` (the ``ignore_paths: []`` precedent).
     """
 
     id: uuid.UUID | None
@@ -525,22 +534,25 @@ class GitSourceRow(BaseModel):
     path: str | None
     added_at: datetime | None
     ignore_paths: list[str]
+    include_hidden: bool
 
 
-class GitSourceIgnoreIn(BaseModel):
-    """``PATCH /api/git-sources/{source_id}`` body (phase 89, A5).
+class GitSourcePatchIn(BaseModel):
+    """``PATCH /api/git-sources/{source_id}`` body (phase 89 A5;
+    extended phase 105).
 
-    ``ignore_paths`` is REQUIRED — the box's lines in REPLACE semantics:
-    the body list (normalized + A4-validated in the API layer, with the
-    fixed-detail 422s) becomes the row's whole list — an empty list
-    clears all; an absent field is a 422 with the model's own detail.
-    Entries are RAW box lines: normalization + the A4 limits are
-    enforced in the API layer so the 422 details stay fixed strings
-    (the router's credential-safety discipline, applied for
-    consistency).
+    Each field is independent and OPTIONAL: absent/None leaves the
+    row's value unchanged; PRESENT applies. ``ignore_paths`` when
+    present keeps the phase-89 A5 REPLACE semantics (the body list,
+    normalized + A4-validated, becomes the row's whole list — empty
+    list clears all; every pre-phase-105 client always sends the
+    list, so their behavior is byte-identical). ``include_hidden``
+    (phase 105) when present sets the stored flag. Both absent →
+    200 no-op (the row is untouched).
     """
 
-    ignore_paths: list[str]
+    ignore_paths: list[str] | None = Field(default=None)
+    include_hidden: bool | None = Field(default=None)
 
 
 class GitSourceList(BaseModel):