Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 105_hidden_folders_toggle
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
This commit is contained in:
ducoterra
2026-09-12 21:26:30 -04:00
parent ecc921098a
commit d731169b8b
49 changed files with 3553 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/importer.py
+51 -9
View File
@@ -19,9 +19,11 @@ the two-phase upsert:
Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
or hidden files) is skipped, plus the well-known exclusion list. A token
may also name extensionless files by their exact lowercased full filename
(``Dockerfile`` under the ``dockerfile`` token — phase 102).
or hidden files) is skipped, plus the well-known exclusion list — UNLESS
the source's phase-105 hidden-folders flag admits dot-prefixed paths; the
exclusion list always applies. A token may also name extensionless files
by their exact lowercased full filename (``Dockerfile`` under the
``dockerfile`` token — phase 102).
``prune=True`` deletes documents (of the imported sources only) whose files
no longer exist **or no longer match the format filter** — this is how
@@ -160,6 +162,20 @@ def _ignore_for_root(
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
def _include_hidden_for_root(
root: Path, include_hidden_by_root: dict[str, bool] | None
) -> bool:
"""The per-root hidden-folders flag (phase 105, A1).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (the ``_ignore_for_root`` convention,
phase 89): ``True`` only for roots the caller lists as True;
unlisted/``None`` roots are ``False`` — every existing caller
behaves byte-identically (A4).
"""
return bool((include_hidden_by_root or {}).get(str(root), False))
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
"""The bare lowercased token *path* imports under, or ``None``.
@@ -183,18 +199,24 @@ def iter_importable_files(
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
include_hidden: bool = False,
) -> list[Path]:
"""All importable files under *root* (sorted), per the A9 scope rules.
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
Skips: any path with a dot-prefixed component (hidden dirs/files —
vendored caches like ``.esphome/.espressif/**``) and the well-known
non-content directories in *excluded*. *ignore* (phase 89, A1) is a
Skips: when ``include_hidden`` is False (the default), any path with a
dot-prefixed component (hidden dirs/files — vendored caches like
``.esphome/.espressif/**``); when True, dot-prefixed components are
ADMITTED (files inside hidden folders, and hidden files) and only
*excluded* is consulted (A1 — caches/VCS internals are never content).
The well-known non-content directories in *excluded* are skipped in
BOTH states. *ignore* (phase 89, A1) is a
tuple of ALREADY-normalized, non-empty source-relative path prefixes
(the importer's ``_ignore_for_root`` is the normalization choke point
— raw box lines never reach this function): a file is skipped when its
source-relative POSIX path starts with any entry; the default ``()``
keeps every existing caller byte-identical.
keeps every existing caller byte-identical. The *ignore* tuple
composes additively in both states.
"""
if not root.is_dir():
return []
@@ -203,7 +225,10 @@ def iter_importable_files(
if not path.is_file():
continue
rel = path.relative_to(root)
if any(part.startswith(".") or part in excluded for part in rel.parts):
if any(
(not include_hidden and part.startswith(".")) or part in excluded
for part in rel.parts
):
continue
if ignore and is_ignored(rel.as_posix(), ignore):
continue
@@ -222,6 +247,7 @@ async def import_sources(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring).
@@ -253,6 +279,15 @@ async def import_sources(
junk-precedent). ``None`` (the default) changes nothing: the map is read
per root, unlisted roots get an empty tuple, and every existing caller
behaves byte-identically.
``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the
stored flag: ``True`` admits dot-prefixed components for that root
(``EXCLUDED_DIRS`` and the extension filter still apply; the ignore
tuple composes additively). Unlisted/``None`` roots are ``False`` —
byte-identical to pre-phase-105. A file that was indexed with the
flag ON and is walked again with it OFF simply never enters
``seen``, so the next ``prune=True`` run deletes its row
automatically (A2 — the A9/phase-89 precedent).
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -274,6 +309,9 @@ async def import_sources(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
include_hidden=_include_hidden_for_root(
root, include_hidden_by_root
),
)
)
try:
@@ -286,8 +324,12 @@ async def import_sources(
source = root.name
source_names.add(source)
ignore = _ignore_for_root(root, ignore_by_root)
include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
for path in iter_importable_files(
root, llm.settings.import_extension_set, ignore=ignore
root,
llm.settings.import_extension_set,
ignore=ignore,
include_hidden=include_hidden,
):
if limit is not None and summary.files >= limit:
break