Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 105_hidden_folders_toggle
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
This commit is contained in:
ducoterra
2026-09-12 21:26:30 -04:00
parent ecc921098a
commit d731169b8b
49 changed files with 3553 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/105_hidden_folders_toggle/00_phase.md → .agents/phases/complete/105_hidden_folders_toggle/00_phase.md
View File
.agents/phases/todo/105_hidden_folders_toggle/01_include_hidden_column.md → .agents/phases/complete/105_hidden_folders_toggle/01_include_hidden_column.md
View File
.agents/phases/todo/105_hidden_folders_toggle/02_importer_include_hidden.md → .agents/phases/complete/105_hidden_folders_toggle/02_importer_include_hidden.md
View File
.agents/phases/todo/105_hidden_folders_toggle/03_include_hidden_api.md → .agents/phases/complete/105_hidden_folders_toggle/03_include_hidden_api.md
View File
.agents/phases/todo/105_hidden_folders_toggle/04_include_hidden_pipelines.md → .agents/phases/complete/105_hidden_folders_toggle/04_include_hidden_pipelines.md
View File
.agents/phases/todo/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md → .agents/phases/complete/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md
View File
.agents/phases/todo/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md → .agents/phases/complete/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.md
+16
View File
@@ -0,0 +1,16 @@
All completion criteria verified. Everything is green.
**Phase 105 final verification pass — all criteria verified**
- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)
**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).
**Deviations:** none — no defects found; no code changes were needed on this pass.
**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 97%]
............................................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 117 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 200 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 311 0 100%
--------------------------------------------------
TOTAL 3879 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.md
+18
View File
@@ -0,0 +1,18 @@
All gates green. Task 01 is complete:
**Implemented**
- `app/models.py`: `GitSource.include_hidden` — `BOOLEAN NOT NULL`, `server_default=text("false")`, `default=False`, A1/A4 provenance docstring; module-header `git_sources` inventory updated (phase 105).
- `alembic/versions/0019_git_source_include_hidden.py`: additive reversible column (revises 0018); dev/test DB now at `0019 (head)`.
- Unit pins in `tests/unit/test_models.py`: column contract (type/NOT NULL/server default) + Python-side default resolves to `False`.
- New `tests/integration/test_migration_0019.py` (4 tests): 0018→0019 upgrade contract, pre-0019 backfill `false`, omitted-column server default, explicit-`true` round-trip (SQL + ORM fresh-session), downgrade drops column with rows surviving, round-trip upgrade.
**Results**
- `uv run pytest tests/unit/ -q` → exit 0
- `uv run alembic upgrade head` → `0018 -> 0019` applied, `alembic current` = `0019 (head)`
- `uv run pytest tests/integration/test_migration_0019.py -v --no-cov` → 4 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2108 passed, TOTAL **99%** (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors (fixed one `reportOptionalMemberAccess` in the new test)
**Decisions/deviations**: none — followed the 0013/0018 house migration format and the 0007/0018 migration-test patterns verbatim; no existing suite asserted the full `git_sources` column set, so no expectation lists needed updating.
**Next pending task**: `.agents/phases/todo/105_hidden_folders_toggle/02_importer_include_hidden.md` (importer `iter_importable_files`/`import_sources` flag support).
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 17%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 34%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 99%]
.................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 229 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 115 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 197 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 307 0 100%
--------------------------------------------------
TOTAL 3867 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.md
+20
View File
@@ -0,0 +1,20 @@
All gates green. Task 02 is complete.
## Report
**Implemented (task 02 — importer flag support, phase 105):**
- `app/rag/importer.py::iter_importable_files` — new `include_hidden: bool = False` param; skip check is now `(not include_hidden and part.startswith(".")) or part in excluded` — byte-identical at the default; docstring updated (A1 flag clause, ignore composes additively).
- `app/rag/importer.py::import_sources` — new kw-only `include_hidden_by_root: dict[str, bool] | None` (same `str(root)` keying as `ignore_by_root`); new `_include_hidden_for_root` helper is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop; module docstring "Scope" + `import_sources` docstring updated.
- `tests/unit/test_importer_include_hidden.py` (new, 12 tests): default walk pinned byte-identically (sorted verbatim), flag-ON admits hidden dirs + hidden files, `EXCLUDED_DIRS`/extension filter stay in force, ignore-tuple composition, `_include_hidden_for_root` pins incl. `str(root)` keying.
- `tests/integration/test_importer_include_hidden.py` (new, 5 tests): default run → no hidden rows/embeds; flag ON → hidden md+yaml indexed, yaml summarized (phase-30 path), `.venv` excluded; flag-off re-run prunes 2 hidden docs (A2); progress `total` agrees in both states; unlisted root stays hidden.
- Fixed one pre-existing regression surfaced by the signature change: the phase-89 counting sentinel in `tests/unit/test_importer.py::test_no_progress_means_no_prewalk` now accepts/forwards `include_hidden` (same drop-in pattern as its `ignore` handling).
**Results:**
- `uv run pytest tests/unit/test_importer_include_hidden.py tests/integration/test_importer_include_hidden.py tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -q` → 44 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate)
- `uv run pytest` → **2125 passed**
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
**Decisions:** none — followed the task design verbatim; no deviations.
**Next pending task:** `.agents/phases/todo/105_hidden_folders_toggle/03_include_hidden_api.md`
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 71%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 98%]
..................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 229 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 115 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 200 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 307 0 100%
--------------------------------------------------
TOTAL 3870 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.md
+16
View File
@@ -0,0 +1,16 @@
All green. Task 03 complete.
**Implemented**
- `app/schemas.py`: `include_hidden` added to `GitSourceIn` (optional, absent → `False`), `GitSourceOut` and `GitSourceRow` (env rows report `False`); `GitSourceIgnoreIn` renamed → `GitSourcePatchIn` with two optional present-wins fields
- `app/api/git_sources.py`: GET reports stored flag (DB rows) / `false` (env rows); POST stores `bool(payload.include_hidden)` for both kinds; PATCH now applies each present field independently (list replace keeps phase-89 A5 semantics + fixed 422s, flag set, both-absent → 200 no-op, 404 unchanged); router + endpoint docstrings updated
- `tests/integration/test_git_sources_api.py`: extended — env/row shape pins gain `include_hidden`, anonymous 403 extended to bool-only body, retired `{}`-422 pin replaced with no-op 200, new phase-105 matrix (GET default false, POST both kinds, bool-only, list-only, both, 422-never-half-applies)
**Results**
- `uv run pytest tests/integration/test_git_sources_api.py -q` → 39 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2131 passed, TOTAL 99% (`app/api/git_sources.py` 100%, `app/schemas.py` 100%)
- `uv run ruff check . && uv run pyright` → clean (0 errors, 0 warnings)
- `grep GitSourceIgnoreIn` repo-wide → zero hits (rename verified)
**Decisions** — `{}` PATCH body changed 422→200 no-op per the locked task-03 contract (both fields now optional); existing dialog client (always sends list) stays byte-identical.
**Next pending task** — `105_hidden_folders_toggle/04_include_hidden_pipelines.md`
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 115 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 200 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 311 0 100%
--------------------------------------------------
TOTAL 3877 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.md
+17
View File
@@ -0,0 +1,17 @@
All completion criteria for task 04 are met and every gate is green.
**Report — Task 04 (105_hidden_folders_toggle): pipelines honor the per-row flag**
- `app/api/sync.py::_run_sync` — builds `include_hidden_by_root` in the existing per-row loop (same `str(root)` keying, OR on shared-root collision — the ignore-union's boolean mirror) and passes it to `import_sources`; module docstring step 4 extended.
- `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple on all three return paths (manual `--source` → `{}, {}`; rows → per-root flag map; legacy fallback → `{}, {}`); `main` unpacks and passes `include_hidden_by_root=` to `import_sources`; module + function docstrings extended.
- `scripts/load_test_kb.py` and the upload run untouched (defaults / no scan).
- Tests: `test_sync_api.py` — real-import test (same local row: flag off → hidden file has no `documents` row, `files=1`; flipped on → hidden file indexed, `files=2`) + fake-import wiring test pinning the OR-collision map; `test_import_docs_git.py` — fake-import wiring test (flag True → `{root: True}`; default row → `{root: False}`), two real-import CLI tests (flag on indexes `.hidden/note.md` `added=2`; off skips it `added=1`), and all 8 existing `_resolve_sources`/`main` tests updated to the 3-tuple with flag-map assertions; fakes in `test_sync_button.py`/`test_sync_api.py`/`test_import_docs_git.py` gained the new kwarg.
Results:
- `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q` → 43 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2136 passed, TOTAL **99%** (>90%; `app/api/sync.py` 100%)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
Decisions: flag-off rows contribute an explicit `{root: False}` entry (per the task's unconditional OR-assignment snippet) — pinned by tests in both suites.
Next pending task: `.agents/phases/todo/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md`
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 97%]
................................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 117 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 200 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 311 0 100%
--------------------------------------------------
TOTAL 3879 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.md
+14
View File
@@ -0,0 +1,14 @@
All green. Task 05 complete.
**Report**
- **index.html**: page-level `<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden>` inserted inside `#view-git-sources`, directly after `#git-sources-table-wrap` (L658).
- **git-sources.js**: module docstring gained the phase-105 contract entry; `hiddenErrorEl` grabber; `hideHiddenError()` on `loadSources` success path (healed-load clear); `makeRow` builds the "hidden on" text tag (source cell, iff `s.id && s.include_hidden === true`) and the native labeled Hidden checkbox in the actions cell — DOM order **Hidden · Ignore paths · Remove**, stored rows only (A3), `checked` from server state only, aria-label via `setAttribute`; new `toggleHidden` + `showHiddenError`/`hideHiddenError` with the §7.4 lifecycle (disable → `PATCH {include_hidden}` → 200: clear error → `await loadSources()` → confirm announced LAST; failure: revert + re-enable + page alert line).
- **styles.css**: `.git-source-hidden` (44px hit height), checkbox `accent-color: var(--brand)`, `:disabled` wait state, `.git-source-hidden-count` tag — checked-state contrast verified + recorded in the comment (built-in `--bg` on `--brand` 5.2:1, measured 5.35:1; themed palettes gated by the theme tab's 8-pair AA check, `bg` on `brand` ≥ 4.5:1).
- **tests/unit/test_hidden_folders_toggle.py** (new): 12 source-level pins — row wiring/DOM order, tag text, toggle lifecycle ordering, revert-on-both-failure-branches, healed-load clear, cross-file wire-contract pin (JS body key/render field vs `GitSourcePatchIn`/`GitSourceRow` in `app/schemas.py`), HTML placement/attributes, CSS rules, docstring.
- `uv run pytest tests/unit/test_hidden_folders_toggle.py -q` → 12 passed
- `uv run pytest --cov=app --cov-report=term-missing` → **2148 passed**, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- JS syntax verified via `node --check` (ESM); CSS brace balance 0.
- Deviations: none (task snippet's `credentials: "same-origin"` kept verbatim — it's the browser default).
- **Next pending task**: `.agents/phases/todo/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md`.
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 97%]
............................................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 117 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 200 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 311 0 100%
--------------------------------------------------
TOTAL 3879 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__06_e2e_hidden_folders_toggle.a1.err
View File
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__06_e2e_hidden_folders_toggle.a1.md
+16
View File
@@ -0,0 +1,16 @@
All green. Phase 105's final task is complete — here's the report:
**Task 06 (105_hidden_folders_toggle) — done**
- Implemented `tests/e2e/test_hidden_folders_toggle.py` — 6 Playwright tests mapped 1:1 to the contract: anonymous gate/403s (incl. bool-only toggle PATCH), A4 default-off byte-identity (checkbox unchecked, no tag, `files=1`, no hidden `documents` row — verified via catalog AND direct DB read), A1 toggle-on (real click → tag + last-announce confirmation → sync indexes `.hidden/note.md`, `.venv/junk.md` excluded, box re-renders checked, API round-trip), A2 toggle-off prune (`detail.pruned=1`, catalog clears), A3 env rows have no controls, a11y + error surface (full-value aria-name, real Tab-key reachability, `role=alert` hidden on happy path, network-aborted PATCH → canned message + box revert/re-enable)
- One API adaptation: Playwright 1.62 Python has no `is_focusable`/`to_be_focusable` — used a 60×Tab loop + `to_be_focused` (stronger: proves Tab actually reaches it)
**Results (all in isolation, DB up):**
- `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed**
- Regressions: `test_source_ignore_paths.py` 6 ✓, `test_git_sources_admin.py` 6 ✓, `test_local_directory_sources.py` 3 ✓, `test_sync_button.py` 3 ✓, `test_smoke.py` 3 ✓ (phase-89 canary first)
- `uv run pytest --cov=app --cov-report=term-missing` → **2148 passed, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → clean (0 errors, 0 warnings)
**Decisions:** no defects in tasks 01-05 — all prior work passed untouched. Per executor rules, no `git add`/`commit` — the whole phase (app + alembic 0019 + scripts + frontend + tests + phase files) is left in the working tree for the harness's atomic commit.
**Next pending task:** none in phase 105 (06 was its last) — phase dir is ready for the harness to move to `complete/` and commit.
.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__06_e2e_hidden_folders_toggle.a1.validate
+97
View File
@@ -0,0 +1,97 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 97%]
............................................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 181 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 134 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 117 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 140 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 21 0 100%
app/main.py 66 0 100%
app/models.py 126 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 316 1 99%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/folder_summaries.py 121 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 200 3 98%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 150 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 311 0 100%
--------------------------------------------------
TOTAL 3879 15 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
alembic/versions/0019_git_source_include_hidden.py
+52
View File
@@ -0,0 +1,52 @@
"""git_sources.include_hidden: per-source hidden-folders flag (phase 105)
Revision ID: 0019
Revises: 0018
Create Date: 2026-09-12
Phase 105 (per-source hidden-folders toggle: dot-prefixed paths become
indexable per input — the owner flips a "Hidden" checkbox next to the
"Ignore paths" button on the Sources page): when the flag is ON, the
importer walk (``app.rag.importer.iter_importable_files``) no longer
skips dot-prefixed path components for that source — files inside
hidden folders AND hidden files with an importable extension are
indexed, embedded, and summarized exactly like visible files (A1).
``EXCLUDED_DIRS`` (``.venv``, ``node_modules``, ``.git``, …) stay
excluded in BOTH states, and the extension filter always applies. The
flag takes effect on the NEXT sync (no auto-sync — the ignore-list
precedent, phase 89); a previously indexed hidden file is pruned on
the next sync after the flag flips OFF (A2). One additive, fully
reversible column (A13), no other schema change:
* ``git_sources.include_hidden`` — BOOLEAN NOT NULL with server
default ``false``: every pre-phase-105 row reads ``False`` —
byte-identical import behavior until the owner flips it (A4).
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "0019"
down_revision = "0018"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"git_sources",
sa.Column(
"include_hidden",
sa.Boolean(),
server_default=sa.text("false"),
nullable=False,
),
)
def downgrade() -> None:
# The column is the only 0019 artefact — dropping it leaves
# 0018's schema byte-identical (A13, fully reversible).
op.drop_column("git_sources", "include_hidden")
app/api/git_sources.py
+53 -20
View File
@@ -16,15 +16,20 @@ Routes: ``GET`` (DB rows oldest-first, or the env list with
``from_env: true`` while the table is empty; rows carry ``kind`` +
``path``, git rows — and env rows — report ``path: null``; every row
reports ``ignore_paths`` — DB rows their stored normalized list, env
rows ``[]``, phase 89), ``POST``
rows ``[]``, phase 89 — and ``include_hidden`` — DB rows their stored
flag, env rows ``False``, phase 105), ``POST``
(201, validated create; ``kind`` selects the validation: git → exactly
the phase-35 URL contract, local → an existing absolute directory, else
422 naming the path; both kinds accept ``ignore_paths`` — optional,
absent → ``[]``, stored normalized, phase 89),
``PATCH /{source_id}`` (phase 89, A5 — replace one source's ignore
list: 404 unknown id, the required body list is normalized + A4-
validated with fixed-detail 422s and REPLACES the row's list wholesale
— an empty list clears all; 200 → the ``GitSourceOut`` shape),
absent → ``[]``, stored normalized, phase 89 — and ``include_hidden``
— optional, absent → ``False``, stored as sent, phase 105),
``PATCH /{source_id}`` (phase 89, A5 + phase 105 — the ignore list
(replace) and/or the hidden-folders flag (phase 105) — each optional,
present-wins: 404 unknown id; ``ignore_paths`` when present is
normalized + A4-validated with fixed-detail 422s and REPLACES the
row's list wholesale — an empty list clears all; ``include_hidden``
when present sets the flag; both absent → 200 no-op; 200 → the
``GitSourceOut`` shape),
``POST /upload`` (phase 49, backgrounded in phase
64 task 03, scan deferred in phase 90 — admin archive upload: the
``.tar``/``.tar.gz``/``.tgz``/``.zip`` name/format gate + the 1 MiB-
@@ -116,10 +121,10 @@ from app.rag.source_removal import (
)
from app.rag.sources_meta import bump_sources_version
from app.schemas import (
GitSourceIgnoreIn,
GitSourceIn,
GitSourceList,
GitSourceOut,
GitSourcePatchIn,
GitSourceRow,
UploadAccepted,
)
@@ -208,11 +213,13 @@ def list_git_sources(
same-timestamp inserts) with ``from_env: false`` — each row carries
its ``kind``, its ``ignore_paths`` (phase 89 — the stored,
normalized list; ``or []`` guards a row that predated the column),
and, for local rows, the stored ``path`` (git rows and env rows
report ``path: null``); while the table is empty, the
its ``include_hidden`` (phase 105 — the stored flag), and, for
local rows, the stored ``path`` (git rows and env rows report
``path: null``); while the table is empty, the
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
git-only) with null ``id``/``added_at``, ``ignore_paths: []`` (no
DB row to store a list on), and ``from_env: true``.
git-only) with null ``id``/``added_at``, ``ignore_paths: []`` and
``include_hidden: False`` (no DB row to store a list or a flag on),
and ``from_env: true``.
"""
rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
@@ -229,6 +236,7 @@ def list_git_sources(
path=row.path,
added_at=row.added_at,
ignore_paths=row.ignore_paths or [],
include_hidden=row.include_hidden,
)
for row in rows
],
@@ -237,7 +245,13 @@ def list_git_sources(
return GitSourceList(
sources=[
GitSourceRow(
id=None, kind="git", url=url, path=None, added_at=None, ignore_paths=[]
id=None,
kind="git",
url=url,
path=None,
added_at=None,
ignore_paths=[],
include_hidden=False,
)
for url in get_settings().git_source_list
],
@@ -272,10 +286,18 @@ def create_git_source(
lines are normalized + A4-validated (``_validate_ignore_paths`` —
the fixed-detail 422s) and the normalized list is what is stored and
reported.
``include_hidden`` (phase 105) — optional, both kinds: absent →
stored ``False`` (A4), present → stored as sent; the stored flag is
what is reported.
"""
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
return GitSourceOut(
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
id=row.id,
url=row.url,
added_at=row.added_at,
ignore_paths=row.ignore_paths,
include_hidden=row.include_hidden,
)
@@ -331,6 +353,7 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
url=url,
kind="git",
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
include_hidden=bool(payload.include_hidden),
),
"a git source with this URL already exists",
db,
@@ -364,6 +387,7 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
kind="local",
path=path,
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
include_hidden=bool(payload.include_hidden),
),
f"a local source with this path already exists: {path}",
db,
@@ -373,24 +397,33 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
@router.patch("/{source_id}", response_model=GitSourceOut)
def patch_git_source(
source_id: uuid.UUID,
payload: GitSourceIgnoreIn,
payload: GitSourcePatchIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Replace one source's ignore list (phase 89, A5).
"""Edit one source's ignore list and/or hidden-folders flag.
404 unknown id. The body list (required) is normalized +
A4-validated (fixed 422 details) and REPLACES the row's list
wholesale — an empty list clears all. Returns the updated
row's public shape (id, url, added_at, ignore_paths).
Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
id; each PRESENT body field applies independently —
``ignore_paths`` REPLACES the list (normalized + A4-validated,
fixed 422 details); ``include_hidden`` sets the flag. Both
absent → 200 no-op. Returns the updated row's public shape
(id, url, added_at, ignore_paths, include_hidden).
"""
row = db.get(GitSource, source_id)
if row is None:
raise HTTPException(status_code=404, detail="git source not found")
if payload.ignore_paths is not None:
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
if payload.include_hidden is not None:
row.include_hidden = payload.include_hidden
db.commit()
db.refresh(row)
return GitSourceOut(
id=row.id, url=row.url, added_at=row.added_at, ignore_paths=row.ignore_paths
id=row.id,
url=row.url,
added_at=row.added_at,
ignore_paths=row.ignore_paths,
include_hidden=row.include_hidden,
)
app/api/sync.py
+14 -2
View File
@@ -36,7 +36,9 @@ decisions):
4. ``import_sources(..., prune=True)`` over the single combined list
(git checkouts + local dirs), honoring each row's ``ignore_paths``
(phase 89 — the per-root ignore map is built in the same per-row
loop as the source list) — prune so files deleted upstream, out of
loop as the source list) and its ``include_hidden`` flag (phase 105
— the per-root hidden-folders map, same per-row construction) —
prune so files deleted upstream, out of
a local dir, or newly matching an ignore pattern leave the index
(pruning covers the union; the CLI's no-prune default is unchanged);
5. when the import changed the KB (added + updated > 0),
@@ -231,6 +233,7 @@ async def _run_sync() -> None:
sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
include_hidden_by_root: dict[str, bool] = {}
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
@@ -250,6 +253,14 @@ async def _run_sync() -> None:
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
# Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
# the SAME root string the importer sees; a shared-root
# collision ORs — if EITHER row says "index hidden", the
# root does (the ignore-map union's boolean mirror).
include_hidden_by_root[str(root)] = (
include_hidden_by_root.get(str(root), False)
or bool(row.include_hidden)
)
# Phase 64 (task 02): the per-file progress hook — the status
# endpoint reports the file being processed right now. The
# closure captures the module ``_status`` exactly like the state
@@ -260,7 +271,8 @@ async def _run_sync() -> None:
_status.files_total = total
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
)
overview = False
if summary.added + summary.updated > 0:
app/models.py
+16 -1
View File
@@ -18,7 +18,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
import (phase 35; ``kind`` discriminator added in
phase 38; ``ignore_paths`` (phase 89 — JSONB list
of normalized path prefixes, server default
``'[]'``)).
``'[]'``); ``include_hidden`` (phase 105 — index
dot-prefixed paths from this source, server
default ``false``)).
* ``saved_chats`` — owner-saved chat conversations: one row per
explicitly Saved conversation (auto-``title`` +
the ``bor.chat.v1`` message list as JSONB, phase
@@ -248,6 +250,19 @@ class GitSource(Base):
ignore_paths: Mapped[list] = mapped_column(
JSONB, default=list, server_default=text("'[]'"), nullable=False
)
#: Index hidden (dot-prefixed) paths from this source (phase 105,
#: A1): True → the walk (app.rag.importer.iter_importable_files)
#: does not skip dot-prefixed components — files inside hidden
#: folders AND hidden files with an importable extension are
#: indexed; ``EXCLUDED_DIRS`` (``.venv``, ``node_modules``,
#: ``.git``, …) are excluded in BOTH states, and the extension
#: filter always applies. Takes effect on the next sync (no
#: auto-sync — the ignore-list precedent, phase 89). Server
#: default False: every pre-phase-105 row imports exactly as
#: before (A4).
include_hidden: Mapped[bool] = mapped_column(
Boolean, default=False, server_default=text("false"), nullable=False
)
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
app/rag/importer.py
+51 -9
View File
@@ -19,9 +19,11 @@ the two-phase upsert:
Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
or hidden files) is skipped, plus the well-known exclusion list. A token
may also name extensionless files by their exact lowercased full filename
(``Dockerfile`` under the ``dockerfile`` token — phase 102).
or hidden files) is skipped, plus the well-known exclusion list — UNLESS
the source's phase-105 hidden-folders flag admits dot-prefixed paths; the
exclusion list always applies. A token may also name extensionless files
by their exact lowercased full filename (``Dockerfile`` under the
``dockerfile`` token — phase 102).
``prune=True`` deletes documents (of the imported sources only) whose files
no longer exist **or no longer match the format filter** — this is how
@@ -160,6 +162,20 @@ def _ignore_for_root(
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
def _include_hidden_for_root(
root: Path, include_hidden_by_root: dict[str, bool] | None
) -> bool:
"""The per-root hidden-folders flag (phase 105, A1).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (the ``_ignore_for_root`` convention,
phase 89): ``True`` only for roots the caller lists as True;
unlisted/``None`` roots are ``False`` — every existing caller
behaves byte-identically (A4).
"""
return bool((include_hidden_by_root or {}).get(str(root), False))
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
"""The bare lowercased token *path* imports under, or ``None``.
@@ -183,18 +199,24 @@ def iter_importable_files(
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
include_hidden: bool = False,
) -> list[Path]:
"""All importable files under *root* (sorted), per the A9 scope rules.
*extensions* is a set of lowercased dotted suffixes (``{'.md', '.py'}``).
Skips: any path with a dot-prefixed component (hidden dirs/files —
vendored caches like ``.esphome/.espressif/**``) and the well-known
non-content directories in *excluded*. *ignore* (phase 89, A1) is a
Skips: when ``include_hidden`` is False (the default), any path with a
dot-prefixed component (hidden dirs/files — vendored caches like
``.esphome/.espressif/**``); when True, dot-prefixed components are
ADMITTED (files inside hidden folders, and hidden files) and only
*excluded* is consulted (A1 — caches/VCS internals are never content).
The well-known non-content directories in *excluded* are skipped in
BOTH states. *ignore* (phase 89, A1) is a
tuple of ALREADY-normalized, non-empty source-relative path prefixes
(the importer's ``_ignore_for_root`` is the normalization choke point
— raw box lines never reach this function): a file is skipped when its
source-relative POSIX path starts with any entry; the default ``()``
keeps every existing caller byte-identical.
keeps every existing caller byte-identical. The *ignore* tuple
composes additively in both states.
"""
if not root.is_dir():
return []
@@ -203,7 +225,10 @@ def iter_importable_files(
if not path.is_file():
continue
rel = path.relative_to(root)
if any(part.startswith(".") or part in excluded for part in rel.parts):
if any(
(not include_hidden and part.startswith(".")) or part in excluded
for part in rel.parts
):
continue
if ignore and is_ignored(rel.as_posix(), ignore):
continue
@@ -222,6 +247,7 @@ async def import_sources(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring).
@@ -253,6 +279,15 @@ async def import_sources(
junk-precedent). ``None`` (the default) changes nothing: the map is read
per root, unlisted roots get an empty tuple, and every existing caller
behaves byte-identically.
``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the
stored flag: ``True`` admits dot-prefixed components for that root
(``EXCLUDED_DIRS`` and the extension filter still apply; the ignore
tuple composes additively). Unlisted/``None`` roots are ``False`` —
byte-identical to pre-phase-105. A file that was indexed with the
flag ON and is walked again with it OFF simply never enters
``seen``, so the next ``prune=True`` run deletes its row
automatically (A2 — the A9/phase-89 precedent).
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -274,6 +309,9 @@ async def import_sources(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
include_hidden=_include_hidden_for_root(
root, include_hidden_by_root
),
)
)
try:
@@ -286,8 +324,12 @@ async def import_sources(
source = root.name
source_names.add(source)
ignore = _ignore_for_root(root, ignore_by_root)
include_hidden = _include_hidden_for_root(root, include_hidden_by_root)
for path in iter_importable_files(
root, llm.settings.import_extension_set, ignore=ignore
root,
llm.settings.import_extension_set,
ignore=ignore,
include_hidden=include_hidden,
):
if limit is not None and summary.files >= limit:
break
app/schemas.py
+24 -12
View File
@@ -469,12 +469,16 @@ class GitSourceIn(BaseModel):
in the API layer, not the schema, so the A4 422 details stay fixed
strings (the router's credential-safety discipline, applied for
consistency).
``include_hidden`` (phase 105) is optional at create time (absent →
stored ``False`` — A4).
"""
kind: Literal["git", "local"] = "git"
url: str | None = Field(default=None, min_length=1, max_length=500)
path: str | None = Field(default=None, min_length=1, max_length=2000)
ignore_paths: list[str] | None = Field(default=None)
include_hidden: bool | None = Field(default=None)
@field_validator("url", mode="before")
@classmethod
@@ -497,12 +501,15 @@ class GitSourceOut(BaseModel):
reports its path in ``url`` and the full row (``kind`` + ``path``)
via ``GET``. ``ignore_paths`` (phase 89) is the stored, normalized
list — non-null (a row created without it reports ``[]``).
``include_hidden`` (phase 105) is the stored flag — a row created
without it reports ``False`` (A4).
"""
id: uuid.UUID | None
url: str
added_at: datetime | None
ignore_paths: list[str]
include_hidden: bool
class GitSourceRow(BaseModel):
@@ -516,7 +523,9 @@ class GitSourceRow(BaseModel):
location column). ``id`` / ``added_at`` are nullable: env-fallback
rows (table empty) carry neither. ``ignore_paths`` (phase 89) is the
row's stored, normalized list — env-fallback rows (no DB row to
store a list on) report ``[]``.
store a list on) report ``[]``. ``include_hidden`` (phase 105) is
the row's stored flag — env-fallback rows (no DB row to store a flag
on) report ``False`` (the ``ignore_paths: []`` precedent).
"""
id: uuid.UUID | None
@@ -525,22 +534,25 @@ class GitSourceRow(BaseModel):
path: str | None
added_at: datetime | None
ignore_paths: list[str]
include_hidden: bool
class GitSourceIgnoreIn(BaseModel):
"""``PATCH /api/git-sources/{source_id}`` body (phase 89, A5).
class GitSourcePatchIn(BaseModel):
"""``PATCH /api/git-sources/{source_id}`` body (phase 89 A5;
extended phase 105).
``ignore_paths`` is REQUIRED — the box's lines in REPLACE semantics:
the body list (normalized + A4-validated in the API layer, with the
fixed-detail 422s) becomes the row's whole list — an empty list
clears all; an absent field is a 422 with the model's own detail.
Entries are RAW box lines: normalization + the A4 limits are
enforced in the API layer so the 422 details stay fixed strings
(the router's credential-safety discipline, applied for
consistency).
Each field is independent and OPTIONAL: absent/None leaves the
row's value unchanged; PRESENT applies. ``ignore_paths`` when
present keeps the phase-89 A5 REPLACE semantics (the body list,
normalized + A4-validated, becomes the row's whole list — empty
list clears all; every pre-phase-105 client always sends the
list, so their behavior is byte-identical). ``include_hidden``
(phase 105) when present sets the stored flag. Both absent →
200 no-op (the row is untouched).
"""
ignore_paths: list[str]
ignore_paths: list[str] | None = Field(default=None)
include_hidden: bool | None = Field(default=None)
class GitSourceList(BaseModel):
frontend/assets/git-sources.js
+119
View File
@@ -178,6 +178,31 @@
* tag next to the location <code> (text + a distinct background —
* never color alone, WCAG 1.4.1).
*
* Phase 105 (task 05) — the per-row "Hidden" toggle: every STORED
* row (s.id truthy — env-fallback rows get NO checkbox, A3: nothing
* is stored to flag) gets a native labeled checkbox (visible
* "Hidden" text + the checkbox's own aria-label naming the full
* source value — setAttribute, never innerHTML) in the actions
* cell, LEFT of the "Ignore paths" button (Remove stays last).
* Checked state comes ONLY from the server row (s.include_hidden —
* makeRow renders server state, never a prior local flip). When on,
* the source cell shows a "hidden on" text tag (text + background,
* never color alone — WCAG 1.4.1). Flipping runs the §7.4
* never-stale lifecycle (toggleHidden): the box disables at once
* (no double-flip), PATCH /api/git-sources/{id} with ONLY
* { include_hidden } (task 03's optional field — the row's ignore
* list is untouched); 200 → the error line clears, loadSources()
* (the row re-renders from the server, the "hidden on" tag lands)
* and the confirmation is the LAST announcement (the reload's "N
* sources listed." lands first — the phase-89 order); failure →
* the box REVERTS to the server state and re-enables, and the
* detail lands in the page-level #git-sources-hidden-error line
* (role=alert — the checkbox is a table-cell control, so its error
* is at page level; the ignore dialog carries its own INSIDE the
* modal). A healed load clears the line (loadSources' success path
* calls hideHiddenError — the phase-89 "happy path heals the error
* state" precedent).
*
* Scope boundary (phase locked decisions): adding a git repo does
* NOT clone — the sync service (server-side) does that. Removing a
* source, however, performs the FULL cleanup server-side (phase 69):
@@ -253,6 +278,10 @@ export async function mount(root) {
const ignoreErrorEl = root.querySelector("#ignore-editor-error");
const ignoreCancelBtn = root.querySelector("#ignore-editor-cancel");
const ignoreSaveBtn = root.querySelector("#ignore-editor-save");
/* Phase 105: the per-row "Hidden" toggle's page-level error line
(the checkbox is a table-cell control — no dialog of its own —
so its failure announces here, next to the table). */
const hiddenErrorEl = root.querySelector("#git-sources-hidden-error");
/* Polite live region: the screen-reader confirmation for loads, adds,
and removals (the phase-15 announcer pattern). */
@@ -312,6 +341,10 @@ export async function mount(root) {
return;
}
hideLoadError();
/* Phase 105: a healed list clears the stale hidden-toggle error
line (the phase-89 "happy path heals the error state" — the
same hideLoadError precedent). */
hideHiddenError();
const sources = Array.isArray(data.sources) ? data.sources : [];
renderSources(sources, data.from_env === true);
announce(`${sources.length} source${sources.length === 1 ? "" : "s"} listed.`);
@@ -381,6 +414,16 @@ export async function mount(root) {
count.textContent = `${s.ignore_paths.length} ignored`;
urlTd.append(count);
}
/* Phase 105 (A5): the "hidden on" state tag — the
.git-source-ignore-count idiom (TEXT + background, never
color alone — WCAG 1.4.1), so the flag is readable at a
glance without hovering the checkbox. */
if (s.id && s.include_hidden === true) {
const hiddenTag = document.createElement("span");
hiddenTag.className = "git-source-hidden-count";
hiddenTag.textContent = "hidden on";
urlTd.append(hiddenTag);
}
tr.appendChild(urlTd);
const addedTd = document.createElement("td");
@@ -390,6 +433,29 @@ export async function mount(root) {
const actTd = document.createElement("td");
actTd.className = "git-source-actions-cell";
if (s.id) {
/* Phase 105 (A5): the per-row hidden-folders toggle — a native
labeled checkbox (the WCAG focus/label idiom) LEFT of the
"Ignore paths" button; Remove stays last. Stored rows only
(A3 — env-fallback rows fall through to the "from .env"
tag). The aria-label is the ONLY place `value` appears
(setAttribute — never innerHTML). Checked state comes from
the SERVER row (s.include_hidden), never from a prior local
flip (§7.4 — makeRow only ever renders server state). */
const hiddenLabel = document.createElement("label");
hiddenLabel.className = "git-source-hidden";
hiddenLabel.title =
"When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.";
const hiddenBox = document.createElement("input");
hiddenBox.type = "checkbox";
hiddenBox.className = "git-source-hidden-box";
hiddenBox.checked = s.include_hidden === true;
hiddenBox.setAttribute(
"aria-label",
`Index hidden folders for ${kindLabel} source: ${value}`,
);
hiddenLabel.append(hiddenBox, document.createTextNode("Hidden"));
hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox));
actTd.appendChild(hiddenLabel);
/* Phase 89: the per-row ignore-paths editor trigger (A3: stored
rows only — env-fallback rows fall through to the "from
.env" tag below). The aria-label is the ONLY place `value`
@@ -747,6 +813,59 @@ export async function mount(root) {
}
}
/* ---------- hidden folders (PATCH /api/git-sources/{id}) — the per-row toggle ----------
* Phase 105: the hidden-folders toggle — PATCH { include_hidden }
* only (the row's list is untouched — the PATCH body's optional
* fields, task 03). §7.4 never-stale: the box disables at once
* (no double-flip while the PATCH is out); on 200 the row
* re-renders from the server (loadSources) and the CONFIRMATION is
* the LAST announcement (the reload's "N sources listed." lands
* first — the phase-89 order); on failure the box REVERTS to the
* server state and the detail lands in #git-sources-hidden-error
* (role=alert). The checkbox is a table-cell control (no dialog of
* its own — the ignore editor carries its error INSIDE the modal),
* so this toggle's error is the PAGE-level line; a healed
* loadSources clears it (hideHiddenError on the success path). */
function toggleHidden(s, box) {
const value = s.kind === "local" ? (s.path ?? s.url) : s.url;
const wanted = box.checked;
box.disabled = true; // a PATCH is out — the box must not flip twice
fetch(`/api/git-sources/${s.id}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
credentials: "same-origin",
body: JSON.stringify({ include_hidden: wanted }),
})
.then(async (r) => {
if (r.ok) {
hideHiddenError();
await loadSources();
announce(`Hidden folders ${wanted ? "enabled" : "disabled"} for ${value}.`);
return;
}
const detail = await apiDetail(
r, `Could not update the hidden-folders setting (${r.status}).`,
);
showHiddenError(detail);
box.checked = s.include_hidden === true; // revert to server state
box.disabled = false;
})
.catch(() => {
showHiddenError("Could not reach the server — the setting was not changed.");
box.checked = s.include_hidden === true;
box.disabled = false;
});
}
function showHiddenError(message) {
if (hiddenErrorEl) hiddenErrorEl.textContent = message;
if (hiddenErrorEl) hiddenErrorEl.hidden = false;
}
function hideHiddenError() {
if (hiddenErrorEl) hiddenErrorEl.textContent = "";
if (hiddenErrorEl) hiddenErrorEl.hidden = true;
}
/* The dialog's own buttons (static markup — wired once). */
if (ignoreCancelBtn) ignoreCancelBtn.addEventListener("click", cancelIgnoreEditor);
if (ignoreBackdrop) ignoreBackdrop.addEventListener("click", cancelIgnoreEditor);
frontend/assets/styles.css
+50
View File
@@ -2729,6 +2729,56 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
white-space: nowrap;
}
/* Phase 105 (A5): the per-row "Hidden" checkbox — the actions
cell, LEFT of the "Ignore paths" button (JS builds it,
git-sources.js makeRow). A native labeled checkbox: the
visible "Hidden" text + the checkbox's own aria-label
(full source value). ~44px hit height matches the action
buttons; :focus-visible via the GLOBAL rule (L146) — no
per-control ring needed. */
.git-source-hidden {
display: inline-flex;
align-items: center;
gap: 0.4rem;
height: 44px;
padding: 0 0.5rem;
font-size: 0.8rem;
color: var(--ink);
cursor: pointer;
user-select: none;
white-space: nowrap;
}
/* --brand checkbox fill: the native check renders in --bg on
--brand — VERIFIED (phase 105): built-in dark theme
#0f0a0a on #f43f5e = 5.2:1 (the house brand pairing;
measured 5.35:1 — WCAG AA ≥4.5:1); every theme-tab-saved
palette is gated by the tab's eight-pair contrast check
(theme.js PAIRS — "bg" on "brand" at AA_MIN 4.5), so the
checked-state ratio stays AA in all themes. */
.git-source-hidden input[type="checkbox"] {
width: 1.05rem;
height: 1.05rem;
margin: 0;
accent-color: var(--brand);
cursor: pointer;
}
.git-source-hidden:disabled { opacity: 0.5; cursor: wait; }
/* Phase 105 (A5): the "hidden on" state tag — a copy of the
.git-source-ignore-count idiom (TEXT + background, never
color alone — WCAG 1.4.1: --ink on --bg 16.7:1). */
.git-source-hidden-count {
display: inline-block;
margin-left: 0.55rem;
padding: 0.08rem 0.5rem;
border: 1px solid var(--line);
border-radius: 999px;
font-size: 0.72rem;
font-weight: 600;
color: var(--ink);
background: var(--bg);
white-space: nowrap;
}
/* Env-fallback rows carry no Remove (nothing is stored to remove) —
the tag says where the row comes from (brand pair, 6.9:1). */
.git-source-env-tag {
frontend/index.html
+6
View File
@@ -651,6 +651,12 @@
</table>
</div>
<!-- Phase 105: the per-row "Hidden" toggle's error line — the
checkbox is a table-cell control (no dialog of its own), so
its failure announces here (role=alert; git-sources.js
showHiddenError). Hidden until a PATCH fails. -->
<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>
<!-- Empty state — no stored rows AND no env fallback. With
from_env, the env note above already explains where the
active list comes from. -->
scripts/import_docs.py
+37 -13
View File
@@ -29,7 +29,11 @@ precedence order:
Phase 89: resolution also returns each row's ignore paths, keyed by
the resolved root string (the importer normalizes them); manual
``--source`` dirs and the legacy fallback have no rows, so they import
with no ignore.
with no ignore. Phase 105 extends the same resolution with each row's
``include_hidden`` flag — a second per-root map keyed by the same root
strings (the importer reads it per root); manual ``--source`` dirs and
the legacy fallback have no rows, so they import with the empty map
(hidden paths skipped — A4).
Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by
default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet
@@ -163,9 +167,10 @@ def repo_name(url: str) -> str:
def _resolve_sources(
cli_sources: list[Path] | None, settings: Settings
) -> tuple[list[Path], dict[str, list[str]]]:
) -> tuple[list[Path], dict[str, list[str]], dict[str, bool]]:
"""Resolve the directories to import (phase 28, extended in phases
35 and 38; per-root ignore maps, phase 89).
35 and 38; per-root ignore maps, phase 89; per-root hidden-folders
flag maps, phase 105).
Precedence: ``--source`` (explicit manual paths — always wins) >
the effective sources — the ``git_sources`` DB rows (git + local),
@@ -178,18 +183,23 @@ def _resolve_sources(
stored directory, re-verified ``.is_dir()`` at run time) > the
legacy ``DEFAULT_SOURCES``.
Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed
by the resolved root string, exactly as the importer sees it (two
rows sharing a root string get the union — extend, not replace);
manual ``--source`` dirs and the legacy fallback have no rows, so
they import with an empty map (no ignore).
Returns ``(sources, ignore_by_root, include_hidden_by_root)``
(phase 89; phase 105 adds the per-root flag map — the flag is
stored per row, manual ``--source`` dirs and the legacy fallback
have no rows and import with the empty map: hidden paths skipped,
A4): both maps are keyed by the resolved root string, exactly as
the importer sees it (two rows sharing a root string get the union
— extend, not replace — for the ignore lists, and the OR of their
flags for the hidden map); manual ``--source`` dirs and the legacy
fallback have no rows, so they import with empty maps (no ignore,
hidden skipped).
A :class:`GitSyncError` from a failing clone/pull — or a missing
local directory (``local source missing: <path>``) — propagates to
:func:`main`, which aborts the run before importing anything.
"""
if cli_sources:
return [path.expanduser() for path in cli_sources], {}
return [path.expanduser() for path in cli_sources], {}, {}
db = SessionLocal()
try:
rows, origin = effective_sources(db)
@@ -204,6 +214,7 @@ def _resolve_sources(
sources_root = Path(settings.sources_dir).expanduser()
sources: list[Path] = []
ignore_by_root: dict[str, list[str]] = {}
include_hidden_by_root: dict[str, bool] = {}
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
@@ -223,8 +234,16 @@ def _resolve_sources(
# edge.
if row.ignore_paths:
ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths)
return sources, ignore_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}
# Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
# the SAME root string the importer sees; a shared-root
# collision ORs — if EITHER row says "index hidden", the
# root does (the ignore-map union's boolean mirror).
include_hidden_by_root[str(root)] = (
include_hidden_by_root.get(str(root), False)
or bool(row.include_hidden)
)
return sources, ignore_by_root, include_hidden_by_root
return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}
def _overview_row_exists() -> bool:
@@ -264,9 +283,13 @@ def main(argv: list[str] | None = None) -> int:
# Git sources resolve (and clone/pull) *before* any import: a failing
# repo aborts the run with a non-zero exit, naming the failure — a bad
# URL must never silently import partial junk. The second element is
# the phase-89 per-root ignore map (empty for manual/fallback paths).
# the phase-89 per-root ignore map and the third the phase-105
# per-root hidden-folders flag map (both empty for manual/fallback
# paths).
try:
sources, ignore_by_root = _resolve_sources(args.source, settings)
sources, ignore_by_root, include_hidden_by_root = _resolve_sources(
args.source, settings
)
except GitSyncError as e:
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
return 1
@@ -332,6 +355,7 @@ def main(argv: list[str] | None = None) -> int:
summary = await import_sources(
sources, llm, prune=args.prune, limit=args.limit,
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
)
if args.limit is not None:
# An incomplete walk is debug-only — it must never advance
tests/e2e/test_hidden_folders_toggle.py
+676
View File
@@ -0,0 +1,676 @@
"""Phase 105 story E2E (Playwright): the per-source hidden-folders toggle —
the "Hidden" checkbox next to the "Ignore paths" button makes a source's
dot-prefixed paths indexable (``/git-sources.html``).
Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-14 —
TODO-derived, no separate user-story file).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov
The story gate proves the whole item end to end through the REAL page +
REAL API + REAL sync pipeline (the in-app Sync's background task, mock
LLM). **No git, no network** — the suite's single source is a local
directory row (``kind="local"``, phase 38) over a fixture tree with one
visible file, one hidden-folder file (the phase-105 subject), and one
``.venv`` file (``EXCLUDED_DIRS`` — never indexed in either state, A1).
The module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL
(the ``test_source_ignore_paths.py`` module-env pattern) that is NEVER
synced or cloned: every sync in this suite runs with the local DB row in
place (DB rows win over the env fallback), and the one env-fallback test
never triggers a sync.
Contract under test:
* anonymous: the ``#git-sources-gate`` sign-in gate, the manager hidden,
NO ``/api/git-sources`` call on load, 403 on
``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}``
with a bool-only body (the phase-89 anonymous pin extended to the
toggle payload);
* A4: with the default row, a sync indexes ``visible.md`` ONLY —
``detail.files`` counts one, ``.hidden/note.md`` has no ``documents``
row, the checkbox renders UNCHECKED and no "hidden on" tag;
* A1: flipping the checkbox on (the real click) → the PATCH 200 lands
(the "hidden on" tag appears, the announcer ``role=status`` fires the
confirmation AFTER the reload line) → sync → ``.hidden/note.md`` IS
indexed (``documents`` row present; the KB catalog lists it — the
tree/catalog is DB-driven, no extra surface) and ``.venv/junk.md`` is
STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders
CHECKED (server state);
* A2: flipping it OFF (real click) → sync → ``detail.pruned`` includes
the hidden doc, the catalog no longer lists it, the tag is gone;
* A3: the env-fallback view (table-empty state) renders the "from .env"
tag with NO Hidden checkbox and NO "Ignore paths" button;
* a11y + error surface: the checkbox has a full accessible name
containing the source location (``Index hidden folders for local
source: …``), is keyboard-focusable, the tag text is "hidden on"
(never color alone); ``#git-sources-hidden-error`` exists with
``role="alert"`` and stays ``hidden`` through the happy path — and a
network-aborted PATCH shows the canned "not changed" message and
reverts the box to the server state (§7.4 never-stale).
Test → contract mapping (one test per bullet, the phase-89 suite's
shape):
1. ``test_anonymous_gate_and_403s``
2. ``test_hidden_off_by_default``
3. ``test_toggle_on_indexes_hidden_folders``
4. ``test_toggle_off_prunes_hidden``
5. ``test_env_fallback_rows_have_no_toggle``
6. ``test_toggle_a11y_and_error_surface``
"""
from __future__ import annotations
import os
import subprocess
import sys
import time
import uuid
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_HIDDEN", "8141"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is
#: NEVER cloned (the env-fallback test reads it; the sync tests all run
#: with the local DB row in place, so the env list never reaches a
#: clone).
ENV_SOURCE = "https://github.com/reese/env-alpha.git"
#: The three fixture files (source-relative POSIX paths — exactly the
#: strings ``documents.path`` stores).
VISIBLE_MD = "visible.md"
HIDDEN_NOTE = ".hidden/note.md"
VENV_JUNK = ".venv/junk.md"
#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the
#: test_sync_button.py polling idiom) — real import of ≤2 small files
#: against the mock LLM; generous budget.
SYNC_TIMEOUT_S = 60.0
SYNC_TICK_S = 2.0
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The local source's on-disk tree (task 06 step 1).
Module-scoped (the same reasoning as ``test_sync_button.py``'s
module-scoped fixture note): ``tmp_path`` is function-scoped while
the module-scoped app + the seeded row reference the dir for the
module's lifetime, so it is built under ``tmp_path_factory`` (the
same pytest-managed temp area, module-safe). Plain files — no git:
a ``kind="local"`` row is walked directly by the sync (phase 38).
"""
root = tmp_path_factory.mktemp("hidden_src")
(root / ".hidden").mkdir()
(root / ".venv").mkdir()
(root / VISIBLE_MD).write_text(
"# Visible\n"
"\n"
"The visible doc — indexed in EVERY state (flag on and off).\n",
encoding="utf-8",
)
# The phase-105 subject: a file INSIDE a hidden (dot-prefixed)
# folder — indexed only when the row's flag is on (A1).
(root / HIDDEN_NOTE).write_text(
"# Hidden note\n"
"\n"
"A file inside a hidden (dot) folder — the toggle's subject:\n"
"indexed, embedded, and summarized like any visible file when\n"
"the flag is on; pruned from the KB when it flips off (A2).\n",
encoding="utf-8",
)
# EXCLUDED_DIRS content: ``.venv`` is a cache, never content —
# skipped in BOTH flag states (A1).
(root / VENV_JUNK).write_text(
"venv junk — EXCLUDED_DIRS content: never indexed, both states.\n",
encoding="utf-8",
)
return root
@pytest.fixture(scope="module")
def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]:
"""The real app under test — per-module env (the conftest pattern,
module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env
fallback's subject — never cloned), module-scratch checkouts/
upload dirs, the mock LLM. No git anywhere in this suite's path."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env
# shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The empty-table env fallback (only read while ``git_sources`` is
# EMPTY — the env-fallback test; never cloned in this suite).
env["BOR_GIT_SOURCES"] = ENV_SOURCE
# Module-scratch dirs (never reached by this suite's local-row
# syncs — kept explicit so a shared checkouts dir cannot leak rows
# into the walk).
scratch = source_dir.parent
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern, the
test_sync_button.py module-env DSN): the sync's counts and the
catalog must be each test's own doing."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources"
)
)
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""This suite owns ``git_sources`` AND the KB tables (the E2E
isolation pattern): suites run in isolation but share one
Postgres, and a leftover row would flip the app from the
``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover
document would skew the prune counts). Empty BOTH before and
after every test."""
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str:
"""(Re)seed the stored row for the module fixture dir via the REAL
admin API (201) — ``kind="local"`` — with NO ``ignore_paths`` and
NO ``include_hidden`` (A4: absent at create → stored ``False``) —
then reload the view so the page's table re-fetches the row (the
API seed is server-side: the mounted view does not know about it
without a re-show). Returns the stored (expanded) path — the
row's ``url`` location value, the same string the page renders
and aria-labels."""
r = page.request.post(
f"{app_url}/api/git-sources",
data={"kind": "local", "path": str(source_dir)},
)
assert r.status == 201, r.text
body = r.json()
assert body["ignore_paths"] == [] # the default list, round-tripped
assert body["include_hidden"] is False # A4: default off
page.reload()
_admin_git_sources_page(page, app_url)
return body["url"]
def run_sync(page: Page, app_url: str) -> dict[str, Any]:
"""``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal
state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s
budget). Returns the terminal status body."""
r = page.request.post(f"{app_url}/api/sync")
assert r.status == 202, r.text
deadline = time.monotonic() + SYNC_TIMEOUT_S
body: dict[str, Any] = {}
while True:
s = page.request.get(f"{app_url}/api/sync/status")
assert s.status == 200, s.text
body = s.json()
if body["state"] in ("success", "failed"):
return body
assert time.monotonic() < deadline, (
f"sync did not reach a terminal state: {body}"
)
time.sleep(SYNC_TICK_S)
def _catalog_paths(page: Page, app_url: str) -> list[str]:
"""The RAG catalog's data source (``GET /api/docs`` — the Sources
page's table): every ``documents.path`` (source-relative POSIX).
The tree/catalog is DB-driven — a newly indexed hidden document
appears here automatically (no extra surface)."""
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200, r.text
return [d["path"] for d in r.json()["documents"]]
def _db_document_paths() -> list[str]:
"""Every ``documents.path`` straight from the DB — the literal
"no ``documents`` row" assertion (the catalog is the same table,
read through the API)."""
with SessionLocal() as db:
rows = db.execute(text("SELECT path FROM documents")).fetchall()
return [row[0] for row in rows]
def _row(page: Page, value: str) -> Any:
"""The table row whose mono location cell shows ``value`` (re-resolved
on every call — the row re-renders from the server after each
successful toggle)."""
return page.locator("#git-sources-tbody tr", has_text=value)
def _hidden_box(page: Page, value: str) -> Any:
"""The row's Hidden checkbox (fresh locator — the element is
re-created by the post-toggle re-render)."""
return _row(page, value).locator(".git-source-hidden-box")
def _flip_hidden(page: Page, value: str, wanted_on: bool) -> None:
"""Click the row's Hidden checkbox and wait for the §7.4 success
lifecycle to settle: the PATCH 200 landed, the row re-rendered
from the server, and the CONFIRMATION is the last announcer message
(the reload's "N sources listed." cannot overwrite it — the
phase-89 order)."""
_hidden_box(page, value).click()
tag = _row(page, value).locator(".git-source-hidden-count")
if wanted_on:
expect(tag).to_have_text("hidden on", timeout=30_000)
else:
expect(tag).to_have_count(0, timeout=30_000)
expect(
page.locator("#git-sources-announcer")
).to_have_text(
f"Hidden folders {'enabled' if wanted_on else 'disabled'} for {value}."
)
# ---------------------------------------------------------------------------
# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. the
# bool-only toggle PATCH)
# ---------------------------------------------------------------------------
def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None:
page.set_default_timeout(30_000)
# Track every /api/git-sources request the page itself makes — the
# gate must be reached WITHOUT touching the admin API (the
# test_source_ignore_paths.py pattern).
api_calls: list[str] = []
page.on(
"request",
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
)
page.goto(app_url + GIT_SOURCES_URL)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
# The sign-in gate (the #sources-gate pattern, phase 16)…
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
expect(
gate.locator("a[href='/login.html?next=/git-sources.html']")
).to_have_count(1)
# …and the manager is absent/inert: table + form + env note, all
# inside the hidden #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers on GET/POST AND the
# phase-105 toggle PATCH with a bool-only body (the row's list is
# untouched — the toggle's exact payload is gated too).
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
f"{app_url}/api/git-sources", data={"url": ENV_SOURCE}
).status
== 403
)
assert (
page.request.patch(
f"{app_url}/api/git-sources/{uuid.uuid4()}",
data={"include_hidden": True},
).status
== 403
)
# ---------------------------------------------------------------------------
# 2. A4: the default row is byte-identical to pre-phase-105 — the sync
# indexes visible.md ONLY; the checkbox renders UNCHECKED, no tag
# ---------------------------------------------------------------------------
def test_hidden_off_by_default(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# A4: the default row renders the NEW control in its default state —
# the checkbox UNCHECKED (checked comes only from server state) and
# NO "hidden on" tag (pre-phase-105 surface + the control).
box = _hidden_box(page, stored_path)
expect(box).to_have_count(1)
expect(box).not_to_be_checked()
expect(_row(page, stored_path).locator(".git-source-hidden-count")).to_have_count(0)
# The real sync: ``visible.md`` ONLY — the walk skipped the
# dot-prefixed ``.hidden/`` component (A4 byte-identical default)
# AND ``.venv/`` (EXCLUDED_DIRS in both states).
sync = run_sync(page, app_url)
assert sync["state"] == "success", sync
detail = sync["detail"]
assert detail["files"] == 1, detail
assert detail["added"] == 1, detail
assert detail["pruned"] == 0, detail
assert detail["errors"] == 0, detail
# The catalog holds exactly the visible file — no ``documents`` row
# for the hidden note (the direct DB read makes "no row" literal)…
paths = _catalog_paths(page, app_url)
assert VISIBLE_MD in paths, f"visible.md missing from the catalog: {paths}"
assert HIDDEN_NOTE not in paths, (
f"hidden note leaked into the catalog with the flag off: {paths}"
)
assert VENV_JUNK not in paths, f"EXCLUDED_DIRS leaked into the catalog: {paths}"
assert _db_document_paths() == [VISIBLE_MD]
# ---------------------------------------------------------------------------
# 3. A1: the real click flips the flag on — the PATCH 200 lands (tag +
# last-announce confirmation), the next sync indexes the hidden
# folder; EXCLUDED_DIRS stays excluded; the box re-renders CHECKED
# ---------------------------------------------------------------------------
def test_toggle_on_indexes_hidden_folders(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# Baseline: the default sync indexes the visible file only.
first = run_sync(page, app_url)
assert first["state"] == "success", first
assert first["detail"]["files"] == 1, first["detail"]
# A1: flip the checkbox on (the real click) — the §7.4 lifecycle:
# PATCH 200 → the "hidden on" tag appears in the source cell and
# the confirmation is the LAST announcer message (the reload's
# "1 source listed." landed first).
_flip_hidden(page, stored_path, wanted_on=True)
# The checkbox re-renders CHECKED from the server state (never a
# local flip) and the API round-trips include_hidden: true.
expect(_hidden_box(page, stored_path)).to_be_checked()
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
sources = r.json()["sources"]
assert [s["include_hidden"] for s in sources] == [True]
assert [s["ignore_paths"] for s in sources] == [[]] # the list untouched
# The NEXT sync indexes the hidden-folder file like any visible
# file (embedded + chunked by the mock LLM)…
second = run_sync(page, app_url)
assert second["state"] == "success", second
detail = second["detail"]
assert detail["files"] == 2, detail
assert detail["added"] == 1, detail # the hidden note only
assert detail["unchanged"] == 1, detail # visible.md
# …and ``.venv/junk.md`` is STILL absent — EXCLUDED_DIRS content is
# never indexed, in either state (A1).
paths = _catalog_paths(page, app_url)
assert HIDDEN_NOTE in paths, (
f"hidden note missing from the catalog after the on-sync: {paths}"
)
assert VISIBLE_MD in paths, paths
assert VENV_JUNK not in paths, (
f"EXCLUDED_DIRS leaked into the catalog with the flag on: {paths}"
)
assert set(_db_document_paths()) == {VISIBLE_MD, HIDDEN_NOTE}
# ---------------------------------------------------------------------------
# 4. A2: flipping the flag OFF prunes the previously indexed hidden doc
# from the KB on the next sync
# ---------------------------------------------------------------------------
def test_toggle_off_prunes_hidden(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# On + sync: the hidden note is in the KB.
_flip_hidden(page, stored_path, wanted_on=True)
first = run_sync(page, app_url)
assert first["state"] == "success", first
assert first["detail"]["files"] == 2, first["detail"]
assert first["detail"]["added"] == 2, first["detail"]
assert HIDDEN_NOTE in _catalog_paths(page, app_url)
# A2: flip it OFF (the real click) — the previously indexed hidden
# file leaves the KB on the next sync (the seen-set prune — the
# phase-89 A2 / A9 precedent; the tag is gone).
_flip_hidden(page, stored_path, wanted_on=False)
r = page.request.get(f"{app_url}/api/git-sources")
assert [s["include_hidden"] for s in r.json()["sources"]] == [False]
second = run_sync(page, app_url)
assert second["state"] == "success", second
detail = second["detail"]
assert detail["pruned"] == 1, detail # the hidden doc
assert detail["files"] == 1, detail # the walk is back to visible-only
paths = _catalog_paths(page, app_url)
assert HIDDEN_NOTE not in paths, (
f"the hidden doc survived the flag-off prune: {paths}"
)
assert VISIBLE_MD in paths, f"visible.md must survive the prune: {paths}"
assert _db_document_paths() == [VISIBLE_MD]
# ---------------------------------------------------------------------------
# 5. A3: env-fallback rows have no toggle (no DB row to store a flag on)
# ---------------------------------------------------------------------------
def test_env_fallback_rows_have_no_toggle(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# The autouse fixture truncated git_sources — the table is EMPTY,
# so the module app's BOR_GIT_SOURCES URL is the effective list
# (the test_source_ignore_paths.py module-env pattern).
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
row = page.locator("#git-sources-tbody tr")
expect(row).to_contain_text(ENV_SOURCE)
# The env-fallback surface: the "from .env" tag (never color
# alone)…
expect(row.locator(".git-source-env-tag")).to_have_count(1)
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
# …and A3: NO per-row controls at all — neither Remove, nor the
# phase-89 "Ignore paths" button, nor the phase-105 Hidden checkbox
# (and no "hidden on" tag).
expect(row.locator(".git-source-remove")).to_have_count(0)
expect(row.locator(".git-source-ignore")).to_have_count(0)
expect(row.locator(".git-source-hidden")).to_have_count(0)
expect(row.locator(".git-source-hidden-box")).to_have_count(0)
expect(row.locator(".git-source-hidden-count")).to_have_count(0)
# The env-fallback note explains the active list's origin…
expect(page.locator("#git-sources-env-note")).to_be_visible()
# …and the API agrees: from_env true, null ids, the env URL, an
# empty ignore list, and include_hidden False (no DB row to store a
# list or a flag on — the GitSourceRow contract, task 03).
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert body["from_env"] is True
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE]
assert all(
s["id"] is None and s["ignore_paths"] == [] and s["include_hidden"] is False
for s in body["sources"]
)
# ---------------------------------------------------------------------------
# 6. a11y + the error surface: the accessible name, keyboard focus, the
# text tag, and the role=alert line (hidden on the happy path, the
# canned message + revert on a network failure)
# ---------------------------------------------------------------------------
def test_toggle_a11y_and_error_surface(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
box = _hidden_box(page, stored_path)
expect(box).to_have_count(1)
# The full accessible name contains the source location (the
# aria-label — the ONLY place the value appears, setAttribute,
# never innerHTML)…
expect(box).to_have_attribute(
"aria-label", f"Index hidden folders for local source: {stored_path}"
)
expect(
page.get_by_role(
"checkbox", name=f"Index hidden folders for local source: {stored_path}"
)
).to_have_count(1)
# …it is keyboard-reachable — a plain Tab sequence from the page
# top reaches it (one full focus cycle covers every focusable
# element; the visible ring is the GLOBAL :focus-visible rule,
# styles.css — unit-pinned, no per-control rule)…
for _ in range(60):
page.keyboard.press("Tab")
if box.evaluate("el => el === document.activeElement"):
break
expect(box).to_be_focused()
# …and it receives programmatic focus too (the a11y trees agree).
box.focus()
expect(box).to_be_focused()
# …and the visible "Hidden" label wraps the box (text + control —
# never color or icon alone).
label = page.locator("label.git-source-hidden")
expect(label).to_have_count(1)
expect(label).to_contain_text("Hidden")
# The page-level error line exists, is a real role=alert, and stays
# hidden through the happy path.
error = page.locator("#git-sources-hidden-error")
expect(error).to_have_count(1)
assert error.get_attribute("role") == "alert"
expect(error).to_be_hidden()
# The tag is TEXT (never color alone — WCAG 1.4.1): flip on…
_flip_hidden(page, stored_path, wanted_on=True)
expect(
_row(page, stored_path).locator(".git-source-hidden-count")
).to_have_text("hidden on")
expect(error).to_be_hidden() # the happy path never touches the line
# §7.4 failure path: a network-level abort of the PATCH → the
# canned "not changed" message in the role=alert line and the box
# reverts to the SERVER state (still ON — the request never
# landed) and re-enables. The tag stays (the server is unchanged).
def _abort_patch(route: Any) -> None:
if route.request.method == "PATCH":
route.abort()
else:
route.continue_()
page.route("**/api/git-sources/**", _abort_patch)
try:
_hidden_box(page, stored_path).click()
expect(error).to_be_visible(timeout=30_000)
expect(error).to_have_text(
"Could not reach the server — the setting was not changed."
)
expect(
_row(page, stored_path).locator(".git-source-hidden-count")
).to_have_count(1) # the server state is unchanged: still on
reverted = _hidden_box(page, stored_path)
expect(reverted).to_be_checked() # reverted to the server state
expect(reverted).to_be_enabled() # re-enabled — never stale
finally:
page.unroute("**/api/git-sources/**")
tests/integration/test_git_sources_api.py
+197 -20
View File
@@ -32,9 +32,16 @@ Contract under test:
* ignore paths (phase 89) — POST accepts the RAW box lines for both
kinds (optional, absent → ``[]``), stored normalized (A1) with the A4
fixed-detail 422s (shared gate with PATCH); GET reports each row's
stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only)
replaces the list wholesale (A5 — an empty list clears all), 404
unknown id, 422 fixed details for the A4 limits, the row otherwise
stored list (env rows ``[]``);
* hidden-folders flag (phase 105) — POST accepts ``include_hidden`` for
both kinds (optional, absent → stored ``False``, A4); GET reports the
stored flag (env rows ``False`` — no DB row to store a flag on);
* ``PATCH /{source_id}`` (admin-only) — phase 89 A5 + phase 105: each
field is optional, present-wins — ``ignore_paths`` when present
REPLACES the list wholesale (A5 — an empty list clears all),
``include_hidden`` when present sets the flag, both absent → 200
no-op, 404 unknown id, 422 fixed details for the A4 limits, a
rejected list never half-applies the flag, the row otherwise
unchanged.
``git_sources`` is global state: truncated around every test.
@@ -99,10 +106,14 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# The phase-89 ignore-list PATCH is gated the same way.
# The phase-89 ignore-list PATCH is gated the same way — and so is
# the phase-105 bool-only payload (the toggle's exact request).
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"include_hidden": True})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# Nothing landed in the table.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
@@ -132,8 +143,10 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "https://a.example.com/one.git",
"path": None,
"added_at": None,
# Env rows have no DB row to store a list on (phase 89).
# Env rows have no DB row to store a list on (phase 89) or
# a flag on (phase 105).
"ignore_paths": [],
"include_hidden": False,
},
{
"id": None,
@@ -142,6 +155,7 @@ def test_get_empty_table_with_env_returns_env_rows(
"path": None,
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
},
]
@@ -208,9 +222,11 @@ def test_post_creates_trimmed_and_list_stops_using_env(
uuid.UUID(body["id"])
assert body["added_at"] is not None
# Phase 89: the response gains ``ignore_paths`` — absent at create
# time → ``[]``.
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
# time → ``[]``; phase 105 adds ``include_hidden`` — absent →
# ``False`` (A4).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert body["ignore_paths"] == []
assert body["include_hidden"] is False
# The DB row now wins: from_env False, the env URL is gone from the list.
body = admin_client.get("/api/git-sources").json()
@@ -321,9 +337,11 @@ def test_post_local_creates_stored_row_with_expanded_path(
body = r.json()
# The phase-35 response shape is unchanged — the local row reports
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
# phase 89 adds ``ignore_paths`` (absent → ``[]``).
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
# phase 89 adds ``ignore_paths`` (absent → ``[]``); phase 105 adds
# ``include_hidden`` (absent → ``False``, A4).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert body["ignore_paths"] == []
assert body["include_hidden"] is False
uuid.UUID(body["id"])
assert body["url"] == str(real_dir)
assert body["added_at"] is not None
@@ -556,6 +574,7 @@ def test_delete_removes_row_and_falls_back_to_env(
"path": None,
"added_at": None,
"ignore_paths": [], # env rows: no DB row to store a list on
"include_hidden": False, # … or a flag on (phase 105)
}
]
db.execute(text("TRUNCATE chunks, documents"))
@@ -671,6 +690,7 @@ def test_get_reports_stored_ignore_paths_and_env_rows_empty(
"path": None,
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
}
]
@@ -690,8 +710,9 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert body["ignore_paths"] == ["a", "b"] # normalized
assert body["include_hidden"] is False # untouched by a list-only PATCH
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Round-trip through GET.
@@ -706,17 +727,34 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == []
def test_patch_missing_field_is_422(admin_client: TestClient) -> None:
"""``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent
field is a 422 with the model's own detail."""
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"})
assert created.status_code == 201
r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={})
assert r.status_code == 422
# The model's own (Pydantic) detail — the missing required field.
assert any(
item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"]
def test_patch_empty_body_is_a_noop_200(admin_client: TestClient) -> None:
"""Phase 105: BOTH fields absent (or explicit None) → 200 no-op —
the row is untouched. (The pre-phase-105 pin of an absent
``ignore_paths`` as a 422 is retired — the list is optional now; a
PRESENT list keeps the phase-89 A5 replace semantics.)"""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/noop.git", "ignore_paths": ["a/b"]},
)
assert created.status_code == 201
before = created.json()
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={})
assert r.status_code == 200, r.text
body = r.json()
assert body["ignore_paths"] == ["a/b"]
assert body["include_hidden"] is False
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Explicit None values are "absent" too — still a no-op.
r = admin_client.patch(
f"/api/git-sources/{before['id']}", json={"ignore_paths": None, "include_hidden": None}
)
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == ["a/b"]
assert r.json()["include_hidden"] is False
# The row never changed — round-trip through GET.
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a/b"]
def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None:
@@ -772,3 +810,142 @@ def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None:
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]})
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == [long_entry]
# --- hidden-folders flag (phase 105) ----------------------------------------
def test_get_reports_include_hidden_default_false(admin_client: TestClient, db: Session) -> None:
"""A fresh stored row (no flag passed) reports ``include_hidden:
false`` — in the GET round-trip AND on the model (the column's
server default, A4)."""
db.add(GitSource(url="https://example.com/fresh.git"))
db.commit()
body = admin_client.get("/api/git-sources").json()
assert body["sources"][0]["include_hidden"] is False
row = db.scalars(select(GitSource)).one()
assert row.include_hidden is False
def test_post_stores_include_hidden_both_kinds(
admin_client: TestClient, tmp_path: Path
) -> None:
"""POST accepts ``include_hidden`` for both kinds (present → stored
as sent); the 201 body and the GET round-trip report it."""
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/hidden-git.git", "include_hidden": True},
)
assert r.status_code == 201, r.text
assert r.json()["include_hidden"] is True
real_dir = tmp_path / "hidden-local"
real_dir.mkdir()
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "path": str(real_dir), "include_hidden": True},
)
assert r.status_code == 201, r.text
assert r.json()["include_hidden"] is True
by_url = {
s["url"]: s["include_hidden"]
for s in admin_client.get("/api/git-sources").json()["sources"]
}
assert by_url["https://example.com/hidden-git.git"] is True
assert by_url[str(real_dir)] is True
def test_patch_bool_only_sets_flag_leaves_list(admin_client: TestClient) -> None:
"""The toggle's exact payload — ``{"include_hidden": …}`` alone:
the flag is set, the ignore list is UNCHANGED (present-wins, phase
105)."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/toggle.git", "ignore_paths": ["a/b"]},
)
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": True})
assert r.status_code == 200, r.text
body = r.json()
assert body["include_hidden"] is True
assert body["ignore_paths"] == ["a/b"] # untouched
assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True
# And back off again.
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": False})
assert r.status_code == 200, r.text
assert r.json()["include_hidden"] is False
assert r.json()["ignore_paths"] == ["a/b"]
def test_patch_list_only_replaces_list_leaves_flag(admin_client: TestClient) -> None:
"""The dialog's exact payload — ``{"ignore_paths": …}`` alone:
the list is REPLACED (normalized, phase-89 A5), the flag is
UNCHANGED — byte-identical to the pre-phase-105 dialog PATCH."""
created = admin_client.post(
"/api/git-sources",
json={
"url": "https://example.com/dialog.git",
"ignore_paths": ["a/b"],
"include_hidden": True,
},
)
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": ["c/d", " e "]})
assert r.status_code == 200, r.text
body = r.json()
assert body["ignore_paths"] == ["c/d", "e"] # normalized
assert body["include_hidden"] is True # untouched
assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True
def test_patch_both_fields_apply_independently(admin_client: TestClient) -> None:
"""``{"ignore_paths": [], "include_hidden": true}`` — the list is
cleared AND the flag is set in one request."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/both.git", "ignore_paths": ["old/one"]},
)
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": [], "include_hidden": True}
)
assert r.status_code == 200, r.text
body = r.json()
assert body["ignore_paths"] == [] # cleared
assert body["include_hidden"] is True # set
def test_patch_422_does_not_half_apply_flag(admin_client: TestClient) -> None:
"""A PRESENT bad list 422s with the fixed A4 details — and the
OTHER field in the same body never half-applies: after each 422
the flag is still the stored value (validation raises before any
assignment, request transaction untouched)."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/half.git", "ignore_paths": ["keep"]},
)
assert created.status_code == 201
sid = created.json()["id"]
for payload, detail in (
([" "], "ignore paths must be non-empty"),
([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"),
(["a" * 501], "an ignore path exceeds 500 characters"),
):
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": payload, "include_hidden": True}
)
assert r.status_code == 422, f"{detail!r}: {r.text}"
assert r.json()["detail"] == detail
# Nothing half-applied: list AND flag are both still the originals.
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["ignore_paths"] == ["keep"]
assert row["include_hidden"] is False
tests/integration/test_import_docs_git.py
+190 -11
View File
@@ -27,15 +27,19 @@ is asserted against the canned value).
from __future__ import annotations
import re
from collections.abc import Iterator
from pathlib import Path
import pytest
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import GitSource
from app.rag.importer import ImportSummary
from scripts import import_docs
from scripts.git_sync import GitSyncError
from tests.fakes import FakeEmbedder
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
@@ -47,11 +51,31 @@ def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource:
def _local_row(
path: str,
ignore_paths: list[str] | None = None,
include_hidden: bool = False,
) -> GitSource:
"""A local row as the phase-38 API stores it: the expanded path in
both ``path`` and the NOT-NULL ``url`` location column (plus the
phase-89 ignore list, empty by default)."""
return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or [])
phase-89 ignore list and the phase-105 hidden-folders flag, both
defaulting off — A4)."""
return GitSource(
url=path, kind="local", path=path,
ignore_paths=ignore_paths or [], include_hidden=include_hidden,
)
@pytest.fixture()
def clean_documents(db: Session) -> Iterator[None]:
"""Phase 105: the real-import CLI tests write ``documents``/
``chunks`` (the canonical KB state) — global, truncated around
every such test."""
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
class FakeImportSources:
@@ -68,10 +92,12 @@ class FakeImportSources:
prune: bool = False,
limit: int | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
) -> ImportSummary:
self.calls.append(
{"sources": list(sources), "prune": prune, "limit": limit,
"ignore_by_root": ignore_by_root}
"ignore_by_root": ignore_by_root,
"include_hidden_by_root": include_hidden_by_root}
)
return ImportSummary(files=1, added=1)
@@ -168,10 +194,16 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
sources_dir=str(tmp_path / "bor"),
)
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
# Phase 105: flag-off rows still contribute their root — with False
# (A4), keyed by the SAME root string the importer sees.
assert hidden_map == {
str(tmp_path / "bor" / "homelab"): False,
str(tmp_path / "bor" / "deploy"): False,
}
assert calls == [
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
@@ -186,10 +218,11 @@ def test_resolve_sources_cli_source_wins(
settings = _settings(git_sources="https://host/a/repo.git")
manual = tmp_path / "Manual"
sources, ignore_map = import_docs._resolve_sources([manual], settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources([manual], settings)
assert sources == [manual]
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
assert hidden_map == {} # phase 105: manual dirs have no rows → hidden skipped
assert calls == [] # git is never touched when --source is given
@@ -210,10 +243,12 @@ def test_resolve_sources_db_rows_win_over_env(
sources_dir=str(tmp_path / "bor"),
)
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
# Phase 105: the default-flag row contributes its root with False (A4).
assert hidden_map == {str(tmp_path / "bor" / "only"): False}
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
@@ -222,9 +257,10 @@ def test_resolve_sources_defaults_when_nothing_configured(
) -> None:
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
sources, ignore_map = import_docs._resolve_sources(None, _settings())
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, _settings())
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
assert ignore_map == {} # phase 89: the legacy fallback has no rows
assert hidden_map == {} # phase 105: the legacy fallback has no rows
def test_resolve_sources_rows_branch_builds_ignore_map(
@@ -250,11 +286,13 @@ def test_resolve_sources_rows_branch_builds_ignore_map(
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
# Keyed by the SAME string the importer sees (the root, not the name).
assert ignore_map == {str(local_dir): ["ignore/"]}
# Phase 105: both rows are flag-off → per-root False entries (A4).
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
def test_resolve_sources_two_rows_sharing_root_string_extend(
@@ -279,11 +317,14 @@ def test_resolve_sources_two_rows_sharing_root_string_extend(
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
shared = str(tmp_path / "bor" / "shared")
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
assert ignore_map == {shared: ["a/", "b"]} # union, row order
# Phase 105 collision: the shared root gets the OR of the flags —
# both rows off here, so one False entry for the one root string.
assert hidden_map == {shared: False}
def test_main_rows_branch_passes_ignore_map_to_import(
@@ -318,9 +359,145 @@ def test_main_rows_branch_passes_ignore_map_to_import(
call = fake_import.calls[0]
assert call["sources"] == [local_dir]
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
# Phase 105: the default-flag row passes the per-root map too — a
# False entry, not an absent key (the importer reads it per root).
assert call["include_hidden_by_root"] == {str(local_dir): False}
assert call["prune"] is False # the CLI's no-prune default is unchanged
# --- phase 105: per-row hidden-folders flag ---------------------------------
def test_main_rows_branch_passes_include_hidden_map_to_import(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Phase 105: a ``kind=local`` row with ``include_hidden=True`` →
``main`` passes the per-root flag map to ``import_sources`` (keyed
by the directory string — the map is built, not lost)."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
assert rc == 0
call = fake_import.calls[0]
assert call["sources"] == [local_dir]
assert call["include_hidden_by_root"] == {str(local_dir): True}
assert call["ignore_by_root"] == {} # the row carries no ignore list
assert call["prune"] is False # the CLI's no-prune default is unchanged
def _stub_overview(monkeypatch: pytest.MonkeyPatch) -> None:
"""Stub the phase-31 overview regeneration (the CLI's real-import
tests: the deterministic ``FakeEmbedder`` must not burn its canned
``chat`` on the KB outline — the import is what is under test)."""
async def fake_overview(llm: object, session: object = None) -> bool:
return False
monkeypatch.setattr(import_docs, "regenerate_overview", fake_overview)
def _kb_docs(db: Session) -> set[tuple[str, str]]:
"""The (source, path) pairs of the ``documents`` table."""
rows = db.execute(text("SELECT source, path FROM documents")).all()
return {(row.source, row.path) for row in rows}
def test_main_local_row_include_hidden_true_indexes_hidden_file(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
db: Session,
capsys: pytest.CaptureFixture[str],
clean_documents: None,
) -> None:
"""Phase 105 (A1): the CLI's DB-row path with the flag ON — the
file inside the hidden folder is indexed, embedded, and counted
like any visible file (real import over a host temp dir, the
deterministic ``FakeEmbedder``; the regression this phase most
plausibly breaks — the CLI's map built but lost — would leave it
out)."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
)
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
_stub_overview(monkeypatch)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
assert rc == 0
# The hidden file has a documents row next to the visible one.
assert _kb_docs(db) == {
("LocalDocs", "visible.md"),
("LocalDocs", ".hidden/note.md"),
}
out = capsys.readouterr().out
assert "added=2" in out # both files were imported
def test_main_local_row_include_hidden_false_skips_hidden_file(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
db: Session,
capsys: pytest.CaptureFixture[str],
clean_documents: None,
) -> None:
"""Phase 105 (A4): the same fixture with the default flag (off) —
the hidden-folder file never enters the KB (the byte-identical
pre-phase-105 walk): no documents row, and the summary line counts
only the visible file."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir))], "db"), # flag defaults to False
)
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
_stub_overview(monkeypatch)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
assert rc == 0
assert _kb_docs(db) == {("LocalDocs", "visible.md")}
out = capsys.readouterr().out
assert "added=1" in out # the hidden file was never walked
# --- main() ----------------------------------------------------------------
@@ -428,10 +605,12 @@ def test_resolve_sources_mixed_git_and_local(
sources_dir=str(tmp_path / "bor"),
)
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
assert ignore_map == {} # phase 89: neither row carries a list
# Phase 105: both rows default-flag → per-root False entries (A4).
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
tests/integration/test_importer_include_hidden.py
+241
View File
@@ -0,0 +1,241 @@
"""Integration: the phase-105 hidden-folders flag through the real import
pipeline.
Phase 105 (TODO.md L3 — "…a toggle per input … to allow indexing hidden
.folders."): ``import_sources`` gains ``include_hidden_by_root`` (same
``str(root)`` keying as phase 89's ``ignore_by_root``). A1
(owner-confirmed 2026-09-14): flag ON admits dot-prefixed components —
hidden files are indexed, embedded, and summarized exactly like visible
files — while ``EXCLUDED_DIRS`` stay excluded in both states. A2: a file
indexed with the flag ON leaves the KB on the next ``prune=True`` run
with the flag OFF (the untouched ``seen`` set does the work). A4: no map
→ byte-identical to pre-phase-105. Mirrors the fixture-tree + mock-LLM
pattern of ``test_importer_ignore.py``: a ``tmp_path`` source dir run
through the real ``import_sources`` into the compose Postgres, with
:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.models import Chunk, Document
from app.rag.importer import import_sources
from tests.fakes import FakeEmbedder
NAME = "HiddenFix"
def _write(root: Path, rel: str, content: str) -> None:
path = root / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
def _tree(tmp_path: Path, name: str = NAME) -> Path:
"""One visible md, a hidden dir (md + non-markdown yaml), an excluded dir."""
root = tmp_path / name
_write(root, "visible.md", "# Visible\n\nvisible body\n")
_write(root, ".hidden/note.md", "# Note\n\nHIDDEN-MD-CONTENT\n")
_write(root, ".hidden/data.yaml", "key: HIDDEN-YAML-VALUE\n")
_write(root, ".venv/junk.md", "# Junk\n\nEXCLUDED-CONTENT\n")
return root
def _reset(db: Session) -> None:
# House cleanup pattern (tests/integration/test_importer_ignore.py).
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
def test_hidden_paths_not_indexed_by_default(db: Session, tmp_path: Path) -> None:
# A4: no map → today's behavior, byte-identical — the hidden files
# never produce a Document/Chunk row and are never embedded.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
summary = asyncio.run(import_sources([root], llm, session=db))
assert summary.files == 1
assert summary.added == 1
assert summary.errors == 0
assert summary.summaries == 0
assert summary.summary_errors == 0
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {(NAME, "visible.md")}
for rel in (".hidden/note.md", ".hidden/data.yaml", ".venv/junk.md"):
assert not any(d.path == rel for d in docs)
for texts in llm.calls: # every embed batch
for t in texts:
assert "HIDDEN-MD-CONTENT" not in t and "HIDDEN-YAML-VALUE" not in t
assert "EXCLUDED-CONTENT" not in t
# The hidden yaml never reached the lite model.
assert not llm.chat_calls
_reset(db)
def test_hidden_paths_indexed_when_flag_on(db: Session, tmp_path: Path) -> None:
# A1: with the map, hidden files are indexed, embedded, and
# summarized exactly like visible files — EXCLUDED_DIRS stay out.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
summary = asyncio.run(
import_sources(
[root], llm, session=db, include_hidden_by_root={str(root): True}
)
)
assert summary.files == 3
assert summary.added == 3
assert summary.errors == 0
assert summary.summary_errors == 0
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
(NAME, "visible.md"),
(NAME, ".hidden/note.md"),
(NAME, ".hidden/data.yaml"),
}
# EXCLUDED_DIRS is excluded in BOTH states (A1).
assert not any(d.path == ".venv/junk.md" for d in docs)
chunks = db.scalars(select(Chunk)).all()
assert not any("EXCLUDED-CONTENT" in c.content for c in chunks)
# The hidden md was embedded like any visible md (no summary — the
# markdown path skips the lite model).
note = db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
assert note is not None and note.summary is None
assert any("HIDDEN-MD-CONTENT" in c.content for c in chunks)
# The hidden yaml went through the FULL non-markdown path (phase 30):
# a stored summary plus one embedded is_summary chunk on top of the
# content chunks.
yaml_doc = db.scalar(select(Document).where(Document.path == ".hidden/data.yaml"))
assert yaml_doc is not None and yaml_doc.summary is not None
yaml_chunks = [
c for c in chunks if c.document_id == yaml_doc.id
]
assert any(c.is_summary for c in yaml_chunks)
assert any(not c.is_summary for c in yaml_chunks)
assert any("HIDDEN-YAML-VALUE" in c.content for c in yaml_chunks)
# Only the yaml reached the lite model.
assert summary.summaries == 1
assert len(llm.chat_calls) == 1
user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user")
assert "HIDDEN-YAML-VALUE" in user
_reset(db)
def test_flag_off_re_run_prunes_hidden_docs(db: Session, tmp_path: Path) -> None:
# A2: the owner flips the flag off — the next prune run walks with
# the default rules, the hidden files never enter ``seen``, and their
# rows (summary chunk included) leave the index.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
s1 = asyncio.run(
import_sources(
[root], llm, session=db, include_hidden_by_root={str(root): True}
)
)
assert (s1.files, s1.added) == (3, 3)
assert (
db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
is not None
)
s2 = asyncio.run(import_sources([root], llm, session=db, prune=True))
assert s2.files == 1
assert s2.unchanged == 1 # visible.md untouched
assert s2.pruned == 2 # both hidden documents
assert (
db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
is None
)
assert (
db.scalar(select(Document).where(Document.path == ".hidden/data.yaml"))
is None
)
assert (
db.scalar(select(Document).where(Document.path == "visible.md")) is not None
)
# The summary chunk rows went with their documents (cascade).
assert not any(
"HIDDEN-YAML-VALUE" in c.content for c in db.scalars(select(Chunk)).all()
)
_reset(db)
def test_progress_total_agrees_with_walk_in_both_states(
db: Session, tmp_path: Path
) -> None:
# The phase-64 pre-walk uses the same per-root flag as the loop, so
# ``total`` agrees with the walk in both states.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
on_totals: set[int] = set()
def on_progress(source: str, rel: str, done: int, total: int) -> None:
on_totals.add(total)
s_on = asyncio.run(
import_sources(
[root],
llm,
session=db,
progress=on_progress,
include_hidden_by_root={str(root): True},
)
)
assert s_on.files == 3
assert on_totals == {3} # hidden files counted in the denominator
_reset(db)
llm2 = FakeEmbedder()
off_totals: set[int] = set()
def off_progress(source: str, rel: str, done: int, total: int) -> None:
off_totals.add(total)
s_off = asyncio.run(
import_sources([root], llm2, session=db, progress=off_progress)
)
assert s_off.files == 1
assert off_totals == {1} # visible only, the pre-phase-105 count
_reset(db)
def test_unlisted_root_stays_hidden(db: Session, tmp_path: Path) -> None:
# The map is per-root, not global: listing one root as True leaves
# the other root exactly as pre-phase-105.
_reset(db)
root_a = _tree(tmp_path, name="HiddenFixA")
root_b = _tree(tmp_path, name="HiddenFixB")
llm = FakeEmbedder()
summary = asyncio.run(
import_sources(
[root_a, root_b],
llm,
session=db,
include_hidden_by_root={str(root_a): True},
)
)
# A: 3 (flag on) · B: 1 (unlisted → False)
assert summary.files == 4
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
("HiddenFixA", "visible.md"),
("HiddenFixA", ".hidden/note.md"),
("HiddenFixA", ".hidden/data.yaml"),
("HiddenFixB", "visible.md"),
}
assert not any(
d.source == "HiddenFixB" and d.path == ".hidden/note.md" for d in docs
)
_reset(db)
tests/integration/test_migration_0019.py
+271
View File
@@ -0,0 +1,271 @@
"""Integration: migration 0019 (git_sources.include_hidden) schema
contract (phase 105, task 01).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0007.py`` (git_sources column contract — information_schema
assertions on the state the migration must leave). The tests target the
0018 → 0019 step explicitly so later migrations cannot break them:
* upgrade 0018 → 0019 → the ``include_hidden`` column exists with the
full contract — BOOLEAN NOT NULL, server default ``false`` — while
the 0018 ``git_sources`` schema (url, kind, path, ignore_paths)
survives;
* pre-0019 rows backfill ``false`` and a row written without the
column takes the server default (A4 — byte-identical import
behavior until the owner flips the flag);
* the ORM contract agrees: a freshly inserted ``GitSource`` (no flag
passed) reads ``include_hidden is False`` (the Python ``default=False``
and the server default agree), and an explicit ``True`` round-trips
through a fresh session;
* downgrade to 0018 → the column is GONE (A13 — reversible) while the
rows + their ignore lists survive;
* upgrade back to 0019 → the column is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
import uuid
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.orm import Session
from alembic import command
from app.db import SessionLocal, db_available
from app.models import GitSource
URL_BASE = "https://git.example.com/mig0019"
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown upgrades
to head no matter what happened, so the dev DB is never left below
head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
# on ``git_sources``, which would deadlock the repair's
# ``ALTER TABLE`` (0019) forever.
db.rollback()
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one git_sources column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'git_sources' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _flag(db: Session, url: str) -> Any:
return db.execute(
text("SELECT include_hidden FROM git_sources WHERE url = :u"),
{"u": url},
).scalar_one()
def _insert_sql(db: Session, url: str, include_hidden: Any = None) -> None:
"""Insert one git_sources row (kind/path omitted → the API default
shape); ``include_hidden`` omitted → pre-0019 insert shape."""
if include_hidden is None:
db.execute(
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
{"u": url},
)
else:
db.execute(
text(
"INSERT INTO git_sources (id, url, include_hidden)"
" VALUES (gen_random_uuid(), :u, :f)"
),
{"u": url, "f": include_hidden},
)
db.commit()
def _delete_by_url(db: Session, url: str) -> None:
db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url})
db.commit()
def test_upgrade_to_0019_adds_include_hidden(db: Session, alembic: Config) -> None:
"""Upgrade 0018 → 0019: the column exists with the full contract
(BOOLEAN NOT NULL, server default ``false``), is ABSENT at 0018,
pre-0019 rows backfill ``false`` (A4), a new row without the column
takes the default, and an explicit ``true`` round-trips — while the
0018 table contract survives."""
command.downgrade(alembic, "0018") # start from the pre-0019 state
assert _version(db) == "0018"
assert _column(db, "include_hidden") is None, (
"the flag must be absent at 0018"
)
url_pre = f"{URL_BASE}/pre-existing.git"
_insert_sql(db, url_pre) # no include_hidden column exists at 0018
try:
command.upgrade(alembic, "0019")
assert _version(db) == "0019", "alembic_version must be at 0019"
flag = _column(db, "include_hidden")
assert flag is not None, "git_sources.include_hidden is missing"
assert flag[0] == "boolean", "include_hidden must be BOOLEAN"
assert flag[1] == "NO", "include_hidden must be NOT NULL"
assert flag[2] is not None and "false" in str(flag[2]), (
"include_hidden must carry the `false` server default"
)
# The pre-0019 row backfilled ``false`` (A4 — unchanged imports).
assert _flag(db, url_pre) is False
# A row written without the column takes the server default.
url_new = f"{URL_BASE}/new-row.git"
_insert_sql(db, url_new)
try:
assert _flag(db, url_new) is False, (
"an omitted flag takes the `false` server default"
)
# The flag round-trips through an explicit ``true``.
db.execute(
text("UPDATE git_sources SET include_hidden = true WHERE url = :u"),
{"u": url_new},
)
db.commit()
assert _flag(db, url_new) is True, (
"include_hidden = true must round-trip"
)
finally:
_delete_by_url(db, url_new)
# The 0018 schema survives the additive upgrade.
ignore = _column(db, "ignore_paths")
assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", (
"git_sources.ignore_paths (0013) must survive the upgrade"
)
kind = _column(db, "kind")
assert kind is not None and kind[0] == "text" and kind[1] == "NO", (
"git_sources.kind (0007) must survive the upgrade"
)
finally:
_delete_by_url(db, url_pre)
def test_orm_fresh_row_defaults_false_and_true_round_trips(
db: Session, alembic: Config
) -> None:
"""The ORM contract agrees with the column contract: a freshly
inserted ``GitSource`` (no flag passed) reads ``include_hidden is
False`` — the Python ``default=False`` and the server default
agree (A4) — and an explicit ``True`` round-trips through a fresh
session."""
command.upgrade(alembic, "head")
url_off = f"{URL_BASE}/orm-default.git"
url_on = f"{URL_BASE}/orm-true.git"
try:
# Fresh row, flag omitted → False (the Python-side default).
row_off = GitSource(url=url_off, kind="git")
db.add(row_off)
db.commit()
db.expire_all()
reloaded_off = db.get(GitSource, row_off.id)
assert reloaded_off is not None, "the fresh row must be readable"
assert reloaded_off.include_hidden is False, (
"a fresh row must read include_hidden is False (A4)"
)
# Explicit True round-trips through a FRESH session.
row_on = GitSource(url=url_on, kind="git", include_hidden=True)
db.add(row_on)
db.commit()
with SessionLocal() as fresh:
reloaded = fresh.get(GitSource, row_on.id)
assert reloaded is not None, "the row must exist in a fresh session"
assert reloaded.include_hidden is True, (
"include_hidden=True must round-trip through the DB"
)
finally:
_delete_by_url(db, url_off)
_delete_by_url(db, url_on)
def test_downgrade_to_0018_drops_the_column(db: Session, alembic: Config) -> None:
"""Downgrade 0019 → 0018: the column is gone (A13 — fully
reversible) while the rows + their ignore lists survive, and the
rest of the 0018 table contract (``path``) is intact."""
command.upgrade(alembic, "head")
url = f"{URL_BASE}/survivor.git"
db.execute(
text(
"INSERT INTO git_sources (id, url, include_hidden)"
" VALUES (:i, :u, true)"
),
{"i": uuid.uuid4(), "u": url},
)
db.commit()
try:
command.downgrade(alembic, "0018")
assert _version(db) == "0018"
assert _column(db, "include_hidden") is None, (
"the flag must be dropped"
)
row = db.execute(
text("SELECT url, kind, ignore_paths FROM git_sources WHERE url = :u"),
{"u": url},
).fetchone()
assert row is not None and row[0] == url, (
"the row must survive the column drop"
)
assert row[1] == "git" and row[2] == [], (
"kind + the ignore list must survive the column drop"
)
doc_path = _column(db, "path")
assert doc_path is not None and doc_path[0] == "text", (
"git_sources.path must survive the downgrade"
)
finally:
_delete_by_url(db, url)
# Repair: the fixture teardown re-upgrades to head.
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
"""Downgrade to 0018, then upgrade back to 0019: the column is back
with the full contract (BOOLEAN NOT NULL, the `false` default)."""
command.downgrade(alembic, "0018")
command.upgrade(alembic, "0019")
assert _version(db) == "0019", "round-trip upgrade must land at 0019"
flag = _column(db, "include_hidden")
assert flag is not None, "git_sources.include_hidden must be back"
assert flag[0] == "boolean", "include_hidden must be BOOLEAN after the round-trip"
assert flag[1] == "NO", "include_hidden must be NOT NULL after the round-trip"
assert flag[2] is not None and "false" in str(flag[2]), (
"the `false` server default must survive the round-trip"
)
tests/integration/test_sync_api.py
+120 -1
View File
@@ -77,7 +77,7 @@ from typing import Any
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import sync as sync_api
@@ -262,6 +262,11 @@ class FakeImportSources:
# rows' ``ignore_paths`` (keyed by the root string the importer
# sees; two rows sharing a root string get the union).
self.ignore_maps: list[dict[str, list[str]]] = []
# Phase 105: the per-root hidden-folders flag map the runner
# builds from the rows' ``include_hidden`` (same root-string
# keying; a shared-root collision ORs — if either row says
# "index hidden", the root does).
self.include_hidden_maps: list[dict[str, bool]] = []
async def __call__(
self,
@@ -273,12 +278,14 @@ class FakeImportSources:
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
) -> ImportSummary:
self.sources.append(list(sources))
self.llms.append(llm)
self.prune_flags.append(prune)
self.progress_hooks.append(progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
if self.delay:
await asyncio.sleep(self.delay)
return self.summary
@@ -823,6 +830,7 @@ def test_import_error_is_reported_with_credentials_masked(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
) -> ImportSummary:
raise EmbeddingError(
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
@@ -1100,3 +1108,114 @@ def test_sync_without_ignore_lists_passes_empty_map(
assert fake_import.sources == [[local_dir]]
assert fake_import.ignore_maps == [{}] # no row carried a list
# --- phase 105: per-row hidden-folders flag --------------------------------
def test_local_row_hidden_flag_off_then_on_indexes_hidden_paths(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 105 (A4 then A1): the SAME local row, synced first with the
default flag (off) — the file inside the hidden folder never lands
in the KB (no document row — hence no embedding, no summary), the
visible file imports as usual; then the row is flipped on (direct
model set — the PATCH round-trip is task 03's layer) and the next
sync walks the hidden file too: it is indexed, embedded, and counted
like any visible file."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
_seed_local(db, local_dir) # include_hidden defaults to False (A4)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# Flag off (A4): only the visible file is walked and indexed — the
# hidden file has NO documents row.
assert body["detail"]["files"] == 1
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
docs = {
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
}
assert docs == {("LocalDocs", "visible.md")}
# Flip the SAME row on — the next sync re-reads the flag per row.
row = db.execute(select(GitSource).where(GitSource.url == str(local_dir))).scalar_one()
row.include_hidden = True
db.commit()
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# Flag on (A1): the hidden file is walked (detail.files counts it),
# embedded, and indexed alongside the visible file.
assert body["detail"]["files"] == 2
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
docs = {
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
}
assert docs == {("LocalDocs", "visible.md"), ("LocalDocs", ".hidden/note.md")}
def test_sync_builds_include_hidden_map_by_root_string_with_or(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 105 wiring (fake import): the runner keys the flag map by
the SAME root string the importer sees, and two rows sharing that
root string (the sibling/repo-name edge — ``…/shared`` and
``…/shared.git`` clone into the same checkout dir) get the OR of
their flags — if EITHER row says "index hidden", the root does
(the ignore-map union's boolean mirror)."""
url_a = f"file://{tmp_path / 'shared'}"
url_b = f"{url_a}.git" # same repo name → same checkout dir
# Distinct added_at: the resolver orders by (added_at, id) — a
# same-timestamp pair would tie-break on the random uuid.
db.add(GitSource(url=url_a, kind="git", include_hidden=False,
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
db.add(GitSource(url=url_b, kind="git", include_hidden=True,
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
shared = str(tmp_path / "bor" / "shared")
# Both rows resolve to the SAME checkout (the collision itself) and
# the map holds the OR of their flags, keyed by that one root
# string — the flag-off row's False is overridden by the True.
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
assert fake_import.include_hidden_maps == [{shared: True}]
tests/unit/test_hidden_folders_toggle.py
+436
View File
@@ -0,0 +1,436 @@
"""Unit: the per-row "Hidden" toggle on the Sources page (phase 105,
task 05).
``TODO.md`` L3: "There should be a toggle per input (next to the
ignores button) to allow indexing hidden .folders." Every STORED
``git_sources`` row (git or local) gets a native labeled **Hidden**
checkbox in the actions cell, LEFT of the "Ignore paths" button (DOM
order Hidden · Ignore paths · Remove — A5); when on, the source cell
shows a "hidden on" text tag (text + background, never color alone —
WCAG 1.4.1); the flip PATCHes ``{"include_hidden": …}`` with the
§7.4 never-stale lifecycle (box disables at once; 200 → reload +
confirm LAST; failure → the box reverts to the SERVER state and the
detail lands in the page-level ``#git-sources-hidden-error``
``role="alert"`` line — the checkbox is a table-cell control with no
dialog of its own). Env-fallback rows (``id`` null) get no checkbox
(A3 — nothing is stored to flag).
The browser behavior itself is E2E-gated by the phase-105 story suite
(``tests/e2e/test_hidden_folders_toggle.py``, task 06); like the
phase-89 ``test_source_ignore_paths.py`` house pattern, this module
pins the source-level contract a silent regression would break:
* the JS row wiring — ``makeRow`` builds the checkbox ONLY in the
``s.id`` branch (class ``git-source-hidden-box``, ``type=checkbox``,
``checked`` from ``s.include_hidden === true`` — server state only,
never a prior local flip), the aria-label is the ONLY place the
value appears (setAttribute — never innerHTML), the visible
"Hidden" text label, and the append ORDER (hidden label →
ignore button → Remove); the "hidden on" tag is TEXT in the source
cell, iff ``s.id && s.include_hidden === true``;
* the JS lifecycle — ``toggleHidden``: the box disables BEFORE the
``PATCH`` (one flip at a time), the body is
``JSON.stringify({ include_hidden: wanted })`` ONLY (the row's
ignore list is untouched — task 03's optional field), 200 →
clear the error → ``await loadSources()`` → announce (the
confirmation is the LAST announcement); non-2xx AND network
failure → the box REVERTS to ``s.include_hidden === true`` +
re-enables and the detail lands in the page-level alert line;
a healed ``loadSources`` clears the line (the phase-89 "happy
path heals the error state" precedent);
* the cross-file wire contract (SINGLE SOURCE OF TRUTH for the
field name) — the JS PATCH body key and the render field both
match their ``app/schemas.py`` counterparts
(``GitSourcePatchIn.include_hidden`` / ``GitSourceRow``) — a
rename on either side breaks the test;
* the static shell markup — ``#git-sources-hidden-error`` is a
``role="alert"`` line, hidden by default, reusing the
``.git-source-error`` class, INSIDE ``#view-git-sources`` and
AFTER ``#git-sources-table-wrap`` in source order;
* styles.css — the ``.git-source-hidden`` family on the house
palette: ~44px hit height, the checkbox's ``accent-color:
var(--brand)`` (checked-state contrast verified + recorded in the
comment), the ``:disabled`` wait state, and the "hidden on" tag
(text + a distinct background, never color alone).
"""
from __future__ import annotations
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
FRONTEND = ROOT / "frontend"
# Phase 76 (task 02): the Sources view lives in the ONE-document shell.
SHELL_HTML = FRONTEND / "index.html"
JS = FRONTEND / "assets" / "git-sources.js"
CSS = FRONTEND / "assets" / "styles.css"
SCHEMAS = ROOT / "app" / "schemas.py"
#: The checkbox + tag classes (pinned verbatim).
HIDDEN_BOX_CLASS = "git-source-hidden-box"
HIDDEN_LABEL_CLASS = "git-source-hidden"
HIDDEN_TAG_CLASS = "git-source-hidden-count"
HIDDEN_TAG_TEXT = "hidden on"
#: The checkbox's aria-label template — the ONLY place the full
#: source value appears (setAttribute, never innerHTML).
ARIA_LABEL_TEMPLATE = "`Index hidden folders for ${kindLabel} source: ${value}`"
#: The §7.4 lifecycle strings (pinned verbatim).
REVERT_STATE = "box.checked = s.include_hidden === true"
NETWORK_MESSAGE = "Could not reach the server — the setting was not changed."
PATCH_BODY = "JSON.stringify({ include_hidden: wanted })"
def _text(path: Path) -> str:
return path.read_text(encoding="utf-8")
def _js() -> str:
return _text(JS)
def _css() -> str:
return _text(CSS)
def _fn(js: str, name: str) -> str:
"""The source of a (possibly async) top-level function via
balanced-brace counting (the test_source_ignore_paths.py helper)."""
for prefix in ("async function ", "function "):
start = js.find(f"{prefix}{name}(")
if start != -1:
depth = 0
for i in range(js.find("{", start), len(js)):
if js[i] == "{":
depth += 1
elif js[i] == "}":
depth -= 1
if depth == 0:
return js[start : i + 1]
raise AssertionError(f"unbalanced braces in {name}()")
raise AssertionError(f"{name}() must exist in git-sources.js")
def _css_rule(css: str, selector: str) -> str:
"""The declarations of a simple one-line-opening rule (comments
stripped first — a house comment may legally carry braces)."""
clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
start = clean.find(f"{selector} {{")
assert start != -1, f"missing rule {selector} in styles.css"
depth = 0
for i in range(clean.find("{", start), len(clean)):
if clean[i] == "{":
depth += 1
elif clean[i] == "}":
depth -= 1
if depth == 0:
return clean[start : i + 1]
raise AssertionError(f"unbalanced braces in {selector}")
def _python_class_field(class_name: str) -> str:
"""The single bool field name declared on a Pydantic class in
``app/schemas.py`` (``bool`` or ``bool | None`` — regex-parsed,
no import: the pin is on the source text, so a rename breaks it
without a schema load)."""
schemas = _text(SCHEMAS)
m = re.search(
rf"class {class_name}\(BaseModel\):.*?(?=\nclass |\Z)", schemas, re.S
)
assert m, f"class {class_name} must exist in app/schemas.py"
fields = re.findall(
r"^\s*(\w+):\s*bool(?:\s*\|\s*None)?\s*(?==|$)", m.group(0), re.M
)
assert len(fields) == 1, f"{class_name} must declare exactly one bool field"
return fields[0]
# ---------- the row wiring in makeRow ----------
def test_row_checkbox_is_built_in_the_stored_row_branch() -> None:
"""makeRow: the Hidden checkbox is created ONLY in the ``s.id``
branch (A3 — env-fallback rows get the "from .env" tag, no
checkbox). It is a NATIVE labeled checkbox: class
git-source-hidden-box, type=checkbox, the visible "Hidden" text,
``checked`` from the SERVER row (``s.include_hidden === true`` —
never a prior local flip, §7.4), and the aria-label template is
the ONLY place ``value`` appears (setAttribute — never
innerHTML). Its change handler runs toggleHidden."""
make = _fn(_js(), "makeRow")
branch_i = make.find("if (s.id) {")
else_i = make.find("} else {", branch_i)
assert branch_i != -1 and else_i > branch_i
box_i = make.find('const hiddenBox = document.createElement("input")', branch_i)
assert -1 < box_i < else_i, "the checkbox is created in the s.id branch (A3)"
assert 'hiddenBox.type = "checkbox"' in make, "a native checkbox"
assert f"hiddenBox.className = \"{HIDDEN_BOX_CLASS}\"" in make
assert "hiddenBox.checked = s.include_hidden === true" in make, (
"checked state comes ONLY from the server row"
)
assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)"
assert 'hiddenBox.setAttribute(' in make, "the label is set via setAttribute"
assert "hiddenBox.innerHTML" not in _js(), "XSS contract: no innerHTML on the box"
assert "hiddenLabel.innerHTML" not in _js(), "XSS contract: no innerHTML on the label"
assert 'document.createTextNode("Hidden")' in make, (
"the visible text label — never aria-label-only (WCAG)"
)
assert "hiddenLabel.append(hiddenBox, document.createTextNode(\"Hidden\"))" in make
assert 'hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox))' in make
# The env-fallback branch (else) has no checkbox.
else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)]
assert HIDDEN_BOX_CLASS not in else_slice and "toggleHidden" not in else_slice, (
"env-fallback rows get no checkbox (A3)"
)
def test_row_dom_order_is_hidden_ignore_remove() -> None:
"""makeRow: the append order in the actions cell is Hidden ·
Ignore paths · Remove (A5 — the toggle sits LEFT of the "Ignore
paths" button; Remove stays last)."""
make = _fn(_js(), "makeRow")
branch_i = make.find("if (s.id) {")
else_i = make.find("} else {", branch_i)
hidden_append = make.find("actTd.appendChild(hiddenLabel)", branch_i)
ignore_i = make.find('const ignoreBtn = document.createElement("button")', branch_i)
ignore_append = make.find("actTd.appendChild(ignoreBtn)", branch_i)
remove_append = make.find("actTd.appendChild(btn)", branch_i)
assert -1 < hidden_append < ignore_i, "the hidden label precedes the ignore button"
assert -1 < ignore_append < remove_append, "Remove stays last"
assert remove_append < else_i, "all three controls are in the s.id branch"
def test_hidden_on_tag_renders_text_in_the_source_cell() -> None:
"""makeRow: a stored row with the flag ON gets the "hidden on"
tag in the SOURCE cell (urlTd — the .git-source-ignore-count
idiom: TEXT, never color alone, WCAG 1.4.1), appended after the
location <code>; the tag is the state copy (pinned verbatim)."""
make = _fn(_js(), "makeRow")
assert "s.id && s.include_hidden === true" in make, (
"the tag only for stored rows with the flag on"
)
assert f"hiddenTag.className = \"{HIDDEN_TAG_CLASS}\"" in make
assert f"hiddenTag.textContent = \"{HIDDEN_TAG_TEXT}\"" in make, (
"the tag copy is TEXT — never color alone"
)
code_i = make.find("urlTd.append(badge, code)")
append_i = make.find("urlTd.append(hiddenTag)", code_i)
assert -1 < code_i < append_i, "the tag is appended to the source cell"
# ---------- the JS lifecycle (toggleHidden) ----------
def test_toggle_sends_patch_and_disables_the_box_first() -> None:
"""toggleHidden: the box disables BEFORE the PATCH goes out
(no double-flip — §7.4); the request is a PATCH to
/api/git-sources/{id} whose body is ``{"include_hidden": …}``
ONLY (the row's ignore list is untouched — task 03's optional
field)."""
body = _fn(_js(), "toggleHidden")
disable_i = body.find("box.disabled = true")
fetch_i = body.find("`/api/git-sources/${s.id}`")
method_i = body.find('method: "PATCH"', fetch_i)
body_i = body.find(PATCH_BODY, method_i)
assert -1 < disable_i < fetch_i < method_i < body_i, (
"disable → PATCH {include_hidden} (and nothing else)"
)
assert "ignore_paths" not in body, "the PATCH body never carries the ignore list"
def test_toggle_success_clears_reloads_and_announces_last() -> None:
"""toggleHidden 200: the error line clears, ``loadSources()``
AWAITS (the row re-renders from the server — the "hidden on" tag
lands), and the confirmation is the LAST announcement (the
reload's "N sources listed." lands first — the phase-89 order)."""
body = _fn(_js(), "toggleHidden")
ok_i = body.find("if (r.ok)")
hide_i = body.find("hideHiddenError()", ok_i)
reload_i = body.find("await loadSources()", ok_i)
announce_i = body.find("announce(`Hidden folders", reload_i)
assert -1 < ok_i < hide_i < reload_i < announce_i, (
"200: clear error → await loadSources → announce (LAST)"
)
assert "for ${value}.`)" in body[announce_i:], "the confirmation names the source"
assert "enabled" in body[announce_i:] and "disabled" in body[announce_i:], (
"the confirmation states the direction of the flip"
)
def test_toggle_failure_reverts_the_box_and_shows_the_page_alert() -> None:
"""toggleHidden failure: non-2xx → the server detail (apiDetail)
into the page-level alert line + the box REVERTS to the server
state + re-enables; network failure → the fixed reachable? line,
same revert. The UI never claims a state the server didn't save
(PLAN §7.4). The revert + re-enable happen on BOTH failure
branches (exactly twice each in the function)."""
body = _fn(_js(), "toggleHidden")
catch_i = body.find(".catch(")
assert catch_i != -1
fail_slice = body[body.find("if (r.ok)"):catch_i]
assert "await apiDetail(" in fail_slice, "the server detail is apiDetail-extracted"
assert "showHiddenError(" in fail_slice, "the detail lands in the page alert line"
assert REVERT_STATE in fail_slice, "the box reverts to the server state"
assert "box.disabled = false" in fail_slice, "the box re-enables"
net_slice = body[catch_i:]
assert NETWORK_MESSAGE in net_slice, "the fixed network copy"
assert "showHiddenError(" in net_slice
assert REVERT_STATE in net_slice
assert "box.disabled = false" in net_slice
assert body.count(REVERT_STATE) == 2, "revert on BOTH failure branches"
assert body.count("box.disabled = false") == 2, "re-enable on BOTH failure branches"
assert "box.disabled = true" in body[: body.find("if (r.ok)")], (
"the box disables at once, before any outcome"
)
def test_error_line_helpers_and_the_healed_load_clear() -> None:
"""showHiddenError/hideHiddenError drive the page-level
``hiddenErrorEl`` (textContent + hidden); ``loadSources``'s
success path calls hideHiddenError() AFTER hideLoadError() — a
healed list clears the stale line (the phase-89 "happy path
heals the error state" precedent)."""
js = _js()
assert 'const hiddenErrorEl = root.querySelector("#git-sources-hidden-error")' in js, (
"the page-local element grabber (root-scoped, the shell idiom)"
)
show = _fn(js, "showHiddenError")
assert "hiddenErrorEl.textContent = message" in show
assert "hiddenErrorEl.hidden = false" in show
hide = _fn(js, "hideHiddenError")
assert 'hiddenErrorEl.textContent = ""' in hide
assert "hiddenErrorEl.hidden = true" in hide
load = _fn(js, "loadSources")
hide_load_i = load.find("hideLoadError()")
hide_hidden_i = load.find("hideHiddenError()", hide_load_i)
assert -1 < hide_load_i < hide_hidden_i, (
"a healed load clears the hidden-toggle line (after the load-error clear)"
)
# ---------- the cross-file wire contract (single source of truth) ----------
def test_js_field_name_matches_the_python_patch_schema() -> None:
"""The cross-file pin: the JS PATCH body key and the render field
both equal their ``app/schemas.py`` counterparts —
``GitSourcePatchIn.include_hidden`` (the write side) and
``GitSourceRow.include_hidden`` (the read side). A rename on
either side breaks the wire contract and this test."""
js = _js()
body = _fn(js, "toggleHidden")
m = re.search(r"JSON\.stringify\(\{\s*(\w+)\s*:\s*wanted\s*\}\)", body)
assert m, "the PATCH body carries a single {<field>: wanted} object"
js_body_key = m.group(1)
assert js_body_key == _python_class_field("GitSourcePatchIn"), (
"the JS PATCH body key must match GitSourcePatchIn's field"
)
make = _fn(js, "makeRow")
rm = re.search(r"s\.(\w+)\s*===\s*true", make)
assert rm, "makeRow reads the flag off the server row (s.<field> === true)"
js_read_key = rm.group(1)
assert js_read_key == _python_class_field("GitSourceRow"), (
"the render field must match GitSourceRow's field"
)
assert js_body_key == js_read_key == "include_hidden", (
"one field name for the whole wire contract"
)
# ---------- the static shell markup ----------
def test_page_error_line_is_a_role_alert_after_the_table() -> None:
"""index.html: ``#git-sources-hidden-error`` is a ``<p>`` reusing
the existing ``.git-source-error`` class, ``role="alert"`` +
hidden by default, INSIDE ``#view-git-sources`` and AFTER
``#git-sources-table-wrap`` in source order (the checkbox is a
table-cell control, so its failure announces at page level — the
ignore dialog carries its own error INSIDE the modal). It
appears exactly once."""
html = _text(SHELL_HTML)
tag = re.search(r'<p[^>]*id="git-sources-hidden-error"[^>]*>', html)
assert tag, "#git-sources-hidden-error must be a real <p>"
open_tag = tag.group(0)
for attr in ('class="git-source-error"', 'role="alert"', "hidden"):
assert attr in open_tag, f"the error line carries {attr}"
view_i = html.find('id="view-git-sources"')
wrap_i = html.find('id="git-sources-table-wrap"')
err_i = html.find('id="git-sources-hidden-error"')
next_i = html.find('id="view-history"')
assert -1 < view_i < wrap_i < err_i < next_i, (
"the line sits inside #view-git-sources, after #git-sources-table-wrap"
)
assert html.count('id="git-sources-hidden-error"') == 1, (
"the id is unique in the shell"
)
# ---------- styles.css ----------
def test_hidden_checkbox_css_rules_exist_with_accent_color() -> None:
"""styles.css carries the phase-105 family: .git-source-hidden
(inline-flex, 44px hit height matching the action buttons, the
house ink), the checkbox (sized, ``accent-color: var(--brand)``),
the :disabled wait state (the .git-source-remove:disabled
idiom), and the "hidden on" tag (text + a distinct background —
never color alone). The checkbox comment RECORDS the
verified checked-state contrast (house style)."""
css = _css()
label = _css_rule(css, ".git-source-hidden")
assert "display: inline-flex" in label
assert "height: 44px" in label, "the ~44px hit height matches the action buttons"
assert "var(--ink)" in label
box = _css_rule(css, '.git-source-hidden input[type="checkbox"]')
assert "accent-color: var(--brand)" in box, "the --brand checkbox fill"
disabled = _css_rule(css, ".git-source-hidden:disabled")
assert "opacity" in disabled and "cursor: wait" in disabled, (
"the .git-source-remove:disabled idiom"
)
tag = _css_rule(css, ".git-source-hidden-count")
assert "var(--ink)" in tag and "var(--bg)" in tag, (
"text + a distinct background (never color alone)"
)
assert "url(http" not in css and "@import url(" not in css, (
"no CDN (AGENTS.md rule 6)"
)
def test_checkbox_contrast_is_verified_and_recorded() -> None:
"""House style: the checked-state contrast of the native widget
is VERIFIED and the ratio RECORDED in the comment above the
checkbox rule (the executor note the task template carries):
the --bg-on---brand house pairing (5.2:1) on the built-in theme,
and the theme tab's AA gate (4.5:1) for every saved palette."""
css = _css()
rule_i = css.find('.git-source-hidden input[type="checkbox"] {')
assert rule_i != -1
header = css[css.rfind("/*", 0, rule_i):rule_i]
assert "VERIFIED" in header, "the contrast is verified (the executor note)"
assert "5.2:1" in header, "the built-in --bg-on---brand ratio is recorded"
assert "4.5:1" in header, "the AA gate for themed palettes is recorded"
# ---------- the module docstring ----------
def test_module_docstring_carries_the_phase_105_contract() -> None:
"""The git-sources.js module docstring gained the phase-105
entry: the per-row Hidden checkbox (makeRow) →
PATCH {include_hidden} (task 03's optional field) →
loadSources + announce; failure reverts the box +
#git-sources-hidden-error (role=alert); env-fallback rows get
no checkbox (A3)."""
doc = _js().split("*/", 2)[0] # the module docstring (first block)
for frag in (
"Phase 105 (task 05)",
"toggleHidden",
"{ include_hidden }",
"#git-sources-hidden-error",
"env-fallback rows get NO checkbox",
"REVERTS to the server state",
"LAST announcement",
):
assert frag in doc, f"the module docstring lost: {frag!r}"
tests/unit/test_importer.py
+5 -3
View File
@@ -849,12 +849,14 @@ def test_no_progress_means_no_prewalk(
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
include_hidden: bool = False,
) -> list[Path]:
# Phase 89: the walker gained the ``ignore`` keyword — the sentinel
# accepts (and forwards) it to stay a drop-in.
# Phase 89: the walker gained the ``ignore`` keyword; phase 105:
# the ``include_hidden`` flag — the sentinel accepts (and forwards)
# both to stay a drop-in.
nonlocal walk_calls
walk_calls += 1
return real_walker(r, extensions, excluded, ignore)
return real_walker(r, extensions, excluded, ignore, include_hidden)
monkeypatch.setattr(importer, "iter_importable_files", counting)
try:
tests/unit/test_importer_include_hidden.py
+161
View File
@@ -0,0 +1,161 @@
"""Unit: the phase-105 hidden-folders flag in ``iter_importable_files``.
Phase 105 (TODO.md L3 — "…a toggle per input (next to the ignores
button) to allow indexing hidden .folders."): ``include_hidden`` lifts
ONLY the dot-prefixed-component skip for one source. A1
(owner-confirmed 2026-09-14): when True, files inside hidden folders
AND hidden files with an importable extension become importable;
``EXCLUDED_DIRS`` (``.venv``, ``node_modules``, ``.git``, …) stay
excluded in BOTH states; the extension filter always applies; the
phase-89 *ignore* tuple composes additively. A4: the default
(``False``) is byte-identical to the pre-phase-105 walk — pinned here
against a tmp fixture tree. The DB-facing behavior (hidden files
indexed / pruned through ``import_sources``) is integration-gated by
``tests/integration/test_importer_include_hidden.py``.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from app.rag.importer import _include_hidden_for_root, iter_importable_files
#: Explicit extension set (not the A9 config default) — the tests pin
#: the walk rules, not the config.
EXTS = frozenset({".md"})
@pytest.fixture()
def tree(tmp_path: Path) -> Path:
"""The task's fixture layout (plus the no-extension ``.env``).
visible.md .hidden/note.md .notes.md
.venv/junk.md node_modules/x.md .hidden/.deep.md
keep/ok.md .env
"""
root = tmp_path / "HiddenFix"
(root / ".hidden").mkdir(parents=True)
(root / ".venv").mkdir()
(root / "node_modules").mkdir()
(root / "keep").mkdir()
(root / "visible.md").write_text("visible\n", encoding="utf-8")
(root / ".hidden" / "note.md").write_text("note\n", encoding="utf-8")
(root / ".notes.md").write_text("notes\n", encoding="utf-8")
(root / ".venv" / "junk.md").write_text("junk\n", encoding="utf-8")
(root / "node_modules" / "x.md").write_text("x\n", encoding="utf-8")
(root / ".hidden" / ".deep.md").write_text("deep\n", encoding="utf-8")
(root / "keep" / "ok.md").write_text("ok\n", encoding="utf-8")
(root / ".env").write_text("SECRET=1\n", encoding="utf-8")
return root
def _rels(paths: list[Path], root: Path) -> set[str]:
return {p.relative_to(root).as_posix() for p in paths}
# --- default (flag False): today's behavior, byte-identical -----------------
def test_iter_default_flag_skips_all_hidden(tree: Path) -> None:
# A4: the default is the pre-phase-105 walk — hidden dir, hidden
# file, and the excluded dirs are all absent. Sorted order pinned
# verbatim (not just the set).
got = iter_importable_files(tree, EXTS)
assert got == [tree / "keep" / "ok.md", tree / "visible.md"]
def test_iter_default_flag_rejects_explicit_false(tree: Path) -> None:
assert iter_importable_files(tree, EXTS, include_hidden=False) == [
tree / "keep" / "ok.md",
tree / "visible.md",
]
# --- flag True (A1) -----------------------------------------------------------
def test_iter_flag_on_admits_hidden_dirs_and_hidden_files(tree: Path) -> None:
got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree)
assert got == {
".hidden/note.md", # inside a hidden dir
".notes.md", # hidden file with an importable extension
".hidden/.deep.md", # both at once
"visible.md",
"keep/ok.md",
}
def test_iter_flag_on_keeps_excluded_dirs_out(tree: Path) -> None:
# A1: EXCLUDED_DIRS are skipped in BOTH states — caches are never
# content, even with the flag on.
got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree)
assert ".venv/junk.md" not in got
assert "node_modules/x.md" not in got
def test_iter_flag_on_keeps_extension_filter_in_force(tree: Path) -> None:
# ``.env`` has no A9 extension — the extension filter is the real
# content gate, so a secret-flavoured file is never listed.
for flag in (False, True):
got = _rels(iter_importable_files(tree, EXTS, include_hidden=flag), tree)
assert ".env" not in got
# --- composition with the phase-89 ignore tuple -------------------------------
def test_iter_flag_on_ignore_composes_additively(tree: Path) -> None:
# The ignored prefix still bites when the flag is ON; everything
# else hidden is admitted.
got = _rels(
iter_importable_files(tree, EXTS, include_hidden=True, ignore=(".hidden",)),
tree,
)
assert got == {".notes.md", "visible.md", "keep/ok.md"}
def test_iter_ignore_only_flag_off(tree: Path) -> None:
# The pre-phase-105 combination still works untouched: ignore
# prefix on a visible subtree, flag at its default.
got = _rels(iter_importable_files(tree, EXTS, ignore=("keep",)), tree)
assert got == {"visible.md"}
# --- _include_hidden_for_root: the single read point --------------------------
def test_include_hidden_for_root_none_map_is_false(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, None) is False
def test_include_hidden_for_root_unlisted_root_is_false(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, {"/other/root": True}) is False
def test_include_hidden_for_root_listed_true_is_true(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, {str(root): True}) is True
def test_include_hidden_for_root_listed_false_is_false(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, {str(root): False}) is False
def test_include_hidden_for_root_keys_by_str_root(tmp_path: Path) -> None:
# Two distinct Path objects whose ``str()`` agrees — the answer must
# be the same (the ``_ignore_for_root`` keying convention, phase 89).
key = str(tmp_path / "src")
a = tmp_path / "src"
b = tmp_path / "src" / "." # different object, same str()
assert str(a) == str(b) == key
mp = {key: True}
assert _include_hidden_for_root(a, mp) is True
assert _include_hidden_for_root(b, mp) is True
tests/unit/test_models.py
+36
View File
@@ -134,6 +134,42 @@ def test_git_source_python_default_empty_list() -> None:
assert col.default.arg(None) == [], "the default must resolve to []"
def test_git_sources_include_hidden_column_contract() -> None:
"""Phase 105: every source row carries its hidden-folders flag —
BOOLEAN, NOT NULL, server default ``false`` (a pre-phase-105 row
reads ``False``, so every existing source imports byte-identically
until the owner flips it — A4)."""
sources = Base.metadata.tables["git_sources"]
assert "include_hidden" in sources.c, (
"git_sources must have the include_hidden column (phase 105)"
)
col = sources.c["include_hidden"]
assert col.nullable is False, "git_sources.include_hidden must be NOT NULL"
sd = col.server_default
assert isinstance(sd, DefaultClause), "include_hidden needs a server default"
assert isinstance(sd.arg, TextClause), (
"the server default must be the literal SQL text false"
)
assert sd.arg.text == "false", "include_hidden server default must be false"
def test_git_source_python_default_include_hidden_false() -> None:
"""A freshly constructed row (no DB) resolves the flag to ``False``
via the Python-side default (``default=False``) — the ORM INSERT-time
default, so an ORM insert that omits the column inserts ``false``
rather than NULL (the server default ``false`` independently covers
non-ORM inserts — A4)."""
from app.models import GitSource
row = GitSource(url="https://example.com/r.git", kind="git")
col = row.__table__.c["include_hidden"]
assert col.default is not None, (
"include_hidden needs a Python-side (INSERT-time) default"
)
# A scalar default: the arg IS the value (not a callable).
assert col.default.arg is False, "the default must resolve to False"
def test_doc_drafts_column_contract() -> None:
"""Phase 59: the editable triple (title/path/body) + status +
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
tests/unit/test_sync_button.py
+1
View File
@@ -733,6 +733,7 @@ class _GatedImport:
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
) -> ImportSummary:
self.prune_flags.append(prune)
if progress is not None: