diff --git a/.agents/phases/todo/105_hidden_folders_toggle/00_phase.md b/.agents/phases/complete/105_hidden_folders_toggle/00_phase.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/00_phase.md rename to .agents/phases/complete/105_hidden_folders_toggle/00_phase.md diff --git a/.agents/phases/todo/105_hidden_folders_toggle/01_include_hidden_column.md b/.agents/phases/complete/105_hidden_folders_toggle/01_include_hidden_column.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/01_include_hidden_column.md rename to .agents/phases/complete/105_hidden_folders_toggle/01_include_hidden_column.md diff --git a/.agents/phases/todo/105_hidden_folders_toggle/02_importer_include_hidden.md b/.agents/phases/complete/105_hidden_folders_toggle/02_importer_include_hidden.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/02_importer_include_hidden.md rename to .agents/phases/complete/105_hidden_folders_toggle/02_importer_include_hidden.md diff --git a/.agents/phases/todo/105_hidden_folders_toggle/03_include_hidden_api.md b/.agents/phases/complete/105_hidden_folders_toggle/03_include_hidden_api.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/03_include_hidden_api.md rename to .agents/phases/complete/105_hidden_folders_toggle/03_include_hidden_api.md diff --git a/.agents/phases/todo/105_hidden_folders_toggle/04_include_hidden_pipelines.md b/.agents/phases/complete/105_hidden_folders_toggle/04_include_hidden_pipelines.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/04_include_hidden_pipelines.md rename to .agents/phases/complete/105_hidden_folders_toggle/04_include_hidden_pipelines.md diff --git a/.agents/phases/todo/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md b/.agents/phases/complete/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md rename to .agents/phases/complete/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md diff --git a/.agents/phases/todo/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md b/.agents/phases/complete/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md similarity index 100% rename from .agents/phases/todo/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md rename to .agents/phases/complete/105_hidden_folders_toggle/06_e2e_hidden_folders_toggle.md diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.err b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.md b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.md new file mode 100644 index 0000000..2956b8f --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.md @@ -0,0 +1,16 @@ +All completion criteria verified. Everything is green. + +**Phase 105 final verification pass — all criteria verified** + +- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page +- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified) +- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓ +- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation) +- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed +- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings) + +**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched). + +**Deviations:** none — no defects found; no code changes were needed on this pass. + +**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`. diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.validate b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.validate new file mode 100644 index 0000000..fcced81 --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__00_phase.a1.validate @@ -0,0 +1,97 @@ +........................................................................ [ 3%] +........................................................................ [ 6%] +........................................................................ [ 10%] +........................................................................ [ 13%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 46%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 56%] +........................................................................ [ 60%] +........................................................................ [ 63%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 73%] +........................................................................ [ 77%] +........................................................................ [ 80%] +........................................................................ [ 83%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 97%] +............................................................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 181 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 94 0 100% +app/api/docs.py 134 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 117 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 140 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 21 0 100% +app/main.py 66 0 100% +app/models.py 126 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 316 1 99% +app/rag/archive_upload.py 128 0 100% +app/rag/chunker.py 206 4 98% +app/rag/folder_summaries.py 121 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 200 3 98% +app/rag/llm.py 243 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 88 0 100% +app/rag/retriever.py 150 3 98% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 311 0 100% +-------------------------------------------------- +TOTAL 3879 15 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.err b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.md b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.md new file mode 100644 index 0000000..2e69904 --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.md @@ -0,0 +1,18 @@ +All gates green. Task 01 is complete: + +**Implemented** +- `app/models.py`: `GitSource.include_hidden` — `BOOLEAN NOT NULL`, `server_default=text("false")`, `default=False`, A1/A4 provenance docstring; module-header `git_sources` inventory updated (phase 105). +- `alembic/versions/0019_git_source_include_hidden.py`: additive reversible column (revises 0018); dev/test DB now at `0019 (head)`. +- Unit pins in `tests/unit/test_models.py`: column contract (type/NOT NULL/server default) + Python-side default resolves to `False`. +- New `tests/integration/test_migration_0019.py` (4 tests): 0018→0019 upgrade contract, pre-0019 backfill `false`, omitted-column server default, explicit-`true` round-trip (SQL + ORM fresh-session), downgrade drops column with rows surviving, round-trip upgrade. + +**Results** +- `uv run pytest tests/unit/ -q` → exit 0 +- `uv run alembic upgrade head` → `0018 -> 0019` applied, `alembic current` = `0019 (head)` +- `uv run pytest tests/integration/test_migration_0019.py -v --no-cov` → 4 passed +- `uv run pytest --cov=app --cov-report=term-missing` → 2108 passed, TOTAL **99%** (>90%) +- `uv run ruff check .` → clean; `uv run pyright` → 0 errors (fixed one `reportOptionalMemberAccess` in the new test) + +**Decisions/deviations**: none — followed the 0013/0018 house migration format and the 0007/0018 migration-test patterns verbatim; no existing suite asserted the full `git_sources` column set, so no expectation lists needed updating. + +**Next pending task**: `.agents/phases/todo/105_hidden_folders_toggle/02_importer_include_hidden.md` (importer `iter_importable_files`/`import_sources` flag support). diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.validate b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.validate new file mode 100644 index 0000000..11d78ab --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__01_include_hidden_column.a1.validate @@ -0,0 +1,97 @@ +........................................................................ [ 3%] +........................................................................ [ 6%] +........................................................................ [ 10%] +........................................................................ [ 13%] +........................................................................ [ 17%] +........................................................................ [ 20%] +........................................................................ [ 23%] +........................................................................ [ 27%] +........................................................................ [ 30%] +........................................................................ [ 34%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 44%] +........................................................................ [ 47%] +........................................................................ [ 51%] +........................................................................ [ 54%] +........................................................................ [ 58%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 99%] +.................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 181 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 94 0 100% +app/api/docs.py 134 1 99% +app/api/git_sources.py 229 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 115 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 140 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 21 0 100% +app/main.py 66 0 100% +app/models.py 126 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 316 1 99% +app/rag/archive_upload.py 128 0 100% +app/rag/chunker.py 206 4 98% +app/rag/folder_summaries.py 121 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 197 3 98% +app/rag/llm.py 243 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 88 0 100% +app/rag/retriever.py 150 3 98% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 307 0 100% +-------------------------------------------------- +TOTAL 3867 15 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.err b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.md b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.md new file mode 100644 index 0000000..5a9e9b7 --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.md @@ -0,0 +1,20 @@ +All gates green. Task 02 is complete. + +## Report + +**Implemented (task 02 — importer flag support, phase 105):** +- `app/rag/importer.py::iter_importable_files` — new `include_hidden: bool = False` param; skip check is now `(not include_hidden and part.startswith(".")) or part in excluded` — byte-identical at the default; docstring updated (A1 flag clause, ignore composes additively). +- `app/rag/importer.py::import_sources` — new kw-only `include_hidden_by_root: dict[str, bool] | None` (same `str(root)` keying as `ignore_by_root`); new `_include_hidden_for_root` helper is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop; module docstring "Scope" + `import_sources` docstring updated. +- `tests/unit/test_importer_include_hidden.py` (new, 12 tests): default walk pinned byte-identically (sorted verbatim), flag-ON admits hidden dirs + hidden files, `EXCLUDED_DIRS`/extension filter stay in force, ignore-tuple composition, `_include_hidden_for_root` pins incl. `str(root)` keying. +- `tests/integration/test_importer_include_hidden.py` (new, 5 tests): default run → no hidden rows/embeds; flag ON → hidden md+yaml indexed, yaml summarized (phase-30 path), `.venv` excluded; flag-off re-run prunes 2 hidden docs (A2); progress `total` agrees in both states; unlisted root stays hidden. +- Fixed one pre-existing regression surfaced by the signature change: the phase-89 counting sentinel in `tests/unit/test_importer.py::test_no_progress_means_no_prewalk` now accepts/forwards `include_hidden` (same drop-in pattern as its `ignore` handling). + +**Results:** +- `uv run pytest tests/unit/test_importer_include_hidden.py tests/integration/test_importer_include_hidden.py tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -q` → 44 passed +- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate) +- `uv run pytest` → **2125 passed** +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors + +**Decisions:** none — followed the task design verbatim; no deviations. + +**Next pending task:** `.agents/phases/todo/105_hidden_folders_toggle/03_include_hidden_api.md` diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.validate b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.validate new file mode 100644 index 0000000..3eb93df --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__02_importer_include_hidden.a1.validate @@ -0,0 +1,97 @@ +........................................................................ [ 3%] +........................................................................ [ 6%] +........................................................................ [ 10%] +........................................................................ [ 13%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 23%] +........................................................................ [ 27%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 44%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 60%] +........................................................................ [ 64%] +........................................................................ [ 67%] +........................................................................ [ 71%] +........................................................................ [ 74%] +........................................................................ [ 77%] +........................................................................ [ 81%] +........................................................................ [ 84%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 94%] +........................................................................ [ 98%] +..................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 181 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 94 0 100% +app/api/docs.py 134 1 99% +app/api/git_sources.py 229 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 115 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 140 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 21 0 100% +app/main.py 66 0 100% +app/models.py 126 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 316 1 99% +app/rag/archive_upload.py 128 0 100% +app/rag/chunker.py 206 4 98% +app/rag/folder_summaries.py 121 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 200 3 98% +app/rag/llm.py 243 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 88 0 100% +app/rag/retriever.py 150 3 98% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 307 0 100% +-------------------------------------------------- +TOTAL 3870 15 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.err b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.md b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.md new file mode 100644 index 0000000..469f4b6 --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.md @@ -0,0 +1,16 @@ +All green. Task 03 complete. + +**Implemented** +- `app/schemas.py`: `include_hidden` added to `GitSourceIn` (optional, absent → `False`), `GitSourceOut` and `GitSourceRow` (env rows report `False`); `GitSourceIgnoreIn` renamed → `GitSourcePatchIn` with two optional present-wins fields +- `app/api/git_sources.py`: GET reports stored flag (DB rows) / `false` (env rows); POST stores `bool(payload.include_hidden)` for both kinds; PATCH now applies each present field independently (list replace keeps phase-89 A5 semantics + fixed 422s, flag set, both-absent → 200 no-op, 404 unchanged); router + endpoint docstrings updated +- `tests/integration/test_git_sources_api.py`: extended — env/row shape pins gain `include_hidden`, anonymous 403 extended to bool-only body, retired `{}`-422 pin replaced with no-op 200, new phase-105 matrix (GET default false, POST both kinds, bool-only, list-only, both, 422-never-half-applies) + +**Results** +- `uv run pytest tests/integration/test_git_sources_api.py -q` → 39 passed +- `uv run pytest --cov=app --cov-report=term-missing` → 2131 passed, TOTAL 99% (`app/api/git_sources.py` 100%, `app/schemas.py` 100%) +- `uv run ruff check . && uv run pyright` → clean (0 errors, 0 warnings) +- `grep GitSourceIgnoreIn` repo-wide → zero hits (rename verified) + +**Decisions** — `{}` PATCH body changed 422→200 no-op per the locked task-03 contract (both fields now optional); existing dialog client (always sends list) stays byte-identical. + +**Next pending task** — `105_hidden_folders_toggle/04_include_hidden_pipelines.md` diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.validate b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.validate new file mode 100644 index 0000000..cc899f6 --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__03_include_hidden_api.a1.validate @@ -0,0 +1,97 @@ +........................................................................ [ 3%] +........................................................................ [ 6%] +........................................................................ [ 10%] +........................................................................ [ 13%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 23%] +........................................................................ [ 27%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 60%] +........................................................................ [ 64%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 74%] +........................................................................ [ 77%] +........................................................................ [ 81%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 181 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 94 0 100% +app/api/docs.py 134 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 115 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 140 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 21 0 100% +app/main.py 66 0 100% +app/models.py 126 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 316 1 99% +app/rag/archive_upload.py 128 0 100% +app/rag/chunker.py 206 4 98% +app/rag/folder_summaries.py 121 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 200 3 98% +app/rag/llm.py 243 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 88 0 100% +app/rag/retriever.py 150 3 98% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 311 0 100% +-------------------------------------------------- +TOTAL 3877 15 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.err b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.md b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.md new file mode 100644 index 0000000..adbf8bc --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.md @@ -0,0 +1,17 @@ +All completion criteria for task 04 are met and every gate is green. + +**Report — Task 04 (105_hidden_folders_toggle): pipelines honor the per-row flag** + +- `app/api/sync.py::_run_sync` — builds `include_hidden_by_root` in the existing per-row loop (same `str(root)` keying, OR on shared-root collision — the ignore-union's boolean mirror) and passes it to `import_sources`; module docstring step 4 extended. +- `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple on all three return paths (manual `--source` → `{}, {}`; rows → per-root flag map; legacy fallback → `{}, {}`); `main` unpacks and passes `include_hidden_by_root=` to `import_sources`; module + function docstrings extended. +- `scripts/load_test_kb.py` and the upload run untouched (defaults / no scan). +- Tests: `test_sync_api.py` — real-import test (same local row: flag off → hidden file has no `documents` row, `files=1`; flipped on → hidden file indexed, `files=2`) + fake-import wiring test pinning the OR-collision map; `test_import_docs_git.py` — fake-import wiring test (flag True → `{root: True}`; default row → `{root: False}`), two real-import CLI tests (flag on indexes `.hidden/note.md` `added=2`; off skips it `added=1`), and all 8 existing `_resolve_sources`/`main` tests updated to the 3-tuple with flag-map assertions; fakes in `test_sync_button.py`/`test_sync_api.py`/`test_import_docs_git.py` gained the new kwarg. + +Results: +- `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q` → 43 passed +- `uv run pytest --cov=app --cov-report=term-missing` → 2136 passed, TOTAL **99%** (>90%; `app/api/sync.py` 100%) +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings + +Decisions: flag-off rows contribute an explicit `{root: False}` entry (per the task's unconditional OR-assignment snippet) — pinned by tests in both suites. + +Next pending task: `.agents/phases/todo/105_hidden_folders_toggle/05_hidden_toggle_sources_ui.md` diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.validate b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.validate new file mode 100644 index 0000000..54f515b --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__04_include_hidden_pipelines.a1.validate @@ -0,0 +1,97 @@ +........................................................................ [ 3%] +........................................................................ [ 6%] +........................................................................ [ 10%] +........................................................................ [ 13%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 60%] +........................................................................ [ 64%] +........................................................................ [ 67%] +........................................................................ [ 70%] +........................................................................ [ 74%] +........................................................................ [ 77%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 94%] +........................................................................ [ 97%] +................................................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 181 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 94 0 100% +app/api/docs.py 134 1 99% +app/api/git_sources.py 232 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 117 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 140 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 21 0 100% +app/main.py 66 0 100% +app/models.py 126 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 316 1 99% +app/rag/archive_upload.py 128 0 100% +app/rag/chunker.py 206 4 98% +app/rag/folder_summaries.py 121 0 100% +app/rag/git_sources.py 14 0 100% +app/rag/importer.py 200 3 98% +app/rag/llm.py 243 1 99% +app/rag/overview.py 71 0 100% +app/rag/prompts.py 88 0 100% +app/rag/retriever.py 150 3 98% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 24 0 100% +app/schemas.py 311 0 100% +-------------------------------------------------- +TOTAL 3879 15 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.err b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.md b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.md new file mode 100644 index 0000000..bcf0346 --- /dev/null +++ b/.agents/reports/105_hidden_folders_toggle/105_hidden_folders_toggle__05_hidden_toggle_sources_ui.a1.md @@ -0,0 +1,14 @@ +All green. Task 05 complete. + +**Report** + +- **index.html**: page-level ` + diff --git a/scripts/import_docs.py b/scripts/import_docs.py index 076546f..95df44c 100644 --- a/scripts/import_docs.py +++ b/scripts/import_docs.py @@ -29,7 +29,11 @@ precedence order: Phase 89: resolution also returns each row's ignore paths, keyed by the resolved root string (the importer normalizes them); manual ``--source`` dirs and the legacy fallback have no rows, so they import -with no ignore. +with no ignore. Phase 105 extends the same resolution with each row's +``include_hidden`` flag — a second per-root map keyed by the same root +strings (the importer reads it per root); manual ``--source`` dirs and +the legacy fallback have no rows, so they import with the empty map +(hidden paths skipped — A4). Imported formats (PLAN anchor A9, revised; phase 56): the A9 family by default — ``md, markdown, txt, yaml, yml, json, py`` plus the quadlet @@ -163,9 +167,10 @@ def repo_name(url: str) -> str: def _resolve_sources( cli_sources: list[Path] | None, settings: Settings -) -> tuple[list[Path], dict[str, list[str]]]: +) -> tuple[list[Path], dict[str, list[str]], dict[str, bool]]: """Resolve the directories to import (phase 28, extended in phases - 35 and 38; per-root ignore maps, phase 89). + 35 and 38; per-root ignore maps, phase 89; per-root hidden-folders + flag maps, phase 105). Precedence: ``--source`` (explicit manual paths — always wins) > the effective sources — the ``git_sources`` DB rows (git + local), @@ -178,18 +183,23 @@ def _resolve_sources( stored directory, re-verified ``.is_dir()`` at run time) > the legacy ``DEFAULT_SOURCES``. - Returns ``(sources, ignore_by_root)`` (phase 89): the map is keyed - by the resolved root string, exactly as the importer sees it (two - rows sharing a root string get the union — extend, not replace); - manual ``--source`` dirs and the legacy fallback have no rows, so - they import with an empty map (no ignore). + Returns ``(sources, ignore_by_root, include_hidden_by_root)`` + (phase 89; phase 105 adds the per-root flag map — the flag is + stored per row, manual ``--source`` dirs and the legacy fallback + have no rows and import with the empty map: hidden paths skipped, + A4): both maps are keyed by the resolved root string, exactly as + the importer sees it (two rows sharing a root string get the union + — extend, not replace — for the ignore lists, and the OR of their + flags for the hidden map); manual ``--source`` dirs and the legacy + fallback have no rows, so they import with empty maps (no ignore, + hidden skipped). A :class:`GitSyncError` from a failing clone/pull — or a missing local directory (``local source missing: ``) — propagates to :func:`main`, which aborts the run before importing anything. """ if cli_sources: - return [path.expanduser() for path in cli_sources], {} + return [path.expanduser() for path in cli_sources], {}, {} db = SessionLocal() try: rows, origin = effective_sources(db) @@ -204,6 +214,7 @@ def _resolve_sources( sources_root = Path(settings.sources_dir).expanduser() sources: list[Path] = [] ignore_by_root: dict[str, list[str]] = {} + include_hidden_by_root: dict[str, bool] = {} for row in rows: if row.kind == "git": root = clone_or_pull(row.url, sources_root / repo_name(row.url)) @@ -223,8 +234,16 @@ def _resolve_sources( # edge. if row.ignore_paths: ignore_by_root.setdefault(str(root), []).extend(row.ignore_paths) - return sources, ignore_by_root - return [path.expanduser() for path in DEFAULT_SOURCES], {} + # Phase 105 (A1/A4): the row's hidden-folders flag, keyed by + # the SAME root string the importer sees; a shared-root + # collision ORs — if EITHER row says "index hidden", the + # root does (the ignore-map union's boolean mirror). + include_hidden_by_root[str(root)] = ( + include_hidden_by_root.get(str(root), False) + or bool(row.include_hidden) + ) + return sources, ignore_by_root, include_hidden_by_root + return [path.expanduser() for path in DEFAULT_SOURCES], {}, {} def _overview_row_exists() -> bool: @@ -264,9 +283,13 @@ def main(argv: list[str] | None = None) -> int: # Git sources resolve (and clone/pull) *before* any import: a failing # repo aborts the run with a non-zero exit, naming the failure — a bad # URL must never silently import partial junk. The second element is - # the phase-89 per-root ignore map (empty for manual/fallback paths). + # the phase-89 per-root ignore map and the third the phase-105 + # per-root hidden-folders flag map (both empty for manual/fallback + # paths). try: - sources, ignore_by_root = _resolve_sources(args.source, settings) + sources, ignore_by_root, include_hidden_by_root = _resolve_sources( + args.source, settings + ) except GitSyncError as e: print(f"import_docs: source sync failed: {e}", file=sys.stderr) return 1 @@ -332,6 +355,7 @@ def main(argv: list[str] | None = None) -> int: summary = await import_sources( sources, llm, prune=args.prune, limit=args.limit, ignore_by_root=ignore_by_root, + include_hidden_by_root=include_hidden_by_root, ) if args.limit is not None: # An incomplete walk is debug-only — it must never advance diff --git a/tests/e2e/test_hidden_folders_toggle.py b/tests/e2e/test_hidden_folders_toggle.py new file mode 100644 index 0000000..0823769 --- /dev/null +++ b/tests/e2e/test_hidden_folders_toggle.py @@ -0,0 +1,676 @@ +"""Phase 105 story E2E (Playwright): the per-source hidden-folders toggle — +the "Hidden" checkbox next to the "Ignore paths" button makes a source's +dot-prefixed paths indexable (``/git-sources.html``). + +Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-14 — +TODO-derived, no separate user-story file). +Run in isolation (DB must be up: ``podman compose up -d db``): + + uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov + +The story gate proves the whole item end to end through the REAL page + +REAL API + REAL sync pipeline (the in-app Sync's background task, mock +LLM). **No git, no network** — the suite's single source is a local +directory row (``kind="local"``, phase 38) over a fixture tree with one +visible file, one hidden-folder file (the phase-105 subject), and one +``.venv`` file (``EXCLUDED_DIRS`` — never indexed in either state, A1). +The module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL +(the ``test_source_ignore_paths.py`` module-env pattern) that is NEVER +synced or cloned: every sync in this suite runs with the local DB row in +place (DB rows win over the env fallback), and the one env-fallback test +never triggers a sync. + +Contract under test: + +* anonymous: the ``#git-sources-gate`` sign-in gate, the manager hidden, + NO ``/api/git-sources`` call on load, 403 on + ``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}`` + with a bool-only body (the phase-89 anonymous pin extended to the + toggle payload); +* A4: with the default row, a sync indexes ``visible.md`` ONLY — + ``detail.files`` counts one, ``.hidden/note.md`` has no ``documents`` + row, the checkbox renders UNCHECKED and no "hidden on" tag; +* A1: flipping the checkbox on (the real click) → the PATCH 200 lands + (the "hidden on" tag appears, the announcer ``role=status`` fires the + confirmation AFTER the reload line) → sync → ``.hidden/note.md`` IS + indexed (``documents`` row present; the KB catalog lists it — the + tree/catalog is DB-driven, no extra surface) and ``.venv/junk.md`` is + STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders + CHECKED (server state); +* A2: flipping it OFF (real click) → sync → ``detail.pruned`` includes + the hidden doc, the catalog no longer lists it, the tag is gone; +* A3: the env-fallback view (table-empty state) renders the "from .env" + tag with NO Hidden checkbox and NO "Ignore paths" button; +* a11y + error surface: the checkbox has a full accessible name + containing the source location (``Index hidden folders for local + source: …``), is keyboard-focusable, the tag text is "hidden on" + (never color alone); ``#git-sources-hidden-error`` exists with + ``role="alert"`` and stays ``hidden`` through the happy path — and a + network-aborted PATCH shows the canned "not changed" message and + reverts the box to the server state (§7.4 never-stale). + +Test → contract mapping (one test per bullet, the phase-89 suite's +shape): +1. ``test_anonymous_gate_and_403s`` +2. ``test_hidden_off_by_default`` +3. ``test_toggle_on_indexes_hidden_folders`` +4. ``test_toggle_off_prunes_hidden`` +5. ``test_env_fallback_rows_have_no_toggle`` +6. ``test_toggle_a11y_and_error_surface`` +""" +from __future__ import annotations + +import os +import subprocess +import sys +import time +import uuid +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest +from playwright.sync_api import Page, expect +from sqlalchemy import text + +from app.db import SessionLocal +from e2e.auth_helpers import login +from e2e.conftest import ( + ADMIN_PASSWORD, + SESSION_SECRET, + USE_REAL_LLM, + _wait_http, +) + +REPO = Path(__file__).resolve().parents[2] + +# Phase 79 (task 04, full inventory): the conftest session app owns its +# port in a combined run — this module app binds its own port instead +# (a same-port second uvicorn dies on bind and would drive the wrong +# server). Env-overridable. +APP_PORT = int(os.environ.get("E2E_APP_PORT_HIDDEN", "8141")) +APP_URL = f"http://127.0.0.1:{APP_PORT}" + +GIT_SOURCES_URL = "/git-sources.html" + +#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is +#: NEVER cloned (the env-fallback test reads it; the sync tests all run +#: with the local DB row in place, so the env list never reaches a +#: clone). +ENV_SOURCE = "https://github.com/reese/env-alpha.git" + +#: The three fixture files (source-relative POSIX paths — exactly the +#: strings ``documents.path`` stores). +VISIBLE_MD = "visible.md" +HIDDEN_NOTE = ".hidden/note.md" +VENV_JUNK = ".venv/junk.md" + +#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the +#: test_sync_button.py polling idiom) — real import of ≤2 small files +#: against the mock LLM; generous budget. +SYNC_TIMEOUT_S = 60.0 +SYNC_TICK_S = 2.0 + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path: + """The local source's on-disk tree (task 06 step 1). + + Module-scoped (the same reasoning as ``test_sync_button.py``'s + module-scoped fixture note): ``tmp_path`` is function-scoped while + the module-scoped app + the seeded row reference the dir for the + module's lifetime, so it is built under ``tmp_path_factory`` (the + same pytest-managed temp area, module-safe). Plain files — no git: + a ``kind="local"`` row is walked directly by the sync (phase 38). + """ + root = tmp_path_factory.mktemp("hidden_src") + (root / ".hidden").mkdir() + (root / ".venv").mkdir() + (root / VISIBLE_MD).write_text( + "# Visible\n" + "\n" + "The visible doc — indexed in EVERY state (flag on and off).\n", + encoding="utf-8", + ) + # The phase-105 subject: a file INSIDE a hidden (dot-prefixed) + # folder — indexed only when the row's flag is on (A1). + (root / HIDDEN_NOTE).write_text( + "# Hidden note\n" + "\n" + "A file inside a hidden (dot) folder — the toggle's subject:\n" + "indexed, embedded, and summarized like any visible file when\n" + "the flag is on; pruned from the KB when it flips off (A2).\n", + encoding="utf-8", + ) + # EXCLUDED_DIRS content: ``.venv`` is a cache, never content — + # skipped in BOTH flag states (A1). + (root / VENV_JUNK).write_text( + "venv junk — EXCLUDED_DIRS content: never indexed, both states.\n", + encoding="utf-8", + ) + return root + + +@pytest.fixture(scope="module") +def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]: + """The real app under test — per-module env (the conftest pattern, + module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env + fallback's subject — never cloned), module-scratch checkouts/ + upload dirs, the mock LLM. No git anywhere in this suite's path.""" + env = dict(os.environ) + env.pop("DEBUGPY", None) + env["BOR_ENVIRONMENT"] = "e2e" + env["BOR_STATIC_DIR"] = str(REPO / "frontend") + env["BOR_LLM_BASE_URL"] = ( + "https://aipi.reeseapps.com/v1" + if USE_REAL_LLM + else f"http://127.0.0.1:{mock_llm}/v1" + ) + # Mock-calibrated threshold (conftest pattern) — no chat turn is + # ever sent in this suite, but the app boots with the same env + # shape. + env["BOR_RELEVANCE_THRESHOLD"] = "0.30" + env.setdefault( + "BOR_DATABASE_URL", + "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", + ) + # Phase 16: admin auth must be set or create_app() refuses to boot. + env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD + env["BOR_SESSION_SECRET"] = SESSION_SECRET + # The empty-table env fallback (only read while ``git_sources`` is + # EMPTY — the env-fallback test; never cloned in this suite). + env["BOR_GIT_SOURCES"] = ENV_SOURCE + # Module-scratch dirs (never reached by this suite's local-row + # syncs — kept explicit so a shared checkouts dir cannot leak rows + # into the walk). + scratch = source_dir.parent + env["BOR_SOURCES_DIR"] = str(scratch / "checkouts") + env["BOR_UPLOAD_DIR"] = str(scratch / "uploads") + proc = subprocess.Popen( + [sys.executable, "-m", "uvicorn", "app.main:app", + "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], + cwd=REPO, + env=env, + ) + try: + _wait_http(f"{APP_URL}/api/health") + yield APP_URL + finally: + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + + +@pytest.fixture(scope="module") +def app_url(app_server: str) -> str: + return app_server + + +def _truncate_all() -> None: + """Fresh registry + KB per test (the E2E isolation pattern, the + test_sync_button.py module-env DSN): the sync's counts and the + catalog must be each test's own doing.""" + with SessionLocal() as db: + db.execute( + text( + "TRUNCATE chunks, documents, query_log, kb_overview, " + "git_sources" + ) + ) + db.commit() + + +@pytest.fixture(autouse=True) +def _clean(db_ready: None) -> Iterator[None]: + """This suite owns ``git_sources`` AND the KB tables (the E2E + isolation pattern): suites run in isolation but share one + Postgres, and a leftover row would flip the app from the + ``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover + document would skew the prune counts). Empty BOTH before and + after every test.""" + _truncate_all() + yield + _truncate_all() + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _admin_git_sources_page(page: Page, app_url: str) -> None: + """Real form login landing on the git sources page (admin settled: + Sign out visible, the manager revealed by the page module).""" + login(page, app_url, next=GIT_SOURCES_URL) + expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000) + expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000) + expect(page.locator("#git-sources-gate")).to_be_hidden() + expect(page.locator("#git-sources-content")).to_be_visible() + + +def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str: + """(Re)seed the stored row for the module fixture dir via the REAL + admin API (201) — ``kind="local"`` — with NO ``ignore_paths`` and + NO ``include_hidden`` (A4: absent at create → stored ``False``) — + then reload the view so the page's table re-fetches the row (the + API seed is server-side: the mounted view does not know about it + without a re-show). Returns the stored (expanded) path — the + row's ``url`` location value, the same string the page renders + and aria-labels.""" + r = page.request.post( + f"{app_url}/api/git-sources", + data={"kind": "local", "path": str(source_dir)}, + ) + assert r.status == 201, r.text + body = r.json() + assert body["ignore_paths"] == [] # the default list, round-tripped + assert body["include_hidden"] is False # A4: default off + page.reload() + _admin_git_sources_page(page, app_url) + return body["url"] + + +def run_sync(page: Page, app_url: str) -> dict[str, Any]: + """``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal + state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s + budget). Returns the terminal status body.""" + r = page.request.post(f"{app_url}/api/sync") + assert r.status == 202, r.text + deadline = time.monotonic() + SYNC_TIMEOUT_S + body: dict[str, Any] = {} + while True: + s = page.request.get(f"{app_url}/api/sync/status") + assert s.status == 200, s.text + body = s.json() + if body["state"] in ("success", "failed"): + return body + assert time.monotonic() < deadline, ( + f"sync did not reach a terminal state: {body}" + ) + time.sleep(SYNC_TICK_S) + + +def _catalog_paths(page: Page, app_url: str) -> list[str]: + """The RAG catalog's data source (``GET /api/docs`` — the Sources + page's table): every ``documents.path`` (source-relative POSIX). + The tree/catalog is DB-driven — a newly indexed hidden document + appears here automatically (no extra surface).""" + r = page.request.get(f"{app_url}/api/docs") + assert r.status == 200, r.text + return [d["path"] for d in r.json()["documents"]] + + +def _db_document_paths() -> list[str]: + """Every ``documents.path`` straight from the DB — the literal + "no ``documents`` row" assertion (the catalog is the same table, + read through the API).""" + with SessionLocal() as db: + rows = db.execute(text("SELECT path FROM documents")).fetchall() + return [row[0] for row in rows] + + +def _row(page: Page, value: str) -> Any: + """The table row whose mono location cell shows ``value`` (re-resolved + on every call — the row re-renders from the server after each + successful toggle).""" + return page.locator("#git-sources-tbody tr", has_text=value) + + +def _hidden_box(page: Page, value: str) -> Any: + """The row's Hidden checkbox (fresh locator — the element is + re-created by the post-toggle re-render).""" + return _row(page, value).locator(".git-source-hidden-box") + + +def _flip_hidden(page: Page, value: str, wanted_on: bool) -> None: + """Click the row's Hidden checkbox and wait for the §7.4 success + lifecycle to settle: the PATCH 200 landed, the row re-rendered + from the server, and the CONFIRMATION is the last announcer message + (the reload's "N sources listed." cannot overwrite it — the + phase-89 order).""" + _hidden_box(page, value).click() + tag = _row(page, value).locator(".git-source-hidden-count") + if wanted_on: + expect(tag).to_have_text("hidden on", timeout=30_000) + else: + expect(tag).to_have_count(0, timeout=30_000) + expect( + page.locator("#git-sources-announcer") + ).to_have_text( + f"Hidden folders {'enabled' if wanted_on else 'disabled'} for {value}." + ) + + +# --------------------------------------------------------------------------- +# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. the +# bool-only toggle PATCH) +# --------------------------------------------------------------------------- + + +def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None: + page.set_default_timeout(30_000) + + # Track every /api/git-sources request the page itself makes — the + # gate must be reached WITHOUT touching the admin API (the + # test_source_ignore_paths.py pattern). + api_calls: list[str] = [] + page.on( + "request", + lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None, + ) + + page.goto(app_url + GIT_SOURCES_URL) + expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000) + + # The sign-in gate (the #sources-gate pattern, phase 16)… + gate = page.locator("#git-sources-gate") + expect(gate).to_be_visible() + expect(gate).to_contain_text("Sign in to manage the git sources") + expect( + gate.locator("a[href='/login.html?next=/git-sources.html']") + ).to_have_count(1) + # …and the manager is absent/inert: table + form + env note, all + # inside the hidden #git-sources-content. + expect(page.locator("#git-sources-content")).to_be_hidden() + expect(page.locator("#git-sources-table")).to_be_hidden() + expect(page.locator("#git-source-form")).to_be_hidden() + expect(page.locator("#git-sources-env-note")).to_be_hidden() + + # The gate never called the admin API… + assert api_calls == [], f"anonymous page called the git sources API: {api_calls}" + # …and the API 403s anonymous callers on GET/POST AND the + # phase-105 toggle PATCH with a bool-only body (the row's list is + # untouched — the toggle's exact payload is gated too). + assert page.request.get(f"{app_url}/api/git-sources").status == 403 + assert ( + page.request.post( + f"{app_url}/api/git-sources", data={"url": ENV_SOURCE} + ).status + == 403 + ) + assert ( + page.request.patch( + f"{app_url}/api/git-sources/{uuid.uuid4()}", + data={"include_hidden": True}, + ).status + == 403 + ) + + +# --------------------------------------------------------------------------- +# 2. A4: the default row is byte-identical to pre-phase-105 — the sync +# indexes visible.md ONLY; the checkbox renders UNCHECKED, no tag +# --------------------------------------------------------------------------- + + +def test_hidden_off_by_default( + page: Page, app_url: str, db_ready: None, source_dir: Path +) -> None: + page.set_default_timeout(30_000) + _admin_git_sources_page(page, app_url) + + stored_path = _seed_local_source(page, app_url, source_dir) + expect(page.locator("#git-sources-tbody tr")).to_have_count(1) + + # A4: the default row renders the NEW control in its default state — + # the checkbox UNCHECKED (checked comes only from server state) and + # NO "hidden on" tag (pre-phase-105 surface + the control). + box = _hidden_box(page, stored_path) + expect(box).to_have_count(1) + expect(box).not_to_be_checked() + expect(_row(page, stored_path).locator(".git-source-hidden-count")).to_have_count(0) + + # The real sync: ``visible.md`` ONLY — the walk skipped the + # dot-prefixed ``.hidden/`` component (A4 byte-identical default) + # AND ``.venv/`` (EXCLUDED_DIRS in both states). + sync = run_sync(page, app_url) + assert sync["state"] == "success", sync + detail = sync["detail"] + assert detail["files"] == 1, detail + assert detail["added"] == 1, detail + assert detail["pruned"] == 0, detail + assert detail["errors"] == 0, detail + + # The catalog holds exactly the visible file — no ``documents`` row + # for the hidden note (the direct DB read makes "no row" literal)… + paths = _catalog_paths(page, app_url) + assert VISIBLE_MD in paths, f"visible.md missing from the catalog: {paths}" + assert HIDDEN_NOTE not in paths, ( + f"hidden note leaked into the catalog with the flag off: {paths}" + ) + assert VENV_JUNK not in paths, f"EXCLUDED_DIRS leaked into the catalog: {paths}" + assert _db_document_paths() == [VISIBLE_MD] + + +# --------------------------------------------------------------------------- +# 3. A1: the real click flips the flag on — the PATCH 200 lands (tag + +# last-announce confirmation), the next sync indexes the hidden +# folder; EXCLUDED_DIRS stays excluded; the box re-renders CHECKED +# --------------------------------------------------------------------------- + + +def test_toggle_on_indexes_hidden_folders( + page: Page, app_url: str, db_ready: None, source_dir: Path +) -> None: + page.set_default_timeout(30_000) + _admin_git_sources_page(page, app_url) + stored_path = _seed_local_source(page, app_url, source_dir) + + # Baseline: the default sync indexes the visible file only. + first = run_sync(page, app_url) + assert first["state"] == "success", first + assert first["detail"]["files"] == 1, first["detail"] + + # A1: flip the checkbox on (the real click) — the §7.4 lifecycle: + # PATCH 200 → the "hidden on" tag appears in the source cell and + # the confirmation is the LAST announcer message (the reload's + # "1 source listed." landed first). + _flip_hidden(page, stored_path, wanted_on=True) + + # The checkbox re-renders CHECKED from the server state (never a + # local flip) and the API round-trips include_hidden: true. + expect(_hidden_box(page, stored_path)).to_be_checked() + r = page.request.get(f"{app_url}/api/git-sources") + assert r.status == 200, r.text + sources = r.json()["sources"] + assert [s["include_hidden"] for s in sources] == [True] + assert [s["ignore_paths"] for s in sources] == [[]] # the list untouched + + # The NEXT sync indexes the hidden-folder file like any visible + # file (embedded + chunked by the mock LLM)… + second = run_sync(page, app_url) + assert second["state"] == "success", second + detail = second["detail"] + assert detail["files"] == 2, detail + assert detail["added"] == 1, detail # the hidden note only + assert detail["unchanged"] == 1, detail # visible.md + # …and ``.venv/junk.md`` is STILL absent — EXCLUDED_DIRS content is + # never indexed, in either state (A1). + paths = _catalog_paths(page, app_url) + assert HIDDEN_NOTE in paths, ( + f"hidden note missing from the catalog after the on-sync: {paths}" + ) + assert VISIBLE_MD in paths, paths + assert VENV_JUNK not in paths, ( + f"EXCLUDED_DIRS leaked into the catalog with the flag on: {paths}" + ) + assert set(_db_document_paths()) == {VISIBLE_MD, HIDDEN_NOTE} + + +# --------------------------------------------------------------------------- +# 4. A2: flipping the flag OFF prunes the previously indexed hidden doc +# from the KB on the next sync +# --------------------------------------------------------------------------- + + +def test_toggle_off_prunes_hidden( + page: Page, app_url: str, db_ready: None, source_dir: Path +) -> None: + page.set_default_timeout(30_000) + _admin_git_sources_page(page, app_url) + stored_path = _seed_local_source(page, app_url, source_dir) + + # On + sync: the hidden note is in the KB. + _flip_hidden(page, stored_path, wanted_on=True) + first = run_sync(page, app_url) + assert first["state"] == "success", first + assert first["detail"]["files"] == 2, first["detail"] + assert first["detail"]["added"] == 2, first["detail"] + assert HIDDEN_NOTE in _catalog_paths(page, app_url) + + # A2: flip it OFF (the real click) — the previously indexed hidden + # file leaves the KB on the next sync (the seen-set prune — the + # phase-89 A2 / A9 precedent; the tag is gone). + _flip_hidden(page, stored_path, wanted_on=False) + r = page.request.get(f"{app_url}/api/git-sources") + assert [s["include_hidden"] for s in r.json()["sources"]] == [False] + + second = run_sync(page, app_url) + assert second["state"] == "success", second + detail = second["detail"] + assert detail["pruned"] == 1, detail # the hidden doc + assert detail["files"] == 1, detail # the walk is back to visible-only + paths = _catalog_paths(page, app_url) + assert HIDDEN_NOTE not in paths, ( + f"the hidden doc survived the flag-off prune: {paths}" + ) + assert VISIBLE_MD in paths, f"visible.md must survive the prune: {paths}" + assert _db_document_paths() == [VISIBLE_MD] + + +# --------------------------------------------------------------------------- +# 5. A3: env-fallback rows have no toggle (no DB row to store a flag on) +# --------------------------------------------------------------------------- + + +def test_env_fallback_rows_have_no_toggle( + page: Page, app_url: str, db_ready: None +) -> None: + page.set_default_timeout(30_000) + # The autouse fixture truncated git_sources — the table is EMPTY, + # so the module app's BOR_GIT_SOURCES URL is the effective list + # (the test_source_ignore_paths.py module-env pattern). + _admin_git_sources_page(page, app_url) + + expect(page.locator("#git-sources-tbody tr")).to_have_count(1) + row = page.locator("#git-sources-tbody tr") + expect(row).to_contain_text(ENV_SOURCE) + # The env-fallback surface: the "from .env" tag (never color + # alone)… + expect(row.locator(".git-source-env-tag")).to_have_count(1) + expect(row.locator(".git-source-env-tag")).to_have_text("from .env") + # …and A3: NO per-row controls at all — neither Remove, nor the + # phase-89 "Ignore paths" button, nor the phase-105 Hidden checkbox + # (and no "hidden on" tag). + expect(row.locator(".git-source-remove")).to_have_count(0) + expect(row.locator(".git-source-ignore")).to_have_count(0) + expect(row.locator(".git-source-hidden")).to_have_count(0) + expect(row.locator(".git-source-hidden-box")).to_have_count(0) + expect(row.locator(".git-source-hidden-count")).to_have_count(0) + # The env-fallback note explains the active list's origin… + expect(page.locator("#git-sources-env-note")).to_be_visible() + # …and the API agrees: from_env true, null ids, the env URL, an + # empty ignore list, and include_hidden False (no DB row to store a + # list or a flag on — the GitSourceRow contract, task 03). + r = page.request.get(f"{app_url}/api/git-sources") + assert r.status == 200, r.text + body = r.json() + assert body["from_env"] is True + assert [s["url"] for s in body["sources"]] == [ENV_SOURCE] + assert all( + s["id"] is None and s["ignore_paths"] == [] and s["include_hidden"] is False + for s in body["sources"] + ) + + +# --------------------------------------------------------------------------- +# 6. a11y + the error surface: the accessible name, keyboard focus, the +# text tag, and the role=alert line (hidden on the happy path, the +# canned message + revert on a network failure) +# --------------------------------------------------------------------------- + + +def test_toggle_a11y_and_error_surface( + page: Page, app_url: str, db_ready: None, source_dir: Path +) -> None: + page.set_default_timeout(30_000) + _admin_git_sources_page(page, app_url) + stored_path = _seed_local_source(page, app_url, source_dir) + + box = _hidden_box(page, stored_path) + expect(box).to_have_count(1) + # The full accessible name contains the source location (the + # aria-label — the ONLY place the value appears, setAttribute, + # never innerHTML)… + expect(box).to_have_attribute( + "aria-label", f"Index hidden folders for local source: {stored_path}" + ) + expect( + page.get_by_role( + "checkbox", name=f"Index hidden folders for local source: {stored_path}" + ) + ).to_have_count(1) + # …it is keyboard-reachable — a plain Tab sequence from the page + # top reaches it (one full focus cycle covers every focusable + # element; the visible ring is the GLOBAL :focus-visible rule, + # styles.css — unit-pinned, no per-control rule)… + for _ in range(60): + page.keyboard.press("Tab") + if box.evaluate("el => el === document.activeElement"): + break + expect(box).to_be_focused() + # …and it receives programmatic focus too (the a11y trees agree). + box.focus() + expect(box).to_be_focused() + # …and the visible "Hidden" label wraps the box (text + control — + # never color or icon alone). + label = page.locator("label.git-source-hidden") + expect(label).to_have_count(1) + expect(label).to_contain_text("Hidden") + + # The page-level error line exists, is a real role=alert, and stays + # hidden through the happy path. + error = page.locator("#git-sources-hidden-error") + expect(error).to_have_count(1) + assert error.get_attribute("role") == "alert" + expect(error).to_be_hidden() + + # The tag is TEXT (never color alone — WCAG 1.4.1): flip on… + _flip_hidden(page, stored_path, wanted_on=True) + expect( + _row(page, stored_path).locator(".git-source-hidden-count") + ).to_have_text("hidden on") + expect(error).to_be_hidden() # the happy path never touches the line + + # §7.4 failure path: a network-level abort of the PATCH → the + # canned "not changed" message in the role=alert line and the box + # reverts to the SERVER state (still ON — the request never + # landed) and re-enables. The tag stays (the server is unchanged). + def _abort_patch(route: Any) -> None: + if route.request.method == "PATCH": + route.abort() + else: + route.continue_() + + page.route("**/api/git-sources/**", _abort_patch) + try: + _hidden_box(page, stored_path).click() + expect(error).to_be_visible(timeout=30_000) + expect(error).to_have_text( + "Could not reach the server — the setting was not changed." + ) + expect( + _row(page, stored_path).locator(".git-source-hidden-count") + ).to_have_count(1) # the server state is unchanged: still on + reverted = _hidden_box(page, stored_path) + expect(reverted).to_be_checked() # reverted to the server state + expect(reverted).to_be_enabled() # re-enabled — never stale + finally: + page.unroute("**/api/git-sources/**") diff --git a/tests/integration/test_git_sources_api.py b/tests/integration/test_git_sources_api.py index db94352..71975c1 100644 --- a/tests/integration/test_git_sources_api.py +++ b/tests/integration/test_git_sources_api.py @@ -32,9 +32,16 @@ Contract under test: * ignore paths (phase 89) — POST accepts the RAW box lines for both kinds (optional, absent → ``[]``), stored normalized (A1) with the A4 fixed-detail 422s (shared gate with PATCH); GET reports each row's - stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only) - replaces the list wholesale (A5 — an empty list clears all), 404 - unknown id, 422 fixed details for the A4 limits, the row otherwise + stored list (env rows ``[]``); +* hidden-folders flag (phase 105) — POST accepts ``include_hidden`` for + both kinds (optional, absent → stored ``False``, A4); GET reports the + stored flag (env rows ``False`` — no DB row to store a flag on); +* ``PATCH /{source_id}`` (admin-only) — phase 89 A5 + phase 105: each + field is optional, present-wins — ``ignore_paths`` when present + REPLACES the list wholesale (A5 — an empty list clears all), + ``include_hidden`` when present sets the flag, both absent → 200 + no-op, 404 unknown id, 422 fixed details for the A4 limits, a + rejected list never half-applies the flag, the row otherwise unchanged. ``git_sources`` is global state: truncated around every test. @@ -99,10 +106,14 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No r = client.delete(f"/api/git-sources/{uuid.uuid4()}") assert r.status_code == 403 assert r.json() == {"detail": "admin only"} - # The phase-89 ignore-list PATCH is gated the same way. + # The phase-89 ignore-list PATCH is gated the same way — and so is + # the phase-105 bool-only payload (the toggle's exact request). r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]}) assert r.status_code == 403 assert r.json() == {"detail": "admin only"} + r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"include_hidden": True}) + assert r.status_code == 403 + assert r.json() == {"detail": "admin only"} # Nothing landed in the table. assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0 @@ -132,8 +143,10 @@ def test_get_empty_table_with_env_returns_env_rows( "url": "https://a.example.com/one.git", "path": None, "added_at": None, - # Env rows have no DB row to store a list on (phase 89). + # Env rows have no DB row to store a list on (phase 89) or + # a flag on (phase 105). "ignore_paths": [], + "include_hidden": False, }, { "id": None, @@ -142,6 +155,7 @@ def test_get_empty_table_with_env_returns_env_rows( "path": None, "added_at": None, "ignore_paths": [], + "include_hidden": False, }, ] @@ -208,9 +222,11 @@ def test_post_creates_trimmed_and_list_stops_using_env( uuid.UUID(body["id"]) assert body["added_at"] is not None # Phase 89: the response gains ``ignore_paths`` — absent at create - # time → ``[]``. - assert set(body) == {"id", "url", "added_at", "ignore_paths"} + # time → ``[]``; phase 105 adds ``include_hidden`` — absent → + # ``False`` (A4). + assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"} assert body["ignore_paths"] == [] + assert body["include_hidden"] is False # The DB row now wins: from_env False, the env URL is gone from the list. body = admin_client.get("/api/git-sources").json() @@ -321,9 +337,11 @@ def test_post_local_creates_stored_row_with_expanded_path( body = r.json() # The phase-35 response shape is unchanged — the local row reports # its (expanded) path in ``url``; ``kind`` + ``path`` via GET; - # phase 89 adds ``ignore_paths`` (absent → ``[]``). - assert set(body) == {"id", "url", "added_at", "ignore_paths"} + # phase 89 adds ``ignore_paths`` (absent → ``[]``); phase 105 adds + # ``include_hidden`` (absent → ``False``, A4). + assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"} assert body["ignore_paths"] == [] + assert body["include_hidden"] is False uuid.UUID(body["id"]) assert body["url"] == str(real_dir) assert body["added_at"] is not None @@ -556,6 +574,7 @@ def test_delete_removes_row_and_falls_back_to_env( "path": None, "added_at": None, "ignore_paths": [], # env rows: no DB row to store a list on + "include_hidden": False, # … or a flag on (phase 105) } ] db.execute(text("TRUNCATE chunks, documents")) @@ -671,6 +690,7 @@ def test_get_reports_stored_ignore_paths_and_env_rows_empty( "path": None, "added_at": None, "ignore_paths": [], + "include_hidden": False, } ] @@ -690,8 +710,9 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) - r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]}) assert r.status_code == 200, r.text body = r.json() - assert set(body) == {"id", "url", "added_at", "ignore_paths"} + assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"} assert body["ignore_paths"] == ["a", "b"] # normalized + assert body["include_hidden"] is False # untouched by a list-only PATCH for key in ("id", "url", "added_at"): assert body[key] == before[key] # Round-trip through GET. @@ -706,17 +727,34 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) - assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == [] -def test_patch_missing_field_is_422(admin_client: TestClient) -> None: - """``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent - field is a 422 with the model's own detail.""" - created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"}) - assert created.status_code == 201 - r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={}) - assert r.status_code == 422 - # The model's own (Pydantic) detail — the missing required field. - assert any( - item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"] +def test_patch_empty_body_is_a_noop_200(admin_client: TestClient) -> None: + """Phase 105: BOTH fields absent (or explicit None) → 200 no-op — + the row is untouched. (The pre-phase-105 pin of an absent + ``ignore_paths`` as a 422 is retired — the list is optional now; a + PRESENT list keeps the phase-89 A5 replace semantics.)""" + created = admin_client.post( + "/api/git-sources", + json={"url": "https://example.com/noop.git", "ignore_paths": ["a/b"]}, ) + assert created.status_code == 201 + before = created.json() + + r = admin_client.patch(f"/api/git-sources/{before['id']}", json={}) + assert r.status_code == 200, r.text + body = r.json() + assert body["ignore_paths"] == ["a/b"] + assert body["include_hidden"] is False + for key in ("id", "url", "added_at"): + assert body[key] == before[key] + # Explicit None values are "absent" too — still a no-op. + r = admin_client.patch( + f"/api/git-sources/{before['id']}", json={"ignore_paths": None, "include_hidden": None} + ) + assert r.status_code == 200, r.text + assert r.json()["ignore_paths"] == ["a/b"] + assert r.json()["include_hidden"] is False + # The row never changed — round-trip through GET. + assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a/b"] def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None: @@ -772,3 +810,142 @@ def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None: r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]}) assert r.status_code == 200, r.text assert r.json()["ignore_paths"] == [long_entry] + + +# --- hidden-folders flag (phase 105) ---------------------------------------- + + +def test_get_reports_include_hidden_default_false(admin_client: TestClient, db: Session) -> None: + """A fresh stored row (no flag passed) reports ``include_hidden: + false`` — in the GET round-trip AND on the model (the column's + server default, A4).""" + db.add(GitSource(url="https://example.com/fresh.git")) + db.commit() + body = admin_client.get("/api/git-sources").json() + assert body["sources"][0]["include_hidden"] is False + row = db.scalars(select(GitSource)).one() + assert row.include_hidden is False + + +def test_post_stores_include_hidden_both_kinds( + admin_client: TestClient, tmp_path: Path +) -> None: + """POST accepts ``include_hidden`` for both kinds (present → stored + as sent); the 201 body and the GET round-trip report it.""" + r = admin_client.post( + "/api/git-sources", + json={"url": "https://example.com/hidden-git.git", "include_hidden": True}, + ) + assert r.status_code == 201, r.text + assert r.json()["include_hidden"] is True + + real_dir = tmp_path / "hidden-local" + real_dir.mkdir() + r = admin_client.post( + "/api/git-sources", + json={"kind": "local", "path": str(real_dir), "include_hidden": True}, + ) + assert r.status_code == 201, r.text + assert r.json()["include_hidden"] is True + + by_url = { + s["url"]: s["include_hidden"] + for s in admin_client.get("/api/git-sources").json()["sources"] + } + assert by_url["https://example.com/hidden-git.git"] is True + assert by_url[str(real_dir)] is True + + +def test_patch_bool_only_sets_flag_leaves_list(admin_client: TestClient) -> None: + """The toggle's exact payload — ``{"include_hidden": …}`` alone: + the flag is set, the ignore list is UNCHANGED (present-wins, phase + 105).""" + created = admin_client.post( + "/api/git-sources", + json={"url": "https://example.com/toggle.git", "ignore_paths": ["a/b"]}, + ) + assert created.status_code == 201 + sid = created.json()["id"] + + r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": True}) + assert r.status_code == 200, r.text + body = r.json() + assert body["include_hidden"] is True + assert body["ignore_paths"] == ["a/b"] # untouched + assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True + + # And back off again. + r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": False}) + assert r.status_code == 200, r.text + assert r.json()["include_hidden"] is False + assert r.json()["ignore_paths"] == ["a/b"] + + +def test_patch_list_only_replaces_list_leaves_flag(admin_client: TestClient) -> None: + """The dialog's exact payload — ``{"ignore_paths": …}`` alone: + the list is REPLACED (normalized, phase-89 A5), the flag is + UNCHANGED — byte-identical to the pre-phase-105 dialog PATCH.""" + created = admin_client.post( + "/api/git-sources", + json={ + "url": "https://example.com/dialog.git", + "ignore_paths": ["a/b"], + "include_hidden": True, + }, + ) + assert created.status_code == 201 + sid = created.json()["id"] + + r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": ["c/d", " e "]}) + assert r.status_code == 200, r.text + body = r.json() + assert body["ignore_paths"] == ["c/d", "e"] # normalized + assert body["include_hidden"] is True # untouched + assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True + + +def test_patch_both_fields_apply_independently(admin_client: TestClient) -> None: + """``{"ignore_paths": [], "include_hidden": true}`` — the list is + cleared AND the flag is set in one request.""" + created = admin_client.post( + "/api/git-sources", + json={"url": "https://example.com/both.git", "ignore_paths": ["old/one"]}, + ) + assert created.status_code == 201 + sid = created.json()["id"] + + r = admin_client.patch( + f"/api/git-sources/{sid}", json={"ignore_paths": [], "include_hidden": True} + ) + assert r.status_code == 200, r.text + body = r.json() + assert body["ignore_paths"] == [] # cleared + assert body["include_hidden"] is True # set + + +def test_patch_422_does_not_half_apply_flag(admin_client: TestClient) -> None: + """A PRESENT bad list 422s with the fixed A4 details — and the + OTHER field in the same body never half-applies: after each 422 + the flag is still the stored value (validation raises before any + assignment, request transaction untouched).""" + created = admin_client.post( + "/api/git-sources", + json={"url": "https://example.com/half.git", "ignore_paths": ["keep"]}, + ) + assert created.status_code == 201 + sid = created.json()["id"] + + for payload, detail in ( + ([" "], "ignore paths must be non-empty"), + ([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"), + (["a" * 501], "an ignore path exceeds 500 characters"), + ): + r = admin_client.patch( + f"/api/git-sources/{sid}", json={"ignore_paths": payload, "include_hidden": True} + ) + assert r.status_code == 422, f"{detail!r}: {r.text}" + assert r.json()["detail"] == detail + # Nothing half-applied: list AND flag are both still the originals. + row = admin_client.get("/api/git-sources").json()["sources"][0] + assert row["ignore_paths"] == ["keep"] + assert row["include_hidden"] is False diff --git a/tests/integration/test_import_docs_git.py b/tests/integration/test_import_docs_git.py index d55c3cf..e84e0c2 100644 --- a/tests/integration/test_import_docs_git.py +++ b/tests/integration/test_import_docs_git.py @@ -27,15 +27,19 @@ is asserted against the canned value). from __future__ import annotations import re +from collections.abc import Iterator from pathlib import Path import pytest +from sqlalchemy import text +from sqlalchemy.orm import Session from app.config import Settings from app.models import GitSource from app.rag.importer import ImportSummary from scripts import import_docs from scripts.git_sync import GitSyncError +from tests.fakes import FakeEmbedder def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings: @@ -47,11 +51,31 @@ def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource: return GitSource(url=url, kind="git", ignore_paths=ignore_paths or []) -def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource: +def _local_row( + path: str, + ignore_paths: list[str] | None = None, + include_hidden: bool = False, +) -> GitSource: """A local row as the phase-38 API stores it: the expanded path in both ``path`` and the NOT-NULL ``url`` location column (plus the - phase-89 ignore list, empty by default).""" - return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or []) + phase-89 ignore list and the phase-105 hidden-folders flag, both + defaulting off — A4).""" + return GitSource( + url=path, kind="local", path=path, + ignore_paths=ignore_paths or [], include_hidden=include_hidden, + ) + + +@pytest.fixture() +def clean_documents(db: Session) -> Iterator[None]: + """Phase 105: the real-import CLI tests write ``documents``/ + ``chunks`` (the canonical KB state) — global, truncated around + every such test.""" + db.execute(text("TRUNCATE chunks, documents")) + db.commit() + yield + db.execute(text("TRUNCATE chunks, documents")) + db.commit() class FakeImportSources: @@ -68,10 +92,12 @@ class FakeImportSources: prune: bool = False, limit: int | None = None, ignore_by_root: dict[str, list[str]] | None = None, # phase 89 + include_hidden_by_root: dict[str, bool] | None = None, # phase 105 ) -> ImportSummary: self.calls.append( {"sources": list(sources), "prune": prune, "limit": limit, - "ignore_by_root": ignore_by_root} + "ignore_by_root": ignore_by_root, + "include_hidden_by_root": include_hidden_by_root} ) return ImportSummary(files=1, added=1) @@ -168,10 +194,16 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir( sources_dir=str(tmp_path / "bor"), ) - sources, ignore_map = import_docs._resolve_sources(None, settings) + sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings) assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"] assert ignore_map == {} # phase 89: no row carries a list → empty map + # Phase 105: flag-off rows still contribute their root — with False + # (A4), keyed by the SAME root string the importer sees. + assert hidden_map == { + str(tmp_path / "bor" / "homelab"): False, + str(tmp_path / "bor" / "deploy"): False, + } assert calls == [ ("https://host/a/homelab.git", tmp_path / "bor" / "homelab"), ("git@host:user/deploy.git", tmp_path / "bor" / "deploy"), @@ -186,10 +218,11 @@ def test_resolve_sources_cli_source_wins( settings = _settings(git_sources="https://host/a/repo.git") manual = tmp_path / "Manual" - sources, ignore_map = import_docs._resolve_sources([manual], settings) + sources, ignore_map, hidden_map = import_docs._resolve_sources([manual], settings) assert sources == [manual] assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore + assert hidden_map == {} # phase 105: manual dirs have no rows → hidden skipped assert calls == [] # git is never touched when --source is given @@ -210,10 +243,12 @@ def test_resolve_sources_db_rows_win_over_env( sources_dir=str(tmp_path / "bor"), ) - sources, ignore_map = import_docs._resolve_sources(None, settings) + sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings) assert sources == [tmp_path / "bor" / "only"] assert ignore_map == {} # phase 89: no row carries a list → empty map + # Phase 105: the default-flag row contributes its root with False (A4). + assert hidden_map == {str(tmp_path / "bor" / "only"): False} assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")] @@ -222,9 +257,10 @@ def test_resolve_sources_defaults_when_nothing_configured( ) -> None: # Both origins empty (the resolver's ``([], "env")``) → legacy dirs. monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env")) - sources, ignore_map = import_docs._resolve_sources(None, _settings()) + sources, ignore_map, hidden_map = import_docs._resolve_sources(None, _settings()) assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES] assert ignore_map == {} # phase 89: the legacy fallback has no rows + assert hidden_map == {} # phase 105: the legacy fallback has no rows def test_resolve_sources_rows_branch_builds_ignore_map( @@ -250,11 +286,13 @@ def test_resolve_sources_rows_branch_builds_ignore_map( ) settings = _settings(sources_dir=str(tmp_path / "bor")) - sources, ignore_map = import_docs._resolve_sources(None, settings) + sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings) assert sources == [tmp_path / "bor" / "only", local_dir] # Keyed by the SAME string the importer sees (the root, not the name). assert ignore_map == {str(local_dir): ["ignore/"]} + # Phase 105: both rows are flag-off → per-root False entries (A4). + assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False} def test_resolve_sources_two_rows_sharing_root_string_extend( @@ -279,11 +317,14 @@ def test_resolve_sources_two_rows_sharing_root_string_extend( ) settings = _settings(sources_dir=str(tmp_path / "bor")) - sources, ignore_map = import_docs._resolve_sources(None, settings) + sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings) shared = str(tmp_path / "bor" / "shared") assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"] assert ignore_map == {shared: ["a/", "b"]} # union, row order + # Phase 105 collision: the shared root gets the OR of the flags — + # both rows off here, so one False entry for the one root string. + assert hidden_map == {shared: False} def test_main_rows_branch_passes_ignore_map_to_import( @@ -318,9 +359,145 @@ def test_main_rows_branch_passes_ignore_map_to_import( call = fake_import.calls[0] assert call["sources"] == [local_dir] assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]} + # Phase 105: the default-flag row passes the per-root map too — a + # False entry, not an absent key (the importer reads it per root). + assert call["include_hidden_by_root"] == {str(local_dir): False} assert call["prune"] is False # the CLI's no-prune default is unchanged +# --- phase 105: per-row hidden-folders flag --------------------------------- + + +def test_main_rows_branch_passes_include_hidden_map_to_import( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + """Phase 105: a ``kind=local`` row with ``include_hidden=True`` → + ``main`` passes the per-root flag map to ``import_sources`` (keyed + by the directory string — the map is built, not lost).""" + settings = _settings(sources_dir=str(tmp_path / "bor")) + monkeypatch.setattr(import_docs, "get_settings", lambda: settings) + local_dir = tmp_path / "LocalDocs" + local_dir.mkdir() + (local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8") + (local_dir / ".hidden").mkdir() + (local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8") + monkeypatch.setattr( + import_docs, + "effective_sources", + lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"), + ) + fake_import = FakeImportSources() + monkeypatch.setattr(import_docs, "import_sources", fake_import) + _stub_bump(monkeypatch) + _stub_folder_summaries(monkeypatch) + + rc = import_docs.main([]) + + assert rc == 0 + call = fake_import.calls[0] + assert call["sources"] == [local_dir] + assert call["include_hidden_by_root"] == {str(local_dir): True} + assert call["ignore_by_root"] == {} # the row carries no ignore list + assert call["prune"] is False # the CLI's no-prune default is unchanged + + +def _stub_overview(monkeypatch: pytest.MonkeyPatch) -> None: + """Stub the phase-31 overview regeneration (the CLI's real-import + tests: the deterministic ``FakeEmbedder`` must not burn its canned + ``chat`` on the KB outline — the import is what is under test).""" + + async def fake_overview(llm: object, session: object = None) -> bool: + return False + + monkeypatch.setattr(import_docs, "regenerate_overview", fake_overview) + + +def _kb_docs(db: Session) -> set[tuple[str, str]]: + """The (source, path) pairs of the ``documents`` table.""" + rows = db.execute(text("SELECT source, path FROM documents")).all() + return {(row.source, row.path) for row in rows} + + +def test_main_local_row_include_hidden_true_indexes_hidden_file( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + db: Session, + capsys: pytest.CaptureFixture[str], + clean_documents: None, +) -> None: + """Phase 105 (A1): the CLI's DB-row path with the flag ON — the + file inside the hidden folder is indexed, embedded, and counted + like any visible file (real import over a host temp dir, the + deterministic ``FakeEmbedder``; the regression this phase most + plausibly breaks — the CLI's map built but lost — would leave it + out).""" + settings = _settings(sources_dir=str(tmp_path / "bor")) + monkeypatch.setattr(import_docs, "get_settings", lambda: settings) + local_dir = tmp_path / "LocalDocs" + local_dir.mkdir() + (local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8") + (local_dir / ".hidden").mkdir() + (local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8") + monkeypatch.setattr( + import_docs, + "effective_sources", + lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"), + ) + monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder()) + _stub_overview(monkeypatch) + _stub_bump(monkeypatch) + _stub_folder_summaries(monkeypatch) + + rc = import_docs.main([]) + + assert rc == 0 + # The hidden file has a documents row next to the visible one. + assert _kb_docs(db) == { + ("LocalDocs", "visible.md"), + ("LocalDocs", ".hidden/note.md"), + } + out = capsys.readouterr().out + assert "added=2" in out # both files were imported + + +def test_main_local_row_include_hidden_false_skips_hidden_file( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + db: Session, + capsys: pytest.CaptureFixture[str], + clean_documents: None, +) -> None: + """Phase 105 (A4): the same fixture with the default flag (off) — + the hidden-folder file never enters the KB (the byte-identical + pre-phase-105 walk): no documents row, and the summary line counts + only the visible file.""" + settings = _settings(sources_dir=str(tmp_path / "bor")) + monkeypatch.setattr(import_docs, "get_settings", lambda: settings) + local_dir = tmp_path / "LocalDocs" + local_dir.mkdir() + (local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8") + (local_dir / ".hidden").mkdir() + (local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8") + monkeypatch.setattr( + import_docs, + "effective_sources", + lambda db: ([_local_row(str(local_dir))], "db"), # flag defaults to False + ) + monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder()) + _stub_overview(monkeypatch) + _stub_bump(monkeypatch) + _stub_folder_summaries(monkeypatch) + + rc = import_docs.main([]) + + assert rc == 0 + assert _kb_docs(db) == {("LocalDocs", "visible.md")} + out = capsys.readouterr().out + assert "added=1" in out # the hidden file was never walked + + # --- main() ---------------------------------------------------------------- @@ -428,10 +605,12 @@ def test_resolve_sources_mixed_git_and_local( sources_dir=str(tmp_path / "bor"), ) - sources, ignore_map = import_docs._resolve_sources(None, settings) + sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings) assert sources == [tmp_path / "bor" / "only", local_dir] assert ignore_map == {} # phase 89: neither row carries a list + # Phase 105: both rows default-flag → per-root False entries (A4). + assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False} assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")] diff --git a/tests/integration/test_importer_include_hidden.py b/tests/integration/test_importer_include_hidden.py new file mode 100644 index 0000000..fb13eb9 --- /dev/null +++ b/tests/integration/test_importer_include_hidden.py @@ -0,0 +1,241 @@ +"""Integration: the phase-105 hidden-folders flag through the real import +pipeline. + +Phase 105 (TODO.md L3 — "…a toggle per input … to allow indexing hidden +.folders."): ``import_sources`` gains ``include_hidden_by_root`` (same +``str(root)`` keying as phase 89's ``ignore_by_root``). A1 +(owner-confirmed 2026-09-14): flag ON admits dot-prefixed components — +hidden files are indexed, embedded, and summarized exactly like visible +files — while ``EXCLUDED_DIRS`` stay excluded in both states. A2: a file +indexed with the flag ON leaves the KB on the next ``prune=True`` run +with the flag OFF (the untouched ``seen`` set does the work). A4: no map +→ byte-identical to pre-phase-105. Mirrors the fixture-tree + mock-LLM +pattern of ``test_importer_ignore.py``: a ``tmp_path`` source dir run +through the real ``import_sources`` into the compose Postgres, with +:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in. +""" +from __future__ import annotations + +import asyncio +from pathlib import Path + +from sqlalchemy import select, text +from sqlalchemy.orm import Session + +from app.models import Chunk, Document +from app.rag.importer import import_sources +from tests.fakes import FakeEmbedder + +NAME = "HiddenFix" + + +def _write(root: Path, rel: str, content: str) -> None: + path = root / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + + +def _tree(tmp_path: Path, name: str = NAME) -> Path: + """One visible md, a hidden dir (md + non-markdown yaml), an excluded dir.""" + root = tmp_path / name + _write(root, "visible.md", "# Visible\n\nvisible body\n") + _write(root, ".hidden/note.md", "# Note\n\nHIDDEN-MD-CONTENT\n") + _write(root, ".hidden/data.yaml", "key: HIDDEN-YAML-VALUE\n") + _write(root, ".venv/junk.md", "# Junk\n\nEXCLUDED-CONTENT\n") + return root + + +def _reset(db: Session) -> None: + # House cleanup pattern (tests/integration/test_importer_ignore.py). + db.execute(text("TRUNCATE chunks, documents, query_log")) + db.commit() + + +def test_hidden_paths_not_indexed_by_default(db: Session, tmp_path: Path) -> None: + # A4: no map → today's behavior, byte-identical — the hidden files + # never produce a Document/Chunk row and are never embedded. + _reset(db) + root = _tree(tmp_path) + llm = FakeEmbedder() + + summary = asyncio.run(import_sources([root], llm, session=db)) + assert summary.files == 1 + assert summary.added == 1 + assert summary.errors == 0 + assert summary.summaries == 0 + assert summary.summary_errors == 0 + + docs = db.scalars(select(Document)).all() + assert {(d.source, d.path) for d in docs} == {(NAME, "visible.md")} + for rel in (".hidden/note.md", ".hidden/data.yaml", ".venv/junk.md"): + assert not any(d.path == rel for d in docs) + for texts in llm.calls: # every embed batch + for t in texts: + assert "HIDDEN-MD-CONTENT" not in t and "HIDDEN-YAML-VALUE" not in t + assert "EXCLUDED-CONTENT" not in t + # The hidden yaml never reached the lite model. + assert not llm.chat_calls + _reset(db) + + +def test_hidden_paths_indexed_when_flag_on(db: Session, tmp_path: Path) -> None: + # A1: with the map, hidden files are indexed, embedded, and + # summarized exactly like visible files — EXCLUDED_DIRS stay out. + _reset(db) + root = _tree(tmp_path) + llm = FakeEmbedder() + + summary = asyncio.run( + import_sources( + [root], llm, session=db, include_hidden_by_root={str(root): True} + ) + ) + assert summary.files == 3 + assert summary.added == 3 + assert summary.errors == 0 + assert summary.summary_errors == 0 + + docs = db.scalars(select(Document)).all() + assert {(d.source, d.path) for d in docs} == { + (NAME, "visible.md"), + (NAME, ".hidden/note.md"), + (NAME, ".hidden/data.yaml"), + } + # EXCLUDED_DIRS is excluded in BOTH states (A1). + assert not any(d.path == ".venv/junk.md" for d in docs) + chunks = db.scalars(select(Chunk)).all() + assert not any("EXCLUDED-CONTENT" in c.content for c in chunks) + # The hidden md was embedded like any visible md (no summary — the + # markdown path skips the lite model). + note = db.scalar(select(Document).where(Document.path == ".hidden/note.md")) + assert note is not None and note.summary is None + assert any("HIDDEN-MD-CONTENT" in c.content for c in chunks) + # The hidden yaml went through the FULL non-markdown path (phase 30): + # a stored summary plus one embedded is_summary chunk on top of the + # content chunks. + yaml_doc = db.scalar(select(Document).where(Document.path == ".hidden/data.yaml")) + assert yaml_doc is not None and yaml_doc.summary is not None + yaml_chunks = [ + c for c in chunks if c.document_id == yaml_doc.id + ] + assert any(c.is_summary for c in yaml_chunks) + assert any(not c.is_summary for c in yaml_chunks) + assert any("HIDDEN-YAML-VALUE" in c.content for c in yaml_chunks) + # Only the yaml reached the lite model. + assert summary.summaries == 1 + assert len(llm.chat_calls) == 1 + user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user") + assert "HIDDEN-YAML-VALUE" in user + _reset(db) + + +def test_flag_off_re_run_prunes_hidden_docs(db: Session, tmp_path: Path) -> None: + # A2: the owner flips the flag off — the next prune run walks with + # the default rules, the hidden files never enter ``seen``, and their + # rows (summary chunk included) leave the index. + _reset(db) + root = _tree(tmp_path) + llm = FakeEmbedder() + + s1 = asyncio.run( + import_sources( + [root], llm, session=db, include_hidden_by_root={str(root): True} + ) + ) + assert (s1.files, s1.added) == (3, 3) + assert ( + db.scalar(select(Document).where(Document.path == ".hidden/note.md")) + is not None + ) + + s2 = asyncio.run(import_sources([root], llm, session=db, prune=True)) + assert s2.files == 1 + assert s2.unchanged == 1 # visible.md untouched + assert s2.pruned == 2 # both hidden documents + assert ( + db.scalar(select(Document).where(Document.path == ".hidden/note.md")) + is None + ) + assert ( + db.scalar(select(Document).where(Document.path == ".hidden/data.yaml")) + is None + ) + assert ( + db.scalar(select(Document).where(Document.path == "visible.md")) is not None + ) + # The summary chunk rows went with their documents (cascade). + assert not any( + "HIDDEN-YAML-VALUE" in c.content for c in db.scalars(select(Chunk)).all() + ) + _reset(db) + + +def test_progress_total_agrees_with_walk_in_both_states( + db: Session, tmp_path: Path +) -> None: + # The phase-64 pre-walk uses the same per-root flag as the loop, so + # ``total`` agrees with the walk in both states. + _reset(db) + root = _tree(tmp_path) + llm = FakeEmbedder() + on_totals: set[int] = set() + + def on_progress(source: str, rel: str, done: int, total: int) -> None: + on_totals.add(total) + + s_on = asyncio.run( + import_sources( + [root], + llm, + session=db, + progress=on_progress, + include_hidden_by_root={str(root): True}, + ) + ) + assert s_on.files == 3 + assert on_totals == {3} # hidden files counted in the denominator + + _reset(db) + llm2 = FakeEmbedder() + off_totals: set[int] = set() + + def off_progress(source: str, rel: str, done: int, total: int) -> None: + off_totals.add(total) + + s_off = asyncio.run( + import_sources([root], llm2, session=db, progress=off_progress) + ) + assert s_off.files == 1 + assert off_totals == {1} # visible only, the pre-phase-105 count + _reset(db) + + +def test_unlisted_root_stays_hidden(db: Session, tmp_path: Path) -> None: + # The map is per-root, not global: listing one root as True leaves + # the other root exactly as pre-phase-105. + _reset(db) + root_a = _tree(tmp_path, name="HiddenFixA") + root_b = _tree(tmp_path, name="HiddenFixB") + llm = FakeEmbedder() + + summary = asyncio.run( + import_sources( + [root_a, root_b], + llm, + session=db, + include_hidden_by_root={str(root_a): True}, + ) + ) + # A: 3 (flag on) · B: 1 (unlisted → False) + assert summary.files == 4 + docs = db.scalars(select(Document)).all() + assert {(d.source, d.path) for d in docs} == { + ("HiddenFixA", "visible.md"), + ("HiddenFixA", ".hidden/note.md"), + ("HiddenFixA", ".hidden/data.yaml"), + ("HiddenFixB", "visible.md"), + } + assert not any( + d.source == "HiddenFixB" and d.path == ".hidden/note.md" for d in docs + ) + _reset(db) diff --git a/tests/integration/test_migration_0019.py b/tests/integration/test_migration_0019.py new file mode 100644 index 0000000..e463a21 --- /dev/null +++ b/tests/integration/test_migration_0019.py @@ -0,0 +1,271 @@ +"""Integration: migration 0019 (git_sources.include_hidden) schema +contract (phase 105, task 01). + +Drives the **real Alembic engine** against the live dev database +(``podman compose up -d db``), mirroring the house pattern of +``test_migration_0007.py`` (git_sources column contract — information_schema +assertions on the state the migration must leave). The tests target the +0018 → 0019 step explicitly so later migrations cannot break them: + +* upgrade 0018 → 0019 → the ``include_hidden`` column exists with the + full contract — BOOLEAN NOT NULL, server default ``false`` — while + the 0018 ``git_sources`` schema (url, kind, path, ignore_paths) + survives; +* pre-0019 rows backfill ``false`` and a row written without the + column takes the server default (A4 — byte-identical import + behavior until the owner flips the flag); +* the ORM contract agrees: a freshly inserted ``GitSource`` (no flag + passed) reads ``include_hidden is False`` (the Python ``default=False`` + and the server default agree), and an explicit ``True`` round-trips + through a fresh session; +* downgrade to 0018 → the column is GONE (A13 — reversible) while the + rows + their ignore lists survive; +* upgrade back to 0019 → the column is back (round-trip). + +The ``alembic`` fixture guarantees the DB ends at head even if a test +fails or the process is interrupted. +""" +from __future__ import annotations + +import uuid +from collections.abc import Iterator +from typing import Any + +import pytest +from alembic.config import Config +from sqlalchemy import text +from sqlalchemy.orm import Session + +from alembic import command +from app.db import SessionLocal, db_available +from app.models import GitSource + +URL_BASE = "https://git.example.com/mig0019" + + +@pytest.fixture() +def alembic(db: Session) -> Iterator[Config]: + """Real Alembic config bound to the dev DB (URL from app settings). + + Starts at head (repairs an interrupted earlier run); teardown upgrades + to head no matter what happened, so the dev DB is never left below + head. + """ + if not db_available(): + pytest.skip("Postgres not reachable — run `podman compose up -d db` first") + cfg = Config() # no alembic.ini file — env.py gets the URL from app config + cfg.set_main_option("script_location", "alembic") + command.upgrade(cfg, "head") + try: + yield cfg + finally: + # Release the test session's open transaction BEFORE the repair + # DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock + # on ``git_sources``, which would deadlock the repair's + # ``ALTER TABLE`` (0019) forever. + db.rollback() + command.upgrade(cfg, "head") + + +def _version(db: Session) -> str | None: + return db.execute(text("SELECT version_num FROM alembic_version")).scalar() + + +def _column(db: Session, column: str) -> tuple[Any, ...] | None: + """(data_type, is_nullable, column_default) for one git_sources column.""" + row = db.execute( + text( + "SELECT data_type, is_nullable, column_default" + " FROM information_schema.columns" + " WHERE table_name = 'git_sources' AND column_name = :c" + ), + {"c": column}, + ).fetchone() + return tuple(row) if row is not None else None + + +def _flag(db: Session, url: str) -> Any: + return db.execute( + text("SELECT include_hidden FROM git_sources WHERE url = :u"), + {"u": url}, + ).scalar_one() + + +def _insert_sql(db: Session, url: str, include_hidden: Any = None) -> None: + """Insert one git_sources row (kind/path omitted → the API default + shape); ``include_hidden`` omitted → pre-0019 insert shape.""" + if include_hidden is None: + db.execute( + text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"), + {"u": url}, + ) + else: + db.execute( + text( + "INSERT INTO git_sources (id, url, include_hidden)" + " VALUES (gen_random_uuid(), :u, :f)" + ), + {"u": url, "f": include_hidden}, + ) + db.commit() + + +def _delete_by_url(db: Session, url: str) -> None: + db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url}) + db.commit() + + +def test_upgrade_to_0019_adds_include_hidden(db: Session, alembic: Config) -> None: + """Upgrade 0018 → 0019: the column exists with the full contract + (BOOLEAN NOT NULL, server default ``false``), is ABSENT at 0018, + pre-0019 rows backfill ``false`` (A4), a new row without the column + takes the default, and an explicit ``true`` round-trips — while the + 0018 table contract survives.""" + command.downgrade(alembic, "0018") # start from the pre-0019 state + assert _version(db) == "0018" + assert _column(db, "include_hidden") is None, ( + "the flag must be absent at 0018" + ) + + url_pre = f"{URL_BASE}/pre-existing.git" + _insert_sql(db, url_pre) # no include_hidden column exists at 0018 + try: + command.upgrade(alembic, "0019") + assert _version(db) == "0019", "alembic_version must be at 0019" + + flag = _column(db, "include_hidden") + assert flag is not None, "git_sources.include_hidden is missing" + assert flag[0] == "boolean", "include_hidden must be BOOLEAN" + assert flag[1] == "NO", "include_hidden must be NOT NULL" + assert flag[2] is not None and "false" in str(flag[2]), ( + "include_hidden must carry the `false` server default" + ) + + # The pre-0019 row backfilled ``false`` (A4 — unchanged imports). + assert _flag(db, url_pre) is False + + # A row written without the column takes the server default. + url_new = f"{URL_BASE}/new-row.git" + _insert_sql(db, url_new) + try: + assert _flag(db, url_new) is False, ( + "an omitted flag takes the `false` server default" + ) + + # The flag round-trips through an explicit ``true``. + db.execute( + text("UPDATE git_sources SET include_hidden = true WHERE url = :u"), + {"u": url_new}, + ) + db.commit() + assert _flag(db, url_new) is True, ( + "include_hidden = true must round-trip" + ) + finally: + _delete_by_url(db, url_new) + + # The 0018 schema survives the additive upgrade. + ignore = _column(db, "ignore_paths") + assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", ( + "git_sources.ignore_paths (0013) must survive the upgrade" + ) + kind = _column(db, "kind") + assert kind is not None and kind[0] == "text" and kind[1] == "NO", ( + "git_sources.kind (0007) must survive the upgrade" + ) + finally: + _delete_by_url(db, url_pre) + + +def test_orm_fresh_row_defaults_false_and_true_round_trips( + db: Session, alembic: Config +) -> None: + """The ORM contract agrees with the column contract: a freshly + inserted ``GitSource`` (no flag passed) reads ``include_hidden is + False`` — the Python ``default=False`` and the server default + agree (A4) — and an explicit ``True`` round-trips through a fresh + session.""" + command.upgrade(alembic, "head") + url_off = f"{URL_BASE}/orm-default.git" + url_on = f"{URL_BASE}/orm-true.git" + try: + # Fresh row, flag omitted → False (the Python-side default). + row_off = GitSource(url=url_off, kind="git") + db.add(row_off) + db.commit() + db.expire_all() + reloaded_off = db.get(GitSource, row_off.id) + assert reloaded_off is not None, "the fresh row must be readable" + assert reloaded_off.include_hidden is False, ( + "a fresh row must read include_hidden is False (A4)" + ) + + # Explicit True round-trips through a FRESH session. + row_on = GitSource(url=url_on, kind="git", include_hidden=True) + db.add(row_on) + db.commit() + with SessionLocal() as fresh: + reloaded = fresh.get(GitSource, row_on.id) + assert reloaded is not None, "the row must exist in a fresh session" + assert reloaded.include_hidden is True, ( + "include_hidden=True must round-trip through the DB" + ) + finally: + _delete_by_url(db, url_off) + _delete_by_url(db, url_on) + + +def test_downgrade_to_0018_drops_the_column(db: Session, alembic: Config) -> None: + """Downgrade 0019 → 0018: the column is gone (A13 — fully + reversible) while the rows + their ignore lists survive, and the + rest of the 0018 table contract (``path``) is intact.""" + command.upgrade(alembic, "head") + url = f"{URL_BASE}/survivor.git" + db.execute( + text( + "INSERT INTO git_sources (id, url, include_hidden)" + " VALUES (:i, :u, true)" + ), + {"i": uuid.uuid4(), "u": url}, + ) + db.commit() + try: + command.downgrade(alembic, "0018") + assert _version(db) == "0018" + assert _column(db, "include_hidden") is None, ( + "the flag must be dropped" + ) + row = db.execute( + text("SELECT url, kind, ignore_paths FROM git_sources WHERE url = :u"), + {"u": url}, + ).fetchone() + assert row is not None and row[0] == url, ( + "the row must survive the column drop" + ) + assert row[1] == "git" and row[2] == [], ( + "kind + the ignore list must survive the column drop" + ) + + doc_path = _column(db, "path") + assert doc_path is not None and doc_path[0] == "text", ( + "git_sources.path must survive the downgrade" + ) + finally: + _delete_by_url(db, url) + # Repair: the fixture teardown re-upgrades to head. + + +def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None: + """Downgrade to 0018, then upgrade back to 0019: the column is back + with the full contract (BOOLEAN NOT NULL, the `false` default).""" + command.downgrade(alembic, "0018") + command.upgrade(alembic, "0019") + assert _version(db) == "0019", "round-trip upgrade must land at 0019" + + flag = _column(db, "include_hidden") + assert flag is not None, "git_sources.include_hidden must be back" + assert flag[0] == "boolean", "include_hidden must be BOOLEAN after the round-trip" + assert flag[1] == "NO", "include_hidden must be NOT NULL after the round-trip" + assert flag[2] is not None and "false" in str(flag[2]), ( + "the `false` server default must survive the round-trip" + ) diff --git a/tests/integration/test_sync_api.py b/tests/integration/test_sync_api.py index e08b6fa..b794c0c 100644 --- a/tests/integration/test_sync_api.py +++ b/tests/integration/test_sync_api.py @@ -77,7 +77,7 @@ from typing import Any import pytest from fastapi.testclient import TestClient -from sqlalchemy import text +from sqlalchemy import select, text from sqlalchemy.orm import Session from app.api import sync as sync_api @@ -262,6 +262,11 @@ class FakeImportSources: # rows' ``ignore_paths`` (keyed by the root string the importer # sees; two rows sharing a root string get the union). self.ignore_maps: list[dict[str, list[str]]] = [] + # Phase 105: the per-root hidden-folders flag map the runner + # builds from the rows' ``include_hidden`` (same root-string + # keying; a shared-root collision ORs — if either row says + # "index hidden", the root does). + self.include_hidden_maps: list[dict[str, bool]] = [] async def __call__( self, @@ -273,12 +278,14 @@ class FakeImportSources: session: Session | None = None, progress: Callable[[str, str, int, int], None] | None = None, ignore_by_root: dict[str, list[str]] | None = None, + include_hidden_by_root: dict[str, bool] | None = None, ) -> ImportSummary: self.sources.append(list(sources)) self.llms.append(llm) self.prune_flags.append(prune) self.progress_hooks.append(progress) self.ignore_maps.append(ignore_by_root or {}) + self.include_hidden_maps.append(include_hidden_by_root or {}) if self.delay: await asyncio.sleep(self.delay) return self.summary @@ -823,6 +830,7 @@ def test_import_error_is_reported_with_credentials_masked( session: Session | None = None, progress: Callable[[str, str, int, int], None] | None = None, ignore_by_root: dict[str, list[str]] | None = None, + include_hidden_by_root: dict[str, bool] | None = None, ) -> ImportSummary: raise EmbeddingError( "embeddings request to https://user:secret@aipi.reeseapps.com/v1 " @@ -1100,3 +1108,114 @@ def test_sync_without_ignore_lists_passes_empty_map( assert fake_import.sources == [[local_dir]] assert fake_import.ignore_maps == [{}] # no row carried a list + + +# --- phase 105: per-row hidden-folders flag -------------------------------- + + +def test_local_row_hidden_flag_off_then_on_indexes_hidden_paths( + sync_client: TestClient, + monkeypatch: pytest.MonkeyPatch, + db: Session, + tmp_path: Path, + clean_documents: None, +) -> None: + """Phase 105 (A4 then A1): the SAME local row, synced first with the + default flag (off) — the file inside the hidden folder never lands + in the KB (no document row — hence no embedding, no summary), the + visible file imports as usual; then the row is flipped on (direct + model set — the PATCH round-trip is task 03's layer) and the next + sync walks the hidden file too: it is indexed, embedded, and counted + like any visible file.""" + local_dir = tmp_path / "LocalDocs" + local_dir.mkdir() + (local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8") + (local_dir / ".hidden").mkdir() + (local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8") + _seed_local(db, local_dir) # include_hidden defaults to False (A4) + _stub_env(monkeypatch) + monkeypatch.setattr( + sync_api, + "get_settings", + lambda: _settings(sources_dir=str(tmp_path / "bor")), + ) + _real_llm(monkeypatch) # real import_sources, deterministic embeddings + + _login(sync_client) + assert sync_client.post("/api/sync").status_code == 202 + body = _poll(sync_client, "success") + + # Flag off (A4): only the visible file is walked and indexed — the + # hidden file has NO documents row. + assert body["detail"]["files"] == 1 + assert body["detail"]["added"] == 1 + assert body["detail"]["errors"] == 0 + docs = { + (d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"] + } + assert docs == {("LocalDocs", "visible.md")} + + # Flip the SAME row on — the next sync re-reads the flag per row. + row = db.execute(select(GitSource).where(GitSource.url == str(local_dir))).scalar_one() + row.include_hidden = True + db.commit() + + assert sync_client.post("/api/sync").status_code == 202 + body = _poll(sync_client, "success") + + # Flag on (A1): the hidden file is walked (detail.files counts it), + # embedded, and indexed alongside the visible file. + assert body["detail"]["files"] == 2 + assert body["detail"]["added"] == 1 + assert body["detail"]["errors"] == 0 + docs = { + (d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"] + } + assert docs == {("LocalDocs", "visible.md"), ("LocalDocs", ".hidden/note.md")} + + +def test_sync_builds_include_hidden_map_by_root_string_with_or( + sync_client: TestClient, + monkeypatch: pytest.MonkeyPatch, + db: Session, + tmp_path: Path, +) -> None: + """Phase 105 wiring (fake import): the runner keys the flag map by + the SAME root string the importer sees, and two rows sharing that + root string (the sibling/repo-name edge — ``…/shared`` and + ``…/shared.git`` clone into the same checkout dir) get the OR of + their flags — if EITHER row says "index hidden", the root does + (the ignore-map union's boolean mirror).""" + url_a = f"file://{tmp_path / 'shared'}" + url_b = f"{url_a}.git" # same repo name → same checkout dir + # Distinct added_at: the resolver orders by (added_at, id) — a + # same-timestamp pair would tie-break on the random uuid. + db.add(GitSource(url=url_a, kind="git", include_hidden=False, + added_at=datetime(2026, 1, 1, tzinfo=UTC))) + db.add(GitSource(url=url_b, kind="git", include_hidden=True, + added_at=datetime(2026, 1, 2, tzinfo=UTC))) + db.commit() + _stub_env(monkeypatch) + monkeypatch.setattr( + sync_api, + "get_settings", + lambda: _settings(sources_dir=str(tmp_path / "bor")), + ) + _, fake_clone = _fake_clone() + monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone) + _stub_probe(monkeypatch) + fake_import = FakeImportSources(ImportSummary(files=1, added=1)) + monkeypatch.setattr(sync_api, "import_sources", fake_import) + monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True)) + monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries()) + + _login(sync_client) + assert sync_client.post("/api/sync").status_code == 202 + _poll(sync_client, "success") + + shared = str(tmp_path / "bor" / "shared") + # Both rows resolve to the SAME checkout (the collision itself) and + # the map holds the OR of their flags, keyed by that one root + # string — the flag-off row's False is overridden by the True. + assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]] + assert fake_import.include_hidden_maps == [{shared: True}] diff --git a/tests/unit/test_hidden_folders_toggle.py b/tests/unit/test_hidden_folders_toggle.py new file mode 100644 index 0000000..1e6f03b --- /dev/null +++ b/tests/unit/test_hidden_folders_toggle.py @@ -0,0 +1,436 @@ +"""Unit: the per-row "Hidden" toggle on the Sources page (phase 105, +task 05). + +``TODO.md`` L3: "There should be a toggle per input (next to the +ignores button) to allow indexing hidden .folders." Every STORED +``git_sources`` row (git or local) gets a native labeled **Hidden** +checkbox in the actions cell, LEFT of the "Ignore paths" button (DOM +order Hidden · Ignore paths · Remove — A5); when on, the source cell +shows a "hidden on" text tag (text + background, never color alone — +WCAG 1.4.1); the flip PATCHes ``{"include_hidden": …}`` with the +§7.4 never-stale lifecycle (box disables at once; 200 → reload + +confirm LAST; failure → the box reverts to the SERVER state and the +detail lands in the page-level ``#git-sources-hidden-error`` +``role="alert"`` line — the checkbox is a table-cell control with no +dialog of its own). Env-fallback rows (``id`` null) get no checkbox +(A3 — nothing is stored to flag). + +The browser behavior itself is E2E-gated by the phase-105 story suite +(``tests/e2e/test_hidden_folders_toggle.py``, task 06); like the +phase-89 ``test_source_ignore_paths.py`` house pattern, this module +pins the source-level contract a silent regression would break: + +* the JS row wiring — ``makeRow`` builds the checkbox ONLY in the + ``s.id`` branch (class ``git-source-hidden-box``, ``type=checkbox``, + ``checked`` from ``s.include_hidden === true`` — server state only, + never a prior local flip), the aria-label is the ONLY place the + value appears (setAttribute — never innerHTML), the visible + "Hidden" text label, and the append ORDER (hidden label → + ignore button → Remove); the "hidden on" tag is TEXT in the source + cell, iff ``s.id && s.include_hidden === true``; +* the JS lifecycle — ``toggleHidden``: the box disables BEFORE the + ``PATCH`` (one flip at a time), the body is + ``JSON.stringify({ include_hidden: wanted })`` ONLY (the row's + ignore list is untouched — task 03's optional field), 200 → + clear the error → ``await loadSources()`` → announce (the + confirmation is the LAST announcement); non-2xx AND network + failure → the box REVERTS to ``s.include_hidden === true`` + + re-enables and the detail lands in the page-level alert line; + a healed ``loadSources`` clears the line (the phase-89 "happy + path heals the error state" precedent); +* the cross-file wire contract (SINGLE SOURCE OF TRUTH for the + field name) — the JS PATCH body key and the render field both + match their ``app/schemas.py`` counterparts + (``GitSourcePatchIn.include_hidden`` / ``GitSourceRow``) — a + rename on either side breaks the test; +* the static shell markup — ``#git-sources-hidden-error`` is a + ``role="alert"`` line, hidden by default, reusing the + ``.git-source-error`` class, INSIDE ``#view-git-sources`` and + AFTER ``#git-sources-table-wrap`` in source order; +* styles.css — the ``.git-source-hidden`` family on the house + palette: ~44px hit height, the checkbox's ``accent-color: + var(--brand)`` (checked-state contrast verified + recorded in the + comment), the ``:disabled`` wait state, and the "hidden on" tag + (text + a distinct background, never color alone). +""" +from __future__ import annotations + +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +FRONTEND = ROOT / "frontend" +# Phase 76 (task 02): the Sources view lives in the ONE-document shell. +SHELL_HTML = FRONTEND / "index.html" +JS = FRONTEND / "assets" / "git-sources.js" +CSS = FRONTEND / "assets" / "styles.css" +SCHEMAS = ROOT / "app" / "schemas.py" + +#: The checkbox + tag classes (pinned verbatim). +HIDDEN_BOX_CLASS = "git-source-hidden-box" +HIDDEN_LABEL_CLASS = "git-source-hidden" +HIDDEN_TAG_CLASS = "git-source-hidden-count" +HIDDEN_TAG_TEXT = "hidden on" + +#: The checkbox's aria-label template — the ONLY place the full +#: source value appears (setAttribute, never innerHTML). +ARIA_LABEL_TEMPLATE = "`Index hidden folders for ${kindLabel} source: ${value}`" + +#: The §7.4 lifecycle strings (pinned verbatim). +REVERT_STATE = "box.checked = s.include_hidden === true" +NETWORK_MESSAGE = "Could not reach the server — the setting was not changed." +PATCH_BODY = "JSON.stringify({ include_hidden: wanted })" + + +def _text(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _js() -> str: + return _text(JS) + + +def _css() -> str: + return _text(CSS) + + +def _fn(js: str, name: str) -> str: + """The source of a (possibly async) top-level function via + balanced-brace counting (the test_source_ignore_paths.py helper).""" + for prefix in ("async function ", "function "): + start = js.find(f"{prefix}{name}(") + if start != -1: + depth = 0 + for i in range(js.find("{", start), len(js)): + if js[i] == "{": + depth += 1 + elif js[i] == "}": + depth -= 1 + if depth == 0: + return js[start : i + 1] + raise AssertionError(f"unbalanced braces in {name}()") + raise AssertionError(f"{name}() must exist in git-sources.js") + + +def _css_rule(css: str, selector: str) -> str: + """The declarations of a simple one-line-opening rule (comments + stripped first — a house comment may legally carry braces).""" + clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S) + start = clean.find(f"{selector} {{") + assert start != -1, f"missing rule {selector} in styles.css" + depth = 0 + for i in range(clean.find("{", start), len(clean)): + if clean[i] == "{": + depth += 1 + elif clean[i] == "}": + depth -= 1 + if depth == 0: + return clean[start : i + 1] + raise AssertionError(f"unbalanced braces in {selector}") + + +def _python_class_field(class_name: str) -> str: + """The single bool field name declared on a Pydantic class in + ``app/schemas.py`` (``bool`` or ``bool | None`` — regex-parsed, + no import: the pin is on the source text, so a rename breaks it + without a schema load).""" + schemas = _text(SCHEMAS) + m = re.search( + rf"class {class_name}\(BaseModel\):.*?(?=\nclass |\Z)", schemas, re.S + ) + assert m, f"class {class_name} must exist in app/schemas.py" + fields = re.findall( + r"^\s*(\w+):\s*bool(?:\s*\|\s*None)?\s*(?==|$)", m.group(0), re.M + ) + assert len(fields) == 1, f"{class_name} must declare exactly one bool field" + return fields[0] + + +# ---------- the row wiring in makeRow ---------- + + +def test_row_checkbox_is_built_in_the_stored_row_branch() -> None: + """makeRow: the Hidden checkbox is created ONLY in the ``s.id`` + branch (A3 — env-fallback rows get the "from .env" tag, no + checkbox). It is a NATIVE labeled checkbox: class + git-source-hidden-box, type=checkbox, the visible "Hidden" text, + ``checked`` from the SERVER row (``s.include_hidden === true`` — + never a prior local flip, §7.4), and the aria-label template is + the ONLY place ``value`` appears (setAttribute — never + innerHTML). Its change handler runs toggleHidden.""" + make = _fn(_js(), "makeRow") + branch_i = make.find("if (s.id) {") + else_i = make.find("} else {", branch_i) + assert branch_i != -1 and else_i > branch_i + box_i = make.find('const hiddenBox = document.createElement("input")', branch_i) + assert -1 < box_i < else_i, "the checkbox is created in the s.id branch (A3)" + assert 'hiddenBox.type = "checkbox"' in make, "a native checkbox" + assert f"hiddenBox.className = \"{HIDDEN_BOX_CLASS}\"" in make + assert "hiddenBox.checked = s.include_hidden === true" in make, ( + "checked state comes ONLY from the server row" + ) + assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)" + assert 'hiddenBox.setAttribute(' in make, "the label is set via setAttribute" + assert "hiddenBox.innerHTML" not in _js(), "XSS contract: no innerHTML on the box" + assert "hiddenLabel.innerHTML" not in _js(), "XSS contract: no innerHTML on the label" + assert 'document.createTextNode("Hidden")' in make, ( + "the visible text label — never aria-label-only (WCAG)" + ) + assert "hiddenLabel.append(hiddenBox, document.createTextNode(\"Hidden\"))" in make + assert 'hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox))' in make + # The env-fallback branch (else) has no checkbox. + else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)] + assert HIDDEN_BOX_CLASS not in else_slice and "toggleHidden" not in else_slice, ( + "env-fallback rows get no checkbox (A3)" + ) + + +def test_row_dom_order_is_hidden_ignore_remove() -> None: + """makeRow: the append order in the actions cell is Hidden · + Ignore paths · Remove (A5 — the toggle sits LEFT of the "Ignore + paths" button; Remove stays last).""" + make = _fn(_js(), "makeRow") + branch_i = make.find("if (s.id) {") + else_i = make.find("} else {", branch_i) + hidden_append = make.find("actTd.appendChild(hiddenLabel)", branch_i) + ignore_i = make.find('const ignoreBtn = document.createElement("button")', branch_i) + ignore_append = make.find("actTd.appendChild(ignoreBtn)", branch_i) + remove_append = make.find("actTd.appendChild(btn)", branch_i) + assert -1 < hidden_append < ignore_i, "the hidden label precedes the ignore button" + assert -1 < ignore_append < remove_append, "Remove stays last" + assert remove_append < else_i, "all three controls are in the s.id branch" + + +def test_hidden_on_tag_renders_text_in_the_source_cell() -> None: + """makeRow: a stored row with the flag ON gets the "hidden on" + tag in the SOURCE cell (urlTd — the .git-source-ignore-count + idiom: TEXT, never color alone, WCAG 1.4.1), appended after the + location ; the tag is the state copy (pinned verbatim).""" + make = _fn(_js(), "makeRow") + assert "s.id && s.include_hidden === true" in make, ( + "the tag only for stored rows with the flag on" + ) + assert f"hiddenTag.className = \"{HIDDEN_TAG_CLASS}\"" in make + assert f"hiddenTag.textContent = \"{HIDDEN_TAG_TEXT}\"" in make, ( + "the tag copy is TEXT — never color alone" + ) + code_i = make.find("urlTd.append(badge, code)") + append_i = make.find("urlTd.append(hiddenTag)", code_i) + assert -1 < code_i < append_i, "the tag is appended to the source cell" + + +# ---------- the JS lifecycle (toggleHidden) ---------- + + +def test_toggle_sends_patch_and_disables_the_box_first() -> None: + """toggleHidden: the box disables BEFORE the PATCH goes out + (no double-flip — §7.4); the request is a PATCH to + /api/git-sources/{id} whose body is ``{"include_hidden": …}`` + ONLY (the row's ignore list is untouched — task 03's optional + field).""" + body = _fn(_js(), "toggleHidden") + disable_i = body.find("box.disabled = true") + fetch_i = body.find("`/api/git-sources/${s.id}`") + method_i = body.find('method: "PATCH"', fetch_i) + body_i = body.find(PATCH_BODY, method_i) + assert -1 < disable_i < fetch_i < method_i < body_i, ( + "disable → PATCH {include_hidden} (and nothing else)" + ) + assert "ignore_paths" not in body, "the PATCH body never carries the ignore list" + + +def test_toggle_success_clears_reloads_and_announces_last() -> None: + """toggleHidden 200: the error line clears, ``loadSources()`` + AWAITS (the row re-renders from the server — the "hidden on" tag + lands), and the confirmation is the LAST announcement (the + reload's "N sources listed." lands first — the phase-89 order).""" + body = _fn(_js(), "toggleHidden") + ok_i = body.find("if (r.ok)") + hide_i = body.find("hideHiddenError()", ok_i) + reload_i = body.find("await loadSources()", ok_i) + announce_i = body.find("announce(`Hidden folders", reload_i) + assert -1 < ok_i < hide_i < reload_i < announce_i, ( + "200: clear error → await loadSources → announce (LAST)" + ) + assert "for ${value}.`)" in body[announce_i:], "the confirmation names the source" + assert "enabled" in body[announce_i:] and "disabled" in body[announce_i:], ( + "the confirmation states the direction of the flip" + ) + + +def test_toggle_failure_reverts_the_box_and_shows_the_page_alert() -> None: + """toggleHidden failure: non-2xx → the server detail (apiDetail) + into the page-level alert line + the box REVERTS to the server + state + re-enables; network failure → the fixed reachable? line, + same revert. The UI never claims a state the server didn't save + (PLAN §7.4). The revert + re-enable happen on BOTH failure + branches (exactly twice each in the function).""" + body = _fn(_js(), "toggleHidden") + catch_i = body.find(".catch(") + assert catch_i != -1 + fail_slice = body[body.find("if (r.ok)"):catch_i] + assert "await apiDetail(" in fail_slice, "the server detail is apiDetail-extracted" + assert "showHiddenError(" in fail_slice, "the detail lands in the page alert line" + assert REVERT_STATE in fail_slice, "the box reverts to the server state" + assert "box.disabled = false" in fail_slice, "the box re-enables" + net_slice = body[catch_i:] + assert NETWORK_MESSAGE in net_slice, "the fixed network copy" + assert "showHiddenError(" in net_slice + assert REVERT_STATE in net_slice + assert "box.disabled = false" in net_slice + assert body.count(REVERT_STATE) == 2, "revert on BOTH failure branches" + assert body.count("box.disabled = false") == 2, "re-enable on BOTH failure branches" + assert "box.disabled = true" in body[: body.find("if (r.ok)")], ( + "the box disables at once, before any outcome" + ) + + +def test_error_line_helpers_and_the_healed_load_clear() -> None: + """showHiddenError/hideHiddenError drive the page-level + ``hiddenErrorEl`` (textContent + hidden); ``loadSources``'s + success path calls hideHiddenError() AFTER hideLoadError() — a + healed list clears the stale line (the phase-89 "happy path + heals the error state" precedent).""" + js = _js() + assert 'const hiddenErrorEl = root.querySelector("#git-sources-hidden-error")' in js, ( + "the page-local element grabber (root-scoped, the shell idiom)" + ) + show = _fn(js, "showHiddenError") + assert "hiddenErrorEl.textContent = message" in show + assert "hiddenErrorEl.hidden = false" in show + hide = _fn(js, "hideHiddenError") + assert 'hiddenErrorEl.textContent = ""' in hide + assert "hiddenErrorEl.hidden = true" in hide + load = _fn(js, "loadSources") + hide_load_i = load.find("hideLoadError()") + hide_hidden_i = load.find("hideHiddenError()", hide_load_i) + assert -1 < hide_load_i < hide_hidden_i, ( + "a healed load clears the hidden-toggle line (after the load-error clear)" + ) + + +# ---------- the cross-file wire contract (single source of truth) ---------- + + +def test_js_field_name_matches_the_python_patch_schema() -> None: + """The cross-file pin: the JS PATCH body key and the render field + both equal their ``app/schemas.py`` counterparts — + ``GitSourcePatchIn.include_hidden`` (the write side) and + ``GitSourceRow.include_hidden`` (the read side). A rename on + either side breaks the wire contract and this test.""" + js = _js() + body = _fn(js, "toggleHidden") + m = re.search(r"JSON\.stringify\(\{\s*(\w+)\s*:\s*wanted\s*\}\)", body) + assert m, "the PATCH body carries a single {: wanted} object" + js_body_key = m.group(1) + assert js_body_key == _python_class_field("GitSourcePatchIn"), ( + "the JS PATCH body key must match GitSourcePatchIn's field" + ) + make = _fn(js, "makeRow") + rm = re.search(r"s\.(\w+)\s*===\s*true", make) + assert rm, "makeRow reads the flag off the server row (s. === true)" + js_read_key = rm.group(1) + assert js_read_key == _python_class_field("GitSourceRow"), ( + "the render field must match GitSourceRow's field" + ) + assert js_body_key == js_read_key == "include_hidden", ( + "one field name for the whole wire contract" + ) + + +# ---------- the static shell markup ---------- + + +def test_page_error_line_is_a_role_alert_after_the_table() -> None: + """index.html: ``#git-sources-hidden-error`` is a ``

`` reusing + the existing ``.git-source-error`` class, ``role="alert"`` + + hidden by default, INSIDE ``#view-git-sources`` and AFTER + ``#git-sources-table-wrap`` in source order (the checkbox is a + table-cell control, so its failure announces at page level — the + ignore dialog carries its own error INSIDE the modal). It + appears exactly once.""" + html = _text(SHELL_HTML) + tag = re.search(r']*id="git-sources-hidden-error"[^>]*>', html) + assert tag, "#git-sources-hidden-error must be a real

" + open_tag = tag.group(0) + for attr in ('class="git-source-error"', 'role="alert"', "hidden"): + assert attr in open_tag, f"the error line carries {attr}" + view_i = html.find('id="view-git-sources"') + wrap_i = html.find('id="git-sources-table-wrap"') + err_i = html.find('id="git-sources-hidden-error"') + next_i = html.find('id="view-history"') + assert -1 < view_i < wrap_i < err_i < next_i, ( + "the line sits inside #view-git-sources, after #git-sources-table-wrap" + ) + assert html.count('id="git-sources-hidden-error"') == 1, ( + "the id is unique in the shell" + ) + + +# ---------- styles.css ---------- + + +def test_hidden_checkbox_css_rules_exist_with_accent_color() -> None: + """styles.css carries the phase-105 family: .git-source-hidden + (inline-flex, 44px hit height matching the action buttons, the + house ink), the checkbox (sized, ``accent-color: var(--brand)``), + the :disabled wait state (the .git-source-remove:disabled + idiom), and the "hidden on" tag (text + a distinct background — + never color alone). The checkbox comment RECORDS the + verified checked-state contrast (house style).""" + css = _css() + label = _css_rule(css, ".git-source-hidden") + assert "display: inline-flex" in label + assert "height: 44px" in label, "the ~44px hit height matches the action buttons" + assert "var(--ink)" in label + box = _css_rule(css, '.git-source-hidden input[type="checkbox"]') + assert "accent-color: var(--brand)" in box, "the --brand checkbox fill" + disabled = _css_rule(css, ".git-source-hidden:disabled") + assert "opacity" in disabled and "cursor: wait" in disabled, ( + "the .git-source-remove:disabled idiom" + ) + tag = _css_rule(css, ".git-source-hidden-count") + assert "var(--ink)" in tag and "var(--bg)" in tag, ( + "text + a distinct background (never color alone)" + ) + assert "url(http" not in css and "@import url(" not in css, ( + "no CDN (AGENTS.md rule 6)" + ) + + +def test_checkbox_contrast_is_verified_and_recorded() -> None: + """House style: the checked-state contrast of the native widget + is VERIFIED and the ratio RECORDED in the comment above the + checkbox rule (the executor note the task template carries): + the --bg-on---brand house pairing (5.2:1) on the built-in theme, + and the theme tab's AA gate (4.5:1) for every saved palette.""" + css = _css() + rule_i = css.find('.git-source-hidden input[type="checkbox"] {') + assert rule_i != -1 + header = css[css.rfind("/*", 0, rule_i):rule_i] + assert "VERIFIED" in header, "the contrast is verified (the executor note)" + assert "5.2:1" in header, "the built-in --bg-on---brand ratio is recorded" + assert "4.5:1" in header, "the AA gate for themed palettes is recorded" + + +# ---------- the module docstring ---------- + + +def test_module_docstring_carries_the_phase_105_contract() -> None: + """The git-sources.js module docstring gained the phase-105 + entry: the per-row Hidden checkbox (makeRow) → + PATCH {include_hidden} (task 03's optional field) → + loadSources + announce; failure reverts the box + + #git-sources-hidden-error (role=alert); env-fallback rows get + no checkbox (A3).""" + doc = _js().split("*/", 2)[0] # the module docstring (first block) + for frag in ( + "Phase 105 (task 05)", + "toggleHidden", + "{ include_hidden }", + "#git-sources-hidden-error", + "env-fallback rows get NO checkbox", + "REVERTS to the server state", + "LAST announcement", + ): + assert frag in doc, f"the module docstring lost: {frag!r}" diff --git a/tests/unit/test_importer.py b/tests/unit/test_importer.py index 67f9328..7fc1bc4 100644 --- a/tests/unit/test_importer.py +++ b/tests/unit/test_importer.py @@ -849,12 +849,14 @@ def test_no_progress_means_no_prewalk( extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS, ignore: tuple[str, ...] = (), + include_hidden: bool = False, ) -> list[Path]: - # Phase 89: the walker gained the ``ignore`` keyword — the sentinel - # accepts (and forwards) it to stay a drop-in. + # Phase 89: the walker gained the ``ignore`` keyword; phase 105: + # the ``include_hidden`` flag — the sentinel accepts (and forwards) + # both to stay a drop-in. nonlocal walk_calls walk_calls += 1 - return real_walker(r, extensions, excluded, ignore) + return real_walker(r, extensions, excluded, ignore, include_hidden) monkeypatch.setattr(importer, "iter_importable_files", counting) try: diff --git a/tests/unit/test_importer_include_hidden.py b/tests/unit/test_importer_include_hidden.py new file mode 100644 index 0000000..ffbb7f3 --- /dev/null +++ b/tests/unit/test_importer_include_hidden.py @@ -0,0 +1,161 @@ +"""Unit: the phase-105 hidden-folders flag in ``iter_importable_files``. + +Phase 105 (TODO.md L3 — "…a toggle per input (next to the ignores +button) to allow indexing hidden .folders."): ``include_hidden`` lifts +ONLY the dot-prefixed-component skip for one source. A1 +(owner-confirmed 2026-09-14): when True, files inside hidden folders +AND hidden files with an importable extension become importable; +``EXCLUDED_DIRS`` (``.venv``, ``node_modules``, ``.git``, …) stay +excluded in BOTH states; the extension filter always applies; the +phase-89 *ignore* tuple composes additively. A4: the default +(``False``) is byte-identical to the pre-phase-105 walk — pinned here +against a tmp fixture tree. The DB-facing behavior (hidden files +indexed / pruned through ``import_sources``) is integration-gated by +``tests/integration/test_importer_include_hidden.py``. +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from app.rag.importer import _include_hidden_for_root, iter_importable_files + +#: Explicit extension set (not the A9 config default) — the tests pin +#: the walk rules, not the config. +EXTS = frozenset({".md"}) + + +@pytest.fixture() +def tree(tmp_path: Path) -> Path: + """The task's fixture layout (plus the no-extension ``.env``). + + visible.md .hidden/note.md .notes.md + .venv/junk.md node_modules/x.md .hidden/.deep.md + keep/ok.md .env + """ + root = tmp_path / "HiddenFix" + (root / ".hidden").mkdir(parents=True) + (root / ".venv").mkdir() + (root / "node_modules").mkdir() + (root / "keep").mkdir() + (root / "visible.md").write_text("visible\n", encoding="utf-8") + (root / ".hidden" / "note.md").write_text("note\n", encoding="utf-8") + (root / ".notes.md").write_text("notes\n", encoding="utf-8") + (root / ".venv" / "junk.md").write_text("junk\n", encoding="utf-8") + (root / "node_modules" / "x.md").write_text("x\n", encoding="utf-8") + (root / ".hidden" / ".deep.md").write_text("deep\n", encoding="utf-8") + (root / "keep" / "ok.md").write_text("ok\n", encoding="utf-8") + (root / ".env").write_text("SECRET=1\n", encoding="utf-8") + return root + + +def _rels(paths: list[Path], root: Path) -> set[str]: + return {p.relative_to(root).as_posix() for p in paths} + + +# --- default (flag False): today's behavior, byte-identical ----------------- + + +def test_iter_default_flag_skips_all_hidden(tree: Path) -> None: + # A4: the default is the pre-phase-105 walk — hidden dir, hidden + # file, and the excluded dirs are all absent. Sorted order pinned + # verbatim (not just the set). + got = iter_importable_files(tree, EXTS) + assert got == [tree / "keep" / "ok.md", tree / "visible.md"] + + +def test_iter_default_flag_rejects_explicit_false(tree: Path) -> None: + assert iter_importable_files(tree, EXTS, include_hidden=False) == [ + tree / "keep" / "ok.md", + tree / "visible.md", + ] + + +# --- flag True (A1) ----------------------------------------------------------- + + +def test_iter_flag_on_admits_hidden_dirs_and_hidden_files(tree: Path) -> None: + got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree) + assert got == { + ".hidden/note.md", # inside a hidden dir + ".notes.md", # hidden file with an importable extension + ".hidden/.deep.md", # both at once + "visible.md", + "keep/ok.md", + } + + +def test_iter_flag_on_keeps_excluded_dirs_out(tree: Path) -> None: + # A1: EXCLUDED_DIRS are skipped in BOTH states — caches are never + # content, even with the flag on. + got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree) + assert ".venv/junk.md" not in got + assert "node_modules/x.md" not in got + + +def test_iter_flag_on_keeps_extension_filter_in_force(tree: Path) -> None: + # ``.env`` has no A9 extension — the extension filter is the real + # content gate, so a secret-flavoured file is never listed. + for flag in (False, True): + got = _rels(iter_importable_files(tree, EXTS, include_hidden=flag), tree) + assert ".env" not in got + + +# --- composition with the phase-89 ignore tuple ------------------------------- + + +def test_iter_flag_on_ignore_composes_additively(tree: Path) -> None: + # The ignored prefix still bites when the flag is ON; everything + # else hidden is admitted. + got = _rels( + iter_importable_files(tree, EXTS, include_hidden=True, ignore=(".hidden",)), + tree, + ) + assert got == {".notes.md", "visible.md", "keep/ok.md"} + + +def test_iter_ignore_only_flag_off(tree: Path) -> None: + # The pre-phase-105 combination still works untouched: ignore + # prefix on a visible subtree, flag at its default. + got = _rels(iter_importable_files(tree, EXTS, ignore=("keep",)), tree) + assert got == {"visible.md"} + + +# --- _include_hidden_for_root: the single read point -------------------------- + + +def test_include_hidden_for_root_none_map_is_false(tmp_path: Path) -> None: + root = tmp_path / "src" + root.mkdir() + assert _include_hidden_for_root(root, None) is False + + +def test_include_hidden_for_root_unlisted_root_is_false(tmp_path: Path) -> None: + root = tmp_path / "src" + root.mkdir() + assert _include_hidden_for_root(root, {"/other/root": True}) is False + + +def test_include_hidden_for_root_listed_true_is_true(tmp_path: Path) -> None: + root = tmp_path / "src" + root.mkdir() + assert _include_hidden_for_root(root, {str(root): True}) is True + + +def test_include_hidden_for_root_listed_false_is_false(tmp_path: Path) -> None: + root = tmp_path / "src" + root.mkdir() + assert _include_hidden_for_root(root, {str(root): False}) is False + + +def test_include_hidden_for_root_keys_by_str_root(tmp_path: Path) -> None: + # Two distinct Path objects whose ``str()`` agrees — the answer must + # be the same (the ``_ignore_for_root`` keying convention, phase 89). + key = str(tmp_path / "src") + a = tmp_path / "src" + b = tmp_path / "src" / "." # different object, same str() + assert str(a) == str(b) == key + mp = {key: True} + assert _include_hidden_for_root(a, mp) is True + assert _include_hidden_for_root(b, mp) is True diff --git a/tests/unit/test_models.py b/tests/unit/test_models.py index 98052be..49ba419 100644 --- a/tests/unit/test_models.py +++ b/tests/unit/test_models.py @@ -134,6 +134,42 @@ def test_git_source_python_default_empty_list() -> None: assert col.default.arg(None) == [], "the default must resolve to []" +def test_git_sources_include_hidden_column_contract() -> None: + """Phase 105: every source row carries its hidden-folders flag — + BOOLEAN, NOT NULL, server default ``false`` (a pre-phase-105 row + reads ``False``, so every existing source imports byte-identically + until the owner flips it — A4).""" + sources = Base.metadata.tables["git_sources"] + assert "include_hidden" in sources.c, ( + "git_sources must have the include_hidden column (phase 105)" + ) + col = sources.c["include_hidden"] + assert col.nullable is False, "git_sources.include_hidden must be NOT NULL" + sd = col.server_default + assert isinstance(sd, DefaultClause), "include_hidden needs a server default" + assert isinstance(sd.arg, TextClause), ( + "the server default must be the literal SQL text false" + ) + assert sd.arg.text == "false", "include_hidden server default must be false" + + +def test_git_source_python_default_include_hidden_false() -> None: + """A freshly constructed row (no DB) resolves the flag to ``False`` + via the Python-side default (``default=False``) — the ORM INSERT-time + default, so an ORM insert that omits the column inserts ``false`` + rather than NULL (the server default ``false`` independently covers + non-ORM inserts — A4).""" + from app.models import GitSource + + row = GitSource(url="https://example.com/r.git", kind="git") + col = row.__table__.c["include_hidden"] + assert col.default is not None, ( + "include_hidden needs a Python-side (INSERT-time) default" + ) + # A scalar default: the arg IS the value (not a callable). + assert col.default.arg is False, "the default must resolve to False" + + def test_doc_drafts_column_contract() -> None: """Phase 59: the editable triple (title/path/body) + status + timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL diff --git a/tests/unit/test_sync_button.py b/tests/unit/test_sync_button.py index b5da085..587834f 100644 --- a/tests/unit/test_sync_button.py +++ b/tests/unit/test_sync_button.py @@ -733,6 +733,7 @@ class _GatedImport: session: object = None, progress: Callable[[str, str, int, int], None] | None = None, ignore_by_root: dict[str, list[str]] | None = None, # phase 89 + include_hidden_by_root: dict[str, bool] | None = None, # phase 105 ) -> ImportSummary: self.prune_flags.append(prune) if progress is not None: