phase: 105_hidden_folders_toggle
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s

All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
This commit is contained in:
2026-09-12 21:26:30 -04:00
parent ecc921098a
commit d731169b8b
49 changed files with 3553 additions and 93 deletions
+676
View File
@@ -0,0 +1,676 @@
"""Phase 105 story E2E (Playwright): the per-source hidden-folders toggle —
the "Hidden" checkbox next to the "Ignore paths" button makes a source's
dot-prefixed paths indexable (``/git-sources.html``).
Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-14 —
TODO-derived, no separate user-story file).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov
The story gate proves the whole item end to end through the REAL page +
REAL API + REAL sync pipeline (the in-app Sync's background task, mock
LLM). **No git, no network** — the suite's single source is a local
directory row (``kind="local"``, phase 38) over a fixture tree with one
visible file, one hidden-folder file (the phase-105 subject), and one
``.venv`` file (``EXCLUDED_DIRS`` — never indexed in either state, A1).
The module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL
(the ``test_source_ignore_paths.py`` module-env pattern) that is NEVER
synced or cloned: every sync in this suite runs with the local DB row in
place (DB rows win over the env fallback), and the one env-fallback test
never triggers a sync.
Contract under test:
* anonymous: the ``#git-sources-gate`` sign-in gate, the manager hidden,
NO ``/api/git-sources`` call on load, 403 on
``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}``
with a bool-only body (the phase-89 anonymous pin extended to the
toggle payload);
* A4: with the default row, a sync indexes ``visible.md`` ONLY —
``detail.files`` counts one, ``.hidden/note.md`` has no ``documents``
row, the checkbox renders UNCHECKED and no "hidden on" tag;
* A1: flipping the checkbox on (the real click) → the PATCH 200 lands
(the "hidden on" tag appears, the announcer ``role=status`` fires the
confirmation AFTER the reload line) → sync → ``.hidden/note.md`` IS
indexed (``documents`` row present; the KB catalog lists it — the
tree/catalog is DB-driven, no extra surface) and ``.venv/junk.md`` is
STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders
CHECKED (server state);
* A2: flipping it OFF (real click) → sync → ``detail.pruned`` includes
the hidden doc, the catalog no longer lists it, the tag is gone;
* A3: the env-fallback view (table-empty state) renders the "from .env"
tag with NO Hidden checkbox and NO "Ignore paths" button;
* a11y + error surface: the checkbox has a full accessible name
containing the source location (``Index hidden folders for local
source: …``), is keyboard-focusable, the tag text is "hidden on"
(never color alone); ``#git-sources-hidden-error`` exists with
``role="alert"`` and stays ``hidden`` through the happy path — and a
network-aborted PATCH shows the canned "not changed" message and
reverts the box to the server state (§7.4 never-stale).
Test → contract mapping (one test per bullet, the phase-89 suite's
shape):
1. ``test_anonymous_gate_and_403s``
2. ``test_hidden_off_by_default``
3. ``test_toggle_on_indexes_hidden_folders``
4. ``test_toggle_off_prunes_hidden``
5. ``test_env_fallback_rows_have_no_toggle``
6. ``test_toggle_a11y_and_error_surface``
"""
from __future__ import annotations
import os
import subprocess
import sys
import time
import uuid
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_HIDDEN", "8141"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is
#: NEVER cloned (the env-fallback test reads it; the sync tests all run
#: with the local DB row in place, so the env list never reaches a
#: clone).
ENV_SOURCE = "https://github.com/reese/env-alpha.git"
#: The three fixture files (source-relative POSIX paths — exactly the
#: strings ``documents.path`` stores).
VISIBLE_MD = "visible.md"
HIDDEN_NOTE = ".hidden/note.md"
VENV_JUNK = ".venv/junk.md"
#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the
#: test_sync_button.py polling idiom) — real import of ≤2 small files
#: against the mock LLM; generous budget.
SYNC_TIMEOUT_S = 60.0
SYNC_TICK_S = 2.0
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The local source's on-disk tree (task 06 step 1).
Module-scoped (the same reasoning as ``test_sync_button.py``'s
module-scoped fixture note): ``tmp_path`` is function-scoped while
the module-scoped app + the seeded row reference the dir for the
module's lifetime, so it is built under ``tmp_path_factory`` (the
same pytest-managed temp area, module-safe). Plain files — no git:
a ``kind="local"`` row is walked directly by the sync (phase 38).
"""
root = tmp_path_factory.mktemp("hidden_src")
(root / ".hidden").mkdir()
(root / ".venv").mkdir()
(root / VISIBLE_MD).write_text(
"# Visible\n"
"\n"
"The visible doc — indexed in EVERY state (flag on and off).\n",
encoding="utf-8",
)
# The phase-105 subject: a file INSIDE a hidden (dot-prefixed)
# folder — indexed only when the row's flag is on (A1).
(root / HIDDEN_NOTE).write_text(
"# Hidden note\n"
"\n"
"A file inside a hidden (dot) folder — the toggle's subject:\n"
"indexed, embedded, and summarized like any visible file when\n"
"the flag is on; pruned from the KB when it flips off (A2).\n",
encoding="utf-8",
)
# EXCLUDED_DIRS content: ``.venv`` is a cache, never content —
# skipped in BOTH flag states (A1).
(root / VENV_JUNK).write_text(
"venv junk — EXCLUDED_DIRS content: never indexed, both states.\n",
encoding="utf-8",
)
return root
@pytest.fixture(scope="module")
def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]:
"""The real app under test — per-module env (the conftest pattern,
module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env
fallback's subject — never cloned), module-scratch checkouts/
upload dirs, the mock LLM. No git anywhere in this suite's path."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env
# shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The empty-table env fallback (only read while ``git_sources`` is
# EMPTY — the env-fallback test; never cloned in this suite).
env["BOR_GIT_SOURCES"] = ENV_SOURCE
# Module-scratch dirs (never reached by this suite's local-row
# syncs — kept explicit so a shared checkouts dir cannot leak rows
# into the walk).
scratch = source_dir.parent
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern, the
test_sync_button.py module-env DSN): the sync's counts and the
catalog must be each test's own doing."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources"
)
)
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""This suite owns ``git_sources`` AND the KB tables (the E2E
isolation pattern): suites run in isolation but share one
Postgres, and a leftover row would flip the app from the
``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover
document would skew the prune counts). Empty BOTH before and
after every test."""
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str:
"""(Re)seed the stored row for the module fixture dir via the REAL
admin API (201) — ``kind="local"`` — with NO ``ignore_paths`` and
NO ``include_hidden`` (A4: absent at create → stored ``False``) —
then reload the view so the page's table re-fetches the row (the
API seed is server-side: the mounted view does not know about it
without a re-show). Returns the stored (expanded) path — the
row's ``url`` location value, the same string the page renders
and aria-labels."""
r = page.request.post(
f"{app_url}/api/git-sources",
data={"kind": "local", "path": str(source_dir)},
)
assert r.status == 201, r.text
body = r.json()
assert body["ignore_paths"] == [] # the default list, round-tripped
assert body["include_hidden"] is False # A4: default off
page.reload()
_admin_git_sources_page(page, app_url)
return body["url"]
def run_sync(page: Page, app_url: str) -> dict[str, Any]:
"""``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal
state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s
budget). Returns the terminal status body."""
r = page.request.post(f"{app_url}/api/sync")
assert r.status == 202, r.text
deadline = time.monotonic() + SYNC_TIMEOUT_S
body: dict[str, Any] = {}
while True:
s = page.request.get(f"{app_url}/api/sync/status")
assert s.status == 200, s.text
body = s.json()
if body["state"] in ("success", "failed"):
return body
assert time.monotonic() < deadline, (
f"sync did not reach a terminal state: {body}"
)
time.sleep(SYNC_TICK_S)
def _catalog_paths(page: Page, app_url: str) -> list[str]:
"""The RAG catalog's data source (``GET /api/docs`` — the Sources
page's table): every ``documents.path`` (source-relative POSIX).
The tree/catalog is DB-driven — a newly indexed hidden document
appears here automatically (no extra surface)."""
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200, r.text
return [d["path"] for d in r.json()["documents"]]
def _db_document_paths() -> list[str]:
"""Every ``documents.path`` straight from the DB — the literal
"no ``documents`` row" assertion (the catalog is the same table,
read through the API)."""
with SessionLocal() as db:
rows = db.execute(text("SELECT path FROM documents")).fetchall()
return [row[0] for row in rows]
def _row(page: Page, value: str) -> Any:
"""The table row whose mono location cell shows ``value`` (re-resolved
on every call — the row re-renders from the server after each
successful toggle)."""
return page.locator("#git-sources-tbody tr", has_text=value)
def _hidden_box(page: Page, value: str) -> Any:
"""The row's Hidden checkbox (fresh locator — the element is
re-created by the post-toggle re-render)."""
return _row(page, value).locator(".git-source-hidden-box")
def _flip_hidden(page: Page, value: str, wanted_on: bool) -> None:
"""Click the row's Hidden checkbox and wait for the §7.4 success
lifecycle to settle: the PATCH 200 landed, the row re-rendered
from the server, and the CONFIRMATION is the last announcer message
(the reload's "N sources listed." cannot overwrite it — the
phase-89 order)."""
_hidden_box(page, value).click()
tag = _row(page, value).locator(".git-source-hidden-count")
if wanted_on:
expect(tag).to_have_text("hidden on", timeout=30_000)
else:
expect(tag).to_have_count(0, timeout=30_000)
expect(
page.locator("#git-sources-announcer")
).to_have_text(
f"Hidden folders {'enabled' if wanted_on else 'disabled'} for {value}."
)
# ---------------------------------------------------------------------------
# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. the
# bool-only toggle PATCH)
# ---------------------------------------------------------------------------
def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None:
page.set_default_timeout(30_000)
# Track every /api/git-sources request the page itself makes — the
# gate must be reached WITHOUT touching the admin API (the
# test_source_ignore_paths.py pattern).
api_calls: list[str] = []
page.on(
"request",
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
)
page.goto(app_url + GIT_SOURCES_URL)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
# The sign-in gate (the #sources-gate pattern, phase 16)…
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
expect(
gate.locator("a[href='/login.html?next=/git-sources.html']")
).to_have_count(1)
# …and the manager is absent/inert: table + form + env note, all
# inside the hidden #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers on GET/POST AND the
# phase-105 toggle PATCH with a bool-only body (the row's list is
# untouched — the toggle's exact payload is gated too).
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
f"{app_url}/api/git-sources", data={"url": ENV_SOURCE}
).status
== 403
)
assert (
page.request.patch(
f"{app_url}/api/git-sources/{uuid.uuid4()}",
data={"include_hidden": True},
).status
== 403
)
# ---------------------------------------------------------------------------
# 2. A4: the default row is byte-identical to pre-phase-105 — the sync
# indexes visible.md ONLY; the checkbox renders UNCHECKED, no tag
# ---------------------------------------------------------------------------
def test_hidden_off_by_default(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# A4: the default row renders the NEW control in its default state —
# the checkbox UNCHECKED (checked comes only from server state) and
# NO "hidden on" tag (pre-phase-105 surface + the control).
box = _hidden_box(page, stored_path)
expect(box).to_have_count(1)
expect(box).not_to_be_checked()
expect(_row(page, stored_path).locator(".git-source-hidden-count")).to_have_count(0)
# The real sync: ``visible.md`` ONLY — the walk skipped the
# dot-prefixed ``.hidden/`` component (A4 byte-identical default)
# AND ``.venv/`` (EXCLUDED_DIRS in both states).
sync = run_sync(page, app_url)
assert sync["state"] == "success", sync
detail = sync["detail"]
assert detail["files"] == 1, detail
assert detail["added"] == 1, detail
assert detail["pruned"] == 0, detail
assert detail["errors"] == 0, detail
# The catalog holds exactly the visible file — no ``documents`` row
# for the hidden note (the direct DB read makes "no row" literal)…
paths = _catalog_paths(page, app_url)
assert VISIBLE_MD in paths, f"visible.md missing from the catalog: {paths}"
assert HIDDEN_NOTE not in paths, (
f"hidden note leaked into the catalog with the flag off: {paths}"
)
assert VENV_JUNK not in paths, f"EXCLUDED_DIRS leaked into the catalog: {paths}"
assert _db_document_paths() == [VISIBLE_MD]
# ---------------------------------------------------------------------------
# 3. A1: the real click flips the flag on — the PATCH 200 lands (tag +
# last-announce confirmation), the next sync indexes the hidden
# folder; EXCLUDED_DIRS stays excluded; the box re-renders CHECKED
# ---------------------------------------------------------------------------
def test_toggle_on_indexes_hidden_folders(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# Baseline: the default sync indexes the visible file only.
first = run_sync(page, app_url)
assert first["state"] == "success", first
assert first["detail"]["files"] == 1, first["detail"]
# A1: flip the checkbox on (the real click) — the §7.4 lifecycle:
# PATCH 200 → the "hidden on" tag appears in the source cell and
# the confirmation is the LAST announcer message (the reload's
# "1 source listed." landed first).
_flip_hidden(page, stored_path, wanted_on=True)
# The checkbox re-renders CHECKED from the server state (never a
# local flip) and the API round-trips include_hidden: true.
expect(_hidden_box(page, stored_path)).to_be_checked()
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
sources = r.json()["sources"]
assert [s["include_hidden"] for s in sources] == [True]
assert [s["ignore_paths"] for s in sources] == [[]] # the list untouched
# The NEXT sync indexes the hidden-folder file like any visible
# file (embedded + chunked by the mock LLM)…
second = run_sync(page, app_url)
assert second["state"] == "success", second
detail = second["detail"]
assert detail["files"] == 2, detail
assert detail["added"] == 1, detail # the hidden note only
assert detail["unchanged"] == 1, detail # visible.md
# …and ``.venv/junk.md`` is STILL absent — EXCLUDED_DIRS content is
# never indexed, in either state (A1).
paths = _catalog_paths(page, app_url)
assert HIDDEN_NOTE in paths, (
f"hidden note missing from the catalog after the on-sync: {paths}"
)
assert VISIBLE_MD in paths, paths
assert VENV_JUNK not in paths, (
f"EXCLUDED_DIRS leaked into the catalog with the flag on: {paths}"
)
assert set(_db_document_paths()) == {VISIBLE_MD, HIDDEN_NOTE}
# ---------------------------------------------------------------------------
# 4. A2: flipping the flag OFF prunes the previously indexed hidden doc
# from the KB on the next sync
# ---------------------------------------------------------------------------
def test_toggle_off_prunes_hidden(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
# On + sync: the hidden note is in the KB.
_flip_hidden(page, stored_path, wanted_on=True)
first = run_sync(page, app_url)
assert first["state"] == "success", first
assert first["detail"]["files"] == 2, first["detail"]
assert first["detail"]["added"] == 2, first["detail"]
assert HIDDEN_NOTE in _catalog_paths(page, app_url)
# A2: flip it OFF (the real click) — the previously indexed hidden
# file leaves the KB on the next sync (the seen-set prune — the
# phase-89 A2 / A9 precedent; the tag is gone).
_flip_hidden(page, stored_path, wanted_on=False)
r = page.request.get(f"{app_url}/api/git-sources")
assert [s["include_hidden"] for s in r.json()["sources"]] == [False]
second = run_sync(page, app_url)
assert second["state"] == "success", second
detail = second["detail"]
assert detail["pruned"] == 1, detail # the hidden doc
assert detail["files"] == 1, detail # the walk is back to visible-only
paths = _catalog_paths(page, app_url)
assert HIDDEN_NOTE not in paths, (
f"the hidden doc survived the flag-off prune: {paths}"
)
assert VISIBLE_MD in paths, f"visible.md must survive the prune: {paths}"
assert _db_document_paths() == [VISIBLE_MD]
# ---------------------------------------------------------------------------
# 5. A3: env-fallback rows have no toggle (no DB row to store a flag on)
# ---------------------------------------------------------------------------
def test_env_fallback_rows_have_no_toggle(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# The autouse fixture truncated git_sources — the table is EMPTY,
# so the module app's BOR_GIT_SOURCES URL is the effective list
# (the test_source_ignore_paths.py module-env pattern).
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
row = page.locator("#git-sources-tbody tr")
expect(row).to_contain_text(ENV_SOURCE)
# The env-fallback surface: the "from .env" tag (never color
# alone)…
expect(row.locator(".git-source-env-tag")).to_have_count(1)
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
# …and A3: NO per-row controls at all — neither Remove, nor the
# phase-89 "Ignore paths" button, nor the phase-105 Hidden checkbox
# (and no "hidden on" tag).
expect(row.locator(".git-source-remove")).to_have_count(0)
expect(row.locator(".git-source-ignore")).to_have_count(0)
expect(row.locator(".git-source-hidden")).to_have_count(0)
expect(row.locator(".git-source-hidden-box")).to_have_count(0)
expect(row.locator(".git-source-hidden-count")).to_have_count(0)
# The env-fallback note explains the active list's origin…
expect(page.locator("#git-sources-env-note")).to_be_visible()
# …and the API agrees: from_env true, null ids, the env URL, an
# empty ignore list, and include_hidden False (no DB row to store a
# list or a flag on — the GitSourceRow contract, task 03).
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert body["from_env"] is True
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE]
assert all(
s["id"] is None and s["ignore_paths"] == [] and s["include_hidden"] is False
for s in body["sources"]
)
# ---------------------------------------------------------------------------
# 6. a11y + the error surface: the accessible name, keyboard focus, the
# text tag, and the role=alert line (hidden on the happy path, the
# canned message + revert on a network failure)
# ---------------------------------------------------------------------------
def test_toggle_a11y_and_error_surface(
page: Page, app_url: str, db_ready: None, source_dir: Path
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
stored_path = _seed_local_source(page, app_url, source_dir)
box = _hidden_box(page, stored_path)
expect(box).to_have_count(1)
# The full accessible name contains the source location (the
# aria-label — the ONLY place the value appears, setAttribute,
# never innerHTML)…
expect(box).to_have_attribute(
"aria-label", f"Index hidden folders for local source: {stored_path}"
)
expect(
page.get_by_role(
"checkbox", name=f"Index hidden folders for local source: {stored_path}"
)
).to_have_count(1)
# …it is keyboard-reachable — a plain Tab sequence from the page
# top reaches it (one full focus cycle covers every focusable
# element; the visible ring is the GLOBAL :focus-visible rule,
# styles.css — unit-pinned, no per-control rule)…
for _ in range(60):
page.keyboard.press("Tab")
if box.evaluate("el => el === document.activeElement"):
break
expect(box).to_be_focused()
# …and it receives programmatic focus too (the a11y trees agree).
box.focus()
expect(box).to_be_focused()
# …and the visible "Hidden" label wraps the box (text + control —
# never color or icon alone).
label = page.locator("label.git-source-hidden")
expect(label).to_have_count(1)
expect(label).to_contain_text("Hidden")
# The page-level error line exists, is a real role=alert, and stays
# hidden through the happy path.
error = page.locator("#git-sources-hidden-error")
expect(error).to_have_count(1)
assert error.get_attribute("role") == "alert"
expect(error).to_be_hidden()
# The tag is TEXT (never color alone — WCAG 1.4.1): flip on…
_flip_hidden(page, stored_path, wanted_on=True)
expect(
_row(page, stored_path).locator(".git-source-hidden-count")
).to_have_text("hidden on")
expect(error).to_be_hidden() # the happy path never touches the line
# §7.4 failure path: a network-level abort of the PATCH → the
# canned "not changed" message in the role=alert line and the box
# reverts to the SERVER state (still ON — the request never
# landed) and re-enables. The tag stays (the server is unchanged).
def _abort_patch(route: Any) -> None:
if route.request.method == "PATCH":
route.abort()
else:
route.continue_()
page.route("**/api/git-sources/**", _abort_patch)
try:
_hidden_box(page, stored_path).click()
expect(error).to_be_visible(timeout=30_000)
expect(error).to_have_text(
"Could not reach the server — the setting was not changed."
)
expect(
_row(page, stored_path).locator(".git-source-hidden-count")
).to_have_count(1) # the server state is unchanged: still on
reverted = _hidden_box(page, stored_path)
expect(reverted).to_be_checked() # reverted to the server state
expect(reverted).to_be_enabled() # re-enabled — never stale
finally:
page.unroute("**/api/git-sources/**")
+197 -20
View File
@@ -32,9 +32,16 @@ Contract under test:
* ignore paths (phase 89) — POST accepts the RAW box lines for both
kinds (optional, absent → ``[]``), stored normalized (A1) with the A4
fixed-detail 422s (shared gate with PATCH); GET reports each row's
stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only)
replaces the list wholesale (A5 — an empty list clears all), 404
unknown id, 422 fixed details for the A4 limits, the row otherwise
stored list (env rows ``[]``);
* hidden-folders flag (phase 105) — POST accepts ``include_hidden`` for
both kinds (optional, absent → stored ``False``, A4); GET reports the
stored flag (env rows ``False`` — no DB row to store a flag on);
* ``PATCH /{source_id}`` (admin-only) — phase 89 A5 + phase 105: each
field is optional, present-wins — ``ignore_paths`` when present
REPLACES the list wholesale (A5 — an empty list clears all),
``include_hidden`` when present sets the flag, both absent → 200
no-op, 404 unknown id, 422 fixed details for the A4 limits, a
rejected list never half-applies the flag, the row otherwise
unchanged.
``git_sources`` is global state: truncated around every test.
@@ -99,10 +106,14 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# The phase-89 ignore-list PATCH is gated the same way.
# The phase-89 ignore-list PATCH is gated the same way — and so is
# the phase-105 bool-only payload (the toggle's exact request).
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"include_hidden": True})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# Nothing landed in the table.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
@@ -132,8 +143,10 @@ def test_get_empty_table_with_env_returns_env_rows(
"url": "https://a.example.com/one.git",
"path": None,
"added_at": None,
# Env rows have no DB row to store a list on (phase 89).
# Env rows have no DB row to store a list on (phase 89) or
# a flag on (phase 105).
"ignore_paths": [],
"include_hidden": False,
},
{
"id": None,
@@ -142,6 +155,7 @@ def test_get_empty_table_with_env_returns_env_rows(
"path": None,
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
},
]
@@ -208,9 +222,11 @@ def test_post_creates_trimmed_and_list_stops_using_env(
uuid.UUID(body["id"])
assert body["added_at"] is not None
# Phase 89: the response gains ``ignore_paths`` — absent at create
# time → ``[]``.
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
# time → ``[]``; phase 105 adds ``include_hidden`` — absent →
# ``False`` (A4).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert body["ignore_paths"] == []
assert body["include_hidden"] is False
# The DB row now wins: from_env False, the env URL is gone from the list.
body = admin_client.get("/api/git-sources").json()
@@ -321,9 +337,11 @@ def test_post_local_creates_stored_row_with_expanded_path(
body = r.json()
# The phase-35 response shape is unchanged — the local row reports
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
# phase 89 adds ``ignore_paths`` (absent → ``[]``).
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
# phase 89 adds ``ignore_paths`` (absent → ``[]``); phase 105 adds
# ``include_hidden`` (absent → ``False``, A4).
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert body["ignore_paths"] == []
assert body["include_hidden"] is False
uuid.UUID(body["id"])
assert body["url"] == str(real_dir)
assert body["added_at"] is not None
@@ -556,6 +574,7 @@ def test_delete_removes_row_and_falls_back_to_env(
"path": None,
"added_at": None,
"ignore_paths": [], # env rows: no DB row to store a list on
"include_hidden": False, # … or a flag on (phase 105)
}
]
db.execute(text("TRUNCATE chunks, documents"))
@@ -671,6 +690,7 @@ def test_get_reports_stored_ignore_paths_and_env_rows_empty(
"path": None,
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
}
]
@@ -690,8 +710,9 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
assert r.status_code == 200, r.text
body = r.json()
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert body["ignore_paths"] == ["a", "b"] # normalized
assert body["include_hidden"] is False # untouched by a list-only PATCH
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Round-trip through GET.
@@ -706,17 +727,34 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == []
def test_patch_missing_field_is_422(admin_client: TestClient) -> None:
"""``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent
field is a 422 with the model's own detail."""
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"})
assert created.status_code == 201
r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={})
assert r.status_code == 422
# The model's own (Pydantic) detail — the missing required field.
assert any(
item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"]
def test_patch_empty_body_is_a_noop_200(admin_client: TestClient) -> None:
"""Phase 105: BOTH fields absent (or explicit None) → 200 no-op —
the row is untouched. (The pre-phase-105 pin of an absent
``ignore_paths`` as a 422 is retired — the list is optional now; a
PRESENT list keeps the phase-89 A5 replace semantics.)"""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/noop.git", "ignore_paths": ["a/b"]},
)
assert created.status_code == 201
before = created.json()
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={})
assert r.status_code == 200, r.text
body = r.json()
assert body["ignore_paths"] == ["a/b"]
assert body["include_hidden"] is False
for key in ("id", "url", "added_at"):
assert body[key] == before[key]
# Explicit None values are "absent" too — still a no-op.
r = admin_client.patch(
f"/api/git-sources/{before['id']}", json={"ignore_paths": None, "include_hidden": None}
)
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == ["a/b"]
assert r.json()["include_hidden"] is False
# The row never changed — round-trip through GET.
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a/b"]
def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None:
@@ -772,3 +810,142 @@ def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None:
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]})
assert r.status_code == 200, r.text
assert r.json()["ignore_paths"] == [long_entry]
# --- hidden-folders flag (phase 105) ----------------------------------------
def test_get_reports_include_hidden_default_false(admin_client: TestClient, db: Session) -> None:
"""A fresh stored row (no flag passed) reports ``include_hidden:
false`` — in the GET round-trip AND on the model (the column's
server default, A4)."""
db.add(GitSource(url="https://example.com/fresh.git"))
db.commit()
body = admin_client.get("/api/git-sources").json()
assert body["sources"][0]["include_hidden"] is False
row = db.scalars(select(GitSource)).one()
assert row.include_hidden is False
def test_post_stores_include_hidden_both_kinds(
admin_client: TestClient, tmp_path: Path
) -> None:
"""POST accepts ``include_hidden`` for both kinds (present → stored
as sent); the 201 body and the GET round-trip report it."""
r = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/hidden-git.git", "include_hidden": True},
)
assert r.status_code == 201, r.text
assert r.json()["include_hidden"] is True
real_dir = tmp_path / "hidden-local"
real_dir.mkdir()
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "path": str(real_dir), "include_hidden": True},
)
assert r.status_code == 201, r.text
assert r.json()["include_hidden"] is True
by_url = {
s["url"]: s["include_hidden"]
for s in admin_client.get("/api/git-sources").json()["sources"]
}
assert by_url["https://example.com/hidden-git.git"] is True
assert by_url[str(real_dir)] is True
def test_patch_bool_only_sets_flag_leaves_list(admin_client: TestClient) -> None:
"""The toggle's exact payload — ``{"include_hidden": …}`` alone:
the flag is set, the ignore list is UNCHANGED (present-wins, phase
105)."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/toggle.git", "ignore_paths": ["a/b"]},
)
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": True})
assert r.status_code == 200, r.text
body = r.json()
assert body["include_hidden"] is True
assert body["ignore_paths"] == ["a/b"] # untouched
assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True
# And back off again.
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": False})
assert r.status_code == 200, r.text
assert r.json()["include_hidden"] is False
assert r.json()["ignore_paths"] == ["a/b"]
def test_patch_list_only_replaces_list_leaves_flag(admin_client: TestClient) -> None:
"""The dialog's exact payload — ``{"ignore_paths": …}`` alone:
the list is REPLACED (normalized, phase-89 A5), the flag is
UNCHANGED — byte-identical to the pre-phase-105 dialog PATCH."""
created = admin_client.post(
"/api/git-sources",
json={
"url": "https://example.com/dialog.git",
"ignore_paths": ["a/b"],
"include_hidden": True,
},
)
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": ["c/d", " e "]})
assert r.status_code == 200, r.text
body = r.json()
assert body["ignore_paths"] == ["c/d", "e"] # normalized
assert body["include_hidden"] is True # untouched
assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True
def test_patch_both_fields_apply_independently(admin_client: TestClient) -> None:
"""``{"ignore_paths": [], "include_hidden": true}`` — the list is
cleared AND the flag is set in one request."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/both.git", "ignore_paths": ["old/one"]},
)
assert created.status_code == 201
sid = created.json()["id"]
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": [], "include_hidden": True}
)
assert r.status_code == 200, r.text
body = r.json()
assert body["ignore_paths"] == [] # cleared
assert body["include_hidden"] is True # set
def test_patch_422_does_not_half_apply_flag(admin_client: TestClient) -> None:
"""A PRESENT bad list 422s with the fixed A4 details — and the
OTHER field in the same body never half-applies: after each 422
the flag is still the stored value (validation raises before any
assignment, request transaction untouched)."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://example.com/half.git", "ignore_paths": ["keep"]},
)
assert created.status_code == 201
sid = created.json()["id"]
for payload, detail in (
([" "], "ignore paths must be non-empty"),
([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"),
(["a" * 501], "an ignore path exceeds 500 characters"),
):
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": payload, "include_hidden": True}
)
assert r.status_code == 422, f"{detail!r}: {r.text}"
assert r.json()["detail"] == detail
# Nothing half-applied: list AND flag are both still the originals.
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["ignore_paths"] == ["keep"]
assert row["include_hidden"] is False
+190 -11
View File
@@ -27,15 +27,19 @@ is asserted against the canned value).
from __future__ import annotations
import re
from collections.abc import Iterator
from pathlib import Path
import pytest
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import GitSource
from app.rag.importer import ImportSummary
from scripts import import_docs
from scripts.git_sync import GitSyncError
from tests.fakes import FakeEmbedder
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
@@ -47,11 +51,31 @@ def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource:
def _local_row(
path: str,
ignore_paths: list[str] | None = None,
include_hidden: bool = False,
) -> GitSource:
"""A local row as the phase-38 API stores it: the expanded path in
both ``path`` and the NOT-NULL ``url`` location column (plus the
phase-89 ignore list, empty by default)."""
return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or [])
phase-89 ignore list and the phase-105 hidden-folders flag, both
defaulting off — A4)."""
return GitSource(
url=path, kind="local", path=path,
ignore_paths=ignore_paths or [], include_hidden=include_hidden,
)
@pytest.fixture()
def clean_documents(db: Session) -> Iterator[None]:
"""Phase 105: the real-import CLI tests write ``documents``/
``chunks`` (the canonical KB state) — global, truncated around
every such test."""
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
class FakeImportSources:
@@ -68,10 +92,12 @@ class FakeImportSources:
prune: bool = False,
limit: int | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
) -> ImportSummary:
self.calls.append(
{"sources": list(sources), "prune": prune, "limit": limit,
"ignore_by_root": ignore_by_root}
"ignore_by_root": ignore_by_root,
"include_hidden_by_root": include_hidden_by_root}
)
return ImportSummary(files=1, added=1)
@@ -168,10 +194,16 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
sources_dir=str(tmp_path / "bor"),
)
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
# Phase 105: flag-off rows still contribute their root — with False
# (A4), keyed by the SAME root string the importer sees.
assert hidden_map == {
str(tmp_path / "bor" / "homelab"): False,
str(tmp_path / "bor" / "deploy"): False,
}
assert calls == [
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
@@ -186,10 +218,11 @@ def test_resolve_sources_cli_source_wins(
settings = _settings(git_sources="https://host/a/repo.git")
manual = tmp_path / "Manual"
sources, ignore_map = import_docs._resolve_sources([manual], settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources([manual], settings)
assert sources == [manual]
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
assert hidden_map == {} # phase 105: manual dirs have no rows → hidden skipped
assert calls == [] # git is never touched when --source is given
@@ -210,10 +243,12 @@ def test_resolve_sources_db_rows_win_over_env(
sources_dir=str(tmp_path / "bor"),
)
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only"]
assert ignore_map == {} # phase 89: no row carries a list → empty map
# Phase 105: the default-flag row contributes its root with False (A4).
assert hidden_map == {str(tmp_path / "bor" / "only"): False}
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
@@ -222,9 +257,10 @@ def test_resolve_sources_defaults_when_nothing_configured(
) -> None:
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
sources, ignore_map = import_docs._resolve_sources(None, _settings())
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, _settings())
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
assert ignore_map == {} # phase 89: the legacy fallback has no rows
assert hidden_map == {} # phase 105: the legacy fallback has no rows
def test_resolve_sources_rows_branch_builds_ignore_map(
@@ -250,11 +286,13 @@ def test_resolve_sources_rows_branch_builds_ignore_map(
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
# Keyed by the SAME string the importer sees (the root, not the name).
assert ignore_map == {str(local_dir): ["ignore/"]}
# Phase 105: both rows are flag-off → per-root False entries (A4).
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
def test_resolve_sources_two_rows_sharing_root_string_extend(
@@ -279,11 +317,14 @@ def test_resolve_sources_two_rows_sharing_root_string_extend(
)
settings = _settings(sources_dir=str(tmp_path / "bor"))
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
shared = str(tmp_path / "bor" / "shared")
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
assert ignore_map == {shared: ["a/", "b"]} # union, row order
# Phase 105 collision: the shared root gets the OR of the flags —
# both rows off here, so one False entry for the one root string.
assert hidden_map == {shared: False}
def test_main_rows_branch_passes_ignore_map_to_import(
@@ -318,9 +359,145 @@ def test_main_rows_branch_passes_ignore_map_to_import(
call = fake_import.calls[0]
assert call["sources"] == [local_dir]
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
# Phase 105: the default-flag row passes the per-root map too — a
# False entry, not an absent key (the importer reads it per root).
assert call["include_hidden_by_root"] == {str(local_dir): False}
assert call["prune"] is False # the CLI's no-prune default is unchanged
# --- phase 105: per-row hidden-folders flag ---------------------------------
def test_main_rows_branch_passes_include_hidden_map_to_import(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Phase 105: a ``kind=local`` row with ``include_hidden=True`` →
``main`` passes the per-root flag map to ``import_sources`` (keyed
by the directory string — the map is built, not lost)."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
assert rc == 0
call = fake_import.calls[0]
assert call["sources"] == [local_dir]
assert call["include_hidden_by_root"] == {str(local_dir): True}
assert call["ignore_by_root"] == {} # the row carries no ignore list
assert call["prune"] is False # the CLI's no-prune default is unchanged
def _stub_overview(monkeypatch: pytest.MonkeyPatch) -> None:
"""Stub the phase-31 overview regeneration (the CLI's real-import
tests: the deterministic ``FakeEmbedder`` must not burn its canned
``chat`` on the KB outline — the import is what is under test)."""
async def fake_overview(llm: object, session: object = None) -> bool:
return False
monkeypatch.setattr(import_docs, "regenerate_overview", fake_overview)
def _kb_docs(db: Session) -> set[tuple[str, str]]:
"""The (source, path) pairs of the ``documents`` table."""
rows = db.execute(text("SELECT source, path FROM documents")).all()
return {(row.source, row.path) for row in rows}
def test_main_local_row_include_hidden_true_indexes_hidden_file(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
db: Session,
capsys: pytest.CaptureFixture[str],
clean_documents: None,
) -> None:
"""Phase 105 (A1): the CLI's DB-row path with the flag ON — the
file inside the hidden folder is indexed, embedded, and counted
like any visible file (real import over a host temp dir, the
deterministic ``FakeEmbedder``; the regression this phase most
plausibly breaks — the CLI's map built but lost — would leave it
out)."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
)
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
_stub_overview(monkeypatch)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
assert rc == 0
# The hidden file has a documents row next to the visible one.
assert _kb_docs(db) == {
("LocalDocs", "visible.md"),
("LocalDocs", ".hidden/note.md"),
}
out = capsys.readouterr().out
assert "added=2" in out # both files were imported
def test_main_local_row_include_hidden_false_skips_hidden_file(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
db: Session,
capsys: pytest.CaptureFixture[str],
clean_documents: None,
) -> None:
"""Phase 105 (A4): the same fixture with the default flag (off) —
the hidden-folder file never enters the KB (the byte-identical
pre-phase-105 walk): no documents row, and the summary line counts
only the visible file."""
settings = _settings(sources_dir=str(tmp_path / "bor"))
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
monkeypatch.setattr(
import_docs,
"effective_sources",
lambda db: ([_local_row(str(local_dir))], "db"), # flag defaults to False
)
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
_stub_overview(monkeypatch)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
assert rc == 0
assert _kb_docs(db) == {("LocalDocs", "visible.md")}
out = capsys.readouterr().out
assert "added=1" in out # the hidden file was never walked
# --- main() ----------------------------------------------------------------
@@ -428,10 +605,12 @@ def test_resolve_sources_mixed_git_and_local(
sources_dir=str(tmp_path / "bor"),
)
sources, ignore_map = import_docs._resolve_sources(None, settings)
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only", local_dir]
assert ignore_map == {} # phase 89: neither row carries a list
# Phase 105: both rows default-flag → per-root False entries (A4).
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
@@ -0,0 +1,241 @@
"""Integration: the phase-105 hidden-folders flag through the real import
pipeline.
Phase 105 (TODO.md L3 — "…a toggle per input … to allow indexing hidden
.folders."): ``import_sources`` gains ``include_hidden_by_root`` (same
``str(root)`` keying as phase 89's ``ignore_by_root``). A1
(owner-confirmed 2026-09-14): flag ON admits dot-prefixed components —
hidden files are indexed, embedded, and summarized exactly like visible
files — while ``EXCLUDED_DIRS`` stay excluded in both states. A2: a file
indexed with the flag ON leaves the KB on the next ``prune=True`` run
with the flag OFF (the untouched ``seen`` set does the work). A4: no map
→ byte-identical to pre-phase-105. Mirrors the fixture-tree + mock-LLM
pattern of ``test_importer_ignore.py``: a ``tmp_path`` source dir run
through the real ``import_sources`` into the compose Postgres, with
:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.models import Chunk, Document
from app.rag.importer import import_sources
from tests.fakes import FakeEmbedder
NAME = "HiddenFix"
def _write(root: Path, rel: str, content: str) -> None:
path = root / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
def _tree(tmp_path: Path, name: str = NAME) -> Path:
"""One visible md, a hidden dir (md + non-markdown yaml), an excluded dir."""
root = tmp_path / name
_write(root, "visible.md", "# Visible\n\nvisible body\n")
_write(root, ".hidden/note.md", "# Note\n\nHIDDEN-MD-CONTENT\n")
_write(root, ".hidden/data.yaml", "key: HIDDEN-YAML-VALUE\n")
_write(root, ".venv/junk.md", "# Junk\n\nEXCLUDED-CONTENT\n")
return root
def _reset(db: Session) -> None:
# House cleanup pattern (tests/integration/test_importer_ignore.py).
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
def test_hidden_paths_not_indexed_by_default(db: Session, tmp_path: Path) -> None:
# A4: no map → today's behavior, byte-identical — the hidden files
# never produce a Document/Chunk row and are never embedded.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
summary = asyncio.run(import_sources([root], llm, session=db))
assert summary.files == 1
assert summary.added == 1
assert summary.errors == 0
assert summary.summaries == 0
assert summary.summary_errors == 0
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {(NAME, "visible.md")}
for rel in (".hidden/note.md", ".hidden/data.yaml", ".venv/junk.md"):
assert not any(d.path == rel for d in docs)
for texts in llm.calls: # every embed batch
for t in texts:
assert "HIDDEN-MD-CONTENT" not in t and "HIDDEN-YAML-VALUE" not in t
assert "EXCLUDED-CONTENT" not in t
# The hidden yaml never reached the lite model.
assert not llm.chat_calls
_reset(db)
def test_hidden_paths_indexed_when_flag_on(db: Session, tmp_path: Path) -> None:
# A1: with the map, hidden files are indexed, embedded, and
# summarized exactly like visible files — EXCLUDED_DIRS stay out.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
summary = asyncio.run(
import_sources(
[root], llm, session=db, include_hidden_by_root={str(root): True}
)
)
assert summary.files == 3
assert summary.added == 3
assert summary.errors == 0
assert summary.summary_errors == 0
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
(NAME, "visible.md"),
(NAME, ".hidden/note.md"),
(NAME, ".hidden/data.yaml"),
}
# EXCLUDED_DIRS is excluded in BOTH states (A1).
assert not any(d.path == ".venv/junk.md" for d in docs)
chunks = db.scalars(select(Chunk)).all()
assert not any("EXCLUDED-CONTENT" in c.content for c in chunks)
# The hidden md was embedded like any visible md (no summary — the
# markdown path skips the lite model).
note = db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
assert note is not None and note.summary is None
assert any("HIDDEN-MD-CONTENT" in c.content for c in chunks)
# The hidden yaml went through the FULL non-markdown path (phase 30):
# a stored summary plus one embedded is_summary chunk on top of the
# content chunks.
yaml_doc = db.scalar(select(Document).where(Document.path == ".hidden/data.yaml"))
assert yaml_doc is not None and yaml_doc.summary is not None
yaml_chunks = [
c for c in chunks if c.document_id == yaml_doc.id
]
assert any(c.is_summary for c in yaml_chunks)
assert any(not c.is_summary for c in yaml_chunks)
assert any("HIDDEN-YAML-VALUE" in c.content for c in yaml_chunks)
# Only the yaml reached the lite model.
assert summary.summaries == 1
assert len(llm.chat_calls) == 1
user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user")
assert "HIDDEN-YAML-VALUE" in user
_reset(db)
def test_flag_off_re_run_prunes_hidden_docs(db: Session, tmp_path: Path) -> None:
# A2: the owner flips the flag off — the next prune run walks with
# the default rules, the hidden files never enter ``seen``, and their
# rows (summary chunk included) leave the index.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
s1 = asyncio.run(
import_sources(
[root], llm, session=db, include_hidden_by_root={str(root): True}
)
)
assert (s1.files, s1.added) == (3, 3)
assert (
db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
is not None
)
s2 = asyncio.run(import_sources([root], llm, session=db, prune=True))
assert s2.files == 1
assert s2.unchanged == 1 # visible.md untouched
assert s2.pruned == 2 # both hidden documents
assert (
db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
is None
)
assert (
db.scalar(select(Document).where(Document.path == ".hidden/data.yaml"))
is None
)
assert (
db.scalar(select(Document).where(Document.path == "visible.md")) is not None
)
# The summary chunk rows went with their documents (cascade).
assert not any(
"HIDDEN-YAML-VALUE" in c.content for c in db.scalars(select(Chunk)).all()
)
_reset(db)
def test_progress_total_agrees_with_walk_in_both_states(
db: Session, tmp_path: Path
) -> None:
# The phase-64 pre-walk uses the same per-root flag as the loop, so
# ``total`` agrees with the walk in both states.
_reset(db)
root = _tree(tmp_path)
llm = FakeEmbedder()
on_totals: set[int] = set()
def on_progress(source: str, rel: str, done: int, total: int) -> None:
on_totals.add(total)
s_on = asyncio.run(
import_sources(
[root],
llm,
session=db,
progress=on_progress,
include_hidden_by_root={str(root): True},
)
)
assert s_on.files == 3
assert on_totals == {3} # hidden files counted in the denominator
_reset(db)
llm2 = FakeEmbedder()
off_totals: set[int] = set()
def off_progress(source: str, rel: str, done: int, total: int) -> None:
off_totals.add(total)
s_off = asyncio.run(
import_sources([root], llm2, session=db, progress=off_progress)
)
assert s_off.files == 1
assert off_totals == {1} # visible only, the pre-phase-105 count
_reset(db)
def test_unlisted_root_stays_hidden(db: Session, tmp_path: Path) -> None:
# The map is per-root, not global: listing one root as True leaves
# the other root exactly as pre-phase-105.
_reset(db)
root_a = _tree(tmp_path, name="HiddenFixA")
root_b = _tree(tmp_path, name="HiddenFixB")
llm = FakeEmbedder()
summary = asyncio.run(
import_sources(
[root_a, root_b],
llm,
session=db,
include_hidden_by_root={str(root_a): True},
)
)
# A: 3 (flag on) · B: 1 (unlisted → False)
assert summary.files == 4
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == {
("HiddenFixA", "visible.md"),
("HiddenFixA", ".hidden/note.md"),
("HiddenFixA", ".hidden/data.yaml"),
("HiddenFixB", "visible.md"),
}
assert not any(
d.source == "HiddenFixB" and d.path == ".hidden/note.md" for d in docs
)
_reset(db)
+271
View File
@@ -0,0 +1,271 @@
"""Integration: migration 0019 (git_sources.include_hidden) schema
contract (phase 105, task 01).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0007.py`` (git_sources column contract — information_schema
assertions on the state the migration must leave). The tests target the
0018 → 0019 step explicitly so later migrations cannot break them:
* upgrade 0018 → 0019 → the ``include_hidden`` column exists with the
full contract — BOOLEAN NOT NULL, server default ``false`` — while
the 0018 ``git_sources`` schema (url, kind, path, ignore_paths)
survives;
* pre-0019 rows backfill ``false`` and a row written without the
column takes the server default (A4 — byte-identical import
behavior until the owner flips the flag);
* the ORM contract agrees: a freshly inserted ``GitSource`` (no flag
passed) reads ``include_hidden is False`` (the Python ``default=False``
and the server default agree), and an explicit ``True`` round-trips
through a fresh session;
* downgrade to 0018 → the column is GONE (A13 — reversible) while the
rows + their ignore lists survive;
* upgrade back to 0019 → the column is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
import uuid
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.orm import Session
from alembic import command
from app.db import SessionLocal, db_available
from app.models import GitSource
URL_BASE = "https://git.example.com/mig0019"
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown upgrades
to head no matter what happened, so the dev DB is never left below
head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
# on ``git_sources``, which would deadlock the repair's
# ``ALTER TABLE`` (0019) forever.
db.rollback()
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one git_sources column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'git_sources' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _flag(db: Session, url: str) -> Any:
return db.execute(
text("SELECT include_hidden FROM git_sources WHERE url = :u"),
{"u": url},
).scalar_one()
def _insert_sql(db: Session, url: str, include_hidden: Any = None) -> None:
"""Insert one git_sources row (kind/path omitted → the API default
shape); ``include_hidden`` omitted → pre-0019 insert shape."""
if include_hidden is None:
db.execute(
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
{"u": url},
)
else:
db.execute(
text(
"INSERT INTO git_sources (id, url, include_hidden)"
" VALUES (gen_random_uuid(), :u, :f)"
),
{"u": url, "f": include_hidden},
)
db.commit()
def _delete_by_url(db: Session, url: str) -> None:
db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url})
db.commit()
def test_upgrade_to_0019_adds_include_hidden(db: Session, alembic: Config) -> None:
"""Upgrade 0018 → 0019: the column exists with the full contract
(BOOLEAN NOT NULL, server default ``false``), is ABSENT at 0018,
pre-0019 rows backfill ``false`` (A4), a new row without the column
takes the default, and an explicit ``true`` round-trips — while the
0018 table contract survives."""
command.downgrade(alembic, "0018") # start from the pre-0019 state
assert _version(db) == "0018"
assert _column(db, "include_hidden") is None, (
"the flag must be absent at 0018"
)
url_pre = f"{URL_BASE}/pre-existing.git"
_insert_sql(db, url_pre) # no include_hidden column exists at 0018
try:
command.upgrade(alembic, "0019")
assert _version(db) == "0019", "alembic_version must be at 0019"
flag = _column(db, "include_hidden")
assert flag is not None, "git_sources.include_hidden is missing"
assert flag[0] == "boolean", "include_hidden must be BOOLEAN"
assert flag[1] == "NO", "include_hidden must be NOT NULL"
assert flag[2] is not None and "false" in str(flag[2]), (
"include_hidden must carry the `false` server default"
)
# The pre-0019 row backfilled ``false`` (A4 — unchanged imports).
assert _flag(db, url_pre) is False
# A row written without the column takes the server default.
url_new = f"{URL_BASE}/new-row.git"
_insert_sql(db, url_new)
try:
assert _flag(db, url_new) is False, (
"an omitted flag takes the `false` server default"
)
# The flag round-trips through an explicit ``true``.
db.execute(
text("UPDATE git_sources SET include_hidden = true WHERE url = :u"),
{"u": url_new},
)
db.commit()
assert _flag(db, url_new) is True, (
"include_hidden = true must round-trip"
)
finally:
_delete_by_url(db, url_new)
# The 0018 schema survives the additive upgrade.
ignore = _column(db, "ignore_paths")
assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", (
"git_sources.ignore_paths (0013) must survive the upgrade"
)
kind = _column(db, "kind")
assert kind is not None and kind[0] == "text" and kind[1] == "NO", (
"git_sources.kind (0007) must survive the upgrade"
)
finally:
_delete_by_url(db, url_pre)
def test_orm_fresh_row_defaults_false_and_true_round_trips(
db: Session, alembic: Config
) -> None:
"""The ORM contract agrees with the column contract: a freshly
inserted ``GitSource`` (no flag passed) reads ``include_hidden is
False`` — the Python ``default=False`` and the server default
agree (A4) — and an explicit ``True`` round-trips through a fresh
session."""
command.upgrade(alembic, "head")
url_off = f"{URL_BASE}/orm-default.git"
url_on = f"{URL_BASE}/orm-true.git"
try:
# Fresh row, flag omitted → False (the Python-side default).
row_off = GitSource(url=url_off, kind="git")
db.add(row_off)
db.commit()
db.expire_all()
reloaded_off = db.get(GitSource, row_off.id)
assert reloaded_off is not None, "the fresh row must be readable"
assert reloaded_off.include_hidden is False, (
"a fresh row must read include_hidden is False (A4)"
)
# Explicit True round-trips through a FRESH session.
row_on = GitSource(url=url_on, kind="git", include_hidden=True)
db.add(row_on)
db.commit()
with SessionLocal() as fresh:
reloaded = fresh.get(GitSource, row_on.id)
assert reloaded is not None, "the row must exist in a fresh session"
assert reloaded.include_hidden is True, (
"include_hidden=True must round-trip through the DB"
)
finally:
_delete_by_url(db, url_off)
_delete_by_url(db, url_on)
def test_downgrade_to_0018_drops_the_column(db: Session, alembic: Config) -> None:
"""Downgrade 0019 → 0018: the column is gone (A13 — fully
reversible) while the rows + their ignore lists survive, and the
rest of the 0018 table contract (``path``) is intact."""
command.upgrade(alembic, "head")
url = f"{URL_BASE}/survivor.git"
db.execute(
text(
"INSERT INTO git_sources (id, url, include_hidden)"
" VALUES (:i, :u, true)"
),
{"i": uuid.uuid4(), "u": url},
)
db.commit()
try:
command.downgrade(alembic, "0018")
assert _version(db) == "0018"
assert _column(db, "include_hidden") is None, (
"the flag must be dropped"
)
row = db.execute(
text("SELECT url, kind, ignore_paths FROM git_sources WHERE url = :u"),
{"u": url},
).fetchone()
assert row is not None and row[0] == url, (
"the row must survive the column drop"
)
assert row[1] == "git" and row[2] == [], (
"kind + the ignore list must survive the column drop"
)
doc_path = _column(db, "path")
assert doc_path is not None and doc_path[0] == "text", (
"git_sources.path must survive the downgrade"
)
finally:
_delete_by_url(db, url)
# Repair: the fixture teardown re-upgrades to head.
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
"""Downgrade to 0018, then upgrade back to 0019: the column is back
with the full contract (BOOLEAN NOT NULL, the `false` default)."""
command.downgrade(alembic, "0018")
command.upgrade(alembic, "0019")
assert _version(db) == "0019", "round-trip upgrade must land at 0019"
flag = _column(db, "include_hidden")
assert flag is not None, "git_sources.include_hidden must be back"
assert flag[0] == "boolean", "include_hidden must be BOOLEAN after the round-trip"
assert flag[1] == "NO", "include_hidden must be NOT NULL after the round-trip"
assert flag[2] is not None and "false" in str(flag[2]), (
"the `false` server default must survive the round-trip"
)
+120 -1
View File
@@ -77,7 +77,7 @@ from typing import Any
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import sync as sync_api
@@ -262,6 +262,11 @@ class FakeImportSources:
# rows' ``ignore_paths`` (keyed by the root string the importer
# sees; two rows sharing a root string get the union).
self.ignore_maps: list[dict[str, list[str]]] = []
# Phase 105: the per-root hidden-folders flag map the runner
# builds from the rows' ``include_hidden`` (same root-string
# keying; a shared-root collision ORs — if either row says
# "index hidden", the root does).
self.include_hidden_maps: list[dict[str, bool]] = []
async def __call__(
self,
@@ -273,12 +278,14 @@ class FakeImportSources:
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
) -> ImportSummary:
self.sources.append(list(sources))
self.llms.append(llm)
self.prune_flags.append(prune)
self.progress_hooks.append(progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
if self.delay:
await asyncio.sleep(self.delay)
return self.summary
@@ -823,6 +830,7 @@ def test_import_error_is_reported_with_credentials_masked(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
) -> ImportSummary:
raise EmbeddingError(
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
@@ -1100,3 +1108,114 @@ def test_sync_without_ignore_lists_passes_empty_map(
assert fake_import.sources == [[local_dir]]
assert fake_import.ignore_maps == [{}] # no row carried a list
# --- phase 105: per-row hidden-folders flag --------------------------------
def test_local_row_hidden_flag_off_then_on_indexes_hidden_paths(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 105 (A4 then A1): the SAME local row, synced first with the
default flag (off) — the file inside the hidden folder never lands
in the KB (no document row — hence no embedding, no summary), the
visible file imports as usual; then the row is flipped on (direct
model set — the PATCH round-trip is task 03's layer) and the next
sync walks the hidden file too: it is indexed, embedded, and counted
like any visible file."""
local_dir = tmp_path / "LocalDocs"
local_dir.mkdir()
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
(local_dir / ".hidden").mkdir()
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
_seed_local(db, local_dir) # include_hidden defaults to False (A4)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# Flag off (A4): only the visible file is walked and indexed — the
# hidden file has NO documents row.
assert body["detail"]["files"] == 1
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
docs = {
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
}
assert docs == {("LocalDocs", "visible.md")}
# Flip the SAME row on — the next sync re-reads the flag per row.
row = db.execute(select(GitSource).where(GitSource.url == str(local_dir))).scalar_one()
row.include_hidden = True
db.commit()
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# Flag on (A1): the hidden file is walked (detail.files counts it),
# embedded, and indexed alongside the visible file.
assert body["detail"]["files"] == 2
assert body["detail"]["added"] == 1
assert body["detail"]["errors"] == 0
docs = {
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
}
assert docs == {("LocalDocs", "visible.md"), ("LocalDocs", ".hidden/note.md")}
def test_sync_builds_include_hidden_map_by_root_string_with_or(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 105 wiring (fake import): the runner keys the flag map by
the SAME root string the importer sees, and two rows sharing that
root string (the sibling/repo-name edge — ``…/shared`` and
``…/shared.git`` clone into the same checkout dir) get the OR of
their flags — if EITHER row says "index hidden", the root does
(the ignore-map union's boolean mirror)."""
url_a = f"file://{tmp_path / 'shared'}"
url_b = f"{url_a}.git" # same repo name → same checkout dir
# Distinct added_at: the resolver orders by (added_at, id) — a
# same-timestamp pair would tie-break on the random uuid.
db.add(GitSource(url=url_a, kind="git", include_hidden=False,
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
db.add(GitSource(url=url_b, kind="git", include_hidden=True,
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
db.commit()
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
shared = str(tmp_path / "bor" / "shared")
# Both rows resolve to the SAME checkout (the collision itself) and
# the map holds the OR of their flags, keyed by that one root
# string — the flag-off row's False is overridden by the True.
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
assert fake_import.include_hidden_maps == [{shared: True}]
+436
View File
@@ -0,0 +1,436 @@
"""Unit: the per-row "Hidden" toggle on the Sources page (phase 105,
task 05).
``TODO.md`` L3: "There should be a toggle per input (next to the
ignores button) to allow indexing hidden .folders." Every STORED
``git_sources`` row (git or local) gets a native labeled **Hidden**
checkbox in the actions cell, LEFT of the "Ignore paths" button (DOM
order Hidden · Ignore paths · Remove — A5); when on, the source cell
shows a "hidden on" text tag (text + background, never color alone —
WCAG 1.4.1); the flip PATCHes ``{"include_hidden": …}`` with the
§7.4 never-stale lifecycle (box disables at once; 200 → reload +
confirm LAST; failure → the box reverts to the SERVER state and the
detail lands in the page-level ``#git-sources-hidden-error``
``role="alert"`` line — the checkbox is a table-cell control with no
dialog of its own). Env-fallback rows (``id`` null) get no checkbox
(A3 — nothing is stored to flag).
The browser behavior itself is E2E-gated by the phase-105 story suite
(``tests/e2e/test_hidden_folders_toggle.py``, task 06); like the
phase-89 ``test_source_ignore_paths.py`` house pattern, this module
pins the source-level contract a silent regression would break:
* the JS row wiring — ``makeRow`` builds the checkbox ONLY in the
``s.id`` branch (class ``git-source-hidden-box``, ``type=checkbox``,
``checked`` from ``s.include_hidden === true`` — server state only,
never a prior local flip), the aria-label is the ONLY place the
value appears (setAttribute — never innerHTML), the visible
"Hidden" text label, and the append ORDER (hidden label →
ignore button → Remove); the "hidden on" tag is TEXT in the source
cell, iff ``s.id && s.include_hidden === true``;
* the JS lifecycle — ``toggleHidden``: the box disables BEFORE the
``PATCH`` (one flip at a time), the body is
``JSON.stringify({ include_hidden: wanted })`` ONLY (the row's
ignore list is untouched — task 03's optional field), 200 →
clear the error → ``await loadSources()`` → announce (the
confirmation is the LAST announcement); non-2xx AND network
failure → the box REVERTS to ``s.include_hidden === true`` +
re-enables and the detail lands in the page-level alert line;
a healed ``loadSources`` clears the line (the phase-89 "happy
path heals the error state" precedent);
* the cross-file wire contract (SINGLE SOURCE OF TRUTH for the
field name) — the JS PATCH body key and the render field both
match their ``app/schemas.py`` counterparts
(``GitSourcePatchIn.include_hidden`` / ``GitSourceRow``) — a
rename on either side breaks the test;
* the static shell markup — ``#git-sources-hidden-error`` is a
``role="alert"`` line, hidden by default, reusing the
``.git-source-error`` class, INSIDE ``#view-git-sources`` and
AFTER ``#git-sources-table-wrap`` in source order;
* styles.css — the ``.git-source-hidden`` family on the house
palette: ~44px hit height, the checkbox's ``accent-color:
var(--brand)`` (checked-state contrast verified + recorded in the
comment), the ``:disabled`` wait state, and the "hidden on" tag
(text + a distinct background, never color alone).
"""
from __future__ import annotations
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
FRONTEND = ROOT / "frontend"
# Phase 76 (task 02): the Sources view lives in the ONE-document shell.
SHELL_HTML = FRONTEND / "index.html"
JS = FRONTEND / "assets" / "git-sources.js"
CSS = FRONTEND / "assets" / "styles.css"
SCHEMAS = ROOT / "app" / "schemas.py"
#: The checkbox + tag classes (pinned verbatim).
HIDDEN_BOX_CLASS = "git-source-hidden-box"
HIDDEN_LABEL_CLASS = "git-source-hidden"
HIDDEN_TAG_CLASS = "git-source-hidden-count"
HIDDEN_TAG_TEXT = "hidden on"
#: The checkbox's aria-label template — the ONLY place the full
#: source value appears (setAttribute, never innerHTML).
ARIA_LABEL_TEMPLATE = "`Index hidden folders for ${kindLabel} source: ${value}`"
#: The §7.4 lifecycle strings (pinned verbatim).
REVERT_STATE = "box.checked = s.include_hidden === true"
NETWORK_MESSAGE = "Could not reach the server — the setting was not changed."
PATCH_BODY = "JSON.stringify({ include_hidden: wanted })"
def _text(path: Path) -> str:
return path.read_text(encoding="utf-8")
def _js() -> str:
return _text(JS)
def _css() -> str:
return _text(CSS)
def _fn(js: str, name: str) -> str:
"""The source of a (possibly async) top-level function via
balanced-brace counting (the test_source_ignore_paths.py helper)."""
for prefix in ("async function ", "function "):
start = js.find(f"{prefix}{name}(")
if start != -1:
depth = 0
for i in range(js.find("{", start), len(js)):
if js[i] == "{":
depth += 1
elif js[i] == "}":
depth -= 1
if depth == 0:
return js[start : i + 1]
raise AssertionError(f"unbalanced braces in {name}()")
raise AssertionError(f"{name}() must exist in git-sources.js")
def _css_rule(css: str, selector: str) -> str:
"""The declarations of a simple one-line-opening rule (comments
stripped first — a house comment may legally carry braces)."""
clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
start = clean.find(f"{selector} {{")
assert start != -1, f"missing rule {selector} in styles.css"
depth = 0
for i in range(clean.find("{", start), len(clean)):
if clean[i] == "{":
depth += 1
elif clean[i] == "}":
depth -= 1
if depth == 0:
return clean[start : i + 1]
raise AssertionError(f"unbalanced braces in {selector}")
def _python_class_field(class_name: str) -> str:
"""The single bool field name declared on a Pydantic class in
``app/schemas.py`` (``bool`` or ``bool | None`` — regex-parsed,
no import: the pin is on the source text, so a rename breaks it
without a schema load)."""
schemas = _text(SCHEMAS)
m = re.search(
rf"class {class_name}\(BaseModel\):.*?(?=\nclass |\Z)", schemas, re.S
)
assert m, f"class {class_name} must exist in app/schemas.py"
fields = re.findall(
r"^\s*(\w+):\s*bool(?:\s*\|\s*None)?\s*(?==|$)", m.group(0), re.M
)
assert len(fields) == 1, f"{class_name} must declare exactly one bool field"
return fields[0]
# ---------- the row wiring in makeRow ----------
def test_row_checkbox_is_built_in_the_stored_row_branch() -> None:
"""makeRow: the Hidden checkbox is created ONLY in the ``s.id``
branch (A3 — env-fallback rows get the "from .env" tag, no
checkbox). It is a NATIVE labeled checkbox: class
git-source-hidden-box, type=checkbox, the visible "Hidden" text,
``checked`` from the SERVER row (``s.include_hidden === true`` —
never a prior local flip, §7.4), and the aria-label template is
the ONLY place ``value`` appears (setAttribute — never
innerHTML). Its change handler runs toggleHidden."""
make = _fn(_js(), "makeRow")
branch_i = make.find("if (s.id) {")
else_i = make.find("} else {", branch_i)
assert branch_i != -1 and else_i > branch_i
box_i = make.find('const hiddenBox = document.createElement("input")', branch_i)
assert -1 < box_i < else_i, "the checkbox is created in the s.id branch (A3)"
assert 'hiddenBox.type = "checkbox"' in make, "a native checkbox"
assert f"hiddenBox.className = \"{HIDDEN_BOX_CLASS}\"" in make
assert "hiddenBox.checked = s.include_hidden === true" in make, (
"checked state comes ONLY from the server row"
)
assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)"
assert 'hiddenBox.setAttribute(' in make, "the label is set via setAttribute"
assert "hiddenBox.innerHTML" not in _js(), "XSS contract: no innerHTML on the box"
assert "hiddenLabel.innerHTML" not in _js(), "XSS contract: no innerHTML on the label"
assert 'document.createTextNode("Hidden")' in make, (
"the visible text label — never aria-label-only (WCAG)"
)
assert "hiddenLabel.append(hiddenBox, document.createTextNode(\"Hidden\"))" in make
assert 'hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox))' in make
# The env-fallback branch (else) has no checkbox.
else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)]
assert HIDDEN_BOX_CLASS not in else_slice and "toggleHidden" not in else_slice, (
"env-fallback rows get no checkbox (A3)"
)
def test_row_dom_order_is_hidden_ignore_remove() -> None:
"""makeRow: the append order in the actions cell is Hidden ·
Ignore paths · Remove (A5 — the toggle sits LEFT of the "Ignore
paths" button; Remove stays last)."""
make = _fn(_js(), "makeRow")
branch_i = make.find("if (s.id) {")
else_i = make.find("} else {", branch_i)
hidden_append = make.find("actTd.appendChild(hiddenLabel)", branch_i)
ignore_i = make.find('const ignoreBtn = document.createElement("button")', branch_i)
ignore_append = make.find("actTd.appendChild(ignoreBtn)", branch_i)
remove_append = make.find("actTd.appendChild(btn)", branch_i)
assert -1 < hidden_append < ignore_i, "the hidden label precedes the ignore button"
assert -1 < ignore_append < remove_append, "Remove stays last"
assert remove_append < else_i, "all three controls are in the s.id branch"
def test_hidden_on_tag_renders_text_in_the_source_cell() -> None:
"""makeRow: a stored row with the flag ON gets the "hidden on"
tag in the SOURCE cell (urlTd — the .git-source-ignore-count
idiom: TEXT, never color alone, WCAG 1.4.1), appended after the
location <code>; the tag is the state copy (pinned verbatim)."""
make = _fn(_js(), "makeRow")
assert "s.id && s.include_hidden === true" in make, (
"the tag only for stored rows with the flag on"
)
assert f"hiddenTag.className = \"{HIDDEN_TAG_CLASS}\"" in make
assert f"hiddenTag.textContent = \"{HIDDEN_TAG_TEXT}\"" in make, (
"the tag copy is TEXT — never color alone"
)
code_i = make.find("urlTd.append(badge, code)")
append_i = make.find("urlTd.append(hiddenTag)", code_i)
assert -1 < code_i < append_i, "the tag is appended to the source cell"
# ---------- the JS lifecycle (toggleHidden) ----------
def test_toggle_sends_patch_and_disables_the_box_first() -> None:
"""toggleHidden: the box disables BEFORE the PATCH goes out
(no double-flip — §7.4); the request is a PATCH to
/api/git-sources/{id} whose body is ``{"include_hidden": …}``
ONLY (the row's ignore list is untouched — task 03's optional
field)."""
body = _fn(_js(), "toggleHidden")
disable_i = body.find("box.disabled = true")
fetch_i = body.find("`/api/git-sources/${s.id}`")
method_i = body.find('method: "PATCH"', fetch_i)
body_i = body.find(PATCH_BODY, method_i)
assert -1 < disable_i < fetch_i < method_i < body_i, (
"disable → PATCH {include_hidden} (and nothing else)"
)
assert "ignore_paths" not in body, "the PATCH body never carries the ignore list"
def test_toggle_success_clears_reloads_and_announces_last() -> None:
"""toggleHidden 200: the error line clears, ``loadSources()``
AWAITS (the row re-renders from the server — the "hidden on" tag
lands), and the confirmation is the LAST announcement (the
reload's "N sources listed." lands first — the phase-89 order)."""
body = _fn(_js(), "toggleHidden")
ok_i = body.find("if (r.ok)")
hide_i = body.find("hideHiddenError()", ok_i)
reload_i = body.find("await loadSources()", ok_i)
announce_i = body.find("announce(`Hidden folders", reload_i)
assert -1 < ok_i < hide_i < reload_i < announce_i, (
"200: clear error → await loadSources → announce (LAST)"
)
assert "for ${value}.`)" in body[announce_i:], "the confirmation names the source"
assert "enabled" in body[announce_i:] and "disabled" in body[announce_i:], (
"the confirmation states the direction of the flip"
)
def test_toggle_failure_reverts_the_box_and_shows_the_page_alert() -> None:
"""toggleHidden failure: non-2xx → the server detail (apiDetail)
into the page-level alert line + the box REVERTS to the server
state + re-enables; network failure → the fixed reachable? line,
same revert. The UI never claims a state the server didn't save
(PLAN §7.4). The revert + re-enable happen on BOTH failure
branches (exactly twice each in the function)."""
body = _fn(_js(), "toggleHidden")
catch_i = body.find(".catch(")
assert catch_i != -1
fail_slice = body[body.find("if (r.ok)"):catch_i]
assert "await apiDetail(" in fail_slice, "the server detail is apiDetail-extracted"
assert "showHiddenError(" in fail_slice, "the detail lands in the page alert line"
assert REVERT_STATE in fail_slice, "the box reverts to the server state"
assert "box.disabled = false" in fail_slice, "the box re-enables"
net_slice = body[catch_i:]
assert NETWORK_MESSAGE in net_slice, "the fixed network copy"
assert "showHiddenError(" in net_slice
assert REVERT_STATE in net_slice
assert "box.disabled = false" in net_slice
assert body.count(REVERT_STATE) == 2, "revert on BOTH failure branches"
assert body.count("box.disabled = false") == 2, "re-enable on BOTH failure branches"
assert "box.disabled = true" in body[: body.find("if (r.ok)")], (
"the box disables at once, before any outcome"
)
def test_error_line_helpers_and_the_healed_load_clear() -> None:
"""showHiddenError/hideHiddenError drive the page-level
``hiddenErrorEl`` (textContent + hidden); ``loadSources``'s
success path calls hideHiddenError() AFTER hideLoadError() — a
healed list clears the stale line (the phase-89 "happy path
heals the error state" precedent)."""
js = _js()
assert 'const hiddenErrorEl = root.querySelector("#git-sources-hidden-error")' in js, (
"the page-local element grabber (root-scoped, the shell idiom)"
)
show = _fn(js, "showHiddenError")
assert "hiddenErrorEl.textContent = message" in show
assert "hiddenErrorEl.hidden = false" in show
hide = _fn(js, "hideHiddenError")
assert 'hiddenErrorEl.textContent = ""' in hide
assert "hiddenErrorEl.hidden = true" in hide
load = _fn(js, "loadSources")
hide_load_i = load.find("hideLoadError()")
hide_hidden_i = load.find("hideHiddenError()", hide_load_i)
assert -1 < hide_load_i < hide_hidden_i, (
"a healed load clears the hidden-toggle line (after the load-error clear)"
)
# ---------- the cross-file wire contract (single source of truth) ----------
def test_js_field_name_matches_the_python_patch_schema() -> None:
"""The cross-file pin: the JS PATCH body key and the render field
both equal their ``app/schemas.py`` counterparts —
``GitSourcePatchIn.include_hidden`` (the write side) and
``GitSourceRow.include_hidden`` (the read side). A rename on
either side breaks the wire contract and this test."""
js = _js()
body = _fn(js, "toggleHidden")
m = re.search(r"JSON\.stringify\(\{\s*(\w+)\s*:\s*wanted\s*\}\)", body)
assert m, "the PATCH body carries a single {<field>: wanted} object"
js_body_key = m.group(1)
assert js_body_key == _python_class_field("GitSourcePatchIn"), (
"the JS PATCH body key must match GitSourcePatchIn's field"
)
make = _fn(js, "makeRow")
rm = re.search(r"s\.(\w+)\s*===\s*true", make)
assert rm, "makeRow reads the flag off the server row (s.<field> === true)"
js_read_key = rm.group(1)
assert js_read_key == _python_class_field("GitSourceRow"), (
"the render field must match GitSourceRow's field"
)
assert js_body_key == js_read_key == "include_hidden", (
"one field name for the whole wire contract"
)
# ---------- the static shell markup ----------
def test_page_error_line_is_a_role_alert_after_the_table() -> None:
"""index.html: ``#git-sources-hidden-error`` is a ``<p>`` reusing
the existing ``.git-source-error`` class, ``role="alert"`` +
hidden by default, INSIDE ``#view-git-sources`` and AFTER
``#git-sources-table-wrap`` in source order (the checkbox is a
table-cell control, so its failure announces at page level — the
ignore dialog carries its own error INSIDE the modal). It
appears exactly once."""
html = _text(SHELL_HTML)
tag = re.search(r'<p[^>]*id="git-sources-hidden-error"[^>]*>', html)
assert tag, "#git-sources-hidden-error must be a real <p>"
open_tag = tag.group(0)
for attr in ('class="git-source-error"', 'role="alert"', "hidden"):
assert attr in open_tag, f"the error line carries {attr}"
view_i = html.find('id="view-git-sources"')
wrap_i = html.find('id="git-sources-table-wrap"')
err_i = html.find('id="git-sources-hidden-error"')
next_i = html.find('id="view-history"')
assert -1 < view_i < wrap_i < err_i < next_i, (
"the line sits inside #view-git-sources, after #git-sources-table-wrap"
)
assert html.count('id="git-sources-hidden-error"') == 1, (
"the id is unique in the shell"
)
# ---------- styles.css ----------
def test_hidden_checkbox_css_rules_exist_with_accent_color() -> None:
"""styles.css carries the phase-105 family: .git-source-hidden
(inline-flex, 44px hit height matching the action buttons, the
house ink), the checkbox (sized, ``accent-color: var(--brand)``),
the :disabled wait state (the .git-source-remove:disabled
idiom), and the "hidden on" tag (text + a distinct background —
never color alone). The checkbox comment RECORDS the
verified checked-state contrast (house style)."""
css = _css()
label = _css_rule(css, ".git-source-hidden")
assert "display: inline-flex" in label
assert "height: 44px" in label, "the ~44px hit height matches the action buttons"
assert "var(--ink)" in label
box = _css_rule(css, '.git-source-hidden input[type="checkbox"]')
assert "accent-color: var(--brand)" in box, "the --brand checkbox fill"
disabled = _css_rule(css, ".git-source-hidden:disabled")
assert "opacity" in disabled and "cursor: wait" in disabled, (
"the .git-source-remove:disabled idiom"
)
tag = _css_rule(css, ".git-source-hidden-count")
assert "var(--ink)" in tag and "var(--bg)" in tag, (
"text + a distinct background (never color alone)"
)
assert "url(http" not in css and "@import url(" not in css, (
"no CDN (AGENTS.md rule 6)"
)
def test_checkbox_contrast_is_verified_and_recorded() -> None:
"""House style: the checked-state contrast of the native widget
is VERIFIED and the ratio RECORDED in the comment above the
checkbox rule (the executor note the task template carries):
the --bg-on---brand house pairing (5.2:1) on the built-in theme,
and the theme tab's AA gate (4.5:1) for every saved palette."""
css = _css()
rule_i = css.find('.git-source-hidden input[type="checkbox"] {')
assert rule_i != -1
header = css[css.rfind("/*", 0, rule_i):rule_i]
assert "VERIFIED" in header, "the contrast is verified (the executor note)"
assert "5.2:1" in header, "the built-in --bg-on---brand ratio is recorded"
assert "4.5:1" in header, "the AA gate for themed palettes is recorded"
# ---------- the module docstring ----------
def test_module_docstring_carries_the_phase_105_contract() -> None:
"""The git-sources.js module docstring gained the phase-105
entry: the per-row Hidden checkbox (makeRow) →
PATCH {include_hidden} (task 03's optional field) →
loadSources + announce; failure reverts the box +
#git-sources-hidden-error (role=alert); env-fallback rows get
no checkbox (A3)."""
doc = _js().split("*/", 2)[0] # the module docstring (first block)
for frag in (
"Phase 105 (task 05)",
"toggleHidden",
"{ include_hidden }",
"#git-sources-hidden-error",
"env-fallback rows get NO checkbox",
"REVERTS to the server state",
"LAST announcement",
):
assert frag in doc, f"the module docstring lost: {frag!r}"
+5 -3
View File
@@ -849,12 +849,14 @@ def test_no_progress_means_no_prewalk(
extensions: frozenset[str],
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
include_hidden: bool = False,
) -> list[Path]:
# Phase 89: the walker gained the ``ignore`` keyword — the sentinel
# accepts (and forwards) it to stay a drop-in.
# Phase 89: the walker gained the ``ignore`` keyword; phase 105:
# the ``include_hidden`` flag — the sentinel accepts (and forwards)
# both to stay a drop-in.
nonlocal walk_calls
walk_calls += 1
return real_walker(r, extensions, excluded, ignore)
return real_walker(r, extensions, excluded, ignore, include_hidden)
monkeypatch.setattr(importer, "iter_importable_files", counting)
try:
+161
View File
@@ -0,0 +1,161 @@
"""Unit: the phase-105 hidden-folders flag in ``iter_importable_files``.
Phase 105 (TODO.md L3 — "…a toggle per input (next to the ignores
button) to allow indexing hidden .folders."): ``include_hidden`` lifts
ONLY the dot-prefixed-component skip for one source. A1
(owner-confirmed 2026-09-14): when True, files inside hidden folders
AND hidden files with an importable extension become importable;
``EXCLUDED_DIRS`` (``.venv``, ``node_modules``, ``.git``, …) stay
excluded in BOTH states; the extension filter always applies; the
phase-89 *ignore* tuple composes additively. A4: the default
(``False``) is byte-identical to the pre-phase-105 walk — pinned here
against a tmp fixture tree. The DB-facing behavior (hidden files
indexed / pruned through ``import_sources``) is integration-gated by
``tests/integration/test_importer_include_hidden.py``.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from app.rag.importer import _include_hidden_for_root, iter_importable_files
#: Explicit extension set (not the A9 config default) — the tests pin
#: the walk rules, not the config.
EXTS = frozenset({".md"})
@pytest.fixture()
def tree(tmp_path: Path) -> Path:
"""The task's fixture layout (plus the no-extension ``.env``).
visible.md .hidden/note.md .notes.md
.venv/junk.md node_modules/x.md .hidden/.deep.md
keep/ok.md .env
"""
root = tmp_path / "HiddenFix"
(root / ".hidden").mkdir(parents=True)
(root / ".venv").mkdir()
(root / "node_modules").mkdir()
(root / "keep").mkdir()
(root / "visible.md").write_text("visible\n", encoding="utf-8")
(root / ".hidden" / "note.md").write_text("note\n", encoding="utf-8")
(root / ".notes.md").write_text("notes\n", encoding="utf-8")
(root / ".venv" / "junk.md").write_text("junk\n", encoding="utf-8")
(root / "node_modules" / "x.md").write_text("x\n", encoding="utf-8")
(root / ".hidden" / ".deep.md").write_text("deep\n", encoding="utf-8")
(root / "keep" / "ok.md").write_text("ok\n", encoding="utf-8")
(root / ".env").write_text("SECRET=1\n", encoding="utf-8")
return root
def _rels(paths: list[Path], root: Path) -> set[str]:
return {p.relative_to(root).as_posix() for p in paths}
# --- default (flag False): today's behavior, byte-identical -----------------
def test_iter_default_flag_skips_all_hidden(tree: Path) -> None:
# A4: the default is the pre-phase-105 walk — hidden dir, hidden
# file, and the excluded dirs are all absent. Sorted order pinned
# verbatim (not just the set).
got = iter_importable_files(tree, EXTS)
assert got == [tree / "keep" / "ok.md", tree / "visible.md"]
def test_iter_default_flag_rejects_explicit_false(tree: Path) -> None:
assert iter_importable_files(tree, EXTS, include_hidden=False) == [
tree / "keep" / "ok.md",
tree / "visible.md",
]
# --- flag True (A1) -----------------------------------------------------------
def test_iter_flag_on_admits_hidden_dirs_and_hidden_files(tree: Path) -> None:
got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree)
assert got == {
".hidden/note.md", # inside a hidden dir
".notes.md", # hidden file with an importable extension
".hidden/.deep.md", # both at once
"visible.md",
"keep/ok.md",
}
def test_iter_flag_on_keeps_excluded_dirs_out(tree: Path) -> None:
# A1: EXCLUDED_DIRS are skipped in BOTH states — caches are never
# content, even with the flag on.
got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree)
assert ".venv/junk.md" not in got
assert "node_modules/x.md" not in got
def test_iter_flag_on_keeps_extension_filter_in_force(tree: Path) -> None:
# ``.env`` has no A9 extension — the extension filter is the real
# content gate, so a secret-flavoured file is never listed.
for flag in (False, True):
got = _rels(iter_importable_files(tree, EXTS, include_hidden=flag), tree)
assert ".env" not in got
# --- composition with the phase-89 ignore tuple -------------------------------
def test_iter_flag_on_ignore_composes_additively(tree: Path) -> None:
# The ignored prefix still bites when the flag is ON; everything
# else hidden is admitted.
got = _rels(
iter_importable_files(tree, EXTS, include_hidden=True, ignore=(".hidden",)),
tree,
)
assert got == {".notes.md", "visible.md", "keep/ok.md"}
def test_iter_ignore_only_flag_off(tree: Path) -> None:
# The pre-phase-105 combination still works untouched: ignore
# prefix on a visible subtree, flag at its default.
got = _rels(iter_importable_files(tree, EXTS, ignore=("keep",)), tree)
assert got == {"visible.md"}
# --- _include_hidden_for_root: the single read point --------------------------
def test_include_hidden_for_root_none_map_is_false(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, None) is False
def test_include_hidden_for_root_unlisted_root_is_false(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, {"/other/root": True}) is False
def test_include_hidden_for_root_listed_true_is_true(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, {str(root): True}) is True
def test_include_hidden_for_root_listed_false_is_false(tmp_path: Path) -> None:
root = tmp_path / "src"
root.mkdir()
assert _include_hidden_for_root(root, {str(root): False}) is False
def test_include_hidden_for_root_keys_by_str_root(tmp_path: Path) -> None:
# Two distinct Path objects whose ``str()`` agrees — the answer must
# be the same (the ``_ignore_for_root`` keying convention, phase 89).
key = str(tmp_path / "src")
a = tmp_path / "src"
b = tmp_path / "src" / "." # different object, same str()
assert str(a) == str(b) == key
mp = {key: True}
assert _include_hidden_for_root(a, mp) is True
assert _include_hidden_for_root(b, mp) is True
+36
View File
@@ -134,6 +134,42 @@ def test_git_source_python_default_empty_list() -> None:
assert col.default.arg(None) == [], "the default must resolve to []"
def test_git_sources_include_hidden_column_contract() -> None:
"""Phase 105: every source row carries its hidden-folders flag —
BOOLEAN, NOT NULL, server default ``false`` (a pre-phase-105 row
reads ``False``, so every existing source imports byte-identically
until the owner flips it — A4)."""
sources = Base.metadata.tables["git_sources"]
assert "include_hidden" in sources.c, (
"git_sources must have the include_hidden column (phase 105)"
)
col = sources.c["include_hidden"]
assert col.nullable is False, "git_sources.include_hidden must be NOT NULL"
sd = col.server_default
assert isinstance(sd, DefaultClause), "include_hidden needs a server default"
assert isinstance(sd.arg, TextClause), (
"the server default must be the literal SQL text false"
)
assert sd.arg.text == "false", "include_hidden server default must be false"
def test_git_source_python_default_include_hidden_false() -> None:
"""A freshly constructed row (no DB) resolves the flag to ``False``
via the Python-side default (``default=False``) — the ORM INSERT-time
default, so an ORM insert that omits the column inserts ``false``
rather than NULL (the server default ``false`` independently covers
non-ORM inserts — A4)."""
from app.models import GitSource
row = GitSource(url="https://example.com/r.git", kind="git")
col = row.__table__.c["include_hidden"]
assert col.default is not None, (
"include_hidden needs a Python-side (INSERT-time) default"
)
# A scalar default: the arg IS the value (not a callable).
assert col.default.arg is False, "the default must resolve to False"
def test_doc_drafts_column_contract() -> None:
"""Phase 59: the editable triple (title/path/body) + status +
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
+1
View File
@@ -733,6 +733,7 @@ class _GatedImport:
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
) -> ImportSummary:
self.prune_flags.append(prune)
if progress is not None: