phase: 105_hidden_folders_toggle
All completion criteria verified. Everything is green. **Phase 105 final verification pass — all criteria verified** - Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page - Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified) - Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓ - Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation) - Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed - Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings) **Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched). **Deviations:** none — no defects found; no code changes were needed on this pass. **Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
This commit is contained in:
@@ -0,0 +1,676 @@
|
||||
"""Phase 105 story E2E (Playwright): the per-source hidden-folders toggle —
|
||||
the "Hidden" checkbox next to the "Ignore paths" button makes a source's
|
||||
dot-prefixed paths indexable (``/git-sources.html``).
|
||||
|
||||
Story source: ``TODO.md`` L3 (owner roadmap confirmation 2026-09-14 —
|
||||
TODO-derived, no separate user-story file).
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov
|
||||
|
||||
The story gate proves the whole item end to end through the REAL page +
|
||||
REAL API + REAL sync pipeline (the in-app Sync's background task, mock
|
||||
LLM). **No git, no network** — the suite's single source is a local
|
||||
directory row (``kind="local"``, phase 38) over a fixture tree with one
|
||||
visible file, one hidden-folder file (the phase-105 subject), and one
|
||||
``.venv`` file (``EXCLUDED_DIRS`` — never indexed in either state, A1).
|
||||
The module app boots with one deterministic ``BOR_GIT_SOURCES`` env URL
|
||||
(the ``test_source_ignore_paths.py`` module-env pattern) that is NEVER
|
||||
synced or cloned: every sync in this suite runs with the local DB row in
|
||||
place (DB rows win over the env fallback), and the one env-fallback test
|
||||
never triggers a sync.
|
||||
|
||||
Contract under test:
|
||||
|
||||
* anonymous: the ``#git-sources-gate`` sign-in gate, the manager hidden,
|
||||
NO ``/api/git-sources`` call on load, 403 on
|
||||
``GET``/``POST /api/git-sources`` AND ``PATCH /api/git-sources/{id}``
|
||||
with a bool-only body (the phase-89 anonymous pin extended to the
|
||||
toggle payload);
|
||||
* A4: with the default row, a sync indexes ``visible.md`` ONLY —
|
||||
``detail.files`` counts one, ``.hidden/note.md`` has no ``documents``
|
||||
row, the checkbox renders UNCHECKED and no "hidden on" tag;
|
||||
* A1: flipping the checkbox on (the real click) → the PATCH 200 lands
|
||||
(the "hidden on" tag appears, the announcer ``role=status`` fires the
|
||||
confirmation AFTER the reload line) → sync → ``.hidden/note.md`` IS
|
||||
indexed (``documents`` row present; the KB catalog lists it — the
|
||||
tree/catalog is DB-driven, no extra surface) and ``.venv/junk.md`` is
|
||||
STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders
|
||||
CHECKED (server state);
|
||||
* A2: flipping it OFF (real click) → sync → ``detail.pruned`` includes
|
||||
the hidden doc, the catalog no longer lists it, the tag is gone;
|
||||
* A3: the env-fallback view (table-empty state) renders the "from .env"
|
||||
tag with NO Hidden checkbox and NO "Ignore paths" button;
|
||||
* a11y + error surface: the checkbox has a full accessible name
|
||||
containing the source location (``Index hidden folders for local
|
||||
source: …``), is keyboard-focusable, the tag text is "hidden on"
|
||||
(never color alone); ``#git-sources-hidden-error`` exists with
|
||||
``role="alert"`` and stays ``hidden`` through the happy path — and a
|
||||
network-aborted PATCH shows the canned "not changed" message and
|
||||
reverts the box to the server state (§7.4 never-stale).
|
||||
|
||||
Test → contract mapping (one test per bullet, the phase-89 suite's
|
||||
shape):
|
||||
1. ``test_anonymous_gate_and_403s``
|
||||
2. ``test_hidden_off_by_default``
|
||||
3. ``test_toggle_on_indexes_hidden_folders``
|
||||
4. ``test_toggle_off_prunes_hidden``
|
||||
5. ``test_env_fallback_rows_have_no_toggle``
|
||||
6. ``test_toggle_a11y_and_error_surface``
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import SessionLocal
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_HIDDEN", "8141"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
GIT_SOURCES_URL = "/git-sources.html"
|
||||
|
||||
#: The module app's ``BOR_GIT_SOURCES`` — one deterministic URL that is
|
||||
#: NEVER cloned (the env-fallback test reads it; the sync tests all run
|
||||
#: with the local DB row in place, so the env list never reaches a
|
||||
#: clone).
|
||||
ENV_SOURCE = "https://github.com/reese/env-alpha.git"
|
||||
|
||||
#: The three fixture files (source-relative POSIX paths — exactly the
|
||||
#: strings ``documents.path`` stores).
|
||||
VISIBLE_MD = "visible.md"
|
||||
HIDDEN_NOTE = ".hidden/note.md"
|
||||
VENV_JUNK = ".venv/junk.md"
|
||||
|
||||
#: ``POST /api/sync`` → terminal ``GET /api/sync/status`` (the
|
||||
#: test_sync_button.py polling idiom) — real import of ≤2 small files
|
||||
#: against the mock LLM; generous budget.
|
||||
SYNC_TIMEOUT_S = 60.0
|
||||
SYNC_TICK_S = 2.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The local source's on-disk tree (task 06 step 1).
|
||||
|
||||
Module-scoped (the same reasoning as ``test_sync_button.py``'s
|
||||
module-scoped fixture note): ``tmp_path`` is function-scoped while
|
||||
the module-scoped app + the seeded row reference the dir for the
|
||||
module's lifetime, so it is built under ``tmp_path_factory`` (the
|
||||
same pytest-managed temp area, module-safe). Plain files — no git:
|
||||
a ``kind="local"`` row is walked directly by the sync (phase 38).
|
||||
"""
|
||||
root = tmp_path_factory.mktemp("hidden_src")
|
||||
(root / ".hidden").mkdir()
|
||||
(root / ".venv").mkdir()
|
||||
(root / VISIBLE_MD).write_text(
|
||||
"# Visible\n"
|
||||
"\n"
|
||||
"The visible doc — indexed in EVERY state (flag on and off).\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
# The phase-105 subject: a file INSIDE a hidden (dot-prefixed)
|
||||
# folder — indexed only when the row's flag is on (A1).
|
||||
(root / HIDDEN_NOTE).write_text(
|
||||
"# Hidden note\n"
|
||||
"\n"
|
||||
"A file inside a hidden (dot) folder — the toggle's subject:\n"
|
||||
"indexed, embedded, and summarized like any visible file when\n"
|
||||
"the flag is on; pruned from the KB when it flips off (A2).\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
# EXCLUDED_DIRS content: ``.venv`` is a cache, never content —
|
||||
# skipped in BOTH flag states (A1).
|
||||
(root / VENV_JUNK).write_text(
|
||||
"venv junk — EXCLUDED_DIRS content: never indexed, both states.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, source_dir: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module env (the conftest pattern,
|
||||
module-scoped): one deterministic ``BOR_GIT_SOURCES`` URL (the env
|
||||
fallback's subject — never cloned), module-scratch checkouts/
|
||||
upload dirs, the mock LLM. No git anywhere in this suite's path."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
||||
# ever sent in this suite, but the app boots with the same env
|
||||
# shape.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The empty-table env fallback (only read while ``git_sources`` is
|
||||
# EMPTY — the env-fallback test; never cloned in this suite).
|
||||
env["BOR_GIT_SOURCES"] = ENV_SOURCE
|
||||
# Module-scratch dirs (never reached by this suite's local-row
|
||||
# syncs — kept explicit so a shared checkouts dir cannot leak rows
|
||||
# into the walk).
|
||||
scratch = source_dir.parent
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB per test (the E2E isolation pattern, the
|
||||
test_sync_button.py module-env DSN): the sync's counts and the
|
||||
catalog must be each test's own doing."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""This suite owns ``git_sources`` AND the KB tables (the E2E
|
||||
isolation pattern): suites run in isolation but share one
|
||||
Postgres, and a leftover row would flip the app from the
|
||||
``BOR_GIT_SOURCES`` env fallback to the DB list (and a leftover
|
||||
document would skew the prune counts). Empty BOTH before and
|
||||
after every test."""
|
||||
_truncate_all()
|
||||
yield
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
||||
"""Real form login landing on the git sources page (admin settled:
|
||||
Sign out visible, the manager revealed by the page module)."""
|
||||
login(page, app_url, next=GIT_SOURCES_URL)
|
||||
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-content")).to_be_visible()
|
||||
|
||||
|
||||
def _seed_local_source(page: Page, app_url: str, source_dir: Path) -> str:
|
||||
"""(Re)seed the stored row for the module fixture dir via the REAL
|
||||
admin API (201) — ``kind="local"`` — with NO ``ignore_paths`` and
|
||||
NO ``include_hidden`` (A4: absent at create → stored ``False``) —
|
||||
then reload the view so the page's table re-fetches the row (the
|
||||
API seed is server-side: the mounted view does not know about it
|
||||
without a re-show). Returns the stored (expanded) path — the
|
||||
row's ``url`` location value, the same string the page renders
|
||||
and aria-labels."""
|
||||
r = page.request.post(
|
||||
f"{app_url}/api/git-sources",
|
||||
data={"kind": "local", "path": str(source_dir)},
|
||||
)
|
||||
assert r.status == 201, r.text
|
||||
body = r.json()
|
||||
assert body["ignore_paths"] == [] # the default list, round-tripped
|
||||
assert body["include_hidden"] is False # A4: default off
|
||||
page.reload()
|
||||
_admin_git_sources_page(page, app_url)
|
||||
return body["url"]
|
||||
|
||||
|
||||
def run_sync(page: Page, app_url: str) -> dict[str, Any]:
|
||||
"""``POST /api/sync`` → poll ``GET /api/sync/status`` to a terminal
|
||||
state (the test_sync_button.py polling idiom — ~2 s ticks, 60 s
|
||||
budget). Returns the terminal status body."""
|
||||
r = page.request.post(f"{app_url}/api/sync")
|
||||
assert r.status == 202, r.text
|
||||
deadline = time.monotonic() + SYNC_TIMEOUT_S
|
||||
body: dict[str, Any] = {}
|
||||
while True:
|
||||
s = page.request.get(f"{app_url}/api/sync/status")
|
||||
assert s.status == 200, s.text
|
||||
body = s.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
assert time.monotonic() < deadline, (
|
||||
f"sync did not reach a terminal state: {body}"
|
||||
)
|
||||
time.sleep(SYNC_TICK_S)
|
||||
|
||||
|
||||
def _catalog_paths(page: Page, app_url: str) -> list[str]:
|
||||
"""The RAG catalog's data source (``GET /api/docs`` — the Sources
|
||||
page's table): every ``documents.path`` (source-relative POSIX).
|
||||
The tree/catalog is DB-driven — a newly indexed hidden document
|
||||
appears here automatically (no extra surface)."""
|
||||
r = page.request.get(f"{app_url}/api/docs")
|
||||
assert r.status == 200, r.text
|
||||
return [d["path"] for d in r.json()["documents"]]
|
||||
|
||||
|
||||
def _db_document_paths() -> list[str]:
|
||||
"""Every ``documents.path`` straight from the DB — the literal
|
||||
"no ``documents`` row" assertion (the catalog is the same table,
|
||||
read through the API)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(text("SELECT path FROM documents")).fetchall()
|
||||
return [row[0] for row in rows]
|
||||
|
||||
|
||||
def _row(page: Page, value: str) -> Any:
|
||||
"""The table row whose mono location cell shows ``value`` (re-resolved
|
||||
on every call — the row re-renders from the server after each
|
||||
successful toggle)."""
|
||||
return page.locator("#git-sources-tbody tr", has_text=value)
|
||||
|
||||
|
||||
def _hidden_box(page: Page, value: str) -> Any:
|
||||
"""The row's Hidden checkbox (fresh locator — the element is
|
||||
re-created by the post-toggle re-render)."""
|
||||
return _row(page, value).locator(".git-source-hidden-box")
|
||||
|
||||
|
||||
def _flip_hidden(page: Page, value: str, wanted_on: bool) -> None:
|
||||
"""Click the row's Hidden checkbox and wait for the §7.4 success
|
||||
lifecycle to settle: the PATCH 200 landed, the row re-rendered
|
||||
from the server, and the CONFIRMATION is the last announcer message
|
||||
(the reload's "N sources listed." cannot overwrite it — the
|
||||
phase-89 order)."""
|
||||
_hidden_box(page, value).click()
|
||||
tag = _row(page, value).locator(".git-source-hidden-count")
|
||||
if wanted_on:
|
||||
expect(tag).to_have_text("hidden on", timeout=30_000)
|
||||
else:
|
||||
expect(tag).to_have_count(0, timeout=30_000)
|
||||
expect(
|
||||
page.locator("#git-sources-announcer")
|
||||
).to_have_text(
|
||||
f"Hidden folders {'enabled' if wanted_on else 'disabled'} for {value}."
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Anonymous: gate, inert manager, no API calls, 403s (incl. the
|
||||
# bool-only toggle PATCH)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_anonymous_gate_and_403s(page: Page, app_url: str, db_ready: None) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
|
||||
# Track every /api/git-sources request the page itself makes — the
|
||||
# gate must be reached WITHOUT touching the admin API (the
|
||||
# test_source_ignore_paths.py pattern).
|
||||
api_calls: list[str] = []
|
||||
page.on(
|
||||
"request",
|
||||
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
|
||||
)
|
||||
|
||||
page.goto(app_url + GIT_SOURCES_URL)
|
||||
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
|
||||
|
||||
# The sign-in gate (the #sources-gate pattern, phase 16)…
|
||||
gate = page.locator("#git-sources-gate")
|
||||
expect(gate).to_be_visible()
|
||||
expect(gate).to_contain_text("Sign in to manage the git sources")
|
||||
expect(
|
||||
gate.locator("a[href='/login.html?next=/git-sources.html']")
|
||||
).to_have_count(1)
|
||||
# …and the manager is absent/inert: table + form + env note, all
|
||||
# inside the hidden #git-sources-content.
|
||||
expect(page.locator("#git-sources-content")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-table")).to_be_hidden()
|
||||
expect(page.locator("#git-source-form")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-env-note")).to_be_hidden()
|
||||
|
||||
# The gate never called the admin API…
|
||||
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
|
||||
# …and the API 403s anonymous callers on GET/POST AND the
|
||||
# phase-105 toggle PATCH with a bool-only body (the row's list is
|
||||
# untouched — the toggle's exact payload is gated too).
|
||||
assert page.request.get(f"{app_url}/api/git-sources").status == 403
|
||||
assert (
|
||||
page.request.post(
|
||||
f"{app_url}/api/git-sources", data={"url": ENV_SOURCE}
|
||||
).status
|
||||
== 403
|
||||
)
|
||||
assert (
|
||||
page.request.patch(
|
||||
f"{app_url}/api/git-sources/{uuid.uuid4()}",
|
||||
data={"include_hidden": True},
|
||||
).status
|
||||
== 403
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. A4: the default row is byte-identical to pre-phase-105 — the sync
|
||||
# indexes visible.md ONLY; the checkbox renders UNCHECKED, no tag
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_hidden_off_by_default(
|
||||
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
|
||||
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||
|
||||
# A4: the default row renders the NEW control in its default state —
|
||||
# the checkbox UNCHECKED (checked comes only from server state) and
|
||||
# NO "hidden on" tag (pre-phase-105 surface + the control).
|
||||
box = _hidden_box(page, stored_path)
|
||||
expect(box).to_have_count(1)
|
||||
expect(box).not_to_be_checked()
|
||||
expect(_row(page, stored_path).locator(".git-source-hidden-count")).to_have_count(0)
|
||||
|
||||
# The real sync: ``visible.md`` ONLY — the walk skipped the
|
||||
# dot-prefixed ``.hidden/`` component (A4 byte-identical default)
|
||||
# AND ``.venv/`` (EXCLUDED_DIRS in both states).
|
||||
sync = run_sync(page, app_url)
|
||||
assert sync["state"] == "success", sync
|
||||
detail = sync["detail"]
|
||||
assert detail["files"] == 1, detail
|
||||
assert detail["added"] == 1, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["errors"] == 0, detail
|
||||
|
||||
# The catalog holds exactly the visible file — no ``documents`` row
|
||||
# for the hidden note (the direct DB read makes "no row" literal)…
|
||||
paths = _catalog_paths(page, app_url)
|
||||
assert VISIBLE_MD in paths, f"visible.md missing from the catalog: {paths}"
|
||||
assert HIDDEN_NOTE not in paths, (
|
||||
f"hidden note leaked into the catalog with the flag off: {paths}"
|
||||
)
|
||||
assert VENV_JUNK not in paths, f"EXCLUDED_DIRS leaked into the catalog: {paths}"
|
||||
assert _db_document_paths() == [VISIBLE_MD]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. A1: the real click flips the flag on — the PATCH 200 lands (tag +
|
||||
# last-announce confirmation), the next sync indexes the hidden
|
||||
# folder; EXCLUDED_DIRS stays excluded; the box re-renders CHECKED
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toggle_on_indexes_hidden_folders(
|
||||
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||
|
||||
# Baseline: the default sync indexes the visible file only.
|
||||
first = run_sync(page, app_url)
|
||||
assert first["state"] == "success", first
|
||||
assert first["detail"]["files"] == 1, first["detail"]
|
||||
|
||||
# A1: flip the checkbox on (the real click) — the §7.4 lifecycle:
|
||||
# PATCH 200 → the "hidden on" tag appears in the source cell and
|
||||
# the confirmation is the LAST announcer message (the reload's
|
||||
# "1 source listed." landed first).
|
||||
_flip_hidden(page, stored_path, wanted_on=True)
|
||||
|
||||
# The checkbox re-renders CHECKED from the server state (never a
|
||||
# local flip) and the API round-trips include_hidden: true.
|
||||
expect(_hidden_box(page, stored_path)).to_be_checked()
|
||||
r = page.request.get(f"{app_url}/api/git-sources")
|
||||
assert r.status == 200, r.text
|
||||
sources = r.json()["sources"]
|
||||
assert [s["include_hidden"] for s in sources] == [True]
|
||||
assert [s["ignore_paths"] for s in sources] == [[]] # the list untouched
|
||||
|
||||
# The NEXT sync indexes the hidden-folder file like any visible
|
||||
# file (embedded + chunked by the mock LLM)…
|
||||
second = run_sync(page, app_url)
|
||||
assert second["state"] == "success", second
|
||||
detail = second["detail"]
|
||||
assert detail["files"] == 2, detail
|
||||
assert detail["added"] == 1, detail # the hidden note only
|
||||
assert detail["unchanged"] == 1, detail # visible.md
|
||||
# …and ``.venv/junk.md`` is STILL absent — EXCLUDED_DIRS content is
|
||||
# never indexed, in either state (A1).
|
||||
paths = _catalog_paths(page, app_url)
|
||||
assert HIDDEN_NOTE in paths, (
|
||||
f"hidden note missing from the catalog after the on-sync: {paths}"
|
||||
)
|
||||
assert VISIBLE_MD in paths, paths
|
||||
assert VENV_JUNK not in paths, (
|
||||
f"EXCLUDED_DIRS leaked into the catalog with the flag on: {paths}"
|
||||
)
|
||||
assert set(_db_document_paths()) == {VISIBLE_MD, HIDDEN_NOTE}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. A2: flipping the flag OFF prunes the previously indexed hidden doc
|
||||
# from the KB on the next sync
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toggle_off_prunes_hidden(
|
||||
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||
|
||||
# On + sync: the hidden note is in the KB.
|
||||
_flip_hidden(page, stored_path, wanted_on=True)
|
||||
first = run_sync(page, app_url)
|
||||
assert first["state"] == "success", first
|
||||
assert first["detail"]["files"] == 2, first["detail"]
|
||||
assert first["detail"]["added"] == 2, first["detail"]
|
||||
assert HIDDEN_NOTE in _catalog_paths(page, app_url)
|
||||
|
||||
# A2: flip it OFF (the real click) — the previously indexed hidden
|
||||
# file leaves the KB on the next sync (the seen-set prune — the
|
||||
# phase-89 A2 / A9 precedent; the tag is gone).
|
||||
_flip_hidden(page, stored_path, wanted_on=False)
|
||||
r = page.request.get(f"{app_url}/api/git-sources")
|
||||
assert [s["include_hidden"] for s in r.json()["sources"]] == [False]
|
||||
|
||||
second = run_sync(page, app_url)
|
||||
assert second["state"] == "success", second
|
||||
detail = second["detail"]
|
||||
assert detail["pruned"] == 1, detail # the hidden doc
|
||||
assert detail["files"] == 1, detail # the walk is back to visible-only
|
||||
paths = _catalog_paths(page, app_url)
|
||||
assert HIDDEN_NOTE not in paths, (
|
||||
f"the hidden doc survived the flag-off prune: {paths}"
|
||||
)
|
||||
assert VISIBLE_MD in paths, f"visible.md must survive the prune: {paths}"
|
||||
assert _db_document_paths() == [VISIBLE_MD]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. A3: env-fallback rows have no toggle (no DB row to store a flag on)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_env_fallback_rows_have_no_toggle(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
# The autouse fixture truncated git_sources — the table is EMPTY,
|
||||
# so the module app's BOR_GIT_SOURCES URL is the effective list
|
||||
# (the test_source_ignore_paths.py module-env pattern).
|
||||
_admin_git_sources_page(page, app_url)
|
||||
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||
row = page.locator("#git-sources-tbody tr")
|
||||
expect(row).to_contain_text(ENV_SOURCE)
|
||||
# The env-fallback surface: the "from .env" tag (never color
|
||||
# alone)…
|
||||
expect(row.locator(".git-source-env-tag")).to_have_count(1)
|
||||
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
|
||||
# …and A3: NO per-row controls at all — neither Remove, nor the
|
||||
# phase-89 "Ignore paths" button, nor the phase-105 Hidden checkbox
|
||||
# (and no "hidden on" tag).
|
||||
expect(row.locator(".git-source-remove")).to_have_count(0)
|
||||
expect(row.locator(".git-source-ignore")).to_have_count(0)
|
||||
expect(row.locator(".git-source-hidden")).to_have_count(0)
|
||||
expect(row.locator(".git-source-hidden-box")).to_have_count(0)
|
||||
expect(row.locator(".git-source-hidden-count")).to_have_count(0)
|
||||
# The env-fallback note explains the active list's origin…
|
||||
expect(page.locator("#git-sources-env-note")).to_be_visible()
|
||||
# …and the API agrees: from_env true, null ids, the env URL, an
|
||||
# empty ignore list, and include_hidden False (no DB row to store a
|
||||
# list or a flag on — the GitSourceRow contract, task 03).
|
||||
r = page.request.get(f"{app_url}/api/git-sources")
|
||||
assert r.status == 200, r.text
|
||||
body = r.json()
|
||||
assert body["from_env"] is True
|
||||
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE]
|
||||
assert all(
|
||||
s["id"] is None and s["ignore_paths"] == [] and s["include_hidden"] is False
|
||||
for s in body["sources"]
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6. a11y + the error surface: the accessible name, keyboard focus, the
|
||||
# text tag, and the role=alert line (hidden on the happy path, the
|
||||
# canned message + revert on a network failure)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toggle_a11y_and_error_surface(
|
||||
page: Page, app_url: str, db_ready: None, source_dir: Path
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
stored_path = _seed_local_source(page, app_url, source_dir)
|
||||
|
||||
box = _hidden_box(page, stored_path)
|
||||
expect(box).to_have_count(1)
|
||||
# The full accessible name contains the source location (the
|
||||
# aria-label — the ONLY place the value appears, setAttribute,
|
||||
# never innerHTML)…
|
||||
expect(box).to_have_attribute(
|
||||
"aria-label", f"Index hidden folders for local source: {stored_path}"
|
||||
)
|
||||
expect(
|
||||
page.get_by_role(
|
||||
"checkbox", name=f"Index hidden folders for local source: {stored_path}"
|
||||
)
|
||||
).to_have_count(1)
|
||||
# …it is keyboard-reachable — a plain Tab sequence from the page
|
||||
# top reaches it (one full focus cycle covers every focusable
|
||||
# element; the visible ring is the GLOBAL :focus-visible rule,
|
||||
# styles.css — unit-pinned, no per-control rule)…
|
||||
for _ in range(60):
|
||||
page.keyboard.press("Tab")
|
||||
if box.evaluate("el => el === document.activeElement"):
|
||||
break
|
||||
expect(box).to_be_focused()
|
||||
# …and it receives programmatic focus too (the a11y trees agree).
|
||||
box.focus()
|
||||
expect(box).to_be_focused()
|
||||
# …and the visible "Hidden" label wraps the box (text + control —
|
||||
# never color or icon alone).
|
||||
label = page.locator("label.git-source-hidden")
|
||||
expect(label).to_have_count(1)
|
||||
expect(label).to_contain_text("Hidden")
|
||||
|
||||
# The page-level error line exists, is a real role=alert, and stays
|
||||
# hidden through the happy path.
|
||||
error = page.locator("#git-sources-hidden-error")
|
||||
expect(error).to_have_count(1)
|
||||
assert error.get_attribute("role") == "alert"
|
||||
expect(error).to_be_hidden()
|
||||
|
||||
# The tag is TEXT (never color alone — WCAG 1.4.1): flip on…
|
||||
_flip_hidden(page, stored_path, wanted_on=True)
|
||||
expect(
|
||||
_row(page, stored_path).locator(".git-source-hidden-count")
|
||||
).to_have_text("hidden on")
|
||||
expect(error).to_be_hidden() # the happy path never touches the line
|
||||
|
||||
# §7.4 failure path: a network-level abort of the PATCH → the
|
||||
# canned "not changed" message in the role=alert line and the box
|
||||
# reverts to the SERVER state (still ON — the request never
|
||||
# landed) and re-enables. The tag stays (the server is unchanged).
|
||||
def _abort_patch(route: Any) -> None:
|
||||
if route.request.method == "PATCH":
|
||||
route.abort()
|
||||
else:
|
||||
route.continue_()
|
||||
|
||||
page.route("**/api/git-sources/**", _abort_patch)
|
||||
try:
|
||||
_hidden_box(page, stored_path).click()
|
||||
expect(error).to_be_visible(timeout=30_000)
|
||||
expect(error).to_have_text(
|
||||
"Could not reach the server — the setting was not changed."
|
||||
)
|
||||
expect(
|
||||
_row(page, stored_path).locator(".git-source-hidden-count")
|
||||
).to_have_count(1) # the server state is unchanged: still on
|
||||
reverted = _hidden_box(page, stored_path)
|
||||
expect(reverted).to_be_checked() # reverted to the server state
|
||||
expect(reverted).to_be_enabled() # re-enabled — never stale
|
||||
finally:
|
||||
page.unroute("**/api/git-sources/**")
|
||||
@@ -32,9 +32,16 @@ Contract under test:
|
||||
* ignore paths (phase 89) — POST accepts the RAW box lines for both
|
||||
kinds (optional, absent → ``[]``), stored normalized (A1) with the A4
|
||||
fixed-detail 422s (shared gate with PATCH); GET reports each row's
|
||||
stored list (env rows ``[]``); ``PATCH /{source_id}`` (admin-only)
|
||||
replaces the list wholesale (A5 — an empty list clears all), 404
|
||||
unknown id, 422 fixed details for the A4 limits, the row otherwise
|
||||
stored list (env rows ``[]``);
|
||||
* hidden-folders flag (phase 105) — POST accepts ``include_hidden`` for
|
||||
both kinds (optional, absent → stored ``False``, A4); GET reports the
|
||||
stored flag (env rows ``False`` — no DB row to store a flag on);
|
||||
* ``PATCH /{source_id}`` (admin-only) — phase 89 A5 + phase 105: each
|
||||
field is optional, present-wins — ``ignore_paths`` when present
|
||||
REPLACES the list wholesale (A5 — an empty list clears all),
|
||||
``include_hidden`` when present sets the flag, both absent → 200
|
||||
no-op, 404 unknown id, 422 fixed details for the A4 limits, a
|
||||
rejected list never half-applies the flag, the row otherwise
|
||||
unchanged.
|
||||
|
||||
``git_sources`` is global state: truncated around every test.
|
||||
@@ -99,10 +106,14 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
|
||||
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
|
||||
assert r.status_code == 403
|
||||
assert r.json() == {"detail": "admin only"}
|
||||
# The phase-89 ignore-list PATCH is gated the same way.
|
||||
# The phase-89 ignore-list PATCH is gated the same way — and so is
|
||||
# the phase-105 bool-only payload (the toggle's exact request).
|
||||
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"ignore_paths": ["a"]})
|
||||
assert r.status_code == 403
|
||||
assert r.json() == {"detail": "admin only"}
|
||||
r = client.patch(f"/api/git-sources/{uuid.uuid4()}", json={"include_hidden": True})
|
||||
assert r.status_code == 403
|
||||
assert r.json() == {"detail": "admin only"}
|
||||
# Nothing landed in the table.
|
||||
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
||||
|
||||
@@ -132,8 +143,10 @@ def test_get_empty_table_with_env_returns_env_rows(
|
||||
"url": "https://a.example.com/one.git",
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
# Env rows have no DB row to store a list on (phase 89).
|
||||
# Env rows have no DB row to store a list on (phase 89) or
|
||||
# a flag on (phase 105).
|
||||
"ignore_paths": [],
|
||||
"include_hidden": False,
|
||||
},
|
||||
{
|
||||
"id": None,
|
||||
@@ -142,6 +155,7 @@ def test_get_empty_table_with_env_returns_env_rows(
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
"ignore_paths": [],
|
||||
"include_hidden": False,
|
||||
},
|
||||
]
|
||||
|
||||
@@ -208,9 +222,11 @@ def test_post_creates_trimmed_and_list_stops_using_env(
|
||||
uuid.UUID(body["id"])
|
||||
assert body["added_at"] is not None
|
||||
# Phase 89: the response gains ``ignore_paths`` — absent at create
|
||||
# time → ``[]``.
|
||||
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
|
||||
# time → ``[]``; phase 105 adds ``include_hidden`` — absent →
|
||||
# ``False`` (A4).
|
||||
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
|
||||
assert body["ignore_paths"] == []
|
||||
assert body["include_hidden"] is False
|
||||
|
||||
# The DB row now wins: from_env False, the env URL is gone from the list.
|
||||
body = admin_client.get("/api/git-sources").json()
|
||||
@@ -321,9 +337,11 @@ def test_post_local_creates_stored_row_with_expanded_path(
|
||||
body = r.json()
|
||||
# The phase-35 response shape is unchanged — the local row reports
|
||||
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET;
|
||||
# phase 89 adds ``ignore_paths`` (absent → ``[]``).
|
||||
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
|
||||
# phase 89 adds ``ignore_paths`` (absent → ``[]``); phase 105 adds
|
||||
# ``include_hidden`` (absent → ``False``, A4).
|
||||
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
|
||||
assert body["ignore_paths"] == []
|
||||
assert body["include_hidden"] is False
|
||||
uuid.UUID(body["id"])
|
||||
assert body["url"] == str(real_dir)
|
||||
assert body["added_at"] is not None
|
||||
@@ -556,6 +574,7 @@ def test_delete_removes_row_and_falls_back_to_env(
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
"ignore_paths": [], # env rows: no DB row to store a list on
|
||||
"include_hidden": False, # … or a flag on (phase 105)
|
||||
}
|
||||
]
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
@@ -671,6 +690,7 @@ def test_get_reports_stored_ignore_paths_and_env_rows_empty(
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
"ignore_paths": [],
|
||||
"include_hidden": False,
|
||||
}
|
||||
]
|
||||
|
||||
@@ -690,8 +710,9 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
|
||||
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={"ignore_paths": ["a/", "b"]})
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert set(body) == {"id", "url", "added_at", "ignore_paths"}
|
||||
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
|
||||
assert body["ignore_paths"] == ["a", "b"] # normalized
|
||||
assert body["include_hidden"] is False # untouched by a list-only PATCH
|
||||
for key in ("id", "url", "added_at"):
|
||||
assert body[key] == before[key]
|
||||
# Round-trip through GET.
|
||||
@@ -706,17 +727,34 @@ def test_patch_replaces_ignore_paths_including_clear(admin_client: TestClient) -
|
||||
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == []
|
||||
|
||||
|
||||
def test_patch_missing_field_is_422(admin_client: TestClient) -> None:
|
||||
"""``ignore_paths`` is REQUIRED (replace semantics, A5) — an absent
|
||||
field is a 422 with the model's own detail."""
|
||||
created = admin_client.post("/api/git-sources", json={"url": "https://example.com/req.git"})
|
||||
assert created.status_code == 201
|
||||
r = admin_client.patch(f"/api/git-sources/{created.json()['id']}", json={})
|
||||
assert r.status_code == 422
|
||||
# The model's own (Pydantic) detail — the missing required field.
|
||||
assert any(
|
||||
item.get("loc") == ["body", "ignore_paths"] for item in r.json()["detail"]
|
||||
def test_patch_empty_body_is_a_noop_200(admin_client: TestClient) -> None:
|
||||
"""Phase 105: BOTH fields absent (or explicit None) → 200 no-op —
|
||||
the row is untouched. (The pre-phase-105 pin of an absent
|
||||
``ignore_paths`` as a 422 is retired — the list is optional now; a
|
||||
PRESENT list keeps the phase-89 A5 replace semantics.)"""
|
||||
created = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"url": "https://example.com/noop.git", "ignore_paths": ["a/b"]},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
before = created.json()
|
||||
|
||||
r = admin_client.patch(f"/api/git-sources/{before['id']}", json={})
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["ignore_paths"] == ["a/b"]
|
||||
assert body["include_hidden"] is False
|
||||
for key in ("id", "url", "added_at"):
|
||||
assert body[key] == before[key]
|
||||
# Explicit None values are "absent" too — still a no-op.
|
||||
r = admin_client.patch(
|
||||
f"/api/git-sources/{before['id']}", json={"ignore_paths": None, "include_hidden": None}
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["ignore_paths"] == ["a/b"]
|
||||
assert r.json()["include_hidden"] is False
|
||||
# The row never changed — round-trip through GET.
|
||||
assert admin_client.get("/api/git-sources").json()["sources"][0]["ignore_paths"] == ["a/b"]
|
||||
|
||||
|
||||
def test_patch_unknown_id_returns_404(admin_client: TestClient) -> None:
|
||||
@@ -772,3 +810,142 @@ def test_patch_accepts_a4_boundaries(admin_client: TestClient) -> None:
|
||||
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": [long_entry]})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["ignore_paths"] == [long_entry]
|
||||
|
||||
|
||||
# --- hidden-folders flag (phase 105) ----------------------------------------
|
||||
|
||||
|
||||
def test_get_reports_include_hidden_default_false(admin_client: TestClient, db: Session) -> None:
|
||||
"""A fresh stored row (no flag passed) reports ``include_hidden:
|
||||
false`` — in the GET round-trip AND on the model (the column's
|
||||
server default, A4)."""
|
||||
db.add(GitSource(url="https://example.com/fresh.git"))
|
||||
db.commit()
|
||||
body = admin_client.get("/api/git-sources").json()
|
||||
assert body["sources"][0]["include_hidden"] is False
|
||||
row = db.scalars(select(GitSource)).one()
|
||||
assert row.include_hidden is False
|
||||
|
||||
|
||||
def test_post_stores_include_hidden_both_kinds(
|
||||
admin_client: TestClient, tmp_path: Path
|
||||
) -> None:
|
||||
"""POST accepts ``include_hidden`` for both kinds (present → stored
|
||||
as sent); the 201 body and the GET round-trip report it."""
|
||||
r = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"url": "https://example.com/hidden-git.git", "include_hidden": True},
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
assert r.json()["include_hidden"] is True
|
||||
|
||||
real_dir = tmp_path / "hidden-local"
|
||||
real_dir.mkdir()
|
||||
r = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"kind": "local", "path": str(real_dir), "include_hidden": True},
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
assert r.json()["include_hidden"] is True
|
||||
|
||||
by_url = {
|
||||
s["url"]: s["include_hidden"]
|
||||
for s in admin_client.get("/api/git-sources").json()["sources"]
|
||||
}
|
||||
assert by_url["https://example.com/hidden-git.git"] is True
|
||||
assert by_url[str(real_dir)] is True
|
||||
|
||||
|
||||
def test_patch_bool_only_sets_flag_leaves_list(admin_client: TestClient) -> None:
|
||||
"""The toggle's exact payload — ``{"include_hidden": …}`` alone:
|
||||
the flag is set, the ignore list is UNCHANGED (present-wins, phase
|
||||
105)."""
|
||||
created = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"url": "https://example.com/toggle.git", "ignore_paths": ["a/b"]},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
sid = created.json()["id"]
|
||||
|
||||
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": True})
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["include_hidden"] is True
|
||||
assert body["ignore_paths"] == ["a/b"] # untouched
|
||||
assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True
|
||||
|
||||
# And back off again.
|
||||
r = admin_client.patch(f"/api/git-sources/{sid}", json={"include_hidden": False})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["include_hidden"] is False
|
||||
assert r.json()["ignore_paths"] == ["a/b"]
|
||||
|
||||
|
||||
def test_patch_list_only_replaces_list_leaves_flag(admin_client: TestClient) -> None:
|
||||
"""The dialog's exact payload — ``{"ignore_paths": …}`` alone:
|
||||
the list is REPLACED (normalized, phase-89 A5), the flag is
|
||||
UNCHANGED — byte-identical to the pre-phase-105 dialog PATCH."""
|
||||
created = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={
|
||||
"url": "https://example.com/dialog.git",
|
||||
"ignore_paths": ["a/b"],
|
||||
"include_hidden": True,
|
||||
},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
sid = created.json()["id"]
|
||||
|
||||
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": ["c/d", " e "]})
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["ignore_paths"] == ["c/d", "e"] # normalized
|
||||
assert body["include_hidden"] is True # untouched
|
||||
assert admin_client.get("/api/git-sources").json()["sources"][0]["include_hidden"] is True
|
||||
|
||||
|
||||
def test_patch_both_fields_apply_independently(admin_client: TestClient) -> None:
|
||||
"""``{"ignore_paths": [], "include_hidden": true}`` — the list is
|
||||
cleared AND the flag is set in one request."""
|
||||
created = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"url": "https://example.com/both.git", "ignore_paths": ["old/one"]},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
sid = created.json()["id"]
|
||||
|
||||
r = admin_client.patch(
|
||||
f"/api/git-sources/{sid}", json={"ignore_paths": [], "include_hidden": True}
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
assert body["ignore_paths"] == [] # cleared
|
||||
assert body["include_hidden"] is True # set
|
||||
|
||||
|
||||
def test_patch_422_does_not_half_apply_flag(admin_client: TestClient) -> None:
|
||||
"""A PRESENT bad list 422s with the fixed A4 details — and the
|
||||
OTHER field in the same body never half-applies: after each 422
|
||||
the flag is still the stored value (validation raises before any
|
||||
assignment, request transaction untouched)."""
|
||||
created = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"url": "https://example.com/half.git", "ignore_paths": ["keep"]},
|
||||
)
|
||||
assert created.status_code == 201
|
||||
sid = created.json()["id"]
|
||||
|
||||
for payload, detail in (
|
||||
([" "], "ignore paths must be non-empty"),
|
||||
([f"e{i}" for i in range(201)], "a source has at most 200 ignore paths"),
|
||||
(["a" * 501], "an ignore path exceeds 500 characters"),
|
||||
):
|
||||
r = admin_client.patch(
|
||||
f"/api/git-sources/{sid}", json={"ignore_paths": payload, "include_hidden": True}
|
||||
)
|
||||
assert r.status_code == 422, f"{detail!r}: {r.text}"
|
||||
assert r.json()["detail"] == detail
|
||||
# Nothing half-applied: list AND flag are both still the originals.
|
||||
row = admin_client.get("/api/git-sources").json()["sources"][0]
|
||||
assert row["ignore_paths"] == ["keep"]
|
||||
assert row["include_hidden"] is False
|
||||
|
||||
@@ -27,15 +27,19 @@ is asserted against the canned value).
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import GitSource
|
||||
from app.rag.importer import ImportSummary
|
||||
from scripts import import_docs
|
||||
from scripts.git_sync import GitSyncError
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
|
||||
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
|
||||
@@ -47,11 +51,31 @@ def _git_row(url: str, ignore_paths: list[str] | None = None) -> GitSource:
|
||||
return GitSource(url=url, kind="git", ignore_paths=ignore_paths or [])
|
||||
|
||||
|
||||
def _local_row(path: str, ignore_paths: list[str] | None = None) -> GitSource:
|
||||
def _local_row(
|
||||
path: str,
|
||||
ignore_paths: list[str] | None = None,
|
||||
include_hidden: bool = False,
|
||||
) -> GitSource:
|
||||
"""A local row as the phase-38 API stores it: the expanded path in
|
||||
both ``path`` and the NOT-NULL ``url`` location column (plus the
|
||||
phase-89 ignore list, empty by default)."""
|
||||
return GitSource(url=path, kind="local", path=path, ignore_paths=ignore_paths or [])
|
||||
phase-89 ignore list and the phase-105 hidden-folders flag, both
|
||||
defaulting off — A4)."""
|
||||
return GitSource(
|
||||
url=path, kind="local", path=path,
|
||||
ignore_paths=ignore_paths or [], include_hidden=include_hidden,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def clean_documents(db: Session) -> Iterator[None]:
|
||||
"""Phase 105: the real-import CLI tests write ``documents``/
|
||||
``chunks`` (the canonical KB state) — global, truncated around
|
||||
every such test."""
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
|
||||
|
||||
class FakeImportSources:
|
||||
@@ -68,10 +92,12 @@ class FakeImportSources:
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
) -> ImportSummary:
|
||||
self.calls.append(
|
||||
{"sources": list(sources), "prune": prune, "limit": limit,
|
||||
"ignore_by_root": ignore_by_root}
|
||||
"ignore_by_root": ignore_by_root,
|
||||
"include_hidden_by_root": include_hidden_by_root}
|
||||
)
|
||||
return ImportSummary(files=1, added=1)
|
||||
|
||||
@@ -168,10 +194,16 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
|
||||
sources_dir=str(tmp_path / "bor"),
|
||||
)
|
||||
|
||||
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
|
||||
|
||||
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
|
||||
assert ignore_map == {} # phase 89: no row carries a list → empty map
|
||||
# Phase 105: flag-off rows still contribute their root — with False
|
||||
# (A4), keyed by the SAME root string the importer sees.
|
||||
assert hidden_map == {
|
||||
str(tmp_path / "bor" / "homelab"): False,
|
||||
str(tmp_path / "bor" / "deploy"): False,
|
||||
}
|
||||
assert calls == [
|
||||
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
|
||||
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
|
||||
@@ -186,10 +218,11 @@ def test_resolve_sources_cli_source_wins(
|
||||
settings = _settings(git_sources="https://host/a/repo.git")
|
||||
manual = tmp_path / "Manual"
|
||||
|
||||
sources, ignore_map = import_docs._resolve_sources([manual], settings)
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources([manual], settings)
|
||||
|
||||
assert sources == [manual]
|
||||
assert ignore_map == {} # phase 89: manual dirs have no rows → no ignore
|
||||
assert hidden_map == {} # phase 105: manual dirs have no rows → hidden skipped
|
||||
assert calls == [] # git is never touched when --source is given
|
||||
|
||||
|
||||
@@ -210,10 +243,12 @@ def test_resolve_sources_db_rows_win_over_env(
|
||||
sources_dir=str(tmp_path / "bor"),
|
||||
)
|
||||
|
||||
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
|
||||
|
||||
assert sources == [tmp_path / "bor" / "only"]
|
||||
assert ignore_map == {} # phase 89: no row carries a list → empty map
|
||||
# Phase 105: the default-flag row contributes its root with False (A4).
|
||||
assert hidden_map == {str(tmp_path / "bor" / "only"): False}
|
||||
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
||||
|
||||
|
||||
@@ -222,9 +257,10 @@ def test_resolve_sources_defaults_when_nothing_configured(
|
||||
) -> None:
|
||||
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
|
||||
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
|
||||
sources, ignore_map = import_docs._resolve_sources(None, _settings())
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, _settings())
|
||||
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
|
||||
assert ignore_map == {} # phase 89: the legacy fallback has no rows
|
||||
assert hidden_map == {} # phase 105: the legacy fallback has no rows
|
||||
|
||||
|
||||
def test_resolve_sources_rows_branch_builds_ignore_map(
|
||||
@@ -250,11 +286,13 @@ def test_resolve_sources_rows_branch_builds_ignore_map(
|
||||
)
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
|
||||
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
|
||||
|
||||
assert sources == [tmp_path / "bor" / "only", local_dir]
|
||||
# Keyed by the SAME string the importer sees (the root, not the name).
|
||||
assert ignore_map == {str(local_dir): ["ignore/"]}
|
||||
# Phase 105: both rows are flag-off → per-root False entries (A4).
|
||||
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
|
||||
|
||||
|
||||
def test_resolve_sources_two_rows_sharing_root_string_extend(
|
||||
@@ -279,11 +317,14 @@ def test_resolve_sources_two_rows_sharing_root_string_extend(
|
||||
)
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
|
||||
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
|
||||
|
||||
shared = str(tmp_path / "bor" / "shared")
|
||||
assert sources == [tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]
|
||||
assert ignore_map == {shared: ["a/", "b"]} # union, row order
|
||||
# Phase 105 collision: the shared root gets the OR of the flags —
|
||||
# both rows off here, so one False entry for the one root string.
|
||||
assert hidden_map == {shared: False}
|
||||
|
||||
|
||||
def test_main_rows_branch_passes_ignore_map_to_import(
|
||||
@@ -318,9 +359,145 @@ def test_main_rows_branch_passes_ignore_map_to_import(
|
||||
call = fake_import.calls[0]
|
||||
assert call["sources"] == [local_dir]
|
||||
assert call["ignore_by_root"] == {str(local_dir): ["ignore/"]}
|
||||
# Phase 105: the default-flag row passes the per-root map too — a
|
||||
# False entry, not an absent key (the importer reads it per root).
|
||||
assert call["include_hidden_by_root"] == {str(local_dir): False}
|
||||
assert call["prune"] is False # the CLI's no-prune default is unchanged
|
||||
|
||||
|
||||
# --- phase 105: per-row hidden-folders flag ---------------------------------
|
||||
|
||||
|
||||
def test_main_rows_branch_passes_include_hidden_map_to_import(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Phase 105: a ``kind=local`` row with ``include_hidden=True`` →
|
||||
``main`` passes the per-root flag map to ``import_sources`` (keyed
|
||||
by the directory string — the map is built, not lost)."""
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
||||
(local_dir / ".hidden").mkdir()
|
||||
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
|
||||
)
|
||||
fake_import = FakeImportSources()
|
||||
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||
_stub_bump(monkeypatch)
|
||||
_stub_folder_summaries(monkeypatch)
|
||||
|
||||
rc = import_docs.main([])
|
||||
|
||||
assert rc == 0
|
||||
call = fake_import.calls[0]
|
||||
assert call["sources"] == [local_dir]
|
||||
assert call["include_hidden_by_root"] == {str(local_dir): True}
|
||||
assert call["ignore_by_root"] == {} # the row carries no ignore list
|
||||
assert call["prune"] is False # the CLI's no-prune default is unchanged
|
||||
|
||||
|
||||
def _stub_overview(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Stub the phase-31 overview regeneration (the CLI's real-import
|
||||
tests: the deterministic ``FakeEmbedder`` must not burn its canned
|
||||
``chat`` on the KB outline — the import is what is under test)."""
|
||||
|
||||
async def fake_overview(llm: object, session: object = None) -> bool:
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(import_docs, "regenerate_overview", fake_overview)
|
||||
|
||||
|
||||
def _kb_docs(db: Session) -> set[tuple[str, str]]:
|
||||
"""The (source, path) pairs of the ``documents`` table."""
|
||||
rows = db.execute(text("SELECT source, path FROM documents")).all()
|
||||
return {(row.source, row.path) for row in rows}
|
||||
|
||||
|
||||
def test_main_local_row_include_hidden_true_indexes_hidden_file(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
db: Session,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Phase 105 (A1): the CLI's DB-row path with the flag ON — the
|
||||
file inside the hidden folder is indexed, embedded, and counted
|
||||
like any visible file (real import over a host temp dir, the
|
||||
deterministic ``FakeEmbedder``; the regression this phase most
|
||||
plausibly breaks — the CLI's map built but lost — would leave it
|
||||
out)."""
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
||||
(local_dir / ".hidden").mkdir()
|
||||
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(local_dir), include_hidden=True)], "db"),
|
||||
)
|
||||
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
|
||||
_stub_overview(monkeypatch)
|
||||
_stub_bump(monkeypatch)
|
||||
_stub_folder_summaries(monkeypatch)
|
||||
|
||||
rc = import_docs.main([])
|
||||
|
||||
assert rc == 0
|
||||
# The hidden file has a documents row next to the visible one.
|
||||
assert _kb_docs(db) == {
|
||||
("LocalDocs", "visible.md"),
|
||||
("LocalDocs", ".hidden/note.md"),
|
||||
}
|
||||
out = capsys.readouterr().out
|
||||
assert "added=2" in out # both files were imported
|
||||
|
||||
|
||||
def test_main_local_row_include_hidden_false_skips_hidden_file(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
db: Session,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Phase 105 (A4): the same fixture with the default flag (off) —
|
||||
the hidden-folder file never enters the KB (the byte-identical
|
||||
pre-phase-105 walk): no documents row, and the summary line counts
|
||||
only the visible file."""
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
||||
(local_dir / ".hidden").mkdir()
|
||||
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(local_dir))], "db"), # flag defaults to False
|
||||
)
|
||||
monkeypatch.setattr(import_docs, "LLMClient", lambda: FakeEmbedder())
|
||||
_stub_overview(monkeypatch)
|
||||
_stub_bump(monkeypatch)
|
||||
_stub_folder_summaries(monkeypatch)
|
||||
|
||||
rc = import_docs.main([])
|
||||
|
||||
assert rc == 0
|
||||
assert _kb_docs(db) == {("LocalDocs", "visible.md")}
|
||||
out = capsys.readouterr().out
|
||||
assert "added=1" in out # the hidden file was never walked
|
||||
|
||||
|
||||
# --- main() ----------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -428,10 +605,12 @@ def test_resolve_sources_mixed_git_and_local(
|
||||
sources_dir=str(tmp_path / "bor"),
|
||||
)
|
||||
|
||||
sources, ignore_map = import_docs._resolve_sources(None, settings)
|
||||
sources, ignore_map, hidden_map = import_docs._resolve_sources(None, settings)
|
||||
|
||||
assert sources == [tmp_path / "bor" / "only", local_dir]
|
||||
assert ignore_map == {} # phase 89: neither row carries a list
|
||||
# Phase 105: both rows default-flag → per-root False entries (A4).
|
||||
assert hidden_map == {str(tmp_path / "bor" / "only"): False, str(local_dir): False}
|
||||
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
"""Integration: the phase-105 hidden-folders flag through the real import
|
||||
pipeline.
|
||||
|
||||
Phase 105 (TODO.md L3 — "…a toggle per input … to allow indexing hidden
|
||||
.folders."): ``import_sources`` gains ``include_hidden_by_root`` (same
|
||||
``str(root)`` keying as phase 89's ``ignore_by_root``). A1
|
||||
(owner-confirmed 2026-09-14): flag ON admits dot-prefixed components —
|
||||
hidden files are indexed, embedded, and summarized exactly like visible
|
||||
files — while ``EXCLUDED_DIRS`` stay excluded in both states. A2: a file
|
||||
indexed with the flag ON leaves the KB on the next ``prune=True`` run
|
||||
with the flag OFF (the untouched ``seen`` set does the work). A4: no map
|
||||
→ byte-identical to pre-phase-105. Mirrors the fixture-tree + mock-LLM
|
||||
pattern of ``test_importer_ignore.py``: a ``tmp_path`` source dir run
|
||||
through the real ``import_sources`` into the compose Postgres, with
|
||||
:class:`tests.fakes.FakeEmbedder` as the deterministic LLM stand-in.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
NAME = "HiddenFix"
|
||||
|
||||
|
||||
def _write(root: Path, rel: str, content: str) -> None:
|
||||
path = root / rel
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
def _tree(tmp_path: Path, name: str = NAME) -> Path:
|
||||
"""One visible md, a hidden dir (md + non-markdown yaml), an excluded dir."""
|
||||
root = tmp_path / name
|
||||
_write(root, "visible.md", "# Visible\n\nvisible body\n")
|
||||
_write(root, ".hidden/note.md", "# Note\n\nHIDDEN-MD-CONTENT\n")
|
||||
_write(root, ".hidden/data.yaml", "key: HIDDEN-YAML-VALUE\n")
|
||||
_write(root, ".venv/junk.md", "# Junk\n\nEXCLUDED-CONTENT\n")
|
||||
return root
|
||||
|
||||
|
||||
def _reset(db: Session) -> None:
|
||||
# House cleanup pattern (tests/integration/test_importer_ignore.py).
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_hidden_paths_not_indexed_by_default(db: Session, tmp_path: Path) -> None:
|
||||
# A4: no map → today's behavior, byte-identical — the hidden files
|
||||
# never produce a Document/Chunk row and are never embedded.
|
||||
_reset(db)
|
||||
root = _tree(tmp_path)
|
||||
llm = FakeEmbedder()
|
||||
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert summary.files == 1
|
||||
assert summary.added == 1
|
||||
assert summary.errors == 0
|
||||
assert summary.summaries == 0
|
||||
assert summary.summary_errors == 0
|
||||
|
||||
docs = db.scalars(select(Document)).all()
|
||||
assert {(d.source, d.path) for d in docs} == {(NAME, "visible.md")}
|
||||
for rel in (".hidden/note.md", ".hidden/data.yaml", ".venv/junk.md"):
|
||||
assert not any(d.path == rel for d in docs)
|
||||
for texts in llm.calls: # every embed batch
|
||||
for t in texts:
|
||||
assert "HIDDEN-MD-CONTENT" not in t and "HIDDEN-YAML-VALUE" not in t
|
||||
assert "EXCLUDED-CONTENT" not in t
|
||||
# The hidden yaml never reached the lite model.
|
||||
assert not llm.chat_calls
|
||||
_reset(db)
|
||||
|
||||
|
||||
def test_hidden_paths_indexed_when_flag_on(db: Session, tmp_path: Path) -> None:
|
||||
# A1: with the map, hidden files are indexed, embedded, and
|
||||
# summarized exactly like visible files — EXCLUDED_DIRS stay out.
|
||||
_reset(db)
|
||||
root = _tree(tmp_path)
|
||||
llm = FakeEmbedder()
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db, include_hidden_by_root={str(root): True}
|
||||
)
|
||||
)
|
||||
assert summary.files == 3
|
||||
assert summary.added == 3
|
||||
assert summary.errors == 0
|
||||
assert summary.summary_errors == 0
|
||||
|
||||
docs = db.scalars(select(Document)).all()
|
||||
assert {(d.source, d.path) for d in docs} == {
|
||||
(NAME, "visible.md"),
|
||||
(NAME, ".hidden/note.md"),
|
||||
(NAME, ".hidden/data.yaml"),
|
||||
}
|
||||
# EXCLUDED_DIRS is excluded in BOTH states (A1).
|
||||
assert not any(d.path == ".venv/junk.md" for d in docs)
|
||||
chunks = db.scalars(select(Chunk)).all()
|
||||
assert not any("EXCLUDED-CONTENT" in c.content for c in chunks)
|
||||
# The hidden md was embedded like any visible md (no summary — the
|
||||
# markdown path skips the lite model).
|
||||
note = db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
|
||||
assert note is not None and note.summary is None
|
||||
assert any("HIDDEN-MD-CONTENT" in c.content for c in chunks)
|
||||
# The hidden yaml went through the FULL non-markdown path (phase 30):
|
||||
# a stored summary plus one embedded is_summary chunk on top of the
|
||||
# content chunks.
|
||||
yaml_doc = db.scalar(select(Document).where(Document.path == ".hidden/data.yaml"))
|
||||
assert yaml_doc is not None and yaml_doc.summary is not None
|
||||
yaml_chunks = [
|
||||
c for c in chunks if c.document_id == yaml_doc.id
|
||||
]
|
||||
assert any(c.is_summary for c in yaml_chunks)
|
||||
assert any(not c.is_summary for c in yaml_chunks)
|
||||
assert any("HIDDEN-YAML-VALUE" in c.content for c in yaml_chunks)
|
||||
# Only the yaml reached the lite model.
|
||||
assert summary.summaries == 1
|
||||
assert len(llm.chat_calls) == 1
|
||||
user = next(m["content"] for m in llm.chat_calls[0] if m["role"] == "user")
|
||||
assert "HIDDEN-YAML-VALUE" in user
|
||||
_reset(db)
|
||||
|
||||
|
||||
def test_flag_off_re_run_prunes_hidden_docs(db: Session, tmp_path: Path) -> None:
|
||||
# A2: the owner flips the flag off — the next prune run walks with
|
||||
# the default rules, the hidden files never enter ``seen``, and their
|
||||
# rows (summary chunk included) leave the index.
|
||||
_reset(db)
|
||||
root = _tree(tmp_path)
|
||||
llm = FakeEmbedder()
|
||||
|
||||
s1 = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db, include_hidden_by_root={str(root): True}
|
||||
)
|
||||
)
|
||||
assert (s1.files, s1.added) == (3, 3)
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
|
||||
is not None
|
||||
)
|
||||
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db, prune=True))
|
||||
assert s2.files == 1
|
||||
assert s2.unchanged == 1 # visible.md untouched
|
||||
assert s2.pruned == 2 # both hidden documents
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.path == ".hidden/note.md"))
|
||||
is None
|
||||
)
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.path == ".hidden/data.yaml"))
|
||||
is None
|
||||
)
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.path == "visible.md")) is not None
|
||||
)
|
||||
# The summary chunk rows went with their documents (cascade).
|
||||
assert not any(
|
||||
"HIDDEN-YAML-VALUE" in c.content for c in db.scalars(select(Chunk)).all()
|
||||
)
|
||||
_reset(db)
|
||||
|
||||
|
||||
def test_progress_total_agrees_with_walk_in_both_states(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
# The phase-64 pre-walk uses the same per-root flag as the loop, so
|
||||
# ``total`` agrees with the walk in both states.
|
||||
_reset(db)
|
||||
root = _tree(tmp_path)
|
||||
llm = FakeEmbedder()
|
||||
on_totals: set[int] = set()
|
||||
|
||||
def on_progress(source: str, rel: str, done: int, total: int) -> None:
|
||||
on_totals.add(total)
|
||||
|
||||
s_on = asyncio.run(
|
||||
import_sources(
|
||||
[root],
|
||||
llm,
|
||||
session=db,
|
||||
progress=on_progress,
|
||||
include_hidden_by_root={str(root): True},
|
||||
)
|
||||
)
|
||||
assert s_on.files == 3
|
||||
assert on_totals == {3} # hidden files counted in the denominator
|
||||
|
||||
_reset(db)
|
||||
llm2 = FakeEmbedder()
|
||||
off_totals: set[int] = set()
|
||||
|
||||
def off_progress(source: str, rel: str, done: int, total: int) -> None:
|
||||
off_totals.add(total)
|
||||
|
||||
s_off = asyncio.run(
|
||||
import_sources([root], llm2, session=db, progress=off_progress)
|
||||
)
|
||||
assert s_off.files == 1
|
||||
assert off_totals == {1} # visible only, the pre-phase-105 count
|
||||
_reset(db)
|
||||
|
||||
|
||||
def test_unlisted_root_stays_hidden(db: Session, tmp_path: Path) -> None:
|
||||
# The map is per-root, not global: listing one root as True leaves
|
||||
# the other root exactly as pre-phase-105.
|
||||
_reset(db)
|
||||
root_a = _tree(tmp_path, name="HiddenFixA")
|
||||
root_b = _tree(tmp_path, name="HiddenFixB")
|
||||
llm = FakeEmbedder()
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root_a, root_b],
|
||||
llm,
|
||||
session=db,
|
||||
include_hidden_by_root={str(root_a): True},
|
||||
)
|
||||
)
|
||||
# A: 3 (flag on) · B: 1 (unlisted → False)
|
||||
assert summary.files == 4
|
||||
docs = db.scalars(select(Document)).all()
|
||||
assert {(d.source, d.path) for d in docs} == {
|
||||
("HiddenFixA", "visible.md"),
|
||||
("HiddenFixA", ".hidden/note.md"),
|
||||
("HiddenFixA", ".hidden/data.yaml"),
|
||||
("HiddenFixB", "visible.md"),
|
||||
}
|
||||
assert not any(
|
||||
d.source == "HiddenFixB" and d.path == ".hidden/note.md" for d in docs
|
||||
)
|
||||
_reset(db)
|
||||
@@ -0,0 +1,271 @@
|
||||
"""Integration: migration 0019 (git_sources.include_hidden) schema
|
||||
contract (phase 105, task 01).
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the house pattern of
|
||||
``test_migration_0007.py`` (git_sources column contract — information_schema
|
||||
assertions on the state the migration must leave). The tests target the
|
||||
0018 → 0019 step explicitly so later migrations cannot break them:
|
||||
|
||||
* upgrade 0018 → 0019 → the ``include_hidden`` column exists with the
|
||||
full contract — BOOLEAN NOT NULL, server default ``false`` — while
|
||||
the 0018 ``git_sources`` schema (url, kind, path, ignore_paths)
|
||||
survives;
|
||||
* pre-0019 rows backfill ``false`` and a row written without the
|
||||
column takes the server default (A4 — byte-identical import
|
||||
behavior until the owner flips the flag);
|
||||
* the ORM contract agrees: a freshly inserted ``GitSource`` (no flag
|
||||
passed) reads ``include_hidden is False`` (the Python ``default=False``
|
||||
and the server default agree), and an explicit ``True`` round-trips
|
||||
through a fresh session;
|
||||
* downgrade to 0018 → the column is GONE (A13 — reversible) while the
|
||||
rows + their ignore lists survive;
|
||||
* upgrade back to 0019 → the column is back (round-trip).
|
||||
|
||||
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||
fails or the process is interrupted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import GitSource
|
||||
|
||||
URL_BASE = "https://git.example.com/mig0019"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run); teardown upgrades
|
||||
to head no matter what happened, so the dev DB is never left below
|
||||
head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
|
||||
# on ``git_sources``, which would deadlock the repair's
|
||||
# ``ALTER TABLE`` (0019) forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||
"""(data_type, is_nullable, column_default) for one git_sources column."""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT data_type, is_nullable, column_default"
|
||||
" FROM information_schema.columns"
|
||||
" WHERE table_name = 'git_sources' AND column_name = :c"
|
||||
),
|
||||
{"c": column},
|
||||
).fetchone()
|
||||
return tuple(row) if row is not None else None
|
||||
|
||||
|
||||
def _flag(db: Session, url: str) -> Any:
|
||||
return db.execute(
|
||||
text("SELECT include_hidden FROM git_sources WHERE url = :u"),
|
||||
{"u": url},
|
||||
).scalar_one()
|
||||
|
||||
|
||||
def _insert_sql(db: Session, url: str, include_hidden: Any = None) -> None:
|
||||
"""Insert one git_sources row (kind/path omitted → the API default
|
||||
shape); ``include_hidden`` omitted → pre-0019 insert shape."""
|
||||
if include_hidden is None:
|
||||
db.execute(
|
||||
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
|
||||
{"u": url},
|
||||
)
|
||||
else:
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO git_sources (id, url, include_hidden)"
|
||||
" VALUES (gen_random_uuid(), :u, :f)"
|
||||
),
|
||||
{"u": url, "f": include_hidden},
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _delete_by_url(db: Session, url: str) -> None:
|
||||
db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url})
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_upgrade_to_0019_adds_include_hidden(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0018 → 0019: the column exists with the full contract
|
||||
(BOOLEAN NOT NULL, server default ``false``), is ABSENT at 0018,
|
||||
pre-0019 rows backfill ``false`` (A4), a new row without the column
|
||||
takes the default, and an explicit ``true`` round-trips — while the
|
||||
0018 table contract survives."""
|
||||
command.downgrade(alembic, "0018") # start from the pre-0019 state
|
||||
assert _version(db) == "0018"
|
||||
assert _column(db, "include_hidden") is None, (
|
||||
"the flag must be absent at 0018"
|
||||
)
|
||||
|
||||
url_pre = f"{URL_BASE}/pre-existing.git"
|
||||
_insert_sql(db, url_pre) # no include_hidden column exists at 0018
|
||||
try:
|
||||
command.upgrade(alembic, "0019")
|
||||
assert _version(db) == "0019", "alembic_version must be at 0019"
|
||||
|
||||
flag = _column(db, "include_hidden")
|
||||
assert flag is not None, "git_sources.include_hidden is missing"
|
||||
assert flag[0] == "boolean", "include_hidden must be BOOLEAN"
|
||||
assert flag[1] == "NO", "include_hidden must be NOT NULL"
|
||||
assert flag[2] is not None and "false" in str(flag[2]), (
|
||||
"include_hidden must carry the `false` server default"
|
||||
)
|
||||
|
||||
# The pre-0019 row backfilled ``false`` (A4 — unchanged imports).
|
||||
assert _flag(db, url_pre) is False
|
||||
|
||||
# A row written without the column takes the server default.
|
||||
url_new = f"{URL_BASE}/new-row.git"
|
||||
_insert_sql(db, url_new)
|
||||
try:
|
||||
assert _flag(db, url_new) is False, (
|
||||
"an omitted flag takes the `false` server default"
|
||||
)
|
||||
|
||||
# The flag round-trips through an explicit ``true``.
|
||||
db.execute(
|
||||
text("UPDATE git_sources SET include_hidden = true WHERE url = :u"),
|
||||
{"u": url_new},
|
||||
)
|
||||
db.commit()
|
||||
assert _flag(db, url_new) is True, (
|
||||
"include_hidden = true must round-trip"
|
||||
)
|
||||
finally:
|
||||
_delete_by_url(db, url_new)
|
||||
|
||||
# The 0018 schema survives the additive upgrade.
|
||||
ignore = _column(db, "ignore_paths")
|
||||
assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", (
|
||||
"git_sources.ignore_paths (0013) must survive the upgrade"
|
||||
)
|
||||
kind = _column(db, "kind")
|
||||
assert kind is not None and kind[0] == "text" and kind[1] == "NO", (
|
||||
"git_sources.kind (0007) must survive the upgrade"
|
||||
)
|
||||
finally:
|
||||
_delete_by_url(db, url_pre)
|
||||
|
||||
|
||||
def test_orm_fresh_row_defaults_false_and_true_round_trips(
|
||||
db: Session, alembic: Config
|
||||
) -> None:
|
||||
"""The ORM contract agrees with the column contract: a freshly
|
||||
inserted ``GitSource`` (no flag passed) reads ``include_hidden is
|
||||
False`` — the Python ``default=False`` and the server default
|
||||
agree (A4) — and an explicit ``True`` round-trips through a fresh
|
||||
session."""
|
||||
command.upgrade(alembic, "head")
|
||||
url_off = f"{URL_BASE}/orm-default.git"
|
||||
url_on = f"{URL_BASE}/orm-true.git"
|
||||
try:
|
||||
# Fresh row, flag omitted → False (the Python-side default).
|
||||
row_off = GitSource(url=url_off, kind="git")
|
||||
db.add(row_off)
|
||||
db.commit()
|
||||
db.expire_all()
|
||||
reloaded_off = db.get(GitSource, row_off.id)
|
||||
assert reloaded_off is not None, "the fresh row must be readable"
|
||||
assert reloaded_off.include_hidden is False, (
|
||||
"a fresh row must read include_hidden is False (A4)"
|
||||
)
|
||||
|
||||
# Explicit True round-trips through a FRESH session.
|
||||
row_on = GitSource(url=url_on, kind="git", include_hidden=True)
|
||||
db.add(row_on)
|
||||
db.commit()
|
||||
with SessionLocal() as fresh:
|
||||
reloaded = fresh.get(GitSource, row_on.id)
|
||||
assert reloaded is not None, "the row must exist in a fresh session"
|
||||
assert reloaded.include_hidden is True, (
|
||||
"include_hidden=True must round-trip through the DB"
|
||||
)
|
||||
finally:
|
||||
_delete_by_url(db, url_off)
|
||||
_delete_by_url(db, url_on)
|
||||
|
||||
|
||||
def test_downgrade_to_0018_drops_the_column(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade 0019 → 0018: the column is gone (A13 — fully
|
||||
reversible) while the rows + their ignore lists survive, and the
|
||||
rest of the 0018 table contract (``path``) is intact."""
|
||||
command.upgrade(alembic, "head")
|
||||
url = f"{URL_BASE}/survivor.git"
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO git_sources (id, url, include_hidden)"
|
||||
" VALUES (:i, :u, true)"
|
||||
),
|
||||
{"i": uuid.uuid4(), "u": url},
|
||||
)
|
||||
db.commit()
|
||||
try:
|
||||
command.downgrade(alembic, "0018")
|
||||
assert _version(db) == "0018"
|
||||
assert _column(db, "include_hidden") is None, (
|
||||
"the flag must be dropped"
|
||||
)
|
||||
row = db.execute(
|
||||
text("SELECT url, kind, ignore_paths FROM git_sources WHERE url = :u"),
|
||||
{"u": url},
|
||||
).fetchone()
|
||||
assert row is not None and row[0] == url, (
|
||||
"the row must survive the column drop"
|
||||
)
|
||||
assert row[1] == "git" and row[2] == [], (
|
||||
"kind + the ignore list must survive the column drop"
|
||||
)
|
||||
|
||||
doc_path = _column(db, "path")
|
||||
assert doc_path is not None and doc_path[0] == "text", (
|
||||
"git_sources.path must survive the downgrade"
|
||||
)
|
||||
finally:
|
||||
_delete_by_url(db, url)
|
||||
# Repair: the fixture teardown re-upgrades to head.
|
||||
|
||||
|
||||
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0018, then upgrade back to 0019: the column is back
|
||||
with the full contract (BOOLEAN NOT NULL, the `false` default)."""
|
||||
command.downgrade(alembic, "0018")
|
||||
command.upgrade(alembic, "0019")
|
||||
assert _version(db) == "0019", "round-trip upgrade must land at 0019"
|
||||
|
||||
flag = _column(db, "include_hidden")
|
||||
assert flag is not None, "git_sources.include_hidden must be back"
|
||||
assert flag[0] == "boolean", "include_hidden must be BOOLEAN after the round-trip"
|
||||
assert flag[1] == "NO", "include_hidden must be NOT NULL after the round-trip"
|
||||
assert flag[2] is not None and "false" in str(flag[2]), (
|
||||
"the `false` server default must survive the round-trip"
|
||||
)
|
||||
@@ -77,7 +77,7 @@ from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api import sync as sync_api
|
||||
@@ -262,6 +262,11 @@ class FakeImportSources:
|
||||
# rows' ``ignore_paths`` (keyed by the root string the importer
|
||||
# sees; two rows sharing a root string get the union).
|
||||
self.ignore_maps: list[dict[str, list[str]]] = []
|
||||
# Phase 105: the per-root hidden-folders flag map the runner
|
||||
# builds from the rows' ``include_hidden`` (same root-string
|
||||
# keying; a shared-root collision ORs — if either row says
|
||||
# "index hidden", the root does).
|
||||
self.include_hidden_maps: list[dict[str, bool]] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -273,12 +278,14 @@ class FakeImportSources:
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
) -> ImportSummary:
|
||||
self.sources.append(list(sources))
|
||||
self.llms.append(llm)
|
||||
self.prune_flags.append(prune)
|
||||
self.progress_hooks.append(progress)
|
||||
self.ignore_maps.append(ignore_by_root or {})
|
||||
self.include_hidden_maps.append(include_hidden_by_root or {})
|
||||
if self.delay:
|
||||
await asyncio.sleep(self.delay)
|
||||
return self.summary
|
||||
@@ -823,6 +830,7 @@ def test_import_error_is_reported_with_credentials_masked(
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
) -> ImportSummary:
|
||||
raise EmbeddingError(
|
||||
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
||||
@@ -1100,3 +1108,114 @@ def test_sync_without_ignore_lists_passes_empty_map(
|
||||
|
||||
assert fake_import.sources == [[local_dir]]
|
||||
assert fake_import.ignore_maps == [{}] # no row carried a list
|
||||
|
||||
|
||||
# --- phase 105: per-row hidden-folders flag --------------------------------
|
||||
|
||||
|
||||
def test_local_row_hidden_flag_off_then_on_indexes_hidden_paths(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Phase 105 (A4 then A1): the SAME local row, synced first with the
|
||||
default flag (off) — the file inside the hidden folder never lands
|
||||
in the KB (no document row — hence no embedding, no summary), the
|
||||
visible file imports as usual; then the row is flipped on (direct
|
||||
model set — the PATCH round-trip is task 03's layer) and the next
|
||||
sync walks the hidden file too: it is indexed, embedded, and counted
|
||||
like any visible file."""
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "visible.md").write_text("# Visible\nin scope\n", encoding="utf-8")
|
||||
(local_dir / ".hidden").mkdir()
|
||||
(local_dir / ".hidden" / "note.md").write_text("# Note\nhidden\n", encoding="utf-8")
|
||||
_seed_local(db, local_dir) # include_hidden defaults to False (A4)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# Flag off (A4): only the visible file is walked and indexed — the
|
||||
# hidden file has NO documents row.
|
||||
assert body["detail"]["files"] == 1
|
||||
assert body["detail"]["added"] == 1
|
||||
assert body["detail"]["errors"] == 0
|
||||
docs = {
|
||||
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
|
||||
}
|
||||
assert docs == {("LocalDocs", "visible.md")}
|
||||
|
||||
# Flip the SAME row on — the next sync re-reads the flag per row.
|
||||
row = db.execute(select(GitSource).where(GitSource.url == str(local_dir))).scalar_one()
|
||||
row.include_hidden = True
|
||||
db.commit()
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# Flag on (A1): the hidden file is walked (detail.files counts it),
|
||||
# embedded, and indexed alongside the visible file.
|
||||
assert body["detail"]["files"] == 2
|
||||
assert body["detail"]["added"] == 1
|
||||
assert body["detail"]["errors"] == 0
|
||||
docs = {
|
||||
(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]
|
||||
}
|
||||
assert docs == {("LocalDocs", "visible.md"), ("LocalDocs", ".hidden/note.md")}
|
||||
|
||||
|
||||
def test_sync_builds_include_hidden_map_by_root_string_with_or(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 105 wiring (fake import): the runner keys the flag map by
|
||||
the SAME root string the importer sees, and two rows sharing that
|
||||
root string (the sibling/repo-name edge — ``…/shared`` and
|
||||
``…/shared.git`` clone into the same checkout dir) get the OR of
|
||||
their flags — if EITHER row says "index hidden", the root does
|
||||
(the ignore-map union's boolean mirror)."""
|
||||
url_a = f"file://{tmp_path / 'shared'}"
|
||||
url_b = f"{url_a}.git" # same repo name → same checkout dir
|
||||
# Distinct added_at: the resolver orders by (added_at, id) — a
|
||||
# same-timestamp pair would tie-break on the random uuid.
|
||||
db.add(GitSource(url=url_a, kind="git", include_hidden=False,
|
||||
added_at=datetime(2026, 1, 1, tzinfo=UTC)))
|
||||
db.add(GitSource(url=url_b, kind="git", include_hidden=True,
|
||||
added_at=datetime(2026, 1, 2, tzinfo=UTC)))
|
||||
db.commit()
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
_poll(sync_client, "success")
|
||||
|
||||
shared = str(tmp_path / "bor" / "shared")
|
||||
# Both rows resolve to the SAME checkout (the collision itself) and
|
||||
# the map holds the OR of their flags, keyed by that one root
|
||||
# string — the flag-off row's False is overridden by the True.
|
||||
assert fake_import.sources == [[tmp_path / "bor" / "shared", tmp_path / "bor" / "shared"]]
|
||||
assert fake_import.include_hidden_maps == [{shared: True}]
|
||||
|
||||
@@ -0,0 +1,436 @@
|
||||
"""Unit: the per-row "Hidden" toggle on the Sources page (phase 105,
|
||||
task 05).
|
||||
|
||||
``TODO.md`` L3: "There should be a toggle per input (next to the
|
||||
ignores button) to allow indexing hidden .folders." Every STORED
|
||||
``git_sources`` row (git or local) gets a native labeled **Hidden**
|
||||
checkbox in the actions cell, LEFT of the "Ignore paths" button (DOM
|
||||
order Hidden · Ignore paths · Remove — A5); when on, the source cell
|
||||
shows a "hidden on" text tag (text + background, never color alone —
|
||||
WCAG 1.4.1); the flip PATCHes ``{"include_hidden": …}`` with the
|
||||
§7.4 never-stale lifecycle (box disables at once; 200 → reload +
|
||||
confirm LAST; failure → the box reverts to the SERVER state and the
|
||||
detail lands in the page-level ``#git-sources-hidden-error``
|
||||
``role="alert"`` line — the checkbox is a table-cell control with no
|
||||
dialog of its own). Env-fallback rows (``id`` null) get no checkbox
|
||||
(A3 — nothing is stored to flag).
|
||||
|
||||
The browser behavior itself is E2E-gated by the phase-105 story suite
|
||||
(``tests/e2e/test_hidden_folders_toggle.py``, task 06); like the
|
||||
phase-89 ``test_source_ignore_paths.py`` house pattern, this module
|
||||
pins the source-level contract a silent regression would break:
|
||||
|
||||
* the JS row wiring — ``makeRow`` builds the checkbox ONLY in the
|
||||
``s.id`` branch (class ``git-source-hidden-box``, ``type=checkbox``,
|
||||
``checked`` from ``s.include_hidden === true`` — server state only,
|
||||
never a prior local flip), the aria-label is the ONLY place the
|
||||
value appears (setAttribute — never innerHTML), the visible
|
||||
"Hidden" text label, and the append ORDER (hidden label →
|
||||
ignore button → Remove); the "hidden on" tag is TEXT in the source
|
||||
cell, iff ``s.id && s.include_hidden === true``;
|
||||
* the JS lifecycle — ``toggleHidden``: the box disables BEFORE the
|
||||
``PATCH`` (one flip at a time), the body is
|
||||
``JSON.stringify({ include_hidden: wanted })`` ONLY (the row's
|
||||
ignore list is untouched — task 03's optional field), 200 →
|
||||
clear the error → ``await loadSources()`` → announce (the
|
||||
confirmation is the LAST announcement); non-2xx AND network
|
||||
failure → the box REVERTS to ``s.include_hidden === true`` +
|
||||
re-enables and the detail lands in the page-level alert line;
|
||||
a healed ``loadSources`` clears the line (the phase-89 "happy
|
||||
path heals the error state" precedent);
|
||||
* the cross-file wire contract (SINGLE SOURCE OF TRUTH for the
|
||||
field name) — the JS PATCH body key and the render field both
|
||||
match their ``app/schemas.py`` counterparts
|
||||
(``GitSourcePatchIn.include_hidden`` / ``GitSourceRow``) — a
|
||||
rename on either side breaks the test;
|
||||
* the static shell markup — ``#git-sources-hidden-error`` is a
|
||||
``role="alert"`` line, hidden by default, reusing the
|
||||
``.git-source-error`` class, INSIDE ``#view-git-sources`` and
|
||||
AFTER ``#git-sources-table-wrap`` in source order;
|
||||
* styles.css — the ``.git-source-hidden`` family on the house
|
||||
palette: ~44px hit height, the checkbox's ``accent-color:
|
||||
var(--brand)`` (checked-state contrast verified + recorded in the
|
||||
comment), the ``:disabled`` wait state, and the "hidden on" tag
|
||||
(text + a distinct background, never color alone).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
FRONTEND = ROOT / "frontend"
|
||||
# Phase 76 (task 02): the Sources view lives in the ONE-document shell.
|
||||
SHELL_HTML = FRONTEND / "index.html"
|
||||
JS = FRONTEND / "assets" / "git-sources.js"
|
||||
CSS = FRONTEND / "assets" / "styles.css"
|
||||
SCHEMAS = ROOT / "app" / "schemas.py"
|
||||
|
||||
#: The checkbox + tag classes (pinned verbatim).
|
||||
HIDDEN_BOX_CLASS = "git-source-hidden-box"
|
||||
HIDDEN_LABEL_CLASS = "git-source-hidden"
|
||||
HIDDEN_TAG_CLASS = "git-source-hidden-count"
|
||||
HIDDEN_TAG_TEXT = "hidden on"
|
||||
|
||||
#: The checkbox's aria-label template — the ONLY place the full
|
||||
#: source value appears (setAttribute, never innerHTML).
|
||||
ARIA_LABEL_TEMPLATE = "`Index hidden folders for ${kindLabel} source: ${value}`"
|
||||
|
||||
#: The §7.4 lifecycle strings (pinned verbatim).
|
||||
REVERT_STATE = "box.checked = s.include_hidden === true"
|
||||
NETWORK_MESSAGE = "Could not reach the server — the setting was not changed."
|
||||
PATCH_BODY = "JSON.stringify({ include_hidden: wanted })"
|
||||
|
||||
|
||||
def _text(path: Path) -> str:
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return _text(JS)
|
||||
|
||||
|
||||
def _css() -> str:
|
||||
return _text(CSS)
|
||||
|
||||
|
||||
def _fn(js: str, name: str) -> str:
|
||||
"""The source of a (possibly async) top-level function via
|
||||
balanced-brace counting (the test_source_ignore_paths.py helper)."""
|
||||
for prefix in ("async function ", "function "):
|
||||
start = js.find(f"{prefix}{name}(")
|
||||
if start != -1:
|
||||
depth = 0
|
||||
for i in range(js.find("{", start), len(js)):
|
||||
if js[i] == "{":
|
||||
depth += 1
|
||||
elif js[i] == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return js[start : i + 1]
|
||||
raise AssertionError(f"unbalanced braces in {name}()")
|
||||
raise AssertionError(f"{name}() must exist in git-sources.js")
|
||||
|
||||
|
||||
def _css_rule(css: str, selector: str) -> str:
|
||||
"""The declarations of a simple one-line-opening rule (comments
|
||||
stripped first — a house comment may legally carry braces)."""
|
||||
clean = re.sub(r"/\*.*?\*/", "", css, flags=re.S)
|
||||
start = clean.find(f"{selector} {{")
|
||||
assert start != -1, f"missing rule {selector} in styles.css"
|
||||
depth = 0
|
||||
for i in range(clean.find("{", start), len(clean)):
|
||||
if clean[i] == "{":
|
||||
depth += 1
|
||||
elif clean[i] == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return clean[start : i + 1]
|
||||
raise AssertionError(f"unbalanced braces in {selector}")
|
||||
|
||||
|
||||
def _python_class_field(class_name: str) -> str:
|
||||
"""The single bool field name declared on a Pydantic class in
|
||||
``app/schemas.py`` (``bool`` or ``bool | None`` — regex-parsed,
|
||||
no import: the pin is on the source text, so a rename breaks it
|
||||
without a schema load)."""
|
||||
schemas = _text(SCHEMAS)
|
||||
m = re.search(
|
||||
rf"class {class_name}\(BaseModel\):.*?(?=\nclass |\Z)", schemas, re.S
|
||||
)
|
||||
assert m, f"class {class_name} must exist in app/schemas.py"
|
||||
fields = re.findall(
|
||||
r"^\s*(\w+):\s*bool(?:\s*\|\s*None)?\s*(?==|$)", m.group(0), re.M
|
||||
)
|
||||
assert len(fields) == 1, f"{class_name} must declare exactly one bool field"
|
||||
return fields[0]
|
||||
|
||||
|
||||
# ---------- the row wiring in makeRow ----------
|
||||
|
||||
|
||||
def test_row_checkbox_is_built_in_the_stored_row_branch() -> None:
|
||||
"""makeRow: the Hidden checkbox is created ONLY in the ``s.id``
|
||||
branch (A3 — env-fallback rows get the "from .env" tag, no
|
||||
checkbox). It is a NATIVE labeled checkbox: class
|
||||
git-source-hidden-box, type=checkbox, the visible "Hidden" text,
|
||||
``checked`` from the SERVER row (``s.include_hidden === true`` —
|
||||
never a prior local flip, §7.4), and the aria-label template is
|
||||
the ONLY place ``value`` appears (setAttribute — never
|
||||
innerHTML). Its change handler runs toggleHidden."""
|
||||
make = _fn(_js(), "makeRow")
|
||||
branch_i = make.find("if (s.id) {")
|
||||
else_i = make.find("} else {", branch_i)
|
||||
assert branch_i != -1 and else_i > branch_i
|
||||
box_i = make.find('const hiddenBox = document.createElement("input")', branch_i)
|
||||
assert -1 < box_i < else_i, "the checkbox is created in the s.id branch (A3)"
|
||||
assert 'hiddenBox.type = "checkbox"' in make, "a native checkbox"
|
||||
assert f"hiddenBox.className = \"{HIDDEN_BOX_CLASS}\"" in make
|
||||
assert "hiddenBox.checked = s.include_hidden === true" in make, (
|
||||
"checked state comes ONLY from the server row"
|
||||
)
|
||||
assert ARIA_LABEL_TEMPLATE in make, "the aria-label template (the only value site)"
|
||||
assert 'hiddenBox.setAttribute(' in make, "the label is set via setAttribute"
|
||||
assert "hiddenBox.innerHTML" not in _js(), "XSS contract: no innerHTML on the box"
|
||||
assert "hiddenLabel.innerHTML" not in _js(), "XSS contract: no innerHTML on the label"
|
||||
assert 'document.createTextNode("Hidden")' in make, (
|
||||
"the visible text label — never aria-label-only (WCAG)"
|
||||
)
|
||||
assert "hiddenLabel.append(hiddenBox, document.createTextNode(\"Hidden\"))" in make
|
||||
assert 'hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox))' in make
|
||||
# The env-fallback branch (else) has no checkbox.
|
||||
else_slice = make[else_i : make.find("tr.appendChild(actTd)", else_i)]
|
||||
assert HIDDEN_BOX_CLASS not in else_slice and "toggleHidden" not in else_slice, (
|
||||
"env-fallback rows get no checkbox (A3)"
|
||||
)
|
||||
|
||||
|
||||
def test_row_dom_order_is_hidden_ignore_remove() -> None:
|
||||
"""makeRow: the append order in the actions cell is Hidden ·
|
||||
Ignore paths · Remove (A5 — the toggle sits LEFT of the "Ignore
|
||||
paths" button; Remove stays last)."""
|
||||
make = _fn(_js(), "makeRow")
|
||||
branch_i = make.find("if (s.id) {")
|
||||
else_i = make.find("} else {", branch_i)
|
||||
hidden_append = make.find("actTd.appendChild(hiddenLabel)", branch_i)
|
||||
ignore_i = make.find('const ignoreBtn = document.createElement("button")', branch_i)
|
||||
ignore_append = make.find("actTd.appendChild(ignoreBtn)", branch_i)
|
||||
remove_append = make.find("actTd.appendChild(btn)", branch_i)
|
||||
assert -1 < hidden_append < ignore_i, "the hidden label precedes the ignore button"
|
||||
assert -1 < ignore_append < remove_append, "Remove stays last"
|
||||
assert remove_append < else_i, "all three controls are in the s.id branch"
|
||||
|
||||
|
||||
def test_hidden_on_tag_renders_text_in_the_source_cell() -> None:
|
||||
"""makeRow: a stored row with the flag ON gets the "hidden on"
|
||||
tag in the SOURCE cell (urlTd — the .git-source-ignore-count
|
||||
idiom: TEXT, never color alone, WCAG 1.4.1), appended after the
|
||||
location <code>; the tag is the state copy (pinned verbatim)."""
|
||||
make = _fn(_js(), "makeRow")
|
||||
assert "s.id && s.include_hidden === true" in make, (
|
||||
"the tag only for stored rows with the flag on"
|
||||
)
|
||||
assert f"hiddenTag.className = \"{HIDDEN_TAG_CLASS}\"" in make
|
||||
assert f"hiddenTag.textContent = \"{HIDDEN_TAG_TEXT}\"" in make, (
|
||||
"the tag copy is TEXT — never color alone"
|
||||
)
|
||||
code_i = make.find("urlTd.append(badge, code)")
|
||||
append_i = make.find("urlTd.append(hiddenTag)", code_i)
|
||||
assert -1 < code_i < append_i, "the tag is appended to the source cell"
|
||||
|
||||
|
||||
# ---------- the JS lifecycle (toggleHidden) ----------
|
||||
|
||||
|
||||
def test_toggle_sends_patch_and_disables_the_box_first() -> None:
|
||||
"""toggleHidden: the box disables BEFORE the PATCH goes out
|
||||
(no double-flip — §7.4); the request is a PATCH to
|
||||
/api/git-sources/{id} whose body is ``{"include_hidden": …}``
|
||||
ONLY (the row's ignore list is untouched — task 03's optional
|
||||
field)."""
|
||||
body = _fn(_js(), "toggleHidden")
|
||||
disable_i = body.find("box.disabled = true")
|
||||
fetch_i = body.find("`/api/git-sources/${s.id}`")
|
||||
method_i = body.find('method: "PATCH"', fetch_i)
|
||||
body_i = body.find(PATCH_BODY, method_i)
|
||||
assert -1 < disable_i < fetch_i < method_i < body_i, (
|
||||
"disable → PATCH {include_hidden} (and nothing else)"
|
||||
)
|
||||
assert "ignore_paths" not in body, "the PATCH body never carries the ignore list"
|
||||
|
||||
|
||||
def test_toggle_success_clears_reloads_and_announces_last() -> None:
|
||||
"""toggleHidden 200: the error line clears, ``loadSources()``
|
||||
AWAITS (the row re-renders from the server — the "hidden on" tag
|
||||
lands), and the confirmation is the LAST announcement (the
|
||||
reload's "N sources listed." lands first — the phase-89 order)."""
|
||||
body = _fn(_js(), "toggleHidden")
|
||||
ok_i = body.find("if (r.ok)")
|
||||
hide_i = body.find("hideHiddenError()", ok_i)
|
||||
reload_i = body.find("await loadSources()", ok_i)
|
||||
announce_i = body.find("announce(`Hidden folders", reload_i)
|
||||
assert -1 < ok_i < hide_i < reload_i < announce_i, (
|
||||
"200: clear error → await loadSources → announce (LAST)"
|
||||
)
|
||||
assert "for ${value}.`)" in body[announce_i:], "the confirmation names the source"
|
||||
assert "enabled" in body[announce_i:] and "disabled" in body[announce_i:], (
|
||||
"the confirmation states the direction of the flip"
|
||||
)
|
||||
|
||||
|
||||
def test_toggle_failure_reverts_the_box_and_shows_the_page_alert() -> None:
|
||||
"""toggleHidden failure: non-2xx → the server detail (apiDetail)
|
||||
into the page-level alert line + the box REVERTS to the server
|
||||
state + re-enables; network failure → the fixed reachable? line,
|
||||
same revert. The UI never claims a state the server didn't save
|
||||
(PLAN §7.4). The revert + re-enable happen on BOTH failure
|
||||
branches (exactly twice each in the function)."""
|
||||
body = _fn(_js(), "toggleHidden")
|
||||
catch_i = body.find(".catch(")
|
||||
assert catch_i != -1
|
||||
fail_slice = body[body.find("if (r.ok)"):catch_i]
|
||||
assert "await apiDetail(" in fail_slice, "the server detail is apiDetail-extracted"
|
||||
assert "showHiddenError(" in fail_slice, "the detail lands in the page alert line"
|
||||
assert REVERT_STATE in fail_slice, "the box reverts to the server state"
|
||||
assert "box.disabled = false" in fail_slice, "the box re-enables"
|
||||
net_slice = body[catch_i:]
|
||||
assert NETWORK_MESSAGE in net_slice, "the fixed network copy"
|
||||
assert "showHiddenError(" in net_slice
|
||||
assert REVERT_STATE in net_slice
|
||||
assert "box.disabled = false" in net_slice
|
||||
assert body.count(REVERT_STATE) == 2, "revert on BOTH failure branches"
|
||||
assert body.count("box.disabled = false") == 2, "re-enable on BOTH failure branches"
|
||||
assert "box.disabled = true" in body[: body.find("if (r.ok)")], (
|
||||
"the box disables at once, before any outcome"
|
||||
)
|
||||
|
||||
|
||||
def test_error_line_helpers_and_the_healed_load_clear() -> None:
|
||||
"""showHiddenError/hideHiddenError drive the page-level
|
||||
``hiddenErrorEl`` (textContent + hidden); ``loadSources``'s
|
||||
success path calls hideHiddenError() AFTER hideLoadError() — a
|
||||
healed list clears the stale line (the phase-89 "happy path
|
||||
heals the error state" precedent)."""
|
||||
js = _js()
|
||||
assert 'const hiddenErrorEl = root.querySelector("#git-sources-hidden-error")' in js, (
|
||||
"the page-local element grabber (root-scoped, the shell idiom)"
|
||||
)
|
||||
show = _fn(js, "showHiddenError")
|
||||
assert "hiddenErrorEl.textContent = message" in show
|
||||
assert "hiddenErrorEl.hidden = false" in show
|
||||
hide = _fn(js, "hideHiddenError")
|
||||
assert 'hiddenErrorEl.textContent = ""' in hide
|
||||
assert "hiddenErrorEl.hidden = true" in hide
|
||||
load = _fn(js, "loadSources")
|
||||
hide_load_i = load.find("hideLoadError()")
|
||||
hide_hidden_i = load.find("hideHiddenError()", hide_load_i)
|
||||
assert -1 < hide_load_i < hide_hidden_i, (
|
||||
"a healed load clears the hidden-toggle line (after the load-error clear)"
|
||||
)
|
||||
|
||||
|
||||
# ---------- the cross-file wire contract (single source of truth) ----------
|
||||
|
||||
|
||||
def test_js_field_name_matches_the_python_patch_schema() -> None:
|
||||
"""The cross-file pin: the JS PATCH body key and the render field
|
||||
both equal their ``app/schemas.py`` counterparts —
|
||||
``GitSourcePatchIn.include_hidden`` (the write side) and
|
||||
``GitSourceRow.include_hidden`` (the read side). A rename on
|
||||
either side breaks the wire contract and this test."""
|
||||
js = _js()
|
||||
body = _fn(js, "toggleHidden")
|
||||
m = re.search(r"JSON\.stringify\(\{\s*(\w+)\s*:\s*wanted\s*\}\)", body)
|
||||
assert m, "the PATCH body carries a single {<field>: wanted} object"
|
||||
js_body_key = m.group(1)
|
||||
assert js_body_key == _python_class_field("GitSourcePatchIn"), (
|
||||
"the JS PATCH body key must match GitSourcePatchIn's field"
|
||||
)
|
||||
make = _fn(js, "makeRow")
|
||||
rm = re.search(r"s\.(\w+)\s*===\s*true", make)
|
||||
assert rm, "makeRow reads the flag off the server row (s.<field> === true)"
|
||||
js_read_key = rm.group(1)
|
||||
assert js_read_key == _python_class_field("GitSourceRow"), (
|
||||
"the render field must match GitSourceRow's field"
|
||||
)
|
||||
assert js_body_key == js_read_key == "include_hidden", (
|
||||
"one field name for the whole wire contract"
|
||||
)
|
||||
|
||||
|
||||
# ---------- the static shell markup ----------
|
||||
|
||||
|
||||
def test_page_error_line_is_a_role_alert_after_the_table() -> None:
|
||||
"""index.html: ``#git-sources-hidden-error`` is a ``<p>`` reusing
|
||||
the existing ``.git-source-error`` class, ``role="alert"`` +
|
||||
hidden by default, INSIDE ``#view-git-sources`` and AFTER
|
||||
``#git-sources-table-wrap`` in source order (the checkbox is a
|
||||
table-cell control, so its failure announces at page level — the
|
||||
ignore dialog carries its own error INSIDE the modal). It
|
||||
appears exactly once."""
|
||||
html = _text(SHELL_HTML)
|
||||
tag = re.search(r'<p[^>]*id="git-sources-hidden-error"[^>]*>', html)
|
||||
assert tag, "#git-sources-hidden-error must be a real <p>"
|
||||
open_tag = tag.group(0)
|
||||
for attr in ('class="git-source-error"', 'role="alert"', "hidden"):
|
||||
assert attr in open_tag, f"the error line carries {attr}"
|
||||
view_i = html.find('id="view-git-sources"')
|
||||
wrap_i = html.find('id="git-sources-table-wrap"')
|
||||
err_i = html.find('id="git-sources-hidden-error"')
|
||||
next_i = html.find('id="view-history"')
|
||||
assert -1 < view_i < wrap_i < err_i < next_i, (
|
||||
"the line sits inside #view-git-sources, after #git-sources-table-wrap"
|
||||
)
|
||||
assert html.count('id="git-sources-hidden-error"') == 1, (
|
||||
"the id is unique in the shell"
|
||||
)
|
||||
|
||||
|
||||
# ---------- styles.css ----------
|
||||
|
||||
|
||||
def test_hidden_checkbox_css_rules_exist_with_accent_color() -> None:
|
||||
"""styles.css carries the phase-105 family: .git-source-hidden
|
||||
(inline-flex, 44px hit height matching the action buttons, the
|
||||
house ink), the checkbox (sized, ``accent-color: var(--brand)``),
|
||||
the :disabled wait state (the .git-source-remove:disabled
|
||||
idiom), and the "hidden on" tag (text + a distinct background —
|
||||
never color alone). The checkbox comment RECORDS the
|
||||
verified checked-state contrast (house style)."""
|
||||
css = _css()
|
||||
label = _css_rule(css, ".git-source-hidden")
|
||||
assert "display: inline-flex" in label
|
||||
assert "height: 44px" in label, "the ~44px hit height matches the action buttons"
|
||||
assert "var(--ink)" in label
|
||||
box = _css_rule(css, '.git-source-hidden input[type="checkbox"]')
|
||||
assert "accent-color: var(--brand)" in box, "the --brand checkbox fill"
|
||||
disabled = _css_rule(css, ".git-source-hidden:disabled")
|
||||
assert "opacity" in disabled and "cursor: wait" in disabled, (
|
||||
"the .git-source-remove:disabled idiom"
|
||||
)
|
||||
tag = _css_rule(css, ".git-source-hidden-count")
|
||||
assert "var(--ink)" in tag and "var(--bg)" in tag, (
|
||||
"text + a distinct background (never color alone)"
|
||||
)
|
||||
assert "url(http" not in css and "@import url(" not in css, (
|
||||
"no CDN (AGENTS.md rule 6)"
|
||||
)
|
||||
|
||||
|
||||
def test_checkbox_contrast_is_verified_and_recorded() -> None:
|
||||
"""House style: the checked-state contrast of the native widget
|
||||
is VERIFIED and the ratio RECORDED in the comment above the
|
||||
checkbox rule (the executor note the task template carries):
|
||||
the --bg-on---brand house pairing (5.2:1) on the built-in theme,
|
||||
and the theme tab's AA gate (4.5:1) for every saved palette."""
|
||||
css = _css()
|
||||
rule_i = css.find('.git-source-hidden input[type="checkbox"] {')
|
||||
assert rule_i != -1
|
||||
header = css[css.rfind("/*", 0, rule_i):rule_i]
|
||||
assert "VERIFIED" in header, "the contrast is verified (the executor note)"
|
||||
assert "5.2:1" in header, "the built-in --bg-on---brand ratio is recorded"
|
||||
assert "4.5:1" in header, "the AA gate for themed palettes is recorded"
|
||||
|
||||
|
||||
# ---------- the module docstring ----------
|
||||
|
||||
|
||||
def test_module_docstring_carries_the_phase_105_contract() -> None:
|
||||
"""The git-sources.js module docstring gained the phase-105
|
||||
entry: the per-row Hidden checkbox (makeRow) →
|
||||
PATCH {include_hidden} (task 03's optional field) →
|
||||
loadSources + announce; failure reverts the box +
|
||||
#git-sources-hidden-error (role=alert); env-fallback rows get
|
||||
no checkbox (A3)."""
|
||||
doc = _js().split("*/", 2)[0] # the module docstring (first block)
|
||||
for frag in (
|
||||
"Phase 105 (task 05)",
|
||||
"toggleHidden",
|
||||
"{ include_hidden }",
|
||||
"#git-sources-hidden-error",
|
||||
"env-fallback rows get NO checkbox",
|
||||
"REVERTS to the server state",
|
||||
"LAST announcement",
|
||||
):
|
||||
assert frag in doc, f"the module docstring lost: {frag!r}"
|
||||
@@ -849,12 +849,14 @@ def test_no_progress_means_no_prewalk(
|
||||
extensions: frozenset[str],
|
||||
excluded: frozenset[str] = EXCLUDED_DIRS,
|
||||
ignore: tuple[str, ...] = (),
|
||||
include_hidden: bool = False,
|
||||
) -> list[Path]:
|
||||
# Phase 89: the walker gained the ``ignore`` keyword — the sentinel
|
||||
# accepts (and forwards) it to stay a drop-in.
|
||||
# Phase 89: the walker gained the ``ignore`` keyword; phase 105:
|
||||
# the ``include_hidden`` flag — the sentinel accepts (and forwards)
|
||||
# both to stay a drop-in.
|
||||
nonlocal walk_calls
|
||||
walk_calls += 1
|
||||
return real_walker(r, extensions, excluded, ignore)
|
||||
return real_walker(r, extensions, excluded, ignore, include_hidden)
|
||||
|
||||
monkeypatch.setattr(importer, "iter_importable_files", counting)
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Unit: the phase-105 hidden-folders flag in ``iter_importable_files``.
|
||||
|
||||
Phase 105 (TODO.md L3 — "…a toggle per input (next to the ignores
|
||||
button) to allow indexing hidden .folders."): ``include_hidden`` lifts
|
||||
ONLY the dot-prefixed-component skip for one source. A1
|
||||
(owner-confirmed 2026-09-14): when True, files inside hidden folders
|
||||
AND hidden files with an importable extension become importable;
|
||||
``EXCLUDED_DIRS`` (``.venv``, ``node_modules``, ``.git``, …) stay
|
||||
excluded in BOTH states; the extension filter always applies; the
|
||||
phase-89 *ignore* tuple composes additively. A4: the default
|
||||
(``False``) is byte-identical to the pre-phase-105 walk — pinned here
|
||||
against a tmp fixture tree. The DB-facing behavior (hidden files
|
||||
indexed / pruned through ``import_sources``) is integration-gated by
|
||||
``tests/integration/test_importer_include_hidden.py``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.rag.importer import _include_hidden_for_root, iter_importable_files
|
||||
|
||||
#: Explicit extension set (not the A9 config default) — the tests pin
|
||||
#: the walk rules, not the config.
|
||||
EXTS = frozenset({".md"})
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def tree(tmp_path: Path) -> Path:
|
||||
"""The task's fixture layout (plus the no-extension ``.env``).
|
||||
|
||||
visible.md .hidden/note.md .notes.md
|
||||
.venv/junk.md node_modules/x.md .hidden/.deep.md
|
||||
keep/ok.md .env
|
||||
"""
|
||||
root = tmp_path / "HiddenFix"
|
||||
(root / ".hidden").mkdir(parents=True)
|
||||
(root / ".venv").mkdir()
|
||||
(root / "node_modules").mkdir()
|
||||
(root / "keep").mkdir()
|
||||
(root / "visible.md").write_text("visible\n", encoding="utf-8")
|
||||
(root / ".hidden" / "note.md").write_text("note\n", encoding="utf-8")
|
||||
(root / ".notes.md").write_text("notes\n", encoding="utf-8")
|
||||
(root / ".venv" / "junk.md").write_text("junk\n", encoding="utf-8")
|
||||
(root / "node_modules" / "x.md").write_text("x\n", encoding="utf-8")
|
||||
(root / ".hidden" / ".deep.md").write_text("deep\n", encoding="utf-8")
|
||||
(root / "keep" / "ok.md").write_text("ok\n", encoding="utf-8")
|
||||
(root / ".env").write_text("SECRET=1\n", encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
def _rels(paths: list[Path], root: Path) -> set[str]:
|
||||
return {p.relative_to(root).as_posix() for p in paths}
|
||||
|
||||
|
||||
# --- default (flag False): today's behavior, byte-identical -----------------
|
||||
|
||||
|
||||
def test_iter_default_flag_skips_all_hidden(tree: Path) -> None:
|
||||
# A4: the default is the pre-phase-105 walk — hidden dir, hidden
|
||||
# file, and the excluded dirs are all absent. Sorted order pinned
|
||||
# verbatim (not just the set).
|
||||
got = iter_importable_files(tree, EXTS)
|
||||
assert got == [tree / "keep" / "ok.md", tree / "visible.md"]
|
||||
|
||||
|
||||
def test_iter_default_flag_rejects_explicit_false(tree: Path) -> None:
|
||||
assert iter_importable_files(tree, EXTS, include_hidden=False) == [
|
||||
tree / "keep" / "ok.md",
|
||||
tree / "visible.md",
|
||||
]
|
||||
|
||||
|
||||
# --- flag True (A1) -----------------------------------------------------------
|
||||
|
||||
|
||||
def test_iter_flag_on_admits_hidden_dirs_and_hidden_files(tree: Path) -> None:
|
||||
got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree)
|
||||
assert got == {
|
||||
".hidden/note.md", # inside a hidden dir
|
||||
".notes.md", # hidden file with an importable extension
|
||||
".hidden/.deep.md", # both at once
|
||||
"visible.md",
|
||||
"keep/ok.md",
|
||||
}
|
||||
|
||||
|
||||
def test_iter_flag_on_keeps_excluded_dirs_out(tree: Path) -> None:
|
||||
# A1: EXCLUDED_DIRS are skipped in BOTH states — caches are never
|
||||
# content, even with the flag on.
|
||||
got = _rels(iter_importable_files(tree, EXTS, include_hidden=True), tree)
|
||||
assert ".venv/junk.md" not in got
|
||||
assert "node_modules/x.md" not in got
|
||||
|
||||
|
||||
def test_iter_flag_on_keeps_extension_filter_in_force(tree: Path) -> None:
|
||||
# ``.env`` has no A9 extension — the extension filter is the real
|
||||
# content gate, so a secret-flavoured file is never listed.
|
||||
for flag in (False, True):
|
||||
got = _rels(iter_importable_files(tree, EXTS, include_hidden=flag), tree)
|
||||
assert ".env" not in got
|
||||
|
||||
|
||||
# --- composition with the phase-89 ignore tuple -------------------------------
|
||||
|
||||
|
||||
def test_iter_flag_on_ignore_composes_additively(tree: Path) -> None:
|
||||
# The ignored prefix still bites when the flag is ON; everything
|
||||
# else hidden is admitted.
|
||||
got = _rels(
|
||||
iter_importable_files(tree, EXTS, include_hidden=True, ignore=(".hidden",)),
|
||||
tree,
|
||||
)
|
||||
assert got == {".notes.md", "visible.md", "keep/ok.md"}
|
||||
|
||||
|
||||
def test_iter_ignore_only_flag_off(tree: Path) -> None:
|
||||
# The pre-phase-105 combination still works untouched: ignore
|
||||
# prefix on a visible subtree, flag at its default.
|
||||
got = _rels(iter_importable_files(tree, EXTS, ignore=("keep",)), tree)
|
||||
assert got == {"visible.md"}
|
||||
|
||||
|
||||
# --- _include_hidden_for_root: the single read point --------------------------
|
||||
|
||||
|
||||
def test_include_hidden_for_root_none_map_is_false(tmp_path: Path) -> None:
|
||||
root = tmp_path / "src"
|
||||
root.mkdir()
|
||||
assert _include_hidden_for_root(root, None) is False
|
||||
|
||||
|
||||
def test_include_hidden_for_root_unlisted_root_is_false(tmp_path: Path) -> None:
|
||||
root = tmp_path / "src"
|
||||
root.mkdir()
|
||||
assert _include_hidden_for_root(root, {"/other/root": True}) is False
|
||||
|
||||
|
||||
def test_include_hidden_for_root_listed_true_is_true(tmp_path: Path) -> None:
|
||||
root = tmp_path / "src"
|
||||
root.mkdir()
|
||||
assert _include_hidden_for_root(root, {str(root): True}) is True
|
||||
|
||||
|
||||
def test_include_hidden_for_root_listed_false_is_false(tmp_path: Path) -> None:
|
||||
root = tmp_path / "src"
|
||||
root.mkdir()
|
||||
assert _include_hidden_for_root(root, {str(root): False}) is False
|
||||
|
||||
|
||||
def test_include_hidden_for_root_keys_by_str_root(tmp_path: Path) -> None:
|
||||
# Two distinct Path objects whose ``str()`` agrees — the answer must
|
||||
# be the same (the ``_ignore_for_root`` keying convention, phase 89).
|
||||
key = str(tmp_path / "src")
|
||||
a = tmp_path / "src"
|
||||
b = tmp_path / "src" / "." # different object, same str()
|
||||
assert str(a) == str(b) == key
|
||||
mp = {key: True}
|
||||
assert _include_hidden_for_root(a, mp) is True
|
||||
assert _include_hidden_for_root(b, mp) is True
|
||||
@@ -134,6 +134,42 @@ def test_git_source_python_default_empty_list() -> None:
|
||||
assert col.default.arg(None) == [], "the default must resolve to []"
|
||||
|
||||
|
||||
def test_git_sources_include_hidden_column_contract() -> None:
|
||||
"""Phase 105: every source row carries its hidden-folders flag —
|
||||
BOOLEAN, NOT NULL, server default ``false`` (a pre-phase-105 row
|
||||
reads ``False``, so every existing source imports byte-identically
|
||||
until the owner flips it — A4)."""
|
||||
sources = Base.metadata.tables["git_sources"]
|
||||
assert "include_hidden" in sources.c, (
|
||||
"git_sources must have the include_hidden column (phase 105)"
|
||||
)
|
||||
col = sources.c["include_hidden"]
|
||||
assert col.nullable is False, "git_sources.include_hidden must be NOT NULL"
|
||||
sd = col.server_default
|
||||
assert isinstance(sd, DefaultClause), "include_hidden needs a server default"
|
||||
assert isinstance(sd.arg, TextClause), (
|
||||
"the server default must be the literal SQL text false"
|
||||
)
|
||||
assert sd.arg.text == "false", "include_hidden server default must be false"
|
||||
|
||||
|
||||
def test_git_source_python_default_include_hidden_false() -> None:
|
||||
"""A freshly constructed row (no DB) resolves the flag to ``False``
|
||||
via the Python-side default (``default=False``) — the ORM INSERT-time
|
||||
default, so an ORM insert that omits the column inserts ``false``
|
||||
rather than NULL (the server default ``false`` independently covers
|
||||
non-ORM inserts — A4)."""
|
||||
from app.models import GitSource
|
||||
|
||||
row = GitSource(url="https://example.com/r.git", kind="git")
|
||||
col = row.__table__.c["include_hidden"]
|
||||
assert col.default is not None, (
|
||||
"include_hidden needs a Python-side (INSERT-time) default"
|
||||
)
|
||||
# A scalar default: the arg IS the value (not a callable).
|
||||
assert col.default.arg is False, "the default must resolve to False"
|
||||
|
||||
|
||||
def test_doc_drafts_column_contract() -> None:
|
||||
"""Phase 59: the editable triple (title/path/body) + status +
|
||||
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
|
||||
|
||||
@@ -733,6 +733,7 @@ class _GatedImport:
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
) -> ImportSummary:
|
||||
self.prune_flags.append(prune)
|
||||
if progress is not None:
|
||||
|
||||
Reference in New Issue
Block a user