phase: 105_hidden_folders_toggle

All completion criteria verified. Everything is green. **Phase 105 final verification pass — all criteria verified** - Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page - Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified) - Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓ - Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation) - Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed - Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings) **Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched). **Deviations:** none — no defects found; no code changes were needed on this pass. **Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
2026-09-12 21:26:30 -04:00
parent ecc921098a
commit d731169b8b
49 changed files with 3553 additions and 93 deletions
@@ -0,0 +1,271 @@
+"""Integration: migration 0019 (git_sources.include_hidden) schema
+contract (phase 105, task 01).
+
+Drives the **real Alembic engine** against the live dev database
+(``podman compose up -d db``), mirroring the house pattern of
+``test_migration_0007.py`` (git_sources column contract — information_schema
+assertions on the state the migration must leave). The tests target the
+0018 → 0019 step explicitly so later migrations cannot break them:
+
+* upgrade 0018 → 0019 → the ``include_hidden`` column exists with the
+  full contract — BOOLEAN NOT NULL, server default ``false`` — while
+  the 0018 ``git_sources`` schema (url, kind, path, ignore_paths)
+  survives;
+* pre-0019 rows backfill ``false`` and a row written without the
+  column takes the server default (A4 — byte-identical import
+  behavior until the owner flips the flag);
+* the ORM contract agrees: a freshly inserted ``GitSource`` (no flag
+  passed) reads ``include_hidden is False`` (the Python ``default=False``
+  and the server default agree), and an explicit ``True`` round-trips
+  through a fresh session;
+* downgrade to 0018 → the column is GONE (A13 — reversible) while the
+  rows + their ignore lists survive;
+* upgrade back to 0019 → the column is back (round-trip).
+
+The ``alembic`` fixture guarantees the DB ends at head even if a test
+fails or the process is interrupted.
+"""
+from __future__ import annotations
+
+import uuid
+from collections.abc import Iterator
+from typing import Any
+
+import pytest
+from alembic.config import Config
+from sqlalchemy import text
+from sqlalchemy.orm import Session
+
+from alembic import command
+from app.db import SessionLocal, db_available
+from app.models import GitSource
+
+URL_BASE = "https://git.example.com/mig0019"
+
+
+@pytest.fixture()
+def alembic(db: Session) -> Iterator[Config]:
+    """Real Alembic config bound to the dev DB (URL from app settings).
+
+    Starts at head (repairs an interrupted earlier run); teardown upgrades
+    to head no matter what happened, so the dev DB is never left below
+    head.
+    """
+    if not db_available():
+        pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
+    cfg = Config()  # no alembic.ini file — env.py gets the URL from app config
+    cfg.set_main_option("script_location", "alembic")
+    command.upgrade(cfg, "head")
+    try:
+        yield cfg
+    finally:
+        # Release the test session's open transaction BEFORE the repair
+        # DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
+        # on ``git_sources``, which would deadlock the repair's
+        # ``ALTER TABLE`` (0019) forever.
+        db.rollback()
+        command.upgrade(cfg, "head")
+
+
+def _version(db: Session) -> str | None:
+    return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
+
+
+def _column(db: Session, column: str) -> tuple[Any, ...] | None:
+    """(data_type, is_nullable, column_default) for one git_sources column."""
+    row = db.execute(
+        text(
+            "SELECT data_type, is_nullable, column_default"
+            " FROM information_schema.columns"
+            " WHERE table_name = 'git_sources' AND column_name = :c"
+        ),
+        {"c": column},
+    ).fetchone()
+    return tuple(row) if row is not None else None
+
+
+def _flag(db: Session, url: str) -> Any:
+    return db.execute(
+        text("SELECT include_hidden FROM git_sources WHERE url = :u"),
+        {"u": url},
+    ).scalar_one()
+
+
+def _insert_sql(db: Session, url: str, include_hidden: Any = None) -> None:
+    """Insert one git_sources row (kind/path omitted → the API default
+    shape); ``include_hidden`` omitted → pre-0019 insert shape."""
+    if include_hidden is None:
+        db.execute(
+            text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
+            {"u": url},
+        )
+    else:
+        db.execute(
+            text(
+                "INSERT INTO git_sources (id, url, include_hidden)"
+                " VALUES (gen_random_uuid(), :u, :f)"
+            ),
+            {"u": url, "f": include_hidden},
+        )
+    db.commit()
+
+
+def _delete_by_url(db: Session, url: str) -> None:
+    db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url})
+    db.commit()
+
+
+def test_upgrade_to_0019_adds_include_hidden(db: Session, alembic: Config) -> None:
+    """Upgrade 0018 → 0019: the column exists with the full contract
+    (BOOLEAN NOT NULL, server default ``false``), is ABSENT at 0018,
+    pre-0019 rows backfill ``false`` (A4), a new row without the column
+    takes the default, and an explicit ``true`` round-trips — while the
+    0018 table contract survives."""
+    command.downgrade(alembic, "0018")  # start from the pre-0019 state
+    assert _version(db) == "0018"
+    assert _column(db, "include_hidden") is None, (
+        "the flag must be absent at 0018"
+    )
+
+    url_pre = f"{URL_BASE}/pre-existing.git"
+    _insert_sql(db, url_pre)  # no include_hidden column exists at 0018
+    try:
+        command.upgrade(alembic, "0019")
+        assert _version(db) == "0019", "alembic_version must be at 0019"
+
+        flag = _column(db, "include_hidden")
+        assert flag is not None, "git_sources.include_hidden is missing"
+        assert flag[0] == "boolean", "include_hidden must be BOOLEAN"
+        assert flag[1] == "NO", "include_hidden must be NOT NULL"
+        assert flag[2] is not None and "false" in str(flag[2]), (
+            "include_hidden must carry the `false` server default"
+        )
+
+        # The pre-0019 row backfilled ``false`` (A4 — unchanged imports).
+        assert _flag(db, url_pre) is False
+
+        # A row written without the column takes the server default.
+        url_new = f"{URL_BASE}/new-row.git"
+        _insert_sql(db, url_new)
+        try:
+            assert _flag(db, url_new) is False, (
+                "an omitted flag takes the `false` server default"
+            )
+
+            # The flag round-trips through an explicit ``true``.
+            db.execute(
+                text("UPDATE git_sources SET include_hidden = true WHERE url = :u"),
+                {"u": url_new},
+            )
+            db.commit()
+            assert _flag(db, url_new) is True, (
+                "include_hidden = true must round-trip"
+            )
+        finally:
+            _delete_by_url(db, url_new)
+
+        # The 0018 schema survives the additive upgrade.
+        ignore = _column(db, "ignore_paths")
+        assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", (
+            "git_sources.ignore_paths (0013) must survive the upgrade"
+        )
+        kind = _column(db, "kind")
+        assert kind is not None and kind[0] == "text" and kind[1] == "NO", (
+            "git_sources.kind (0007) must survive the upgrade"
+        )
+    finally:
+        _delete_by_url(db, url_pre)
+
+
+def test_orm_fresh_row_defaults_false_and_true_round_trips(
+    db: Session, alembic: Config
+) -> None:
+    """The ORM contract agrees with the column contract: a freshly
+    inserted ``GitSource`` (no flag passed) reads ``include_hidden is
+    False`` — the Python ``default=False`` and the server default
+    agree (A4) — and an explicit ``True`` round-trips through a fresh
+    session."""
+    command.upgrade(alembic, "head")
+    url_off = f"{URL_BASE}/orm-default.git"
+    url_on = f"{URL_BASE}/orm-true.git"
+    try:
+        # Fresh row, flag omitted → False (the Python-side default).
+        row_off = GitSource(url=url_off, kind="git")
+        db.add(row_off)
+        db.commit()
+        db.expire_all()
+        reloaded_off = db.get(GitSource, row_off.id)
+        assert reloaded_off is not None, "the fresh row must be readable"
+        assert reloaded_off.include_hidden is False, (
+            "a fresh row must read include_hidden is False (A4)"
+        )
+
+        # Explicit True round-trips through a FRESH session.
+        row_on = GitSource(url=url_on, kind="git", include_hidden=True)
+        db.add(row_on)
+        db.commit()
+        with SessionLocal() as fresh:
+            reloaded = fresh.get(GitSource, row_on.id)
+            assert reloaded is not None, "the row must exist in a fresh session"
+            assert reloaded.include_hidden is True, (
+                "include_hidden=True must round-trip through the DB"
+            )
+    finally:
+        _delete_by_url(db, url_off)
+        _delete_by_url(db, url_on)
+
+
+def test_downgrade_to_0018_drops_the_column(db: Session, alembic: Config) -> None:
+    """Downgrade 0019 → 0018: the column is gone (A13 — fully
+    reversible) while the rows + their ignore lists survive, and the
+    rest of the 0018 table contract (``path``) is intact."""
+    command.upgrade(alembic, "head")
+    url = f"{URL_BASE}/survivor.git"
+    db.execute(
+        text(
+            "INSERT INTO git_sources (id, url, include_hidden)"
+            " VALUES (:i, :u, true)"
+        ),
+        {"i": uuid.uuid4(), "u": url},
+    )
+    db.commit()
+    try:
+        command.downgrade(alembic, "0018")
+        assert _version(db) == "0018"
+        assert _column(db, "include_hidden") is None, (
+            "the flag must be dropped"
+        )
+        row = db.execute(
+            text("SELECT url, kind, ignore_paths FROM git_sources WHERE url = :u"),
+            {"u": url},
+        ).fetchone()
+        assert row is not None and row[0] == url, (
+            "the row must survive the column drop"
+        )
+        assert row[1] == "git" and row[2] == [], (
+            "kind + the ignore list must survive the column drop"
+        )
+
+        doc_path = _column(db, "path")
+        assert doc_path is not None and doc_path[0] == "text", (
+            "git_sources.path must survive the downgrade"
+        )
+    finally:
+        _delete_by_url(db, url)
+    # Repair: the fixture teardown re-upgrades to head.
+
+
+def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
+    """Downgrade to 0018, then upgrade back to 0019: the column is back
+    with the full contract (BOOLEAN NOT NULL, the `false` default)."""
+    command.downgrade(alembic, "0018")
+    command.upgrade(alembic, "0019")
+    assert _version(db) == "0019", "round-trip upgrade must land at 0019"
+
+    flag = _column(db, "include_hidden")
+    assert flag is not None, "git_sources.include_hidden must be back"
+    assert flag[0] == "boolean", "include_hidden must be BOOLEAN after the round-trip"
+    assert flag[1] == "NO", "include_hidden must be NOT NULL after the round-trip"
+    assert flag[2] is not None and "false" in str(flag[2]), (
+        "the `false` server default must survive the round-trip"
+    )