feat(import): index quadlet unit files and jinja templates (A9 revision)

Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the full Podman quadlet family (.container, .network, .volume, .image, .pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the allowed + default A9 import formats, chunked as plain text (owner decision — no TOML/Jinja-aware splitter). No env configuration needed: a default import now indexes them. - app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default import_extensions CSV gain the ten names (the original seven first); the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and still rejects truly unknown extensions. - app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged); docstring/comments cite the A9 revision 2026-08-27. - tests/fixtures/docs/homelab/: quadlet/compose.container (realistic quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections, RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network, quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the fixture tree updates its 9 -> 13 document-count constants. - tests/unit/test_config.py: allowed set carries all seventeen formats, default CSV + dotted import_extension_set include the ten, the validator accepts the new names and still rejects unknowns. - tests/unit/test_chunker.py: dispatch parity with chunk_text for every new suffix (parametrized), the .container fixture chunks >=2 under the cap with the sentinel surviving, the .j2 fixture keeps {{ }} verbatim, the unknown-suffix fallback is unchanged. - tests/unit/test_importer.py: a default-extensions walk over a temp tree indexes exactly the ten new files (unknown/hidden/excluded filtered), the original seven still walk, stem-title fallback holds. - tests/integration/test_import_quadlet_jinja.py (new): import_sources over a temp tree with .container/.volume/.j2 -> documents + chunks rows with stem titles; delta re-import updates only the changed .j2 doc; prune drops the deleted .volume doc with cascade. - tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only, isolation): GET /api/docs (admin session) lists the four new-format docs with non-zero chunk counts and stem titles; the Sources table renders a row + .doc-link per file; the phase-26 modal shows the .container TOML ([Container] section + sentinel) with stem title and the container format badge; a RESE-JINJA-SENTINEL-33dd question FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero FTS hits) — the bubble is not .is-deflected and a source chip names templates/deploy.j2. - README.md + .env.example: the extended default format set (A9 revised 2026-08-27, plain-text chunking, narrow-only rule intact). - .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status, the revision note under the anchors table, and the §5 chunking-policy + §11 workflow lines. The only PLAN edit this phase. Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%); ruff check + pyright clean; story E2E 4/4 in isolation (DB up); regression E2E suites test_import_documents (3) / test_sync_button (3) / test_git_sources_admin (6) green in isolation. Also records the 47_quadlet_jinja_import task-file moves (01–03) todo/ -> complete/.
2026-08-28 07:02:24 -04:00
parent 6be692d999
commit 5d679f5184
40 changed files with 811 additions and 73 deletions
@@ -0,0 +1,204 @@
+"""Phase 47 E2E (Playwright): quadlet unit files + Jinja templates ride the
+A9 import path end to end (A9 revised 2026-08-27, owner permission).
+
+Story: ``.agent/user_stories/quadlet-jinja-import.md``
+Run in isolation (DB must be up: ``podman compose up -d db``):
+
+    uv run pytest tests/e2e/test_quadlet_jinja_import.py -v --no-cov
+
+Seeding reuses the real import function against ``tests/fixtures/docs/``
+with the deterministic mock embeddings — the ``test_import_documents.py``
+pattern (truncate the KB, ``import_sources`` in a worker thread). This
+suite's re-import changes the KB for the session; it is run in isolation
+(A16), so there is no cross-suite interference.
+
+Test → story mapping (Playwright Mapping Rule):
+1. ``test_quadlet_and_jinja_indexed`` — ``GET /api/docs`` (admin session)
+   lists the four new-format fixtures with non-zero chunk counts and the
+   file stem as title — no env configuration needed (default set).
+2. ``test_sources_table_shows_them`` (admin) — the Sources table renders
+   a row per new file, each with its ``.doc-link`` path link.
+3. ``test_container_content_viewable`` — the phase-26 modal shows the
+   ``.container`` file's TOML content (``[Container]`` section + sentinel)
+   with the stem as title.
+4. ``test_jinja_retrievable_not_deflected`` — a question carrying the
+   ``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine
+   below threshold **and** zero FTS hits) → honest-positive: the answer
+   bubble is not ``.is-deflected`` and a source chip names
+   ``templates/deploy.j2``.
+"""
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+from threading import Thread
+from typing import Any
+
+from playwright.sync_api import Page, expect
+from sqlalchemy import text
+
+from app.config import Settings
+from app.db import SessionLocal
+from app.rag.importer import ImportSummary, import_sources
+from app.rag.llm import LLMClient
+from e2e.auth_helpers import login
+
+REPO = Path(__file__).resolve().parents[2]
+FIXTURES = REPO / "tests" / "fixtures" / "docs"
+
+#: The four new-format fixtures (A9 revised 2026-08-27): path → stem title
+#: (no H1 → ``extract_title`` falls back to the file stem).
+NEW_FORMAT_DOCS = {
+    "homelab/quadlet/compose.container": "compose",
+    "homelab/quadlet/lan.network": "lan",
+    "homelab/quadlet/cache.volume": "cache",
+    "homelab/templates/deploy.j2": "deploy",
+}
+
+#: Carries the ``.j2`` fixture's sentinel — the hyphens split into the
+#: ``rese | jinja | sentinel | 33dd`` tsquery tokens that FTS-match the
+#: chunk holding ``{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %}``.
+JINJA_QUESTION = "What is RESE-JINJA-SENTINEL-33dd?"
+
+
+async def _import_fixtures(mock_port: int) -> ImportSummary:
+    kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
+    settings = Settings(**kwargs)  # pyright: ignore[reportCallIssue]
+    return await import_sources([FIXTURES], LLMClient(settings))
+
+
+def _run_in_thread(coro: Any) -> Any:
+    """Run a coroutine on a worker thread.
+
+    Playwright's sync API keeps an asyncio loop running on the test thread,
+    so ``asyncio.run`` cannot be called directly from a test body.
+    """
+    box: dict[str, Any] = {}
+
+    def runner() -> None:
+        try:
+            box["value"] = asyncio.run(coro)
+        except BaseException as e:  # noqa: BLE001 — re-raised on the test thread
+            box["error"] = e
+
+    t = Thread(target=runner)
+    t.start()
+    t.join()
+    if "error" in box:
+        raise box["error"]
+    return box["value"]
+
+
+def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
+    """Truncate the KB (and query log), then optionally re-import fixtures."""
+    with SessionLocal() as db:
+        db.execute(text("TRUNCATE chunks, documents, query_log"))
+        db.commit()
+    if not seed:
+        return None
+    return _run_in_thread(_import_fixtures(mock_port))
+
+
+# ---------------------------------------------------------------------------
+# 1. Default-extensions import indexes the new formats (API view)
+# ---------------------------------------------------------------------------
+
+
+def test_quadlet_and_jinja_indexed(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None:
+    summary = _reset_db(mock_llm, seed=True)
+    assert summary is not None
+    # The four fixtures joined the default set (A9 revised 2026-08-27) —
+    # the import needed no BOR_IMPORT_EXTENSIONS configuration at all.
+    assert summary.added == 13, f"expected all 13 fixture docs, added {summary.added}"
+
+    login(page, app_url)  # phase 16: the catalog is admin-only
+    r = page.request.get(f"{app_url}/api/docs")
+    assert r.status == 200
+    by_path = {d["path"]: d for d in r.json()["documents"]}
+    for path, stem in NEW_FORMAT_DOCS.items():
+        doc = by_path.get(path)
+        assert doc is not None, f"{path} missing from GET /api/docs"
+        assert doc["chunks"] > 0, f"{path} indexed zero chunks"
+        # No H1 → the stem is the title, exactly like the other
+        # non-markdown formats.
+        assert doc["title"] == stem, f"{path} title {doc['title']!r}, want stem {stem!r}"
+        assert doc["source"] == "docs"
+
+
+# ---------------------------------------------------------------------------
+# 2. The Sources table renders a row (with path link) per new file
+# ---------------------------------------------------------------------------
+
+
+def test_sources_table_shows_them(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None:
+    _reset_db(mock_llm, seed=True)
+    login(page, app_url)  # phase 16: the Sources catalog is admin-only
+    for path in NEW_FORMAT_DOCS:
+        row = page.locator("#docs-tbody tr", has_text=path)
+        expect(row).to_have_count(1)
+        link = row.locator("td:nth-child(2) a.doc-link")
+        expect(link).to_have_count(1)
+        # The full path is the link's accessible context (ellipsis is
+        # visual only) — the same contract the other rows carry.
+        expect(link).to_have_attribute("title", path)
+
+
+# ---------------------------------------------------------------------------
+# 3. The .container file's TOML is viewable in the modal (stem title)
+# ---------------------------------------------------------------------------
+
+
+def test_container_content_viewable(
+    page: Page, app_url: str, mock_llm: int, db_ready: None
+) -> None:
+    _reset_db(mock_llm, seed=True)
+    login(page, app_url)  # phase 16: the Sources catalog is admin-only
+
+    row = page.locator("#docs-tbody tr", has_text="homelab/quadlet/compose.container")
+    expect(row).to_have_count(1)
+    link = row.locator("td:nth-child(2) a.doc-link")
+    expect(link).to_have_attribute(
+        "href", "/document.html?source=docs&path=homelab%2Fquadlet%2Fcompose.container"
+    )
+    expect(link).not_to_have_attribute("target")
+
+    link.click()
+    expect(page.locator(".doc-modal")).to_be_visible()
+    # Stem title (no H1 in a quadlet unit file) + the extension badge.
+    expect(page.locator("#doc-modal-title")).to_have_text("compose")
+    expect(page.locator("#doc-modal-meta .format-badge")).to_have_text("container")
+    # Non-markdown content renders as escaped monospace text in a pre —
+    # the whole TOML, [Container] section and sentinel included.
+    pre = page.locator("#doc-modal-content pre.doc-raw")
+    expect(pre).to_have_count(1)
+    expect(pre).to_contain_text("[Container]")
+    expect(pre).to_contain_text("Image=docker.io/reese/compose-gateway:1.4.2")
+    expect(pre).to_contain_text("RESE-QUADLET-SENTINEL-77aa")
+    # Still on the Sources page: the modal is same-page (phase 26).
+    assert page.url == app_url + "/sources.html"
+
+
+# ---------------------------------------------------------------------------
+# 4. A .j2 sentinel question is retrievable and honest-positive (A8)
+# ---------------------------------------------------------------------------
+
+
+def test_jinja_retrievable_not_deflected(
+    page: Page, app_url: str, mock_llm: int, db_ready: None
+) -> None:
+    _reset_db(mock_llm, seed=True)
+    page.set_default_timeout(30_000)
+    page.goto(app_url)
+    page.fill("#message-input", JINJA_QUESTION)
+    page.click("#send-btn")
+
+    # The done event appends source chips — waiting on the .j2 chip means
+    # the turn is finished and the retrieval doc reached the UI.
+    chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2")
+    expect(chip).to_have_count(1, timeout=30_000)
+
+    # A8: LOW requires best cosine < threshold AND zero FTS hits — the
+    # question's sentinel tokens FTS-match the .j2 chunk, so the gate is
+    # honest-positive whatever the mock's cosine says.
+    expect(page.locator(".msg.brain")).to_have_count(1)
+    expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)