feat(import): index quadlet unit files and jinja templates (A9 revision)

Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
2026-08-28 07:02:24 -04:00
parent 6be692d999
commit 5d679f5184
40 changed files with 811 additions and 73 deletions
@@ -0,0 +1,135 @@
"""Integration test: the phase-47 quadlet + j2 formats ride the import
machinery unchanged (walk → delta → prune).
The temp source tree holds byte-identical copies of the real fixture
files (``tests/fixtures/docs/homelab/quadlet/*.container|.volume`` and
``templates/deploy.j2``), so this test tracks the fixtures even if their
bytes change. Runs against the local compose Postgres (the ``db`` fixture
from ``tests/conftest.py``) with the deterministic ``FakeEmbedder`` —
same harness as ``test_importer_e2e.py``. A separate file (not an
extension of that one) because the delta/prune lifecycle mutates the
tree between runs, while the fixture e2e stays a single import +
idempotent re-run over the shared fixture tree.
Runs (DB must be up: ``podman compose up -d db``):
uv run pytest tests/integration/test_import_quadlet_jinja.py -v
"""
from __future__ import annotations
import asyncio
from collections.abc import Iterator
from pathlib import Path
import pytest
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.models import Chunk, Document
from app.rag.importer import import_sources
from tests.fakes import FakeEmbedder
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs" / "homelab"
#: Fixture files copied into the temp source tree (a ``.container``, a
#: ``.volume``, and a ``.j2``).
QUADLET_RELS = ("quadlet/compose.container", "quadlet/cache.volume")
JINJA_RELS = ("templates/deploy.j2",)
ALL_RELS = QUADLET_RELS + JINJA_RELS
def _cleanup_source(db: Session, source: str) -> None:
for doc in db.scalars(select(Document).where(Document.source == source)).all():
db.delete(doc)
db.commit()
@pytest.fixture()
def source_dir(db: Session, tmp_path: Path) -> Iterator[Path]:
"""A temp source tree seeded with copies of the real fixture files."""
root = tmp_path / "quadsrc"
for rel in ALL_RELS:
target = root / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes((FIXTURES / rel).read_bytes())
try:
yield root
finally:
_cleanup_source(db, root.name)
def test_quadlet_and_jinja_import_delta_and_prune(source_dir: Path, db: Session) -> None:
llm = FakeEmbedder()
# --- fresh import: all three docs land with stem titles + chunks ---
s1 = asyncio.run(import_sources([source_dir], llm, prune=False, session=db))
assert (s1.files, s1.added, s1.unchanged, s1.updated, s1.pruned) == (3, 3, 0, 0, 0)
assert s1.formats == {"container": 1, "volume": 1, "j2": 1}
assert s1.errors == 0
docs = {
d.path: d
for d in db.scalars(select(Document).where(Document.source == source_dir.name)).all()
}
assert set(docs) == {
"quadlet/compose.container",
"quadlet/cache.volume",
"templates/deploy.j2",
}
# Non-markdown titles come from the file stem (a leading ``#`` is a
# comment in these formats, not a heading).
assert docs["quadlet/compose.container"].title == "compose"
assert docs["quadlet/cache.volume"].title == "cache"
assert docs["templates/deploy.j2"].title == "deploy"
for path, doc in docs.items():
content = [c for c in doc.chunks if not c.is_summary]
assert content, f"{path} has no content chunks"
assert all(
c.embedding is not None and len(c.embedding) == 768 for c in content
), f"{path} content chunks not embedded"
# Phase 30 parity: the new non-markdown formats also get their
# lite summary + one ``is_summary`` chunk.
assert doc.summary is not None, f"{path} should have a summary"
schunks = [c for c in doc.chunks if c.is_summary]
assert len(schunks) == 1 and schunks[0].position == -1
# --- delta: change only the .j2 → it updates, the others stay ---
j2 = source_dir / "templates" / "deploy.j2"
j2.write_text(j2.read_text(encoding="utf-8") + "\n# RESE-JINJA-DELTA-CHANGED\n")
s2 = asyncio.run(import_sources([source_dir], llm, session=db))
assert (s2.added, s2.updated, s2.unchanged, s2.pruned) == (0, 1, 2, 0)
changed = db.scalar(
select(Document).where(
Document.source == source_dir.name, Document.path == "templates/deploy.j2"
)
)
assert changed is not None and "RESE-JINJA-DELTA-CHANGED" in changed.content
# --- idempotent: a clean re-run re-embeds nothing ---
calls_before = len(llm.calls)
s3 = asyncio.run(import_sources([source_dir], llm, session=db))
assert (s3.added, s3.updated, s3.unchanged, s3.pruned) == (0, 0, 3, 0)
assert len(llm.calls) == calls_before # unchanged → no embedding requests
# --- prune: delete the .volume → its row + chunks cascade away ---
(source_dir / "quadlet" / "cache.volume").unlink()
s4 = asyncio.run(import_sources([source_dir], llm, session=db, prune=True))
assert (s4.unchanged, s4.pruned) == (2, 1)
assert db.scalar(
select(Document).where(
Document.source == source_dir.name, Document.path == "quadlet/cache.volume"
)
) is None
# Its chunks (content + summary) are gone — FK cascade.
assert db.scalar(
select(func.count())
.select_from(Chunk)
.join(Document, Document.id == Chunk.document_id)
.where(
Document.source == source_dir.name,
Document.path == "quadlet/cache.volume",
)
) == 0
# The other two docs survived the prune.
assert db.scalar(
select(func.count()).select_from(Document).where(Document.source == source_dir.name)
) == 2