feat(import): index quadlet unit files and jinja templates (A9 revision)

Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
2026-08-28 07:02:24 -04:00
parent 6be692d999
commit 5d679f5184
40 changed files with 811 additions and 73 deletions
+23 -8
View File
@@ -28,6 +28,11 @@ EXPECTED_DOCS = {
("docs", "homelab/scripts/uptime_probe.py"),
("docs", "homelab/ssh/ssh_aliases.txt"),
("docs", "homelab/tables.md"), # phase 44: the markdown-tables fixture
# phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures
("docs", "homelab/quadlet/compose.container"),
("docs", "homelab/quadlet/lan.network"),
("docs", "homelab/quadlet/cache.volume"),
("docs", "homelab/templates/deploy.j2"),
}
@@ -37,13 +42,19 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
llm = FakeEmbedder()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
# Nine A9-format files (phase 44 added homelab/tables.md);
# .hidden/junk.md is out of scope (A9 revised).
assert (summary.files, summary.added, summary.unchanged) == (9, 9, 0)
assert summary.chunks >= 9
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
# Thirteen A9-format files (phase 44 added homelab/tables.md, phase 47
# added the quadlet + j2 fixtures); .hidden/junk.md is out of scope
# (A9 revised).
assert (summary.files, summary.added, summary.unchanged) == (13, 13, 0)
assert summary.chunks >= 13
assert summary.formats == {
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
}
# PLAN §9 per-format summary line: highest count first, then alpha.
assert summary.format_counts() == "md:5,json:1,py:1,txt:1,yaml:1"
assert summary.format_counts() == (
"md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1"
)
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == EXPECTED_DOCS
@@ -57,6 +68,10 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
assert titles["homelab/scripts/uptime_probe.py"] == "uptime_probe"
assert titles["homelab/networking/static-dns.json"] == "static-dns"
assert titles["homelab/ssh/ssh_aliases.txt"] == "ssh_aliases"
assert titles["homelab/quadlet/compose.container"] == "compose"
assert titles["homelab/quadlet/lan.network"] == "lan"
assert titles["homelab/quadlet/cache.volume"] == "cache"
assert titles["homelab/templates/deploy.j2"] == "deploy"
# Hidden junk was never imported.
assert not any(".hidden" in d.path for d in docs)
# Full content is stored — that is what the RAG context will be.
@@ -92,13 +107,13 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
r = admin_client.get("/api/docs") # phase 16: the catalog is admin-only
assert r.status_code == 200
body = r.json()
assert len(body["documents"]) == 9
assert len(body["documents"]) == 13
assert all(d["chunks"] >= 1 for d in body["documents"])
# Idempotent re-run: nothing re-embedded.
calls_before = len(llm.calls)
s2 = asyncio.run(import_sources([FIXTURES], llm, session=db))
assert s2.unchanged == 9 and s2.added == 0
assert s2.unchanged == 13 and s2.added == 0
assert len(llm.calls) == calls_before # unchanged → no embedding requests
db.execute(text("TRUNCATE chunks, documents, query_log"))