Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(import): index quadlet unit files and jinja templates (A9 revision)

Browse Source 
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
ducoterra
2026-08-28 07:02:24 -04:00
parent 6be692d999
commit 5d679f5184
40 changed files with 811 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/unit/test_importer.py
+126 -2
View File
@@ -18,6 +18,7 @@ from pathlib import Path
import pytest
from sqlalchemy import func, select
from app.config import Settings
from app.models import Chunk, Document
from app.rag.importer import (
EXCLUDED_DIRS,
@@ -29,8 +30,9 @@ from app.rag.importer import (
from app.rag.llm import EmbeddingError
from tests.fakes import FakeEmbedder
#: A9 default extension set as dotted suffixes (what the importer passes to
#: the walker when no override is configured).
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
#: set). The phase-47 walk test uses the live config default
#: (``Settings().import_extension_set`` — now seventeen formats).
DEFAULT_EXTS = frozenset({".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"})
@@ -121,6 +123,70 @@ def test_iter_importable_files_missing_dir_yields_nothing(tmp_path: Path) -> Non
assert iter_importable_files(tmp_path / "definitely-missing", DEFAULT_EXTS) == []
def test_iter_importable_files_walker_picks_up_new_formats_by_default(
tmp_path: Path,
) -> None:
"""Phase 47 (A9 revised 2026-08-27): the ten new formats — the full
quadlet family + j2 — walk under the *default* extension set (no env
configuration needed), the original seven still walk (regression), and
unknown/hidden-dir/excluded-dir filtering is unchanged for them."""
root = tmp_path / "quadproj"
(root / ".esphome").mkdir(parents=True)
(root / "node_modules").mkdir()
files = {
# the ten new formats (A9 revised 2026-08-27):
"web.container": "# RESE-QUADLET-SENTINEL-77aa\n[Container]\nImage=alpine\n",
"lan.network": "[Network]\nDriver=bridge\n",
"cache.volume": "[Volume]\nDriver=local\n",
"alpine.image": "[Image]\nImages=alpine\n",
"app.pod": "[Pod]\nPodName=app\n",
"k3s.kube": "apiVersion: v1\nkind: Pod\n",
"zram.swap": "[Swap]\nFile=/swapfile\n",
"fedora.os": "[OS]\nImage=fedora\n",
"edge.endpoint": "[Endpoint]\nPort=8080\n",
"deploy.j2": "{% for s in services %}\n[{{ s }}]\n{% endfor %}\n",
# the original seven — regression:
"note.md": "# n\n",
"note.markdown": "# m\n",
"note.txt": "t\n",
"cfg.yaml": "a: b\n",
"cfg.yml": "a: b\n",
"data.json": "{}\n",
"agent.py": "x = 1\n",
# must be skipped — filtering is unchanged for the new formats too:
"mystery.xyz": "unknown extension",
".esphome/x.container": "hidden dir (new-format file)",
"node_modules/y.container": "excluded dir (new-format file)",
}
for rel, text in files.items():
(root / rel).write_text(text)
# The live config default (Settings with no env override) — no
# BOR_IMPORT_EXTENSIONS needed for the new formats to walk.
exts = Settings(_env_file=None).import_extension_set # pyright: ignore[reportCallIssue]
found = {p.relative_to(root).as_posix() for p in iter_importable_files(root, exts)}
assert found == {
# all ten new formats:
"web.container",
"lan.network",
"cache.volume",
"alpine.image",
"app.pod",
"k3s.kube",
"zram.swap",
"fedora.os",
"edge.endpoint",
"deploy.j2",
# …and the original seven:
"note.md",
"note.markdown",
"note.txt",
"cfg.yaml",
"cfg.yml",
"data.json",
"agent.py",
}
def test_iter_importable_files_respects_custom_extension_filter(tmp_path: Path) -> None:
"""A narrower filter (e.g. md only) excludes the other A9 formats."""
root = tmp_path / "filtered"
@@ -358,6 +424,64 @@ def test_multi_format_import_counts_per_format_and_titles_stem(db, tmp_path: Pat
_cleanup_source(db, root.name)
def test_quadlet_and_j2_files_get_stem_titles_and_per_format_counts(
db, tmp_path: Path
) -> None:
"""Phase 47: the ten new formats import like any other A9 format — the
per-format counts land in the summary, every file yields content
chunks, and a leading ``#`` line (a TOML comment in quadlet files, not
a heading) never becomes the title: the file stem wins."""
root = tmp_path / "quadsrc"
(root / "quadlet").mkdir(parents=True)
(root / "tpl").mkdir(parents=True)
(root / "quadlet" / "web.container").write_text(
"# RESE-QUADLET-SENTINEL-77aa\n\n[Container]\nImage=alpine\n"
)
(root / "quadlet" / "lan.network").write_text("# net\n[Network]\nDriver=bridge\n")
(root / "quadlet" / "cache.volume").write_text("[Volume]\nDriver=local\n")
(root / "quadlet" / "alpine.image").write_text("[Image]\nImages=alpine\n")
(root / "quadlet" / "app.pod").write_text("[Pod]\nPodName=app\n")
(root / "quadlet" / "k3s.kube").write_text("apiVersion: v1\nkind: Pod\n")
(root / "quadlet" / "zram.swap").write_text("[Swap]\nFile=/swapfile\n")
(root / "quadlet" / "fedora.os").write_text("[OS]\nImage=fedora\n")
(root / "quadlet" / "edge.endpoint").write_text("[Endpoint]\nPort=8080\n")
(root / "tpl" / "deploy.j2").write_text(
"{% for s in services %}\n[{{ s }}]\nport = {{ s.port }}\n{% endfor %}\n"
)
llm = FakeEmbedder()
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert summary.files == 10 and summary.added == 10
assert summary.formats == {
"container": 1, "network": 1, "volume": 1, "image": 1, "pod": 1,
"kube": 1, "swap": 1, "os": 1, "endpoint": 1, "j2": 1,
}
docs = {
d.path: d
for d in db.scalars(select(Document).where(Document.source == root.name)).all()
}
titles = {p: d.title for p, d in docs.items()}
assert titles == {
# the ``#`` line is a comment in these formats — stem titles:
"quadlet/web.container": "web",
"quadlet/lan.network": "lan",
"quadlet/cache.volume": "cache",
"quadlet/alpine.image": "alpine",
"quadlet/app.pod": "app",
"quadlet/k3s.kube": "k3s",
"quadlet/zram.swap": "zram",
"quadlet/fedora.os": "fedora",
"quadlet/edge.endpoint": "edge",
"tpl/deploy.j2": "deploy",
}
for doc in docs.values():
content = [c for c in doc.chunks if not c.is_summary]
assert content # every new-format file yields content chunks
assert all(c.embedding is not None and len(c.embedding) == 768 for c in content)
finally:
_cleanup_source(db, root.name)
# ---------- phase 30: lite-model summaries for non-markdown files ----------