feat(import): index quadlet unit files and jinja templates (A9 revision)

Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
2026-08-28 07:02:24 -04:00
parent 6be692d999
commit 5d679f5184
40 changed files with 811 additions and 73 deletions
+54 -3
View File
@@ -8,7 +8,7 @@ import pytest
from pydantic import ValidationError
from pydantic_settings import SettingsError
from app.config import Settings
from app.config import _ALLOWED_IMPORT_EXTENSIONS, Settings # pyright: ignore[reportPrivateUsage]
def _settings(**kwargs: Any) -> Settings:
@@ -42,12 +42,46 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non
# Phase 17: the model's thinking streams by default (kill-switch off).
assert s.stream_thinking is True
assert len(s.suggestions) >= 3
# A9 (revised): the import scope covers the seven A9 formats.
# A9 (revised 2026-08-27): the import scope covers all seventeen
# A9 formats (original seven + quadlet family + jinja).
assert s.import_extension_set == {
".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"
".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py",
".container", ".network", ".volume", ".image", ".pod",
".kube", ".swap", ".os", ".endpoint", ".j2",
}
NEW_A9_FORMATS = (
"container", "network", "volume", "image", "pod",
"kube", "swap", "os", "endpoint", "j2",
)
def test_allowed_import_extensions_contains_all_seventeen_formats() -> None:
"""The validator's base set is the full A9 set: the original seven
plus the ten added 2026-08-27 (quadlet family + ``j2``). The
never-widen contract bounds :py:data:`import_extensions` against
exactly this set."""
assert {
"md", "markdown", "txt", "yaml", "yml", "json", "py",
*NEW_A9_FORMATS,
} == _ALLOWED_IMPORT_EXTENSIONS
def test_default_import_extensions_include_the_ten_new_formats() -> None:
"""A9 revised 2026-08-27 (owner permission): the quadlet family +
``j2`` are imported by default — no env configuration needed — with
the original seven first (order is cosmetic, the set is what
matters)."""
s = _settings()
for ext in NEW_A9_FORMATS:
assert ext in s.import_extensions
assert s.import_extension_set == (
{".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"}
| {f".{ext}" for ext in NEW_A9_FORMATS}
)
def test_env_override(monkeypatch) -> None:
monkeypatch.setenv("BOR_RELEVANCE_THRESHOLD", "0.42")
monkeypatch.setenv("BOR_LLM_CHAT_MODEL", "juggernaut")
@@ -131,6 +165,23 @@ def test_import_extensions_rejects_empty(monkeypatch) -> None:
_settings()
def test_import_extensions_validator_accepts_new_a9_formats(monkeypatch) -> None:
"""A9 revised 2026-08-27: the new names are first-class — the
never-widen contract now holds against the widened base set, so a
narrowing CSV with quadlet/jinja names is accepted."""
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,container,j2")
s = _settings()
assert s.import_extension_set == {".md", ".container", ".j2"}
def test_import_extensions_validator_still_rejects_unknown(monkeypatch) -> None:
"""Truly unknown extensions still fail loudly at startup (the
validator is intact — only the allowed base set widened)."""
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,xyz")
with pytest.raises(ValidationError, match="xyz"):
_settings()
def test_git_sources_default_empty_and_sources_dir_default() -> None:
"""Phase 28: no git sources by default (backwards-compatible with the
``--source`` / ``DEFAULT_SOURCES`` fallback); the clone location stays