feat(import): index quadlet unit files and jinja templates (A9 revision)

Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
2026-08-28 07:02:24 -04:00
parent 6be692d999
commit 5d679f5184
40 changed files with 811 additions and 73 deletions
+17 -5
View File
@@ -56,7 +56,7 @@ that"* and offers alternatives instead of hallucinating.
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-27) |
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
@@ -83,6 +83,13 @@ that"* and offers alternatives instead of hallucinating.
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
> marker.
>
> **A9 revision (phase 47, owner permission 2026-08-27):** the format
> set extends with the Podman quadlet family (`container, network,
> volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja
> templates) — plain-text chunking (`chunk_text`), owner: `TODO.md`
> L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the
> hidden-dir/exclusion invariants are unchanged.
---
@@ -251,8 +258,11 @@ preceding heading in the text for retrieval quality.
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
`---` separators (key line kept as anchor); `json` pretty-printed, split on
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
hard cap (aipi ~1024-token request limit).
`txt` on paragraphs; markdown unchanged. The quadlet family
(`container, network, volume, image, pod, kube, swap, os, endpoint`)
and `j2` (Jinja templates) are plain-text chunked — no format-specific
splitter (A9 revised 2026-08-27, phase 47). Every format honors the
1200-char hard cap (aipi ~1024-token request limit).
---
@@ -455,8 +465,10 @@ uv run python -m scripts.llm_probe # sanity: models + dim
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
(no re-embedding); changed files are re-chunked + re-embedded (chunks
replaced atomically); `--prune` removes docs whose files disappeared or no
longer match the format filter. Formats per A9 (revised): `md, markdown,
txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
longer match the format filter. Formats per A9 (revised 2026-08-27):
`md, markdown, txt, yaml, yml, json, py`, the quadlet family
(`container, network, volume, image, pod, kube, swap, os, endpoint`), and
`j2` (plain-text chunked) (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
ranks live hybrid results for a question (retrieval tuning).
+1 -1
View File
@@ -41,7 +41,7 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)
# --- Import scope (A9 formats; may only narrow, never widen) ---
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips
# --- Import sources (git; phase 28, admin-managed since phase 35) ---
+8 -4
View File
@@ -293,8 +293,11 @@ embedded.
added=… updated=… unchanged=… pruned=… chunks=… embed_batches=…
formats=md:203,yaml:267,…`), so it is safe to run from a cron job or
after every commit.
- Indexed formats (A9): **`md, markdown, txt, yaml, yml, json, py`**
(case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). Any path with a
- Indexed formats (A9, revised 2026-08-27): **`md, markdown, txt,
yaml, yml, json, py`**, the Podman quadlet family (**`container,
network, volume, image, pod, kube, swap, os, endpoint`**), and **`j2`**
Jinja templates (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`).
Any path with a
**dot-prefixed component** — hidden files or vendored caches like
`.esphome/.espressif/**` — is skipped, along with `.venv`,
`node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`.
@@ -302,7 +305,8 @@ embedded.
that's how previously imported junk leaves the index.
- Non-markdown files get format-aware chunking (YAML top-level keys /
`---` docs, JSON top-level keys, Python top-level defs/classes via
stdlib `ast`) and their title comes from the file stem.
stdlib `ast`; quadlet unit files and `j2` templates are paragraph-
packed as plain text) and their title comes from the file stem.
- Unchanged files are **not re-embedded** — only new/changed ones, so
refreshes are cheap.
- After a run that **changed** the knowledge base (at least one document
@@ -644,7 +648,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn — every call the model emits consumes a round; at the cap the loop forces one final no-tools answer (0 = no tools, the kill switch) |
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
+17 -5
View File
@@ -11,10 +11,19 @@ from functools import lru_cache
from pydantic import field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
#: The A9 import formats (PLAN anchor A9, revised 2026-08-21).
#: ``BOR_IMPORT_EXTENSIONS`` may narrow — but never widen — this set.
#: The A9 import formats (PLAN anchor A9, revised 2026-08-21; revised
#: 2026-08-27, owner permission — the full Podman quadlet family
#: ``container, network, volume, image, pod, kube, swap, os, endpoint``
#: plus Jinja templates ``j2`` join the allowed set, chunked as plain
#: text). ``BOR_IMPORT_EXTENSIONS`` may narrow — but never widen — this
#: set.
_ALLOWED_IMPORT_EXTENSIONS: frozenset[str] = frozenset(
{"md", "markdown", "txt", "yaml", "yml", "json", "py"}
{
"md", "markdown", "txt", "yaml", "yml", "json", "py",
# A9 revised 2026-08-27 (owner permission): quadlet family + jinja.
"container", "network", "volume", "image", "pod",
"kube", "swap", "os", "endpoint", "j2",
}
)
@@ -127,14 +136,17 @@ class Settings(BaseSettings):
session_max_age: int = 43_200
session_cookie: str = "bor_session"
# --- Import scope (A9, revised 2026-08-21) ---
# --- Import scope (A9, revised 2026-08-21 and 2026-08-27) ---
# Comma-separated list of lowercased file extensions (no dot) imported
# by ``scripts/import_docs.py``. Hidden (dot) path components are always
# skipped, plus the importer's exclusion list.
# Stored as a raw CSV string (env-native — no JSON) and parsed on demand
# via :py:meth:`import_extension_set`. ``mode="after"`` validation runs
# against the raw string so a typo fails loudly at startup.
import_extensions: str = "md,markdown,txt,yaml,yml,json,py"
import_extensions: str = (
"md,markdown,txt,yaml,yml,json,py,"
"container,network,volume,image,pod,kube,swap,os,endpoint,j2"
)
#: List of git repo URLs to clone/pull into ``sources_dir`` before
#: indexing (phase 28); comma-separated, stored raw. Empty means no git
#: sources — ``import_docs`` then falls back to ``--source`` / the old
+19 -1
View File
@@ -22,6 +22,10 @@ per-format policies:
module preamble — imports, constants — is its own block); an oversized
definition falls back to line packing.
* **txt** (and any unknown suffix) — paragraph packing.
* **container / network / volume / image / pod / kube / swap / os /
endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and
Jinja templates; plain-text paragraph packing (``chunk_text``) — no
format-specific splitter (owner decision).
Every format honors :data:`HARD_MAX_CHARS` (1200 — the aipi ~1024-token
request cap) and the target/overlap settings; oversized blocks are split
@@ -331,7 +335,8 @@ def _normalize_target_overlap(target_chars: int, overlap_chars: int) -> tuple[in
return target, min(overlap_chars, target - 1)
#: suffix → chunker (A9, revised: md, markdown, txt, yaml, yml, json, py).
#: suffix → chunker (A9, revised 2026-08-21 and 2026-08-27: md, markdown,
#: txt, yaml, yml, json, py + the quadlet family and j2 — plain text).
_FORMAT_CHUNKERS = {
".md": chunk_markdown,
".markdown": chunk_markdown,
@@ -340,6 +345,19 @@ _FORMAT_CHUNKERS = {
".yml": chunk_yaml,
".json": chunk_json,
".py": chunk_python,
# A9 revised 2026-08-27 (owner permission): the full Podman quadlet
# family + Jinja templates — plain-text paragraph packing, no
# TOML/Jinja-aware splitter (owner decision).
".container": chunk_text,
".network": chunk_text,
".volume": chunk_text,
".image": chunk_text,
".pod": chunk_text,
".kube": chunk_text,
".swap": chunk_text,
".os": chunk_text,
".endpoint": chunk_text,
".j2": chunk_text,
}
+2 -2
View File
@@ -208,10 +208,10 @@ def test_admin_login_unlocks_sources_and_tuning(
login(page, app_url)
expect(page).to_have_url(app_url + "/sources.html")
expect(page.locator("#sources-gate")).to_be_hidden()
expect(page.locator("#stat-docs")).to_have_text("9") # phase 44: +tables.md
expect(page.locator("#stat-docs")).to_have_text("13") # phase 47: +quadlet/j2
expect(page.locator("#stat-chunks")).not_to_have_text("–")
expect(page.locator("#docs-table")).to_be_visible()
expect(page.locator("#docs-tbody tr")).to_have_count(9)
expect(page.locator("#docs-tbody tr")).to_have_count(13)
# Chat: the tuning UI is back — header toggle with count badge,
# Sign out instead of Sign in, Tune under the answer.
+1 -1
View File
@@ -128,7 +128,7 @@ def test_conversation_survives_reload(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
page.goto(app_url)
_ask(page, QUESTION)
+1 -1
View File
@@ -76,7 +76,7 @@ def test_on_topic_question_streams_grounded_answer(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
page.goto(app_url)
+1 -1
View File
@@ -114,7 +114,7 @@ def _seed_kb(mock_port: int) -> ImportSummary:
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
summary = _run_in_thread(_import_fixtures(mock_port))
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
return summary
+1 -1
View File
@@ -336,7 +336,7 @@ def test_edit_note_steers_answer(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
_open_tuning(page, app_url)
+1 -1
View File
@@ -84,7 +84,7 @@ def test_off_topic_question_deflects_honestly(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
page.goto(app_url)
expect(page.locator("#kb-banner")).to_be_hidden()
+13 -4
View File
@@ -41,6 +41,11 @@ EXPECTED_ROWS = (
"homelab/scripts/uptime_probe.py",
"homelab/ssh/ssh_aliases.txt",
"homelab/tables.md", # phase 44: the markdown-tables fixture
# phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures
"homelab/quadlet/compose.container",
"homelab/quadlet/lan.network",
"homelab/quadlet/cache.volume",
"homelab/templates/deploy.j2",
)
@@ -86,14 +91,18 @@ def test_sources_page_lists_indexed_docs(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
# Nine A9-format files are imported (phase 44 added homelab/tables.md);
# Thirteen A9-format files are imported (phase 44 added
# homelab/tables.md, phase 47 added the quadlet + j2 fixtures);
# .hidden/junk.md is out of scope (A9 revised — hidden path components
# are never walked).
assert summary is not None and summary.added == 9
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
assert summary is not None and summary.added == 13
assert summary.formats == {
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
}
login(page, app_url) # phase 16: the catalog is admin-only
expect(page.locator("#stat-docs")).to_have_text("9")
expect(page.locator("#stat-docs")).to_have_text("13")
# Phase 30: non-markdown fixtures each gained one ``is_summary`` chunk,
# so the Sources total is content chunks + summary chunks.
expect(page.locator("#stat-chunks")).to_have_text(
+2 -2
View File
@@ -170,7 +170,7 @@ def test_on_topic_answer_echoes_kb_overview(
the mock's echo of the ``<knowledge_base>`` section's first bullet —
only possible if the section reached the LLM prompt."""
summary = _reset_db(mock_llm, seed=True, overview=OVERVIEW)
assert summary is not None and summary.added == 9 # phase 44 added tables.md
assert summary is not None and summary.added == 13 # phase 47 added quadlet+j2
assert _overview_row() is not None # the row the turn must inject
bubble = _ask(page, app_url, QUESTION, KB_ECHO)
@@ -250,7 +250,7 @@ def test_mock_generated_outline_is_stored_and_echoed(
stores the byte-stable 8-token digest of the generator's document
list, and a chat turn echoes its first bullet."""
summary = _reset_db(mock_llm, seed=True, overview=None)
assert summary is not None and summary.added == 9 # phase 44 added tables.md
assert summary is not None and summary.added == 13 # phase 47 added quadlet+j2
assert _overview_row() is None # direct import never regenerates
kwargs: dict[str, Any] = {
+1 -1
View File
@@ -176,7 +176,7 @@ def test_typing_indicator_during_slow_think(
"""AC1/AC5: the 3s mock warm-up must show the typing indicator for
>=2s before any text appears, then it is gone once the answer lands."""
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
page.goto(app_url)
+2 -2
View File
@@ -106,7 +106,7 @@ def _run_in_thread(coro: Any) -> Any:
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
"""Truncate the KB (+ the global prompt-state rows), then optionally
re-import the fixtures (9 docs since phase 44 added tables.md)."""
re-import the fixtures (13 docs since phase 47 added quadlet+j2)."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
@@ -151,7 +151,7 @@ def test_chat_table_renders(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # phase 44: +tables.md
assert summary is not None and summary.added == 13 # phase 47: +quadlet/j2 fixtures
page.set_default_timeout(30_000)
bubble = _ask_table_answer(page, app_url)
+3 -3
View File
@@ -123,11 +123,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
@pytest.fixture()
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
44, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
47, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
skips with clear instructions when Postgres is down."""
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9
assert summary is not None and summary.added == 13
yield
_reset_db(mock_llm, seed=False)
+204
View File
@@ -0,0 +1,204 @@
"""Phase 47 E2E (Playwright): quadlet unit files + Jinja templates ride the
A9 import path end to end (A9 revised 2026-08-27, owner permission).
Story: ``.agent/user_stories/quadlet-jinja-import.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_quadlet_jinja_import.py -v --no-cov
Seeding reuses the real import function against ``tests/fixtures/docs/``
with the deterministic mock embeddings — the ``test_import_documents.py``
pattern (truncate the KB, ``import_sources`` in a worker thread). This
suite's re-import changes the KB for the session; it is run in isolation
(A16), so there is no cross-suite interference.
Test → story mapping (Playwright Mapping Rule):
1. ``test_quadlet_and_jinja_indexed`` — ``GET /api/docs`` (admin session)
lists the four new-format fixtures with non-zero chunk counts and the
file stem as title — no env configuration needed (default set).
2. ``test_sources_table_shows_them`` (admin) — the Sources table renders
a row per new file, each with its ``.doc-link`` path link.
3. ``test_container_content_viewable`` — the phase-26 modal shows the
``.container`` file's TOML content (``[Container]`` section + sentinel)
with the stem as title.
4. ``test_jinja_retrievable_not_deflected`` — a question carrying the
``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine
below threshold **and** zero FTS hits) → honest-positive: the answer
bubble is not ``.is-deflected`` and a source chip names
``templates/deploy.j2``.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from threading import Thread
from typing import Any
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.config import Settings
from app.db import SessionLocal
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
#: The four new-format fixtures (A9 revised 2026-08-27): path → stem title
#: (no H1 → ``extract_title`` falls back to the file stem).
NEW_FORMAT_DOCS = {
"homelab/quadlet/compose.container": "compose",
"homelab/quadlet/lan.network": "lan",
"homelab/quadlet/cache.volume": "cache",
"homelab/templates/deploy.j2": "deploy",
}
#: Carries the ``.j2`` fixture's sentinel — the hyphens split into the
#: ``rese | jinja | sentinel | 33dd`` tsquery tokens that FTS-match the
#: chunk holding ``{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %}``.
JINJA_QUESTION = "What is RESE-JINJA-SENTINEL-33dd?"
async def _import_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return await import_sources([FIXTURES], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread.
Playwright's sync API keeps an asyncio loop running on the test thread,
so ``asyncio.run`` cannot be called directly from a test body.
"""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
"""Truncate the KB (and query log), then optionally re-import fixtures."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
if not seed:
return None
return _run_in_thread(_import_fixtures(mock_port))
# ---------------------------------------------------------------------------
# 1. Default-extensions import indexes the new formats (API view)
# ---------------------------------------------------------------------------
def test_quadlet_and_jinja_indexed(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None
# The four fixtures joined the default set (A9 revised 2026-08-27) —
# the import needed no BOR_IMPORT_EXTENSIONS configuration at all.
assert summary.added == 13, f"expected all 13 fixture docs, added {summary.added}"
login(page, app_url) # phase 16: the catalog is admin-only
r = page.request.get(f"{app_url}/api/docs")
assert r.status == 200
by_path = {d["path"]: d for d in r.json()["documents"]}
for path, stem in NEW_FORMAT_DOCS.items():
doc = by_path.get(path)
assert doc is not None, f"{path} missing from GET /api/docs"
assert doc["chunks"] > 0, f"{path} indexed zero chunks"
# No H1 → the stem is the title, exactly like the other
# non-markdown formats.
assert doc["title"] == stem, f"{path} title {doc['title']!r}, want stem {stem!r}"
assert doc["source"] == "docs"
# ---------------------------------------------------------------------------
# 2. The Sources table renders a row (with path link) per new file
# ---------------------------------------------------------------------------
def test_sources_table_shows_them(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None:
_reset_db(mock_llm, seed=True)
login(page, app_url) # phase 16: the Sources catalog is admin-only
for path in NEW_FORMAT_DOCS:
row = page.locator("#docs-tbody tr", has_text=path)
expect(row).to_have_count(1)
link = row.locator("td:nth-child(2) a.doc-link")
expect(link).to_have_count(1)
# The full path is the link's accessible context (ellipsis is
# visual only) — the same contract the other rows carry.
expect(link).to_have_attribute("title", path)
# ---------------------------------------------------------------------------
# 3. The .container file's TOML is viewable in the modal (stem title)
# ---------------------------------------------------------------------------
def test_container_content_viewable(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
_reset_db(mock_llm, seed=True)
login(page, app_url) # phase 16: the Sources catalog is admin-only
row = page.locator("#docs-tbody tr", has_text="homelab/quadlet/compose.container")
expect(row).to_have_count(1)
link = row.locator("td:nth-child(2) a.doc-link")
expect(link).to_have_attribute(
"href", "/document.html?source=docs&path=homelab%2Fquadlet%2Fcompose.container"
)
expect(link).not_to_have_attribute("target")
link.click()
expect(page.locator(".doc-modal")).to_be_visible()
# Stem title (no H1 in a quadlet unit file) + the extension badge.
expect(page.locator("#doc-modal-title")).to_have_text("compose")
expect(page.locator("#doc-modal-meta .format-badge")).to_have_text("container")
# Non-markdown content renders as escaped monospace text in a pre —
# the whole TOML, [Container] section and sentinel included.
pre = page.locator("#doc-modal-content pre.doc-raw")
expect(pre).to_have_count(1)
expect(pre).to_contain_text("[Container]")
expect(pre).to_contain_text("Image=docker.io/reese/compose-gateway:1.4.2")
expect(pre).to_contain_text("RESE-QUADLET-SENTINEL-77aa")
# Still on the Sources page: the modal is same-page (phase 26).
assert page.url == app_url + "/sources.html"
# ---------------------------------------------------------------------------
# 4. A .j2 sentinel question is retrievable and honest-positive (A8)
# ---------------------------------------------------------------------------
def test_jinja_retrievable_not_deflected(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
_reset_db(mock_llm, seed=True)
page.set_default_timeout(30_000)
page.goto(app_url)
page.fill("#message-input", JINJA_QUESTION)
page.click("#send-btn")
# The done event appends source chips — waiting on the .j2 chip means
# the turn is finished and the retrieval doc reached the UI.
chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2")
expect(chip).to_have_count(1, timeout=30_000)
# A8: LOW requires best cosine < threshold AND zero FTS hits — the
# question's sentinel tokens FTS-match the .j2 chunk, so the gate is
# honest-positive whatever the mock's cosine says.
expect(page.locator(".msg.brain")).to_have_count(1)
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
+11 -7
View File
@@ -84,13 +84,17 @@ def _ask(page: Page, message: str) -> None:
def test_multi_format_import_hidden_doc_excluded(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""A9 (revised): all seven formats import; hidden (dot) paths never do."""
"""A9 (revised): all seventeen formats import; hidden (dot) paths never do."""
summary = _reset_db(mock_llm, seed=True)
assert summary is not None
# Nine A9-format fixture files (phase 44 added homelab/tables.md);
# .hidden/junk.md must never be walked.
assert summary.added == 9
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
# Thirteen A9-format fixture files (phase 44 added homelab/tables.md,
# phase 47 added the quadlet + j2 fixtures); .hidden/junk.md must never
# be walked.
assert summary.added == 13
assert summary.formats == {
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
}
# Phase 16: the catalog is admin-only — perform the real form login,
# then call the API with the signed cookie the browser now holds.
@@ -103,7 +107,7 @@ def test_multi_format_import_hidden_doc_excluded(
r = httpx.get(f"{app_url}/api/docs", timeout=10, cookies=cookies)
assert r.status_code == 200
docs = r.json()["documents"]
assert len(docs) == 9
assert len(docs) == 13
assert all(".hidden" not in d["path"] for d in docs)
assert {d["path"] for d in docs} >= {
"homelab/container_gitlab/gitlab.md",
@@ -114,7 +118,7 @@ def test_multi_format_import_hidden_doc_excluded(
}
# The Sources page (we're already on it, signed in) reflects the set.
expect(page.locator("#stat-docs")).to_have_text("9")
expect(page.locator("#stat-docs")).to_have_text("13")
expect(page.locator("#docs-tbody tr", has_text=".hidden")).to_have_count(0)
+3 -3
View File
@@ -154,11 +154,11 @@ def _no_error_banner(page: Page) -> None:
@pytest.fixture()
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
44, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
47, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
skips with clear instructions when Postgres is down."""
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9
assert summary is not None and summary.added == 13
yield
_reset_db(mock_llm, seed=False)
+1 -1
View File
@@ -127,7 +127,7 @@ def test_tune_under_answer_persists_and_steers(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
page.set_default_timeout(30_000)
page.goto(app_url)
login(page, app_url, next="/") # phase 16: tuning is admin-only
+1 -1
View File
@@ -63,7 +63,7 @@ def _seed_kb(mock_port: int) -> ImportSummary:
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
summary = _run_in_thread(_import_fixtures(mock_port))
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
return summary
+3 -3
View File
@@ -99,11 +99,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
@pytest.fixture()
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
44, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
47, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
skips with clear instructions when Postgres is down."""
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9
assert summary is not None and summary.added == 13
yield
_reset_db(mock_llm, seed=False)
+3 -3
View File
@@ -148,11 +148,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
@pytest.fixture()
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
44, A9 formats), truncated again on teardown (same fixture shape as
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
47, A9 formats), truncated again on teardown (same fixture shape as
the phase-17/21 suites)."""
summary = _reset_db(mock_llm, seed=True)
assert summary is not None and summary.added == 9
assert summary is not None and summary.added == 13
yield
_reset_db(mock_llm, seed=False)
+3 -3
View File
@@ -17,8 +17,8 @@ The oversized documents are seeded directly via SQLAlchemy (a
``documents`` row + 2–3 ``chunks`` rows whose embeddings are the mock's
own deterministic bag-of-words vectors, so the question's live mock
embedding genuinely overlaps — no fixture files added:
``tests/fixtures/docs/`` stays at its 9 files (phase 44), other suites
pin ``summary.added == 9``).
``tests/fixtures/docs/`` stays at its 13 files (phase 47), other suites
pin ``summary.added == 13``).
"""
from __future__ import annotations
@@ -301,7 +301,7 @@ def test_small_document_path_unchanged(
path, byte-identical to before — no marker, kubernetes.md cited."""
_reset_db(None)
summary = _run_in_thread(_import_fixtures(mock_llm))
assert summary.added == 9 # A9 formats (phase 44 added tables.md)
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
bubble = _ask(page, app_url, SMALL_QUESTION)
expect(bubble).to_contain_text(SMALL_QUESTION, timeout=30_000)
+11
View File
@@ -0,0 +1,11 @@
# Quadlet volume: backing storage for the shared cache (btrfs loop on
# the app node — see the app-node playbook for the loop setup).
# RESE-VOLUME-SENTINEL-22cc
[Unit]
Description=Shared cache volume (btrfs loop, 32G)
[Volume]
Driver=local
Type=volume
Device=/var/lib/quadlet-volumes/cache
+51
View File
@@ -0,0 +1,51 @@
# Quadlet unit for the homelab compose stack gateway (Podman >= 4.9,
# quadlet >= 0.5). Kept in version control next to the playbook so a
# fresh reinstall of the app node reproduces the exact stack — the unit
# file is the source of truth, not the running container.
# RESE-QUADLET-SENTINEL-77aa
[Unit]
Description=Compose stack gateway (reverse proxy + metrics scrape)
Wants=lan.network
After=lan.network network-online.target
Documentation=https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
[Service]
Restart=always
RestartSec=5
TimeoutStartSec=300
MemoryMax=4G
CPUQuota=400%
# The gateway must come up before the scrape jobs expect it.
ExecStartPre=/usr/bin/test -f /etc/homelab/compose.env
Environment=COMPOSE_PROJECT_NAME=homelab
EnvironmentFile=/etc/homelab/compose.env
[Container]
Image=docker.io/reese/compose-gateway:1.4.2
ImageVolume=ignore
ContainerName=homelab-compose-gateway
Label=io.podman.quadlet.stack=homelab
Label=io.reese.maintained-by=ansible
PublishPort=127.0.0.1:8443:8443
PublishPort=127.0.0.1:9100:9100
Network=lan
Environment=LOG_LEVEL=info
Environment=METRICS_BIND=0.0.0.0:9100
Environment=UPSTREAM_REGISTRY=10.89.0.10:5000
Volume=/var/lib/compose-gateway/certs:/etc/certs:ro
Volume=/var/lib/compose-gateway/data:/data
Volume=/etc/homelab/compose.env:/etc/compose.env:ro
Exec=serve
# Health probe: the gateway answers /healthz on the metrics port.
HealthCmd=CURL -fsS http://127.0.0.1:9100/healthz
HealthInterval=30s
HealthStartPeriod=15s
HealthTimeout=5s
HealthRetries=3
StartTimeout=60s
# Keep the gateway on the lab bridge — no public interface exposure.
AdditionalCapabilities=CHOWN,SETGID,SETUID
[Install]
WantedBy=default.target
+13
View File
@@ -0,0 +1,13 @@
# Quadlet network: the flat lab bridge for all homelab services.
# RESE-NETWORK-SENTINEL-11bb
[Unit]
Description=Homelab lab network (bridge, 10.89.0.0/24)
[Network]
Driver=bridge
NetworkName=labnet
IPAMDriver=dhcp
Subnets=10.89.0.0/24
Gateway=10.89.0.1
Options=mtu=1500
+15
View File
@@ -0,0 +1,15 @@
{# Service unit template — rendered by the deploy playbook, one file per
host. The playbook loops the services dict from the group vars. #}
{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %}
{% for svc in services %}
[{{ svc.name }}]
host = {{ svc.host }}
port = {{ svc.port }}
enabled = {{ svc.enabled | default(true) }}
{% if svc.env is defined %}
{% for key, value in svc.env.items() %}
{{ key }} = {{ value }}
{% endfor %}
{% endif %}
{% endfor %}
# rendered by ansible on {{ inventory_hostname }} at {{ deployment_stamp }}
+1 -1
View File
@@ -47,7 +47,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
db.commit()
llm = FakeRagLLM()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
yield llm
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
+1 -1
View File
@@ -138,7 +138,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
db.commit()
llm = FakeRagLLM()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
yield llm
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
@@ -0,0 +1,135 @@
"""Integration test: the phase-47 quadlet + j2 formats ride the import
machinery unchanged (walk → delta → prune).
The temp source tree holds byte-identical copies of the real fixture
files (``tests/fixtures/docs/homelab/quadlet/*.container|.volume`` and
``templates/deploy.j2``), so this test tracks the fixtures even if their
bytes change. Runs against the local compose Postgres (the ``db`` fixture
from ``tests/conftest.py``) with the deterministic ``FakeEmbedder`` —
same harness as ``test_importer_e2e.py``. A separate file (not an
extension of that one) because the delta/prune lifecycle mutates the
tree between runs, while the fixture e2e stays a single import +
idempotent re-run over the shared fixture tree.
Runs (DB must be up: ``podman compose up -d db``):
uv run pytest tests/integration/test_import_quadlet_jinja.py -v
"""
from __future__ import annotations
import asyncio
from collections.abc import Iterator
from pathlib import Path
import pytest
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.models import Chunk, Document
from app.rag.importer import import_sources
from tests.fakes import FakeEmbedder
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs" / "homelab"
#: Fixture files copied into the temp source tree (a ``.container``, a
#: ``.volume``, and a ``.j2``).
QUADLET_RELS = ("quadlet/compose.container", "quadlet/cache.volume")
JINJA_RELS = ("templates/deploy.j2",)
ALL_RELS = QUADLET_RELS + JINJA_RELS
def _cleanup_source(db: Session, source: str) -> None:
for doc in db.scalars(select(Document).where(Document.source == source)).all():
db.delete(doc)
db.commit()
@pytest.fixture()
def source_dir(db: Session, tmp_path: Path) -> Iterator[Path]:
"""A temp source tree seeded with copies of the real fixture files."""
root = tmp_path / "quadsrc"
for rel in ALL_RELS:
target = root / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes((FIXTURES / rel).read_bytes())
try:
yield root
finally:
_cleanup_source(db, root.name)
def test_quadlet_and_jinja_import_delta_and_prune(source_dir: Path, db: Session) -> None:
llm = FakeEmbedder()
# --- fresh import: all three docs land with stem titles + chunks ---
s1 = asyncio.run(import_sources([source_dir], llm, prune=False, session=db))
assert (s1.files, s1.added, s1.unchanged, s1.updated, s1.pruned) == (3, 3, 0, 0, 0)
assert s1.formats == {"container": 1, "volume": 1, "j2": 1}
assert s1.errors == 0
docs = {
d.path: d
for d in db.scalars(select(Document).where(Document.source == source_dir.name)).all()
}
assert set(docs) == {
"quadlet/compose.container",
"quadlet/cache.volume",
"templates/deploy.j2",
}
# Non-markdown titles come from the file stem (a leading ``#`` is a
# comment in these formats, not a heading).
assert docs["quadlet/compose.container"].title == "compose"
assert docs["quadlet/cache.volume"].title == "cache"
assert docs["templates/deploy.j2"].title == "deploy"
for path, doc in docs.items():
content = [c for c in doc.chunks if not c.is_summary]
assert content, f"{path} has no content chunks"
assert all(
c.embedding is not None and len(c.embedding) == 768 for c in content
), f"{path} content chunks not embedded"
# Phase 30 parity: the new non-markdown formats also get their
# lite summary + one ``is_summary`` chunk.
assert doc.summary is not None, f"{path} should have a summary"
schunks = [c for c in doc.chunks if c.is_summary]
assert len(schunks) == 1 and schunks[0].position == -1
# --- delta: change only the .j2 → it updates, the others stay ---
j2 = source_dir / "templates" / "deploy.j2"
j2.write_text(j2.read_text(encoding="utf-8") + "\n# RESE-JINJA-DELTA-CHANGED\n")
s2 = asyncio.run(import_sources([source_dir], llm, session=db))
assert (s2.added, s2.updated, s2.unchanged, s2.pruned) == (0, 1, 2, 0)
changed = db.scalar(
select(Document).where(
Document.source == source_dir.name, Document.path == "templates/deploy.j2"
)
)
assert changed is not None and "RESE-JINJA-DELTA-CHANGED" in changed.content
# --- idempotent: a clean re-run re-embeds nothing ---
calls_before = len(llm.calls)
s3 = asyncio.run(import_sources([source_dir], llm, session=db))
assert (s3.added, s3.updated, s3.unchanged, s3.pruned) == (0, 0, 3, 0)
assert len(llm.calls) == calls_before # unchanged → no embedding requests
# --- prune: delete the .volume → its row + chunks cascade away ---
(source_dir / "quadlet" / "cache.volume").unlink()
s4 = asyncio.run(import_sources([source_dir], llm, session=db, prune=True))
assert (s4.unchanged, s4.pruned) == (2, 1)
assert db.scalar(
select(Document).where(
Document.source == source_dir.name, Document.path == "quadlet/cache.volume"
)
) is None
# Its chunks (content + summary) are gone — FK cascade.
assert db.scalar(
select(func.count())
.select_from(Chunk)
.join(Document, Document.id == Chunk.document_id)
.where(
Document.source == source_dir.name,
Document.path == "quadlet/cache.volume",
)
) == 0
# The other two docs survived the prune.
assert db.scalar(
select(func.count()).select_from(Document).where(Document.source == source_dir.name)
) == 2
+23 -8
View File
@@ -28,6 +28,11 @@ EXPECTED_DOCS = {
("docs", "homelab/scripts/uptime_probe.py"),
("docs", "homelab/ssh/ssh_aliases.txt"),
("docs", "homelab/tables.md"), # phase 44: the markdown-tables fixture
# phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures
("docs", "homelab/quadlet/compose.container"),
("docs", "homelab/quadlet/lan.network"),
("docs", "homelab/quadlet/cache.volume"),
("docs", "homelab/templates/deploy.j2"),
}
@@ -37,13 +42,19 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
llm = FakeEmbedder()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
# Nine A9-format files (phase 44 added homelab/tables.md);
# .hidden/junk.md is out of scope (A9 revised).
assert (summary.files, summary.added, summary.unchanged) == (9, 9, 0)
assert summary.chunks >= 9
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
# Thirteen A9-format files (phase 44 added homelab/tables.md, phase 47
# added the quadlet + j2 fixtures); .hidden/junk.md is out of scope
# (A9 revised).
assert (summary.files, summary.added, summary.unchanged) == (13, 13, 0)
assert summary.chunks >= 13
assert summary.formats == {
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
}
# PLAN §9 per-format summary line: highest count first, then alpha.
assert summary.format_counts() == "md:5,json:1,py:1,txt:1,yaml:1"
assert summary.format_counts() == (
"md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1"
)
docs = db.scalars(select(Document)).all()
assert {(d.source, d.path) for d in docs} == EXPECTED_DOCS
@@ -57,6 +68,10 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
assert titles["homelab/scripts/uptime_probe.py"] == "uptime_probe"
assert titles["homelab/networking/static-dns.json"] == "static-dns"
assert titles["homelab/ssh/ssh_aliases.txt"] == "ssh_aliases"
assert titles["homelab/quadlet/compose.container"] == "compose"
assert titles["homelab/quadlet/lan.network"] == "lan"
assert titles["homelab/quadlet/cache.volume"] == "cache"
assert titles["homelab/templates/deploy.j2"] == "deploy"
# Hidden junk was never imported.
assert not any(".hidden" in d.path for d in docs)
# Full content is stored — that is what the RAG context will be.
@@ -92,13 +107,13 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
r = admin_client.get("/api/docs") # phase 16: the catalog is admin-only
assert r.status_code == 200
body = r.json()
assert len(body["documents"]) == 9
assert len(body["documents"]) == 13
assert all(d["chunks"] >= 1 for d in body["documents"])
# Idempotent re-run: nothing re-embedded.
calls_before = len(llm.calls)
s2 = asyncio.run(import_sources([FIXTURES], llm, session=db))
assert s2.unchanged == 9 and s2.added == 0
assert s2.unchanged == 13 and s2.added == 0
assert len(llm.calls) == calls_before # unchanged → no embedding requests
db.execute(text("TRUNCATE chunks, documents, query_log"))
+1 -1
View File
@@ -67,7 +67,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
db.commit()
llm = FakeRagLLM()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
yield llm
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview"))
db.commit()
+1 -1
View File
@@ -51,7 +51,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
db.commit()
llm = FakeRagLLM()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
yield llm
db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes"))
db.commit()
+60
View File
@@ -7,6 +7,7 @@ unchanged); the per-format tests cover the phase-09 dispatcher
from __future__ import annotations
from itertools import pairwise
from pathlib import Path
import pytest
@@ -361,6 +362,63 @@ def test_txt_long_doc_packs_with_overlap() -> None:
assert all(f"para {i}" in "\n".join(chunks) for i in range(6))
# ---------------------------------------------------------------------------
# quadlet family + j2 (A9 revised 2026-08-27) — plain-text dispatch
# ---------------------------------------------------------------------------
#: The ten suffixes added 2026-08-27 (owner decision R1): the full Podman
#: quadlet family + Jinja templates, all chunked as plain text.
NEW_FORMAT_SUFFIXES = [
".container", ".network", ".volume", ".image", ".pod",
".kube", ".swap", ".os", ".endpoint", ".j2",
]
REPO_ROOT = Path(__file__).resolve().parents[2]
FIXTURE_DOCS = REPO_ROOT / "tests" / "fixtures" / "docs" / "homelab"
@pytest.mark.parametrize("suffix", NEW_FORMAT_SUFFIXES)
def test_dispatch_new_suffixes_match_plain_text(suffix: str) -> None:
"""Every new suffix dispatches to plain-text paragraph packing — the
chunks are identical to a direct ``chunk_text`` call."""
content = "[Unit]\nDescription=one\n\n[Service]\nRestart=always\n\n[Container]\nImage=busybox\n"
name = suffix.lstrip(".")
assert chunk_document(content, f"x/{name}{suffix}") == chunk_text(content)
def test_container_fixture_chunks_past_single_paragraph_pack() -> None:
"""The realistic quadlet TOML fixture (≥1 500 chars) splits into ≥2
chunks, every chunk stays under the hard cap, and the sentinel token
survives the split."""
rel = "quadlet/compose.container"
content = (FIXTURE_DOCS / rel).read_text(encoding="utf-8")
assert len(content) >= 1500 # the fixture must exercise sub-splitting
chunks = chunk_document(content, rel)
assert len(chunks) >= 2
assert all(len(c) <= HARD_MAX_CHARS for c in chunks)
assert any("RESE-QUADLET-SENTINEL-77aa" in c for c in chunks)
joined = "\n".join(chunks)
assert "[Container]" in joined and "Restart=always" in joined
def test_j2_fixture_chunks_braces_as_plain_text() -> None:
"""Jinja braces are just text — no special handling; the ``{{ … }}``
line and the ``{% set %}`` sentinel appear verbatim in the chunks."""
rel = "templates/deploy.j2"
content = (FIXTURE_DOCS / rel).read_text(encoding="utf-8")
chunks = chunk_document(content, rel)
assert chunks
assert all(len(c) <= HARD_MAX_CHARS for c in chunks)
joined = "\n".join(chunks)
assert "RESE-JINJA-SENTINEL-33dd" in joined
assert any("[{{ svc.name }}]" in c for c in chunks)
def test_dispatch_still_falls_back_for_unknown_suffix() -> None:
content = "alpha\n\nbeta\n"
assert chunk_document(content, "x/notes.whatever") == chunk_text(content)
# ---------------------------------------------------------------------------
# Hard cap across every format (aipi ~1024-token request cap)
# ---------------------------------------------------------------------------
@@ -374,6 +432,8 @@ def test_txt_long_doc_packs_with_overlap() -> None:
('{"blob": "' + "z" * 5000 + '"}', "big.json"),
("def f():\n" + " x = 1\n" * 1000, "big.py"),
("line of text\n\n" * 800, "big.txt"),
("[Unit]\n" + "key=" + "v" * 5000 + "\n", "big.container"), # phase 47
("{% for x in y %}" + "{{ x }} " * 400 + "{% endfor %}", "big.j2"), # phase 47
],
)
def test_hard_cap_holds_for_every_format(content: str, path: str) -> None:
+54 -3
View File
@@ -8,7 +8,7 @@ import pytest
from pydantic import ValidationError
from pydantic_settings import SettingsError
from app.config import Settings
from app.config import _ALLOWED_IMPORT_EXTENSIONS, Settings # pyright: ignore[reportPrivateUsage]
def _settings(**kwargs: Any) -> Settings:
@@ -42,12 +42,46 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non
# Phase 17: the model's thinking streams by default (kill-switch off).
assert s.stream_thinking is True
assert len(s.suggestions) >= 3
# A9 (revised): the import scope covers the seven A9 formats.
# A9 (revised 2026-08-27): the import scope covers all seventeen
# A9 formats (original seven + quadlet family + jinja).
assert s.import_extension_set == {
".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"
".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py",
".container", ".network", ".volume", ".image", ".pod",
".kube", ".swap", ".os", ".endpoint", ".j2",
}
NEW_A9_FORMATS = (
"container", "network", "volume", "image", "pod",
"kube", "swap", "os", "endpoint", "j2",
)
def test_allowed_import_extensions_contains_all_seventeen_formats() -> None:
"""The validator's base set is the full A9 set: the original seven
plus the ten added 2026-08-27 (quadlet family + ``j2``). The
never-widen contract bounds :py:data:`import_extensions` against
exactly this set."""
assert {
"md", "markdown", "txt", "yaml", "yml", "json", "py",
*NEW_A9_FORMATS,
} == _ALLOWED_IMPORT_EXTENSIONS
def test_default_import_extensions_include_the_ten_new_formats() -> None:
"""A9 revised 2026-08-27 (owner permission): the quadlet family +
``j2`` are imported by default — no env configuration needed — with
the original seven first (order is cosmetic, the set is what
matters)."""
s = _settings()
for ext in NEW_A9_FORMATS:
assert ext in s.import_extensions
assert s.import_extension_set == (
{".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"}
| {f".{ext}" for ext in NEW_A9_FORMATS}
)
def test_env_override(monkeypatch) -> None:
monkeypatch.setenv("BOR_RELEVANCE_THRESHOLD", "0.42")
monkeypatch.setenv("BOR_LLM_CHAT_MODEL", "juggernaut")
@@ -131,6 +165,23 @@ def test_import_extensions_rejects_empty(monkeypatch) -> None:
_settings()
def test_import_extensions_validator_accepts_new_a9_formats(monkeypatch) -> None:
"""A9 revised 2026-08-27: the new names are first-class — the
never-widen contract now holds against the widened base set, so a
narrowing CSV with quadlet/jinja names is accepted."""
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,container,j2")
s = _settings()
assert s.import_extension_set == {".md", ".container", ".j2"}
def test_import_extensions_validator_still_rejects_unknown(monkeypatch) -> None:
"""Truly unknown extensions still fail loudly at startup (the
validator is intact — only the allowed base set widened)."""
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,xyz")
with pytest.raises(ValidationError, match="xyz"):
_settings()
def test_git_sources_default_empty_and_sources_dir_default() -> None:
"""Phase 28: no git sources by default (backwards-compatible with the
``--source`` / ``DEFAULT_SOURCES`` fallback); the clone location stays
+126 -2
View File
@@ -18,6 +18,7 @@ from pathlib import Path
import pytest
from sqlalchemy import func, select
from app.config import Settings
from app.models import Chunk, Document
from app.rag.importer import (
EXCLUDED_DIRS,
@@ -29,8 +30,9 @@ from app.rag.importer import (
from app.rag.llm import EmbeddingError
from tests.fakes import FakeEmbedder
#: A9 default extension set as dotted suffixes (what the importer passes to
#: the walker when no override is configured).
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
#: set). The phase-47 walk test uses the live config default
#: (``Settings().import_extension_set`` — now seventeen formats).
DEFAULT_EXTS = frozenset({".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"})
@@ -121,6 +123,70 @@ def test_iter_importable_files_missing_dir_yields_nothing(tmp_path: Path) -> Non
assert iter_importable_files(tmp_path / "definitely-missing", DEFAULT_EXTS) == []
def test_iter_importable_files_walker_picks_up_new_formats_by_default(
tmp_path: Path,
) -> None:
"""Phase 47 (A9 revised 2026-08-27): the ten new formats — the full
quadlet family + j2 — walk under the *default* extension set (no env
configuration needed), the original seven still walk (regression), and
unknown/hidden-dir/excluded-dir filtering is unchanged for them."""
root = tmp_path / "quadproj"
(root / ".esphome").mkdir(parents=True)
(root / "node_modules").mkdir()
files = {
# the ten new formats (A9 revised 2026-08-27):
"web.container": "# RESE-QUADLET-SENTINEL-77aa\n[Container]\nImage=alpine\n",
"lan.network": "[Network]\nDriver=bridge\n",
"cache.volume": "[Volume]\nDriver=local\n",
"alpine.image": "[Image]\nImages=alpine\n",
"app.pod": "[Pod]\nPodName=app\n",
"k3s.kube": "apiVersion: v1\nkind: Pod\n",
"zram.swap": "[Swap]\nFile=/swapfile\n",
"fedora.os": "[OS]\nImage=fedora\n",
"edge.endpoint": "[Endpoint]\nPort=8080\n",
"deploy.j2": "{% for s in services %}\n[{{ s }}]\n{% endfor %}\n",
# the original seven — regression:
"note.md": "# n\n",
"note.markdown": "# m\n",
"note.txt": "t\n",
"cfg.yaml": "a: b\n",
"cfg.yml": "a: b\n",
"data.json": "{}\n",
"agent.py": "x = 1\n",
# must be skipped — filtering is unchanged for the new formats too:
"mystery.xyz": "unknown extension",
".esphome/x.container": "hidden dir (new-format file)",
"node_modules/y.container": "excluded dir (new-format file)",
}
for rel, text in files.items():
(root / rel).write_text(text)
# The live config default (Settings with no env override) — no
# BOR_IMPORT_EXTENSIONS needed for the new formats to walk.
exts = Settings(_env_file=None).import_extension_set # pyright: ignore[reportCallIssue]
found = {p.relative_to(root).as_posix() for p in iter_importable_files(root, exts)}
assert found == {
# all ten new formats:
"web.container",
"lan.network",
"cache.volume",
"alpine.image",
"app.pod",
"k3s.kube",
"zram.swap",
"fedora.os",
"edge.endpoint",
"deploy.j2",
# …and the original seven:
"note.md",
"note.markdown",
"note.txt",
"cfg.yaml",
"cfg.yml",
"data.json",
"agent.py",
}
def test_iter_importable_files_respects_custom_extension_filter(tmp_path: Path) -> None:
"""A narrower filter (e.g. md only) excludes the other A9 formats."""
root = tmp_path / "filtered"
@@ -358,6 +424,64 @@ def test_multi_format_import_counts_per_format_and_titles_stem(db, tmp_path: Pat
_cleanup_source(db, root.name)
def test_quadlet_and_j2_files_get_stem_titles_and_per_format_counts(
db, tmp_path: Path
) -> None:
"""Phase 47: the ten new formats import like any other A9 format — the
per-format counts land in the summary, every file yields content
chunks, and a leading ``#`` line (a TOML comment in quadlet files, not
a heading) never becomes the title: the file stem wins."""
root = tmp_path / "quadsrc"
(root / "quadlet").mkdir(parents=True)
(root / "tpl").mkdir(parents=True)
(root / "quadlet" / "web.container").write_text(
"# RESE-QUADLET-SENTINEL-77aa\n\n[Container]\nImage=alpine\n"
)
(root / "quadlet" / "lan.network").write_text("# net\n[Network]\nDriver=bridge\n")
(root / "quadlet" / "cache.volume").write_text("[Volume]\nDriver=local\n")
(root / "quadlet" / "alpine.image").write_text("[Image]\nImages=alpine\n")
(root / "quadlet" / "app.pod").write_text("[Pod]\nPodName=app\n")
(root / "quadlet" / "k3s.kube").write_text("apiVersion: v1\nkind: Pod\n")
(root / "quadlet" / "zram.swap").write_text("[Swap]\nFile=/swapfile\n")
(root / "quadlet" / "fedora.os").write_text("[OS]\nImage=fedora\n")
(root / "quadlet" / "edge.endpoint").write_text("[Endpoint]\nPort=8080\n")
(root / "tpl" / "deploy.j2").write_text(
"{% for s in services %}\n[{{ s }}]\nport = {{ s.port }}\n{% endfor %}\n"
)
llm = FakeEmbedder()
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert summary.files == 10 and summary.added == 10
assert summary.formats == {
"container": 1, "network": 1, "volume": 1, "image": 1, "pod": 1,
"kube": 1, "swap": 1, "os": 1, "endpoint": 1, "j2": 1,
}
docs = {
d.path: d
for d in db.scalars(select(Document).where(Document.source == root.name)).all()
}
titles = {p: d.title for p, d in docs.items()}
assert titles == {
# the ``#`` line is a comment in these formats — stem titles:
"quadlet/web.container": "web",
"quadlet/lan.network": "lan",
"quadlet/cache.volume": "cache",
"quadlet/alpine.image": "alpine",
"quadlet/app.pod": "app",
"quadlet/k3s.kube": "k3s",
"quadlet/zram.swap": "zram",
"quadlet/fedora.os": "fedora",
"quadlet/edge.endpoint": "edge",
"tpl/deploy.j2": "deploy",
}
for doc in docs.values():
content = [c for c in doc.chunks if not c.is_summary]
assert content # every new-format file yields content chunks
assert all(c.embedding is not None and len(c.embedding) == 768 for c in content)
finally:
_cleanup_source(db, root.name)
# ---------- phase 30: lite-model summaries for non-markdown files ----------