feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.
- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
import_extensions CSV gain the ten names (the original seven first);
the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
(HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
default CSV + dotted import_extension_set include the ten, the
validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
new suffix (parametrized), the .container fixture chunks >=2 under
the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
tree indexes exactly the ten new files (unknown/hidden/excluded
filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
over a temp tree with .container/.volume/.j2 -> documents + chunks
rows with stem titles; delta re-import updates only the changed .j2
doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
isolation): GET /api/docs (admin session) lists the four new-format
docs with non-zero chunk counts and stem titles; the Sources table
renders a row + .doc-link per file; the phase-26 modal shows the
.container TOML ([Container] section + sentinel) with stem title and
the container format badge; a RESE-JINJA-SENTINEL-33dd question
FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
FTS hits) — the bubble is not .is-deflected and a source chip names
templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
the revision note under the anchors table, and the §5 chunking-policy
+ §11 workflow lines. The only PLAN edit this phase.
Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.
Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
+17
-5
@@ -56,7 +56,7 @@ that"* and offers alternatives instead of hallucinating.
|
||||
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
|
||||
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
|
||||
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
|
||||
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
|
||||
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-27) |
|
||||
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
|
||||
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
|
||||
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
|
||||
@@ -83,6 +83,13 @@ that"* and offers alternatives instead of hallucinating.
|
||||
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
|
||||
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
|
||||
> marker.
|
||||
>
|
||||
> **A9 revision (phase 47, owner permission 2026-08-27):** the format
|
||||
> set extends with the Podman quadlet family (`container, network,
|
||||
> volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja
|
||||
> templates) — plain-text chunking (`chunk_text`), owner: `TODO.md`
|
||||
> L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the
|
||||
> hidden-dir/exclusion invariants are unchanged.
|
||||
|
||||
---
|
||||
|
||||
@@ -251,8 +258,11 @@ preceding heading in the text for retrieval quality.
|
||||
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
|
||||
`---` separators (key line kept as anchor); `json` pretty-printed, split on
|
||||
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
|
||||
`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
|
||||
hard cap (aipi ~1024-token request limit).
|
||||
`txt` on paragraphs; markdown unchanged. The quadlet family
|
||||
(`container, network, volume, image, pod, kube, swap, os, endpoint`)
|
||||
and `j2` (Jinja templates) are plain-text chunked — no format-specific
|
||||
splitter (A9 revised 2026-08-27, phase 47). Every format honors the
|
||||
1200-char hard cap (aipi ~1024-token request limit).
|
||||
|
||||
---
|
||||
|
||||
@@ -455,8 +465,10 @@ uv run python -m scripts.llm_probe # sanity: models + dim
|
||||
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
|
||||
(no re-embedding); changed files are re-chunked + re-embedded (chunks
|
||||
replaced atomically); `--prune` removes docs whose files disappeared or no
|
||||
longer match the format filter. Formats per A9 (revised): `md, markdown,
|
||||
txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
|
||||
longer match the format filter. Formats per A9 (revised 2026-08-27):
|
||||
`md, markdown, txt, yaml, yml, json, py`, the quadlet family
|
||||
(`container, network, volume, image, pod, kube, swap, os, endpoint`), and
|
||||
`j2` (plain-text chunked) (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
|
||||
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
|
||||
ranks live hybrid results for a question (retrieval tuning).
|
||||
|
||||
|
||||
+1
-1
@@ -41,7 +41,7 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
|
||||
# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)
|
||||
|
||||
# --- Import scope (A9 formats; may only narrow, never widen) ---
|
||||
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py
|
||||
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2
|
||||
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips
|
||||
|
||||
# --- Import sources (git; phase 28, admin-managed since phase 35) ---
|
||||
|
||||
@@ -293,8 +293,11 @@ embedded.
|
||||
added=… updated=… unchanged=… pruned=… chunks=… embed_batches=…
|
||||
formats=md:203,yaml:267,…`), so it is safe to run from a cron job or
|
||||
after every commit.
|
||||
- Indexed formats (A9): **`md, markdown, txt, yaml, yml, json, py`**
|
||||
(case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). Any path with a
|
||||
- Indexed formats (A9, revised 2026-08-27): **`md, markdown, txt,
|
||||
yaml, yml, json, py`**, the Podman quadlet family (**`container,
|
||||
network, volume, image, pod, kube, swap, os, endpoint`**), and **`j2`**
|
||||
Jinja templates (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`).
|
||||
Any path with a
|
||||
**dot-prefixed component** — hidden files or vendored caches like
|
||||
`.esphome/.espressif/**` — is skipped, along with `.venv`,
|
||||
`node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`.
|
||||
@@ -302,7 +305,8 @@ embedded.
|
||||
that's how previously imported junk leaves the index.
|
||||
- Non-markdown files get format-aware chunking (YAML top-level keys /
|
||||
`---` docs, JSON top-level keys, Python top-level defs/classes via
|
||||
stdlib `ast`) and their title comes from the file stem.
|
||||
stdlib `ast`; quadlet unit files and `j2` templates are paragraph-
|
||||
packed as plain text) and their title comes from the file stem.
|
||||
- Unchanged files are **not re-embedded** — only new/changed ones, so
|
||||
refreshes are cheap.
|
||||
- After a run that **changed** the knowledge base (at least one document
|
||||
@@ -644,7 +648,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
||||
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
||||
| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn — every call the model emits consumes a round; at the cap the loop forces one final no-tools answer (0 = no tools, the kill switch) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
|
||||
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
|
||||
|
||||
+17
-5
@@ -11,10 +11,19 @@ from functools import lru_cache
|
||||
from pydantic import field_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
#: The A9 import formats (PLAN anchor A9, revised 2026-08-21).
|
||||
#: ``BOR_IMPORT_EXTENSIONS`` may narrow — but never widen — this set.
|
||||
#: The A9 import formats (PLAN anchor A9, revised 2026-08-21; revised
|
||||
#: 2026-08-27, owner permission — the full Podman quadlet family
|
||||
#: ``container, network, volume, image, pod, kube, swap, os, endpoint``
|
||||
#: plus Jinja templates ``j2`` join the allowed set, chunked as plain
|
||||
#: text). ``BOR_IMPORT_EXTENSIONS`` may narrow — but never widen — this
|
||||
#: set.
|
||||
_ALLOWED_IMPORT_EXTENSIONS: frozenset[str] = frozenset(
|
||||
{"md", "markdown", "txt", "yaml", "yml", "json", "py"}
|
||||
{
|
||||
"md", "markdown", "txt", "yaml", "yml", "json", "py",
|
||||
# A9 revised 2026-08-27 (owner permission): quadlet family + jinja.
|
||||
"container", "network", "volume", "image", "pod",
|
||||
"kube", "swap", "os", "endpoint", "j2",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -127,14 +136,17 @@ class Settings(BaseSettings):
|
||||
session_max_age: int = 43_200
|
||||
session_cookie: str = "bor_session"
|
||||
|
||||
# --- Import scope (A9, revised 2026-08-21) ---
|
||||
# --- Import scope (A9, revised 2026-08-21 and 2026-08-27) ---
|
||||
# Comma-separated list of lowercased file extensions (no dot) imported
|
||||
# by ``scripts/import_docs.py``. Hidden (dot) path components are always
|
||||
# skipped, plus the importer's exclusion list.
|
||||
# Stored as a raw CSV string (env-native — no JSON) and parsed on demand
|
||||
# via :py:meth:`import_extension_set`. ``mode="after"`` validation runs
|
||||
# against the raw string so a typo fails loudly at startup.
|
||||
import_extensions: str = "md,markdown,txt,yaml,yml,json,py"
|
||||
import_extensions: str = (
|
||||
"md,markdown,txt,yaml,yml,json,py,"
|
||||
"container,network,volume,image,pod,kube,swap,os,endpoint,j2"
|
||||
)
|
||||
#: List of git repo URLs to clone/pull into ``sources_dir`` before
|
||||
#: indexing (phase 28); comma-separated, stored raw. Empty means no git
|
||||
#: sources — ``import_docs`` then falls back to ``--source`` / the old
|
||||
|
||||
+19
-1
@@ -22,6 +22,10 @@ per-format policies:
|
||||
module preamble — imports, constants — is its own block); an oversized
|
||||
definition falls back to line packing.
|
||||
* **txt** (and any unknown suffix) — paragraph packing.
|
||||
* **container / network / volume / image / pod / kube / swap / os /
|
||||
endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and
|
||||
Jinja templates; plain-text paragraph packing (``chunk_text``) — no
|
||||
format-specific splitter (owner decision).
|
||||
|
||||
Every format honors :data:`HARD_MAX_CHARS` (1200 — the aipi ~1024-token
|
||||
request cap) and the target/overlap settings; oversized blocks are split
|
||||
@@ -331,7 +335,8 @@ def _normalize_target_overlap(target_chars: int, overlap_chars: int) -> tuple[in
|
||||
return target, min(overlap_chars, target - 1)
|
||||
|
||||
|
||||
#: suffix → chunker (A9, revised: md, markdown, txt, yaml, yml, json, py).
|
||||
#: suffix → chunker (A9, revised 2026-08-21 and 2026-08-27: md, markdown,
|
||||
#: txt, yaml, yml, json, py + the quadlet family and j2 — plain text).
|
||||
_FORMAT_CHUNKERS = {
|
||||
".md": chunk_markdown,
|
||||
".markdown": chunk_markdown,
|
||||
@@ -340,6 +345,19 @@ _FORMAT_CHUNKERS = {
|
||||
".yml": chunk_yaml,
|
||||
".json": chunk_json,
|
||||
".py": chunk_python,
|
||||
# A9 revised 2026-08-27 (owner permission): the full Podman quadlet
|
||||
# family + Jinja templates — plain-text paragraph packing, no
|
||||
# TOML/Jinja-aware splitter (owner decision).
|
||||
".container": chunk_text,
|
||||
".network": chunk_text,
|
||||
".volume": chunk_text,
|
||||
".image": chunk_text,
|
||||
".pod": chunk_text,
|
||||
".kube": chunk_text,
|
||||
".swap": chunk_text,
|
||||
".os": chunk_text,
|
||||
".endpoint": chunk_text,
|
||||
".j2": chunk_text,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -208,10 +208,10 @@ def test_admin_login_unlocks_sources_and_tuning(
|
||||
login(page, app_url)
|
||||
expect(page).to_have_url(app_url + "/sources.html")
|
||||
expect(page.locator("#sources-gate")).to_be_hidden()
|
||||
expect(page.locator("#stat-docs")).to_have_text("9") # phase 44: +tables.md
|
||||
expect(page.locator("#stat-docs")).to_have_text("13") # phase 47: +quadlet/j2
|
||||
expect(page.locator("#stat-chunks")).not_to_have_text("–")
|
||||
expect(page.locator("#docs-table")).to_be_visible()
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(9)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(13)
|
||||
|
||||
# Chat: the tuning UI is back — header toggle with count badge,
|
||||
# Sign out instead of Sign in, Tune under the answer.
|
||||
|
||||
@@ -128,7 +128,7 @@ def test_conversation_survives_reload(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
page.goto(app_url)
|
||||
_ask(page, QUESTION)
|
||||
|
||||
@@ -76,7 +76,7 @@ def test_on_topic_question_streams_grounded_answer(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
page.goto(app_url)
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ def _seed_kb(mock_port: int) -> ImportSummary:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
summary = _run_in_thread(_import_fixtures(mock_port))
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
return summary
|
||||
|
||||
|
||||
|
||||
@@ -336,7 +336,7 @@ def test_edit_note_steers_answer(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
_open_tuning(page, app_url)
|
||||
|
||||
|
||||
@@ -84,7 +84,7 @@ def test_off_topic_question_deflects_honestly(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
page.goto(app_url)
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
@@ -41,6 +41,11 @@ EXPECTED_ROWS = (
|
||||
"homelab/scripts/uptime_probe.py",
|
||||
"homelab/ssh/ssh_aliases.txt",
|
||||
"homelab/tables.md", # phase 44: the markdown-tables fixture
|
||||
# phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures
|
||||
"homelab/quadlet/compose.container",
|
||||
"homelab/quadlet/lan.network",
|
||||
"homelab/quadlet/cache.volume",
|
||||
"homelab/templates/deploy.j2",
|
||||
)
|
||||
|
||||
|
||||
@@ -86,14 +91,18 @@ def test_sources_page_lists_indexed_docs(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
# Nine A9-format files are imported (phase 44 added homelab/tables.md);
|
||||
# Thirteen A9-format files are imported (phase 44 added
|
||||
# homelab/tables.md, phase 47 added the quadlet + j2 fixtures);
|
||||
# .hidden/junk.md is out of scope (A9 revised — hidden path components
|
||||
# are never walked).
|
||||
assert summary is not None and summary.added == 9
|
||||
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
|
||||
assert summary is not None and summary.added == 13
|
||||
assert summary.formats == {
|
||||
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
|
||||
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
|
||||
}
|
||||
|
||||
login(page, app_url) # phase 16: the catalog is admin-only
|
||||
expect(page.locator("#stat-docs")).to_have_text("9")
|
||||
expect(page.locator("#stat-docs")).to_have_text("13")
|
||||
# Phase 30: non-markdown fixtures each gained one ``is_summary`` chunk,
|
||||
# so the Sources total is content chunks + summary chunks.
|
||||
expect(page.locator("#stat-chunks")).to_have_text(
|
||||
|
||||
@@ -170,7 +170,7 @@ def test_on_topic_answer_echoes_kb_overview(
|
||||
the mock's echo of the ``<knowledge_base>`` section's first bullet —
|
||||
only possible if the section reached the LLM prompt."""
|
||||
summary = _reset_db(mock_llm, seed=True, overview=OVERVIEW)
|
||||
assert summary is not None and summary.added == 9 # phase 44 added tables.md
|
||||
assert summary is not None and summary.added == 13 # phase 47 added quadlet+j2
|
||||
assert _overview_row() is not None # the row the turn must inject
|
||||
|
||||
bubble = _ask(page, app_url, QUESTION, KB_ECHO)
|
||||
@@ -250,7 +250,7 @@ def test_mock_generated_outline_is_stored_and_echoed(
|
||||
stores the byte-stable 8-token digest of the generator's document
|
||||
list, and a chat turn echoes its first bullet."""
|
||||
summary = _reset_db(mock_llm, seed=True, overview=None)
|
||||
assert summary is not None and summary.added == 9 # phase 44 added tables.md
|
||||
assert summary is not None and summary.added == 13 # phase 47 added quadlet+j2
|
||||
assert _overview_row() is None # direct import never regenerates
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
|
||||
@@ -176,7 +176,7 @@ def test_typing_indicator_during_slow_think(
|
||||
"""AC1/AC5: the 3s mock warm-up must show the typing indicator for
|
||||
>=2s before any text appears, then it is gone once the answer lands."""
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
page.goto(app_url)
|
||||
|
||||
|
||||
@@ -106,7 +106,7 @@ def _run_in_thread(coro: Any) -> Any:
|
||||
|
||||
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
"""Truncate the KB (+ the global prompt-state rows), then optionally
|
||||
re-import the fixtures (9 docs since phase 44 added tables.md)."""
|
||||
re-import the fixtures (13 docs since phase 47 added quadlet+j2)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
@@ -151,7 +151,7 @@ def test_chat_table_renders(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # phase 44: +tables.md
|
||||
assert summary is not None and summary.added == 13 # phase 47: +quadlet/j2 fixtures
|
||||
page.set_default_timeout(30_000)
|
||||
bubble = _ask_table_answer(page, app_url)
|
||||
|
||||
|
||||
@@ -123,11 +123,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
|
||||
44, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
|
||||
47, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
|
||||
skips with clear instructions when Postgres is down."""
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9
|
||||
assert summary is not None and summary.added == 13
|
||||
yield
|
||||
_reset_db(mock_llm, seed=False)
|
||||
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
"""Phase 47 E2E (Playwright): quadlet unit files + Jinja templates ride the
|
||||
A9 import path end to end (A9 revised 2026-08-27, owner permission).
|
||||
|
||||
Story: ``.agent/user_stories/quadlet-jinja-import.md``
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_quadlet_jinja_import.py -v --no-cov
|
||||
|
||||
Seeding reuses the real import function against ``tests/fixtures/docs/``
|
||||
with the deterministic mock embeddings — the ``test_import_documents.py``
|
||||
pattern (truncate the KB, ``import_sources`` in a worker thread). This
|
||||
suite's re-import changes the KB for the session; it is run in isolation
|
||||
(A16), so there is no cross-suite interference.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_quadlet_and_jinja_indexed`` — ``GET /api/docs`` (admin session)
|
||||
lists the four new-format fixtures with non-zero chunk counts and the
|
||||
file stem as title — no env configuration needed (default set).
|
||||
2. ``test_sources_table_shows_them`` (admin) — the Sources table renders
|
||||
a row per new file, each with its ``.doc-link`` path link.
|
||||
3. ``test_container_content_viewable`` — the phase-26 modal shows the
|
||||
``.container`` file's TOML content (``[Container]`` section + sentinel)
|
||||
with the stem as title.
|
||||
4. ``test_jinja_retrievable_not_deflected`` — a question carrying the
|
||||
``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine
|
||||
below threshold **and** zero FTS hits) → honest-positive: the answer
|
||||
bubble is not ``.is-deflected`` and a source chip names
|
||||
``templates/deploy.j2``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from e2e.auth_helpers import login
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
|
||||
#: The four new-format fixtures (A9 revised 2026-08-27): path → stem title
|
||||
#: (no H1 → ``extract_title`` falls back to the file stem).
|
||||
NEW_FORMAT_DOCS = {
|
||||
"homelab/quadlet/compose.container": "compose",
|
||||
"homelab/quadlet/lan.network": "lan",
|
||||
"homelab/quadlet/cache.volume": "cache",
|
||||
"homelab/templates/deploy.j2": "deploy",
|
||||
}
|
||||
|
||||
#: Carries the ``.j2`` fixture's sentinel — the hyphens split into the
|
||||
#: ``rese | jinja | sentinel | 33dd`` tsquery tokens that FTS-match the
|
||||
#: chunk holding ``{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %}``.
|
||||
JINJA_QUESTION = "What is RESE-JINJA-SENTINEL-33dd?"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
|
||||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return await import_sources([FIXTURES], LLMClient(settings))
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread.
|
||||
|
||||
Playwright's sync API keeps an asyncio loop running on the test thread,
|
||||
so ``asyncio.run`` cannot be called directly from a test body.
|
||||
"""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
t = Thread(target=runner)
|
||||
t.start()
|
||||
t.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
"""Truncate the KB (and query log), then optionally re-import fixtures."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
if not seed:
|
||||
return None
|
||||
return _run_in_thread(_import_fixtures(mock_port))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Default-extensions import indexes the new formats (API view)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_quadlet_and_jinja_indexed(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None
|
||||
# The four fixtures joined the default set (A9 revised 2026-08-27) —
|
||||
# the import needed no BOR_IMPORT_EXTENSIONS configuration at all.
|
||||
assert summary.added == 13, f"expected all 13 fixture docs, added {summary.added}"
|
||||
|
||||
login(page, app_url) # phase 16: the catalog is admin-only
|
||||
r = page.request.get(f"{app_url}/api/docs")
|
||||
assert r.status == 200
|
||||
by_path = {d["path"]: d for d in r.json()["documents"]}
|
||||
for path, stem in NEW_FORMAT_DOCS.items():
|
||||
doc = by_path.get(path)
|
||||
assert doc is not None, f"{path} missing from GET /api/docs"
|
||||
assert doc["chunks"] > 0, f"{path} indexed zero chunks"
|
||||
# No H1 → the stem is the title, exactly like the other
|
||||
# non-markdown formats.
|
||||
assert doc["title"] == stem, f"{path} title {doc['title']!r}, want stem {stem!r}"
|
||||
assert doc["source"] == "docs"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The Sources table renders a row (with path link) per new file
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sources_table_shows_them(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None:
|
||||
_reset_db(mock_llm, seed=True)
|
||||
login(page, app_url) # phase 16: the Sources catalog is admin-only
|
||||
for path in NEW_FORMAT_DOCS:
|
||||
row = page.locator("#docs-tbody tr", has_text=path)
|
||||
expect(row).to_have_count(1)
|
||||
link = row.locator("td:nth-child(2) a.doc-link")
|
||||
expect(link).to_have_count(1)
|
||||
# The full path is the link's accessible context (ellipsis is
|
||||
# visual only) — the same contract the other rows carry.
|
||||
expect(link).to_have_attribute("title", path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The .container file's TOML is viewable in the modal (stem title)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_container_content_viewable(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db(mock_llm, seed=True)
|
||||
login(page, app_url) # phase 16: the Sources catalog is admin-only
|
||||
|
||||
row = page.locator("#docs-tbody tr", has_text="homelab/quadlet/compose.container")
|
||||
expect(row).to_have_count(1)
|
||||
link = row.locator("td:nth-child(2) a.doc-link")
|
||||
expect(link).to_have_attribute(
|
||||
"href", "/document.html?source=docs&path=homelab%2Fquadlet%2Fcompose.container"
|
||||
)
|
||||
expect(link).not_to_have_attribute("target")
|
||||
|
||||
link.click()
|
||||
expect(page.locator(".doc-modal")).to_be_visible()
|
||||
# Stem title (no H1 in a quadlet unit file) + the extension badge.
|
||||
expect(page.locator("#doc-modal-title")).to_have_text("compose")
|
||||
expect(page.locator("#doc-modal-meta .format-badge")).to_have_text("container")
|
||||
# Non-markdown content renders as escaped monospace text in a pre —
|
||||
# the whole TOML, [Container] section and sentinel included.
|
||||
pre = page.locator("#doc-modal-content pre.doc-raw")
|
||||
expect(pre).to_have_count(1)
|
||||
expect(pre).to_contain_text("[Container]")
|
||||
expect(pre).to_contain_text("Image=docker.io/reese/compose-gateway:1.4.2")
|
||||
expect(pre).to_contain_text("RESE-QUADLET-SENTINEL-77aa")
|
||||
# Still on the Sources page: the modal is same-page (phase 26).
|
||||
assert page.url == app_url + "/sources.html"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. A .j2 sentinel question is retrievable and honest-positive (A8)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_jinja_retrievable_not_deflected(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db(mock_llm, seed=True)
|
||||
page.set_default_timeout(30_000)
|
||||
page.goto(app_url)
|
||||
page.fill("#message-input", JINJA_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
# The done event appends source chips — waiting on the .j2 chip means
|
||||
# the turn is finished and the retrieval doc reached the UI.
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
|
||||
# A8: LOW requires best cosine < threshold AND zero FTS hits — the
|
||||
# question's sentinel tokens FTS-match the .j2 chunk, so the gate is
|
||||
# honest-positive whatever the mock's cosine says.
|
||||
expect(page.locator(".msg.brain")).to_have_count(1)
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
@@ -84,13 +84,17 @@ def _ask(page: Page, message: str) -> None:
|
||||
def test_multi_format_import_hidden_doc_excluded(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""A9 (revised): all seven formats import; hidden (dot) paths never do."""
|
||||
"""A9 (revised): all seventeen formats import; hidden (dot) paths never do."""
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None
|
||||
# Nine A9-format fixture files (phase 44 added homelab/tables.md);
|
||||
# .hidden/junk.md must never be walked.
|
||||
assert summary.added == 9
|
||||
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
|
||||
# Thirteen A9-format fixture files (phase 44 added homelab/tables.md,
|
||||
# phase 47 added the quadlet + j2 fixtures); .hidden/junk.md must never
|
||||
# be walked.
|
||||
assert summary.added == 13
|
||||
assert summary.formats == {
|
||||
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
|
||||
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
|
||||
}
|
||||
|
||||
# Phase 16: the catalog is admin-only — perform the real form login,
|
||||
# then call the API with the signed cookie the browser now holds.
|
||||
@@ -103,7 +107,7 @@ def test_multi_format_import_hidden_doc_excluded(
|
||||
r = httpx.get(f"{app_url}/api/docs", timeout=10, cookies=cookies)
|
||||
assert r.status_code == 200
|
||||
docs = r.json()["documents"]
|
||||
assert len(docs) == 9
|
||||
assert len(docs) == 13
|
||||
assert all(".hidden" not in d["path"] for d in docs)
|
||||
assert {d["path"] for d in docs} >= {
|
||||
"homelab/container_gitlab/gitlab.md",
|
||||
@@ -114,7 +118,7 @@ def test_multi_format_import_hidden_doc_excluded(
|
||||
}
|
||||
|
||||
# The Sources page (we're already on it, signed in) reflects the set.
|
||||
expect(page.locator("#stat-docs")).to_have_text("9")
|
||||
expect(page.locator("#stat-docs")).to_have_text("13")
|
||||
expect(page.locator("#docs-tbody tr", has_text=".hidden")).to_have_count(0)
|
||||
|
||||
|
||||
|
||||
@@ -154,11 +154,11 @@ def _no_error_banner(page: Page) -> None:
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
|
||||
44, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
|
||||
47, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
|
||||
skips with clear instructions when Postgres is down."""
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9
|
||||
assert summary is not None and summary.added == 13
|
||||
yield
|
||||
_reset_db(mock_llm, seed=False)
|
||||
|
||||
|
||||
@@ -127,7 +127,7 @@ def test_tune_under_answer_persists_and_steers(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
page.set_default_timeout(30_000)
|
||||
page.goto(app_url)
|
||||
login(page, app_url, next="/") # phase 16: tuning is admin-only
|
||||
|
||||
@@ -63,7 +63,7 @@ def _seed_kb(mock_port: int) -> ImportSummary:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
summary = _run_in_thread(_import_fixtures(mock_port))
|
||||
assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
return summary
|
||||
|
||||
|
||||
|
||||
@@ -99,11 +99,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
|
||||
44, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
|
||||
47, A9 formats), truncated again on teardown. ``db_ready`` (conftest)
|
||||
skips with clear instructions when Postgres is down."""
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9
|
||||
assert summary is not None and summary.added == 13
|
||||
yield
|
||||
_reset_db(mock_llm, seed=False)
|
||||
|
||||
|
||||
@@ -148,11 +148,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
|
||||
@pytest.fixture()
|
||||
def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]:
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase
|
||||
44, A9 formats), truncated again on teardown (same fixture shape as
|
||||
"""A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase
|
||||
47, A9 formats), truncated again on teardown (same fixture shape as
|
||||
the phase-17/21 suites)."""
|
||||
summary = _reset_db(mock_llm, seed=True)
|
||||
assert summary is not None and summary.added == 9
|
||||
assert summary is not None and summary.added == 13
|
||||
yield
|
||||
_reset_db(mock_llm, seed=False)
|
||||
|
||||
|
||||
@@ -17,8 +17,8 @@ The oversized documents are seeded directly via SQLAlchemy (a
|
||||
``documents`` row + 2–3 ``chunks`` rows whose embeddings are the mock's
|
||||
own deterministic bag-of-words vectors, so the question's live mock
|
||||
embedding genuinely overlaps — no fixture files added:
|
||||
``tests/fixtures/docs/`` stays at its 9 files (phase 44), other suites
|
||||
pin ``summary.added == 9``).
|
||||
``tests/fixtures/docs/`` stays at its 13 files (phase 47), other suites
|
||||
pin ``summary.added == 13``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -301,7 +301,7 @@ def test_small_document_path_unchanged(
|
||||
path, byte-identical to before — no marker, kubernetes.md cited."""
|
||||
_reset_db(None)
|
||||
summary = _run_in_thread(_import_fixtures(mock_llm))
|
||||
assert summary.added == 9 # A9 formats (phase 44 added tables.md)
|
||||
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||||
|
||||
bubble = _ask(page, app_url, SMALL_QUESTION)
|
||||
expect(bubble).to_contain_text(SMALL_QUESTION, timeout=30_000)
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Quadlet volume: backing storage for the shared cache (btrfs loop on
|
||||
# the app node — see the app-node playbook for the loop setup).
|
||||
# RESE-VOLUME-SENTINEL-22cc
|
||||
|
||||
[Unit]
|
||||
Description=Shared cache volume (btrfs loop, 32G)
|
||||
|
||||
[Volume]
|
||||
Driver=local
|
||||
Type=volume
|
||||
Device=/var/lib/quadlet-volumes/cache
|
||||
@@ -0,0 +1,51 @@
|
||||
# Quadlet unit for the homelab compose stack gateway (Podman >= 4.9,
|
||||
# quadlet >= 0.5). Kept in version control next to the playbook so a
|
||||
# fresh reinstall of the app node reproduces the exact stack — the unit
|
||||
# file is the source of truth, not the running container.
|
||||
# RESE-QUADLET-SENTINEL-77aa
|
||||
|
||||
[Unit]
|
||||
Description=Compose stack gateway (reverse proxy + metrics scrape)
|
||||
Wants=lan.network
|
||||
After=lan.network network-online.target
|
||||
Documentation=https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
TimeoutStartSec=300
|
||||
MemoryMax=4G
|
||||
CPUQuota=400%
|
||||
# The gateway must come up before the scrape jobs expect it.
|
||||
ExecStartPre=/usr/bin/test -f /etc/homelab/compose.env
|
||||
Environment=COMPOSE_PROJECT_NAME=homelab
|
||||
EnvironmentFile=/etc/homelab/compose.env
|
||||
|
||||
[Container]
|
||||
Image=docker.io/reese/compose-gateway:1.4.2
|
||||
ImageVolume=ignore
|
||||
ContainerName=homelab-compose-gateway
|
||||
Label=io.podman.quadlet.stack=homelab
|
||||
Label=io.reese.maintained-by=ansible
|
||||
PublishPort=127.0.0.1:8443:8443
|
||||
PublishPort=127.0.0.1:9100:9100
|
||||
Network=lan
|
||||
Environment=LOG_LEVEL=info
|
||||
Environment=METRICS_BIND=0.0.0.0:9100
|
||||
Environment=UPSTREAM_REGISTRY=10.89.0.10:5000
|
||||
Volume=/var/lib/compose-gateway/certs:/etc/certs:ro
|
||||
Volume=/var/lib/compose-gateway/data:/data
|
||||
Volume=/etc/homelab/compose.env:/etc/compose.env:ro
|
||||
Exec=serve
|
||||
# Health probe: the gateway answers /healthz on the metrics port.
|
||||
HealthCmd=CURL -fsS http://127.0.0.1:9100/healthz
|
||||
HealthInterval=30s
|
||||
HealthStartPeriod=15s
|
||||
HealthTimeout=5s
|
||||
HealthRetries=3
|
||||
StartTimeout=60s
|
||||
# Keep the gateway on the lab bridge — no public interface exposure.
|
||||
AdditionalCapabilities=CHOWN,SETGID,SETUID
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Quadlet network: the flat lab bridge for all homelab services.
|
||||
# RESE-NETWORK-SENTINEL-11bb
|
||||
|
||||
[Unit]
|
||||
Description=Homelab lab network (bridge, 10.89.0.0/24)
|
||||
|
||||
[Network]
|
||||
Driver=bridge
|
||||
NetworkName=labnet
|
||||
IPAMDriver=dhcp
|
||||
Subnets=10.89.0.0/24
|
||||
Gateway=10.89.0.1
|
||||
Options=mtu=1500
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{# Service unit template — rendered by the deploy playbook, one file per
|
||||
host. The playbook loops the services dict from the group vars. #}
|
||||
{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %}
|
||||
{% for svc in services %}
|
||||
[{{ svc.name }}]
|
||||
host = {{ svc.host }}
|
||||
port = {{ svc.port }}
|
||||
enabled = {{ svc.enabled | default(true) }}
|
||||
{% if svc.env is defined %}
|
||||
{% for key, value in svc.env.items() %}
|
||||
{{ key }} = {{ value }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
# rendered by ansible on {{ inventory_hostname }} at {{ deployment_stamp }}
|
||||
@@ -47,7 +47,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
|
||||
db.commit()
|
||||
llm = FakeRagLLM()
|
||||
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
|
||||
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
|
||||
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
|
||||
yield llm
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
|
||||
@@ -138,7 +138,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
|
||||
db.commit()
|
||||
llm = FakeRagLLM()
|
||||
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
|
||||
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
|
||||
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
|
||||
yield llm
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
db.commit()
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
"""Integration test: the phase-47 quadlet + j2 formats ride the import
|
||||
machinery unchanged (walk → delta → prune).
|
||||
|
||||
The temp source tree holds byte-identical copies of the real fixture
|
||||
files (``tests/fixtures/docs/homelab/quadlet/*.container|.volume`` and
|
||||
``templates/deploy.j2``), so this test tracks the fixtures even if their
|
||||
bytes change. Runs against the local compose Postgres (the ``db`` fixture
|
||||
from ``tests/conftest.py``) with the deterministic ``FakeEmbedder`` —
|
||||
same harness as ``test_importer_e2e.py``. A separate file (not an
|
||||
extension of that one) because the delta/prune lifecycle mutates the
|
||||
tree between runs, while the fixture e2e stays a single import +
|
||||
idempotent re-run over the shared fixture tree.
|
||||
|
||||
Runs (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/integration/test_import_quadlet_jinja.py -v
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs" / "homelab"
|
||||
|
||||
#: Fixture files copied into the temp source tree (a ``.container``, a
|
||||
#: ``.volume``, and a ``.j2``).
|
||||
QUADLET_RELS = ("quadlet/compose.container", "quadlet/cache.volume")
|
||||
JINJA_RELS = ("templates/deploy.j2",)
|
||||
ALL_RELS = QUADLET_RELS + JINJA_RELS
|
||||
|
||||
|
||||
def _cleanup_source(db: Session, source: str) -> None:
|
||||
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def source_dir(db: Session, tmp_path: Path) -> Iterator[Path]:
|
||||
"""A temp source tree seeded with copies of the real fixture files."""
|
||||
root = tmp_path / "quadsrc"
|
||||
for rel in ALL_RELS:
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes((FIXTURES / rel).read_bytes())
|
||||
try:
|
||||
yield root
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_quadlet_and_jinja_import_delta_and_prune(source_dir: Path, db: Session) -> None:
|
||||
llm = FakeEmbedder()
|
||||
|
||||
# --- fresh import: all three docs land with stem titles + chunks ---
|
||||
s1 = asyncio.run(import_sources([source_dir], llm, prune=False, session=db))
|
||||
assert (s1.files, s1.added, s1.unchanged, s1.updated, s1.pruned) == (3, 3, 0, 0, 0)
|
||||
assert s1.formats == {"container": 1, "volume": 1, "j2": 1}
|
||||
assert s1.errors == 0
|
||||
docs = {
|
||||
d.path: d
|
||||
for d in db.scalars(select(Document).where(Document.source == source_dir.name)).all()
|
||||
}
|
||||
assert set(docs) == {
|
||||
"quadlet/compose.container",
|
||||
"quadlet/cache.volume",
|
||||
"templates/deploy.j2",
|
||||
}
|
||||
# Non-markdown titles come from the file stem (a leading ``#`` is a
|
||||
# comment in these formats, not a heading).
|
||||
assert docs["quadlet/compose.container"].title == "compose"
|
||||
assert docs["quadlet/cache.volume"].title == "cache"
|
||||
assert docs["templates/deploy.j2"].title == "deploy"
|
||||
for path, doc in docs.items():
|
||||
content = [c for c in doc.chunks if not c.is_summary]
|
||||
assert content, f"{path} has no content chunks"
|
||||
assert all(
|
||||
c.embedding is not None and len(c.embedding) == 768 for c in content
|
||||
), f"{path} content chunks not embedded"
|
||||
# Phase 30 parity: the new non-markdown formats also get their
|
||||
# lite summary + one ``is_summary`` chunk.
|
||||
assert doc.summary is not None, f"{path} should have a summary"
|
||||
schunks = [c for c in doc.chunks if c.is_summary]
|
||||
assert len(schunks) == 1 and schunks[0].position == -1
|
||||
|
||||
# --- delta: change only the .j2 → it updates, the others stay ---
|
||||
j2 = source_dir / "templates" / "deploy.j2"
|
||||
j2.write_text(j2.read_text(encoding="utf-8") + "\n# RESE-JINJA-DELTA-CHANGED\n")
|
||||
s2 = asyncio.run(import_sources([source_dir], llm, session=db))
|
||||
assert (s2.added, s2.updated, s2.unchanged, s2.pruned) == (0, 1, 2, 0)
|
||||
changed = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_dir.name, Document.path == "templates/deploy.j2"
|
||||
)
|
||||
)
|
||||
assert changed is not None and "RESE-JINJA-DELTA-CHANGED" in changed.content
|
||||
|
||||
# --- idempotent: a clean re-run re-embeds nothing ---
|
||||
calls_before = len(llm.calls)
|
||||
s3 = asyncio.run(import_sources([source_dir], llm, session=db))
|
||||
assert (s3.added, s3.updated, s3.unchanged, s3.pruned) == (0, 0, 3, 0)
|
||||
assert len(llm.calls) == calls_before # unchanged → no embedding requests
|
||||
|
||||
# --- prune: delete the .volume → its row + chunks cascade away ---
|
||||
(source_dir / "quadlet" / "cache.volume").unlink()
|
||||
s4 = asyncio.run(import_sources([source_dir], llm, session=db, prune=True))
|
||||
assert (s4.unchanged, s4.pruned) == (2, 1)
|
||||
assert db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_dir.name, Document.path == "quadlet/cache.volume"
|
||||
)
|
||||
) is None
|
||||
# Its chunks (content + summary) are gone — FK cascade.
|
||||
assert db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
.where(
|
||||
Document.source == source_dir.name,
|
||||
Document.path == "quadlet/cache.volume",
|
||||
)
|
||||
) == 0
|
||||
# The other two docs survived the prune.
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Document).where(Document.source == source_dir.name)
|
||||
) == 2
|
||||
@@ -28,6 +28,11 @@ EXPECTED_DOCS = {
|
||||
("docs", "homelab/scripts/uptime_probe.py"),
|
||||
("docs", "homelab/ssh/ssh_aliases.txt"),
|
||||
("docs", "homelab/tables.md"), # phase 44: the markdown-tables fixture
|
||||
# phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures
|
||||
("docs", "homelab/quadlet/compose.container"),
|
||||
("docs", "homelab/quadlet/lan.network"),
|
||||
("docs", "homelab/quadlet/cache.volume"),
|
||||
("docs", "homelab/templates/deploy.j2"),
|
||||
}
|
||||
|
||||
|
||||
@@ -37,13 +42,19 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
|
||||
llm = FakeEmbedder()
|
||||
|
||||
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
|
||||
# Nine A9-format files (phase 44 added homelab/tables.md);
|
||||
# .hidden/junk.md is out of scope (A9 revised).
|
||||
assert (summary.files, summary.added, summary.unchanged) == (9, 9, 0)
|
||||
assert summary.chunks >= 9
|
||||
assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1}
|
||||
# Thirteen A9-format files (phase 44 added homelab/tables.md, phase 47
|
||||
# added the quadlet + j2 fixtures); .hidden/junk.md is out of scope
|
||||
# (A9 revised).
|
||||
assert (summary.files, summary.added, summary.unchanged) == (13, 13, 0)
|
||||
assert summary.chunks >= 13
|
||||
assert summary.formats == {
|
||||
"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1,
|
||||
"container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47
|
||||
}
|
||||
# PLAN §9 per-format summary line: highest count first, then alpha.
|
||||
assert summary.format_counts() == "md:5,json:1,py:1,txt:1,yaml:1"
|
||||
assert summary.format_counts() == (
|
||||
"md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1"
|
||||
)
|
||||
|
||||
docs = db.scalars(select(Document)).all()
|
||||
assert {(d.source, d.path) for d in docs} == EXPECTED_DOCS
|
||||
@@ -57,6 +68,10 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
|
||||
assert titles["homelab/scripts/uptime_probe.py"] == "uptime_probe"
|
||||
assert titles["homelab/networking/static-dns.json"] == "static-dns"
|
||||
assert titles["homelab/ssh/ssh_aliases.txt"] == "ssh_aliases"
|
||||
assert titles["homelab/quadlet/compose.container"] == "compose"
|
||||
assert titles["homelab/quadlet/lan.network"] == "lan"
|
||||
assert titles["homelab/quadlet/cache.volume"] == "cache"
|
||||
assert titles["homelab/templates/deploy.j2"] == "deploy"
|
||||
# Hidden junk was never imported.
|
||||
assert not any(".hidden" in d.path for d in docs)
|
||||
# Full content is stored — that is what the RAG context will be.
|
||||
@@ -92,13 +107,13 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
|
||||
r = admin_client.get("/api/docs") # phase 16: the catalog is admin-only
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert len(body["documents"]) == 9
|
||||
assert len(body["documents"]) == 13
|
||||
assert all(d["chunks"] >= 1 for d in body["documents"])
|
||||
|
||||
# Idempotent re-run: nothing re-embedded.
|
||||
calls_before = len(llm.calls)
|
||||
s2 = asyncio.run(import_sources([FIXTURES], llm, session=db))
|
||||
assert s2.unchanged == 9 and s2.added == 0
|
||||
assert s2.unchanged == 13 and s2.added == 0
|
||||
assert len(llm.calls) == calls_before # unchanged → no embedding requests
|
||||
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||
|
||||
@@ -67,7 +67,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
|
||||
db.commit()
|
||||
llm = FakeRagLLM()
|
||||
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
|
||||
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
|
||||
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
|
||||
yield llm
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview"))
|
||||
db.commit()
|
||||
|
||||
@@ -51,7 +51,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]:
|
||||
db.commit()
|
||||
llm = FakeRagLLM()
|
||||
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
|
||||
assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped
|
||||
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped
|
||||
yield llm
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes"))
|
||||
db.commit()
|
||||
|
||||
@@ -7,6 +7,7 @@ unchanged); the per-format tests cover the phase-09 dispatcher
|
||||
from __future__ import annotations
|
||||
|
||||
from itertools import pairwise
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -361,6 +362,63 @@ def test_txt_long_doc_packs_with_overlap() -> None:
|
||||
assert all(f"para {i}" in "\n".join(chunks) for i in range(6))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# quadlet family + j2 (A9 revised 2026-08-27) — plain-text dispatch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The ten suffixes added 2026-08-27 (owner decision R1): the full Podman
|
||||
#: quadlet family + Jinja templates, all chunked as plain text.
|
||||
NEW_FORMAT_SUFFIXES = [
|
||||
".container", ".network", ".volume", ".image", ".pod",
|
||||
".kube", ".swap", ".os", ".endpoint", ".j2",
|
||||
]
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
FIXTURE_DOCS = REPO_ROOT / "tests" / "fixtures" / "docs" / "homelab"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("suffix", NEW_FORMAT_SUFFIXES)
|
||||
def test_dispatch_new_suffixes_match_plain_text(suffix: str) -> None:
|
||||
"""Every new suffix dispatches to plain-text paragraph packing — the
|
||||
chunks are identical to a direct ``chunk_text`` call."""
|
||||
content = "[Unit]\nDescription=one\n\n[Service]\nRestart=always\n\n[Container]\nImage=busybox\n"
|
||||
name = suffix.lstrip(".")
|
||||
assert chunk_document(content, f"x/{name}{suffix}") == chunk_text(content)
|
||||
|
||||
|
||||
def test_container_fixture_chunks_past_single_paragraph_pack() -> None:
|
||||
"""The realistic quadlet TOML fixture (≥1 500 chars) splits into ≥2
|
||||
chunks, every chunk stays under the hard cap, and the sentinel token
|
||||
survives the split."""
|
||||
rel = "quadlet/compose.container"
|
||||
content = (FIXTURE_DOCS / rel).read_text(encoding="utf-8")
|
||||
assert len(content) >= 1500 # the fixture must exercise sub-splitting
|
||||
chunks = chunk_document(content, rel)
|
||||
assert len(chunks) >= 2
|
||||
assert all(len(c) <= HARD_MAX_CHARS for c in chunks)
|
||||
assert any("RESE-QUADLET-SENTINEL-77aa" in c for c in chunks)
|
||||
joined = "\n".join(chunks)
|
||||
assert "[Container]" in joined and "Restart=always" in joined
|
||||
|
||||
|
||||
def test_j2_fixture_chunks_braces_as_plain_text() -> None:
|
||||
"""Jinja braces are just text — no special handling; the ``{{ … }}``
|
||||
line and the ``{% set %}`` sentinel appear verbatim in the chunks."""
|
||||
rel = "templates/deploy.j2"
|
||||
content = (FIXTURE_DOCS / rel).read_text(encoding="utf-8")
|
||||
chunks = chunk_document(content, rel)
|
||||
assert chunks
|
||||
assert all(len(c) <= HARD_MAX_CHARS for c in chunks)
|
||||
joined = "\n".join(chunks)
|
||||
assert "RESE-JINJA-SENTINEL-33dd" in joined
|
||||
assert any("[{{ svc.name }}]" in c for c in chunks)
|
||||
|
||||
|
||||
def test_dispatch_still_falls_back_for_unknown_suffix() -> None:
|
||||
content = "alpha\n\nbeta\n"
|
||||
assert chunk_document(content, "x/notes.whatever") == chunk_text(content)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hard cap across every format (aipi ~1024-token request cap)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -374,6 +432,8 @@ def test_txt_long_doc_packs_with_overlap() -> None:
|
||||
('{"blob": "' + "z" * 5000 + '"}', "big.json"),
|
||||
("def f():\n" + " x = 1\n" * 1000, "big.py"),
|
||||
("line of text\n\n" * 800, "big.txt"),
|
||||
("[Unit]\n" + "key=" + "v" * 5000 + "\n", "big.container"), # phase 47
|
||||
("{% for x in y %}" + "{{ x }} " * 400 + "{% endfor %}", "big.j2"), # phase 47
|
||||
],
|
||||
)
|
||||
def test_hard_cap_holds_for_every_format(content: str, path: str) -> None:
|
||||
|
||||
@@ -8,7 +8,7 @@ import pytest
|
||||
from pydantic import ValidationError
|
||||
from pydantic_settings import SettingsError
|
||||
|
||||
from app.config import Settings
|
||||
from app.config import _ALLOWED_IMPORT_EXTENSIONS, Settings # pyright: ignore[reportPrivateUsage]
|
||||
|
||||
|
||||
def _settings(**kwargs: Any) -> Settings:
|
||||
@@ -42,12 +42,46 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non
|
||||
# Phase 17: the model's thinking streams by default (kill-switch off).
|
||||
assert s.stream_thinking is True
|
||||
assert len(s.suggestions) >= 3
|
||||
# A9 (revised): the import scope covers the seven A9 formats.
|
||||
# A9 (revised 2026-08-27): the import scope covers all seventeen
|
||||
# A9 formats (original seven + quadlet family + jinja).
|
||||
assert s.import_extension_set == {
|
||||
".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"
|
||||
".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py",
|
||||
".container", ".network", ".volume", ".image", ".pod",
|
||||
".kube", ".swap", ".os", ".endpoint", ".j2",
|
||||
}
|
||||
|
||||
|
||||
NEW_A9_FORMATS = (
|
||||
"container", "network", "volume", "image", "pod",
|
||||
"kube", "swap", "os", "endpoint", "j2",
|
||||
)
|
||||
|
||||
|
||||
def test_allowed_import_extensions_contains_all_seventeen_formats() -> None:
|
||||
"""The validator's base set is the full A9 set: the original seven
|
||||
plus the ten added 2026-08-27 (quadlet family + ``j2``). The
|
||||
never-widen contract bounds :py:data:`import_extensions` against
|
||||
exactly this set."""
|
||||
assert {
|
||||
"md", "markdown", "txt", "yaml", "yml", "json", "py",
|
||||
*NEW_A9_FORMATS,
|
||||
} == _ALLOWED_IMPORT_EXTENSIONS
|
||||
|
||||
|
||||
def test_default_import_extensions_include_the_ten_new_formats() -> None:
|
||||
"""A9 revised 2026-08-27 (owner permission): the quadlet family +
|
||||
``j2`` are imported by default — no env configuration needed — with
|
||||
the original seven first (order is cosmetic, the set is what
|
||||
matters)."""
|
||||
s = _settings()
|
||||
for ext in NEW_A9_FORMATS:
|
||||
assert ext in s.import_extensions
|
||||
assert s.import_extension_set == (
|
||||
{".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"}
|
||||
| {f".{ext}" for ext in NEW_A9_FORMATS}
|
||||
)
|
||||
|
||||
|
||||
def test_env_override(monkeypatch) -> None:
|
||||
monkeypatch.setenv("BOR_RELEVANCE_THRESHOLD", "0.42")
|
||||
monkeypatch.setenv("BOR_LLM_CHAT_MODEL", "juggernaut")
|
||||
@@ -131,6 +165,23 @@ def test_import_extensions_rejects_empty(monkeypatch) -> None:
|
||||
_settings()
|
||||
|
||||
|
||||
def test_import_extensions_validator_accepts_new_a9_formats(monkeypatch) -> None:
|
||||
"""A9 revised 2026-08-27: the new names are first-class — the
|
||||
never-widen contract now holds against the widened base set, so a
|
||||
narrowing CSV with quadlet/jinja names is accepted."""
|
||||
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,container,j2")
|
||||
s = _settings()
|
||||
assert s.import_extension_set == {".md", ".container", ".j2"}
|
||||
|
||||
|
||||
def test_import_extensions_validator_still_rejects_unknown(monkeypatch) -> None:
|
||||
"""Truly unknown extensions still fail loudly at startup (the
|
||||
validator is intact — only the allowed base set widened)."""
|
||||
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,xyz")
|
||||
with pytest.raises(ValidationError, match="xyz"):
|
||||
_settings()
|
||||
|
||||
|
||||
def test_git_sources_default_empty_and_sources_dir_default() -> None:
|
||||
"""Phase 28: no git sources by default (backwards-compatible with the
|
||||
``--source`` / ``DEFAULT_SOURCES`` fallback); the clone location stays
|
||||
|
||||
+126
-2
@@ -18,6 +18,7 @@ from pathlib import Path
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import (
|
||||
EXCLUDED_DIRS,
|
||||
@@ -29,8 +30,9 @@ from app.rag.importer import (
|
||||
from app.rag.llm import EmbeddingError
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: A9 default extension set as dotted suffixes (what the importer passes to
|
||||
#: the walker when no override is configured).
|
||||
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
|
||||
#: set). The phase-47 walk test uses the live config default
|
||||
#: (``Settings().import_extension_set`` — now seventeen formats).
|
||||
DEFAULT_EXTS = frozenset({".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"})
|
||||
|
||||
|
||||
@@ -121,6 +123,70 @@ def test_iter_importable_files_missing_dir_yields_nothing(tmp_path: Path) -> Non
|
||||
assert iter_importable_files(tmp_path / "definitely-missing", DEFAULT_EXTS) == []
|
||||
|
||||
|
||||
def test_iter_importable_files_walker_picks_up_new_formats_by_default(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 47 (A9 revised 2026-08-27): the ten new formats — the full
|
||||
quadlet family + j2 — walk under the *default* extension set (no env
|
||||
configuration needed), the original seven still walk (regression), and
|
||||
unknown/hidden-dir/excluded-dir filtering is unchanged for them."""
|
||||
root = tmp_path / "quadproj"
|
||||
(root / ".esphome").mkdir(parents=True)
|
||||
(root / "node_modules").mkdir()
|
||||
files = {
|
||||
# the ten new formats (A9 revised 2026-08-27):
|
||||
"web.container": "# RESE-QUADLET-SENTINEL-77aa\n[Container]\nImage=alpine\n",
|
||||
"lan.network": "[Network]\nDriver=bridge\n",
|
||||
"cache.volume": "[Volume]\nDriver=local\n",
|
||||
"alpine.image": "[Image]\nImages=alpine\n",
|
||||
"app.pod": "[Pod]\nPodName=app\n",
|
||||
"k3s.kube": "apiVersion: v1\nkind: Pod\n",
|
||||
"zram.swap": "[Swap]\nFile=/swapfile\n",
|
||||
"fedora.os": "[OS]\nImage=fedora\n",
|
||||
"edge.endpoint": "[Endpoint]\nPort=8080\n",
|
||||
"deploy.j2": "{% for s in services %}\n[{{ s }}]\n{% endfor %}\n",
|
||||
# the original seven — regression:
|
||||
"note.md": "# n\n",
|
||||
"note.markdown": "# m\n",
|
||||
"note.txt": "t\n",
|
||||
"cfg.yaml": "a: b\n",
|
||||
"cfg.yml": "a: b\n",
|
||||
"data.json": "{}\n",
|
||||
"agent.py": "x = 1\n",
|
||||
# must be skipped — filtering is unchanged for the new formats too:
|
||||
"mystery.xyz": "unknown extension",
|
||||
".esphome/x.container": "hidden dir (new-format file)",
|
||||
"node_modules/y.container": "excluded dir (new-format file)",
|
||||
}
|
||||
for rel, text in files.items():
|
||||
(root / rel).write_text(text)
|
||||
# The live config default (Settings with no env override) — no
|
||||
# BOR_IMPORT_EXTENSIONS needed for the new formats to walk.
|
||||
exts = Settings(_env_file=None).import_extension_set # pyright: ignore[reportCallIssue]
|
||||
found = {p.relative_to(root).as_posix() for p in iter_importable_files(root, exts)}
|
||||
assert found == {
|
||||
# all ten new formats:
|
||||
"web.container",
|
||||
"lan.network",
|
||||
"cache.volume",
|
||||
"alpine.image",
|
||||
"app.pod",
|
||||
"k3s.kube",
|
||||
"zram.swap",
|
||||
"fedora.os",
|
||||
"edge.endpoint",
|
||||
"deploy.j2",
|
||||
# …and the original seven:
|
||||
"note.md",
|
||||
"note.markdown",
|
||||
"note.txt",
|
||||
"cfg.yaml",
|
||||
"cfg.yml",
|
||||
"data.json",
|
||||
"agent.py",
|
||||
}
|
||||
|
||||
|
||||
def test_iter_importable_files_respects_custom_extension_filter(tmp_path: Path) -> None:
|
||||
"""A narrower filter (e.g. md only) excludes the other A9 formats."""
|
||||
root = tmp_path / "filtered"
|
||||
@@ -358,6 +424,64 @@ def test_multi_format_import_counts_per_format_and_titles_stem(db, tmp_path: Pat
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_quadlet_and_j2_files_get_stem_titles_and_per_format_counts(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 47: the ten new formats import like any other A9 format — the
|
||||
per-format counts land in the summary, every file yields content
|
||||
chunks, and a leading ``#`` line (a TOML comment in quadlet files, not
|
||||
a heading) never becomes the title: the file stem wins."""
|
||||
root = tmp_path / "quadsrc"
|
||||
(root / "quadlet").mkdir(parents=True)
|
||||
(root / "tpl").mkdir(parents=True)
|
||||
(root / "quadlet" / "web.container").write_text(
|
||||
"# RESE-QUADLET-SENTINEL-77aa\n\n[Container]\nImage=alpine\n"
|
||||
)
|
||||
(root / "quadlet" / "lan.network").write_text("# net\n[Network]\nDriver=bridge\n")
|
||||
(root / "quadlet" / "cache.volume").write_text("[Volume]\nDriver=local\n")
|
||||
(root / "quadlet" / "alpine.image").write_text("[Image]\nImages=alpine\n")
|
||||
(root / "quadlet" / "app.pod").write_text("[Pod]\nPodName=app\n")
|
||||
(root / "quadlet" / "k3s.kube").write_text("apiVersion: v1\nkind: Pod\n")
|
||||
(root / "quadlet" / "zram.swap").write_text("[Swap]\nFile=/swapfile\n")
|
||||
(root / "quadlet" / "fedora.os").write_text("[OS]\nImage=fedora\n")
|
||||
(root / "quadlet" / "edge.endpoint").write_text("[Endpoint]\nPort=8080\n")
|
||||
(root / "tpl" / "deploy.j2").write_text(
|
||||
"{% for s in services %}\n[{{ s }}]\nport = {{ s.port }}\n{% endfor %}\n"
|
||||
)
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert summary.files == 10 and summary.added == 10
|
||||
assert summary.formats == {
|
||||
"container": 1, "network": 1, "volume": 1, "image": 1, "pod": 1,
|
||||
"kube": 1, "swap": 1, "os": 1, "endpoint": 1, "j2": 1,
|
||||
}
|
||||
docs = {
|
||||
d.path: d
|
||||
for d in db.scalars(select(Document).where(Document.source == root.name)).all()
|
||||
}
|
||||
titles = {p: d.title for p, d in docs.items()}
|
||||
assert titles == {
|
||||
# the ``#`` line is a comment in these formats — stem titles:
|
||||
"quadlet/web.container": "web",
|
||||
"quadlet/lan.network": "lan",
|
||||
"quadlet/cache.volume": "cache",
|
||||
"quadlet/alpine.image": "alpine",
|
||||
"quadlet/app.pod": "app",
|
||||
"quadlet/k3s.kube": "k3s",
|
||||
"quadlet/zram.swap": "zram",
|
||||
"quadlet/fedora.os": "fedora",
|
||||
"quadlet/edge.endpoint": "edge",
|
||||
"tpl/deploy.j2": "deploy",
|
||||
}
|
||||
for doc in docs.values():
|
||||
content = [c for c in doc.chunks if not c.is_summary]
|
||||
assert content # every new-format file yields content chunks
|
||||
assert all(c.embedding is not None and len(c.embedding) == 768 for c in content)
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 30: lite-model summaries for non-markdown files ----------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user