diff --git a/.agent/PLAN.md b/.agent/PLAN.md index 9e507a4..6077354 100644 --- a/.agent/PLAN.md +++ b/.agent/PLAN.md @@ -56,7 +56,7 @@ that"* and offers alternatives instead of hallucinating. | A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED | | A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) | | A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) | -| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) | +| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-27) | | A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED | | A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED | | A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED | @@ -83,6 +83,13 @@ that"* and offers alternatives instead of hallucinating. > `BOR_MAX_CONTEXT_CHARS` is gone. The steering section > (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared > marker. +> +> **A9 revision (phase 47, owner permission 2026-08-27):** the format +> set extends with the Podman quadlet family (`container, network, +> volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja +> templates) — plain-text chunking (`chunk_text`), owner: `TODO.md` +> L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the +> hidden-dir/exclusion invariants are unchanged. --- @@ -251,8 +258,11 @@ preceding heading in the text for retrieval quality. **Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and `---` separators (key line kept as anchor); `json` pretty-printed, split on top-level keys; `py` split on top-level defs/classes (stdlib `ast`); -`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char -hard cap (aipi ~1024-token request limit). +`txt` on paragraphs; markdown unchanged. The quadlet family +(`container, network, volume, image, pod, kube, swap, os, endpoint`) +and `j2` (Jinja templates) are plain-text chunked — no format-specific +splitter (A9 revised 2026-08-27, phase 47). Every format honors the +1200-char hard cap (aipi ~1024-token request limit). --- @@ -455,8 +465,10 @@ uv run python -m scripts.llm_probe # sanity: models + dim Behavior: sha256 delta per `(source, path)` — unchanged files are skipped (no re-embedding); changed files are re-chunked + re-embedded (chunks replaced atomically); `--prune` removes docs whose files disappeared or no -longer match the format filter. Formats per A9 (revised): `md, markdown, -txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot) +longer match the format filter. Formats per A9 (revised 2026-08-27): +`md, markdown, txt, yaml, yml, json, py`, the quadlet family +(`container, network, volume, image, pod, kube, swap, os, endpoint`), and +`j2` (plain-text chunked) (`BOR_IMPORT_EXTENSIONS`), hidden (dot) directories skipped, exclusion list applied. `scripts/eval_retrieval.py` ranks live hybrid results for a question (retrieval tuning). diff --git a/.agent/phases/todo/47_quadlet_jinja_import/01_config_formats.md b/.agent/phases/complete/47_quadlet_jinja_import/01_config_formats.md similarity index 100% rename from .agent/phases/todo/47_quadlet_jinja_import/01_config_formats.md rename to .agent/phases/complete/47_quadlet_jinja_import/01_config_formats.md diff --git a/.agent/phases/todo/47_quadlet_jinja_import/02_chunker_dispatch_fixtures.md b/.agent/phases/complete/47_quadlet_jinja_import/02_chunker_dispatch_fixtures.md similarity index 100% rename from .agent/phases/todo/47_quadlet_jinja_import/02_chunker_dispatch_fixtures.md rename to .agent/phases/complete/47_quadlet_jinja_import/02_chunker_dispatch_fixtures.md diff --git a/.agent/phases/todo/47_quadlet_jinja_import/03_importer_integration.md b/.agent/phases/complete/47_quadlet_jinja_import/03_importer_integration.md similarity index 100% rename from .agent/phases/todo/47_quadlet_jinja_import/03_importer_integration.md rename to .agent/phases/complete/47_quadlet_jinja_import/03_importer_integration.md diff --git a/.env.example b/.env.example index e499f87..762f0df 100644 --- a/.env.example +++ b/.env.example @@ -41,7 +41,7 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant # BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools) # --- Import scope (A9 formats; may only narrow, never widen) --- -# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py +# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2 # BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips # --- Import sources (git; phase 28, admin-managed since phase 35) --- diff --git a/README.md b/README.md index 0bde75a..76e1134 100644 --- a/README.md +++ b/README.md @@ -293,8 +293,11 @@ embedded. added=… updated=… unchanged=… pruned=… chunks=… embed_batches=… formats=md:203,yaml:267,…`), so it is safe to run from a cron job or after every commit. -- Indexed formats (A9): **`md, markdown, txt, yaml, yml, json, py`** - (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). Any path with a +- Indexed formats (A9, revised 2026-08-27): **`md, markdown, txt, + yaml, yml, json, py`**, the Podman quadlet family (**`container, + network, volume, image, pod, kube, swap, os, endpoint`**), and **`j2`** + Jinja templates (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). + Any path with a **dot-prefixed component** — hidden files or vendored caches like `.esphome/.espressif/**` — is skipped, along with `.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`. @@ -302,7 +305,8 @@ embedded. that's how previously imported junk leaves the index. - Non-markdown files get format-aware chunking (YAML top-level keys / `---` docs, JSON top-level keys, Python top-level defs/classes via - stdlib `ast`) and their title comes from the file stem. + stdlib `ast`; quadlet unit files and `j2` templates are paragraph- + packed as plain text) and their title comes from the file stem. - Unchanged files are **not re-embedded** — only new/changed ones, so refreshes are cheap. - After a run that **changed** the knowledge base (at least one document @@ -644,7 +648,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`. | `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion | | `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) | | `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn — every call the model emits consumes a round; at the cap the loop forces one final no-tools answer (0 = no tools, the kill switch) | -| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) | +| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) | | `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) | | `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) | | `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `` (steering notes) prompt section | diff --git a/app/config.py b/app/config.py index 5615e58..83ca049 100644 --- a/app/config.py +++ b/app/config.py @@ -11,10 +11,19 @@ from functools import lru_cache from pydantic import field_validator from pydantic_settings import BaseSettings, SettingsConfigDict -#: The A9 import formats (PLAN anchor A9, revised 2026-08-21). -#: ``BOR_IMPORT_EXTENSIONS`` may narrow — but never widen — this set. +#: The A9 import formats (PLAN anchor A9, revised 2026-08-21; revised +#: 2026-08-27, owner permission — the full Podman quadlet family +#: ``container, network, volume, image, pod, kube, swap, os, endpoint`` +#: plus Jinja templates ``j2`` join the allowed set, chunked as plain +#: text). ``BOR_IMPORT_EXTENSIONS`` may narrow — but never widen — this +#: set. _ALLOWED_IMPORT_EXTENSIONS: frozenset[str] = frozenset( - {"md", "markdown", "txt", "yaml", "yml", "json", "py"} + { + "md", "markdown", "txt", "yaml", "yml", "json", "py", + # A9 revised 2026-08-27 (owner permission): quadlet family + jinja. + "container", "network", "volume", "image", "pod", + "kube", "swap", "os", "endpoint", "j2", + } ) @@ -127,14 +136,17 @@ class Settings(BaseSettings): session_max_age: int = 43_200 session_cookie: str = "bor_session" - # --- Import scope (A9, revised 2026-08-21) --- + # --- Import scope (A9, revised 2026-08-21 and 2026-08-27) --- # Comma-separated list of lowercased file extensions (no dot) imported # by ``scripts/import_docs.py``. Hidden (dot) path components are always # skipped, plus the importer's exclusion list. # Stored as a raw CSV string (env-native — no JSON) and parsed on demand # via :py:meth:`import_extension_set`. ``mode="after"`` validation runs # against the raw string so a typo fails loudly at startup. - import_extensions: str = "md,markdown,txt,yaml,yml,json,py" + import_extensions: str = ( + "md,markdown,txt,yaml,yml,json,py," + "container,network,volume,image,pod,kube,swap,os,endpoint,j2" + ) #: List of git repo URLs to clone/pull into ``sources_dir`` before #: indexing (phase 28); comma-separated, stored raw. Empty means no git #: sources — ``import_docs`` then falls back to ``--source`` / the old diff --git a/app/rag/chunker.py b/app/rag/chunker.py index af6bca7..6eb9c93 100644 --- a/app/rag/chunker.py +++ b/app/rag/chunker.py @@ -22,6 +22,10 @@ per-format policies: module preamble — imports, constants — is its own block); an oversized definition falls back to line packing. * **txt** (and any unknown suffix) — paragraph packing. +* **container / network / volume / image / pod / kube / swap / os / + endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and + Jinja templates; plain-text paragraph packing (``chunk_text``) — no + format-specific splitter (owner decision). Every format honors :data:`HARD_MAX_CHARS` (1200 — the aipi ~1024-token request cap) and the target/overlap settings; oversized blocks are split @@ -331,7 +335,8 @@ def _normalize_target_overlap(target_chars: int, overlap_chars: int) -> tuple[in return target, min(overlap_chars, target - 1) -#: suffix → chunker (A9, revised: md, markdown, txt, yaml, yml, json, py). +#: suffix → chunker (A9, revised 2026-08-21 and 2026-08-27: md, markdown, +#: txt, yaml, yml, json, py + the quadlet family and j2 — plain text). _FORMAT_CHUNKERS = { ".md": chunk_markdown, ".markdown": chunk_markdown, @@ -340,6 +345,19 @@ _FORMAT_CHUNKERS = { ".yml": chunk_yaml, ".json": chunk_json, ".py": chunk_python, + # A9 revised 2026-08-27 (owner permission): the full Podman quadlet + # family + Jinja templates — plain-text paragraph packing, no + # TOML/Jinja-aware splitter (owner decision). + ".container": chunk_text, + ".network": chunk_text, + ".volume": chunk_text, + ".image": chunk_text, + ".pod": chunk_text, + ".kube": chunk_text, + ".swap": chunk_text, + ".os": chunk_text, + ".endpoint": chunk_text, + ".j2": chunk_text, } diff --git a/tests/e2e/test_admin_auth.py b/tests/e2e/test_admin_auth.py index 175cf95..063fd74 100644 --- a/tests/e2e/test_admin_auth.py +++ b/tests/e2e/test_admin_auth.py @@ -208,10 +208,10 @@ def test_admin_login_unlocks_sources_and_tuning( login(page, app_url) expect(page).to_have_url(app_url + "/sources.html") expect(page.locator("#sources-gate")).to_be_hidden() - expect(page.locator("#stat-docs")).to_have_text("9") # phase 44: +tables.md + expect(page.locator("#stat-docs")).to_have_text("13") # phase 47: +quadlet/j2 expect(page.locator("#stat-chunks")).not_to_have_text("–") expect(page.locator("#docs-table")).to_be_visible() - expect(page.locator("#docs-tbody tr")).to_have_count(9) + expect(page.locator("#docs-tbody tr")).to_have_count(13) # Chat: the tuning UI is back — header toggle with count badge, # Sign out instead of Sign in, Tune under the answer. diff --git a/tests/e2e/test_chat_persistence.py b/tests/e2e/test_chat_persistence.py index 59caa94..f50c731 100644 --- a/tests/e2e/test_chat_persistence.py +++ b/tests/e2e/test_chat_persistence.py @@ -128,7 +128,7 @@ def test_conversation_survives_reload( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) page.set_default_timeout(30_000) page.goto(app_url) _ask(page, QUESTION) diff --git a/tests/e2e/test_chat_rag.py b/tests/e2e/test_chat_rag.py index 90f2d62..5696f6a 100644 --- a/tests/e2e/test_chat_rag.py +++ b/tests/e2e/test_chat_rag.py @@ -76,7 +76,7 @@ def test_on_topic_question_streams_grounded_answer( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) page.set_default_timeout(30_000) page.goto(app_url) diff --git a/tests/e2e/test_dark_tech_theme.py b/tests/e2e/test_dark_tech_theme.py index eaeded7..dd54d3d 100644 --- a/tests/e2e/test_dark_tech_theme.py +++ b/tests/e2e/test_dark_tech_theme.py @@ -114,7 +114,7 @@ def _seed_kb(mock_port: int) -> ImportSummary: db.execute(text("TRUNCATE chunks, documents, query_log")) db.commit() summary = _run_in_thread(_import_fixtures(mock_port)) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) return summary diff --git a/tests/e2e/test_global_tuning.py b/tests/e2e/test_global_tuning.py index 3467b31..e9f4619 100644 --- a/tests/e2e/test_global_tuning.py +++ b/tests/e2e/test_global_tuning.py @@ -336,7 +336,7 @@ def test_edit_note_steers_answer( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) page.set_default_timeout(30_000) _open_tuning(page, app_url) diff --git a/tests/e2e/test_honest_deflection.py b/tests/e2e/test_honest_deflection.py index 99b6b1d..991a069 100644 --- a/tests/e2e/test_honest_deflection.py +++ b/tests/e2e/test_honest_deflection.py @@ -84,7 +84,7 @@ def test_off_topic_question_deflects_honestly( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) page.set_default_timeout(30_000) page.goto(app_url) expect(page.locator("#kb-banner")).to_be_hidden() diff --git a/tests/e2e/test_import_documents.py b/tests/e2e/test_import_documents.py index bed8dee..2e2ebba 100644 --- a/tests/e2e/test_import_documents.py +++ b/tests/e2e/test_import_documents.py @@ -41,6 +41,11 @@ EXPECTED_ROWS = ( "homelab/scripts/uptime_probe.py", "homelab/ssh/ssh_aliases.txt", "homelab/tables.md", # phase 44: the markdown-tables fixture + # phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures + "homelab/quadlet/compose.container", + "homelab/quadlet/lan.network", + "homelab/quadlet/cache.volume", + "homelab/templates/deploy.j2", ) @@ -86,14 +91,18 @@ def test_sources_page_lists_indexed_docs( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - # Nine A9-format files are imported (phase 44 added homelab/tables.md); + # Thirteen A9-format files are imported (phase 44 added + # homelab/tables.md, phase 47 added the quadlet + j2 fixtures); # .hidden/junk.md is out of scope (A9 revised — hidden path components # are never walked). - assert summary is not None and summary.added == 9 - assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1} + assert summary is not None and summary.added == 13 + assert summary.formats == { + "md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1, + "container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47 + } login(page, app_url) # phase 16: the catalog is admin-only - expect(page.locator("#stat-docs")).to_have_text("9") + expect(page.locator("#stat-docs")).to_have_text("13") # Phase 30: non-markdown fixtures each gained one ``is_summary`` chunk, # so the Sources total is content chunks + summary chunks. expect(page.locator("#stat-chunks")).to_have_text( diff --git a/tests/e2e/test_kb_overview.py b/tests/e2e/test_kb_overview.py index 8f48adb..8e9a96b 100644 --- a/tests/e2e/test_kb_overview.py +++ b/tests/e2e/test_kb_overview.py @@ -170,7 +170,7 @@ def test_on_topic_answer_echoes_kb_overview( the mock's echo of the ```` section's first bullet — only possible if the section reached the LLM prompt.""" summary = _reset_db(mock_llm, seed=True, overview=OVERVIEW) - assert summary is not None and summary.added == 9 # phase 44 added tables.md + assert summary is not None and summary.added == 13 # phase 47 added quadlet+j2 assert _overview_row() is not None # the row the turn must inject bubble = _ask(page, app_url, QUESTION, KB_ECHO) @@ -250,7 +250,7 @@ def test_mock_generated_outline_is_stored_and_echoed( stores the byte-stable 8-token digest of the generator's document list, and a chat turn echoes its first bullet.""" summary = _reset_db(mock_llm, seed=True, overview=None) - assert summary is not None and summary.added == 9 # phase 44 added tables.md + assert summary is not None and summary.added == 13 # phase 47 added quadlet+j2 assert _overview_row() is None # direct import never regenerates kwargs: dict[str, Any] = { diff --git a/tests/e2e/test_loading_feedback.py b/tests/e2e/test_loading_feedback.py index 4a8272f..9395c24 100644 --- a/tests/e2e/test_loading_feedback.py +++ b/tests/e2e/test_loading_feedback.py @@ -176,7 +176,7 @@ def test_typing_indicator_during_slow_think( """AC1/AC5: the 3s mock warm-up must show the typing indicator for >=2s before any text appears, then it is gone once the answer lands.""" summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) page.set_default_timeout(30_000) page.goto(app_url) diff --git a/tests/e2e/test_markdown_tables.py b/tests/e2e/test_markdown_tables.py index fdb1cd0..b1a7e37 100644 --- a/tests/e2e/test_markdown_tables.py +++ b/tests/e2e/test_markdown_tables.py @@ -106,7 +106,7 @@ def _run_in_thread(coro: Any) -> Any: def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None: """Truncate the KB (+ the global prompt-state rows), then optionally - re-import the fixtures (9 docs since phase 44 added tables.md).""" + re-import the fixtures (13 docs since phase 47 added quadlet+j2).""" with SessionLocal() as db: db.execute( text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview") @@ -151,7 +151,7 @@ def test_chat_table_renders( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # phase 44: +tables.md + assert summary is not None and summary.added == 13 # phase 47: +quadlet/j2 fixtures page.set_default_timeout(30_000) bubble = _ask_table_answer(page, app_url) diff --git a/tests/e2e/test_no_reply_autoscroll.py b/tests/e2e/test_no_reply_autoscroll.py index c8371bb..17bba65 100644 --- a/tests/e2e/test_no_reply_autoscroll.py +++ b/tests/e2e/test_no_reply_autoscroll.py @@ -123,11 +123,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None: @pytest.fixture() def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]: - """A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase - 44, A9 formats), truncated again on teardown. ``db_ready`` (conftest) + """A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase + 47, A9 formats), truncated again on teardown. ``db_ready`` (conftest) skips with clear instructions when Postgres is down.""" summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 + assert summary is not None and summary.added == 13 yield _reset_db(mock_llm, seed=False) diff --git a/tests/e2e/test_quadlet_jinja_import.py b/tests/e2e/test_quadlet_jinja_import.py new file mode 100644 index 0000000..3f486fe --- /dev/null +++ b/tests/e2e/test_quadlet_jinja_import.py @@ -0,0 +1,204 @@ +"""Phase 47 E2E (Playwright): quadlet unit files + Jinja templates ride the +A9 import path end to end (A9 revised 2026-08-27, owner permission). + +Story: ``.agent/user_stories/quadlet-jinja-import.md`` +Run in isolation (DB must be up: ``podman compose up -d db``): + + uv run pytest tests/e2e/test_quadlet_jinja_import.py -v --no-cov + +Seeding reuses the real import function against ``tests/fixtures/docs/`` +with the deterministic mock embeddings — the ``test_import_documents.py`` +pattern (truncate the KB, ``import_sources`` in a worker thread). This +suite's re-import changes the KB for the session; it is run in isolation +(A16), so there is no cross-suite interference. + +Test → story mapping (Playwright Mapping Rule): +1. ``test_quadlet_and_jinja_indexed`` — ``GET /api/docs`` (admin session) + lists the four new-format fixtures with non-zero chunk counts and the + file stem as title — no env configuration needed (default set). +2. ``test_sources_table_shows_them`` (admin) — the Sources table renders + a row per new file, each with its ``.doc-link`` path link. +3. ``test_container_content_viewable`` — the phase-26 modal shows the + ``.container`` file's TOML content (``[Container]`` section + sentinel) + with the stem as title. +4. ``test_jinja_retrievable_not_deflected`` — a question carrying the + ``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine + below threshold **and** zero FTS hits) → honest-positive: the answer + bubble is not ``.is-deflected`` and a source chip names + ``templates/deploy.j2``. +""" +from __future__ import annotations + +import asyncio +from pathlib import Path +from threading import Thread +from typing import Any + +from playwright.sync_api import Page, expect +from sqlalchemy import text + +from app.config import Settings +from app.db import SessionLocal +from app.rag.importer import ImportSummary, import_sources +from app.rag.llm import LLMClient +from e2e.auth_helpers import login + +REPO = Path(__file__).resolve().parents[2] +FIXTURES = REPO / "tests" / "fixtures" / "docs" + +#: The four new-format fixtures (A9 revised 2026-08-27): path → stem title +#: (no H1 → ``extract_title`` falls back to the file stem). +NEW_FORMAT_DOCS = { + "homelab/quadlet/compose.container": "compose", + "homelab/quadlet/lan.network": "lan", + "homelab/quadlet/cache.volume": "cache", + "homelab/templates/deploy.j2": "deploy", +} + +#: Carries the ``.j2`` fixture's sentinel — the hyphens split into the +#: ``rese | jinja | sentinel | 33dd`` tsquery tokens that FTS-match the +#: chunk holding ``{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %}``. +JINJA_QUESTION = "What is RESE-JINJA-SENTINEL-33dd?" + + +async def _import_fixtures(mock_port: int) -> ImportSummary: + kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"} + settings = Settings(**kwargs) # pyright: ignore[reportCallIssue] + return await import_sources([FIXTURES], LLMClient(settings)) + + +def _run_in_thread(coro: Any) -> Any: + """Run a coroutine on a worker thread. + + Playwright's sync API keeps an asyncio loop running on the test thread, + so ``asyncio.run`` cannot be called directly from a test body. + """ + box: dict[str, Any] = {} + + def runner() -> None: + try: + box["value"] = asyncio.run(coro) + except BaseException as e: # noqa: BLE001 — re-raised on the test thread + box["error"] = e + + t = Thread(target=runner) + t.start() + t.join() + if "error" in box: + raise box["error"] + return box["value"] + + +def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None: + """Truncate the KB (and query log), then optionally re-import fixtures.""" + with SessionLocal() as db: + db.execute(text("TRUNCATE chunks, documents, query_log")) + db.commit() + if not seed: + return None + return _run_in_thread(_import_fixtures(mock_port)) + + +# --------------------------------------------------------------------------- +# 1. Default-extensions import indexes the new formats (API view) +# --------------------------------------------------------------------------- + + +def test_quadlet_and_jinja_indexed(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None: + summary = _reset_db(mock_llm, seed=True) + assert summary is not None + # The four fixtures joined the default set (A9 revised 2026-08-27) — + # the import needed no BOR_IMPORT_EXTENSIONS configuration at all. + assert summary.added == 13, f"expected all 13 fixture docs, added {summary.added}" + + login(page, app_url) # phase 16: the catalog is admin-only + r = page.request.get(f"{app_url}/api/docs") + assert r.status == 200 + by_path = {d["path"]: d for d in r.json()["documents"]} + for path, stem in NEW_FORMAT_DOCS.items(): + doc = by_path.get(path) + assert doc is not None, f"{path} missing from GET /api/docs" + assert doc["chunks"] > 0, f"{path} indexed zero chunks" + # No H1 → the stem is the title, exactly like the other + # non-markdown formats. + assert doc["title"] == stem, f"{path} title {doc['title']!r}, want stem {stem!r}" + assert doc["source"] == "docs" + + +# --------------------------------------------------------------------------- +# 2. The Sources table renders a row (with path link) per new file +# --------------------------------------------------------------------------- + + +def test_sources_table_shows_them(page: Page, app_url: str, mock_llm: int, db_ready: None) -> None: + _reset_db(mock_llm, seed=True) + login(page, app_url) # phase 16: the Sources catalog is admin-only + for path in NEW_FORMAT_DOCS: + row = page.locator("#docs-tbody tr", has_text=path) + expect(row).to_have_count(1) + link = row.locator("td:nth-child(2) a.doc-link") + expect(link).to_have_count(1) + # The full path is the link's accessible context (ellipsis is + # visual only) — the same contract the other rows carry. + expect(link).to_have_attribute("title", path) + + +# --------------------------------------------------------------------------- +# 3. The .container file's TOML is viewable in the modal (stem title) +# --------------------------------------------------------------------------- + + +def test_container_content_viewable( + page: Page, app_url: str, mock_llm: int, db_ready: None +) -> None: + _reset_db(mock_llm, seed=True) + login(page, app_url) # phase 16: the Sources catalog is admin-only + + row = page.locator("#docs-tbody tr", has_text="homelab/quadlet/compose.container") + expect(row).to_have_count(1) + link = row.locator("td:nth-child(2) a.doc-link") + expect(link).to_have_attribute( + "href", "/document.html?source=docs&path=homelab%2Fquadlet%2Fcompose.container" + ) + expect(link).not_to_have_attribute("target") + + link.click() + expect(page.locator(".doc-modal")).to_be_visible() + # Stem title (no H1 in a quadlet unit file) + the extension badge. + expect(page.locator("#doc-modal-title")).to_have_text("compose") + expect(page.locator("#doc-modal-meta .format-badge")).to_have_text("container") + # Non-markdown content renders as escaped monospace text in a pre — + # the whole TOML, [Container] section and sentinel included. + pre = page.locator("#doc-modal-content pre.doc-raw") + expect(pre).to_have_count(1) + expect(pre).to_contain_text("[Container]") + expect(pre).to_contain_text("Image=docker.io/reese/compose-gateway:1.4.2") + expect(pre).to_contain_text("RESE-QUADLET-SENTINEL-77aa") + # Still on the Sources page: the modal is same-page (phase 26). + assert page.url == app_url + "/sources.html" + + +# --------------------------------------------------------------------------- +# 4. A .j2 sentinel question is retrievable and honest-positive (A8) +# --------------------------------------------------------------------------- + + +def test_jinja_retrievable_not_deflected( + page: Page, app_url: str, mock_llm: int, db_ready: None +) -> None: + _reset_db(mock_llm, seed=True) + page.set_default_timeout(30_000) + page.goto(app_url) + page.fill("#message-input", JINJA_QUESTION) + page.click("#send-btn") + + # The done event appends source chips — waiting on the .j2 chip means + # the turn is finished and the retrieval doc reached the UI. + chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2") + expect(chip).to_have_count(1, timeout=30_000) + + # A8: LOW requires best cosine < threshold AND zero FTS hits — the + # question's sentinel tokens FTS-match the .j2 chunk, so the gate is + # honest-positive whatever the mock's cosine says. + expect(page.locator(".msg.brain")).to_have_count(1) + expect(page.locator(".msg.brain.is-deflected")).to_have_count(0) diff --git a/tests/e2e/test_retrieval_quality.py b/tests/e2e/test_retrieval_quality.py index c9020db..e2cb7d5 100644 --- a/tests/e2e/test_retrieval_quality.py +++ b/tests/e2e/test_retrieval_quality.py @@ -84,13 +84,17 @@ def _ask(page: Page, message: str) -> None: def test_multi_format_import_hidden_doc_excluded( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: - """A9 (revised): all seven formats import; hidden (dot) paths never do.""" + """A9 (revised): all seventeen formats import; hidden (dot) paths never do.""" summary = _reset_db(mock_llm, seed=True) assert summary is not None - # Nine A9-format fixture files (phase 44 added homelab/tables.md); - # .hidden/junk.md must never be walked. - assert summary.added == 9 - assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1} + # Thirteen A9-format fixture files (phase 44 added homelab/tables.md, + # phase 47 added the quadlet + j2 fixtures); .hidden/junk.md must never + # be walked. + assert summary.added == 13 + assert summary.formats == { + "md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1, + "container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47 + } # Phase 16: the catalog is admin-only — perform the real form login, # then call the API with the signed cookie the browser now holds. @@ -103,7 +107,7 @@ def test_multi_format_import_hidden_doc_excluded( r = httpx.get(f"{app_url}/api/docs", timeout=10, cookies=cookies) assert r.status_code == 200 docs = r.json()["documents"] - assert len(docs) == 9 + assert len(docs) == 13 assert all(".hidden" not in d["path"] for d in docs) assert {d["path"] for d in docs} >= { "homelab/container_gitlab/gitlab.md", @@ -114,7 +118,7 @@ def test_multi_format_import_hidden_doc_excluded( } # The Sources page (we're already on it, signed in) reflects the set. - expect(page.locator("#stat-docs")).to_have_text("9") + expect(page.locator("#stat-docs")).to_have_text("13") expect(page.locator("#docs-tbody tr", has_text=".hidden")).to_have_count(0) diff --git a/tests/e2e/test_sources_midstream_bug.py b/tests/e2e/test_sources_midstream_bug.py index 31444cf..8b75d0a 100644 --- a/tests/e2e/test_sources_midstream_bug.py +++ b/tests/e2e/test_sources_midstream_bug.py @@ -154,11 +154,11 @@ def _no_error_banner(page: Page) -> None: @pytest.fixture() def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]: - """A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase - 44, A9 formats), truncated again on teardown. ``db_ready`` (conftest) + """A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase + 47, A9 formats), truncated again on teardown. ``db_ready`` (conftest) skips with clear instructions when Postgres is down.""" summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 + assert summary is not None and summary.added == 13 yield _reset_db(mock_llm, seed=False) diff --git a/tests/e2e/test_steering.py b/tests/e2e/test_steering.py index e9c0a23..f48a974 100644 --- a/tests/e2e/test_steering.py +++ b/tests/e2e/test_steering.py @@ -127,7 +127,7 @@ def test_tune_under_answer_persists_and_steers( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) page.set_default_timeout(30_000) page.goto(app_url) login(page, app_url, next="/") # phase 16: tuning is admin-only diff --git a/tests/e2e/test_suggestion_chips.py b/tests/e2e/test_suggestion_chips.py index b097bf8..1ed0b5f 100644 --- a/tests/e2e/test_suggestion_chips.py +++ b/tests/e2e/test_suggestion_chips.py @@ -63,7 +63,7 @@ def _seed_kb(mock_port: int) -> ImportSummary: db.execute(text("TRUNCATE chunks, documents, query_log")) db.commit() summary = _run_in_thread(_import_fixtures(mock_port)) - assert summary is not None and summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2) return summary diff --git a/tests/e2e/test_thinking_display.py b/tests/e2e/test_thinking_display.py index 6aa1836..83b8ed6 100644 --- a/tests/e2e/test_thinking_display.py +++ b/tests/e2e/test_thinking_display.py @@ -99,11 +99,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None: @pytest.fixture() def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]: - """A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase - 44, A9 formats), truncated again on teardown. ``db_ready`` (conftest) + """A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase + 47, A9 formats), truncated again on teardown. ``db_ready`` (conftest) skips with clear instructions when Postgres is down.""" summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 + assert summary is not None and summary.added == 13 yield _reset_db(mock_llm, seed=False) diff --git a/tests/e2e/test_thinking_scroll.py b/tests/e2e/test_thinking_scroll.py index 9cfd1ae..5e89434 100644 --- a/tests/e2e/test_thinking_scroll.py +++ b/tests/e2e/test_thinking_scroll.py @@ -148,11 +148,11 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None: @pytest.fixture() def seeded_kb(mock_llm: int, db_ready: None) -> Iterator[None]: - """A fresh KB seeded from ``tests/fixtures/docs`` (9 docs since phase - 44, A9 formats), truncated again on teardown (same fixture shape as + """A fresh KB seeded from ``tests/fixtures/docs`` (13 docs since phase + 47, A9 formats), truncated again on teardown (same fixture shape as the phase-17/21 suites).""" summary = _reset_db(mock_llm, seed=True) - assert summary is not None and summary.added == 9 + assert summary is not None and summary.added == 13 yield _reset_db(mock_llm, seed=False) diff --git a/tests/e2e/test_whole_document_context.py b/tests/e2e/test_whole_document_context.py index 1006ecf..5337947 100644 --- a/tests/e2e/test_whole_document_context.py +++ b/tests/e2e/test_whole_document_context.py @@ -17,8 +17,8 @@ The oversized documents are seeded directly via SQLAlchemy (a ``documents`` row + 2–3 ``chunks`` rows whose embeddings are the mock's own deterministic bag-of-words vectors, so the question's live mock embedding genuinely overlaps — no fixture files added: -``tests/fixtures/docs/`` stays at its 9 files (phase 44), other suites -pin ``summary.added == 9``). +``tests/fixtures/docs/`` stays at its 13 files (phase 47), other suites +pin ``summary.added == 13``). """ from __future__ import annotations @@ -301,7 +301,7 @@ def test_small_document_path_unchanged( path, byte-identical to before — no marker, kubernetes.md cited.""" _reset_db(None) summary = _run_in_thread(_import_fixtures(mock_llm)) - assert summary.added == 9 # A9 formats (phase 44 added tables.md) + assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2) bubble = _ask(page, app_url, SMALL_QUESTION) expect(bubble).to_contain_text(SMALL_QUESTION, timeout=30_000) diff --git a/tests/fixtures/docs/homelab/quadlet/cache.volume b/tests/fixtures/docs/homelab/quadlet/cache.volume new file mode 100644 index 0000000..f38b6dc --- /dev/null +++ b/tests/fixtures/docs/homelab/quadlet/cache.volume @@ -0,0 +1,11 @@ +# Quadlet volume: backing storage for the shared cache (btrfs loop on +# the app node — see the app-node playbook for the loop setup). +# RESE-VOLUME-SENTINEL-22cc + +[Unit] +Description=Shared cache volume (btrfs loop, 32G) + +[Volume] +Driver=local +Type=volume +Device=/var/lib/quadlet-volumes/cache diff --git a/tests/fixtures/docs/homelab/quadlet/compose.container b/tests/fixtures/docs/homelab/quadlet/compose.container new file mode 100644 index 0000000..7299854 --- /dev/null +++ b/tests/fixtures/docs/homelab/quadlet/compose.container @@ -0,0 +1,51 @@ +# Quadlet unit for the homelab compose stack gateway (Podman >= 4.9, +# quadlet >= 0.5). Kept in version control next to the playbook so a +# fresh reinstall of the app node reproduces the exact stack — the unit +# file is the source of truth, not the running container. +# RESE-QUADLET-SENTINEL-77aa + +[Unit] +Description=Compose stack gateway (reverse proxy + metrics scrape) +Wants=lan.network +After=lan.network network-online.target +Documentation=https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html + +[Service] +Restart=always +RestartSec=5 +TimeoutStartSec=300 +MemoryMax=4G +CPUQuota=400% +# The gateway must come up before the scrape jobs expect it. +ExecStartPre=/usr/bin/test -f /etc/homelab/compose.env +Environment=COMPOSE_PROJECT_NAME=homelab +EnvironmentFile=/etc/homelab/compose.env + +[Container] +Image=docker.io/reese/compose-gateway:1.4.2 +ImageVolume=ignore +ContainerName=homelab-compose-gateway +Label=io.podman.quadlet.stack=homelab +Label=io.reese.maintained-by=ansible +PublishPort=127.0.0.1:8443:8443 +PublishPort=127.0.0.1:9100:9100 +Network=lan +Environment=LOG_LEVEL=info +Environment=METRICS_BIND=0.0.0.0:9100 +Environment=UPSTREAM_REGISTRY=10.89.0.10:5000 +Volume=/var/lib/compose-gateway/certs:/etc/certs:ro +Volume=/var/lib/compose-gateway/data:/data +Volume=/etc/homelab/compose.env:/etc/compose.env:ro +Exec=serve +# Health probe: the gateway answers /healthz on the metrics port. +HealthCmd=CURL -fsS http://127.0.0.1:9100/healthz +HealthInterval=30s +HealthStartPeriod=15s +HealthTimeout=5s +HealthRetries=3 +StartTimeout=60s +# Keep the gateway on the lab bridge — no public interface exposure. +AdditionalCapabilities=CHOWN,SETGID,SETUID + +[Install] +WantedBy=default.target diff --git a/tests/fixtures/docs/homelab/quadlet/lan.network b/tests/fixtures/docs/homelab/quadlet/lan.network new file mode 100644 index 0000000..8f96438 --- /dev/null +++ b/tests/fixtures/docs/homelab/quadlet/lan.network @@ -0,0 +1,13 @@ +# Quadlet network: the flat lab bridge for all homelab services. +# RESE-NETWORK-SENTINEL-11bb + +[Unit] +Description=Homelab lab network (bridge, 10.89.0.0/24) + +[Network] +Driver=bridge +NetworkName=labnet +IPAMDriver=dhcp +Subnets=10.89.0.0/24 +Gateway=10.89.0.1 +Options=mtu=1500 diff --git a/tests/fixtures/docs/homelab/templates/deploy.j2 b/tests/fixtures/docs/homelab/templates/deploy.j2 new file mode 100644 index 0000000..8927796 --- /dev/null +++ b/tests/fixtures/docs/homelab/templates/deploy.j2 @@ -0,0 +1,15 @@ +{# Service unit template — rendered by the deploy playbook, one file per + host. The playbook loops the services dict from the group vars. #} +{% set sentinel = "RESE-JINJA-SENTINEL-33dd" %} +{% for svc in services %} +[{{ svc.name }}] +host = {{ svc.host }} +port = {{ svc.port }} +enabled = {{ svc.enabled | default(true) }} +{% if svc.env is defined %} +{% for key, value in svc.env.items() %} +{{ key }} = {{ value }} +{% endfor %} +{% endif %} +{% endfor %} +# rendered by ansible on {{ inventory_hostname }} at {{ deployment_stamp }} diff --git a/tests/integration/test_auth_api.py b/tests/integration/test_auth_api.py index a052b93..d0d6a8c 100644 --- a/tests/integration/test_auth_api.py +++ b/tests/integration/test_auth_api.py @@ -47,7 +47,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]: db.commit() llm = FakeRagLLM() summary = asyncio.run(import_sources([FIXTURES], llm, session=db)) - assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped + assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped yield llm db.execute(text("TRUNCATE chunks, documents, query_log")) db.commit() diff --git a/tests/integration/test_chat_api.py b/tests/integration/test_chat_api.py index deac93c..54c5c68 100644 --- a/tests/integration/test_chat_api.py +++ b/tests/integration/test_chat_api.py @@ -138,7 +138,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]: db.commit() llm = FakeRagLLM() summary = asyncio.run(import_sources([FIXTURES], llm, session=db)) - assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped + assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped yield llm db.execute(text("TRUNCATE chunks, documents, query_log")) db.commit() diff --git a/tests/integration/test_import_quadlet_jinja.py b/tests/integration/test_import_quadlet_jinja.py new file mode 100644 index 0000000..1fa3ad0 --- /dev/null +++ b/tests/integration/test_import_quadlet_jinja.py @@ -0,0 +1,135 @@ +"""Integration test: the phase-47 quadlet + j2 formats ride the import +machinery unchanged (walk → delta → prune). + +The temp source tree holds byte-identical copies of the real fixture +files (``tests/fixtures/docs/homelab/quadlet/*.container|.volume`` and +``templates/deploy.j2``), so this test tracks the fixtures even if their +bytes change. Runs against the local compose Postgres (the ``db`` fixture +from ``tests/conftest.py``) with the deterministic ``FakeEmbedder`` — +same harness as ``test_importer_e2e.py``. A separate file (not an +extension of that one) because the delta/prune lifecycle mutates the +tree between runs, while the fixture e2e stays a single import + +idempotent re-run over the shared fixture tree. + +Runs (DB must be up: ``podman compose up -d db``): + + uv run pytest tests/integration/test_import_quadlet_jinja.py -v +""" +from __future__ import annotations + +import asyncio +from collections.abc import Iterator +from pathlib import Path + +import pytest +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.models import Chunk, Document +from app.rag.importer import import_sources +from tests.fakes import FakeEmbedder + +FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs" / "homelab" + +#: Fixture files copied into the temp source tree (a ``.container``, a +#: ``.volume``, and a ``.j2``). +QUADLET_RELS = ("quadlet/compose.container", "quadlet/cache.volume") +JINJA_RELS = ("templates/deploy.j2",) +ALL_RELS = QUADLET_RELS + JINJA_RELS + + +def _cleanup_source(db: Session, source: str) -> None: + for doc in db.scalars(select(Document).where(Document.source == source)).all(): + db.delete(doc) + db.commit() + + +@pytest.fixture() +def source_dir(db: Session, tmp_path: Path) -> Iterator[Path]: + """A temp source tree seeded with copies of the real fixture files.""" + root = tmp_path / "quadsrc" + for rel in ALL_RELS: + target = root / rel + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes((FIXTURES / rel).read_bytes()) + try: + yield root + finally: + _cleanup_source(db, root.name) + + +def test_quadlet_and_jinja_import_delta_and_prune(source_dir: Path, db: Session) -> None: + llm = FakeEmbedder() + + # --- fresh import: all three docs land with stem titles + chunks --- + s1 = asyncio.run(import_sources([source_dir], llm, prune=False, session=db)) + assert (s1.files, s1.added, s1.unchanged, s1.updated, s1.pruned) == (3, 3, 0, 0, 0) + assert s1.formats == {"container": 1, "volume": 1, "j2": 1} + assert s1.errors == 0 + docs = { + d.path: d + for d in db.scalars(select(Document).where(Document.source == source_dir.name)).all() + } + assert set(docs) == { + "quadlet/compose.container", + "quadlet/cache.volume", + "templates/deploy.j2", + } + # Non-markdown titles come from the file stem (a leading ``#`` is a + # comment in these formats, not a heading). + assert docs["quadlet/compose.container"].title == "compose" + assert docs["quadlet/cache.volume"].title == "cache" + assert docs["templates/deploy.j2"].title == "deploy" + for path, doc in docs.items(): + content = [c for c in doc.chunks if not c.is_summary] + assert content, f"{path} has no content chunks" + assert all( + c.embedding is not None and len(c.embedding) == 768 for c in content + ), f"{path} content chunks not embedded" + # Phase 30 parity: the new non-markdown formats also get their + # lite summary + one ``is_summary`` chunk. + assert doc.summary is not None, f"{path} should have a summary" + schunks = [c for c in doc.chunks if c.is_summary] + assert len(schunks) == 1 and schunks[0].position == -1 + + # --- delta: change only the .j2 → it updates, the others stay --- + j2 = source_dir / "templates" / "deploy.j2" + j2.write_text(j2.read_text(encoding="utf-8") + "\n# RESE-JINJA-DELTA-CHANGED\n") + s2 = asyncio.run(import_sources([source_dir], llm, session=db)) + assert (s2.added, s2.updated, s2.unchanged, s2.pruned) == (0, 1, 2, 0) + changed = db.scalar( + select(Document).where( + Document.source == source_dir.name, Document.path == "templates/deploy.j2" + ) + ) + assert changed is not None and "RESE-JINJA-DELTA-CHANGED" in changed.content + + # --- idempotent: a clean re-run re-embeds nothing --- + calls_before = len(llm.calls) + s3 = asyncio.run(import_sources([source_dir], llm, session=db)) + assert (s3.added, s3.updated, s3.unchanged, s3.pruned) == (0, 0, 3, 0) + assert len(llm.calls) == calls_before # unchanged → no embedding requests + + # --- prune: delete the .volume → its row + chunks cascade away --- + (source_dir / "quadlet" / "cache.volume").unlink() + s4 = asyncio.run(import_sources([source_dir], llm, session=db, prune=True)) + assert (s4.unchanged, s4.pruned) == (2, 1) + assert db.scalar( + select(Document).where( + Document.source == source_dir.name, Document.path == "quadlet/cache.volume" + ) + ) is None + # Its chunks (content + summary) are gone — FK cascade. + assert db.scalar( + select(func.count()) + .select_from(Chunk) + .join(Document, Document.id == Chunk.document_id) + .where( + Document.source == source_dir.name, + Document.path == "quadlet/cache.volume", + ) + ) == 0 + # The other two docs survived the prune. + assert db.scalar( + select(func.count()).select_from(Document).where(Document.source == source_dir.name) + ) == 2 diff --git a/tests/integration/test_importer_e2e.py b/tests/integration/test_importer_e2e.py index ac8f859..e21072e 100644 --- a/tests/integration/test_importer_e2e.py +++ b/tests/integration/test_importer_e2e.py @@ -28,6 +28,11 @@ EXPECTED_DOCS = { ("docs", "homelab/scripts/uptime_probe.py"), ("docs", "homelab/ssh/ssh_aliases.txt"), ("docs", "homelab/tables.md"), # phase 44: the markdown-tables fixture + # phase 47 (A9 revised 2026-08-27): quadlet family + jinja fixtures + ("docs", "homelab/quadlet/compose.container"), + ("docs", "homelab/quadlet/lan.network"), + ("docs", "homelab/quadlet/cache.volume"), + ("docs", "homelab/templates/deploy.j2"), } @@ -37,13 +42,19 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None: llm = FakeEmbedder() summary = asyncio.run(import_sources([FIXTURES], llm, session=db)) - # Nine A9-format files (phase 44 added homelab/tables.md); - # .hidden/junk.md is out of scope (A9 revised). - assert (summary.files, summary.added, summary.unchanged) == (9, 9, 0) - assert summary.chunks >= 9 - assert summary.formats == {"md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1} + # Thirteen A9-format files (phase 44 added homelab/tables.md, phase 47 + # added the quadlet + j2 fixtures); .hidden/junk.md is out of scope + # (A9 revised). + assert (summary.files, summary.added, summary.unchanged) == (13, 13, 0) + assert summary.chunks >= 13 + assert summary.formats == { + "md": 5, "yaml": 1, "json": 1, "py": 1, "txt": 1, + "container": 1, "network": 1, "volume": 1, "j2": 1, # phase 47 + } # PLAN §9 per-format summary line: highest count first, then alpha. - assert summary.format_counts() == "md:5,json:1,py:1,txt:1,yaml:1" + assert summary.format_counts() == ( + "md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1" + ) docs = db.scalars(select(Document)).all() assert {(d.source, d.path) for d in docs} == EXPECTED_DOCS @@ -57,6 +68,10 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None: assert titles["homelab/scripts/uptime_probe.py"] == "uptime_probe" assert titles["homelab/networking/static-dns.json"] == "static-dns" assert titles["homelab/ssh/ssh_aliases.txt"] == "ssh_aliases" + assert titles["homelab/quadlet/compose.container"] == "compose" + assert titles["homelab/quadlet/lan.network"] == "lan" + assert titles["homelab/quadlet/cache.volume"] == "cache" + assert titles["homelab/templates/deploy.j2"] == "deploy" # Hidden junk was never imported. assert not any(".hidden" in d.path for d in docs) # Full content is stored — that is what the RAG context will be. @@ -92,13 +107,13 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None: r = admin_client.get("/api/docs") # phase 16: the catalog is admin-only assert r.status_code == 200 body = r.json() - assert len(body["documents"]) == 9 + assert len(body["documents"]) == 13 assert all(d["chunks"] >= 1 for d in body["documents"]) # Idempotent re-run: nothing re-embedded. calls_before = len(llm.calls) s2 = asyncio.run(import_sources([FIXTURES], llm, session=db)) - assert s2.unchanged == 9 and s2.added == 0 + assert s2.unchanged == 13 and s2.added == 0 assert len(llm.calls) == calls_before # unchanged → no embedding requests db.execute(text("TRUNCATE chunks, documents, query_log")) diff --git a/tests/integration/test_kb_overview_api.py b/tests/integration/test_kb_overview_api.py index a20c7f1..7dda095 100644 --- a/tests/integration/test_kb_overview_api.py +++ b/tests/integration/test_kb_overview_api.py @@ -67,7 +67,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]: db.commit() llm = FakeRagLLM() summary = asyncio.run(import_sources([FIXTURES], llm, session=db)) - assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped + assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped yield llm db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview")) db.commit() diff --git a/tests/integration/test_steering_api.py b/tests/integration/test_steering_api.py index 4847558..77a0832 100644 --- a/tests/integration/test_steering_api.py +++ b/tests/integration/test_steering_api.py @@ -51,7 +51,7 @@ def seeded_kb(db) -> Iterator[FakeRagLLM]: db.commit() llm = FakeRagLLM() summary = asyncio.run(import_sources([FIXTURES], llm, session=db)) - assert summary.added == 9 # A9 formats (phase 44 added tables.md); .hidden/ skipped + assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2); .hidden/ skipped yield llm db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes")) db.commit() diff --git a/tests/unit/test_chunker.py b/tests/unit/test_chunker.py index 82e347d..54cbd34 100644 --- a/tests/unit/test_chunker.py +++ b/tests/unit/test_chunker.py @@ -7,6 +7,7 @@ unchanged); the per-format tests cover the phase-09 dispatcher from __future__ import annotations from itertools import pairwise +from pathlib import Path import pytest @@ -361,6 +362,63 @@ def test_txt_long_doc_packs_with_overlap() -> None: assert all(f"para {i}" in "\n".join(chunks) for i in range(6)) +# --------------------------------------------------------------------------- +# quadlet family + j2 (A9 revised 2026-08-27) — plain-text dispatch +# --------------------------------------------------------------------------- + +#: The ten suffixes added 2026-08-27 (owner decision R1): the full Podman +#: quadlet family + Jinja templates, all chunked as plain text. +NEW_FORMAT_SUFFIXES = [ + ".container", ".network", ".volume", ".image", ".pod", + ".kube", ".swap", ".os", ".endpoint", ".j2", +] + +REPO_ROOT = Path(__file__).resolve().parents[2] +FIXTURE_DOCS = REPO_ROOT / "tests" / "fixtures" / "docs" / "homelab" + + +@pytest.mark.parametrize("suffix", NEW_FORMAT_SUFFIXES) +def test_dispatch_new_suffixes_match_plain_text(suffix: str) -> None: + """Every new suffix dispatches to plain-text paragraph packing — the + chunks are identical to a direct ``chunk_text`` call.""" + content = "[Unit]\nDescription=one\n\n[Service]\nRestart=always\n\n[Container]\nImage=busybox\n" + name = suffix.lstrip(".") + assert chunk_document(content, f"x/{name}{suffix}") == chunk_text(content) + + +def test_container_fixture_chunks_past_single_paragraph_pack() -> None: + """The realistic quadlet TOML fixture (≥1 500 chars) splits into ≥2 + chunks, every chunk stays under the hard cap, and the sentinel token + survives the split.""" + rel = "quadlet/compose.container" + content = (FIXTURE_DOCS / rel).read_text(encoding="utf-8") + assert len(content) >= 1500 # the fixture must exercise sub-splitting + chunks = chunk_document(content, rel) + assert len(chunks) >= 2 + assert all(len(c) <= HARD_MAX_CHARS for c in chunks) + assert any("RESE-QUADLET-SENTINEL-77aa" in c for c in chunks) + joined = "\n".join(chunks) + assert "[Container]" in joined and "Restart=always" in joined + + +def test_j2_fixture_chunks_braces_as_plain_text() -> None: + """Jinja braces are just text — no special handling; the ``{{ … }}`` + line and the ``{% set %}`` sentinel appear verbatim in the chunks.""" + rel = "templates/deploy.j2" + content = (FIXTURE_DOCS / rel).read_text(encoding="utf-8") + chunks = chunk_document(content, rel) + assert chunks + assert all(len(c) <= HARD_MAX_CHARS for c in chunks) + joined = "\n".join(chunks) + assert "RESE-JINJA-SENTINEL-33dd" in joined + assert any("[{{ svc.name }}]" in c for c in chunks) + + +def test_dispatch_still_falls_back_for_unknown_suffix() -> None: + content = "alpha\n\nbeta\n" + assert chunk_document(content, "x/notes.whatever") == chunk_text(content) + + # --------------------------------------------------------------------------- # Hard cap across every format (aipi ~1024-token request cap) # --------------------------------------------------------------------------- @@ -374,6 +432,8 @@ def test_txt_long_doc_packs_with_overlap() -> None: ('{"blob": "' + "z" * 5000 + '"}', "big.json"), ("def f():\n" + " x = 1\n" * 1000, "big.py"), ("line of text\n\n" * 800, "big.txt"), + ("[Unit]\n" + "key=" + "v" * 5000 + "\n", "big.container"), # phase 47 + ("{% for x in y %}" + "{{ x }} " * 400 + "{% endfor %}", "big.j2"), # phase 47 ], ) def test_hard_cap_holds_for_every_format(content: str, path: str) -> None: diff --git a/tests/unit/test_config.py b/tests/unit/test_config.py index 1fc2f59..e16a4b1 100644 --- a/tests/unit/test_config.py +++ b/tests/unit/test_config.py @@ -8,7 +8,7 @@ import pytest from pydantic import ValidationError from pydantic_settings import SettingsError -from app.config import Settings +from app.config import _ALLOWED_IMPORT_EXTENSIONS, Settings # pyright: ignore[reportPrivateUsage] def _settings(**kwargs: Any) -> Settings: @@ -42,12 +42,46 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non # Phase 17: the model's thinking streams by default (kill-switch off). assert s.stream_thinking is True assert len(s.suggestions) >= 3 - # A9 (revised): the import scope covers the seven A9 formats. + # A9 (revised 2026-08-27): the import scope covers all seventeen + # A9 formats (original seven + quadlet family + jinja). assert s.import_extension_set == { - ".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py" + ".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py", + ".container", ".network", ".volume", ".image", ".pod", + ".kube", ".swap", ".os", ".endpoint", ".j2", } +NEW_A9_FORMATS = ( + "container", "network", "volume", "image", "pod", + "kube", "swap", "os", "endpoint", "j2", +) + + +def test_allowed_import_extensions_contains_all_seventeen_formats() -> None: + """The validator's base set is the full A9 set: the original seven + plus the ten added 2026-08-27 (quadlet family + ``j2``). The + never-widen contract bounds :py:data:`import_extensions` against + exactly this set.""" + assert { + "md", "markdown", "txt", "yaml", "yml", "json", "py", + *NEW_A9_FORMATS, + } == _ALLOWED_IMPORT_EXTENSIONS + + +def test_default_import_extensions_include_the_ten_new_formats() -> None: + """A9 revised 2026-08-27 (owner permission): the quadlet family + + ``j2`` are imported by default — no env configuration needed — with + the original seven first (order is cosmetic, the set is what + matters).""" + s = _settings() + for ext in NEW_A9_FORMATS: + assert ext in s.import_extensions + assert s.import_extension_set == ( + {".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"} + | {f".{ext}" for ext in NEW_A9_FORMATS} + ) + + def test_env_override(monkeypatch) -> None: monkeypatch.setenv("BOR_RELEVANCE_THRESHOLD", "0.42") monkeypatch.setenv("BOR_LLM_CHAT_MODEL", "juggernaut") @@ -131,6 +165,23 @@ def test_import_extensions_rejects_empty(monkeypatch) -> None: _settings() +def test_import_extensions_validator_accepts_new_a9_formats(monkeypatch) -> None: + """A9 revised 2026-08-27: the new names are first-class — the + never-widen contract now holds against the widened base set, so a + narrowing CSV with quadlet/jinja names is accepted.""" + monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,container,j2") + s = _settings() + assert s.import_extension_set == {".md", ".container", ".j2"} + + +def test_import_extensions_validator_still_rejects_unknown(monkeypatch) -> None: + """Truly unknown extensions still fail loudly at startup (the + validator is intact — only the allowed base set widened).""" + monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,xyz") + with pytest.raises(ValidationError, match="xyz"): + _settings() + + def test_git_sources_default_empty_and_sources_dir_default() -> None: """Phase 28: no git sources by default (backwards-compatible with the ``--source`` / ``DEFAULT_SOURCES`` fallback); the clone location stays diff --git a/tests/unit/test_importer.py b/tests/unit/test_importer.py index e1d1302..578a726 100644 --- a/tests/unit/test_importer.py +++ b/tests/unit/test_importer.py @@ -18,6 +18,7 @@ from pathlib import Path import pytest from sqlalchemy import func, select +from app.config import Settings from app.models import Chunk, Document from app.rag.importer import ( EXCLUDED_DIRS, @@ -29,8 +30,9 @@ from app.rag.importer import ( from app.rag.llm import EmbeddingError from tests.fakes import FakeEmbedder -#: A9 default extension set as dotted suffixes (what the importer passes to -#: the walker when no override is configured). +#: The original seven A9 formats as dotted suffixes (pre-phase-47 default +#: set). The phase-47 walk test uses the live config default +#: (``Settings().import_extension_set`` — now seventeen formats). DEFAULT_EXTS = frozenset({".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"}) @@ -121,6 +123,70 @@ def test_iter_importable_files_missing_dir_yields_nothing(tmp_path: Path) -> Non assert iter_importable_files(tmp_path / "definitely-missing", DEFAULT_EXTS) == [] +def test_iter_importable_files_walker_picks_up_new_formats_by_default( + tmp_path: Path, +) -> None: + """Phase 47 (A9 revised 2026-08-27): the ten new formats — the full + quadlet family + j2 — walk under the *default* extension set (no env + configuration needed), the original seven still walk (regression), and + unknown/hidden-dir/excluded-dir filtering is unchanged for them.""" + root = tmp_path / "quadproj" + (root / ".esphome").mkdir(parents=True) + (root / "node_modules").mkdir() + files = { + # the ten new formats (A9 revised 2026-08-27): + "web.container": "# RESE-QUADLET-SENTINEL-77aa\n[Container]\nImage=alpine\n", + "lan.network": "[Network]\nDriver=bridge\n", + "cache.volume": "[Volume]\nDriver=local\n", + "alpine.image": "[Image]\nImages=alpine\n", + "app.pod": "[Pod]\nPodName=app\n", + "k3s.kube": "apiVersion: v1\nkind: Pod\n", + "zram.swap": "[Swap]\nFile=/swapfile\n", + "fedora.os": "[OS]\nImage=fedora\n", + "edge.endpoint": "[Endpoint]\nPort=8080\n", + "deploy.j2": "{% for s in services %}\n[{{ s }}]\n{% endfor %}\n", + # the original seven — regression: + "note.md": "# n\n", + "note.markdown": "# m\n", + "note.txt": "t\n", + "cfg.yaml": "a: b\n", + "cfg.yml": "a: b\n", + "data.json": "{}\n", + "agent.py": "x = 1\n", + # must be skipped — filtering is unchanged for the new formats too: + "mystery.xyz": "unknown extension", + ".esphome/x.container": "hidden dir (new-format file)", + "node_modules/y.container": "excluded dir (new-format file)", + } + for rel, text in files.items(): + (root / rel).write_text(text) + # The live config default (Settings with no env override) — no + # BOR_IMPORT_EXTENSIONS needed for the new formats to walk. + exts = Settings(_env_file=None).import_extension_set # pyright: ignore[reportCallIssue] + found = {p.relative_to(root).as_posix() for p in iter_importable_files(root, exts)} + assert found == { + # all ten new formats: + "web.container", + "lan.network", + "cache.volume", + "alpine.image", + "app.pod", + "k3s.kube", + "zram.swap", + "fedora.os", + "edge.endpoint", + "deploy.j2", + # …and the original seven: + "note.md", + "note.markdown", + "note.txt", + "cfg.yaml", + "cfg.yml", + "data.json", + "agent.py", + } + + def test_iter_importable_files_respects_custom_extension_filter(tmp_path: Path) -> None: """A narrower filter (e.g. md only) excludes the other A9 formats.""" root = tmp_path / "filtered" @@ -358,6 +424,64 @@ def test_multi_format_import_counts_per_format_and_titles_stem(db, tmp_path: Pat _cleanup_source(db, root.name) +def test_quadlet_and_j2_files_get_stem_titles_and_per_format_counts( + db, tmp_path: Path +) -> None: + """Phase 47: the ten new formats import like any other A9 format — the + per-format counts land in the summary, every file yields content + chunks, and a leading ``#`` line (a TOML comment in quadlet files, not + a heading) never becomes the title: the file stem wins.""" + root = tmp_path / "quadsrc" + (root / "quadlet").mkdir(parents=True) + (root / "tpl").mkdir(parents=True) + (root / "quadlet" / "web.container").write_text( + "# RESE-QUADLET-SENTINEL-77aa\n\n[Container]\nImage=alpine\n" + ) + (root / "quadlet" / "lan.network").write_text("# net\n[Network]\nDriver=bridge\n") + (root / "quadlet" / "cache.volume").write_text("[Volume]\nDriver=local\n") + (root / "quadlet" / "alpine.image").write_text("[Image]\nImages=alpine\n") + (root / "quadlet" / "app.pod").write_text("[Pod]\nPodName=app\n") + (root / "quadlet" / "k3s.kube").write_text("apiVersion: v1\nkind: Pod\n") + (root / "quadlet" / "zram.swap").write_text("[Swap]\nFile=/swapfile\n") + (root / "quadlet" / "fedora.os").write_text("[OS]\nImage=fedora\n") + (root / "quadlet" / "edge.endpoint").write_text("[Endpoint]\nPort=8080\n") + (root / "tpl" / "deploy.j2").write_text( + "{% for s in services %}\n[{{ s }}]\nport = {{ s.port }}\n{% endfor %}\n" + ) + llm = FakeEmbedder() + try: + summary = asyncio.run(import_sources([root], llm, session=db)) + assert summary.files == 10 and summary.added == 10 + assert summary.formats == { + "container": 1, "network": 1, "volume": 1, "image": 1, "pod": 1, + "kube": 1, "swap": 1, "os": 1, "endpoint": 1, "j2": 1, + } + docs = { + d.path: d + for d in db.scalars(select(Document).where(Document.source == root.name)).all() + } + titles = {p: d.title for p, d in docs.items()} + assert titles == { + # the ``#`` line is a comment in these formats — stem titles: + "quadlet/web.container": "web", + "quadlet/lan.network": "lan", + "quadlet/cache.volume": "cache", + "quadlet/alpine.image": "alpine", + "quadlet/app.pod": "app", + "quadlet/k3s.kube": "k3s", + "quadlet/zram.swap": "zram", + "quadlet/fedora.os": "fedora", + "quadlet/edge.endpoint": "edge", + "tpl/deploy.j2": "deploy", + } + for doc in docs.values(): + content = [c for c in doc.chunks if not c.is_summary] + assert content # every new-format file yields content chunks + assert all(c.embedding is not None and len(c.embedding) == 768 for c in content) + finally: + _cleanup_source(db, root.name) + + # ---------- phase 30: lite-model summaries for non-markdown files ----------