feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.
- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
import_extensions CSV gain the ten names (the original seven first);
the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
(HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
default CSV + dotted import_extension_set include the ten, the
validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
new suffix (parametrized), the .container fixture chunks >=2 under
the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
tree indexes exactly the ten new files (unknown/hidden/excluded
filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
over a temp tree with .container/.volume/.j2 -> documents + chunks
rows with stem titles; delta re-import updates only the changed .j2
doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
isolation): GET /api/docs (admin session) lists the four new-format
docs with non-zero chunk counts and stem titles; the Sources table
renders a row + .doc-link per file; the phase-26 modal shows the
.container TOML ([Container] section + sentinel) with stem title and
the container format badge; a RESE-JINJA-SENTINEL-33dd question
FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
FTS hits) — the bubble is not .is-deflected and a source chip names
templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
the revision note under the anchors table, and the §5 chunking-policy
+ §11 workflow lines. The only PLAN edit this phase.
Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.
Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
@@ -293,8 +293,11 @@ embedded.
|
||||
added=… updated=… unchanged=… pruned=… chunks=… embed_batches=…
|
||||
formats=md:203,yaml:267,…`), so it is safe to run from a cron job or
|
||||
after every commit.
|
||||
- Indexed formats (A9): **`md, markdown, txt, yaml, yml, json, py`**
|
||||
(case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). Any path with a
|
||||
- Indexed formats (A9, revised 2026-08-27): **`md, markdown, txt,
|
||||
yaml, yml, json, py`**, the Podman quadlet family (**`container,
|
||||
network, volume, image, pod, kube, swap, os, endpoint`**), and **`j2`**
|
||||
Jinja templates (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`).
|
||||
Any path with a
|
||||
**dot-prefixed component** — hidden files or vendored caches like
|
||||
`.esphome/.espressif/**` — is skipped, along with `.venv`,
|
||||
`node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`.
|
||||
@@ -302,7 +305,8 @@ embedded.
|
||||
that's how previously imported junk leaves the index.
|
||||
- Non-markdown files get format-aware chunking (YAML top-level keys /
|
||||
`---` docs, JSON top-level keys, Python top-level defs/classes via
|
||||
stdlib `ast`) and their title comes from the file stem.
|
||||
stdlib `ast`; quadlet unit files and `j2` templates are paragraph-
|
||||
packed as plain text) and their title comes from the file stem.
|
||||
- Unchanged files are **not re-embedded** — only new/changed ones, so
|
||||
refreshes are cheap.
|
||||
- After a run that **changed** the knowledge base (at least one document
|
||||
@@ -644,7 +648,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
||||
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
||||
| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn — every call the model emits consumes a round; at the cap the loop forces one final no-tools answer (0 = no tools, the kill switch) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
|
||||
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
|
||||
|
||||
Reference in New Issue
Block a user