feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.
- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
import_extensions CSV gain the ten names (the original seven first);
the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
(HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
default CSV + dotted import_extension_set include the ten, the
validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
new suffix (parametrized), the .container fixture chunks >=2 under
the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
tree indexes exactly the ten new files (unknown/hidden/excluded
filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
over a temp tree with .container/.volume/.j2 -> documents + chunks
rows with stem titles; delta re-import updates only the changed .j2
doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
isolation): GET /api/docs (admin session) lists the four new-format
docs with non-zero chunk counts and stem titles; the Sources table
renders a row + .doc-link per file; the phase-26 modal shows the
.container TOML ([Container] section + sentinel) with stem title and
the container format badge; a RESE-JINJA-SENTINEL-33dd question
FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
FTS hits) — the bubble is not .is-deflected and a source chip names
templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
the revision note under the anchors table, and the §5 chunking-policy
+ §11 workflow lines. The only PLAN edit this phase.
Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.
Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
This commit is contained in:
+17
-5
@@ -56,7 +56,7 @@ that"* and offers alternatives instead of hallucinating.
|
||||
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
|
||||
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
|
||||
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
|
||||
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
|
||||
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-27) |
|
||||
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
|
||||
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
|
||||
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
|
||||
@@ -83,6 +83,13 @@ that"* and offers alternatives instead of hallucinating.
|
||||
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
|
||||
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
|
||||
> marker.
|
||||
>
|
||||
> **A9 revision (phase 47, owner permission 2026-08-27):** the format
|
||||
> set extends with the Podman quadlet family (`container, network,
|
||||
> volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja
|
||||
> templates) — plain-text chunking (`chunk_text`), owner: `TODO.md`
|
||||
> L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the
|
||||
> hidden-dir/exclusion invariants are unchanged.
|
||||
|
||||
---
|
||||
|
||||
@@ -251,8 +258,11 @@ preceding heading in the text for retrieval quality.
|
||||
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
|
||||
`---` separators (key line kept as anchor); `json` pretty-printed, split on
|
||||
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
|
||||
`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
|
||||
hard cap (aipi ~1024-token request limit).
|
||||
`txt` on paragraphs; markdown unchanged. The quadlet family
|
||||
(`container, network, volume, image, pod, kube, swap, os, endpoint`)
|
||||
and `j2` (Jinja templates) are plain-text chunked — no format-specific
|
||||
splitter (A9 revised 2026-08-27, phase 47). Every format honors the
|
||||
1200-char hard cap (aipi ~1024-token request limit).
|
||||
|
||||
---
|
||||
|
||||
@@ -455,8 +465,10 @@ uv run python -m scripts.llm_probe # sanity: models + dim
|
||||
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
|
||||
(no re-embedding); changed files are re-chunked + re-embedded (chunks
|
||||
replaced atomically); `--prune` removes docs whose files disappeared or no
|
||||
longer match the format filter. Formats per A9 (revised): `md, markdown,
|
||||
txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
|
||||
longer match the format filter. Formats per A9 (revised 2026-08-27):
|
||||
`md, markdown, txt, yaml, yml, json, py`, the quadlet family
|
||||
(`container, network, volume, image, pod, kube, swap, os, endpoint`), and
|
||||
`j2` (plain-text chunked) (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
|
||||
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
|
||||
ranks live hybrid results for a question (retrieval tuning).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user