chore(agent): phase roadmap from TODO.md — 8 phases (40–47), 24 tasks
Converts the 9 TODO items into an executable phase roadmap (Protocol B, appended after phase 39): - 40 tuning toggle anonymous flash (TODO L3) - 41 sync fail-fast + modal when a model is down (TODO L4) - 42 no reply autoscroll (TODO L5) - 43 thinking scroll back — user scroll + gated autoscroll (TODO L7) - 44 markdown tables (TODO L6) - 45 agent unlimited tool calls behind BOR_AGENT_MAX_ROUNDS (TODO L8) - 46 mobile hamburger nav (TODO L9) - 47 quadlet + jinja import formats, A9 revision (TODO L10–L11) Each phase carries a user story, a dedicated Playwright E2E suite plan, and owner-locked decisions (R1 A9 format extension, R2 phase-37 budget revision, A1–A5 scope decisions) confirmed 2026-08-27. Also records the completed phases 30–39 todo/ -> complete/ moves that were pending in the working tree. TODO.md is cleared (items now live in .agent/phases/todo/).
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Story: Import quadlet + jinja files
|
||||
|
||||
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md` L10–L11 ·
|
||||
**E2E:** `tests/e2e/test_quadlet_jinja_import.py`
|
||||
|
||||
## Bug reports (verbatim, `TODO.md` L10–L11)
|
||||
|
||||
> "Add \".container\", \".network\", \".volume\" and other quadlet files
|
||||
> to the list of allowed/parsed files"
|
||||
> "Add \".j2\" jinja files to the list of allowed/parsed files"
|
||||
|
||||
## Narrative
|
||||
|
||||
As **the owner**, my homelab notes increasingly live in Podman quadlet
|
||||
unit files (`.container`, `.network`, `.volume`, …) and Jinja templates
|
||||
(`.j2`). Neither is in the A9 import format list, so the KB is blind to
|
||||
exactly the config files I ask questions about. Both families join the
|
||||
allowed + default import formats and are parsed (chunked) by the
|
||||
importer.
|
||||
|
||||
- **Given** a source directory containing quadlet and/or `.j2` files
|
||||
- **When** the importer (CLI or sync) runs
|
||||
- **Then** those files are indexed (chunked, embedded, upserted) like any
|
||||
other A9-format file, and their content is retrievable.
|
||||
|
||||
## Acceptance criteria
|
||||
1. **Allowed set:** `_ALLOWED_IMPORT_EXTENSIONS` in `app/config.py`
|
||||
gains `container, network, volume, image, pod, kube, swap, os,
|
||||
endpoint` (the full Podman quadlet family) and `j2`; a
|
||||
`BOR_IMPORT_EXTENSIONS` env value may name any of them (the
|
||||
never-widen validator keeps rejecting truly unknown extensions).
|
||||
2. **Default set:** the default `import_extensions` CSV includes all ten
|
||||
new formats after the existing seven — a default import now picks
|
||||
them up with no env configuration.
|
||||
3. **Parsing:** `chunker.py` dispatches every new suffix to
|
||||
plain-text paragraph packing (`chunk_text`) — quadlet files are TOML
|
||||
unit files and `.j2` files are templates; no format-specific
|
||||
splitter (owner decision). Every chunk still honors
|
||||
`HARD_MAX_CHARS` (1200).
|
||||
4. **Title:** no H1 → `extract_title` falls back to the file stem, as
|
||||
with other non-markdown formats (no change needed, verified).
|
||||
5. **Behavior parity:** hidden (dot) directories are still skipped, the
|
||||
exclusion list is unchanged, sha256 delta detection / prune work
|
||||
unchanged for the new formats.
|
||||
6. **Docs:** `.env.example`'s `BOR_IMPORT_EXTENSIONS` comment, the
|
||||
README's format list, and an **A9 revision note** in
|
||||
`.agent/PLAN.md` (owner permission 2026-08-27, `TODO.md` L10–L11)
|
||||
record the extended set.
|
||||
|
||||
## Owner-confirmed (2026-08-27, roadmap R1)
|
||||
1. **Full quadlet family:** `container, network, volume, image, pod,
|
||||
kube, swap, os, endpoint` — plus `j2`.
|
||||
2. **Chunked as plain text** — no TOML/Jinja-aware splitting.
|
||||
3. **A9 is revised** with a PLAN.md revision note (the established
|
||||
owner-permission pattern).
|
||||
|
||||
## UI Visualization & Structure
|
||||
- **Config** (`app/config.py`): extend the `_ALLOWED_IMPORT_EXTENSIONS`
|
||||
frozenset + the `import_extensions` default string (comment cites the
|
||||
A9 revision 2026-08-27). No validator change — it already
|
||||
normalizes/dedups and rejects unknowns.
|
||||
- **Chunker** (`app/rag/chunker.py`): ten new `_FORMAT_CHUNKERS` entries
|
||||
→ `chunk_text`; module docstring's format list updated.
|
||||
- **Fixtures:** `tests/fixtures/docs/homelab/quadlet/compose.container`
|
||||
(realistic quadlet TOML: `[Unit]` / `[Service]` / `[Container]`
|
||||
sections, one unique sentinel token, >1200 chars to exercise
|
||||
sub-splitting), `…/quadlet/lan.network`, `…/quadlet/cache.volume`,
|
||||
and `tests/fixtures/docs/homelab/templates/deploy.j2` (Jinja snippet
|
||||
with `{{ … }}` / `{% … %}` tags + its own sentinel).
|
||||
- **Tests:** unit (`test_config.py` allowed-set/default/validator;
|
||||
`test_chunker.py` dispatch for every new suffix + fixture chunking;
|
||||
`test_importer.py` directory walk picks the new files up);
|
||||
integration (import_sources over a temp dir with quadlet+j2 files →
|
||||
documents + chunks rows).
|
||||
- **Non-goals:** no new DB column, no format badge change (the viewer
|
||||
shows the extension it already shows), no summary-model changes.
|
||||
|
||||
## Playwright Mapping Rule
|
||||
**Test Scenario → `tests/e2e/test_quadlet_jinja_import.py`** (mock
|
||||
LLM; DB up):
|
||||
1. `test_quadlet_and_jinja_indexed` — truncate + import the fixture
|
||||
tree (the `test_import_documents.py` pattern): `GET /api/docs` lists
|
||||
the `.container` / `.network` / `.volume` / `.j2` files with
|
||||
non-zero chunk counts.
|
||||
2. `test_sources_table_shows_them` (admin) — the Sources table renders
|
||||
rows for the new files; their path links open the document modal.
|
||||
3. `test_container_content_viewable` — the viewer modal shows the
|
||||
`.container` file's TOML content (sentinel token present) with its
|
||||
stem as the title.
|
||||
4. `test_jinja_retrievable_not_deflected` — ask a question containing
|
||||
the `.j2` file's sentinel word: the FTS hit keeps the honesty gate
|
||||
honest-positive (A8) — the answer bubble is **not**
|
||||
`.is-deflected` and the source chip names the `.j2` document.
|
||||
5. `test_default_walk_includes_new_formats` (regression, unit-backed) —
|
||||
a default-extensions walk over a temp tree with all ten new
|
||||
extensions indexes every file; hidden directories + the exclusion
|
||||
list still filter (covered by `test_importer.py` in the regression
|
||||
pass).
|
||||
Reference in New Issue
Block a user