refactor(agents): migrate .agent/ planning tree to .agents/

Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
2026-09-05 10:57:07 -04:00
parent 766702c750
commit dbf2af26c6
1118 changed files with 664 additions and 664 deletions
@@ -0,0 +1,40 @@
# Phase 56 — User-Extensible Import Extensions
**Source:** `TODO.md` L6 — "Allow the user to specify extensions to be read in .env, don't hard-code working extensions. Supply some by default in .env.example"
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-08-31)
**Context:** `BOR_IMPORT_EXTENSIONS` already exists: the `import_extensions` CSV field in `app/config.py` (default = the A9 family), the `import_extension_set` property, the `_import_extensions_known` validator (`app/config.py:172`), and a commented default line in `.env.example`. The one gap versus the TODO is the A9 lock: the allowed set is hard-coded in `_ALLOWED_IMPORT_EXTENSIONS` (`app/config.py:20`), so the env var may only **narrow** the family, never **add** an extension ("may only narrow, never widen" — `app/config.py` module docstring, `.env.example` "Import scope" comment). The importer filters with `path.suffix.lower() in extensions` (`app/rag/importer.py:137`); non-markdown formats are chunked as plain text and (phase 30) get a `lite` summary, so any new extension flows through the existing pipeline with no importer changes.
## Objective
`BOR_IMPORT_EXTENSIONS` may name **any** extension (the A9 family becomes the built-in default, not the ceiling), fail-loud validation is kept for typos, and `.env.example` ships the default list as a usable, documented example.
## Dependencies
- — (none)
## Tasks
1. `01_lift_extension_allowlist.md` — config change: A9 family becomes the default, the validator accepts any well-formed extension, `.env.example` documents it.
2. `02_e2e_extensions_env.md` — integration + Playwright proof (a novel `.sh` extension imports end-to-end), regressions, atomic commit.
## Testing & Quality
- Unit: `tests/unit/test_config*.py` — the validator now **accepts** novel extensions (`md,sh` → `{'.md', '.sh'}`), still rejects an empty list and malformed tokens; default parsing unchanged.
- Integration: `tests/integration/test_import_extensions_env.py` — in-process `import_sources` against a story-dedicated fixture dir with `import_extensions="md,sh"` (novel extension imported, plain-text chunked, mock summary) and `import_extensions="md"` (narrowing still works).
- E2E (mandatory, A16): `tests/e2e/test_import_extensions_env.py`, run in isolation (DB up) — Sources page (admin) lists the novel-extension doc with its format badge; anonymous still gets the sign-in gate.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `BOR_IMPORT_EXTENSIONS=md,sh` (or any other well-formed novel extension) starts the app and imports `.sh` files; the Sources page shows them with format badge `sh`.
- [ ] `BOR_IMPORT_EXTENSIONS=` (empty) or a malformed token (e.g. `md,sh!`) fails loudly at startup with a named value.
- [ ] `.env.example` carries the default list (A9 family + quadlet + jinja) with a comment stating any extension is allowed.
- [ ] Existing suites stay green: `test_import_documents.py`, `test_quadlet_jinja_import.py` (isolated runs).
- [ ] `uv run pytest` green; coverage TOTAL >90%.
- [ ] `uv run pytest tests/e2e/test_import_extensions_env.py -v --no-cov` green in isolation (DB up).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` (`.agents/` stays untracked — owner instruction, commit 281f355).
## Locked decisions
- **Owner-locked (2026-08-31, roadmap confirmation, D1):** `BOR_IMPORT_EXTENSIONS` may **extend** beyond the A9 family — the hard-coded set becomes the default, not the ceiling. This revises the A9 "narrow-only" clause (2026-08-21/27) by owner permission; the A9 list remains the built-in default and the documented example.
- **Fail-loud kept (house style):** empty list and malformed tokens are rejected at startup (validator), so a typo never walks zero files silently.
## Commit
```bash
git add app/ tests/ .env.example && git commit --no-gpg-sign -m "feat(import): user-extensible BOR_IMPORT_EXTENSIONS — any well-formed extension, A9 family stays the default"
```
@@ -0,0 +1,32 @@
# Task 01 — Lift the extension allowlist to a default
**Phase:** `56_import_extensions_env` · **Source:** `TODO.md:6` — "Allow the user to specify extensions to be read in .env, don't hard-code working extensions. Supply some by default in .env.example"
**Story:** n/a (TODO-derived)
## Objective
`app/config.py` treats the A9 family as the **default** import scope and accepts any well-formed extension named in `BOR_IMPORT_EXTENSIONS`; `.env.example` documents the setting with the default list.
## Work
1. `app/config.py`:
- Rename `_ALLOWED_IMPORT_EXTENSIONS` → `_DEFAULT_IMPORT_EXTENSIONS` (same A9 list, incl. the quadlet family + `j2`). Update its comment: this is the built-in default and the `.env.example` example — no longer a ceiling.
- Rewrite the module docstring and the field comment around `import_extensions` (currently "may only narrow, never widen" — module header ~L12–19, field docstring ~L146): the user may name **any** extension; the default shown is the A9 family.
- Rewrite the validator `_import_extensions_known`: parse parts (strip, `lstrip(".")`, lowercase); reject an empty result (`"import_extensions must name at least one format"`); reject any token that does not match `^[a-z0-9]{1,16}$` with a `ValueError` naming the offending token(s) (fail loud, `agent_max_rounds`-style). Delete the membership test against the fixed set.
- `ASSUMPTION:` the token shape guard `^[a-z0-9]{1,16}$` is the typo guard — it keeps path-ish values (`../x`, `/etc/passwd`, `sh!`) out of the set while still allowing anything a file could actually be suffixed with.
2. `.env.example`: uncomment the `BOR_IMPORT_EXTENSIONS=…` line (full default list) and rewrite the "Import scope" section comment: extensions the importer reads; **any** extension is allowed (lowercase, no dot); the value shown is the built-in default (A9 family + quadlet + jinja).
3. Sweep stale references: `rg -n "narrow|_ALLOWED_IMPORT_EXTENSIONS" app/ scripts/ frontend/ tests/` — update every hit that still claims the set can only narrow (known candidates: `app/config.py` header, `scripts/import_docs.py` docstrings/comments, `app/rag/importer.py` docstring).
4. Adjust the config unit tests that assert unknown-extension rejection (they now assert acceptance of novel extensions — see Testing & Quality).
## Testing & Quality
- Unit (`tests/unit/test_config*.py`, adapt in place):
- default value parses to the full A9 family set (unchanged pin).
- `import_extensions="md,sh"` → set `{".md", ".sh"}` (novel extension accepted).
- `"MD,.Py"` normalizes to `{".md", ".py"}` (case + leading-dot tolerance unchanged).
- `""` / `",,"` → startup `ValueError` naming the field.
- `"md,sh!"` (and `"md,../x"`) → `ValueError` naming the malformed token.
- Existing importer unit tests stay green (they exercise `iter_importable_files` with explicit sets — no behavior change for a fixed set).
## Completion Criteria
- [ ] Validator accepts novel extensions, rejects empty/malformed, with named-value error messages.
- [ ] `_DEFAULT_IMPORT_EXTENSIONS` is the only surviving constant; no "narrow-only" wording remains anywhere.
- [ ] `.env.example` ships the default list as an active, documented example.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,34 @@
# Task 02 — Import proof (novel extension) + E2E + commit
**Phase:** `56_import_extensions_env` · **Source:** `TODO.md:6` — "Allow the user to specify extensions to be read in .env, don't hard-code working extensions. Supply some by default in .env.example"
**Story:** n/a (TODO-derived)
## Objective
Prove a **novel** (non-A9) extension flows end-to-end — config → import → chunks → mock summary → Sources page — run the regressions, and commit the phase.
## Work
1. Story-dedicated fixture dir `tests/fixtures/extension_kb/` (house pattern: `summary_kb/` keeps the shared `docs/` fixtures pinned):
- `homelab/scripts/uptime.sh` — a handful of distinctive lines incl. a sentinel word (e.g. `UPTIME-PROBE-SENTINEL-9c2f`) so retrieval/row assertions are unambiguous.
- `homelab/notes/note.md` — a small markdown control doc.
2. `tests/integration/test_import_extensions_env.py` (in-process, mock LLM port — the `_import_fixtures` pattern from `tests/e2e/test_import_documents.py`):
- `Settings(_env_file=None, llm_base_url=<mock>, import_extensions="md,sh")` → `import_sources([FIXTURE], LLMClient(settings))` → a `homelab/scripts/uptime.sh` row with format `sh`, non-empty `summary` (mock `SUMMARY_MODE` digest), chunked; `note.md` imported as control.
- Same fixture with `import_extensions="md"` → **no** `.sh` row (narrowing still works — the A9-era behavior is preserved as a special case).
3. `tests/e2e/test_import_extensions_env.py` (Playwright, DB up, mock LLM; admin login via `e2e.auth_helpers.login`):
- Seed the same fixture in-process with `import_extensions="md,sh"` (the `test_import_documents.py` seeding thread pattern — the fixture, not the subject).
- Admin: `/sources.html` lists `homelab/scripts/uptime.sh` with format badge `sh` (the row's path cell + badge, house assertion style).
- Anonymous fresh context: the Sources gate renders and **no** `/api/docs` request fires (`page.on("request")` pin — the `test_import_documents.py`/phase-16 pattern).
- DB isolation: the fixture's `source` name (`extension_kb`) is distinctive — never assert on absolute row counts; delete the rows it creates in a `finally` (admin cookie).
4. Regression pass (isolation runs): `test_import_documents.py`, `test_quadlet_jinja_import.py` (both pin the A9 family with the **default** config — must be byte-for-byte unchanged in behavior).
5. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
6. Commit (Conventional Commits, `--no-gpg-sign`) — the message from the phase overview's Commit section; move `.agents/phases/todo/56_import_extensions_env/` → `.agents/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_import_extensions_env.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A `.sh` file imports with `import_extensions="md,sh"` (row, summary, chunks) and is visible on the admin Sources page with badge `sh`.
- [ ] `import_extensions="md"` excludes the `.sh` file (narrowing unchanged).
- [ ] Anonymous Sources gate regression holds (no `/api/docs` call).
- [ ] Regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.