phase: 102_extensionless_filenames
All verification complete — every gate green, no defects found in previously completed work. **Phase 102 final verification pass — report** Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design): - `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct - Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched) Test / lint results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate) - `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up - Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓ - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness. Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# Phase 102 — Extensionless filenames: sync `Dockerfile` / `Containerfile` when their name is in `BOR_IMPORT_EXTENSIONS`
|
||||
|
||||
**Source:** owner report (chat): "files without extensions never get synced, so 'Dockerfile' and 'Containerfile' get skipped even if I put them in BOR_IMPORT_EXTENSIONS."
|
||||
**Story:** n/a (defect — the import pipeline of `38_local_directory_sources` / `56_import_extensions_env`; the badge display rides on the phase-10 viewer).
|
||||
**Context:** the importer's format filter is `path.suffix.lower() in extensions` (`app/rag/importer.py`, `iter_importable_files`) and `Settings.import_extension_set` (the dotted set, `app/config.py`) feeds it. `Path("Dockerfile").suffix` is `""`, so **extensionless files never match — no value of `BOR_IMPORT_EXTENSIONS` can import them**. The validator already accepts `dockerfile` as a well-formed token (lowercase alphanumerics, 1-16 chars, phase 56 — "any extension"), so today the user can *type* the format and it still silently imports nothing. The viewer's format badge (`doc_format`, `app/api/docs.py`, rendered by `frontend/assets/document.js`) falls back to `text` for any suffix-less path; the import `formats=` counter (`ImportSummary.formats`) keys extensionless files under `unknown`.
|
||||
|
||||
## Objective
|
||||
An extensionless file is imported **iff its lowercased full filename is a token in `BOR_IMPORT_EXTENSIONS`** — exact name, case-insensitive. With `dockerfile,containerfile` in the env (the owner's current config), `Dockerfile` and `Containerfile` sync like any other A9 file: chunks, `lite` summary, Sources tree, viewer with a truthful format badge.
|
||||
|
||||
## Dependencies
|
||||
- `56_import_extensions_env` (the env-driven extension scope this phase extends) — complete
|
||||
- `38_local_directory_sources` / `97_kb_tree_catalog` (import pipeline + drill-down tree the E2E asserts on) — complete
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
### D1 — the matching rule (task 01)
|
||||
A file under a source root is in scope **iff**:
|
||||
1. its lowercased dotted suffix is a member of `import_extension_set` (the existing rule — `kubernetes.md` → `.md`), **or**
|
||||
2. it has **no suffix** and its lowercased **full filename** equals a bare token of `import_extension_set` (`Dockerfile` → `dockerfile`).
|
||||
|
||||
Consequences (pinned by tests): `DOCKERFILE` matches (case-insensitive name); `mydockerfile` and `dockerfile`-suffixed lookalikes do **not** (exact name only — the list is user-controlled and exact is the only predictable rule); `Dockerfile.dev` is governed by its `.dev` suffix as today; dot-prefixed files (`.dockerfile`) stay skipped by the pre-existing hidden-component rule; the rule applies at any depth (`services/api/Dockerfile` matches).
|
||||
|
||||
### D2 — one choke point (task 01)
|
||||
`app/rag/importer.py` gains `match_extension(path: Path, extensions: frozenset[str]) -> str | None` — returns the matched **bare lowercased token** (``md``, ``dockerfile``) or `None` when out of scope; `extensions` keeps the existing dotted form (``{'.md'}``), bare tokens are derived by stripping the leading dot. `iter_importable_files` filters with `match_extension(path, extensions) is not None` (the walk and the phase-64 progress pre-walk both call this function — one change covers both, `total` stays honest). The `summary.formats` counter in `import_sources` uses `match_extension(path, …) or "unknown"`, so an imported `Dockerfile` counts under `dockerfile:1` in the PLAN §9 line, not `unknown:1`.
|
||||
|
||||
### D3 — the badge (task 02)
|
||||
`doc_format(path, extensions: frozenset[str] = frozenset())` in `app/api/docs.py`: a **non-empty suffix still returns the suffix token unconditionally** (display must never depend on the import list — an out-of-scope `readme.rst` still badges `rst`); a suffix-less path returns the name token when it matches per D1, else `text` (today's fallback, byte-identical for every existing row). The content endpoint (`GET /api/documents/content`) passes `get_settings().import_extension_set` (`app.config`, lru-cached). `frontend/assets/document.js` needs no change: only `md`/`markdown` trigger markdown rendering, so a `dockerfile` badge renders the raw `<pre>` view — correct for a build file.
|
||||
|
||||
### D4 — no changes (pinned)
|
||||
Title stays `full_path.stem` (`Dockerfile` — already correct); the phase-30 `lite` summary applies (non-markdown); prune semantics unchanged — a file that stops matching (file renamed, or token removed from the env) leaves `seen` and is deleted by the next `prune=True` run (the A9 junk precedent).
|
||||
|
||||
### D5 — docs
|
||||
`app/config.py` `import_extensions` docstring + the `.env.example` "Import scope" comment gain the extensionless rule, with `dockerfile,containerfile` as the example (no default-list change — the built-in A9 default stays exactly as phase 56 locked it).
|
||||
|
||||
## Tasks
|
||||
1. `01_name_token_matching.md` — `match_extension` in the importer + the walk + the `formats` counter (unit-pinned).
|
||||
2. `02_format_badge_and_docs.md` — the viewer badge via `doc_format` + config/`.env.example` docs.
|
||||
3. `03_integration_and_e2e.md` — the dedicated integration suite + the Playwright story suite + regressions + atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_importer.py` — the matching matrix (D1), walk inclusion/exclusion, the `formats` counter; `tests/unit/test_document_viewer.py` — `doc_format` (with/without the token set, suffix precedence, `text` fallback).
|
||||
- Integration: `tests/integration/test_import_extensionless.py` (new) — in-process `import_sources` against a story-dedicated fixture dir with the deterministic mock LLM (the phase-02/56 seeding pattern): positive import (counts, per-format `dockerfile:1,containerfile:1,md:1`, lite summary generated for the extensionless files), negative import (no token → not walked), prune-on-token-removal; `tests/integration/test_document_content.py` — the `format` field carries `dockerfile` for an extensionless row and `text` for an unknown-name row.
|
||||
- E2E (mandatory, house rule 4): `tests/e2e/test_extensionless_import.py`, run in isolation (DB up) — admin Signs in, the drill-down tree lists `Dockerfile`/`Containerfile`, the viewer modal shows badge `dockerfile` + title + summary, `Makefile` (no token) is absent, rows cleaned up in `finally`.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the phase is "Complete" only when the full suite passes and the gate holds.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `BOR_IMPORT_EXTENSIONS` containing `dockerfile,containerfile` (plus the usual formats) makes a `Dockerfile` and a `Containerfile` under a source sync end-to-end: imported, chunked, `lite`-summarized, listed in the Sources drill-down tree, openable in the viewer.
|
||||
- [ ] Case-insensitivity: `DOCKERFILE` imports under the `dockerfile` token; `mydockerfile` does **not**; `Dockerfile.dev` follows its `.dev` suffix; `.dockerfile` stays hidden-skipped.
|
||||
- [ ] The import `formats=` counter reports `dockerfile:1` (not `unknown:1`) for an imported `Dockerfile`.
|
||||
- [ ] The viewer format badge shows `dockerfile`/`containerfile`; a suffix-less file whose name is not in the env still badges `text`; every suffixed row's badge is unchanged.
|
||||
- [ ] Removing a token from the env prunes the previously-imported extensionless documents on the next `prune=True` run.
|
||||
- [ ] The existing suites stay green in isolation: `test_import_documents.py`, `test_import_extensions_env.py`, `test_quadlet_jinja_import.py`, `test_document_viewer.py`, `test_kb_tree.py`.
|
||||
- [ ] `uv run pytest` green; `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` green in isolation (DB up).
|
||||
- [ ] No behavior change to completed phases for any suffixed path (the existing walk rule is untouched — only the suffix-less branch is new).
|
||||
- [ ] One atomic Conventional Commit, `--no-gpg-sign` (e.g. `fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS`), `.agents/` phase files moved to `complete/` with the phase.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS"
|
||||
```
|
||||
@@ -0,0 +1,42 @@
|
||||
# Task 01 — The name-token match: `match_extension`, the walk, the `formats` counter
|
||||
|
||||
**Phase:** `102_extensionless_filenames` · **Source:** owner report — "files without extensions never get synced … 'Dockerfile' and 'Containerfile' get skipped even if I put them in BOR_IMPORT_EXTENSIONS."
|
||||
**Story:** n/a (defect).
|
||||
|
||||
## Objective
|
||||
One pure helper decides import scope; the walk and the per-format counter use it; extensionless files whose lowercased full name is a `BOR_IMPORT_EXTENSIONS` token are finally imported.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py` — new module-level pure function, placed near `iter_importable_files`:
|
||||
```python
|
||||
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
|
||||
"""The bare lowercased token *path* imports under, or ``None``.
|
||||
|
||||
1. Non-empty lowercased dotted suffix in *extensions* (the A9 rule —
|
||||
``kubernetes.md`` → ``md``).
|
||||
2. No suffix: the lowercased FULL filename equals a bare token of
|
||||
*extensions* (``Dockerfile`` → ``dockerfile``) — the phase-102
|
||||
extensionless rule. Exact name only: ``mydockerfile`` never
|
||||
matches the ``dockerfile`` token.
|
||||
"""
|
||||
```
|
||||
*extensions* keeps the existing dotted form (``{'.md', '.dockerfile'}``); derive the bare token set by stripping the leading dot. Rule 1 first; rule 2 only when ``path.suffix`` is empty.
|
||||
2. `app/rag/importer.py` — `iter_importable_files`: replace the `path.suffix.lower() not in extensions` skip with `match_extension(path, extensions) is None`. The hidden-component check and the phase-89 ignore-prefix check stay before it, byte-identical.
|
||||
3. `app/rag/importer.py` — the `summary.formats` line in `import_sources` (`ext = path.suffix.lower().lstrip(".") or "unknown"`): use `match_extension(path, llm.settings.import_extension_set) or "unknown"` so an imported `Dockerfile` counts under `dockerfile`, never `unknown`.
|
||||
4. `app/rag/importer.py` — module docstring: one sentence for the extensionless rule in the scope paragraph (tokens may also name extensionless files by exact lowercased name — phase 102).
|
||||
|
||||
ASSUMPTIONS (pinned in `00_phase.md` D1/D2): exact-name, case-insensitive matching; no partial name matching; dotted set kept as the parameter form so no `app/config.py` change is needed in this task.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/test_importer.py`, new tests next to the existing `iter_importable_files` ones):
|
||||
- `match_extension` matrix: `kubernetes.md` + `{'.md'}` → `md`; `Kubernetes.MD` → `md` (case-insensitive suffix, as today); `Dockerfile` + `{'.md', '.dockerfile'}` → `dockerfile`; `DOCKERFILE` → `dockerfile`; `Dockerfile` without the token → `None`; `mydockerfile` with the token → `None`; `Dockerfile.dev` with only `dockerfile` → `None` (with `.dev` it → `dev`); `readme.rst` + `{'.md'}` → `None`.
|
||||
- walk: a fixture tree with `Dockerfile`, `Containerfile`, `mydockerfile`, `Dockerfile.dev`, `.dockerfile`, `notes.md` — with tokens `md,dockerfile,containerfile` the walk yields exactly `Dockerfile`, `Containerfile`, `notes.md` (sorted); the hidden `.dockerfile` stays skipped.
|
||||
- counter: in-process `import_sources` (the file's existing `db` + fake-LLM pattern) over a tree with one `Dockerfile` + one `.md` → `summary.formats == {"dockerfile": 1, "md": 1}` and the PLAN §9 log line carries `dockerfile:1` (not `unknown`).
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `match_extension` exists, pure, dotted-set parameter, rule order pinned by the matrix above
|
||||
- [ ] `iter_importable_files` + the progress pre-walk import extensionless name-token files (both go through the helper)
|
||||
- [ ] the `formats` counter reports the matched token, never `unknown`, for in-scope files
|
||||
- [ ] full test suite green, coverage >90%, `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change for any suffixed path (existing `test_importer.py` cases byte-identical)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 02 — The truthful badge: `doc_format` learns the name-token rule + config/env docs
|
||||
|
||||
**Phase:** `102_extensionless_filenames` · **Story:** n/a (defect, display half).
|
||||
|
||||
## Objective
|
||||
The viewer's format badge shows `dockerfile`/`containerfile` for extensionless documents that matched by name (instead of the generic `text`), and the config/`.env.example` docs tell the user the extensionless rule exists.
|
||||
|
||||
## Work
|
||||
1. `app/api/docs.py` — `doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str`:
|
||||
- non-empty suffix → the suffix token, **unconditionally** (display never depends on the import list — an out-of-scope `readme.rst` still badges `rst`);
|
||||
- no suffix → reuse the task-01 helper semantics from `app/rag/importer.py` (import `match_extension` — `app.api.docs` already imports from `app.rag`): the lowercased full name returns the token when it is one of `extensions` (dotted form, as passed), else `text`.
|
||||
- Default `extensions=frozenset()` ⇒ today's exact behavior for every caller that passes nothing (byte-identical).
|
||||
- Docstring: update the fallback line — ``text`` when the path has no suffix **and its name is not a configured token (phase 102)**.
|
||||
2. `app/api/docs.py` — the content endpoint (`GET /api/documents/content`, `format=doc_format(doc.path)` at ~L143): pass `get_settings().import_extension_set` (`from app.config import get_settings` — the lru-cached accessor; no per-request Settings construction).
|
||||
3. `app/config.py` — `import_extensions` docstring: one sentence — a token also matches **extensionless files whose lowercased full filename equals it exactly** (`dockerfile` → `Dockerfile`), case-insensitive, no partial names (phase 102).
|
||||
4. `.env.example` — "Import scope" comment: add the same rule with the example `dockerfile,containerfile` (e.g. `add your own (e.g. md,sh,toml,dockerfile,containerfile)`); the default `BOR_IMPORT_EXTENSIONS` line is unchanged.
|
||||
5. No frontend change: `frontend/assets/document.js` renders `doc.format` into the `format-badge` span and only `md`/`markdown` get markdown rendering — a `dockerfile` badge already lands in the raw `<pre>` view, which is the correct treatment for a build file. Verify, don't edit.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration (`tests/integration/test_document_content.py`, extend the `test_content_format_from_suffix` neighborhood — seed via the file's existing `_seed_doc` + TRUNCATE-in-`finally` pattern):
|
||||
- seeded `path="services/api/Dockerfile"` (source `Homelab`) → `body["format"] == "dockerfile"` under the default settings (the endpoint reads the real `get_settings()` — if the test needs the token, set `BOR_IMPORT_EXTENSIONS` for the client/session via the house env-override pattern used by other API tests; the default list does **not** contain `dockerfile`, so the assertion must run with a settings carrying it);
|
||||
- seeded `path="README"` → `format == "text"` (unchanged — `readme` is not a token; the existing line at ~L554 stays green);
|
||||
- seeded `path="notes/README.dev"` → `format == "dev"` (suffix precedence — display ignores the import list);
|
||||
- `doc_format` directly: default-args call on an extensionless path → `text` (the no-arg contract pinned).
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the content endpoint's `format` field carries the name token for an extensionless document whose name is in `BOR_IMPORT_EXTENSIONS`, `text` otherwise, and every suffixed value is unchanged
|
||||
- [ ] `doc_format` with no second argument behaves byte-identically to today
|
||||
- [ ] `app/config.py` docstring + `.env.example` document the extensionless rule with the `dockerfile,containerfile` example
|
||||
- [ ] full test suite green, coverage >90%, `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] `frontend/assets/document.js` verified untouched (raw `<pre>` rendering is correct for the new badge values)
|
||||
@@ -0,0 +1,39 @@
|
||||
# Task 03 — The proof: dedicated integration suite, the Playwright story suite, regressions, commit
|
||||
|
||||
**Phase:** `102_extensionless_filenames` · **Story:** n/a (defect — phase-level E2E per house rule 4).
|
||||
|
||||
## Objective
|
||||
End-to-end proof that `Dockerfile`/`Containerfile` sync when their names are in `BOR_IMPORT_EXTENSIONS` — import → chunks → `lite` summary → drill-down tree → viewer badge — plus the negative and prune proofs, regressions, and the atomic commit.
|
||||
|
||||
## Work
|
||||
1. Fixture `tests/fixtures/extensionless_kb/` (new, story-dedicated, distinctive source name = the dir name, the phase-56 pattern):
|
||||
- `Dockerfile` — a few lines of plausible compose/build content carrying a unique sentinel (e.g. `DOCKERFILE-PROBE-SENTINEL-7a3e`);
|
||||
- `Containerfile` — same shape, its own sentinel;
|
||||
- `Makefile` — a few lines, **no token in the env under test** (the negative control);
|
||||
- `notes.md` — one short markdown doc (the control row + the `md` count).
|
||||
2. Integration `tests/integration/test_import_extensionless.py` (new; the `tests/integration/test_import_extensions_env.py` pattern — in-process `import_sources` against a fixture/tmp tree, `Settings(_env_file=None, import_extensions=…)` + the house mock LLM, `db` fixture, TRUNCATE/delete rows in `finally`):
|
||||
- positive: extensions `md,dockerfile,containerfile` over the fixture → `added == 3`, `errors == 0`, `formats == {"dockerfile": 1, "containerfile": 1, "md": 1}` (NO `unknown` key), the `Dockerfile` row has chunks and a `lite` summary (non-markdown → phase-30 path);
|
||||
- case: a `DOCKERFILE` file + token `dockerfile` imports (its row's path is `DOCKERFILE`);
|
||||
- negative: extensions `md` only → `notes.md` imported, `Dockerfile`/`Containerfile`/`Makefile` absent from `documents`;
|
||||
- prune: import with the tokens, then re-run `prune=True` with extensions `md` → the two extensionless rows are deleted, the `md` row survives (mirrors `test_prune_removes_files_now_excluded_by_format_filter`).
|
||||
3. E2E `tests/e2e/test_extensionless_import.py` (new; **copy the skeleton of `tests/e2e/test_import_extensions_env.py`** — the module-docstring conventions, the in-thread seeding pattern, `_drill`/`_go_top`, the distinctive-source row cleanup in `finally`):
|
||||
- boot/app with the house `mock_llm` + `db_ready` fixtures; seed via `import_sources([FIXTURES], LLMClient(settings))` with `import_extensions="md,dockerfile,containerfile"` (`Settings(_env_file=None, llm_base_url=…, import_extensions=…)`);
|
||||
- test 1 (admin): summary assertions as in the integration positive case, then `login(page, app_url)`, `_drill(page, SOURCE)` — rows for `Dockerfile`, `Containerfile`, `notes.md` each count 1; `Makefile` count 0 (negative control); click the `Dockerfile` path link → same-page modal: `#doc-modal-meta .doc-source-badge` = source, `.format-badge` = `dockerfile` (the D3 badge), title `Dockerfile`, `pre.doc-raw` contains the sentinel, the phase-30 summary line renders (house assertion style — `test_summary_in_viewer.py`); still on `/sources.html`;
|
||||
- test 2 (anonymous): the sign-in gate renders and `GET /api/docs` 403s (the phase-16 contract, same shape as the phase-56 suite's second test).
|
||||
4. Regressions (isolated runs, per AGENTS.md rule 9): `tests/e2e/test_import_documents.py`, `test_import_extensions_env.py`, `test_quadlet_jinja_import.py`, `test_document_viewer.py`, `test_kb_tree.py`.
|
||||
5. Commit: full suite + `--cov=app` >90% + `uv run ruff check . && uv run pyright` clean, then move this phase dir `102_extensionless_filenames/` to `.agents/phases/complete/` and commit **everything together** (AGENTS.md rule 8 — `.agents/` is tracked), one atomic commit:
|
||||
```bash
|
||||
git add app/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS"
|
||||
```
|
||||
|
||||
## Testing & Quality
|
||||
- The suites above ARE this task's tests; coverage gate **>90%** on `app/` (the fixture files and the E2E file need no coverage, `app/` is the measured package).
|
||||
- E2E runs in isolation: `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` (DB up: `podman compose up -d db`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `tests/fixtures/extensionless_kb/` exists with `Dockerfile`, `Containerfile`, `Makefile`, `notes.md` (sentinels unique to this suite)
|
||||
- [ ] integration suite green: positive (counts + `dockerfile:1,containerfile:1,md:1` + summary), case, negative, prune
|
||||
- [ ] E2E green in isolation: tree rows (incl. `Makefile` absent), the `dockerfile` format badge in the modal, the raw-content sentinel, the anonymous gate
|
||||
- [ ] all five regression suites green in isolation
|
||||
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] one atomic Conventional Commit with `--no-gpg-sign` (message above); phase dir moved to `.agents/phases/complete/` in the same commit
|
||||
Reference in New Issue
Block a user