phase: 102_extensionless_filenames
All verification complete — every gate green, no defects found in previously completed work. **Phase 102 final verification pass — report** Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design): - `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct - Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched) Test / lint results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate) - `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up - Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓ - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness. Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
This commit is contained in:
+16
@@ -0,0 +1,16 @@
|
|||||||
|
All verification complete — every gate green, no defects found in previously completed work.
|
||||||
|
|
||||||
|
**Phase 102 final verification pass — report**
|
||||||
|
|
||||||
|
Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design):
|
||||||
|
- `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct
|
||||||
|
- Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched)
|
||||||
|
|
||||||
|
Test / lint results (exact commands):
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate)
|
||||||
|
- `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up
|
||||||
|
- Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
|
||||||
|
|
||||||
|
Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness.
|
||||||
|
Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
|
||||||
+96
@@ -0,0 +1,96 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
.................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 181 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 134 1 99%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 115 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 140 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 125 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 316 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 121 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 197 3 98%
|
||||||
|
app/rag/llm.py 243 1 99%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 307 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3862 15 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+14
@@ -0,0 +1,14 @@
|
|||||||
|
Task 01 complete. All gates green.
|
||||||
|
|
||||||
|
**Implemented (phase 102, task 01 — name-token matching):**
|
||||||
|
- `app/rag/importer.py::match_extension(path, extensions) -> str | None` — pure helper: rule 1 = dotted suffix (existing A9 rule, case-insensitive); rule 2 = suffix-less file whose lowercased full name equals a bare token (`Dockerfile` → `dockerfile`); dotted set kept as parameter form (no `config.py` change)
|
||||||
|
- `iter_importable_files` walk now filters via `match_extension(...) is None` (hidden-component + phase-89 ignore checks untouched, still first) — covers the phase-64 progress pre-walk too, so `total` stays honest
|
||||||
|
- `import_sources` `formats` counter uses the matched token (`dockerfile:1`, never `unknown`); module docstring documents the extensionless rule
|
||||||
|
- Unit tests in `tests/unit/test_importer.py`: the full D1 matching matrix, walk inclusion/exclusion (`Dockerfile`/`Containerfile` in; `mydockerfile`, `Dockerfile.dev`, `.dockerfile` out), in-process counter test asserting `summary.formats == {"dockerfile": 1, "md": 1}` and the PLAN §9 line `formats=dockerfile:1,md:1`
|
||||||
|
|
||||||
|
**Results:**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 2068 passed, `app/` coverage **99%** (>90% gate)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- No behavior change for any suffixed path — all pre-existing importer/phase-56/quadlet/ignore suites pass unmodified; no task files or PLAN touched, no commit made
|
||||||
|
|
||||||
|
**Next pending task:** `.agents/phases/todo/102_extensionless_filenames/02_format_badge_and_docs.md` (viewer `doc_format` badge + config/`.env.example` docs)
|
||||||
+96
@@ -0,0 +1,96 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 73%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
.................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 181 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 127 1 99%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 115 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 140 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 125 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 316 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 121 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 197 3 98%
|
||||||
|
app/rag/llm.py 243 1 99%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 307 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3855 15 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+16
@@ -0,0 +1,16 @@
|
|||||||
|
All green. Final report:
|
||||||
|
|
||||||
|
**Implemented (task 02 — badge + docs):**
|
||||||
|
- `app/api/docs.py`: `doc_format(path, extensions=frozenset())` — non-empty suffix returned unconditionally; suffix-less paths reuse task-01's `match_extension` name-token rule (`Dockerfile` → `dockerfile`), else `text`; default-args call byte-identical. Content endpoint now passes `get_settings().import_extension_set` (lru-cached).
|
||||||
|
- `app/config.py`: `import_extensions` docstring documents the extensionless exact-name rule (phase 102).
|
||||||
|
- `.env.example`: "Import scope" comment gains the rule with the `dockerfile,containerfile` example; default `BOR_IMPORT_EXTENSIONS` line untouched.
|
||||||
|
- `frontend/assets/document.js` verified untouched (badge renders `doc.format`; only `md`/`markdown` get markdown rendering — raw `<pre>` is correct).
|
||||||
|
- Tests: unit `test_document_viewer.py` (token-set matrix, suffix precedence, no-arg `text` pin) + integration `test_document_content.py::test_content_format_name_token_extensionless` (Dockerfile→`dockerfile` via env-override pattern, README.dev→`dev`, Containerfile→`text` without token).
|
||||||
|
|
||||||
|
**Results:**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 2080 passed, `app/` coverage **99%** (gate >90%)
|
||||||
|
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors
|
||||||
|
- Regression: `test_document_content.py` (13), `test_document_viewer.py` + `test_importer.py` (65) green
|
||||||
|
|
||||||
|
**Deviations:** none. (Regression E2E suites + story E2E are task 03's gate.)
|
||||||
|
**Next pending task:** `.agents/phases/todo/102_extensionless_filenames/03_integration_and_e2e.md`
|
||||||
+96
@@ -0,0 +1,96 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
................................................................ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 181 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 134 1 99%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 115 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 140 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 125 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 316 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 121 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 197 3 98%
|
||||||
|
app/rag/llm.py 243 1 99%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 307 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3862 15 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+14
@@ -0,0 +1,14 @@
|
|||||||
|
All gates green. Report:
|
||||||
|
|
||||||
|
**Phase 102 task 03 — extensionless filenames: integration + E2E proof**
|
||||||
|
|
||||||
|
- Fixture `tests/fixtures/extensionless_kb/`: `Dockerfile` (sentinel `…7a3e`), `Containerfile` (sentinel `…4b1c`), `Makefile` (negative control), `notes.md` (md control)
|
||||||
|
- `tests/integration/test_import_extensionless.py` (4 tests, house phase-56 pattern): positive import (3 files, `formats == {dockerfile:1, containerfile:1, md:1}`, no `unknown`, 768-dim chunks, mock `SUMMARY_MODE` digests), `DOCKERFILE` case-insensitive row, `md`-only negative, prune-on-token-removal (`pruned=2`, md survives)
|
||||||
|
- `tests/e2e/test_extensionless_import.py` (2 tests): admin — drill-down tree lists all 3 rows, `Makefile` absent, same-page modal shows badge `dockerfile` + source badge + title + raw sentinel + phase-30 summary panel, still on `/sources.html`; anonymous — gate + zero `/api/docs` page calls + direct GET 403
|
||||||
|
- E2E in isolation: `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed (run twice, stable; no stale servers on E2E ports)
|
||||||
|
- Regressions, isolated: `test_import_documents` 3, `test_import_extensions_env` 2, `test_quadlet_jinja_import` 4, `test_document_viewer` 7, `test_kb_tree` 8 — all passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, app/ coverage **99%** (>90% gate)
|
||||||
|
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors
|
||||||
|
- Decision: the viewer badge is rendered by the app from its own settings, so the E2E file pins `BOR_IMPORT_EXTENSIONS` for the app-under-test at module level (phase-61 leak-guard pattern; process env outranks the local `.env`, isolation makes it deterministic)
|
||||||
|
- Per harness rules I did not commit or move phase files — working tree left ready for the harness's single atomic phase commit (`fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS`)
|
||||||
|
- Next pending task: none in this phase (03 was last) — harness to move phase 102 to `complete/`; next phase in `todo/` is `103_suggestions_session_openers`
|
||||||
+96
@@ -0,0 +1,96 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 31%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
.................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 181 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 134 1 99%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 115 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 140 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 125 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 316 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 121 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 197 3 98%
|
||||||
|
app/rag/llm.py 243 1 99%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 307 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3862 15 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+4
-1
@@ -54,7 +54,10 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
|
|||||||
# Comma-separated file extensions (lowercase, no dot) the importer reads.
|
# Comma-separated file extensions (lowercase, no dot) the importer reads.
|
||||||
# Any extension is allowed — the value below is the built-in default (the
|
# Any extension is allowed — the value below is the built-in default (the
|
||||||
# A9 family: the original seven + the quadlet family + jinja ``j2``); add
|
# A9 family: the original seven + the quadlet family + jinja ``j2``); add
|
||||||
# your own (e.g. md,sh,toml) or narrow it (e.g. md). A blank list or a
|
# your own (e.g. md,sh,toml,dockerfile,containerfile) or narrow it
|
||||||
|
# (e.g. md). A token also matches extensionless files whose lowercased
|
||||||
|
# full filename equals it exactly (dockerfile → Dockerfile — case-
|
||||||
|
# insensitive, no partial names — phase 102). A blank list or a
|
||||||
# malformed token (e.g. md,sh!) fails startup loudly, naming the value.
|
# malformed token (e.g. md,sh!) fails startup loudly, naming the value.
|
||||||
BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2
|
BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2
|
||||||
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON seed chips — shown only before any question has been saved (phase 80)
|
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON seed chips — shown only before any question has been saved (phase 80)
|
||||||
|
|||||||
+20
-5
@@ -35,11 +35,13 @@ from sqlalchemy import func, select
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from app.api.sync import _sanitize_error
|
from app.api.sync import _sanitize_error
|
||||||
|
from app.config import get_settings
|
||||||
from app.core.auth import require_admin, require_user
|
from app.core.auth import require_admin, require_user
|
||||||
from app.db import get_db
|
from app.db import get_db
|
||||||
from app.models import Chunk, Document, FolderSummary
|
from app.models import Chunk, Document, FolderSummary
|
||||||
from app.rag.agent import list_source_names
|
from app.rag.agent import list_source_names
|
||||||
from app.rag.folder_summaries import folder_of
|
from app.rag.folder_summaries import folder_of
|
||||||
|
from app.rag.importer import match_extension
|
||||||
from app.rag.llm import EmbeddingError, LLMClient
|
from app.rag.llm import EmbeddingError, LLMClient
|
||||||
from app.schemas import (
|
from app.schemas import (
|
||||||
DocContent,
|
DocContent,
|
||||||
@@ -58,11 +60,24 @@ from app.schemas import (
|
|||||||
router = APIRouter(tags=["kb"])
|
router = APIRouter(tags=["kb"])
|
||||||
|
|
||||||
|
|
||||||
def doc_format(path: str) -> str:
|
def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
|
||||||
"""Lowercased path suffix without its dot (``kubernetes.md`` → ``md``,
|
"""Lowercased path suffix without its dot (``kubernetes.md`` → ``md``,
|
||||||
``notes/deep.Markdown`` → ``markdown``); ``text`` when the path has no
|
``notes/deep.Markdown`` → ``markdown``) — returned **unconditionally**
|
||||||
suffix — the value shown in the viewer's format badge."""
|
for a non-empty suffix (display never depends on the import list — an
|
||||||
return Path(path).suffix.lower().lstrip(".") or "text"
|
out-of-scope ``readme.rst`` still badges ``rst``); ``text`` when the
|
||||||
|
path has no suffix **and its name is not a configured token** (phase
|
||||||
|
102: a suffix-less ``Dockerfile`` badges ``dockerfile`` when its
|
||||||
|
lowercased full filename is one of *extensions* — the importer's
|
||||||
|
:func:`app.rag.importer.match_extension` rule). The value shown in
|
||||||
|
the viewer's format badge; the default empty *extensions* keeps the
|
||||||
|
pre-phase-102 suffix-only result for every path (byte-identical).
|
||||||
|
"""
|
||||||
|
p = Path(path)
|
||||||
|
suffix = p.suffix.lower().lstrip(".")
|
||||||
|
if suffix:
|
||||||
|
return suffix
|
||||||
|
matched = match_extension(p, extensions)
|
||||||
|
return matched if matched is not None else "text"
|
||||||
|
|
||||||
|
|
||||||
@router.get("/docs", response_model=DocList)
|
@router.get("/docs", response_model=DocList)
|
||||||
@@ -140,7 +155,7 @@ def get_document_content(
|
|||||||
source=doc.source,
|
source=doc.source,
|
||||||
path=doc.path,
|
path=doc.path,
|
||||||
title=doc.title,
|
title=doc.title,
|
||||||
format=doc_format(doc.path),
|
format=doc_format(doc.path, get_settings().import_extension_set),
|
||||||
summary=doc.summary,
|
summary=doc.summary,
|
||||||
content=doc.content,
|
content=doc.content,
|
||||||
indexed_at=doc.indexed_at.isoformat(),
|
indexed_at=doc.indexed_at.isoformat(),
|
||||||
|
|||||||
+4
-1
@@ -207,7 +207,10 @@ class Settings(BaseSettings):
|
|||||||
# doubles as the typo guard); the value below is the built-in default
|
# doubles as the typo guard); the value below is the built-in default
|
||||||
# (the A9 family, incl. the quadlet family + ``j2``) and the documented
|
# (the A9 family, incl. the quadlet family + ``j2``) and the documented
|
||||||
# example in ``.env.example``. Hidden (dot) path components are always
|
# example in ``.env.example``. Hidden (dot) path components are always
|
||||||
# skipped, plus the importer's exclusion list.
|
# skipped, plus the importer's exclusion list. A token also matches
|
||||||
|
# extensionless files whose lowercased full filename equals it exactly
|
||||||
|
# (``dockerfile`` → ``Dockerfile``), case-insensitive, no partial
|
||||||
|
# names (phase 102).
|
||||||
# Stored as a raw CSV string (env-native — no JSON) and parsed on demand
|
# Stored as a raw CSV string (env-native — no JSON) and parsed on demand
|
||||||
# via :py:meth:`import_extension_set`. The validator rejects an empty
|
# via :py:meth:`import_extension_set`. The validator rejects an empty
|
||||||
# list and malformed tokens so a typo fails loudly at startup (it can
|
# list and malformed tokens so a typo fails loudly at startup (it can
|
||||||
|
|||||||
+29
-3
@@ -19,7 +19,9 @@ the two-phase upsert:
|
|||||||
|
|
||||||
Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
|
Scope (A9, revised 2026-08-21): any path containing a dot-prefixed
|
||||||
component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
|
component (hidden dirs — vendored caches like ``.esphome/.espressif/**`` —
|
||||||
or hidden files) is skipped, plus the well-known exclusion list.
|
or hidden files) is skipped, plus the well-known exclusion list. A token
|
||||||
|
may also name extensionless files by their exact lowercased full filename
|
||||||
|
(``Dockerfile`` under the ``dockerfile`` token — phase 102).
|
||||||
|
|
||||||
``prune=True`` deletes documents (of the imported sources only) whose files
|
``prune=True`` deletes documents (of the imported sources only) whose files
|
||||||
no longer exist **or no longer match the format filter** — this is how
|
no longer exist **or no longer match the format filter** — this is how
|
||||||
@@ -158,6 +160,24 @@ def _ignore_for_root(
|
|||||||
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
|
return tuple(p for p in (normalize_ignore_path(e) for e in raw) if p)
|
||||||
|
|
||||||
|
|
||||||
|
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
|
||||||
|
"""The bare lowercased token *path* imports under, or ``None``.
|
||||||
|
|
||||||
|
1. Non-empty lowercased dotted suffix in *extensions* (the A9 rule —
|
||||||
|
``kubernetes.md`` → ``md``).
|
||||||
|
2. No suffix: the lowercased FULL filename equals a bare token of
|
||||||
|
*extensions* (``Dockerfile`` → ``dockerfile``) — the phase-102
|
||||||
|
extensionless rule. Exact name only: ``mydockerfile`` never
|
||||||
|
matches the ``dockerfile`` token.
|
||||||
|
"""
|
||||||
|
suffix = path.suffix.lower()
|
||||||
|
if suffix and suffix in extensions:
|
||||||
|
return suffix.lstrip(".")
|
||||||
|
if not suffix and path.name.lower() in {e.lstrip(".") for e in extensions}:
|
||||||
|
return path.name.lower()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def iter_importable_files(
|
def iter_importable_files(
|
||||||
root: Path,
|
root: Path,
|
||||||
extensions: frozenset[str],
|
extensions: frozenset[str],
|
||||||
@@ -187,7 +207,7 @@ def iter_importable_files(
|
|||||||
continue
|
continue
|
||||||
if ignore and is_ignored(rel.as_posix(), ignore):
|
if ignore and is_ignored(rel.as_posix(), ignore):
|
||||||
continue
|
continue
|
||||||
if path.suffix.lower() not in extensions:
|
if match_extension(path, extensions) is None:
|
||||||
continue
|
continue
|
||||||
files.append(path)
|
files.append(path)
|
||||||
return files
|
return files
|
||||||
@@ -274,7 +294,13 @@ async def import_sources(
|
|||||||
rel = path.relative_to(root).as_posix()
|
rel = path.relative_to(root).as_posix()
|
||||||
seen.add((source, rel))
|
seen.add((source, rel))
|
||||||
summary.files += 1
|
summary.files += 1
|
||||||
ext = path.suffix.lower().lstrip(".") or "unknown"
|
# Phase 102: the matched bare token (``dockerfile`` for an
|
||||||
|
# extensionless ``Dockerfile``), never ``unknown`` — the
|
||||||
|
# file is in scope, so the walk matched it.
|
||||||
|
ext = (
|
||||||
|
match_extension(path, llm.settings.import_extension_set)
|
||||||
|
or "unknown"
|
||||||
|
)
|
||||||
summary.formats[ext] = summary.formats.get(ext, 0) + 1
|
summary.formats[ext] = summary.formats.get(ext, 0) + 1
|
||||||
if progress is not None:
|
if progress is not None:
|
||||||
# phase 64: report the file *before* indexing it — a
|
# phase 64: report the file *before* indexing it — a
|
||||||
|
|||||||
@@ -0,0 +1,245 @@
|
|||||||
|
"""Phase 102 E2E (Playwright): extensionless files (``Dockerfile``,
|
||||||
|
``Containerfile``) sync when their name is a ``BOR_IMPORT_EXTENSIONS``
|
||||||
|
token — import → chunks → mock ``lite`` summary → drill-down tree →
|
||||||
|
viewer badge, plus the negative control and the anonymous gate.
|
||||||
|
|
||||||
|
The owner's defect: files without extensions never got synced, so
|
||||||
|
``Dockerfile`` / ``Containerfile`` were skipped even with
|
||||||
|
``dockerfile,containerfile`` in the env. Phase 102's rule: a suffix-less
|
||||||
|
file imports iff its lowercased FULL filename equals a token (exact
|
||||||
|
name, case-insensitive), the import ``formats=`` counter keys it by the
|
||||||
|
matched token, and the viewer badge is truthful (``dockerfile``, not
|
||||||
|
the generic ``text``).
|
||||||
|
|
||||||
|
The subject is the extensionless scope (``import_extensions=
|
||||||
|
"md,dockerfile,containerfile"``); the story-dedicated fixture
|
||||||
|
(``tests/fixtures/extensionless_kb/``) is seeded in-process against the
|
||||||
|
deterministic mock LLM — the phase-02/56 seeding-thread pattern — with
|
||||||
|
``Dockerfile`` / ``Containerfile`` carrying their own unique sentinels
|
||||||
|
and ``Makefile`` as the negative control (no token names it).
|
||||||
|
|
||||||
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov
|
||||||
|
|
||||||
|
DB isolation: the fixture's source name (``extensionless_kb``) is
|
||||||
|
distinctive — the suite never asserts on absolute row counts and
|
||||||
|
deletes the rows it creates in a ``finally`` (other suites' documents
|
||||||
|
stay untouched in the shared E2E database).
|
||||||
|
|
||||||
|
App-under-test scope pin (the phase-61 leak-guard pattern): the viewer
|
||||||
|
badge is rendered by the APP from its own ``get_settings().
|
||||||
|
import_extension_set`` (``GET /api/documents/content``), so the app
|
||||||
|
process must carry exactly the scope under test — a module-level
|
||||||
|
``BOR_IMPORT_EXTENSIONS`` override (process env ranks above the
|
||||||
|
operator's local gitignored ``.env`` when the ``app_server`` fixture
|
||||||
|
snapshots ``os.environ``). In isolation (AGENTS.md rule 9) the session
|
||||||
|
app is spawned by this file's first test, so the pin lands determinis-
|
||||||
|
tically.
|
||||||
|
|
||||||
|
Phase 97 adaptation: the catalog is the DRILL-DOWN TREE — the fixture's
|
||||||
|
files live at the SOURCE level (no subfolders), so a single drill
|
||||||
|
reaches every asserted row; the drill is the only change, the asserted
|
||||||
|
rows/links/modal are unchanged.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
from threading import Thread
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from playwright.sync_api import Page, expect
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from app.config import Settings
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from app.models import Document
|
||||||
|
from app.rag.importer import ImportSummary, import_sources
|
||||||
|
from app.rag.llm import LLMClient
|
||||||
|
from e2e.auth_helpers import login
|
||||||
|
from tests.e2e.mock_llm import TOKEN_RE
|
||||||
|
|
||||||
|
#: The app-under-test scope pin (see the module docstring) — exactly the
|
||||||
|
#: scope this suite proves, no matter what the operator's local ``.env``
|
||||||
|
#: carries. Must run before the session ``app_server`` fixture snapshots
|
||||||
|
#: ``os.environ`` (module import precedes fixture setup).
|
||||||
|
os.environ["BOR_IMPORT_EXTENSIONS"] = "md,dockerfile,containerfile"
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
FIXTURES = REPO / "tests" / "fixtures" / "extensionless_kb"
|
||||||
|
SOURCE = FIXTURES.name # "extensionless_kb" — distinctive, never asserted by count
|
||||||
|
DOCKER_REL = "Dockerfile"
|
||||||
|
CONTAINER_REL = "Containerfile"
|
||||||
|
NOTES_REL = "notes.md"
|
||||||
|
MAKE_REL = "Makefile" # negative control — no token in the scope under test
|
||||||
|
DOCKER_SENTINEL = "DOCKERFILE-PROBE-SENTINEL-7a3e"
|
||||||
|
|
||||||
|
|
||||||
|
async def _import_fixtures(mock_port: int, extensions: str) -> ImportSummary:
|
||||||
|
kwargs: dict[str, Any] = {
|
||||||
|
"_env_file": None,
|
||||||
|
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
|
||||||
|
"import_extensions": extensions,
|
||||||
|
}
|
||||||
|
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||||
|
return await import_sources([FIXTURES], LLMClient(settings))
|
||||||
|
|
||||||
|
|
||||||
|
def _run_in_thread(coro: Any) -> Any:
|
||||||
|
"""Run a coroutine on a worker thread.
|
||||||
|
|
||||||
|
Playwright's sync API keeps an asyncio loop running on the test
|
||||||
|
thread, so ``asyncio.run`` cannot be called directly from a test
|
||||||
|
body.
|
||||||
|
"""
|
||||||
|
box: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def runner() -> None:
|
||||||
|
try:
|
||||||
|
box["value"] = asyncio.run(coro)
|
||||||
|
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||||
|
box["error"] = e
|
||||||
|
|
||||||
|
t = Thread(target=runner)
|
||||||
|
t.start()
|
||||||
|
t.join()
|
||||||
|
if "error" in box:
|
||||||
|
raise box["error"]
|
||||||
|
return box["value"]
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_source_rows() -> None:
|
||||||
|
"""Delete every row of this suite's distinctive source (chunks
|
||||||
|
cascade with the document rows)."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
for doc in db.scalars(select(Document).where(Document.source == SOURCE)).all():
|
||||||
|
db.delete(doc)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _drill(page: Page, *names: str) -> None:
|
||||||
|
"""Drill one level at a time (phase 97 — client-side, no fetch,
|
||||||
|
no URL change): each name is the EXACT text of the source/folder
|
||||||
|
link at the current level."""
|
||||||
|
for name in names:
|
||||||
|
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||||
|
|
||||||
|
|
||||||
|
def _go_top(page: Page) -> None:
|
||||||
|
"""Back to the top level: the breadcrumb's top-level link (call
|
||||||
|
between drills only — the breadcrumb is hidden at the top)."""
|
||||||
|
page.locator("#kb-crumb a.kb-crumb-link").first.click()
|
||||||
|
|
||||||
|
|
||||||
|
def _summary_lines(path: str) -> tuple[str, str]:
|
||||||
|
"""(digest line, pointer line) of the stored phase-30 summary for a
|
||||||
|
fixture file — mirrors the mock ``SUMMARY_MODE`` branch (first 24
|
||||||
|
tokens of the document content) plus the code pointer line (the
|
||||||
|
``test_summary_in_viewer.py`` house helper)."""
|
||||||
|
content = (FIXTURES / path).read_text(encoding="utf-8")
|
||||||
|
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
|
||||||
|
expected = f"This document covers {digest}.\nSource: {SOURCE}/{path}"
|
||||||
|
digest_line, pointer_line = expected.split("\n", 1)
|
||||||
|
return digest_line, pointer_line
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def extensionless_kb(mock_llm: int, db_ready: None) -> Iterator[ImportSummary]:
|
||||||
|
"""Seed the fixture with the extensionless scope
|
||||||
|
(``md,dockerfile,containerfile``) for one test and delete every row
|
||||||
|
it creates afterwards (DB isolation — see the module docstring)."""
|
||||||
|
_delete_source_rows() # idempotent: leftovers from a crashed run
|
||||||
|
summary = _run_in_thread(_import_fixtures(mock_llm, "md,dockerfile,containerfile"))
|
||||||
|
try:
|
||||||
|
yield summary
|
||||||
|
finally:
|
||||||
|
_delete_source_rows()
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_tree_lists_name_token_files_and_viewer_badge(
|
||||||
|
page: Page, app_url: str, extensionless_kb: ImportSummary
|
||||||
|
) -> None:
|
||||||
|
# The seed saw exactly the three in-scope files in their formats —
|
||||||
|
# the two extensionless build files walked by name token, chunked,
|
||||||
|
# and summarized; the Makefile negative control stayed out.
|
||||||
|
assert extensionless_kb.formats == {"dockerfile": 1, "containerfile": 1, "md": 1}
|
||||||
|
assert "unknown" not in extensionless_kb.formats
|
||||||
|
assert (extensionless_kb.added, extensionless_kb.errors) == (3, 0)
|
||||||
|
|
||||||
|
login(page, app_url) # phase 16: the catalog is admin-only
|
||||||
|
# All three fixture files live at the SOURCE level (no subfolders —
|
||||||
|
# one drill reaches them, phase 97).
|
||||||
|
_drill(page, SOURCE)
|
||||||
|
for name in (DOCKER_REL, CONTAINER_REL, NOTES_REL):
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text=name)).to_have_count(1)
|
||||||
|
# The negative control: no token names Makefile exactly — absent.
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text=MAKE_REL)).to_have_count(0)
|
||||||
|
|
||||||
|
# The Dockerfile row's path link opens the SAME-PAGE modal and its
|
||||||
|
# meta row shows the D3 badge (``dockerfile`` — not the generic
|
||||||
|
# ``text`` fallback) + the source badge + the stem title.
|
||||||
|
row = page.locator("#docs-tbody tr", has_text=DOCKER_REL)
|
||||||
|
link = row.locator("td:nth-child(2) a.doc-link")
|
||||||
|
expect(link).to_have_count(1)
|
||||||
|
expect(link).to_have_attribute("title", DOCKER_REL)
|
||||||
|
before = len(page.context.pages)
|
||||||
|
link.click()
|
||||||
|
assert len(page.context.pages) == before, "clicking a row link must not open a new tab"
|
||||||
|
expect(page.locator("#doc-modal-meta .doc-source-badge")).to_have_text(SOURCE)
|
||||||
|
expect(page.locator("#doc-modal-meta .format-badge")).to_have_text("dockerfile")
|
||||||
|
expect(page.locator("#doc-modal-title")).to_have_text(DOCKER_REL)
|
||||||
|
|
||||||
|
# Non-markdown content renders as escaped monospace text in a pre —
|
||||||
|
# the sentinel proves it is THIS document's content.
|
||||||
|
pre = page.locator("#doc-modal-content pre.doc-raw")
|
||||||
|
expect(pre).to_have_count(1)
|
||||||
|
expect(pre).to_contain_text(DOCKER_SENTINEL)
|
||||||
|
|
||||||
|
# The phase-30 summary line renders above the content (house
|
||||||
|
# assertion style — test_summary_in_viewer.py): the deterministic
|
||||||
|
# mock digest + the code pointer line, in the labeled panel.
|
||||||
|
digest_line, pointer_line = _summary_lines(DOCKER_REL)
|
||||||
|
panel = page.locator("#doc-modal .doc-summary")
|
||||||
|
expect(panel).to_have_count(1)
|
||||||
|
expect(panel).to_be_visible()
|
||||||
|
expect(panel).to_have_attribute("aria-label", "Summary")
|
||||||
|
expect(panel).to_contain_text(digest_line)
|
||||||
|
expect(panel).to_contain_text(pointer_line)
|
||||||
|
|
||||||
|
# Still on the Sources page: no navigation happened.
|
||||||
|
assert page.url == app_url + "/sources.html", f"navigated away: {page.url}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_anonymous_sources_gate_and_no_api_docs(
|
||||||
|
page: Page, app_url: str, extensionless_kb: ImportSummary
|
||||||
|
) -> None:
|
||||||
|
"""A fresh anonymous context (function-scoped ``page`` = new
|
||||||
|
browser context, no cookies): the sign-in gate renders and the page
|
||||||
|
never calls ``/api/docs`` — the phase-16 pin, regression-checked
|
||||||
|
with the extensionless KB seeded; the API itself 403s anonymous
|
||||||
|
callers."""
|
||||||
|
api_docs_calls: list[str] = []
|
||||||
|
page.on(
|
||||||
|
"request",
|
||||||
|
lambda r: api_docs_calls.append(r.url) if "/api/docs" in r.url else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
page.goto(f"{app_url}/sources.html")
|
||||||
|
# The gate, with its sign-in link — not a redirect.
|
||||||
|
gate = page.locator("#sources-gate")
|
||||||
|
expect(gate).to_be_visible()
|
||||||
|
expect(gate).to_contain_text("Sign in to view the full catalog")
|
||||||
|
expect(gate.locator("a[href='/login.html?next=/sources.html']")).to_have_count(1)
|
||||||
|
# Stat cards + table hidden…
|
||||||
|
expect(page.locator("#stat-cards")).to_be_hidden()
|
||||||
|
expect(page.locator("#docs-table")).to_be_hidden()
|
||||||
|
expect(page.locator("#sources-empty")).to_be_hidden()
|
||||||
|
# …and NO /api/docs call was ever made.
|
||||||
|
assert api_docs_calls == [], f"anonymous sources page called /api/docs: {api_docs_calls}"
|
||||||
|
# And the API gate itself: an anonymous GET /api/docs 403s (phase 16
|
||||||
|
# — the router sits behind require_admin).
|
||||||
|
assert page.request.get(f"{app_url}/api/docs").status == 403
|
||||||
+14
@@ -0,0 +1,14 @@
|
|||||||
|
# CONTAINERFILE-PROBE-SENTINEL-4b1c — phase 102 fixture marker: this
|
||||||
|
# token exists nowhere else (the Dockerfile sibling carries its own).
|
||||||
|
#
|
||||||
|
# Containerfile — the local Postgres 17 + pgvector image the compose
|
||||||
|
# stack builds for the Brain of Reese database service.
|
||||||
|
|
||||||
|
FROM postgres:17
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends build-essential git \
|
||||||
|
&& git clone --branch v0.7.4 https://github.com/pgvector/pgvector /tmp/pgvector \
|
||||||
|
&& make -C /tmp/pgvector install \
|
||||||
|
&& rm -rf /tmp/pgvector \
|
||||||
|
&& apt-get purge -y build-essential git
|
||||||
+16
@@ -0,0 +1,16 @@
|
|||||||
|
# DOCKERFILE-PROBE-SENTINEL-7a3e — phase 102 fixture marker: this token
|
||||||
|
# exists nowhere else, so the extensionless_kb rows are unambiguous in
|
||||||
|
# the shared database.
|
||||||
|
#
|
||||||
|
# Dockerfile — the Brain of Reese app image: a slim Python runtime that
|
||||||
|
# serves the FastAPI app behind the homelab reverse proxy.
|
||||||
|
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY pyproject.toml uv.lock ./
|
||||||
|
COPY app ./app
|
||||||
|
RUN pip install --no-cache-dir uv && uv sync --frozen --no-dev
|
||||||
|
|
||||||
|
EXPOSE 8000
|
||||||
|
CMD ["uv", "run", "uvicorn", "app.main:app", "--host", "0.0.0.0"]
|
||||||
+10
@@ -0,0 +1,10 @@
|
|||||||
|
# Makefile — phase 102 negative control: extensionless, and no token in
|
||||||
|
# the env under test names it exactly, so the importer must skip it
|
||||||
|
# (the exact-name rule never reaches for a compound or unrelated name).
|
||||||
|
.PHONY: up down ps
|
||||||
|
up:
|
||||||
|
podman compose up -d
|
||||||
|
down:
|
||||||
|
podman compose down
|
||||||
|
ps:
|
||||||
|
podman compose ps
|
||||||
+6
@@ -0,0 +1,6 @@
|
|||||||
|
# Extensionless fixture notes
|
||||||
|
|
||||||
|
A short markdown control document for the phase 102 extensionless-kb
|
||||||
|
fixture. With `import_extensions="md,dockerfile,containerfile"` the
|
||||||
|
importer walks this file plus the name-token build files (`Dockerfile`,
|
||||||
|
`Containerfile`); with `md` alone, this note is the only import.
|
||||||
@@ -35,6 +35,7 @@ from sqlalchemy import select, text
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
import app.api.docs as docs_api
|
import app.api.docs as docs_api
|
||||||
|
from app.config import get_settings
|
||||||
from app.main import app as fastapi_app
|
from app.main import app as fastapi_app
|
||||||
from app.models import Chunk, Document
|
from app.models import Chunk, Document
|
||||||
from app.rag.llm import EmbeddingError
|
from app.rag.llm import EmbeddingError
|
||||||
@@ -555,3 +556,71 @@ def test_content_format_from_suffix(client, db) -> None:
|
|||||||
finally:
|
finally:
|
||||||
db.execute(text("TRUNCATE chunks, documents"))
|
db.execute(text("TRUNCATE chunks, documents"))
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_content_format_name_token_extensionless(
|
||||||
|
client, db, monkeypatch
|
||||||
|
) -> None:
|
||||||
|
"""Phase 102: the endpoint's ``format`` carries the name token for an
|
||||||
|
extensionless document whose lowercased full filename is a
|
||||||
|
``BOR_IMPORT_EXTENSIONS`` token (``Dockerfile`` → ``dockerfile``),
|
||||||
|
``text`` for an extensionless name that is NOT configured, and every
|
||||||
|
suffixed value is unchanged (display never depends on the import
|
||||||
|
list — ``notes/README.dev`` badges ``dev`` even though ``dev`` is not
|
||||||
|
a configured token). The endpoint reads the real ``get_settings()``,
|
||||||
|
so the token lands via the house env-override pattern (the default
|
||||||
|
list does not contain ``dockerfile``)."""
|
||||||
|
get_settings.cache_clear()
|
||||||
|
try:
|
||||||
|
monkeypatch.setenv("BOR_IMPORT_EXTENSIONS", "md,dockerfile")
|
||||||
|
|
||||||
|
_seed_doc(
|
||||||
|
db,
|
||||||
|
path="services/api/Dockerfile",
|
||||||
|
title="Dockerfile",
|
||||||
|
content="FROM alpine\nCMD [\"/bin/sh\"]",
|
||||||
|
chunks=0,
|
||||||
|
)
|
||||||
|
r = client.get(
|
||||||
|
"/api/documents/content",
|
||||||
|
params={"source": "Homelab", "path": "services/api/Dockerfile"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["format"] == "dockerfile"
|
||||||
|
|
||||||
|
# Suffix precedence under the same override — the import list is
|
||||||
|
# ignored for suffixed paths.
|
||||||
|
_seed_doc(
|
||||||
|
db,
|
||||||
|
path="notes/README.dev",
|
||||||
|
title="README.dev",
|
||||||
|
content="dev note",
|
||||||
|
chunks=0,
|
||||||
|
)
|
||||||
|
r = client.get(
|
||||||
|
"/api/documents/content",
|
||||||
|
params={"source": "Homelab", "path": "notes/README.dev"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["format"] == "dev"
|
||||||
|
|
||||||
|
# Extensionless name NOT in the configured set → the ``text``
|
||||||
|
# fallback (the existing README pin above covers the same case
|
||||||
|
# under the default settings).
|
||||||
|
_seed_doc(
|
||||||
|
db,
|
||||||
|
path="Containerfile",
|
||||||
|
title="Containerfile",
|
||||||
|
content="FROM alpine",
|
||||||
|
chunks=0,
|
||||||
|
)
|
||||||
|
r = client.get(
|
||||||
|
"/api/documents/content",
|
||||||
|
params={"source": "Homelab", "path": "Containerfile"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.json()["format"] == "text"
|
||||||
|
finally:
|
||||||
|
get_settings.cache_clear()
|
||||||
|
db.execute(text("TRUNCATE chunks, documents"))
|
||||||
|
db.commit()
|
||||||
|
|||||||
@@ -0,0 +1,293 @@
|
|||||||
|
"""Integration test: phase 102 — extensionless files named in
|
||||||
|
``BOR_IMPORT_EXTENSIONS`` import end to end.
|
||||||
|
|
||||||
|
The owner's defect: ``Dockerfile`` / ``Containerfile`` never synced even
|
||||||
|
with ``dockerfile`` / ``containerfile`` in the env, because the importer
|
||||||
|
matched on the dotted suffix only (``Path("Dockerfile").suffix`` is
|
||||||
|
``""``). Phase 102's rule (D1/D2): a suffix-less file is in scope iff its
|
||||||
|
lowercased FULL filename equals a token of the set — exact name,
|
||||||
|
case-insensitive — and the ``formats`` counter keys it by the matched
|
||||||
|
token, never ``unknown``.
|
||||||
|
|
||||||
|
Proven here against the story-dedicated fixture directory
|
||||||
|
``tests/fixtures/extensionless_kb/`` (the shared ``extension_kb``
|
||||||
|
fixture of phase 56 stays pinned by its own suite):
|
||||||
|
|
||||||
|
* positive — ``md,dockerfile,containerfile`` walks ``Dockerfile``,
|
||||||
|
``Containerfile`` and ``notes.md`` (the ``Makefile`` negative control
|
||||||
|
stays out): rows + embedded chunks + the deterministic mock
|
||||||
|
``SUMMARY_MODE`` digest for both build files (non-markdown → the
|
||||||
|
phase-30 ``lite`` path), ``formats == {"dockerfile": 1,
|
||||||
|
"containerfile": 1, "md": 1}`` with NO ``unknown`` key;
|
||||||
|
* case — an on-disk ``DOCKERFILE`` imports under the ``dockerfile``
|
||||||
|
token, its row keeps the on-disk case;
|
||||||
|
* negative — ``md`` alone: only the control note imports;
|
||||||
|
* prune — a token removed from the env drops its extensionless rows on
|
||||||
|
the next ``prune=True`` run (D4 — the A9 junk precedent).
|
||||||
|
|
||||||
|
The LLM is the deterministic mock server (``tests/e2e/mock_llm.py``) on
|
||||||
|
a scratch port — the integration analogue of the e2e ``mock_llm``
|
||||||
|
fixture (the phase-56 integration pattern). Runs against the local
|
||||||
|
compose Postgres (the ``db`` fixture from ``tests/conftest.py``); only
|
||||||
|
the distinctive ``extensionless_kb`` / ``casekb`` source rows are
|
||||||
|
created and deleted, so the rest of the shared KB is untouched.
|
||||||
|
|
||||||
|
Runs (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/integration/test_import_extensionless.py -v
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.config import Settings
|
||||||
|
from app.models import Document
|
||||||
|
from app.rag.importer import import_sources
|
||||||
|
from app.rag.llm import LLMClient
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
FIXTURES = REPO / "tests" / "fixtures" / "extensionless_kb"
|
||||||
|
SOURCE = FIXTURES.name # "extensionless_kb" — distinctive, never asserted by count
|
||||||
|
DOCKER_REL = "Dockerfile"
|
||||||
|
CONTAINER_REL = "Containerfile"
|
||||||
|
NOTES_REL = "notes.md"
|
||||||
|
DOCKER_SENTINEL = "DOCKERFILE-PROBE-SENTINEL-7a3e"
|
||||||
|
CONTAINER_SENTINEL = "CONTAINERFILE-PROBE-SENTINEL-4b1c"
|
||||||
|
#: The mock's SUMMARY_MODE tokenizes the document (``[a-z0-9]+``) — the
|
||||||
|
#: hyphenated sentinels land in the digest in their tokenized forms.
|
||||||
|
DOCKER_SENTINEL_TOKENS = "dockerfile probe sentinel 7a3e"
|
||||||
|
CONTAINER_SENTINEL_TOKENS = "containerfile probe sentinel 4b1c"
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_http(url: str, timeout: float = 30.0) -> None:
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
last_err = "unknown"
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
httpx.get(url, timeout=2.0)
|
||||||
|
return
|
||||||
|
except Exception as e: # noqa: BLE001 — retry until deadline
|
||||||
|
last_err = str(e)
|
||||||
|
time.sleep(0.2)
|
||||||
|
raise RuntimeError(f"mock LLM at {url} did not come up: {last_err}")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def mock_llm_port() -> Iterator[int]:
|
||||||
|
"""The deterministic mock LLM (``tests/e2e/mock_llm.py``) on a free
|
||||||
|
scratch port — same server as the e2e ``mock_llm`` fixture, but
|
||||||
|
private to this file (integration tests otherwise run network-free)."""
|
||||||
|
sock = socket.socket()
|
||||||
|
sock.bind(("127.0.0.1", 0))
|
||||||
|
port = sock.getsockname()[1]
|
||||||
|
sock.close()
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "tests.e2e.mock_llm:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"http://127.0.0.1:{port}/v1/models")
|
||||||
|
yield port
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
def _settings(mock_port: int, extensions: str) -> Settings:
|
||||||
|
kwargs: dict[str, Any] = {
|
||||||
|
"_env_file": None,
|
||||||
|
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
|
||||||
|
"import_extensions": extensions,
|
||||||
|
}
|
||||||
|
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||||
|
|
||||||
|
|
||||||
|
def _cleanup_source(db: Session, source: str) -> None:
|
||||||
|
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||||
|
db.delete(doc)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_name_token_files_import_end_to_end(mock_llm_port: int, db: Session) -> None:
|
||||||
|
"""``md,dockerfile,containerfile`` over the fixture imports the two
|
||||||
|
name-token build files + the markdown control: rows, embedded
|
||||||
|
chunks, the mock ``SUMMARY_MODE`` digest for each build file, and
|
||||||
|
the ``formats`` counter keyed by the matched token — never
|
||||||
|
``unknown``. The ``Makefile`` (no token) stays out."""
|
||||||
|
settings = _settings(mock_llm_port, "md,dockerfile,containerfile")
|
||||||
|
assert settings.import_extension_set == {".md", ".dockerfile", ".containerfile"}
|
||||||
|
summary = asyncio.run(import_sources([FIXTURES], LLMClient(settings), session=db))
|
||||||
|
try:
|
||||||
|
# Exactly the three in-scope files walk — the Makefile negative
|
||||||
|
# control and nothing else.
|
||||||
|
assert (
|
||||||
|
summary.files, summary.added, summary.unchanged, summary.updated, summary.errors
|
||||||
|
) == (3, 3, 0, 0, 0)
|
||||||
|
# D2: extensionless files count under their matched token.
|
||||||
|
assert summary.formats == {"dockerfile": 1, "containerfile": 1, "md": 1}
|
||||||
|
assert "unknown" not in summary.formats
|
||||||
|
# Phase 30: both build files are non-markdown → lite summaries;
|
||||||
|
# the markdown control never is.
|
||||||
|
assert (summary.summaries, summary.summary_errors) == (2, 0)
|
||||||
|
|
||||||
|
for rel, sentinel, tokens in (
|
||||||
|
(DOCKER_REL, DOCKER_SENTINEL, DOCKER_SENTINEL_TOKENS),
|
||||||
|
(CONTAINER_REL, CONTAINER_SENTINEL, CONTAINER_SENTINEL_TOKENS),
|
||||||
|
):
|
||||||
|
doc = db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == rel)
|
||||||
|
)
|
||||||
|
assert doc is not None, f"{rel} was not imported by name token"
|
||||||
|
# Suffix-less: the title is the file stem — the whole name.
|
||||||
|
assert doc.title == rel
|
||||||
|
# Plain-text chunking: the content (incl. the sentinel) is
|
||||||
|
# chunked and every content chunk is embedded at the 768-dim
|
||||||
|
# contract.
|
||||||
|
content = [c for c in doc.chunks if not c.is_summary]
|
||||||
|
assert content, f"{rel} has no content chunks"
|
||||||
|
assert all(
|
||||||
|
c.embedding is not None and len(c.embedding) == 768 for c in content
|
||||||
|
)
|
||||||
|
assert any(sentinel in c.content for c in content)
|
||||||
|
# Mock SUMMARY_MODE digest: byte-stable, the tokenized
|
||||||
|
# sentinel inside it, plus the code-appended pointer line.
|
||||||
|
assert doc.summary is not None
|
||||||
|
assert doc.summary.startswith("This document covers")
|
||||||
|
assert tokens in doc.summary
|
||||||
|
assert f"Source: {SOURCE}/{rel}" in doc.summary
|
||||||
|
schunks = [c for c in doc.chunks if c.is_summary]
|
||||||
|
assert len(schunks) == 1 and schunks[0].position == -1
|
||||||
|
assert schunks[0].embedding is not None
|
||||||
|
|
||||||
|
# The markdown control doc imported too — but markdown never
|
||||||
|
# gets a summary (phase 30).
|
||||||
|
note = db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == NOTES_REL)
|
||||||
|
)
|
||||||
|
assert note is not None
|
||||||
|
assert note.summary is None
|
||||||
|
assert [c for c in note.chunks if not c.is_summary]
|
||||||
|
|
||||||
|
# The negative control: no token names Makefile exactly — never
|
||||||
|
# walked.
|
||||||
|
assert (
|
||||||
|
db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == "Makefile")
|
||||||
|
)
|
||||||
|
is None
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, SOURCE)
|
||||||
|
|
||||||
|
|
||||||
|
def test_uppercase_extensionless_name_imports(
|
||||||
|
mock_llm_port: int, db: Session, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Case-insensitive exact name (D1): an on-disk ``DOCKERFILE``
|
||||||
|
imports under the ``dockerfile`` token, and its row keeps the
|
||||||
|
on-disk case in both ``path`` and ``title``."""
|
||||||
|
root = tmp_path / "casekb"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "DOCKERFILE").write_text("FROM alpine\nRUN apk add --no-cache curl\n")
|
||||||
|
settings = _settings(mock_llm_port, "md,dockerfile")
|
||||||
|
summary = asyncio.run(import_sources([root], LLMClient(settings), session=db))
|
||||||
|
try:
|
||||||
|
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||||
|
assert summary.formats == {"dockerfile": 1}
|
||||||
|
doc = db.scalar(
|
||||||
|
select(Document).where(Document.source == "casekb", Document.path == "DOCKERFILE")
|
||||||
|
)
|
||||||
|
assert doc is not None, "DOCKERFILE did not import under the dockerfile token"
|
||||||
|
assert doc.title == "DOCKERFILE"
|
||||||
|
assert [c for c in doc.chunks if not c.is_summary]
|
||||||
|
# Non-markdown → the phase-30 lite path ran.
|
||||||
|
assert summary.summaries == 1
|
||||||
|
assert doc.summary is not None
|
||||||
|
assert doc.summary.startswith("This document covers")
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "casekb")
|
||||||
|
|
||||||
|
|
||||||
|
def test_without_tokens_the_extensionless_files_stay_out(
|
||||||
|
mock_llm_port: int, db: Session
|
||||||
|
) -> None:
|
||||||
|
"""``md`` alone: the control note imports; ``Dockerfile`` /
|
||||||
|
``Containerfile`` (their names are not tokens in this scope) and the
|
||||||
|
``Makefile`` are all out of scope — no value of the scope walks
|
||||||
|
them."""
|
||||||
|
settings = _settings(mock_llm_port, "md")
|
||||||
|
assert settings.import_extension_set == {".md"}
|
||||||
|
summary = asyncio.run(import_sources([FIXTURES], LLMClient(settings), session=db))
|
||||||
|
try:
|
||||||
|
assert (
|
||||||
|
summary.files, summary.added, summary.unchanged, summary.updated, summary.errors
|
||||||
|
) == (1, 1, 0, 0, 0)
|
||||||
|
assert summary.formats == {"md": 1}
|
||||||
|
assert summary.summaries == 0
|
||||||
|
for rel in (DOCKER_REL, CONTAINER_REL, "Makefile"):
|
||||||
|
assert (
|
||||||
|
db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == rel)
|
||||||
|
)
|
||||||
|
is None
|
||||||
|
), f"{rel} was imported although its name is not a token in `md`"
|
||||||
|
assert (
|
||||||
|
db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == NOTES_REL)
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, SOURCE)
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_removes_extensionless_rows_when_the_token_is_removed(
|
||||||
|
mock_llm_port: int, db: Session
|
||||||
|
) -> None:
|
||||||
|
"""D4: a file that stops matching — the token leaves the env —
|
||||||
|
leaves ``seen`` and is deleted by the next ``prune=True`` run (the
|
||||||
|
A9 junk precedent); the in-scope ``notes.md`` row survives."""
|
||||||
|
with_tokens = _settings(mock_llm_port, "md,dockerfile,containerfile")
|
||||||
|
asyncio.run(import_sources([FIXTURES], LLMClient(with_tokens), session=db))
|
||||||
|
try:
|
||||||
|
md_only = _settings(mock_llm_port, "md")
|
||||||
|
summary = asyncio.run(
|
||||||
|
import_sources([FIXTURES], LLMClient(md_only), session=db, prune=True)
|
||||||
|
)
|
||||||
|
assert summary.pruned == 2, "the two extensionless rows must be pruned"
|
||||||
|
assert summary.unchanged == 1, "notes.md must survive the prune"
|
||||||
|
for rel in (DOCKER_REL, CONTAINER_REL):
|
||||||
|
assert (
|
||||||
|
db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == rel)
|
||||||
|
)
|
||||||
|
is None
|
||||||
|
), f"{rel} survived although its token left the env"
|
||||||
|
assert (
|
||||||
|
db.scalar(
|
||||||
|
select(Document).where(Document.source == SOURCE, Document.path == NOTES_REL)
|
||||||
|
)
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, SOURCE)
|
||||||
@@ -2,7 +2,9 @@
|
|||||||
|
|
||||||
Python side:
|
Python side:
|
||||||
* ``doc_format`` — format from the path suffix (incl. ``.markdown`` and the
|
* ``doc_format`` — format from the path suffix (incl. ``.markdown`` and the
|
||||||
no-suffix fallback);
|
no-suffix fallback) + the phase-102 extensionless name-token rule
|
||||||
|
(``Dockerfile`` → ``dockerfile`` with a configured token, suffixes still
|
||||||
|
unconditional, no-arg calls byte-identical to the suffix-only rule);
|
||||||
* the content endpoint's 200/404 mapping — tested WITHOUT a database by
|
* the content endpoint's 200/404 mapping — tested WITHOUT a database by
|
||||||
stubbing the session via FastAPI's dependency override (unknown pairs and
|
stubbing the session via FastAPI's dependency override (unknown pairs and
|
||||||
traversal-style paths map to 404 ``{detail: "document not found"}``;
|
traversal-style paths map to 404 ``{detail: "document not found"}``;
|
||||||
@@ -74,6 +76,43 @@ def test_doc_format_from_suffix(path: str, expected: str) -> None:
|
|||||||
assert doc_format(path) == expected
|
assert doc_format(path) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_doc_format_no_args_extensionless_falls_back_to_text() -> None:
|
||||||
|
"""The no-arg contract: default ``extensions=frozenset()`` keeps the
|
||||||
|
pre-phase-102 result for every path — an extensionless name that LOOKS
|
||||||
|
like a configured token still badges ``text`` without the token set."""
|
||||||
|
assert doc_format("Dockerfile") == "text"
|
||||||
|
assert doc_format("README") == "text"
|
||||||
|
|
||||||
|
|
||||||
|
#: A token set in the dotted form ``import_extension_set`` passes it.
|
||||||
|
_TOKEN_EXTS = frozenset({".md", ".dev", ".dockerfile", ".containerfile"})
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("path", "extensions", "expected"),
|
||||||
|
[
|
||||||
|
# Name-token branch (phase 102): suffix-less name in the set.
|
||||||
|
("services/api/Dockerfile", _TOKEN_EXTS, "dockerfile"),
|
||||||
|
("DOCKERFILE", _TOKEN_EXTS, "dockerfile"), # case-insensitive name
|
||||||
|
("Containerfile", _TOKEN_EXTS, "containerfile"),
|
||||||
|
# Suffix precedence: display never depends on the import list.
|
||||||
|
("Dockerfile.dev", _TOKEN_EXTS, "dev"), # suffixed → its suffix rules
|
||||||
|
("readme.rst", _TOKEN_EXTS, "rst"), # out-of-scope suffix still badges
|
||||||
|
("notes/README.dev", _TOKEN_EXTS, "dev"),
|
||||||
|
("dockerfile.bak", _TOKEN_EXTS, "bak"), # lookalike → suffix, not name
|
||||||
|
("kubernetes.md", frozenset(), "md"), # empty set: suffix unconditional
|
||||||
|
# Extensionless names NOT in the set fall back to text — exact
|
||||||
|
# name only, no partial names.
|
||||||
|
("README", _TOKEN_EXTS, "text"),
|
||||||
|
("mydockerfile", _TOKEN_EXTS, "text"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_doc_format_with_token_set(
|
||||||
|
path: str, extensions: frozenset[str], expected: str
|
||||||
|
) -> None:
|
||||||
|
assert doc_format(path, extensions) == expected
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Content endpoint mapping — stubbed session, no database required
|
# Content endpoint mapping — stubbed session, no database required
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ from app.rag.importer import (
|
|||||||
_store_summary,
|
_store_summary,
|
||||||
import_sources,
|
import_sources,
|
||||||
iter_importable_files,
|
iter_importable_files,
|
||||||
|
match_extension,
|
||||||
)
|
)
|
||||||
from app.rag.llm import EmbeddingError
|
from app.rag.llm import EmbeddingError
|
||||||
from tests.fakes import FakeEmbedder
|
from tests.fakes import FakeEmbedder
|
||||||
@@ -59,6 +60,15 @@ class _CapEmbedder(FakeEmbedder):
|
|||||||
return await super().embed(texts)
|
return await super().embed(texts)
|
||||||
|
|
||||||
|
|
||||||
|
def _embedder_with_extensions(extensions: str) -> FakeEmbedder:
|
||||||
|
"""A :class:`FakeEmbedder` whose settings carry a custom
|
||||||
|
``BOR_IMPORT_EXTENSIONS`` CSV (phase 102 — the import scope the walk
|
||||||
|
and the ``formats`` counter read from ``llm.settings``)."""
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
llm.settings = Settings(_env_file=None, import_extensions=extensions) # pyright: ignore[reportCallIssue]
|
||||||
|
return llm
|
||||||
|
|
||||||
|
|
||||||
def _cleanup_source(db, source: str) -> None:
|
def _cleanup_source(db, source: str) -> None:
|
||||||
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||||
db.delete(doc)
|
db.delete(doc)
|
||||||
@@ -199,6 +209,48 @@ def test_iter_importable_files_respects_custom_extension_filter(tmp_path: Path)
|
|||||||
assert found == {"a.md"}
|
assert found == {"a.md"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_match_extension_matrix() -> None:
|
||||||
|
"""Phase 102, D1 — the matching matrix: the A9 dotted-suffix rule first,
|
||||||
|
then the extensionless exact-name rule (case-insensitive), exact name
|
||||||
|
only — no partial matching, suffixed files governed by their suffix."""
|
||||||
|
exts = frozenset({".md", ".dockerfile"})
|
||||||
|
# Rule 1 — the dotted suffix (case-insensitive, as today):
|
||||||
|
assert match_extension(Path("kubernetes.md"), frozenset({".md"})) == "md"
|
||||||
|
assert match_extension(Path("Kubernetes.MD"), frozenset({".md"})) == "md"
|
||||||
|
# Rule 2 — extensionless files by exact lowercased FULL filename:
|
||||||
|
assert match_extension(Path("Dockerfile"), exts) == "dockerfile"
|
||||||
|
assert match_extension(Path("DOCKERFILE"), exts) == "dockerfile"
|
||||||
|
# …only when the token is actually in the set:
|
||||||
|
assert match_extension(Path("Dockerfile"), frozenset({".md"})) is None
|
||||||
|
# Exact name only — compound lookalikes never match the token:
|
||||||
|
assert match_extension(Path("mydockerfile"), exts) is None
|
||||||
|
# A suffixed file is governed by its suffix, never its name:
|
||||||
|
assert match_extension(Path("Dockerfile.dev"), exts) is None
|
||||||
|
assert match_extension(Path("Dockerfile.dev"), frozenset({".md", ".dev"})) == "dev"
|
||||||
|
# An out-of-scope suffix is out of scope, name be damned:
|
||||||
|
assert match_extension(Path("readme.rst"), frozenset({".md"})) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_iter_importable_files_walks_extensionless_name_tokens(tmp_path: Path) -> None:
|
||||||
|
"""Phase 102, D1 — an extensionless file walks iff its lowercased full
|
||||||
|
name is a token (``md,dockerfile,containerfile`` here): lookalikes and
|
||||||
|
the dot-prefixed hidden file stay skipped by the pre-existing rules."""
|
||||||
|
root = tmp_path / "build"
|
||||||
|
root.mkdir()
|
||||||
|
for name in (
|
||||||
|
"Dockerfile",
|
||||||
|
"Containerfile",
|
||||||
|
"mydockerfile", # compound name — never matches the `dockerfile` token
|
||||||
|
"Dockerfile.dev", # governed by its .dev suffix (not a token here)
|
||||||
|
".dockerfile", # dot-prefixed FILE — the hidden-component rule
|
||||||
|
"notes.md",
|
||||||
|
):
|
||||||
|
(root / name).write_text(f"# {name}\n\nbody {name}\n")
|
||||||
|
exts = frozenset({".md", ".dockerfile", ".containerfile"})
|
||||||
|
found = [p.name for p in iter_importable_files(root, exts)]
|
||||||
|
assert found == ["Containerfile", "Dockerfile", "notes.md"]
|
||||||
|
|
||||||
|
|
||||||
def test_excluded_dirs_match_plan_anchor_a9() -> None:
|
def test_excluded_dirs_match_plan_anchor_a9() -> None:
|
||||||
assert {
|
assert {
|
||||||
".venv", "node_modules", ".git", "__pycache__", ".pytest_cache", "dist", "build"
|
".venv", "node_modules", ".git", "__pycache__", ".pytest_cache", "dist", "build"
|
||||||
@@ -711,6 +763,36 @@ def test_prune_removes_files_now_excluded_by_format_filter(db, tmp_path: Path) -
|
|||||||
_cleanup_source(db, root.name)
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- phase 102: extensionless name-token import ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_formats_counter_counts_extensionless_name_token(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""Phase 102, D2 — an imported ``Dockerfile`` counts under
|
||||||
|
``dockerfile`` in ``summary.formats`` and the PLAN §9 line, never
|
||||||
|
``unknown``."""
|
||||||
|
root = tmp_path / "extless"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "Dockerfile").write_text("FROM alpine\n\nCMD [\"/bin/sh\"]\n")
|
||||||
|
(root / "notes.md").write_text("# Notes\n\nbody\n")
|
||||||
|
llm = _embedder_with_extensions("md,dockerfile")
|
||||||
|
try:
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert summary.files == 2 and summary.added == 2
|
||||||
|
assert summary.formats == {"dockerfile": 1, "md": 1}
|
||||||
|
line = next(
|
||||||
|
r.getMessage()
|
||||||
|
for r in caplog.records
|
||||||
|
if "import: summary files=" in r.getMessage()
|
||||||
|
)
|
||||||
|
assert line.endswith("formats=dockerfile:1,md:1")
|
||||||
|
assert "unknown" not in line
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
# ---------- phase 64 (task 01): optional per-file progress hook ----------
|
# ---------- phase 64 (task 01): optional per-file progress hook ----------
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user