phase: 102_extensionless_filenames
All verification complete — every gate green, no defects found in previously completed work. **Phase 102 final verification pass — report** Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design): - `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct - Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched) Test / lint results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate) - `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up - Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓ - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness. Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
This commit is contained in:
@@ -0,0 +1,245 @@
|
||||
"""Phase 102 E2E (Playwright): extensionless files (``Dockerfile``,
|
||||
``Containerfile``) sync when their name is a ``BOR_IMPORT_EXTENSIONS``
|
||||
token — import → chunks → mock ``lite`` summary → drill-down tree →
|
||||
viewer badge, plus the negative control and the anonymous gate.
|
||||
|
||||
The owner's defect: files without extensions never got synced, so
|
||||
``Dockerfile`` / ``Containerfile`` were skipped even with
|
||||
``dockerfile,containerfile`` in the env. Phase 102's rule: a suffix-less
|
||||
file imports iff its lowercased FULL filename equals a token (exact
|
||||
name, case-insensitive), the import ``formats=`` counter keys it by the
|
||||
matched token, and the viewer badge is truthful (``dockerfile``, not
|
||||
the generic ``text``).
|
||||
|
||||
The subject is the extensionless scope (``import_extensions=
|
||||
"md,dockerfile,containerfile"``); the story-dedicated fixture
|
||||
(``tests/fixtures/extensionless_kb/``) is seeded in-process against the
|
||||
deterministic mock LLM — the phase-02/56 seeding-thread pattern — with
|
||||
``Dockerfile`` / ``Containerfile`` carrying their own unique sentinels
|
||||
and ``Makefile`` as the negative control (no token names it).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov
|
||||
|
||||
DB isolation: the fixture's source name (``extensionless_kb``) is
|
||||
distinctive — the suite never asserts on absolute row counts and
|
||||
deletes the rows it creates in a ``finally`` (other suites' documents
|
||||
stay untouched in the shared E2E database).
|
||||
|
||||
App-under-test scope pin (the phase-61 leak-guard pattern): the viewer
|
||||
badge is rendered by the APP from its own ``get_settings().
|
||||
import_extension_set`` (``GET /api/documents/content``), so the app
|
||||
process must carry exactly the scope under test — a module-level
|
||||
``BOR_IMPORT_EXTENSIONS`` override (process env ranks above the
|
||||
operator's local gitignored ``.env`` when the ``app_server`` fixture
|
||||
snapshots ``os.environ``). In isolation (AGENTS.md rule 9) the session
|
||||
app is spawned by this file's first test, so the pin lands determinis-
|
||||
tically.
|
||||
|
||||
Phase 97 adaptation: the catalog is the DRILL-DOWN TREE — the fixture's
|
||||
files live at the SOURCE level (no subfolders), so a single drill
|
||||
reaches every asserted row; the drill is the only change, the asserted
|
||||
rows/links/modal are unchanged.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from threading import Thread
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from e2e.auth_helpers import login
|
||||
from tests.e2e.mock_llm import TOKEN_RE
|
||||
|
||||
#: The app-under-test scope pin (see the module docstring) — exactly the
|
||||
#: scope this suite proves, no matter what the operator's local ``.env``
|
||||
#: carries. Must run before the session ``app_server`` fixture snapshots
|
||||
#: ``os.environ`` (module import precedes fixture setup).
|
||||
os.environ["BOR_IMPORT_EXTENSIONS"] = "md,dockerfile,containerfile"
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "extensionless_kb"
|
||||
SOURCE = FIXTURES.name # "extensionless_kb" — distinctive, never asserted by count
|
||||
DOCKER_REL = "Dockerfile"
|
||||
CONTAINER_REL = "Containerfile"
|
||||
NOTES_REL = "notes.md"
|
||||
MAKE_REL = "Makefile" # negative control — no token in the scope under test
|
||||
DOCKER_SENTINEL = "DOCKERFILE-PROBE-SENTINEL-7a3e"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int, extensions: str) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {
|
||||
"_env_file": None,
|
||||
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
|
||||
"import_extensions": extensions,
|
||||
}
|
||||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return await import_sources([FIXTURES], LLMClient(settings))
|
||||
|
||||
|
||||
def _run_in_thread(coro: Any) -> Any:
|
||||
"""Run a coroutine on a worker thread.
|
||||
|
||||
Playwright's sync API keeps an asyncio loop running on the test
|
||||
thread, so ``asyncio.run`` cannot be called directly from a test
|
||||
body.
|
||||
"""
|
||||
box: dict[str, Any] = {}
|
||||
|
||||
def runner() -> None:
|
||||
try:
|
||||
box["value"] = asyncio.run(coro)
|
||||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||
box["error"] = e
|
||||
|
||||
t = Thread(target=runner)
|
||||
t.start()
|
||||
t.join()
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["value"]
|
||||
|
||||
|
||||
def _delete_source_rows() -> None:
|
||||
"""Delete every row of this suite's distinctive source (chunks
|
||||
cascade with the document rows)."""
|
||||
with SessionLocal() as db:
|
||||
for doc in db.scalars(select(Document).where(Document.source == SOURCE)).all():
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _drill(page: Page, *names: str) -> None:
|
||||
"""Drill one level at a time (phase 97 — client-side, no fetch,
|
||||
no URL change): each name is the EXACT text of the source/folder
|
||||
link at the current level."""
|
||||
for name in names:
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
|
||||
|
||||
def _go_top(page: Page) -> None:
|
||||
"""Back to the top level: the breadcrumb's top-level link (call
|
||||
between drills only — the breadcrumb is hidden at the top)."""
|
||||
page.locator("#kb-crumb a.kb-crumb-link").first.click()
|
||||
|
||||
|
||||
def _summary_lines(path: str) -> tuple[str, str]:
|
||||
"""(digest line, pointer line) of the stored phase-30 summary for a
|
||||
fixture file — mirrors the mock ``SUMMARY_MODE`` branch (first 24
|
||||
tokens of the document content) plus the code pointer line (the
|
||||
``test_summary_in_viewer.py`` house helper)."""
|
||||
content = (FIXTURES / path).read_text(encoding="utf-8")
|
||||
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
|
||||
expected = f"This document covers {digest}.\nSource: {SOURCE}/{path}"
|
||||
digest_line, pointer_line = expected.split("\n", 1)
|
||||
return digest_line, pointer_line
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def extensionless_kb(mock_llm: int, db_ready: None) -> Iterator[ImportSummary]:
|
||||
"""Seed the fixture with the extensionless scope
|
||||
(``md,dockerfile,containerfile``) for one test and delete every row
|
||||
it creates afterwards (DB isolation — see the module docstring)."""
|
||||
_delete_source_rows() # idempotent: leftovers from a crashed run
|
||||
summary = _run_in_thread(_import_fixtures(mock_llm, "md,dockerfile,containerfile"))
|
||||
try:
|
||||
yield summary
|
||||
finally:
|
||||
_delete_source_rows()
|
||||
|
||||
|
||||
def test_admin_tree_lists_name_token_files_and_viewer_badge(
|
||||
page: Page, app_url: str, extensionless_kb: ImportSummary
|
||||
) -> None:
|
||||
# The seed saw exactly the three in-scope files in their formats —
|
||||
# the two extensionless build files walked by name token, chunked,
|
||||
# and summarized; the Makefile negative control stayed out.
|
||||
assert extensionless_kb.formats == {"dockerfile": 1, "containerfile": 1, "md": 1}
|
||||
assert "unknown" not in extensionless_kb.formats
|
||||
assert (extensionless_kb.added, extensionless_kb.errors) == (3, 0)
|
||||
|
||||
login(page, app_url) # phase 16: the catalog is admin-only
|
||||
# All three fixture files live at the SOURCE level (no subfolders —
|
||||
# one drill reaches them, phase 97).
|
||||
_drill(page, SOURCE)
|
||||
for name in (DOCKER_REL, CONTAINER_REL, NOTES_REL):
|
||||
expect(page.locator("#docs-tbody tr", has_text=name)).to_have_count(1)
|
||||
# The negative control: no token names Makefile exactly — absent.
|
||||
expect(page.locator("#docs-tbody tr", has_text=MAKE_REL)).to_have_count(0)
|
||||
|
||||
# The Dockerfile row's path link opens the SAME-PAGE modal and its
|
||||
# meta row shows the D3 badge (``dockerfile`` — not the generic
|
||||
# ``text`` fallback) + the source badge + the stem title.
|
||||
row = page.locator("#docs-tbody tr", has_text=DOCKER_REL)
|
||||
link = row.locator("td:nth-child(2) a.doc-link")
|
||||
expect(link).to_have_count(1)
|
||||
expect(link).to_have_attribute("title", DOCKER_REL)
|
||||
before = len(page.context.pages)
|
||||
link.click()
|
||||
assert len(page.context.pages) == before, "clicking a row link must not open a new tab"
|
||||
expect(page.locator("#doc-modal-meta .doc-source-badge")).to_have_text(SOURCE)
|
||||
expect(page.locator("#doc-modal-meta .format-badge")).to_have_text("dockerfile")
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(DOCKER_REL)
|
||||
|
||||
# Non-markdown content renders as escaped monospace text in a pre —
|
||||
# the sentinel proves it is THIS document's content.
|
||||
pre = page.locator("#doc-modal-content pre.doc-raw")
|
||||
expect(pre).to_have_count(1)
|
||||
expect(pre).to_contain_text(DOCKER_SENTINEL)
|
||||
|
||||
# The phase-30 summary line renders above the content (house
|
||||
# assertion style — test_summary_in_viewer.py): the deterministic
|
||||
# mock digest + the code pointer line, in the labeled panel.
|
||||
digest_line, pointer_line = _summary_lines(DOCKER_REL)
|
||||
panel = page.locator("#doc-modal .doc-summary")
|
||||
expect(panel).to_have_count(1)
|
||||
expect(panel).to_be_visible()
|
||||
expect(panel).to_have_attribute("aria-label", "Summary")
|
||||
expect(panel).to_contain_text(digest_line)
|
||||
expect(panel).to_contain_text(pointer_line)
|
||||
|
||||
# Still on the Sources page: no navigation happened.
|
||||
assert page.url == app_url + "/sources.html", f"navigated away: {page.url}"
|
||||
|
||||
|
||||
def test_anonymous_sources_gate_and_no_api_docs(
|
||||
page: Page, app_url: str, extensionless_kb: ImportSummary
|
||||
) -> None:
|
||||
"""A fresh anonymous context (function-scoped ``page`` = new
|
||||
browser context, no cookies): the sign-in gate renders and the page
|
||||
never calls ``/api/docs`` — the phase-16 pin, regression-checked
|
||||
with the extensionless KB seeded; the API itself 403s anonymous
|
||||
callers."""
|
||||
api_docs_calls: list[str] = []
|
||||
page.on(
|
||||
"request",
|
||||
lambda r: api_docs_calls.append(r.url) if "/api/docs" in r.url else None,
|
||||
)
|
||||
|
||||
page.goto(f"{app_url}/sources.html")
|
||||
# The gate, with its sign-in link — not a redirect.
|
||||
gate = page.locator("#sources-gate")
|
||||
expect(gate).to_be_visible()
|
||||
expect(gate).to_contain_text("Sign in to view the full catalog")
|
||||
expect(gate.locator("a[href='/login.html?next=/sources.html']")).to_have_count(1)
|
||||
# Stat cards + table hidden…
|
||||
expect(page.locator("#stat-cards")).to_be_hidden()
|
||||
expect(page.locator("#docs-table")).to_be_hidden()
|
||||
expect(page.locator("#sources-empty")).to_be_hidden()
|
||||
# …and NO /api/docs call was ever made.
|
||||
assert api_docs_calls == [], f"anonymous sources page called /api/docs: {api_docs_calls}"
|
||||
# And the API gate itself: an anonymous GET /api/docs 403s (phase 16
|
||||
# — the router sits behind require_admin).
|
||||
assert page.request.get(f"{app_url}/api/docs").status == 403
|
||||
Reference in New Issue
Block a user