Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c73810593218e2770de7ab98d932dc11ee2fabae
brain-of-reese/tests/e2e/test_response_to_docs.py
T
ducoterra baefcde668 fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page 
Fixed: index.html meta description, empty-state sub and composer
placeholder (A1); app/config.py default suggestion chips → the four
neutral A2 defaults (BOR_SUGGESTIONS override unchanged); sources.html
KB page-sub → the current source model (git repos + local dirs +
uploaded archives, Sync pulls/imports); git-sources.html example URL
→ your-repo.git (A3); all 9 footers → neutral default in
span.footer-text (the phase-62 hook); E2E/unit conftests force the
code defaults so a local .env cannot leak corpus copy into tests;
new unit text pins + dedicated E2E suite.

Task 02 verification read-through — no change needed:
- sources.html sync result/error copy (matches the real sync behavior)
- tuning.html page-sub (accurate as written)
- history.html page-sub (accurate as written)
- doc-edit.html page-sub (accurate as written)
- git-sources.html page-sub (accurate as written)
- #sources-gate anonymous copy (accurate as written)
2026-09-01 10:54:50 -04:00

608 lines
23 KiB
Python
Raw Blame History

"""Phase 59 story E2E (Playwright): the save → edit → push loop, with
the BARE REPO as source of truth.
Story: n/a (TODO-derived — "Convert response to documentation that gets
committed back to a repo specified in .env … allows you to modify the
new documentation before [pushing] to the specified repo").
Run in isolation (DB must be up: ``podman compose up -d db``; ``git``
on PATH — the suite skips without it):
uv run pytest tests/e2e/test_response_to_docs.py -v --no-cov
The loop under test: a completed brain bubble carries a bottom-right
"Save as doc" action (admin + a configured ``BOR_DOCS_REPO``) → it
opens ``/doc-edit.html?draft=<token>`` prefilled (auto-title from the
last question, path ``docs/<slug>.md``, body = the answer's MARKDOWN
SOURCE — never the rendered HTML) → Push commits + pushes to the
``.env``-configured branch of the ``.env``-configured repo. Every
success assertion reads the **bare repo itself** (``git show
<branch>:<path>``, ``git rev-list``, ``git rev-parse``) — the UI text
is only the entry point (D3: no PR is ever created or attempted — the
flow ends at the push to the branch).
App boots (the conftest pattern, module-scoped — as in
``test_git_sources_admin.py``):
* the module app boots with ``BOR_DOCS_REPO=<tmp>/docs.git`` (a local
BARE repo seeded with one commit on ``main``), ``BOR_DOCS_BRANCH=
bor-docs``, ``BOR_DOCS_BASE_BRANCH=main``, ``BOR_DOCS_WORK_DIR=
<tmp>/docs-work``;
* ``test_unconfigured_hides_button`` boots a SECOND app (separate
fixture, ``APP_PORT + 1``) with NO docs env — the inert default:
no button for anyone, draft creation still allowed (drafts are
repo-independent), push 409s naming ``BOR_DOCS_REPO``.
The mock LLM keeps every answer byte-deterministic: the suite replays
the same question through ``POST /api/chat`` (raw SSE, the
``test_chat_rag.py`` pattern) to recover the exact markdown source the
draft must carry — so "body == the answer's markdown source" is an
exact-byte assertion, not a contains check.
Test → story mapping (Playwright Mapping Rule):
1. ``test_save_edit_push``
2. ``test_second_push_fast_forwards``
3. ``test_guest_has_no_button``
4. ``test_unconfigured_hides_button``
"""
from __future__ import annotations
import asyncio
import json
import os
import re
import subprocess
import sys
from collections.abc import Iterator
from pathlib import Path
from types import SimpleNamespace
from typing import Any
from urllib.parse import parse_qs, urlsplit
import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.config import Settings
from app.db import SessionLocal
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The unconfigured app's port (task 07: a second app boot WITHOUT
#: ``BOR_DOCS_REPO`` — a separate fixture on the next port, so it can
#: run alongside the module app).
UNCONF_URL = f"http://127.0.0.1:{APP_PORT + 1}"
BRANCH = "bor-docs"
BASE_BRANCH = "main"
#: On-topic fixture questions (the house phrasing — proven HIGH gate in
#: test_chat_rag.py / test_pinned_composer.py, so every turn renders a
#: grounded answer with the deterministic marker, never a deflection).
QUESTION_1 = "How is my Kubernetes cluster set up?"
QUESTION_2 = "What is in the new-service deployment?"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
#: The distinctive line test 1 appends to the body before pushing —
#: ASCII on purpose (git's output must match byte-for-byte), and a
#: module constant so test 2 can reconstruct file 1's expected content
#: (deterministic: the mock answer + this exact suffix).
E2E_MARKER = "E2E-DOCS-MARKER (appended by the response-to-docs story suite)"
#: The edit screen's URL shape (task 05 navigates with the uuid4 token).
DRAFT_URL_RE = re.compile(r"/doc-edit\.html\?draft=[0-9a-f-]{36}")
#: The success line (task 06): `Pushed to <branch> — commit <sha7>.`
SUCCESS_SHA_RE = re.compile(r"commit ([0-9a-f]{7})\.$")
def _git_available() -> bool:
try:
return subprocess.run(
["git", "--version"], capture_output=True, timeout=10
).returncode == 0
except (FileNotFoundError, subprocess.TimeoutExpired):
return False
pytestmark = pytest.mark.skipif(
not _git_available(), reason="git is not on PATH (the docs push is real git)"
)
def _git(args: list[str], cwd: Path | None = None) -> str:
"""One git command (the bare repo is the source of truth); fail loud."""
proc = subprocess.run(
["git", *args], cwd=cwd, capture_output=True, text=True, timeout=60
)
assert proc.returncode == 0, f"git {' '.join(args)} failed: {proc.stderr}"
return proc.stdout
def doc_slug(title: str) -> str:
"""The app.js slug rule (phase 59 locked assumption), ported:
lowercase, runs of non-alphanumerics → ``-``, trimmed, ≤60 chars,
empty → ``note`` (the trailing trim survives a mid-dash 60-cut)."""
slug = (
re.sub(r"[^a-z0-9]+", "-", title.lower())
.strip("-")[:60]
.rstrip("-")
)
return slug or "note"
def _admin_cookies(page: Page) -> dict[str, str]:
"""The signed session cookies the browser holds after a form login
— used to call the admin API with plain httpx (the
``test_cache_busting.py`` pattern)."""
return {
c["name"]: c["value"]
for c in page.context.cookies()
if "name" in c and "value" in c
}
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def docs_repo(tmp_path_factory: pytest.TempPathFactory) -> SimpleNamespace:
"""The local BARE docs repo (the .env remote, D3-generic): one
seed commit (``README.md``) pushed as ``main``. ``work`` is where
the app's ``BOR_DOCS_WORK_DIR`` checkout lands (it persists for the
whole module — the second push exercises the existing-checkout
path)."""
base = tmp_path_factory.mktemp("docs-git")
bare = base / "docs.git"
_git(["init", "--bare", str(bare)])
seed = base / "seed"
_git(["init", "-b", "main", str(seed)])
(seed / "README.md").write_text("# e2e docs repo\n", encoding="utf-8")
_git(["add", "--", "README.md"], cwd=seed)
# -c identity + no GPG signing: the machine's global git config
# (gpgsign=true here) must not leak into the fixture.
_git(
[
"-c", "user.name=E2E Seeder",
"-c", "user.email=e2e@local",
"-c", "commit.gpgsign=false",
"commit", "-m", "seed: README",
],
cwd=seed,
)
_git(["remote", "add", "origin", str(bare)], cwd=seed)
_git(["push", "origin", "main"], cwd=seed)
return SimpleNamespace(bare=bare, work=base / "docs-work")
def _spawn_app(port: int, mock_port: int, docs_env: dict[str, str] | None) -> subprocess.Popen:
"""One uvicorn boot (the conftest app_server env shape); ``None``
docs_env = NO docs variables at all (the unconfigured app)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else f"http://127.0.0.1:{mock_port}/v1"
)
# Mock-calibrated threshold (conftest pattern): the fixture questions
# gate HIGH, so every turn is a grounded answer with the marker.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
if docs_env is None:
for var in (
"BOR_DOCS_REPO",
"BOR_DOCS_BRANCH",
"BOR_DOCS_BASE_BRANCH",
"BOR_DOCS_WORK_DIR",
):
env.pop(var, None)
# Phase 61 (defect fix): the pop only clears the process env —
# pydantic-settings would still pick up ``BOR_DOCS_REPO`` from an
# operator's local (gitignored) ``.env`` (``cwd=REPO``). Force
# empty so the "unconfigured" boot really is the inert default.
env["BOR_DOCS_REPO"] = ""
else:
env.update(docs_env)
return subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=env,
)
def _stop(proc: subprocess.Popen) -> None:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(mock_llm: int, docs_repo: SimpleNamespace) -> Iterator[str]:
"""The configured app under test (module scope — shadows the
conftest session app; an isolated run never starts two)."""
proc = _spawn_app(
APP_PORT,
mock_llm,
{
"BOR_DOCS_REPO": str(docs_repo.bare),
"BOR_DOCS_BRANCH": BRANCH,
"BOR_DOCS_BASE_BRANCH": BASE_BRANCH,
"BOR_DOCS_WORK_DIR": str(docs_repo.work),
},
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
_stop(proc)
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
@pytest.fixture()
def unconfigured_app(mock_llm: int) -> Iterator[str]:
"""The SECOND app boot (task 07): NO ``BOR_DOCS_REPO`` — the inert
default the suite must see as absent-for-everyone + 409 push."""
proc = _spawn_app(APP_PORT + 1, mock_llm, None)
try:
_wait_http(f"{UNCONF_URL}/api/health")
yield UNCONF_URL
finally:
_stop(proc)
# ---------------------------------------------------------------------------
# KB + table hygiene (the E2E isolation pattern — this suite owns the
# KB tables and doc_drafts; both are reset around every test)
# ---------------------------------------------------------------------------
async def _import_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {
"_env_file": None,
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return await import_sources([FIXTURES], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread (the Playwright sync API keeps
an asyncio loop on the test thread — the test_chat_rag.py helper)."""
import threading
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = threading.Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _reset_db(mock_port: int, seed: bool) -> None:
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, doc_drafts"))
db.commit()
if seed:
summary = _run_in_thread(_import_fixtures(mock_port))
assert summary.added == 13 # the A9 fixture set (test_chat_rag.py)
@pytest.fixture(autouse=True)
def _kb_and_clean_drafts(mock_llm: int, db_ready: None) -> Iterator[None]:
"""Fresh KB (the deterministic mock embeddings — the grounded
questions gate HIGH) + an empty ``doc_drafts`` table per test."""
_reset_db(mock_llm, seed=True)
yield
_reset_db(mock_llm, seed=False)
# ---------------------------------------------------------------------------
# Story helpers
# ---------------------------------------------------------------------------
def _stream_chat_answer(app_url: str, message: str) -> str:
"""Replay one turn through the raw SSE endpoint (the
``test_chat_rag.py`` transport pattern) and return the EXACT answer
text — the markdown source the UI accumulates into ``m.text``,
byte-identical for the deterministic mock (same KB, same question)."""
frames: list[dict[str, Any]] = []
with httpx.stream(
"POST", f"{app_url}/api/chat", json={"message": message}, timeout=120.0
) as r:
assert r.status_code == 200
buf = ""
for part in r.iter_text():
buf += part
while "\n\n" in buf:
frame, buf = buf.split("\n\n", 1)
if frame.strip().startswith("data:"):
frames.append(
json.loads(frame.strip().removeprefix("data:").strip())
)
deltas = [f for f in frames if f.get("type") == "delta"]
assert deltas, "the SSE stream must deliver deltas"
return "".join(d["text"] for d in deltas)
def _ask(page: Page, app_url: str, question: str) -> None:
"""One grounded turn to its DONE state (marker in the bubble + the
send button re-enabled — the meta-row buttons land on done)."""
page.fill("#message-input", question)
page.click("#send-btn")
bubble = page.locator(".msg.brain .bubble:not(.typing)").first
expect(bubble).to_contain_text(question, timeout=30_000)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send")
def _login_admin(page: Page, app_url: str) -> None:
"""Real form login landing on the chat (admin settled)."""
login(page, app_url, next="/")
expect(page).to_have_url(app_url + "/", timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=30_000)
def _open_edit_screen(page: Page) -> str:
"""Click the save action, wait for the navigation, return the draft
token from the URL (the uuid4 credential)."""
page.click(".save-as-doc-btn")
page.wait_for_url(DRAFT_URL_RE, timeout=30_000)
token = parse_qs(urlsplit(page.url).query).get("draft", [""])[0]
assert re.fullmatch(r"[0-9a-f-]{36}", token), f"no draft token in {page.url}"
expect(page.locator("#doc-edit-gate")).to_be_hidden(timeout=30_000)
expect(page.locator("#doc-edit-content")).to_be_visible(timeout=30_000)
return token
def _push_and_read_sha(page: Page) -> tuple[str, str]:
"""Submit the edit screen's push; wait for the success line and
return (branch, sha7) exactly as the live region reported them."""
page.click("#push-doc-btn")
status = page.locator("#push-status")
expect(status).to_contain_text(f"Pushed to {BRANCH}", timeout=60_000)
line = status.inner_text().strip()
m = SUCCESS_SHA_RE.search(line)
assert m, f"the success line carries no commit sha: {line!r}"
return BRANCH, m.group(1)
# ---------------------------------------------------------------------------
# 1. The whole loop: save → edit → push → the bare repo agrees
# ---------------------------------------------------------------------------
def test_save_edit_push(
page: Page,
app_url: str,
mock_llm: int,
db_ready: None,
docs_repo: SimpleNamespace,
) -> None:
page.set_default_timeout(30_000)
_login_admin(page, app_url)
_ask(page, app_url, QUESTION_1)
# The "Save as doc" action is on the completed brain bubble…
btn = page.locator(".msg.brain .save-as-doc-btn")
expect(btn).to_have_count(1)
expect(btn).to_contain_text("Save as doc")
# …bottom-right: its left edge sits past the bubble's midline
# (margin-inline-start: auto in the meta row).
msg_box = page.locator(".msg.brain").bounding_box()
btn_box = btn.bounding_box()
assert msg_box is not None and btn_box is not None
midline = msg_box["x"] + msg_box["width"] / 2
assert btn_box["x"] > midline, (
f"save button x={btn_box['x']:.0f} is not past the bubble midline "
f"{midline:.0f} — it must sit bottom-right"
)
# Click → /doc-edit.html?draft=<uuid4>, prefilled.
_open_edit_screen(page)
expect(page.locator("#draft-title")).to_have_value(QUESTION_1) # auto-title
expect(page.locator("#draft-path")).to_have_value(
f"docs/{doc_slug(QUESTION_1)}.md"
)
# Body == the rendered answer's MARKDOWN SOURCE: the SSE replay
# recovers the exact bytes the UI accumulated (the mock is
# byte-deterministic on the same KB + question) — and they are
# plain markdown, not rendered HTML.
raw = _stream_chat_answer(app_url, QUESTION_1)
assert MOCK_ANSWER_MARKER in raw and QUESTION_1 in raw
assert "<" not in raw and ">" not in raw, "the draft body must be markdown, not HTML"
expect(page.locator("#draft-body")).to_have_value(raw)
# Modify the doc (the story's "modify before [pushing]"): a
# distinctive marker line the bare repo must show after the push.
edited = f"{raw}\n\n{E2E_MARKER}"
page.fill("#draft-body", edited)
# Push → the live region reports the branch + a 7-char commit sha…
branch, sha7 = _push_and_read_sha(page)
assert branch == BRANCH
# …and the BARE REPO agrees (the source of truth — not the UI):
# the file on the branch is exactly the edited body…
path = f"docs/{doc_slug(QUESTION_1)}.md"
shown = _git(["-C", str(docs_repo.bare), "show", f"{BRANCH}:{path}"])
assert shown == edited
# …and the branch tip's first 7 chars are the sha the UI reported.
tip = _git(["-C", str(docs_repo.bare), "rev-parse", BRANCH]).strip()
assert tip.startswith(sha7), f"UI sha {sha7} != bare repo tip {tip}"
# First push: the branch exists and is exactly one commit beyond
# main (created by the push — the remote had no bor-docs before).
assert (
_git(["-C", str(docs_repo.bare), "rev-list", "--count", f"{BASE_BRANCH}..{BRANCH}"])
.strip()
== "1"
)
# ---------------------------------------------------------------------------
# 2. A second save fast-forwards: two commits, file 1 untouched
# ---------------------------------------------------------------------------
def test_second_push_fast_forwards(
page: Page,
app_url: str,
mock_llm: int,
db_ready: None,
docs_repo: SimpleNamespace,
) -> None:
page.set_default_timeout(30_000)
_login_admin(page, app_url)
_ask(page, app_url, QUESTION_2)
# Save the second answer (different question → different slug)…
expect(page.locator(".msg.brain .save-as-doc-btn")).to_have_count(1)
_open_edit_screen(page)
expect(page.locator("#draft-title")).to_have_value(QUESTION_2)
expect(page.locator("#draft-path")).to_have_value(
f"docs/{doc_slug(QUESTION_2)}.md"
)
raw2 = _stream_chat_answer(app_url, QUESTION_2)
expect(page.locator("#draft-body")).to_have_value(raw2)
# …and push WITHOUT editing — a new commit on the same branch.
_push_and_read_sha(page)
# The bare repo: exactly two commits beyond main (fast-forward,
# never a force-push or a reset)…
assert (
_git(["-C", str(docs_repo.bare), "rev-list", "--count", f"{BASE_BRANCH}..{BRANCH}"])
.strip()
== "2"
)
# …file 2 landed with its unedited body…
path2 = f"docs/{doc_slug(QUESTION_2)}.md"
assert _git(["-C", str(docs_repo.bare), "show", f"{BRANCH}:{path2}"]) == raw2
# …and file 1 from test 1 is still at its path, byte-for-byte
# (deterministic reconstruction: the mock answer + the marker line).
path1 = f"docs/{doc_slug(QUESTION_1)}.md"
expected_first = f"{_stream_chat_answer(app_url, QUESTION_1)}\n\n{E2E_MARKER}"
assert _git(["-C", str(docs_repo.bare), "show", f"{BRANCH}:{path1}"]) == expected_first
# ---------------------------------------------------------------------------
# 3. Guest: no button, 403 on the draft API, the edit screen gates
# ---------------------------------------------------------------------------
def test_guest_has_no_button(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# No login (the conftest page fixture is a fresh context). Track
# every /api/doc-drafts request the PAGES make — the guest flow
# must never reach the admin API.
drafts_calls: list[str] = []
page.on(
"request",
lambda r: drafts_calls.append(r.url) if "/api/doc-drafts" in r.url else None,
)
page.goto(app_url)
expect(page.locator("#sign-in-link")).to_be_visible(timeout=30_000)
_ask(page, app_url, QUESTION_1) # the grounded answer streams for guests too
# The "Save as doc" action is admin-only: ABSENT (not hidden) on
# the completed bubble, whatever the docs config says.
expect(page.locator(".save-as-doc-btn")).to_have_count(0)
# The draft API 403s anonymous callers (httpx, no cookie at all).
r = httpx.post(
f"{app_url}/api/doc-drafts",
json={"title": "guest", "path": "docs/guest.md", "body": "nope"},
timeout=10,
)
assert r.status_code == 403
# The edit screen renders the admin gate with NO draft data in the
# DOM, and the page itself made zero draft API calls.
page.goto(app_url + "/doc-edit.html")
expect(page.locator("#doc-edit-gate")).to_be_visible(timeout=30_000)
expect(page.locator("#doc-edit-content")).to_be_hidden()
assert page.input_value("#draft-title") == ""
assert page.input_value("#draft-path") == ""
assert page.input_value("#draft-body") == ""
assert drafts_calls == [], f"guest pages called the draft API: {drafts_calls}"
# ---------------------------------------------------------------------------
# 4. Unconfigured (BOR_DOCS_REPO empty): inert for everyone, 409 push
# ---------------------------------------------------------------------------
def test_unconfigured_hides_button(
page: Page, unconfigured_app: str, mock_llm: int, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_login_admin(page, unconfigured_app)
_ask(page, unconfigured_app, QUESTION_1)
# docs_repo_configured false → the button is hidden for EVERYONE,
# admin included (the optional-feature pattern — inert by default).
expect(page.locator(".save-as-doc-btn")).to_have_count(0)
# Drafts are repo-independent: an admin can still create one…
cookies = _admin_cookies(page)
r = httpx.post(
f"{unconfigured_app}/api/doc-drafts",
json={
"title": "Unconfigured draft",
"path": "docs/unconfigured.md",
"body": "A draft while no docs repo is configured.",
},
timeout=10,
cookies=cookies,
)
assert r.status_code == 201, r.text
token = r.json()["token"]
# …but pushing 409s, naming the missing variable (D3: fail loud,
# inert by default).
r = httpx.post(
f"{unconfigured_app}/api/doc-drafts/{token}/push", timeout=10, cookies=cookies
)
assert r.status_code == 409
assert "BOR_DOCS_REPO" in r.json()["detail"]
Reference in New Issue View Git Blame Copy Permalink