"""Phase 182 E2E (Playwright): the exact text an answer shows, byte for byte. Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_verbatim_show_text.py -v --no-cov The phase's incident (TODO.md, 2026-10-14): an answer that RE-TYPES a file drifts from it by a character and the user cannot tell. The fix is that the model only ever names a LOCATION — the ``show_text`` tool or a `` ```bor [range] `` locator fence in the answer — and the SERVER resolves the bytes out of the index and displays them. This suite is the end-to-end proof of that contract on the real user path, including the clipboard truth (the mock LLM's scripted branches live in ``tests/e2e/mock_llm.py``; the KB is seeded straight into the shared e2e Postgres so the fixtures' exact text is the assertion baseline): * **App A (``BOR_ANSWER_TEXT=true``, ``BOR_ANSWER_IMAGES=true``)** - *tool flow* — ``show me the exact text of `` drives the mock's scripted ``show_text`` call, then an answer whose fence carries a DELIBERATELY CORRUPTED copy of the slice (one flipped character + a trailing space + a sentinel line). The panel is the FILE: the caption's extent is the server's, the body equals the stored text, the gutter runs 1…N, the VERBATIM badge and the viewer link are there — and NONE of the model's bytes are anywhere in the DOM (no `` ```bor ``, no sentinel, no flipped line). One ``.verbatim`` per locator (no second copy from the append path), no placeholder left pending, the answer's ordinary `` ```bash `` block wrapped as ``.code-block`` with its own Copy and NO badge (locked A5). Wire pins: exactly FIVE tools offered (the three base + ``show_image`` + ``show_text``), exactly one executed call, and — from an in-page fetch tap over the real ``/api/chat`` stream — the turn's SSE ``type`` set stays inside the pre-phase seven with ``answer_text`` riding INSIDE ``done`` (no eighth event type). - *append path* — the same turn with ``describe it without a block`` (no fence in the answer): the shown block renders as a panel APPENDED to the bubble body, below the answer text and ABOVE the source clusters (the ``.msg-meta`` family — the phase-124/130 DOM order). - *clipboard truth* — ``.verbatim-copy`` puts the server string on the clipboard EXACTLY (read back through the async clipboard API), and the ordinary block's Copy puts the model's line on it exactly. - *standalone fence* — ``put the block inline`` answers with one locator fence and ZERO tool calls; the panel resolves from the answer text (lines 2–3 of the note file, the server's extent). - *honest failures* — a locator with no document, and a range past the end (the tool refuses with ``OFFSET_PAST_END``): the panel is the pinned sentence, with no badge, no body, no line numbers and none of the model's claimed numbers on screen; the record carries no ``answer_text``. - *wrong range corrected* — a fence claiming ``2-999`` on a 9-line file: the panel shows the SERVER's extent (``lines 2–9 of 9``) and ``999`` appears nowhere. - *reload, history, share* — the panel survives the reload (the ``bor.chat.v1`` restore), re-renders from the SERVER's saved row when the chat is opened off ``/history.html`` (``/?chat=``, with local storage cleared first so the pass cannot be vacuous), and renders identically on the shared page for an anonymous reader (the one owner exception to that page's zero-controls rule: the panel's Copy button). - *image regression* — a ``show_image`` turn still shows exactly ONE ``.answer-image`` and NO panel, and ``show_text`` on an image document is REFUSED by name (the refusal copy naming ``show_image`` reaches the model; LOCKED A6) instead of silently converting it. - *phone* — at 390 px the panel's long line scrolls INSIDE the panel (no horizontal page scroll) and both Copy controls clear the 44 px touch floor. * **App B (``BOR_ANSWER_TEXT=false`` explicitly)** — the kill switch: exactly FOUR tools offered (no ``show_text``), the show-me trigger never fires (the clause is gone, so the mock stays pre-phase: one call, no tool, no panel anywhere), and the `` ```bor `` fence the inline trigger still lands never paints the model's bytes — with no ``answer_text`` on the wire it settles to the same honest sentence (locked A7 is flag-independent: the client never renders the model's re-typed body), while the ordinary code block STILL gets its Copy button (A5 is flag-independent too). The saved record carries no ``answer_text`` key. DB isolation: the suites share one Postgres; ``saved_chats`` survives the truncation (the house pattern), so every question text here is UNIQUE (its auto-title is therefore unique) and the row lookups key on it. :func:`_cleanup_saved_rows` deletes this suite's rows at both module boundaries. App B re-seeds after App A's tests (file order) and never looks back. """ from __future__ import annotations import base64 import hashlib import json import os import re import subprocess import sys import time from collections.abc import Iterator from datetime import UTC, datetime from pathlib import Path from typing import Any import httpx import pytest from playwright.sync_api import Browser, Page, expect from sqlalchemy import text from app.config import Settings from app.db import SessionLocal from app.models import Chunk, Document from e2e.auth_helpers import login from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http from e2e.mock_llm import ( IMAGE_DESCRIPTION_ANSWER, SHOW_TEXT_MISSING_TARGET, SHOW_TEXT_MODEL_BODY_MARKER, SHOW_TEXT_PLAIN_BODY, corrupt_verbatim_line, embed_text, ) REPO = Path(__file__).resolve().parents[2] #: The one source the suite seeds (a name no other suite uses). SOURCE = "baselab" DOC_A_PATH = "quadlets/gufo.container" DOC_B_PATH = "notes/verbatim-b.md" DOC_IMG_PATH = "diagrams/network-map.png" #: Fixture A: a quadlet unit file — nine lines with an obvious per-line #: identity, line 6 deliberately long so the panel MUST scroll sideways on a #: phone instead of stretching the bubble (PLAN §7.1 / phase 116 A1). NO #: trailing newline: the resolver's ``splitlines`` drops it, so the panel's #: text equals this constant byte for byte. DOC_A_CONTENT = "\n".join( [ "# baselab quadlets — the gufo container's exact text, stored in the KB", "[Container]", "ContainerName=gufo", "Image=ghcr.io/home-operations/gufo:2.4.1", "Network=baselab", ( 'Environment=GUFO_MODE=exact GUFO_TEXT_SOURCE=baselab ' 'GUFO_NOTE="the exact text of this baselab quadlet is what the ' 'user copies — gufo container, line six, long enough that a phone ' 'panel must scroll sideways instead of stretching the answer bubble"' ), "Volume=/var/home/baselab/gufo:/data:z", "RestartPolicy=always", "# The baselab quadlet above is the gufo container's exact text.", ] ) DOC_A_SUMMARY = ( "The exact text of the baselab quadlet for the gufo container: image, " "network, ports, volume and restart policy." ) #: Fixture B: a note whose text carries the shapes a markdown renderer likes #: to eat — a URL, backticks and an em dash — so the panel's escaping is #: proved against real content. Lines 2–3 are the standalone fence's target. DOC_B_CONTENT = "\n".join( [ "# Verbatim notes B — the baselab inline block reference", ( "The inline block is quoted — exactly — from " "https://lab.example/runbook#verbatim-b" ), ( "Run `podman generate quadlet gufo`, then paste the notes block " "inline for baselab." ), "GUFO_BLOCK_MARKER_4 — the fourth line of the verbatim notes file.", ( "GUFO_BLOCK_MARKER_5 — the fifth line, with `backticks`, a URL " "https://lab.example/tokens and a long tail so the panel scrolls." ), ] ) DOC_B_SUMMARY = ( "Baselab notes about quoting a block inline — the runbook URL, the " "podman command and the block markers." ) #: A real 1×1 transparent PNG (the phase-122/124 fixture bytes): the image #: document is the LOCKED-A6 refusal subject and the ``show_image`` figure. PNG_1X1 = base64.b64decode( "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ" "AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII=" ) #: The mock's deterministic answer tail (the turn-settled wait signal). MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E" #: The unresolved state's pinned sentence (``frontend/assets/verbatim.js``'s #: ``verbatimUnresolvedCopy`` — the copy the reader gets for a locator the #: server did not resolve; mirrored here on purpose: it is the contract). UNRESOLVED_COPY = "I could not verify this against the knowledge base" # --- questions (every text UNIQUE: the shared DB keeps saved_chats) ------- #: The happy path: the tool flow + one fence per locator + the escaping proof. SHOW_QUESTION = "Show me the exact text of `baselab/quadlets/gufo.container`?" #: The append path: the answer carries NO fence (``SHOW_TEXT_NO_FENCE_MARKER``). APPEND_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container` and " "describe it without a block?" ) #: The clipboard turn (a unique question; the panel is the same file). COPY_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container`? (clipboard)" ) #: The wrong-range half: the fence CLAIMS ``2-999`` of a nine-line file. WRONG_RANGE_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container` 2-999?" ) #: The past-the-end half: the tool refuses (``OFFSET_PAST_END``) and the #: fence resolves to nothing — the honest state, no claimed numbers. PAST_END_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container` 40-60?" ) #: The standalone fence: zero tool calls, one fence, lines 2–3 of fixture B. INLINE_QUESTION = ( "Please put the block inline for `baselab/notes/verbatim-b.md` 2-3?" ) #: The hallucinated locator (no document, no tool call). MISSING_QUESTION = ( "Show me a block I cannot find in the notes — " "`baselab/notes/the-file-that-is-not-there.md`?" ) #: LOCKED A6: ``show_text`` pointed at the image document. IMAGE_TARGET_QUESTION = ( "Show me the exact text of `baselab/diagrams/network-map.png`?" ) #: The phase-124 image flow (one figure, no panel). PICTURE_QUESTION = "Show me the picture of the homelab network diagram?" #: The reload/share turn. PERSIST_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container`? (persist)" ) #: The phone turn. MOBILE_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container`? (phone)" ) #: App B: the trigger with the clause gone (no tool, no fence). OFF_SHOW_QUESTION = ( "Show me the exact text of `baselab/quadlets/gufo.container`? (feature off)" ) #: App B: the fence the client must still refuse to paint. OFF_INLINE_QUESTION = ( "Please put the block inline for `baselab/notes/verbatim-b.md` 2-3? (feature off)" ) DOC_A_LINES = DOC_A_CONTENT.split("\n") DOC_B_LINES = DOC_B_CONTENT.split("\n") #: The SSE ``type`` values the pre-phase stream already carried #: (``app/schemas.py``'s chat events): the phase added NO eighth type. PRE_PHASE_EVENT_TYPES = { "delta", "thinking", "tool", "tool_result", "done", "error", "retry", } # --------------------------------------------------------------------------- # Module apps: verbatim ON (App A) and the kill switch OFF (App B). # --------------------------------------------------------------------------- def _app_env( mock_llm: int, *, answer_text: str, answer_images: str, scratch: Path, ) -> dict[str, str]: """The conftest app env (leak guards included) with the phase-182 knobs pinned EXPLICITLY — process env ranks above an operator's local gitignored ``.env``, so neither direction of the contract can leak in (the phase-122/124 default-app convention). ``BOR_IMAGES``/ ``BOR_ANSWER_IMAGES`` stay ON on both apps so the FIVE-tool offer (App A) and the FOUR-tool offer (App B) are the only difference between them, and the image-document fixtures behave identically on either.""" env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = ( "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else f"http://127.0.0.1:{mock_llm}/v1" ) # Mock-calibrated gate (conftest pattern) — every question above grounds # on these values against the two text fixtures. env["BOR_RELEVANCE_THRESHOLD"] = "0.30" env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15" env["BOR_LLM_RETRY_DELAY"] = "0" env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default) env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET # Leak guards (conftest pattern). env["BOR_GIT_SOURCES"] = "" env["BOR_DOCS_REPO"] = "" env["BOR_SUGGESTIONS"] = json.dumps(Settings.model_fields["suggestions"].default) env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default # The phase under test — pinned EXPLICITLY per app, plus the caps at the # CODE defaults (the suite's fixtures are far under both). env["BOR_ANSWER_TEXT"] = answer_text env["BOR_SHOW_TEXT_MAX_CHARS"] = str( Settings.model_fields["show_text_max_chars"].default ) env["BOR_SHOW_TEXT_MAX_BLOCKS"] = str( Settings.model_fields["show_text_max_blocks"].default ) env["BOR_IMAGES"] = "true" env["BOR_ANSWER_IMAGES"] = answer_images env["BOR_UPLOAD_DIR"] = str(scratch / "uploads") env["BOR_SOURCES_DIR"] = str(scratch / "checkouts") env["BOR_IMAGE_DIR"] = str(scratch / "images") return env def _serve( mock_llm: int, port: int, env: dict[str, str] ) -> Iterator[str]: proc = subprocess.Popen( [ sys.executable, "-m", "uvicorn", "app.main:app", "--host", "127.0.0.1", "--port", str(port), "--log-level", "warning", ], cwd=REPO, env=env, ) try: _wait_http(f"http://127.0.0.1:{port}/api/health") yield f"http://127.0.0.1:{port}" finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_server( mock_llm: int, tmp_path_factory: pytest.TempPathFactory ) -> Iterator[str]: """App A — ``BOR_ANSWER_TEXT=true`` (explicit, the code default) with images on, so the offer is the full FIVE tools.""" scratch = tmp_path_factory.mktemp("bor_verbatim_on") port = int(os.environ.get("E2E_APP_PORT_VERBATIM_ON", "8197")) yield from _serve( mock_llm, port, _app_env(mock_llm, answer_text="true", answer_images="true", scratch=scratch), ) @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server @pytest.fixture(scope="module") def off_app_server( mock_llm: int, tmp_path_factory: pytest.TempPathFactory ) -> Iterator[str]: """App B — ``BOR_ANSWER_TEXT=false`` pinned explicitly (the kill switch; only the negative tests start it, and they run AFTER App A's).""" scratch = tmp_path_factory.mktemp("bor_verbatim_off") port = int(os.environ.get("E2E_APP_PORT_VERBATIM_OFF", "8198")) yield from _serve( mock_llm, port, _app_env(mock_llm, answer_text="false", answer_images="true", scratch=scratch), ) @pytest.fixture(scope="module") def off_app_url(off_app_server: str) -> str: return off_app_server # --------------------------------------------------------------------------- # KB seeding (straight into the shared e2e Postgres — the fixtures' exact # text is the assertion baseline) + DB/UI/mock helpers # --------------------------------------------------------------------------- def _truncate_all() -> None: """The E2E isolation pattern (``test_answer_images.py``): the suites share one Postgres, so the KB tables reset per module. ``saved_chats`` is never truncated — the unique question texts keep the row lookups unambiguous and :func:`_cleanup_saved_rows` owns this suite's rows.""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, kb_overview, " "git_sources, sources_meta, folder_summaries" ) ) db.commit() def _make_doc( content: str, path: str, title: str, summary: str, *, is_image: bool = False ) -> Document: """An un-persisted document row for the fixture KB.""" return Document( source=SOURCE, path=path, full_path=f"/tmp/{path}", title=title, content=content, content_hash=hashlib.sha256(content.encode()).hexdigest(), summary=summary, is_image=is_image, indexed_at=datetime.now(UTC), # Byte-stable dates (the prompt seed's date attribute and the tool # result's date line are both rendered from this). created_at=datetime(2026, 1, 15, tzinfo=UTC), ) def _seed_kb(scratch: Path) -> dict[str, Any]: """The three-document KB (the quadlet, the note, the standalone image). Every document carries ONE chunk embedded with the MOCK's own :func:`embed_text` — the only way a question's tokens overlap the fixture so the turn grounds instead of deflecting. The image document gets a real bytes file at :attr:`image_path` (the public bytes route serves it, so the figure renders): the upload flow is phase 124's subject, not this phase's. """ _truncate_all() image_file = scratch / "network-map.png" image_file.write_bytes(PNG_1X1) with SessionLocal() as db: doc_a = _make_doc(DOC_A_CONTENT, DOC_A_PATH, "gufo.container", DOC_A_SUMMARY) doc_b = _make_doc(DOC_B_CONTENT, DOC_B_PATH, "verbatim-b", DOC_B_SUMMARY) doc_img = _make_doc( IMAGE_DESCRIPTION_ANSWER, DOC_IMG_PATH, "network-map", IMAGE_DESCRIPTION_ANSWER, is_image=True, ) doc_img.image_path = str(image_file) for doc in (doc_a, doc_b, doc_img): db.add(doc) db.flush() db.add( Chunk( document_id=doc.id, position=0, content=doc.content, embedding=embed_text(doc.content), ) ) db.commit() ids = (str(doc_a.id), str(doc_b.id), str(doc_img.id)) return { "a_id": ids[0], "b_id": ids[1], "img_id": ids[2], "image_url": f"/api/documents/{ids[2]}/image", "a_content": DOC_A_CONTENT, "b_content": DOC_B_CONTENT, "a_locator": f"{SOURCE}/{DOC_A_PATH}", "b_locator": f"{SOURCE}/{DOC_B_PATH}", } @pytest.fixture(scope="module") def seeded( app_url: str, tmp_path_factory: pytest.TempPathFactory ) -> Iterator[dict[str, Any]]: """App A's KB. Runs once for the module, after the health check.""" if not _health_db(app_url): pytest.skip("Postgres not reachable — run `podman compose up -d db` first") _cleanup_saved_rows() yield _seed_kb(tmp_path_factory.mktemp("bor_verbatim_img_on")) _cleanup_saved_rows() _truncate_all() @pytest.fixture(scope="module") def seeded_off( off_app_url: str, tmp_path_factory: pytest.TempPathFactory ) -> Iterator[dict[str, Any]]: """App B's KB — the SAME three documents (the kill switch changes the app's behaviour, never the corpus). Runs after App A's tests.""" if not _health_db(off_app_url): pytest.skip("Postgres not reachable — run `podman compose up -d db` first") _cleanup_saved_rows() yield _seed_kb(tmp_path_factory.mktemp("bor_verbatim_img_off")) _cleanup_saved_rows() _truncate_all() def _health_db(url: str) -> bool: try: return httpx.get(f"{url}/api/health", timeout=5).json()["db"] == "up" except Exception: # noqa: BLE001 — unreachable is the skip case return False def _mock_capture(mock_llm: int) -> list[dict[str, Any]]: """The mock's recent chat-completions request bodies (oldest first) — the wire lens on the tool OFFER and the executed CALLS.""" r = httpx.get(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured", timeout=5) assert r.status_code == 200 return r.json() def _mock_capture_reset(mock_llm: int) -> None: r = httpx.post(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured/reset", timeout=5) assert r.status_code == 200 def _offered_tool_names(body: dict[str, Any]) -> list[str]: return [t["function"]["name"] for t in (body.get("tools") or [])] def _executed_tool_calls(captured: list[dict[str, Any]]) -> list[str]: return [ tc["function"]["name"] for body in captured for m in body["messages"] if m.get("role") == "assistant" and m.get("tool_calls") for tc in m["tool_calls"] ] def _tool_results(body: dict[str, Any]) -> list[str]: return [ str(m.get("content") or "") for m in body.get("messages", []) if m.get("role") == "tool" ] #: In-page tap over the app's own ``fetch("/api/chat")`` stream: the SSE #: frames' ``type`` fields + the parsed ``done`` frame, collected on a second #: tee of the body so the app's own reader is untouched. The ONLY lens on the #: frame vocabulary from the browser (the phase added no eighth event type). _SSE_TAP_JS = """ (() => { const orig = window.fetch.bind(window); window.__borSse = []; window.fetch = async (...args) => { const res = await orig(...args); const url = String((args[0] && args[0].url) || args[0] || ""); if (!url.includes("/api/chat") || !res.body) return res; let keep, tap; try { [keep, tap] = res.body.tee(); } catch { return res; } const reader = tap.getReader(); const decoder = new TextDecoder(); (async () => { let buf = ""; for (;;) { const { done, value } = await reader.read(); if (done) break; buf += decoder.decode(value, { stream: true }); let idx; while ((idx = buf.indexOf("\\n\\n")) >= 0) { const chunk = buf.slice(0, idx); buf = buf.slice(idx + 2); for (const line of chunk.split("\\n")) { if (!line.startsWith("data:")) continue; try { window.__borSse.push(JSON.parse(line.slice(5))); } catch {} } } } })(); return new Response(keep, { status: res.status, statusText: res.statusText, headers: res.headers, }); }; })(); """ def _sse_frames(page: Page) -> list[dict[str, Any]]: return page.evaluate("() => window.__borSse || []") def _ask(page: Page, question: str) -> None: """Send *question* and wait for the turn to SETTLE (the house signal: the answer's marker text plus the re-enabled Send pill).""" page.fill("#message-input", question) page.click("#send-btn") bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=60_000) expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000) expect(page.locator("#send-label")).to_have_text("Send") def _panel(page: Page, locator: str) -> Any: """The resolved panels carrying one ``data-bor-locator`` (the SERVER's canonical key — the attribute the client hydrated against).""" return page.locator(f".msg.brain .verbatim[data-bor-locator='{locator}']") def _panel_texts(page: Page) -> list[str]: """Every panel's ``data-bor-locator`` in DOM order (the placement pin).""" return page.evaluate( """() => [...document.querySelectorAll(".msg.brain .verbatim[data-bor-locator]")] .map((el) => el.getAttribute("data-bor-locator"))""" ) def _body_child_classes(page: Page) -> list[str]: """The LAST brain bubble's ``.msg-body`` direct children's classes, in DOM order (the A4/append position pin).""" return page.evaluate( """() => { const body = [...document.querySelectorAll(".msg.brain")].pop() .querySelector(".msg-body"); return [...body.children].map((el) => el.className); }""" ) def _admin_cookies(page: Page) -> dict[str, str]: return { c["name"]: c["value"] for c in page.context.cookies() if "name" in c and "value" in c } def _auto_title(question: str) -> str: return " ".join(question.split())[:120] _SUITE_TITLES = tuple( _auto_title(q) for q in ( SHOW_QUESTION, APPEND_QUESTION, COPY_QUESTION, WRONG_RANGE_QUESTION, PAST_END_QUESTION, INLINE_QUESTION, MISSING_QUESTION, IMAGE_TARGET_QUESTION, PICTURE_QUESTION, PERSIST_QUESTION, MOBILE_QUESTION, OFF_SHOW_QUESTION, OFF_INLINE_QUESTION, ) ) def _cleanup_saved_rows() -> None: """Delete THIS suite's saved rows by title (the house pattern — a re-run must not find the previous run's record with a stale document id).""" try: with SessionLocal() as db: db.execute( text("DELETE FROM saved_chats WHERE title = ANY(:titles)"), {"titles": list(_SUITE_TITLES)}, ) db.commit() except Exception: # noqa: BLE001 — DB unreachable: the skip handles it pass def _wait_saved_row(url: str, cookies: dict[str, str], title: str) -> dict[str, Any]: """Poll the admin list until the turn's auto-saved row appears (auto-saves are SILENT — phase 55 A2 — so there is no status line to wait on).""" deadline = time.monotonic() + 20 last: Any = None while time.monotonic() < deadline: r = httpx.get(f"{url}/api/chats", timeout=10, cookies=cookies) assert r.status_code == 200 rows = r.json()["chats"] last = next((c for c in rows if c["title"] == title), None) if last is not None and last["message_count"] >= 2: return last time.sleep(0.2) raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})") def _brain_record( url: str, cookies: dict[str, str], chat_id: str ) -> dict[str, Any]: r = httpx.get(f"{url}/api/chats/{chat_id}", timeout=10, cookies=cookies) assert r.status_code == 200 brains = [m for m in r.json()["messages"] if m.get("who") == "brain"] assert brains, "no brain record in the saved conversation" return brains[-1] def _grant_clipboard(page: Page, url: str) -> None: page.context.grant_permissions( ["clipboard-read", "clipboard-write"], origin=url ) def _clipboard(page: Page) -> str: return page.evaluate("async () => await navigator.clipboard.readText()") def _assert_no_model_bytes(page: Page) -> None: """LOCKED A7 on the rendered surface: none of the mock's re-typed fence body reached the DOM — no sentinel line, no `` ```bor `` fence, and no flipped-character drift of a real fixture line (the corruption the phase exists for, derived from the fixture with the mock's own function).""" html = page.content() assert SHOW_TEXT_MODEL_BODY_MARKER not in html, ( "the model's re-typed fence body painted" ) assert "```bor" not in html, "a raw ```bor locator fence painted" drift = corrupt_verbatim_line(DOC_A_LINES[0]) assert drift != DOC_A_LINES[0] + " ", ( "the mock's corruption helper stopped drifting — the assertion below " "would prove nothing" ) # The flipped character itself (``# b4selab …``): the one-character drift # a retyping model produces, which is exactly what must never paint. assert "b4selab" not in html, "the model's one-character drift painted" # --------------------------------------------------------------------------- # 1. App A — the tool flow: the panel IS the file, the model's bytes are not # --------------------------------------------------------------------------- def test_show_text_turn_paints_the_files_bytes( page: Page, app_url: str, seeded: dict[str, Any], mock_llm: int ) -> None: """The incident, end to end: the mock's scripted ``show_text`` call shows fixture A, and the answer's fence carries a DELIBERATELY CORRUPTED copy of it. The panel is the file (caption extent, body bytes, gutter numbers 1…N, VERBATIM badge, viewer link, path-naming aria-label); the model's bytes are nowhere; the ordinary code block is wrapped, badged nothing and still the model's own; one panel per locator and nothing left pending. The wire pins: FIVE tools offered, one executed ``show_text``, and the turn's SSE ``type`` set inside the pre-phase seven with ``answer_text`` riding inside ``done``. Plus the persistence pin: the saved brain record carries the same blocks.""" page.add_init_script(_SSE_TAP_JS) login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, SHOW_QUESTION) # No placeholder survived the terminal frame, and none of the model's # re-typed bytes reached the DOM. expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) # --- the fixture-A panel: the SERVER's bytes, caption and gutter ----- panel = _panel(page, seeded["a_locator"]) expect(panel).to_have_count(1) expect(panel).to_have_attribute("role", "group") expect(panel.locator(".verbatim-caption")).to_have_text( f"{SOURCE}/{DOC_A_PATH} · all {len(DOC_A_LINES)} lines" ) expect(panel.locator(".verbatim-badge")).to_have_text("VERBATIM") body = panel.locator(".verbatim-text") expect(body).to_have_text(DOC_A_CONTENT) # The panel's text really is the stored row (not the trimmed render). assert body.text_content() == DOC_A_CONTENT gutter = panel.locator(".verbatim-lines li") expect(gutter).to_have_count(len(DOC_A_LINES)) expect(gutter.first).to_have_text("1") expect(gutter.last).to_have_text(str(len(DOC_A_LINES))) # The Copy control names the file and the SERVER's line count. copy_btn = panel.locator(".verbatim-copy") expect(copy_btn).to_have_count(1) expect(copy_btn).to_have_attribute( "aria-label", f"Copy {len(DOC_A_LINES)} lines from {SOURCE}/{DOC_A_PATH}", ) # The viewer link targets the document (where the git origin lives) — # the chip's own URL builder, back → the chat page. open_link = panel.locator(".verbatim-open") expect(open_link).to_have_count(1) expect(open_link).to_have_attribute( "href", f"/document.html?source={SOURCE}&path={DOC_A_PATH.replace('/', '%2F')}" "&back=%2F", ) # --- the second fence: the OTHER file, resolved the same way --------- panel_b = _panel(page, seeded["b_locator"]) expect(panel_b).to_have_count(1) expect(panel_b.locator(".verbatim-text")).to_have_text(DOC_B_CONTENT) # The escaping proof: a URL, backticks and an em dash survive as TEXT # (never markup, never eaten by the markdown pass). text_b = panel_b.locator(".verbatim-text").text_content() or "" assert "https://lab.example/runbook#verbatim-b" in text_b assert "`backticks`" in text_b assert "—" in text_b # --- placement and the no-second-copy rule -------------------------- # Both fences live inside the answer, so both panels render AT THE FENCE # (inside .bubble), in answer order; the append path adds nothing. inside = page.locator(".msg.brain .bubble .verbatim") expect(inside).to_have_count(2) assert _panel_texts(page) == [seeded["a_locator"], seeded["b_locator"]] locators = _panel_texts(page) assert len(locators) == len(set(locators)), ( f"a locator rendered twice: {locators}" ) # --- the ordinary code block: model's own, Copy, NO badge ----------- plain = page.locator(".msg.brain .bubble .code-block") expect(plain).to_have_count(1) expect(plain.locator("pre")).to_have_text(SHOW_TEXT_PLAIN_BODY) expect(plain.locator(".code-block-copy")).to_have_count(1) expect(plain.locator(".verbatim-badge")).to_have_count(0) # …and the badge belongs to exactly the two resolved panels. expect(page.locator(".msg.brain .verbatim-badge")).to_have_count(2) # --- the wire -------------------------------------------------------- captured = _mock_capture(mock_llm) assert len(captured) == 2, ( f"expected 2 model calls (show_text + answer), got {len(captured)}" ) assert _offered_tool_names(captured[0]) == [ "ls", "read", "grep", "show_image", "show_text", ], "the feature-on offer must be exactly the five tools" assert _executed_tool_calls(captured) == ["show_text"] assert any( "Text shown to the user" in r for r in _tool_results(captured[1]) ), "the show_text confirming result never reached the model" # --- the frame vocabulary (no eighth SSE type) ---------------------- frames = _sse_frames(page) types: set[str] = {str(f.get("type")) for f in frames} assert types <= PRE_PHASE_EVENT_TYPES, ( f"the stream invented an event type: {sorted(types)}" ) assert len(types) <= 7 assert "answer_text" not in types, "answer_text leaked out as its own event" done = [f for f in frames if f.get("type") == "done"] assert len(done) == 1 blocks = done[0].get("answer_text") assert isinstance(blocks, list) and [b["locator"] for b in blocks] == [ seeded["a_locator"], seeded["b_locator"], ], f"the done frame's answer_text is wrong: {blocks!r}" assert blocks[0]["text"] == DOC_A_CONTENT assert blocks[1]["text"] == DOC_B_CONTENT # --- the persistence pin -------------------------------------------- row = _wait_saved_row( app_url, _admin_cookies(page), _auto_title(SHOW_QUESTION) ) record = _brain_record( app_url, _admin_cookies(page), str(row["id"]) ) assert [b["locator"] for b in record.get("answer_text") or []] == [ seeded["a_locator"], seeded["b_locator"], ] assert record["answer_text"][0]["text"] == DOC_A_CONTENT # --------------------------------------------------------------------------- # 2. App A — the block the answer never locates APPENDS above .sources # --------------------------------------------------------------------------- def test_unlocated_block_appends_above_sources( page: Page, app_url: str, seeded: dict[str, Any] ) -> None: """The same tool turn with the answer's fences REMOVED (``describe it without a block``): the block the tool showed has no fence to live in, so it renders as a panel APPENDED to the bubble body — below the answer text and ABOVE the ``.sources`` clusters (the phase-124/130 order), still the server's bytes and still badged.""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _ask(page, APPEND_QUESTION) expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) # The panel is OUTSIDE the answer bubble (nothing located it there). expect(page.locator(".msg.brain .bubble .verbatim")).to_have_count(0) panel = _panel(page, seeded["a_locator"]) expect(panel).to_have_count(1) expect(panel.locator(".verbatim-text")).to_have_text(DOC_A_CONTENT) expect(panel.locator(".verbatim-badge")).to_have_text("VERBATIM") classes = _body_child_classes(page) assert "bubble" in classes, f"no answer bubble in .msg-body: {classes}" assert "verbatim" in classes, f"no appended panel in .msg-body: {classes}" # The source clusters are the .msg-meta family (phase 130: the chips row # is a .msg-meta, an image doc cites by figure in .msg-meta.msg-images — # here the auto-cited PNG, since a show_text turn performs no read). sources_idx = next( (i for i, c in enumerate(classes) if c.split(" ")[0] == "msg-meta"), None ) assert sources_idx is not None, f"no source cluster in .msg-body: {classes}" assert ( classes.index("bubble") < classes.index("verbatim") < sources_idx ), f"append order violated (bubble < verbatim < sources): {classes}" # --------------------------------------------------------------------------- # 3. App A — the clipboard equals the server's string, for both controls # --------------------------------------------------------------------------- def test_copy_buttons_copy_the_panel_and_the_plain_block( page: Page, app_url: str, seeded: dict[str, Any] ) -> None: """``.verbatim-copy`` puts the resolved block on the clipboard EXACTLY (read back through the async clipboard API — the byte-for-byte equality the phase exists for), the pill settles to ``Copied`` text, and the ordinary answer code block's Copy puts the MODEL's line on it exactly — the two controls, the two different sources, one shared builder.""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _grant_clipboard(page, app_url) _ask(page, COPY_QUESTION) panel = _panel(page, seeded["a_locator"]) expect(panel).to_have_count(1) panel.locator(".verbatim-copy").click() expect(panel.locator(".verbatim-copy")).to_have_text("Copied") assert _clipboard(page) == DOC_A_CONTENT, ( "the panel's clipboard content is not the server's string exactly" ) plain = page.locator(".msg.brain .bubble .code-block .code-block-copy") expect(plain).to_have_count(1) plain.click() expect(plain).to_have_text("Copied") assert _clipboard(page) == SHOW_TEXT_PLAIN_BODY, ( "the ordinary block's clipboard content is not its own text" ) # The line-number gutter never rides into the clipboard (user-select: # none): the panel's text has no leading digits on any line. assert not re.search(r"(?m)^\d+\s", _clipboard(page)) # --------------------------------------------------------------------------- # 4. App A — a standalone locator fence resolves with ZERO tool calls # --------------------------------------------------------------------------- def test_standalone_fence_resolves_without_a_tool_call( page: Page, app_url: str, seeded: dict[str, Any], mock_llm: int ) -> None: """The half of the contract the tool does not cover: an answer that only WRITES a locator fence. Zero tool calls on the wire, one panel resolved from the answer text, the server's extent in the caption (``lines 2–3 of 5``) and the model's claimed body nowhere.""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, INLINE_QUESTION) captured = _mock_capture(mock_llm) assert len(captured) == 1, ( f"a fence-only answer must make exactly 1 model call, got {len(captured)}" ) assert _executed_tool_calls(captured) == [], "the fence-only turn called a tool" expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) locator = f"{seeded['b_locator']}#2-3" panel = _panel(page, locator) expect(panel).to_have_count(1) expect(panel.locator(".verbatim-caption")).to_have_text( f"{SOURCE}/{DOC_B_PATH} · lines 2–3 of {len(DOC_B_LINES)}" ) expect(panel.locator(".verbatim-badge")).to_have_text("VERBATIM") expected = "\n".join(DOC_B_LINES[1:3]) assert panel.locator(".verbatim-text").text_content() == expected # The panel is INSIDE the bubble at the fence position, and the answer's # prose still rendered around it. expect(page.locator(".msg.brain .bubble .verbatim")).to_have_count(1) expect(page.locator(".msg.brain .bubble")).to_contain_text( "the block itself" ) # The gutter carries the SERVER's numbers (2, 3), not 1-based ones. gutter = panel.locator(".verbatim-lines li") expect(gutter).to_have_count(2) expect(gutter.first).to_have_text("2") expect(gutter.last).to_have_text("3") # --------------------------------------------------------------------------- # 5. App A — a locator the server cannot resolve says so # --------------------------------------------------------------------------- def test_hallucinated_locator_renders_the_honest_state( page: Page, app_url: str, seeded: dict[str, Any], mock_llm: int ) -> None: """A fence naming a file with no row: NO panel — the pinned sentence, with no badge, no body, no gutter and none of the model's bytes; the viewer link still targets the named file (the reader can go look). The saved record carries no ``answer_text`` key at all (no block displayed).""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, MISSING_QUESTION) assert _executed_tool_calls(_mock_capture(mock_llm)) == [] expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) honest = page.locator(".msg.brain .verbatim.verbatim-unresolved") expect(honest).to_have_count(1) expect(honest.locator(".verbatim-state")).to_have_text(UNRESOLVED_COPY) expect(honest.locator(".verbatim-badge")).to_have_count(0) expect(honest.locator(".verbatim-body")).to_have_count(0) expect(honest.locator(".verbatim-lines")).to_have_count(0) expect(page.locator(".msg.brain .verbatim-copy")).to_have_count(0) # The honest state still names the file the model asked for, and still # links to it (the reader can go look for themselves). expect(honest).to_have_attribute( "data-bor-locator", f"{SHOW_TEXT_MISSING_TARGET}#3-4" ) expect(honest.locator(".verbatim-open")).to_have_attribute( "href", "/document.html?source=baselab&path=notes%2Fthe-file-that-is-not-there.md" "&back=%2F", ) # Nothing resolved → nothing displayed → no resolved panel anywhere. expect(page.locator(".msg.brain .verbatim:not(.verbatim-unresolved)")).to_have_count( 0 ) row = _wait_saved_row(app_url, _admin_cookies(page), _auto_title(MISSING_QUESTION)) record = _brain_record(app_url, _admin_cookies(page), str(row["id"])) assert "answer_text" not in record, ( f"a turn with no block must persist no answer_text: {record!r}" ) # --------------------------------------------------------------------------- # 6. App A — a wrong range: the server's extent, or the honest state # --------------------------------------------------------------------------- def test_wrong_range_shows_the_servers_extent( page: Page, app_url: str, seeded: dict[str, Any] ) -> None: """The fence claims ``2-999`` of a nine-line file. The tool clamps and displays 2–9, so the panel shows the SERVER's extent (caption, gutter and bytes all 2–9) under the model's own placeholder key — and the claimed ``999`` appears nowhere in the DOM.""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _ask(page, WRONG_RANGE_QUESTION) expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) server_locator = f"{seeded['a_locator']}#2-{len(DOC_A_LINES)}" panel = _panel(page, server_locator) expect(panel).to_have_count(1) expect(panel.locator(".verbatim-caption")).to_have_text( f"{SOURCE}/{DOC_A_PATH} · lines 2–{len(DOC_A_LINES)} of {len(DOC_A_LINES)}" ) assert ( panel.locator(".verbatim-text").text_content() == "\n".join(DOC_A_LINES[1:]) ) expect(panel.locator(".verbatim-badge")).to_have_text("VERBATIM") expect(panel.locator(".verbatim-lines li").first).to_have_text("2") expect(panel.locator(".verbatim-lines li").last).to_have_text( str(len(DOC_A_LINES)) ) # The show SUCCEEDED (clamped), so the answer's second fence resolves too # — the model's claimed ``999`` is still nowhere on screen. expect(_panel(page, seeded["b_locator"])).to_have_count(1) # VISIBLE text (the unresolved/pending node may carry the model's own # locator as its identity attribute — that is how its link targets the # file; what must never appear is the claim ON SCREEN). visible = page.locator(".msg.brain").last.inner_text() assert "999" not in visible, ( f"the model's claimed end line painted instead of the server's: {visible!r}" ) def test_past_end_range_refuses_and_says_so( page: Page, app_url: str, seeded: dict[str, Any], mock_llm: int ) -> None: """The other wrong-range half: ``offset`` past the last line. The tool REFUSES (``OFFSET_PAST_END`` — the refusal names the true line count to the model), nothing displays, and the client's honest state renders with NONE of the model's claimed numbers on screen.""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, PAST_END_QUESTION) captured = _mock_capture(mock_llm) assert _executed_tool_calls(captured) == ["show_text"] refusals = _tool_results(captured[1]) assert any( "is past the end" in r and f"is {len(DOC_A_LINES)} lines long" in r and "`offset` 40" in r for r in refusals ), f"the OFFSET_PAST_END teaching never reached the model: {refusals}" expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) honest = page.locator(".msg.brain .verbatim.verbatim-unresolved") expect(honest).to_have_count(1) expect(honest.locator(".verbatim-state")).to_have_text(UNRESOLVED_COPY) expect(page.locator(".msg.brain .verbatim-body")).to_have_count(0) expect(page.locator(".msg.brain .verbatim-badge")).to_have_count(0) visible = page.locator(".msg.brain").last.inner_text() assert "40-60" not in visible, f"the refused range painted: {visible!r}" # --------------------------------------------------------------------------- # 7. App A — reload restores the panel; the shared page renders it too # --------------------------------------------------------------------------- def test_reload_and_share_re_render_the_panel( page: Page, app_url: str, seeded: dict[str, Any], browser: Browser, ) -> None: """The same turn, four surfaces, one renderer: LIVE → RELOAD (the ``bor.chat.v1`` restore re-hydrates from the stored record) → the chat OPENED FROM ``/history.html`` (the ``?chat=`` boot load, which pulls the row from ``GET /api/chats/{id}`` — the SERVER's record, not local storage) → the SHARED PAGE for an anonymous reader. The panel's bytes, caption and Copy button are identical on all four (the shared page's zero-controls rule has exactly this exception).""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _ask(page, PERSIST_QUESTION) panel = _panel(page, seeded["a_locator"]) expect(panel).to_have_count(1) assert panel.locator(".verbatim-text").text_content() == DOC_A_CONTENT cookies = _admin_cookies(page) saved = _wait_saved_row(app_url, cookies, _auto_title(PERSIST_QUESTION)) # RELOAD — the restore path renders the panel from stored state. page.reload() expect(page.locator("#kb-banner")).to_be_hidden() restored = _panel(page, seeded["a_locator"]) expect(restored).to_have_count(1, timeout=30_000) assert restored.locator(".verbatim-text").text_content() == DOC_A_CONTENT expect(restored.locator(".verbatim-caption")).to_have_text( f"{SOURCE}/{DOC_A_PATH} · all {len(DOC_A_LINES)} lines" ) expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) # OPEN FROM HISTORY — the row's Open link (``/?chat=``) boots the # chat page from the SAVED record through the admin API, so this is the # panel rendering off the persisted ``answer_text`` rather than local # storage. chat_id = str(saved["id"]) # Drop the local record FIRST: after this the panel can only come from # the API's saved row (a vacuous pass through localStorage is impossible). page.evaluate("() => window.localStorage.clear()") page.goto(f"{app_url}/history.html") open_link = page.locator(f".history-title-link[href='/?chat={chat_id}']") expect(open_link).to_have_count(1, timeout=30_000) open_link.click() from_history = _panel(page, seeded["a_locator"]) expect(from_history).to_have_count(1, timeout=30_000) assert from_history.locator(".verbatim-text").text_content() == DOC_A_CONTENT expect(from_history.locator(".verbatim-badge")).to_have_text("VERBATIM") expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) # SHARE — a fresh anonymous context, no cookies. _grant_clipboard(page, app_url) page.locator("#share-chat-btn").click() if page.evaluate("() => !!navigator.clipboard"): expect(page.locator("#send-status")).to_have_text( "Share link copied.", timeout=15_000 ) else: expect(page.locator(".share-link-fallback")).to_be_visible() rows = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies).json()[ "chats" ] shared = next( (c for c in rows if c["title"] == _auto_title(PERSIST_QUESTION) and c.get("share_url")), None, ) assert shared is not None and shared.get("share_url"), f"no share_url: {rows!r}" anon = browser.new_page(viewport={"width": 1280, "height": 800}) try: anon.goto(app_url + str(shared["share_url"])) expect(anon.locator("#shared-title")).to_have_text( _auto_title(PERSIST_QUESTION), timeout=30_000 ) shared_panel = anon.locator( f".msg.brain .verbatim[data-bor-locator='{seeded['a_locator']}']" ) expect(shared_panel).to_have_count(1, timeout=30_000) assert shared_panel.locator(".verbatim-text").text_content() == DOC_A_CONTENT expect(shared_panel.locator(".verbatim-badge")).to_have_text("VERBATIM") # The one exception to the shared page's zero-controls rule: the # panel's Copy button (and its plain viewer link — no modal there). expect(shared_panel.locator(".verbatim-copy")).to_have_count(1) expect(shared_panel.locator(".verbatim-open")).to_have_attribute( "href", "/document.html?source=baselab&path=quadlets%2Fgufo.container", ) assert SHOW_TEXT_MODEL_BODY_MARKER not in anon.content() finally: anon.close() # --------------------------------------------------------------------------- # 8. App A — the image surfaces: one figure, no panel; show_text refuses an # image document by name (LOCKED A6) # --------------------------------------------------------------------------- def test_answer_image_turn_shows_one_figure_and_no_panel( page: Page, app_url: str, seeded: dict[str, Any], mock_llm: int ) -> None: """The phase-124 surface is untouched: the ``show me the picture of`` trigger still streams exactly ONE ``.answer-image`` figure (the bytes route serves 200 with the fixture bytes) and NO verbatim panel — and no ``answer_text`` on the done frame, so nothing else renders.""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, PICTURE_QUESTION) figure = page.locator(".msg.brain .answer-image") expect(figure).to_have_count(1) img = figure.locator("img") expect(img).to_have_attribute("src", seeded["image_url"]) expect(img).to_be_visible(timeout=15_000) resp = httpx.get(app_url + seeded["image_url"], timeout=10) assert resp.status_code == 200 assert resp.content == PNG_1X1 expect(page.locator(".msg.brain .verbatim")).to_have_count(0) assert _executed_tool_calls(_mock_capture(mock_llm)) == ["show_image"] def test_show_text_on_an_image_document_is_refused_by_name( page: Page, app_url: str, seeded: dict[str, Any], mock_llm: int ) -> None: """LOCKED A6: ``show_text`` pointed at an image document is REFUSED, never silently converted — the refusal copy naming ``show_image`` is what reaches the model, and the surface shows the honest state (no panel, no figure: the answer called no display tool).""" login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, IMAGE_TARGET_QUESTION) captured = _mock_capture(mock_llm) assert _executed_tool_calls(captured) == ["show_text"] results = _tool_results(captured[1]) assert any( "standalone image document" in r and "show_image" in r for r in results ), f"the A6 refusal naming show_image never reached the model: {results}" expect(page.locator(".msg.brain .verbatim-body")).to_have_count(0) expect(page.locator(".msg.brain .verbatim-badge")).to_have_count(0) expect(page.locator(".msg.brain .answer-image")).to_have_count(0) expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) expect(page.locator(".msg.brain .verbatim.verbatim-unresolved")).to_have_count(1) # --------------------------------------------------------------------------- # 9. App A — the phone: the panel scrolls inside itself, controls are 44px # --------------------------------------------------------------------------- def test_phone_panel_scrolls_and_controls_clear_44px( browser: Browser, app_url: str, seeded: dict[str, Any] ) -> None: """At 390 px the panel's long line (fixture A line 6) scrolls INSIDE the bounded panel — the page itself never scrolls sideways — and both Copy controls clear the 44 px WCAG touch floor.""" page = browser.new_page(viewport={"width": 390, "height": 844}) try: login(page, app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _ask(page, MOBILE_QUESTION) panel = _panel(page, seeded["a_locator"]) expect(panel).to_have_count(1, timeout=30_000) metrics = panel.locator(".verbatim-body").evaluate( """(el) => ({ scrollWidth: el.scrollWidth, clientWidth: el.clientWidth, pageScroll: document.documentElement.scrollWidth, viewport: window.innerWidth, })""" ) assert metrics["scrollWidth"] > metrics["clientWidth"], ( f"the long line does not scroll inside the panel: {metrics}" ) assert metrics["pageScroll"] <= metrics["viewport"] + 1, ( f"the page itself scrolls sideways at 390px: {metrics}" ) for selector in (".verbatim-copy", ".code-block-copy"): box = page.locator(f".msg.brain {selector}").first.bounding_box() assert box is not None, f"no {selector} to measure" assert box["height"] >= 44, f"{selector} height {box['height']} < 44px" assert box["width"] >= 44, f"{selector} width {box['width']} < 44px" finally: page.close() # --------------------------------------------------------------------------- # 10. App B (BOR_ANSWER_TEXT=false) — the kill switch # --------------------------------------------------------------------------- def test_answer_text_off_no_tool_no_panel( page: Page, off_app_url: str, seeded_off: dict[str, Any], mock_llm: int ) -> None: """The escape hatch, end to end: the clause is gone, so the mock's show-me flow never fires (ONE model call, zero tool calls) and the offer is exactly FOUR tools with no ``show_text``. The answer renders as plain prose — no ``.verbatim`` element of any kind, and the saved record carries no ``answer_text`` key.""" login(page, off_app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _mock_capture_reset(mock_llm) _ask(page, OFF_SHOW_QUESTION) captured = _mock_capture(mock_llm) assert len(captured) == 1, ( f"the clause-gated flow must not fire, got {len(captured)} model calls" ) assert _offered_tool_names(captured[0]) == ["ls", "read", "grep", "show_image"], ( "show_text leaked into the kill-switch offer" ) assert _executed_tool_calls(captured) == [] expect(page.locator(".msg.brain .verbatim")).to_have_count(0) expect(page.locator(".verbatim-pending")).to_have_count(0) row = _wait_saved_row( off_app_url, _admin_cookies(page), _auto_title(OFF_SHOW_QUESTION) ) record = _brain_record(off_app_url, _admin_cookies(page), str(row["id"])) assert "answer_text" not in record, ( f"the kill-switch record must carry no answer_text: {record!r}" ) def test_answer_text_off_fence_never_paints_the_model( page: Page, off_app_url: str, seeded_off: dict[str, Any], mock_llm: int ) -> None: """A `` ```bor `` fence on the kill-switch app: nothing on the wire resolves it, so the client settles it to the honest sentence — the model's re-typed body STILL never reaches the DOM (locked A7 is flag-independent), there is no badge and no body, while the ordinary code block keeps its working Copy button (A5 is flag-independent too).""" login(page, off_app_url, next="/") expect(page.locator("#kb-banner")).to_be_hidden() _grant_clipboard(page, off_app_url) _mock_capture_reset(mock_llm) _ask(page, OFF_INLINE_QUESTION) captured = _mock_capture(mock_llm) assert len(captured) == 1 and _executed_tool_calls(captured) == [] assert "show_text" not in _offered_tool_names(captured[0]) expect(page.locator(".verbatim-pending")).to_have_count(0) _assert_no_model_bytes(page) honest = page.locator(".msg.brain .verbatim.verbatim-unresolved") expect(honest).to_have_count(1) expect(honest.locator(".verbatim-state")).to_have_text(UNRESOLVED_COPY) expect(honest.locator(".verbatim-badge")).to_have_count(0) expect(honest.locator(".verbatim-body")).to_have_count(0) plain = page.locator(".msg.brain .bubble .code-block") expect(plain).to_have_count(1) expect(plain.locator("pre")).to_have_text(SHOW_TEXT_PLAIN_BODY) expect(plain.locator(".verbatim-badge")).to_have_count(0) plain.locator(".code-block-copy").click() expect(plain.locator(".code-block-copy")).to_have_text("Copied") assert _clipboard(page) == SHOW_TEXT_PLAIN_BODY row = _wait_saved_row( off_app_url, _admin_cookies(page), _auto_title(OFF_INLINE_QUESTION) ) record = _brain_record(off_app_url, _admin_cookies(page), str(row["id"])) assert "answer_text" not in record # The raw fence text is what persists (the answer is stored as typed) — # it renders as the honest state, never as the model's claim. assert "```bor" in (record["text"] or "")