Phase 192 — final verification pass complete. - **Verified (all 4 tasks already complete):** additive `verbatim_block` SSE frame (`VerbatimBlockPiece` → `ChatVerbatimBlockEvent` → chat dispatch, after the round's `tool`/`tool_result`, before the next round); `hydrateVerbatimNow` matched-only mid-turn pass + app.js branch; `dropPlacedTwin` per-locator done/restore dedupe; the dedicated E2E suite. - **Root cause of the rejected attempt:** my own concurrent background `pytest` run deadlocked the harness validation run on the shared Postgres (`psycopg.errors.DeadlockDetected`). No code defect — re-ran every gate **exclusively**. - `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, `TOTAL 7181 16 99%` (>90%) - `uv run pytest tests/e2e/test_verbatim_inline_midturn.py --no-cov` → **6 passed** (isolated) - `uv run pytest tests/e2e/test_verbatim_show_text.py tests/e2e/test_chat_history.py --no-cov` → **18 passed** (`test_chat_history.py` byte-unedited; the phase-182 suite's "no eighth type" pin was tightened, not loosened, to seven + `verbatim_block` with frame-order asserts) - `uv run ruff check .` → All checks passed (exit 0) · `uv run pyright` → **0 errors** - **Completion criteria:** single frame per success + zero per refusal ✅ · `BOR_ANSWER_TEXT=0` wire byte-identical (integration spelled baseline + E2E App B) ✅ · fence hydrates in place / unfenced lands at the call moment, before `done` ✅ · append+fence → 1 panel, two ranges → 2 panels ✅ · restore 1 panel/locator, `done.answer_text` + record key + `verbatim_blocks=N` unchanged ✅ · suite green, coverage 99%, lint/types clean ✅ · no-show_text turn byte-identical ✅ · commit → left to the harness (no git index/history commands run). - **Docs:** README reviewed line-by-line (2399 lines) + AGENTS.md in full; mechanically verified all 86 `BOR_*` settings, every `.env.example` var, table defaults, internal anchors (0 broken), ports 8123/8901, tree depth 1–8, pagination 25/1–100, 17 palette colors, `0029`/`0030` migrations, and every cited test file. Fixed 1 inconsistency: the verbatim "Parity" bullet claimed the live turn/reload/shared page share "the same three globals" — now names the settle-path trio and states that `hydrateVerbatimNow` is the ONE chat-only addition (a shared page has no stream). - **Notable:** `run_agent`'s four repeated piece unions were collapsed into the ONE exported `AgentPiece` alias, so a future piece type is spelled in one place; `test_frontend_edit_message.py`'s `runTurn` byte-pin was regenerated, not weakened. - **Next pending phase:** `193_sync_remaining_count`.
1297 lines
52 KiB
Python
1297 lines
52 KiB
Python
"""Phase 192 E2E (Playwright): the panel lands WHILE the answer is still typing.
|
|
|
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
|
|
|
uv run pytest tests/e2e/test_verbatim_inline_midturn.py -v --no-cov
|
|
|
|
The TODO item (TODO.md:6, 2026-10-16): a ``show_text`` block the server had
|
|
already resolved used to appear only when the turn was OVER — the model kept
|
|
typing for another ten seconds and the panel popped in afterwards, so the
|
|
answer never read as "here is the text" at the point the model introduced it.
|
|
Phase 192 sends the block on its OWN additive SSE frame
|
|
(``verbatim_block``, the eighth event type) the moment the call succeeds —
|
|
after that round's ``tool``/``tool_result`` frames, before the next model
|
|
round — and the client settles the placeholders that frame names through the
|
|
ONE matched-only pass (``hydrateVerbatimNow``) plus the answer-position
|
|
append, both mid-turn.
|
|
|
|
This suite is the owner-visible proof of that contract, and of the clause
|
|
that keeps it safe (LOCKED **A4 — the unchanged contract**): the answer's
|
|
``delta`` payloads, the ``done`` frame's shape for every surface the phase
|
|
does not touch, the record shape, the restore pass and the shared page are
|
|
BYTE-IDENTICAL to phase 182. The kill switch (``BOR_ANSWER_TEXT=0``) is the
|
|
extreme case, and it is pinned here as such: the same mock turn against App B
|
|
ships the pre-phase wire — the same ``delta`` bytes in the same order, no
|
|
``verbatim_block`` frame, no ``answer_text`` key, no panel, no placeholder.
|
|
|
|
The arms (the mock's scripted branches live in ``tests/e2e/mock_llm.py``
|
|
under ``MIDTURN_TRIGGER``; the KB is seeded straight into the shared e2e
|
|
Postgres so the fixture's exact text is the assertion baseline). Every mock
|
|
turn leaves a deliberate ``MIDTURN_GAP_S`` of silence between the
|
|
``verbatim_block`` frame and the answer's first ``delta``, so "mid-turn" is
|
|
witnessed by the DOM rather than raced against it: an in-page
|
|
``MutationObserver`` snapshots, at the instant each panel is inserted,
|
|
whether the ``done`` frame had been read, whether the answer's sentinel token
|
|
was on screen and whether the Send pill still said Stop.
|
|
|
|
* **(a) mid-turn (the TODO regression)** — an intro ``delta`` → the
|
|
``show_text`` call for ``midturnlab/runbooks/valve-panel.md#12-20`` → the
|
|
continuation deltas (the last carrying the sentinel) → ``done``. The panel
|
|
is in the DOM BEFORE ``done`` (its witness record: no ``done`` read, no
|
|
sentinel painted, the pill still "Stop"), it sits BELOW the intro text the
|
|
model had already written, its caption/gutter/body are the SERVER's, and
|
|
the ``done`` pass adds nothing (final count 1). The wire: the frame order
|
|
is ``delta* tool verbatim_block delta* done`` and the concatenated
|
|
``delta`` payload equals the mock's answer byte for byte.
|
|
* **(b) fence in place** — the fence rides round 1, so the placeholder
|
|
EXISTS when the frame lands: the mid-turn pass resolves it IN PLACE (the
|
|
panel is inside ``.bubble`` at the fence position, not appended at the
|
|
answer tail) before ``done``, and the total for that locator stays 1.
|
|
* **(c) no double panel** — the realistic ordering the task describes: the
|
|
call first, the fence for the SAME locator inside the final answer. The
|
|
block lands mid-turn at the answer position, the fence streams as a
|
|
placeholder, and ``done`` re-homes it onto the fence — exactly ONE panel
|
|
afterwards, at the fence position, the appended twin gone (task 03's
|
|
``dropPlacedTwin``, owner-visible).
|
|
* **(d) two ranges** — TWO calls, ``1-10`` then ``11-20`` of the same file,
|
|
no fences: TWO panels at the answer position in call order, each with its
|
|
own caption (the per-locator, not per-identity pin).
|
|
* **(e) kill switch** — App B (``BOR_ANSWER_TEXT=0``): the same attempt hits
|
|
the phase-182 ``Unknown tool.`` refusal, the captured frame sequence equals
|
|
the documented pre-phase baseline, and the DOM carries no panel and no
|
|
placeholder.
|
|
* **(f) restore** — arm (a) to completion, then reload: exactly ONE panel
|
|
for the locator from the record's ``answer_text`` through the same
|
|
functions (the A4 parity, owner-visible).
|
|
|
|
DB isolation: the suites share one Postgres and ``saved_chats`` survives the
|
|
truncation (the house pattern), so every question text here is UNIQUE and
|
|
:func:`_cleanup_saved_rows` deletes this suite's rows at both module
|
|
boundaries. App B re-seeds after App A's tests (file order).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
from collections.abc import Iterator
|
|
from datetime import UTC, datetime
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import httpx
|
|
import pytest
|
|
from playwright.sync_api import Browser, BrowserContext, Page, expect
|
|
from sqlalchemy import text
|
|
|
|
from app.config import Settings
|
|
from app.db import SessionLocal
|
|
from app.models import Chunk, Document
|
|
from e2e.auth_helpers import login
|
|
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
|
from e2e.mock_llm import (
|
|
MIDTURN_COMBINED,
|
|
MIDTURN_FENCE_BEFORE_MARKER,
|
|
MIDTURN_FENCE_BODY_MARKER,
|
|
MIDTURN_INTRO,
|
|
MIDTURN_PATH,
|
|
MIDTURN_PLAIN_ANSWER,
|
|
MIDTURN_SENTINEL,
|
|
MIDTURN_SOURCE,
|
|
MIDTURN_TRIGGER,
|
|
embed_text,
|
|
)
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
|
|
#: The fixture document (a source name no other suite uses).
|
|
DOC_PATH = MIDTURN_PATH
|
|
DOC_TITLE = "valve-panel"
|
|
DOC_SUMMARY = (
|
|
"The valve panel runbook: walk through the stored text of the panel "
|
|
"line by line, close VALVE A on the west header, and store every "
|
|
"deviation against the runbook line."
|
|
)
|
|
|
|
#: NO trailing newline: the resolver's ``splitlines`` drops it, so the
|
|
#: panel's text equals this constant byte for byte. The vocabulary is
|
|
#: deliberately repetitive — the mock's embeddings are token-overlap, and the
|
|
#: turn must GROUND (a deflected turn offers no tools and would never show a
|
|
#: panel at all).
|
|
DOC_CONTENT = "\n".join(
|
|
[
|
|
"---",
|
|
"title: valve-panel runbook",
|
|
"source: midturnlab",
|
|
"---",
|
|
"",
|
|
"# Valve panel runbook — walk through the stored text",
|
|
"",
|
|
"Walk through the stored text of the valve panel line by line.",
|
|
"",
|
|
"## Panel layout",
|
|
"",
|
|
"VALVE A feeds the west header; the stored text closes it first.",
|
|
"VALVE B feeds the east header while the tank drains.",
|
|
"The panel gauge, not the pump, is the stored reading.",
|
|
"Walk through the west header before the pump room door opens.",
|
|
"The valve panel drawing is the only stored layout.",
|
|
"VALVE B packing is replaced with the header dry.",
|
|
"Every panel line is walked, stored and countersigned.",
|
|
"The runbook walk ends at the pump room log.",
|
|
"A deviation is stored against this runbook line.",
|
|
"",
|
|
"## After the walk",
|
|
"",
|
|
"The stored runbook text is signed by the shift lead.",
|
|
]
|
|
)
|
|
DOC_LINES = DOC_CONTENT.split("\n")
|
|
TOTAL_LINES = len(DOC_LINES)
|
|
|
|
#: The ranges the arms request (the mock reads them straight out of the
|
|
#: question, so the question and these constants cannot drift apart).
|
|
PRIMARY_START, PRIMARY_END = 12, 20
|
|
FIRST_START, FIRST_END = 1, 10
|
|
SECOND_START, SECOND_END = 11, 20
|
|
|
|
|
|
def _locator(start: int, end: int) -> str:
|
|
"""The SERVER's canonical locator for one range."""
|
|
return f"{MIDTURN_SOURCE}/{DOC_PATH}#{start}-{end}"
|
|
|
|
|
|
def _block_text(start: int, end: int) -> str:
|
|
"""The stored lines the panel must show for one range."""
|
|
return "\n".join(DOC_LINES[start - 1 : end])
|
|
|
|
|
|
def _caption(start: int, end: int) -> str:
|
|
"""The caption: the identity is the DERIVED ``source/path`` and the
|
|
numbers are the SERVER's extent (``frontend/assets/verbatim.js``)."""
|
|
return (
|
|
f"{MIDTURN_SOURCE}/{DOC_PATH} · "
|
|
f"lines {start}\u2013{end} of {TOTAL_LINES}"
|
|
)
|
|
|
|
|
|
# --- questions (every text UNIQUE: the shared DB keeps saved_chats) -------
|
|
|
|
#: (a) the mid-turn arm — one call, no fence anywhere.
|
|
MIDTURN_QUESTION = (
|
|
f"{MIDTURN_TRIGGER}: {MIDTURN_SOURCE}/{DOC_PATH} "
|
|
f"{PRIMARY_START}-{PRIMARY_END}?"
|
|
)
|
|
#: (b) the fence rides round 1 — the placeholder exists at the frame.
|
|
FENCE_BEFORE_QUESTION = (
|
|
f"{MIDTURN_TRIGGER}, {MIDTURN_FENCE_BEFORE_MARKER}: "
|
|
f"{MIDTURN_SOURCE}/{DOC_PATH} {PRIMARY_START}-{PRIMARY_END}?"
|
|
)
|
|
#: (c) the fence rides the final answer — the same locator, no double panel.
|
|
FENCE_AFTER_QUESTION = (
|
|
f"{MIDTURN_TRIGGER}, fence it in the answer: {MIDTURN_SOURCE}/{DOC_PATH} "
|
|
f"{PRIMARY_START}-{PRIMARY_END}?"
|
|
)
|
|
#: (d) two ranges of one file, two calls, no fences.
|
|
TWO_RANGES_QUESTION = (
|
|
f"{MIDTURN_TRIGGER}: {MIDTURN_SOURCE}/{DOC_PATH} {FIRST_START}-{FIRST_END} "
|
|
f"then the next block {SECOND_START}-{SECOND_END}?"
|
|
)
|
|
#: (e) App B: the same attempt, the refusal instead of the block.
|
|
OFF_QUESTION = f"{MIDTURN_QUESTION} (feature off)"
|
|
#: (f) the restore turn (arm (a) again, on a page that then reloads).
|
|
RESTORE_QUESTION = f"{MIDTURN_QUESTION} (restore)"
|
|
|
|
#: The turn's answer tail (the house settle signal).
|
|
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
|
|
|
#: The SSE ``type`` values the pre-phase stream already carried
|
|
#: (``app/schemas.py``'s chat events). Phase 192 adds exactly ONE type —
|
|
#: ``verbatim_block`` — and App B's stream carries none of it.
|
|
PRE_PHASE_EVENT_TYPES = {
|
|
"delta",
|
|
"thinking",
|
|
"tool",
|
|
"tool_result",
|
|
"done",
|
|
"error",
|
|
"retry",
|
|
}
|
|
|
|
#: The whole turn's answer bytes on the wire, A and B alike: the mid-round
|
|
#: intro the model wrote BEFORE the call, then the answer it wrote AFTER the
|
|
#: result. The A4 clause is that adding the frame changed neither.
|
|
WIRE_ANSWER_BYTES = MIDTURN_INTRO + MIDTURN_PLAIN_ANSWER
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Module apps: verbatim ON (App A) and the kill switch OFF (App B).
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _app_env(
|
|
mock_llm: int, *, answer_text: str, scratch: Path
|
|
) -> dict[str, str]:
|
|
"""The conftest app env (leak guards included) with the phase-192 knob
|
|
pinned EXPLICITLY — process env ranks above an operator's local gitignored
|
|
``.env``, so neither direction of the contract can leak in (the phase-182
|
|
convention). Images are OFF on both apps: this phase has no image surface,
|
|
and a figure would only add nodes to the DOM the position pins read."""
|
|
env = dict(os.environ)
|
|
env.pop("DEBUGPY", None)
|
|
env["BOR_ENVIRONMENT"] = "e2e"
|
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
|
env["BOR_LLM_BASE_URL"] = (
|
|
"https://aipi.reeseapps.com/v1"
|
|
if USE_REAL_LLM
|
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
|
)
|
|
# Mock-calibrated gate (conftest pattern) — every question above grounds
|
|
# on these values against the one fixture document.
|
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
|
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
|
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
|
env["BOR_LLM_RETRY_DELAY"] = "0"
|
|
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
|
env.setdefault(
|
|
"BOR_DATABASE_URL",
|
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
|
)
|
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
|
# Leak guards (conftest pattern).
|
|
env["BOR_GIT_SOURCES"] = ""
|
|
env["BOR_DOCS_REPO"] = ""
|
|
env["BOR_SUGGESTIONS"] = json.dumps(Settings.model_fields["suggestions"].default)
|
|
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
|
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
|
# The phase under test — pinned EXPLICITLY per app; the caps stay at the
|
|
# CODE defaults (this fixture is far under both).
|
|
env["BOR_ANSWER_TEXT"] = answer_text
|
|
env["BOR_SHOW_TEXT_MAX_CHARS"] = str(
|
|
Settings.model_fields["show_text_max_chars"].default
|
|
)
|
|
env["BOR_SHOW_TEXT_MAX_BLOCKS"] = str(
|
|
Settings.model_fields["show_text_max_blocks"].default
|
|
)
|
|
env["BOR_IMAGES"] = "false"
|
|
env["BOR_ANSWER_IMAGES"] = "false"
|
|
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
|
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
|
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
|
return env
|
|
|
|
|
|
def _serve(mock_llm: int, port: int, env: dict[str, str]) -> Iterator[str]:
|
|
proc = subprocess.Popen(
|
|
[
|
|
sys.executable,
|
|
"-m",
|
|
"uvicorn",
|
|
"app.main:app",
|
|
"--host",
|
|
"127.0.0.1",
|
|
"--port",
|
|
str(port),
|
|
"--log-level",
|
|
"warning",
|
|
],
|
|
cwd=REPO,
|
|
env=env,
|
|
)
|
|
try:
|
|
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
|
yield f"http://127.0.0.1:{port}"
|
|
finally:
|
|
proc.terminate()
|
|
try:
|
|
proc.wait(timeout=10)
|
|
except subprocess.TimeoutExpired:
|
|
proc.kill()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_server(
|
|
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
|
) -> Iterator[str]:
|
|
"""App A — ``BOR_ANSWER_TEXT=true`` (explicit: the mid-turn frame is ON)."""
|
|
scratch = tmp_path_factory.mktemp("bor_midturn_on")
|
|
port = int(os.environ.get("E2E_APP_PORT_MIDTURN_ON", "8209"))
|
|
yield from _serve(
|
|
mock_llm, port, _app_env(mock_llm, answer_text="true", scratch=scratch)
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_url(app_server: str) -> str:
|
|
return app_server
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def off_app_server(
|
|
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
|
) -> Iterator[str]:
|
|
"""App B — ``BOR_ANSWER_TEXT=false`` pinned explicitly (the kill switch;
|
|
only the wire test starts it, and it runs AFTER App A's)."""
|
|
scratch = tmp_path_factory.mktemp("bor_midturn_off")
|
|
port = int(os.environ.get("E2E_APP_PORT_MIDTURN_OFF", "8210"))
|
|
yield from _serve(
|
|
mock_llm, port, _app_env(mock_llm, answer_text="false", scratch=scratch)
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def off_app_url(off_app_server: str) -> str:
|
|
return off_app_server
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# KB seeding (straight into the shared e2e Postgres) + DB / mock helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _truncate_all() -> None:
|
|
"""The E2E isolation pattern: the suites share one Postgres, so the KB
|
|
tables reset per module. ``saved_chats`` is never truncated — the unique
|
|
question texts keep the row lookups unambiguous and
|
|
:func:`_cleanup_saved_rows` owns this suite's rows."""
|
|
with SessionLocal() as db:
|
|
db.execute(
|
|
text(
|
|
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
|
"git_sources, sources_meta, folder_summaries"
|
|
)
|
|
)
|
|
db.commit()
|
|
|
|
|
|
def _seed_kb() -> dict[str, Any]:
|
|
"""The one-document KB, embedded with the MOCK's own :func:`embed_text` —
|
|
the only way a question's tokens overlap the fixture so the turn grounds
|
|
instead of deflecting (a deflected turn offers no tools at all, so a
|
|
deflecting suite could never show a panel)."""
|
|
_truncate_all()
|
|
with SessionLocal() as db:
|
|
doc = Document(
|
|
source=MIDTURN_SOURCE,
|
|
path=DOC_PATH,
|
|
full_path=f"/tmp/{DOC_PATH}",
|
|
title=DOC_TITLE,
|
|
content=DOC_CONTENT,
|
|
content_hash=hashlib.sha256(DOC_CONTENT.encode()).hexdigest(),
|
|
summary=DOC_SUMMARY,
|
|
is_image=False,
|
|
indexed_at=datetime.now(UTC),
|
|
created_at=datetime(2026, 1, 15, tzinfo=UTC),
|
|
)
|
|
db.add(doc)
|
|
db.flush()
|
|
db.add(
|
|
Chunk(
|
|
document_id=doc.id,
|
|
position=0,
|
|
content=DOC_CONTENT,
|
|
embedding=embed_text(DOC_CONTENT),
|
|
)
|
|
)
|
|
db.commit()
|
|
doc_id = str(doc.id)
|
|
return {"doc_id": doc_id, "locator": f"{MIDTURN_SOURCE}/{DOC_PATH}"}
|
|
|
|
|
|
@pytest.fixture()
|
|
def ctx(browser: Browser) -> Iterator[BrowserContext]:
|
|
"""One browser context per test (the conftest ``page`` fixture builds an
|
|
implicit context per page, which would split a test's own session in two
|
|
— the restore arm needs the logged-in context to re-open the chat).
|
|
Viewport at the desktop reading width; the phone shape is phase 182's
|
|
subject, not this phase's."""
|
|
context = browser.new_context(viewport={"width": 1280, "height": 900})
|
|
yield context
|
|
context.close()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def seeded(app_url: str) -> Iterator[dict[str, Any]]:
|
|
"""App A's KB. Runs once for the module, after the health check."""
|
|
if not _health_db(app_url):
|
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
|
_cleanup_saved_rows()
|
|
yield _seed_kb()
|
|
_cleanup_saved_rows()
|
|
_truncate_all()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def seeded_off(off_app_url: str) -> Iterator[dict[str, Any]]:
|
|
"""App B's KB — the SAME document (the kill switch changes the app's
|
|
behaviour, never the corpus). Runs after App A's tests."""
|
|
if not _health_db(off_app_url):
|
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
|
_cleanup_saved_rows()
|
|
yield _seed_kb()
|
|
_cleanup_saved_rows()
|
|
_truncate_all()
|
|
|
|
|
|
def _health_db(url: str) -> bool:
|
|
try:
|
|
return httpx.get(f"{url}/api/health", timeout=5).json()["db"] == "up"
|
|
except Exception: # noqa: BLE001 — unreachable is the skip case
|
|
return False
|
|
|
|
|
|
def _mock_capture(mock_llm: int) -> list[dict[str, Any]]:
|
|
"""The mock's recent chat-completions request bodies (oldest first)."""
|
|
r = httpx.get(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured", timeout=5)
|
|
assert r.status_code == 200
|
|
return r.json()
|
|
|
|
|
|
def _mock_capture_reset(mock_llm: int) -> None:
|
|
r = httpx.post(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured/reset", timeout=5)
|
|
assert r.status_code == 200
|
|
|
|
|
|
def _tool_results(body: dict[str, Any]) -> list[str]:
|
|
return [
|
|
str(m.get("content") or "")
|
|
for m in body.get("messages", [])
|
|
if m.get("role") == "tool"
|
|
]
|
|
|
|
|
|
def _offered_tool_names(body: dict[str, Any]) -> list[str]:
|
|
return [t["function"]["name"] for t in (body.get("tools") or [])]
|
|
|
|
|
|
#: In-page witness + SSE tap (the house capture pattern, extended). The tap
|
|
#: tees the app's own ``fetch("/api/chat")`` body so the app's reader is
|
|
#: untouched; the ``MutationObserver`` snapshots the WORLD AT THE INSTANT a
|
|
#: verbatim panel is inserted into the thread — whether ``done`` had been
|
|
#: READ, whether the answer's sentinel token was on screen, whether the Send
|
|
#: pill still said Stop, and whether the node is inside the answer bubble.
|
|
#: That is the only way to pin "the panel landed mid-turn" without racing it.
|
|
_MIDTURN_JS = """
|
|
(() => {
|
|
const SENTINEL = "__SENTINEL__";
|
|
const SELECTOR = ".msg.brain .verbatim[data-bor-locator]";
|
|
window.__borSse = [];
|
|
window.__borPanels = [];
|
|
const orig = window.fetch.bind(window);
|
|
window.fetch = async (...args) => {
|
|
const res = await orig(...args);
|
|
const url = String((args[0] && args[0].url) || args[0] || "");
|
|
if (!url.includes("/api/chat") || !res.body) return res;
|
|
let keep, tap;
|
|
try { [keep, tap] = res.body.tee(); } catch { return res; }
|
|
const reader = tap.getReader();
|
|
const decoder = new TextDecoder();
|
|
(async () => {
|
|
let buf = "";
|
|
for (;;) {
|
|
const { done, value } = await reader.read();
|
|
if (done) break;
|
|
buf += decoder.decode(value, { stream: true });
|
|
let idx;
|
|
while ((idx = buf.indexOf("\\n\\n")) >= 0) {
|
|
const chunk = buf.slice(0, idx);
|
|
buf = buf.slice(idx + 2);
|
|
for (const line of chunk.split("\\n")) {
|
|
if (!line.startsWith("data:")) continue;
|
|
try { window.__borSse.push(JSON.parse(line.slice(5))); } catch {}
|
|
}
|
|
}
|
|
}
|
|
})();
|
|
return new Response(keep, {
|
|
status: res.status,
|
|
statusText: res.statusText,
|
|
headers: res.headers,
|
|
});
|
|
};
|
|
const witness = () => {
|
|
const thread = document.getElementById("messages");
|
|
if (!thread) return;
|
|
const threadText = thread.innerText || "";
|
|
const label = document.getElementById("send-label");
|
|
document.querySelectorAll(SELECTOR).forEach((el) => {
|
|
if (el.__borWitnessed) return;
|
|
el.__borWitnessed = true;
|
|
const locator = el.getAttribute("data-bor-locator") || "";
|
|
window.__borPanels.push({
|
|
locator,
|
|
pending: el.classList.contains("verbatim-pending"),
|
|
inBubble: !!el.closest(".bubble"),
|
|
doneSeen: window.__borSse.some((f) => f && f.type === "done"),
|
|
toolSeen: window.__borSse.some((f) => f && f.type === "tool"),
|
|
sentinelInDom: threadText.includes(SENTINEL),
|
|
streaming: !!label && label.textContent === "Stop",
|
|
forLocator: document.querySelectorAll(
|
|
SELECTOR + '[data-bor-locator="' + CSS.escape(locator) + '"]'
|
|
).length,
|
|
});
|
|
});
|
|
};
|
|
const start = () => {
|
|
const thread = document.getElementById("messages");
|
|
if (!thread) {
|
|
setTimeout(start, 20);
|
|
return;
|
|
}
|
|
new MutationObserver(witness).observe(thread, { childList: true, subtree: true });
|
|
witness();
|
|
};
|
|
start();
|
|
})();
|
|
"""
|
|
|
|
|
|
def _open(page: Page) -> None:
|
|
"""Installs the witness BEFORE the first navigation, then loads the app."""
|
|
page.add_init_script(_MIDTURN_JS.replace("__SENTINEL__", MIDTURN_SENTINEL))
|
|
|
|
|
|
def _sse_frames(page: Page) -> list[dict[str, Any]]:
|
|
return page.evaluate("() => window.__borSse || []")
|
|
|
|
|
|
def _witnesses(page: Page) -> list[dict[str, Any]]:
|
|
"""Every panel insertion of this page's turn, in the order they landed."""
|
|
return page.evaluate("() => window.__borPanels || []")
|
|
|
|
|
|
def _delta_text(frame: dict[str, Any]) -> str:
|
|
"""One ``delta`` frame's answer bytes (``ChatDeltaEvent.text`` — the
|
|
pre-phase key, unchanged by this phase)."""
|
|
return str(frame.get("text") or "")
|
|
|
|
|
|
def _ask(page: Page, question: str) -> None:
|
|
"""Send *question* WITHOUT waiting — the mid-turn assertions run while the
|
|
stream is live; :func:`_settle` closes the turn."""
|
|
page.fill("#message-input", question)
|
|
page.click("#send-btn")
|
|
|
|
|
|
def _settle(page: Page) -> None:
|
|
"""The house settle signal: the answer's marker text + the re-enabled pill."""
|
|
bubble = page.locator(".msg.brain .bubble").last
|
|
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=60_000)
|
|
expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000)
|
|
expect(page.locator("#send-label")).to_have_text("Send")
|
|
|
|
|
|
def _panel(page: Page, locator: str) -> Any:
|
|
"""Every verbatim node carrying one ``data-bor-locator`` (the SERVER's
|
|
canonical key — the attribute the client hydrated against)."""
|
|
return page.locator(f".msg.brain .verbatim[data-bor-locator='{locator}']")
|
|
|
|
|
|
def _resolved(page: Page, locator: str) -> Any:
|
|
"""The RESOLVED panels for one locator (a pending placeholder is a
|
|
different node: ``.verbatim-pending`` never carries the badge)."""
|
|
return page.locator(
|
|
f".msg.brain .verbatim:not(.verbatim-pending)[data-bor-locator='{locator}']"
|
|
)
|
|
|
|
|
|
def _appended(page: Page, locator: str) -> Any:
|
|
"""The panels hanging DIRECTLY off ``.msg-body`` — the answer-position
|
|
append (``appendUnlocatedVerbatim``). A fence-hydrated panel lives inside
|
|
``.bubble`` and therefore never matches this."""
|
|
return page.locator(
|
|
f".msg.brain > .msg-body > .verbatim[data-bor-locator='{locator}']"
|
|
)
|
|
|
|
|
|
def _inside_bubble(page: Page, locator: str) -> Any:
|
|
"""The RESOLVED panel for one locator INSIDE the answer (the fence
|
|
position) — a pending placeholder is a different state."""
|
|
return page.locator(
|
|
f".msg.brain .bubble "
|
|
f".verbatim:not(.verbatim-pending)[data-bor-locator='{locator}']"
|
|
)
|
|
|
|
|
|
def _pending(page: Page, locator: str) -> Any:
|
|
"""The fence's placeholder — the state between the streamed fence and the
|
|
frame (or ``done``) that resolves it."""
|
|
return page.locator(
|
|
f".msg.brain .verbatim-pending[data-bor-locator='{locator}']"
|
|
)
|
|
|
|
|
|
def _body_child_classes(page: Page) -> list[str]:
|
|
"""The LAST brain answer's ``.msg-body`` direct children's classes, in DOM
|
|
order (the answer-band position pin)."""
|
|
return page.evaluate(
|
|
"""() => {
|
|
const body = [...document.querySelectorAll(".msg.brain")].pop()
|
|
.querySelector(".msg-body");
|
|
return [...body.children].map((el) => el.className);
|
|
}"""
|
|
)
|
|
|
|
|
|
def _bubble_order(page: Page, tokens: dict[str, str]) -> dict[str, int]:
|
|
"""Where each token sits in the LAST bubble's rendered text (``-1`` when
|
|
absent) — the inline-position pin: the panel's own bytes between the
|
|
intro the model wrote before the call and the sentinel it wrote after."""
|
|
return page.evaluate(
|
|
"""(tokens) => {
|
|
const bubble = [...document.querySelectorAll(".msg.brain .bubble")].pop();
|
|
const text = bubble ? bubble.innerText : "";
|
|
const out = {};
|
|
for (const [key, token] of Object.entries(tokens)) {
|
|
out[key] = text.indexOf(token);
|
|
}
|
|
return out;
|
|
}""",
|
|
tokens,
|
|
)
|
|
|
|
|
|
def _assert_no_model_fence_bytes(page: Page) -> None:
|
|
"""LOCKED A7 on the rendered surface: the model's re-typed fence body and
|
|
the raw fence never painted."""
|
|
html = page.content()
|
|
assert MIDTURN_FENCE_BODY_MARKER not in html, (
|
|
"the model's re-typed fence body painted"
|
|
)
|
|
assert "```bor" not in html, "a raw ```bor locator fence painted"
|
|
|
|
|
|
def _grant_clipboard(page: Page) -> None:
|
|
page.context.grant_permissions(
|
|
["clipboard-read", "clipboard-write"], origin=page.url.rsplit("/", 1)[0]
|
|
)
|
|
|
|
|
|
def _clipboard(page: Page) -> str:
|
|
return page.evaluate("async () => await navigator.clipboard.readText()")
|
|
|
|
|
|
def _admin_cookies(page: Page) -> dict[str, str]:
|
|
return {
|
|
c["name"]: c["value"]
|
|
for c in page.context.cookies()
|
|
if "name" in c and "value" in c
|
|
}
|
|
|
|
|
|
def _auto_title(question: str) -> str:
|
|
return " ".join(question.split())[:120]
|
|
|
|
|
|
_SUITE_TITLES = tuple(
|
|
_auto_title(q)
|
|
for q in (
|
|
MIDTURN_QUESTION,
|
|
FENCE_BEFORE_QUESTION,
|
|
FENCE_AFTER_QUESTION,
|
|
TWO_RANGES_QUESTION,
|
|
OFF_QUESTION,
|
|
RESTORE_QUESTION,
|
|
)
|
|
)
|
|
|
|
|
|
def _cleanup_saved_rows() -> None:
|
|
"""Delete THIS suite's saved rows by title (the house pattern — a re-run
|
|
must not find the previous run's record)."""
|
|
try:
|
|
with SessionLocal() as db:
|
|
db.execute(
|
|
text("DELETE FROM saved_chats WHERE title = ANY(:titles)"),
|
|
{"titles": list(_SUITE_TITLES)},
|
|
)
|
|
db.commit()
|
|
except Exception: # noqa: BLE001 — DB unreachable: the skip handles it
|
|
pass
|
|
|
|
|
|
def _wait_saved_row(url: str, cookies: dict[str, str], title: str) -> dict[str, Any]:
|
|
"""Poll the admin list until the turn's auto-saved row appears (auto-saves
|
|
are SILENT — phase 55 A2 — so there is no status line to wait on)."""
|
|
deadline = time.monotonic() + 20
|
|
last: Any = None
|
|
while time.monotonic() < deadline:
|
|
r = httpx.get(f"{url}/api/chats", timeout=10, cookies=cookies)
|
|
assert r.status_code == 200
|
|
rows = r.json()["chats"]
|
|
last = next((c for c in rows if c["title"] == title), None)
|
|
if last is not None and last["message_count"] >= 2:
|
|
return last
|
|
time.sleep(0.2)
|
|
raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})")
|
|
|
|
|
|
def _brain_record(
|
|
url: str, cookies: dict[str, str], chat_id: str
|
|
) -> dict[str, Any]:
|
|
r = httpx.get(f"{url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
|
|
assert r.status_code == 200
|
|
brains = [m for m in r.json()["messages"] if m.get("who") == "brain"]
|
|
assert brains, "no brain record in the saved conversation"
|
|
return brains[-1]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# (a) The mid-turn arm — the TODO regression
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_midturn_panel_lands_before_the_answer_streams(
|
|
ctx: BrowserContext, app_url: str, seeded: dict[str, Any], mock_llm: int
|
|
) -> None:
|
|
"""The regression, end to end: the panel is ON SCREEN while the answer is
|
|
still streaming, BELOW the text the model had already written, and the
|
|
``done`` pass adds nothing.
|
|
|
|
The witness record is the load-bearing assertion — it is taken by an
|
|
in-page ``MutationObserver`` at the instant the panel node is inserted,
|
|
so it cannot be lapped by the stream: no ``done`` read, no sentinel
|
|
painted, the Send pill still "Stop".
|
|
|
|
A note on the position clause. The task phrases the mid-turn panel as
|
|
"below the intro text and ABOVE the final sentinel delta". Both halves
|
|
hold, but not as one sibling order: an UNFENCED block has no fence to
|
|
return to, so ``appendUnlocatedVerbatim`` parks it at the phase-182
|
|
answer band (below the answer text, above the source clusters) — and the
|
|
sentinel, which streams later INTO the bubble, is therefore below the
|
|
panel by the end. What is true at the instant the panel lands is the
|
|
stronger claim the witness pins: the sentinel did not exist yet, so the
|
|
panel preceded it in TIME. The literal between-the-two position belongs
|
|
to the fenced arms (tests (b) and (c)), where the panel does sit between
|
|
the intro and the sentinel inside the bubble.
|
|
"""
|
|
locator = _locator(PRIMARY_START, PRIMARY_END)
|
|
page = ctx.new_page()
|
|
_open(page)
|
|
login(page, app_url, next="/")
|
|
expect(page.locator("#kb-banner")).to_be_hidden()
|
|
|
|
_mock_capture_reset(mock_llm)
|
|
_ask(page, MIDTURN_QUESTION)
|
|
|
|
# --- the panel exists while the stream is still running --------------
|
|
panel = _resolved(page, locator)
|
|
expect(panel).to_have_count(1, timeout=30_000)
|
|
expect(page.locator("#send-label")).to_have_text("Stop")
|
|
|
|
witness = _witnesses(page)
|
|
# One insertion, and NOTHING else: no fence streamed, so no placeholder
|
|
# ever existed to settle (the append path is the only pass that ran).
|
|
assert len(witness) == 1, witness
|
|
landed = witness[0]
|
|
assert landed["locator"] == locator
|
|
assert landed["pending"] is False, "the panel painted as a placeholder"
|
|
assert landed["doneSeen"] is False, "the panel only landed after `done`"
|
|
assert landed["sentinelInDom"] is False, (
|
|
"the answer's sentinel was already on screen when the panel landed"
|
|
)
|
|
assert landed["streaming"] is True, "the turn was not in flight"
|
|
assert landed["toolSeen"] is True, "the panel landed before its own tool call"
|
|
assert landed["forLocator"] == 1, "a doubled panel mid-turn"
|
|
|
|
# Below the intro the model wrote BEFORE the call, and (mid-turn) the
|
|
# answer band the phase-182 append owns: nothing but the bubble above it.
|
|
bubble = page.locator(".msg.brain .bubble").last
|
|
assert MIDTURN_INTRO in bubble.inner_text()
|
|
assert MIDTURN_SENTINEL not in bubble.inner_text()
|
|
# Mid-turn the panel is the LAST thing in the answer band, right after
|
|
# the bubble (the reasoning/tool rows the same turn created sit above it).
|
|
classes = _body_child_classes(page)
|
|
assert classes[-1] == "verbatim", classes
|
|
assert classes[-2] == "bubble", classes
|
|
bubble_box = bubble.bounding_box()
|
|
panel_box = panel.bounding_box()
|
|
assert bubble_box and panel_box and panel_box["y"] > bubble_box["y"], (
|
|
"the panel did not land below the text already on screen"
|
|
)
|
|
|
|
# --- the stream CONTINUES: the sentinel lands after the panel --------
|
|
expect(bubble).to_contain_text(MIDTURN_SENTINEL, timeout=60_000)
|
|
_settle(page)
|
|
|
|
# The done pass added NOTHING: one panel, still no placeholder.
|
|
expect(_panel(page, locator)).to_have_count(1)
|
|
expect(panel).to_have_count(1)
|
|
expect(page.locator(".verbatim-pending")).to_have_count(0)
|
|
_assert_no_model_fence_bytes(page)
|
|
|
|
# --- the panel IS the file: the SERVER's caption, gutter and body ----
|
|
expect(panel).to_have_attribute("role", "group")
|
|
expect(panel.locator(".verbatim-caption")).to_have_text(
|
|
_caption(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
expect(panel.locator(".verbatim-badge")).to_have_text("VERBATIM")
|
|
text_el = panel.locator(".verbatim-text")
|
|
expect(text_el).to_have_text(_block_text(PRIMARY_START, PRIMARY_END))
|
|
assert text_el.text_content() == _block_text(PRIMARY_START, PRIMARY_END)
|
|
gutter = panel.locator(".verbatim-lines li")
|
|
expect(gutter).to_have_count(PRIMARY_END - PRIMARY_START + 1)
|
|
expect(gutter.first).to_have_text(str(PRIMARY_START))
|
|
expect(gutter.last).to_have_text(str(PRIMARY_END))
|
|
expect(panel.locator(".verbatim-copy")).to_have_attribute(
|
|
"aria-label",
|
|
f"Copy {PRIMARY_END - PRIMARY_START + 1} lines from "
|
|
f"{MIDTURN_SOURCE}/{DOC_PATH}",
|
|
)
|
|
# The clipboard truth survives the mid-turn builder: the panel Copy puts
|
|
# the SERVER's string on the clipboard exactly (the phase-182 contract).
|
|
_grant_clipboard(page)
|
|
panel.locator(".verbatim-copy").click()
|
|
assert _clipboard(page) == _block_text(PRIMARY_START, PRIMARY_END)
|
|
# The block is the range, not the file: the last line's neighbour and
|
|
# the front matter are both absent from the panel.
|
|
panel_text = text_el.text_content() or ""
|
|
assert "signed by the shift lead" not in panel_text
|
|
assert "title: valve-panel runbook" not in panel_text
|
|
|
|
# --- the answer band after the terminal frame ------------------------
|
|
# The panel keeps the phase-182 answer band: straight after the answer
|
|
# text, ahead of the .msg-meta family (the chips row, the actions row).
|
|
classes = _body_child_classes(page)
|
|
assert classes.index("verbatim") == classes.index("bubble") + 1, classes
|
|
meta = next(
|
|
(i for i, c in enumerate(classes) if c.split(" ")[0] == "msg-meta"), None
|
|
)
|
|
assert meta is not None and meta > classes.index("verbatim"), classes
|
|
|
|
# --- the wire: the frame order and the untouched answer bytes --------
|
|
frames = _sse_frames(page)
|
|
types = [f["type"] for f in frames]
|
|
assert "verbatim_block" in types
|
|
assert set(types) <= PRE_PHASE_EVENT_TYPES | {"verbatim_block"}
|
|
# delta* → tool → verbatim_block (AFTER the tool line, BEFORE the answer)
|
|
tool_at = types.index("tool")
|
|
assert types.count("tool") == 1
|
|
assert types.count("verbatim_block") == 1
|
|
# The frame ORDER is the contract: the mid-turn copy sits AFTER that
|
|
# round's tool frame and BEFORE the next model round's first delta.
|
|
assert set(types[:tool_at]) == {"delta"}, types
|
|
assert types[tool_at + 1] == "verbatim_block", types
|
|
assert types[-1] == "done", types
|
|
assert set(types[tool_at + 2 : -1]) == {"delta"}, types
|
|
# …and the answer's BYTES are the pre-phase bytes, segment by segment.
|
|
assert "".join(_delta_text(f) for f in frames[:tool_at]) == MIDTURN_INTRO, (
|
|
"the mid-round intro changed"
|
|
)
|
|
assert "".join(_delta_text(f) for f in frames[tool_at + 2 : -1]) == (
|
|
MIDTURN_PLAIN_ANSWER
|
|
), "the answer's delta payload changed with the frame"
|
|
assert "".join(_delta_text(f) for f in frames if f["type"] == "delta") == (
|
|
WIRE_ANSWER_BYTES
|
|
)
|
|
done = frames[-1]
|
|
assert done["type"] == "done"
|
|
assert [b["locator"] for b in done["answer_text"]] == [
|
|
locator
|
|
], "the mid-turn frame left the done payload alone"
|
|
|
|
captured = _mock_capture(mock_llm)
|
|
assert len(captured) == 2, f"expected 2 model calls, got {len(captured)}"
|
|
assert any(
|
|
"Unknown tool" in r for r in _tool_results(captured[1])
|
|
) is False, "the show_text call was refused on the feature-on app"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# (b) The fence in place — the mid-turn hydrateVerbatimNow pass
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_midturn_fence_hydrates_in_place_not_at_the_answer_tail(
|
|
ctx: BrowserContext, app_url: str, seeded: dict[str, Any], mock_llm: int
|
|
) -> None:
|
|
"""The fence is on screen BEFORE the call (the model quotes, then asks), so
|
|
the placeholder exists when the frame lands and the matched-only pass
|
|
resolves it IN PLACE: inside ``.bubble`` at the fence position, never
|
|
appended at the answer tail, and never twice.
|
|
|
|
This is also the phase-182 invariant surviving the mid-turn pass: the
|
|
panel's body is the SERVER's text and none of the model's re-typed fence
|
|
body paints.
|
|
|
|
Honest note on the middle of the turn: every later ``delta`` re-renders
|
|
the bubble from the accumulated answer text, and the accumulated text
|
|
still contains the fence — so between this hydration and the terminal
|
|
frame the node is back to ``.verbatim-pending`` and ``done`` settles it
|
|
once more. Both ends are asserted (the witness for mid-turn, the DOM for
|
|
the end state) and the count never leaves 1; the interim is the
|
|
phase-182 placeholder state, never the model's bytes.
|
|
"""
|
|
locator = _locator(PRIMARY_START, PRIMARY_END)
|
|
page = ctx.new_page()
|
|
_open(page)
|
|
login(page, app_url, next="/")
|
|
|
|
_mock_capture_reset(mock_llm)
|
|
_ask(page, FENCE_BEFORE_QUESTION)
|
|
|
|
# 1) the fence streams as a PLACEHOLDER inside the bubble (the model
|
|
# quoted before it called, so the node exists BEFORE the frame) …
|
|
expect(_pending(page, locator)).to_have_count(1, timeout=30_000)
|
|
# 2) … and the frame resolves it IN PLACE — same node position, inside
|
|
# the answer, while the turn is still streaming.
|
|
inside = _inside_bubble(page, locator)
|
|
expect(inside).to_have_count(1, timeout=30_000)
|
|
expect(page.locator("#send-label")).to_have_text("Stop")
|
|
|
|
witness = _witnesses(page)
|
|
pending = [w for w in witness if w["pending"]]
|
|
resolved = [w for w in witness if not w["pending"]]
|
|
assert pending, "the fence never painted a placeholder first"
|
|
assert pending[0]["inBubble"] is True
|
|
assert resolved, "no resolved panel was witnessed"
|
|
landed = resolved[0]
|
|
assert landed["locator"] == locator
|
|
assert landed["inBubble"] is True, "the fence hydrated OUTSIDE the answer"
|
|
assert landed["doneSeen"] is False, "the fence only hydrated after `done`"
|
|
assert landed["sentinelInDom"] is False, "the answer had already streamed"
|
|
assert landed["streaming"] is True
|
|
assert landed["forLocator"] == 1, "the mid-turn pass built a second panel"
|
|
|
|
# Nothing was appended at the answer position: the placeholder resolved
|
|
# where it stood, so the block never got a second home at the tail.
|
|
expect(_appended(page, locator)).to_have_count(0)
|
|
|
|
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
|
MIDTURN_SENTINEL, timeout=60_000
|
|
)
|
|
_settle(page)
|
|
|
|
# --- after `done`: ONE panel, still at the fence ----------------------
|
|
expect(_panel(page, locator)).to_have_count(1)
|
|
expect(_inside_bubble(page, locator)).to_have_count(1)
|
|
expect(_appended(page, locator)).to_have_count(0)
|
|
expect(page.locator(".verbatim-pending")).to_have_count(0)
|
|
_assert_no_model_fence_bytes(page)
|
|
|
|
# The inline position: intro → panel → sentinel, inside the bubble.
|
|
order = _bubble_order(
|
|
page,
|
|
{
|
|
"intro": "MIDTURN_INTRO_4412",
|
|
"panel": "VALVE A feeds the west header",
|
|
"sentinel": MIDTURN_SENTINEL,
|
|
},
|
|
)
|
|
assert -1 not in order.values(), order
|
|
assert order["intro"] < order["panel"] < order["sentinel"], order
|
|
|
|
panel = _resolved(page, locator)
|
|
expect(panel.locator(".verbatim-caption")).to_have_text(
|
|
_caption(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
assert (
|
|
panel.locator(".verbatim-text").text_content()
|
|
== _block_text(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
# The answer band carries no appended copy of the block at all — the
|
|
# panel has exactly one home, the fence (the turn's reasoning disclosure
|
|
# and action rows are the band's other nodes).
|
|
band = [c.split(" ")[0] for c in _body_child_classes(page)]
|
|
assert "verbatim" not in band, band
|
|
assert band.count("bubble") == 1, band
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# (c) No double panel — the mid-turn append + the same-locator fence
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_midturn_append_and_a_later_fence_leave_one_panel(
|
|
ctx: BrowserContext, app_url: str, seeded: dict[str, Any], mock_llm: int
|
|
) -> None:
|
|
"""The realistic ordering: the model calls, keeps talking, and quotes the
|
|
SAME locator in its answer. The block lands mid-turn at the answer
|
|
position; the fence streams as a placeholder; ``done`` re-homes the panel
|
|
onto the fence and DROPS the appended twin (task 03's
|
|
``dropPlacedTwin``, the per-locator-not-per-identity dedupe) — exactly one
|
|
panel afterwards, at the fence position.
|
|
"""
|
|
locator = _locator(PRIMARY_START, PRIMARY_END)
|
|
page = ctx.new_page()
|
|
_open(page)
|
|
login(page, app_url, next="/")
|
|
|
|
_mock_capture_reset(mock_llm)
|
|
_ask(page, FENCE_AFTER_QUESTION)
|
|
|
|
# Mid-turn: the block is already on screen, appended below the answer
|
|
# text, while the answer (and its fence) has not streamed a token.
|
|
appended = _appended(page, locator)
|
|
expect(appended).to_have_count(1, timeout=30_000)
|
|
expect(page.locator("#send-label")).to_have_text("Stop")
|
|
landed = [w for w in _witnesses(page) if not w["pending"]][0]
|
|
assert landed["inBubble"] is False, "the unfenced block hydrated somewhere odd"
|
|
assert landed["doneSeen"] is False
|
|
assert landed["sentinelInDom"] is False
|
|
assert landed["streaming"] is True
|
|
assert landed["forLocator"] == 1
|
|
assert MIDTURN_SENTINEL not in page.locator(".msg.brain .bubble").last.inner_text()
|
|
|
|
# --- after `done`: ONE panel, at the fence, the twin gone -------------
|
|
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
|
MIDTURN_SENTINEL, timeout=60_000
|
|
)
|
|
_settle(page)
|
|
expect(_panel(page, locator)).to_have_count(1)
|
|
expect(_inside_bubble(page, locator)).to_have_count(1)
|
|
expect(_appended(page, locator)).to_have_count(0)
|
|
expect(page.locator(".verbatim-pending")).to_have_count(0)
|
|
_assert_no_model_fence_bytes(page)
|
|
|
|
order = _bubble_order(
|
|
page,
|
|
{
|
|
"intro": "MIDTURN_INTRO_4412",
|
|
"panel": "VALVE A feeds the west header",
|
|
"sentinel": MIDTURN_SENTINEL,
|
|
},
|
|
)
|
|
assert -1 not in order.values(), order
|
|
assert order["intro"] < order["panel"] < order["sentinel"], order
|
|
|
|
# The LAST witnessed node is the re-homed panel — inside the bubble.
|
|
assert [w for w in _witnesses(page) if not w["pending"]][-1]["inBubble"] is True
|
|
assert (
|
|
_resolved(page, locator).locator(".verbatim-text").text_content()
|
|
== _block_text(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# (d) Two ranges of one file — two panels, in call order
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_two_ranges_of_one_file_land_two_panels(
|
|
ctx: BrowserContext, app_url: str, seeded: dict[str, Any], mock_llm: int
|
|
) -> None:
|
|
"""``shown_texts`` dedupes per LOCATOR, not per document: two ranges of one
|
|
file are two blocks, two mid-turn frames and two panels at the answer
|
|
position, in call order, each captioning its own SERVER extent.
|
|
"""
|
|
first = _locator(FIRST_START, FIRST_END)
|
|
second = _locator(SECOND_START, SECOND_END)
|
|
page = ctx.new_page()
|
|
_open(page)
|
|
login(page, app_url, next="/")
|
|
|
|
_mock_capture_reset(mock_llm)
|
|
_ask(page, TWO_RANGES_QUESTION)
|
|
|
|
expect(_appended(page, first)).to_have_count(1, timeout=30_000)
|
|
expect(_appended(page, second)).to_have_count(1, timeout=30_000)
|
|
expect(page.locator("#send-label")).to_have_text("Stop")
|
|
mid = _witnesses(page)
|
|
assert [w for w in mid if not w["pending"]], "no panel landed mid-turn"
|
|
assert all(w["sentinelInDom"] is False for w in mid if not w["pending"]), (
|
|
"a panel landed only after the answer had streamed"
|
|
)
|
|
|
|
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
|
MIDTURN_SENTINEL, timeout=60_000
|
|
)
|
|
_settle(page)
|
|
|
|
# Exactly two panels — the second never collapsed onto the first.
|
|
assert [
|
|
w["locator"] for w in _witnesses(page) if not w["pending"]
|
|
] == [first, second]
|
|
expect(page.locator(".msg.brain .verbatim-pending")).to_have_count(0)
|
|
_assert_no_model_fence_bytes(page)
|
|
|
|
panels = page.locator(".msg.brain .verbatim[data-bor-locator]")
|
|
expect(panels).to_have_count(2)
|
|
expect(panels.nth(0)).to_have_attribute("data-bor-locator", first)
|
|
expect(panels.nth(1)).to_have_attribute("data-bor-locator", second)
|
|
expect(panels.nth(0).locator(".verbatim-caption")).to_have_text(
|
|
_caption(FIRST_START, FIRST_END)
|
|
)
|
|
expect(panels.nth(1).locator(".verbatim-caption")).to_have_text(
|
|
_caption(SECOND_START, SECOND_END)
|
|
)
|
|
assert (
|
|
panels.nth(0).locator(".verbatim-text").text_content()
|
|
== _block_text(FIRST_START, FIRST_END)
|
|
)
|
|
assert (
|
|
panels.nth(1).locator(".verbatim-text").text_content()
|
|
== _block_text(SECOND_START, SECOND_END)
|
|
)
|
|
# Two panels, two gutter starts: the second names its OWN first line.
|
|
expect(panels.nth(0).locator(".verbatim-lines li").first).to_have_text(
|
|
str(FIRST_START)
|
|
)
|
|
expect(panels.nth(1).locator(".verbatim-lines li").first).to_have_text(
|
|
str(SECOND_START)
|
|
)
|
|
|
|
frames = _sse_frames(page)
|
|
types = [f["type"] for f in frames]
|
|
assert types.count("verbatim_block") == 2, types
|
|
first_block = types.index("verbatim_block")
|
|
second_block = len(types) - 1 - types[::-1].index("verbatim_block")
|
|
assert first_block < second_block
|
|
# Each copy rides its OWN round: the tool frames bracket them in call
|
|
# order (tool → block(1-10) → tool → block(11-20) → the answer → done).
|
|
assert [types[i] for i in range(first_block - 1, second_block + 2)] == [
|
|
"tool",
|
|
"verbatim_block",
|
|
"tool",
|
|
"verbatim_block",
|
|
"delta",
|
|
], types
|
|
done = frames[-1]
|
|
assert [b["locator"] for b in done["answer_text"]] == [first, second]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# (e) The kill switch — the pre-phase wire, byte for byte
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_kill_switch_off_ships_the_pre_phase_wire(
|
|
ctx: BrowserContext, off_app_url: str, seeded_off: dict[str, Any], mock_llm: int
|
|
) -> None:
|
|
"""``BOR_ANSWER_TEXT=0``: the tool is not offered, so the same scripted
|
|
attempt hits the phase-182 refusal and NOTHING new reaches the wire.
|
|
|
|
The documented pre-phase baseline for this turn shape (a mid-round tool
|
|
call, phase 161's flush ordering):
|
|
|
|
delta(intro)* → tool → delta(answer)* → done
|
|
|
|
i.e. exactly the phase-192-minus-one frame sequence, and the ``delta``
|
|
payload is the SAME bytes App A streams (``WIRE_ANSWER_BYTES``) — adding
|
|
the mid-turn frame changed neither the answer nor its framing (LOCKED A4).
|
|
No ``verbatim_block``, no ``answer_text`` on ``done``, no panel and no
|
|
placeholder in the DOM.
|
|
"""
|
|
page = ctx.new_page()
|
|
_open(page)
|
|
login(page, off_app_url, next="/")
|
|
|
|
_mock_capture_reset(mock_llm)
|
|
_ask(page, OFF_QUESTION)
|
|
_settle(page)
|
|
|
|
frames = _sse_frames(page)
|
|
types = [f["type"] for f in frames]
|
|
assert "verbatim_block" not in types
|
|
assert set(types) <= PRE_PHASE_EVENT_TYPES, types
|
|
assert types[0] == "delta", types
|
|
assert types[-1] == "done", types
|
|
tool_at = types.index("tool")
|
|
assert types.count("tool") == 1
|
|
assert frames[tool_at] == {
|
|
"type": "tool",
|
|
"name": "show_text",
|
|
"argument": MIDTURN_COMBINED,
|
|
}, "the attempt's tool frame drifted from the pre-phase shape"
|
|
# Nothing new slipped in around the tool line: the stream is
|
|
# delta* → tool → delta* → done with ONE interruption-free tail.
|
|
assert set(types[:tool_at]) == {"delta"}
|
|
assert set(types[tool_at + 1 : -1]) == {"delta"}
|
|
assert "".join(_delta_text(f) for f in frames if f["type"] == "delta") == (
|
|
WIRE_ANSWER_BYTES
|
|
), "the answer bytes are not the feature-on answer bytes"
|
|
done = frames[-1]
|
|
assert "answer_text" not in done, "the kill switch shipped a verbatim payload"
|
|
|
|
# And nothing on screen: no panel, no placeholder, no witness record.
|
|
expect(page.locator(".msg.brain .verbatim")).to_have_count(0)
|
|
expect(page.locator(".verbatim-pending")).to_have_count(0)
|
|
assert _witnesses(page) == []
|
|
|
|
# The attempt really did reach the server-side refusal (the tool is not
|
|
# offered while the flag is off — phase 182's defensive branch).
|
|
captured = _mock_capture(mock_llm)
|
|
assert len(captured) == 2, f"expected 2 model calls, got {len(captured)}"
|
|
assert "show_text" not in _offered_tool_names(captured[0])
|
|
assert any(
|
|
"Unknown tool." in r for r in _tool_results(captured[1])
|
|
), "the show_text attempt never hit the phase-182 refusal"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# (f) The restore — one panel per locator, from the record
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_restore_repaints_one_panel_for_the_midturn_locator(
|
|
ctx: BrowserContext, app_url: str, seeded: dict[str, Any], mock_llm: int
|
|
) -> None:
|
|
"""Arm (a)'s turn, re-rendered from the SAVED record: the auto-save carried
|
|
the block in ``answer_text`` (the shape phase 182 defined — the mid-turn
|
|
frame added nothing to the record), and the restore pass paints exactly ONE
|
|
panel for the locator with the SERVER's caption and bytes, no placeholder
|
|
and none of the model's prose retyped.
|
|
"""
|
|
locator = _locator(PRIMARY_START, PRIMARY_END)
|
|
page = ctx.new_page()
|
|
_open(page)
|
|
login(page, app_url, next="/")
|
|
|
|
_mock_capture_reset(mock_llm)
|
|
_ask(page, RESTORE_QUESTION)
|
|
expect(_resolved(page, locator)).to_have_count(1, timeout=30_000)
|
|
_settle(page)
|
|
|
|
title = _auto_title(RESTORE_QUESTION)
|
|
row = _wait_saved_row(app_url, _admin_cookies(page), title)
|
|
record = _brain_record(app_url, _admin_cookies(page), row["id"])
|
|
assert [b["locator"] for b in record["answer_text"]] == [
|
|
locator
|
|
], "the saved record changed shape with the mid-turn frame"
|
|
assert record["answer_text"][0]["text"] == _block_text(
|
|
PRIMARY_START, PRIMARY_END
|
|
)
|
|
assert MIDTURN_SENTINEL in record["text"]
|
|
|
|
# --- (1) the literal reload: the bor.chat.v1 pass paints the panel ----
|
|
_mock_capture_reset(mock_llm)
|
|
page.reload(wait_until="networkidle")
|
|
expect(page.locator(".msg.brain")).to_have_count(1)
|
|
expect(_panel(page, locator)).to_have_count(1)
|
|
expect(page.locator(".verbatim-pending")).to_have_count(0)
|
|
restored = _resolved(page, locator)
|
|
expect(restored.locator(".verbatim-caption")).to_have_text(
|
|
_caption(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
assert (
|
|
restored.locator(".verbatim-text").text_content()
|
|
== _block_text(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
expect(restored.locator(".verbatim-badge")).to_have_text("VERBATIM")
|
|
expect(restored.locator(".verbatim-lines li").first).to_have_text(
|
|
str(PRIMARY_START)
|
|
)
|
|
_assert_no_model_fence_bytes(page)
|
|
|
|
# --- (2) the SERVER's record: the local store cleared first, so the pass
|
|
# cannot be vacuous (the phase-182 convention) ----------------------
|
|
page.evaluate("() => window.localStorage.clear()")
|
|
page.goto(f"{app_url}/?chat={row['id']}", wait_until="networkidle")
|
|
expect(page.locator(".msg.brain")).to_have_count(1)
|
|
expect(_panel(page, locator)).to_have_count(1)
|
|
expect(page.locator(".verbatim-pending")).to_have_count(0)
|
|
assert (
|
|
_resolved(page, locator).locator(".verbatim-text").text_content()
|
|
== _block_text(PRIMARY_START, PRIMARY_END)
|
|
)
|
|
_assert_no_model_fence_bytes(page)
|
|
# Opening a saved chat never re-runs the turn.
|
|
assert all(
|
|
MIDTURN_TRIGGER not in json.dumps(body).lower()
|
|
for body in _mock_capture(mock_llm)
|
|
), "the restore ran a model turn"
|