ducoterra
2ac3fc89c2
Phase 117 (owner request, live-mockup-confirmed): the phase-109 3-dot cue becomes a compact ECG trace (49px, P/QRS/T) with a brand sweep traveling the path (bwdraw, 42/140 dash segment, 0.9s loop), the loader repositioned left of the button so its appearance never shifts it. setUiState stays the sole owner of the loader's hidden attribute; the reduced-motion variant stills the sweep. Unit + lifecycle-E2E pins updated for the new contract.
331 lines
15 KiB
Python
331 lines
15 KiB
Python
"""Phase 109 E2E (Playwright): the never-frozen turn — the reported
|
||
repro (delta → tool → thinking-after-delta) replayed deterministically.
|
||
|
||
Story: n/a — owner request 2026-09-16 (TODO.md L3): "Thinking can
|
||
happen after the model starts responding. This sometimes results in a
|
||
the chat appearing 'frozen' because the model responds, calls a tool,
|
||
then continues thinking without re-expanding the thinking block. There
|
||
should be a visual that the chat is still progressing regardless of
|
||
what state it's in (some kind of loader will do)."
|
||
|
||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||
|
||
uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov
|
||
|
||
Marker contract: the mock's ``answer first, then list, then think``
|
||
marker (``TURN_PROGRESS_TRIGGER`` — see the ``mock_llm.py`` module
|
||
docstring) drives the deterministic reported repro. Model call 1
|
||
streams a short content delta (3 chunks, NO reasoning) after a ~2 s
|
||
pre-delay (model latency — the loader's start-state window), then the
|
||
no-arg ``ls`` tool call; model call 2 (after the server ran the ``ls``)
|
||
streams a ~7 s frameless gap (past the phase-87 tool-line ``(Ns)``
|
||
counter's 5 s gate), then a ``reasoning_content`` stream (10 × 0.3 s),
|
||
a content delta (3 chunks ending in the distinctive final sentence
|
||
carrying ``marker-progress-42``), and a FINAL ``reasoning_content``
|
||
chunk group (3 × 0.3 s). The server is position-independent over the
|
||
wire (``app/rag/llm.py`` — each ``reasoning_content`` chunk → a
|
||
``thinking`` SSE frame, each ``content`` chunk → a ``delta`` frame;
|
||
the tool call materializes after its request's stream), so the turn's
|
||
SSE is exactly ``delta → tool → thinking → delta → thinking → done`` —
|
||
the owner's repro, and every window is deterministic (the mock's
|
||
baked delays, the ``slow_llm.py`` pacing precedent).
|
||
|
||
Each test sends the marker question in its OWN fresh conversation
|
||
(one full turn per test — the ``page`` fixture is a fresh browser
|
||
context, ``_reset_db`` reseeds the KB deterministically):
|
||
|
||
* ``test_loader_visible_from_send_through_the_tool_gap`` — the loader
|
||
is visible right after the send (the thinking state, no frame yet —
|
||
the start-state window), is STILL visible when the ``.tool-call``
|
||
line lands (the turn is provably in flight), the tool line carries
|
||
the phase-87 elapsed counter in the tool gap, and ``#send-status``
|
||
carries a state text (not empty) throughout.
|
||
* ``test_thinking_block_reopens_after_delta_with_visible_loader`` —
|
||
THE reported symptom's state: the thinking block re-opens after the
|
||
answer has started (open + non-empty ``.thinking-text`` while the
|
||
bubble already carries call 1's content) with the loader STILL
|
||
visible (the frozen window is gone); the terminal state is clean
|
||
(loader hidden, the scratchpad carries both thinking rounds, the
|
||
bubble carries BOTH answers, the send button reads "Send" not
|
||
"Stop", ``#send-status`` back to the idle shape).
|
||
* ``test_loader_a11y_and_reduced_motion`` — after a full turn: the
|
||
loader element is ``aria-hidden="true"`` in the DOM and
|
||
``#send-status`` (the sole ``aria-live`` announcer) is back to the
|
||
``SEND_STATUS`` idle shape (empty — not stuck on a mid-turn label);
|
||
then, in a context with ``reducedMotion: "reduce"`` (the Playwright
|
||
context option), a second turn shows the loader STILL visible
|
||
mid-turn — the reduced-motion variant stills the sweep (the phase-
|
||
117 brain wave: computed ``animation-name: none`` on the ``.bw-pulse``
|
||
path), it does not HIDE the cue: visibility is
|
||
owned by the JS ``hidden`` attribute in ``setUiState`` (D16), not by
|
||
CSS.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import asyncio
|
||
import re
|
||
from pathlib import Path
|
||
from threading import Thread
|
||
from typing import Any
|
||
|
||
from playwright.sync_api import Browser, Page, expect
|
||
from sqlalchemy import text
|
||
|
||
from app.config import Settings
|
||
from app.db import SessionLocal
|
||
from app.rag.importer import ImportSummary, import_sources
|
||
from app.rag.llm import LLMClient
|
||
from e2e.auth_helpers import login
|
||
|
||
REPO = Path(__file__).resolve().parents[2]
|
||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||
|
||
#: The marker question — the mock's ``TURN_PROGRESS_TRIGGER`` phrase +
|
||
#: an on-topic tail (HIGH gate → the ``<tools>`` section, so the
|
||
#: scripted ``ls`` round is reachable — the phase-37/70 convention).
|
||
QUESTION = "answer first, then list, then think — how is my Kubernetes cluster set up?"
|
||
|
||
#: The mock's byte-stable sentinels (``mock_llm.py``): call 1's
|
||
#: content, call 2's final sentence, call 2's scratchpad, call 2's
|
||
#: FINAL scratchpad (the thinking after the answer's last delta).
|
||
SENTINEL_FIRST = "marker-progress-41"
|
||
SENTINEL_FINAL = "marker-progress-42"
|
||
SENTINEL_THOUGHT = "marker-thought-42"
|
||
SENTINEL_FINAL_THOUGHT = "marker-final-thought-42"
|
||
|
||
|
||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
|
||
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||
return await import_sources([FIXTURES], LLMClient(settings))
|
||
|
||
|
||
def _run_in_thread(coro: Any) -> Any:
|
||
"""Run a coroutine on a worker thread.
|
||
|
||
Playwright's sync API keeps an asyncio loop running on the test
|
||
thread, so ``asyncio.run`` cannot be called directly from a test
|
||
body.
|
||
"""
|
||
box: dict[str, Any] = {}
|
||
|
||
def runner() -> None:
|
||
try:
|
||
box["value"] = asyncio.run(coro)
|
||
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||
box["error"] = e
|
||
|
||
t = Thread(target=runner)
|
||
t.start()
|
||
t.join()
|
||
if "error" in box:
|
||
raise box["error"]
|
||
return box["value"]
|
||
|
||
|
||
def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||
"""Truncate the KB (+ query log + steering notes — deterministic
|
||
mock answers), then optionally re-import fixtures."""
|
||
with SessionLocal() as db:
|
||
db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes"))
|
||
db.commit()
|
||
if not seed:
|
||
return None
|
||
return _run_in_thread(_import_fixtures(mock_port))
|
||
|
||
|
||
def _send(page: Page) -> None:
|
||
"""Send the marker question (one full reported-repro turn)."""
|
||
page.fill("#message-input", QUESTION)
|
||
page.click("#send-btn")
|
||
|
||
|
||
def _answer_bubble(page: Page) -> Any:
|
||
"""The answer bubble (the typing indicator's bubble is ``.typing``
|
||
— scoped out)."""
|
||
return page.locator(".msg.brain .bubble:not(.typing)")
|
||
|
||
|
||
def test_loader_visible_from_send_through_the_tool_gap(
|
||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||
) -> None:
|
||
summary = _reset_db(mock_llm, seed=True)
|
||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||
page.set_default_timeout(30_000)
|
||
login(page, app_url, next="/")
|
||
|
||
_send(page)
|
||
|
||
# Right after the send — inside call 1's ~2 s pre-delay window:
|
||
# the turn is in flight (the thinking state, NO frame has arrived
|
||
# yet) and the loader is ALREADY visible — the start-state cue.
|
||
# ``setUiState(thinking)`` runs synchronously in the submit path,
|
||
# so this is race-free: the button is the Stop control, the
|
||
# loader is out, and no frame-derived surface exists yet (no
|
||
# answer bubble, no thinking block, no tool line — only the
|
||
# pre-delta typing bubble).
|
||
loader = page.locator("#turn-loader")
|
||
expect(loader).to_be_visible()
|
||
expect(page.locator("#send-label")).to_have_text("Stop")
|
||
expect(page.locator("#typing-indicator")).to_be_visible()
|
||
expect(_answer_bubble(page)).to_have_count(0)
|
||
expect(page.locator(".msg.brain .thinking")).to_have_count(0)
|
||
expect(page.locator(".tool-call")).to_have_count(0)
|
||
|
||
# The ``.tool-call`` line lands (call 1's content + the ls): the
|
||
# turn is provably in flight — and the loader is STILL visible.
|
||
expect(page.locator(".tool-call")).to_be_visible(timeout=30_000)
|
||
expect(loader).to_be_visible()
|
||
expect(_answer_bubble(page)).to_contain_text(SENTINEL_FIRST)
|
||
|
||
# The tool gap (call 2's ~7 s frameless delay — deliberately past
|
||
# the phase-87 counter's 5 s gate): the tool line's elapsed
|
||
# counter appears and ticks BEFORE the first thinking frame
|
||
# settles the line ...
|
||
elapsed = page.locator(".tool-call .tool-elapsed")
|
||
expect(elapsed).to_be_visible(timeout=20_000)
|
||
expect(elapsed).to_have_text(re.compile(r"\(\d+s\)"))
|
||
# ... and the loader + the #send-status state text hold through
|
||
# the whole gap (the status line carries the streaming state's
|
||
# text — it is never empty while a frame is in flight).
|
||
expect(loader).to_be_visible()
|
||
expect(page.locator("#send-status")).not_to_be_empty()
|
||
|
||
|
||
def test_thinking_block_reopens_after_delta_with_visible_loader(
|
||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||
) -> None:
|
||
summary = _reset_db(mock_llm, seed=True)
|
||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||
page.set_default_timeout(30_000)
|
||
login(page, app_url, next="/")
|
||
|
||
_send(page)
|
||
|
||
bubble = _answer_bubble(page)
|
||
block = page.locator(".msg.brain .thinking")
|
||
text_el = page.locator(".msg.brain .thinking-text")
|
||
loader = page.locator("#turn-loader")
|
||
|
||
# THE reported symptom's state (D15): the thinking block is open
|
||
# with non-empty text WHILE the answer bubble already carries
|
||
# call 1's content — the post-delta re-open (call 2's thinking
|
||
# frames re-opened the block the first delta had closed).
|
||
page.wait_for_function(
|
||
"""() => {
|
||
const b = document.querySelector('.msg.brain .thinking');
|
||
const t = b ? b.querySelector('.thinking-text') : null;
|
||
return !!b && b.open && !!t && t.textContent.trim().length > 0;
|
||
}""",
|
||
timeout=30_000,
|
||
)
|
||
expect(block).to_have_count(1)
|
||
expect(block).to_have_attribute("open", "")
|
||
expect(text_el).to_contain_text(SENTINEL_THOUGHT)
|
||
# The re-open is genuinely AFTER a delta: call 1's content is
|
||
# already in the bubble ...
|
||
expect(bubble).to_contain_text(SENTINEL_FIRST)
|
||
# ... and the loader is STILL visible — the frozen window is gone.
|
||
expect(loader).to_be_visible()
|
||
|
||
# Terminal approach: call 2's distinctive final sentence landed in
|
||
# the bubble ...
|
||
expect(bubble).to_contain_text(SENTINEL_FINAL, timeout=30_000)
|
||
# ... and the FINAL thinking round (the last frames before done —
|
||
# thinking AFTER the answer's last delta) re-opened the block
|
||
# AGAIN: the scratchpad carries the last round's text and the
|
||
# block is open (the `thinking` handler opens it before it
|
||
# renders — D15), with the loader still visible (the turn is
|
||
# still in flight — no terminal frame yet).
|
||
expect(text_el).to_contain_text(SENTINEL_FINAL_THOUGHT, timeout=30_000)
|
||
expect(block).to_have_attribute("open", "")
|
||
expect(loader).to_be_visible()
|
||
|
||
# The turn settles (done → idle through setUiState): the loader is
|
||
# HIDDEN (its sole owner — D16, no per-handler cleanup) ...
|
||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||
expect(loader).to_be_hidden()
|
||
# ... the scratchpad text is intact (BOTH thinking rounds) ...
|
||
expect(text_el).to_contain_text(SENTINEL_THOUGHT)
|
||
expect(text_el).to_contain_text(SENTINEL_FINAL_THOUGHT)
|
||
# ... the bubble carries BOTH answers (call 1 + call 2) ...
|
||
expect(bubble).to_contain_text(SENTINEL_FIRST)
|
||
expect(bubble).to_contain_text(SENTINEL_FINAL)
|
||
# ... the send button is back to "Send" (asserted above via the
|
||
# wait) and NOT the Stop treatment ...
|
||
expect(page.locator("#send-btn")).not_to_have_class(re.compile(r"is-stop"))
|
||
# ... and #send-status is back to the idle shape (not stuck on a
|
||
# mid-turn label).
|
||
expect(page.locator("#send-status")).to_have_text("")
|
||
|
||
|
||
def test_loader_a11y_and_reduced_motion(
|
||
page: Page, browser: Browser, app_url: str, mock_llm: int, db_ready: None
|
||
) -> None:
|
||
summary = _reset_db(mock_llm, seed=True)
|
||
assert summary is not None and summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
|
||
page.set_default_timeout(30_000)
|
||
login(page, app_url, next="/")
|
||
|
||
_send(page)
|
||
# The full turn lands (the distinctive final sentence + done →
|
||
# the Send button back).
|
||
expect(_answer_bubble(page)).to_contain_text(SENTINEL_FINAL, timeout=60_000)
|
||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||
|
||
# The a11y split: the loader element is ``aria-hidden="true"`` in
|
||
# the DOM (decorative — it never carries meaning) ...
|
||
expect(page.locator("#turn-loader")).to_have_attribute("aria-hidden", "true")
|
||
# ... and #send-status (the sole ``aria-live`` announcer) is back
|
||
# to the SEND_STATUS idle shape after done — EMPTY, not stuck on a
|
||
# mid-turn label ("… is thinking" / "… is answering").
|
||
status = page.locator("#send-status")
|
||
expect(status).to_have_attribute("aria-live", "polite")
|
||
expect(status).to_have_text("")
|
||
# Control (default-motion context): the CONTAINER carries no
|
||
# animation — the SWEEP path runs the phase-117 bwdraw travel —
|
||
# the reduced-motion check below stills it.
|
||
assert (
|
||
page.evaluate(
|
||
"getComputedStyle(document.querySelector('#turn-loader')).animationName"
|
||
)
|
||
== "none"
|
||
)
|
||
assert (
|
||
page.evaluate(
|
||
"getComputedStyle(document.querySelector('#turn-loader .bw-pulse')).animationName"
|
||
)
|
||
== "bwdraw"
|
||
)
|
||
|
||
# Reduced motion: a FRESH context with ``reducedMotion: "reduce"``
|
||
# (the Playwright context option) — send a second turn and pin
|
||
# that the loader is STILL visible mid-turn: the reduced-motion
|
||
# variant stills the dots (the CSS rule changes — computed
|
||
# ``animation-name: none``), but visibility is owned by the JS
|
||
# ``hidden`` attribute in setUiState (D16), not by CSS.
|
||
rm_context = browser.new_context(
|
||
reduced_motion="reduce", viewport={"width": 1280, "height": 800}
|
||
)
|
||
try:
|
||
rm_page = rm_context.new_page()
|
||
rm_page.set_default_timeout(30_000)
|
||
login(rm_page, app_url, next="/")
|
||
_send(rm_page)
|
||
# Mid-turn (the tool gap — call 2's ~7 s frameless delay): the
|
||
# loader is still the visible cue ...
|
||
expect(rm_page.locator(".tool-call")).to_be_visible(timeout=30_000)
|
||
expect(rm_page.locator("#turn-loader")).to_be_visible()
|
||
# ... and the reduced-motion CSS variant IS applied: the full
|
||
# trace shows static, no travel (the animation is gone from
|
||
# the sweep path's computed style — the loader itself is
|
||
# unchanged).
|
||
assert (
|
||
rm_page.evaluate(
|
||
"getComputedStyle(document.querySelector('#turn-loader .bw-pulse')).animationName"
|
||
)
|
||
== "none"
|
||
)
|
||
finally:
|
||
rm_context.close()
|