"""Phase 120 E2E (Playwright): failed-turn retry — network errors and refresh survive a failed turn. Source: ``TODO.md`` L3–4 — "Retry doesn't seem to work on network error" + "Refreshing the page after an error shows only the chat message you sent and no options to retry the message …". Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the failure shapes are the deterministic injections in ``tests/e2e/mock_llm.py`` (the phase-67 counter machinery, the phase-120 triggers): * ``fail first turn`` (``FAIL_FIRST_TURN_TRIGGER``): the WHOLE first turn dies — every app-level attempt of the forced-default budget (conftest pins ``BOR_LLM_RETRIES`` to the code default 3 → 4 attempts) 500s, ZERO frames — the retry-budget exhaustion (``tests/e2e/test_llm_retry.py``'s ``always fail`` past its budget) — and the second turn's first request streams the normal answer (the Retry's re-ask). * ``partial then fail`` (``PARTIAL_FAIL_TRIGGER``): the first request streams ``PARTIAL_FAIL_TEXT`` in ordinary delta chunks, then the mock's generator RAISES — a genuine mid-stream connection reset: the app's ``chat_stream_retried`` re-raises (a piece already emitted — the locked retry-before-first-frame rule) and the chat endpoint settles with the terminal SSE ``error`` frame AFTER the partial deltas. The second request streams the normal answer. The e2e app server boots with ``BOR_LLM_RETRY_DELAY=0`` (conftest) so the dead attempts are instant; ``BOR_LLM_RETRIES`` is forced to the real default (3) so the exhaustion turn burns the REAL budget — 4 attempts, the last ``retry`` frame reads "(4 of 4)". KB seed (the ``test_llm_retry.py`` direct-seed pattern): ONE fixture document (``homelab/kubernetes.md``) — the "How is my Kubernetes cluster set up?" questions FTS-hit it → HIGH → the grounded path (the re-asked turn streams the grounded ``MOCK_ANSWER_MARKER`` answer). Test → contract mapping (locked A1: a failed turn persists as a brain record with the ``failed`` marker + the capped ``error`` detail; the banner Retry stays as-is — ``lastBrainWrap`` simply EXISTS on the error paths now): 1. ``test_network_error_shows_banner_retry_and_failed_bubble`` — A (zero-frame network error): the banner (role=alert) is visible WITH the phase-111 Retry button (its ``lastBrainWrap`` precondition finally holds), a failed bubble with the fixed honest line + the in-bubble error note (the detail) + the in-bubble Retry exists, the record persists ``failed: true`` + ``error`` in localStorage, and the wire is retry×3 + terminal error with NO delta/done. Clicking the BANNER Retry re-asks WITHOUT re-typing (no second user bubble), the mock now answers, the grounded answer streams in place, and the failed bubble is gone (redo-in-place). 2. ``test_partial_then_error_keeps_partial_with_note_and_retry`` — B (SSE error frame after partial deltas): the partial bubble KEEPS its streamed text, gains the in-bubble error note (the detail) + the in-bubble Retry, the record persists the RAW partial + ``failed: true`` + ``error``, and the wire is deltas-then-terminal- error with no done. Clicking the IN-BUBBLE Retry re-asks in place (the failed record is replaced by the fresh grounded answer — no re-typing, no failed note). 3. ``test_failed_turn_survives_a_refresh`` — C (the refresh case): after a zero-frame failure, ``page.reload()`` restores the question + the failed bubble (error detail visible) WITH a working Retry button on it — no "new chat" required — and clicking it re-asks (the grounded answer replaces the failed record). 4. ``test_stopped_turn_is_not_a_failed_turn`` — negative (phase 48 unchanged): a user-stopped turn restores with the "Stopped" note and NOT a failed note — the two markers are mutually exclusive by construction (the stop path never touches the failed funnel). """ from __future__ import annotations import hashlib import json import re import time from collections.abc import Iterator from datetime import UTC, datetime from pathlib import Path import pytest from playwright.sync_api import Page, expect from sqlalchemy import text from sqlalchemy.orm import Session from app.db import SessionLocal from app.models import Chunk, Document from e2e.auth_helpers import login from tests.e2e.mock_llm import PARTIAL_FAIL_TEXT, embed_text REPO = Path(__file__).resolve().parents[2] # -------------------------------------------------------------------------- # Seed + questions (see the module docstring for the gate notes) # -------------------------------------------------------------------------- SEED_SOURCE = "docs" SEED_PATH = "homelab/kubernetes.md" KUB_CONTENT = (REPO / "tests" / "fixtures" / "docs" / SEED_PATH).read_text() #: A (zero-frame network error) + C (refresh) — HIGH turn (FTS-hits the #: seed) so the re-asked turn streams the GROUNDED mock answer. FAIL_Q = "How is my Kubernetes cluster set up? fail first turn" #: B (partial deltas + the terminal error frame) — HIGH turn, same #: re-ask contract. PARTIAL_Q = "How is my Kubernetes cluster set up? partial then fail" #: Negative: the mock's ~8 s long answer (12 chars / 0.02 s) — the #: comfortable stop window (the phase-48 suite's phrasing). STOP_Q = "How is my Kubernetes cluster set up? write a long answer" #: The app's terminal LLM-failure copy (app/api/chat.py's LLMError #: frame) — the error detail the note/banner/record all carry. ERROR_COPY = "The chat model dropped the connection — try again?" MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E" #: The conftest forces BOR_LLM_RETRIES to its default (3) → 4 attempts #: total; the retry-frame math in the assertions is fixed by that. MAX_ATTEMPTS = 4 STORAGE_KEY = "bor.chat.v1" def _js_const(name: str) -> str: """A frontend string constant, read from app.js — the E2E asserts against the SAME text the page renders (no JS/Python drift).""" js = (REPO / "frontend" / "assets" / "app.js").read_text(encoding="utf-8") m = re.search(rf'const {name}\s*=\s*\n?\s*"([^"]*)"', js) assert m, f"const {name} not found in app.js" return m.group(1) FAILED_TURN_TEXT = _js_const("FAILED_TURN_TEXT") # -------------------------------------------------------------------------- # DB seeding (TRUNCATE-then-seed, cf. test_llm_retry.py) # -------------------------------------------------------------------------- def _seed(db: Session) -> None: """The single fixture document (see the module docstring).""" md = Document( source=SEED_SOURCE, path=SEED_PATH, full_path=f"/tmp/{SEED_PATH}", title="Kubernetes", content=KUB_CONTENT, content_hash=hashlib.sha256(KUB_CONTENT.encode()).hexdigest(), indexed_at=datetime.now(UTC), ) db.add(md) db.flush() # One chunk carrying the mock's own embedding → genuine token overlap # for the grounded questions (the FTS path carries them to HIGH). db.add( Chunk( document_id=md.id, position=0, content=KUB_CONTENT, embedding=embed_text(KUB_CONTENT), ) ) def _reset_db() -> None: """Truncate the KB (plus the prompt-shaping tables), then re-seed. ``steering_notes`` / ``kb_overview`` are truncated too, so the prompts are byte-stable regardless of leftovers from other suites. """ with SessionLocal() as db: db.execute( text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview") ) db.commit() _seed(db) db.commit() @pytest.fixture(autouse=True) def clean_chats() -> Iterator[None]: """The send auto-saves a ``saved_chats`` row per turn (phase 55) — truncate around every test so the suite starts from (and leaves) an empty deployment (the phase-80/103 isolation pattern).""" with SessionLocal() as db: db.execute(text("TRUNCATE saved_chats")) db.commit() yield # The auto-save is fire-and-forget (the browser never awaits the # PUT): a short margin lets the last turn's PUT land server-side # before the TRUNCATE, so the teardown never 500s an in-flight # upsert (the "Could not refresh instance" race). This runs AFTER # the page closes (LIFO finalization), so the PUT is either done # or aborted — never in flight. time.sleep(0.75) with SessionLocal() as db: db.execute(text("TRUNCATE saved_chats")) db.commit() # -------------------------------------------------------------------------- # Page hooks (the SSE capture — the phase-37/67 pattern from # test_llm_retry.py) + localStorage reads # -------------------------------------------------------------------------- SSE_HOOK = """ () => { if (window.__sseInstalled) return; window.__sseInstalled = true; window.__sseFrames = []; const origFetch = window.fetch; window.fetch = async function (...args) { const res = await origFetch.apply(this, args); try { const url = typeof args[0] === 'string' ? args[0] : args[0].url; if (url.includes('/api/chat')) { res.clone().text().then((bodyText) => { for (const block of bodyText.split('\\n\\n')) { const line = block.trim(); if (line.startsWith('data: ')) { window.__sseFrames.push(line.slice(6)); } } }); } } catch (e) { /* non-clonable responses: ignored */ } return res; }; } """ def _install_sse_hook(page: Page) -> None: page.evaluate(SSE_HOOK) def _frames(page: Page, terminal: str = "done") -> list[dict]: """The captured SSE frames of the CURRENT turn, once the *terminal* frame lands (``error`` for the failure scenarios).""" deadline = time.monotonic() + 30.0 while True: raw = page.evaluate("() => window.__sseFrames || []") parsed = [json.loads(line) for line in raw if line] if any(f.get("type") == terminal for f in parsed): return parsed if time.monotonic() > deadline: raise AssertionError( f"SSE hook captured no `{terminal}` frame (frames so far: " f"{len(parsed)}) — hook install failed?" ) time.sleep(0.05) def _stored(page: Page) -> dict: """The persisted ``bor.chat.v1`` conversation (localStorage — the phase-14 store the restore reads on refresh).""" raw = page.evaluate(f"() => localStorage.getItem('{STORAGE_KEY}')") assert raw is not None, "the conversation key must exist in localStorage" return json.loads(raw) def _submit(page: Page, question: str) -> None: page.fill("#message-input", question) page.click("#send-btn") # The user bubble lands synchronously with the submit handler. expect(page.locator(".msg.user .bubble").last).to_contain_text(question) def _assert_failed_bubble( page: Page, *, expected_text: str, note_detail: str ) -> None: """The failed-bubble contract shared by the scenarios: ONE brain bubble with *expected_text*, the in-bubble error note (the "Failed" label + *note_detail*), and the in-bubble Retry button (``markLastRetryable`` on the failed bubble — the last brain wrap).""" brain = page.locator(".msg.brain") expect(brain).to_have_count(1) expect(brain.locator(".bubble")).to_have_text(expected_text) note = brain.locator(".failed-note") expect(note).to_have_count(1) expect(note).to_contain_text("Failed") expect(note).to_contain_text(note_detail) expect(brain.locator(".retry-btn")).to_have_count(1) # A note, not an answer: no Save-as-doc button on the failed bubble. expect(brain.locator(".save-as-doc-btn")).to_have_count(0) def _assert_settled_composer(page: Page) -> None: """The failed turn settles to idle: the Send button recovered (never a zombified Stop — the §7.4 never-stale contract).""" expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000) expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) def _assert_redo_succeeded(page: Page) -> None: """After a Retry re-ask: exactly question + the fresh GROUNDED answer (the mock now answers) — the failed record is REPLACED in place (no second user bubble, no failed note, no failed marker on the new record).""" expect(page.locator(".msg.user .bubble")).to_have_count(1) expect(page.locator(".msg.brain")).to_have_count(1) bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) expect(page.locator(".failed-note")).to_have_count(0) # The turn has FULLY settled: the ``done`` frame is processed (the # answer record persisted through rememberBrainTurn — synchronously # in the frame handler) BEFORE the finally's idle state flips the # label to Send. The label is the settle signal, so the local # storage read below is race-free (the marker can be visible a few # frames before ``done`` — the last deltas carry it). expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) stored = _stored(page) assert [m["who"] for m in stored["messages"]] == ["user", "brain"] last = stored["messages"][-1] assert last.get("failed") in (None, False), "the fresh answer is not failed" assert MOCK_ANSWER_MARKER in last["text"] # The fresh grounded record's text is the rendered answer (no # failed-marker leftovers in the conversation). assert FAILED_TURN_TEXT not in last["text"] # -------------------------------------------------------------------------- # A — zero-frame network error: the banner WITH a working Retry + the # failed bubble; clicking the banner Retry re-asks without re-typing # -------------------------------------------------------------------------- def test_network_error_shows_banner_retry_and_failed_bubble( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _reset_db() page.set_default_timeout(30_000) login(page, app_url, next="/") _install_sse_hook(page) _submit(page, FAIL_Q) # After the 4th dead attempt (the forced-default budget) the turn # dies with ZERO frames: the existing terminal error banner # (role=alert, the "dropped the connection" copy) — NOW with its # Retry button revealed (the phase-111 condition # ``opts.retryable && lastBrainWrap`` finally holds: the failed # bubble's wrap exists before the error state). banner = page.locator("#kb-banner") expect(banner).to_have_attribute("role", "alert", timeout=60_000) expect(banner).to_contain_text(ERROR_COPY) expect(page.locator("#banner-retry")).to_be_visible(timeout=10_000) _assert_settled_composer(page) # The zero-frame failed bubble: the fixed honest line (NOT the raw # detail — that rides the note), the in-bubble error note carrying # the detail, and the in-bubble Retry. _assert_failed_bubble(page, expected_text=FAILED_TURN_TEXT, note_detail=ERROR_COPY) # Persisted: the question's user record + the failed brain record — # ``failed: true`` + the capped detail (the phase-48 ``stopped`` # precedent in localStorage; the phase-55 auto-save rides the same # call, so the server-side saved chat carries it too). stored = _stored(page) assert [m["who"] for m in stored["messages"]] == ["user", "brain"] assert stored["messages"][0]["text"] == FAIL_Q failed = stored["messages"][-1] assert failed["text"] == FAILED_TURN_TEXT assert failed["failed"] is True assert failed["error"] == ERROR_COPY # Wire: the three retry frames (attempts 2–4 of 4 — the real # budget), then the terminal error frame LAST — no delta, no done # (ZERO frames reached the client). frames = _frames(page, terminal="error") assert [f for f in frames if f["type"] == "retry"] == [ {"type": "retry", "attempt": a, "max_attempts": MAX_ATTEMPTS} for a in (2, 3, 4) ], frames assert frames[-1]["type"] == "error" assert ERROR_COPY in frames[-1]["detail"] assert not [f for f in frames if f.get("type") in ("done", "delta")], frames # The BANNER Retry: re-asks WITHOUT re-typing (no second user # bubble) — the mock now answers (the sequence resets after its # guarded success) and the grounded answer streams in place of the # failed bubble (redo-in-place — the failed record is popped). page.click("#banner-retry") _assert_redo_succeeded(page) # -------------------------------------------------------------------------- # B — SSE error frame after partial deltas: the partial keeps its text # + the error note + the in-bubble Retry; clicking it re-asks in # place (the failed record is replaced by the fresh answer) # -------------------------------------------------------------------------- def test_partial_then_error_keeps_partial_with_note_and_retry( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _reset_db() page.set_default_timeout(30_000) login(page, app_url, next="/") _install_sse_hook(page) _submit(page, PARTIAL_Q) # The partial streams, then the mock's generator dies mid-stream: # the app settles the turn with the terminal error banner AFTER the # partial deltas — and the partial bubble KEEPS its text. banner = page.locator("#kb-banner") expect(banner).to_have_attribute("role", "alert", timeout=60_000) expect(banner).to_contain_text(ERROR_COPY) _assert_settled_composer(page) # The failed partial: the streamed text kept verbatim + the # in-bubble error note (the detail) + the in-bubble Retry. _assert_failed_bubble( page, expected_text=PARTIAL_FAIL_TEXT, note_detail=ERROR_COPY ) # Persisted: the RAW partial (what the user saw is what is stored — # the phase-17/20 convention, now with the failed marker + detail). stored = _stored(page) assert [m["who"] for m in stored["messages"]] == ["user", "brain"] failed = stored["messages"][-1] assert failed["text"] == PARTIAL_FAIL_TEXT assert failed["failed"] is True assert failed["error"] == ERROR_COPY # Wire: the delta frames (the partial, in order) precede the # terminal error frame — no done (the stream never settled), no # retry (a piece already emitted — the locked retry-before-first- # frame rule means the failure is terminal, never redone). frames = _frames(page, terminal="error") deltas = [f for f in frames if f["type"] == "delta"] assert deltas, "no delta frames before the error" assert "".join(d["text"] for d in deltas) == PARTIAL_FAIL_TEXT assert not [f for f in frames if f.get("type") == "retry"], frames assert frames[-1]["type"] == "error" assert not [f for f in frames if f.get("type") == "done"], frames # The IN-BUBBLE Retry: redo-in-place — the failed partial record is # popped + re-asked (no re-typing), and the fresh grounded answer # (the mock now answers — the sequence resets) replaces it. page.click(".msg.brain .retry-btn") _assert_redo_succeeded(page) # -------------------------------------------------------------------------- # C — the refresh case: reload restores the failed bubble WITH a # working Retry; clicking it re-asks (no "new chat" required) # -------------------------------------------------------------------------- def test_failed_turn_survives_a_refresh( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _reset_db() page.set_default_timeout(30_000) login(page, app_url, next="/") _install_sse_hook(page) _submit(page, FAIL_Q) # The zero-frame failure lands (same shape as scenario A). banner = page.locator("#kb-banner") expect(banner).to_have_attribute("role", "alert", timeout=60_000) expect(banner).to_contain_text(ERROR_COPY) expect(page.locator("#banner-retry")).to_be_visible(timeout=10_000) _assert_failed_bubble( page, expected_text=FAILED_TURN_TEXT, note_detail=ERROR_COPY ) # The refresh: the page restores from localStorage (the failed # record persisted by the funnel) — the question + the failed # bubble with its error detail, and the Retry button on the failed # bubble (the restore loop's lastBrainWrap + markLastRetryable land # it on the LAST restored brain bubble — the failed one). page.reload() expect(page.locator("#messages > .msg")).to_have_count(2) expect(page.locator(".msg.user .bubble")).to_have_text(FAIL_Q) _assert_failed_bubble( page, expected_text=FAILED_TURN_TEXT, note_detail=ERROR_COPY ) # The error state itself does NOT restore (the banner is a live- # turn state) — the recovery affordance is the in-bubble Retry. expect(page.locator("#kb-banner")).to_be_hidden() _assert_settled_composer(page) # The restored record still carries the marker (the restore is # lossless: text + detail + marker). stored = _stored(page) assert [m["who"] for m in stored["messages"]] == ["user", "brain"] assert stored["messages"][-1]["failed"] is True assert stored["messages"][-1]["error"] == ERROR_COPY # The restored Retry WORKS: clicking it re-asks the question that # precedes the failed record (retryLastTurn, unchanged) — the # grounded answer streams and replaces the failed record. page.click(".msg.brain .retry-btn") _assert_redo_succeeded(page) # -------------------------------------------------------------------------- # Negative — a STOPPED turn (phase 48) is NOT a failed turn: the stop # path restores with the "Stopped" note and never the failed note # (mutually exclusive by construction — the stop branch is the catch's # own, the failed funnel is the catch's else) # -------------------------------------------------------------------------- def test_stopped_turn_is_not_a_failed_turn( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _reset_db() page.set_default_timeout(30_000) login(page, app_url, next="/") # The ~8 s long answer — a comfortable stop window (phase 48). _submit(page, STOP_Q) answer = page.locator(".msg.brain .bubble:not(.typing)") answer.wait_for(state="visible", timeout=30_000) partial = "" deadline = time.monotonic() + 15 while time.monotonic() < deadline: partial = answer.inner_text() if len(partial.split()) >= 8: break time.sleep(0.05) assert len(partial.split()) >= 8, "no answer deltas before the stop" expect(page.locator("#send-label")).to_have_text("Stop") page.click("#send-btn") # the in-flight button IS the Stop control # Settled to idle: no error banner (the stop path never shows one), # the partial kept + the "Stopped" note — and NO failed note. expect(page.locator("#send-label")).to_have_text("Send", timeout=5_000) expect(page.locator("#kb-banner")).to_be_hidden() expect(page.locator(".msg.brain .stopped-note")).to_have_count(1) expect(page.locator(".msg.brain .stopped-note")).to_contain_text("Stopped") expect(page.locator(".msg.brain .failed-note")).to_have_count(0) stopped_text = answer.inner_text() assert stopped_text.strip() # Persisted: the stopped marker, NOT the failed marker (the two are # mutually exclusive by construction). stored = _stored(page) last = stored["messages"][-1] assert last["stopped"] is True assert last.get("failed") in (None, False), "a stopped turn is not a failed turn" assert "error" not in last or last["error"] is None # The refresh: the stopped partial restores with the "Stopped" # note and NOT the failed note — phase 48's restore is unchanged. page.reload() expect(page.locator("#messages > .msg")).to_have_count(2) expect(page.locator(".msg.user .bubble")).to_have_text(STOP_Q) expect(page.locator(".msg.brain .bubble:not(.typing)")).to_have_text( stopped_text ) expect(page.locator(".msg.brain .stopped-note")).to_have_count(1) expect(page.locator(".msg.brain .failed-note")).to_have_count(0) expect(page.locator("#kb-banner")).to_be_hidden() stored = _stored(page) assert stored["messages"][-1]["stopped"] is True assert stored["messages"][-1].get("failed") in (None, False)