brain-of-reese/tests/e2e/test_failed_turn_retry.py

"""Phase 120 E2E (Playwright): failed-turn retry — network errors and
refresh survive a failed turn.

Source: ``TODO.md`` L3–4 — "Retry doesn't seem to work on network
error" + "Refreshing the page after an error shows only the chat
message you sent and no options to retry the message …".
Run in isolation (DB must be up: ``podman compose up -d db``):

    uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov

MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the failure
shapes are the deterministic injections in ``tests/e2e/mock_llm.py``
(the phase-67 counter machinery, the phase-120 triggers):

* ``fail first turn`` (``FAIL_FIRST_TURN_TRIGGER``): the WHOLE first
  turn dies — every app-level attempt of the forced-default budget
  (conftest pins ``BOR_LLM_RETRIES`` to the code default 3 → 4
  attempts) 500s, ZERO frames — the retry-budget exhaustion
  (``tests/e2e/test_llm_retry.py``'s ``always fail`` past its budget) —
  and the second turn's first request streams the normal answer (the
  Retry's re-ask).
* ``partial then fail`` (``PARTIAL_FAIL_TRIGGER``): the first request
  streams ``PARTIAL_FAIL_TEXT`` in ordinary delta chunks, then the
  mock's generator RAISES — a genuine mid-stream connection reset: the
  app's ``chat_stream_retried`` re-raises (a piece already emitted —
  the locked retry-before-first-frame rule) and the chat endpoint
  settles with the terminal SSE ``error`` frame AFTER the partial
  deltas. The second request streams the normal answer.

The e2e app server boots with ``BOR_LLM_RETRY_DELAY=0`` (conftest) so
the dead attempts are instant; ``BOR_LLM_RETRIES`` is forced to the
real default (3) so the exhaustion turn burns the REAL budget — 4
attempts, the last ``retry`` frame reads "(4 of 4)".

KB seed (the ``test_llm_retry.py`` direct-seed pattern): ONE fixture
document (``homelab/kubernetes.md``) — the "How is my Kubernetes
cluster set up?" questions FTS-hit it → HIGH → the grounded path (the
re-asked turn streams the grounded ``MOCK_ANSWER_MARKER`` answer).

Test → contract mapping (locked A1: a failed turn persists as a brain
record with the ``failed`` marker + the capped ``error`` detail; the
banner Retry stays as-is — ``lastBrainWrap`` simply EXISTS on the error
paths now):

1. ``test_network_error_shows_banner_retry_and_failed_bubble`` —
   A (zero-frame network error): the banner (role=alert) is visible
   WITH the phase-111 Retry button (its ``lastBrainWrap`` precondition
   finally holds), a failed bubble with the fixed honest line + the
   in-bubble error note (the detail) + the in-bubble Retry exists, the
   record persists ``failed: true`` + ``error`` in localStorage, and
   the wire is retry×3 + terminal error with NO delta/done. Clicking
   the BANNER Retry re-asks WITHOUT re-typing (no second user
   bubble), the mock now answers, the grounded answer streams in
   place, and the failed bubble is gone (redo-in-place).
2. ``test_partial_then_error_keeps_partial_with_note_and_retry`` —
   B (SSE error frame after partial deltas): the partial bubble KEEPS
   its streamed text, gains the in-bubble error note (the detail) +
   the in-bubble Retry, the record persists the RAW partial +
   ``failed: true`` + ``error``, and the wire is deltas-then-terminal-
   error with no done. Clicking the IN-BUBBLE Retry re-asks in place
   (the failed record is replaced by the fresh grounded answer — no
   re-typing, no failed note).
3. ``test_failed_turn_survives_a_refresh`` — C (the refresh case):
   after a zero-frame failure, ``page.reload()`` restores the question
   + the failed bubble (error detail visible) WITH a working Retry
   button on it — no "new chat" required — and clicking it re-asks
   (the grounded answer replaces the failed record).
4. ``test_stopped_turn_is_not_a_failed_turn`` — negative (phase 48
   unchanged): a user-stopped turn restores with the "Stopped" note
   and NOT a failed note — the two markers are mutually exclusive by
   construction (the stop path never touches the failed funnel).
"""
from __future__ import annotations

import hashlib
import json
import re
import time
from collections.abc import Iterator
from datetime import UTC, datetime
from pathlib import Path

import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from sqlalchemy.orm import Session

from app.db import SessionLocal
from app.models import Chunk, Document
from e2e.auth_helpers import login
from tests.e2e.mock_llm import PARTIAL_FAIL_TEXT, embed_text

REPO = Path(__file__).resolve().parents[2]

# --------------------------------------------------------------------------
# Seed + questions (see the module docstring for the gate notes)
# --------------------------------------------------------------------------

SEED_SOURCE = "docs"
SEED_PATH = "homelab/kubernetes.md"
KUB_CONTENT = (REPO / "tests" / "fixtures" / "docs" / SEED_PATH).read_text()

#: A (zero-frame network error) + C (refresh) — HIGH turn (FTS-hits the
#: seed) so the re-asked turn streams the GROUNDED mock answer.
FAIL_Q = "How is my Kubernetes cluster set up? fail first turn"
#: B (partial deltas + the terminal error frame) — HIGH turn, same
#: re-ask contract.
PARTIAL_Q = "How is my Kubernetes cluster set up? partial then fail"
#: Negative: the mock's ~8 s long answer (12 chars / 0.02 s) — the
#: comfortable stop window (the phase-48 suite's phrasing).
STOP_Q = "How is my Kubernetes cluster set up? write a long answer"

#: The app's terminal LLM-failure copy (app/api/chat.py's LLMError
#: frame) — the error detail the note/banner/record all carry.
ERROR_COPY = "The chat model dropped the connection — try again?"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"

#: The conftest forces BOR_LLM_RETRIES to its default (3) → 4 attempts
#: total; the retry-frame math in the assertions is fixed by that.
MAX_ATTEMPTS = 4

STORAGE_KEY = "bor.chat.v1"


def _js_const(name: str) -> str:
    """A frontend string constant, read from app.js — the E2E asserts
    against the SAME text the page renders (no JS/Python drift)."""
    js = (REPO / "frontend" / "assets" / "app.js").read_text(encoding="utf-8")
    m = re.search(rf'const {name}\s*=\s*\n?\s*"([^"]*)"', js)
    assert m, f"const {name} not found in app.js"
    return m.group(1)


FAILED_TURN_TEXT = _js_const("FAILED_TURN_TEXT")


# --------------------------------------------------------------------------
# DB seeding (TRUNCATE-then-seed, cf. test_llm_retry.py)
# --------------------------------------------------------------------------


def _seed(db: Session) -> None:
    """The single fixture document (see the module docstring)."""
    md = Document(
        source=SEED_SOURCE,
        path=SEED_PATH,
        full_path=f"/tmp/{SEED_PATH}",
        title="Kubernetes",
        content=KUB_CONTENT,
        content_hash=hashlib.sha256(KUB_CONTENT.encode()).hexdigest(),
        indexed_at=datetime.now(UTC),
    )
    db.add(md)
    db.flush()
    # One chunk carrying the mock's own embedding → genuine token overlap
    # for the grounded questions (the FTS path carries them to HIGH).
    db.add(
        Chunk(
            document_id=md.id,
            position=0,
            content=KUB_CONTENT,
            embedding=embed_text(KUB_CONTENT),
        )
    )


def _reset_db() -> None:
    """Truncate the KB (plus the prompt-shaping tables), then re-seed.

    ``steering_notes`` / ``kb_overview`` are truncated too, so the
    prompts are byte-stable regardless of leftovers from other suites.
    """
    with SessionLocal() as db:
        db.execute(
            text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
        )
        db.commit()
        _seed(db)
        db.commit()


@pytest.fixture(autouse=True)
def clean_chats() -> Iterator[None]:
    """The send auto-saves a ``saved_chats`` row per turn (phase 55) —
    truncate around every test so the suite starts from (and leaves)
    an empty deployment (the phase-80/103 isolation pattern)."""
    with SessionLocal() as db:
        db.execute(text("TRUNCATE saved_chats"))
        db.commit()
    yield
    # The auto-save is fire-and-forget (the browser never awaits the
    # PUT): a short margin lets the last turn's PUT land server-side
    # before the TRUNCATE, so the teardown never 500s an in-flight
    # upsert (the "Could not refresh instance" race). This runs AFTER
    # the page closes (LIFO finalization), so the PUT is either done
    # or aborted — never in flight.
    time.sleep(0.75)
    with SessionLocal() as db:
        db.execute(text("TRUNCATE saved_chats"))
        db.commit()


# --------------------------------------------------------------------------
# Page hooks (the SSE capture — the phase-37/67 pattern from
# test_llm_retry.py) + localStorage reads
# --------------------------------------------------------------------------

SSE_HOOK = """
() => {
  if (window.__sseInstalled) return;
  window.__sseInstalled = true;
  window.__sseFrames = [];
  const origFetch = window.fetch;
  window.fetch = async function (...args) {
    const res = await origFetch.apply(this, args);
    try {
      const url = typeof args[0] === 'string' ? args[0] : args[0].url;
      if (url.includes('/api/chat')) {
        res.clone().text().then((bodyText) => {
          for (const block of bodyText.split('\\n\\n')) {
            const line = block.trim();
            if (line.startsWith('data: ')) {
              window.__sseFrames.push(line.slice(6));
            }
          }
        });
      }
    } catch (e) { /* non-clonable responses: ignored */ }
    return res;
  };
}
"""


def _install_sse_hook(page: Page) -> None:
    page.evaluate(SSE_HOOK)


def _frames(page: Page, terminal: str = "done") -> list[dict]:
    """The captured SSE frames of the CURRENT turn, once the *terminal*
    frame lands (``error`` for the failure scenarios)."""
    deadline = time.monotonic() + 30.0
    while True:
        raw = page.evaluate("() => window.__sseFrames || []")
        parsed = [json.loads(line) for line in raw if line]
        if any(f.get("type") == terminal for f in parsed):
            return parsed
        if time.monotonic() > deadline:
            raise AssertionError(
                f"SSE hook captured no `{terminal}` frame (frames so far: "
                f"{len(parsed)}) — hook install failed?"
            )
        time.sleep(0.05)


def _stored(page: Page) -> dict:
    """The persisted ``bor.chat.v1`` conversation (localStorage — the
    phase-14 store the restore reads on refresh)."""
    raw = page.evaluate(f"() => localStorage.getItem('{STORAGE_KEY}')")
    assert raw is not None, "the conversation key must exist in localStorage"
    return json.loads(raw)


def _submit(page: Page, question: str) -> None:
    page.fill("#message-input", question)
    page.click("#send-btn")
    # The user bubble lands synchronously with the submit handler.
    expect(page.locator(".msg.user .bubble").last).to_contain_text(question)


def _assert_failed_bubble(
    page: Page, *, expected_text: str, note_detail: str
) -> None:
    """The failed-bubble contract shared by the scenarios: ONE brain
    bubble with *expected_text*, the in-bubble error note (the "Failed"
    label + *note_detail*), and the in-bubble Retry button
    (``markLastRetryable`` on the failed bubble — the last brain
    wrap)."""
    brain = page.locator(".msg.brain")
    expect(brain).to_have_count(1)
    expect(brain.locator(".bubble")).to_have_text(expected_text)
    note = brain.locator(".failed-note")
    expect(note).to_have_count(1)
    expect(note).to_contain_text("Failed")
    expect(note).to_contain_text(note_detail)
    expect(brain.locator(".retry-btn")).to_have_count(1)
    # A note, not an answer: no Save-as-doc button on the failed bubble.
    expect(brain.locator(".save-as-doc-btn")).to_have_count(0)


def _assert_settled_composer(page: Page) -> None:
    """The failed turn settles to idle: the Send button recovered
    (never a zombified Stop — the §7.4 never-stale contract)."""
    expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
    expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)


def _assert_redo_succeeded(page: Page) -> None:
    """After a Retry re-ask: exactly question + the fresh GROUNDED
    answer (the mock now answers) — the failed record is REPLACED in
    place (no second user bubble, no failed note, no failed marker on
    the new record)."""
    expect(page.locator(".msg.user .bubble")).to_have_count(1)
    expect(page.locator(".msg.brain")).to_have_count(1)
    bubble = page.locator(".msg.brain .bubble").last
    expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
    expect(page.locator(".failed-note")).to_have_count(0)
    # The turn has FULLY settled: the ``done`` frame is processed (the
    # answer record persisted through rememberBrainTurn — synchronously
    # in the frame handler) BEFORE the finally's idle state flips the
    # label to Send. The label is the settle signal, so the local
    # storage read below is race-free (the marker can be visible a few
    # frames before ``done`` — the last deltas carry it).
    expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
    stored = _stored(page)
    assert [m["who"] for m in stored["messages"]] == ["user", "brain"]
    last = stored["messages"][-1]
    assert last.get("failed") in (None, False), "the fresh answer is not failed"
    assert MOCK_ANSWER_MARKER in last["text"]
    # The fresh grounded record's text is the rendered answer (no
    # failed-marker leftovers in the conversation).
    assert FAILED_TURN_TEXT not in last["text"]


# --------------------------------------------------------------------------
# A — zero-frame network error: the banner WITH a working Retry + the
#    failed bubble; clicking the banner Retry re-asks without re-typing
# --------------------------------------------------------------------------


def test_network_error_shows_banner_retry_and_failed_bubble(
    page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
    _reset_db()
    page.set_default_timeout(30_000)
    login(page, app_url, next="/")
    _install_sse_hook(page)

    _submit(page, FAIL_Q)

    # After the 4th dead attempt (the forced-default budget) the turn
    # dies with ZERO frames: the existing terminal error banner
    # (role=alert, the "dropped the connection" copy) — NOW with its
    # Retry button revealed (the phase-111 condition
    # ``opts.retryable && lastBrainWrap`` finally holds: the failed
    # bubble's wrap exists before the error state).
    banner = page.locator("#kb-banner")
    expect(banner).to_have_attribute("role", "alert", timeout=60_000)
    expect(banner).to_contain_text(ERROR_COPY)
    expect(page.locator("#banner-retry")).to_be_visible(timeout=10_000)
    _assert_settled_composer(page)

    # The zero-frame failed bubble: the fixed honest line (NOT the raw
    # detail — that rides the note), the in-bubble error note carrying
    # the detail, and the in-bubble Retry.
    _assert_failed_bubble(page, expected_text=FAILED_TURN_TEXT, note_detail=ERROR_COPY)

    # Persisted: the question's user record + the failed brain record —
    # ``failed: true`` + the capped detail (the phase-48 ``stopped``
    # precedent in localStorage; the phase-55 auto-save rides the same
    # call, so the server-side saved chat carries it too).
    stored = _stored(page)
    assert [m["who"] for m in stored["messages"]] == ["user", "brain"]
    assert stored["messages"][0]["text"] == FAIL_Q
    failed = stored["messages"][-1]
    assert failed["text"] == FAILED_TURN_TEXT
    assert failed["failed"] is True
    assert failed["error"] == ERROR_COPY

    # Wire: the three retry frames (attempts 2–4 of 4 — the real
    # budget), then the terminal error frame LAST — no delta, no done
    # (ZERO frames reached the client).
    frames = _frames(page, terminal="error")
    assert [f for f in frames if f["type"] == "retry"] == [
        {"type": "retry", "attempt": a, "max_attempts": MAX_ATTEMPTS}
        for a in (2, 3, 4)
    ], frames
    assert frames[-1]["type"] == "error"
    assert ERROR_COPY in frames[-1]["detail"]
    assert not [f for f in frames if f.get("type") in ("done", "delta")], frames

    # The BANNER Retry: re-asks WITHOUT re-typing (no second user
    # bubble) — the mock now answers (the sequence resets after its
    # guarded success) and the grounded answer streams in place of the
    # failed bubble (redo-in-place — the failed record is popped).
    page.click("#banner-retry")
    _assert_redo_succeeded(page)


# --------------------------------------------------------------------------
# B — SSE error frame after partial deltas: the partial keeps its text
#    + the error note + the in-bubble Retry; clicking it re-asks in
#    place (the failed record is replaced by the fresh answer)
# --------------------------------------------------------------------------


def test_partial_then_error_keeps_partial_with_note_and_retry(
    page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
    _reset_db()
    page.set_default_timeout(30_000)
    login(page, app_url, next="/")
    _install_sse_hook(page)

    _submit(page, PARTIAL_Q)

    # The partial streams, then the mock's generator dies mid-stream:
    # the app settles the turn with the terminal error banner AFTER the
    # partial deltas — and the partial bubble KEEPS its text.
    banner = page.locator("#kb-banner")
    expect(banner).to_have_attribute("role", "alert", timeout=60_000)
    expect(banner).to_contain_text(ERROR_COPY)
    _assert_settled_composer(page)

    # The failed partial: the streamed text kept verbatim + the
    # in-bubble error note (the detail) + the in-bubble Retry.
    _assert_failed_bubble(
        page, expected_text=PARTIAL_FAIL_TEXT, note_detail=ERROR_COPY
    )

    # Persisted: the RAW partial (what the user saw is what is stored —
    # the phase-17/20 convention, now with the failed marker + detail).
    stored = _stored(page)
    assert [m["who"] for m in stored["messages"]] == ["user", "brain"]
    failed = stored["messages"][-1]
    assert failed["text"] == PARTIAL_FAIL_TEXT
    assert failed["failed"] is True
    assert failed["error"] == ERROR_COPY

    # Wire: the delta frames (the partial, in order) precede the
    # terminal error frame — no done (the stream never settled), no
    # retry (a piece already emitted — the locked retry-before-first-
    # frame rule means the failure is terminal, never redone).
    frames = _frames(page, terminal="error")
    deltas = [f for f in frames if f["type"] == "delta"]
    assert deltas, "no delta frames before the error"
    assert "".join(d["text"] for d in deltas) == PARTIAL_FAIL_TEXT
    assert not [f for f in frames if f.get("type") == "retry"], frames
    assert frames[-1]["type"] == "error"
    assert not [f for f in frames if f.get("type") == "done"], frames

    # The IN-BUBBLE Retry: redo-in-place — the failed partial record is
    # popped + re-asked (no re-typing), and the fresh grounded answer
    # (the mock now answers — the sequence resets) replaces it.
    page.click(".msg.brain .retry-btn")
    _assert_redo_succeeded(page)


# --------------------------------------------------------------------------
# C — the refresh case: reload restores the failed bubble WITH a
#    working Retry; clicking it re-asks (no "new chat" required)
# --------------------------------------------------------------------------


def test_failed_turn_survives_a_refresh(
    page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
    _reset_db()
    page.set_default_timeout(30_000)
    login(page, app_url, next="/")
    _install_sse_hook(page)

    _submit(page, FAIL_Q)

    # The zero-frame failure lands (same shape as scenario A).
    banner = page.locator("#kb-banner")
    expect(banner).to_have_attribute("role", "alert", timeout=60_000)
    expect(banner).to_contain_text(ERROR_COPY)
    expect(page.locator("#banner-retry")).to_be_visible(timeout=10_000)
    _assert_failed_bubble(
        page, expected_text=FAILED_TURN_TEXT, note_detail=ERROR_COPY
    )

    # The refresh: the page restores from localStorage (the failed
    # record persisted by the funnel) — the question + the failed
    # bubble with its error detail, and the Retry button on the failed
    # bubble (the restore loop's lastBrainWrap + markLastRetryable land
    # it on the LAST restored brain bubble — the failed one).
    page.reload()
    expect(page.locator("#messages > .msg")).to_have_count(2)
    expect(page.locator(".msg.user .bubble")).to_have_text(FAIL_Q)
    _assert_failed_bubble(
        page, expected_text=FAILED_TURN_TEXT, note_detail=ERROR_COPY
    )
    # The error state itself does NOT restore (the banner is a live-
    # turn state) — the recovery affordance is the in-bubble Retry.
    expect(page.locator("#kb-banner")).to_be_hidden()
    _assert_settled_composer(page)

    # The restored record still carries the marker (the restore is
    # lossless: text + detail + marker).
    stored = _stored(page)
    assert [m["who"] for m in stored["messages"]] == ["user", "brain"]
    assert stored["messages"][-1]["failed"] is True
    assert stored["messages"][-1]["error"] == ERROR_COPY

    # The restored Retry WORKS: clicking it re-asks the question that
    # precedes the failed record (retryLastTurn, unchanged) — the
    # grounded answer streams and replaces the failed record.
    page.click(".msg.brain .retry-btn")
    _assert_redo_succeeded(page)


# --------------------------------------------------------------------------
# Negative — a STOPPED turn (phase 48) is NOT a failed turn: the stop
# path restores with the "Stopped" note and never the failed note
# (mutually exclusive by construction — the stop branch is the catch's
# own, the failed funnel is the catch's else)
# --------------------------------------------------------------------------


def test_stopped_turn_is_not_a_failed_turn(
    page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
    _reset_db()
    page.set_default_timeout(30_000)
    login(page, app_url, next="/")

    # The ~8 s long answer — a comfortable stop window (phase 48).
    _submit(page, STOP_Q)
    answer = page.locator(".msg.brain .bubble:not(.typing)")
    answer.wait_for(state="visible", timeout=30_000)
    partial = ""
    deadline = time.monotonic() + 15
    while time.monotonic() < deadline:
        partial = answer.inner_text()
        if len(partial.split()) >= 8:
            break
        time.sleep(0.05)
    assert len(partial.split()) >= 8, "no answer deltas before the stop"

    expect(page.locator("#send-label")).to_have_text("Stop")
    page.click("#send-btn")  # the in-flight button IS the Stop control

    # Settled to idle: no error banner (the stop path never shows one),
    # the partial kept + the "Stopped" note — and NO failed note.
    expect(page.locator("#send-label")).to_have_text("Send", timeout=5_000)
    expect(page.locator("#kb-banner")).to_be_hidden()
    expect(page.locator(".msg.brain .stopped-note")).to_have_count(1)
    expect(page.locator(".msg.brain .stopped-note")).to_contain_text("Stopped")
    expect(page.locator(".msg.brain .failed-note")).to_have_count(0)
    stopped_text = answer.inner_text()
    assert stopped_text.strip()

    # Persisted: the stopped marker, NOT the failed marker (the two are
    # mutually exclusive by construction).
    stored = _stored(page)
    last = stored["messages"][-1]
    assert last["stopped"] is True
    assert last.get("failed") in (None, False), "a stopped turn is not a failed turn"
    assert "error" not in last or last["error"] is None

    # The refresh: the stopped partial restores with the "Stopped"
    # note and NOT the failed note — phase 48's restore is unchanged.
    page.reload()
    expect(page.locator("#messages > .msg")).to_have_count(2)
    expect(page.locator(".msg.user .bubble")).to_have_text(STOP_Q)
    expect(page.locator(".msg.brain .bubble:not(.typing)")).to_have_text(
        stopped_text
    )
    expect(page.locator(".msg.brain .stopped-note")).to_have_count(1)
    expect(page.locator(".msg.brain .failed-note")).to_have_count(0)
    expect(page.locator("#kb-banner")).to_be_hidden()
    stored = _stored(page)
    assert stored["messages"][-1]["stopped"] is True
    assert stored["messages"][-1].get("failed") in (None, False)