phase: 120_failed_turn_retry
Build and Push Containers / build-and-push-app (push) Successful in 3m5s
Build and Push Containers / build-and-push-db (push) Failing after 2m13s

All verification complete. Final report:

**Phase 120 — Failed-turn retry: verification pass (all 3 tasks were done; final verification + 1 regression fix)**

**Verified:** `ChatMessage.failed`/`error` (≤500, `extra="forbid"` intact); `finalizeFailedTurn` funnel on the 3 failure paths (catch-else, stream-drop guard, zero-frame fallback) with `failed: true` + capped detail + `markLastRetryable`; `appendFailedNote` restore branch (Save-as-doc/Tune excluded); `showErrorBanner`/`retryLastTurn` byte-pinned untouched; only the three paths persist `failed: true` (grep + unit pin); no test asserts the old broken behavior.

**Defect found & fixed (rule 7):** a real navigate-away mid-turn let the browser's teardown fetch rejection (TypeError, not AbortError) leak into the failed funnel, persisting a phantom failed brain record — `test_sources_midstream_bug.py::test_no_orphan_brain_message_when_navigated_before_first_token` failed (2 `.msg` after reload) and violated the phase-20 navigate-away convention. Fixed: turn-scoped `leftThePage` flag (set unconditionally on `pagehide`, reset in `runTurn`) skips the funnel in the catch-else branch; pinned by new unit test `test_navigate_away_is_not_a_failed_turn`. No phase-overview/PLAN/todo/complete files touched; no commits made.

**Gates (exact):**
- `uv run pytest` → 2577 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 4271 stmts, 99% (>90%)
- `uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov` → 4 passed (isolated)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- Regression E2E, isolated: `test_sources_midstream_bug.py` 6/6 (was 5/6); `test_llm_retry`/`test_tool_scaffolding_guardrails`/`test_stop_generation`/`test_navbar_refresh` 17/17

**Completion criteria:** (1) network error → banner + in-bubble Retry, re-ask without re-typing ✅ (E2E A); (2) refresh restores failed bubble + working Retry, no "new chat" ✅ (E2E C); (3) stopped/successful turns byte-identical ✅ (negative E2E, stop suite, byte-identity units); (4) pytest/coverage/lint/types ✅; (5) commit + phase move — left to the harness per pass rules.

**Notable:** deviation = the regression fix above (a navigation is not a failed turn; phase-20 partial-persist convention restored). Next pending phase: `121_git_source_tokens`.
This commit is contained in:
2026-09-24 18:50:36 -04:00
parent 0ff1f8c4d6
commit 3a0fc3db05
29 changed files with 2298 additions and 28 deletions
+112
View File
@@ -521,6 +521,36 @@ test_llm_retry.py``). The mock is single-conversation per e2e server, so
strings are this suite's own folder names, so no other E2E can hit
them (they seed different trees).
Failure injection (phase 120, failed-turn retry, TODO.md L3–4) —
deterministic failures for the failed-turn E2E suite
(``tests/e2e/test_failed_turn_retry.py``), reusing the phase-67
counter machinery:
- user message containing ``fail first turn``
(``FAIL_FIRST_TURN_TRIGGER``): EVERY app-level attempt of the FIRST
turn responds 500 — the full forced-default budget (conftest pins
``BOR_LLM_RETRIES`` to the code default 3 → 4 app-level attempts =
12 POSTs, ``FAIL_FIRST_TURN_DEAD_ATTEMPTS`` ×
``_HTTPS_PER_DEAD_ATTEMPT``) — the retry-budget exhaustion with
ZERO frames (the terminal ``error`` before any delta) — and the
SECOND turn's first request streams the normal composed answer (the
banner/in-bubble Retry's re-ask). The sequence resets after that
success, so a second question carrying the trigger re-drives the
failure from zero (the phase-67 convention).
- user message containing ``partial then fail``
(``PARTIAL_FAIL_TRIGGER``): the FIRST matching streaming request
sends ``PARTIAL_FAIL_TEXT`` in normal 12-char ``delta.content``
chunks and then the generator RAISES — a genuine mid-stream
connection reset (the body ends without ``finish_reason`` /
``[DONE]``; a mid-stream failure never re-POSTs — the body was
already flowing). The app's ``chat_stream_retried`` re-raises the
wrapped ``LLMError`` (a piece already emitted — the locked
retry-before-first-frame rule) and the chat endpoint settles the
turn with the terminal SSE ``error`` frame AFTER the partial
deltas (the partial-then-error wire: the partial bubble keeps its
text + the error note + the Retry). The SECOND request streams the
normal composed answer (the retry's re-ask) and resets the
sequence.
``max_tokens`` is honored deterministically (token ≈ whitespace word),
like a real endpoint: an answer longer than the cap is truncated. This
is what makes the phase-11 truncation regression observable.
@@ -750,6 +780,46 @@ ALWAYS_FAIL_TRIGGER = "always fail"
#: bag-of-words vector — the endpoint's pre-stream embedding retry loop.
EMBED_FAIL_TRIGGER = "embed fail once"
# ---------------------------------------------------------------------------
# Phase 120 (failed-turn retry, TODO.md L3–4): deterministic failure
# injections for the failed-turn E2E suite (tests/e2e/
# test_failed_turn_retry.py) — see the module docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) dies for
#: the ENTIRE first turn — every app-level attempt 500s (the
#: retry-budget exhaustion, the zero-frame terminal ``error``) — and
#: streams the normal composed answer from the SECOND turn on (the
#: banner/in-bubble Retry's re-ask, phase 111/120). The dead window is
#: exactly ONE turn: ``FAIL_FIRST_TURN_DEAD_ATTEMPTS`` = the e2e
#: forced-default budget (conftest pins ``BOR_LLM_RETRIES`` to the code
#: default 3 → 4 app-level attempts = 12 POSTs at
#: ``_HTTPS_PER_DEAD_ATTEMPT``). Each sequence resets after the success
#: it guards, so a second question carrying the trigger re-drives the
#: failure from zero (the phase-67 ``_fail_posts`` convention).
FAIL_FIRST_TURN_TRIGGER = "fail first turn"
#: One full turn under the forced-default budget (3 retries → 4
#: attempts): the whole first turn dies, the second turn's first
#: attempt streams (the Retry's re-ask).
FAIL_FIRST_TURN_DEAD_ATTEMPTS = 4
#: A user message containing this substring (case-insensitive) streams
#: ``PARTIAL_FAIL_TEXT`` as ordinary ``delta.content`` chunks on its
#: FIRST matching request and then the mock's generator RAISES (a
#: mid-stream connection reset — the SSE body ends without
#: ``finish_reason``/``[DONE]``; a mid-stream failure never re-POSTs).
#: The app's ``chat_stream_retried`` re-raises the wrapped ``LLMError``
#: (a piece already emitted — the locked retry-before-first-frame rule)
#: and the chat endpoint settles with the terminal SSE ``error`` frame
#: AFTER the partial deltas. The SECOND request streams the normal
#: composed answer (the retry's re-ask) and resets the sequence.
PARTIAL_FAIL_TRIGGER = "partial then fail"
#: Matching requests that partial-then-die (one POST — see above).
PARTIAL_FAIL_DEAD_REQUESTS = 1
#: The deterministic partial answer (byte-stable — the E2E asserts the
#: bubble KEEPS exactly this text, with the error note appended).
PARTIAL_FAIL_TEXT = "Here is the start of the answer that never finished."
# ---------------------------------------------------------------------------
# Phase 71 (tool-scaffolding guardrails, 2026-09-03 incident):
# deterministic raw-markup flows — see the module docstring
@@ -987,6 +1057,32 @@ def _chat_dead(key: str, dead_attempts: int) -> bool:
return _bump_fail(key) <= dead_attempts * _HTTPS_PER_DEAD_ATTEMPT
def _partial_fail_stream() -> Any:
"""The phase-120 partial-then-fail stream (``PARTIAL_FAIL_TRIGGER``):
``PARTIAL_FAIL_TEXT`` in 12-char ``delta.content`` chunks (the
mock's default cadence — NO ``finish_reason``, NO ``[DONE]``), then
a RAISE that ends the SSE body abruptly (a connection reset
mid-stream — the client SDK sees a truncated body, the app wraps it
in ``LLMError`` after the pieces already emitted, and the chat
endpoint's terminal ``error`` frame lands AFTER the partial deltas
on the client's stream)."""
model = "turbo"
chunk_id = f"chatcmpl-{uuid.uuid4()}"
for piece in re.findall(r".{1,12}", PARTIAL_FAIL_TEXT, re.S):
payload = {
"id": chunk_id,
"object": "chat.completion.chunk",
"created": int(time.time()),
"model": model,
"choices": [
{"index": 0, "delta": {"content": piece}, "finish_reason": None}
],
}
yield f"data: {json_dumps(payload)}\n\n"
time.sleep(0.02)
raise RuntimeError("e2e mid-stream failure (phase 120 partial-then-fail)")
#: The agent's ``read`` tool-result prefix (app.rag.agent
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
_READ_RESULT_PREFIX = "Document "
@@ -2471,6 +2567,22 @@ def chat_completions(body: dict[str, Any]) -> Any:
if _chat_dead(RETRY_TRIGGER, RETRY_DEAD_ATTEMPTS):
return _llm_500(RETRY_TRIGGER)
_fail_posts[RETRY_TRIGGER] = 0 # the answer streamed — restart
# Phase 120 (failed-turn retry): the failed-turn suite's two
# injections — the whole-first-turn death (the zero-frame
# exhaustion whose Retry's re-ask the mock then answers) and the
# partial-then-die (the partial deltas + terminal error frame).
if FAIL_FIRST_TURN_TRIGGER in user_lower:
if _chat_dead(FAIL_FIRST_TURN_TRIGGER, FAIL_FIRST_TURN_DEAD_ATTEMPTS):
return _llm_500(FAIL_FIRST_TURN_TRIGGER)
_fail_posts[FAIL_FIRST_TURN_TRIGGER] = 0 # the answer streamed — restart
if PARTIAL_FAIL_TRIGGER in user_lower:
if _bump_fail(PARTIAL_FAIL_TRIGGER) <= PARTIAL_FAIL_DEAD_REQUESTS:
return StreamingResponse(
_partial_fail_stream(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
_fail_posts[PARTIAL_FAIL_TRIGGER] = 0 # the answer streamed — restart
# Phase 71 (tool-scaffolding guardrails): the deterministic raw-
# markup flow — checked BEFORE the search/tool marker flows (the
# trigger is independent of the ``<tools>`` marker, so both