feat(rag): retry a failed LLM request before the first token lands — BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY with a live 'retrying' status

app/rag/agent.py

@@ -43,16 +43,27 @@ task 04):
    the assistant tool-call message + the tool result (refusals included),
    consumes one round, and the model is called again. At the round cap —
    ``max_rounds = settings.agent_max_rounds`` (``BOR_AGENT_MAX_ROUNDS``,
-   default 10) — the loop forces one final ``chat_stream(messages,
-   tools=None)`` and returns: the cap is the **only** forced exit
-   (besides "the stream carried no calls"), and it bounds pathological
-   rejected-call streams.
+   default 10) — the loop forces one final retried no-tools request
+   (``chat_stream_retried`` with ``tools=None``) and returns: the cap is
+   the **only** forced exit (besides "the stream carried no calls"), and
+   it bounds pathological rejected-call streams.
 5. A rare stream that carries both content and a tool call keeps the
    content (it was already emitted) **and** still runs the tool.
 6. *holder* (an :class:`AgentHolder`) records the read documents and the
    number of executed tool calls (re-lists included); the API layer
    (task 04) reads it after the stream to extend ``done.sources`` /
    ``query_log.sources`` and the per-turn log line (``tool_calls=N``).
+7. Retries (phase 67, owner-locked A2): every model request — each tool
+   round and the forced final ``tools=None`` call — goes through
+   ``chat_stream_retried``: a round that dies before its first piece is
+   restarted with the SAME messages (up to ``settings.llm_retries``
+   restarts, a flat ``settings.llm_retry_delay`` between attempts, each
+   preceded by a :class:`app.rag.llm.RetryPiece` the API layer turns into
+   an SSE ``retry`` frame); a round that already streamed a piece fails
+   the turn as before (no partial answer is ever redone). Retries are
+   invisible to the round cap: a round that needed a retry still consumes
+   exactly one round. With ``settings.llm_retries=0`` every request is a
+   single plain attempt (the pre-phase-67 path).
 
 The DB accessors (:func:`list_catalog`, :func:`find_document`) are
 module-level functions so unit tests can monkeypatch them without a
@@ -71,7 +82,13 @@ from sqlalchemy.orm import Session
 
 from app.config import Settings
 from app.models import Document
-from app.rag.llm import LLMClient, StreamPiece, ToolCallPiece
+from app.rag.llm import (
+    LLMClient,
+    RetryPiece,
+    StreamPiece,
+    ToolCallPiece,
+    chat_stream_retried,
+)
 
 logger = logging.getLogger("app.agent")
 
@@ -224,13 +241,20 @@ async def run_agent(
     seed_docs: Sequence[Document],
     settings: Settings,
     holder: AgentHolder,
-) -> AsyncIterator[StreamPiece | ToolCallPiece]:
+) -> AsyncIterator[StreamPiece | ToolCallPiece | RetryPiece]:
     """Run the grounded-turn tool loop, yielding every stream piece.
 
-    Every piece (``thinking`` / ``content`` / tool calls) is yielded as it
-    arrives; the API layer (task 04) turns tool-call pieces into SSE
-    ``tool`` events. After the loop finishes, *holder* carries the read
-    documents and the executed tool-call count (re-lists included).
+    Every piece (``thinking`` / ``content`` / tool calls /
+    :class:`RetryPiece`) is yielded as it arrives; the API layer (task 04)
+    turns tool-call pieces into SSE ``tool`` events and retry pieces into
+    SSE ``retry`` events. After the loop finishes, *holder* carries the
+    read documents and the executed tool-call count (re-lists included).
+
+    Retries (phase 67, owner-locked A2): every model request goes through
+    :func:`chat_stream_retried` — a failed round is retried **before** its
+    first piece (same messages, ``settings.llm_retries`` restarts, a flat
+    ``settings.llm_retry_delay``); a round that already streamed pieces
+    fails the turn as before.
 
     ``seed_docs`` are the documents the retrieval already put in context
     (they shape the *system_prompt* the caller built); re-reading one of
@@ -252,10 +276,22 @@ async def run_agent(
         calls: list[ToolCallPiece] = []
         # Phase 48: bind the round's stream so a consumer abandon
         # (GeneratorExit into the yield below) tears down the in-flight
-        # model stream deterministically — not GC-dependent. Awaiting
-        # ``aclose()`` in the ``finally`` is safe because it does not
-        # yield; on a fully consumed round it is a quiet no-op.
-        stream = llm.chat_stream(cast("list[dict[str, str]]", messages), tools=tools)
+        # model stream deterministically — not GC-dependent. Phase 67:
+        # the round goes through the retry primitive — a failure before
+        # the first piece restarts the request (locked A2) after a
+        # RetryPiece; closing the OUTER generator propagates GeneratorExit
+        # into ``chat_stream_retried``, whose own ``finally`` closes the
+        # in-flight inner ``chat_stream``, so teardown stays deterministic
+        # on consumer abandon. Awaiting ``aclose()`` in the ``finally`` is
+        # safe because it does not yield; on a fully consumed round it is
+        # a quiet no-op.
+        stream = chat_stream_retried(
+            llm,
+            cast("list[dict[str, str]]", messages),
+            tools=tools,
+            retries=settings.llm_retries,
+            delay=settings.llm_retry_delay,
+        )
         try:
             async for piece in stream:
                 if isinstance(piece, ToolCallPiece):
@@ -300,8 +336,16 @@ async def run_agent(
             )
             # Phase 48: the forced final answer gets the same explicit
             # teardown as the loop rounds (consumer abandon mid-final
-            # answer must still close the model's stream).
-            final = llm.chat_stream(cast("list[dict[str, str]]", messages), tools=None)
+            # answer must still close the model's stream). Phase 67: the
+            # forced call retries under the same locked-A2 rule as the
+            # loop rounds.
+            final = chat_stream_retried(
+                llm,
+                cast("list[dict[str, str]]", messages),
+                tools=None,
+                retries=settings.llm_retries,
+                delay=settings.llm_retry_delay,
+            )
             try:
                 async for piece in final:
                     yield piece