phase: 96_oneshot_resilience
All checks complete. Final report: **Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design) - `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff - `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe - `.env.example` comments updated (chat-turn stream + one-shot summary calls) **Test / lint / coverage results** - `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%) - `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated) - Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files). **Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
This commit is contained in:
+96
-17
@@ -207,6 +207,23 @@ class _TooLarge(RuntimeError):
|
||||
"""Internal: the endpoint rejected the request's input size."""
|
||||
|
||||
|
||||
class _EmptyContentError(RuntimeError):
|
||||
"""Internal: the one-shot reply parsed but carries no usable content.
|
||||
|
||||
Raised by :meth:`LLMClient._chat_once` for the retryable failure
|
||||
class (phase 96: ``content`` None or whitespace). Carries the
|
||||
reply's ``finish_reason`` (``None`` when the provider omits it) so
|
||||
:meth:`LLMClient.chat` can log the greppable retry line. Never
|
||||
escapes the client — ``chat()`` converts it to the public
|
||||
:class:`LLMError` (D3). Not an :class:`LLMError` on purpose: callers
|
||||
catching :class:`LLMError` must only ever see final, public errors.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, finish_reason: str | None) -> None:
|
||||
super().__init__(message)
|
||||
self.finish_reason = finish_reason
|
||||
|
||||
|
||||
class LLMClient:
|
||||
"""Thin async wrapper over the aipi OpenAI-compatible API."""
|
||||
|
||||
@@ -332,24 +349,19 @@ class LLMClient:
|
||||
(vec,) = await self.embed([text])
|
||||
return vec
|
||||
|
||||
async def chat(
|
||||
self, messages: list[dict[str, Any]], model: str | None = None
|
||||
) -> str:
|
||||
"""One-shot (non-streaming) completion (A5 extended, phase 30).
|
||||
async def _chat_once(self, messages: list[dict[str, Any]], model: str) -> str:
|
||||
"""One non-streaming completion attempt (phase 96, private).
|
||||
|
||||
Short, low-temperature request (``temperature=0.2``, 2048-token
|
||||
cap — summaries and outlines are small, so a fixed budget is
|
||||
enough) against ``BOR_LLM_SUMMARY_MODEL`` (default ``lite``)
|
||||
unless *model* names another. Used by the document summarizer
|
||||
(phase 30) and the KB overview generator (phase 31).
|
||||
|
||||
Any transport/HTTP/malformed failure, a choiceless reply, or an
|
||||
empty/missing ``content`` field raises :class:`LLMError` — a
|
||||
silent empty summary must never be stored.
|
||||
The single-attempt body of :meth:`chat` — the transport wrap
|
||||
(→ :class:`LLMError` with the sanitized base URL), the choiceless
|
||||
check, and the empty-content check. Raises
|
||||
:class:`_EmptyContentError` (carrying the reply's
|
||||
``finish_reason``) for an empty reply — the one failure class
|
||||
:meth:`chat` retries — and :class:`LLMError` for everything else.
|
||||
"""
|
||||
try:
|
||||
resp = await self._client.chat.completions.create(
|
||||
model=model or self.settings.llm_summary_model,
|
||||
model=model,
|
||||
messages=cast("list[ChatCompletionMessageParam]", messages),
|
||||
temperature=0.2,
|
||||
max_tokens=2048,
|
||||
@@ -368,14 +380,81 @@ class LLMClient:
|
||||
f"chat completion from {sanitize_error(self.settings.llm_base_url)} "
|
||||
"returned no choices"
|
||||
)
|
||||
content = resp.choices[0].message.content
|
||||
choice = resp.choices[0]
|
||||
content = choice.message.content
|
||||
if content is None or not content.strip():
|
||||
raise LLMError(
|
||||
raise _EmptyContentError(
|
||||
f"chat completion from {sanitize_error(self.settings.llm_base_url)} "
|
||||
"returned empty content — refusing to store a silent summary"
|
||||
"returned empty content — refusing to store a silent summary",
|
||||
choice.finish_reason,
|
||||
)
|
||||
return content.strip()
|
||||
|
||||
async def chat(
|
||||
self, messages: list[dict[str, Any]], model: str | None = None
|
||||
) -> str:
|
||||
"""One-shot (non-streaming) completion (A5 extended, phase 30).
|
||||
|
||||
Short, low-temperature request (``temperature=0.2``, 2048-token
|
||||
cap — summaries and outlines are small, so a fixed budget is
|
||||
enough) against ``BOR_LLM_SUMMARY_MODEL`` (default ``lite``)
|
||||
unless *model* names another. Used by the document summarizer
|
||||
(phase 30), the KB overview generator (phase 31), folder
|
||||
summaries (phase 94), and the pre-sync probe
|
||||
(``check_models``, phase 41).
|
||||
|
||||
Any transport/HTTP/malformed failure, a choiceless reply, or an
|
||||
empty/missing ``content`` field raises :class:`LLMError` — a
|
||||
silent empty summary must never be stored.
|
||||
|
||||
Empty-reply retry (phase 96, house LLM-retry policy D1–D3): the
|
||||
empty-content failure class — the model answered but said
|
||||
nothing (``content`` None or whitespace) — is retried up to
|
||||
``settings.llm_retries`` times (``BOR_LLM_RETRIES``, default 3;
|
||||
``0`` = off) with a flat ``settings.llm_retry_delay``
|
||||
(``BOR_LLM_RETRY_DELAY``) between attempts, one WARNING per
|
||||
retry naming the model, the reply's ``finish_reason`` (``None``
|
||||
when the provider omits it), and the attempt count. After
|
||||
``1 + llm_retries`` empty attempts it raises
|
||||
:class:`LLMError` naming the attempts; with ``llm_retries=0``
|
||||
the single-attempt message stays byte-identical to the
|
||||
pre-phase-96 behavior. NO other failure retries at the app level:
|
||||
a choiceless reply and transport failures raise immediately (the
|
||||
openai SDK's own ``max_retries=2`` already re-POSTs wire-level
|
||||
failures — an app-level transport retry would stack on top of
|
||||
it).
|
||||
"""
|
||||
total = 1 + self.settings.llm_retries
|
||||
chosen = model or self.settings.llm_summary_model
|
||||
for attempt in range(1, total + 1):
|
||||
try:
|
||||
return await self._chat_once(messages, chosen)
|
||||
except _EmptyContentError as e:
|
||||
if attempt >= total:
|
||||
# Retries exhausted — refuse to store the silent
|
||||
# summary. total == 1 (BOR_LLM_RETRIES=0 kill switch)
|
||||
# keeps the pre-phase-96 message byte-identical.
|
||||
if total == 1:
|
||||
raise LLMError(str(e)) from e
|
||||
raise LLMError(
|
||||
f"chat completion from "
|
||||
f"{sanitize_error(self.settings.llm_base_url)} "
|
||||
f"returned empty content on all {total} attempts — "
|
||||
"refusing to store a silent summary"
|
||||
) from e
|
||||
logger.warning(
|
||||
"one-shot LLM reply returned empty content (model=%s, "
|
||||
"finish_reason=%s, attempt %d/%d) — retrying in %.1fs",
|
||||
chosen,
|
||||
e.finish_reason,
|
||||
attempt,
|
||||
total,
|
||||
self.settings.llm_retry_delay,
|
||||
)
|
||||
# Flat delay — the phase-67 convention, no backoff.
|
||||
await asyncio.sleep(self.settings.llm_retry_delay)
|
||||
raise AssertionError("unreachable: every attempt returned or raised")
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
|
||||
Reference in New Issue
Block a user