phase: 96_oneshot_resilience
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s

All checks complete. Final report:

**Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design)

- `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff
- `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe
- `.env.example` comments updated (chat-turn stream + one-shot summary calls)

**Test / lint / coverage results**
- `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%)
- `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated)
- Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files).

**Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
This commit is contained in:
2026-09-11 13:16:20 -04:00
parent bcaef800c5
commit a49be80b8e
42 changed files with 2893 additions and 143 deletions
+96 -17
View File
@@ -207,6 +207,23 @@ class _TooLarge(RuntimeError):
"""Internal: the endpoint rejected the request's input size."""
class _EmptyContentError(RuntimeError):
"""Internal: the one-shot reply parsed but carries no usable content.
Raised by :meth:`LLMClient._chat_once` for the retryable failure
class (phase 96: ``content`` None or whitespace). Carries the
reply's ``finish_reason`` (``None`` when the provider omits it) so
:meth:`LLMClient.chat` can log the greppable retry line. Never
escapes the client — ``chat()`` converts it to the public
:class:`LLMError` (D3). Not an :class:`LLMError` on purpose: callers
catching :class:`LLMError` must only ever see final, public errors.
"""
def __init__(self, message: str, finish_reason: str | None) -> None:
super().__init__(message)
self.finish_reason = finish_reason
class LLMClient:
"""Thin async wrapper over the aipi OpenAI-compatible API."""
@@ -332,24 +349,19 @@ class LLMClient:
(vec,) = await self.embed([text])
return vec
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
"""One-shot (non-streaming) completion (A5 extended, phase 30).
async def _chat_once(self, messages: list[dict[str, Any]], model: str) -> str:
"""One non-streaming completion attempt (phase 96, private).
Short, low-temperature request (``temperature=0.2``, 2048-token
cap — summaries and outlines are small, so a fixed budget is
enough) against ``BOR_LLM_SUMMARY_MODEL`` (default ``lite``)
unless *model* names another. Used by the document summarizer
(phase 30) and the KB overview generator (phase 31).
Any transport/HTTP/malformed failure, a choiceless reply, or an
empty/missing ``content`` field raises :class:`LLMError` — a
silent empty summary must never be stored.
The single-attempt body of :meth:`chat` — the transport wrap
(→ :class:`LLMError` with the sanitized base URL), the choiceless
check, and the empty-content check. Raises
:class:`_EmptyContentError` (carrying the reply's
``finish_reason``) for an empty reply — the one failure class
:meth:`chat` retries — and :class:`LLMError` for everything else.
"""
try:
resp = await self._client.chat.completions.create(
model=model or self.settings.llm_summary_model,
model=model,
messages=cast("list[ChatCompletionMessageParam]", messages),
temperature=0.2,
max_tokens=2048,
@@ -368,14 +380,81 @@ class LLMClient:
f"chat completion from {sanitize_error(self.settings.llm_base_url)} "
"returned no choices"
)
content = resp.choices[0].message.content
choice = resp.choices[0]
content = choice.message.content
if content is None or not content.strip():
raise LLMError(
raise _EmptyContentError(
f"chat completion from {sanitize_error(self.settings.llm_base_url)} "
"returned empty content — refusing to store a silent summary"
"returned empty content — refusing to store a silent summary",
choice.finish_reason,
)
return content.strip()
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
"""One-shot (non-streaming) completion (A5 extended, phase 30).
Short, low-temperature request (``temperature=0.2``, 2048-token
cap — summaries and outlines are small, so a fixed budget is
enough) against ``BOR_LLM_SUMMARY_MODEL`` (default ``lite``)
unless *model* names another. Used by the document summarizer
(phase 30), the KB overview generator (phase 31), folder
summaries (phase 94), and the pre-sync probe
(``check_models``, phase 41).
Any transport/HTTP/malformed failure, a choiceless reply, or an
empty/missing ``content`` field raises :class:`LLMError` — a
silent empty summary must never be stored.
Empty-reply retry (phase 96, house LLM-retry policy D1–D3): the
empty-content failure class — the model answered but said
nothing (``content`` None or whitespace) — is retried up to
``settings.llm_retries`` times (``BOR_LLM_RETRIES``, default 3;
``0`` = off) with a flat ``settings.llm_retry_delay``
(``BOR_LLM_RETRY_DELAY``) between attempts, one WARNING per
retry naming the model, the reply's ``finish_reason`` (``None``
when the provider omits it), and the attempt count. After
``1 + llm_retries`` empty attempts it raises
:class:`LLMError` naming the attempts; with ``llm_retries=0``
the single-attempt message stays byte-identical to the
pre-phase-96 behavior. NO other failure retries at the app level:
a choiceless reply and transport failures raise immediately (the
openai SDK's own ``max_retries=2`` already re-POSTs wire-level
failures — an app-level transport retry would stack on top of
it).
"""
total = 1 + self.settings.llm_retries
chosen = model or self.settings.llm_summary_model
for attempt in range(1, total + 1):
try:
return await self._chat_once(messages, chosen)
except _EmptyContentError as e:
if attempt >= total:
# Retries exhausted — refuse to store the silent
# summary. total == 1 (BOR_LLM_RETRIES=0 kill switch)
# keeps the pre-phase-96 message byte-identical.
if total == 1:
raise LLMError(str(e)) from e
raise LLMError(
f"chat completion from "
f"{sanitize_error(self.settings.llm_base_url)} "
f"returned empty content on all {total} attempts — "
"refusing to store a silent summary"
) from e
logger.warning(
"one-shot LLM reply returned empty content (model=%s, "
"finish_reason=%s, attempt %d/%d) — retrying in %.1fs",
chosen,
e.finish_reason,
attempt,
total,
self.settings.llm_retry_delay,
)
# Flat delay — the phase-67 convention, no backoff.
await asyncio.sleep(self.settings.llm_retry_delay)
raise AssertionError("unreachable: every attempt returned or raised")
async def chat_stream(
self,
messages: list[dict[str, Any]],